Lab 01 - Tokenization and Embedding Lookup

Overview

Lab 01: Tokenization and Embedding Lookup

Annotated code reading lab. Running code is optional.

Related handbook section

Tokenization and Embedding Lookup

This lab maps directly to the handbook section. Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.

Open handbook section Open starter file

Concept Goal

Tokenization and Embedding Lookup

Map raw text to token ids, then to embedding rows. Notice that model code sees integer ids, not words.

Mental Model

Mechanism to keep in mind

`vocab` records the contract between string pieces and ids.
`token_ids` is the compact sequence the model receives.
`embedding_table[id]` is a row lookup; it creates the first `(S,D)` tensor.

Annotated Code Preview

Starter preview

Excerpt from code/lab-01-tokenization-embedding/tokenizer_embedding.py. The linked starter file is the source of truth.

Open starter file

# Tokenization and Embedding Lookup
# Annotated reading material. Running this file is optional.
# Source-of-truth focus: Map raw text to token ids, then to embedding rows. Notice that model code sees integer ids, not words.

text = "attention cache"
vocab = {"<bos>": 0, "attention": 1, "cache": 2}
token_ids = [vocab["<bos>"], vocab["attention"], vocab["cache"]]
# Each id selects one learned row; the output shape becomes (seq_len, hidden_dim).
embedding_table_shape = (len(vocab), 8)
embedding_output_shape = (len(token_ids), embedding_table_shape[1])

# What to explain while reading:
# - vocab records the contract between string pieces and ids.
# - token_ids is the compact sequence the model receives.
# - embedding_table[id] is a row lookup; it creates the first (S,D) tensor.
#
# Common traps:
# - Token is not the same as word.
# - Embedding lookup is not where context mixing happens.

Line-by-line Explanation

What each block is doing

Setup / contract: `vocab` records the contract between string pieces and ids.
Main transition: `token_ids` is the compact sequence the model receives.
Interview hook: `embedding_table[id]` is a row lookup; it creates the first `(S,D)` tensor.

What to Notice

Reading checkpoints

Token boundaries are model-specific.
Embedding lookup changes representation, not sequence length.
Unknown/special tokens are part of the API contract.

Common Misunderstandings

What this lab prevents

Token is not the same as word.
Embedding lookup is not where context mixing happens.

Interview Explanation

How to say it out loud

Map raw text to token ids, then to embedding rows. Notice that model code sees integer ids, not words. Then explain the code by naming the state being transformed, the axis or shape that matters, and the tradeoff that would appear in a real system.

External intuition notes

Additional intuition

Use official docs and papers for API behavior and factual claims; use blogs only to improve the mental picture.
If support matrices, performance behavior or backend choices are version-sensitive, check current docs before repeating them.
A strong interview answer names the state object, the shape or axis it changes, and the tradeoff it creates.

InfraLens