InfraLens - Transformer Annotated Labs

Overview

Transformer Annotated Code Reading Labs

Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.

Goals

How to use these labs

Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.

Read this after the Transformer Systems handbook and, when useful, the parent AI Infra systems map. Then open the lab page and starter file. For each line, ask what concept it encodes and how you would explain it in an interview.

Related practice: Transformer interview practice.

Handbook links

Handbook-to-lab crosslink table

#	Lab	Related handbook section	Starter
01	Tokenization and Embedding Lookup	Tokenization and Embeddings	tokenizer_embedding.py
02	Q/K/V Shape Walkthrough	Q/K/V and Attention Shapes	qkv_shapes.py
03	Causal Mask and Attention Scores	Decoder-only Transformer Block	causal_mask.py
04	Multi-Head Attention Minimal Code	Multi-Head Attention	multi_head_attention.py
05	RoPE Mental Model	Positional Encoding / RoPE	rope_reading.py
06	FFN / SwiGLU Parameter Count	MLP / FFN / SwiGLU	ffn_swiglu_params.py
07	Training Loop and Cross Entropy Loss	Training Loop and Loss	training_loop.py
08	KV Cache Step-by-Step	KV Cache and Autoregressive Inference	kv_cache.py
09	Attention Backend Comparison Reading	Attention Backends: SDPA / FlashAttention / PagedAttention	attention_backends.md
10	Quantization Reading Lab	Quantization and Inference Memory	quantization_plan.md

Annotated reading material

Starter code directory

Every starter under labs/code/ is a compact reading artifact with teaching comments. Treat it as source of truth for the code preview.