InfraLens

Focus on execution speed and scaling: the Transformer turned sequence modeling from a sequential recurrent dependency into a parallelizable matrix multiplication problem, enabling the training of models with billions or trillions of parameters on massive datasets.

Thinking attention is the only important part. In practice, the Feed-Forward Network (FFN) layers contain the majority of the model's parameters and act as a key-value storage for factual knowledge, while residual connections and LayerNorm/RMSNorm are what make deep training stable.

Vocabulary size represents a memory-compute tradeoff: a larger vocabulary V decreases the sequence length S for the same text (reducing attention complexity), but increases the embedding parameters (V * D), which dominates the weight footprint of smaller models.

Assuming that a token always equals a word. Many technical words, code keywords, or foreign characters are split into multiple subwords or individual characters by the tokenizer, which changes the effective sequence length processed by the attention blocks.

Explain that Pre-LN (normalization on the input branch before sub-layers) is critical for scaling. Unlike Post-LN which normalizes the main residual path and requires a warm-up phase to avoid gradient explosion, Pre-LN allows stable training from initialization.

Assuming that decoder-only blocks are less capable of processing input prompts because they mask future tokens. The causal mask only prevents future tokens from affecting past positions, which aligns perfectly with how autoregressive generation is computed during serving.

Walk through the shapes at each step. Emphasize that the scaling factor 1/sqrt(Dh) is crucial: as the head dimension Dh grows, dot-products grow in magnitude, which would push softmax into flat regions with near-zero gradients without scaling.

Thinking Q, K, and V are separate source arrays. They are linear transformations of the exact same input representation X. In encoder-decoder cross-attention, Q is projected from the decoder, while K and V come from the encoder's output.

Explain the serving trade-offs: MQA uses 1 KV head for all Q heads, which saves cache memory but can hurt representation capacity. GQA groups query heads to share a smaller subset of KV heads (e.g. 8 Q heads per KV head), offering a sweet spot in memory and performance.

Thinking multi-head attention increases computational complexity compared to a hypothetical single head of dimension D. For a fixed dimension D, splitting it into H heads preserves the total operations, though layout permutations add minor overhead.

Explain that RoPE is superior to absolute positional embeddings because it encodes relative distances directly in the attention computation. This allows length extrapolation techniques (like NTK-aware scaling) to work by modifying the base frequency theta.

Thinking RoPE is added to the word embeddings at the bottom of the network. Unlike absolute encodings, RoPE is applied dynamically to the Queries and Keys at every single self-attention layer, keeping positional information intact as representation flows through deep blocks.

Explain the dimension math: SwiGLU requires about 1.5x more parameter capacity than standard FFN for a hidden dimension H. To keep total parameter count and FLOPs equivalent to a standard FFN (where H = 4D), the SwiGLU hidden dimension is typically scaled down to ~8/3 D.

Believing the attention layers store most of the model's factual knowledge. In large models, FFN blocks make up roughly 2/3 of the total parameters. They act as key-value memory banks, while attention operates as the router selecting which memories to read.

Highlight the computational benefit of RMSNorm: by removing the mean computation step, RMSNorm saves memory bandwidth and kernel execution time (avoiding two passes over the data) while achieving identical convergence quality as standard LayerNorm.

Confusing LayerNorm with BatchNorm. BatchNorm normalizes features across the batch dimension, making token representations dependent on other batch items. LayerNorm normalizes independently per token, preserving causal isolation during text generation.

Explain that teacher forcing speeds up training but creates "exposure bias" because the model never learns to correct its own errors during training. During inference, it relies on its own generated tokens, where small errors can compound over steps.

Thinking training runs sequentially. Because the entire ground-truth sequence is available during training, we compute predictions and loss for all token positions concurrently in a single forward pass, which is highly efficient.

Discuss the KV Cache memory footprint calculations. As batch size and sequence length grow, the cache size can grow to tens of gigabytes, making KV Cache the primary limiting factor for batch size and serving density.

Thinking the KV Cache reduces memory usage. It actually exchanges GPU memory footprint (storing gigabytes of key/value states) to save massive amounts of compute and HBM read/write cycles during generation.

FlashAttention is an exact mathematical implementation of standard attention, not an approximation. It is an IO-aware algorithm that reduces memory access overhead by keeping intermediate scores in fast SRAM and recomputing them in the backward pass.

Assuming PagedAttention speeds up individual kernel executions. PagedAttention is a system-level memory management technique that allows serving systems to allocate cache in page blocks, eliminating external fragmentation and enabling higher batch concurrency.

Explain the execution difference: weight-only quantization accelerates the memory-bandwidth-bound decode phase by reducing weight retrieval volume from HBM. Weight-activation quantization accelerates the compute-bound prefill phase by utilizing fast INT8 tensor math.

Assuming quantization always improves latency. If the batch size is small, weight-only quantization requires dequantization to float16 at runtime, which adds compute overhead. The speedup only registers if weight retrieval is the dominant bottleneck.