CUDA and GPU kernels: attention, matmul, quantization and numerics. 1 note.
Most dense Transformer arithmetic is matmul. GPU matmul speed depends on tiling and data movement. Small decode batches are frequently memory-bound.
The Life of a Token Across the LLM Stack · Part 2
02 Jun 2026 · 15 min read
See all tags.