Series

The Life of a Token Across the LLM Stack

This series has six articles about one request from prompt to completed answer. The articles examine Transformer math, GPU kernels, KV cache, request scheduling, performance diagnosis and reasoning effort.

Transformer math GPU matmul kernels KV-cache memory Request scheduling Performance diagnosis Reasoning effort
  1. Overview

    LLM Inference Is a Full-Stack Systems Problem

    A slow LLM response can have different causes. This article examines prefill, decode, GPU kernels, KV-cache memory and request scheduling.

  2. Part 1

    The Life of a Token Inside a Transformer

    A Transformer maps token IDs to next-token logits. This article examines embeddings, attention, MLPs, RMSNorm, RoPE, GQA and the KV cache.

  3. Part 2

    Why Transformer Performance Is Mostly a Matmul Problem

    Most dense Transformer arithmetic is matmul. GPU matmul speed depends on tiling and data movement. Small decode batches are frequently memory-bound.

  4. Part 3

    The Life of a Request Inside vLLM

    This article gives engineers an introduction to vLLM serving. It compares batching, cache paging, prefix reuse and speculative decoding.

  5. Part 4

    A Token's Journey Across the LLM Stack

    Find LLM inference bottlenecks with one worked example. Calculate KV-cache bytes for each token and request capacity in 8 GiB. Examine why long outputs keep cache occupied.

  6. Part 5

    Reasoning Effort Is an Inference-Time Systems Knob

    Reasoning effort can change internal work, latency, cost and cache occupancy. Compare tasks before you select low, medium or high.