To understand modern LLM inference, examine the model, GPU kernels and request serving.

When a streamed response slows down, “is the model slow?” is too broad a question for diagnosis. The application programming interface (API) still looks like text in, text out. In the server, the call becomes token IDs, temporary activation tensors, GPU launches, cached attention history and scheduled work.

A long prompt can delay the first token. A small decode batch can underuse tensor cores, the GPU’s matrix arithmetic units. Key/value (KV) cache pressure can delay a new request’s start. It can pause a running request (preemption) and cause later cache recomputation, or reject work despite available compute. A scheduler can keep the GPU busy while an interactive user waits.

This post follows one request through three views: the model, the GPU kernels and the serving engine. The tokenizer maps text to token IDs. The Transformer maps them to contextual vectors and next-token scores, or logits. GPU kernels run those calculations. The serving engine selects requests and batches, and manages their cached state.

These three views help connect a symptom to the next measurement: model cost, kernel efficiency, cache pressure or scheduling delay.

LLM inference is the coordination of token computation, GPU execution, and request scheduling.

Track token count, GPU batch shape, live cache blocks and requests that wait for the next scheduler step. One diagnostic view of request state at step $t$ is

This vector groups the possible bottlenecks. Engines split, overlap or omit its parts according to their executor, cache manager and protocol.

One request

Follow the computation, memory and scheduling work for one ordinary request.

Suppose a user sends this short chat prompt:

Explain why KV cache matters for LLM inference.

The server maps this text to token IDs and request metadata: sampling parameters, maximum output length, stop conditions, priority and arrival time. Chat formatting may add system, assistant or tool tokens before tokenization. The model reads that formatted sequence, not the raw message visible in the user interface (UI).

During prefill, the model reads all prompt tokens together and makes their KV cache. Each layer calculates hidden vectors. Attention projects them into queries, keys and values. A multilayer perceptron (MLP) transforms each token independently. The last prompt position gives next-token logits, from which the sampler selects the first output token.

After prefill, GPU time is necessary for each request step. The request keeps KV-cache memory between steps.

Matrix multiplications (matmuls) use most of that GPU time: the query/key/value (QKV) and MLP projections. Their GPU kernels move weights and activation tiles through high-bandwidth memory (HBM), caches, shared memory and registers to tensor cores. A poor data path wastes theoretical floating-point operations per second (FLOP/s).

Dozens or thousands of requests may compete. Some still process prompts. Others generate tokens. Some of these requests have long contexts, and some are near completion. The scheduler selects requests for the next batch. It decides whether to split a long prefill, how many tokens this step’s budget permits and whether KV blocks are free.

After the sampler selects the first output token, the request enters decode, which writes the answer one token at a time. Each forward pass processes only the newest position and reads previous keys and values from KV cache. Decode does less arithmetic for each step than prefill. But each output token requires another cache read and scheduling decision.

For diagnosis, split total latency into queueing, prefill, repeated decode and streaming:

The terms name causes of delay. Serialization, networking, sampling and scheduling can overlap GPU execution. APIs also account for streaming differently. Queueing plus prefill sets the time to first token (TTFT). The decode steps set the inter-token latency (ITL), the gap between streamed tokens. Measure the relevant component before you call the response slow.

A single prompt on a warm GPU measures one clean path. Production traffic also combines prompt/output lengths, cache pressure, batch shape, kernel efficiency and scheduling. For this reason, the same model can benchmark quickly but feel slow in a product.

Changes from each optimization

I group the papers by the changes they make in a request. This keeps an implementation improvement separate from a model change. Multi-head, multi-query and grouped-query attention (MHA, MQA, GQA) have different arrangements for query heads to share key/value heads.

The last row asks two different questions. Speculative decoding tries to deliver a target model’s generation more efficiently. Reasoning effort changes the work that the model spends to solve a task. Each needs its own quality check.

I track three quantities: work required, resources occupied, and time occupied. The Transformer sets the work and the state that each request keeps. Kernels set the cost to run a batch. The engine selects requests to share a batch and decides the time that state stays in memory. Optimize the part that limits the workload you actually serve.

View 1: Model computation

The model view asks which computations are necessary for the request.

A dense decoder-only Transformer alternates per-token operations with attention, where tokens read other positions. These operations have different functions.

The MLP changes the information in one token vector. Attention gives the token information from permitted positions.

The MLP expands each token vector, applies a nonlinearity or gate, then projects it to its previous size. Each position uses the same learned weights.

Attention compares queries with keys, applies masks and combines values with the resulting weights. In a causal model, a token reads its position and previous positions. The mask prevents reads from future positions. When several documents share one sequence, segmentation masks prevent attention between documents. Long-context models may combine local and global attention to reduce cost and retain distant access.

Inference has these two phases:

• Prefill processes many prompt tokens at one time and makes KV cache.
• Decode processes one or a few new tokens for each request and reads the existing KV cache.

Those phases use the same model weights, but they stress the system differently.

View 2: GPU kernels

The kernel view asks whether the accelerator can execute those calculations efficiently.

Query/key/value, MLP and vocabulary projections are matmuls. Attention also includes matrix products. Optimized attention kernels, for example FlashAttention, do not write the full attention matrix to memory.

The model specifies the matrices to multiply. The GPU implementation determines the quantity of available throughput that the multiplication uses.

Tensor cores help only when data arrives on time in the required layout. Much of matmul optimization arranges memory movement around those arithmetic units.

GPU storage ranges from global memory through L2 and L1 caches to shared memory and registers. Reuse of nearby data costs less than repeated global reads. Tiling splits a matmul into blocks that fit in shared memory and registers. Good kernels reuse those blocks and overlap transfers with computation. This keeps tensor cores busy.

The equation for a linear layer becomes work scheduled across warps (groups of GPU threads), registers, shared memory and tensor cores.

View 3: Request serving

The serving view asks how requests share hardware and meet latency goals.

Engines, for example vLLM, select forward passes to run, batch requests, assign KV cache and stream output. A model executor runs one forward pass. The engine decides which passes run and when.

Production traffic is irregular. One user sends a short prompt and asks for a paragraph. Another sends a long document and asks one question. Many requests have the same system prompt, which prefix caching can reuse.

Some requests are in prefill, others in decode, and some are near completion. Some need structured-output constraints or may benefit from speculative decoding.

The engine uses those differences to make scheduling decisions.

Continuous batching rebuilds the batch at each step. It does not wait for a fixed batch to finish. The engine admits new requests, continues decode for active requests, retires completed requests and schedules prefill or decode work. This matches online traffic. Requests of different lengths arrive all the time in online traffic.

Continuous batching alone cannot guarantee high utilization or low latency. Long prefill, cache pressure, small decode batches and poor admission choices can still idle capacity or delay streaming requests.

At this layer, KV cache is memory that the engine must allocate, address, reuse and release. PagedAttention maps logical token blocks to physical cache blocks, much like virtual memory. It reduces waste and fragmentation compared with a reservation of one large contiguous region for each request.

Scheduling also affects fairness. A preference for GPU utilization can delay short interactive requests. A preference for latency can reduce throughput. Too much prefill can delay requests that already stream. These choices change the product’s responsiveness, but the model’s quality stays the same.

Suppose two requests arrive together. One has a 200-token prompt and asks for 50 output tokens. The other has a 20,000-token prompt and asks for one sentence. The long prompt may dominate prefill compute and cache despite its short output.

Execution of that prefill as one job can delay the short request. A split into chunks (chunked prefill) may let the short request stream sooner, at the cost of more scheduling steps. Select according to the product goal.

Why an optimization can move the bottleneck

An optimization can expose the next bottleneck. Fewer weight bytes may make KV reads dominant. Prefix reuse shortens setup and keeps shared blocks resident. More requests can improve matmul efficiency and increase each user’s gap between decode steps. An essay on why an optimization often moves the bottleneck gives the same point for TTFT, ITL and cost per task.

Test those effects against recorded prompt and output lengths, arrivals, cache reuse, hardware and latency targets. Follow the resource that the optimization changes.

The synthesis article examines one hypothetical dense GQA workload. It calculates cache capacity, the arithmetic each projection does for each byte moved (arithmetic intensity), and the time that requests hold memory. It then asks which constraint each method removes and which one remains. Parts 1–3 explain the machinery necessary to read that calculation.

The full stack in one diagram

The loop shows these operations: generated tokens return to decode, active sequences extend KV cache, and the scheduler examines requests at each iteration.

Why I follow one token

I follow one token because it connects all three views. In the model, an ID becomes a vector and then logits. On the GPU, kernels run those transformations. In the server, the token belongs to a request that holds cache and waits for scheduling. Prompt tokens fill the cache, generated tokens extend it, and completion releases the memory.

Follow that token to help locate a delay. A component diagram shows separate boxes, but the token crosses all of them.

The series continues in five parts:

1. Inside the Transformer: A token becomes a vector and then logits.
2. Inside GPU matmul: Matrix multiplication and memory movement dominate.
3. Inside vLLM-style serving: vLLM schedules requests around prefill, decode and KV cache.
4. Across the full stack: The article follows token work, compute, memory and request lifetime.
5. Inside reasoning effort: Hidden generated work changes quality, latency, cost and capacity.

To explain a slow next token, measure architecture cost, kernel utilization, cache pressure and batching policy. Then measure their interactions. The symptom alone cannot identify the failing component.

Key takeaways

• I recommend that readers diagnose a slow token across model architecture, GPU kernels, KV-cache memory and request scheduling.
• The Transformer specifies the work. Kernels determine the fraction of hardware capacity that this work can use.
• KV cache makes attention history a capacity and allocation problem.
• Serving policy shapes time to first token (TTFT), inter-token latency (ITL), throughput and tail behavior.
• Give the name of the limiting layer before you select an optimization.

Series navigation

• Series index: The Life of a Token Across the LLM Stack
• Part 0: LLM Inference Is a Full-Stack Systems Problem (current)
• Part 1: The Life of a Token Inside a Transformer
• Part 2: Why Transformer Performance Is Mostly a Matmul Problem
• Part 3: The Life of a Request Inside vLLM
• Part 4: A Token's Journey Across the LLM Stack
• Part 5: Reasoning Effort Is an Inference-Time Systems Knob

@article{ji2026llminferencefullstacksystems,
  title = {LLM Inference Is a Full-Stack Systems Problem},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {May},
  url = {https://fusheng-ji.github.io/blog/posts/llm-inference-full-stack-systems/}
}

References

• Aleksa Gordic, Inside the Transformer: The Life of a Token.
• Aleksa Gordic, Inside NVIDIA GPUs: Anatomy of high performance matmul kernels.
• Aleksa Gordic, Inside vLLM: Anatomy of a High-Throughput LLM Inference System.
• Vaswani et al., Attention Is All You Need.
• Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding.
• Agrawal et al., SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.
• Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.

Note: This blog was drafted and polished with assistance from ChatGPT. Its sources include Aleksa Gordic's tutorials and the cited primary papers. The cross-layer framework is an explanatory synthesis.

Updated 03 November 2026