Calculate token work, GPU reuse, cache size and time held for the same request.

A service starts to accept longer reasoning traces. Answers stay short, but requests stay active for more time. The key-value (KV) cache fills, and interactive traffic waits. A faster matmul may not let any more requests start. A larger cache pool lets more requests start, but then each decode step reads more cached keys and values.

This article uses one hypothetical request to find LLM inference bottlenecks. It examines the computation that the request uses, its GPU memory and its memory occupancy time. These quantities give KV-cache bytes for each token and request capacity in 8 GiB. They show the bottleneck after an optimization.

The previous articles explain the machinery. The kernel and serving articles use the same dimensions.

1. Start with one hypothetical request

A token starts as an ID, becomes a hidden state and reads permitted previous context. It supplies information for next-token logits. Each generated token changes the work of the next step.

Examine the request in four connected views. The token view gives the necessary computation. The GPU view shows how hardware does the computation. The memory view shows the KV cache kept between steps. The request view shows when that cache gets service. They are views of one task, not stages you can make better in isolation.

For this hypothetical example, one replica uses a dense decoder with these properties:

KiB, MiB and GiB are powers of two. Rates use decimal prefixes. BF16 is bfloat16. Generated tokens include any reasoning and control tokens kept in the cache.

2. Token rows set the cost of each step

Use the query projection $A_{M\times4096}B_{4096\times4096}$. $M$ is the number of token rows, one for each token in this step. A prompt chunk supplies many rows. Ordinary decode adds one row for each active request.

Count two FLOPs for each multiply-add. In this optimistic model, each input is read one time and the output is written one time:

$F$ is the number of floating-point operations (FLOPs), and $D$ is the number of bytes moved. Here $K=N=4096$. Arithmetic intensity (AI) is FLOPs for each byte moved.

Use an illustrative compute ceiling $P=200$ TFLOP/s and high-bandwidth memory (HBM) bandwidth $B=1$ TB/s. The Roofline ridge point is $P/B=200$ FLOPs/byte.

A kernel is memory-bound when its intensity is less than this value. It is compute-bound when its intensity is more than this value. The optimistic minimum time is $\max(F/P,D/B)$.

At $M=16$, arithmetic intensity is about 15.9 FLOPs/byte, and memory traffic gives a 33.8 μs minimum. At $M=256$, it is 227.6 FLOPs/byte, and arithmetic gives a 42.9 μs minimum. More rows use more time but do more token work for each weight load. The two estimates do not include the rest of the layer, attention, kernel launches, communication and imperfect utilization.

Token rows and active requests are separate budgets. A 256-token prefill chunk uses projection weights again without admission of 256 requests. Admission lets a request start only when free KV-cache blocks are available.

SARATHI puts prefill and decode tokens into the same linear operations. This mixed batching can help when cache capacity limits decode batch size. An online latency check is still necessary for the selected chunk size.

FlashAttention does not write the full score and probability matrices to HBM. FlashAttention-2 makes work partitioning and execution better. The matmul and FlashAttention article explains the two methods. They decrease execution cost, not the KV cache that stays after each step.

3. Stored history limits concurrent requests

The KV cache in Fig. 3 contains keys and values for each kept position in each layer. For ordinary full attention, the KV-cache size formula gives:

This request holds 512 MiB after prefill and 576 MiB at length 4608. An 8 GiB pool has capacity for at most 14 such full-length unshared histories (no prefix caching between requests). This is an equal-length bound; alignment, block slack, temporary memory and shared prefixes change actual use.

Multi-head, multi-query and grouped-query attention (MHA/MQA/GQA) change $H_{\mathrm{kv}}$, the number of K/V heads. With other dimensions fixed, 32 K/V heads use 512 KiB/token, and one head uses 16 KiB/token. Quality evaluation is necessary for these architecture choices. They are not runtime switches that preserve the checkpoint's output distribution.

Paging and prefix caching in vLLM change how the cache is allocated and shared. A filled token entry still uses 128 KiB here. With unshared 16-token blocks, the last block wastes less than 2 MiB. A shared prefix prevents duplicate history storage when requests have the same model and full prefix identity: the tokens plus any other cache-key inputs.

Attention kernels and cache allocation change different memory costs. Decreased temporary attention reads and writes to HBM do not remove the stored KV cache. Paging decreases wasted reservations, but decode still reads each filled entry. FlashAttention makes each step cheaper. With paging, the pool can hold more requests.

4. Count how long each request holds cache

Cache size alone does not show how long a request holds memory while other requests wait.

For this example only, decode adds one token at intervals of $\Delta t=0.04$ seconds, with no pauses and immediate cleanup. Do not include prefill. Use continuous growth. For $N_g$ generated tokens, the time held is $W=N_g\Delta t$. The cache's byte-seconds (bytes held multiplied by seconds) are

The first term is the prompt's cache. The second is the growing suffix. Discrete steps give slightly different values. A measured occupancy trace is necessary for variable step times, eviction, sharing or compression.

Eight times the generation holds cache eight times as long and, because the suffix increases, uses about 11.3 times the byte-seconds. This shows the relation between reasoning length and hidden reasoning tokens and capacity.

For stable admitted traffic, Little's law gives average active requests as arrival rate times average time held: $\bar R=\lambda E[W]$, with $\lambda$ in admissions for each second. At a hypothetical $\lambda=0.02$ requests/s, the two rows give averages of 0.4096 and 3.2768 active decode requests. For all active decode requests, the total size of unshared cache has an average of $\lambda E[A_{\mathrm{KV}}]$: 0.2176 and 2.4576 GiB. These values do not include prefill and bursts. If offered load is more than capacity, there is no stable average: queue lengths increase or admission changes.

Longer output increases generated work, peak KV cache memory and time held. This decreases serving capacity. Final-answer length or peak cache alone does not show the full cost.

5. Compare the methods in one table

The table shows each method’s changes:

Quality checks are different. The objective of exact attention and exact speculative sampling is the same computation or output distribution, apart from numerical effects. GQA and reasoning controls change the computation or the model's behavior. Outcome tests and speed tests are necessary.

6. Follow the bottleneck after each change

In this equal-length example, 16 ordinary decode requests give a projection about 15.9 FLOPs/byte. Their final unshared cache is 9 GiB. Whatever the compute capacity, the 8 GiB pool cannot hold them unless stored state or admission changes.

Apply the changes in this sequence:

1. Make allocation better. Paging can decrease unused reservations. It cannot hold 9 GiB of filled unshared KV cache in 8 GiB. To increase this example's 14-request limit, use sharing, fewer stored bytes, shorter histories or more memory.
2. Use a shared prefix again. In this hypothetical case, 14 requests share the same 4096-token prefix and each keeps its own 512-token tail. Their ideal filled cache is $512+14\times64=1408$ MiB. Sharing removes this capacity limit, but the skipped prefill also removes rows for mixed batching. Measure execution again.
3. Supply a prefill chunk. With available prompt work after a cache miss, chunking can make projection reuse better with few extra active requests. It also puts more work into one step, which can delay an interactive token.
4. Decrease target steps. Speculative decoding decreases completion and cache occupancy time if accepted tokens save more than draft and verification cost. Draft memory can use all the capacity just recovered. A product of isolated speedups does not include this capacity cost.
5. Use more reasoning. If 4096 generated tokens are necessary for the task, each equal-length request’s final unshared cache is 1 GiB. The unshared capacity limit decreases to eight. The longer trace must give a result that passes the grader and is worth that work. A short visible answer alone does not show this.

More memory, faster attention and larger batches each change a different limit.

7. Select the next measurement

Select measurements that show differences between explanations. For a slow first useful token, measure queue time, uncached prompt work and hidden tokens before visible output. For later tokens, measure scheduler gaps, batch rows, step time and KV-cache reads. For capacity, measure filled and reserved bytes, sharing and occupancy over time.

Use the symptom and previous change to select the first view. It is not necessary to start with model cost.

Keep arrivals, lengths, prefix repetition, checkpoint, sampling and latency target fixed. Examine correctness when behavior changes. Give throughput at that quality and latency target, rather than unconstrained tokens for each second. The infrastructure essay applies this test to cost for each successful task.

Show the four views together:

The arrows make a loop because each generated token adds context, cache and a scheduling decision. Predict which resource a change affects. Calculate how much limit it removes. Then measure the next bottleneck.

Key takeaways

• More token rows use each weight load again. Each active request holds its own KV cache. Use different budgets for rows and requests.
• Attention architecture, FlashAttention kernels, paging and prefix reuse each change a different cost.
• Cache byte-seconds show the relation between generated length, time held and average cache use.
• Different quality checks are necessary for exact execution improvements and reasoning controls.
• Examine an optimization after its bottleneck changes, under the same task and service target.

Series navigation

• Series index: The Life of a Token Across the LLM Stack
• Part 0: LLM Inference Is a Full-Stack Systems Problem
• Part 1: The Life of a Token Inside a Transformer
• Part 2: Why Transformer Performance Is Mostly a Matmul Problem
• Part 3: The Life of a Request Inside vLLM
• Part 4: A Token's Journey Across the LLM Stack (current)
• Part 5: Reasoning Effort Is an Inference-Time Systems Knob

@article{ji2026tokenjourneyacrossllmstack,
  title = {A Token's Journey Across the LLM Stack},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {June},
  url = {https://fusheng-ji.github.io/blog/posts/token-journey-across-llm-stack/}
}

References

• Aleksa Gordic, Inside the Transformer: The Life of a Token.
• Aleksa Gordic, Inside NVIDIA GPUs: Anatomy of high performance matmul kernels.
• Aleksa Gordic, Inside vLLM: Anatomy of a High-Throughput LLM Inference System.
• vLLM Project, Architecture Overview.
• Vaswani et al., Attention Is All You Need.
• Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding.
• Agrawal et al., SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.
• Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.
• Williams, Waterman, and Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures.

Note: This blog was drafted and polished with assistance from ChatGPT and uses Aleksa Gordic's tutorials and the cited primary papers. The workload is hypothetical.

Updated 03 November 2026