High-throughput serving combines Transformer execution, scheduling and cache management for an online service.

A model calculates next-token logits from input tensors and weights. The logits have one score for each vocabulary entry. An inference engine has different questions at intervals of a few milliseconds. Which requests run next? How should the engine batch them? Where should it keep their KV cache?

A forward pass runs the input through all model layers. This is the service's primary expensive computation. The service also accepts and tokenizes requests. It does these three jobs while it keeps the GPU busy:

• It schedules prefill (prompt processing) and decode (answer generation, one token at a time).
• It allocates and releases KV-cache blocks. They keep attention keys and values from previous tokens, so decode does not recompute them.
• It samples and streams tokens, then completes each request.

vLLM is an open-source engine for large language model (LLM) serving. Its design is for high throughput. It tracks each request's state and cache blocks. This lets it select the next computation, the requests that wait and the memory location for each request.

High-throughput LLM serving and one fast request are different goals. Requests of different lengths compete for GPU compute, KV-cache memory and batch capacity. Good kernels can still give bad product latency.

This article follows one request through a vLLM-style engine. It examines prefill, decode, continuous batching, chunked prefill, PagedAttention, prefix caching and speculative decoding. Each feature saves a resource and has a cost. The resource numbers are from the series' hypothetical model.

1. From model execution to a serving loop

The engine has these three distinct parts:

• The request state: prompt tokens, generated tokens, sampling parameters, status and output stream
• The model executor: the component that runs model computation on one or more GPUs
• The scheduler and memory manager: logic for request selection and KV-cache locations

A vLLM-style engine runs a serving loop. The scheduler selects work, and the KV cache manager allocates cache blocks. The model executor runs the selected batch. Sampling gives token IDs from logits. The engine returns new tokens to the caller. Unfinished requests enter the loop again until each request is completed.

The unit of control changes. A library call runs one prompt to completion. The engine runs the next step that its scheduler can select for the current active requests.

Each request keeps state between steps. A waiting request may have text and sampling parameters but no cache yet. A running request may hold cache blocks during the wait for its next decode step. With chunked prefill enabled, a request may have only part of its prompt processed. For a completed request, cleanup is still necessary: output delivery, stop-condition handling and cache release.

2. Prefill and decode scheduling

An autoregressive request generates one token at a time. It uses each new token as input to the next step. Its two phases put different loads on the system.

Prefill processes the prompt. For a 2,000-token prompt, the model makes hidden states (one vector for each position) and KV cache for those 2,000 positions. The model can process prompt tokens together. This gives prefill more parallelism.

Decode generates new tokens. At each step, the model usually processes one new token for each active request and uses attention to cached keys and values. This phase repeats many times and reads the KV cache at each step.

Prefill is often compute-heavy. Decode is often memory-bound: weight and KV-cache reads from GPU memory cause more delay than arithmetic. The balance changes with model architecture, batch shape, context length, quantization, attention implementation and hardware. Thus, "compute-heavy" and "memory-bound" are common workload labels. They are not fixed bottlenecks for each phase.

The two phases compete for the same GPU. A long prefill can delay other requests' first tokens. Each decode step is smaller, but hundreds or thousands of these steps may be necessary for a request. The engine must interleave the phases and prevent starvation of either phase.

Two user-facing metrics correspond to the phases. Time to first token (TTFT) is the wait for the first output token. The engine samples that token from the logits of the prefill pass. Inter-token latency (ITL) is the gap between subsequent tokens.

Chunked prefill splits prefill for a long prompt into chunks. Other requests' decode steps can run between chunks. Each prefill interruption is shorter. This can decrease ITL for requests that already stream output. But the user can receive the long prompt's first token later. Measure TTFT, ITL and throughput (completed requests or tokens for each second) separately.

Chunk size has a tradeoff. Smaller chunks can make fairness better and decrease decode starvation. But each step then does less large, efficient prefill work. Larger chunks can increase throughput, but they can block interactive decode. The setting depends on prompt-length distribution, latency targets and cache pressure (proximity to a KV-cache state with no free blocks).

SARATHI (Sections 4 and 5.4) gives chunking a second purpose. Its mixed batching puts one prefill chunk and current decode tokens in the same weight matmuls. The two share each weight load. Attention still runs separately.

Smaller chunks split a prompt into more steps. This gives decode more chances to share those loads. But smaller chunks can decrease matmul efficiency. Each chunk also reads the KV cache of the previous chunks again.

The paper studies throughput. It leaves online fairness and queueing policy as open questions.

In each weight matmul (a projection), each token in the step is one row of the input matrix. This example uses the hypothetical Part 2 model. A projection over 16 decode rows has an arithmetic intensity of about 15.9 floating-point operations for each byte moved (FLOPs/byte). A prefill chunk can add hundreds of rows in one step. This increases reuse of each loaded weight. For decode alone, hundreds more active requests would be necessary to add the same rows.

In mixed batching, rows set compute reuse and each active request holds its own KV cache. For this reason, the scheduler tracks rows and requests as separate budgets.

Continuous batching rebuilds the batch at each iteration

Traditional batching works when a fixed set of examples starts together and is completed together. Online LLM traffic is irregular. The server receives requests at different times, with different prompt and output lengths. Some requests wait for prefill, some run decode, and some are in a stopped state.

Continuous batching rebuilds the batch at each scheduler iteration. At each iteration, the engine can do these actions:

• It can admit waiting requests if capacity is available (these requests start to run).
• It can continue decode steps for active requests.
• It can process prefill chunks.
• It can remove completed requests.
• It can reclaim KV cache.

Continuous batching can increase throughput. With runnable work and cache capacity available, the GPU takes new work. It does not wait until the longest request in a fixed batch is completed.

The cost is a more complex scheduler. Rules are necessary for these decisions:

• The quantity of prefill to admit
• The decode priority
• The memory limits
• The prevention of extreme tail latency (the slowest requests, for example the 99th percentile)

This pseudocode gives the approximate structure of one scheduler iteration:

inspect waiting and running requests
estimate token and cache budgets
choose prefill chunks and decode steps
allocate or reuse KV-cache blocks
run the model executor
sample and stream tokens
update request state
release finished cache

In Fig. 2, B is completed in iteration 3 and exits. A continues decode at that time. C enters in iteration 4, after A started.

3. KV-cache locations: paged blocks

The KV cache is primarily a Transformer optimization. During autoregressive generation, previous keys and values never change. The cache keeps them, so recomputation is not necessary. Part 1 shows why decode can reuse cached keys and values.

In a serving system, the cache becomes a memory-management problem. The engine must do these tasks:

• It must allocate cache for each request.
• It must address the cache during attention.
• It must release the cache at request completion.
• It must reuse the cache when prefixes match.
• It must decide the action when memory is full.

A serving estimate starts with bytes for each cached token. This quantity depends on layer count $L$, key/value head count $H_{\mathrm{kv}}$ and head dimension $d_{\mathrm{head}}$. The dtype sets the bytes for each number $b$. The parallelism strategy sometimes matters too. Multiply by each request's sequence length $S_r$. Then add the values for all active requests.

Long contexts and many concurrent requests can make KV cache dominate GPU memory (VRAM). The memory can hold a model at load time, but the service can still fail with concurrent long-context traffic.

The factor of 2 counts keys and values. This is an estimate of the total cache footprint for active requests:

The formula is a capacity estimate that assumes ordinary cached keys and values and excludes alignment, block rounding, metadata, temporary buffers, prefix sharing, cache compression and tensor/pipeline-parallel effects. Use it to show memory scaling. Then measure the deployment's actual cache manager.

The cache manager is like a memory allocator. It tracks free blocks, used blocks and mappings from request positions to physical locations. It also works with the scheduler. The scheduler can admit a waiting request only if cache capacity is available for its prompt or expected generation.

PagedAttention maps logical tokens to physical blocks

PagedAttention, from Kwon et al.'s paper, made vLLM known. It divides KV cache into fixed-size blocks, not one contiguous allocation for each request. PagedAttention changes cache allocation and addressing. The model's learned attention rule stays the same.

The analogy is virtual memory. A process has a logical address space. The operating system maps its pages to physical memory frames. In vLLM, a request has a logical token history, and the engine maps it to KV-cache blocks. KV-cache blocks are in GPU memory, with a serving-engine design for attention kernels.

Paging can decrease waste. A request that increases token by token gets new blocks as necessary. It does not reserve a large maximum-context region at the start. At completion, the engine returns the request's blocks to the pool. With variable request lengths, block allocation can decrease fragmentation and let more concurrent requests run.

Attention kernels use a block table (or an equivalent mapping) to read the non-contiguous cache. It maps logical token blocks to physical KV-cache blocks:

$B_r$ is the block table for request $r$, $p$ is a logical token position, and $P$ is the number of tokens for each cache block.

The cost is indirection. The block table is necessary for kernels to find the memory location for a logical sequence's cache. That lookup must be fast.

For this estimate, the cache is unshared, with no prefix sharing. Its block size is $P$ tokens. A sequence of length $S$ allocates $\lceil S/P\rceil P$ token slots. At most $P-1$ slots in the last block are unused.

In the series' hypothetical 128 KiB-per-token model, this unused capacity with $P=16$ is less than 2 MiB for each request. Paging decreases reserved but unused space. The bytes written for each filled token stay the same.

The PagedAttention paper's block-size ablation shows the tradeoff. Very small blocks decrease kernel utilization. Large blocks waste more space and decrease sharing. Use the block sizes that your engine supports. Use measurements of your own workload.

4. Prefix caching and work reuse

Serving workloads often repeat prompt prefixes. A chat application may reuse the same system prompt. A coding assistant may include repository instructions in repeated requests. A retrieval pipeline may share boilerplate formatting.

Prefix caching keeps KV cache for a shared prompt prefix. Subsequent requests can reuse it. If two requests start with the same tokens and the blocks are still in memory, the second may skip prefix recomputation.

Reuse changes the cost model. A long prompt costs less if much of it is reusable. Rules are also necessary for caching. The engine must identify prefixes, store cache entries, handle cache pressure and select entries for eviction.

The cited vLLM docs give a hash-based method that caches only full blocks. A cache key identifies the prefix and the inputs necessary to reuse the saved computation. A compatible model, compatible cache context and blocks still in memory are necessary for reuse. After a weight update, cached prefixes are also invalid.

Prefix caching gives a benefit only when the workload repeats. Tool schemas and multi-turn conversations that keep previous context repeat too. The benefit is smaller when prompts are mostly unique or cache pressure forces frequent eviction. Prefix caching saves computation but uses memory and bookkeeping.

The scheduler must also account for cache hits. For a request with a hit, less prefill may be necessary than its raw prompt length suggests.

For prompt length $S$ and reusable cached-prefix length $S_{\mathrm{hit}}$, the remaining prefill work is closer to:

5. Serial-step reduction with speculative decoding

Autoregressive decoding is sequential. Each token waits for the previous token. Speculative decoding decreases the cost of that dependency. A low-cost draft (a small model or another proposer) suggests several tokens. The target model is the model that the engine serves. It verifies all draft tokens in one forward pass.

The exact speculative sampling algorithm (Section 2.3 and Appendix A.1) keeps the target model's output distribution unchanged.

At one prefix, $p(x)$ is the target token distribution and $q(x)$ is the draft distribution. These distributions include the effects of temperature, top-p and other sampling settings. Verification accepts a proposal $x\sim q$ with probability $\min(1,p(x)/q(x))$. On rejection, it samples from the normalized positive residual $[p-q]_+$. Acceptance contributes probability mass $\min(p(x),q(x))$ for token $x$, and the residual contributes the rest. Together they recover $p(x)$.

The guarantee covers the output distribution. It does not promise identical text with a shared random seed. Greedy decoding is a different special case. The target keeps a draft token only if it equals the target's own top token.

Approximate acceptance rules, inconsistent sampling settings and output constraints that the verifier does not apply each need their own correctness check. The engine must drop the rejected token and all subsequent draft tokens from the target's KV cache and the draft's state. It must do this before it continues.

If the target accepts several draft tokens, the service adds several tokens to the sequence with fewer expensive target-model iterations. Speculative decoding adds scheduling and state for the engine. The engine must coordinate draft generation, target verification, cache updates and sampling correctness.

Speculative decoding can increase throughput or decrease latency. The gain depends on draft quality, target-model cost, acceptance rate, implementation overhead and the workload. Compare time for each accepted token. Add the time for draft work, target verification and bookkeeping. Divide it by the accepted-token count.

A target batch that already uses compute efficiently may gain less than a sparse, memory-bandwidth-limited decode batch. Extra draft weights and state can also decrease admission capacity. Acceptance rate alone cannot select a deployment.

6. Serving with more than one GPU

If one GPU cannot hold the model weights, the executor distributes computation across GPUs. Tensor parallelism splits each matrix operation across GPUs. Pipeline parallelism gives a range of layers to each GPU or GPU group. Data parallelism runs full model copies to serve more traffic.

vLLM hides much of this in its engine and executor processes. The process layout changes with version and deployment mode (local, multiprocess GPU or larger data-parallel). This can occur although the model API looks the same.

Model shape still matters. Hidden size, layer count and vocabulary projection affect the weight split across GPUs. Kernels still matter too. Communication and computation must be arranged to prevent unnecessary GPU waits. The serving layer adds admission, routing and load balancing.

7. Compare the features on the same workload

The four features save different resources:

The features interact. A prefix hit can remove prefill rows that a mixed batch would otherwise use. The scheduler must react to the smaller batch. Speculation can decrease the time that a request stays active, but it adds draft memory. Which resource limits the workload after the proposed change? This is the deciding question.

A serving-system evaluation uses these criteria:

• Time to first token
• Inter-token latency
• Request throughput
• Token throughput
• Tail latency
• GPU utilization
• KV-cache memory efficiency
• Cost for each generated token
• Stability under bursty traffic

Benchmarks must represent workload shape. Short prompts and short outputs may hide long-context problems. A benchmark with only prefill may miss decode bottlenecks. Fixed batches may overstate online-serving performance.

Batch limits, prefill chunk size, parallelism, cache block size and decoding options depend on the model and hardware. They also depend on sequence lengths, request mix and latency targets. This makes auto-tuning important.

I recommend that serving-benchmark authors report these measurement conditions:

• The prompt length distribution
• The output length distribution
• The request arrival pattern
• The concurrency level
• The time-to-first-token target
• The inter-token latency target
• The cache reuse assumptions
• The hardware and parallelism setup

Without these conditions, a tokens-per-second number can mislead. The number may be for a throughput-oriented batch workload, not interactive chat.

Request flow in an inference engine

Serving in the full stack

The Transformer defines the computation. Matmul kernels set the efficiency of GPU execution. vLLM-style serving schedules real traffic for that computation with memory limits. The serving layer often decides product performance. It cannot override model and hardware limits.

A token flows through layers. A request flows through queues, cache blocks, executor calls and streaming outputs. At the request level, LLM inference is an online system with scarce compute, scarce memory and irregular demand. Users experience queueing, TTFT, ITL, streaming behavior and tail latency under load.

Throughput and latency are coupled. A batch with more work can raise GPU utilization, but it can also increase waiting time. The engine makes these tradeoffs continuously.

Key takeaways

• The model executor calculates tensors. The engine selects the request state for the next computation.
• Prefill is prompt-wide setup work. Decode repeats cache reads.
• Continuous batching continues work as the server receives requests and as requests are completed and change length.
• KV cache is GPU memory that the engine must allocate, address, reuse and release.
• PagedAttention, prefix caching and speculative decoding help specific workloads and add scheduling state.

Series navigation

• Series index: The Life of a Token Across the LLM Stack
• Part 0: LLM Inference Is a Full-Stack Systems Problem
• Part 1: The Life of a Token Inside a Transformer
• Part 2: Why Transformer Performance Is Mostly a Matmul Problem
• Part 3: The Life of a Request Inside vLLM (current)
• Part 4: A Token's Journey Across the LLM Stack
• Part 5: Reasoning Effort Is an Inference-Time Systems Knob

@article{ji2026lifeofarequestvllm,
  title = {The Life of a Request Inside vLLM},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {June},
  url = {https://fusheng-ji.github.io/blog/posts/life-of-a-request-vllm/}
}

References

• Aleksa Gordic, Inside vLLM: Anatomy of a High-Throughput LLM Inference System.
• vLLM Project, Paged Attention.
• vLLM Project, Automatic Prefix Caching.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding.
• Agrawal et al., SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.

Note: This blog was drafted and polished with assistance from ChatGPT. Its sources include Aleksa Gordic's tutorials, engine documentation and the cited primary papers. Resource examples are hypothetical. Paper mechanisms are distinct from current engine policy.

Updated 08 November 2026