LLM Inference
How LLMs run at inference time: tokens, attention, KV cache and decoding. 8 notes.
-
Same Skeleton, Different Boundaries: How Open-Source AI Inference Frameworks Are Built
A source-level comparison of vLLM, SGLang, TensorRT-LLM, LMDeploy, vLLM-Omni, SGLang-Omni, verl, slime, OpenRLHF and AReaL at November 2026 releases.
-
Reasoning Effort Is an Inference-Time Systems Knob
Reasoning effort can change internal work, latency, cost and cache occupancy. Compare tasks before you select low, medium or high.
The Life of a Token Across the LLM Stack · Part 5
-
AI Infrastructure Is Not Plumbing
AI infrastructure can change numerical results, inference cost and reliability. Examine kernels, memory, scheduling and research checks together.
-
A Token's Journey Across the LLM Stack
Find LLM inference bottlenecks with one worked example. Calculate KV-cache bytes for each token and request capacity in 8 GiB. Examine why long outputs keep cache occupied.
The Life of a Token Across the LLM Stack · Part 4
-
The Life of a Request Inside vLLM
This article gives engineers an introduction to vLLM serving. It compares batching, cache paging, prefix reuse and speculative decoding.
The Life of a Token Across the LLM Stack · Part 3
-
Why Transformer Performance Is Mostly a Matmul Problem
Most dense Transformer arithmetic is matmul. GPU matmul speed depends on tiling and data movement. Small decode batches are frequently memory-bound.
The Life of a Token Across the LLM Stack · Part 2
-
The Life of a Token Inside a Transformer
A Transformer maps token IDs to next-token logits. This article examines embeddings, attention, MLPs, RMSNorm, RoPE, GQA and the KV cache.
The Life of a Token Across the LLM Stack · Part 1
-
LLM Inference Is a Full-Stack Systems Problem
A slow LLM response can have different causes. This article examines prefill, decode, GPU kernels, KV-cache memory and request scheduling.
The Life of a Token Across the LLM Stack · Overview
See all tags.