Model Serving
Serving engines and runtimes: requests, caches, scheduling and failure handling. 3 notes.
-
Same Skeleton, Different Boundaries: How Open-Source AI Inference Frameworks Are Built
A source-level comparison of vLLM, SGLang, TensorRT-LLM, LMDeploy, vLLM-Omni, SGLang-Omni, verl, slime, OpenRLHF and AReaL at November 2026 releases.
-
AI Infrastructure Is Not Plumbing
AI infrastructure can change numerical results, inference cost and reliability. Examine kernels, memory, scheduling and research checks together.
-
The Life of a Request Inside vLLM
This article gives engineers an introduction to vLLM serving. It compares batching, cache paging, prefix reuse and speculative decoding.
The Life of a Token Across the LLM Stack · Part 3
See all tags.