A checkpoint loads. It answers a prompt. Its offline score matches the number in the experiment log. This is frequently the moment when a model is declared to “work.”

That test does not answer all the questions. Does batching with other requests change outputs? Do rollout and optimization evaluate the same policy? Does latency stay acceptable with long prompts, bursty arrivals and a nearly full key/value (KV) cache? If a tool call fails halfway through an agent trajectory, can the system distinguish that failure from a poor model decision?

The weights do not answer any of these questions. Infrastructure does.

By AI infrastructure, I mean the mechanisms that turn weights into observed behavior. These include hardware mappings, memory management, schedulers, runtimes, data paths, evaluation and operational controls. The definition includes more than a GPU cluster. These mechanisms determine the model’s computations, the execution, the records and the output that a user receives.

“Plumbing” suggests carrying an unchanged substance between places. Here the path changes the result. Batches can change floating-point sum order. Cache policy changes admission and latency. A stale reward service changes the optimization target. An opaque retry can duplicate a tool action.

I want infrastructure to answer three practical questions:

• What did an experiment test?
• Can the product afford its workload?
• What happens when that workload changes or fails?

The checkpoint illusion

A checkpoint stores learned parameters. A working system continues to execute. Its behavior can differ from the behavior that a saved score appears to promise.

Hidden Technical Debt in Machine Learning Systems predates today’s foundation-model stack. Sculley and colleagues traced costly change to data dependencies, feedback, configuration debt and undeclared consumers. Model code is a small part of that surrounding system. The subsequent ML Test Score turns this into production tests and monitoring across data, models and infrastructure.

One checkpoint can run in a trainer, optimized inference engine, speculative decoder, quantized deployment, reinforcement-learning (RL) rollout service or agent harness. Kernels, precision, batches, tokenization, sampling, state layout and failure handling may differ. Those are different executions of “the model.”

Researchers are correct that infrastructure should not hide weak algorithms. Engineers are correct that evaluation depends on implementation. Find the properties necessary for this particular result to be valid. Then do tests of these properties.

Five responsibilities in model execution

I use five responsibilities to find failures and show the interactions.

1. Compute

Compute maps a mathematical graph to processors. This includes kernels, precision, partitioning, compilers, communication collectives and synchronization. The equation names a multiplication. The implementation selects GPU kernels, tiles, accumulation order, device and utilization.

Megatron-LM partitions computation in Transformer layers. ZeRO partitions optimizer states, gradients and parameters. The two methods let the combined GPU memory hold larger models. Sharded execution must still be sufficiently equivalent to the intended optimization. Communication and precision choices must stay controlled.

2. Memory and state

Memory management tracks the state, its holder process and the time when the state can be freed. Training includes parameters, gradients, optimizer state, activations, input buffers and checkpoints. Inference adds weights, temporary activations, CUDA graph pools, prefixes and per-request KV. Agents also keep conversations, files, tool results, credentials and sandbox snapshots.

An out-of-memory (OOM) error can start with oversized weights or reserved pools, or arise subsequently from cache growth, fragmentation, leaked preprocessing objects or another process. Chenyang Zhao’s When SGLang OOMs separates startup and runtime pressure before memory fractions are adjusted. His Torch Memory Snapshot case study attributes one vision-language-model RL (VLM-RL) failure to rollout image-processing fragmentation. This engineering report shows how to trace allocations and their lifetimes.

3. Execution and scheduling

Scheduling selects the work, its location and its execution time. This includes batching, admission, placement, queues, preemption, retries, backpressure (limiting incoming work) and communication between stages.

Ray was designed for the mix of task-parallel and actor-based computation in emerging AI applications. LLM serving engines have a narrower workload: irregular prompts and generations share model replicas and memory. Agents and RL add tools, sandboxes, rewards and weight synchronization. The scheduler selects the goal that has priority when throughput, fairness, deadlines, cost and weight freshness conflict.

4. Correctness and observability

For numerical correctness, first select the comparison criterion: identical bits, numerical tolerance, equivalent distributions or stable task scores. Then select applicable kernels, tests and error bounds for that criterion.

Observability is the recording of sufficient information to reconstruct an execution. This includes versioned configuration, structured traces, resource use, numerical checks, request history and clear reference paths. These records help explain a change. A faster result alone does not explain the change.

5. Data and feedback

Data handling covers inputs and returned outcomes: dataset versions, filtering, tokenization, templates, rollout trajectories, labels, reward models, evaluation sets and production traces.

A serving request can appear to have only one direction. In post-training and agents, trajectories become training data, evaluators supply rewards and production failures become tests. The data’s source and delay can thus change future learning.

These responsibilities interact. Low GPU utilization can come from slow kernels, poor batches or starved input workers. Lower rewards can reflect a model regression, stale rollout weights, evaluator timeouts or changed templates. “Infra issue” begins the investigation. It does not find a cause.

Infrastructure determines what an experiment means

The most important infrastructure failures do not cause a crash. They return plausible numerical results.

Floating-point sums depend on order. Parallel kernels and changing batch shapes may select different groupings. Thinking Machines Lab’s Defeating Nondeterminism in LLM Inference shows variation from missing batch invariance. This can occur even with greedy decoding. One request’s output depends on the other requests in its batch.

The authors show that deterministic kernels must preserve outputs across batch composition.

For a small illustration, round each addition to float32.

With $a=10^8$, $b=-10^8$, and $c=1$, $(a+b)+c$ gives $1$, but $a+(b+c)$ gives $0$. The second grouping loses the unit when the unit is added to $-10^8$.

A kernel can use the same grouping for all repeated launches. It can still select a different grouping when its batch shape changes. Thus, repetition of one fixed batch is a weaker test than serving the same request in different batches.

For some applications, quality distributions are necessary rather than identical bits. Sampling deliberately adds randomness. Evaluation must separate that variance from batch-dependent numerics before it attributes a small score change to a checkpoint. Kernel dispatch is another hidden choice: plausible output alone cannot identify which backend executed.

RL consumes its own inference results. Rollout samples a policy. Training then scores and differentiates those trajectories at synchronized weights. In Diagnosing Training-Inference Mismatch in LLM Reinforcement Learning, Zhong and colleagues isolate small token-level numerical disagreements that can independently collapse training in their diagnostic setting. That scoped result makes training–inference mismatch a first-order systems perturbation, not harmless noise to average away.

A method that gives better results only with a particular mismatch may use the runtime difference to its advantage. If training collapses with mismatch, the learning objective may still be sound. The runtime difference may be the cause. Before you interpret curves, compare tokenizers, prompt templates, weight freshness, log probabilities, precision, kernels, sampling and trajectory sources.

Zhao’s Awesome-ML-SYS-Tutorial warns that flawed RL infrastructure can invalidate the conclusion built on it. I would extend that: infrastructure can be an experimental variable even when treated as apparatus.

Keep an optimized path and a clear reference. These reference paths are the research checks. Serving might use one request, eager execution, fixed precision and no speculation as its reference. RL can use a small model and compare rollout and training token by token. An agent can replay a saved tool transcript without live external effects.

The reference is a ruler for testing whether an optimization preserved the intended computation.

Infrastructure determines whether the economics close

Correct execution is necessary. But inference cost can still be too high for a correct product.

For training cost, first examine the capacity for all state and its memory layout. Parameters are only part of this state. It also includes gradients, optimizer state, activations, temporary buffers and communication staging.

Tensor, pipeline and data parallelism trade device capacity for communication and idle time. Activation checkpointing calculates activations again to save memory. ZeRO shards replicated state. Measure learning progress alongside that coordination.

At inference time, the workload splits again. The Transformer model defines the computation, but matrix shapes and data movement decide how well the GPU can execute it. Prefill processes many prompt tokens in parallel and frequently supplies compute-rich matrix multiplication. Decode repeatedly processes a small number of new tokens. It also reads weights and a growing KV cache at the same time. Thus, memory traffic and batching become increasingly important.

“Tokens per second” may mean server throughput or one user’s decode rate. It omits first-token waiting, uneven streaming and 99th-percentile behavior near a full cache. A serving report should separate these measurements:

• Time to first token (TTFT): queueing, prompt processing, and first-token emission
• Inter-token latency (ITL): the cadence of subsequent decode steps for a streaming user
• Throughput: completed input and output work per unit time, with the accounting stated
• Tail latency: behavior with the slow requests and resource contention a mean hides
• Memory headroom: capacity for weights, runtime pools, and live per-request state
• Cost per successful task: the unit the product ultimately has to afford

PagedAttention and vLLM map logical KV blocks to physical blocks. This decreases waste from contiguous reservations and fragmentation. Better allocation lets more batching run before the system uses all capacity.

SGLang also makes structured language-model programs available. RadixAttention reuses shared-prefix KV. The frontend shows generation and parallelism patterns. It makes prefixes, dependencies and structured decoding available to the runtime. This lets the runtime schedule and reuse resources beyond independent calls.

My full-stack map and vLLM request walkthrough explain why optimizations move bottlenecks. Faster attention can expose scheduler cost. Quantized weights can expose KV-cache bandwidth. Prefix reuse saves prefill work but keeps cache memory occupied. Larger batches can increase throughput but delay interactive users. The token’s journey follows those interactions.

A LinkedIn production report makes the coupling concrete. In Turbocharging LinkedIn's Recommendation Systems with SGLang, the team shows ranking workloads with shared member context and item-specific context. Their “knock-knock” design overlaps member-context prefill with candidate retrieval, then routes the follow-up to the data-parallel worker that holds the relevant KV cache.

In that application, they report a decrease in overall latency from 520 ms to 200 ms. They also acknowledge the extra GPU compute. The endpoints give a decrease of about 62%. The article itself gives a “~38% decrease,” and apparently uses the remaining fraction instead.

This application uses extra compute early to hide a critical-path wait. It keeps state placement consistent across the two calls.

The economics close when the system makes the product’s limiting resource or delay better. A better isolated engine score is not sufficient.

Let the workload shape the system

Labels such as LLM, video, speech and robotics help organize teams. They can hide the resource profile that should shape a runtime.

Zhao’s SGLang Omni argues for grouping multi-stage generative models by decoding computation rather than modality. A thinker’s prefill may be compute-bound, its decode memory-bound, and a lightweight talker or multi-token predictor latency-sensitive. Different schedulers may be necessary for different shapes, cache lifetimes and service goals.

The same reasoning is applicable to other modalities. A streaming autoregressive video generator has a causal backbone, visual memory, a sampler, a decoder, and a display deadline. The sampler may be compute-heavy, the memory path may grow with history, and the decoder must deliver frames before the user notices a stall. Reporting one frames-per-second number hides time to first frame, inter-frame jitter, resolution, denoising steps, and hardware.

An embodied world model adds another constraint: actions arrive online and stale predictions can be useless even if they are visually good. Robot data collection, policy inference, safety checks, and actuator control run at different rates. The system must decide which state is persistent, which observation can be dropped, and which deadline is hard. In the open-pi family, action chunking and high-level subgoals are model ideas, but they also define scheduling and control interfaces. A chunk that is elegant in a training objective must still be available before the chunk is necessary for the controller.

Three questions usually show more than the modality label:

1. Where is the critical path? Compute, memory movement, communication, queueing, or an external service?
2. Which state grows with the request? KV cache, visual memory, simulator state, tool transcripts, or intermediate artifacts?
3. What can be batched if product requirements are met? Prompts, decode steps, frames, environments, reward calls, or nothing at all?

Different queues and memory budgets can be necessary for similar Transformers. Independent scheduling with explicit communication may be necessary for stages. One batch that waits for its slowest component may be unsuitable.

Agents and RL turn pipelines into feedback loops

In agents and RL, outputs return as subsequent inputs. Thus, a one-way pipeline diagram is incomplete.

RL can generate trajectories, execute tools and code, score and filter data, update weights and install them in rollout workers. Agents make plans, use tools, edit files, retry and delegate. Subsequent steps depend on external changes from earlier ones.

This cyclic structure changes the infrastructure problem in four ways.

First, resources differ: GPUs run models, CPUs prepare data, sandboxes execute untrusted code, databases store trajectories and services return tool results and rewards. Faster kernels cannot unblock a saturated sandbox pool.

Second, state persists. Failure after twenty successful tool calls makes replay and cleanup costly. External systems rarely all support exactly-once execution. Idempotency keys let a supporting service recognize repeated requests as one operation. Checkpoints, leases and compensating actions help recovery.

Third, versions move. Record what rollout, reward and training workers used. Weight transfers use bandwidth. They also set policy lag, and therefore which policy version generated each batch of data.

Fourth, evaluators are services with code versions, dependencies, timeouts and possible bias. A sandbox exit may indicate policy failure, environment failure or termination of a correct slow solution. Without end-to-end traces, those cases collapse into one score.

Zhao uses the barrel effect, where the shortest board limits capacity, in his LinkedIn article and X post. More rollouts may only make a serial reward queue longer. Faster training may barely help when environment resets dominate. Local utilization differs from completed work.

Aaron Levie’s related industry argument forecasts agents running continuously and in parallel, stressing sandboxes, search, files, permissions and services. The argument identifies non-model capacity limits that parallel model calls can expose.

Automated research can increase trials before judgment improves. Leakage, silent retries, mixed code versions or irreproducible environments then increase confidence without knowledge. I would build source tracking, isolation, budgets and replay into automation.

Follow one trajectory across the system

Examine an illustrative agent trajectory with a serial critical path. Model computation takes 0.6 seconds. The tool wait takes 2.0 seconds. Evaluation takes 0.4 seconds.

Assume there is no overlap or queueing in this example. It takes 3.0 seconds. With half the model time, the total is 2.7 seconds. Twice the model speed gives a 10% decrease in trajectory latency.

If the tool can run concurrently with other work, first redraw the critical path. Then add the durations.

Suppose an evaluator timeout causes a tool retry.

Compute traces explain the 0.6 seconds. Retained state determines whether the first result survives. Scheduling determines whether the retry blocks other trajectories. Correctness checks distinguish one action from duplicate attempts. Data handling determines whether the timeout is recorded as an infrastructure failure or used in training as a failed policy decision.

A trace connects those questions. Faster computation helps only on the critical path. Better recovery can preserve a result without accelerating a kernel. Measurements of each subsystem are still necessary.

Properties of good AI infrastructure

Different stacks are necessary for a workstation, public API, robot lab and RL cluster. I use the following principles to select between them.

Measure before optimizing

Start with a causal timeline and a resource budget. For serving, record queue time, prefill, decode, cache occupancy, batch shape, and tail latency. For training, measure each component: input stalls, forward and backward compute, communication, optimizer work, checkpointing and failed-step recovery. For agents, record traces of model calls, tool queues, sandbox startup, external latency, retries and evaluator time.

Measurement has overhead. Full traces cannot always run on every request. The answer is deliberate sampling and escalation: cheap counters continuously, representative traces routinely, and high-detail capture around incidents or regressions.

Select the numerical target before the speed comparison

Select a reference and checks for each optimized path. Include batch invariance where it is necessary. Compare log probabilities from rollout and training. Replay agent trajectories. Examine cache isolation across tenants and prompts. Examine retries for duplicated side effects.

Write the necessary equivalence. Then do a test of that equivalence. Identical bits may be unnecessary, but speed alone cannot show the computation was preserved.

Reuse components with explicit inputs and responsibilities

Zhao’s reflection on SGLang for RL favors reused components exposed as libraries rather than one framework for all workloads. I agree: shared components concentrate debugging and performance work. They also let unusual loops compose them.

Integration stays our responsibility. Components can disagree on tokenization, which process frees device memory, cancellation or versions, although they pass their individual tests. Their APIs should show those responsibilities.

Name the state holders and cleanup paths

Give each large object a named holder, lifetime, budget and cleanup path. These objects include KV blocks, prefixes, CUDA graph pools, optimizer shards, trajectories, sandbox disks and model versions. Several processes can reserve the same resource. Thus, name the responsible process.

A named holder also makes failure recovery better. A restarted worker should have the following information:

• The leases that expired
• The requests that can resume
• The side effects that may already have happened
• The cached state that is unsafe to reuse

Treat observability and recovery as product capabilities

Logs after an incident are weaker than traces designed around the execution graph. Carry request, trajectory, model, dataset, and code-version identifiers across process and service calls. Record cancellation and retry reasons. Keep a path to reconstruct the configuration that produced a result.

Recovery deserves equal status. Checkpoint frequency, resumable queues, idempotent tools, circuit breakers, and degraded modes determine how much useful work survives a failure. In long-running agents and RL loops, the expected cost of failure grows with trajectory length. Recovery policy is part of the cost model.

Optimize the end-to-end objective

The metric should agree with the system's purpose. A chat product may give priority to TTFT and readable streaming with a cost ceiling. Batch generation may give priority to completions per GPU-hour. A robot policy may give priority to deadline misses and task success. An RL run may give priority to accepted trajectories or learning progress per wall-clock hour.

Component metrics stay necessary for diagnosis. They should not become the goal by convenience. A fully utilized GPU can be in a system that is slow, wasteful, or wrong.

Select capabilities from the workload

Select capabilities for the workload. Start with a reproducible run and resource control.

For a service, add load management. For multi-node training, add recovery. For an agent, add tracking of trajectories and side effects. These branches can combine. A fixed-model agent can make a reliable loop on one machine without distributed training.

Reproduce one complete run

One experiment runs from a clean environment with pinned code, configuration, dataset version, random seeds, and an archived result. A small reference case has checks of basic numerical behavior. Failure is shown rather than converted into a partial success.

Exit criterion: another machine or clean environment can reproduce the run in the declared tolerance range, and the artifacts identify exactly what executed.

Measure resource limits

The system measures peak and steady-state memory. It records startup and runtime allocation in different categories. It measures execution and resource use for the critical kernels. It frees what it allocates. It can explain why the available capacity is sufficient for a workload. It can also find the resource that limits concurrency and explain the events after cancellation.

Exit criterion: increasing model size, sequence length, batch, or concurrent work produces a predictable capacity limit instead of a mysterious OOM.

For concurrent users: a load-aware service

Requests have queueing, admission, batching, caching, timeouts, and backpressure policies. Measurements of TTFT, ITL, throughput, memory headroom and tail latency use a stated request distribution. The service supports graceful overload rather than accepting work it cannot complete.

Exit criterion: a load test that resembles production meets its service objective. The team can attribute a regression to queueing, compute, memory or downstream delivery.

For multi-node learning: distributed recovery and provenance

The system tracks data and checkpoint versions across workers. Communication and idle time are shown. Interrupted work resumes safely. When learning uses rollouts and rewards, policy and evaluator versions also enter the lineage (a record of which data, code and weights produced each result). Rollout and training paths have an agreement test, and evaluator failures are separated from model outcomes.

Exit criterion: a multi-node failure or worker replacement loses a bounded amount of work without corrupting the experiment's provenance or silently changing its objective.

For closed loops: trajectory tracking and feedback integrity

Model inference, tools, sandboxes, and evaluation share end-to-end lineage. Side effects are idempotent or can be undone by a compensating action. The system has budgets for time, tokens, compute and external calls. The system can replay representative trajectories and explain which version made each decision. A fixed-model agent can stop there. If its outputs subsequently become rewards or training data, that additional feedback path must preserve the same provenance.

Exit criterion: a trajectory can be audited and recovered without classification of an infrastructure failure as a model outcome. If the system learns from those trajectories, only validated outcomes enter its training signal.

For many research claims, only a reproducible, resource-aware single run is necessary. This is easier to examine. Add distribution, concurrency or feedback checks when they are necessary for the workload.

The path is part of the result

Weights are easy to save, upload, name and compare. Execution is spread across kernels, queues, caches, services and operational habits, so it is easier to overlook.

But execution contains the central questions. Did the intended policy run? Can useful work stay in its resource budget? Will behavior survive traffic and failures beyond a demo?

Compute implements the graph. Memory management tracks state and lifetimes. Scheduling uses the product's priorities. Observability records what ran, what was compared and what failed. Data handling tracks sources as outputs become new inputs. Together they make those questions testable.

Open research questions concern coordination as much as speed. Which equivalence tests should engines share? How should deadlines span model calls, sandboxes, rewards and tools? What state should persist through long multimodal tasks or be recomputed? How do we benchmark full loops without a reward for a fast component in a failing system?

Those decisions determine what the model computes and delivers in practice.

@article{ji2026aiinfrastructureisnotplumbing,
  title = {AI Infrastructure Is Not Plumbing},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/ai-infrastructure-is-not-plumbing/}
}

References

• Hidden Technical Debt in Machine Learning Systems, D. Sculley et al., NeurIPS, 2015.
• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, Eric Breck et al., 2017.
• Defeating Nondeterminism in LLM Inference, Horace He et al., Thinking Machines Lab, 2025.
• Diagnosing Training-Inference Mismatch in LLM Reinforcement Learning, Tianle Zhong et al., arXiv:2605.14220, 2026.
• Efficient Memory Management for Large Language Model Serving with PagedAttention, Woosuk Kwon et al., SOSP, 2023.
• SGLang: Efficient Execution of Structured Language Model Programs, Lianmin Zheng et al., arXiv:2312.07104, 2023.
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, Mohammad Shoeybi et al., arXiv:1909.08053, 2019.
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Samyam Rajbhandari et al., SC20, 2020.
• Ray: A Distributed Framework for Emerging AI Applications, Philipp Moritz et al., OSDI, 2018.
• Turbocharging LinkedIn's Recommendation Systems with SGLang, Steven Shimizu et al., LinkedIn Engineering, 2025.
• Awesome-ML-SYS-Tutorial, Chenyang Zhao et al., GitHub repository, accessed July 13, 2026.
• RL Infra: The Board You Polished the Most Might Matter the Least, Chenyang Zhao, LinkedIn, 2026. X version.
• SGLang Omni: Redesigning the Inference Framework for Multi-Stage Generative Models from Decoding Computation Characteristics, Chenyang Zhao, LinkedIn, 2026.
• When SGLang OOMs: What Exactly Runs Out of Memory?, Chenyang Zhao, LinkedIn, 2026.
• Analyzing GPU Memory Leaks with Torch Memory Snapshot for VLM RL, Chenyang Zhao, LinkedIn, 2025.
• Reflecting on a Year of Building SGLang for Reinforcement Learning, Chenyang Zhao, LinkedIn, 2026.
• Agent Infrastructure Observation, Aaron Levie, X, 2026.

Updated 03 November 2026