AI Infrastructure Is Not Plumbing

Figure 1. A checkpoint becomes useful only when compute, state, scheduling, evidence, and feedback remain connected. Generated with GPT Image 2.

A checkpoint loads. It answers a prompt. Its offline score matches the number in the experiment log. This is often the moment when a model is declared to “work.”

But that test leaves most of the hard questions unanswered. Does the same input produce meaningfully equivalent behavior when it shares a batch with other requests? Does the rollout engine evaluate the same policy that the optimizer updates? Can the service keep its latency promise when prompts are long, arrivals are bursty, and KV cache is nearly full? If a tool call fails halfway through an agent trajectory, can the system tell failure from a bad model decision?

The weights do not answer any of these questions. Infrastructure does.

I use AI infrastructure to mean the execution contract between a model and the world: the hardware mappings, memory ownership, schedulers, runtimes, data paths, evaluation machinery, and operational controls that turn weights into observed behavior. That definition is wider than a GPU cluster and narrower than “everything around AI.” It includes the mechanisms that can change what the model computes, which work gets to run, what evidence we record, and whether the result reaches a user intact.

This is why calling AI infrastructure “plumbing” is misleading. Plumbing is supposed to carry an unchanged substance from one place to another. AI systems do not have that property. Batch composition can alter floating-point reduction paths. A cache policy can change admission and latency. A stale reward service can change the optimization target. An opaque retry can turn one tool action into two. The path participates in the result.

The argument of this post is therefore practical: infrastructure determines whether an experiment means what we think it means, whether the economics of running the model close, and whether a product remains reliable under the workload it actually receives.

The checkpoint illusion

A checkpoint is a compact description of learned parameters. A working AI system is a continuing process. The distance between the two is where many confident conclusions become fragile.

The classic warning predates the current foundation-model stack. In Hidden Technical Debt in Machine Learning Systems, Sculley and colleagues described how data dependencies, feedback loops, configuration debt, and undeclared consumers make ML systems expensive to change. The point was not that models are unimportant. It was that model code occupies a small region inside a much larger system whose couplings determine long-term behavior. The later ML Test Score made the operational consequence explicit: production readiness requires tests and monitoring across data, models, and infrastructure.

Foundation models deepen that coupling. The same checkpoint may be used by a training engine, an optimized inference runtime, a speculative decoder, a quantized deployment, an RL rollout service, and an agent harness. Those paths can differ in kernels, precision, batching, tokenization, sampling, state layout, and failure handling. “The model” is no longer one execution path.

This distinction helps resolve a common disagreement. Model researchers are right that infrastructure should not hide algorithmic weaknesses. Systems engineers are right that an algorithm cannot be evaluated independently of its realization. The useful boundary is not model versus system. It is which contract must hold for this result to be valid.

Five contracts beneath every model

I find five contracts useful because they separate failure ownership without pretending the layers are independent.

1. Compute

The compute contract maps a mathematical graph onto processors. It covers kernels, precision, parallel decomposition, compiler choices, communication collectives, and synchronization. A Transformer equation says which multiplication should happen; the compute path decides its tile shapes, accumulation order, device placement, and delivered utilization.

Training systems such as Megatron-LM made large models feasible by partitioning computation within Transformer layers. ZeRO attacked a related constraint by partitioning optimizer states, gradients, and parameters. These are often presented as scale techniques. They are also semantic obligations: sharded execution must remain equivalent enough to the intended optimization while communication and precision choices stay controlled.

2. Memory and state

The memory contract says what state exists, who owns it, where it lives, and when it may be reclaimed. During training this includes parameters, gradients, optimizer states, activations, dataloader buffers, and checkpoints. During inference it includes weights, temporary activations, CUDA graph pools, prefix entries, and per-request KV cache. In an agent system it extends to conversation state, files, tool outputs, credentials, and sandbox snapshots.

An out-of-memory error is therefore not a single diagnosis. It can occur during process startup because weights and reserved pools do not fit, or later because dynamic cache growth, fragmentation, a leaked preprocessing object, or a different process owns the memory. Chenyang Zhao's post When SGLang OOMs is valuable precisely because it separates startup and runtime pressure instead of treating an OOM as “use a smaller memory fraction.” His later Torch Memory Snapshot case study attributes one VLM-RL failure to image-processing memory fragmentation in the rollout path. This is an engineering report, not a universal benchmark, but the diagnostic lesson travels well: locate ownership and lifetime before tuning capacity.

3. Execution and scheduling

The execution contract determines which work runs, where, and in what order. It includes batching, admission control, placement, queueing, preemption, retries, backpressure, and communication between stages.

Ray was designed around the mix of task-parallel and actor-based computation found in emerging AI applications. LLM serving engines solve a more specialized version: irregular prompts and generations must share model replicas and memory. Agent and RL systems add external tools, sandbox lifetimes, reward services, and weight synchronization. The scheduler becomes the place where product priorities—throughput, fairness, deadline, cost, or freshness—turn into actual execution.

4. Correctness and observability

The correctness contract defines equivalence. Must runs be bitwise identical, numerically close, distributionally equivalent, or only stable on a task metric? The answer changes which kernels, tests, and tolerances are acceptable.

Observability makes that contract inspectable. It includes versioned configuration, structured traces, resource measurements, numerical checks, request lineage, and reference paths that favor clarity over speed. Without them, a faster system can only report that something changed. It cannot establish why.

5. Data and feedback

The data contract covers what enters the system and how outcomes return. Dataset versions, filtering, tokenization, prompt templates, rollout trajectories, human labels, reward models, evaluation sets, and production traces all belong here.

In conventional serving, this path can look one-way: request in, prediction out. In post-training and agents it becomes a loop. Generated trajectories become training data; evaluator outputs become rewards; production failures become new tests. Once outputs can alter future models, provenance and delay become part of the learning algorithm.

The contracts interact, but naming them prevents vague remedies. Low GPU utilization can be a compute problem, a scheduler that cannot form useful batches, or a data pipeline that starves workers. A reward drop can be a model regression, stale rollout weights, evaluator timeouts, or an unnoticed template change. “Infra issue” is not a root cause. It is the start of a search across explicit contracts.

Infrastructure determines what an experiment means

The most important infra failures do not crash. They return plausible numbers.

Floating-point arithmetic is order-sensitive. Parallel kernels may reduce values in different orders, and serving batches change the shapes over which those kernels operate. Thinking Machines Lab's Defeating Nondeterminism in LLM Inference traces a striking source of variation to the lack of batch invariance: the output for one request can depend on which other requests share its batch, even with greedy decoding. The authors show that deterministic kernels must be invariant to batch composition, not merely repeat an operation under a fixed launch shape.

That result does not mean every product needs bitwise-identical generation. Sampling is intentionally stochastic, and many applications care about quality distributions rather than exact tokens. It does mean that an evaluation harness must know which variability it is measuring. If batch-dependent numerics are mixed with sampling variance, a small benchmark delta may say more about runtime shape than about the checkpoint.

Reinforcement learning raises the stakes because inference produces the data that training consumes. The rollout engine samples actions from a policy; the training engine computes probabilities and gradients for those trajectories. They are intended to represent the same policy at synchronized weights. In Diagnosing Training-Inference Mismatch in LLM Reinforcement Learning, Zhong and colleagues isolate small token-level numerical disagreements between the two paths and show that they can independently cause training collapse in their diagnostic setting. Their conclusion is carefully scoped, but it is strong: training–inference mismatch is a first-order systems perturbation, not harmless noise to average away.

This changes how an RL experiment should be read. If a method improves only when rollout and training runtimes disagree in a particular way, the result may not identify an algorithmic gain. Conversely, a run that collapses under mismatch may not falsify the learning objective. Before explaining the curve, the system should check tokenizer and template equality, weight freshness, log-probability agreement, precision, kernel settings, sampling parameters, and trajectory provenance.

Zhao makes the same point in practical terms in the introduction to Awesome-ML-SYS-Tutorial: a flawed RL infrastructure can invalidate the conclusion built on top of it. I would state the implication more broadly. Infrastructure is sometimes an independent variable in the experiment, even when the paper treats it as apparatus.

A credible setup therefore needs two paths. The optimized path is allowed to be complicated. The reference path should be small, slow, and easy to compare. For a serving engine, that may mean one request, eager execution, fixed precision, and no speculative decoding. For RL, it may mean a small model where rollout and training log probabilities can be checked token by token. For an agent, it may mean replaying a recorded tool transcript without live external effects.

The reference path is not wasted engineering. It is the ruler used to decide whether an optimization preserved the thing being measured.

Infrastructure determines whether the economics close

Correct execution is necessary, but an AI product can be correct and still be unaffordable.

At training time, the first economic question is whether the model fits. Parameters are only one part of the memory budget. Gradients, optimizer states, activations, temporary buffers, and communication staging can dominate. Tensor, pipeline, and data parallelism trade device capacity against communication and idle time. Activation checkpointing saves memory by recomputing work. ZeRO saves replicated state by sharding it. Each technique changes the ratio between useful learning and expensive coordination.

At inference time, the workload splits again. The Transformer model defines the computation, but matrix shapes and data movement decide how well the GPU can execute it. Prefill processes many prompt tokens in parallel and often exposes compute-rich matrix multiplication. Decode repeatedly processes a small number of new tokens while reading weights and a growing KV cache, making memory traffic and batching increasingly important.

This is why “tokens per second” is not a sufficient product metric. It can mean aggregate server throughput or one user's decode rate. It says nothing about how long the user waited for the first token, how uneven token delivery became, or how the 99th percentile behaved when the cache filled. A useful serving report separates at least:

• Time to first token (TTFT): queueing, prompt processing, and first decode.
• Inter-token latency (ITL): the cadence of later decode steps for a streaming user.
• Throughput: completed input and output work per unit time, with the accounting stated.
• Tail latency: behavior under the slow requests and resource contention a mean hides.
• Memory headroom: capacity for weights, runtime pools, and live per-request state.
• Cost per successful task: the unit the product ultimately has to afford.

PagedAttention and vLLM provide a clean example of infrastructure changing the feasible operating point. KV cache grows and shrinks with each request, so contiguous allocation wastes memory through fragmentation and reservation. PagedAttention maps logical KV blocks onto physical blocks, borrowing the logic of virtual memory. The important result is not a timeless throughput multiplier—the paper's reported comparisons depend on its models, hardware, sequences, and baselines. It is the mechanism: better state allocation allows more useful batching before memory becomes the limiter.

SGLang extends the serving problem to structured language-model programs. RadixAttention reuses KV state across shared prefixes, while the frontend exposes generation and parallelism patterns to the runtime. Again, the useful idea is the contract between program structure and resource reuse. A runtime can schedule more effectively when repeated prefixes, dependencies, and structured decoding are visible instead of buried in independent API calls.

My earlier full-stack map of LLM inference and request-level walkthrough of vLLM go deeper into these mechanisms. The short version here is that an optimization often moves the bottleneck. A faster attention kernel can expose scheduler overhead. Quantized weights can make KV-cache bandwidth more visible. Prefix caching can save prefill compute while consuming memory longer. Larger batches can improve aggregate throughput while hurting interactive tail latency. The token's journey across the stack is coupled all the way to the response stream.

A LinkedIn production report makes the coupling concrete. In Turbocharging LinkedIn's Recommendation Systems with SGLang, the team describes ranking workloads with shared member context and item-specific context. Their “knock-knock” design overlaps member-context prefill with candidate retrieval, then routes the follow-up to the data-parallel worker that holds the relevant KV cache. In that application, they report overall latency dropping from 520 ms to 200 ms, while acknowledging that the technique uses extra GPU compute. The endpoints imply a reduction of about 62%; the article itself calls it a “~38% decrease,” apparently using the remaining fraction instead. This is not a generic SGLang speed claim. It is a workload-specific exchange: spend compute early to hide a critical-path wait and preserve state placement across two calls.

That is what it means for the economics to close. The system must optimize the resource and latency that constrain the product, not the number that makes the engine look best in isolation.

The workload—not the modality—should shape the system

Infrastructure is often organized around model labels: LLM serving, video inference, speech, robotics. Those categories help teams and repositories. They can obscure the resource profile that should drive the runtime.

Zhao's SGLang Omni article argues for classifying multi-stage generative models by decoding computation rather than modality. A “thinker” prefill can be compute-bound; its token-by-token decode can be memory-bound; a lightweight talker or multi-token prediction stage can be latency-sensitive. If these stages have different batch shapes, cache lifetimes, and service-level goals, forcing them through one scheduler creates false simplicity.

The same reasoning applies beyond language. A streaming autoregressive video generator has a causal backbone, visual memory, a sampler, a decoder, and a display deadline. The sampler may be compute-heavy, the memory path may grow with history, and the decoder must deliver frames before the user notices a stall. Reporting one frames-per-second number hides time to first frame, inter-frame jitter, resolution, denoising steps, and hardware.

An embodied world model adds another constraint: actions arrive online and stale predictions can be useless even if they are visually good. Robot data collection, policy inference, safety checks, and actuator control run at different rates. The system must decide which state is persistent, which observation can be dropped, and which deadline is hard. In the open-pi family, action chunking and high-level subgoals are model ideas, but they also define scheduling and control interfaces. A chunk that is elegant in a training objective still has to arrive before the controller needs it.

Three questions usually reveal more than the modality label:

1. Where is the critical path? Compute, memory movement, communication, queueing, or an external service?
2. Which state grows with the request? KV cache, visual memory, simulator state, tool transcripts, or intermediate artifacts?
3. What can be batched without violating the product contract? Prompts, decode steps, frames, environments, reward calls, or nothing at all?

These questions lead to different schedulers and memory budgets even when two systems use similar Transformers. They also reveal when stages should be separated. A multi-stage model may need independent queues joined by explicit communication, not one global batch that waits for its slowest component.

Agents and RL turn pipelines into feedback loops

Supervised training and conventional inference can often be drawn as pipelines. Agent systems and RL resist that picture because they revisit their own outputs.

An RL post-training loop may generate trajectories, execute tools or code, compute rewards, filter data, update weights, and publish those weights back to rollout workers. An agent may plan, call a tool, inspect the result, alter files, retry, and delegate work. The next step depends on external state created by the previous one.

This cyclic structure changes the infrastructure problem in four ways.

First, resources are heterogeneous. GPUs run models, CPUs preprocess data, sandboxes execute untrusted code, databases hold trajectories, and network services provide tools or rewards. Optimizing GPU kernels cannot repair a saturated sandbox pool.

Second, state is long-lived. A failure may occur after twenty successful tool calls, when replay is expensive and cleanup matters. Exactly-once semantics are rarely available across every external system, so idempotency keys, checkpoints, leases, and compensating actions become part of product correctness.

Third, policy versions move. Rollout workers, reward services, and trainers can observe different versions unless the system records them explicitly. Weight transfer is not merely bandwidth; it determines policy lag and therefore what distribution generated the data.

Fourth, evaluation becomes operational. A reward is produced by code with versions, dependencies, timeouts, and possible bias. A sandbox exit may mean the agent failed, the environment failed, or the evaluator killed a correct long-running solution. Without end-to-end traces, those cases collapse into one score.

Zhao describes RL infrastructure through the barrel effect: the shortest board determines how much water the barrel holds. His LinkedIn article and corresponding X post frame this as an engineering observation, not a theorem. The metaphor is useful because RL loops are coupled. Doubling rollout throughput may add no end-to-end capacity when rewards queue behind a serial service. A polished training kernel may barely move wall-clock time when environment resets dominate. Local utilization is not the same as system progress.

Aaron Levie has made a related industry argument: agents that run continuously and in parallel put new pressure on sandboxes, search, files, permissions, and surrounding services. I treat that as a product forecast rather than evidence of a universal workload. Its value is in naming the inversion: as model calls become easier to parallelize, non-model dependencies become visible capacity limits.

This is also the infrastructure side of automated research. Automation can increase experiment volume long before it improves experimental judgment. If the harness leaks test information, silently retries failures, mixes code versions, or cannot reproduce an environment, running more experiments produces more confidence without more knowledge. The correct response is not to avoid automation. It is to make evidence lineage, isolation, budgets, and replay part of the research system.

What good AI infrastructure looks like

There is no single good stack. A research workstation, a high-throughput API, a robot lab, and an RL cluster need different systems. The durable principles are about how the stack is reasoned about.

Measure before optimizing

Start with a causal timeline and a resource budget. For serving, record queue time, prefill, decode, cache occupancy, batch shape, and tail latency. For training, separate input stalls, forward and backward compute, communication, optimizer work, checkpointing, and failed-step recovery. For agents, trace model calls, tool queues, sandbox startup, external latency, retries, and evaluator time.

Measurement has overhead. Full traces cannot always run on every request. The answer is deliberate sampling and escalation: cheap counters continuously, representative traces routinely, and high-detail capture around incidents or regressions.

Correctness before leaderboards

Every optimized path needs invariants and a reference. Test batch invariance when it matters. Compare rollout and training log probabilities. Replay a known agent trajectory. Validate that caching does not cross tenant or prompt boundaries. Check that retries do not duplicate side effects.

This does not require bitwise determinism everywhere. It requires a written equivalence target and tests aligned with it. Performance without such a target is only a different program that happens to be faster.

Reuse mature components, but keep boundaries visible

Zhao's reflection on building SGLang for RL favors reusing open-source components and exposing a library rather than forcing every workload into one framework. I agree with the direction. Reuse concentrates debugging and performance work; library boundaries preserve the ability to compose an unusual loop.

The tradeoff is integration responsibility. A component can be individually correct while two components disagree about tokenization, process ownership, device memory, cancellation, or versioning. Stable interfaces should expose these contracts instead of hiding them behind a convenient call.

Make state ownership explicit

Every large object should have an owner, lifetime, budget, and cleanup path. That includes KV blocks, prefix caches, CUDA graph pools, optimizer shards, trajectory batches, sandbox disks, and model versions. “The runtime manages it” is not enough when multiple processes or services can reserve the same resource.

Explicit ownership also improves failure recovery. A restarted worker should know which leases expired, which requests can resume, which side effects may already have happened, and which cached state is unsafe to reuse.

Treat observability and recovery as product capabilities

Logs after an incident are weaker than traces designed around the execution graph. Carry request, trajectory, model, dataset, and code-version identifiers across boundaries. Record cancellation and retry reasons. Keep a path to reconstruct the configuration that produced a result.

Recovery deserves equal status. Checkpoint frequency, resumable queues, idempotent tools, circuit breakers, and degraded modes determine how much useful work survives a failure. In long-running agents and RL loops, the expected cost of failure grows with trajectory length; recovery policy is part of the cost model.

Optimize the end-to-end objective

The metric should match the system's purpose. A chat product may prioritize TTFT and readable streaming under a cost ceiling. Batch generation may prioritize completions per GPU-hour. A robot policy may prioritize deadline misses and task success. An RL run may prioritize accepted trajectories or learning progress per wall-clock hour.

Component metrics remain necessary for diagnosis. They should not become the goal by convenience. A fully utilized GPU can sit inside a system that is slow, wasteful, or wrong.

A practical maturity ladder

Infrastructure maturity is easier to reason about as earned capabilities than as company size. The levels below are cumulative. Moving upward before the previous contract is stable creates a larger debugging surface, not a more advanced system.

Level 0: Reproducible single run

One experiment runs from a clean environment with pinned code, configuration, data identity, random seeds, and an archived result. A small reference case checks basic numerical behavior. Failure is visible rather than converted into a partial success.

Exit criterion: another machine or clean environment can reproduce the run within the declared tolerance, and the artifacts identify exactly what executed.

Level 1: Resource-aware single node

The system measures peak and steady-state memory, separates startup from runtime allocation, profiles the critical kernels, and owns cleanup. It can explain why a workload fits, which resource limits concurrency, and what happens after cancellation.

Exit criterion: increasing model size, sequence length, batch, or concurrent work produces a predictable capacity boundary instead of a mysterious OOM.

Level 2: Load-aware service

Requests have queueing, admission, batching, caching, timeouts, and backpressure policies. TTFT, ITL, throughput, memory headroom, and tail latency are measured against a stated request distribution. The service supports graceful overload rather than accepting work it cannot finish.

Exit criterion: a load test that resembles production meets its service objective, and the team can attribute a regression to queueing, compute, memory, or downstream delivery.

Level 3: Distributed training or post-training

The system tracks data, policy, reward, and checkpoint versions across workers. Communication and idle time are visible. Interrupted work resumes safely. Rollout and training paths have an agreement test, and evaluator failures are separated from model outcomes.

Exit criterion: a multi-node failure or worker replacement loses a bounded amount of work without corrupting the experiment's provenance or silently changing its objective.

Level 4: Closed-loop agents and RL

Model inference, tools, sandboxes, rewards, evaluation, and retraining share end-to-end lineage. Side effects are idempotent or compensatable. Budgets apply to time, tokens, compute, and external calls. The system can replay representative trajectories and explain which version made each decision.

Exit criterion: the loop can improve from its own data without turning infrastructure failures into training signal or allowing a local bottleneck to dictate system behavior invisibly.

No level is automatically better. Many research questions should stay at Level 0 or 1 because a small system is easier to understand. Maturity means meeting the contracts the claim requires, not accumulating distributed components.

The path is part of the result

The checkpoint illusion persists because weights are tangible. They can be saved, uploaded, compared, and named. The execution path is distributed across kernels, queues, caches, services, and operational habits. It is harder to point at, so it is easy to treat as support work.

Yet three of the most important questions live there. Did the experiment evaluate the policy it intended to evaluate? Can the model deliver useful work within its memory, latency, and cost budget? Will the product preserve its behavior when traffic, tools, and failures stop looking like a demo?

AI infrastructure answers those questions by enforcing contracts. Compute must implement the intended graph. Memory must give state an explicit owner and lifetime. Scheduling must turn product priorities into execution. Observability must make equivalence and failure inspectable. Data and feedback must preserve lineage as outputs become future inputs.

The remaining research problems are as much about interfaces as speed. What equivalence tests should be standard across training and inference engines? How should a scheduler express deadlines across model calls, sandboxes, rewards, and tools? Which state should persist across long multimodal trajectories, and which can be recomputed? How do we benchmark an RL or agent system without rewarding a fast component inside a slow or unreliable loop?

Those questions do not sit underneath the model. They determine what the model is allowed to mean in practice.

@article{ji2026aiinfrastructureisnotplumbing,
  title = {AI Infrastructure Is Not Plumbing},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/ai-infrastructure-is-not-plumbing/}
}

References

• Hidden Technical Debt in Machine Learning Systems, D. Sculley et al., NeurIPS, 2015.
• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, Eric Breck et al., 2017.
• Defeating Nondeterminism in LLM Inference, Horace He et al., Thinking Machines Lab, 2025.
• Diagnosing Training-Inference Mismatch in LLM Reinforcement Learning, Tianle Zhong et al., arXiv:2605.14220, 2026.
• Efficient Memory Management for Large Language Model Serving with PagedAttention, Woosuk Kwon et al., SOSP, 2023.
• SGLang: Efficient Execution of Structured Language Model Programs, Lianmin Zheng et al., arXiv:2312.07104, 2023.
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, Mohammad Shoeybi et al., arXiv:1909.08053, 2019.
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Samyam Rajbhandari et al., SC20, 2020.
• Ray: A Distributed Framework for Emerging AI Applications, Philipp Moritz et al., OSDI, 2018.
• Turbocharging LinkedIn's Recommendation Systems with SGLang, Steven Shimizu et al., LinkedIn Engineering, 2025.
• Awesome-ML-SYS-Tutorial, Chenyang Zhao et al., GitHub repository, accessed July 13, 2026.
• RL Infra: The Board You Polished the Most Might Matter the Least, Chenyang Zhao, LinkedIn, 2026. X version.
• SGLang Omni: Redesigning the Inference Framework for Multi-Stage Generative Models from Decoding Computation Characteristics, Chenyang Zhao, LinkedIn, 2026.
• When SGLang OOMs: What Exactly Runs Out of Memory?, Chenyang Zhao, LinkedIn, 2026.
• Analyzing GPU Memory Leaks with Torch Memory Snapshot for VLM RL, Chenyang Zhao, LinkedIn, 2025.
• Reflecting on a Year of Building SGLang for Reinforcement Learning, Chenyang Zhao, LinkedIn, 2026.
• Agent Infrastructure Observation, Aaron Levie, X, 2026. Cited as an attributed industry viewpoint, not academic evidence.