Streaming Autoregressive Video Generation

Streaming video generation begins when the output stops being a closed clip and becomes a running state. The model has to emit frames under latency, remember enough of its own past, accept controls while it is already moving, and keep generating from the distribution created by its previous choices.

The problem is stateful serving

The obvious baseline is to generate a short clip, feed the last few frames back in, and ask for the next clip. That can work for demos or bounded extensions. It is not sufficient as a definition of stable streaming generation.

The failure is not only visual quality. A repeated short-video system has to keep identity, geometry, motion, lighting, object layout, and causal state coherent after its own predictions become the new context. If a generated car drifts a few pixels, that drift is no longer only an output error. It becomes input state. If a face becomes slightly overexposed, the next chunk may treat that exposure as the true appearance. If an occluded object is not stored in a usable form, the model may reintroduce it with a different shape when it reappears.

A model can produce a long sequence by looping, fading, slowing down, or hiding motion. A useful streaming model has to maintain enough state for the future to remain responsive without becoming visually unrelated to the past.

The serving contract is stricter than "make the video longer":

• completed frames can be emitted without waiting for the whole future;
• the model can keep useful past state without reprocessing all past frames;
• new control signals can enter while generation is ongoing;
• sampling is cheap enough for low latency;
• training exposes the model to the distribution it will create at inference time.

Those requirements make causal, autoregressive, memory-aware systems a useful framing for streaming. They do not rule out all offline components or short-window refinements. Length still matters, but state is the more diagnostic abstraction.

How the field got here

The earlier long-video literature did not start with native streaming. A 2024 survey on long video generation grouped methods into two broad paradigms: divide-and-conquer systems that split a long video into manageable parts, and temporal autoregressive systems that extend generation step by step.

That taxonomy is still useful, but it hides a difference that matters for serving. Divide-and-conquer methods can create longer videos while remaining offline. They may stitch scenes, interpolate motion, extend a clip, or plan a sequence of short generations. Temporal autoregression looks closer to streaming, but many systems still operate as chunk-wise generators: make the next block, concatenate, then continue.

The chunk-wise long-video survey explains why this became attractive. Standard diffusion video models denoise a whole tensor of fixed resolution and length, so memory grows quickly and out-of-memory failures appear when length or resolution increases. Chunking is the practical escape hatch: reduce each subproblem to a short clip, then use inter-chunk conditioning to connect them.

The cost is that chunking inherits the weaknesses of short-clip generation. The model sees only a limited active window, the cache grows or gets truncated, and local errors become historical facts. That is the bridge from long-video extension to streaming autoregression: the field had to move from "generate many clips" to "serve a persistent causal state."

Offline diffusion and the chunking limit

Conventional video diffusion is strong because the model sees a fixed temporal window. Bidirectional attention lets early and late frames inform each other. The model can repeatedly process the full window and denoise all frames together.

That is a good fit for a closed clip. It is a poor fit for an online generator.

First, bidirectional attention has no natural notion of "this frame is final, emit it now." If the model keeps revising the full window, the output is a batch artifact, not a stream. Second, repeated full-window denoising gets more expensive as context grows. Third, standard bidirectional attention does not map cleanly onto KV-cache reuse, because future and past are mixed during denoising. Fourth, new controls arrive at awkward times: a camera turn, a text edit, a pose update, or a robot action may need to influence frames that are not yet generated without rewriting the entire clip.

This is the same serving constraint that made autoregressive language models practical: the system needs a causal state that can be cached, advanced, and queried again. Video makes the state heavier and less discrete. A wrong token in text is visible; a wrong latent in video can become camera motion, geometry, lighting, or identity.

Diffusion Forcing and the frontier

Diffusion Forcing is one of the cleanest conceptual bridges between autoregression and diffusion. Instead of assigning one global noise level to a whole sequence, it lets different tokens or frames have different noise levels. A useful shorthand is:

Here $z_i$ is the latent at temporal position $i$, $\tau_i$ is its noise level, and $\epsilon_i$ is noise. The useful move is not the notation. It is that the sequence can contain clean past, partially denoised present, and noisy future at the same time.

This creates the idea of a diagonal denoising frontier. Rather than denoising every frame uniformly, the model can progressively move a boundary across time. Frames behind the frontier are closer to final. Frames ahead of it remain uncertain. That is the shape a streaming generator wants: preserve completed history, work on the near future, and leave distant future flexible.

Diffusion Forcing addresses a temporal interface problem. It gives a causal model a way to mix stable history and uncertain future inside one sequence. It does not remove exposure bias, nor does it decide which history should remain in memory. Later methods target those gaps separately.

Causalizing the teacher

The next move is distillation. A strong bidirectional video diffusion model can produce good clips, but it is slow and non-causal. A streaming system wants a causal student that can generate incrementally, reuse KV cache, and sample in a few steps.

CausVid, formally From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, frames this as converting a bidirectional teacher into a fast autoregressive video diffusion model. The paper reports distilling a 50-step teacher into a 4-step generator and gives a single-GPU FPS number. I do not treat that as a leaderboard result, because runtime numbers across these papers often mix different resolutions, hardware, model sizes, and latency definitions. The useful idea is the teacher-student structure: use a strong offline model to supervise a causal model that is shaped for streaming.

That teacher-student setup is powerful but also delicate. A bidirectional teacher models a different conditional distribution from a causal student. The teacher can use future frames while shaping the present. The student cannot. Causal Forcing argues that directly transferring a bidirectional probability-flow ODE trajectory to a causal student is theoretically inconsistent, and it proposes using an autoregressive teacher trajectory for initialization instead.

Context Forcing exposes a related supervision mismatch. A long-context student may roll out for far longer than the short-window teacher that supervises it. If the teacher cannot see the long history, it cannot correct global dependencies. The proposed repair is to train with a long-context teacher and manage the growing context through slow-fast memory.

This is the architecture gap in plain terms: the teacher knows too much, too little, or the wrong kind of history. A streaming student has to live one direction at a time.

Self-Forcing and exposure bias

The next gap is distributional.

During teacher-forced training, the model sees real history:

During inference, it sees generated history:

For language, this mismatch appears as exposure bias. In video latents, the problem is often harsher because small continuous errors can become visual state. A slight color shift can turn into overexposure. A small geometry error can become a new wall boundary. A tiny identity drift can become the face the model now conditions on.

Self Forcing addresses that gap directly. Instead of training only on clean ground-truth history, the student performs autoregressive rollouts during training using its own generated past and rolling KV cache. The teacher still supplies distribution-level supervision, but the context belongs to the student. The model has to learn under the same kind of self-generated state that it will face at inference time.

This is easy to understate. Self-Forcing is more than an autoregressive loss. It forces the model to see its own failure modes while gradients and teacher supervision can still correct them. Long rollouts are expensive, so the method uses truncated or stochastic gradient propagation rather than fully backpropagating through every generated step. That is an engineering compromise, but it keeps the right distributional constraint: train on the kind of history the model will actually create.

Self-Forcing++ and the horizon mismatch

Self-generated context narrows the train-test distribution gap, but it does not guarantee long-horizon stability. A model may train on short rollouts and then be asked to generate for a minute or longer. Train-self/test-self alignment is not enough if training never reaches the point where failures appear.

Self-Forcing++ makes that temporal horizon gap explicit. The student rolls out a long autoregressive trajectory. Short windows are then sampled from later points in that trajectory. A short-horizon teacher provides local distribution correction on those windows. The teacher does not need to generate the entire long video. It only needs to say whether local windows sampled from a long student rollout remain on the plausible video manifold.

That separation of labor is the point. The student bears the long-horizon state problem. The teacher provides short-window correction. The paper reports minute-scale extension claims, including videos up to 4 minutes and 15 seconds, but the conceptual contribution is more general: long-horizon failures should be encountered during training, not discovered at demo time.

This changes how to read long-video demos. The useful question is not only "how many seconds can the model output?" It is "how far into a self-generated rollout did training place corrective signal?"

MilliVid and hierarchical temporal memory

After Self-Forcing++ and Context Forcing, there is another question that training alone cannot answer:

What should hundreds of frames of history look like inside the model?

The naive representation is uniform. If a latent video has $T$ frames and each frame uses an $H \times W$ token grid, context cost scales like:

That is a bad bargain. Recent frames need spatial detail because they constrain immediate motion, appearance, and continuity. Distant frames often need a different kind of information: room layout, camera path, object identity, scene revisitation, and rough geometry. Representing a frame from twenty seconds ago with the same spatial token resolution as the last frame is usually wasteful.

MilliVid targets this representation-granularity problem. It trains a hierarchical autoencoder with multiple latent levels. Fine levels retain ordinary spatial detail. Coarse levels compress a frame down to very few tokens. Each level can be decoded independently, and generation proceeds coarse to fine: first produce a long coarse plan, then refine the temporally relevant region into higher spatial detail.

The paper's own example shows why this matters. A 30-second, 20 fps video with a 16 by 16 latent grid per frame gives 600 x 16 x 16 = 153,600 tokens before the model has even decided which history matters. Simply retaining more frames does not guarantee the model can use them. It may be too expensive to attend to them, or the relevant signal may be buried among too many fine tokens.

MilliVid's LOOPCRAFT setting is useful because it separates video quality from world consistency. The task uses long Minecraft-style sequences with actions and pose metadata, then asks whether the model can preserve and revisit a world over long context. A video can look locally plausible while forgetting where the camera came from. Conversely, a model can preserve global structure while still being imperfect at fine texture. Those are not the same metric.

The distinction is important: packing more frames into a bounded context is not the same as changing what an old frame means. MilliVid argues that the representation itself should be hierarchical, and that coarse levels should be independently decodable so distant history can be stored at the right granularity.

MilliVid has clear limitations. Its hierarchical memory depends on a newly trained hierarchical autoencoder. That makes it harder to bolt onto existing large video models with fixed latent spaces such as WAN- or Hunyuan-style systems. The paper also reports extra rollout steps relative to simpler packing baselines. So MilliVid is not a replacement for Self-Forcing. It complements it.

Self-Forcing asks: does the model train on its own generated context?

Self-Forcing++ asks: does training reach the horizon where failures appear?

MilliVid asks: is the long-range context represented at a useful granularity?

A five-layer streaming framework

The field looks messy if every paper is treated as a separate recipe. A better reading is a five-layer system: each layer answers a different question about how a video stream remains causal, coherent, controllable, and measurable. Few-step sampling crosses all five layers; it is the serving budget that determines whether the stack can respond in time.

The layers are complementary. Self-Forcing changes the rollout distribution; MilliVid changes how history is represented; Causal Forcing changes the supervision path. None substitutes for the others. A system may be causally correct yet forget a revisited room, or retain rich memory yet fail to obey a new action.

KV cache as visual memory

In language serving, KV cache is often described as an inference optimization. In streaming video generation, it starts to look more like visual memory. It decides what parts of the generated world remain accessible.

The memory question splits into at least four parts.

Memory selection: which historical tokens should remain available?

Memory compression: how many bytes can each historical token consume?

Memory retrieval: can the model find relevant non-local history instead of only using the recent window?

Representation granularity: at what spatial and semantic resolution should historical information be stored?

Sliding-window attention answers selection crudely: keep the recent window and discard the rest. Attention sinks preserve a few early tokens so the model keeps stable anchors. Slow-fast memory separates recent dense state from slower persistent state. Sparse cache policies keep selected blocks.

Quant VideoGen treats KV cache as a system bottleneck. Its 2-bit KV-cache quantization is not a new world model, but it matters because memory pressure limits how much generated history can remain usable. Future Forcing attacks selection from another angle: estimate which KV entries future queries may need, rather than judging token importance only from the current or past query. LongLive-RAG moves toward retrieval by treating previously generated latents as a searchable history instead of an irreversible recent window.

VideoSSM points in a different direction again. It treats long video generation as a recurrent dynamical process with local window memory plus an evolving global state-space memory. That is closer to a persistent scene state than a pure attention cache, though it still has to prove itself under broad interactive controls.

These axes should not be collapsed. A future model might use retrieval to select relevant distant episodes, quantization to keep the cache affordable, hierarchical latents to store those episodes at multiple resolutions, and a recurrent or object-centric state outside the 2D latent grid. KV cache is useful, but it is still a token memory. It does not automatically know which object is behind a wall, which identity should persist after occlusion, or which camera pose corresponds to a revisited room.

Sampling is a cross-cutting serving budget

Even a perfectly causal memory system is not real-time if it needs dozens of denoising steps per chunk. Sampling is therefore not a sixth layer: it constrains how often the model can update causal state, train on self-generated rollouts, use long memory, and accept online controls.

Distribution matching distillation, consistency distillation, diagonal distillation, and adversarial losses are different answers to the same latency constraint. Diagonal Distillation is especially aligned with streaming because it spends more denoising compute where it matters early, then reduces steps for later chunks. One-Forcing pushes toward stable one-step autoregressive generation by combining a DMD-style objective with an auxiliary GAN loss. Causal Forcing++ targets frame-wise autoregression with one or two sampling steps. Ultra Flash makes the high-resolution real-time constraint explicit.

Again, the reported speed numbers should be read carefully. CausVid, Self Forcing, Diagonal Distillation, Causal Forcing++, One-Forcing, and Ultra Flash all report impressive runtime claims, but they do not define a single comparable benchmark. Resolution, hardware, model size, first-frame latency, inter-frame latency, denoising steps, and output frame-rate conventions all matter.

The stable conclusion is not that one number wins. Denoising-step count, time to first frame, inter-frame latency, and hardware belong beside quality and consistency metrics whenever a paper claims streaming behavior.

Online control and evaluation

Once a video generator can stream, the control interface changes. The model is no longer mapping a prompt to a clip. It is updating a latent world state:

Here $z$ is visual or latent world state, $a$ may be an action, and $c$ may be text, camera motion, pose, trajectory, or another control signal.

That formulation connects streaming video generation to interactive world models, embodied AI, simulators, and robot learning. But it should not be overstated. Current video generators do not become physically correct world models just because they generate long sequences. They may learn useful visual regularities without maintaining object permanence, contact physics, or reliable causal state.

The hard interface is online correction. A user edits the prompt. A camera path changes. A robot action updates. A human pose arrives from a tracker. An agent chooses a new action. The generator has to integrate that information without resetting the world.

VRAG is useful here because it names the two problems that keep reappearing: compounding error and insufficient memory. It adds action conditioning and retrieval-augmented state, then shows that naive long context or naive retrieval is not enough for video. CausalCine pushes the same issue into multi-shot narratives, where the stream has to cross shot boundaries, accept dynamic prompts, and route memory by content instead of temporal proximity. Directing the World makes the control problem more explicit by combining human-motion and camera-trajectory control in a fast autoregressive setup.

These papers are not all solving the same benchmark. Taken together, they motivate the fifth layer: a controlled serving loop. A generator may need online controls, persistent state, and latency-aware sampling rather than only a longer prompt-to-clip context. The matching evaluation question is whether a new condition is obeyed while identity, geometry, memory retrieval, and user-visible latency remain acceptable.

This is where long-video research and robot world-model research begin to touch. A robot does not need a beautiful movie of every possible future. It needs a state representation that preserves the parts of the world that matter for action. Streaming video research is valuable when it clarifies which visual state must persist, which details can be forgotten, and how errors can be corrected before they become policy inputs.

What remains unsolved

The next bottleneck is not longer context by itself. Context length helps only when the model can represent, retrieve, and use the relevant history within its latency and memory budget.

First, visual memory needs content addressability. A sliding window assumes the recent past is the most relevant past. That is often false. A room visited thirty seconds ago may matter more than the last four frames if the camera turns back. LongLive-RAG is an early version of this idea, but retrieval still needs better keys: object identity, scene layout, camera pose, action state, and semantic intent.

Second, the state representation is still too flat for many interactive tasks. KV cache stores token traces. Hierarchical latents store coarse-to-fine visual evidence. State-space memory stores a recurrent summary. None of these mechanisms alone guarantees an object- or scene-level state with occlusion, affordances, contact, and revisitation. A plausible systems direction is hybrid: token cache for local detail, hierarchical latents for long visual context, retrieval for non-local episodes, and explicit state for objects or geometry when action matters.

Third, control can destabilize rollout. A prompt edit, pose condition, camera path, or robot action changes the future distribution. The model has to preserve the old world while obeying the new control. If the control module simply overwrites the prior, the video may follow the instruction but lose identity or geometry. If the prior dominates, the interface feels unresponsive.

Fourth, evaluation is behind the systems. FVD and short-clip preference metrics do not measure whether a model revisits a room consistently after thirty seconds. Motion stability can be misread as success when the model has actually collapsed into repetitive loops. Runtime reporting needs the same discipline: time to first frame, inter-frame latency, throughput, resolution, denoising steps, and hardware should not be collapsed into one FPS number.

Finally, the layers need to compose. Causal rollout defines what can be emitted; Self-Forcing aligns training with generated histories; Causal Forcing targets supervision mismatch; MilliVid, VideoSSM, Quant VideoGen, Future Forcing, and LongLive-RAG define a memory budget and access path; action-conditioned models decide how new controls enter. Causal Forcing++, One-Forcing, Diagonal Distillation, and Ultra Flash constrain the latency budget across that stack. A practical system may need several of these at once, but the combination has to be evaluated as a system rather than inferred from isolated paper results.

Research outlook

The next generation of streaming video models should not be judged by whether they remember every pixel. The harder test is whether they preserve the state that makes the next frame belong to the same world.

That is the arc from long-video generation to streaming autoregression. The surveys framed the early divide between decomposing a long video and generating it temporally. Chunk-wise generation made the engineering pressure concrete. Diffusion Forcing gave the field a temporal denoising interface. Causal distillation made streaming plausible. Self-Forcing made generated history part of training. Self-Forcing++ pushed correction into late rollout states. MilliVid, VideoSSM, Quant VideoGen, Future Forcing, and LongLive-RAG showed that memory is a design space rather than a cache size.

The next step is not a single new trick. It is a five-layer serving architecture: causal interface, rollout-distribution alignment, teacher-student consistency, persistent memory, and online control with evaluation. Few-step sampling is the budget that makes the architecture responsive enough to use.

At that point, streaming video generation starts to look less like producing a longer movie and more like running a fragile simulator whose state is partly visual, partly latent, partly cached, and partly controlled. The hard question is not whether the model can generate another frame. It is whether that frame still comes from the same world state.

@article{ji2026streamingautoregressivevideogeneration,
  title = {Streaming Autoregressive Video Generation},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {June},
  url = {https://fusheng-ji.github.io/blog/posts/streaming-autoregressive-video-generation/}
}

References

• A Survey on Long Video Generation: Challenges, Methods, and Prospects, Chengxuan Li et al., 2024.
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion, Boyuan Chen et al., 2024.
• Towards Chunk-Wise Generation for Long Videos, Siyang Zhang and Ser-Nam Lim, 2024.
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, Tianwei Yin et al., 2024.
• VRAG: Learning World Models for Interactive Video Generation, Taiye Chen et al., 2025.
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, Tianwei Yin et al., 2025.
• Self-Forcing++: Towards Minute-Scale High-Quality Video Generation, Justin Cui et al., 2025.
• VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory, Yifei Yu et al., 2025.
• Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation, Hongzhou Zhu et al., 2026. Project page.
• Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization, Haocheng Xi et al., 2026.
• Context Forcing: Consistent Autoregressive Video Generation with Long Context, Shuo Chen et al., 2026.
• Streaming Autoregressive Video Generation via Diagonal Distillation, Jinxiu Liu et al., 2026.
• CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narratives, Yihao Meng et al., 2026.
• Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation, Min Zhao et al., 2026.
• One-Forcing: Towards Stable One-Step Autoregressive Video Generation, Jiaqi Feng et al., 2026.
• Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation, Jiayi Luo et al., 2026.
• LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation, Qixin Hu et al., 2026.
• MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation, Ishaan Preetam Chandratreya et al., 2026.
• Ultra Flash: Scaling Real-Time Streaming Video Generation to High Resolutions, Luxury et al., 2026.
• Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control, Haoyuan Wang et al., 2026.