FAR-Drive: Frame-AutoRegressive Video Generation for Autonomous Driving

arXiv:2603.14938 · cs.CV · Submitted 2026-03-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "FAR-Drive: Frame-AutoRegressive Video Generation for Autonomous Driving".

Paper summary: Tom: So, looking at the "FAR-Drive: Frame-AutoRegressive Video Generation for Autonomous Driving" paper, what we're seeing is a comprehensive framework that addresses the core problems of long-horizon consistency, autoregressive degradation, and low latency in autonomous driving simulation. Jane It’s really about moving away from open loop generation to something that can actively interact with its environment step by step.

Lu: The authors propose this multi view diffusion transformer setup coupled with their two stage training process to handle both the visual coherence over time and the error propagation issues inherent in sequential generation > Page ten. Meng What this means for deployment is that we might finally have simulation environments that are reliable enough to train complex driving AI systems without them drifting into physically impossible states.

Jane: It’s about giving the agent a predictable, consistent world to learn in, which is crucial because if the environment keeps changing randomly, you can't reliably test or improve the driving policy > Page three <ref:2603.14938#pg3>. Lalam The authors confirm that their results on the nuScenes dataset show state of the art performance across several visual and geometric metrics > Page eight.

Tom: In simple terms, they’ve made a system where you can give an AI action, it generates the next frame consistently across multiple views with low latency, and that's what makes this paper interesting for us as machine learning engineers > Page one <ref:2603.14938#pg1>. Jane It’s a lot of engineering working together to solve the tough constraints of real world autonomous driving simulation > Page two <ref:2603.14938#pg1>.

Conclusion: Tom: So we've been looking at FAR-Drive, this paper about frame level autoregressive video generation for autonomous driving, and honestly, the title itself is pretty descriptive of what they built > Jane

Jane: It really is. The authors are tackling these big headaches: making sure a generated video looks consistent over a long time and stopping those errors from just piling up as the AI keeps predicting future frames > Lu

Tom: Yeah, because in simulation, if the scene starts drifting or the car suddenly appears in a weird place on frame twenty-seven, it's useless for training real driving policies > Meng

Jane: The core idea is using this unified architecture and their two stage training to get that consistency right from the start without those annoying degradation errors we see in most sequential generation models > Lu

Tom: And they seem pretty confident about the results on nuScenes, showing state of the art across visual quality and geometric alignment metrics > Lalam

Jane: It shows that by explicitly modeling those structured control signals—the road layouts and bounding boxes—and enforcing this frame level consistency, you genuinely improve how good the driving simulation environment is > Lu

Tom: It's interesting because they also focused on keeping the inference fast, so it’s not just a pretty picture; they managed to get sub second latency on a single GPU > Meng

Jane: And that’s what I find really important for deployment. If you can generate these consistent driving scenes quickly, it opens up ways for AI systems to learn more reliably in simulation > Lalam

Tom: It’s a lot of work getting all those pieces—the multi view transformer, the dual control branch—to talk to each other correctly > Lu

Jane: Exactly. And I think what they've done here is create a tool that makes the simulation environment itself more robust for training complex autonomous agents > Tom

Yaoru Li, Federico Landi, Marco Godi, Xin Jin, Ruiju Fu, Yufei Ma, Muyang Sun, Heyu Si

Zhejiang University · Huawei

cs.CV

Submitted: 2026-03-16

Updated: 2026-11-17

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 99/100

The gist: FAR-Drive proposes a frame-level autoregressive video generation framework for autonomous driving that addresses long-horizon consistency, autoregressive degradation…

Key concepts

Long-Horizon Consistency
This refers to maintaining visual coherence across many consecutive frames in a generated video. In autonomous driving simulation, if this is poor, the scene layout can drift or cross-view correspondences might break over time. FAR-Drive addresses this by conditioning the model on preceding reference frames.
Autoregressive Degradation
This is the problem where errors made in predicting an early frame negatively affect all subsequent predicted frames. It causes a mismatch between training data and real-world testing (exposure bias). The paper uses a two-stage training strategy to mitigate this degradation by gradually blending ground truth with model predictions.
Dual-MMDiT Architecture
This is the core backbone of the framework, consisting of two interconnected Multi-Modal Diffusion Transformers. One branch handles visual generation, while the other processes external control inputs like road layouts and bounding boxes. This dual design allows for precise structural conditioning of the generated video.
Blend-Forcing Training
This is Stage II of the training process that improves autoregressive quality. It uses a blending coefficient to gradually shift the training objective from relying entirely on ground-truth frames to relying on the model's own generated conditioning, leading to more robust and consistent outputs.

Terminology

Summary

FAR-Drive proposes a frame-level autoregressive video generation framework for autonomous driving that addresses long-horizon consistency, autoregressive degradation, and low-latency inference constraints through a unified architecture and two-stage training strategy<ref:2603. p. 414938#pg16>.

Problem Statement

The core challenge addressed is the lack of scalable and interactive simulation environments for autonomous driving systems<ref:2603.14938#pg3>. Frame-level autoregressive rollout introduces two tightly coupled challenges: long-horizon consistency and autoregressive degradation<ref:2603. p. 214938#pg4>. Long-horizon consistency is a prerequisite for reliable simulation because minor inconsistencies can accumulate over time, leading to drifting scene layouts or broken cross-view correspondences<ref:2603. p. 214938#pg5>. Autoregressive generation suffers from iterative degradation where errors in early predicted frames propagate through conditioning, causing a train–test distribution mismatch known as exposure bias<ref:2603. p. 214938#pg6>. Furthermore, practical deployment imposes low-latency constraints because the simulated environment must evolve synchronously with agent actions<ref:2603. p. 214938#pg7>.

FAR-Drive Architecture

The proposed framework, FAR-Drive, introduces a multi-view diffusion transformer with fine-grained structured control to enable geometrically consistent multi-camera generation<ref:2603.14938#pg8>. The architecture consists of a multiview causal multi-modal diffusion transformer (MMDiT) backbone for coherent visual generation<ref:2603. p. 414938#pg9>. This backbone is augmented with a control-aware architecture designed to inject structured driving constraints<ref:2603. p. 414938#pg10>. Specifically, the model utilizes a dual-MMDiT architecture where the control branch processes external control inputs like road layouts and bounding boxes<ref:2603. p. 614938#pg8>. This dual-branch design allows for precise structural conditioning by injecting the control signal into the backbone layer-wise through zero-initialized projection layers<ref:2603. p. 614938#pg10>.

Training Strategy

The training process employs a two-stage strategy to jointly improve long-horizon consistency and mitigate autoregressive degradation<ref:2603.14938#pg8>. Stage I is Adaptive Reference-Horizon Conditioning (ARHC), which enhances temporal consistency by conditioning the model on a variable number of preceding reference frames, with higher probability given to short horizons<ref:2603. p. 714938#pg10>. Stage II is Blend-Forcing Autoregressive Training, a hybrid strategy that gradually transitions from ground-truth conditioning to model-generated conditioning<ref:2603. p. 714938#pg8>. This transition is controlled by a blending coefficient α that increases linearly over time, and the training objective minimizes a flow matching loss defined over the blended latent sequence<ref:2603. p. 714938#pg10>.

Inference Optimizations

To meet low-latency interaction requirements, several engineering optimizations are integrated into the inference process<ref:2603.14938#pg8>. These include:

  1. Sampling Step Reduction, where a reduced number of sampling steps is employed during inference after blend-forcing training to substantially reduce latency<ref:2603. p. 814938#pg8>.

  2. Frame-Level VAE Design, which uses a VAE architecture with spatial-only compression to support frame-level controllability while preserving inference throughput<ref:2603. p. 814938#pg8>.

  3. Latent Caching for Reference Frames, which utilizes a KV-cache-style mechanism to reuse attention keys and values from cached reference frames, avoiding redundant feature encoding<ref:2603. p. 914938#pg8>.

  4. Condition Caching Across Diffusion Steps, where structured control inputs are encoded once and cached across all diffusion steps to reduce overhead<ref:2603. p. 914938#pg8>.

Experimental Results

Experiments on the nuScenes dataset demonstrate that FAR-Drive achieves state-of-the-art performance among existing closed-loop autonomous driving simulation approaches<ref:2603.14938#pg8>. The method consistently outperforms competitors across Fréchet Inception Distance (FID), Fréchet Video Distance (FVD), mean Average Precision (mAP), and mean Intersectionover-Union (mIoU)<ref:2603. p. 014938#pg8>. Notably, the results validate that explicitly modeling structured control signals and enforcing frame-level consistency significantly improves both visual-temporal quality and geometric alignment<ref:2603. p. 414938#pg8>. The framework supports flexible quality–efficiency trade-offs, achieving sub-second latency on a single GPU under typical configurations<ref:2603. p. 014938#pg8>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved long-horizon consistency in simulation allows for reliable training of driving systems by maintaining temporally coherent and geometrically aligned multi-view observations over extended horizons.

  2. Mitigated autoregressive degradation ensures that errors do not compound over time, leading to a model that can maintain stable scene layout, object integrity, and multiview coherence throughout the entire sequence.

  3. Enhanced geometric consistency enables the system to generate geometrically consistent multi-camera generation by explicitly projecting BEV road layout and 3D object bounding boxes into each camera view based on known camera intrinsics and extrinsics.

  4. Low latency inference constraints are satisfied by integrating optimizations like a Frame-Level VAE Design (1x32x32 compression) and Latent Caching for Reference Frames (KV-cache mechanism), enabling sub-second latency on a single GPU.

  5. The system can perform frame-level autoregressive rollout, meaning it can generate future observations step by step based on agent actions, which enables the creation of an interactive rollout manner.

Sources

Related papers