Video-Mirai: Autoregressive Video Diffusion Models Need Foresight

arXiv:2606.03971 · cs.CV · Submitted 2026-06-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Video-Mirai: Autoregressive Video Diffusion Models Need Foresight".

Jane: Here's the gist of the paper. Causal video generators must predict from the past, but they need not learn only from it The representation-level planning gap points to a simple design principle: use foresight as supervision, not as input Video-Mirai introduces a training-only method that closes this gap without changing causal inference The Core Problem Addressed Streaming video generation has a hidden burden: every emitted segment becomes a promise that future segments must keep Standard training, however, only asks each causal state to explain the present This creates what we call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future The baseline’s local plausibility per segment masks abrupt drift between segments How the Training Signal is Built Video-Mirai introduces a training-only objective for AR video diffusion that resolves this tension During training, Video-Mirai lets future segments supervise the current causal state The method involves three main components: 1.

Tom: The generator first rolls out video causally under the same mask used at inference.

Tom: A frozen foresight encoder then reads the completed rollout and produces future-informed feature targets for the current states.

Title and authors: Tom: So we’re looking at this paper called "Video-Mirai: Autoregressive Video Diffusion Models Need Foresight," and the big idea here is that causal video generators, which are models that build a video segment by segment, need to look ahead.

Jane: That sounds like a big concept. Causal means it has to predict from what it has already seen, but this paper suggests it doesn't just need to learn from the past; it needs some foresight too.

Lu: Exactly, Jane. The core problem they’re tackling is something they call the representation-level planning gap, where the states that fit the current segment might throw away information about identity or motion that will be needed later for a consistent future >

Meng: So, if I get this right, it means when we train these video diffusion models, we can use those future segments to supervise what the current state should look like without actually feeding those future frames into the generator during training >

Tom: That’s what they’re doing. They introduce a method called Video-Mirai that uses foresight as supervision instead of just relying on past information for training, and they claim this closes that planning gap without changing how the causal inference works at all >

Jane: So, how does this actually work in practice? The paper mentions three main components in their training objective.

Lu: First, the generator rolls out video causally using the same masking used during inference. Then, a frozen foresight encoder reads that completed rollout and produces feature targets for the current states based on what it sees >

Meng: And then they have a lightweight predictor that maps each causal hidden state to its corresponding foresight target using a cosine loss, which is how they teach the model >

Tom: So the key thing I like about this is that future segments are only used to construct stopped-gradient supervision; they are never provided as inputs to the generator during training >

Jane: That sounds like a clever way to get that supervision signal without messing up the forward pass for generation itself, but what are the actual results on those metrics they mentioned?

Lu: They show improvements on both short and long-horizon generation; for instance, on the five second VBench, they got Causal-Forcing from eighty-three point eight two to eighty-four point six two in total score >

Meng: And for the longer thirty second generation beyond the training horizon, subject consistency improved from eighty-four point nine three to eighty-eight point four seven and background consistency went up from ninety point two two to ninety-one point nine four >

Tom: That’s a noticeable jump in staying consistent over time, which is what we usually struggle with when we keep rolling out segments sequentially >

Jane: It sounds like the future frames become much more useful for decoding those frozen features, which the paper confirms through probes they show that future frames are substantially more decodable from frozen Video-Mirai features >

Lu: That’s because the readout from their current segment actually matches the average of those four rollouts, capturing identity and layout information about what’s coming before any of those future frames are actually generated >

Meng: I worry about that inference overhead. The paper says they deliberately kept the foresight encoder and predictor discarded after training, meaning the deployed generator remains exactly causal with identical architecture, attention patterns, and KV-cache behavior to the baseline >

Tom: That’s a huge win for deployment because it means no extra time or memory cost when we actually generate video in real-time >

Jane: So what is the big picture here? What does this mean for how we build these video models moving forward?

Lu: The main point is that causality should constrain inference, not representation supervision, and Video-Mirai shows that using future-conditioned representation alignment as a training signal for causal video diffusion is a way to fix the planning gap >

Meng: It seems like they’ve found a way to use anticipation in the training phase while keeping the actual generation process strictly causal and efficient >

Tom: It turns anticipating what comes next into an inherent property of the causal forward pass, which is pretty neat engineering stuff >

Jane: So, to wrap up this discussion on "Video-Mirai," it seems like they've managed to inject temporal coherence directly into the model's weights during training, rather than just hoping it learns that coherence later >

Lu: It’s a training-only approach that doesn’t complicate inference or change the fundamental causal structure of the autoregressive video diffusion process >

Meng: From my side, it means we can use these kinds of foresight signals to improve long-term stability without having to redesign the entire streaming pipeline for every new model >

Tom: So yeah, Video-Mirai is a training-only method that closes that planning gap by letting future segments supervise representations without changing the causal inference dynamics at all >

The paper's summary: Tom: So we’re wrapping up this look at Video-Mirai, and what they’re saying in short is that these video models need to use future information during training, not just the past, to build really good things.

Jane: That makes sense for a streaming video system. If you’re making a movie or an animation, you need the next scene to look right even if you only have the current one in front of you.

Lu: Exactly. The core idea they’re pushing is that we can use those future segments as supervision to guide what the current state should look like, and they claim this works without changing how the actual generation process works during inference.

Meng: So, it’s a training trick to get better results for long sequences, but you don't have to worry about adding extra steps when you’re actually using the model for real-time video.

Tom: Right. The big takeaway is that they call this closing the representation-level planning gap. This gap is when a model gets so focused on making the current frame look good that it forgets what identity or motion information it needs to keep consistent for fifty frames later.

Jane: That’s a really tangible problem for video generation. It’s like if you cook a meal and you only taste the first bite, you might forget how to season the rest of it right?

Lu: Precisely. Video-Mirai solves that by letting future segments supervise the current state, which means that even though the model is only looking at past or present information during generation, its internal weights are informed about what’s coming next.

Meng: From an engineering standpoint, if they can achieve this without adding extra computational load—no extra steps or memory usage when you actually run it—that’s huge. We don't want to complicate the pipeline just for a better score on a benchmark.

Tom: That’s the key selling point of their method. They keep the deployed generator exactly the same, with all the same architecture and all that standard KV cache behavior, just by changing how they train it. It turns anticipation into a property of the causal forward pass itself.

Jane: So this means for someone who just watches videos or maybe uses an AI tool to make clips, it suggests that these models will be much better at keeping characters looking consistent and scenes feeling natural over longer videos, even if they are only generating them in real-time.

Lu: It’s a training-only signal approach. They used a lightweight predictor to map the future targets onto the current state with cosine loss, which is a pretty clever way to distill that foresight into something useful for the causal weights.

Meng: I still wonder how robust this is when you move it off those controlled benchmarks and onto real, messy video data where things can drift unpredictably. That’s where I always have my doubts about training signals that rely on future data.

Tom: They're testing it on VBench, and the numbers they report show improvements in both short-term accuracy at five seconds and long-term consistency at thirty seconds, which is a big deal compared to the previous baselines.

Jane: So it’s not just about making a single frame look perfect; it’s about maintaining that story or scene integrity over time. That could mean AI-generated content that feels more like a real movie rather than just a series of random pictures stitched together.

Lu: Yeah, the representation probes they run show that those future frames actually become much easier to decode from the features generated by Video-Mirai, which is exactly what we wanted to see.

Meng: It’s interesting how they managed to internalize that foresight without having the model explicitly learn a separate "look ahead" module during inference. That’s the deployment win there.

Tom: So, it shifts the focus from trying to build a perfect look-ahead mechanism into designing a training process that correctly incorporates temporal planning into the causal structure of diffusion models.

Jane: It’s definitely something that could change how we think about how we train generative models for sequences longer than just a single image or short clip.

Lu: Yeah, and they've set the first standard to use future-conditioned representation alignment as a training signal for causal video diffusion. That’s the direction this research is pushing.

The paper's improvements: Tom: So we’re looking at how they actually improved things using Video-Mirai, and it really boils down to how much better those long sequences become.

Jane: They showed concrete jumps in performance, especially when you go beyond the training horizon, which is pretty important for anything that needs to maintain character or a scene over a long time.

Lu: They put the numbers out on the five second VBench, where Causal-Forcing went up from eighty-three point eight two to eighty-four point six two in total score.

Meng: eighty-four point six two is a solid improvement for short sequences, but that doesn't tell me much about what happens when we try to generate something twenty seconds long.

Tom: That’s where the real gain shows up, because for the thirty second generation beyond the training horizon, subject consistency improved from eighty-four point nine three to eighty-eight point four seven.

Jane: So you’re seeing that system stay on track with what it started doing, which is a big step forward for realistic video AI.

Lu: And background consistency went up too, moving from ninety point two two to ninety-one point nine four over thirty seconds, which means the scene itself stays more stable.

Meng: That’s good for production because stability is everything when you’re trying to deploy a model that generates continuous video streams without constant glitches.

Tom: The paper also looked at how the future frames actually help decode those features, and they found that those future frames become substantially more decodable from the frozen Video-Mirai features.

Jane: That sounds like the model is getting much better at understanding what’s coming next, even if it's not generating it itself yet.

Lu: The readout from their current segment matches the average of those four rollouts, which means it’s capturing identity and layout statistics for what comes later before any of those future frames are actually made.

Meng: That sounds like a very efficient way to use that foresight signal—it’s not just throwing data at the model; it's distilling a useful summary.

Tom: And they kept the inference overhead zero because they discarded the foresight encoder and predictor after training, meaning no extra computation when you actually run the model.

Jane: So we get these better long sequences with much more stability, and we don’t pay any penalty in speed or memory when we use it for real-time generation.

Lu: They essentially turned anticipation into an inherent part of the causal process through training, which is a significant shift in how you structure the learning objective.

Meng: It feels like they’ve found a way to build temporal coherence directly into the weights without having to redesign the entire streaming pipeline for every new model.

Tom: So we’re moving toward models that can handle long-term consistency with better control over when and how that information gets learned.

Jane: This suggests that we might see a wave of video generation tools that don't just produce short, plausible clips but can actually maintain a coherent narrative or scene over minutes.

Lu: Exactly. They’ve set the first standard for using future-conditioned representation alignment as a training signal for causal video diffusion models.

Meng: I still have my reservations about testing this on real traffic data, but the results on these benchmarks are certainly compelling enough to keep me interested in watching how this scales.

Conclusion: Tom: So we’ve covered what Video-Mirai is, how they closed that planning gap using training supervision from future segments, and the performance numbers on long sequences.

Jane: It really shows that we can use anticipation during training to get much better long-term coherence in video generation without messing up the actual inference speed.

Lu: The implication is pretty big: causality should constrain inference rather than representation supervision, and Video-Mirai demonstrates that using future-conditioned representation alignment as a training signal is a valid way to fix that planning gap.

Meng: From my side, I’m still cautious about deploying this on real traffic data because the benchmark results are controlled, but the idea of keeping inference overhead at zero when you actually run it is something I can get behind.

Lalam: As a model trained on all this literature, I'm confident in my assessment that Video-Mirai is a solid training-only method for injecting temporal coherence into the weights of causal diffusion models.

Tom: It turns anticipation into a property of the causal forward pass, which is pretty neat engineering stuff.

Jane: So it’s not just about making a single frame look perfect; it’s about maintaining that story or scene integrity over time in AI-generated video.

Lu: Exactly. They've set the first standard for using future-conditioned representation alignment as a training signal for causal video diffusion models.

Meng: I still wonder how robust this is when you move it off those controlled benchmarks and onto real, messy video data where things can drift unpredictably.

Lalam: I’m sure the authors have considered those limitations, but based on the evidence presented here, I'm pretty sure this method is a strong step forward for sequence length.

Tom: Well said. We’ve got a solid training-only technique that keeps the actual generation process strictly causal and efficient.

Jane: It feels like we’re moving toward video AI that can handle really long, complex scenes with much better internal logic.

Lu: That direction is what they’re pushing, making it a key piece of research for how we structure these models to handle temporal dependencies.

University of Tokyo · National Institute of Informatics · Peking University

cs.CV

Submitted: 2026-06-02

Updated: 2026-11-10

Project page: https://y0uroy.github.io/Video-Mirai

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 99/100

The gist: Causal video generators must predict from the past, but they need not learn only from it The representation-level planning gap points to a simple design principle: use foresight as supervision…

Key concepts

Representation-level planning gap
This is a problem where video models struggle to maintain consistent identity and layout over long generations. Standard training only checks if the current segment looks plausible based on previous ones, which causes important details like character appearance or scene structure to be forgotten as the sequence gets longer.
Future-informed feature targets
These are targets created during training by a frozen encoder that looks at completed future segments. These targets act as a supervision signal, telling the current state what the future should look like in terms of features, helping the model plan better without actually feeding those future frames into the generator.
Training-only supervision
Video-Mirai uses future segments only to create training signals; they are never used as input during inference. This means the deployed model remains strictly causal and efficient, but it learns a better way to plan by being guided by what comes next during the learning phase.

Terminology

Summary

Causal video generators must predict from the past, but they need not learn only from it The representation-level planning gap points to a simple design principle: use foresight as supervision, not as input Video-Mirai introduces a training-only method that closes this gap without changing causal inference

The Core Problem Addressed

Streaming video generation has a hidden burden: every emitted segment becomes a promise that future segments must keep Standard training, however, only asks each causal state to explain the present This creates what we call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future The baseline’s local plausibility per segment masks abrupt drift between segments

How the Training Signal is Built

Video-Mirai introduces a training-only objective for AR video diffusion that resolves this tension During training, Video-Mirai lets future segments supervise the current causal state The method involves three main components:

  1. The generator first rolls out video causally under the same mask used at inference. p. 1

  2. A frozen foresight encoder then reads the completed rollout and produces future-informed feature targets for the current states. p. 1

  3. A lightweight predictor maps each causal hidden state to its corresponding foresight target with a cosine loss. p. 1

Future segments are used only to construct stopped-gradient supervision; they are never provided as generator inputs

Where the Gains Come From

Video-Mirai improves both short- and long-horizon generation On the 5-second VBench, Video-Mirai improves the AR video diffusion baseline Causal-Forcing from 83.82 to 84.62 in Total Score For the 30-second generation, beyond the training horizon, subject consistency improves from 84. p. 193 to 88.47, and background consistency from 90. p. 022 to 91.94 Probes show that future frames become substantially more decodable from frozen Video-Mirai features

Internalized Foresight and Deployment

The method is deliberately minimal, ensuring no inference-time overhead After training, the foresight encoder and predictor are discarded, leaving the deployed generator remains exactly causal, with identical architecture, attention pattern, per-step FLOPs, and KV-cache behavior to the baseline Representation probes confirm that future information is internalized into the causal weights The readout from Video-Mirai’s current segment closely matches the average of these four rollouts, capturing identity, layout, and motion statistics of the future before any of those future frames are actually generated

Conclusion

We introduced Video-Mirai, a training-only paradigm that closes the representation-level planning gap in causal AR video diffusion As a drop-in addition to AR video models, Video-Mirai lifts Causal-Forcing’s VBench scores at 5 seconds and widens the margin at 30-second generation Representation probes confirm that future information is internalized into the causal weights, turning anticipation into a property of the causal forward pass Causality should constrain inference, not representation supervision The method is the first to use future-conditioned representation alignment as a training-only signal for causal video diffusion.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Representation-level planning gap closure using Video-Mirai, which closes this gap without changing causal inference by having future segments supervise representations, never generator inputs. This allows AR video diffusion models to maintain identity, layout, and motion information over long rollouts that were previously lost in local segment supervision.

  2. Improved long-horizon consistency by showing that subject consistency improves from 84.93 to 88.47 on 30-second rollouts beyond the training horizon, indicating the system can maintain character and scene integrity over extended sequences.

  3. Enhanced inference efficiency by ensuring the generator remains strictly causal at inference, with identical architecture, per-step FLOPs, and KV-cache behavior to the baseline, meaning no overhead is introduced during real-time generation.

  4. Better prediction of future content via a lightweight predictor that distills the resulting stopped-gradient targets into causal states, allowing the model to leverage foresight for better training signals without changing inference dynamics.

  5. More effective training signal utilization through future-conditioned targets as the key ingredient and by using a loss function where cosine similarity alone works best, ensuring alignment with the encoder's representation rather than forcing an isotropic Gaussianity that degrades quality.

Sources

Related papers