Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generation

arXiv:2606.02479 · cs.CV · Submitted 2026-06-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Retrieve What's Missing".

Jane: Here's the gist of the paper. Maintaining long-term geometric consistency remains challenging for long-horizon autoregressive video generation The authors propose Coverage-Maximizing Retrieval (COVRAG), a depth-based memory retrieval framework that uses pretrained 3D priors to construct a target-view coverage map as lightweight 3D memory evidence This method improves long-horizon geometric consistency while maintaining low latency compared to baselines Pruning redundant reasoning tokens The paper addresses the limitations of existing memory-augmented video generation methods by focusing on two key design choices: what 3D-geometric evidence should represent past observations, and how memory frames should be selected from this evidence Existing methods often rely on camera poses or field-of-view overlap, which are lightweight but too coarse to reason about pixel-wise visibility, or use explicit 3D reconstruction, which provides fine-grained evidence but is costly to maintain over long rollouts COVRAG uses depth and camera poses predicted by a pretrained 3D prior model

21: to construct a target-view coverage map as lightweight 3D memory evidence This depth-based evidence provides a practical middle ground between FoV overlap and explicit 3D reconstruction, being sufficiently fine-grained for pixelwise visibility reasoning yet lightweight enough to avoid maintaining a globally reconstructed scene How the training signal is built The retrieval process in COVRAG involves three main components: constructing target-view coverage maps, selecting memory frames according to residual coverage gain, and introducing sliding-window depth caching for scalable geometry estimation The target-view coverage map indicates which target pixels are geometrically supported by a given source frame through back-projection using estimated depth and camera poses Frame selection is based on residual coverage gain, defined as gj = X v 1 - mctx t (v) mj→t(v), where this score counts the number of currently uncovered target-view pixels that would become covered if frame j were retrieved This iterative selection process encourages retrieved frames to be complementary rather than redundant, as it maximizes residual coverage gain Where the gains come from The primary gains stem from using depth-based target-view coverage maps as 3D-aware memory evidence, which enables pixel-level visibility reasoning without explicit 3D reconstruction The residual coverage gain criterion ensures that retrieved frames cover currently uncovered target-view regions, explicitly discouraging redundancy among retrieved memories Furthermore, the sliding-window depth caching strategy keeps geometry estimation efficient by estimating depth only within a fixed local window and reusing cached depth maps thereafter, keeping inference cost nearly constant as the video grows Experiments on RealEstate10K and DL3DV10K show that COVRAG improves long-horizon geometric consistency while maintaining low latency compared to baselines Pruning redundant reasoning tokens The authors demonstrate that target-view coverage provides more useful evidence for long-horizon revisitation than view-level overlap or explicit geometric memory construction, as COVRAG reduces MEt3R by 29% compared to WorldMem and VMem The method achieves the best generation quality, reducing FVD to 226.40 and FID to 23.00 on DL3DV10K The ablation study confirms that combining target-view coverage with residual gain achieves the best results, reducing MEt3R to 0.100 and LPIPS to 0.156 Where the gains come from COVRAG consistently outperforms WorldMem across all tested budgets and metrics, achieving strong performance even with n = 1 The qualitative results show that COVRAG better preserves scene geometry and appearance when revisiting previously observed views compared to WorldMem and VMem The scalability analysis shows that while both methods degrade as the number of generated frames increases, COVRAG remains better at every horizon COVRAG achieves an average latency of 8.70s over a 100-frame generation, which is comparable to WorldMem and 5.21× faster than VMem The method remains relatively stable under moderate perturbations to geometry estimation, suggesting depth-based target-view coverage can tolerate moderate local errors Where the gains come from The retrieval latency per step rises only from 0.98s to 1.19s, and the incremental cost remains about 21.2s/frame, consistent with the sliding-window and caching design This confirms that depth-based target-view coverage provides a practical middle ground between coarse but efficient FoV-based retrieval and fine-grained but costly explicit 3D memory The authors expect geometry-aware coverage-based retrieval to generalize better as models are exposed to richer scene variations and more diverse camera motion This work aims to improve long-term consistency in video generation models, benefiting applications such as 3D content creation and embodied simulation Pruning redundant reasoning tokens COVRAG depends on a pretrained geometry estimator to obtain depth maps for target-view coverage computation, and it does not directly modify the generative prior of the base video model Inaccurate depth estimates can still produce imperfect coverage maps and affect retrieval decisions, suggesting that integrating retrieval more tightly with the generator is a promising direction for future work The method is model-agnostic and can be integrated in a plugand-play manner, making it well-suited for large pretrained long video diffusion models trained on diverse datasets Responsible deployment and appropriate safeguards are necessary to mitigate risks associated with creating misleading visual content Where the gains come from The authors provide a controlled loop-closing evaluation on RealEstate10K, showing that COVRAG better preserves scene geometry and appearance across revisitation compared to WorldMem and VMem The results indicate that target-view coverage provides more informative memory evidence for long-horizon generation in diverse 3D scenes These findings support the central design principle of COVRAG, which is to use depth-based target-view coverage maps and residual coverage gain for retrieval The paper concludes that target-view coverage is an effective retrieval objective for 3D-consistent long-horizon generation Pruning redundant reasoning tokens The authors present Coverage-Maximizing Retrieval (COVRAG), a depth-based memory retrieval framework to improve geometric consistency in autoregressive video generation COVRAG constructs target-view coverage maps as lightweight 3D-aware evidence and selects memory frames by residual coverage gain, encouraging retrieved frames to provide complementary support for uncovered target-view regions This framework is shown to be effective in both controlled loop-closing settings and diverse camera trajectories on DL3DV10K The authors conclude that target-view coverage is an effective retrieval objective for 3D-consistent long-horizon generation The paper is available as arXiv:2606.02479v1

cs.CV: 1 Jun 2026 Where the gains come from The authors present Coverage-Maximizing Retrieval (COVRAG), a depth-based memory retrieval framework to improve geometric consistency in autoregressive video generation COVRAG constructs target-view coverage maps as lightweight 3D-aware evidence and selects memory frames by residual coverage gain, encouraging retrieved frames to provide complementary support for uncovered target-view regions This framework is shown to be effective in both controlled loop-closing settings and diverse camera trajectories on DL3DV10K The authors conclude that target-view coverage is an effective retrieval objective for 3D-consistent long-horizon generation The paper is available as arXiv:2606.02479v1 [cs.

Paper summary: Tom: Alright everyone, we're looking at this paper today called "Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generation". It tackles the problem of keeping a video consistent when you try to generate it over a long time.

Jane: So the big question here is how do we get that consistency back when the model revisits a part of the scene? The paper claims they solved this by focusing on two main things: what kind of three dimensional geometric information should we use as evidence from past observations, and how should we pick which past frames to bring back into the process.

Lu: Exactly. They look at existing methods that use camera poses or field of view overlap, which is lightweight but they can't reason about individual pixels being visible. Or they use explicit three dimensional reconstruction, which gives them fine detail but it gets too expensive to keep up over a long sequence like a video one <ref:2606.02479#pg1>.

Meng: I always worry about the cost. Explicit three dee reconstruction sounds powerful, but if you're trying to run this at scale for real applications, maintaining that level of detail across hundreds of frames seems like it could kill your inference speed two <ref:2606.02479#pg2>.

Tom: That’s where this paper tries to find a middle ground. They propose using depth and camera poses from a pre-trained three dimensional prior model to create something they call a target view coverage map. This acts as lightweight evidence that helps the system understand pixel visibility without needing a full, explicit three dee reconstruction three <ref:2606.02479#pg1>.

Jane: That sounds like it’s trying to give the memory framework some spatial awareness, right? How does this coverage map actually work in practice for choosing which frames to retrieve?

Lu: The retrieval process involves three main steps. First, they construct those target view coverage maps using the depth and camera poses to see which target pixels are supported by each historical frame through back projection three <ref:2606.02479#pg1>. Then, they select memory frames based on something called residual coverage gain.

Meng: Residual coverage gain sounds like a way to make sure we aren't just picking redundant frames. It’s about maximizing how much new area is covered when you add the next frame you retrieve, instead of just picking whatever looks most similar to the current view four <ref:2606.02479#pg2>.

Paper summary: Tom: Right, so if we pick Frame One and it covers part A, but Frame Two only covers part B that's already covered, the gain is low. If Frame Two covers C and D which are new targets, that gain is high. It encourages complementary memories rather than just repeating what you already have four <ref:2606.02479#pg2>.

Jane: So the core idea seems to be using this coverage map as a lightweight three dee-aware memory evidence and then using that gain score to pick frames that fill in the gaps in the scene <ref:2606.02479#pg1>. That’s pretty clever because it’s pixel level reasoning without having to reconstruct the whole scene from scratch three <ref:2606.02479#pg1>.

Lu: And they show results on datasets like RealEstate10K and DLthree deeV10K where they improve long horizon geometric consistency while keeping latency low compared to other methods like WorldMem and VMem one <ref:2606.02479#pg1>. They also claim this approach can reduce the number of memory frames needed, MEt3R, by twenty-nine percent compared to WorldMem two.

Meng: A twenty nine percent reduction in memory usage sounds promising for deployment. But how does that translate to a real application? Does it keep up with the generation speed when you're generating a hundred frames?

Tom: They show that COVRAG achieves an average latency of about eight point seven seconds over a one hundred frame generation, which is comparable to WorldMem but is five point two times faster than VMem one <ref:2606.02479#pg1>. And they also note that retrieval latency per step only goes up from zero point nine eight seconds to one point one nine seconds, with the cost remaining around twenty one point two seconds per frame due to their sliding window and caching strategy one <ref:2606.02479#pg1>.

Jane: So, even though they are using this coverage map for geometric reasoning, they’ve managed to keep the overall generation speed manageable. That keeps it practical for video generation tasks.

Lu: They also found that this depth-based target view coverage provides more useful evidence for long horizon revisitation than just view level overlap or explicit geometric memory construction two <ref:2606.02479#pg2>. The ablation study shows that combining target view coverage with residual gain gets them the best results, reducing MEt3R down to zero point one zero zero and LPIPS to zero point one five six two <ref:2606.02479#pg2>.

Paper summary: Meng: I'm interested in the dependency on that pre-trained three dimensional prior model. If your deployment environment doesn't have access to that prior model, does the whole system fall apart?

Tom: The paper is model agnostic in how it’s integrated, but they rely on a pretrained geometry estimator to get those depth maps for the coverage computation three <ref:2606.02479#pg1>. So if you don't have that specific prior, you need a separate depth estimation step, which introduces its own inaccuracies.

Jane: Those inaccuracies could definitely affect the quality of the coverage map and how good the frame selection decisions are. It’s a trade-off between using powerful priors and needing more robust local estimations three <ref:2606.02479#pg1>.

Lu: They acknowledge that inaccurate depth estimates can still produce imperfect coverage maps and influence the retrieval decisions, suggesting that integrating this retrieval more tightly with the generator might be where future work needs to focus three <ref:2606.02479#pg1>.

Meng: So, if I were deploying this right now for production, I'd need a reliable way to handle those local geometry errors without making the whole thing too slow or brittle. That’s a real concern for me at scale.

Tom: The authors do state that they expect geometry aware coverage based retrieval to generalize better when models see more different scene variations and camera motions one <ref:2606.02479#pg1>. They are aiming to improve long term consistency in things like three dee content creation and embodied simulation one <ref:2606.02479#pg1>.

Jane: So, the main point is that using this specific target view coverage objective, driven by residual gain, seems like a really effective way to build three dee-consistent long horizon generation <ref:2606.02479#pg1>. It’s proven in controlled loop closing experiments on RealEstate10K two <ref:2606.02479#pg2>.

Lu: It validates the central design principle: depth based target view coverage maps plus residual gain for retrieval is an effective objective for this kind of generation two <ref:2606.02479#pg2>. The paper concludes that this coverage maximizing retrieval is a viable way to achieve three dee consistent long horizon generation zero.

Tom: So, it’s not just about making the video look smooth; it’s about ensuring the underlying three dimensional structure stays coherent over a long sequence. That's what this research focuses on with Coverage-Maximizing Retrieval for Consistent Long Video Generation.

Conclusion: Tom: So we've been talking about this paper, "Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generation."

Jane: Right, so basically, they’re saying that keeping a video consistent when you generate it for a long time is hard.

Tom: Yeah, and the authors propose this framework called COVRAG. It uses depth information to build memory evidence that's more spatially aware than just camera angles or simple overlaps.

Jane: And they use this map, which they call a target-view coverage map, to decide which past frames are most useful for filling in the gaps in the future generation.

Tom: Exactly, it’s not just picking random old frames; it’s picking them based on maximizing what new area gets covered. The authors show that this helps maintain geometric consistency over long sequences.

Jane: So what does this actually change for someone watching a video made by an AI? Does it make the scene look more stable?

Tom: It does, especially in scenes with complex three dee structures because it forces the generation process to respect the underlying geometry more strictly. The results on datasets like RealEstate10K show that COVRAG preserves scene geometry and appearance better than previous methods we've looked at.

Jane: That’s a big deal if it means we can trust long-form video content to stay geometrically sound instead of just looking plausible moment by moment.

Tom: It does, but there are still caveats. The paper notes that the quality of this whole system really depends on the accuracy of the initial depth maps they use from that pre-trained prior model.

Jane: So if that initial depth estimation is off, the memory evidence gets messy, and that could impact how well the retrieval works. That’s a key limitation they point out.

Tom: Right, so it's a trade-off between using rich geometric priors and dealing with potential local estimation errors in real deployment. But overall, they prove that this coverage-based retrieval objective is effective for getting long horizon consistency in video generation.

Jane: And that means for future work, the focus might shift to making that depth estimation more robust or integrating the retrieval process even more tightly with the generative model itself.

Minseok Joo, Dogyun Park, Taehoon Lee, Kyujin Lee, Hyunwoo J. Kim

Korea University · KAIST

cs.CV

Submitted: 2026-06-01

Updated: 2026-11-07

Comments: Accepted at NeurIPS 2026. Project page: https://mlvlab.github.io/CovRAG

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Maintaining long-term geometric consistency remains challenging for long-horizon autoregressive video generation The authors propose Coverage-Maximizing Retrieval (COVRAG)…

Key concepts

Target-View Coverage Map
This map shows which pixels in the future video frame are geometrically supported by a specific past source frame. It is created using estimated depth and camera poses from a pretrained 3D model. This provides fine-grained, pixel-level evidence of what geometry is visible where, acting as lightweight 3D memory.
Residual Coverage Gain
This score measures how much new area would be covered in the target view if a specific past frame were retrieved. By selecting frames that maximize this gain, the method ensures retrieved memories are complementary rather than redundant, actively encouraging coverage of currently uncovered regions.
Sliding-Window Depth Caching
This strategy keeps geometry estimation efficient by only estimating depth within a small local window and reusing cached depth maps for subsequent frames. This significantly reduces inference cost over long video rollouts while maintaining sufficient geometric detail for retrieval decisions.

Terminology

Summary

Maintaining long-term geometric consistency remains challenging for long-horizon autoregressive video generation The authors propose Coverage-Maximizing Retrieval (COVRAG), a depth-based memory retrieval framework that uses pretrained 3D priors to construct a target-view coverage map as lightweight 3D memory evidence This method improves long-horizon geometric consistency while maintaining low latency compared to baselines

Pruning redundant reasoning tokens

The paper addresses the limitations of existing memory-augmented video generation methods by focusing on two key design choices: what 3D-geometric evidence should represent past observations, and how memory frames should be selected from this evidence Existing methods often rely on camera poses or field-of-view overlap, which are lightweight but too coarse to reason about pixel-wise visibility, or use explicit 3D reconstruction, which provides fine-grained evidence but is costly to maintain over long rollouts COVRAG uses depth and camera poses predicted by a pretrained 3D prior model [21] to construct a target-view coverage map as lightweight 3D memory evidence This depth-based evidence provides a practical middle ground between FoV overlap and explicit 3D reconstruction, being sufficiently fine-grained for pixelwise visibility reasoning yet lightweight enough to avoid maintaining a globally reconstructed scene

How the training signal is built

The retrieval process in COVRAG involves three main components: constructing target-view coverage maps, selecting memory frames according to residual coverage gain, and introducing sliding-window depth caching for scalable geometry estimation The target-view coverage map indicates which target pixels are geometrically supported by a given source frame through back-projection using estimated depth and camera poses Frame selection is based on residual coverage gain, defined as gj = X v 1 - mctx t (v) mj→t(v), where this score counts the number of currently uncovered target-view pixels that would become covered if frame j were retrieved This iterative selection process encourages retrieved frames to be complementary rather than redundant, as it maximizes residual coverage gain

Where the gains come from

The primary gains stem from using depth-based target-view coverage maps as 3D-aware memory evidence, which enables pixel-level visibility reasoning without explicit 3D reconstruction The residual coverage gain criterion ensures that retrieved frames cover currently uncovered target-view regions, explicitly discouraging redundancy among retrieved memories Furthermore, the sliding-window depth caching strategy keeps geometry estimation efficient by estimating depth only within a fixed local window and reusing cached depth maps thereafter, keeping inference cost nearly constant as the video grows Experiments on RealEstate10K and DL3DV10K show that COVRAG improves long-horizon geometric consistency while maintaining low latency compared to baselines

The authors demonstrate that target-view coverage provides more useful evidence for long-horizon revisitation than view-level overlap or explicit geometric memory construction, as COVRAG reduces MEt3R by 29% compared to WorldMem and VMem The method achieves the best generation quality, reducing FVD to 226.40 and FID to 23. p. 600 on DL3DV10K The ablation study confirms that combining target-view coverage with residual gain achieves the best results, reducing MEt3R to 0. p. 8100 and LPIPS to 0. p. 8156

COVRAG consistently outperforms WorldMem across all tested budgets and metrics, achieving strong performance even with n = 1 The qualitative results show that COVRAG better preserves scene geometry and appearance when revisiting previously observed views compared to WorldMem and VMem The scalability analysis shows that while both methods degrade as the number of generated frames increases, COVRAG remains better at every horizon COVRAG achieves an average latency of 8.70s over a 100-frame generation, which is comparable to WorldMem and 5. p. 821× faster than VMem The method remains relatively stable under moderate perturbations to geometry estimation, suggesting depth-based target-view coverage can tolerate moderate local errors

The retrieval latency per step rises only from 0.98s to 1.19s, and the incremental cost remains about 21. p. 22s/frame, consistent with the sliding-window and caching design This confirms that depth-based target-view coverage provides a practical middle ground between coarse but efficient FoV-based retrieval and fine-grained but costly explicit 3D memory The authors expect geometry-aware coverage-based retrieval to generalize better as models are exposed to richer scene variations and more diverse camera motion This work aims to improve long-term consistency in video generation models, benefiting applications such as 3D content creation and embodied simulation

COVRAG depends on a pretrained geometry estimator to obtain depth maps for target-view coverage computation, and it does not directly modify the generative prior of the base video model Inaccurate depth estimates can still produce imperfect coverage maps and affect retrieval decisions, suggesting that integrating retrieval more tightly with the generator is a promising direction for future work The method is model-agnostic and can be integrated in a plugand-play manner, making it well-suited for large pretrained long video diffusion models trained on diverse datasets Responsible deployment and appropriate safeguards are necessary to mitigate risks associated with creating misleading visual content

The authors provide a controlled loop-closing evaluation on RealEstate10K, showing that COVRAG better preserves scene geometry and appearance across revisitation compared to WorldMem and VMem The results indicate that target-view coverage provides more informative memory evidence for long-horizon generation in diverse 3D scenes These findings support the central design principle of COVRAG, which is to use depth-based target-view coverage maps and residual coverage gain for retrieval The paper concludes that target-view coverage is an effective retrieval objective for 3D-consistent long-horizon generation

The authors present Coverage-Maximizing Retrieval (COVRAG), a depth-based memory retrieval framework to improve geometric consistency in autoregressive video generation COVRAG constructs target-view coverage maps as lightweight 3D-aware evidence and selects memory frames by residual coverage gain, encouraging retrieved frames to provide complementary support for uncovered target-view regions This framework is shown to be effective in both controlled loop-closing settings and diverse camera trajectories on DL3DV10K The authors conclude that target-view coverage is an effective retrieval objective for 3D-consistent long-horizon generation The paper is available as arXiv:2606.02479v1 [cs. p. 0 p. 0CV] 1 Jun 2026

The authors present Coverage-Maximizing Retrieval (COVRAG), a depth-based memory retrieval framework to improve geometric consistency in autoregressive video generation COVRAG constructs target-view coverage maps as lightweight 3D-aware evidence and selects memory frames by residual coverage gain, encouraging retrieved frames to provide complementary support for uncovered target-view regions This framework is shown to be effective in both controlled loop-closing settings and diverse camera trajectories on DL3DV10K The authors conclude that target-view coverage is an effective retrieval objective for 3D-consistent long-horizon generation The paper is available as arXiv:2606.02479v1 [cs.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Coverage-Maximizing Retrieval (COVRAG) can maintain long-horizon geometric consistency by using target-view coverage maps as lightweight 3D memory evidence, which provides a practical middle ground between FoV overlap and explicit 3D reconstruction.

  2. The system can select memory frames by maximizing residual coverage gain, which encourages retrieved frames to be complementary rather than redundant, ensuring that the retrieved set collectively covers the target view instead of selecting top-n frames independently.

  3. Sliding-window depth caching allows for scalable geometry estimation, keeping inference cost nearly constant as the generated sequence grows by reusing cached depth maps within a fixed local window, thereby avoiding the growing overhead of explicit 3D reconstruction.

  4. The improved system can achieve superior long-horizon geometric consistency, reducing MEt3R by up to 29% compared to WorldMem and achieving the best performance across all consistency and quality metrics on RealEstate10K and DL3DV10K.

  5. The retrieval module is model-agnostic, allowing for a plugand play manner integration into large pretrained long video diffusion models, enabling geometry-aware coverage-based retrieval to generalize better as models are exposed to richer scene variations and more diverse camera motion.

Sources

Related papers