PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

arXiv:2609.06948 · cs.CV · Submitted 2026-09-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis".

Jane: Here's the gist of the paper. Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation <ref:2609.06948#pg2> The paper proposes PRG-Fusion, a framework that uses reconstruction evidence to orchestrate generative priors across regions <ref:2609.06948#pg4>.

Tom: Core Problem Addressed The challenge lies in balancing the strengths of reconstruction-based methods, which provide geometric consistency, and generative models, which offer flexibility along arbitrary trajectories <ref:2609.06948#pg2> Specifically, when the target trajectory deviates from the recorded trajectory, 3DGS renderings exhibit spatially varying degradation ranging from high fidelity to artifacts or missing content <ref:2609.06948#pg2> Conversely, generative models often lack stable 3D geometric consistency across frames, leading to temporal drift in scene structures <ref:2609.06948#pg2>.

Jane: The PRG Framework PRG-Fusion introduces a region-aware video generation framework that explicitly formulates the spatially varying reliability of off-trajectory 3DGS renderings through pixel-wise Preserve, Repair, and Generate (PRG) labels <ref:2609.06948#pg2> These PRG labels serve as a unified routing policy for region-aware spatiotemporal synthesis <ref:2609.06948#pg2>.

Tom: The framework coordinates 3DGS appearance preservation, LiDAR-guided structural correction, and video-prior-driven content completion across Preserve, Repair, and Generate regions, respectively <ref:2609.06948#pg2>.

Title and authors: Tom: So we're talking about this paper now called PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis. Jane, can you tell us what this is trying to do in plain English?

Jane: Well, it’s tackling a big problem where you want to make realistic driving videos along a path that wasn't in your original training data. The main issue is that the existing methods have trade-offs: reconstruction methods give you geometric consistency but they mess up when you go off-path, and generative models can follow any path but they lose their sense of three dee structure over time <ref:2609.06948#pg2>. PRG-Fusion tries to fix both by using evidence from the original reconstructions to decide what kind of information—visual appearance, geometric shape, or just raw generation—to use in different parts of the scene.

Lu: Exactly. The core idea is that we don't treat all parts of the novel view equally when we go off-trajectory. We break the view down into three zones: Preserve, Repair, and Generate—those are your PRG labels—and those labels act like a set of instructions for how to combine the three dee Gaussian Splatting appearance, the LiDAR geometry, and that video generative prior <ref:2609.06948#pg2>.

Meng: That sounds complicated. So if I'm deploying this at scale, what does that mean for my production pipeline? Does it just make the training longer or more expensive?

Tom: It means we have a much smarter way to use our data inputs. Instead of blindly using everything, the system decides pixel by pixel whether it should trust the dense three deeGS appearance, reinforce it with LiDAR structure, or let the generative model take over <ref:2609.06948#pg2>. That adaptive routing policy is what they call PRG-Fusion.

Jane: Let's dig into how they figure out those labels. How do we get that pixel-wise Preserve, Repair, Generate map?

Lu: They derive this by looking at reconstruction evidence. First, they quantify trajectory unreliability using something called Expected Information Gain to see how sensitive the parameters of each Gaussian are when you move from the training views to your target trajectory; that gives them a measure of how much trouble a specific part is.

Tom: And then they combine that with observation and geometric support, which involves checking visibility frequency and directional coverage for observation support, along with depth consistency across all the training views to gauge how well that spot is supported geometrically.

Meng: So it’s like a local reliability score? If a patch looks really weird or unsupported when you look at it from a new angle, it gets labeled differently?

Title and authors: Jane: Precisely. If the unreliability and lack of geometric support are high, the label shifts toward Repair or Generate; if everything is solid, it goes to Preserve. This entire process creates that pixel-wise map that acts as a unified routing policy for the synthesis.

Lu: And then they use that map to control two separate branches: a visual branch based on three deeGS appearance and a geometry branch based on LiDAR projections <ref:2609.06948#pg2>. The PRG labels modulate those signals, allowing them to adapt which guidance—visual or geometric—is most useful at any given spot in the video.

Tom: That dual-branch setup is interesting because it prevents one source from completely dominating. How do they train these branches together?

Lu: They use a two-stage progressive training strategy. In the first stage, they optimize the geometry branch using only the sparse LiDAR projections so those observations get encoded into hierarchical geometric control features.

Jane: And then in the second stage, they remove that initial geometry input and freeze it, while optimizing just the visual branch under joint conditioning from both three deeGS renderings and LiDAR projections <ref:2609.06948#pg2>. That step is crucial because it teaches the visual side to rely on its own learned appearance rather than being overly reliant on the sparse geometry data.

Meng: From a deployment standpoint, that two-stage process sounds like it’s designed to stabilize things early on, which is good for production stability. But what happens if we need to change the scene structure drastically? Does this framework handle that?

Tom: It handles it by having a mechanism where regions with insufficient reconstruction support are granted greater freedom to the video generative prior, allowing flexibility when the data is missing. It’s about balancing control and freedom based on local evidence.

Jane: So, despite all these sophisticated routing policies and dual controls, what are the actual results they show? What does this mean for someone who just wants a better driving simulator?

Lu: They show state-of-the-art overall performance on novel trajectory video synthesis when tested on the Waymo Open Dataset. Specifically, PRG-Fusion achieves superior visual quality and geometric fidelity while maintaining competitive view consistency even under large trajectory shifts, like three meters or six meters lateral shifts.

Tom: Superior quality is good, but the metrics matter most. They report better FVD and CLIP-V scores compared to competing methods when you look at how well it preserves scene content and semantic consistency across those different viewpoints.

Meng: So, in simple terms for me, this means we can synthesize much more realistic driving scenarios that look consistent across many different camera angles without needing a massive amount of perfectly aligned training data for every single path.

Title and authors: Jane: That’s the practical implication. It moves us closer to scalable closed-loop simulation because we get better quality video synthesis from less perfect, off-path reconstructions.

Lu: And looking at the ablation study, they confirm that this PRG-guided fusion gets the best scores for metrics like NTA and NTL IoU compared to all other variants tested. They show that the real gains come from adapting how much visual and geometric guidance each region gets, not just combining them blindly.

Tom: I always love seeing an ablation study show exactly which part of a mechanism is doing the heavy lifting. So, it seems the region-aware control is definitely where the power lies here.

Jane: It does seem to be. It turns what used to be a uniform input into something that’s tailored to the specific local conditions of every single pixel in that video.

Meng: If we were building this for production, I'd worry about the overhead of extracting all those PRG labels per frame; how much processing time is added there compared to just using one method?

Tom: That’s a fair concern. The paper admits that this current framework still relies on scene-specific three deeGS reconstruction and LiDAR geometry, which introduces additional preprocessing overhead before the synthesis starts <ref:2609.06948#pg2>.

Jane: So the authors are acknowledging that while it’s powerful, it's not completely sensor-free yet. What about future directions? Where does this research go next?

Lu: They point out that for future work, they should explore feed-forward reconstruction models and predicted depth as alternatives to improve scalability and reduce dependence on the LiDAR geometry input.

Meng: That makes sense; relying on sparse LiDAR projections at scale is a bottleneck. If we could predict depth better, that would be a huge win for reducing the reliance on those specific geometric sensors.

Tom: So, to wrap up this discussion on PRG-Fusion: it’s a framework that uses reconstruction evidence to create pixel-wise Preserve, Repair, and Generate labels to adaptively coordinate three deeGS appearance, LiDAR geometry, and video priors based on local reconstruction states <ref:2609.06948#pg2,3DGS appearance, LiDAR geometry, and>.

Jane: It’s a sophisticated way of blending the strengths of geometric consistency and generative flexibility for driving view synthesis.

Lu: And their two-stage progressive training strategy helps ensure that the geometry branch gets stable control before it starts optimizing the appearance branch under joint conditioning.

Meng: For me, it’s a solid framework, but we need to keep watching those future work suggestions about making it less dependent on dense scene reconstruction data.

Tom: That’s right. PRG-Fusion is definitely a strong piece of work for scaling simulation, and we’ll be keeping an eye on those next steps.

The paper's summary: Tom: So, PRG-Fusion is basically an approach that uses clues from how the scene was reconstructed to decide which type of generation to use in different parts of a video when you change your viewpoint.

Jane: Right, it’s this idea that instead of treating every pixel the same when you look at something new, the system looks at what kind of evidence supports it—whether it's a clear visual image from the original reconstruction or just sparse geometric data from LiDAR.

Lu: The framework partitions your novel view into three zones: Preserve, Repair, and Generate—these are called PRG labels. These labels become a routing policy that tells the system how to blend the dense three dee Gaussian Splatting appearance, the sparse LiDAR geometry, and those video generative priors.

Meng: So for me at scale, it means we get to be much smarter about mixing inputs; we aren't just feeding everything into one big box.

Tom: Exactly. And they use a dual-branch control architecture where one branch handles the visual appearance from the three deeGS, and the other handles the geometry from LiDAR projections.

Jane: But those two branches don't just work together; their blending is controlled by those PRG labels that change depending on whether a spot is reliable or degraded.

Lu: And to get those labels, they extract evidence from the original reconstruction. They use something called Expected Information Gain to see how unreliable a specific Gaussian is when you look at your target trajectory.

Tom: That sounds like they are measuring the sensitivity of the model parameters as you move away from the training data.

Jane: And they combine that with observation support, checking how much visible and directional coverage a spot has, plus consistency across all the training views to gauge geometric support.

Lu: So they get a pixel-wise map where every spot is labeled Preserve, Repair, or Generate based on its local unreliability and geometric support.

Meng: That’s really interesting—turning raw reconstruction data into actionable routing instructions for the generation process.

Tom: And then they train this whole thing in two stages. First, they get the geometry branch stable using only the LiDAR projections, then they fine-tune the visual branch under joint conditioning from both three deeGS and LiDAR once that’s set up.

Jane: The results on Waymo are pretty strong; they claim state of the art performance in novel trajectory video synthesis. They show superior visual quality and geometric fidelity even when you shift your viewpoint by three or six meters laterally.

Lu: They also report better metrics like FVD and CLIP-V, which means the scene content and semantics are preserved better across those different viewpoints.

Tom: So, the core idea is that this adaptive routing based on reconstruction evidence allows the system to handle large trajectory shifts much more gracefully than previous methods.

Jane: It's about making sure that when you go off-path, you don't just get garbage; you get a scene that still looks and feels right because the system knows exactly what it can trust in every single spot.

Lu: And the ablation study backs this up; they show that adapting how much visual versus geometric guidance each region gets based on those PRG states is what really drives the best scores, not just combining them blindly.

Tom: So, while it's a bit more complex with all these labels and branches, the payoff seems to be a much more robust way to synthesize realistic driving videos for simulation.

Jane: It definitely shifts the focus from just building one perfect reconstruction to building a system that understands where its data is good and where it needs to rely on generative creativity.

Lu: And looking ahead, they suggest exploring feed-forward reconstruction models and predicted depth as ways to make this approach even more scalable and less dependent on the LiDAR geometry input.

Tom: So, the next big question for this research is how we can make this whole system work better without needing that heavy preprocessing step of building a perfect three deeGS model first.

The paper's improvements: Tom: So, PRG-Fusion isn't just a single method; it’s an entire strategy where they adapt how much visual guidance versus geometric guidance each part of the scene gets based on local reconstruction states.

Jane: Right, so if you’re looking at a spot that looks super clear from the original data, it gets to keep that dense appearance from the three deeGS.

Lu: But if that spot is off-trajectory and looks weird or unsupported, it shifts to needing more geometric guidance from LiDAR projections.

Meng: It's like they’re setting up a dynamic switch: visual guidance for things that look good, and geometric structure for things that are only partially visible or degraded.

Tom: They use this pixel-wise Preserve, Repair, Generate labeling system to explicitly encode that routing policy using a fixed matrix called lambda.

Jane: It’s smart because it takes the complex evidence from the reconstruction—like how unreliable a specific Gaussian is—and turns it into something simple that tells the generation process exactly what to prioritize at every single pixel location.

Lu: That’s what they call orchestrating generative priors across regions, meaning they are coordinating the visual appearance preservation, LiDAR-guided structural correction, and video prior content completion simultaneously.

Tom: And they do this with a dual-branch control architecture where the visual branch gets conditioned on the three deeGS and the geometry branch gets conditioned on those sparse LiDAR projections.

Jane: The way they train it is progressive; they establish the geometry control first using only LiDAR, and then they train the visual branch under joint conditioning from both sources.

Meng: That two-stage training is smart for production because it lets them get a solid foundation with the geometry before trying to optimize everything together.

Lu: And in terms of results, they’re seeing superior view consistency under large trajectory shifts compared to other methods, achieving the best metrics like NTA and NTL IoU.

Tom: So, the implication for simulation is that you can synthesize much higher quality driving videos that look consistent even when you take the virtual camera far away from where you trained it.

Jane: It means we’re moving closer to a simulator where the visual fidelity doesn't totally collapse when we extrapolate beyond our original training views.

Lu: The limitation they mention is that this current framework still relies on those detailed scene-specific three deeGS reconstructions and LiDAR geometry, which adds some preprocessing overhead.

Meng: That’s the practical catch; it’s powerful, but it's not sensor-free yet because you still have to build that initial detailed three dee model first.

Tom: So they are looking toward future work where they want to explore feed-forward reconstruction models and predicted depth as alternatives to reduce that dependency on the explicit three deeGS geometry input.

Jane: That would be a big step for scalability, because predicting depth or using a different reconstruction method could make this framework much faster and more sensor-agnostic.

Conclusion: Tom: So, to wrap up PRG-Fusion, it’s about using reconstruction evidence to guide the synthesis process by assigning Preserve, Repair, or Generate labels based on local reliability and geometric support.

Jane: Exactly; it’s a sophisticated way to blend dense appearance and sparse geometry using that adaptive routing policy across three different regions of the scene.

Lu: The implication is that we can achieve better view consistency under big trajectory shifts because the system knows exactly what information to trust locally at every pixel location.

Meng: For us shipping models, it means our output won't degrade as much when we extrapolate the driving path outside the training data boundaries.

Tom: It’s a way to make the synthesis robust by not treating all parts of the novel view uniformly degraded inputs.

Jane: They showed strong results on Waymo data, proving that this adaptive coordination actually leads to superior visual quality and geometric fidelity compared to existing methods.

Lu: The core idea they're pushing is that the gains come from adapting how much visual and geometric guidance each region gets based on those reconstruction states, not just combining them blindly.

Tom: They are pushing toward a more intelligent system where the control signals are modulated by the local reconstruction state rather than being fixed inputs.

Jane: It’s a great way to think about it; it’s like giving the AI a map that tells it, "In this area, trust your eyes; over there, trust your ruler."

Meng: I just wonder about the cost of running that system at scale now—does all that evidence extraction and routing add too much latency to a real-time deployment?

Tom: That’s a fair question. The authors did admit that this current framework still relies on detailed scene-specific three deeGS reconstructions and LiDAR geometry, which adds some preprocessing overhead before the synthesis starts.

Jane: So the next big thing they are looking at is feed-forward reconstruction models and predicted depth to try and get rid of that heavy upfront reconstruction step.

Lu: That would be a huge improvement for scalability because it means we could potentially reduce dependence on those explicit LiDAR geometry inputs over time.

Meng: If they can simplify the input data requirements, it makes deployment much more feasible for real-world applications instead of just high-fidelity research environments.

Tom: So, to sum up PRG-Fusion: it’s a framework that uses reconstruction evidence to create pixel-wise Preserve, Repair, or Generate labels based on local reliability and geometric support.

Jane: It’s a sophisticated way to blend dense appearance and sparse geometry using that adaptive routing policy across three different regions of the scene.

Lu: The implication is that we can achieve better view consistency under big trajectory shifts because the system knows exactly what information to trust locally at every pixel location.

Tom: It’s a way to make the synthesis robust by not treating all parts of the novel view uniformly degraded inputs.

Jane: They showed strong results on Waymo data, proving that this adaptive coordination actually leads to superior visual quality and geometric fidelity compared to existing methods.

Lu: The core idea they're pushing is that the gains come from adapting how much visual and geometric guidance each region gets based on those reconstruction states, not just combining them blindly.

Tom: They are pushing toward a more intelligent system where the control signals are modulated by the local reconstruction state rather than being fixed inputs.

Jane: It’s a great way to think about it; it’s like giving the AI a map that tells it, "In this area, trust your eyes; over there, trust your ruler."

Meng: I just wonder about the cost of running that system at scale now—does all that evidence extraction and routing add too much latency to a real-time deployment?

Tom: That’s a fair question. The authors did admit that this current framework still relies on detailed scene-specific three deeGS reconstructions and LiDAR geometry, which adds some preprocessing overhead before the synthesis starts.

Jane: So the next big thing they are looking at is feed-forward reconstruction models and predicted depth to try and get rid of that heavy upfront reconstruction step.

University of Science and Technology of China

cs.CV

Submitted: 2026-09-07

Updated: 2026-11-14

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation <ref:2609.06948#pg2> The paper proposes PRG-Fusion…

Key concepts

Reconstruction Evidence
This is the data derived from a 3DGS scene when viewed from a target trajectory that differs from the original training data. It is analyzed to determine how reliable or unreliably the rendered pixels are at specific locations, providing a map of local scene quality.
Preserve, Repair, Generate (PRG) Labels
These are pixel-wise labels assigned to every point in the novel view. 'Preserve' regions keep the high-quality 3DGS appearance; 'Repair' regions use LiDAR geometry to fix structural issues; and 'Generate' regions rely more on the video generative model for content.
Dual-Branch Control Architecture
The system uses two separate control pathways: one branch handles visual appearance guided by 3DGS, and the other handles geometric structure guided by sparse LiDAR projections. These branches are modulated by the PRG labels to ensure adaptive fusion based on local scene needs.

Terminology

Summary

Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation <ref:2609. p. 006948#pg2> The paper proposes PRG-Fusion, a framework that uses reconstruction evidence to orchestrate generative priors across regions <ref:2609. p. 006948#pg4>.

Core Problem Addressed

The challenge lies in balancing the strengths of reconstruction-based methods, which provide geometric consistency, and generative models, which offer flexibility along arbitrary trajectories <ref:2609.06948#pg2> Specifically, when the target trajectory deviates from the recorded trajectory, 3DGS renderings exhibit spatially varying degradation ranging from high fidelity to artifacts or missing content <ref:2609. p. 006948#pg2> Conversely, generative models often lack stable 3D geometric consistency across frames, leading to temporal drift in scene structures <ref:2609. p. 006948#pg2>.

The PRG Framework

PRG-Fusion introduces a region-aware video generation framework that explicitly formulates the spatially varying reliability of off-trajectory 3DGS renderings through pixel-wise Preserve, Repair, and Generate (PRG) labels <ref:2609.06948#pg2> These PRG labels serve as a unified routing policy for region-aware spatiotemporal synthesis <ref:2609. p. 006948#pg2>. The framework coordinates 3DGS appearance preservation, LiDAR-guided structural correction, and video-prior-driven content completion across Preserve, Repair, and Generate regions, respectively <ref:2609. p. 006948#pg2>.

Evidence Extraction Mechanism

The extraction of reconstruction evidence involves deriving pixel-wise PRG labels from reconstruction evidence to characterize the local reliability of off-trajectory renderings <ref:2609.06948#pg2> This process involves two main components:

  1. Trajectory unreliability is quantified using Expected Information Gain (EIG) to measure parameter sensitivity between training and target views, resulting in a Gaussian-level unreliability score <ref:2609. p. 306948#pg2>.

  2. Observation and geometric support are estimated by combining visibility frequency with directional coverage for observation support, and depth consistency across training views to determine geometric support <ref:2609. p. 306948#pg2>.

Dual-Branch Control Architecture

The model employs a dual-branch control architecture where a visual branch encodes 3DGS appearance and a geometry branch encodes LiDAR projections <ref:2609.06948#pg2> The two conditions are encoded by separate control branches, and the PRG labels are used to modulate the two control signals during generation, enabling adaptive fusion according to local reconstruction states <ref:2609. p. 206948#pg2>. A two-stage progressive training strategy is adopted where the geometry branch is optimized first using LiDAR projections, and then the visual branch is optimized under joint crender and clidar conditioning <ref:2609. p. 106948#pg2>.

PRG-Guided Fusion Strategy

The PRG labels are used as a routing policy to modulate the two control branches, where Preserve regions favor visual guidance to retain reliable 3DGS appearance, Repair regions emphasize geometric guidance for degraded but supported content, and Generate regions reduce reconstruction guidance to leave greater freedom to the video generative prior <ref:2609.06948#pg2>. This routing is explicitly encoded via a fixed routing matrix Λ that defines the routing strengths for the visual and geometry branches across Preserve, Repair, and Generate states <ref:2609. p. 506948#pg2>.

Experimental Validation

Extensive experiments on the Waymo Open Dataset demonstrate that PRG-Fusion achieves state-of-the-art overall performance in novel trajectory video synthesis <ref:2609.06948#pg2> The framework consistently outperforms existing methods in novel-trajectory video synthesis, achieving superior visual quality, geometric fidelity, and view consistency under large trajectory shifts <ref:2609. p. 106948#pg2>. Qualitative results show that PRG-Fusion better preserves vehicle appearance and lane topology compared to competing methods under both 3m and 6m lateral trajectory shifts <ref:2609. p. 706948#pg2>. The lower FVD and CLIP-V metrics indicate better preservation of scene content and semantic consistency across viewpoint changes <ref:2609. p. 606948#pg2>.

Ablation Study Insights

The ablation study confirms that PRG-guided fusion achieves the best NTA-IoU and NTL-IoU among all variants, showing that gains arise not only from combining visual and geometric cues but also from adapting their contributions to local reconstruction states <ref:2609.06948#pg2>. The results indicate that LiDAR guidance better preserves road geometry and lane structure, while conditioning on 3DGS renderings yields a pronounced improvement in scene-specific appearance <ref:2609. p. 706948#pg2>. The PRG-guided fusion variant shows the highest NTA-IoU and NTL-IoU among all variants, supporting the effectiveness of region-aware control <ref:2609. p. 706948#pg2>.

Conclusion

In summary, PRG-Fusion presents a region-aware video generation framework that adaptively coordinates dense 3DGS appearance, sparse LiDAR geometry, and video generative priors according to the local reconstruction state <ref:2609.06948#pg2> By converting spatially varying reconstruction evidence into pixel-wise Preserve, Repair, and Generate (PRG) labels, PRG-Fusion adaptively coordinates dense 3DGS appearance, sparse LiDAR geometry, and video generative priors according to the local reconstruction state <ref:2609. p. 706948#pg2> A dual-branch control architecture with two-stage progressive training further promotes complementary visual and geometric guidance while mitigating over-reliance on degraded 3DGS renderings <ref:2609. p. 706948#pg2>. Extensive experiments demonstrate that PRG-Fusion achieves improved visual quality, geometric fidelity, and view consistency <ref:2609. p. 706948#pg2>. The current framework still relies on scene-specific 3DGS reconstruction and LiDAR geometry, introducing additional preprocessing overhead <ref:2609. p. 706948#pg2>. Future work will explore feed-forward reconstruction models and predicted depth as alternatives to improve scalability and reduce sensor dependence <ref:2609. p. 706948#pg2>. The paper is 2609. p. 006948<ref:2609. p. 006948#pg2>.

--- Page 1 ---

PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

Sipeng He1 Jialei Chen2† Zhen Fang1 Dongchun Ren2 Feng Zhao1 Abstract Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation <ref:2609.06948#pg2> Reconstruction-based methods leverage neural rendering to synthesize geometrically consistent views, but often exhibit diverse artifacts and missing content when the viewpoint deviates from the training trajectory <ref:2609. p. 006948#pg2> In contrast, generative models can synthesize realistic views along arbitrary trajectories from vehicle sensor data, yet often struggle to maintain temporal and geometric consistency across frames <ref:2609. p. 006948#pg2> To combine the strengths of both, we propose PRG-Fusion, a framework for driving view synthesis that uses reconstruction evidence to orchestrate generative priors across regions <ref:2609. p. 006948#pg2> Specifically, we extract region-wise degradation evidence from reconstructed driving scenes and convert it into Preserve, Repair, and Generate (PRG) labels <ref:2609. p. 006948#pg2>. At inference, these labels serve as a unified routing policy for regionaware spatiotemporal synthesis <ref:2609. p. 006948#pg2> We then follow a two-stage training paradigm, first establish geometric control from sparse LiDAR projections and subsequently learning appearance control from dense 3DGS renderings <ref:2609. p. 006948#pg2> Extensive experiments on Waymo demonstrate that PRG-Fusion achieves state-of-the-art overall performance in novel trajectory video synthesis, with superior visual quality and geometric fidelity while maintaining competitive view consistency under large trajectory shifts <ref:2609. p. 006948#pg2> 1. Introduction Closed-loop simulation has become a key component in driving the continuous iteration and self-evolution of autonomous driving systems, with its core focus on building a virtual driving environment capable of interacting with driving strategies in real time [37, 44] <ref:2609. p. 006948#pg2> However, repeatedly collecting real-world data to capture diverse driving behaviors and environmental conditions is not only costly but also difficult to scale <ref:2609. p. 006948#pg2> Therefore, synthesizing videos along different trajectories from a single real-world driving log provides a more scalable alternative for closed-loop simulation <ref:2609. p. 006948#pg2> Mainstream methods typically rely on scene reconstruction techniques such as NeRF and 3DGS [15, 22] to construct renderable virtual environments, but their rendering quality is limited by the coverage of captured viewpoints and the scene representation capabilities <ref:2609. p. 006948#pg2> For driving scenes represented using 3DGS [7, 12, 13, 34], when the target trajectory deviates from the recorded trajectory, anisotropic Gaussians may exhibit varying degrees of degradation under novel view projection and composition: some areas retain high fidelity, while others exhibit artifacts or geometric distortions; newly exposed areas, however, form holes due to a lack of reliable geometric and appearance information <ref:2609. p. 006948#pg2> Compared with reconstruction-based approaches, recent driving world models [1, 8, 9, 11, 17, 29, 35] can generate diverse videos conditioned on specific driving instructions <ref:2609. p. 006948#pg2> However, their generated results often lack stable 3D geometric consistency across frames <ref:2609. p. 006948#pg2> To address these challenges, we propose PRG-Fusion, a region-aware video generation framework for novel trajectory synthesis that combines geometric constraints from reconstruction with the generalization capability of video generative models <ref:2609. p. 006948#pg2> Our key insight is that off-trajectory 3DGS renderings should not be treated as uniformly degraded conditioning inputs <ref:2609. p. 006948#pg2> As the viewpoint deviates from the recorded trajectory, different regions exhibit substantially different levels of reconstruction reliability and observational support <ref:2609. p. 006948#pg2> Based on this observation, we partition the novel view into three types of regions: Preserve, Repair, and Generate (PRG), and produce pixel-wise PRG labels <ref:2609. p. 106948#pg2>. The PRG labels further guide how reconstruction and generative information are utilized across different regions <ref:2609. p. 106948#pg2> Reliable regions preferentially retain the dense appearance provided by 3DGS; degraded regions that still retain structure are reinforced with more stable geometric constraints from LiDAR; and regions with insufficient reconstruction support are granted greater freedom to the video generative prior <ref:2609. p. 106948#pg2>.

--- Page 2 ---

Figure 1. Motivation of PRG-Fusion. p. 1 Trajectory deviation causes spatially varying degradation in 3DGS renderings, ranging from reliable appearance to geometric artifacts and missing content <ref:2609. p. 106948#pg2> the novel view into three types of regions: Preserve, Repair, and Generate (PRG), and produce pixel-wise PRG labels <ref:2609. p. 106948#pg2>

--- Page 3 ---

  1. Method Given a recorded driving sequence and a target novel trajectory, our goal is to synthesize a temporally consistent video that preserves scene-specific appearance while remaining geometrically faithful under viewpoint changes <ref:2609. p. 206948#pg2> To this end, we propose PRG-Fusion, which combines dense 3DGS renderings, sparse LiDAR geometry, and video generative priors through region-aware control <ref:2609. p. 206948#pg2> Specifically, we first derive pixel-wise Preserve, Repair, and Generate (PRG) labels from reconstruction evidence to characterize the local reliability of off-trajectory renderings <ref:2609. p. 206948#pg2> We then introduce a dual-branch control architecture, where a visual branch encodes 3DGS appearance and a geometry branch encodes LiDAR projections <ref:2609. p. 206948#pg2> The PRG labels are used to modulate the two control signals during generation, enabling adaptive fusion according to local reconstruction states <ref:2609. p. 206948#pg2>

--- Page 4 ---

3.1. Preliminaries 3D Gaussian Splatting [15] represents a scene with an anisotropic Gaussians G = G i N i=1 Each Gaussian has position µi ∈ R cubed, rotation quaternion qi ∈ R 4, scale σi ∈ R cubed, opacity oi ∈ R, and spherical harmonic coefficients γi; we collect them as ωi = (µi, qi, σi, oi, γi) <ref:2609. p. 206948#pg2> After projection and alpha blending the rendered pixel color is C = X i∈M c iα' i i Y-1 j=1 1 − α' j (1) <ref:2609.06948#pg2>

--- Page 5 ---

3.2. Reconstruction Evidence Extraction Given a reconstructed 3DGS scene G = G i N i=1 and training views V train = V train k K k=1, and a target trajectory V tar = V tar f T f=1 Our goal is to convert spatially varying reconstruction evidence into a pixel-wise PRG label map Lf ∈ P, R, G H×W, where each target pixel is assigned to Preserve, Repair, or Generate <ref:2609. p. 306948#pg2> Trajectory unreliability is quantified using Expected Information Gain (EIG) to measure the viewdependent unreliability of each Gaussian along the target trajectory <ref:2609. p. 306948#pg2> Observation and geometric support are estimated by combining visibility frequency with directional coverage for observation support, and depth consistency across training views to determine geometric support <ref:2609. p. 306948#pg2> Finally, we combine the projected observation support with the geometric support using the standard alphacompositing weights to derive Sf (p) = Gf (p) X i∈M s iα' i i Y-1 j=1 1 − α' j (24) <ref:2609.06948#pg2> PRG assignment follows a hierarchical decision rule based on unreliability and support maps to assign Lf (p) = P if Uf (p)

--- Page 6 ---

3.3. Dual-Branch Conditional Video Generation Off-trajectory 3DGS renderings retain dense, scene-specific visual context, but their reliability degrades as the target trajectory departs from the training views <ref:2609. p. 406948#pg2> LiDAR projections, by contrast, provide reliable metric geometry, but their sparsity prevents them from describing complete visual appearance <ref:2609. p. 406948#pg2> We use a frozen video DiT as the generative backbone and construct a visual branch conditioned on 3DGS renderings and a geometry branch conditioned on LiDAR projections <ref:2609. p. 406948#pg2> The two condition latents are independently mapped to the video-token grid by their control embedders and passed to the corresponding ControlDiT blocks <ref:2609. p. 406948#pg2> We train the two branches progressively while keeping the shared VAE and pretrained video DiT frozen <ref:2609. p. 506948#pg2> In the first stage, we optimize the geometry branch using clidar so that sparse observations are encoded into hierarchical geometric control features <ref:2609. p. 506948#pg2> In the second stage, we remove Iref, freeze the trained geometry branch, and optimize the visual branch under joint crender and clidar conditioning <ref:2609. p. 506948#pg2>

--- Page 7 ---

3.4. PRG-Guided Fusion The visual and geometry branches provide complementary controls, whose relative contributions should vary with the local reconstruction state <ref:2609. p. 506948#pg2> We therefore use the pixel-wise PRG label map L ∈ P, R, G F ×H×W as a routing policy for the two control branches <ref:2609. p. 506948#pg2> To explicitly encode this routing policy, we define a fixed routing matrix Λ = Pvis Rvis Gvis Pgeo Rgeo Ggeo (11) <ref:2609. p. 506948#pg2> The PRG-guided fusion achieves the best NTA-IoU and NTL-IoU among all variants, showing that the gains arise not only from combining visual and geometric cues but also from adapting their contributions to local reconstruction states <ref:2609. p. 706948#pg2>

--- Page 8 ---

4.1. Experimental Setup Implementation Details We conduct experiments on the Waymo Open Dataset [26] Following the data curation strategy in Sec. p. 5 3.3, we process 114 scenes into 684 aligned triplets <ref:2609. p. 506948#pg2> The training resolution is set to 480 × 832, and each video consists of 121 frames <ref:2609. p. 506948#pg2> We adopt Cosmos Transfer 2. p. 55 [1] as the frozen video DiT backbone and train the control branches in two stages <ref:2609. p. 506948#pg2> Eac…

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Region-aware synthesis using PRG labels allows for adaptive coordination of 3DGS appearance, LiDAR geometry, and video priors based on local reconstruction states. This enables a unified routing policy that orchestrates 3DGS appearance preservation, LiDAR-guided structural correction, and video-prior-driven content completion across Preserve, Repair, and Generate regions.

  2. A dual-branch control architecture with a two-stage progressive training strategy effectively integrates dense 3DGS appearance and sparse LiDAR geometry by first establishing stable geometric control before optimizing the visual branch under joint conditioning. This mitigates the issue where dense 3DGS rendering is highly correlated with the target video and can dominate the much sparser LiDAR signal.

  3. The PRG extraction method converts spatially varying reconstruction evidence into a pixel-wise Preserve, Repair, and Generate (PRG) label map, which explicitly guides how information is utilized across different regions. This allows for evidence-guided correction where sufficient support exists by assigning pixels to Repair or Generate based on their local reliability and geometric support.

  4. The system achieves superior view consistency under large trajectory shifts, maintaining superior visual quality and geometric fidelity while maintaining competitive view consistency under large trajectory shifts, as evidenced by achieving the best FVD and CLIP-V metrics in experiments.

  5. The training data curation strategy constructs strictly aligned triplets consisting of a degraded 3DGS rendering video Vrender, a colored LiDAR projection video Vlidar, and the corresponding real video Vrgb, ensuring that the two control branches are trained following the two-stage progressive strategy described in Sec. 3.3.

Sources

Related papers