CityRAG: Stepping Into a City via Spatially-Grounded Video Generation

arXiv:2604.19741 · cs.CV · Submitted 2026-04-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "CityRAG: Stepping Into a City via Spatially-Grounded Video Generation".

Jane: Here's the gist of the paper. CityRAG presents video generative model that leverages large corpora of geo-registered data as context to ground generation to the physical scene while maintaining learned priors for complex motion and appearance changes The capability to reconstruct the real world under arbitrary weather conditions and dynamic object configurations is essential for downstream applications including autonomous driving and robotics simulation Core Problem Addressed The paper addresses the problem of generating a 3D-consistent, navigable environment that respects both transient attributes of a first image condition, such as weather and pedestrians, and static attributes derived from geospatial conditions like buildings and roads This capability would unlock applications in virtual tourism, gaming, and simulation for autonomous driving and robotics CityRAG focuses on the domain of Street View for its dense coverage and semantic cues of the arrangement of static and dynamic elements to ground generation in real-world environments Data Curation Strategy The authors curated a dataset of paired Street View videos that capture the same physical location at different times, such as morning versus sunset Specifically, they collected a total of 5.5M Street View panoramas and their poses across 10 cities They created a training pair if they find a continuous path in a city of length N where there exists 2 sets of panoramas located along the same path with an average distance threshold smaller than epsilon meters This process resulted in 1.3M panoramas for training and a few thousand held out for testing Model Architecture and Conditioning CityRAG relies on a state-of-the-art I2V model, Wan 2.1

Wan et al., 2025: , which consists of a spatio-temporal VAE and a diffusion transformer (DiT) The generator is conditioned on three inputs: the first image, a defined trajectory, and geo-registered videos along the trajectory First image conditioning initializes the scene by processing it through the VAE and concatenating it to target latents Trajectory conditioning involves converting 4x4 extrinsic matrices into relative poses on-the-fly, which are then processed through an MLP and added to the DiT blocks Geospatial conditioning is passed via cross-attention, where the condition video is processed by the VAE and its latents serve as keys and values for cross-attention with target noisy latents Inference Pipeline and Robustness Inference involves a pipeline where a user selects a location and first image, inputs a trajectory, retrieves geo-registered Street View data along the defined path as geospatial conditioning, and then generates the video autoregressively The model is robust to non-continuous geospatial conditioning because it can stitch frames from multiple videos to navigate arbitrary trajectories Qualitative results show that CityRAG successfully handles scenarios where the geospatial condition stops in traffic, yet the generated video follows the defined trajectory and renders structures that appear later in the condition frames Performance and Evaluation Quantitative comparisons show that CityRAG outperforms baselines across various metrics, including PSNR, LPIPS, and FID In the user study, users rated CityRAG videos as both smooth continuations from first images and faithful renders of the real physical location The model demonstrates strong 3D understanding of the underlying scene and disentangles dynamic and static elements without heuristics CityRAG is highlighted as the first video generative model that emphasizes adherence to our real world The method is not real-time, focusing on spatial grounding for realism and long-horizon temporal consistency Limitations and Future Work A limitation noted is that the model performs autoregression by only providing the generated last frame as the first frame of the subsequent sample, which could be improved with existing methods The disentanglement of static vs.

Tom: transient objects is completely data-driven, and future work could include fine-grained control and annotations over individual elements The method is not real-time, focusing on spatial grounding for realism and long-horizon temporal consistency Furthermore, the model currently freezes original text cross-attention blocks and uses a fixed prompt instead of responding to new text captions The data biases include not including snowy, rainy, or nighttime conditions due to hardware and sensor limitations Inference Costs and Training Optimization The inference costs are divided into retrieving geospatial conditions from the Street View database and the video generation cost Retrieval time is O(log N) for map lookup, and sequential frames can be retrieved in O(1) constant time Video generation cost on an 8 A100 GPU node was measured at 90 seconds for 73 frames, with an amortized cost of roughly 0.

Title and authors: Tom: So, if we summarize the actual process of CityRAG, it uses a state-of-the-art I2V model called Wan two point one, which has both a spatio-temporal VAE and a diffusion transformer <ref:2604.19741#pg1>.

Lu: The generator is conditioned on three main inputs: the first image you give it, a defined trajectory you want the camera to follow, and geo-registered videos along that specific path.

Jane: The first image helps initialize the scene by processing it through the VAE and adding those features to target latents.

Meng: Then there’s trajectory conditioning where they convert four by four extrinsic matrices into relative poses on the fly, which an MLP then processes before being added to diffusion transformer blocks.

Lalam: And for geospatial conditioning, they use cross-attention where the condition video is processed by the VAE, and its latents serve as keys and values for cross-attention with the target noisy latents.

Tom: What’s really interesting is how they handle training data. They use temporally unaligned training data to teach the model to semantically disentangle the underlying scene from its transient attributes.

Jane: They create a training pair if there is a continuous path where there exist two sets of captures at different times, like morning versus afternoon, but with an average distance threshold smaller than five meters <ref:2604.19741#pg3> so the model learns to disentangle static and transient attributes like roads and buildings from things like weather and cars.

Lu: This training process results in one thousand three hundred panoramas for training and just a few thousand held out for testing <ref:2604.19741#pg3>

The paper's summary: Tom: The authors highlight several key improvements in CityRAG, focusing on making the generation more robust and controllable than previous methods.

Jane: One big improvement is the ability to generate minutes-long, spatially grounded video sequences that respect both the transient attributes of a first image condition, such as weather and pedestrians, and the static attributes derived from geospatial conditions <ref:2604.19741#pg2>.

Lu: This capability unlocks applications in autonomous driving and robotics simulation because it respects both what’s static, like buildings and roads, while allowing for changes in things like lighting or cars <ref:2604.19741#pg2>

Meng: They also improved consistency by showing that the system can follow user-defined trajectories and perform loop closure even when there are discontinuities in the geospatial conditioning data <ref:2604.19741#pg2>.

Lalam: They specifically show robustness to condition mismatch because if the geospatial video gets stuck in traffic, the generated video still follows the defined trajectory and renders structures that appear later in those condition frames <ref:2604.19741#pg2>.

Tom: It seems they’ve made a solid step toward achieving a model that understands the underlying scene without needing manual heuristics for disentangling static and dynamic elements.

Jane: They are showing how learning from unaligned data can force the model to learn a representation that cleanly separates static structure from things like weather or cars, which is exactly what we need for reliable simulation <ref:2604.19741#pg3>.

The paper's improvements: Jane: So, wrapping up, CityRAG is presented as the first video generative model that really emphasizes adherence to our real world by leveraging geo-registered data for spatial grounding.

Tom: It’s not about making perfect real-time videos; it focuses on spatial grounding for realism and long-horizon temporal consistency <ref:2604.19741#pg2>.

Lu: The main implication is that we now have a way to generate physically grounded video sequences that respect both the static layout of a city and the transient changes happening in it, which is crucial for simulation.

Meng: From my side, while the results are strong across PSNR and LPIPS metrics, I’m still focused on how much computational cost this actually means when you try to deploy this at scale in a production environment.

Lalam: The paper notes that it doesn't handle certain conditions well because the data biases include not including snowy, rainy, or nighttime conditions due to hardware and sensor limitations <ref:2604.19741#pg3>.

Tom: That limitation on weather is something we need to keep in mind as we look toward making these systems more versatile for real-world deployment.

Jane: So, CityRAG is a significant piece of work that bridges the gap between generative video and physically grounded simulation by using geo-registered data as context.

Lu: It shows how learning from unaligned data can force the model to learn a representation that cleanly separates static structure from things like weather or cars <ref:2604.19741#pg3>.

Meng: It’s a solid foundation, but the next step for me would be to see if we can reduce the inference cost significantly so this moves out of the lab and into actual robot testing.

Lalam: I'm pretty confident in what they achieved regarding scene consistency, but fine-grained control over individual elements is something that future work will need to focus on.

Conclusion: Jane: So, we’ve talked about CityRAG: Stepping Into a City via Spatially-Grounded Video Generation. Basically, this paper introduces a system that uses real geography to ground video generation instead of just making up pretty scenes from text or an image prompt.

Tom: Right. It’s about creating videos that respect the physical layout of a place, even when you introduce things like rain or cars moving around.

Lu: The authors are tackling the problem of generating a three dimensional consistent navigable environment that is spatially grounded, meaning it simulates a real location based on physical data.

Meng: They’re trying to solve the issue where current video models can make something plausible based on text or an image prompt, but they can’t actually reconstruct the real world when you introduce things like arbitrary weather or moving objects.

Lalam: CityRAG is presented as a video generative model that uses large corpora of geo-registered data as context to ground the generation to the physical scene while maintaining learned priors for complex motion and appearance changes.

Tom: So, it’s not just making pretty videos; it’s about creating something that respects the physical layout of a place while still allowing for changes in dynamic things like cars or rain.

Jane: Exactly. The implications here are big because this capability is essential for applications like autonomous driving and robotics simulation where you need to know exactly what’s around you physically.

Lu: It shows how learning from unaligned data can force the model to learn a representation that cleanly separates static structure from things like weather or cars.

Meng: From my side, while the results are strong across PSNR and LPIPS metrics, I’m still focused on how much computational cost this actually means when you try to deploy this at scale in a production environment.

Lalam: The paper notes that it doesn't handle certain conditions well because the data biases include not including snowy, rainy, or nighttime conditions due to hardware and sensor limitations.

Tom: That limitation on weather is something we need to keep in mind as we look toward making these systems more versatile for real-world deployment.

Jane: So, CityRAG is a significant piece of work that bridges the gap between generative video and physically grounded simulation by using geo-registered data as context.

Lu: It shows how learning from unaligned data can force the model to learn a representation that cleanly separates static structure from things like weather or cars.

Meng: It’s a solid foundation, but the next step for me would be to see if we can reduce the inference cost significantly so this moves out of the lab and into actual robot testing.

Lalam: I'm pretty confident in what they achieved regarding scene consistency, but fine-grained control over individual elements is something that future work will need to focus on.

Tom: Alright, well, CityRAG is a major step forward in making AI understand the physical world through context. Next time we’ll be looking at how we can make these models actually run faster so they aren't just lab curiosities.

Google University of Cornell University Stanford University

cs.CV

Submitted: 2026-04-21

Updated: 2026-11-05

Comments: Project page: cityrag.github.io

Code: https://github.com/krea-ai/realtime-video

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: CityRAG presents video generative model that leverages large corpora of geo-registered data as context to ground generation to the physical scene while maintaining learned priors for complex motion…

Key concepts

Geo-registered Data
This refers to Street View panoramas and their poses collected across multiple cities. This data provides the fixed, static spatial context—like building layouts and roads—that anchors the generated video to a real physical location.
Conditioning Inputs
The model is guided by three main inputs: the first image sets the initial scene look; a defined trajectory dictates where the camera moves; and geo-registered videos provide contextual information about the environment along that path. These inputs work together to shape the final video generation.
Attribute Disentanglement
CityRAG learns to separate what is permanent (static elements like structures) from what changes (transient attributes like lighting or pedestrians). By training on paired images of the same location under different conditions, it can reconstruct a consistent scene even when environmental factors shift.
Autoregressive Generation
The model generates video frame by frame, using the previously generated frame as input for the next one. This sequential process allows it to create long, temporally consistent video sequences that follow a specific path or trajectory.

Terminology

Summary

CityRAG presents video generative model that leverages large corpora of geo-registered data as context to ground generation to the physical scene while maintaining learned priors for complex motion and appearance changes The capability to reconstruct the real world under arbitrary weather conditions and dynamic object configurations is essential for downstream applications including autonomous driving and robotics simulation

Core Problem Addressed

The paper addresses the problem of generating a 3D-consistent, navigable environment that respects both transient attributes of a first image condition, such as weather and pedestrians, and static attributes derived from geospatial conditions like buildings and roads This capability would unlock applications in virtual tourism, gaming, and simulation for autonomous driving and robotics CityRAG focuses on the domain of Street View for its dense coverage and semantic cues of the arrangement of static and dynamic elements to ground generation in real-world environments

Data Curation Strategy

The authors curated a dataset of paired Street View videos that capture the same physical location at different times, such as morning versus sunset Specifically, they collected a total of 5.5M Street View panoramas and their poses across 10 cities They created a training pair if they find a continuous path in a city of length N where there exists 2 sets of panoramas located along the same path with an average distance threshold smaller than epsilon meters This process resulted in 1. p. 33M panoramas for training and a few thousand held out for testing

Model Architecture and Conditioning

CityRAG relies on a state-of-the-art I2V model, Wan 2.1 [Wan et al. p. 1, 2025], which consists of a spatio-temporal VAE and a diffusion transformer (DiT) The generator is conditioned on three inputs: the first image, a defined trajectory, and geo-registered videos along the trajectory First image conditioning initializes the scene by processing it through the VAE and concatenating it to target latents Trajectory conditioning involves converting 4x4 extrinsic matrices into relative poses on-the-fly, which are then processed through an MLP and added to the DiT blocks Geospatial conditioning is passed via cross-attention, where the condition video is processed by the VAE and its latents serve as keys and values for cross-attention with target noisy latents

Inference Pipeline and Robustness

Inference involves a pipeline where a user selects a location and first image, inputs a trajectory, retrieves geo-registered Street View data along the defined path as geospatial conditioning, and then generates the video autoregressively The model is robust to non-continuous geospatial conditioning because it can stitch frames from multiple videos to navigate arbitrary trajectories Qualitative results show that CityRAG successfully handles scenarios where the geospatial condition stops in traffic, yet the generated video follows the defined trajectory and renders structures that appear later in the condition frames

Performance and Evaluation

Quantitative comparisons show that CityRAG outperforms baselines across various metrics, including PSNR, LPIPS, and FID In the user study, users rated CityRAG videos as both smooth continuations from first images and faithful renders of the real physical location The model demonstrates strong 3D understanding of the underlying scene and disentangles dynamic and static elements without heuristics CityRAG is highlighted as the first video generative model that emphasizes adherence to our real world The method is not real-time, focusing on spatial grounding for realism and long-horizon temporal consistency

Limitations and Future Work

A limitation noted is that the model performs autoregression by only providing the generated last frame as the first frame of the subsequent sample, which could be improved with existing methods The disentanglement of static vs. transient objects is completely data-driven, and future work could include fine-grained control and annotations over individual elements The method is not real-time, focusing on spatial grounding for realism and long-horizon temporal consistency Furthermore, the model currently freezes original text cross-attention blocks and uses a fixed prompt instead of responding to new text captions The data biases include not including snowy, rainy, or nighttime conditions due to hardware and sensor limitations

Inference Costs and Training Optimization

The inference costs are divided into retrieving geospatial conditions from the Street View database and the video generation cost Retrieval time is O(log N) for map lookup, and sequential frames can be retrieved in O(1) constant time Video generation cost on an 8 A100 GPU node was measured at 90 seconds for 73 frames, with an amortized cost of roughly 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Grounded Video Generation for Simulation and Robotics: CityRAG can generate minutes-long, physically grounded video sequences that respect both the transient attributes of a first image condition, such as weather and pedestrians, and the static attributes derived from geospatial conditions. This capability unlocks applications in autonomous driving and robotics simulation.

  2. Arbitrary Trajectory Navigation with Scene Consistency: The system can generate videos following user-defined trajectories while maintaining consistency across samples, even when dealing with discontinuities in the geospatial conditioning, as shown by its ability to navigate arbitrary paths by stitching frames from multiple videos.

  3. Dynamic Scene Understanding via Attribute Disentanglement: CityRAG teaches the model to semantically disentangle the underlying scene from its transient attributes by forcing it to derive a static layout from morning frames to reconstruct a scene at night, allowing it to maintain weather and lighting conditions over thousands of frames.

  4. Robustness to Condition Mismatch: The model demonstrates robustness when geospatial conditioning is mismatched with the trajectory, as evidenced by the ability to follow the defined trajectory even when the geospatial video is stuck in traffic, showing an understanding of static and transient elements in a scene.

Sources

Related papers