Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving".

Jane: Here's the gist of the paper. The gist: Zero-shot cross-city transfer in end-to-end autonomous driving reveals that representation learning significantly influences robustness under geographic domain shift, with domain-specific self-supervised pretraining substantially reducing transfer degradation in several settings How it works The study formulates zero-shot cross-city transfer as a controlled representation-level stress test for end-to-end autonomous driving.

Tom: This involves training models exclusively on data from one city and evaluating them on a geographically distinct city without adaptation in both open-loop (nuScenes) and closed-loop (NAVSIM) settings.

Jane: The core question investigated is how visual pretraining affects transfer behavior under geographic domain shift.

Tom: The researchers conducted a comprehensive study by integrating self-supervised backbones I-JEPA, DINOv2, and MAE into planning frameworks.

Paper summary: Tom: So we're looking at this paper, "Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations." The main idea here is that models trained on one city often struggle when they see a completely different city later on.

Jane: Exactly. They're asking how visual pretraining affects how well these driving models can transfer across different geographic areas without any retraining for each new location. It’s about testing the limits of their knowledge before they hit the real world in a new place <ref:2603.11417#pg2>.

Lu: What they claim is that when training and evaluation data are mixed geographically, models might accidentally rely on cues specific to that city, which hides the actual failure modes when you move to a completely new location later on <ref:2603.11417#pg1>.

Meng: But what does that mean for us? It means if we just train something once on one city dataset and then deploy it somewhere else, we might be setting ourselves up for a big problem when the road layouts or traffic rules change drastically <ref:2603.11417#pg2>.

Lalam: I see. So the paper sets this up as a stress test by using strict geographic splits on nuScenes in the open-loop setting and NAVSIM in the closed-loop evaluation protocol <ref:2603.11417#pg2>.

Tom: Right, so they’re testing how different visual pretraining methods—like I-JEPA, DINOv2, and MAE—do up when we force them to transfer between cities that have different road topologies and traffic conventions <ref:2603.11417#pg1>.

Jane: And the core question is how visual pretraining influences this transfer behavior under a geographic domain shift <ref:2603.11417#pg2>.

Lu: They integrate these self-supervised backbones into planning frameworks, using the LAW latent world model and the TransFuser model for end-to-end trajectory prediction <ref:2603.11417#pg3>.

Meng: I'm curious about what they actually tested. Are we talking about just comparing accuracy on a single city, or is it really testing that zero-shot cross-city transfer capability?

Tom: It’s the latter. They evaluate performance under strict geographic splits on nuScenes in the open-loop setting and on NAVSIM in the closed-loop evaluation protocol <ref:2603.11417#pg2>.

Jane: And they measure performance using two main protocols: L2 displacement error and collision rate for open-loop, and a predictive driver model score, or PDMS, for closed-loop <ref:2603.11417#pg2>.

Lu: They compare supervised ImageNetpretrained backbones against generic self-supervised backbones like I-JEPA, DINOv2, and MAE across these settings <ref:2603.11417#pg2>.

Meng: What's the key finding there? Did the supervised ones win out, or did these self-supervised methods actually show some benefit?

Tom: Well, they found a substantial generalization gap when transferring models across cities with different road topologies and traffic conventions <ref:2603.11417#pg2>.

Jane: That's a big deal because it shows that simply having good in-domain performance on one city doesn't guarantee robustness when you move to another city <ref:2603.11417#pg2>.

Lu: In the open-loop evaluation, they found that a supervised backbone exhibits severe degradation when transferring between cities <ref:2603.11417#pg2>.

Meng: That sounds like a failure mode we see all the time in deployment, so what's the counterpoint? Are there any self-supervised methods that actually help reduce that degradation?

Tom: Yes, some domain-specific self-supervised methods can substantially reduce both displacement and collision degradation <ref:2603.11417#pg2>.

Jane: In the closed-loop evaluation on NAVSIM, they found that self-supervised pretraining actually improves the average out-of-distribution PDMS in several single-city training settings <ref:2603.11417#pg2>.

Lu: That suggests that representation learning, specifically self-supervised pretraining, does influence the robustness of cross-city planning <ref:2603.11417#pg2>.

Meng: So if I were deploying a model right now, this paper suggests that using a domain-specific SSL approach might be more stable than just using an ImageNetpretrained backbone <ref:2603.11417#pg2>.

Tom: It does, but it depends on the SSL objective, the input geometry they use—they used a ViT-S backbone with two hundred twenty-four by three hundred ninety-two input resolution for domain-specific pretraining—and the metric they pick <ref:2603.11417#pg3>.

Jane: It’s important to note that the transfer behavior itself is directional <ref:2603.11417#pg2>.

Lu: The results show that strong in-domain performance does not imply robustness under geographic domain shift, and performance differences between backbones are small when you train on all cities but large when you train on a single city <ref:2603.11417#pg3>.

Meng: Okay, so what about the practical implications for scaling up? Does this mean we need to collect data from more different cities to make our systems safe?

Tom: It suggests that city selection and geographic diversity in data collection are critical factors for scalable autonomous driving deployment <ref:2603.11417#pg3>.

Jane: So, while single-city training reveals strong sensitivity to representation initialization, the paper emphasizes that domain-specific self-supervised pretraining can help narrow that generalization gap <ref:2603.11417#pg3>.

Lu: The authors conclude by presenting this controlled study of zero-shot cross-city generalization in end-to-end autonomous driving <ref:2603.11417#pg2>.

Meng: It sounds like they are using this cross-city test as a practical way to stress test the representation robustness of these systems before we deploy them in varied real environments <ref:2603.11417#pg3>.

Tom: Exactly. They show that transferring across cities induces pronounced and asymmetric degradation, which is something we need to account for when designing systems <ref:2603.11417#pg2>.

Conclusion: Tom: So, we're wrapping up this look at this paper, "Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations." The authors are looking at how these driving models handle moving from one city to a completely different one without retraining.

Jane: Exactly. They're investigating if the way the AI learns its visual world representation—the backbone it uses—actually helps it generalize when the geography changes drastically, like switching between different road layouts.

Lu: The main thing they found is that strong performance in one city doesn't guarantee robustness in another city with a different traffic convention or road topology.

Meng: So, if I’m deploying this stuff, this means I can't just pick the best model for Boston and assume it works fine in Singapore without testing that transfer capability first.

Lalam: The paper shows that self-supervised pretraining can actually reduce the performance drop when we try to transfer models across cities.

Tom: Right, so the takeaway is that how you train those initial visual representations matters more than just training on one dataset. It's not just about getting a high score on Boston data; it's about building a foundation that handles new cities better.

Jane: And they’re using this zero-shot transfer test as a stress test for the entire system, which is pretty important because real-world deployment involves those kinds of sudden geographic shifts.

Lu: They showed that supervised models often degrade really badly when transferring, but some self-supervised methods manage to substantially reduce that displacement and collision degradation.

Meng: That’s useful information for me because it tells me exactly where I need to focus my research if I want to build something that doesn't break when we expand our service area.

Tom: It really highlights that representation learning isn't just about accuracy; it’s about structural robustness under domain shift. This points toward a new way of thinking about how we pretrain these massive visual models for driving tasks.

Jane: So, if we think about the future, this suggests that geographic diversity in data collection is going to be critical not just for training, but for ensuring the safety and reliability of autonomous systems when they operate globally.