Towards Zero-Shot Transfer Across Embodiments For Driving VLAs

arXiv:2609.02341 · cs.CV · Submitted 2026-09-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Towards Zero-Shot Transfer Across Embodiments For Driving VLAs".

Jane: Here's the gist of the paper. Vision-Language-Action models (VLAs) have shown strong potential in autonomous driving by leveraging multimodal pretraining for instruction following, visual reasoning, and scene-level generalization The paper investigates multi-dataset training and an auxiliary objective called BEV-Forcing to improve zero-shot transfer across different camera rigs for driving VLAs The Core Problem Addressed VLAs often lack geometric transfer when adapted for driving, meaning they may fail when asked to output spatially grounded trajectories on a new dataset collected with a different camera setup This is because the model learns spatial understanding implicitly from a low number of sensor rigs and fewer examples than in its pretraining The research hypothesizes that improved zero-shot transfer may arise if the model can learn a unified spatial representation from multiview camera images coming from varying rigs The Proposed Solution The authors propose BEV-Forcing, an auxiliary objective inspired by Spatial Forcing

19: , which transfers ground-plane object-layout information from a specialized Bird’s-Eye-View model into the VLA backbone This is achieved by extracting teacher occupancy maps from a specialized BEV model, SimpleBEV

11: , and using them to supervise the last-layer image hidden states with these teacher maps The BEV head is designed to initialize queries with a resolution of NX ×NY and then cross-attend these queries to the projected image embeddings Training and Evaluation Strategy The training involves combining scaling training embodiments with the auxiliary objective, where the total loss is defined as L = LNTP + λ · LBEV (4) The BEV loss, LBEV, is calculated using a binary cross entropy loss with targets coming from the occupancy map FSimpleBEV

11: The study evaluates performance in both seen embodiments and at zero-shot transfer when scaling up the number of training embodiments Key Findings The results show that BEV-Forcing can serve as a geometry-aware regularizer, providing large improvements when training on a limited number of camera rigs, with improvements diminishing as training-rig diversity increases Specifically, applying the BEV loss while training on only WOD-E2E provides a 10.

Title and authors: Jane: Moving on to what this paper is actually about, the title is "Towards Zero-Shot Transfer Across Embodiments For Driving VLAs," and it immediately tells you the goal they’re aiming for.

Tom: It’s focusing on bridging that gap where vision-language action models struggle to transfer their driving knowledge when the camera rig changes.

Lu: Specifically, they want to move beyond just having a model perform well on one specific driving dataset and instead have it generalize to entirely new camera rigs without any extra training for those setups.

Jane: The authors are tackling the problem that naively adding more datasets doesn't automatically make the model better when you’re working with different physical embodiments, like different camera setups.

Tom: They hypothesize that if a model can learn a unified spatial representation from multiview camera images coming from varying rigs, zero-shot transfer might improve.

Lu: The key mechanism they introduce to help this is BEV-Forcing, which acts as an auxiliary objective to guide the VLA backbone toward learning that unified spatial awareness.

Jane: Essentially, they are trying to teach the model how things are spatially laid out on the ground plane so it doesn't get confused by different camera perspectives.

Tom: So, instead of just relying on whatever spatial understanding is implicitly learned during pretraining, they’re explicitly supervising it with a geometric signal.

Jane: It’s about giving the model an explicit hint about ground-plane object positions, which is crucial for accurate trajectory planning in driving tasks.

Lu: The paper sets out to solve that question of what specifically makes a model good at cross-embodiment zero-shot transfer in particular for autonomous driving, which they say is still underexplored.

Tom: So the focus shifts from just making the VLA perform well on seen data to making it robust enough to handle unseen camera rigs effectively.

The paper's summary: Jane: So, let’s get into the specifics of what they found in "Towards Zero-Shot Transfer Across Embodiments For Driving VLAs." They introduced BEV-Forcing as the solution.

Tom: To summarize, they extract teacher occupancy maps from a specialized Bird’s Eye View model called SimpleBEV and use those maps to supervise the last layer image hidden states of the VLA backbone.

Lu: This process effectively transfers ground plane object position awareness into the backbone, which is what they call transferring ground plane object layout information.

Jane: This forces the VLA to learn how objects are positioned relative to each other on a map, which should help with spatial reasoning for driving.

Tom: The training strategy involves combining standard scaling training embodiments with this auxiliary objective, defined by the loss L equals LNTP plus lambda times LBEV.

Jane: They found that applying BEV-Forcing when you’re only training on a few embodiments can lead to gains comparable to what you might get from just adding extra training embodiments.

Lu: But they also provided a series of ablations to show exactly how this auxiliary task interacts with other techniques aimed at increasing robustness, like image augmentations or passing calibration parameters as inputs.

Tom: So, the summary is that BEV-Forcing acts as a geometry-aware regularizer that improves in-distribution and out-of-distribution performance when training on a limited number of camera rigs.

The paper's improvements: Jane: The paper highlights several key improvements they found, and I want to make sure we unpack what those mean for us.

Tom: First, BEV-Forcing can improve both in-distribution and out-of-distribution performance when training on a limited number of camera rigs by transferring ground plane object layout information from a specialized Bird’s Eye View model into the VLA backbone.

Lu: That means we get better planning performance in both seen and unseen environments, which is what we need for reliable autonomous driving adaptation.

Jane: Second, it shows that the improved AI system can perform zero-shot transfer to unseen camera rigs for autonomous driving by learning a unified spatial representation from multiview camera images coming from varying rigs when trained on multiple datasets.

Tom: That’s the big payoff: getting better generalization across those different sensor perspectives without needing a new dataset for every single rig.

Jane: Third, they also showed that the system demonstrates better in-dataset performance on single embodiments when using BEV-Forcing, as the paper shows that BEV-Forcing helps in-dataset performance when training on a single embodiment.

Lu: That’s an interesting finding because it suggests the geometric regularization isn't just for generalization; it actually sharpens the understanding of what's happening in the specific training scenario.

Tom: And they also confirmed that combining this BEV task with common robustness techniques like image augmentations or injecting calibration information retains improvements in-dataset as measured on the validation set of WOD-E2E.

Conclusion: Jane: So, to wrap up this discussion on "Towards Zero-Shot Transfer Across Embodiments For Driving VLAs," the main implication is that geometry awareness during training is a powerful tool.

Tom: They concluded that BEV-Forcing serves as a geometry-aware regularizer, providing large improvements when training on a limited number of camera rigs, even if the diversity increases later.

Lu: The final point they make is that this technique helps in-dataset performance when training on a single embodiment, which is important for understanding the components better.

Meng: From my side at startup, I’m thinking about deployment: if we can make these models more robust to camera changes through techniques like this, it significantly reduces the effort required to deploy new versions of our driving AI.

Tom: Exactly. The system can achieve better planning performance in both seen and unseen environments when trained on a limited number of camera rigs because it learns that unified spatial representation from multiview camera images coming from varying rigs.

Jane: It’s a solid piece of research showing how to inject explicit geometric knowledge into vision-language models to make them more reliable for real-world driving applications.

Caio Azevedo, Stefano Sabatini, Sascha Hornauer, Fabien Moutarde

Ecole des Mines de Paris · Stellantis

cs.CV

Submitted: 2026-09-02

Updated: 2026-11-02

Comments: Paper accepted for WACV 2027

Code: https://github.com/caiocj1/ad-vla

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: Vision-Language-Action models (VLAs) have shown strong potential in autonomous driving by leveraging multimodal pretraining for instruction following, visual reasoning…

Key concepts

Vision-Language-Action Models (VLAs)
These are AI models that combine vision (seeing images), language (understanding instructions), and action (planning movement). They are being tested for autonomous driving tasks, aiming to understand visual scenes and generate appropriate driving actions based on natural language commands.
BEV-Forcing
This is an auxiliary training objective inspired by Spatial Forcing. It forces the VLA backbone to incorporate geometric knowledge by using teacher occupancy maps from a specialized Bird's Eye View (BEV) model. This helps the model understand spatial relationships and ground-plane layouts.
Zero-Shot Transfer
This refers to the model's ability to perform a task successfully on data or environments it has never seen before, specifically in this context, driving VLAs trained on one set of camera rigs performing well when tested on a completely new camera setup.

Terminology

Summary

Vision-Language-Action models (VLAs) have shown strong potential in autonomous driving by leveraging multimodal pretraining for instruction following, visual reasoning, and scene-level generalization The paper investigates multi-dataset training and an auxiliary objective called BEV-Forcing to improve zero-shot transfer across different camera rigs for driving VLAs

The Core Problem Addressed

VLAs often lack geometric transfer when adapted for driving, meaning they may fail when asked to output spatially grounded trajectories on a new dataset collected with a different camera setup This is because the model learns spatial understanding implicitly from a low number of sensor rigs and fewer examples than in its pretraining The research hypothesizes that improved zero-shot transfer may arise if the model can learn a unified spatial representation from multiview camera images coming from varying rigs

The Proposed Solution

The authors propose BEV-Forcing, an auxiliary objective inspired by Spatial Forcing [19], which transfers ground-plane object-layout information from a specialized Bird’s-Eye-View model into the VLA backbone This is achieved by extracting teacher occupancy maps from a specialized BEV model, SimpleBEV [11], and using them to supervise the last-layer image hidden states with these teacher maps The BEV head is designed to initialize queries with a resolution of NX ×NY and then cross-attend these queries to the projected image embeddings

Training and Evaluation Strategy

The training involves combining scaling training embodiments with the auxiliary objective, where the total loss is defined as L = LNTP + λ · LBEV (4) The BEV loss, LBEV, is calculated using a binary cross entropy loss with targets coming from the occupancy map FSimpleBEV [11] The study evaluates performance in both seen embodiments and at zero-shot transfer when scaling up the number of training embodiments

Key Findings

The results show that BEV-Forcing can serve as a geometry-aware regularizer, providing large improvements when training on a limited number of camera rigs, with improvements diminishing as training-rig diversity increases Specifically, applying the BEV loss while training on only WOD-E2E provides a 10.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. BEV-Forcing as an auxiliary objective can improve both in-distribution and out-of-distribution performance when training on a limited number of camera rigs by transferring ground plane object layout information from a specialized Bird’s Eye-View model into the VLA backbone.

  2. The improved AI system can perform zero-shot transfer to unseen camera rigs for autonomous driving by learning a unified spatial representation from multiview camera images coming from varying rigs when trained on multiple datasets.

  3. The system demonstrates better in-dataset performance on single embodiments when using BEV-Forcing, as the paper shows that BEV-Forcing helps in-dataset performance when training on a single embodiment.

  4. The system can effectively utilize common robustness techniques like image augmentations or injecting calibration information by showing that combining the BEV task with such robustness techniques retains improvements in-dataset as measured on the validation set of WOD-E2E.

  5. The improved system can achieve better planning performance in both seen and unseen environments when trained on a limited number of camera rigs, as BEV-Forcing acts as a geometry-aware regularizer.

Sources

Related papers