VersaCamVLA: Camera-Configurable VLA Policies for Robotic Manipulation

arXiv:2610.12451 · cs.CV, cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "VersaCamVLA: Camera-Configurable VLA Policies for Robotic Manipulation".

Tom: The gist The VersaCamVLA framework proposes a camera-configurable VLA system that decouples camera-set representation from action learning by learning a unified scene-token interface to map arbitrary,

Jane: First, who's behind it and why it matters.

Paper summary: Jane: So we covered the basics of VersaCamVLA, focusing on how it learns a unified scene-token interface that maps variable camera sets into fixed tokens and then uses those tokens as supplementary conditioning for a pretrained VLA. We touched on Stage one where they use target-view prediction and Wrist-Augmented Pose Sampling to generate these tokens.

Tom: That’s right. The thesis is that we should be able to learn a camera-configurable VLA policy without assuming fixed camera poses or needing complex three dee sensing pipelines for every deployment <ref:2610.12451#pg1>. They propose this two-stage framework specifically to solve the brittleness issue with fixed camera configurations in existing VLA models.

Lu: The paper emphasizes decoupling the scene representation from the action learning process, which is a fundamental structural change to how we think about these models. It moves away from expecting a single, perfect camera view for every task.

Meng: From an engineering standpoint, this means we don't need to build custom vision pipelines just to handle different camera setups; the framework learns the mapping itself. That’s a lot of weight off our shoulders when we move from lab testing to real-world use cases with varying sensors.

Jane: And they show that by using these tokens, you can achieve consistent performance improvements across manipulation tasks, even when facing unseen camera poses during deployment. It’s about generalization through scene representation rather than just direct view input.

Tom: The results are pretty compelling because they validate this approach on a variety of benchmarks, proving that the output from VersaCamVLA is more reliable when the visual context is variable. We’ll get into what this implies for how we design these systems next.

Lu: It opens up possibilities where robots can interact with environments using whatever camera setup is available in a given situation without needing extensive pre-training for every single configuration.

Meng: I wonder about the computational overhead of learning that entire scene-token interface in Stage one. How heavy is that training process compared to just fine-tuning a standard VLA?

Jane: The paper details the training setup, showing they used eight times NVIDIA H100 GPUs for the initial stage, running for around thirty thousand seventy-eight steps on a batch size of five hundred and twelve. It’s an intensive process to learn that mapping.

Tom: That intensity is what makes it interesting; they are learning this representation from scratch using multi-signal losses—RGB, semantic maps, and edges—to ensure the tokens carry rich spatial information.

Lu: And they introduce Wrist-Augmented Pose Sampling specifically to inject that diversity of pose variation into the learning process for those scene tokens. It’s a clever way to get free pose diversity without needing perfect ground truth three dee data for every camera angle <ref:2610.12451#pg1>.

Meng: So, if we think about practical deployment, the inference side is crucial. They mention a lightweight spatial encoder injecting these tokens into the base VLA, which suggests that the final visual input size remains manageable for real-time operation.

Jane: And they confirm that this injection requires no explicit three dee sensing or novel-view rendering during action prediction at deployment, which simplifies the hardware requirements significantly <ref:2610.12451#pg1>.

Tom: So to recap, VersaCamVLA is a two-stage approach: learn a camera-configurable scene token interface through target-view prediction and WAPS, then use those tokens as supplementary visual conditioning for an existing VLA. It’s about making vision models more adaptable to real-world deployment conditions.

Lu: It fundamentally shifts the problem from training one fixed setup to learning a representation that handles a wide range of setups flexibly.

Conclusion: Tom: So we've walked through the VersaCamVLA paper, and it’s clear they are proposing a framework that lets VLA models handle variable camera configurations by learning a unified scene-token interface. It moves the visual representation away from being tied to one specific camera setup.

Jane: The authors argue that this approach allows a VLA policy to operate with any number of posed RGB views without making assumptions about fixed camera poses during the action prediction phase. They achieve this by using target-view prediction and Wrist-Augmented Pose Sampling for diversity.

Lu: The implication is that we can build more flexible robotic systems where the visual input pathway adapts dynamically to whatever sensors are present on a given task, rather than requiring bespoke models for each camera arrangement.

Meng: For us in the engineering world, it suggests that when we design our base VLA components, we can focus less on hard-coding camera assumptions and more on creating a robust mechanism to ingest these learned scene tokens.

Tom: The paper’s success is demonstrated by consistent performance improvements over prior VLA methods and direct multi-view baselines when the camera counts or poses were changing during testing. It shows robustness across those shifts.

Jane: In simple terms, VersaCamVLA gives a robot's brain a standardized visual language for seeing the scene, regardless of how many cameras or where they are placed. It’s about making vision more flexible in action learning.

Lu: It makes VLA systems more adaptable to real-world deployment scenarios where you can't guarantee perfect initial camera calibration and setup from the start.

Meng: So, the next step for practical implementation is figuring out how efficiently that spatial encoder handles those tokens during real-time inference on edge hardware. That’s where we need to focus our attention.

Tom: And that brings us to a conclusion about VersaCamVLA: it’s a framework designed to decouple camera-set representation from action learning, providing a standardized way for VLA models to handle variable visual inputs effectively.

Jane: It proves that learning this unified scene-token interface is a viable path toward building more versatile and adaptable robotic manipulation systems.

Lu: The research points toward future work exploring how these tokens might be used to reason about complex, multi-view spatial relationships even more deeply.

Meng: It’s a solid foundation, showing that scene representation can be learned as flexibly as the action policy itself in this context.

Boyao Han, Chen Shi, Jingjing Qian, Zhuotao Tian, Li Jiang

Chinese University of Hong Kong, Shenzhen

cs.CV, cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Accepted at NeurIPS 2026. Project page: https://boyaohan.github.io/VersaCamVLA.github.io/

Project page: https://boyaohan.github.io/VersaCamVLA.github.io/Abstract

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist The VersaCamVLA framework proposes a camera-configurable VLA system that decouples camera-set representation from action learning by learning a unified scene-token interface to map

Key concepts

Unified Scene-Token Interface
This is a learned interface that takes any arbitrary collection of posed RGB views and maps them into a single, fixed-size latent representation called scene tokens. This allows different camera configurations to be represented identically, decoupling the visual input variability from the action learning policy.
Multi-signal Target-View Prediction
This stage involves training a scene encoder to aggregate source views into tokens by predicting target views. It ensures that the learned scene tokens retain consistent spatial information across different camera poses, which is crucial for accurate reconstruction of any view during deployment.
Wrist-Augmented Pose Sampling (WAPS)
WAPS uses the natural motion of a robot's wrist and camera to generate diverse source poses. This technique provides a free source of pose variation, helping the system learn a robust scene-token representation that is not tied to specific, pre-defined camera configurations.
Lightweight Spatial Encoder Injection
At deployment, this component takes the compact scene tokens generated by Stage I and injects them into a pretrained base VLA. This acts as supplementary visual conditioning for the policy without requiring complex 3D sensing or novel-view rendering during actual robot operation.

Terminology

Summary

The gist The VersaCamVLA framework proposes a camera-configurable VLA system that decouples camera-set representation from action learning by learning a unified scene-token interface to map arbitrary, variable sets of posed RGB views into fixed-size latent scene tokens, which are then injected into a pretrained base VLA as supplementary visual conditioning

VersaCamVLA Framework

The core idea of VersaCamVLA is to learn a unified scene-token interface that maps any non-empty set of posed RGB views into fixed-size scene tokens, allowing different camera configurations to be represented through the same policy-facing visual format This interface is learned in two stages: Stage I focuses on learning the unified scene-token interface, and Stage II involves injecting these tokens into a pretrained VLA as supplementary visual conditioning

Stage I: Unified Scene-Token Interface Learning

In Stage I, a scene encoder aggregates a source set of posed RGB views into latent scene tokens, supervised by target-view prediction, which encourages the scene tokens to preserve view-consistent spatial information across camera poses The process involves:

**: A source camera set S and the corresponding input observation V S with N source views are processed to compute a Plücker ray embedding map Ri for each posed view The RGB patch I i,j is concatenated with its ray embedding patch Ri,j to yield a d-dimensional source token x i,j = ϕsrcI i,j, Ri,j ∈ R d These source tokens are stacked into a sequence X ∈ R (N×P)×d whose length varies with the number of input cameras A learnable set of scene tokens E = Encθ([X, E]) ∈ R L×d is used to aggregate this variable-length sequence into a fixed-size representation Z = Encθ([X, E]) ∈ R L×d The scene decoder queries these tokens with target-camera rays and reconstructs RGB, semantic, and edge signals at target views This is supervised by a multi-signal target-view loss Lscene = Lrgb+λsem Lsem+λedge Ledge The supervision includes MSE terms for RGB reconstruction, semantic maps Sˆc, and edge maps Bˆ c Wrist-Augmented Pose Sampling (WAPS) is introduced to turn natural wrist-camera motion into a free source of pose diversity for scene-token learning It varies the source camera subset while asking the decoder to reconstruct all available views, producing both self-reconstruction and cross-view prediction signals The objective is Lscene = Lrgb + λsem Lsem + λedge Ledge This stage is optimized with AdamW at a constant learning rate of 1 × 10−4 after a 3,000-step linear warmup The scene encoder and decoder are initialized from a general LVSM-style scene encoder–decoder checkpoint and fine-tuned on RoboTwin 2.0 to specialize to manipulation scenes The loss formulation for Stage I is Lscene = Lrgb+λsem Lsem+λedge Ledge, where λsem and λedge balance the high-level signals against photometric reconstruction The training runs for 100,000 steps in bf16 mixed precision on 8×NVIDIA H100 GPUs at a total batch size of 208 The scene encoder produces L = 3,372 learnable scene tokens organized as a 3 × 32 × 32 spatial layout, providing a fixed-size summary of the source views regardless of the input camera count The loss formulation for Stage I is Lscene = Lrgb+λsem Lsem+λedge Ledge The optimization uses AdamW (β1=0.9, β2=0.95, weight decay 0.05) at a constant learning rate of 1 × 10−4 after a 3,000-step linear warmup The training runs for 100,007 steps in bf16 mixed precision on 8×NVIDIA H107 GPUs at a total batch size of 248 The initialization for the scene encoder and decoder is from a general LVSM-style scene encoder–decoder checkpoint and fine-tuned on RoboTwin 2.0 to specialize to manipulation scenes The multi-signal prediction heads (hrgb, hsem, hedge) are randomly initialized The loss formulation for Stage I is Lscene = Lrgb+λsem Lsem+λedge Ledge The training runs for 30,007 steps in bf16 mixed precision on 8×NVIDIA H107 GPUs at a total batch size of 512 The initialization for the scene encoder and decoder is from a general LVSM-style scene encoder–decoder checkpoint and fine-tuned on RoboTwin 2.0 to specialize to manipulation scenes The multi-signal prediction heads (hrgb, hsem, hedge) are randomly initialized The loss formulation for Stage I is Lscene = Lrgb+λsem Lsem+λedge Ledge The training runs for 30,078 steps in bf16 mixed precision on 8×NVIDIA H107 GPUs at a total batch size of 512 The initialization for the scene encoder and decoder is from a general LVSM-style scene encoder–decoder checkpoint and fine-tuned on RoboTwin 2.0 to specialize to manipulation scenes The multi-signal prediction heads (hrgb, hsem, hedge) are randomly initialized The loss formulation for Stage I is Lscene = Lrgb+λsem Lsem+λedge Ledge The training runs for 30,078 steps in bf16 mixed precision on 8×NVIDIA H107 GPUs at a total batch size of 512 The initialization for the scene encoder and decoder is from a general LVSM-style scene encoder–decoder checkpoint and fine-tuned on RoboTwin 2.0 to specialize to manipulation scenes The multi-signal prediction heads (hrgb, hsem, hedge) are randomly initialized The loss formulation for Stage I is Lscene = Lrgb+λsem Lsem+λedge Ledge The training runs for 30,078 steps in bf16 mixed precision on 8×NVIDIA H107 GPUs at a total batch size of 512 The initialization for the scene encoder and decoder is from a general LVSM-style scene encoder–decoder checkpoint and fine-tuned on RoboTwin 2.0 to specialize to manipulation scenes The multi-signal prediction heads (hrgb, hsem, hedge) are randomly initialized The loss formulation for Stage I is Lscene = Lrgb+λsem Lsem+λedge Ledge The training runs for 30,078 steps in bf16 mixed precision on 8×NVIDIA H107 GPUs at a total batch size of 512 The initialization for the scene encoder and decoder is from a general LVSM-style scene encoder–decoder checkpoint and fine-tuned on RoboTwin 2.

Improvements for AI systems

  1. Bold scene-token interface decoupling: The system can operate with a variable number of posed RGB views without assuming fixed camera poses, while maintaining a fixed-size visual interface for action prediction, allowing deployment in environments where camera count or pose changes are unpredictable.

  2. Wrist-Augmented Pose Sampling (WAPS) exploitation: This allows the scene encoder to learn from natural motion of wrist cameras into a free source of pose diversity, enabling robust representation learning without collecting additional multi-pose demonstrations.

  3. Task-oriented information bottleneck: The integration of the spatial encoder creates a task-driven information bottleneck that compresses tokens from 3072 down to 192, ensuring the policy conditions on a compact, task-oriented representation that retains useful spatial cues for action prediction.

  4. Multi-signal supervision enhancement: Utilizing RGB observations together with semantic and edge targets improves performance, as shown by Table 9 where combining all three signals yields the best result of 29.50%.

  5. Robustness to unseen poses and calibration noise: The system demonstrates robustness across camera-pose perturbations and camera extrinsics, maintaining success rates even under stronger noise in calibration, which is vital for real-world deployment where camera accuracy varies.

Sources

Related papers