AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models
summary
The gist
AnyCamVLA proposes a zero-shot camera adaptation framework that virtually adjusts test-time camera observations to match training camera configurations in real-time without requiring additional…
In short
AnyCamVLA proposes a zero-shot method to adapt Vision-Language Action Models (VLAs) to new camera views in real-time without needing extra training or fine-tuning. It uses a novel view synthesis model to virtually adjust test images to match training configurations, allowing VLAs to perform robustly across different camera setups instantly.
Key concepts
- Zero-Shot Camera Adaptation
- This is a framework that allows an existing VLA, trained on one set of cameras, to work successfully with entirely new camera views it has never seen before. It achieves this by virtually transforming the test-time images to resemble the training conditions in real-time, eliminating the need for new demonstrations or retraining.
- Novel View Synthesis Model
- This is a feed-forward model used to generate high-quality target view images from a source image and camera parameters. It is crucial because it handles both changes in camera position (extrinsic) and internal lens settings (intrinsic), enabling the virtual adjustment of test observations.
- Policy Inference Formulation
- The method reformulates how the VLA policy makes decisions. Instead of feeding raw test images, it feeds the synthesized, adapted images into the frozen VLA policy. This keeps the original visual reasoning capabilities intact while adapting to arbitrary camera configurations at inference time.
- LVSM Fine-tuning for Domain Adaptation
- This refers to a one-time process where a smaller model (LVSM) is fine-tuned using multi-view data. This pre-adaptation step creates a universal view adapter that can be plugged into any VLA, making the overall adaptation highly efficient and practical.
Terminology used across episodes
This episode discusses
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models · Paper Radio
- RT-1: Robotics Transformer for Real-World Control at Scale
- Octo: An Open-Source Generalist Robot Policy
- OpenVLA: An Open-Source Vision-Language-Action Model
- pi 0: A Vision-Language-Action Flow Model for General Robot Control
- pi 0.5: a Vision-Language-Action Model with Open-World Generalization
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Evaluating Real-World Robot Manipulation Policies in Simulation
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
- GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Model
- Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learning
- FP3: A 3D Foundation Policy for Robotic Manipulation
- How Do VLAs Effectively Inherit from VLMs?
- From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors
- Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping
- Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- Stereo Magnification: Learning View Synthesis using Multiplane Images
- Decoupled Weight Decay Regularization
The paper
AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models · Read on arXiv
Seoul National University · Massachusetts Institute of Technology
Despite remarkable progress in Vision-Language-Action models (VLAs) for robot manipulation, these large pre-trained models require fine-tuning to be deployed in specific environments. These fine-tuned models are highly sensitive to camera viewpoint changes that frequently occur in unstructured environments. In this paper, we propose a zero-shot camera adaptation framework without additional demonstration data, policy fine-tuning, or architectural modification. Our key idea is to virtually adjust test-time camera observations to match the training camera configuration in real-time. For that, we use a recent feed-forward novel view synthesis model which outputs high-quality target view images, handling both extrinsic and intrinsic parameters. This plug-and-play approach preserves the pre-trained capabilities of VLAs and applies to any RGB-based policy. Through extensive experiments on the LIBERO benchmark, our method consistently outperforms baselines that use data augmentation for policy fine-tuning or additional 3D-aware features for visual input. We further validate that our approach constantly enhances viewpoint robustness in real-world robotic manipulation scenarios, including settings with varying camera extrinsics, intrinsics, and freely moving handheld cameras. Project Page: https://heo0224.github.io/AnyCamVLA/
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models".
Dev: Here's the gist of the paper. AnyCamVLA proposes a zero-shot camera adaptation framework that virtually adjusts test-time camera observations to match training camera configurations in real-time without requiring additional demonstration data, policy fine-tuning, or architectural modification.
Rosa: This approach preserves the pre-trained capabilities of Vision-Language Action Models (VLAs) and enables robust task execution across diverse setups including different extrinsics, intrinsics, and freely moving handheld cameras.
Dev: Pruning redundant reasoning tokens The core idea of the framework is to virtually adjust test-time camera observations to match the training camera configuration in real-time.
Rosa: This is achieved by using a recent feed-forward novel view synthesis model which outputs high-quality target view images, handling both extrinsic and intrinsic parameters.
Paper summary: Rosa: So, we're looking at this paper now, AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models. The main idea here is that you can take a vision language action model and use it in totally different camera setups without needing any extra training data or changing the model itself.
Dev: Right, so the core thesis is to virtually adjust what the camera sees at test time to match how it was trained, all in real-time. It claims this lets you handle different camera positions, intrinsics, and even handheld cameras while keeping those pre-trained capabilities intact.
Taro: But that sounds like a lot of work for the robot. How does this actually work without needing to retrain anything? And what kind of robustness are they talking about when the world misbehaves?
Rosa: Well, it says they use a feed-forward novel view synthesis model to generate high-quality target view images that handle both extrinsic and intrinsic parameters for the test camera. This adaptation runs in real-time and plugs directly into any Vision Language Action Model.
Dev: The math behind it is that they transform the test time observation, which we call I test t, using a module F to get a synthetic image called I train t, where I train t equals F of I test t, C test, C train.
Taro: So you're taking the current view and projecting it to look like it came from the training camera configuration using this synthesis model? That sounds like a clever trick for geometry.
Rosa: Exactly. They are drawing inspiration from novel view synthesis to generate precise projective warping of three dimensional geometry, which they say is more robust than just traditional image warping when the multi-view geometry is inaccurate.
Dev: The point they make is that this model can handle various changes in pose, focal length, and lighting without having to modify the original Vision Language Action Model. It’s plug-and-play for any RGB based policy.
Taro: But if it's plugging into an existing VLA, what's the risk? Does this approach inject extra features or change how the model reasons internally?
Rosa: It preserves the pre-trained capabilities of those VLAs by only adjusting the input observation, not modifying the policy itself. They say this avoids catastrophic forgetting associated with retraining, which is a big deal for deployment.
Dev: The performance claims are pretty strong on their benchmark, LIBERO. They state that their method consistently outperforms baselines that use data augmentation for policy fine-tuning or additional three dimensional aware features for visual input.
Taro: I saw they mention something about agent camera perturbations achieving a ninety nine point zero percent success rate on unseen agent camera viewpoint across all LIBERO suites. That’s a high bar for generalization in the real world.
Paper summary: Rosa: And they also showed that on the most challenging LIBERO Long suite with wrist camera perturbations, their method, called Ours-pi, achieved an eighty eight point six percent average success rate when compared to data augmentation and GeoAwareVLA.
Dev: The ablation study on viewpoint adaptation methods is interesting too. They found that the feed-forward novel view synthesis approach significantly outperforms geometric baselines on both image synthesis quality and task success rate across all perturbation levels.
Taro: Why did they find that? Because depth based projection showed improvement over no adaptation and homography, but it had these non photorealistic artifacts from point cloud projection when there were large viewpoint shifts.
Rosa: That’s because the feed-forward novel view synthesis leverages learned priors to accurately reconstruct three dimensional geometry from multi-view images while naturally synthesizing photorealistic images, which they link to that superior image quality.
Dev: The paper notes that the camera adaptation module F is flexible with respect to how many input and output cameras there are, meaning N and M can be different in some cases. This flexibility is important for deployment scenarios where calibration happens once during setup.
Taro: But what about the practical side? They mention that feed-forward novel view synthesis introduces a latency of approximately thirty milliseconds per frame, which could be an issue if you're dealing with extremely dynamic camera movements.
Rosa: That’s a known trade-off. The authors acknowledge that this latency might pose challenges in very fast, dynamic scenarios and requires more GPU memory during inference to run smoothly.
Dev: So, the paper says it can operate under dynamic camera conditions by generalizing not just to changes in camera extrinsics but also to variations in other parameters like intrinsics and image characteristics across different camera models.
Taro: It sounds like they are tackling a lot of real-world variation without needing any new demonstrations or retraining cycles, which is a huge practical win for deploying VLAs on robots.
Rosa: The conclusion is that this zero shot camera adaptation framework effectively generalizes to novel viewpoints in real world manipulation scenarios while maintaining task success rates comparable to those achieved with the training viewpoint.
Dev: The title, AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models, points directly at how they are achieving this without the usual heavy lifting of demonstration data or policy fine tuning.
Taro: I think what this means for people using these models is that you don't have to spend all your time collecting massive datasets just to make the model work in a slightly different camera position on a new robot setup.
Rosa: It simplifies the deployment pipeline significantly, making it much more practical for researchers and engineers working outside of perfectly controlled lab settings.
Dev: So, in short, they've found a way to adapt models to arbitrary camera configurations in real time just by using this novel view synthesis approach, which is much more efficient than fine tuning the entire VLA.
Conclusion: Rosa: So, we're looking at AnyCamVLA, and the title says zero shot camera adaptation for viewpoint robust vision language action models. Basically, they figured out how to make a robot model work in a camera position it was never trained on just by adjusting the picture in real time.
Dev: Right. They’re talking about this framework that virtually adjusts test time camera observations to match the training camera configuration without needing any extra training data or policy fine tuning.
Taro: So, no new robot demonstrations required for every new viewpoint? That sounds like a huge win for getting these models out into the real world quickly.
Rosa: Exactly. The core idea is using a feed forward novel view synthesis model to generate those target view images in real time, handling both the camera position and its internal settings.
Dev: And that’s how they build the training signal; they transform the test time observation by feeding it through this module F, so the policy runs on that synthetic image instead of the actual one.
Taro: But what about when things go wrong? If you're driving or cooking and your camera suddenly moves in a weird way, how does this system handle that surprise?
Rosa: Well, they show strong performance on the LIBERO benchmark. They got ninety nine point zero percent success rate on unseen agent camera viewpoints across all those tests.
Dev: And they also achieved eighty eight point six percent average success rate on the most difficult suite involving wrist camera perturbations. That’s solid generalization for a system that hasn't seen that specific setup before.
Taro: What I'm really interested in is the trade off, since we know this runs in real time. They admitted there’s about thirty milliseconds of latency per frame, and it needs more GPU memory during inference.
Rosa: Yeah, that latency is something to watch for when you deploy it on a very fast dynamic system. But they say it still works under dynamic camera conditions, handling changes in intrinsics and image characteristics too.
Dev: So the big picture here is that this method bypasses the major headaches of getting viewpoint generalization across many different tasks or risking catastrophic forgetting from policy fine tuning.
Taro: It makes deployment way more practical because you don't need massive datasets just to make it work in a slightly different camera setup.
Rosa: It really simplifies the entire pipeline, moving us closer to deploying these vision language models in messy real-world scenarios without needing perfect lab conditions or endless training cycles.
Dev: Next up, we're going to look at how they compared this novel view synthesis approach against other geometric methods and why that choice matters for quality.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies