Observing and Controlling Features in Vision-Language-Action Models
cs.RO
Submitted: 2026-03-05
Updated: 2026-09-21
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-Language-Action models (VLAs) have shown remarkable progress towards embodied intelligence.
Terminology
Abstract
Vision-Language-Action models (VLAs) have shown remarkable progress towards embodied intelligence. While their architecture partially resembles that of Large Language Models (LLMs), VLAs exhibit higher complexity due to their multi-modal inputs/outputs and often hybrid nature of transformer and diffusion heads. This is part of the reason why insights from mechanistic interpretability in LLMs, which explain how the internal model representations relate to their output behavior, do not trivially transfer to VLA counterparts. In this work, we investigate whether VLA internal representations support lightweight behavioral steering without retraining. Across four frontier VLA models, linear observers recover state- and action-relevant information in both autoregressive and transformer--flow-matching architectures, and provide robust directions to causally alter VLA outputs. Building on this, we introduce a controller that minimally modifies representations to place observer predictions within prescribed target intervals. Closed-loop experiments with π 0.5 in the LIBERO simulator and on DROID hardware demonstrate improved constraint satisfaction while retaining task performance, with only approximately 1% inference overhead. Together, these experiments show that lightweight linear interventions can reliably steer VLA behavior while preserving closed-loop capabilities, enabling alignment with user preferences and task requirements without fine-tuning.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- LiSeCo: Linear Semantic Control for Language Generation
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- Mechanistic interpretability for steering vision-language-action models
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- VLAs are Confined yet Capable of Generalizing to Novel Instructions
- Probing a Vision-Language-Action Model for Symbolic States and Integration into a Cognitive Architecture
- Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
- HyperSteer: Activation Steering at Scale with Hypernetworks
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies