Say the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAs
cs.LG, cs.CR, cs.CV, cs.RO
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: Accepted to NeurIPS 2026. 23 pages, 3 figures, 8 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reasoning vision-language-action (VLA) models pair physical control with natural-language rationales that supervisors can read to monitor embodied safety.
Terminology
Abstract
Reasoning vision-language-action (VLA) models pair physical control with natural-language rationales that supervisors can read to monitor embodied safety. Prior attacks on VLA reasoning corrupt the rationale itself and leave visible evidence in the text. We show that visual perturbations can instead alter the physical action while keeping the generated rationale word-for-word identical. At the shared text/action interface of Alpamayo for driving and InstructVLA for manipulation, our text-neutral action-subspace (TNAS) diagnostic shows that the locally most action-sensitive direction is nearly orthogonal to the active text-loss gradients, with 1.87% and 5.28% norm overlap. At equal hidden-state perturbation norms across both models, interventions along this text-neutral residual shift actions 4.64x to 47.8x more than text-gradient perturbations, whereas text-gradient perturbations induce 21.2x to 277x more text drift. Steering visual perturbations along this direction reaches the same decoupling from the image. Across 154 obstacle-annotated driving scenes in Alpamayo, TNAS-guided attacks induce a new collision under a word-for-word identical rationale in 52.6% of scenes under full-image perturbations and 35.1% under a localized forward-camera patch, compared with 4.55% for unguided action attacks. Relative to a text-preserving baseline, TNAS guidance raises exact rationale preservation from 83.0% to 89.5% and cuts teacher-forced text drift by 68.4% at a comparable collision rate. In InstructVLA, TNAS guidance cuts text drift by 40.5% and preserves the exact response on 85.0% of examples alongside a 1.73-unit action-chunk shift. An unchanged rationale therefore cannot certify a safe action, so monitors for reasoning VLAs must verify the predicted action channel directly.
Sources
- DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
- Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving
- Robotic Control via Embodied Chain-of-Thought Reasoning
- Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
- InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
- TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches
- Altered Thoughts, Altered Actions: Reasoning Chain as Control Surface for a Vision-Language-Action Policy
- Qwen3 Technical Report
- RT-1: Robotics Transformer for Real-World Control at Scale
- PaLM-E: An Embodied Multimodal Language Model
- OpenVLA: An Open-Source Vision-Language-Action Model
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
- ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
- DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning
- VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models
- Adversarial Patch
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis