Decoupling the Declarative from the Procedural in Vision-Language-Action Models

arXiv:2606.21496 · cs.RO, cs.AI, cs.CV, cs.LG · Submitted 2026-06-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Decoupling the Declarative from the Procedural in Vision-Language-Action Models".

Rosa: The gist: w2VLA, a new Vision-Language-Action (VLA) model, successfully decouples declarative knowledge from procedural knowledge to enable zero-shot skill transfer to novel objects.

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: So what is the paper actually suggesting is in terms of how we fix this mixed-up knowledge problem? They introduce a new information flow architecture that conditions the robot state sequence compositionally, using visual and skill information in a way that’s interpretable.

Dev: It’s about explicitly isolating "the where," which they define as the declarative, object-centric knowledge encoded by the Vision Language Model, from "the what," which is the specific behavior derived only from demonstration data.

Taro: I like that separation idea. If you can tell the system exactly where something is and separately tell it what action to take, then when you swap out the object, you only need to change its location information.

Rosa: Right. The architecture they describe involves a few modules working in sequence: first there’s encoding the proprioceptive state of the robot into a latent space, which we call hidden robot states.

Dev: Then those states get modulated by visual tokens from the Vision Foundation Model, which they process through something called Attentive Feature Aggregation to compress everything down into task-relevant summary tokens.

Taro: And then they add spatial conditioning, which is where they inject information about the object's location derived from heatmaps that compute similarity scores between visual patches and text embeddings.

Rosa: And finally, they have skill conditioning, where a specific skill embedding is projected into a single vector that broadcasts across the temporal dimension of the rollout.

Dev: This allows the policy to map that localized object representation directly to the motor trajectory dictated by the language instruction. It’s a very structured way to feed information into that action head.

The paper's summary: Rosa: So, summarizing what this paper actually found, it’s that w2VLA manages skill transfer between objects from demonstration data in a way that other models simply can’t do. They show this with an example where they train three different VLAs on a skill-object pair, but only w2VLA reliably performs the skill when you test it on a different object.

Dev: They evaluate this in a skill transfer scenario—for example, if you trained it to rotate by ninety degrees the carrot, and then asked it to do that rotation on a banana <ref:2606.21496#pg2,in a skill transfer scenario>. The other models, pi0 point 5 and OTTER, fail to transfer the learned skill because they don't understand the object difference well enough <ref:2606.21496#pg1>.

Taro: That’s significant because it means we aren't just seeing better in-distribution performance; we are actually seeing a mechanism that allows for true generalization across different entities.

Rosa: They found that all three models perform almost perfectly when deployed over seen pairs, but only w2VLA manages to reliably perform the skill transfer. They also tested its robustness in out-of-distribution scenarios.

Dev: In one test with distractors, their success rates dropped by about sixteen point six percent and thirteen point nine percent. But the drop was mainly due to a lower success rate with the target object itself, not because they picked the wrong object entirely.

Taro: So it’s not just about being smart enough to pick the right thing; it’s about having a better internal structure for understanding what that thing means declaratively before you even try to act on it.

The paper's improvements: Rosa: The authors suggested two main ways to improve this system, focusing heavily on how they train it to make sure the transfer works reliably. First, they keep the foundation models for vision and language encoding frozen throughout policy training.

Dev: Freezing those backbones is important because it prevents them from getting skewed by the low-data regime of imitation learning, keeping their pre-trained representations intact during policy training.

Taro: I think that makes sense because if you constantly fine-tune the vision model based on every little demonstration, you risk diluting the general knowledge it already has.

Rosa: Second, they introduced structured dropout during training to combat visual appearance biases tied to specific skill-object pairs. They mask Visual Foundation Model patches entirely with a probability of q equals zero point five during training.

Dev: That masking is designed to force the model to rely more on the spatial and skill information coming from those new where and what conditioning modules, rather than just relying on specific visual cues for every object.

Taro: That structured dropout sounds like a clever way to encourage the AI to look at the higher-level, decoupled information we set up earlier.

Conclusion: Rosa: So, wrapping up this paper on "Decoupling the Declarative from the Procedural in Vision-Language-Action Models," they show that by structuring how visual and skill info modulates robot states compositionally, w2VLA can reliably perform zero-shot skill transfer to novel objects.

Dev: The main implication for us is that we don't necessarily need massive datasets of demonstrations for every single new object if the underlying VLM already understands the object’s identity well enough.

Taro: It points toward a future where agents can learn how to use their existing world knowledge to adapt their physical actions without needing a complete retraining cycle for every new scenario.

Rosa: They also pointed out that successful skill transfer depends on applying the where visual conditioning earlier in the sequence than the what module, which they found mimics human sensorimotor coordination very well.

Dev: That ordering is crucial because grounding the spatial area of interaction first gives a solid location reference before you tell the model what specific skill to execute for that location.

Taro: It suggests that understanding and explicitly modeling the object's spatial context is a more important prerequisite than just knowing which action to take next.

Rosa: Overall, w2VLA shows how we can unlock a new capacity for compositional generalization by separating the declarative knowledge of what something is from the procedural knowledge of how to manipulate it.

Dev: It moves us away from just scaling models and datasets toward identifying those foundational priors in the world that make true generalist capabilities possible.

University of Edinburgh

cs.RO, cs.AI, cs.CV, cs.LG

Submitted: 2026-06-19

Updated: 2026-10-09

Code: https://github.com/huggingface/lerobot

Project page: https://tsagkas.github.io/w2vla

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist: w2VLA, a new Vision-Language-Action (VLA) model, successfully decouples declarative knowledge from procedural knowledge to enable zero-shot skill transfer to novel objects.

Key concepts

Declarative Knowledge
This refers to the 'what' and 'where' of a task—the conceptual understanding of objects and entities provided by the Vision-Language Model (VLM). It encodes the semantic information about what an object is, its properties, and its spatial location in the scene. This knowledge is largely static and comes from pre-trained VLMs.
Procedural Knowledge
This refers to the 'how' of a task—the specific sequence of motor actions required to achieve a goal. In w2VLA, this procedural knowledge is derived exclusively from demonstration data through Imitation Learning. It represents the learned skill or behavior needed to execute a specific manipulation.
Spatial Conditioning (where)
This mechanism injects information about the object's location into the policy. It uses heatmaps to calculate similarity scores between visual patches and textual object embeddings, creating spatial tokens that guide the policy on where in the scene it needs to focus its actions for manipulation.
Skill Conditioning (what)
This component conditions the model on the intended behavior or 'what' action to perform. It takes skill embeddings derived from demonstrations and projects them into a single vector broadcast across time, telling the policy which specific motor trajectory it should follow for that task.

Terminology

Summary

The gist: w2VLA, a new Vision-Language-Action (VLA) model, successfully decouples declarative knowledge from procedural knowledge to enable zero-shot skill transfer to novel objects.

Motivation

The pursuit of generalist, autonomous robot agents requires policies that can generalize beyond specific object demonstrations for data collection requirements to become tractable. Current state-of-the-art VLAs, which fine-tune deep Vision-Language Models (VLMs), remain brittle to minor spatial, semantic, and task variations. This brittleness is attributed to the inability of current models to decouple the declarative (i.e., concepts and entity semantics) from the procedural knowledge (i.e., how to do something) encoded in their parameters. The primary contribution of this work is a novel, end-to-end trainable VLA model that, to the best of our knowledge, is the first to achieve zero-shot skill transfer to unseen objects.

How it works

w2VLA introduces a restructured information flow that modulates the robot state sequence with visual, spatial, and skill information in a compositional and interpretable manner. This approach addresses the limitation of current VLAs where all multimodal tokens from the VLM encoder are fed into a large, opaque transformer-based action expert. The model explicitly isolates two critical components of an interaction: the where, which constitutes the declarative object-centric knowledge encoded by the VLM, and the what, i.e., the specific behavior to be executed, which is derived exclusively from demonstration data.

The architecture involves several key modules that sequentially condition the action prediction process:

  1. Proprioceptive State Encoding: The proprioceptive observation sequence P is first mapped via an MLP (i.e., the Proprio Encoder) into the policy’s D-dimensional latent space, yielding hidden robot states H.

  2. Visual Modulation: Hidden robot states H are first modulated with the dense VFM patch tokens F, which are processed through Attentive Feature Aggregation (AFA) to compress visual features into task-relevant summary tokens F'. This is followed by a Causal Self-Attention (CSA) block for temporal reasoning.

  3. Spatial Conditioning (where): Spatial conditioning is achieved by injecting a sequence of tokens Σ spatial into the first conditioning block, derived from spatial localization heatmaps Mτ that compute raw similarity scores between VLM visual patch embeddings and the textual object embedding. These heatmaps are projected into the transformer’s hidden dimension to form the final conditioning tokens σ spatial.

  4. Skill Conditioning (what): To condition the policy on behavior intent, skill embeddings eskill are projected via an MLP into a single skill vector σ skill, which is broadcast across the temporal dimension T of a rollout. This allows the policy to map localized object representation to the specific motor trajectory dictated by the language instruction.

  5. Action Head: Finally, the fully conditioned sequence of hidden robot states is processed by an MLP that predicts a chunk of future actions Aˆ = at+1, at+2,..., at+L.

Training and Robustness

The training recipe for w2VLA complements its architecture with two key elements to facilitate zero-shot skill transfer:

  1. Foundation Model Freezing: The foundation models employed for vision and language encoding (i.e., VFM and VLM backbones) remain frozen throughout policy training, ensuring their pre-trained representations remain undiluted by the traditionally low-data regime of Imitation Learning.

  2. Structured Dropout: To mitigate visual appearance biases tied to specific skill-object pairs, a structured dropout approach is applied during training, masking VFM patches entirely with a probability q = 0.5. This encourages the model to rely on the rich spatial and skill information from the newly introduced where and what conditioning modules.

The model's robustness was tested in two out-of-distribution cases:

  1. Robustness to distractors (Scenario 1 with Distractor Objects): w2VLA retained competitive performance in both settings, with success rates decreasing by 16.6% and 13.9%. This drop was primarily driven by a lower interaction success rate with the target object, not wrong object selection.

  2. Skill transfer to novel objects (Scenario 2 with Unseen Objects): w2VLA maintained its high success rate, showing a decrease in success score by 8.3%. This was attributed to the ability of the model to effectively leverage the encoded declarative knowledge of the backbone VLM.

Ablation and Conclusion

Ablations confirmed that omitting visual modulation qualitatively enabled skill transfer but resulted in imprecise manipulation, while including it improved in-domain performance without reliably enabling skill transfer. Furthermore, successful skill transfer heavily relied on applying the where visual conditioning earlier in the sequence than the what module. In conclusion, w2VLA successfully unlocks a new capacity for compositional generalization beyond what is currently possible with popular VLAs by decoupling and sequentially processing the declarative and procedural information of each target task. The model relies on the declarative knowledge of pre-trained VLMs, and the procedural knowledge that is encoded in our policy via Imitation Learning. This work encourages the community to look beyond scaling models and datasets, and towards identifying the foundational priors and biases necessary to unlock true generalist capabilities. The limitations include evaluating only primitive robot skills, but w2VLA is well-suited to operate within a hierarchical VLA framework. Future extensions could add a “how” module, conditioning the policy on unique, fine-grained geometric features. This work successfully unlocks an new capacity for compositional generalization beyond what is currently possible with popular VLAs.

--- Page 1 ---

Decoupling the Declarative from the Procedural in Vision-Language-Action Models

Training and Robustness

The training recipe for w2VLA complements its architecture with two key elements to facilitate zero-shot skill transfer:

  1. Foundation Model Freezing: The foundation models employed for vision and language encoding (i.e., VFM and VLM backbones) remain frozen throughout policy training, ensuring their pre-trained representations remain undiluted by the traditionally low-data regime of Imitation Learning.

  2. Structured Dropout: To mitigate visual appearance biases tied to specific skill-object pairs, a structured dropout approach is applied during training, masking VFM patches entirely with a probability q = 0.5.

Improvements for AI systems

  1. textbfWeight-Decoupled Information Flow for Robust Generalization: Introduce w2VLA architecture to decouple declarative from procedural knowledge by sequentially modulating past proprioceptive robot states with visual, spatial, and skill information in a compositional and interpretable manner. This allows the policy to reliably direct the robot to manipulate the target object, which does not have to be the one the behavior was demonstrated on while maintaining in-domain behavior cloning performance on par with state-of-the-art VLAs.

  2. textbfZero-Shot Skill Transfer Capability: Enable robust skill transfer to novel objects by explicitly isolating the where (declarative object-centric knowledge) from the what (procedural knowledge). This is achieved by conditioning the action expert on decoupled representations of the target object's location and the required skill, which prevents overfitting on spurious skill-object correlations.

  3. textbf Enhanced Robustness to Distractors: Improve in-domain performance under visual noise by employing a structured dropout approach during training, specifically masking VFM patches entirely with a probability q = 0.5. This encourages the model to densely rely on the rich spatial and skill information from the newly introduced where and what conditioning modules rather than relying solely on specific visual cues.

  4. textbf Optimal Conditioning Sequence: Implement a critical ordering of conditioning modules to maximize transfer success by ensuring the where visual conditioning earlier in the sequence. This mirrors human sensorimotor coordination, grounding the spatial area of interaction before the what module specifies the target skill to be executed, leading to higher skill transfer rates (e.g., 94.4% ST for VM+Mask → where → what).

Sources

Related papers