PRISM: Primitive Routing via In-context Skill Mixing for Retentive VLA Adaptation
cs.RO
Submitted: 2026-11-06
Updated: 2026-11-06
Comments: 19 pages, 6 figures, 4 tables. Accepted at NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-Language-Action (VLA) models promise a single language-conditioned policy adaptable to many manipulation behaviors…
Terminology
Abstract
Vision-Language-Action (VLA) models promise a single language-conditioned policy adaptable to many manipulation behaviors, yet deployment additionally requires skill retention and long-horizon composition: a policy must retain previously acquired skills and reuse them inside composite tasks whose behavior changes across phases. The standard recipe absorbs every new task into one shared update, which provides neither a parameter-level locus for reusable skills nor a mechanism for invoking each skill along the trajectory. We observe that this missing structure is latent rather than absent: per-skill low-rank updates of a pretrained VLA occupy a robustly distinguishable, low-dimensional subspace, and a single composite-task LoRA concentrates roughly half of its Frobenius energy in this subspace, against a near-zero matched random-rotation control. Building on this geometry, we introduce PRISM, which represents reusable skills as parameter-level primitives: each skill family is stored as a dedicated low-rank action-expert adapter in a primitive bank, primitives are augmented with skill-specific low-rank residuals and composed at inference by a per-chunk VLM-conditioned router that reads the prefix representation to track task progress, and the router is supervised by per-chunk labels from a deterministic, task-conditioned, action-derived oracle. During composite training, the vision encoder, the language LoRA, the skill residuals, the router, and the shared action projections and time MLP are updated; every primitive is held frozen, so the base per-skill action-expert adapters are preserved by construction. On RoboCasa365, PRISM achieves 56.43% atomic-skill success and reaches 20.00% on long-horizon composite tasks, while a standard π 0.5-LoRA two-stage fine-tune reaches 17.86% atomic and 17.50% composite success. We additionally validate PRISM on an AgileX PIPER real-robot setup.
Sources
- Qwen2.5-VL Technical Report
- PaliGemma: A versatile 3B VLM for transfer
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- RT-1: Robotics Transformer for Real-World Control at Scale
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs
- PaLM-E: An Embodied Multimodal Language Model
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Gemma: Open Models Based on Gemini Research and Technology
- GLaD: Geometric Latent Distillation for Vision-Language-Action Models
- Mixtral of Experts
- CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
- OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning
- Flow Matching for Generative Modeling
- Skill Expansion and Composition in Parameter Space
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Octo: An Open-Source Generalist Robot Policy
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- FAST: Efficient Action Tokenization for Vision-Language-Action Models
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation