Rethinking Causal Action Tokenization with Conditional Annealing in Flow Matching
cs.RO, cs.AI, cs.LG
Submitted: 2026-09-28
Updated: 2026-09-28
Project page: https://chenyuzhangx.github.io/CATok/Abstract
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autoregressive Vision-Language-Action (VLA) models offer a scalable path to robot learning, yet existing action tokenizers treat tokenization as a compression problem, producing representations that
Terminology
Abstract
Autoregressive Vision-Language-Action (VLA) models offer a scalable path to robot learning, yet existing action tokenizers treat tokenization as a compression problem, producing representations that are semantically misaligned with the autoregressive backbone. We propose CATok, a causal action tokenizer that reframes tokenization as a causally structured generative process. CATok introduces a conditional annealing mechanism that extracts action tokens by progressively annealing a flow-matching process: each token is conditioned on all preceding tokens and encodes the residual reconstruction signal at a specific noise level, establishing a coarse-to-fine causal token space whose generative semantics are structurally aligned with autoregressive modeling. A token-conditioned flow-matching decoder built on Multimodal Diffusion Transformer (MMDiT) reconstructs continuous action chunks from these discrete tokens with the precision of hybrid diffusion-head architectures. This discrete bottleneck enforces knowledge insulation by design, cleanly separating high-level semantic reasoning from low-level motor execution without requiring explicit attention masking. Extensive evaluations across three simulation benchmarks and real-world robotic manipulation tasks demonstrate that CATok consistently surpasses existing tokenization methods in both reconstruction fidelity-compression tradeoff and inference efficiency, while improving VLA task success rate and training efficiency, establishing a high-performance, scalable foundation for purely autoregressive VLA systems.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- OpenVLA: An Open-Source Vision-Language-Action Model
- PaLM-E: An Embodied Multimodal Language Model
- Octo: An Open-Source Generalist Robot Policy
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
- Motus: A Unified Latent Action World Model
- SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model
- Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training
- FAST: Efficient Action Tokenization for Vision-Language-Action Models
- FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenization
- OAT: Ordered Action Tokenization
- Exploring Diffusion Time-steps for Unsupervised Representation Learning
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- PaLI-X: On Scaling up a Multilingual Vision and Language Model
- PaliGemma: A versatile 3B VLM for transfer
- VLA-0: Building State-of-the-Art VLAs with Zero Modification
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies