Decoupling Intention from Trajectory: A Representational Deduction Framework for World Action Models

The paper addresses a structural bottleneck in World Action Models (WAMs), which aim to construct a unified architecture capable of understanding world state evolution and guiding to generative motion planning. The authors identify a critical flaw in existing WAMs: existing visual branches focus on predicting static visual observation, rather than reflecting potential transition information that captures the evolution of world states under motion interactions. This leads to "representational entanglement between high-level physical condition evolution and low-level action trajectory generation within the Action Model, creating a structural bottleneck while weakening the predictive capability of world evolution modeling for action generation."

The authors note that "future states are confined to the visual latent space constructed by frozen VAEs. This structural bottleneck causes the joint optimization objective of WAMs to primarily target visual reconstruction over trajectory generation. Furthermore, static visual representations at a single moment fail to reflect the evolution process of physical states under action interactions. Without explicit modeling of state transitions, directly generating actions through the diffusion process leads to representational entanglement. Existing Latent Action Models (LAMs) are also criticized because this kind of visual-aware-only supervision, fails to prompt the model to understand what action conditions led to the variation in the visual state."

The authors propose PILOT (Physical Inference for Latent Optimized Trajectories), "an end-to-end framework grounded in the Representational Deduction (RD) mechanism that bridges the gap between static visual prediction and dynamic action manipulation by establishing motion semantic CoT reasoning as an innate capability of WAMs."

The theoretical formulation factorizes the policy as:

pΘ(at ot, l, st) = ∫ pΘ(at m, st) × pΘ(m ot, l, st) dm,

where m ∈ RK×d is a latent motion-semantic variable. The central thesis is that the quality of a world-action model hinges on whether m captures the action-conditioned state transition ot → ot+∆.

Architecturally, PILOT comprises three cooperating components:

1. A World-Model branch built on a pre-trained Wan2.2, which encodes the current observation and instruction into vision-language context and provides auxiliary future-frame supervision. The context is obtained as c = WMθ(ot, l) ∈ RNc×dc, where the pre-trained video prior in WMθ provides visual and physical cues for downstream action generation.

2. An Action Model branch — a from-scratch trained Action Model branch that distills motion-semantic context with learnable query tokens and decodes actions through causally-decoupled flow matching. It uses K learnable query tokens Q ∈ RK×d, and a Perceiver-style flow-matching head. The authors introduce a Causally-decoupled attention mechanism: To prevent motion semantics from absorbing diffusion noise, we decouple the attention inside each Perceiver block. The non-action latents Z = [S; Q] update as Z ← Z + Attn(Z, [c; Z]), while action tokens follow Aτ ← Aτ + Attn(Aτ, [c; Z; Aτ]). This asymmetric flow lets motion semantics condition action decoding while blocking reverse noise leakage, enabling the model to focus on trajectory refinement. The action decoding uses a flow-matching velocity objective: Lact = Eτ,ϵ ‖Projact(O[:, K+1:,:]) − (at − ϵ)‖2.

3. A Representational Deduction (RD) branch — a RD branch that supervises the motion semantics by predicting VJEPA2-AC future-state representations with a trainable CDE. The Causal Dynamics Engine (CDE) Gξ is initialized from the pre-trained VJEPA2-AC predictor and conditioned on motion semantics and robotic state: r̂t+∆ = Gξ(rt; m̃, s̃) ∈ RP×dr. The supervision uses a representational loss: LRD = SmoothL1(r̂t+∆, rt+∆). Because the supervision target is representational rather than pixel-level, RD teaches the model how the state evolves under the action, rather than merely what the future looks like.

The overall training objective combines three losses: L = Lact + λwm Lwm + λRD LRD, where Lwm is a change-weighted future-frame latent prediction loss.

The paper summarizes three contributions:

• "We propose PILOT, a novel WAM framework that decouples high-level motion intention from low-level trajectory generation via a Representational Deduction mechanism, and internalizes latent state transition information as Motion CoT to alleviate representational entanglement."

• "We introduce a Causal Dynamics Engine with causally-decoupled attention, that endows the Action Model with explicit physical causality learned from VJEPA latent spaces, while bypassing the computational bottleneck of pixel-level future frame reconstruction during inference."

• "PILOT achieves state-of-the-art performance with 97.9% and 62.6% success rates on LIBERO and RoboCasa-GR1 benchmarks respectively, and attains an 83.1% average success rate in real-world manipulation tasks, demonstrating superior generalization and 90% lower inference latency compared to predict-then-act paradigms."

LIBERO: "PILOT achieves 97.9% average success rate with only 5.4B parameters, surpassing all baselines including Motus (8B, 97.7%) and π0.5 (3.5B with internet-scale pre-training, 96.2%). Notably, PILOT excels on the challenging Goal (98.1%) and Long-horizon (97.2%) suites."

LIBERO-Plus (perturbation robustness): PILOT achieves 81.0% total success rate, surpassing the previous best PokeVLA (79.3%) by 1.7% absolute. The improvement is particularly pronounced on Robot (70.0% vs. 46.1%) and Layout (82.0% vs. 77.2%).

RoboCasa-GR1: "PILOT achieves 62.6% average success rate, outperforming all baselines including FastWAM (56.7%, 6B) and LDA (55.4%, 7B). The performance gap is largest on PnP Novel From Tray (74.8% vs. 55.1%) and PnP Close (59.7% vs. 56.3%)."

Deployed on an Agibot-G1 dual-arm humanoid robot:

• Standard Tasks: 8 pick-and-place tasks, PILOT achieves 83.1% average success rate, surpassing Fast-WAM (73.3%) and π0.5 (71.3%).

• Generalization Transfer (Flash+Cam.Offs., Desk, Color): "PILOT achieves 68.3% generalization average, compared to 50.0% for Fast-WAM and 48.0% for π0.5. The 18.3% gap over Fast-WAM indicates that the Representational Deduction branch encodes transition dynamics rather than superficial visual appearance."

• Few-Shot Fine-Tuning (10% data): PILOT retains 62.4% average success rate, a modest 20.7% relative drop from the full-data 83.1%, whereas Fast-WAM drops from 73.3% to 40.8% (44.3% relative decline).

The decoupled design achieves a major speedup over predict-then-act paradigms. On RoboCasa: the decoupled pipeline runs at 111.60 ms per step (8.26 Hz), while predict-then-act requires 1329.95 ms (0.75 Hz)—an 11.9× slowdown, corresponding to 90% lower inference latency. This is because the generation pipeline is only activated during training (to provide future-frame supervision via Lwm) and is entirely bypassed at inference.

The cumulative ablation (Table 5) shows incremental gains on LIBERO/RoboCasa: bare baseline 91.3/51.2; +future-frame prediction 93.6/54.8; +Motion-CoT 94.5/56.7; +RD 96.1/59.8; +CDE 96.9/61.3; +causally-decoupled attention 97.9/62.6. This confirms that the two complementary objectives—pixel-level visual grounding and semantic-level transition prediction—jointly achieve optimal performance.

t-SNE visualization reveals Motion-CoT tokens naturally group by action type without any explicit clustering supervision, demonstrating that PILOT decouples motion semantics from background and object appearance variations. PCA visualization of CDE predictions shows only the foreground regions undergoing manipulation exhibit significant color changes, while static background regions remain largely unchanged, providing direct evidence that the Representational Deduction branch successfully decouples action-conditioned dynamics from irrelevant background information.

The paper concludes: "We present PILOT, a framework that decouples high-level motion semantics from low-level trajectory generation via Representational Deduction. By compressing state transitions into Motion-CoT, PILOT mitigates representational entanglement in World Action Models, achieving real-time efficiency and strong few-shot transferability. Future work will extend RD to long-horizon planning by incorporating hierarchical Motion-CoT for multi-stage embodied tasks."

Based on the paper, here are specific improvements we can make to AI systems and the resulting capabilities:

1. Add a motion-semantic CoT bottleneck between perception and action generation. Instead of feeding visual features directly into a diffusion policy, we insert a latent motion-semantic variable that is forced to predict the action-conditioned state transition, not just the next-frame appearance. This decouples high-level intention from low-level trajectory details.

2. Supervise with representation-space future states rather than pixel-space future frames. Use a frozen, pre-trained world representation model (e.g., VJEPA-style) as a supervision target: the system learns to predict the change in abstract state representations under an action. This avoids the compute cost of pixel reconstruction while capturing physical causality.

3. Use a causal dynamics engine (CDE) to explicitly model state evolution. A lightweight, pre-initialized predictor consumes current state representations plus motion-semantic tokens and produces the predicted future representation. This gives the action model a direct, differentiable physical consequence signal.

4. Implement causally-decoupled attention in the action head. Split the latent state into world/context tokens and action trajectory tokens. Let context tokens condition action tokens, but block attention from noisy action tokens back into the motion semantics. This prevents diffusion noise from corrupting the motion-intention representation during training.

5. Bypass future-frame generation at inference time. Train with future-frame prediction only as an auxiliary loss, but at deployment, run only the action model + causal dynamics engine. This gives a decoupled pipeline with ~90% lower latency than predict-then-act systems.

6. Add change-weighted future-frame loss. Weight future-frame reconstruction to focus on regions that actually change under manipulation, teaching the system to ignore static background and attend to actionable dynamics.

7. Exploit representational grouping via unlabeled motion semantics. The motion-semantic tokens naturally cluster by action type even without clustering supervision, enabling zero-shot action classification, semantic grounding, and interpretable intermediate representations.

• Achieve higher task success with fewer parameters: The improved system can reach ~98% success on LIBERO and ~63% on RoboCasa-GR1, surpassing much larger models, because it learns the transition dynamics rather than memorizing visual appearance.

• Generalize robustly to environment perturbations: It can maintain high success rates when the robot type, background layout, object color, camera offset, or flash lighting changes, because the motion semantics are decoupled from superficial visual features. It outperforms prior systems by ~18% absolute in real-world generalization transfer.

• Operate in real-time on physical robots: By skipping pixel-level future-frame prediction during inference, it can run at ~8 Hz rather than ~0.75 Hz, enabling reactive closed-loop control on dual-arm humanoid robots.

• Learn new tasks from very little data: After fine-tuning on only 10% of task demonstrations, it retains ~62% success rate, while existing models collapse to ~41%. This is useful for quickly adapting to new manipulation scenarios.

• Provide interpretable motion intention: Because the motion-semantic tokens encode the kind of motion (e.g., pick, place, push) rather than the visual scene, the system can be inspected via clustering or PCA to see what high-level physical intention it is carrying out, even before the trajectory is decoded.

• Explicitly reason about what action causes what state change: The causal dynamics engine lets the system anticipate the next latent state under its own motion intention, enabling it to verify that an action will produce the intended physical effect, not merely render a plausible future frame.

• Scale efficiently to long-horizon tasks: By factoring out state-transition prediction from trajectory generation, the architecture can be extended to hierarchical motion semantics for multi-stage embodied tasks, with each stage carrying its own compact motion-CoT.

World Action Models (WAMs) aim to construct a unified architecture capable of understanding world state evolution and guiding to generative motion planning. However, existing visual branches focus on predicting static visual observation, rather than reflecting potential transition information that captures the evolution of world states under motion interactions. This leads to representational entanglement between high-level physical condition evolution and low-level action trajectory generation within the Action Model, creating a structural bottleneck while weakening the predictive capability of world evolution modeling for action generation. We propose PILOT (Physical Inference for Latent Optimized Trajectories), whose core Representational Deduction (RD) bridges this gap by integrating motion thought-of-chain (CoT) guidance as a native model capability. Specifically, RD aims to encourage the action branch to explicitly model potential state transition tokens, which are retained as CoT in the reasoning space to guide fine-grained motion trajectory. Experiments demonstrate that RD not only significantly improves the success rate and generalization ability of WAMs in complex robotic manipulation tasks but also enhances the model's physical interpretability by decoupling high-level motion semantics from low-level trajectory details. Furthermore, the abundant state transition supervision signals introduced by RD effectively alleviate the sparse supervision in action generation, enabling it to serve as an efficient few-shot real-robot fine-tuning strategy and demonstrating superior scalability for migration to mainstream WAM architectures.