TrAct: Bridging Robot Control and Visual Prediction with Visual Tracks

The gist Robot actions are inherently embodiment-specific and only weakly aligned with image-space visual changes, limiting their effectiveness as conditioning signals for robot world models, but visual tracks provide an embodiment-agnostic representation of how task-relevant points move through a scene, offering dense image-space guidance for accurate and spatially precise future video prediction <ref:2608.24101#pg6>.

TrAct Framework Overview

TrAct is a worldmodel-based robot decision-making framework that uses visual tracks as an intermediate interface between control and prediction. It consists of three components:

1. A Vision-Language-Action-and-Track model (VLAT) that jointly predicts candidate actions and corresponding visual tracks from the current observation and language instruction.

2. A track-conditioned world model (TWM) that predicts future visual outcomes conditioned on the proposed tracks.

3. A vision-language reward model (VLAC) that scores the predicted outcomes.

At inference time, VLAT generates candidate action–track pairs, TWM rolls out their visual consequences conditioned on the tracks, and VLAC selects the track whose predicted outcome best satisfies the instruction.

Policy Architecture and Training

The Vision-Language-Action-and-Track Policy (VLAT) is built on top of a pre-trained flow-matching VLA policy π0.5, modifying its action head to jointly output executable robot actions and 2D point tracks. The VLAT policy is optimized using a joint flow-matching objective: LVLAT = Lflow(a, a∗) + λLflow(τ, τ ∗). It predicts tracks for two types of points: (1) key mesh vertices of the robotic gripper, and (2) uniformly sampled background and object points on a 5×5 grid.

VLAT is pretrained on large-scale real-world robot datasets, including 76K DROID trajectories and 150K EgoDex trajectories using a 1:2 dataset mixing ratio. For track supervision, seven predefined gripper points corresponding to fixed 3D offsets on the end-effector mesh are used, and a uniform 5×5 grid of background points is tracked over time using CoTracker.

World Model Comparison

The track-conditioned world model (TWM) consistently outperforms action-conditioned world model baselines (AWM) for future prediction in both settings. TWM produces more spatially accurate and temporally coherent predictions across both agent-view and wrist-view observations, while AWM often exhibits incorrect gripper poses, disappearing objects, and cross-view object position inconsistencies.

Experimental Results

Experiments on the LIBERO-INTEGRAL benchmark show that TrAct improves success from 49% to 55% in simulation and 49% to 76% on real-world tasks compared to the strong VLA baseline π0.5. Furthermore, TWM consistently improves video prediction quality over AWM. On physical Franka Panda robot experiments, TrAct achieves the best overall result of 98.3% compared to VLAT+AWM achieving 98.0%.

Key Contributions

The main contributions are:

• We propose 2D point tracks as an intermediate representation between robot policies and world models, improving both planning and future prediction.

• We introduce TrAct, a framework that uses a VLAT policy to jointly predict visual tracks and actions, and conditions a world model on tracks for reward-guided action selection.

• We validate TrAct in simulation and on physical robots, showing that TWM improves future prediction over action-conditioned alternatives and that TWM-based action selection increases downstream task success.

Conclusion

Visual tracks provide an effective shared interface between robot control and visual prediction, enabling more accurate world modeling and stronger robot generalization. The results suggest that visual tracks provide an effective intermediate representation between planning and execution, improving both future prediction and task completion.

Pseudo-code for Inference

Algorithm 1 summarizes the inference procedure of TrAct, which involves proposing candidate action–track pairs from VLAT, rolling out their visual consequences via the track-conditioned world model, and selecting the action chunk whose predicted rollout best satisfies the instruction using VLAC. The inference procedure is summarized as follows:

1. while task not finished do

2. (ai, τi) K i=1 ← PROPOSE(πθ, ot, l, K) ▷ sample candidate action–track pairs

3. for i = 1 to K do

4. vˆi ← pϕ(ot, τi) ▷ roll out future video with predicted tracks

5. si ← Rψ(ˆvi, l) ▷ score predicted outcome

6. end for

7. i⋆ ← arg maxi si

8. Execute selected action chunk ai⋆

9. Observe new state ot+H

10. t ← t + H

Unified Track Representation

The unified embodiment-agnostic track representation uses a fixed slot layout to denote a possible track type, such as a gripper/hand keypoint or a grid-sampled scene point. Each slot corresponds to a possible track type, and the model is encouraged to organize observations according to interaction dynamics rather than robot morphology or dataset-specific action formats.

Pretraining Data

The main TrAct model is pretrained on DROID and EgoDex. The pretraining data includes 76K DROID trajectories and 150K EgoDex trajectories using a 1:2 dataset mixing ratio.

Scaling Pretraining Mixture

TrAct+ is pretrained on a mixture of 76K DROID [39] trajectories, 60K BridgeData V2 [41] trajectories, and the full EgoDex [40] dataset. This stronger variant improves the average success rate from 0.32 to 0.38 on the five most challenging UR5 cross-embodiment tasks.

Candidate Scaling

TrAct selects among K candidate action–track pairs at each planning step, where K is varied in simulation experiments from K = 5 to K = 20. Success improves monotonically with K but with diminishing returns, rising by 2 points from K=5 to K=10 and by a further 1 point from K=10 to K=20.

Physical Experiment Robustness

On physical experiments, TrAct averages 0.76 versus 0.66 for VLAT+AWM. Under the unseen background, the gap widens: VLAT+AWM drops to 0.4, matching VLAT without reranking, while TrAct remains at 0.7.

Final Summary

TrAct demonstrates that visual tracks provide a robust interface between robot control and visual prediction. The results show that track conditioning is consistently stronger, especially under visual domain shift, anchoring the world model to image-space motion that remains robust to background changes. This supports the central hypothesis that visual tracks provide a robust interface between robot control and visual prediction, improving generalization under observation, environment, and embodiment changes. The framework enables reward-guided action selection grounded in predicted image-space dynamics <ref:2608.

1. textbf Real-time multimodal action selection via visual tracks in TrAct: This system can select actions by conditioning a world model on visual tracks as an intermediate interface between control and prediction, enabling reward-guided action selection grounded in image-space dynamics.

2. textbf Improved future video prediction quality through track conditioning: The Track-Conditioned World Model (TWM) produces more spatially accurate and temporally coherent predictions across both agent-view and wrist-view observations compared to action-conditioned alternatives, which often exhibit incorrect gripper poses, disappearing objects, and cross-view object position inconsistencies.

3. textbf Enhanced generalization under distribution shift: TrAct demonstrates superior robustness to changes in observation, environment, and embodiment shifts; specifically, Track conditioning is consistently stronger: TrAct averages 0.76 versus 0.66 for VLAT+AWM, and it shows the gap appears even without domain shift on contact-rich tasks.

4. textbf Robustness to visual domain shift: The system maintains performance under background changes, as visual tracks anchor the world model to image-space motion that remains robust to background changes, leading to a significant gain in success rates when moving from training backgrounds (R) to visually distinct backgrounds (D).

5. textbf Scalable and diverse candidate sampling: By selecting from K candidates at each step, TrAct manages complexity by varying the number of sampled candidates, showing that success improves monotonically with K but with diminishing returns, allowing for a trade-off between compute and the quality of the best available candidate.