TrAct: Bridging Robot Control and Visual Prediction with Visual Tracks
Robot actions are difficult to align with visual changes, limiting world models. TrAct introduces a framework using 'visual tracks'—embodiment-agnostic representations of point movements in a scene—as an interface between robot control and prediction. It combines a Vision-Language-Action model to propose actions and tracks, a track-conditioned world model to predict outcomes, and a reward model to select the best action, significantly boosting task success rates.