Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer
ACT3 is an Action-Centric Tri-Stream Transformer that combines semantic understanding and physical dynamics predictions to generate control actions. It uses three separate streams—semantic, action, and dynamics—and allows the action stream to query both context streams independently. This fusion improves robotic manipulation performance by leveraging both task knowledge and predictive physics.