Unifying Policy Learning and State Prediction through Spatial Language Modeling
The research introduces Spatial Language Modeling to unify policy learning and state prediction in manipulation tasks like Push-T. It uses a shared vocabulary of discrete coordinates and semantic tokens to represent scene geometry, goals, action targets, and future states. The model learns action generation and state prediction through sequence completion tasks trained on both random interactions and expert demonstrations.