Learning to Think and Explore in 3D World
Act3D enables vision-language models to think and explore in 3D space by treating 3D representations as interactive environments. The model learns a policy by alternating between reasoning and interacting with the environment, acquiring visual evidence through movement. This outcome-driven reinforcement learning allows the model to learn not just how to reason over existing views, but also which new views it needs to acquire to answer a question.