SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenes

arXiv:2603.29798 · cs.CV · Submitted 2026-03-31 · Read on arXiv

cs.CV

Submitted: 2026-03-31

Updated: 2026-09-17

Comments: Project page: https://sceneteract.github.io/

Code: https://github.com/PhyScene/PhyScene

Project page: https://sceneteract.github.io

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: Indoor 3D scenes are ultimately meant to be used: an embodied agent should be able to navigate, reach objects, and complete diverse activities.

Terminology

Abstract

Indoor 3D scenes are ultimately meant to be used: an embodied agent should be able to navigate, reach objects, and complete diverse activities. Yet whether a given scene actually supports these activities for a specific agent profile is rarely verified. Existing evaluations of indoor 3D scenes typically focus on visual quality and semantic plausibility. In contrast, the feasibility of an activity depends on geometric, agent-specific constraints such as reach, clearance, and navigable space availability. These properties are not captured by visual plausibility metrics, and, as we show, VLMs, which are increasingly used to reason about 3D scenes, often fail to determine action feasibility in a single shot. We present SceneTeract, a verification interface that separates semantic action understanding from physical feasibility. Given a scene, an activity, and an embodied agent profile, we decompose the activity into atomic actions on scene objects. Explicit geometric checks then decide whether each step is executable and return a diagnostic trace explaining failures. In synthetic indoor scenes, SceneTeract reveals widespread functional and accessibility failures across diverse agent profiles. Moreover, when benchmarked against our verification, we find that existing VLMs systematically over-predict action feasibility, highlighting limited awareness of embodied functional constraints. In response, we post-train a lightweight VLM with verifier feedback, improving its assessment of physical feasibility. Although trained only on renders of synthetic scenes, we demonstrate that scene understanding improvements also generalize to real-world scenes. We will release our verification suite, benchmark labels, and diagnostic trace datasets.

Sources

Related papers