MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes

arXiv:2611.15937 · cs.RO, cs.CV · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes".

Dev: Here's the gist of the paper. Vision-Language-Action (VLA) models have demonstrated strong performance in robotic manipulation, yet most rely primarily on 2D visual observations and language instructions, leaving scene geometry and obstacle relationships to be inferred implicitly.

Rosa: The proposed MOSAIC framework augments the VLA backbone with auxiliary semantic grounding tasks that predict role-specific masks for the target object, destination region, and surrounding obstacles.

Dev: This method constructs role-specific geometric tokens from target and destination masks combined with depth estimated from 3D scene reconstruction to inject them directly into the action expert.

Rosa: Furthermore, obstacle-aware attention supervision emphasizes obstacles near the robot body during task execution by injecting resulting attention features into later action-expert layers.

Paper summary: Rosa: So, we're looking at this paper called MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes. Basically, it tackles a problem with Vision-Language-Action models—they work well with 2D pictures and language instructions but they guess the three dee shape of the scene and where everything is just by looking at the picture <ref:2611.15937#pg1>.

Dev: And that guessing game is really risky when things get cluttered, you know? Small errors in spatial understanding can lead to a robot bumping into something or crashing. So, this paper proposes MOSAIC as a modular framework to fix that by giving the models explicit geometric information about what's around them.

Taro: I'm interested in how they handle those surrounding obstacles because that’s where the real-world failures happen when you're trying to grab something in a messy room.

Rosa: Exactly. The core idea of MOSAIC is adding an auxiliary task to the existing VLA backbone so it learns to predict role specific masks for three things: the target object, where the destination region is, and all the surrounding obstacles.

Dev: And they don't just stop there; they combine those target and destination masks with depth information from a three dee scene reconstruction to build these role specific geometric tokens <ref:2611.15937#pg1,target and destination masks with depth>. These tokens are then put directly into the action expert to guide what the robot does.

Taro: So, it’s injecting this explicit geometry right into the action generation process instead of just letting the model infer it implicitly through its standard layers. That sounds like a solid way to handle clutter.

Rosa: Right. And they also propose something where they make the VLA attention focus more on those surrounding obstacles during task execution, which is important because when you're actually moving, you need to pay close attention to potential collisions near your body.

Dev: That means they are re-injecting features from that obstacle-aware attention into later layers of the action expert so it helps guide the robot's movement in those cluttered spots. It’s about making the policy smarter about where to look during its steps.

Taro: So, if I were driving or cooking, I’d want that model to know exactly where my hands are relative to a cupboard or another person, not just what's visible on the screen.

Rosa: Precisely. The paper shows that applying these modules to models already trained for tasks like RoboPRO actually improves performance in cluttered scenes by five point one percent and by nine point seven percent when there are visual changes happening.

Dev: That improvement is significant, especially when you look at the most challenging environments where the gains hit up to twenty five point zero percent. The paper also notes that the full model gets a nine point nine percent boost in success rate and a seven point one percent boost in human success rate specifically in an office environment tested on RoboPRO.

Paper summary: Taro: It’s interesting how much robustness they see when testing against visual changes like lighting or blur, because the spatial structure they learned seems to stay relevant even when the picture gets messy.

Rosa: That suggests that these explicit grounding modules are providing a level of task relevant spatial structure that holds up better than just relying on raw pixels. It confirms their idea that the model learns to identify those target object and surrounding obstacle roles consistently across different visual conditions.

Dev: The authors also show evidence through ablation studies, where they break down each module to see exactly what part contributes most to the performance boost, which helps prove that the modular approach actually adds value. It’s not just one big thing doing all the work, but specific pieces adding their own layer of help.

Taro: What about when you think about putting this out into the real world? Does this framework work well outside of a clean lab setting, or does it need that perfect three dee scene reconstruction data they mentioned <ref:2611.15937#pg1>?

Rosa: Well, the paper tests it on real-world robots, which is good because that shows it holds up beyond just synthetic benchmarks. The results from those experiments confirm the model identifies the target object and surrounding obstacles and uses that information throughout its entire action generation process when deployed.

Dev: But you have to remember what they admit as a limitation; they focus heavily on injecting these geometric tokens derived from three dee scene reconstruction, which means their performance is heavily dependent on having good depth estimation available for the task <ref:2611.15937#pg1>.

Taro: So, if the robot encounters a completely novel object or a situation where depth estimation is really poor, does this framework still manage to act gracefully instead of failing totally?

Rosa: The paper suggests that while it improves manipulation in cluttered scenes significantly, the authors are looking at ways to distill those object level three dimensional shape representations into the VLA backbone itself. That would be a way to strengthen the model’s own intrinsic understanding of geometry so it doesn't rely as heavily on external depth inputs.

Dev: And that sounds like a good direction for future work, moving away from just injecting tokens and toward building that spatial reasoning directly into the foundation of the VLA model. It moves the understanding from an external correction to an internal feature.

Taro: So, it’s about making the AI understand "where things are" at a deeper level, not just learning a better way to map pixels to movements in one specific setup. That's a big shift for autonomy if that works out.

Rosa: Yep, that’s the high-level idea behind MOSAIC—it’s about making spatial reasoning explicit and robust enough to handle the mess of real life manipulation. We'll take a quick break.

Conclusion: Rosa: So we’re wrapping up our look at MOSAIC, this Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes, and what it means for robots handling real stuff.

Dev: Yeah, it’s a framework that adds extra layers to those Vision-Language-Action models so they get much better at navigating messy spaces.

Rosa: Exactly. The authors basically show how you can inject these role specific masks and geometric tokens directly into the robot’s action expert to make it more aware of what's right there in front of it.

Dev: And they found that when you use this with models already trained for tasks like RoboPRO, the success rate jumps by about five point one percent in cluttered scenes.

Rosa: That's a solid improvement, especially when things get really cluttered where the gains can hit up to twenty five point zero percent under challenging conditions.

Dev: The robustness against visual changes, like lighting or blur, also improved quite a bit when they tested it against that base policy.

Rosa: It shows that this isn't just about getting better at one perfect picture; it’s about building a spatial understanding that sticks even when the world gets weird.

Dev: So what does this mean for the future? It suggests we can move away from just hoping the AI guesses the geometry correctly and start giving it explicit geometric rules to follow.

Rosa: Right. The next step they’re looking at is taking those object level three dimensional shape representations and actually baking them into the VLA backbone so the model understands geometry intrinsically instead of relying on external depth data every time.

Huawei Noah’s Ark Lab · University of Toronto · Simon Fraser University

cs.RO, cs.CV

Submitted: 2026-11-11

Updated: 2026-11-11

Project page: https://mmahdavian.github.io/MOSAIC

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: Vision-Language-Action (VLA) models have demonstrated strong performance in robotic manipulation, yet most rely primarily on 2D visual observations and language instructions…

Key concepts

Semantic Mask Generation Module
This module is an auxiliary task where the VLA backbone learns to predict specific masks for three roles: the target object, the destination region, and surrounding obstacles. These masks clearly define the boundaries of these elements, helping the model understand what it needs to interact with in a complex scene.
Role-Specific Geometric Tokens
These tokens are constructed by combining semantic masks from the VLA backbone with depth information from 3D scene reconstruction. They represent the geometry of specific elements like the target object or destination region, allowing the model to use explicit spatial information during action planning.
Obstacle-Aware Attention Supervision
This technique focuses on guiding the robot's attention toward obstacles near its body during task execution. Features resulting from this supervision are injected into later layers of the action expert, ensuring the policy pays close attention to relevant spatial constraints while moving.
Pruning Redundant Reasoning Tokens
The framework uses semantic masks and VGGT-Ω to extract role-specific object geometry, which are then encoded as tokens. This process prunes unnecessary reasoning by focusing the model's understanding on the essential geometric relationships needed for effective manipulation in cluttered environments.

Terminology

Summary

Vision-Language-Action (VLA) models have demonstrated strong performance in robotic manipulation, yet most rely primarily on 2D visual observations and language instructions, leaving scene geometry and obstacle relationships to be inferred implicitly. p. 0 The proposed MOSAIC framework augments the VLA backbone with auxiliary semantic grounding tasks that predict role-specific masks for the target object, destination region, and surrounding obstacles. p. 0 This method constructs role-specific geometric tokens from target and destination masks combined with depth estimated from 3D scene reconstruction to inject them directly into the action expert. p. 0 Furthermore, obstacle-aware attention supervision emphasizes obstacles near the robot body during task execution by injecting resulting attention features into later action-expert layers. p. 0

Pruning redundant reasoning tokens

The core of MOSAIC is a modular framework that enhances object-centric understanding and spatial reasoning in VLA models, enabling more effective manipulation in cluttered environments. This is achieved by combining semantic masks from the VLA backbone with VGGT-omega to extract role-specific object geometry which are then encoded as tokens and injected into the action expert for clutter-aware manipulation. p. 0 The framework also introduces contact-point-aware attention supervision over obstacles and re-injecting the resulting attention features into later action expert layers to guide action generation. p. 0

How the training signal is built

The semantic mask generation module proposes an auxiliary task where the backbone learns to predict role-specific masks for the target object, destination region, and surrounding obstacles jointly with its standard representations. These masks are divided into three task-specific roles: the target object, destination region, and surrounding obstacles. p. 2 The resulting masks preserve clear role distinctions between the target object, destination region, and surrounding obstacles while maintaining accurate boundaries. p. 3

Where the gains come from

The gains are observed when applying these modules to pretrained VLA models using the RoboPRO benchmark and experimenting with real-world robots. The results show that MOSAIC improves the success rate by 5. p. 01% in cluttered scenes and by 9. p. 07% under visual perturbations, with gains of up to 25. p. 00% in the most challenging environment. p. 0 Specifically, the full model improves SR by 9. p. 59% and HSR by 7. p. 51% in the Office environment when tested on RoboPRO. p. 5 The largest gains occur in challenging cluttered settings where explicit grounding improves the policy’s ability to act in clutter, while further improvements in execution safety remain helpful. p. 5

Robustness to visual changes

The framework demonstrates improved robustness when evaluated against the base π0.5 (RoboPRO) policy under RoboPRO’s visual perturbation setting, consistently increasing overall SR from 65. p. 62% to 74.9% (+9.7%) and HSR from 63. p. 54% to 72.7% (+9.3%). This suggests that the proposed modules provide task-relevant spatial structure that remains robust to changes in visual appearance. p. 6 The real-world experiments confirm that the model identifies the target object, destination region, and surrounding obstacles and use this information throughout action generation. p. 6

Conclusion

We proposed MOSAIC, a modular spatial- and obstacleaware framework that augments pretrained VLA policies with role-specific semantic grounding, object-level geometric tokens, and obstacle-aware attention guidance. Evaluations on RoboPRO and experiments on the real-world tasks demonstrate improved robustness and performance in cluttered environments. p. 7 Future work will explore distilling object-level 3D shape representations into the VLA backbone to strengthen its intrinsic geometric understanding and support spatial reasoning and generalization. p. 7

--- Page 4 ---

Table I reports per-environment ablation results for π0.5 on RoboPRO. p. 0 p. 0 We evaluate the contribution of each introduced module. p. 6 p. 6 Best per column within each scene in bold, second best underlined. p. 6 p. 6 All variants are evaluated on the same episodes. p. 6 p. 6

--- Page 7 ---

TABLE I: Per-environment ablation results for π0.5 on RoboPRO. We evaluate the contribution of each introduced module. Best per column within each scene in bold, second best underlined. All variants are evaluated on the same episodes.

--- Page 8 ---

TABLE II: Robustness to visual perturbation (lighting, blur and per-frame pixel shift) on clean RoboPRO scenes. Best results within each model pair are shown in bold, including ties. p. 7

--- Page 9 ---

Fig. 5 provides further qualitative evidence. p. 6 Our model consistently attends to potential collision regions throughout the rollout, whereas the base π0. p. 65 (RoboPRO) model often allocates attention to less relevant regions. p. 6

--- Page 6 ---

Fig. 4(a) shows examples from all four environments. p. 4 In (a), rows show RGB images, mask overlays, and binary target (T), destination (D), and obstacle (O) masks. p. 5

--- Page 5 ---

Fig. 3: The pipeline for groundtruth attention map generation. p. 4 (a) The original RGB countertop camera view. p. 4 In (b), Per-object contact disk generation. p. 4

--- Page 4 ---

(a) Ground the scene Multi-view RGB Main + left / right wrist cameras “Place the mug in the tray.”.

--- Page 2 ---

h pre, h suf = Transformerz pre, z suf, M (1) where z pre contains the visual and language tokens, while z suf is obtained from the noisy action chunk.

--- Page 3 ---

h pre, h suf = Transformerz pre, z suf, M (1) where z pre contains the visual and language tokens, while z suf is obtained from the noisy action chunk.

--- Page 4 ---

h pre, h suf = Transformerz pre, z suf, M (1) where z pre contains the visual and language tokens, while z suf is obtained from the noisy action chunk.

--- Page 5 ---

h pre and h suf denote the corresponding hidden representations (h) with only h suf decoded into an action-space velocity.

--- Page 7 ---

vt = Wout h suf ∈ R H×A (2) where vt is the instantaneous velocity of the action chunk in flow space.

--- Page 3 ---

h pre, h suf = Transformerz pre, z suf, M (1) where z pre contains the visual and language tokens, while z suf is obtained from the noisy action chunk.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. To improve object-level scene understanding, augment a pretrained Vision-Language Action (VLA) backbone with an auxiliary semantic grounding task that predicts role-specific masks for target object, destination region, and surrounding obstacles. This allows the model to better preserve distinct semantics for different spatial elements during action generation.

  2. Inject role-specific geometric tokens into the action expert by combining target and destination masks with depth estimated from 3D scene reconstruction, creating role-specific geometric tokens that are then injected directly into the action expert. This provides explicit semantic and geometric cues to guide manipulation in cluttered scenes.

  3. Implement obstacle-aware attention supervision that emphasizes obstacles near the robot body during task execution, and subsequently inject these resulting attention features into later action-expert layers to further guide action generation. This mechanism helps direct VLA attention beyond the target object and destination region toward surrounding obstacles.

  4. Develop contact-point-aware object attention supervision by calculating proximity weights based on minimum distance to objects over a temporal window, and use these masks to generate groundtruth attention maps that are compared against the model's output using KL divergence loss. This teaches the model to look at objects leading to unsafe scenarios, improving execution safety.

Sources

Related papers