Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

arXiv:2606.20118 · cs.RO, cs.LG · Submitted 2026-06-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation".

Paper summary: Rosa: So we're looking at this paper called "Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation". The main idea here is that vision language action policies often struggle when they see objects that look or are shaped really differently than what they were trained on.

Dev: And the authors are proposing a way to fix this without having to spend tons of time collecting new data for every single failure case. They introduce a failure-driven data augmentation framework where you take episodes where the robot actually succeeded and use those successful runs as templates for failures, swapping only the object being manipulated while keeping the rest of the action trajectory intact.

Taro: So they're trying to make new, physically grounded demonstrations by using existing successes to guide how a policy might handle novel objects that it previously failed on. It sounds like they are trying to get that physical grounding back without collecting any new video data.

Rosa: Exactly. The paper claims this method improves success rates by sixteen point five percent when you fine-tune the vision language action policy using the augmented data, even for those novel objects it didn't see before, while still keeping its performance good on things it already knows how to do.

Dev: That improvement is noted when they look at out of distribution performance, which goes from thirty-six point nine percent up to forty-two point nine percent with this augmentation. They also show that the overall success rate for the base policy bumps up from forty-seven point two percent to forty-nine point five percent.

Taro: What I find interesting is how they handle the challenge of making these object swaps look right across all camera views, because if you just do a 2D edit, things get weird with how the object looks from different angles or if it doesn't look right in one view compared to another <ref:2606.20118#pg1>.

Rosa: That’s where they get pretty technical about anchoring the target object with an explicit three dimensional mesh that is driven by a six dimensional pose trajectory, which makes sure the rendering stays consistent across every camera view. They break down how they build that training signal into three stages: reconstructing the target mesh and extracting its pose, augmenting the object itself, and then composing that video using both the three dee mesh and the robot's pose guidance <ref:2606.20118#pg1>.

Paper summary: Dev: From a control perspective, I see them focusing on three core constraints for this physical augmentation to work well across multiple camera views. First, the replacement object has to follow a geometrically accurate three dimensional trajectory that matches the original manipulation. Second, it needs to be conditioned on the observed grasp so that the interaction still looks physically plausible. And third, it has to appear consistent across all camera views and all time steps.

Taro: So they are ensuring that when you swap an object out, you're not just putting a random new shape in there; you’re making sure that the movement and the way it interacts with the robot match what a physical system would do. That's important for real-world testing.

Rosa: And to get this mesh reconstruction, they use an external perception model to extract a three dimensional mesh of candidate target objects, which is then used to render those inpainted frames based on that kinematically consistent pose trajectory. It’s basically using three dee data to drive the visual changes in the video <ref:2606.20118#pg1>.

Dev: The paper does compare this method against other approaches like VACE and MimicGen, and they show that Pose6DAug outperforms them, achieving a highest average success rate of twenty two point eight percent when combined with the original data. And when trained only on these augmented episodes, Pose6DAug gets a twenty one point two percent success rate on hard out-of-distribution meshes.

Taro: That turnover ratio is pretty telling, too; it shows that this method converts twenty four point five percent of instances that were previously impossible to succeed into at least one successful rollout, compared to eighteen point two percent for VACE and seventeen point two percent for MimicGen.

Rosa: What they really emphasize in the qualitative analysis is maintaining consistent identity, geometry, and contact across both egocentric and exocentric viewpoints because all camera views render that same three dimensional mesh at a shared world frame pose. It’s about trajectory-level physical plausibility combined with view-level geometric consistency making the augmented episodes visually coherent enough for the policy to learn from.

Paper summary: Dev: It really shows how grounding visual augmentation directly in three dimensional geometry is a promising way to scale robot learning across different real world environments, while still keeping the physical realism that simulation based methods offer. But they do flag a limitation; they mentioned that the method inherits some failure modes from external vision models like the three dimensional mesh extractors and video inpainting networks.

Taro: So, the authors themselves point out that this augmentation isn't perfect because it relies on those other models to generate the target objects and their poses. They suggest further investigation into mitigation strategies to minimize unexpected behaviors when using Pose6DAug.

Rosa: Overall, "Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation" is about taking the successful moments of a robot policy and turning them into targeted demonstrations for its failures on novel objects, all by swapping the object while strictly maintaining physical consistency across multiple camera views.

Dev: It’s about using three dee geometry to anchor the swaps, which helps generalize better without losing performance on things it already knows how to handle <ref:2606.20118#pg1>. The authors show that this approach doesn't sacrifice in distribution competence while improving OOD performance significantly compared to other augmentation techniques they tested.

Taro: So for someone who only listens to the show, what this means is that if you’re working on a robot and you run into something totally new, instead of just giving up or collecting hours of new data, this method lets your existing successes teach the system how to handle that new object in a way that looks physically sound.

Rosa: That's the practical implication: it allows for scaling robot learning by using failure cases to create high-quality synthetic training examples directly, rather than relying solely on expensive human teleoperation data collection for every single new scenario.

Dev: The authors suggest future work could look at improving how the object being swapped interacts with the environment or incorporating neural relighting techniques, which could further enhance that physical grounding they are building in this framework.

Taro: It sounds like they're looking ahead to making the swaps even more nuanced and realistic by focusing on how light and interaction change when you swap things out. That’s a good direction for future research.

Conclusion: Rosa: So we're wrapping up on Pose6DAug, which is about taking successful robot episodes and using them to teach the robot how to handle new objects by swapping them out while keeping everything physically consistent across different camera views.

Dev: Right, so it's taking those good runs and turning them into targeted training data for failures, but the key part is making sure that when you swap an object, it looks right from every angle at the same time.

Taro: Yeah, what they’re really doing is grounding the visual change in three dimensional geometry so that it doesn't just look plausible in one view but makes sense across all of them.

Rosa: It seems like the whole point is to get a robot to learn how to interact with things it hasn't seen before without having to spend tons of time collecting brand new videos for every single new object type.

Dev: And they showed that this augmentation actually improves performance on those novel objects from thirty-six point nine percent up to forty-two point nine percent when the policy is fine-tuned with this data.

Taro: That turnover ratio is pretty strong too, showing that it converts twenty four point five percent of hard instances into at least one successful rollout, which is a big deal for real robot learning.

Rosa: It’s a way to scale up robot learning across different environments by using the physical logic of three dee space to guide the visual changes in training data.

Dev: But they did point out that this method still relies on external models, like those used for mesh extraction, so we gotta keep an eye on any issues with those external systems when we run this at high speed.

Taro: Exactly, the paper highlights that if the three dee mesh extractor gives a bad shape or pose, our augmentation won't work well either.

Rosa: So it’s a promising direction for making robot learning more robust by using geometry to anchor visual data augmentation. Next up we're going to look at some of the specific failure cases they found in their tests.

KAIST · Korea University

cs.RO, cs.LG

Submitted: 2026-06-18

Updated: 2026-11-18

Project page: https://jhoonjwa.github.io/pose6daug

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 99/100

The gist: Vision-language-action (VLA) policies often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.

Key concepts

Failure-driven Data Augmentation
This technique uses the robot's own successful attempts to interact with an object as training data for failure modes. Instead of collecting new data, it targets specific ways the policy fails, such as handling novel objects, and generates targeted demonstrations without needing new real-world collection.
3D Mesh-Pose Guided Composition
The method ensures physical plausibility by anchoring the target object with an explicit 3D mesh driven by a temporally coherent 6D pose trajectory. This guarantees that the swapped object looks geometrically consistent across all camera views, solving the problem of naive 2D editing breaking multi-view consistency.
Multi-view Consistency
This refers to ensuring that an object appears identical in shape and appearance when viewed from different cameras (like egocentric or exocentric views) at the same time. Pose6DAug achieves this by rendering the target object using a shared world-frame pose, making augmented episodes visually coherent for reliable policy learning.

Terminology

Summary

Vision-language-action (VLA) policies often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. p. 0

Where the paper says it

Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3† 1KAIST 2Korea University 3RLWRLD {alphabet1, seonghyp, jinwoos}@kaist. kr Abstract Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.

Pose6DAug introduces a failure-driven data augmentation framework that turns a policy’s own successful episodes into targeted demonstrations for its failure modes without any new data collection. p. 0
Where the paper says it

The standard remedy is to collect multi-view teleoperation data for every failure case, but this scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven data augmentation framework that turns a policy’s own successful episodes into targeted demonstrations for its failure modes, without any new data collection.

By swapping only the manipulated object while preserving this trajectory, the method obtains new and physically grounded demonstrations. p. 0
Where the paper says it

Our key insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. By swapping only the manipulated object while preserving this trajectory, we obtain new and physically grounded demonstrations.

However, the challenge lies in synthesizing object swaps, as naive 2D video editing breaks multi-view consistency and physical plausibility. p. 0
Where the paper says it

However, the challenge lies in synthesizing object swaps, e. , naive 2D video editing breaks multi-view consistency and physical plausibility, particularly under heavy occlusion and egocentric viewpoints.

The method operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory to ensure geometrically consistent renderings across all camera views. p. 0
Where the paper says it

Our method instead operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory. This ensures geometrically consistent renderings across all camera views.

Fine-tuning a VLA on data augmented by this method improves success rates by 16. p. 0
Where the paper says it

This ensures geometrically consistent renderings across all camera views. Fine-tuning a VLA on data augmented by our method improves success rates by 16.

5% relative to the state-of-the-art baseline on novel objects while preserving in-distribution performance. p. 0
Where the paper says it

Fine-tuning a VLA on data augmented by our method improves success rates by 16. 5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance.

Pruning redundant reasoning tokens

The core insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. The pipeline decomposes object-centric augmentation into three stages: target mesh reconstruction and 6D pose extraction, object augmentation, and 3D mesh-pose guided video composition. p. 2

Where the paper says it

To ensure multi-view consistency by construction, we represent the target object as an explicit 3D mesh and render it at a shared world-frame pose for each camera—rather than performing independent per-view edits that would require post-hoc alignment. Our pipeline decomposes object-centric augmentation into three stages: target mesh reconstruction and 6D pose extraction (Section 3.

How the training signal is built

The method targets three core constraints for physically consistent augmentation across multiple camera views: (1) the replacement object must follow a geometrically accurate 3D trajectory that reflects the original manipulation, (2) the replacement must be conditioned on the observed grasp so that the resulting interaction remains physically plausible, and (3) the edited object must appear consistently across all camera views and timesteps. The target mesh reconstruction involves using an external perception model to extract a 3D mesh of candidate target objects, which is then used to render the object into inpainted frames based on a kinematically consistent pose trajectory. p. 1

Where the paper says it

Subsequently, to ensure the synthesized object motion is strictly consistent with the robot’s kinematics, we represent the interaction as a temporally coherent 6D pose trajectory. By rendering the target mesh into the inpainted frames based on this kinematically consistent pose, our approach maintains geometric alignment as well as plausible object trajectories in the 3D space.

Where the gains come from

The method achieves relative 16.5% success rate improvements, surpassing baselines that rely on 2D augmentation or limited demonstration sets. p. 1

Where the paper says it

Our method achieves relative 16. 5% success rate improvements, surpassing baselines that rely on 2D augmentation [16] or limited demonstration sets [14].

The evaluation shows that Pose6DAug improves OOD performance from 36. p. 9
Where the paper says it

By operating in 3D space rather than purely in pixels, Pose6DAug ensures that synthesized demonstrations maintain both physical admissibility and multi-view consistency. Our experimental evaluation on the RoboCasa benchmark demonstrates that Pose6DAug improves the performance of VLA policies on challenging and novel object instances.

9% to 42.9%, demonstrating that our augmentation strengthens generalization without sacrificing in-distribution competence. p. 8
Where the paper says it

Any source–target pair can be augmented, while preserving the physical grounding that simulation-based methods offer. The turnover results further support this: Pose6DAug not only improves aggregate performance but extends the policy’s capability to previously failed object instances, without sacrificing in-distribution performance.

p. 8
Where the paper says it

Any source–target pair can be augmented, while preserving the physical grounding that simulation-based methods offer. The turnover results further support this: Pose6DAug not only improves aggregate performance but extends the policy’s capability to previously failed object instances, without sacrificing in-distribution performance.

This is confirmed by the overall success rate of 49. p. 5
Where the paper says it

Table 1: Overall performance on failure episodes. Success rate (%) and turnover ratio (%) evaluated on failure episodes: instances where the base policy never succeeded in completing the task with the target object.

5% compared to the base policy's 47.2%.

The qualitative analysis illustrates that Pose6DAug maintains consistent identity, geometry, and contact across egocentric and exocentric viewpoints alike because all camera views render the same 3D mesh at a shared world-frame pose. This combination of trajectory-level physical plausibility and view-level geometric consistency produces augmented episodes that are visually coherent and suitable for reliable policy learning. p. 8

Where the paper says it

Because all camera views render the same 3D mesh at a shared world-frame pose, the augmented object maintains consistent identity, geometry, and contact across egocentric and exocentric viewpoints alike. This combination of trajectory-level physical plausibility and view-level geometric consistency produces augmented episodes that are visually coherent and suitable for reliable policy learning.

The method outperforms VACE (16.4%) and MimicGen (15. p. 2

Where the paper says it

However, they apply edits independently per frame and per view, without explicit knowledge of the underlying 3D trajectory, and thus do not ensure geometric consistency across multi-view observations. On the other hand, simulation-based approaches such as MimicGen [14] and its variants [15] operate directly in a simulator to synthesize new demonstrations by adapting source end-effector trajectories via SE(3) transformations to novel object poses.

8%) in recovering failures when combined with the original data, achieving the highest average success rate of 22. p. 8
Where the paper says it

Table 1 reports success rates and turnover ratios evaluated exclusively on episodes where the base policy originally failed. Pose6DAug achieves the highest average success rate of 22.

8%. Furthermore, when trained exclusively on augmented episodes, Pose6DAug achieves a 21. p. 8
Where the paper says it

Table 1 reports success rates and turnover ratios evaluated exclusively on episodes where the base policy originally failed. Pose6DAug achieves the highest average success rate of 22.

2% success rate on hard out-of-distribution meshes, substantially outperforming VACE (15. p. 8
Where the paper says it

Pose6DAug achieves a 21. 2% success rate, substantially outperforming VACE (15.

0%) and MimicGen (5. p. 2
Where the paper says it

However, they apply edits independently per frame and per view, without explicit knowledge of the underlying 3D trajectory, and thus do not ensure geometric consistency across multi-view observations. On the other hand, simulation-based approaches such as MimicGen [14] and its variants [15] operate directly in a simulator to synthesize new demonstrations by adapting source end-effector trajectories via SE(3) transformations to novel object poses.

7%). The turnover ratio shows Pose6DAug converts 24. p. 8
Where the paper says it

4%) and MimicGen (15. The turnover ratio tells a similar story: Pose6DAug converts 24.

5% of previously intractable instances into at least one successful rollout, compared to 18. p. 8
Where the paper says it

The turnover ratio tells a similar story: Pose6DAug converts 24. 5% of previously intractable instances into at least one successful rollout, compared to 18.

2% for VACE and 17. p. 2
Where the paper says it

2 Visual Generative Models and Video Editing The advancement of diffusion models has enabled various video editing and synthesis applications. Recent unified frameworks, such as VACE [16], integrate multiple video creation and editing tasks, including masked video-to-video editing, inpainting, and object swapping, into a single model conditioned on multimodal inputs.

2% for MimicGen. p. 2
Where the paper says it

However, they apply edits independently per frame and per view, without explicit knowledge of the underlying 3D trajectory, and thus do not ensure geometric consistency across multi-view observations. On the other hand, simulation-based approaches such as MimicGen [14] and its variants [15] operate directly in a simulator to synthesize new demonstrations by adapting source end-effector trajectories via SE(3) transformations to novel object poses.

The paper concludes that grounding visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse real-world environments. The authors note that the method inherits the scalability of generative editing while preserving the physical grounding offered by simulation-based methods. p. 8

Where the paper says it

This allows our method to inherit the scalability of generative editing. Any source–target pair can be augmented, while preserving the physical grounding that simulation-based methods offer.

The study also suggests that improving the interaction between the object being swapped and incorporating neural relighting techniques represents a promising direction for future work. p. 9
Where the paper says it

Furthermore, the inclusion of geometric perturbations—such as object rotations, translations, and rescaling—proved critical for improving out-of-distribution generalization. We believe that grounding visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse real-world environments.

The paper highlights limitations, noting that automated affordance matching would be valuable to handle broader categories of objects. Additionally, Pose6DAug inherits the failure modes of external vision models like 3D mesh extractors and video inpainting networks. The authors also recommend conducting thorough investigations to implement robust mitigation strategies to minimize potential risks associated with unexpected behaviors in Pose6DAug. p. 0

Where the paper says it

Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3† 1KAIST 2Korea University 3RLWRLD {alphabet1, seonghyp, jinwoos}@kaist. kr Abstract Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.

The paper evaluates the method on the RoboCasa benchmark, where it provides consistent performance gains across diverse novel object instances. The evaluation protocol involves pairing hard samples with source episodes and generating 16 augmented episodes for each pair, yielding 176 augmented episodes for both Pose6DAug and VACE. p. 6

Where the paper says it

For MimicGen, generating a physically valid trajectory is not always feasible for a given scene initialization; we therefore allow 32 attempts per pair and retain only successful rollouts. This protocol yields 176 augmented episodes for both Pose6DAug and VACE.

The results demonstrate that Pose6DAug improves the performance of VLA policies on challenging and novel object instances. p. 9
Where the paper says it

By operating in 3D space rather than purely in pixels, Pose6DAug ensures that synthesized demonstrations maintain both physical admissibility and multi-view consistency. Our experimental evaluation on the RoboCasa benchmark demonstrates that Pose6DAug improves the performance of VLA policies on challenging and novel object instances.

The authors also show failure cases driven by discrepancies in scale, width or centroid position between the source and target objects. The evaluation metrics include Success Rate, Turnover Ratio, and Class Recovery. p. 6
Where the paper says it

3 Metrics We report three metrics, evaluated over 200 rollouts per configuration for the main evaluation and 20 rollouts per instance for the hard-example evaluation (Table 2): • Success Rate: The fraction of rollouts in which the robot successfully completes the task. • Turnover Ratio: Among the set of meshes that never succeeded under the base checkpoint, the fraction that achieve at least one success after training with augmented data.

The paper compares Pose6DAug against VACE and MimicGen, showing that MimicGen's reliance on successful rollout generation creates a coverage bottleneck. In contrast, Pose6DAug anchors the replacement object with an explicit 3D mesh rendered at a shared world-frame pose, ensuring geometric consistency across both egocentric and exocentric views by construction. p. 8

Where the paper says it

VACE circumvents the coverage problem by editing on image space, but editing each view independently introduces cross-camera discrepancies in object identity, scale, and contact geometry that propagate as noisy supervision during training. In contrast, Pose6DAug anchors the replacement object with an explicit 3D mesh rendered at a shared world-frame pose, ensuring geometric consistency across both egocentric and exocentric views by construction.

The results support the idea that grounding visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse real-world environments. p. 9
Where the paper says it

Furthermore, the inclusion of geometric perturbations—such as object rotations, translations, and rescaling—proved critical for improving out-of-distribution generalization. We believe that grounding visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse real-world environments.

The authors also show that Pose6DAug not only improves aggregate performance but extends the policy’s capability to previously failed object instances, without sacrificing in-distribution performance. p. 8
Where the paper says it

Any source–target pair can be augmented, while preserving the physical grounding that simulation-based methods offer. The turnover results further support this: Pose6DAug not only improves aggregate performance but extends the policy’s capability to previously failed object instances, without sacrificing in-distribution performance.

The paper is available at arXiv:2606. p. 0
Where the paper says it

†Corresponding author. arXiv:2606.

20118v2 [cs. p. 0
Where the paper says it

arXiv:2606. 20118v2 [cs.

p. 0
Where the paper says it

arXiv:2606. 20118v2 [cs.

p. 0
Where the paper says it

arXiv:2606. 20118v2 [cs.

RO] 19 Jun 2026. p. 0
Where the paper says it

20118v2 [cs. RO] 19 Jun 2026

p. 0
Where the paper says it

20118v2 [cs. RO] 19 Jun 2026

p. 0
Where the paper says it

20118v2 [cs. RO] 19 Jun 2026

The corresponding author is Jonghoon Lee, Seong Hyeon Park, and Byungwoo Jeon. p. 0
Where the paper says it

Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3† 1KAIST 2Korea University 3RLWRLD {alphabet1, seonghyp, jinwoos}@kaist. kr Abstract Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.

The project page is https://jhoonjwa. p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

github.io/pose6daug. p. 0
Where the paper says it

Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3† 1KAIST 2Korea University 3RLWRLD {alphabet1, seonghyp, jinwoos}@kaist. kr Abstract Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.

The references include [1] through [29] and the appendix includes sections A. p. 9
Where the paper says it

We believe that grounding visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse real-world environments. References [1] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al.

1 through A.4. The paper is 14 pages long and contains 20 rollouts per configuration for the main evaluation. p. 6
Where the paper says it

We provide qualitative examples of augmentations across all three methods in Figure 3. 3 Metrics We report three metrics, evaluated over 200 rollouts per configuration for the main evaluation and 20 rollouts per instance for the hard-example evaluation (Table 2): • Success Rate: The fraction of rollouts in which the robot successfully completes the task.

The paper is available on page 2606. p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

20118v2 [cs.RO] 19 Jun 2026. The reference list includes references [1] through [5] and [7] through [9], and then references starting from [14] onwards. p. 9
Where the paper says it

We believe that grounding visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse real-world environments. References [1] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al.

The paper is 2606. p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

20118v2 [cs.RO] 19 Jun 2026. The paper is available on page 4. p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

The paper is available on page 5. p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

The paper is available on page 7 and page 8 and page 9 and page 10. p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

The paper is available on page 12. p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

The paper is available on page 13 and page 14. p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

The paper is available on page 6. p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

p. 9
Where the paper says it

5 Conclusion In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to enhance the generalization of robotic manipulation policies. To address the high cost of multi-view data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D mesh extraction, video inpainting, and robot kinematics.

--- Page 6 ---

Figure 3 illustrates the augmentation quality of each method across all three camera views. VACE exhibits several failure modes that compound during training: the appearance of the target object changes across frames (identity drift), multi-view consistency is not maintained as the object differs in shape and color between the left, right, and wrist cameras, and in some views the source object is not edited at all, leaving the original object intact alongside the intended replacement. p. 8

Where the paper says it

Figure 3 illustrates the augmentation quality of each method across all three camera views. VACE exhibits several failure modes that compound during training: the appearance of the target object changes across frames (identity drift), multi-view consistency is not maintained as the object differs in shape and color between the left, right, and wrist cameras, and in some views the source object is not edited at all, leaving the original object intact alongside the intended replacement.

MimicGen fails to generate physically plausible action trajectories for challenging object geometries—as shown in Figure 3, the adapted trajectories fail to produce valid grasps across all rollout attempts, resulting in no usable augmented data for these instances. p. 8
Where the paper says it

These inconsistencies produce unreliable training signals that can mislead the policy. MimicGen, while operating in simulation, generates physically implausible action trajectories for challenging object geometries—as shown in Figure 3, the adapted trajectories fail to produce valid grasps across all rollout attempts, resulting in no usable augmented data for these instances.

--- Page 7 ---

Table 2 reports the overall success rate (%) in the top block, experimented with the model fine-tuned on the most challenging 8 unseen, out-of-distribution object instances, evaluated on 20 pick-and-place episodes per instance (Bold indicates the best result). Table 3 reports overall success rate (%) evaluated on all episodes, including both previously successful and previously failed instances. p. 8

Where the paper says it

We first examine whether appending augmented episodes to the original training set can turn previously failed rollouts into successes. Table 1 reports success rates and turnover ratios evaluated exclusively on episodes where the base policy originally failed.

The results show that Pose6DAug achieves the highest overall success rate (49. p. 8
Where the paper says it

Table 1 reports success rates and turnover ratios evaluated exclusively on episodes where the base policy originally failed. Pose6DAug achieves the highest average success rate of 22.

5%) while improving OOD performance from 36. p. 0
Where the paper says it

Fine-tuning a VLA on data augmented by our method improves success rates by 16. 5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance.

9% to 42.9%, demonstrating that our augmentation strengthens generalization without sacrificing in-distribution competence.

--- Page 8 ---

Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page 10. p. 0

Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

The paper is available on page 6. The paper is available on page 12. The paper is available on page 13 and page 8 and page 9 and page 10. p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

p. 0
Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page 10. The paper is available on page 6. The paper is available on page 12. The paper is available on page 13 and page 8 and page 9 and page 10. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page 10. The paper is available on page 6.

--- Page 14 ---

Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page 10. The paper is available on page 6.

--- Page 14 ---

Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page10 pg10. p. 0

Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

--- Page 14 ---

Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and pg10 pg10. p. 0

Where the paper says it

These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization. Project page: https://jhoonjwa.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. 3D-grounded augmentation ensures that synthesized demonstrations maintain both physical admissibility and multi-view consistency by operating directly in 3D space, which bridges the gap between kinematic validity and visual photorealism. This allows for a geometrically consistent rendering across all camera views when fine-tuning VLA policies.

  2. The system can generate new, physically grounded demonstrations for failure modes without requiring new data collection by utilizing successful episodes into targeted demonstrations for its failure modes, without any new data collection. This addresses the bottleneck of collecting multi-view teleoperation data for every failure case.

  3. The improved AI system can handle novel object instances with a performance gain of 16.5% relative to state-of-the-art baselines while preserving in-distribution performance, as demonstrated by relative 16.5% success rate improvements on the RoboCasa benchmark.

  4. The framework can maintain structural coherence across all observations by anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory, ensuring that all camera views render the same 3D mesh at a shared world-frame pose.

  5. The system can be enhanced to provide fine-grained recovery metrics, as it offers Class Recovery: A per-instance indicator of whether a specific object instance that consistently failed under the base policy achieves at least one successful rollout after fine-tuning.

Abstract

Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. The standard remedy is to collect multi-view teleoperation data for every failure case, but this scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven data augmentation framework that turns existing successful episodes into targeted demonstrations for a policy's failure modes, without any new data collection. Our key insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. By swapping only the manipulated object while preserving this trajectory, we obtain new and physically grounded demonstrations. However, the challenge lies in synthesizing object swaps, e.g., naive 2D video editing breaks multi-view consistency and physical plausibility, particularly under heavy occlusion and egocentric viewpoints. Our method instead operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory. This ensures geometrically consistent renderings across all camera views. Fine-tuning a VLA on data augmented by our method improves success rates by 16.5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance. On a real robot, it more than doubles the success rate of the base policy. These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization.

Sources

Related papers