Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation".
Paper summary: Rosa: So we're looking at this paper called "Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation". The main idea here is that vision language action policies often struggle when they see objects that look or are shaped really differently than what they were trained on.
Dev: And the authors are proposing a way to fix this without having to spend tons of time collecting new data for every single failure case. They introduce a failure-driven data augmentation framework where you take episodes where the robot actually succeeded and use those successful runs as templates for failures, swapping only the object being manipulated while keeping the rest of the action trajectory intact.
Taro: So they're trying to make new, physically grounded demonstrations by using existing successes to guide how a policy might handle novel objects that it previously failed on. It sounds like they are trying to get that physical grounding back without collecting any new video data.
Rosa: Exactly. The paper claims this method improves success rates by sixteen point five percent when you fine-tune the vision language action policy using the augmented data, even for those novel objects it didn't see before, while still keeping its performance good on things it already knows how to do.
Dev: That improvement is noted when they look at out of distribution performance, which goes from thirty-six point nine percent up to forty-two point nine percent with this augmentation. They also show that the overall success rate for the base policy bumps up from forty-seven point two percent to forty-nine point five percent.
Taro: What I find interesting is how they handle the challenge of making these object swaps look right across all camera views, because if you just do a 2D edit, things get weird with how the object looks from different angles or if it doesn't look right in one view compared to another <ref:2606.20118#pg1>.
Rosa: That’s where they get pretty technical about anchoring the target object with an explicit three dimensional mesh that is driven by a six dimensional pose trajectory, which makes sure the rendering stays consistent across every camera view. They break down how they build that training signal into three stages: reconstructing the target mesh and extracting its pose, augmenting the object itself, and then composing that video using both the three dee mesh and the robot's pose guidance <ref:2606.20118#pg1>.
Paper summary: Dev: From a control perspective, I see them focusing on three core constraints for this physical augmentation to work well across multiple camera views. First, the replacement object has to follow a geometrically accurate three dimensional trajectory that matches the original manipulation. Second, it needs to be conditioned on the observed grasp so that the interaction still looks physically plausible. And third, it has to appear consistent across all camera views and all time steps.
Taro: So they are ensuring that when you swap an object out, you're not just putting a random new shape in there; you’re making sure that the movement and the way it interacts with the robot match what a physical system would do. That's important for real-world testing.
Rosa: And to get this mesh reconstruction, they use an external perception model to extract a three dimensional mesh of candidate target objects, which is then used to render those inpainted frames based on that kinematically consistent pose trajectory. It’s basically using three dee data to drive the visual changes in the video <ref:2606.20118#pg1>.
Dev: The paper does compare this method against other approaches like VACE and MimicGen, and they show that Pose6DAug outperforms them, achieving a highest average success rate of twenty two point eight percent when combined with the original data. And when trained only on these augmented episodes, Pose6DAug gets a twenty one point two percent success rate on hard out-of-distribution meshes.
Taro: That turnover ratio is pretty telling, too; it shows that this method converts twenty four point five percent of instances that were previously impossible to succeed into at least one successful rollout, compared to eighteen point two percent for VACE and seventeen point two percent for MimicGen.
Rosa: What they really emphasize in the qualitative analysis is maintaining consistent identity, geometry, and contact across both egocentric and exocentric viewpoints because all camera views render that same three dimensional mesh at a shared world frame pose. It’s about trajectory-level physical plausibility combined with view-level geometric consistency making the augmented episodes visually coherent enough for the policy to learn from.
Paper summary: Dev: It really shows how grounding visual augmentation directly in three dimensional geometry is a promising way to scale robot learning across different real world environments, while still keeping the physical realism that simulation based methods offer. But they do flag a limitation; they mentioned that the method inherits some failure modes from external vision models like the three dimensional mesh extractors and video inpainting networks.
Taro: So, the authors themselves point out that this augmentation isn't perfect because it relies on those other models to generate the target objects and their poses. They suggest further investigation into mitigation strategies to minimize unexpected behaviors when using Pose6DAug.
Rosa: Overall, "Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation" is about taking the successful moments of a robot policy and turning them into targeted demonstrations for its failures on novel objects, all by swapping the object while strictly maintaining physical consistency across multiple camera views.
Dev: It’s about using three dee geometry to anchor the swaps, which helps generalize better without losing performance on things it already knows how to handle <ref:2606.20118#pg1>. The authors show that this approach doesn't sacrifice in distribution competence while improving OOD performance significantly compared to other augmentation techniques they tested.
Taro: So for someone who only listens to the show, what this means is that if you’re working on a robot and you run into something totally new, instead of just giving up or collecting hours of new data, this method lets your existing successes teach the system how to handle that new object in a way that looks physically sound.
Rosa: That's the practical implication: it allows for scaling robot learning by using failure cases to create high-quality synthetic training examples directly, rather than relying solely on expensive human teleoperation data collection for every single new scenario.
Dev: The authors suggest future work could look at improving how the object being swapped interacts with the environment or incorporating neural relighting techniques, which could further enhance that physical grounding they are building in this framework.
Taro: It sounds like they're looking ahead to making the swaps even more nuanced and realistic by focusing on how light and interaction change when you swap things out. That’s a good direction for future research.
Conclusion: Rosa: So we're wrapping up on Pose6DAug, which is about taking successful robot episodes and using them to teach the robot how to handle new objects by swapping them out while keeping everything physically consistent across different camera views.
Dev: Right, so it's taking those good runs and turning them into targeted training data for failures, but the key part is making sure that when you swap an object, it looks right from every angle at the same time.
Taro: Yeah, what they’re really doing is grounding the visual change in three dimensional geometry so that it doesn't just look plausible in one view but makes sense across all of them.
Rosa: It seems like the whole point is to get a robot to learn how to interact with things it hasn't seen before without having to spend tons of time collecting brand new videos for every single new object type.
Dev: And they showed that this augmentation actually improves performance on those novel objects from thirty-six point nine percent up to forty-two point nine percent when the policy is fine-tuned with this data.
Taro: That turnover ratio is pretty strong too, showing that it converts twenty four point five percent of hard instances into at least one successful rollout, which is a big deal for real robot learning.
Rosa: It’s a way to scale up robot learning across different environments by using the physical logic of three dee space to guide the visual changes in training data.
Dev: But they did point out that this method still relies on external models, like those used for mesh extraction, so we gotta keep an eye on any issues with those external systems when we run this at high speed.
Taro: Exactly, the paper highlights that if the three dee mesh extractor gives a bad shape or pose, our augmentation won't work well either.
Rosa: So it’s a promising direction for making robot learning more robust by using geometry to anchor visual data augmentation. Next up we're going to look at some of the specific failure cases they found in their tests.
KAIST · Korea University
cs.RO, cs.LG
Submitted: 2026-06-18
Updated: 2026-11-18
Project page: https://jhoonjwa.github.io/pose6daug
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 99/100
The gist: Vision-language-action (VLA) policies often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.
Key concepts
- Failure-driven Data Augmentation
- This technique uses the robot's own successful attempts to interact with an object as training data for failure modes. Instead of collecting new data, it targets specific ways the policy fails, such as handling novel objects, and generates targeted demonstrations without needing new real-world collection.
- 3D Mesh-Pose Guided Composition
- The method ensures physical plausibility by anchoring the target object with an explicit 3D mesh driven by a temporally coherent 6D pose trajectory. This guarantees that the swapped object looks geometrically consistent across all camera views, solving the problem of naive 2D editing breaking multi-view consistency.
- Multi-view Consistency
- This refers to ensuring that an object appears identical in shape and appearance when viewed from different cameras (like egocentric or exocentric views) at the same time. Pose6DAug achieves this by rendering the target object using a shared world-frame pose, making augmented episodes visually coherent for reliable policy learning.
Terminology
Summary
Vision-language-action (VLA) policies often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. p. 0 Pose6DAug: Physically Plausible Multi-view Object
Swapping for Robot Data Augmentation
Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3†
1KAIST 2Korea University 3RLWRLD
{alphabet1, seonghyp, jinwoos}@kaist. kr
Abstract
Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects
whose appearance or geometry deviates from the training distribution. The standard
remedy is to collect multi-view teleoperation data for every failure case, but this
scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven
data augmentation framework that turns a policy’s own successful episodes into
targeted demonstrations for its failure modes, without any new data collection. Our
key insight is that each successful episode already encodes a physically valid action
trajectory together with calibrated multi-view observations. By swapping only the
manipulated object while preserving this trajectory, we obtain new and physically
grounded demonstrations. However, the challenge lies in synthesizing object swaps,
e. , naive 2D video editing breaks multi-view consistency and physical plausibility,
particularly under heavy occlusion and egocentric viewpoints. Our method instead
operates directly in 3D, anchoring the target object with an explicit mesh driven by
a temporally coherent 6D pose trajectory. This ensures geometrically consistent
renderings across all camera views. This ensures geometrically consistent
renderings across all camera views. Fine-tuning a VLA on data augmented by our
method improves success rates by 16. Fine-tuning a VLA on data augmented by our
method improves success rates by 16. 5% relative to the state-of-the-art baseline on
novel objects, while preserving in-distribution performance.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The core insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. The pipeline decomposes object-centric augmentation into three stages: target mesh reconstruction and 6D pose extraction, object augmentation, and 3D mesh-pose guided video composition. p. 2 To ensure multi-view consistency by construction, we represent the target
object as an explicit 3D mesh and render it at a shared world-frame pose for each camera—rather
than performing independent per-view edits that would require post-hoc alignment. Our pipeline decomposes object-centric augmentation into three stages: target mesh reconstruction
and 6D pose extraction (Section 3.Where the paper says it
How the training signal is built
The method targets three core constraints for physically consistent augmentation across multiple camera views: (1) the replacement object must follow a geometrically accurate 3D trajectory that reflects the original manipulation, (2) the replacement must be conditioned on the observed grasp so that the resulting interaction remains physically plausible, and (3) the edited object must appear consistently across all camera views and timesteps. The target mesh reconstruction involves using an external perception model to extract a 3D mesh of candidate target objects, which is then used to render the object into inpainted frames based on a kinematically consistent pose trajectory. p. 1 Subsequently, to ensure the synthesized
object motion is strictly consistent with the robot’s kinematics, we represent the interaction as a
temporally coherent 6D pose trajectory. By rendering the target mesh into the inpainted frames
based on this kinematically consistent pose, our approach maintains geometric alignment as well
as plausible object trajectories in the 3D space.Where the paper says it
Where the gains come from
The method achieves relative 16.5% success rate improvements, surpassing baselines that rely on 2D augmentation or limited demonstration sets. p. 1 Our method achieves relative 16. 5% success rate improvements, surpassing baselines that rely on 2D augmentation [16] or limited demonstration sets [14]. By operating in 3D space rather than purely
in pixels, Pose6DAug ensures that synthesized demonstrations maintain both physical admissibility
and multi-view consistency. Our experimental evaluation on the RoboCasa benchmark demonstrates
that Pose6DAug improves the performance of VLA policies on challenging and novel object instances. Any source–target
pair can be augmented, while preserving the physical grounding that simulation-based methods offer. The turnover results further support this: Pose6DAug not only improves aggregate performance but
extends the policy’s capability to previously failed object instances, without sacrificing in-distribution
performance. Any source–target
pair can be augmented, while preserving the physical grounding that simulation-based methods offer. The turnover results further support this: Pose6DAug not only improves aggregate performance but
extends the policy’s capability to previously failed object instances, without sacrificing in-distribution
performance. Table 1: Overall performance on failure episodes. Success rate (%) and turnover ratio (%) evaluated
on failure episodes: instances where the base policy never succeeded in completing the task with
the target object.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The qualitative analysis illustrates that Pose6DAug maintains consistent identity, geometry, and contact across egocentric and exocentric viewpoints alike because all camera views render the same 3D mesh at a shared world-frame pose. This combination of trajectory-level physical plausibility and view-level geometric consistency produces augmented episodes that are visually coherent and suitable for reliable policy learning. p. 8 Because all camera views render the same 3D
mesh at a shared world-frame pose, the augmented object maintains consistent identity, geometry,
and contact across egocentric and exocentric viewpoints alike. This combination of trajectory-level
physical plausibility and view-level geometric consistency produces augmented episodes that are
visually coherent and suitable for reliable policy learning.Where the paper says it
The method outperforms VACE (16.4%) and MimicGen (15. p. 2 However, they apply edits independently per frame and per view, without explicit knowledge
of the underlying 3D trajectory, and thus do not ensure geometric consistency across multi-view
observations. On the other hand, simulation-based approaches such as MimicGen [14] and its variants [15] operate
directly in a simulator to synthesize new demonstrations by adapting source end-effector trajectories
via SE(3) transformations to novel object poses. Table 1 reports success rates and turnover ratios evaluated exclusively on
episodes where the base policy originally failed. Pose6DAug achieves the highest average success
rate of 22. Table 1 reports success rates and turnover ratios evaluated exclusively on
episodes where the base policy originally failed. Pose6DAug achieves the highest average success
rate of 22. Pose6DAug achieves
a 21. 2% success rate, substantially outperforming VACE (15. However, they apply edits independently per frame and per view, without explicit knowledge
of the underlying 3D trajectory, and thus do not ensure geometric consistency across multi-view
observations. On the other hand, simulation-based approaches such as MimicGen [14] and its variants [15] operate
directly in a simulator to synthesize new demonstrations by adapting source end-effector trajectories
via SE(3) transformations to novel object poses. 4%) and MimicGen (15. The turnover ratio
tells a similar story: Pose6DAug converts 24. The turnover ratio
tells a similar story: Pose6DAug converts 24. 5% of previously intractable instances into at least
one successful rollout, compared to 18. 2 Visual Generative Models and Video Editing
The advancement of diffusion models has enabled various video editing and synthesis applications. Recent unified frameworks, such as VACE [16], integrate multiple video creation and editing tasks,
including masked video-to-video editing, inpainting, and object swapping, into a single model
conditioned on multimodal inputs. However, they apply edits independently per frame and per view, without explicit knowledge
of the underlying 3D trajectory, and thus do not ensure geometric consistency across multi-view
observations. On the other hand, simulation-based approaches such as MimicGen [14] and its variants [15] operate
directly in a simulator to synthesize new demonstrations by adapting source end-effector trajectories
via SE(3) transformations to novel object poses.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The paper concludes that grounding visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse real-world environments. The authors note that the method inherits the scalability of generative editing while preserving the physical grounding offered by simulation-based methods. p. 8 This allows our method to inherit the scalability of generative editing. Any source–target
pair can be augmented, while preserving the physical grounding that simulation-based methods offer. Furthermore, the inclusion of geometric perturbations—such as object rotations, translations, and
rescaling—proved critical for improving out-of-distribution generalization. We believe that grounding
visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse
real-world environments.Where the paper says it
Where the paper says it
The paper highlights limitations, noting that automated affordance matching would be valuable to handle broader categories of objects. Additionally, Pose6DAug inherits the failure modes of external vision models like 3D mesh extractors and video inpainting networks. The authors also recommend conducting thorough investigations to implement robust mitigation strategies to minimize potential risks associated with unexpected behaviors in Pose6DAug. p. 0 Pose6DAug: Physically Plausible Multi-view Object
Swapping for Robot Data Augmentation
Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3†
1KAIST 2Korea University 3RLWRLD
{alphabet1, seonghyp, jinwoos}@kaist. kr
Abstract
Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects
whose appearance or geometry deviates from the training distribution.Where the paper says it
The paper evaluates the method on the RoboCasa benchmark, where it provides consistent performance gains across diverse novel object instances. The evaluation protocol involves pairing hard samples with source episodes and generating 16 augmented episodes for each pair, yielding 176 augmented episodes for both Pose6DAug and VACE. p. 6 For MimicGen, generating a physically valid trajectory is not always feasible for a given scene
initialization; we therefore allow 32 attempts per pair and retain only successful rollouts. This
protocol yields 176 augmented episodes for both Pose6DAug and VACE. By operating in 3D space rather than purely
in pixels, Pose6DAug ensures that synthesized demonstrations maintain both physical admissibility
and multi-view consistency. Our experimental evaluation on the RoboCasa benchmark demonstrates
that Pose6DAug improves the performance of VLA policies on challenging and novel object instances. 3 Metrics
We report three metrics, evaluated over 200 rollouts per configuration for the main evaluation and 20
rollouts per instance for the hard-example evaluation (Table 2):
• Success Rate: The fraction of rollouts in which the robot successfully completes the task. • Turnover Ratio: Among the set of meshes that never succeeded under the base checkpoint,
the fraction that achieve at least one success after training with augmented data.Where the paper says it
Where the paper says it
Where the paper says it
The paper compares Pose6DAug against VACE and MimicGen, showing that MimicGen's reliance on successful rollout generation creates a coverage bottleneck. In contrast, Pose6DAug anchors the replacement object with an explicit 3D mesh rendered at a shared world-frame pose, ensuring geometric consistency across both egocentric and exocentric views by construction. p. 8 VACE circumvents the coverage problem by
editing on image space, but editing each view independently introduces cross-camera discrepancies
in object identity, scale, and contact geometry that propagate as noisy supervision during training. In
contrast, Pose6DAug anchors the replacement object with an explicit 3D mesh rendered at a shared
world-frame pose, ensuring geometric consistency across both egocentric and exocentric views by
construction. Furthermore, the inclusion of geometric perturbations—such as object rotations, translations, and
rescaling—proved critical for improving out-of-distribution generalization. We believe that grounding
visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse
real-world environments. Any source–target
pair can be augmented, while preserving the physical grounding that simulation-based methods offer. The turnover results further support this: Pose6DAug not only improves aggregate performance but
extends the policy’s capability to previously failed object instances, without sacrificing in-distribution
performance. †Corresponding author. arXiv:2606. arXiv:2606. 20118v2 [cs. arXiv:2606. 20118v2 [cs. arXiv:2606. 20118v2 [cs. 20118v2 [cs. RO] 19 Jun 2026 20118v2 [cs. RO] 19 Jun 2026 20118v2 [cs. RO] 19 Jun 2026 Pose6DAug: Physically Plausible Multi-view Object
Swapping for Robot Data Augmentation
Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3†
1KAIST 2Korea University 3RLWRLD
{alphabet1, seonghyp, jinwoos}@kaist. kr
Abstract
Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects
whose appearance or geometry deviates from the training distribution. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. Pose6DAug: Physically Plausible Multi-view Object
Swapping for Robot Data Augmentation
Jonghoon Lee1∗Seong Hyeon Park1∗Byungwoo Jeon1 Minha Lee2 Jinwoo Shin1,3†
1KAIST 2Korea University 3RLWRLD
{alphabet1, seonghyp, jinwoos}@kaist. kr
Abstract
Vision-language-action (VLA) policies have shown strong potential for generalpurpose manipulation, yet they often fail on novel, out-of-distribution objects
whose appearance or geometry deviates from the training distribution. We believe that grounding
visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse
real-world environments. References
[1] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan
Welker, Ayzaan Wahid, et al. We provide qualitative
examples of augmentations across all three methods in Figure 3. 3 Metrics
We report three metrics, evaluated over 200 rollouts per configuration for the main evaluation and 20
rollouts per instance for the hard-example evaluation (Table 2):
• Success Rate: The fraction of rollouts in which the robot successfully completes the task. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. We believe that grounding
visual augmentation in 3D geometry is a promising direction for scaling robot learning across diverse
real-world environments. References
[1] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan
Welker, Ayzaan Wahid, et al. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics. 5 Conclusion
In this paper, we introduced Pose6DAug, a 3D-aware data augmentation framework designed to
enhance the generalization of robotic manipulation policies. To address the high cost of multi-view
data collection, our approach leverages a pose-aware object swapping pipeline that integrates 3D
mesh extraction, video inpainting, and robot kinematics.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 6 ---
Figure 3 illustrates the augmentation quality of each method across all three camera views. VACE exhibits several failure modes that compound during training: the appearance of the target object changes across frames (identity drift), multi-view consistency is not maintained as the object differs in shape and color between the left, right, and wrist cameras, and in some views the source object is not edited at all, leaving the original object intact alongside the intended replacement. p. 8 Figure 3 illustrates the augmentation quality of each method across all three
camera views. VACE exhibits several failure modes that compound during training: the appearance
of the target object changes across frames (identity drift), multi-view consistency is not maintained as
the object differs in shape and color between the left, right, and wrist cameras, and in some views the
source object is not edited at all, leaving the original object intact alongside the intended replacement. These inconsistencies produce unreliable training signals that can mislead the policy. MimicGen,
while operating in simulation, generates physically implausible action trajectories for challenging
object geometries—as shown in Figure 3, the adapted trajectories fail to produce valid grasps across
all rollout attempts, resulting in no usable augmented data for these instances.Where the paper says it
Where the paper says it
--- Page 7 ---
Table 2 reports the overall success rate (%) in the top block, experimented with the model fine-tuned on the most challenging 8 unseen, out-of-distribution object instances, evaluated on 20 pick-and-place episodes per instance (Bold indicates the best result). Table 3 reports overall success rate (%) evaluated on all episodes, including both previously successful and previously failed instances. p. 8 We first
examine whether appending augmented episodes to the original training set can turn previously failed
rollouts into successes. Table 1 reports success rates and turnover ratios evaluated exclusively on
episodes where the base policy originally failed. Table 1 reports success rates and turnover ratios evaluated exclusively on
episodes where the base policy originally failed. Pose6DAug achieves the highest average success
rate of 22. Fine-tuning a VLA on data augmented by our
method improves success rates by 16. 5% relative to the state-of-the-art baseline on
novel objects, while preserving in-distribution performance.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 8 ---
Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page 10. p. 0 These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa. These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 14 ---
Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page 10. The paper is available on page 6.
--- Page 14 ---
Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and page10 pg10. p. 0 These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa.Where the paper says it
--- Page 14 ---
Figure 4 shows failure cases of Pose6DAug. The paper is available on page 4. The paper is available on page 5 and page 7 and page 8 and page 9 and pg10 pg10. p. 0 These results show that
multi-view and physically consistent augmentation is a practical path to scalable
VLA generalization. Project page: https://jhoonjwa.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
3D-grounded augmentation ensures that synthesized demonstrations maintain both physical admissibility and multi-view consistency by operating directly in 3D space, which
bridges the gap between kinematic validity and visual photorealism.
This allows for ageometrically consistent rendering across all camera views
when fine-tuning VLA policies. -
The system can generate new, physically grounded demonstrations for failure modes without requiring new data collection by utilizing
successful episodes into targeted demonstrations for its failure modes, without any new data collection.
This addresses the bottleneck of collecting multi-view teleoperation data for every failure case. -
The improved AI system can handle novel object instances with a performance gain of 16.5% relative to state-of-the-art baselines while preserving in-distribution performance, as demonstrated by
relative 16.5% success rate improvements
on the RoboCasa benchmark. -
The framework can maintain structural coherence across all observations by
anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory,
ensuring thatall camera views render the same 3D mesh at a shared world-frame pose.
-
The system can be enhanced to provide fine-grained recovery metrics, as it offers
Class Recovery: A per-instance indicator of whether a specific object instance that consistently failed under the base policy achieves at least one successful rollout after fine-tuning.
Abstract
Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. The standard remedy is to collect multi-view teleoperation data for every failure case, but this scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven data augmentation framework that turns existing successful episodes into targeted demonstrations for a policy's failure modes, without any new data collection. Our key insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. By swapping only the manipulated object while preserving this trajectory, we obtain new and physically grounded demonstrations. However, the challenge lies in synthesizing object swaps, e.g., naive 2D video editing breaks multi-view consistency and physical plausibility, particularly under heavy occlusion and egocentric viewpoints. Our method instead operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory. This ensures geometrically consistent renderings across all camera views. Fine-tuning a VLA on data augmented by our method improves success rates by 16.5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance. On a real robot, it more than doubles the success rate of the base policy. These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization.
Sources
- OpenVLA: An Open-Source Vision-Language-Action Model
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Octo: An Open-Source Generalist Robot Policy
- RLDX-1 Technical Report
- Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Models
- ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation
- LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
- Wan: Open and Advanced Large-Scale Video Generative Models
- LTX-Video: Realtime Video Latent Diffusion
- VOID: Video Object and Interaction Deletion
- RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfer
- RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots
- Scaling Robot Learning with Semantically Imagined Experience
- GenAug: Retargeting behaviors to unseen situations via Generative Augmentation
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- Decoupled Weight Decay Regularization
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation