Mixture of Frames Policy: Multi-Frame Action Denoising for Bimanual Mobile Manipulation

arXiv:2607.11884 · cs.RO · Submitted 2026-07-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Mixture of Frames Policy".

Dev: Here's the gist of the paper. Mixture of Frames Policy (MoF) is a diffusion policy that performs synchronized action denoising across multiple coordinate frames to improve performance in bimanual mobile manipulation tasks The best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines Problem Statement Robotic manipulation systems are inherently multi-frame, as local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame Modern visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame This suggests that frames should not only be provided as observations, but should also define alternative action spaces in which denoising can be performed Consistent Multi-Frame Denoising MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame Specifically, let Fc denote the canonical frame in which the scheduler state x k c is stored For each frame Fm ∈ F, we transform the canonical-frame noisy action sample x k c into the expert frame via mTc(·), feed it to a frame-specialized denoiser ϵ θ m, transform the predicted noise back to the canonical frame using cTm(·), and fuse the aligned predictions This synchronized update has two benefits: First, all experts contribute to the same underlying diffusion trajectory, avoiding the mode-mismatch problem Second, because fusion occurs in the canonical noise space, the diffusion scheduler remains unchanged Transformation-Compatible Action Representation The synchronized denoising procedure in Sec.

Rosa: 3.2 requires frame transformations to be applied directly to intermediate diffusion states, whose rotation channels generally do not form a valid element of SO(3) This poses a challenge for the row-vector 6D rotation representation used in standard Diffusion Policy implementations, which stores the first two rows of the rotation matrix We instead use a column-vector rotation representation For one robot arm at one action timestep, the pose/action channels expressed in the canonical frame Fc are xc =

pc, c 1 c, c 2 c, g: , where pc is the end-effector position This operation is well-defined even when c 1 c and c 2 c are arbitrary noisy vectors rather than columns of a valid rotation matrix Where the gains come from The within-episode router dynamics track the visible phase structure of each task, suggesting that MoF-MoE’s advantage over Oracle Frame stems primarily from its ability to assign different frames to different task phases The two tasks with the largest router modulation, Threading and Drawer Cleanup, are also the tasks where MoF-MoE most improves over Oracle Frame This relationship suggests that MoF-MoE’s advantage over the best single frame stems primarily from its ability to assign different frames to different task phases, rather than from a static preference for any one frame Pruning redundant reasoning tokens The orthogonalization ablation validates the transformation-compatible action representation When noisy rotation channels are projected back to SO(3) before applying frame transformations, MoFMoE drops by 7.0 percentage points on average, and MoF-Ensemble collapses completely This is because the orthogonalization completely destroys all non-canonical-frame experts Removing the best-performing single frame from the expert set reduces performance by 6.2 percentage points, showing that the expert set is not redundant and that task-relevant frames contribute complementary denoising structure How the training signal is built The mixed objective Lmix supervises only the fused canonical-frame prediction, so experts with small router weights may receive weak gradients To stabilize training, we add a per-expert auxiliary denoising loss that directly supervises each expert in its own frame The final training objective is L = Lmix +λauxLaux This suggests that directly supervising each expert in its own frame is important for keeping all frame experts meaningful during training We present Mixture of Frames Policy (MoF), a diffusion-policy framework for denoising actions across multiple coordinate-frame parameterizations Our experiments show that action-frame choice has a substantial and task-dependent effect on diffusion-policy performance, and that fusing framespecialized denoisers improves over baselines including oracle task-level frame selection MoF also matches or outperforms Oracle Frame on most tasks The largest improvement is on Threading (+7.7 points) In the pouring task, MoF-MoE achieves 85% success, with only three failures in picking up (1 left, 2 right) The Left policy achieves only 15% success, with 12 right-cup pickup failures, two left-cup pickup failures, two pouring failures, and one rollout that repeatedly continues the pouring motion even after successful transfer The Right policy achieves 70% success, but all of its failures are left-side failures The Rel Traj policy achieves only 5% success and is highly unstable during precise pickup The Left policy achieves 55% success and fails nine times at cup pickup The Right policy fails in all 20 rollouts and exhibits highly unstable navigation throughout Limitations First, while theoretically adaptable to larger vision-language action (VLA) models, our current experiments focus on smaller diffusion policies Second, the router is supervised solely via denoising loss, which might not strictly correspond to policy performance Third, MoF relies on a designer-specified set of candidate frames and accurate frame transformations from robot proprioception Future work could investigate learning or automatically discovering task-relevant frames and study how frame uncertainty affects multi-frame denoising in real-world deployment Acknowledgments The authors would like to thank Yifan Hou for the help during RB-Y1 controller implementation, and Mengda Xu for the helpful discussion during project brainstorming This work was supported in part by the NSF Award and Apple, and Toyota Research Institute The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors References

1: C.

Dev: Chi, Z.

Rosa: Xu, S.

Title and authors: Rosa: So what's the actual mechanism they propose? They describe it as maintaining a single canonical diffusion state, but then re-expressing that state into several task-relevant frames and feeding those into frame-specialized denoisers.

Dev: They do this by taking the canonical frame, say Fc, and transforming it to an expert frame Fm using a transformation mTc(·), running the expert denoiser epsilon theta m m, and then transforming that noise prediction back to the canonical frame using cTm(·).

Rosa: That sounds complicated for real-time control. How do you make sure all these experts are actually contributing to the same thing?

Dev: The point is that this synchronized update has two main benefits: first, all experts contribute to the same underlying diffusion trajectory, which avoids that mode mismatch problem we often see in standard Mixture-of-Experts or MoE baselines. Second, because the fusion happens back in the canonical noise space, the diffusion scheduler itself doesn't have to change.

Rosa: That makes sense for stability. So they’re trying to keep the overall process consistent even while switching between frame perspectives?

Dev: Yeah, they are trying to make sure that when you fuse those predictions, you're fusing them in a way that respects the diffusion process itself. But there’s a practical hurdle they run into with how they represent rotation channels.

The paper's summary: Rosa: They hit this problem where the transformation-compatible action representation is tricky because the rotation channels don't always form a valid element of SO(three), which clashes with how standard Diffusion Policy implementations handle their 6D rotation representations, like using row vectors <ref:2607.11884#pg3>.

Dev: So they switch things up and use a column-vector representation instead, which makes the frame transformations work even when the input vectors for c1 c and c2 c are just arbitrary noisy vectors instead of guaranteed columns of a valid rotation matrix.

Rosa: That’s clever, adapting the math to fit the physics of what they're trying to model. But where do you see this actually helping a robot on the ground?

Dev: The paper suggests that the gains come from how within-episode router dynamics track visible phase structure of each task. They found that MoF outperforms oracle frame selection and standard Mixture-of-Experts baselines because it can assign different frames to different task phases.

Rosa: So, it's not about picking one perfect frame for the whole episode; it's about letting the system decide which frame to use based on what the robot is currently doing?

Dev: Precisely. They point out that the two tasks with the biggest router modulation, Threading and Drawer Cleanup, are where MoF-MoE gets its most improvement over Oracle Frame selection. This suggests it's about adapting to phase changes rather than just sticking to one static frame.

The paper's improvements: Rosa: Beyond just the frame switching, they also talk about how they make the training signal better. They use a mixed objective Lmix that only supervises the fused canonical-frame prediction, which can leave experts with small router weights getting weak gradients.

Dev: To stabilize that, they add a per-expert auxiliary denoising loss that directly supervises each expert in its own frame. That way, even if an expert isn't heavily used by the router, it still gets a direct signal about its performance in its specific frame.

Rosa: So they’re essentially giving each frame expert a personal training objective so they don't just drift away when the main objective is focused on the fused result?

Dev: That’s right. And looking at the pruning ablation, when you project noisy rotation channels back to SO(three) before applying frame transformations, MoF-MoE drops by seven point zero percentage points on average, and MoF-Ensemble collapses completely because those non-canonical experts are just noise <ref:2607.11884#pg3>.

Rosa: That suggests that the expert set isn't redundant; each task-relevant frame actually contributes something complementary to the overall denoising structure.

Conclusion: Dev: So wrapping up, the authors of "Mixture of Frames Policy: Multi-Frame Action Denoising for Bimanual Mobile Manipulation" show that action-frame choice has a substantial effect on performance and that it’s task dependent. They confirm that fusing frame-specialized denoisers improves over baselines including oracle frame selection and standard Mixture-of-Experts (MoE) baselines.

Rosa: The biggest win they highlight is on the Threading task, where MoF-MoE showed a gain of seven point seven points compared to the best single frame approach <ref:2607.11884#pg3>. They also show it matches or outperforms Oracle Frame on most tasks overall, though that improvement is most significant when the task changes its requirements.

Dev: And looking at their results on the pouring task, MoF-MoE achieved eighty-five percent success with only three failures in picking up objects—one left, two right. That’s a big jump compared to the other policies they tested.

Taro: From my testing standpoint, it’s interesting that the router dynamics track visible phase structure of each task; it means MoF-MoE's advantage over Oracle Frame stems primarily from its ability to assign different frames to different task phases, not just a static preference for any one frame.

Rosa: So if you only listen to the show, what does this mean for us? It means that for complex tasks like bimanual manipulation, we shouldn't just pick one fixed way of looking at the action; we need a system that can dynamically switch between frames as the task progresses.

Dev: That’s right. We’ve got a framework here, the Mixture of Frames Policy, which lets us leverage different coordinate systems to model complex movements more effectively in diffusion policies. We'll keep an eye on how they handle real-world deployment and frame uncertainty next.

Dian Wang, Jisang Park, Xiaomeng Xu, Han Zhang, Shuran Song

Stanford University

cs.RO

Submitted: 2026-07-13

Updated: 2026-10-28

Project page: https://mofpo.github.io

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 93/100

The gist: Mixture of Frames Policy (MoF) is a diffusion policy that performs synchronized action denoising across multiple coordinate frames to improve performance in bimanual mobile manipulation tasks The…

Key concepts

Consistent Multi-Frame Denoising
This method keeps a single main state but transforms it into several task-relevant frames. Specialized denoisers process each frame's view, and their noise predictions are fused back into the main frame's noise space. This ensures all experts work on the same underlying trajectory, avoiding confusion between different views.
Transformation-Compatible Action Representation
The paper solves a problem where rotating action vectors don't fit standard 6D rotation formats. They use a column-vector representation for actions in the canonical frame, which allows them to apply necessary frame transformations directly to intermediate diffusion states without breaking the mathematical structure.
Task-Dependent Frame Selection
MoF leverages the idea that different frames are best for different parts of a task. The system learns to assign specific frames to specific phases of manipulation, such as using one frame for end-effector movement and another for whole-body coordination, maximizing performance gains.
Per-Expert Auxiliary Denoising Loss
During training, the main loss only focuses on the final fused prediction. To keep all frame experts useful, a secondary loss is added that forces each expert to accurately predict noise in its own specific frame. This prevents less important experts from becoming inactive during learning.

Terminology

Summary

Mixture of Frames Policy (MoF) is a diffusion policy that performs synchronized action denoising across multiple coordinate frames to improve performance in bimanual mobile manipulation tasks The best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines

Problem Statement

Robotic manipulation systems are inherently multi-frame, as local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame Modern visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame This suggests that frames should not only be provided as observations, but should also define alternative action spaces in which denoising can be performed

Consistent Multi-Frame Denoising

MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame Specifically, let Fc denote the canonical frame in which the scheduler state x k c is stored For each frame Fm ∈ F, we transform the canonical-frame noisy action sample x k c into the expert frame via mTc(·), feed it to a frame-specialized denoiser ϵ θ m, transform the predicted noise back to the canonical frame using cTm(·), and fuse the aligned predictions This synchronized update has two benefits: First, all experts contribute to the same underlying diffusion trajectory, avoiding the mode-mismatch problem Second, because fusion occurs in the canonical noise space, the diffusion scheduler remains unchanged

Transformation-Compatible Action Representation

The synchronized denoising procedure in Sec. 3.2 requires frame transformations to be applied directly to intermediate diffusion states, whose rotation channels generally do not form a valid element of SO(3) This poses a challenge for the row-vector 6D rotation representation used in standard Diffusion Policy implementations, which stores the first two rows of the rotation matrix We instead use a column-vector rotation representation For one robot arm at one action timestep, the pose/action channels expressed in the canonical frame Fc are xc = [pc, c 1 c, c 2 c, g], where pc is the end-effector position This operation is well-defined even when c 1 c and c 2 c are arbitrary noisy vectors rather than columns of a valid rotation matrix

Where the gains come from

The within-episode router dynamics track the visible phase structure of each task, suggesting that MoF-MoE’s advantage over Oracle Frame stems primarily from its ability to assign different frames to different task phases The two tasks with the largest router modulation, Threading and Drawer Cleanup, are also the tasks where MoF-MoE most improves over Oracle Frame This relationship suggests that MoF-MoE’s advantage over the best single frame stems primarily from its ability to assign different frames to different task phases, rather than from a static preference for any one frame

Pruning redundant reasoning tokens

The orthogonalization ablation validates the transformation-compatible action representation When noisy rotation channels are projected back to SO(3) before applying frame transformations, MoFMoE drops by 7.0 percentage points on average, and MoF-Ensemble collapses completely This is because the orthogonalization completely destroys all non-canonical-frame experts Removing the best-performing single frame from the expert set reduces performance by 6. p. 6

Where the paper says it

5 points, suggesting that MoF is robust to the arbitrary choice of canonical frame. In contrast, removing the best-performing single frame from the expert set reduces performance by 6.

2 percentage points, showing that the expert set is not redundant and that task-relevant frames contribute complementary denoising structure

How the training signal is built

The mixed objective Lmix supervises only the fused canonical-frame prediction, so experts with small router weights may receive weak gradients To stabilize training, we add a per-expert auxiliary denoising loss that directly supervises each expert in its own frame The final training objective is L = Lmix +λauxLaux This suggests that directly supervising each expert in its own frame is important for keeping all frame experts meaningful during training

Conclusion

We present Mixture of Frames Policy (MoF), a diffusion-policy framework for denoising actions across multiple coordinate-frame parameterizations Our experiments show that action-frame choice has a substantial and task-dependent effect on diffusion-policy performance, and that fusing framespecialized denoisers improves over baselines including oracle task-level frame selection MoF also matches or outperforms Oracle Frame on most tasks The largest improvement is on Threading (+7.7 points) In the pouring task, MoF-MoE achieves 85% success, with only three failures in picking up (1 left, 2 right) The Left policy achieves only 15% success, with 12 right-cup pickup failures, two left-cup pickup failures, two pouring failures, and one rollout that repeatedly continues the pouring motion even after successful transfer The Right policy achieves 70% success, but all of its failures are left-side failures The Rel Traj policy achieves only 5% success and is highly unstable during precise pickup The Left policy achieves 55% success and fails nine times at cup pickup The Right policy fails in all 20 rollouts and exhibits highly unstable navigation throughout

Limitations

First, while theoretically adaptable to larger vision-language action (VLA) models, our current experiments focus on smaller diffusion policies Second, the router is supervised solely via denoising loss, which might not strictly correspond to policy performance Third, MoF relies on a designer-specified set of candidate frames and accurate frame transformations from robot proprioception Future work could investigate learning or automatically discovering task-relevant frames and study how frame uncertainty affects multi-frame denoising in real-world deployment

Acknowledgments

The authors would like to thank Yifan Hou for the help during RB-Y1 controller implementation, and Mengda Xu for the helpful discussion during project brainstorming This work was supported in part by the NSF Award and Apple, and Toyota Research Institute The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors

References

[1] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. p. 8

Where the paper says it

References [1] C. Cousineau, Y.

Du, B. Burchfiel, R. p. 8
Where the paper says it

Cousineau, Y. Burchfiel, R.

Tedrake, and S. p. 8
Where the paper says it

Burchfiel, R. Tedrake, and S.

Song

[2] M. Liu, X. Li, Z. Ling, Y. Li, and H. Su

[3] M. R. Montero, G. p. 8

Where the paper says it

PMLR, 2023. Montero, G.

Franzese, J. p. 8
Where the paper says it

Montero, G. Franzese, J.

Kober, and C. p. 8
Where the paper says it

Franzese, J. Kober, and C.

Della Santina

[4] J. Grannen, Y. p. 8

Where the paper says it

IEEE, 2024. Grannen, Y.

Wu, B. Vu, and D. Sadigh

[5] A. C.-W. Lee, I. Chuang, L.-Y Chen, and I Soltani

[6] J.-J Jiang, X.-M Wu, Y.-X He, L.-A Zeng, Y.-L Wei, D Zhang, and W-S Zheng

[7] Z. Chen, N. T Chan, Y Hou, C Tie, Z Liu, H Chen, J Chen, J Shi et al

[8] J Yang, I Huang, B Vu, M Bajracharya R Antonova and J Bohg

[9] H Ha, Y Gao, Z Fu, J Tan, and S Song

[10] P Sundaresan, R Malhotra, P Miao, J Yang et al

[11] Z Fu, T Z Zhao, and C Finn

[12] L Y Zhu, P Kuppili, R Punamiya et al

[13] Y Jiang, R Zhang, J Wong, C Wang et al

[14] X Xu, J Park, H Zhang, E Cousineau A Bhat et al

[15] L Righetti, M Kalakrishnan, P Pastor et al

[16] Y Huang, J Silvério, L Rozo, and D Caldwell

[17] H Zhao, D Wang, Y Zhu, X Zhu et al

[18] H Zhao, Y Qi, B Hu, Y Zhu et al

[19] J Gao, X Jin, F Krebs, N Jaquier and T Asfour

[20] A Bahety, P Mandikal, B Abbatematteo and R Martín-Martín

[21] R A Jacobs, M I Jordan, S J Nowlan and G E Hinton

[22] N Shazeer, A Mirhoseini, K Maziarz et al

[23] O Celik, D Zhou, G Li, P Becker and G Neumann

[24] O Celik, A Taranovic, and G Neumann

[25] Y Wang, Y Zhang, M Huo, T Tian et al

[26] K Guo, H Liu, Y Sun, R Zhao J Zhou and J Ma

[27] C Hao, X Zhai, Y Liu, and H Soh

[28] A Rodriguez, C Li, L Mazza, R Younis et al

[29] B Cheng, T Liang, S Huang, M Shao et al

[30] M Reuss, J Pari, P Agrawal and R Lioutikov

[31] H Zhou, D Blessing, G Li, O Celik X Jia et al

[32] J Yu, H Liu, Q Yu, J Ren C Hao et al

[33] D Wang, S Hart, D Surovik T Kelestemur H Huang et al

[34] X Xu, D Bauer, and S Song

[35] D Wang, B Hu, S Song R Walters and R Platt

[36] X Xu, Y Hou, Z Liu, and S Song

[37] N Chernyadev, N Backshall, X Ma, Y Lu et al

[38] Z Jiang, Y Xie, K Lin, Z Xu W Wan A Mandlekar L J Fan and Y Zhu

[39] J Song, C Meng, and S Ermon

(Note: The paper provided only goes up to Page 20, so page numbers in the references will be based on the structure provided.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved action representation allows for exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold, which enables consistent multi-frame denoising across different coordinate systems during action generation.

  2. The Mixture of Frames Policy (MoF) can reason over multiple coordinate frames simultaneously by instantiating one expert denoiser per reference frame and fusing their predictions at each denoising step through either a mixture-of-experts pipeline or simple ensembling.

  3. The system can adapt its action representation based on the task phase, as seen in the router analysis where the rel-traj expert is the dominant frame during the first part of the episode, where each gripper reaches and grasps its object, and the base expert takes over for the later contact-rich phase.

  4. The policy can achieve superior performance by leveraging task-dependent frame choice, as demonstrated by MoF improving over oracle frame selection and standard Mixture-of-Experts (MoE) baselines, leading to gains like +7.7 points on the Threading task.

  5. The system can be made more robust to arbitrary canonical frame choices because ablation studies show that Changing the canonical frame has only a modest effect, indicating that MoF is robust to the arbitrary choice of canonical frame.

  6. The system can be trained more effectively by incorporating per-expert auxiliary denoising loss, which supervises each expert directly against the diffusion noise expressed in its own frame, preventing experts with small router weights from drifting away.

Abstract

Robotic manipulation is inherently multi-frame: local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame. However, modern diffusion-based visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame. We propose Mixture of Frames Policy (MoF), a diffusion policy that performs synchronized action denoising across multiple coordinate frames. MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame. To make this possible for intermediate noisy diffusion states, we adopt a column-based 6D rotation representation within an SE(3) action parameterization that supports exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold. Across nine simulated bimanual manipulation tasks, we show that the best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines. We further evaluate MoF on two real-world bimanual mobile manipulation tasks, demonstrating that it outperforms all constituent single-frame baselines.

Sources

Related papers