MATE: Asynchronous Multi-Rate Diffusion Policy Composition

summary

The gist

Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance.

In short

MATE introduces LAG-Fusion, a framework for asynchronously composing diffusion policies from different modalities that operate at varying inference rates. This allows high-frequency policies, like force feedback, to run at their native speed without slowing down slower ones. It achieves better performance in contact manipulation by fusing guidance intelligently based on temporal freshness and aligning delayed guidance through reference-frame rebasing.

Key concepts

Asynchronous Multimodal Diffusion Policy Composition
This is the core idea of MATE, allowing different sensory policies (like vision and force) to run at their own speeds. Instead of forcing them into one slow pipeline, they work independently and communicate guidance when they are ready, improving efficiency.
Reference-Frame Rebase
This mechanism ensures consistency when combining guidance from policies running at different times or frequencies. It mathematically transforms delayed guidance signals into the current action's reference frame so that the fused information is always aligned correctly for the next step of denoising.
Latency-Aware Guidance Fusion
The fusion strategy weights visual and force guidance based on how 'fresh' they are. Early actions rely more on vision, while later actions prioritize high-frequency force feedback, treating latency as a measure of confidence in the current action.

Terminology used across episodes

This episode discusses

The paper

MATE: Asynchronous Multi-Rate Diffusion Policy Composition · Read on arXiv

Zihao He, Hongjie Fang, Shirun Tang, Cewu Lu, Hao-Shu Fang

Shanghai Jiao Tong University · Noematrix Institute

Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance. However, these modalities often differ not only in information content but also in sensing rates and inference latencies. Existing multimodal diffusion policies typically rely on synchronous fusion or manually designed multi-frequency architectures, which either slow down high-frequency feedback or limit extensibility to new modality combinations. We propose LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. LAG-Fusion allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available. To make asynchronous composition consistent, we derive a reference-frame rebasing rule for diffusion variables under relative action representations, enabling delayed guidance to be aligned before fusion. We instantiate LAG-Fusion in contact-rich manipulation by composing a low-frequency vision policy with a high-frequency force policy. Experiments under heterogeneous modality latencies show that LAG-Fusion improves policy responsiveness and task performance over synchronous fusion and specially designed force-aware baselines.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "MATE: Asynchronous Multi-Rate Diffusion Policy Composition".

Dev: Here's the gist of the paper. Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance.

Rosa: Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance.

Dev: The core contribution of the paper is the introduction of LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition.

Rosa: This framework allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available.

Title and authors: Rosa: The paper is MATE: Asynchronous Multi-Rate Diffusion Policy Composition, and the authors are Zihao He, Hongjie Fang, Shirun Tang, Cewu Lu, Hao-Shu Fang. They are focusing on this idea of combining policies that run at different rates without forcing them into a single pipeline.

Dev: It seems like the name itself explains the big idea—asynchronous means they operate at different times and frequencies, and multi-rate means they handle multiple inputs with varying update speeds.

Taro: I wonder if this is something we can actually test outside of a perfectly controlled lab setting, or if it’s just theoretical stuff for now.

Rosa: The authors are showing that you can have one policy running at five times per second, and another running at twenty-five times per second, and MATE handles the composition between those two different rates.

Dev: They are showing that you can have a vision policy providing long-horizon guidance, which is slow, while a force feedback policy provides local contact guidance that is very fast.

Taro: So they’re suggesting that we don't have to make everything run at the same speed just to get better control in physical tasks.

Rosa: That’s right. They show how you can use the force/torque feedback for immediate, reactive corrections while still using the vision guidance for long-term planning and context.

The paper's summary: Dev: The paper summarizes the issue clearly by pointing out that current multimodal diffusion policies aren't built to handle information that arrives at different temporal rates and with different latencies.

Rosa: They explain that when you have vision, it’s usually heavy processing, and force or tactile signals can be lightweight and high-frequency, so fusing them all together forces the fast signal to wait for the slow one.

Taro: So the problem is that high-frequency feedback gets stuck waiting for a slow visual observation before it can be used to make a correction.

Dev: They propose LAG-Fusion as their solution, which allows each modality-specific policy to operate at its natural inference rate, meaning they don't have to synchronize perfectly.

Rosa: The key mechanism is that when guidance from the slower policy arrives, they rebase it into the current action frame and then fuse it with a latency-aware weight schedule before updating the noise prediction.

The paper's improvements: Taro: I’m interested in how they handle that alignment part. Rebase is usually tricky when you have different reference frames for action representations, especially in diffusion models where you are predicting noise based on a future state.

Dev: They introduce a specific reference-frame rebasing rule derived for diffusion variables under relative action representations to ensure consistency when aligning the delayed guidance.

Rosa: They also use a latency-aware guidance fusion strategy where the weighting schedule changes depending on time, giving more influence to fresher feedback, like the force signal when you're making an immediate contact.

Taro: So they’re essentially saying that you treat latency as a proxy for confidence in the guidance and weigh it accordingly during fusion.

Dev: The result is that they can combine the long-horizon visual guidance with the fast local contact guidance, leading to better policy responsiveness and task performance in contact-rich manipulation tasks.

Conclusion: Rosa: So to wrap up, MATE uses LAG-Fusion to allow different modalities to run at their native frequencies without creating a bottleneck for high-frequency inputs like force feedback.

Dev: It achieves this by aligning delayed guidance through rebase and fusing it with weights that prioritize temporal freshness, which lets the fast signal contribute immediately.

Taro: The implication for me is that we can finally build systems where physical interaction correction is truly reactive, not just reacting to a delayed visual state.

Rosa: It’s a modular way to exploit heterogeneous sensory modalities in diffusion policies, and it successfully avoids bottlenecking high-frequency branches by letting them run at their native frequency.

Dev: Future work could look into adding control priors, like hybrid force-position control, into the learned policies for more stable contact regulation in real-world scenarios.

Taro: Extending this framework to things like flow matching or dexterous hands seems like a promising path forward for making these compositions even more robust.

More episodes

← Home