MATE: Asynchronous Multi-Rate Diffusion Policy Composition
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "MATE: Asynchronous Multi-Rate Diffusion Policy Composition".
Dev: Here's the gist of the paper. Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance.
Rosa: Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance.
Dev: The core contribution of the paper is the introduction of LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition.
Rosa: This framework allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available.
Title and authors: Rosa: The paper is MATE: Asynchronous Multi-Rate Diffusion Policy Composition, and the authors are Zihao He, Hongjie Fang, Shirun Tang, Cewu Lu, Hao-Shu Fang. They are focusing on this idea of combining policies that run at different rates without forcing them into a single pipeline.
Dev: It seems like the name itself explains the big idea—asynchronous means they operate at different times and frequencies, and multi-rate means they handle multiple inputs with varying update speeds.
Taro: I wonder if this is something we can actually test outside of a perfectly controlled lab setting, or if it’s just theoretical stuff for now.
Rosa: The authors are showing that you can have one policy running at five times per second, and another running at twenty-five times per second, and MATE handles the composition between those two different rates.
Dev: They are showing that you can have a vision policy providing long-horizon guidance, which is slow, while a force feedback policy provides local contact guidance that is very fast.
Taro: So they’re suggesting that we don't have to make everything run at the same speed just to get better control in physical tasks.
Rosa: That’s right. They show how you can use the force/torque feedback for immediate, reactive corrections while still using the vision guidance for long-term planning and context.
The paper's summary: Dev: The paper summarizes the issue clearly by pointing out that current multimodal diffusion policies aren't built to handle information that arrives at different temporal rates and with different latencies.
Rosa: They explain that when you have vision, it’s usually heavy processing, and force or tactile signals can be lightweight and high-frequency, so fusing them all together forces the fast signal to wait for the slow one.
Taro: So the problem is that high-frequency feedback gets stuck waiting for a slow visual observation before it can be used to make a correction.
Dev: They propose LAG-Fusion as their solution, which allows each modality-specific policy to operate at its natural inference rate, meaning they don't have to synchronize perfectly.
Rosa: The key mechanism is that when guidance from the slower policy arrives, they rebase it into the current action frame and then fuse it with a latency-aware weight schedule before updating the noise prediction.
The paper's improvements: Taro: I’m interested in how they handle that alignment part. Rebase is usually tricky when you have different reference frames for action representations, especially in diffusion models where you are predicting noise based on a future state.
Dev: They introduce a specific reference-frame rebasing rule derived for diffusion variables under relative action representations to ensure consistency when aligning the delayed guidance.
Rosa: They also use a latency-aware guidance fusion strategy where the weighting schedule changes depending on time, giving more influence to fresher feedback, like the force signal when you're making an immediate contact.
Taro: So they’re essentially saying that you treat latency as a proxy for confidence in the guidance and weigh it accordingly during fusion.
Dev: The result is that they can combine the long-horizon visual guidance with the fast local contact guidance, leading to better policy responsiveness and task performance in contact-rich manipulation tasks.
Conclusion: Rosa: So to wrap up, MATE uses LAG-Fusion to allow different modalities to run at their native frequencies without creating a bottleneck for high-frequency inputs like force feedback.
Dev: It achieves this by aligning delayed guidance through rebase and fusing it with weights that prioritize temporal freshness, which lets the fast signal contribute immediately.
Taro: The implication for me is that we can finally build systems where physical interaction correction is truly reactive, not just reacting to a delayed visual state.
Rosa: It’s a modular way to exploit heterogeneous sensory modalities in diffusion policies, and it successfully avoids bottlenecking high-frequency branches by letting them run at their native frequency.
Dev: Future work could look into adding control priors, like hybrid force-position control, into the learned policies for more stable contact regulation in real-world scenarios.
Taro: Extending this framework to things like flow matching or dexterous hands seems like a promising path forward for making these compositions even more robust.
Zihao He, Hongjie Fang, Shirun Tang, Cewu Lu, Hao-Shu Fang
Shanghai Jiao Tong University · Noematrix Institute
cs.RO, cs.AI
Submitted: 2026-07-19
Updated: 2026-11-06
Project page: https://lag-fusion.github.io
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 96/100
The gist: Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance.
Key concepts
- Asynchronous Multimodal Diffusion Policy Composition
- This is the core idea of MATE, allowing different sensory policies (like vision and force) to run at their own speeds. Instead of forcing them into one slow pipeline, they work independently and communicate guidance when they are ready, improving efficiency.
- Reference-Frame Rebase
- This mechanism ensures consistency when combining guidance from policies running at different times or frequencies. It mathematically transforms delayed guidance signals into the current action's reference frame so that the fused information is always aligned correctly for the next step of denoising.
- Latency-Aware Guidance Fusion
- The fusion strategy weights visual and force guidance based on how 'fresh' they are. Early actions rely more on vision, while later actions prioritize high-frequency force feedback, treating latency as a measure of confidence in the current action.
Terminology
Summary
Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance. p. 0 (D) Our asynchronous fusion policy
decouples modality-specific inference and composes their outputs with latency-aware guidance, preserving
both modularity and high-frequency reactivity. Abstract: Diffusion policies have shown strong potential for robotic imitation
learning, and recent extensions incorporate additional modalities to improve manipulation performance. (D) Our asynchronous fusion policy
decouples modality-specific inference and composes their outputs with latency-aware guidance, preserving
both modularity and high-frequency reactivity. Abstract: Diffusion policies have shown strong potential for robotic imitation
learning, and recent extensions incorporate additional modalities to improve manipulation performance.Where the paper says it
Where the paper says it
The core contribution of the paper is the introduction of LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. p. 0 Asynchronous Multimodal Diffusion Policy
Composition via Latency-Aware Guidance Fusion
Zihao He∗,1
, Hongjie Fang∗,1
, Shirun Tang2
, Cewu Lu1,2,3,†
, Hao-Shu Fang△,†
1Shanghai Jiao Tong University 2Noematrix 3Shanghai Innovation Institute
△Independent Researcher ∗Equal Contribution †Corresponding Authors
https://lag-fusion. io/
Single Policy Slow Policy
Fast Policy
Vision
Enc. We propose LAGFusion, a latency-aware guidance fusion framework for asynchronous multimodal
diffusion policy composition. LAG-Fusion allows modality-specific policies to
operate at their native inference rates and contribute denoising guidance whenever
available.Where the paper says it
Where the paper says it
Framework Overview and Composition Mechanism
The LAG-Fusion framework is designed for settings where modality-conditioned policies operate at different sensing and inference rates, allowing them to compose without forcing all modalities into a single pipeline. The process involves integrating denoising guidance during sampling by rebasing delayed guidance into the current action reference frame and fusing this aligned guidance with latency-aware weights. p. 3 1), LAG-Fusion handles
asynchronous queries by rebasing delayed guidance into the current action reference frame (§3. 2)
and fusing the aligned guidance with latency-aware weights (§3.Where the paper says it
The key steps in the asynchronous composition are as follows:
-
Modality-specific policies operate at their native inference rates. p. 0
Where the paper says it
We propose LAGFusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. LAG-Fusion allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available.
-
Delayed guidance is aligned through reference-frame rebasing before being fused with latency-aware weights during sampling. p. 1
Where the paper says it
LAG-Fusion allows modalityspecific policies to operate at their native inference rates and contribute denoising guidance whenever available. To make asynchronous composition consistent, delayed guidance is aligned through reference-frame rebasing before being fused with latency-aware weights during sampling.
-
The final noise prediction is updated using the composed guidance, which involves mapping the fused guidance back to frame T1. p. 2
Where the paper says it
(Right) In the diffusion composition head, the overlapping noisy action segment is rebased into the current force frame for force denoising. The corresponding vision guidance is rebased to the same frame, fused with force guidance, and mapped back to update the full-horizon noise prediction.
Reference-Frame Rebase for Consistency
To ensure consistency in asynchronous composition, a reference-frame rebasing rule is derived for diffusion variables under relative action representations. This rebase operation addresses the issue where delayed guidance from a lower-frequency policy might be expressed in an outdated reference frame. p. 3 2 Reference-Frame Rebase for Diffusion Variables
In asynchronous composition, all modality guidance must be expressed in a common action reference frame. We use relative action chunks [45, 1, 46, 47], where each action is defined with respect
to the robot state at the policy query time.Where the paper says it
The transformation involves defining an action representation that has an orthogonal linear part and a separate affine offset, leading to the rule (3) for the full action-space transformation. p. 3 Since composition is performed in the noise space,
the rebase operation must be compatible with clean actions, noisy samples, and predicted noises. We thus need an action representation whose frame transformation has an orthogonal linear part and
a separate affine offset. We now derive how the affine rebase applies to different
diffusion variables. For the clean actions, they naturally follow the full affine transform in Eq. (4)
Since R2←1 ∈ SO(3), the linear part A2←1 is orthogonal. Next, we show that this affine rebase leads
to different rules for different diffusion variables: clean actions use the full affine transform, noisy
samples include a scaled offset, and noises are transformed only by the orthogonal linear part A2←1.Where the paper says it
Where the paper says it
Where the paper says it
Latency-Aware Guidance Fusion Strategy
The fusion strategy employs a latency-aware weighting schedule to combine the aligned guidance signals. This schedule is index-dependent, where the vision weight linearly decreases from a high value (0. p. 5 Therefore, we assign a larger vision weight at the
beginning of the segment and gradually decrease it along the horizon, while increasing the force
weight accordingly. This schedule treats latency as a proxy for confidence: older or farther-ahead
guidance is down-weighted, whereas fresher high-frequency feedback is given stronger influence. To make asynchronous composition consistent, we derive a referenceframe rebasing rule for diffusion variables under relative action representations,
enabling delayed guidance to be aligned before fusion. We instantiate LAGFusion in contact-rich manipulation by composing a low-frequency vision policy
with a high-frequency force policy. In contrast, the force policy is queried at the
current time t2 and provides more up-to-date local contact feedback, making it more reliable for
reactive corrections within the selected segment. Therefore, we assign a larger vision weight at the
beginning of the segment and gradually decrease it along the horizon, while increasing the force
weight accordingly.Where the paper says it
Where the paper says it
Where the paper says it
This weighting schedule reflects temporal freshness: early actions rely more on vision guidance, while later actions rely more on force/torque guidance. p. 5 For
latency-aware fusion, the weights are designed to reflect the temporal freshness of each modality. The vision policy provides long-horizon guidance from the last visual observation at t1, which is
reliable near the beginning of the selected segment but becomes less certain for later steps as the
robot state evolves without updated visual inference. Therefore, we assign a larger vision weight at the
beginning of the segment and gradually decrease it along the horizon, while increasing the force
weight accordingly. This schedule treats latency as a proxy for confidence: older or farther-ahead
guidance is down-weighted, whereas fresher high-frequency feedback is given stronger influence.Where the paper says it
Where the paper says it
Experimental Validation and Performance Gains
Experiments on contact-rich manipulation tasks demonstrate that LAG-Fusion improves policy responsiveness and task performance over synchronous fusion and specially designed force-aware baselines. Table 1 shows that LAG-Fusion achieves the best performance compared to other fusion methods. p. 6 0%
Table 1: Evaluation Results. LAG-Fusion achieves the best performance, indicating that asynchronous policy
composition and latency-aware guidance fusion are effective in contact-rich manipulation.Where the paper says it
The effect of inference frequency analysis confirms that preserving native inference rates benefits both standalone and composed policies. p. 7 Experiments on two contact-rich tasks show that LAG-Fusion
improves the performance over synchronized fusion and slow-fast baselines. Our frequency analysis
further confirms that preserving native inference rates benefits both standalone and composed policies. Standalone force policies benefit from
native high-frequency inference (Q3). We first isolate the low-dimensional force
policy to examine whether force/torque
feedback should be treated as a highfrequency control signal.Where the paper says it
Where the paper says it
Conclusion and Future Directions
LAG-Fusion provides a modular and practical interface for exploiting heterogeneous sensory modalities in diffusion-policy-based robot control. The framework successfully avoids bottlenecking high-frequency branches by allowing them to run at their native frequency while fusing guidance. p. 7 Therefore, LAG-Fusion avoids synchronizing all modalities to the slowest policy and instead allows each modality, especially lightweight
high-frequency modalities, to operate its native frequency while fusing their guidance. 5 Conclusion
We presented LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. A promising direction is to incorporate control priors, such as hybrid force-position control [51, 11], into the learned policies, which
may provide more stable contact regulation during asynchronous composition. Second, our current
framework is developed for diffusion-based policies, and extending it to flow matching [52, 53] or
other generative action modeling frameworks remains future work.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Asynchronous Multimodal Diffusion Policy Composition (LAG-Fusion) allows modality-specific policies to operate at their native inference rates, enabling lightweight high-frequency modalities like force feedback to run at their native frequency without being bottlenecked by a slower policy. This enables
latency-aware multimodal control while preserving the modularity of existing pretrained policies.
-
The system can improve policy responsiveness and task performance in contact-rich manipulation tasks by utilizing high-frequency force/torque feedback for local contact guidance, which is crucial for rapid corrections during physical interaction.
-
The proposed framework allows the combination of long-horizon visual guidance (from a low-frequency vision policy) with high-frequency local contact guidance (from a fast force policy), leading to better task success rates on manipulation benchmarks like Flip and Polish Curve.
-
The reference frame rebasing rule for diffusion variables ensures that delayed guidance is aligned before fusion, making the asynchronous composition consistent by transforming delayed diffusion variables into the current action frame.
Abstract
Diffusion policies have shown strong potential for robotic imitation learning, and recent extensions incorporate additional modalities to improve manipulation performance. However, these modalities often differ not only in information content but also in sensing rates and inference latencies. Existing multimodal diffusion policies typically rely on synchronous fusion or manually designed multi-frequency architectures, which either slow down high-frequency feedback or limit extensibility to new modality combinations. We propose LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. LAG-Fusion allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available. To make asynchronous composition consistent, we derive a reference-frame rebasing rule for diffusion variables under relative action representations, enabling delayed guidance to be aligned before fusion. We instantiate LAG-Fusion in contact-rich manipulation by composing a low-frequency vision policy with a high-frequency force policy. Experiments under heterogeneous modality latencies show that LAG-Fusion improves policy responsiveness and task performance over synchronous fusion and specially designed force-aware baselines.
Sources
- Multi-Modal Manipulation via Multi-Modal Policy Consensus
- Force Policy: Learning Hybrid Force-Position Control Policy under Interaction Frame for Contact-Rich Manipulation
- Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation
- DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation
- TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
- OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer
- RISE: 3D Perception Makes Real-World Robot Imitation Simple and Effective
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation