Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Robots that Collaborate".
Dev: Here's the gist of the paper. Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning We propose Sequential Asymmetric Imitation (SAI), a single-teleoperator curriculum for learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures.
Rosa: The Core Problem Addressed Dual-robot manipulation success requires more than two locally competent policies because each robot's motion changes the load, contact state, and feasible motion of the other Small timing errors can therefore produce large interaction forces, object deformation, loss of grasp, or task failure.
Dev: Independent policies often fail because each robot follows its own nominal motion phase, even when the partner is delayed, stalled, or out of position.
Rosa: The key coordination challenges include temporal phase coupling, partner-contingent yielding, and interaction conflict mitigation.
Title and authors: Rosa: So we're talking about this paper now, "Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies." The title itself suggests they’re tackling how robots can work together when they have to interact physically through shared objects while one of them is partially observed.
Dev: Right, and the authors are Rosa and Dev. They're focusing on a specific way to learn this collaboration without needing two operators guiding both robots at the same time or any direct communication between them.
Rosa: Exactly. The implication here is that we don't need those synchronized demonstrations anymore to teach robots how to handle those tricky coordination issues, like waiting or yielding when the partner moves unexpectedly.
Dev: It sounds like they’re proposing a staged learning process where Robot A gets started first from a single human demonstration, then Robot B learns against that learned behavior before the system is refined with targeted corrections.
The paper's summary: Rosa: Basically, the core idea of this "Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies" paper is to learn coupled multi-robot behaviors without needing synchronized dual-operator demonstrations or explicit inter-robot communication. They propose something called Sequential Asymmetric Imitation, or SAI.
Dev: It breaks the coordination learning down into three asymmetric stages: first, Robot A learns basic task execution from unilateral demonstrations with a compliant human partner; then, Robot B learns against the deployed Robot A policy; and finally, they refine Robot A using sparse interventions near coordination failures.
Rosa: That's what I mean. The paper says this staged process is key because it exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch, insufficient yielding, and interaction conflict.
Dev: And the main question they’re trying to answer is whether coupled dual-robot policies can actually be learned from single-teleoperator demonstrations without having that synchronized dual operator data or reinforcement learning or explicit inter-robot communication.
The paper's improvements: Rosa: Now, what they suggest as improvements is this entire curriculum structure itself. They hypothesize that inducing inter-robot coordination can happen by structuring the partner distribution seen during imitation learning. It shifts the training from just compliant support to a deployed learned partner, and finally to closed-loop robot-robot execution with targeted corrections.
Dev: That shift exposes the policies to realistic coordination errors, such as partner delay, phase mismatch, insufficient yielding, and interaction conflict. The paper shows that SAI actually improves task success and process-level coupling metrics across several real-world tasks like deformable spreading and rigid-object transport compared to just using independent imitation.
Rosa: And they also touched on the policy architecture itself. They use an ACT style action chunking backbone with temporal history encoding and a cascaded whole-body action head, which helps stabilize primitive execution by conditioning arm commands on the predicted base motion rather than having an independent flat head.
Dev: That architecture actually shows some specific wins, too. The authors found that the temporal history encoding reduces premature release in tasks like Laundry Collection from sixty-four point five percent down to sixteen point one percent because the policy can distinguish intermediate transport states from terminal release states.
Conclusion: Rosa: So to wrap up, this paper shows that Sequential Asymmetric Imitation successfully learns physically coupled dual-robot manipulation without needing synchronized dual-operator demonstrations or explicit inter-robot communication. The staged curriculum is effective because it exposes policies to those realistic coordination errors through the different stages.
Dev: It suggests that coordinated multi-robot behavior can emerge from structured imitation data, without needing reinforcement learning, explicit inter-robot communication, or dense synchronized demonstrations for every single interaction.
Rosa: I think the most important sentence is that Sequential Asymmetric Imitation trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures.
Dev: Taro, what’s your take on these results? Does this work outside of the lab seem practical for real-world deployment?
Taro: Well, it seems like they assume each robot can learn useful standalone primitives and that failures can be corrected through meaningful single-agent interventions. The main limitation they flag is failure-mode coverage, because the Stage three intervention set is only about thirty percent of the baseline dataset size, meaning recovery hasn't fully saturated yet.
Rosa: That makes sense for real-world deployment concerns about robustness when things go seriously wrong. What does this mean for our next steps?
Taro: They suggest that more targeted interventions around rare delays and near-failure contact states might actually improve robustness, and they also note that the current A centric design might not fit tasks where both robots need simultaneous correction.
Dev: So we’re left with a method that learns coupling through data structure rather than direct communication or reinforcement learning, but it still has some gaps in covering every possible failure mode. That's what we’ve got for now on this paper, "Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies."
Yincong Chen, Ranpeng Qiu, Zihao Li, Yanan Zhou
Zhejiang University of Technology · University of Sydney
cs.RO
Submitted: 2026-06-15
Updated: 2026-11-13
Project page: http://cyc0429.github.io/sai-project-page
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects This is difficult because failures often arise…
Key concepts
- Sequential Asymmetric Imitation (SAI)
- A training strategy where Robot A learns from a single demonstration first, then Robot B learns against the resulting policy of Robot A, and finally, both are refined using sparse human interventions near coordination errors. This decomposes complex coordination into three asymmetric stages.
- Partner-masking Augmentation
- A technique used in Stage 1 where the visual identity of a compliant human partner is masked or replaced with randomized elements like Gaussian blur or Stable Diffusion Inpainting. This forces Robot A to focus on the actual task object and state rather than relying on the specific appearance of its supporting partner.
- Data Curriculum
- The structured sequence of training data used in SAI, moving from unilateral demonstrations (Robot A) to paired demonstration (Robot B|A), and finally to sparse interventions near failures. This curriculum is designed to systematically expose policies to increasingly realistic coordination challenges.
- Temporal History Encoding
- An architecture component using a GRU network with a history window of 30 steps to encode the sequence of past states. This allows the policy to distinguish between intermediate transport phases and final release states, which helps stabilize primitive execution and reduce premature actions like releasing an object too early.
Terminology
Summary
Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning We propose Sequential Asymmetric Imitation (SAI), a single-teleoperator curriculum for learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures. p. 0 Robots that Collaborate: Sequential Asymmetric
Imitation for Learning Coupled Robot Policies
Yincong Chen Ranpeng Qiu Zihao Li Yanan Zhou
Guoqiang Ren Weiming Zhi
Abstract: Collaborative mobile manipulation requires robots to coordinate with a
partially observed partner while physically interacting through shared objects. This
is difficult because failures often arise not from poor local skills, but from mistimed
waiting, yielding, pulling, releasing, or repositioning.Where the paper says it
The Core Problem Addressed
Dual-robot manipulation success requires more than two locally competent policies because each robot's motion changes the load, contact state, and feasible motion of the other Small timing errors can therefore produce large interaction forces, object deformation, loss of grasp, or task failure. Independent policies often fail because each robot follows its own nominal motion phase, even when the partner is delayed, stalled, or out of position. p. 0 The robots must align task phases, adapt to
partner delays, yield under interaction conflict, and resume coordinated motion after perturbations. Independent policies often fail because each robot follows its own nominal motion phase, even when
the partner is delayed, stalled, or out of position. • Partner-contingent yielding: each robot must slow, pause, or re-time its motion when the partner
is delayed, stalled, obstructed, or out of phase. • Interaction conflict mitigation: the joint behavior must avoid excessive internal forces, object
deformation, opposing motions, and workspace collisions.Where the paper says it
Where the paper says it
How the Training Signal is Built
SAI decomposes coordination learning into three asymmetric stages. The training follows the data-collection sequence Dsolo A → Ddemo BA → Dint A. p. 3 SAI follows the data-collection sequence
D
solo
A → Ddemo
B|A → Dint
A . (2)
Here, Dsolo
A contains unilateral demonstrations for Robot A, Ddemo
B|A
contains Robot-B demonstrations
collected while Robot A executes its learned policy, and Dint
A contains corrective Robot-A interventions collected during joint deployment.Where the paper says it
-
Stage 1 involves Robot A learning basic task execution from unilateral demonstrations with a compliant human or passive partner. p. 1
Partner-masking augmentation is applied to encourage Robot A to focus on task-relevant object state rather than the visual identity of the supporting partner. p. 4Where the paper says it
SAI decomposes coordination learning into three asymmetric stages. First, Robot A learns basic task execution from unilateral demonstrations with a compliant human or passive partner.
Where the paper says it
We therefore apply partner-masking augmentation to frames in which the supporting partner is visible: the partner region is replaced with randomized appearance perturbations, such as color jitter, blur, noise, or texture changes, with details in Appendix B. This encourages Robot A to focus on taskrelevant object state rather than the visual identity of the supporting partner.
-
Stage 2 deploys Robot A as a frozen policy while a single operator teleoperates Robot B, producing Ddemo B|A. This allows Robot B to learn against the timing, motion profile, and errors of its future robot partner. p. 1
Where the paper says it
First, Robot A learns basic task execution from unilateral demonstrations with a compliant human or passive partner. Second, Robot A is deployed as a frozen policy while a single operator teleoperates Robot B, allowing Robot B to learn against the timing, motion profile, and errors of its future robot partner.
-
Stage 3 involves deploying both policies and refining Robot A using sparse human interventions near coordination failures. p. 0
This intervention process is DAgger-style, where corrective labels are provided only near states where coordination begins to fail. p. 4Where the paper says it
SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures. This staged process exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch, insufficient yielding, and interaction conflict.
Where the paper says it
To correct these failures, we deploy π (1) A and π (2) B together and collect sparse human interventions on Robot A. The intervention process is DAgger-style: the learned policies induce the closed-loop states, and the operator provides corrective labels only near states where coordination begins to fail.
Where the Gains Come From
The authors hypothesize that inter-robot coordination can be induced by structuring the partner distribution seen during imitation learning. The training shifts from compliant support to a deployed learned partner, and finally to closed-loop robot-robot execution with targeted corrections. p. 1 Our key hypothesis is that inter-robot coordination can be induced by structuring the partner distribution seen during imitation learning. SAI shifts training from compliant support, to a deployed learned
partner, to closed-loop robot-robot execution with targeted corrections. SAI shifts training from compliant support, to a deployed learned
partner, to closed-loop robot-robot execution with targeted corrections. This exposes the policies
to realistic coordination errors, including partner delay, phase mismatch, insufficient yielding, and
interaction conflict, enabling coordination through local observations and shared physical interaction
without explicit communication or centralized action prediction. Table 1: Per-task evaluation. SAI improves task success
and process-level coupling metrics across deformable spreading, shared-workspace collection, and rigid-object transport.Where the paper says it
Where the paper says it
Where the paper says it
Policy Architecture and Ablations
The policy instantiation uses an ACT-style action chunking backbone with temporal history encoding and a cascaded whole-body action head. This architecture includes a history window of H = 30 steps, using a single-layer GRU to encode the final hidden state as the history token zt hist. p. 5 To provide temporal context for coordination decisions, we maintain a history window of H = 30
steps and select 12 frames using logarithmic temporal sampling. Multi-view visual features and
full-body state features are fused and encoded by a single-layer GRU, whose final hidden state
forms the history token. 7%
Table 2 and Fig. 5 show that temporal history reduces premature release in Laundry Collection from 64. 5 show that temporal history reduces premature release in Laundry Collection from 64. 1%,
because the policy can distinguish intermediate transport
states from terminal release states. Without Cascaded Head With Cascaded Head
Without History Token With Hitory Token
(a) Early release
Without Cascaded Head With Cascaded Head
Without History Token With Hitory Token
(b) Early lowering
Figure 5: Policy ablations on execution failures. The history token reduces premature release, while the
cascaded base-arm head reduces premature lowering by improving whole-body action consistency. The cascaded base-arm
head reduces premature lowering from 51. 7%, because arm commands are conditioned on the predicted base
motion rather than produced by an independent flat head. These components improve local execution reliability, but they do not explain the cooperative gains
by themselves: the baseline pipelines in Table 1 use the same architecture and still fail more often
under partner delay and phase mismatch. Thus, the architecture stabilizes primitive execution, while
SAI induces partner-responsive coordination through the data curriculum.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Validation and Generalizability
SAI validates its effectiveness across four real-world dual-robot manipulation tasks: Bedthrow Spreading, Tablecloth Spreading, Laundry Collection, and Painting Transport. The method improves task success from 33. p. 5 SAI improves task success
and process-level coupling metrics across deformable spreading, shared-workspace collection, and rigid-object transport. Task Method Success ↑ Phase Sync. To test backbone compatibility,
we instantiate the same curriculum with
two imitation-learning backbones, Action
Chunking Transformer (ACT) and Diffusion Policy, while keeping the sensorimotor
interface and task protocol fixed. We evaluate this comparison on Painting Transport, a contact-rich task that requires synchronized grasping,
lifting, transport, and placement. 9%
SAI operates at the data-curriculum level
and is not tied to a particular action decoder. To test backbone compatibility,
we instantiate the same curriculum with
two imitation-learning backbones, Action
Chunking Transformer (ACT) and Diffusion Policy, while keeping the sensorimotor
interface and task protocol fixed. With Diffusion Policy, SAI also improves all
three metrics, including a large gain in Yield/Wait from 30. These results support the
interpretation that SAI is a data-curriculum contribution rather than an artifact of a particular action
decoder.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations
The authors note several limitations, including the assumption that each robot can learn useful standalone primitives and that failures can be corrected through meaningful single-agent interventions. The main limitation is failure-mode coverage, as the Stage-3 intervention set is only about 30% of the baseline dataset size, meaning recovery has not saturated. p. 8 The main limitation is failure-mode coverage: our Stage-3 intervention set
is only about 30% of the baseline dataset size. Although this improves Yield/Wait, recovery has not
saturated. Although this improves Yield/Wait, recovery has not
saturated. More targeted interventions around rare delays and near-failure contact states may improve
robustness, while excessive correction may bias the policy toward unnecessary waiting. Limitations: SAI assumes that each robot can learn useful standalone primitives, that partner progress
is partially observable from local sensory inputs, and that failures can be corrected through meaningful
single-agent interventions. Tasks requiring simultaneous correction of both robots may not fit the
current A-centric design.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
SAI successfully learns physically coupled dual-robot manipulation without requiring synchronized dual-operator demonstrations, reinforcement learning, or explicit inter-robot communication. The staged curriculum effectively exposes policies to realistic coordination errors, enabling coordination through local observations and shared physical interaction without explicit communication or centralized action prediction. p. 1 SAI shifts training from compliant support, to a deployed learned
partner, to closed-loop robot-robot execution with targeted corrections. This exposes the policies
to realistic coordination errors, including partner delay, phase mismatch, insufficient yielding, and
interaction conflict, enabling coordination through local observations and shared physical interaction
without explicit communication or centralized action prediction. This staged process exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch, insufficient
yielding, and interaction conflict. Across real-world dual-robot manipulation tasks,
SAI improves task success, phase synchronization, and partner-contingent yielding over independent imitation and curriculum-ablation baselines. Across deformable spreading, sharedworkspace collection, and rigid-object transport, SAI improves task success, phase synchronization,
and partner-contingent yielding over independent imitation. These results suggest that coordinated
multi-robot behavior can emerge from structured imitation data, without reinforcement learning,
explicit inter-robot communication, or dense synchronized demonstrations.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Sequential Asymmetric Imitation (SAI) enables learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication by decomposing coordination into three asymmetric stages: Robot A learns basic task execution from unilateral demonstrations, Robot B learns against the deployed Robot A policy, and then sparse interventions refine Robot A using
behaviors such as waiting, yielding, slowing, and recovery.
-
The partner-distribution curriculum induces phase alignment and partner-contingent yielding by shifting the training partner from
compliant support
to afrozen learned policy,
which trains Robot B to compensate forRobot A’s timing, motion profile, and residual errors.
-
Targeted loss masking in Stage 3 refines Robot A by applying corrective labels only near coordination failures, allowing it to adapt from a human-supported partner to the autonomous behavior of Robot B by teaching it
to slow down, wait, yield, or recover when the joint system deviates from the nominal phase.
-
The policy architecture can be improved using an ACT-style action chunking backbone with temporal history encoding and a cascaded whole-body action head to improve local execution reliability by conditioning arm and gripper commands on the predicted base motion.
-
Temporal history token encoding, consisting of a GRU processing multi-view visual features and state features, reduces premature release in tasks like Laundry Collection from 64.5% to 16.1% because the policy can
distinguish intermediate transport states from terminal release states.
-
The cascaded base-arm action head improves whole-body action consistency by predicting a mobile-base action and then conditioning the arm command on that predicted base motion, reducing premature lowering from 51.6% to 9.7%.
-
Partner-region masking and randomization during training regularizes the policy against partner appearance shortcuts by replacing the human partner region with
RGB Randomization, Gaussian Blur, or Stable Diffusion Inpainting,
encouraging reliance ontask-relevant scene cues
rather than visual identity.
Abstract
Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects. This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning. We study this problem with two bimanual mobile manipulators coupled through rigid and deformable objects. We propose Sequential Asymmetric Imitation (SAI), a single-teleoperator curriculum for learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication. SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures. This staged process exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch,insufficient yielding, and interaction conflict. Across real-world dual-robot manipulation tasks, SAI improves task success, phase synchronization, and partner-contingent yielding over independent imitation and curriculum-ablation baselines. These results suggest that physically coupled collaboration can be learned through the structure of the imitation curriculum, rather than through synchronized multi-operator demonstrations or explicit coordination mechanisms. More videos on project page:http://cyc0429.github.io/sai-project-page/
Sources
- Constraining Streaming Flow Models for Adapting Learned Robot Trajectory Distributions
- TriPilot-FF: Coordinated Whole-Body Teleoperation with Force Feedback
- DINOv3
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation