Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies

arXiv:2606.16490 · cs.RO · Submitted 2026-06-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Robots that Collaborate".

Dev: Here's the gist of the paper. Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning We propose Sequential Asymmetric Imitation (SAI), a single-teleoperator curriculum for learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures.

Rosa: The Core Problem Addressed Dual-robot manipulation success requires more than two locally competent policies because each robot's motion changes the load, contact state, and feasible motion of the other Small timing errors can therefore produce large interaction forces, object deformation, loss of grasp, or task failure.

Dev: Independent policies often fail because each robot follows its own nominal motion phase, even when the partner is delayed, stalled, or out of position.

Rosa: The key coordination challenges include temporal phase coupling, partner-contingent yielding, and interaction conflict mitigation.

Title and authors: Rosa: So we're talking about this paper now, "Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies." The title itself suggests they’re tackling how robots can work together when they have to interact physically through shared objects while one of them is partially observed.

Dev: Right, and the authors are Rosa and Dev. They're focusing on a specific way to learn this collaboration without needing two operators guiding both robots at the same time or any direct communication between them.

Rosa: Exactly. The implication here is that we don't need those synchronized demonstrations anymore to teach robots how to handle those tricky coordination issues, like waiting or yielding when the partner moves unexpectedly.

Dev: It sounds like they’re proposing a staged learning process where Robot A gets started first from a single human demonstration, then Robot B learns against that learned behavior before the system is refined with targeted corrections.

The paper's summary: Rosa: Basically, the core idea of this "Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies" paper is to learn coupled multi-robot behaviors without needing synchronized dual-operator demonstrations or explicit inter-robot communication. They propose something called Sequential Asymmetric Imitation, or SAI.

Dev: It breaks the coordination learning down into three asymmetric stages: first, Robot A learns basic task execution from unilateral demonstrations with a compliant human partner; then, Robot B learns against the deployed Robot A policy; and finally, they refine Robot A using sparse interventions near coordination failures.

Rosa: That's what I mean. The paper says this staged process is key because it exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch, insufficient yielding, and interaction conflict.

Dev: And the main question they’re trying to answer is whether coupled dual-robot policies can actually be learned from single-teleoperator demonstrations without having that synchronized dual operator data or reinforcement learning or explicit inter-robot communication.

The paper's improvements: Rosa: Now, what they suggest as improvements is this entire curriculum structure itself. They hypothesize that inducing inter-robot coordination can happen by structuring the partner distribution seen during imitation learning. It shifts the training from just compliant support to a deployed learned partner, and finally to closed-loop robot-robot execution with targeted corrections.

Dev: That shift exposes the policies to realistic coordination errors, such as partner delay, phase mismatch, insufficient yielding, and interaction conflict. The paper shows that SAI actually improves task success and process-level coupling metrics across several real-world tasks like deformable spreading and rigid-object transport compared to just using independent imitation.

Rosa: And they also touched on the policy architecture itself. They use an ACT style action chunking backbone with temporal history encoding and a cascaded whole-body action head, which helps stabilize primitive execution by conditioning arm commands on the predicted base motion rather than having an independent flat head.

Dev: That architecture actually shows some specific wins, too. The authors found that the temporal history encoding reduces premature release in tasks like Laundry Collection from sixty-four point five percent down to sixteen point one percent because the policy can distinguish intermediate transport states from terminal release states.

Conclusion: Rosa: So to wrap up, this paper shows that Sequential Asymmetric Imitation successfully learns physically coupled dual-robot manipulation without needing synchronized dual-operator demonstrations or explicit inter-robot communication. The staged curriculum is effective because it exposes policies to those realistic coordination errors through the different stages.

Dev: It suggests that coordinated multi-robot behavior can emerge from structured imitation data, without needing reinforcement learning, explicit inter-robot communication, or dense synchronized demonstrations for every single interaction.

Rosa: I think the most important sentence is that Sequential Asymmetric Imitation trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures.

Dev: Taro, what’s your take on these results? Does this work outside of the lab seem practical for real-world deployment?

Taro: Well, it seems like they assume each robot can learn useful standalone primitives and that failures can be corrected through meaningful single-agent interventions. The main limitation they flag is failure-mode coverage, because the Stage three intervention set is only about thirty percent of the baseline dataset size, meaning recovery hasn't fully saturated yet.

Rosa: That makes sense for real-world deployment concerns about robustness when things go seriously wrong. What does this mean for our next steps?

Taro: They suggest that more targeted interventions around rare delays and near-failure contact states might actually improve robustness, and they also note that the current A centric design might not fit tasks where both robots need simultaneous correction.

Dev: So we’re left with a method that learns coupling through data structure rather than direct communication or reinforcement learning, but it still has some gaps in covering every possible failure mode. That's what we’ve got for now on this paper, "Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies."

Yincong Chen, Ranpeng Qiu, Zihao Li, Yanan Zhou

Zhejiang University of Technology · University of Sydney

cs.RO

Submitted: 2026-06-15

Updated: 2026-11-13

Project page: http://cyc0429.github.io/sai-project-page

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 98/100

The gist: Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects This is difficult because failures often arise…

Key concepts

Sequential Asymmetric Imitation (SAI)
A training strategy where Robot A learns from a single demonstration first, then Robot B learns against the resulting policy of Robot A, and finally, both are refined using sparse human interventions near coordination errors. This decomposes complex coordination into three asymmetric stages.
Partner-masking Augmentation
A technique used in Stage 1 where the visual identity of a compliant human partner is masked or replaced with randomized elements like Gaussian blur or Stable Diffusion Inpainting. This forces Robot A to focus on the actual task object and state rather than relying on the specific appearance of its supporting partner.
Data Curriculum
The structured sequence of training data used in SAI, moving from unilateral demonstrations (Robot A) to paired demonstration (Robot B|A), and finally to sparse interventions near failures. This curriculum is designed to systematically expose policies to increasingly realistic coordination challenges.
Temporal History Encoding
An architecture component using a GRU network with a history window of 30 steps to encode the sequence of past states. This allows the policy to distinguish between intermediate transport phases and final release states, which helps stabilize primitive execution and reduce premature actions like releasing an object too early.

Terminology

Summary

Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning We propose Sequential Asymmetric Imitation (SAI), a single-teleoperator curriculum for learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures. p. 0

Where the paper says it

Robots that Collaborate: Sequential Asymmetric Imitation for Learning Coupled Robot Policies Yincong Chen Ranpeng Qiu Zihao Li Yanan Zhou Guoqiang Ren Weiming Zhi Abstract: Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects. This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning.

The Core Problem Addressed

Dual-robot manipulation success requires more than two locally competent policies because each robot's motion changes the load, contact state, and feasible motion of the other Small timing errors can therefore produce large interaction forces, object deformation, loss of grasp, or task failure. Independent policies often fail because each robot follows its own nominal motion phase, even when the partner is delayed, stalled, or out of position. p. 0

Where the paper says it

The robots must align task phases, adapt to partner delays, yield under interaction conflict, and resume coordinated motion after perturbations. Independent policies often fail because each robot follows its own nominal motion phase, even when the partner is delayed, stalled, or out of position.

The key coordination challenges include temporal phase coupling, partner-contingent yielding, and interaction conflict mitigation. p. 2
Where the paper says it

• Partner-contingent yielding: each robot must slow, pause, or re-time its motion when the partner is delayed, stalled, obstructed, or out of phase. • Interaction conflict mitigation: the joint behavior must avoid excessive internal forces, object deformation, opposing motions, and workspace collisions.

How the Training Signal is Built

SAI decomposes coordination learning into three asymmetric stages. The training follows the data-collection sequence Dsolo A → Ddemo BA → Dint A. p. 3

Where the paper says it

SAI follows the data-collection sequence D solo A → Ddemo B|A → Dint A . (2) Here, Dsolo A contains unilateral demonstrations for Robot A, Ddemo B|A contains Robot-B demonstrations collected while Robot A executes its learned policy, and Dint A contains corrective Robot-A interventions collected during joint deployment.

  1. Stage 1 involves Robot A learning basic task execution from unilateral demonstrations with a compliant human or passive partner. p. 1

    Where the paper says it

    SAI decomposes coordination learning into three asymmetric stages. First, Robot A learns basic task execution from unilateral demonstrations with a compliant human or passive partner.

    Partner-masking augmentation is applied to encourage Robot A to focus on task-relevant object state rather than the visual identity of the supporting partner. p. 4
    Where the paper says it

    We therefore apply partner-masking augmentation to frames in which the supporting partner is visible: the partner region is replaced with randomized appearance perturbations, such as color jitter, blur, noise, or texture changes, with details in Appendix B. This encourages Robot A to focus on taskrelevant object state rather than the visual identity of the supporting partner.

  2. Stage 2 deploys Robot A as a frozen policy while a single operator teleoperates Robot B, producing Ddemo B|A. This allows Robot B to learn against the timing, motion profile, and errors of its future robot partner. p. 1

    Where the paper says it

    First, Robot A learns basic task execution from unilateral demonstrations with a compliant human or passive partner. Second, Robot A is deployed as a frozen policy while a single operator teleoperates Robot B, allowing Robot B to learn against the timing, motion profile, and errors of its future robot partner.

  3. Stage 3 involves deploying both policies and refining Robot A using sparse human interventions near coordination failures. p. 0

    Where the paper says it

    SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures. This staged process exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch, insufficient yielding, and interaction conflict.

    This intervention process is DAgger-style, where corrective labels are provided only near states where coordination begins to fail. p. 4
    Where the paper says it

    To correct these failures, we deploy π (1) A and π (2) B together and collect sparse human interventions on Robot A. The intervention process is DAgger-style: the learned policies induce the closed-loop states, and the operator provides corrective labels only near states where coordination begins to fail.

Where the Gains Come From

The authors hypothesize that inter-robot coordination can be induced by structuring the partner distribution seen during imitation learning. The training shifts from compliant support to a deployed learned partner, and finally to closed-loop robot-robot execution with targeted corrections. p. 1

Where the paper says it

Our key hypothesis is that inter-robot coordination can be induced by structuring the partner distribution seen during imitation learning. SAI shifts training from compliant support, to a deployed learned partner, to closed-loop robot-robot execution with targeted corrections.

This exposes policies to realistic coordination errors, including partner delay, phase mismatch, insufficient yielding, and interaction conflict. p. 1
Where the paper says it

SAI shifts training from compliant support, to a deployed learned partner, to closed-loop robot-robot execution with targeted corrections. This exposes the policies to realistic coordination errors, including partner delay, phase mismatch, insufficient yielding, and interaction conflict, enabling coordination through local observations and shared physical interaction without explicit communication or centralized action prediction.

Experimental results show SAI improves task success and process-level coupling metrics across deformable spreading, shared-workspace collection, and rigid-object transport over independent imitation. p. 5
Where the paper says it

Table 1: Per-task evaluation. SAI improves task success and process-level coupling metrics across deformable spreading, shared-workspace collection, and rigid-object transport.

Policy Architecture and Ablations

The policy instantiation uses an ACT-style action chunking backbone with temporal history encoding and a cascaded whole-body action head. This architecture includes a history window of H = 30 steps, using a single-layer GRU to encode the final hidden state as the history token zt hist. p. 5

Where the paper says it

To provide temporal context for coordination decisions, we maintain a history window of H = 30 steps and select 12 frames using logarithmic temporal sampling. Multi-view visual features and full-body state features are fused and encoded by a single-layer GRU, whose final hidden state forms the history token.

Ablation studies on this architecture show that the temporal history reduces premature release in Laundry Collection from 64. p. 7
Where the paper says it

7% Table 2 and Fig. 5 show that temporal history reduces premature release in Laundry Collection from 64.

5% to 16.1%, because the policy can distinguish intermediate transport states from terminal release states. p. 7
Where the paper says it

5 show that temporal history reduces premature release in Laundry Collection from 64. 1%, because the policy can distinguish intermediate transport states from terminal release states.

Similarly, the cascaded base-arm head reduces premature lowering from 51. p. 7
Where the paper says it

Without Cascaded Head With Cascaded Head Without History Token With Hitory Token (a) Early release Without Cascaded Head With Cascaded Head Without History Token With Hitory Token (b) Early lowering Figure 5: Policy ablations on execution failures. The history token reduces premature release, while the cascaded base-arm head reduces premature lowering by improving whole-body action consistency.

6% to 9.7%, because arm commands are conditioned on the predicted base motion rather than produced by an independent flat head. p. 7
Where the paper says it

The cascaded base-arm head reduces premature lowering from 51. 7%, because arm commands are conditioned on the predicted base motion rather than produced by an independent flat head.

These architectural improvements stabilize primitive execution, while SAI induces partner-responsive coordination through the data curriculum itself. p. 7
Where the paper says it

These components improve local execution reliability, but they do not explain the cooperative gains by themselves: the baseline pipelines in Table 1 use the same architecture and still fail more often under partner delay and phase mismatch. Thus, the architecture stabilizes primitive execution, while SAI induces partner-responsive coordination through the data curriculum.

Validation and Generalizability

SAI validates its effectiveness across four real-world dual-robot manipulation tasks: Bedthrow Spreading, Tablecloth Spreading, Laundry Collection, and Painting Transport. The method improves task success from 33. p. 5

Where the paper says it

SAI improves task success and process-level coupling metrics across deformable spreading, shared-workspace collection, and rigid-object transport. Task Method Success ↑ Phase Sync.

3% to 56.7% for the ACT backbone on Painting Transport. p. 7
Where the paper says it

To test backbone compatibility, we instantiate the same curriculum with two imitation-learning backbones, Action Chunking Transformer (ACT) and Diffusion Policy, while keeping the sensorimotor interface and task protocol fixed. We evaluate this comparison on Painting Transport, a contact-rich task that requires synchronized grasping, lifting, transport, and placement.

Furthermore, SAI operates at the data-curriculum level and is not tied to a particular action decoder, as it improves over Independent Imitation for both ACT and Diffusion Policy backbones. p. 7
Where the paper says it

9% SAI operates at the data-curriculum level and is not tied to a particular action decoder. To test backbone compatibility, we instantiate the same curriculum with two imitation-learning backbones, Action Chunking Transformer (ACT) and Diffusion Policy, while keeping the sensorimotor interface and task protocol fixed.

The results support the interpretation that SAI is a data-curriculum contribution rather than an artifact of a particular action decoder. p. 7
Where the paper says it

With Diffusion Policy, SAI also improves all three metrics, including a large gain in Yield/Wait from 30. These results support the interpretation that SAI is a data-curriculum contribution rather than an artifact of a particular action decoder.

Limitations

The authors note several limitations, including the assumption that each robot can learn useful standalone primitives and that failures can be corrected through meaningful single-agent interventions. The main limitation is failure-mode coverage, as the Stage-3 intervention set is only about 30% of the baseline dataset size, meaning recovery has not saturated. p. 8

Where the paper says it

The main limitation is failure-mode coverage: our Stage-3 intervention set is only about 30% of the baseline dataset size. Although this improves Yield/Wait, recovery has not saturated.

More targeted interventions around rare delays and near-failure contact states may improve robustness. p. 8
Where the paper says it

Although this improves Yield/Wait, recovery has not saturated. More targeted interventions around rare delays and near-failure contact states may improve robustness, while excessive correction may bias the policy toward unnecessary waiting.

The current A-centric design may not fit tasks requiring simultaneous correction of both robots. p. 8
Where the paper says it

Limitations: SAI assumes that each robot can learn useful standalone primitives, that partner progress is partially observable from local sensory inputs, and that failures can be corrected through meaningful single-agent interventions. Tasks requiring simultaneous correction of both robots may not fit the current A-centric design.

Conclusion

SAI successfully learns physically coupled dual-robot manipulation without requiring synchronized dual-operator demonstrations, reinforcement learning, or explicit inter-robot communication. The staged curriculum effectively exposes policies to realistic coordination errors, enabling coordination through local observations and shared physical interaction without explicit communication or centralized action prediction. p. 1

Where the paper says it

SAI shifts training from compliant support, to a deployed learned partner, to closed-loop robot-robot execution with targeted corrections. This exposes the policies to realistic coordination errors, including partner delay, phase mismatch, insufficient yielding, and interaction conflict, enabling coordination through local observations and shared physical interaction without explicit communication or centralized action prediction.

SAI improves task success, phase synchronization, and partner-contingent yielding over independent imitation baselines across various contact-rich real-world tasks. p. 0
Where the paper says it

This staged process exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch, insufficient yielding, and interaction conflict. Across real-world dual-robot manipulation tasks, SAI improves task success, phase synchronization, and partner-contingent yielding over independent imitation and curriculum-ablation baselines.

The findings suggest that coordinated multi-robot behavior can emerge from structured imitation data, without reinforcement learning, explicit inter-robot communication, or dense synchronized demonstrations. p. 8
Where the paper says it

Across deformable spreading, sharedworkspace collection, and rigid-object transport, SAI improves task success, phase synchronization, and partner-contingent yielding over independent imitation. These results suggest that coordinated multi-robot behavior can emerge from structured imitation data, without reinforcement learning, explicit inter-robot communication, or dense synchronized demonstrations.

The single most informative sentence about the paper's subject and finding is: Sequential Asymmetric Imitation (SAI) trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Sequential Asymmetric Imitation (SAI) enables learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication by decomposing coordination into three asymmetric stages: Robot A learns basic task execution from unilateral demonstrations, Robot B learns against the deployed Robot A policy, and then sparse interventions refine Robot A using behaviors such as waiting, yielding, slowing, and recovery.

  2. The partner-distribution curriculum induces phase alignment and partner-contingent yielding by shifting the training partner from compliant support to a frozen learned policy, which trains Robot B to compensate for Robot A’s timing, motion profile, and residual errors.

  3. Targeted loss masking in Stage 3 refines Robot A by applying corrective labels only near coordination failures, allowing it to adapt from a human-supported partner to the autonomous behavior of Robot B by teaching it to slow down, wait, yield, or recover when the joint system deviates from the nominal phase.

  4. The policy architecture can be improved using an ACT-style action chunking backbone with temporal history encoding and a cascaded whole-body action head to improve local execution reliability by conditioning arm and gripper commands on the predicted base motion.

  5. Temporal history token encoding, consisting of a GRU processing multi-view visual features and state features, reduces premature release in tasks like Laundry Collection from 64.5% to 16.1% because the policy can distinguish intermediate transport states from terminal release states.

  6. The cascaded base-arm action head improves whole-body action consistency by predicting a mobile-base action and then conditioning the arm command on that predicted base motion, reducing premature lowering from 51.6% to 9.7%.

  7. Partner-region masking and randomization during training regularizes the policy against partner appearance shortcuts by replacing the human partner region with RGB Randomization, Gaussian Blur, or Stable Diffusion Inpainting, encouraging reliance on task-relevant scene cues rather than visual identity.

Abstract

Collaborative mobile manipulation requires robots to coordinate with a partially observed partner while physically interacting through shared objects. This is difficult because failures often arise not from poor local skills, but from mistimed waiting, yielding, pulling, releasing, or repositioning. We study this problem with two bimanual mobile manipulators coupled through rigid and deformable objects. We propose Sequential Asymmetric Imitation (SAI), a single-teleoperator curriculum for learning coupled multi-robot behaviors without synchronized dual-operator demonstrations or explicit inter-robot communication. SAI trains Robot A from unilateral demonstrations with a compliant human partner, trains Robot B against the deployed Robot A policy, and then refines Robot A using sparse interventions near coordination failures. This staged process exposes the policies to increasingly realistic partner behaviors, including delay, phase mismatch,insufficient yielding, and interaction conflict. Across real-world dual-robot manipulation tasks, SAI improves task success, phase synchronization, and partner-contingent yielding over independent imitation and curriculum-ablation baselines. These results suggest that physically coupled collaboration can be learned through the structure of the imitation curriculum, rather than through synchronized multi-operator demonstrations or explicit coordination mechanisms. More videos on project page:http://cyc0429.github.io/sai-project-page/

Sources

Related papers