Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection

arXiv:2610.10601 · cs.RO · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Teaching a Robot Dog New Tricks".

Rosa: The gist This work proposes a three-stage method that trains a single policy to perform distinct tasks such as walking, digging, and hopping,

Dev: First, who's behind it and why it matters.

Title and authors: Dev: Moving on to the structure of this work, the paper, "Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection," lays out how they tackle this multi-task problem step by step.

Rosa: They start by setting up the problem as an MDP where a single policy has to handle various skills, and they define a "single task" as something with its own dedicated teacher policy, and a "composite task" as requiring several tasks to run together.

Taro: So the core idea is separating the skill acquisition from the composition phase, which seems like a necessary first step before you try to make it all work at once.

Dev: Right. The paper describes training multiple teacher policies using reinforcement learning on those narrow tasks first, and then a student policy learns by distilling knowledge from these teachers.

Rosa: And they introduce this combined objective that balances the PPO clipped surrogate loss with a BC loss to drive both performance and imitation simultaneously during the student training phase.

Taro: That balancing act sounds like it’s designed to keep the robot from just ignoring one task while trying to master another, which is a big challenge in multi-task scenarios.

Dev: And then they add this adversarial task selection process that picks the worst-performing tasks to focus training on next, which is meant to correct for any systematic bias in sampling.

Rosa: It seems like the whole structure of teaching a robot dog new tricks comes down to this careful orchestration between learning individual skills and figuring out how to combine them effectively.

Taro: And they show that this method helps them learn these composite tasks, such as walking at a low height when combining leg lift and body height tracking.

The paper's summary: Rosa: To summarize what they found in "Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection," they are proposing a three-stage method to train one policy for many skills, like walking, digging, and hopping.

Dev: The summary highlights that the first stage involves training multiple teacher policies using RL on those specific tasks in isolation. This allows them to create reference motions for the main student policy to imitate later.

Taro: Then comes the second stage where the student policy learns with a multi-teacher distillation setup using that combined RL and imitation learning objective we talked about, plus that adversarial task selection mechanism.

Rosa: The key finding they report is that this method preserves motion quality and tracks commands better than PPO and DtF baselines while showing minimal effects of reward gaming during evaluation.

Dev: They also emphasize the generalization aspect, noting that these policies can meaningfully compose three or more tasks despite never seeing those specific compositions during the training process itself.

Taro: That’s a big deal for real-world use because it suggests the robot can handle situations it hasn't been explicitly shown in training.

Rosa: It really points toward a system that learns to be flexible, not just good at one thing you teach it directly.

The paper's improvements: Dev: So when we look at what the authors suggest for improvement in this work, they focus on how to make the process more robust and less prone to failure during training.

Rosa: They highlight that they retained a nonzero imitation weight throughout the entire training process so that the BC term keeps anchoring motion quality, which is different from some other approaches.

Taro: That means they are actively fighting against style-rewards or adversarial priors by keeping that imitation component strong to maintain naturalistic behavior.

Dev: They also mention using DAgger-style online data collection where teachers are queried under the student’s own state distribution rather than relying on a fixed offline dataset, which helps address distributional shift during online learning.

Rosa: And they specifically point out that their critic pretraining stage occurs before actor optimization in the reinforcement learning primary stage to prevent instability when starting with low imitation weights.

Taro: That critic pretraining step is crucial because it trains the value estimator against the full task distribution before the policy starts making its first big moves, which prevents convergence issues.

Dev: The paper also notes that their reward landscape during reinforcement learning needs to be expanded to specifically target minimizing tracking errors and encouraging new task learning for composition.

Rosa: They also mention that adversarial motion priors, or AMP, are an alternative approach that replaces style-rewards with a discriminator trained on reference motion, which they say has been successfully demonstrated in quadruped locomotion.

Conclusion: Dev: To wrap up this discussion on "Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection," the main implication is that we can train single policies to handle a wide array of distinct tasks and then compose them into novel behaviors.

Rosa: This paper demonstrates how combining reinforcement learning, imitation learning, and adversarial task selection can help build robots capable of performing diverse physical actions in real-world scenarios without needing custom policies for every single situation.

Taro: It’s about moving toward systems that are more flexible because they learn to combine known skills into new ones on the fly instead of being explicitly programmed for every possible combination.

Dev: And one thing they point out is the explicit and implicit performance scores they provide for each task, which lets researchers tell if a robot is tracking a target correctly or if it’s just moving smoothly.

Rosa: Ultimately, this work provides a solid framework for tackling the problem of multi-task learning in embodied AI by focusing on how to manage the trade-off between learning new skills and maintaining high motion quality.

Taro: I think the way they structured the training stages makes it clear that composition is achievable even when you’re dealing with complex, conflicting objectives.

Dev: Yeah, they give us a concrete method for building those composite behaviors from individual skill policies.

Rosa: That’s it for this paper today. Next time we'll be looking at some papers on how AI models are handling visual actions and what bottlenecks that creates in the learning process.

University of California at Santa Cruz

cs.RO

Submitted: 2026-10-06

Updated: 2026-10-06

Comments: 12 pages, 5 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist This work proposes a three-stage method that trains a single policy to perform distinct tasks such as walking, digging, and hopping, and compose them into novel behaviors such as crawling.

Key concepts

Teacher Policies: These are separate policies trained using reinforcement learning for narrowly defined skills (e.g., walking or digging). They generate reference motions that the main student policy will imitate, providing smooth and natural movement guidance.
Student Policy: This is the single policy being trained to perform all tasks. It learns by combining two objectives: imitation learning (to mimic teacher actions) and reinforcement learning (to optimize for task rewards). It is conditioned on the environment state and shared command targets.
Combined Training Objective: The student policy optimizes a combined loss function that mixes a PPO clipped objective (for RL) and a BC loss (for imitation learning). This allows the robot to learn both good control from reinforcement signals and accurate motion tracking from expert demonstrations.
Adversarial Task Selection: This process involves selecting training examples or tasks that are the worst-performing for the current policy. By focusing training on these challenging scenarios, the method ensures faster learning and better generalization across diverse skills.

Terminology

Distillation — 12× in this paper · explained in 2 episodes
Reinforcement Learning — 9× in this paper · explained in 4 episodes
Imitation Learning — 7× in this paper
Generalization — 6× in this paper · explained in 13 episodes
Robustness — 5× in this paper · explained in 5 episodes
Actor — 4× in this paper · explained in 3 episodes
Adversarial Motion Priors — 3× in this paper
Ablation — 2× in this paper · explained in 1 episode
Catastrophic Forgetting — 2× in this paper · explained in 5 episodes
Deep Reinforcement Learning — 2× in this paper · explained in 2 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Imitation Learning (IL) — 2× in this paper · explained in 2 episodes
Mixture of Experts — 2× in this paper · explained in 2 episodes
Reward Function — 2× in this paper · explained in 2 episodes
Action Space — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 3 episodes
Core — 1× in this paper · explained in 2 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Regret — 1× in this paper · explained in 3 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
Systematic Bias — 1× in this paper · explained in 3 episodes

Summary

The gist

This work proposes a three-stage method that trains a single policy to perform distinct tasks such as walking, digging, and hopping, and compose them into novel behaviors such as crawling.

Problem Definition

The problem is modeled as a Markov Decision Process (MDP) defined by the tuple (S, A, T, R, γ). A “single task” is defined as a narrowly-defined behavior for which a dedicated teacher policy exists, and a “composite task” is a task that requires the concurrent execution of two or more single tasks and for which no teacher policy exists. The student policy πθ(a s, cY) is conditioned on the environment state s ∈ S and the command vector cY, which is a set of tracking targets that are shared across Y.

Teacher Policies

For each task xi ∈ X, a corresponding teacher policy π∗i (a s, ci) is trained using proximal policy optimization (PPO) under a reward landscape specific to that task. These teachers serve only to generate reference motion for the student policy to imitate, and are trained on privileged information unavailable to the student, such as foot contact forces, yielding smooth and natural movements while still retaining good command tracking performance. The final teacher policies are frozen prior to student training.

Combined Training Objective

The student is trained using a combined objective consisting of a PPO clipped surrogate objective and a BC loss. The PPO clipped surrogate objective is defined as L PPO(θ) = Et h min rt(θ)Aˆt, clip(rt(θ), 1 − ϵ, 1 + ϵ)Aˆt i. The BC loss is defined as L BC(θ) = E(s,cY,a∗)∼D h∥µθ(s, cY) − a∗∥2 i. By combining both, we obtain the combined objective: L(θ) = L PPO(θ) − λL BC(θ) (3).

Imitation Primary and Reinforcement Primary Stages

Training is split into two stages: an imitation learning primary stage (ILP), and a reinforcement learning primary stage (RLP).

** Stage 1 (Teacher Training) focuses on training multiple teacher policies to perform narrowly defined tasks using RL. During ILP, the student policy is optimized over a combined LPPO − λILLBC objective, with rewards only encouraging body stability and motion smoothness. During RLP, a new critic is first pre-trained on the RLP reward landscape, and the student is optimized over the same combined objective but with a weaker λRL. Here, rewards are extended to also finetune certain behaviors and encourage command tracking. Both stages use adversarial task selection to accumulate more transitions on the worst performing tasks. During RLP, the IL gradients still act as an anchor to teacher policy behaviors so that motion quality is preserved as learning continues. The critic pretraining stage occurs prior to actor optimization in RLP. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The ILP stage uses a high λ coefficient in L(θ) so that BC dominates PPO. The RLP stage uses a low λ coefficient to allow PPO to drive learning. The critic becomes unreliable in value estimation and hinders policy convergence if started directly with low λ at this point. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition. The reward landscape during RLP must be expanded to target minimizing tracking errors and encouraging new task learning for task composition.

Improvements for AI systems

The proposed three-stage training method allows for training a single policy to perform distinct tasks such as walking, digging, and hopping and composing them into novel behaviors like crawling. This enables a single end-to-end policy to perform 22 tasks, demonstrating the capability to learn many distinct tasks from its teachers and composing several tasks together to create new behaviors.
The adversarial task selection strategy corrects systematic bias by allocating more episodes to tasks with the largest performance deficit measured through error channels, ensuring that training effort is focused on improving the most challenging skills. This mechanism is explicitly used to correct for the issue where uniform task sampling in DAgger introduces a systematic bias under multi-task training.
The combined objective, defined as L(θ) = L PPO(θ) − λL BC(θ) with a high initial weight on the imitation component (high λ coefficient in L(θ) so that BC dominates PPO), ensures that the policy preserve[s] motion quality and track[s] commands more accurately than PPO and DtF baselines.
The system can learn to compose tasks, exemplified by a policy trained for body height tracking and omni-directional walking learning to compose the two to create a crawling motion (i.e., walking at a low height). This demonstrates the ability to handle composite tasks by combining single task commands into novel behaviors.
The critic pretraining stage prior to actor optimization in the RLP phase addresses instability by training a new critic against the RLP rewards under the full task distribution, which prevents it from becoming unreliable and hinders convergence, as direct starting updates with low λ at this point causes training instability due to the critic being out of distribution.
The evaluation framework provides detailed metrics, including explicit and implicit performance score[s] for each task, allowing researchers to distinguish between performance on commanded targets versus inherent behavioral quality, as noted by the finding that higher task score does not necessarily indicate that the task is performed correctly.