Chasing Stability: Humanoid Running via Control Lyapunov Function Guided Reinforcement Learning

summary

The gist

Achieving highly dynamic behaviors on humanoid robots, such as running, requires controllers that are both robust and precise…

In short

This research developed a Control Lyapunov Function Guided Reinforcement Learning (CLF-RL) approach for humanoid robots to achieve natural and highly dynamic running behaviors, including flight phases. By embedding stability conditions into the reward function, it eliminates the need for hand-crafted rewards and encourages certifiably stable movement. The method generates full-order trajectories offline, leading to a robust policy capable of tracking desired positions accurately in outdoor environments.

Key concepts

Hybrid Dynamics
Legged locomotion is modeled as a hybrid dynamical system because it switches between different modes, such as single support and flight phases. This means the system's behavior changes abruptly based on which state it is currently in, requiring mathematical tools that handle both continuous motion and discrete transitions.
Control Lyapunov Function (CLF)
A CLF is a mathematical function used to guarantee stability for a control system. In this context, it is incorporated into the reward signal to guide the reinforcement learning agent toward certifiably stable running trajectories, ensuring the robot's movements are predictable and safe.
Multi-domain Trajectory Optimization
This technique is used offline to generate a library of full dynamic reference trajectories for various speeds. By optimizing these paths subject to constraints like state transitions, researchers create principled guidance for the reinforcement learning agent before it starts training.

Terminology used across episodes

This episode discusses

The paper

Chasing Stability: Humanoid Running via Control Lyapunov Function Guided Reinforcement Learning · Read on arXiv

Technology Innovation Institute (TII) · National Science Foundation Graduate Research Fellowship

Achieving highly dynamic behaviors on humanoid robots, such as running, requires controllers that are both robust and precise, and hence difficult to design. Classical control theory offers principled tools for stabilizing such systems but struggles to synthesize real-time controllers for their nonlinear, hybrid dynamics; reinforcement learning (RL) handles these dynamics well but typically depends on extensive hand-tuning of heuristic rewards. In this work, we embed ideas from nonlinear control theory, specifically control Lyapunov functions (CLFs), along with optimized reference trajectories for the hybrid system into the reinforcement learning reward. This approach, CLF-RL, replaces the need to handcraft and tune heuristic reward terms with easy to tune stability informed rewards. We further develop a trajectory optimization routine that can handle the hybrid structure of the system to produce dynamically feasible, multi-domain, trackable references; using full-order dynamics rather than a kinematic approximation yields up to a 1.7x improvement in position tracking. The resulting policy runs on a Unitree G1 humanoid with a full flight phase, operating reliably in tight corridors such as a treadmill as well as in outdoor environments, all while remaining robust to disturbances applied to the torso and feet. Moreover, it achieves accurate global reference tracking utilizing only on-board sensors, making a critical step toward integrating these dynamic motions into a full autonomy stack.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Chasing Stability: Humanoid Running via Control Lyapunov Function Guided Reinforcement Learning".

Dev: Here's the gist of the paper. Achieving highly dynamic behaviors on humanoid robots, such as running, requires controllers that are both robust and precise, and hence difficult to design The resulting policy produces natural and highly dynamic running behaviors with a full flight phase on a humanoid robot This approach, CLF-RL, eliminates the need to handcraft and tune heuristic reward terms, while simultaneously encouraging certifiable stability and providing meaningful intermediate rewards to guide learning System Modeling for Hybrid Dynamics Legged locomotion can be represented as a nonlinear hybrid dynamical system with impulse effects, including both continuous and discrete states The hybrid dynamics of the system are written as x˙ = fD(x) + gD(x) uD x ∈ D without a transition For humanoid running, the system involves multiple domains, specifically the single support phase (SSP) and the flight phase (FLT) The continuous dynamics are derived using equations involving mass-inertia matrix M(q), Coriolis and gravity terms H(q, q˙), actuation matrix B, and Jacobian Jh(q) Transitions between domains are denoted as γ:= (Di, Di+1) ∈ Γ with associated guard and reset maps Trajectory Generation and Reward Synthesis Multi-domain trajectory optimization is used to generate a gait library spanning speeds from 1.1 m/s to 3.0 m/s, including a standing pose This optimization involves formulating the problem as a multipleshooting trajectory optimization, subject to various constraints like state transitions and periodicity Control Lyapunov functions (CLFs) are embedded into the reward function to promote tracking and convergence to steady-state trajectories The CLF tracking reward is defined as rv = wv exp − Vt/σv, where Vt is the value of the CLF based on error η Reinforcement Learning Training Details The reinforcement learning framework seeks to solve the stochastic optimal control problem π∗ = arg max π J(π) where J is the expected reward The total reward R is composed of several terms, including a CLF tracking reward (rv), a CLF decay condition reward (rv˙), and domain rewards (rdom) Domain rewards are broken down into rdom = 1SPPrhol + 1FLTrcon, which encourages the stance foot to remain stationary in SSP and penalizes contact forces in FLT The policy is trained using PPO implementation from Robotics System Lab RL library

34: and is robustified via domain randomization Performance and Robustness Results The learned policy demonstrates strong performance in outdoor environments using only on-board compute and sensors The robot successfully recovers from external disturbances, including impacts from dodge balls to the torso and leg, and wooden planks striking the stance toe and heel The CLF-based tracking reward improves velocity tracking performance, especially for higher speeds compared to a pure tracking reward The policy tracks desired positions and velocities very well, maintaining the magnitude of the y error less than 0.25 m and the yaw error less than 0.

Title and authors: Rosa: So we're talking about this paper today, "Chasing Stability: Humanoid Running via Control Lyapunov Function Guided Reinforcement Learning." It looks like they are tackling the really hard problem of getting humanoid robots to run dynamically, which is tough because you need controllers that are both super robust and incredibly precise.

Dev: Exactly. Classical control methods give us some good ideas on how these systems can stabilize themselves, but actually building a real-time controller for those kinds of nonlinear and hybrid dynamics is still really challenging.

Taro: I always wonder about the boundary between what's feasible in a lab setting versus what happens when the world actually throws a curve at it outside.

Rosa: Well, this paper dives into reinforcement learning, which has gotten popular for locomotion control because it can handle these complex dynamics, and they’re embedding ideas from nonlinear control theory right into the reinforcement learning training process.

Dev: They are specifically using control Lyapunov functions or CLFs to shape the reward signal during training, which is key because this approach aims to eliminate the need for handcrafting those heuristic reward terms that usually cause instability in learning.

Taro: So they're trying to get the robot stable just by how they tell it what success looks like, rather than manually setting up all these complex contact force models?

Rosa: That’s right. They are simultaneously encouraging certifiable stability—meaning you can prove it should be stable—while also giving the learning process meaningful intermediate rewards to guide it toward a good running motion.

Dev: That structure is interesting because it links the desired dynamic behavior directly to a proven stability condition, which hopefully makes the entire learning process much more reliable than just relying on raw experience.

Taro: But what about when things go wrong in the real world? Does this approach handle external disturbances well, or is it just learning on perfect simulated ground?

Rosa: The results show that this policy produces natural and highly dynamic running behaviors with a full flight phase on a humanoid robot, and they tested it using only on-board compute and sensors.

Dev: They even showed that the learned policy demonstrates strong performance in outdoor environments, where the robot successfully recovers from external disturbances like impacts from dodge balls to its torso and leg.

Taro: That's good because robustness is everything; if it can't handle things hitting it while running on a wooden plank, it’s not ready for anything outside a controlled setup.

Rosa: Plus, the CLF-based tracking reward actually improved velocity tracking performance, especially when compared to just using a pure tracking reward.

Dev: That suggests that grounding the learning in those dynamic reference trajectories generated offline through multi-domain trajectory optimization is really paying off in terms of accuracy and speed.

Taro: So they are generating these full-order dynamic reference trajectories beforehand, and then using those as guidance for the reinforcement learning training process via those CLFs?

Title and authors: Rosa: That’s the core idea, yes. They use multi-domain trajectory optimization to create a library of gaits spanning speeds from one point one meters per second up to three meters per second, including a standing pose.

Dev: And they formulate that optimization problem as a multipleshooting trajectory optimization subject to various constraints, including state transitions and periodicity.

Taro: I’m curious about the transition between phases; how precisely does the system handle moving from one contact phase to the next when things get bumpy?

Rosa: The system is modeled as a nonlinear hybrid dynamical system with impulse effects, meaning it has both continuous states and discrete states, specifically the single support phase and the flight phase.

Dev: The continuous dynamics are derived using equations that involve things like the mass inertia matrix, Coriolis and gravity terms, actuation matrix, and Jacobian when you’re dealing with motion.

Taro: And those transitions between domains are defined by guard maps and reset maps, which sounds like they have a very detailed understanding of the robot's physical state changes.

Rosa: They embed Control Lyapunov Functions into the reward function to promote tracking and convergence toward steady-state trajectories, defining that CLF tracking reward as r sub v equals w sub v exp minus V sub t over sigma v.

Dev: That specific form shows how they are using the value of the CLF based on an error η to create a meaningful intermediate reward signal for the learning agent.

Taro: It seems like they’re not just teaching it to walk, but teaching it a stable, repeatable running pattern by rewarding closeness to that stable path defined by the CLF.

Rosa: And the training reward R is composed of several parts: that CLF tracking reward r sub v, a CLF decay condition reward r sub v dot, and domain rewards.

Dev: The domain rewards are broken down into one for the single support phase, which encourages the stance foot to remain stationary in that phase, and one for the flight phase, which penalizes contact forces in the flight phase.

Taro: So they are explicitly telling it to stay put when it’s on its feet and not exert force when it’s airborne. That sounds like a very direct way to enforce physical constraints during learning.

Rosa: And this whole framework allows them to generate these running controllers for full humanoids, not just bipeds, with minimal tuning and without relying on human data.

Dev: The implication there is that you don't need massive datasets of human motion to learn complex dynamic gaits; you can use theoretical control insights to guide the learning from scratch.

Taro: If it works reliably in outdoor environments with only on-board sensors, that’s a critical step for integration into a full autonomy stack, isn't it?

Title and authors: Rosa: It is. The robot achieves accurate global reference tracking utilizing only those on-board sensors, and this capability is supported by using fast light through odometry and PD control to regulate lateral motion.

Dev: That reliance on fast light through odometry and PD control for lateral motion addresses the real-time loop requirements, which is something I always pay close attention to when we design these things.

Taro: So, putting all those pieces together—the offline trajectory optimization, the CLF guidance, and the PPO training—what’s actually missing from this approach?

Rosa: A limitation they flag is that they are focusing on generating trajectories that lie on a stable zero dynamics manifold, which contains the dynamics after virtual constraints are satisfied.

Dev: That means if the actual physical system deviates significantly from that idealized zero dynamics manifold during operation, the guaranteed stability based on their design might not hold up perfectly.

Taro: So they are providing a strong framework for dynamic running, but there’s still a gap in rigorously guaranteeing stability when the real world pushes those limits away from their planned paths.

Rosa: That’s right. They provide a principled way to synthesize these controllers, but the certification of stability depends on how well that offline optimization captured all the necessary constraints of reality.

Dev: It shifts the challenge from tuning a reward function blindly to making sure the underlying reference trajectories are physically feasible and stable for the system being controlled.

Taro: So, it’s less about finding *any* running policy, and more about finding a policy that is provably safe given those pre-computed dynamic paths.

Rosa: Exactly. It moves away from just trial and error learning toward a system where we have some control theory backing the learned behavior.

Dev: It’s a big step in making these robots truly autonomous for complex, dynamic tasks in unstructured settings, which is what we need for things like navigating uneven terrain or even just running around an outdoor area.

Taro: For me, it shows that if you can define the desired stable state mathematically first—the zero dynamics manifold—then reinforcement learning becomes a tool to efficiently find the path to get there.

Rosa: That’s a good way to put it. It’s about using control theory as the blueprint and RL as the efficient construction crew for the actual policy execution.

Dev: So, we've seen how they use CLFs to guide learning, how they use multi-domain optimization for trajectories, and what that means for real-world robustness when things get messy.

Taro: It’s a solid foundation for dynamic locomotion, but I still want to see how this scales if the robot needs to handle much more unpredictable external forces than just impacts from dodge balls.

Rosa: That’s definitely the next area for exploration, pushing those bounds on resilience in truly chaotic environments.

Dev: It sounds like a really promising direction for how we can start building reliable locomotion capabilities for humanoid platforms outside of highly controlled testbeds.

The paper's summary: Rosa: So, this paper is about using Reinforcement Learning to get humanoid robots to run without having to manually design all those complicated stability rules we usually have to handcraft for them.

Dev: Right, they're using Control Lyapunov Functions, or CLFs, embedded right into the reward system. That means the robot learns how to run in a way that is guaranteed to be stable according to that function.

Rosa: Exactly. They’ve found this way of eliminating those tricky heuristic reward terms that usually just make things unstable during training. It’s like giving the robot a built-in stability check while it's learning the steps for running, which encourages certifiable behavior.

Dev: And it also gives the learning process meaningful intermediate rewards to guide it toward a good running motion, instead of just hoping it stumbles into something decent. It links the desired dynamic trajectory directly to that proven stability condition.

Rosa: The paper shows they generated these full-order dynamic reference trajectories offline using multi-domain trajectory optimization. Then they feed those into the reinforcement learning training process using those CLFs as a guide for tracking.

Dev: That means you can synthesize running controllers for full humanoids, not just bipeds, with minimal tuning and without needing massive datasets of human motion. It’s a principled way to design these gaits from scratch.

Rosa: They tested this policy in real outdoor environments using only on-board sensors. The results showed the robot produced natural running behaviors with a full flight phase, even when it had to recover from external disturbances like impacts from dodge balls and hitting wooden planks.

Dev: And the CLF tracking reward actually improved velocity tracking performance, especially at higher speeds compared to just using a pure tracking reward. That suggests grounding the learning in those pre-computed dynamic paths really pays off for accuracy.

Rosa: It's a critical step toward integrating these complex motions into a full autonomy stack because the robot can achieve accurate global reference tracking just using on-board sensors, supported by fast light through odometry and PD control to handle lateral motion.

Dev: That capability is key for real-world use; it means we don't need perfect external sensing just to maintain position and velocity during dynamic tasks. But the paper does flag a limitation: they are focusing on generating trajectories that lie on a stable zero dynamics manifold, which means if the actual physical system deviates too much from that idealized path, the guaranteed stability might not hold up perfectly.

Rosa: So it's not just about finding any running policy; it's about finding one that is provably safe given those pre-computed dynamic paths. It shifts the challenge from tuning a reward function blindly to making sure the underlying reference trajectories are actually physically feasible for the system being controlled.

Dev: That’s right, and that means we have a framework where control theory provides the blueprint, and reinforcement learning acts as an efficient construction crew for executing that policy. But we still need to figure out how to make those dynamic paths robust enough to handle truly unpredictable external forces beyond just impacts.

The paper's improvements: Rosa: We just talked about how they use Control Lyapunov Functions to guide the reinforcement learning training for running on humanoid robots, but what else does this paper suggest we do better?

Dev: It suggests that the next step is focusing more on making those offline reference trajectories really robust against physical deviations. They want to move beyond just finding a path that *looks* stable in theory.

Rosa: So, they are looking at how the system behaves when it’s actually running outside of those perfect planned paths. They want to test how gracefully the robot recovers when the world misbehaves and doesn't follow the expected constraints.

Dev: That makes sense for a controls engineer; you can have a perfect mathematical model, but if your real-world hardware has friction or unexpected mass changes, that model breaks down in practice. So they want to see how much resilience those CLF-guided gaits actually have under stress.

Rosa: They're interested in using this framework to explore different types of external forces beyond just simple impacts from dodge balls, maybe things like uneven terrain where the ground itself is constantly changing.

Dev: Yeah, and they want to see if the method can adapt quickly enough in real time when those external conditions change rapidly. It’s about ensuring that the learning process doesn't get stuck because the environment shifted too fast for the policy to react properly.

Rosa: The implication there is that this isn't just about getting a robot to run smoothly on a treadmill; it’s about building locomotion systems that can operate reliably in messy, real-world scenarios where physics are constantly shifting.

Dev: It shifts the goal from just achieving velocity tracking accuracy to ensuring persistent stability across a wider range of physical conditions. That's where the safety case for real-time execution really gets established.

Rosa: And they’re also exploring how this method scales if we move from a single humanoid robot to a swarm of robots, trying to manage coordination while keeping that CLF guidance active for each unit.

Dev: Scaling up is always a headache, especially with real-time control loops. If you have multiple agents, the communication latency and the complexity of coordinating those stability functions gets much harder to manage in practice.

Rosa: So they’re looking at how to keep that principled way of synthesizing running controllers working when you introduce multiple agents interacting in a complex physical space. It’s a big leap for multi-robot dynamic locomotion.

Conclusion: Rosa: So to wrap up, this paper on "Chasing Stability: Humanoid Running via Control Lyapunov Function Guided Reinforcement Learning" shows how we can use control theory to guide reinforcement learning for dynamic locomotion.

Dev: Exactly, it’s about using those CLFs to create a reward structure that encourages certifiably stable behavior without having to handcraft all those complicated stability terms.

Rosa: It means we can generate these complex running gaits from scratch using offline trajectory optimization and then use reinforcement learning just as an efficient tool to find the policy that actually executes them.

Dev: The big implication is a principled way of synthesizing running controllers for full humanoids, not just bipeds, without needing tons of human motion data or massive manual tuning.

Rosa: It's a huge step toward making these robots truly autonomous for dynamic tasks outside of highly controlled testbeds because they can run robustly in real outdoor environments with only on-board sensors.

Dev: That resilience under disturbance is what matters for the safety case; it’s not just about reaching a speed, it’s about staying stable when things go wrong in the field.

Taro: I just think the focus on zero dynamics manifolds is important because if you can define that ideal stable path mathematically first, then reinforcement learning becomes a tool to efficiently find the path to get there.

Rosa: It really does shift the goal from just getting a robot to move smoothly on a treadmill to building locomotion systems that can operate reliably in messy, real-world scenarios where physics are constantly shifting.

Dev: And that’s where we need to push further, because if you introduce multiple agents into this framework, managing the communication latency and coordinating those stability functions becomes much harder in practice.

Taro: Scaling up is definitely a headache when you start talking about swarms; keeping that CLF guidance active for every unit while they interact in a complex physical space is a major challenge we need to explore next.

Rosa: Yeah, so this work sets up the blueprint for using control theory as the foundation and reinforcement learning as the construction crew for dynamic running.

Dev: It’s solid work, but it highlights that the next hurdle is moving from single-robot stability to multi-agent coordination under real-time constraints.

More episodes

← Home