Combining Constrained Sampling and Reinforcement Learning for Manipulation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Combining Constrained Sampling and Reinforcement Learning for Manipulation".
Dev: Here's the gist of the paper. Training non-prehensile manipulation policies in contact-rich settings is a core challenge in robotics <ref:2602.08557#pg2> While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and discover complex manipulation strategies <ref:2602.08557#pg2> To address this, the authors propose a novel state sampler that boosts the performance of goal-conditioned RL in complex contact-rich manipulation tasks <ref:2602.08557#pg2> By combining constrained sampling resets with projected interpolation and curriculum learning, their approach outperforms RL without constrained sampling and alternative reset methods <ref:2602.08557#pg2>.
Rosa: Problem Formulation The authors define a constrained goal-conditioned MDP as a variant of a standard goal-conditioned MDP where the state space is restricted by constraint functions <ref:2602.08557#pg2> Specifically, they define the constraint space Sc as Sc = S: gc(s) ≤ 0, hc(s) = 0 <ref:2602.08557#pg2> The problem then becomes finding a goal-conditioned policy π: s, g 7→ a to maximize max π Es,g∼p0 of Vπ(s, g) <ref:2602.08557#pg2> where p0(s) is defined as U(Sc) equally over start and goal states as uniform over Sc <ref:2602.08557#pg2>.
Dev: State Sampling Mechanism The state sampler formulates necessary collision, contact, and force constraints that hold for any state under rigid-body dynamics <ref:2602.08557#pg2> The constraints involve binary contact mode variables cij ∈ 0, 1 and continuous variables s ∈ R n as well as auxiliary decision variables for forces and points of attack <ref:2602.08557#pg2>.
Rosa: They use a hierarchical sampling strategy where they first sample the contact mode c:, and then use an Augmented Lagrangian method to solve the proximal problem min s,f: p: 1/2 s − s¯2 s.t.
Paper summary: Rosa: So, we're talking about this paper titled "Combining Constrained Sampling and Reinforcement Learning for Manipulation." Basically, they're looking at how to train robots to do things like picking up or balancing objects when you can only touch them, which is pretty tough because the world is full of contacts.
Dev: Yeah, the core idea here is that standard reinforcement learning often struggles to explore all the complicated ways a robot can manipulate something in these contact-rich situations and discover those complex strategies effectively.
Rosa: Right. And they propose this combination: first, using a novel state sampler to create better start and goal states for the RL agent, which is one way to improve exploration. Then, they pair that with some advanced reset strategies to make sure the RL process actually learns well from those starting points and goals.
Taro: That sounds like it’s trying to give the reinforcement learning algorithm a much better map of where it should start and where it should be going in this complex space, instead of just throwing random states at it.
Rosa: Exactly. The paper claims that by combining their constrained state sampler with an interpolation and curriculum reset strategy, they boost the performance of goal-conditioned RL in these complex contact-rich manipulation tasks compared to using just the constrained sampling or other reset methods alone.
Dev: It seems like the main claim is that this approach outperforms training without any constraints or using other simple reset techniques. It’s about making sure the initial states and goals are actually reachable and meaningful for learning.
Rosa: That's right. They show this works across different domains, testing various non-prehensile manipulation skills, like balancing things or complex whole-body contact maneuvers when dealing with objects on a sphere or a cube.
Taro: I’m interested in those specific results—the paper shows the policy they discover has versatile skill in reaching any possible quasi-static state in the double-sphere domain and complex whole-body contact maneuvers in the panda-sphere domain.
Paper summary: Dev: That versatility is important because it means the learned policy isn't just good at one specific trick; it can handle a lot of different scenarios that involve diverse contact configurations.
Rosa: And they even show this in simulation, but they also propose a way to transfer this knowledge to the real world using OptiTrack state tracking, replacing the simulated sphere robot with an endeffector-attached ball and tracking the cube's position with twelve Flexthirteen cameras.
Taro: Transferring it to the real world is a big step. It moves it from just working in a controlled simulation environment to actually interacting with physical objects under real-world conditions, which is where most of these problems really get messy.
Dev: From an engineering standpoint, the paper addresses how this system handles failure modes when the world misbehaves outside the lab and what kind of safety case you can build around that learned policy.
Rosa: They also looked into how they can generate a dataset for training using zero-order optimization methods to create open loop trajectories between random starts and goals from their constrained sampler, which is interesting because it’s about creating structured training data.
Taro: Creating those structured trajectories might help stabilize the learning process by giving the AI more realistic paths to follow when it's trying to get from a random spot to a target.
Dev: But we have to remember what they say about their limitations, and one of them is that their approach targets privileged training of a universal manipulation skill where the policy has full access to state.
Rosa: That means the method relies on the policy having perfect knowledge of the state, which isn't realistic for many physical systems in practice. Plus, they used a single-CPU constrained sampling method based on a general-purpose NLP solver, which raises substantial computational costs for more complex objects.
Taro: So while the performance on panda-sphere and double-sphere domains looks strong, the practical implementation cost and the assumption of perfect state access are real hurdles we have to consider.
Paper summary: Dev: That's true. The paper shows that while their method works well in simulation and for specific tasks, scaling it up to handle truly arbitrary complexity might require more sophisticated solvers than what they currently use.
Rosa: So, what does this mean for the world of robotics? It suggests a way to overcome the exploration problems that plague reinforcement learning when you’re dealing with things that have lots of unexpected physical interactions.
Taro: It means we might be able to train robots for real-world manipulation skills much faster and more reliably than before, especially in unstructured environments where prehensile grasping isn't the only option.
Dev: It shifts the focus from just trying to find *any* good policy to designing a better way to sample the problem space itself, which is a fundamental change in how we think about goal-conditioned RL.
Rosa: Exactly. The combination of constrained sampling with intelligent reset strategies seems like a solid way forward for tackling those hard manipulation challenges that aren't just simple pick-and-place tasks.
Taro: It gives us tools to see what the robot *can* do in terms of contact and balance, which is crucial for building robots that can work in messy, real-world settings.
Dev: So, to wrap up this discussion on "Combining Constrained Sampling and Reinforcement Learning for Manipulation," the paper offers a way to improve exploration by constraining the starting and goal states using a specific sampler, then enhancing that with interpolation and curriculum resets for better learning.
Rosa: And it demonstrates that this combination leads to policies with versatile skill in complex tasks like reaching any quasi-static state in the double-sphere domain or doing whole-body contact maneuvers on the panda-sphere domain.
Taro: It’s a solid foundation for training more capable, adaptable manipulation systems, even if we still need to work on those computational costs and real-world state access limitations they mentioned.
Dev: That's right. This work is about making the training process itself smarter by structuring the problem space before we let the reinforcement learning agent start exploring it freely.
Conclusion: Rosa: So we're wrapping up this discussion on "Combining Constrained Sampling and Reinforcement Learning for Manipulation."
Dev: Yeah, to put it simply, these authors are showing how you can make reinforcement learning better at tasks that involve lots of physical contact by structuring the starting and goal states.
Rosa: They did this by using a new state sampler to limit the robot's possible positions based on physics constraints and then combining that with some interpolation and curriculum learning resets.
Dev: It means instead of the AI wandering around randomly, it starts its learning from points that are physically plausible for manipulation, which should speed things up significantly.
Rosa: The authors claim this combination beats using just the constrained sampling or other reset methods by showing better performance in complex tasks like whole-body contact maneuvers on a panda-sphere.
Dev: That versatility in the results they show—reaching any possible quasi-static state in the double-sphere domain—is what really tells you this isn't just a small tweak, it’s addressing the core exploration problem.
Rosa: It moves beyond simple pick and place into handling complex interactions where things are constantly touching or balancing.
Dev: The paper also touches on how they can take these learned policies and transfer them to the real world using state tracking from cameras.
Rosa: They even look at using zero-order optimization to generate training data for those trajectories, which is a neat way to create structured experience.
Dev: But you have to remember their caveat about the limitations: this method requires the policy to have full access to the state, and they used a single CPU sampler which can be slow for really complex objects.
Rosa: So it’s a powerful technique for training universal manipulation skills, but we still need to work on making it computationally efficient enough for everything.
Dev: That’s right. And that brings us to how this approach might actually change the way we design those initial setups for robotic learning...
Marc Toussaint, Cornelius V. Braun, Armand Jordana, Sayantan Auddy, Eckart Cobo-Briesewitz, Denis Shcherba, Tilman Burghoff, Justin Carpentier
Technische Universitat Berlin Robotics Institute Germany LAAS-CNRS Inria PSL Research University
cs.RO
Submitted: 2026-02-09
Updated: 2026-10-29
Comments: Conference on Robot Learning (CoRL) 2026
Code: https://github.com/sfujim/TD7
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 99/100
The gist: Training non-prehensile manipulation policies in contact-rich settings is a core challenge in robotics <ref:2602.08557#pg2> While Reinforcement Learning (RL) has demonstrated its strength in such…
Key concepts
- Constrained Goal-Conditioned MDP
- This is a modified Reinforcement Learning problem where the robot's state space is restricted by mathematical functions that define physical constraints, such as collision avoidance or specific contact modes. The goal is to find an optimal policy that maximizes reward while staying within these physically feasible boundaries.
- State Sampler
- This mechanism generates realistic and valid states for training by enforcing necessary physics constraints like rigid-body dynamics, contact modes (binary variables), and force limits. It uses a hierarchical sampling strategy, first determining contact modes and then solving a proximal problem to find physically plausible continuous state variables.
- Projected Interpolation Reset
- This is an advanced exploration technique where the agent resets its state to a point on the constrained space that lies along the straight line between two sampled states (start and goal). This ensures that every training step begins from a physically meaningful, valid configuration within the allowed constraints.
- Curriculum Learning
- This strategy involves gradually introducing complexity during training. In this context, it means scheduling the interpolation factor 't' over time, starting with a small range (near 0) and gradually increasing it towards 1. This helps the RL agent learn simpler skills before tackling more complex, fully interpolated states.
Terminology
Summary
Training non-prehensile manipulation policies in contact-rich settings is a core challenge in robotics <ref:2602. p. 0 Braun1
, Armand Jordana3
, Sayantan Auddy1
,
Eckart Cobo-Briesewitz1
, Denis Shcherba1
, Tilman Burghoff1
, Justin Carpentier4
1Technische Universitat Berlin ¨
2Robotics Institute Germany
3LAAS-CNRS 4
Inria & DI-ENS, PSL Research University
toussaint@tu-berlin. de
Abstract: Training non-prehensile manipulation policies in contact-rich settings
is a core challenge in robotics. de
Abstract: Training non-prehensile manipulation policies in contact-rich settings
is a core challenge in robotics. While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and
discover complex manipulation strategies. Second, while model-based approaches to finding trajectories through manipulation are hard, recent work showed that model-based approaches to sampling states
on constrained manifolds can be highly efficient. Based on these observations, we
propose a novel state sampler that boosts the performance of goal-conditioned
RL in complex contact-rich manipulation tasks. Our sampler explicitly takes into
account the structure of contact in order to provide a rich covering of diverse
contact modes. By combining constrained sampling resets with projected interpolation and curriculum learning, our novel approach outperforms RL without constrained sampling and alternative reset methods, and effectively trains universal,
non-prehensile, and dynamic manipulation policies in contact-rich settings.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Problem Formulation
The authors define a constrained goal-conditioned MDP as a variant of a standard goal-conditioned MDP where the state space is restricted by constraint functions <ref:2602.08557#pg2> Specifically, they define the constraint space Sc as Sc = S: gc(s) ≤ 0, hc(s) = 0 <ref:2602. p. 1 More specifically, we address training goal-conditioned (or universal [18, 19]) policies π : s, g 7→ a
that control from any starting state s ∼ p(s) to any goal g ∼ p(g) of the same distribution. We define
that distribution to diversely cover contact modes and manipulation states using a constraint formulation. Given Sc,
we define a distribution p0(s) = U(Sc) equally over start and goal states as uniform over Sc. The
problem is to find a goal-conditioned policy π : s, g 7→ a to maximize
max
π
Es,g∼p0
{V
π
(s, g)} , (2)
with V
π
(s, g) = E{
P
t
γ
tR(st+1, st, g)| π, st=0 = s} (3)
the universal value function [18]. 1 Problem Formulation
We define a constrained goal-conditioned MDP as a variant of a standard goal-conditioned MDP
[18, 19], where we have a state space S ⊆ R
d
, an action space A, dynamics p(s
′
|s, a), a goalconditioned reward function R(s
′
, s, g), discount factor γ ∈ (0, 1), as well as constraint functions
gc, hc which define the constraint space
Sc = {s ∈ S : gc(s) ≤ 0, hc(s) = 0}, (1)
where gc : R
d → R
gc and hc : R
d → R
hc and their Jacobians can be queried point-wise. Given Sc,
we define a distribution p0(s) = U(Sc) equally over start and goal states as uniform over Sc.Where the paper says it
Where the paper says it
Where the paper says it
State Sampling Mechanism
The state sampler formulates necessary collision, contact, and force constraints that hold for any state under rigid-body dynamics <ref:2602.08557#pg2> The constraints involve binary contact mode variables cij ∈ 0, 1 and continuous variables s ∈ R n as well as auxiliary decision variables for forces and points of attack <ref:2602. p. 3 Non-linear Constrained Sampling. In summary, our decision variables are the discrete contact
mode c: with cij ∈ {0, 1}, and continuous s ∈ R
n, as well as pij , fij ∈ R
3
for pairs with cij = 1. In summary, our decision variables are the discrete contact
mode c: with cij ∈ {0, 1}, and continuous s ∈ R
n, as well as pij , fij ∈ R
3
for pairs with cij = 1. We use a hierarchical sampling strategy, where we first sample the contact mode c:
, and then for
fixed c: use an Augmented Lagrangian method [44] to solve the proximal problem
min
s,f: ,p:
1
2
||s − s¯||2
2
s. de/mtoussai/26-CSRL/ for supplementary material. arXiv:2602. Concerning sampling cij , in this paper we assume a single free object i = 1 (which is subject to
manipulation) and a finite set of potential support shapes Z ⊆ {2, . We limit the potential
number of active support contacts to 3, uniformly first choose between the number 1, 2, or 3 of
support contacts, and then uniformly pick the 1-, 2- or 3-tuples of support contacts from Z.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Advanced Reset Strategies
To mitigate exploration challenges, the authors propose two additional strategies to combine with their constrained sampler <ref:2602.08557#pg2> 1) Projected interpolation resets, which reset to s = argmin s'∈D ϕ(s') − [tϕ(a) + (1 − t)ϕ(g)] with t ∼ U[0, 1] projected onto the constrained space Sc <ref:2602.08557#pg2>. 2) An optional reset curriculum, where a phase variable α ∈ (0, 1] is scheduled linearly from near 0 to 1 during initial training, modifying the interpolation to use t ∼ U[0, α] <ref:2602. p. 4 We therefore project the interpolation to our
constrained space Sc approximately by finding the nearest s ∈ D (using a kd-tree in ϕ-space). Second, we propose an optional reset curriculum, where a phase variable α ∈ (0, 1] is scheduled
linearly from near 0 to 1 during an initial phase of RL training, and the interpolation above is
modified to use t ∼ U[0, α], resetting closer to the goal.Where the paper says it
Experimental Validation
The approach was evaluated on four domains designed to test various non-prehensile manipulation skills <ref:2602.08557#pg2> The results show that constrained sampling outperforms Random Sampling, especially in the panda-sphere domain where random sampling performs poorly <ref:2602. p. 5 On our simplest domain, the
double-sphere, random resets can achieve considerable but sub-optimal success rates when trained
on goals from our constrained sampler. On the panda-sphere problem, random sampling performs
poorly, presumably as it is unlikely to provide reset states where the sphere is held static between
limbs. Projected interpolation resets and curricular further improve performance. Tables 1 and 2 confirm
that adding projected interpolation and a curriculum into the reset strategy further increases performances, across domains and RL methods used inside CSRL. Concerning the doublesphere domain, the optimal policy shows versatile skill in reaching any possible quasi-static state,
including balancing the ball and highly reactive recovery from external perturbations. For the pandasphere domain, we find versatile skill in bringing the sphere to be held with diverse limbs, using
complex whole-body contact maneuvers to scoop and transport the ball into holding positions.Where the paper says it
Where the paper says it
Where the paper says it
Real World Transfer
The authors propose a novel framework for privileged training of fundamental manipulation skills by transferring policies to real world using OptiTrack state tracking <ref:2602.08557#pg2> This involves replacing the simulated sphere robot with an endeffector-attached ball and tracking the cube's position via 12 Flex13 cameras <ref:2602.08557#pg2>.
Comparison to Behavior Cloning
The authors investigated using behavior cloning (BC) within TD7, finding that while state initialization has a drastic impact on training behavior, adding a BC objective has minor impact <ref:2602.08557#pg2>. The core role of trajectories is demonstrating relevant “interpolating” states rather than demonstrating actions <ref:2602.08557#pg2>.
Zero-Order Optimization
They explored using zero-order optimization methods to create a dataset of open loop trajectories between random starts and goals from their constrained sampler <ref:2602.08557#pg2> This involved solving an optimization problem min θ ϕ(g) − ϕ(x(T))2 s. p. 3 In summary, our decision variables are the discrete contact
mode c: with cij ∈ {0, 1}, and continuous s ∈ R
n, as well as pij , fij ∈ R
3
for pairs with cij = 1. We use a hierarchical sampling strategy, where we first sample the contact mode c:
, and then for
fixed c: use an Augmented Lagrangian method [44] to solve the proximal problem
min
s,f: ,p:
1
2
||s − s¯||2
2
s. de/mtoussai/26-CSRL/ for supplementary material. arXiv:2602.Where the paper says it
Where the paper says it
Conclusion
The authors conclude that integrating the constrained sampler into a reset strategy and optionally combining this with an interpolation and curriculum strategy attains an effective method to train universal non-prehensile manipulation policies <ref:2602.08557#pg2> Their evaluations demonstrate contact-rich whole-body manipulation in the panda-sphere scenario, as well as highly agile behaviors to bring a cube to balance on the edge or a corner in the sphere-cube scenario <ref:2602. p. 7 Integrating the constrained sampler in a reset
strategy and optionally combining this with an interpolation and curriculum strategy we attain an
effective method to train universal non-prehensile manipulation policies. Our evaluations demonstrate contact-rich whole-body manipulation in the panda-sphere scenario, as well as highly agile
behaviors to bring a cube to balance on the edge or a corner in the sphere-cube scenario.Where the paper says it
Limitations
A limitation is that their approach targets privileged training of a universal manipulation skill where the policy has full access to state <ref:2602.08557#pg2> Secondly, they used a single-CPU constrained sampling method based on a general-purpose NLP solver, which raises substantial computational costs for more complex objects <ref:2602. p. 7 (ii) In this work we used a singleCPU constrained sampling method based on a general-purpose NLP solver. While sampling for
simple scenes such as the double-sphere is still efficient, the non-linearities in constraints (4-7) raise
substantial computational costs for more complex objects.Where the paper says it
Acknowledgments
This work has received support from the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG); from the Amazon Fulfillment Technologies and Robotics team; from the French government, managed by the National Research Agency, under the France 2030 program with the references Organic Robotics Program (PEPR O2R), “PR[AI]RIE-PSAI” (ANR-23-IACL-0008) and RODEO (ANR-24-CE23-5886); and from the European Union through the ARTIFACT project (GA no.101165695) and the AGIMUS project (GA no. p. 8 Acknowledgments
This work has received support from the German Federal Ministry of Research, Technology and
Space (BMFTR) under the Robotics Institute Germany (RIG); from the Amazon Fulfillment Technologies and Robotics team; from the French government, managed by the National Research
Agency, under the France 2030 program with the references Organic Robotics Program (PEPR
O2R), “PR[AI]RIE-PSAI” (ANR-23-IACL-0008) and RODEO (ANR-24-CE23-5886); and from
the European Union through the ARTIFACT project (GA no. 101165695) and the AGIMUS project
(GA no. de/mtoussai/26-CSRL/ for supplementary material. arXiv:2602.Where the paper says it
Where the paper says it
References
The paper cites numerous works related to manipulation planning, goal-conditioned RL, and various exploration methods <ref:2602.08557#pg2> Key references include those on model-based manipulation planning, universal value functions and goal-conditioned MDPs, and specific techniques like Hindsight experience replay and reverse curriculum generation <ref:2602. p. 2 of objects on which the vision-based policy is applicable. This notion differs somewhat from the
concept of universal value functions and goal-conditioned MDPs [18, 19, 29], where the aim is to
learn policies that maximize rewards under any task in a fixed MDP (with fixed objects). In contrast, our work constructs the reset and
goal distribution from general constraints on manipulation states, allowing us to expose the policy
to diverse contact configurations without committing to a specific interaction primitive. 3 Methods
3.Where the paper says it
Where the paper says it
Gym Environment Details
The authors detail the Mujoco simulation step time τsim = 0.001sec and action time τaction = 0. p. 7 synced view (bottom), paired by action index t (number of policy actions
taken). The synced view mirrors the OptiTrack cube pose and the robot’s joint state; per-action
wall-clock time is set by a blocking IK move. de/mtoussai/26-CSRL/ for supplementary material. arXiv:2602.Where the paper says it
Where the paper says it
Vector Embedding for Projected Interpolation
For interpolation (9), they use the vector embedding ϕ(s) = (wqq, G) ∈ R n+3nG that combines the goal features with the weighted robot pose q <ref:2602.08557#pg2>. The weights w are chosen as 0.5 for sphere domains and 0. p. 5 Table 1 compares alternatives combinations
of using constrained sampling (C) or random sampling (R) for the goal and start distribution, as well
as combining this with projected interpolation (i) and curriculum learning (c). On the double-sphere
and panda-sphere domains, as well as their sparse variants (see Appendix A), using constrained sampling for starts and goals significantly outperforms random sampling. 027
C Crev 0. 093
Table 3: (a) Performance on the cube domains.Where the paper says it
Where the paper says it
Compute Statistics
Table 4 provides compute statistics showing the dimensionality, training requirements, and environment parameters for each environment <ref:2602.08557#pg2>. Table 3(a) confirms the benefit of the interpolation and curriculum strategy in demanding settings like the cube domains <ref:2602. p. 6 Stable
training required to increase the replay buffer size to 4M and 300k training steps. Table 3(a) confirms
the benefit of the interpolation and curriculum strategy in these settings. The domain is a basic test for universal manipulation capable to reach any quasistatic state from any other. (2) The panda-sphere domain replaces the sphere robot with a panda
arm, and the set of S of potential support shapes includes all of panda’s links (made convex due
to MuJoCo’s constraints) in addition to the floor. 2 provides success rates during training. The convergence behaviors confirm the benefit of our
proposed methods not only for the final policy performance, but also for convergence speed and
thereby sample efficiency.Where the paper says it
Where the paper says it
Where the paper says it
Feasible Rates
Table 1 compares alternative goal and reset sampling methods, showing that Constrained Sampling (C) significantly outperforms Random Sampling (R) on the double-sphere and panda-sphere domains <ref:2602.08557#pg2>. Table 3(b) shows that their interpolation approach outperforms reverse curricular, also when using random drop starts <ref:2602. p. 6 Table 3(b) provides results on the double-sphere and panda-sphere domains. For both domains, our
interpolation approach outperforms reverse curricular, also when using random drop starts. Projected interpolation resets and curricular further improve performance. Tables 1 and 2 confirm
that adding projected interpolation and a curriculum into the reset strategy further increases performances, across domains and RL methods used inside CSRL. 2 provides success rates during training. The convergence behaviors confirm the benefit of our
proposed methods not only for the final policy performance, but also for convergence speed and
thereby sample efficiency. The policies we discover are highly dynamic,
robust, and capable of using whole-body and environmental contacts to reach a broad range of goals. We confirm the effectiveness of our approach to training manipulation skills by comparing it to alternative reset strategies. The policies we discover are highly dynamic,
robust, and capable of using whole-body and environmental contacts to reach a broad range of goals. We confirm the effectiveness of our approach to training manipulation skills by comparing it to alternative reset strategies. The policies we discover are highly dynamic,
robust, and capable of using whole-body and environmental contacts to reach a broad range of goals. We confirm the effectiveness of our approach to training manipulation skills by comparing it to alternative reset strategies. The policies we discover are highly dynamic,
robust, and capable of using whole-body and environmental contacts to reach a broad range of goals. We confirm the effectiveness of our approach to training manipulation skills by comparing it to alternative reset strategies. Concerning the doublesphere domain, the optimal policy shows versatile skill in reaching any possible quasi-static state,
including balancing the ball and highly reactive recovery from external perturbations. For the pandasphere domain, we find versatile skill in bringing the sphere to be held with diverse limbs, using
complex whole-body contact maneuvers to scoop and transport the ball into holding positions. Concerning the doublesphere domain, the optimal policy shows versatile skill in reaching any possible quasi-static state,
including balancing the ball and highly reactive recovery from external perturbations. For the pandasphere domain, we find versatile skill in bringing the sphere to be held with diverse limbs, using
complex whole-body contact maneuvers to scoop and transport the ball into holding positions. Concerning the doublesphere domain, the optimal policy shows versatile skill in reaching any possible quasi-static state,
including balancing the ball and highly reactive recovery from external perturbations. For the pandasphere domain, we find versatile skill in bringing the sphere to be held with diverse limbs, using
complex whole-body contact maneuvers to scoop and transport the ball into holding positions. The policies we discover are highly dynamic,
robust, and capable of using whole-body and environmental contacts to reach a broad range of goals. We confirm the effectiveness of our approach to training manipulation skills by comparing it to alternative reset strategies.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Figure 3
Figure 3 illustrates the learned behaviors and success rates, showing versatile skill in reaching any possible quasi-static state in the double-sphere domain, including balancing the ball and highly reactive recovery from external perturbations <ref:2602.08557#pg2>. For the panda-sphere domain, they find versatile skill in bringing the sphere to be held with diverse limbs, using complex whole-body contact maneuvers to scoop and transport the ball into holding positions <ref:2602. p. 6 Concerning the doublesphere domain, the optimal policy shows versatile skill in reaching any possible quasi-static state,
including balancing the ball and highly reactive recovery from external perturbations. For the pandasphere domain, we find versatile skill in bringing the sphere to be held with diverse limbs, using
complex whole-body contact maneuvers to scoop and transport the ball into holding positions. The synced view mirrors the OptiTrack cube pose and the robot’s joint state; per-action
wall-clock time is set by a blocking IK move. For the sphere-cube domain, our policy is highly effective in turning the cube to desired orientations and balancing it on the edge or corner. In the latter case it reaches near-goal configurations
with highly dynamic and reactive behaviors, though sometimes missing the goal precision threshold
slightly. In the panda-cube domain the learned strategies include rotating the cube via a flat endeffector squeeze from top.Where the paper says it
Where the paper says it
Where the paper says it
Figure 4
Figure 4 shows the real world setup, illustrating the Franka Emika Panda manipulator, the custom end-effector and the cube <ref:2602.08557#pg2>. The policy is state-based, tracking the cube’s position by using an Optitrack system with 12 Flex13 cameras <ref:2602. p. 7 To evaluate basic real-world transfer we considered cube manipulation with using OptiTrack
state tracking. In particular, we considered transfer of a sphere-cube policy, where the simulated
sphere robot is replaced by an endeffector-attached ball (see App.Where the paper says it
Figure 5
Figure 4(a) shows a picture of the real world setup, showing the Franka Emika Panda manipulator, the custom end-effector and the cube <ref:2602.08557#pg2>. Figure 4(b) illustrates dimensions of the custom end-effector in millimeter <ref:2602.08557#pg2>.
Figure 6
Figure 6 shows constrained samples from the panda sphere domain <ref:2602.08557#pg2>.
Figure 7
Figure 7 shows constrained samples from the sphere cube domain <ref:2602.08557#pg2>.
Table 1
Table 1 compares alternative goal and reset sampling methods, showing that Constrained Sampling (C) significantly outperforms Random Sampling (R) on the double-sphere and panda-sphere domains <ref:2602.08557#pg2>. Table 2 provides performances when using alternative RL methods inside CSRL <ref:2602. p. 5 The above holds across RL methods used inside CSRL, including HER variants. Table 2 provides
performances when using alternative RL engines inside CSRL.Where the paper says it
Table 3
Table 3 shows performance on the cube domains, confirming the benefit of the interpolation and curriculum strategy in these settings <ref:2602.08557#pg2>. Table 4 provides compute statistics for constrained sampling <ref:2602. p. 4 The study uses the TD7 [46] as default
RL method within CSRL, but we evaluate also a HER extension and TD3 version. Appendix C
provides compute statistics for constrained sampling in our four domains.Where the paper says it
Table 4
Table 4 provides dimensionality, training and environment parameters for each environment <ref:2602.08557#pg2>. The success rates during training figures illustrate the effectiveness of their approach <ref:2602.08557#pg2>.
Figure 2
Figure 2 provides success rates during RL training, confirming the benefit of their proposed methods not only for the final policy performance but also for convergence speed and thereby sample efficiency <ref:2602.08557#pg2>. The optimal policy shows versatile skill in reaching any possible quasi-static state in the double-sphere domain and complex whole-body contact maneuvers in the panda-sphere domain <ref:2602.08557#pg2>.
Table 3(a)
Table 3(a) confirms the benefit of the interpolation and curriculum strategy in these settings <ref:2602.08557#pg2>.
Figure 1
Figure 1 shows a complementary physics simulator that samples dynamics, leveraging a physics-informed state sampler to globally sample starts and goals <ref:2602.08557#pg2>.
Table 3(b)
Table 3(b) provides results on the double-sphere and panda-sphere domains <ref:2602.08557#pg2>.
Figure 8
Figure 1 shows a complementary physics simulator that samples dynamics, leveraging a physics-informed state sampler to globally sample starts and goals <ref:2602.08557#pg2>.
Figure 9
Figure 1 shows a complementary physics simulator that samples dynamics, leveraging a physics-informed state sampler to globally sample starts and goals <ref:2602.08557#pg2>.
Table 4 provides dimensionality, training and environment parameters for each environment <ref:2602.08557#pg2>. The success rates during training figures illustrate the effectiveness of their approach <ref:2602.08557#pg2>. The optimal policy shows versatile skill in reaching any possible quasi-static state in the double-sphere domain and complex whole-body contact maneuvers in the panda-sphere domain <ref:2602.08557#pg2>.
Figure 1 shows a complementary physics simulator that samples dynamics, leveraging a physics-informed state sampler to globally sample starts and goals <ref:2602.08557#pg2>. The authors propose to complement this with a physics-informed constrained sampler of states <ref:2602. p. 7 5 Conclusion
Relating back to Fig. 1, while a physics simulator provides a means to sample the dynamics, with
this paper we propose to complement this with a physics-informed constrained sampler of states. 5 Conclusion
Relating back to Fig. 1, while a physics simulator provides a means to sample the dynamics, with
this paper we propose to complement this with a physics-informed constrained sampler of states.Where the paper says it
Where the paper says it
Table 3(b) provides results on the double-sphere and panda-sphere domains <ref:2602.08557#pg2>. The success rates during training figures illustrate the effectiveness of their approach <ref:2602.08557#pg2>.
Figure 1 shows a complementary physics simulator that samples dynamics, leveraging a physics-informed state sampler to globally sample starts and goals <ref:2602.08557#pg2>. The authors propose to complement this with a physics-informed constrained sampler of states <ref:2602.08557#pg2>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Constrained State Sampler for Non-Prehensile Manipulation: The system can generate a
rich covering of diverse contact modes
by explicitly taking into accountthe structure of contact,
providing areset and goal distribution for goal-conditioned RL.
-
Projected Interpolation Resets: The AI can mitigate the exploration challenge in sparse reward tasks by using a strategy where it resets to the state that is the
uniform convex interpolation
between a sampled start and goal state, projected to stay within the feasible constraint space Sc. -
Curriculum Learning Reset Strategy: The system can improve performance by employing an optional reset curriculum where it modifies the interpolation strategy using
t U[0, α]
and adapts MDP truncation time proportional to alpha, allowing RL to initially collectshort rollouts near to the goal.
-
Universal Goal-Conditioned Policy Training: The improved system will train a policy π that can control from
any starting state s ∼ p(s) to any goal g ∼ p(g) of the same distribution,
enabling it to learnuniversal, non-prehensile, and dynamic manipulation policies in contact-rich settings.
-
Whole-Body Contact Skill Acquisition: The system will acquire versatile skills in complex scenarios, such as
balancing the ball and highly reactive recovery from external perturbations
or performingcomplex whole-body contact maneuvers to scoop and transport the ball into holding positions.
-
Robustness Against Alternative Sampling Methods: By demonstrating that
Constrained Sampling outperforms Random Sampling,
the AI system will exhibit significantly higher success rates across different RL methods, even when compared to behavior cloning from diverse open-loop trajectories.
Abstract
Training manipulation policies in contact-rich settings is a core challenge in robotics. While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and discover complex manipulation strategies. To address this, we combine two basic ideas: First, designing appropriate reset strategies (the start state distribution of episodes) has shown promise in improving RL exploration and effectiveness. Second, while model-based approaches to finding trajectories through manipulation are hard, recent work showed that model-based approaches to sampling states on constrained manifolds can be highly efficient. Based on these observations, we propose a novel state sampler that enables efficient goal-conditioned RL in complex contact-rich manipulation tasks. Our sampler explicitly takes into account the structure of contact in order to provide a rich covering of diverse contact modes. By combining constrained sampling resets with projected interpolation and curriculum learning, our novel approach outperforms RL without constrained sampling and alternative reset methods, and effectively trains universal, prehensile and non-prehensile manipulation policies in contact-rich whole-body and in-hand settings. See the project page https://www.user.tu-berlin.de/mtoussai/26-CSRL/ for videos and full source code access.
Sources
- Solving Rubik's Cube with a Robot Hand
- Epistemically-guided forward-backward exploration
- Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning
- An Introduction to Zero-Order Optimization Techniques for Robotics
- Learning Multi-Level Hierarchies with Hindsight
- Diversity Progress for Goal Selection in Discriminability-Motivated RL
- FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control
- CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation