A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

arXiv:2610.12465 · cs.RO, cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "A Balanced Data Diet".

Dev: The gist: Success Guided Sampling (SGS) is introduced as a simple adaptive sampler that concentrates Reinforcement Learning training on task configurations around the frontier of the policy’s capabilities,

Rosa: First, who's behind it and why it matters.

Title and authors: Dev: We're talking about "A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control," and the paper highlights how this Success Guided Sampling method tackles that exploration problem in large-scale simulations. The authors focus on moving away from wasteful sampling by concentrating training where it matters most.

Rosa: What they are doing is showing how this simple sampler can concentrate Reinforcement Learning training on task configurations around the frontier of the policy’s capabilities, which lets large-scale simulated RL make the most out of the experience in a batch, enabling much more effective scaling.

Taro: It really points to a shift in how we approach autonomy; instead of randomly throwing every possible situation at a robot, we direct that experience toward configurations that are neither too easy nor completely impossible under the current policy. That’s a big difference for building general learners.

Dev: The authors note that this work combines diverse simulator resets, success-based adaptive sampling, and large-scale RL using no demonstrations for RL training and one shared reward function within each domain <ref:2610.12465#pg3>. This structure is key to managing the complexity of general-purpose robotics.

Rosa: And they also show how they distill these learned manipulation policies into RGB-based ones and transfer them zero-shot to real hardware, achieving about ninety-eight percent success on assembly tasks when using a point-cloud teacher with symmetric resets and rewards <ref:2610.12465#pg1>. That shows a path from simulation to actual physical interaction.

The paper's summary: Dev: Building on that, the paper summarizes how this method addresses the engineering burden by using one simple reward function across all locomotion tasks and another across all manipulation tasks, which avoids the huge engineering headache of having to tune a different reward for every single terrain or assembly task.

Rosa: That shared reward design is important because it cuts down on the amount of hand-engineered prior knowledge you need to inject into every new environment. It’s about reducing that per-task structural prior you mentioned earlier, so we don't spend weeks tuning a reward function just for a new type of terrain.

Taro: And they also address the scaling issue by concentrating rollouts on configurations where the policy is actively learning, so when you add more environments, it actually makes the learning signal denser instead of getting diluted <ref:2610.12465#pg3>. That’s a real practical improvement for scaling up simulations to one million parallel environments.

Dev: They show that this SGS method outperforms uniform sampling and Prioritized Level Replay on hard tasks like multi-terrain whole-body locomotion and nut-and-bolt assembly, reaching seventy-three percent mean success on multi-task locomotion versus fifty-four percent for Prioritized Level Replay at one million parallel environments.

Rosa: That performance gain is significant because it shows that this adaptive sampling actually helps the policy get better as you scale up the training data, which is something other methods struggle with. It's not just about getting more data; it's about getting better data efficiently.

The paper's improvements: Dev: So, to wrap up "A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control," they present SGS as a simple success-based sampler that focuses training on task configurations with moderate success rates, combined with diverse resets and large-scale RL.

Rosa: The main implication is that you can train policies for complex behaviors without needing to spend massive amounts of time on hand-engineered reward specifications or manual curriculum engineering. This lets you focus the effort where it matters most, freeing up human intuition for more complex system design instead of just tweaking reward functions.

Taro: For autonomy research, this means we can push systems to handle much wider varieties of tasks just by being smart about how we expose them to experience during training rather than relying on pre-defined task structures for every single scenario. It shifts the focus from building a library of task solutions to building a general learner.

Dev: And they also showed a way to distill these learned manipulation policies into RGB-based ones and transfer them zero-shot to real hardware, achieving about ninety-eight percent success on assembly tasks when using a point-cloud teacher with symmetric resets and rewards <ref:2610.12465#pg1>. That’s not just in simulation anymore.

Rosa: So while they have these limitations—like the fact that their success estimates are based on a discrete set of cells, which means for continuous spaces you might need a different scoring representation—they show a solid path toward making large-scale simulated RL much more effective. It’s about making the exploration process itself smarter, not just throwing more data at the problem.

Conclusion: Dev: So we've been looking at "A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control," which is essentially a paper about Success Guided Sampling, or SGS. Right, it’s that adaptive sampler that concentrates training on configurations where the robot is currently succeeding or struggling moderately.

Taro: It really addresses that problem of how the AI wanders around randomly in millions of simulations without learning anything useful by directing its attention toward those middle ground situations.

Rosa: Exactly, it moves away from just throwing every possible situation at the robot and directs that experience toward configurations that are neither too easy nor completely impossible under the current policy <ref:2610.12465#pg1>.

Dev: The mechanism they use is weighting those success estimates with a beta distribution to prioritize those middle ground configurations, which is defined by a specific density formula using pi i and kappa.

Taro: That weighting seems clever because it stops the AI from just getting stuck on one type of task or ignoring the really hard ones entirely.

Rosa: And when you look at the results, they show that this method outperforms uniform sampling and Prioritized Level Replay on tough tasks like multi-terrain whole-body locomotion at one million parallel environments <ref:2610.12465#pg2>.

Dev: They hit seventy-three percent mean success on multi-task locomotion compared to fifty-four percent for Prioritized Level Replay in those large settings, which is a solid performance jump.

Taro: It proves that this isn't just theoretical; it actually translates into better results when you scale up the training data.

Rosa: The authors also touched on how they reduced the engineering burden by using one simple reward function for all locomotion and another for manipulation tasks, instead of tuning a unique reward for every terrain.

Dev: That shared reward design is important because it cuts down on the amount of hand-engineered prior knowledge you need to inject into every new environment.

Taro: It means we can focus less on creating task-specific rewards and more on building better underlying learning models.

Rosa: They also showed a way to distill these learned manipulation policies into RGB-based ones and transfer them zero-shot to real hardware, achieving about ninety-eight percent success on assembly tasks <ref:2610.12465#pg1>. That zero-shot transfer capability is pretty impressive, showing the policy generalizes well when it moves from simulation to actual physical interaction.

Dev: It’s definitely something the control engineers need to pay attention to because it changes how we think about data density in massive parallel setups.

Taro: Next time, we'll be looking at something completely different, maybe how large models can actually see and navigate the world without relying on all that pre-defined structure.

University of Washington

cs.RO, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: CoRL 2026. Project website: https://sgs-rl.github.io/

Project page: https://sgs-rl.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist: Success Guided Sampling (SGS) is introduced as a simple adaptive sampler that concentrates Reinforcement Learning training on task configurations around the frontier of the policy’s

Key concepts

Success Guided Sampling (SGS)
SGS is an adaptive sampling method that directs Reinforcement Learning training toward task configurations where the policy is actively learning. It uses non-parametric estimates of success rates to prioritize experiences that are neither too easy nor too hard, ensuring the agent receives the most beneficial learning signals from parallel environments.
Task Configuration
A task configuration defines a specific scenario for the robot to perform. For locomotion, this includes terrain type, starting pose, and goal pose. For manipulation tasks, it specifies the assembly task and initial states like reaching or stable grasp.
Beta Distribution Weighting
SGS uses a beta distribution to weight success estimates of configurations. This weighting prioritizes sampling configurations with moderate success rates (the mode of the distribution). This mechanism ensures that training time is spent on experiences that provide the most informative learning signal, concentrating effort where progress is most likely.
Shared Reward Structure
The method employs a shared reward structure across different domains, using one simple reward function for all locomotion tasks and another for all manipulation tasks. This design simplifies the pipeline by avoiding the need to engineer separate reward functions for every unique terrain or assembly task.

Terminology

Summary

The gist: Success Guided Sampling (SGS) is introduced as a simple adaptive sampler that concentrates Reinforcement Learning training on task configurations around the frontier of the policy’s capabilities, enabling large-scale simulated RL to make the most out of experience in a batch

The Problem Addressed

General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation Current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning

Success Guided Sampling (SGS)

SGS is a simple sampler that adapts task-configuration sampling throughout training using non-parametric estimates of the policy’s current success rates The core insight of SGS is that learning complex control requires exposing the policy to the right experience at the right time Learning signal is concentrated in regions of the space that are neither too hard nor too easy under the current policy This is instantiated by weighting success estimates with a beta distribution, which prioritizes configurations with moderate success rates The unnormalized density at pi and the corresponding logit are defined by wi = (pi + ϵ)κt (1 − pi + ϵ)κ(1−t), where wi is the unnormalized density at pi and li the corresponding logit The next configuration is sampled i.i.d. from a softmax over l i with temperature T, defined as p(i) = exp(li/T) PN j=1 exp(lj/T)

Experimental Results and Scaling

Across experiments using up to 2 20 (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve SGS outperforms uniform sampling and Prioritized Level Replay (PLR) [24] on challenging multi-terrain whole-body locomotion and nut-and-bolt assembly, while performing comparably to uniform sampling on simpler tasks At 1M parallel environments, SGS reaches 73% mean success on multi-task locomotion versus 54% for PLR On Franka Panda nut-and-bolt assembly, it reaches 70%, versus 6% for uniform sampling and 5% for PLR Notably, we see monotonic improvement in performance with increased scale, where other methods struggle to benefit or collapse SGS also outperforms Sampling for Learnability (SFL) [47] on locomotion at 4K and 32K environments

Key Methodological Decisions

SGS makes two deliberate algorithmic choices that work together to support training across a wide task configuration space with PPO [30]

  1. Shared rewards within each domain: We use one simple reward function across all locomotion tasks and another across all manipulation tasks This shared-reward design avoids specifying a different reward function for each terrain or assembly task

  2. Scaling parallel environments: SGS addresses this by concentrating rollouts on configurations where the policy is actively learning, so additional environments translate into denser learning signal rather than diminishing returns

Zero-Shot Transfer

We distill learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware This distillation takes about two days on four L40S GPUs with 480 environments each The distilled RGB student achieves approximately 98% success after distillation when using a point-cloud teacher with symmetric resets and rewards

Limitations

We find three primary limitations First, SGS maintains success-rate estimates over a discrete set of K cells; while K = 32,768 is sufficient here, continuous or much larger task spaces would require a learned or hierarchical scoring representation Second, SGS samples from a given distribution but does not construct it: we rely on reset-diversity methods such as OmniReset [12] for broad coverage, and closing the loop on which configurations to make available remains an open problem Third, a sim-to-real transfer gap remains, and closing it to match simulated success rates is left to future work

Conclusion

We present SGS, a simple success-based sampler that focuses training on task configurations with moderate success rates Combined with diverse resets and large-scale RL, SGS solves harder tasks than prior samplers under the same training budget We distill the manipulation policies to RGB and demonstrate zero-shot transfer of three NIST board assembly tasks to UR5e hardware Training policies with SGS enables learning without as much hand-engineered reward specification nor manual curriculum engineering on challenging tasks compared to previous works

How it works

The core insight of SGS is that learning complex control requires exposing the policy to the right experience at the right time We instantiate this by weighting success estimates with a beta distribution, which prioritizes configurations with moderate success rates Before training, we sample a fixed set of N configurations from the task distribution At each episode reset, SGS selects a configuration from this set We maintain the last H Boolean outcomes and estimate its success rate pi as their mean We then score each configuration with a Beta-shaped kernel in mode-concentration form, Beta(1 + κt, 1 + κ(1 − t)), parameterized by a target success rate (i.e. mode) t ∈ [0, 1] and concentration κ ≥ 0 At the end of each episode, the next configuration is sampled i.i.d. from a softmax over l i with temperature T

Task Configuration Details

For locomotion, a task configuration specifies the terrain type and geometry, the robot’s starting pose, and its goal pose For manipulation, a task configuration specifies the assembly task, the initial robot and object states, and the receptacle pose Initial states cover Reaching, Stable Grasp, and Near-Goal resets

Reward Function Structure

The reward r(st, at) is defined as r(st, at) = λtask rtask(st, at) + λreg rreg(st, at) This combines a boolean task success reward obtained at the end of the episode with relatively small robot-specific, per-step regularization terms

Hardware and Control Setup

For UR5e transfer, we use a UR5e arm with a Robotiq 2F-85 gripper and follow OmniReset’s relative Cartesian operational-space control interface [12, Appendices A.3.5–A.3.

Improvements for AI systems

  1. Bold header: Success Guided Sampling (SGS) for efficient exploration. SGS concentrates RL training on task configurations around the frontier of the policy’s capabilities, which allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling. This means that when scaling to millions of parallel environments, learning signal is not wasted on configurations already mastered or unattemptable.

  2. Bold header: Robust zero-shot transfer for dexterous manipulation. The system can distill learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Specifically, the distilled policy achieves 98–99% success on peg insertion when using a point-cloud teacher with symmetric resets and rewards, overcoming the observability gap.

  3. Bold header: Generalization across diverse locomotion terrains. The policy can be trained to traverse a wide range of challenging terrains—including navigation mazes, contoured terrains, and four variations of floating islands—by training a single policy across all types from scratch, unlike prior methods that train per-terrain specialist policies.

  4. Bold header: Reduced engineering burden through shared reward functions. The system avoids the need for task-specific reward tuning by using one simple reward function across all locomotion tasks and another across all manipulation tasks, which addresses the issue where reward functions require expert tuning and frequently induce unintended behaviors.

  5. Bold header: Adaptive sampling for improved learning throughput. By adapting sampling based on success rates, SGS ensures that additional environments translate into denser learning signal rather than diminishing returns, leading to improved learning throughput at scales up to one million parallel environments.

Abstract

General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, we find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial. While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt. This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning. To mitigate this, we introduce Success Guided Sampling (SGS), a simple adaptive sampler that concentrates RL training on task configurations around the frontier of the policy's capabilities. Doing so allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling to large-scale parallel simulation. Across experiments using up to 2 20 (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve. Finally, we distill the learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Project website: https://sgs-rl.github.io/.

Sources

Related papers