Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning

The gist The SCOUT controller introduces an online, learner-agnostic reset controller that gives every context its own curriculum, which addresses the failure of sparse-reward reinforcement learning when rollouts rarely reach later task stages.

How it works

SCOUT operates by maintaining a frontier for each context, which is the current rung of its scaffold ladder. The controller observes only whether each rollout succeeded, so it works with different RL algorithms without changing their reward or optimizer. The controller moves the frontier based on binary success: sustained success moves the frontier toward the unassisted target start, failure restores assistance, and it cautiously tests a harder start when progress stalls.

The controller's active cell sampling uses a score defined by a softmax distribution that favors cells near intermediate competence, transfer-directed terms favoring less assisted cells, and staleness counters to increase coverage of recently unvisited cells. The frontier transition rule dictates movement: high success removes assistance, low success restores it. A third case is the antistall transition, which encourages cautious progress without preventing recovery when a context is neither unsuccessful enough to retreat nor successful enough to advance.

Key findings on scaffold access and allocation

Scaffold access improves learning across six navigation and manipulation settings and enables success in three where target-start training does not succeed within the reported budget. The experiments separate access from allocation, showing that scaffold access improves learning across six natural settings.

No allocation rule is uniformly best when contexts progress at similar rates; allocation matters when progress differs. SCOUT succeeds on both groups in a constructed manipulation conflict, while each tested global schedule fails on one. The counting construction shows that synchronized global pacing can fail when contexts need conflicting amounts of assisted practice.

Pacing granularity and structural limits

The paper identifies reset curricula as a granularity-matching allocation problem, where assistance should be paced at the scale where learning progress differs. A counting construction shows that synchronized global schedules can fail when contexts require different amounts of assisted practice. The structural limit of synchronized pacing is demonstrated by a two-context, two-level scaffold curriculum where a context-local schedule solves both contexts, while every synchronized global schedule fails at least one context.

Experimental results and comparisons

SCOUT achieves higher held-out target-start AUC than the target-start baseline across all six natural settings, with every paired-seed difference positive. Per-context allocation improves performance under pacing conflict, improving median worst-group AUC by nearly 3×. SCOUT maintains shelf-place success at 0.52–0.68 and wins the paired worst-group comparison on all six seeds in the Meta-World conflict.

Trajectory-derived scaffold construction

The paper constructs scaffolds automatically from one successful reference trajectory per training context, defining a scalar progress coordinate p(t) and serializing simulator states nearest each of five fixed progress quantiles. This construction does not use semantic stage labels, and the environment restores the selected state at reset. The retained gain ratio Rretain is the fraction of the hand-built scaffold’s gain recovered by the trajectory-derived scaffold.

The need for allocation depends on the base learner; a contrastive critic removes bottlenecks related to access, but reset curricula help when access, rather than the base learner, limits learning. The trajectory-derived scaffolds retain the hand-built benefit in all three settings.

Conclusion

Reset curricula for sparse-reward learning make two decisions: which intermediate states to train from, and how quickly to remove their assistance. SCOUT adapts each context’s pace from binary rollout success alone. Curriculum pace should match the scale at which learning progress differs. The method does not apply without reset access, as on physical robots without instrumented resets.

--- Page 1 ---

SCOUT: Per-Context Reset Curricula for Sparse-Reward Reinforcement Learning

Siddharth Aphale1, Ayushman Singh2 1Stanford University 2Sesame AI saphale@stanford.edu, ayushman@sesame.com

Abstract

Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reset curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. Such a curriculum faces two decisions: scaffold access, obtaining informative starts, and scaffold allocation, deciding how quickly that assistance is removed. Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce SCOUT, an online, learner-agnostic reset controller that gives every context its own curriculum. Using only binary rollout success, SCOUT removes assistance after sustained success, restores it after failure, and cautiously tests a harder start when progress stalls, without changing the reward, optimizer, or learner. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. In a constructed pacing conflict, each tested global schedule leaves one group unsolved, while SCOUT solves both. Average success can conceal this failure, so we also report the least successful group. Group-level pacing works when learning differences follow known groups but can fail when they occur within one group. SCOUT needs no group labels and remains consistently strong in both cases. A reset curriculum should remove assistance at the scale where learning progress differs.

Introduction

An agent cannot learn the later stages of a task if it almost never reaches them. A reward given only for the final placement provides little guidance to a policy that cannot yet grasp the object in a long-horizon task such as picking it up and placing it on a shelf. Hindsight relabeling can reuse states that the agent visits, but it cannot create experience in stages the agent never reaches (Andrychowicz et al. 2017). Training can instead begin partway through the task in simulators that support state restoration. We study how to choose these training starts so that agents learn long-horizon, sparse-reward tasks from the unassisted starts used for evaluation.

We call a temporary, easier training start a scaffold. A grasped-object state, for example, lets the agent practice placement before it can reliably perform the preceding grasp. Several ordered scaffolds form a ladder from an easy state near success back to the unassisted evaluation start, the target start. A reset curriculum chooses where on this ladder each training rollout begins and gradually removes the assistance (Florensa et al. 2017, 2018).

1. Bold header: Per-Context Curriculum Adaptation

SCOUT implements a reset controller that makes this pacing decision separately for every context, allowing each task instance to receive a tailored training start and removal schedule (SCOUT keeps a frontier for each context). This enables the system to adapt assistance based on individual learning rates, addressing the failure of synchronized global pacing when contexts need conflicting amounts of assisted practice.

2. Bold header: Dynamic Assistance Management

The controller uses binary rollout success to manage assistance: it removes assistance after sustained success, restores it after failure, and cautiously tests a harder start when progress stalls. This mechanism ensures that learning progresses efficiently by moving the context's current rung of its scaffold ladder based on evidence-aware transitions.

3. Bold header: Robustness to Within-Group Heterogeneity

SCOUT is designed to handle complex learning dynamics where contexts within the same group have differing rates, as it needs no group labels and remains consistently strong in both cases. This capability is demonstrated by its superior performance over methods like Pooled frontier (oracle) when heterogeneity occurs within one group, showing that the aggregate AUC can conceal this failure of simpler methods.

4. Bold header: Scaffold Construction Flexibility

The system supports flexible scaffold construction, allowing for both hand-built scaffolds and automatic generation from successful trajectories (Trajectory-derived scaffold construction). This allows researchers to leverage prior knowledge or policy competence by using the retained gain ratio Rretain to validate whether an automatic scaffold matches a manually designed one.

5. Bold header: Granularity Matching for Pacing

The core principle is that assistance should be paced at the scale where learning progress differs, as stated in the contribution: A reset curriculum should remove assistance at the scale where learning progress differs. SCOUT achieves this by operating on a frontier that advances or retreats based on observed success statistics.

Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reverse curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. The main choices are which intermediate starts to use (scaffold access) and how to distribute training across them (scaffold allocation). Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce an online, learner-agnostic reset controller that gives every context its own curriculum. We use binary rollout success to decide whether to move a context's start forward or back. A stalled frontier also gets an occasional harder-start trial. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. None of the global schedules we tested solved both groups in the constructed conflict. The per-context controller did. Group-level pacing works if the labels capture how contexts learn. Without those labels, per-context pacing still handles differences within a group. These results suggest pacing assistance separately for contexts that learn at different rates.