GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences
cs.LG
Submitted: 2026-09-27
Updated: 2026-09-29
Code: https://github.com/monaf-chowdhury/gtrl
License: http://creativecommons.org/licenses/by/4.0/
The gist: In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal.
Terminology
Abstract
In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal. However, under stochastic dynamics, the base case of this rule values the luckiest trajectories through the data. The subgoal must also lie on a shared trajectory, so a state-goal pair that no trajectory connects gets no value update at all. To address both, we present Grounded Transitive RL (GTRL), an offline GCRL value learning algorithm that grounds the divide-and-conquer update with a one-step TD target. Over a single step, TD is correct, as its target averages over the successors and needs no subgoal. GTRL adds this target to the composition rather than replacing it, so every pair receives an update, and the composition still carries the long horizon. GTRL also corrects the bias from hindsight relabeling by reweighting each goal against how reachable it was from other successors. We evaluate our algorithm on nineteen OGBench tasks spanning stochastic, deterministic, and stitching environments, where it achieves the highest average success rate. Code will be released soon.
Sources
- Hindsight Experience Replay
- NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
- Unbiased Methods for Multi-Goal Reinforcement Learning
- Learning Successor States and Goal-Dependent Values: A Mathematical Viewpoint
- Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals
- Planning with Diffusion for Flexible Behavior Synthesis
- Offline Reinforcement Learning with Implicit Q-Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Playing Atari with Deep Reinforcement Learning
- Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planning
- OGBench: Benchmarking Offline Goal-Conditioned RL
- Transitive RL: Value Learning via Divide and Conquer
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Rapid Exploration for Open-World Navigation with Latent Goal Models
- Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings
- Behavior Regularized Offline Reinforcement Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks