1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
cs.LG, cs.AI
Submitted: 2025-03-19
Updated: 2026-10-05
Comments: Published at NeurIPS 2025, Best Paper Award. Link to project website: https://wang-kevin3290.github.io/scaling-crl/
Project page: https://wang-kevin3290.github.io/scaling-crl
License: http://creativecommons.org/licenses/by/4.0/
The gist: Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL).
Terminology
Abstract
Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by 2 times - 50 times, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.
Sources
- Layer Normalization
- Accelerating Goal-Conditioned RL Algorithms and Research
- Emerging Properties in Self-Supervised Vision Transformers
- The Llama 3 Herd of Models
- Understanding and Preventing Capacity Loss in Reinforcement Learning
- Disentangling the Causes of Plasticity Loss in Neural Networks
- Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
- Scaling Laws for a Multi-Agent Reinforcement Learning Model
- Training Larger Networks for Deep Reinforcement Learning
- OGBench: Benchmarking Offline Goal-Conditioned RL
- JaxMARL: Multi-Agent RL Environments and Algorithms in JAX
- Human-Timescale Adaptation in an Open-Ended Task Space
- Deep Reinforcement Learning and the Deadly Triad
- Residual Networks Behave Like Ensembles of Relatively Shallow Networks
- How Does Critical Batch Size Scale in Pre-training?
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks