V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
Donghu Kim, Youngdo Lee, Hojoon Lee, Johan Obando-Ceron, Byungkun Lee, Aaron Courville, Pablo Samuel Castro, Jaegul Choo, Clare Lyle
cs.LG, cs.RO
Submitted: 2026-08-08
Updated: 2026-08-11
Comments: Accepted at RLC'26
Code: https://github.com/DAVIAN-Robotics/V-Simba
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Improving sample efficiency remains a core challenge in reinforcement learning (RL), especially in real-world settings like robotics, where data collection is costly.
Terminology
Abstract
Improving sample efficiency remains a core challenge in reinforcement learning (RL), especially in real-world settings like robotics, where data collection is costly. This challenge is pronounced in visual RL, where high-dimensional inputs often obscure learning signals. While prior work in visual RL has focused on algorithmic solutions, such as better dynamics models or exploration strategies, recent advances in state-based RL show that architectural design alone can lead to significant gains in sample efficiency. This raises an important question: Can these architectural principles transfer to visual RL? In response, we introduce V-Simba, a simple yet effective visual RL architecture inspired by the Simba architecture from state-based RL. Built on top of Soft Actor-Critic (SAC) with data augmentation, V-Simba modifies the architecture by adding normalization layers to stabilize training and using pointwise convolutions to reduce computation. Despite its simplicity, V-Simba matches or outperforms the state-of-the-art methods across the DMC, Adroit, and Meta-World benchmarks, while being more computationally efficient than DrQ-v2. We make our code publicly available at https://github.com/DAVIAN-Robotics/V-Simba.
Sources
- Exploration by Random Network Distillation
- Stabilizing Off-Policy Deep Reinforcement Learning from Pixels
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- A Survey of State Representation Learning for Deep Reinforcement Learning
- Deep Transformer Q-Networks for Partially Observable Reinforcement Learning
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
- Sharpness-Aware Minimization for Efficiently Improving Generalization
- Towards General-Purpose Model-Free Reinforcement Learning
- World Models
- Mastering Diverse Domains through World Models
- FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff
- TD-MPC2: Scalable, Robust World Models for Continuous Control
- A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
- Hadamax Encoding: Elevating Performance in Model-Free Atari
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Dual PatchNorm
- SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
- Hyperspherical Normalization for Scalable Deep Reinforcement Learning
- Continuous control with deep reinforcement learning
- TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks