Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control

State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks <ref:2603.08588#pg2> The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks <ref:2603.08588#pg3> In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of-the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer <ref:2603.08588#pg2> Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious per-environment hyperparameter tuning <ref:2603.08588#pg2>.

Algorithm Design and Compatibility

The proposed algorithms, S2AC and SDAC, are designed to adapt SAC and TD3 for the streaming setting, taking inspiration from the work of Elsayed et al. [6] <ref:2603.08588#pg3> Both methods adopt sparse network initialization, LayerNorm [2] applied to the pre-activations of each layer, and the state normalization and reward scaling schemes of Andrychowicz et al. [1] and Engstrom et al. [7] <ref:2603.08588#pg2>. For S2AC, the critic is updated at each time step by minimizing the instantaneous squared soft Bellman residual <ref:2603.08588#pg2> The actor is updated without eligibility traces, employing the Adam optimizer [17], similarly to batch deep RL algorithms <ref:2603.08588#pg2>. SDAC considers a deterministic policy where actions are augmented with Gaussian noise as at = πθ(st) + ϵ1 <ref:2603.08588#pg2>.

Training Signal Construction

The training signal for S2AC involves minimizing the instantaneous squared soft Bellman residual, which uses a single next transition (st+1, at+1) to compute the target <ref:2603.08588#pg2> The policy improvement objective for S2AC trades off expected return maximization and entropy regularization through the temperature parameter α <ref:2603.08588#pg2>. For SDAC, the critic updates employ ObGD with traces to minimize the instantaneous squared residual, which includes additional noise ϵ2 ∼ N (0, σ2I) to mitigate the risk of critic overfitting to narrow peaks in the value estimate <ref:2603.08588#pg2>.

Performance Gains and Ablations

Both S2AC and SDAC achieve performance comparable to Stream AC(λ) on standard benchmarks <ref:2603.08588#pg2>. The results show that S2AC does not require per-environment optimizer, learning rates, and discount factor tuning <ref:2603.08588#pg2>. For SDAC, the target noise modification is far more critical for SDAC, which fails to learn entirely without it <ref:2603.08588#pg2>. Ablations confirm that observation and reward normalization provide a consistent benefit both for S2AC and SDAC <ref:2603.08588#pg2>.

Batch-to-Streaming Transition

The investigation into the batch-to-streaming transition shows that a naive transition does not guarantee the preservation of the pre-trained policy performance <ref:2603.08588#pg2> A principled approach is proposed to address this challenge <ref:2603.08588#pg2>. This involves replacing Adam with SGDC for the critic during pre-training, motivated by its close resemblance to ObGD, as shown by the lower L2-norm of the critic weights in SDAC <ref:2603.08588#pg2>.

Conclusion and Limitations

This work represents a first step toward bridging batch and streaming deep reinforcement learning, introducing a paradigm with potential for real-world deployment scenarios such as Sim2Real finetuning <ref:2603.08588#pg2>. Several challenges remain, including SDAC finetuning proves ineffective on quadruped-run-v0, the adoption of SGDC for critic pre-training reduces sample efficiency in a few cases, and the batch-to-streaming transition for S2AC remains an open problem <ref:2603.08588#pg2>. The findings suggest that batch and streaming algorithms need not be treated as independent paradigms, but may instead benefit from a shared algorithmic foundation <ref:2603.08588#pg2>.

Pruning redundant reasoning tokens

The state normalization and reward scaling subroutines are detailed in Appendix A <ref:2603.08588#pg2>. The pseudocodes for S2AC and SDAC are reported in Algorithm 5 and Algorithm 6, respectively <ref:2603.08588#pg2>. The network architectures for each streaming algorithm are presented in Pseudocode 1, Pseudocode 2 and Pseudocode 3 <ref:2603.08588#pg2>.

Where the gains come from

The performance of finetuning SDAC after pre-training with TD3-norm using SGDC as the critic optimizer is shown in Figure 5 <ref:2603.08588#pg2>. The results suggest that initializing SDAC from a pre-trained policy consistently accelerates learning and yields final performance that surpasses what is achievable by SDAC trained from scratch, with higher sample efficiency <ref:2603.08588#pg2>. The paper provides pseudocode for S2AC and SDAC in Algorithm 5 and Algorithm 6 <ref:2603.08588#pg2>.

Hyperparameters for SAC and TD3

The hyperparameters for SAC and TD3 follow those of the original works of Haarnoja et al. [13] and Fujimoto et al. [11] <ref:2603.08588#pg3> The network architectures are reported in Pseudocode 4 and Pseudocode 5, where the hidden layer size is reduced from 256 to 128 to match the capacity of the streaming approaches <ref:2603.08588#pg2>.

Additional results for streaming RL

Figure 7 presents additional results on different environments from DM Control Suite and MuJoCo Gym <ref:2603.08588#pg2>. The results reveal a highly inconsistent performance profile for AVG: while it achieves strong results on some environments such as dog-walk-v0, it performs poorly on others such as Ant-v4 <ref:2603.08588#pg2>.

Comparison with AVG

S2AC requires no tuning of these hyperparameters, yet achieves comparable overall performance across the full range of evaluated environments <ref:2603.08588#pg2>. The results, reported in Figure 8, reveal a highly inconsistent performance profile for AVG: while it achieves strong results on some environments such as dog-walk-v0, it performs poorly on others such as Ant-v4 <ref:2603.08588#pg2>.

Finetuning experimental setting

In the Sim2Real scenario, we derive two perturbed variants of the DM Control Suite [33] environment walker-run-v0, denoted as walker-run-v0-p1 and walker-run-v0-p2, and one perturbed variant of dog-walk-v0, denoted dog walk v0 p <ref:2603.08588#pg2>. All experiments concern the finetuning by use of SDAC from a given pre-trained policy obtained with TD3 with state normalization and reward scaling <ref:2603.08588#pg2>.

Connecting ObGD and SGDC

We propose to replace Adam with SGDC [28] for the critic during pre-training, motivated by its close resemblance to ObGD <ref:2603.08588#pg2> This connection is formalized by showing that the update in Equation 7 reduces to that of ObGD under certain assumptions <ref:2603.08588#pg2>.

Final Summary

This work represents a first step toward bridging batch and streaming deep reinforcement learning, introducing a paradigm with potential for real-world deployment scenarios such as Sim2Real finetuning <ref:2603.08588#pg2>. We developed two novel streaming DRL algorithms, S2AC and SDAC, which achieve competitive performance against state-of-the-art methods while maintaining architectural compatibility with established batch algorithms such as SAC and TD3 <ref:2603.08588#pg2>. Building on this foundation, we investigated the practical challenges that emerge when transitioning from batch to streaming learning during the finetuning phase, and proposed an initial framework for addressing them <ref:2603.08588#pg2>.

References

[1] Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990 <ref:2603.08588#pg2>

[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 <ref:2603.08588#pg2>

[3] Shuxiao Chen, Bike Zhang, Mark W Mueller, Akshara Rai, and Koushil Sreenath. Learning torque control for quadrupedal locomotion. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1–8. IEEE, 2023 <ref:2603.08588#pg2>

[4] Sebastien D De Rivaz, Benjamin Goldberg, Neel Doshi, Kaushik Jayaram, Jack Zhou, and Robert J Wood. Inverted and vertical climbing of a quadrupedal microrobot using electroadhesion. Science Robotics, 3(25):eaau3038, 2018 <ref:2603.08588#pg2>

[5] Bardienus P Duisterhof, Shushuai Li, Javier Burgués, Vijay Janapa Reddi, and Guido CHE De Croon. Sniffy bug: A fully autonomous swarm of gas-seeking nano quadcopters in cluttered environments. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9099–9106. IEEE, 2021 <ref:2603.08588#pg2>

[6] Mohamed Elsayed, Gautham Vasan, and A Rupam Mahmood. Streaming deep reinforcement learning finally works. arXiv preprint arXiv:2410.14606 <ref:2603.08588#pg2>

[7] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729 <ref:2603.08588#pg2>

[8] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In 10th International Conference on Learning Representations, ICLR 2022, 2022 <ref:2603.08588#pg2>

[9] Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36:61573–61624, 2023 <ref:2603.08588#pg2>

[10] Scott Fujimoto, Pierluca D’Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. In The Thirteenth International Conference on Learning Representations <ref:2603.08588#pg2>

[11] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018 <ref:2603.08588#pg2>

[12] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352–1361. PMLR, 2017 <ref:2603.08588#pg2>

[13] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 <ref:2603.08588#pg2>

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016 <ref:2603.08588#pg2>

[15] Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson et al. Sim2real in robotics and automation: Applications and challenges. IEEE transactions on automation science and engineering, 18(2):398–400, 2021 <ref:2603.08588#pg2>

[16] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty et al. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022 <ref:2603.08588#pg2>

[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014 <ref:2603.08588#pg2>

[18] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999 <ref:2603.08588#pg2>

[19] Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai et al. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754 <ref:2603.08588#pg2>

[20] Timothy Paul Lillicrap, Jonathan James Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess et al. Continuous control with deep reinforcement learning, September 15 2020. US Patent 10,776,692 <ref:2603.08588#pg2>

[21] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires et al. Understanding plasticity in neural networks. In International Conference on Machine Learning, pages 23190–23211. PMLR, 2023 <ref:2603.08588#pg2>

[25] Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu et al. Optimizers qualitatively alter solutions and we should leverage this. arXiv preprint arXiv:2507.12224 <ref:2603.08588#pg2>

[26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 <ref:2603.08588#pg2>

[27] David Silver, Guy Lever, Nicolas Heess et al. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387–395. Pmlr, 2014 <ref:2603.08588#pg2>

[29] Richard S Sutton and Andrew G Barto. Toward a modern theory of adaptive networks: expectation and prediction. Psychological review, 88(2):135, 1981 <ref:2603.08588#pg2>

[30] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 1998 <ref:2603.08588#pg2>

[31] Richard S Sutton, David McAllester, Satinder Singh et al. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999 <ref:2603.08588#pg2>

[32] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012 <ref:2603.08588#pg2>

[33] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron et al. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020 <ref:2603.08588#pg2>

[34] Gautham Vasan, Mohamed Elsayed, Seyed Alireza Azimi et al. Deep policy gradient methods without batch updates, target networks, or replay buffers. Advances in Neural Information Processing Systems, 37:845–891 <ref:2603.08588#pg2>

[35] Yan Wang, Gautham Vasan, and A Rupam Mahmood. Real-time reinforcement learning for vision-based robotics utilizing local and remote computers. arXiv preprint arXiv:2210.02317, 2022 <ref:2603.08588#pg2>

[36] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279–292, 1992 <ref:2603.08588#pg2>

[37] Barry Payne Welford. Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3):419–420, 1962 <ref:2603.08588#pg2>

[38] Mingdong Wu, Lehong Wu, Yizhuo Wu et al. Simlauncher: Launching sample-efficient real-world robotic reinforcement learning via simulation pre-training. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7933–7940. IEEE, 2025 <ref:2603.08588#pg2>

[39] Patrick Yin, Tyler Westenbroek, Simran Bagaria et al. Rapidly adapting policies to the real world via simulationguided fine-tuning. arXiv preprint arXiv:2502.02705 <ref:2603.

1. Batch to Streaming Transition Framework: Implement a principled approach to address the challenge that a naive transition does not guarantee preservation of pre-trained policy performance. This framework should allow for controlled adaptation when moving from batch pre-training (like TD3) to streaming finetuning (SDAC), ensuring performance is preserved or predictably degraded.

2. State Normalization and Reward Scaling Integration: Integrate the state normalization and reward scaling schemes into batch RL methods like SAC and TD3 during pre-training, as shown in Figure 3. This will yield notable improvements across many environments, particularly for TD3, leading to better sample efficiency when finetuning is later applied.

3. Deterministic Policy Exploration: For the SDAC algorithm, ensure the exploration noise injection is carefully managed by incorporating target noise N into the update rule, as shown in Equation 5. This modification is critical because SDAC fails to learn entirely without it.

4. Entropy Coefficient Adaptation: Implement dynamic scaling of the entropy coefficient using "alpha -> alpha/sigma r in S2AC. This correction ensures that the relative weighting between the reward and entropy terms remains consistent throughout training, regardless of the current reward statistics," preventing undesirable behavior when rewards are normalized.

5. Optimizer Selection for Policy Improvement: For finetuning SDAC from a pre-trained policy, replace Adam with SGDC for the critic optimizer because it closely resembles ObGD, which supports the hypothesis that the two optimizers induce different solution geometries. This is expected to lead to better final performance in perturbed environments.

6. Critic Warm-up Phase: Incorporate a critic warm-up phase where the policy is frozen and only the critic is updated during finetuning of SDAC. This step has been shown to contribute to more stable learning during finetuning, mitigating initial performance drops observed in direct transitions from pre-trained batch methods.

7. Architectural Consistency for Transfer: Utilize network architectures consistent with established batch algorithms, specifically using the architectures proposed by Elsayed et al. [6] for both S2AC and SDAC, to ensure architectural and methodological modifications introduced to enable streaming learning also yield notable performance gains.

8. Performance in Perturbed Environments: The system should be capable of surpassing the final performance of models trained from scratch in perturbed environments, as evidenced by Figure 17. This suggests that finetuning SDAC from a TD3-norm pre-trained checkpoint can surpass what is achievable by SDAC trained from scratch.

State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of-the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious per-environment hyperparameter tuning. We further investigate the batch-to-streaming transition, showing that a naive transition does not guarantee preservation of pre-trained policy performance, and propose a principled approach to address this challenge.