Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

arXiv:2605.15622 · cs.LG · Submitted 2026-05-15 · Read on arXiv

cs.LG

Submitted: 2026-05-15

Updated: 2026-05-18

Comments: Accepted by ICML 2026 Position Paper Track as a Spotlight Paper

Journal ref: Proceedings of Machine Learning Research, 306:171315-171329, 2026 (ICML)

License: http://creativecommons.org/licenses/by/4.0/

The gist: Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation…

Terminology

Abstract

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.

Sources

Related papers