Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
cs.LG
Submitted: 2026-05-15
Updated: 2026-05-18
Comments: Accepted by ICML 2026 Position Paper Track as a Spotlight Paper
Journal ref: Proceedings of Machine Learning Research, 306:171315-171329, 2026 (ICML)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation…
Terminology
Abstract
Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.
Sources
- Gradients without Backpropagation
- Optimization without Backpropagation
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Online convex optimization in the bandit setting: gradient descent without a gradient
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
- The Forward-Forward Algorithm: Some Preliminary Investigations
- The Universal Weight Subspace Hypothesis
- Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization
- The Multi-Query Paradox in Zeroth-Order Optimization
- Muon is Scalable for LLM Training
- Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning
- End-to-End Mesh Optimization of a Hybrid Deep Learning Black-Box PDE Solver
- Zero Bubble Pipeline Parallelism
- Zeroth-Order Optimization is Secretly Single-Step Policy Optimization
- Align, then memorise: the dynamics of learning with feedback alignment
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Gemma 2: Improving Open Language Models at a Practical Size
- Biologically-plausible learning algorithms can scale to large datasets
- Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis