Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization
cs.LG
Submitted: 2026-02-19
Updated: 2026-10-28
Code: https://github.com/OPTML-Group/ZO-Muon
License: http://creativecommons.org/licenses/by/4.0/
The gist: Zeroth-order (ZO) optimization provides a gradient-free alternative to first-order (FO) methods by estimating gradients via finite differences of function evaluations…
Terminology
Abstract
Zeroth-order (ZO) optimization provides a gradient-free alternative to first-order (FO) methods by estimating gradients via finite differences of function evaluations, and has recently emerged as a memory-efficient paradigm for fine-tuning large-scale models by avoiding backpropagation. However, ZO optimization has a fundamental tension between accuracy and query efficiency. In this work, we show that ZO optimization can be substantially improved by unifying two complementary principles: (i) a projection-based subspace view that reduces gradient estimation variance by exploiting the intrinsic low-rank structure of model updates, and (ii) Muon-style spectral optimization that applies gradient orthogonalization to extract informative spectral structure from noisy ZO gradients. These findings form a unified framework of subspace gradient orthogonalization, which we instantiate in a new method, ZO-Muon, admitting a natural interpretation as a low-rank Muon optimizer in the ZO setting. Extensive experiments on large language models (LLMs) and vision transformers (ViTs) demonstrate that ZO-Muon significantly accelerates convergence and achieves a win-win improvement in accuracy and query/runtime efficiency. Notably, compared to the popular MeZO baseline, ZO-Muon requires only 24.7% of the queries to reach the same SST-2 performance for LLM fine-tuning, and improves accuracy by 25.1% on ViT-L fine-tuning on CIFAR-100.
Sources
- GPT-4 Technical Report
- OPT: Open Pre-trained Transformer Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Adam: A Method for Stochastic Optimization
- DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training
- Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
- The Multi-Query Paradox in Zeroth-Order Optimization
- FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
- Muon is Scalable for LLM Training
- The Universal Weight Subspace Hypothesis
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
- Subspace Optimization for Large Language Models with Convergence Guarantees
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining
- Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)
- MuonBP: Faster Muon via Block-Periodic Orthogonalization
- Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
- Dion: Distributed Orthonormalized Updates
- The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
- Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
- The Llama 3 Herd of Models
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning