TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
cs.LG, cs.AI, eess.SP
Submitted: 2026-09-27
Updated: 2026-09-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Fine-tuning anguage models (LLMs) with first-order optimizers requires a memory several times larger than that required for inference.
Terminology
Abstract
Fine-tuning anguage models (LLMs) with first-order optimizers requires a memory several times larger than that required for inference. Memory-efficient zeroth-order optimization (MeZO) sidesteps this cost by estimating gradients from forward passes only. However, for BitNet architectures, a family of LLMs with ternary-1,0,1 weights and 8-bit activations, fine-tuning requires updating full-precision latent weights, and thus the memory footprint of MeZO no longer matches that of inference. A promising solution is to finetune only a subset of the latent weights, but existing sparse zeroth-order (ZO) methods either ignore the ternary structure or require first-order gradient information to build a sparse mask, which is at odds with the purpose of ZO fine-tuning. We propose TerMeZO, a sparse MeZO scheme that exploits the geometry of the ternary quantizer itself to identify the latent weights that are more likely to change values during fine-tuning, at no additional data or memory cost. Our convergence analysis shows that TerMeZO can converge faster than full-parameter MeZO, owing to its optimized reduction of the fine-tuning effective dimension. We run extensive experiments on BitNet models ranging from 1B to 3B parameters, spanning classification, instruction-following, and mathematical reasoning tasks. TerMeZO matches or exceeds the performance of full-parameter MeZO while substantially reducing the fine-tuning memory footprint.
Sources
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- On Adaptivity in Zeroth-Order Optimization
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
- Ternary Weight Networks
- Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
- The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
- BitNet b1.58 2B4T Technical Report
- Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
- A White Paper on Neural Network Quantization
- Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
- BitNet: Scaling 1-bit Transformers for Large Language Models
- Magicoder: Empowering Code Generation with OSS-Instruct
- BitNet Distillation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks