Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning
cs.LG, math.OC
Submitted: 2026-11-07
Updated: 2026-11-07
Terminology
Sources
- GPT-4 Technical Report
- LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
- FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
- Riemannian Accelerated Zeroth-order Algorithm: Improved Robustness and Lower Query Complexity
- Zeroth-order Riemannian Averaging Stochastic Approximation Algorithms
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
- Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
- AROMA: Autonomous Rank-one Matrix Adaptation
- CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
- Low-Rank Adaptation for Foundation Models: A Comprehensive Review
- OPT: Open Pre-trained Transformer Language Models
- Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis