ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
cs.LG, math.OC, stat.ML
Submitted: 2026-10-30
Updated: 2026-10-30
Code: https://github.com/meta-llama/llama-models
Project page: https://kellerjordan.github.io
Terminology
Sources
- LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
- Program Synthesis with Large Language Models
- LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
- Evaluating Large Language Models Trained on Code
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- The Llama 3 Herd of Models
- LoRA+: Efficient Low Rank Adaptation of Large Models
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
- StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
- PoLAR: Polar-Decomposed Low-Rank Adapter Representation
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- Muon is Scalable for LLM Training
- DoRA: Weight-Decomposed Low-Rank Adaptation
- Can Muon Fine-tune Adam-Pretrained Models?
- Qwen2.5 Technical Report
- Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
- AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
- Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis