Stochastic Projections and Rank-Aware Learning Rates for Federated LoRA
cs.LG, cs.AI
Submitted: 2026-11-09
Updated: 2026-11-13
Code: https://github.com/Saber-mm/SPRA-LoRA
Terminology
Sources
- GPT-4 Technical Report
- LoRA: Low-Rank Adaptation of Large Language Models
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
- A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
- SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models
- FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition
- Federated LoRA with Sparse Communication
- Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs
- LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
- Adam: A Method for Stochastic Optimization
- Decoupled Weight Decay Regularization
- Noise-Aware Algorithm for Heterogeneous Differentially Private Federated Learning
- The Llama 3 Herd of Models
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning