FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA
Lele Zheng, Ruijie Hu, Tao Zhang, Ke Cheng, Yulong Shen
cs.CR
Submitted: 2026-08-04
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models.
Terminology
Abstract
Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under epsilon=6 and epsilon=3, respectively.
Sources
- Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Federated Low-Rank Adaptation with Differential Privacy over Wireless Networks
- Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning
- FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
- Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Natural Language Understanding with the Quora Question Pairs Dataset
- Improving LoRA in Privacy-preserving Federated Learning
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix
- FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills