Differentiated Aggregation to Improve Generalization in Federated Learning
cs.LG, cs.AI
Submitted: 2024-04-17
Updated: 2026-08-25
Comments: 25 pages. Published in Transactions on Machine Learning Research (TMLR), 2025
Journal ref: Transactions on Machine Learning Research, 2025
License: http://creativecommons.org/licenses/by/4.0/
The gist: This paper focuses on reducing the communication cost of federated learning by exploring generalization bounds and representation learning.
Terminology
Abstract
This paper focuses on reducing the communication cost of federated learning by exploring generalization bounds and representation learning. We first characterize a tighter generalization bound for one-round federated learning based on local clients' generalizations and heterogeneity of data distribution (non-iid scenario). We also characterize a generalization bound in R-round federated learning and its relation to the number of local updates (local stochastic gradient descents (SGDs)). Then, based on our generalization bound analysis and its interpretation through representation learning, we infer that less frequent aggregations for the representation extractor (typically corresponds to initial layers) compared to the head (usually the final layers) leads to the creation of more generalizable models, particularly in non-iid scenarios. We design a novel Federated Learning with Adaptive Local Steps (FedALS) algorithm based on our generalization bound and representation learning analysis. FedALS employs varying aggregation frequencies for different parts of the model, so reduces the communication cost. The paper is followed with experimental results showing the effectiveness of FedALS. Our codes are available at for reproducibility.
Sources
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Federated Learning: Strategies for Improving Communication Efficiency
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Decentralized Gradient Tracking with Local Steps
- Agnostic Federated Learning
- Local SGD Converges Fast and Communicates Little
- The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Minibatch vs Local SGD for Heterogeneous Distributed Learning
- Is Local SGD Better than Minibatch SGD?
- Salvaging Federated Learning by Local Adaptation
- OPT: Open Pre-trained Transformer Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks