Breaking the sqrt d Communication Barrier in Federated Sampling with Adaptive Hamiltonian Monte Carlo
stat.ML, cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/laaaarrywang/Adaptive-Federated-HMC
License: http://creativecommons.org/licenses/by/4.0/
The gist: This work proposes the Adaptive Federated Hamiltonian Monte Carlo (AFHMC) for Bayesian federated learning (FL) tasks.
Terminology
Abstract
This work proposes the Adaptive Federated Hamiltonian Monte Carlo (AFHMC) for Bayesian federated learning (FL) tasks. The prior work on HMC applications for FL achieves O(sqrt d /ε) communication cost under log-convex distributions, already demonstrating its advantage over the Langevin dynamic-based counterpart. Based on a refined analysis, this paper further reveals the benefit of HMC for the FL regime by incorporating a control mechanism for local node shifts. We prove that AFHMC at most requires O(d/ε 2 1/3) communication cost up to a logarithmic term, which is significantly better than the existing results. Moreover, the proposed method adapts to client heterogeneity. Under low-heterogeneity settings (i.e., the target sampling precision ε squared is higher than the heterogeneity level), the communication rate of AFHMC can be as good as O((d/ε 2)), resembling the communication costs for FL optimization tools. To the best of our knowledge, this provides the first logarithmic communication complexity result for federated sampling, even though it requires low heterogeneity. Numerically, we also justify AFHMC's advantage under the low-heterogeneity regime. Our code is available at https://github.com/laaaarrywang/Adaptive-Federated-HMC
Sources
- Federated Optimization:Distributed Optimization Beyond the Datacenter
- Federated Learning: Strategies for Improving Communication Efficiency
- Rapid Mixing of Hamiltonian Monte Carlo on Strongly Log-Concave Distributions
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference