Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
cs.LG, cs.AI
Submitted: 2026-08-25
Updated: 2026-08-25
License: http://creativecommons.org/licenses/by/4.0/
The gist: Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend.
Terminology
Abstract
Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend. Existing evaluations are often tied to custom infrastructure, released as incomplete research code, and conducted primarily in simulation, which limits portability and practical relevance. We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications. We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation workflows. We instantiate this approach with a multi-domain benchmark suite spanning cross-silo and cross-device settings, and including tasks in medical imaging, financial tabular learning, legal instruction tuning, phishing URL detection, and audio tagging. We further demonstrate that the same benchmarking application can run across both simulation and deployment runtimes without changing the application code, enabling unified evaluation across varying learning environments. Beyond model quality, our benchmark design supports system-aware reporting, including runtime and communication metrics. This work advances benchmarking in FL settings from ad hoc code artifacts towards portable, executable, and reusable benchmark applications.
Sources
- FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things
- Flower: A Friendly Federated Learning Research Framework
- LEAF: A Benchmark for Federated Settings
- FedEval: A Holistic Evaluation Framework for Federated Learning
- Automatic tagging using deep convolutional neural networks
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
- FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations
- FedML: A Research Library and Benchmark for Federated Machine Learning
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection
- UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworks
- Federated Learning: Opportunities and Challenges
- Adaptive Federated Optimization
- OpenFL: An open-source framework for Federated Learning
- NVIDIA FLARE: Federated Learning from Simulation to Real-World
- Capabilities of Gemini Models in Medicine
- FederatedScope: A Flexible Federated Learning Platform for Heterogeneity
- DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services
- Federated Learning with Non-IID Data
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks