Protocol Choices Matter More Than the Aggregation Algorithm: Federated Learning for In-Hospital Mortality Prediction in Care-Unit and Multi-Hospital Federations

arXiv:2610.17265 · cs.LG, cs.CY, cs.DC · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Protocol Choices Matter More Than the Aggregation Algorithm".

Jane: The gist: This study compared seven aggregation strategies for federated learning on in-hospital mortality prediction under controlled protocols, revealing that protocol choices often matter more than the aggregation algorithm itself.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper, "Protocol Choices Matter More Than the Aggregation Algorithm: Federated Learning for In-Hospital Mortality Prediction in Care-Unit and Multi-Hospital Federations." It basically tackles a problem where hospitals want to train a shared mortality model without sharing their actual patient records, but they don't agree on how to evaluate the results.

Jane: That sounds like a real headache for anyone trying to get this kind of medical AI deployed. So what is the main point this paper is making about this situation?

Tom: The core claim here is that published comparisons of aggregation strategies rarely control the evaluation protocol, and they're showing how much those protocol choices actually matter. They compared seven different aggregation strategies—FedAvg, FedProx, SCAFFOLD, FedAdagrad, FedAdam, FedYogi and FedCluster—on two different setups: a small intra-hospital federation with twelve care units from MIMIC-IV and a larger multi-hospital federation with forty-nine hospitals from eICU-CRD <ref:2610.17265#pg1,strategies—FedAvg, FedProx, SCAFFOLD, FedAdagrad, FedAdam, FedYogi and FedCluster—on two>.

Lu: What they are doing is systematically tuning every single strategy on validation data using the same rules, and then repeating every comparison across ten seeds using Friedman and Holm-corrected tests to make sure their findings are robust <ref:2610.17265#pg1,Friedman and Holm-corrected tests>.

Meng: That sounds rigorous, but when you're talking about medical risk prediction, how does that controlled setup actually translate into something that runs reliably in a real hospital environment?

Jane: Well, the paper sets up these two distinct environments so they can see if the performance of the aggregation method changes depending on whether you're looking at one hospital or many. It asks three main questions: which strategy performs best under a protocol treating every strategy alike, how close federated training is to centralized training once the protocol is controlled, and finally, how much common protocol choices actually change those conclusions.

Tom: And the results are pretty clear on the first question: SCAFFOLD ranked first in both federations. It got an AUC-ROC of zero point eight eight six in the care-unit federation and zero point eight one nine in the multi-hospital federation, and it was selected over FedAvg in seventeen of eighteen variants, but its advantage over the runner-up never exceeded zero point zero zero eight <ref:2610.17265#pg1,over the runner-up never exceeded 0.008>.

Jane: That small margin is interesting, Tom, because they also point out that SCAFFOLD's main edge over FedAvg seems to come from drift correction rather than the optimizer change itself.

Lalam: I see that SCAFFOLD ranked first in both federations (AUC-ROC zero point eight eight six and zero point eight one nine) and in seventeen of eighteen variants, but its margin over the runner-up never exceeded zero point zero zero eight, and a control sharing its local optimizer attributed most of its margin over FedAvg to drift correction <ref:2610.17265#pg1>.

Tom: Right, so even the best strategy doesn't always win by a huge margin; it's more about how stable the training process is. Now they move on to how protocols shift things around.

Paper summary: Jane: They look at four specific protocol choices and quantify exactly how much they change discrimination in AUC-ROC for both federations. They found that three of those choices changed discrimination by more than the choice of strategy did in the care-unit federation, and one choice changed it in both federations.

Lu: Specifically, they look at discharge-coded predictors, client-level SMOTE, evaluating the last instead of the best validation round, and tuning hyperparameters. The results show that discharge-coded diagnosis and procedure counts raised AUC-ROC from zero point eight eight eight to zero point nine two five in MIMIC-IV (+zero point zero three eight) <ref:2610.17265#pg3>.

Meng: I’m interested in client-level SMOTE because that sounds like a way to handle class imbalance, which is huge in mortality prediction, but the paper shows it actually reduced performance across all strategies in both databases.

Jane: It did reduce it; they found SMOTE reducing AUC-ROC by zero point zero five one on average in MIMIC-IV and about zero point zero one nine and zero point zero two three in eICU-CRD <ref:2610.17265#pg3>. They also tested how reporting the last validation round instead of the best one affected results, which cost FedAvg zero point zero eight four of AUC-ROC on MIMIC-IV <ref:2610.17265#pg3>.

Tom: That’s a big signal because it shows that just tweaking the data preparation or the evaluation timing can swing performance by more than switching from one aggregation method to another.

Lu: It suggests that what you are feeding the model, or when you stop training and check its score, has a much bigger impact on discrimination than picking FedAdam versus FedProx <ref:2610.17265#pg2>.

Jane: And they also touched on calibration issues; models trained with class weights were poorly calibrated, but Platt scaling on validation predictions fixed that risk distortion without losing discrimination <ref:2610.17265#pg3>.

Meng: From a production standpoint, that recalibration step is something we have to worry about because if the model overestimates risk before you apply the fix, you’re back to square one.

Tom: So, looking at the overall findings of this paper on "Protocol Choices Matter More Than the Aggregation Algorithm," it seems like while SCAFFOLD is a strong performer, its advantage is small and consistent. The protocol choices have a much larger impact on discrimination than the choice of aggregation strategy itself.

Jane: That implies for practitioners, we shouldn't just focus on picking the best algorithm in isolation; we need to pay just as much attention to how we set up the evaluation protocol before ranking algorithms.

Lu: The paper concludes that predictors should be restricted to information available at the moment of prediction, and that handling class imbalance is better done by weighting or no correction followed by recalibration rather than oversampling each client <ref:2610.17265#pg3>.

Meng: So for deployment, it sounds like a simple weight adjustment or just careful calibration might be more robust than introducing complex data augmentation techniques like SMOTE if you're aiming for the best general performance across different hospital settings.

Tom: Exactly. The choice of aggregation strategy still matters, but by less than the protocol does <ref:2610.17265#pg1>. It’s a reminder that the setup is half the battle in federated learning for healthcare data.

Conclusion: Tom: So, we’ve looked at seven different ways to combine hospital data using federated learning for predicting mortality, and this paper is arguing that how you set up your evaluation protocol matters more than which aggregation strategy you pick.

Jane: Exactly. The title itself puts it right there: "Protocol Choices Matter More Than the Aggregation Algorithm." It’s about showing that even if you use the absolute best math algorithm, like SCAFFOLD, if your testing rules are wrong, you get a bad result.

Lu: I think what this is really pushing is that we need to stop just optimizing for the model and start thinking about how we measure success in a way that reflects real-world deployment constraints. It’s not just about achieving the highest AUC score on a paper; it's about building a system that works reliably across different hospital settings.

Meng: From my side, I see this as important because when we try to ship these models to actual hospitals, we can’t just run one single test. We have to consider all these protocol choices—like which validation round you pick or how you handle class imbalance—and those tiny tweaks have a huge impact on whether the model is actually useful for clinicians.

Jane: Right, so it’s not about finding the perfect algorithm in a vacuum; it’s about understanding that the setup of your experiment dictates the final outcome. It suggests we need to treat protocol design with just as much rigor as our math and model selection.

Tom: And looking at the authors, they did a really thorough job by testing these strategies on two different types of federations—one within one hospital and one across many hospitals—which really strengthens the argument that you can’t rely on any single setting being perfect.

Lu: The implication is that for anyone working in this space, whether you’re training a model for cancer risk or mortality, your first question should probably be about the testing protocol itself before you even look at FedAvg or FedProx.

Jane: It really shifts the focus from just algorithm tuning to system design. It tells us that we need a standardized way of evaluating these models across different data sources and deployment scenarios.

Meng: And it means when we build our next generation of federated systems, we have to bake in choices about things like client-level data balancing right from the start, because those protocol decisions are what’re actually going to make or break the real-world performance.

Tom: So, while SCAFFOLD is strong, this paper is a clear warning that you can't just trust the best math alone without controlling the environment it runs in. The question now becomes how do we design protocols that work for everyone?

Rodrigo Tertulino, Laercio Alencar, Ricardo Almeida

Instituto Federal de Educac¸ao, Ci ˜ encia e Tecnologia do Rio Grande do Norte (IFRN)

cs.LG, cs.CY, cs.DC

Submitted: 2026-10-12

Updated: 2026-10-12

Comments: 23 pages, 6 figures, 10 tables. Code: https://anonymous.4open.science/r/protocol-choices-federated-mortality-D18F/

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: The gist: This study compared seven aggregation strategies for federated learning on in-hospital mortality prediction under controlled protocols, revealing that protocol choices often matter more

Key concepts

Federated Learning (FL): A machine learning approach where models are trained across multiple decentralized devices (like hospitals) without ever sharing the raw patient data. Instead, only model updates are sent back to a central server for aggregation, protecting patient privacy while allowing collaborative training.
Aggregation Strategies: These are the different mathematical methods used to combine the local model updates received from various hospitals during federated learning. The study tested seven distinct methods (like FedAvg and SCAFFOLD) to see which best combines the decentralized knowledge into a single, accurate global model.
Evaluation Protocol: The specific rules or settings used when testing and comparing the performance of different models. This includes things like how patient data is labeled (e.g., discharge codes), how data imbalance is handled (e.g., SMOTE), and which specific validation step is used to judge a model's final performance.
SCAFFOLD: One of the seven aggregation strategies tested, which was found to perform best in both hospital settings. It uses a control mechanism that tracks and corrects for 'drift' (changes in data distribution over time) by comparing local updates against a global reference, leading to high accuracy.

Terminology

Aggregation — 1× in this paper
Federated Learning — 1× in this paper · explained in 17 episodes

Summary

The gist: This study compared seven aggregation strategies for federated learning on in-hospital mortality prediction under controlled protocols, revealing that protocol choices often matter more than the aggregation algorithm itself.

How it works

The research addressed whether different aggregation strategies perform best when evaluation protocols are treated equally across two distinct federations: one intra-hospital (12 care units of a hospital in MIMIC-IV) and one inter-hospital (49 hospitals in eICU-CRD) The study compared seven strategies (FedAvg, FedProx, SCAFFOLD, FedAdagrad, FedAdam, FedYogi and FedCluster) on these two federations under a controlled protocol where every strategy was tuned and every comparison was repeated across 10 seeds with Friedman and Holm-corrected tests

The study quantified the effect of four specific protocol choices to determine how much they changed the conclusions, finding that three of them changed discrimination by more than the choice of strategy did in the care-unit federation, and one in both federations These protocol choices included discharge-coded predictors (+0.038 in AUC-ROC), client-level SMOTE (up to −0.074), evaluating the last instead of the best validation round (up to −0.084), and tuning of hyperparameters

Aggregation Strategies and Performance

SCAFFOLD ranked first in both federations, achieving an AUC-ROC of 0.886 in the care-unit federation and 0.819 in the multi-hospital federation However, its margin over the runner-up never exceeded 0.008, and its advantage over FedAvg was attributed most to drift correction rather than optimizer change The study identified SCAFFOLD as the most accurate strategy in both federations, contingent on plain local gradient descent with a tuned learning rate and a control strategy

The results showed that federated training matched centralized training on the multi-hospital federation In contrast, in the care-unit federation, three protocol choices changed discrimination by more than the choice of strategy did: discharge-coded predictors (+0.038 in AUC-ROC), client-level SMOTE (up to −0.074), and evaluating the last instead of the best validation round (up to −0.084)

Protocol Choices and Discrimination

The protocol choices significantly impacted results, with discharge-coded diagnosis and procedure counts raising AUC-ROC from 0.888 to 0.925 (+0.038) on MIMIC-IV Client-level oversampling reduced performance across all strategies in both databases, with SMOTE reducing AUC-ROC by 0.051 on average in MIMIC-IV and about 0.019 and 0.023 in eICU-CRD The choice of which round to report mattered only in the care-unit federation, costing FedAvg 0.084 of AUC-ROC on MIMIC-IV

Calibration and Robustness

Models trained with class weights discriminated well but were poorly calibrated, overestimating risk before recalibration Platt scaling on validation predictions repaired this bias for all seven strategies, bringing Brier scores down to 0.0196 and 0.074 in MIMIC-IV The ranking of the strategies proved stable across variants, with SCAFFOLD maintaining the highest AUC-ROC in 17 of the 18 configurations tested

Conclusion

The study concluded that while SCAFFOLD was the best strategy, its advantage is small and consistent, and that evaluation protocol choices often have a larger impact on discrimination than the choice of aggregation strategy itself For practitioners, this implies that the evaluation protocol deserves as much attention as the choice of aggregation strategy The results suggest that predictors should be restricted to information available at the moment of prediction, and class imbalance is better handled by weighting or no correction followed by recalibration than by oversampling each client The choice of aggregation strategy still matters, but by less than the protocol does

Limitations

The study has limitations including restricting predictors to demographics and a laboratory panel available in both databases, which kept discrimination below that of models with richer clinical data The federations were simulated on a single machine, meaning communication costs and secure aggregation of shared statistics were not modeled Furthermore, the same pooled validation set served three purposes—tuning, round selection and recalibration—which may have made validation performance slightly optimistic The tuning budget also differed between strategies, which might narrow the gap to SCAFFOLD The degree of heterogeneity between clients was not varied synthetically, as the two natural federations exhibited extreme imbalance in one case The MIMIC-IV federation partitions a single hospital by care unit, representing an extreme intra-hospital imbalance by design The eICU-CRD federation, conversely, represents a federation of independent institutions

Data and Code Availability

MIMIC-IV (version 3.1) and eICU-CRD (version 2.0) are available from PhysioNet to credentialed users who complete the required training and sign the corresponding data use agreements The code that builds the cohorts from the PhysioNet files, runs all experiments, and produces all figures and tables is available in a public mirror of the study repository The authors used generative artificial intelligence tools to revise the manuscript’s language and to review and refactor code The study was implemented in Python 3.12 with PyTorch 2.13, scikit-learn 1.9 and imbalancedlearn 0.14 The ethics of the study involved using only de-identified data, approved by the institutional review boards of the contributing institutions with a waiver of informed consent No specific funding was received for this research The authors declare no conflicts of interest

References

[1] Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data [2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data [3] Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletar´ı, Holger R. Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N. Galtier, Bennett A. Landman, Klaus Maier-Hein, Sebastien Ourselin´Micah Sheller and Ronald M. Summers [4] Peter Kairouz, H. Brendan McMahan, et al. Advances and open problems in federated learning [5] Trung Kien Dang, Xiang Lan, Jianshu Weng, and Mengling Feng. Federated learning for electronic health records [6] Ruben van den Goorbergh, Maarten van Smeden, Dirk Timmerman, and Ben Van Calster. The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression [7] Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in machine-learningbased science [8] Theodora S.

Improvements for AI systems

textbfBold Header: Incorporate Strategy-Specific Tuning for SCAFFOLD Advantage in Unbalanced Federations. The paper shows that the remainder of this section interprets these findings by noting that tuning was decisive for SCAFFOLD, suggesting AI systems should implement a control strategy like FedAvg-SGD to leverage the drift correction mechanism, which is shown to be responsible for the margin over alternatives in unbalanced settings where a single client dominates the federation.
textbfBold Header: Implement Protocol-Aware Risk Estimation and Recalibration. The study demonstrated that "three common protocol choices, namely predictors recorded at discharge, client-level oversampling, and the evaluation of the last round, each moved discrimination in the care-unit federation by more than the whole range between the strategies," necessitating a system that explicitly manages these protocol variables before reporting performance metrics.
textbfBold Header: Adopt Class Weighting or No Correction Over Oversampling for Imbalance Handling. The paper concluded that class weighting avoided this loss but still distorted the predicted risks, and recommended handling imbalance by weighting the loss, or by no correction at all, followed by recalibration, than by oversampling each client. This implies that AI training pipelines should prioritize robust weighting schemes over synthetic data generation techniques like SMOTE-Tomek to maintain calibration.
textbfBold Header: Employ Robust Model Selection Criteria Based on Validation Performance Across Multiple Seeds. The researchers tested every comparison repeated across 10 seeds with Friedman and Holm-corrected tests, indicating that AI systems should select final model architectures or aggregation strategies based on metrics derived from these controlled, multi-seed comparisons rather than relying on a single run.
textbfBold Header: Utilize Control Strategies to Isolate Optimization Components. The comparison between SCAFFOLD and FedAvg-SGD showed that the control variates [account] for most of its margin over FedAvg to drift correction, suggesting that AI systems should use control strategies to decouple the effect of drift correction from the inherent changes in the local optimizer.

Abstract

Federated learning allows hospitals to train a shared mortality model without pooling their records, but published comparisons of aggregation strategies rarely control the evaluation protocol. This study compared seven strategies (FedAvg, FedProx, SCAFFOLD, FedAdagrad, FedAdam, FedYogi and FedCluster) on two federations built with one pipeline: 12 care units of a hospital in MIMIC-IV and 49 hospitals in eICU-CRD. Each strategy was tuned and selected on validation data by the same rule, and every comparison was repeated across 10 seeds with Friedman and Holm-corrected tests. SCAFFOLD ranked first in both federations (AUC-ROC 0.886 and 0.819) and in 17 of 18 variants, but its margin over the runner-up never exceeded 0.008, and a control sharing its local optimizer attributed most of its margin over FedAvg to drift correction. On the multi-hospital federation, federated training matched centralized training. In the care-unit federation, three protocol choices changed discrimination by more than the choice of strategy did: discharge-coded predictors (+0.038 in AUC-ROC), client-level SMOTE (up to-0.074), and evaluating the last instead of the best validation round (up to-0.084). Platt scaling on validation data repaired the risks distorted by imbalance corrections without loss of discrimination. Studies should therefore fix the protocol before ranking algorithms.