FedReason: Large Language Models as Weight-Space Meta-Aggregators for Heterogeneous Federated Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "FedReason: Large Language Models as Weight-Space Meta-Aggregators for Heterogeneous Federated Learning".
Elias: Here's the gist of the paper. Federated learning servers combine client models with a fixed aggregation rule, and each rule in common use is designed for one kind of heterogeneity The paper investigates whether a large language model (LLM) that inspects the client models can choose the rule itself, without any training on the server FedReason Framework FedReason is a meta-aggregator for problem (5) that changes only the server FedAvg is the active rule until the first program is accepted In every round with (t − 1) mod T = 0, the server opens a new tool-use conversation with the LLM, loads the round context into an analysis sandbox and lets the LLM investigate it until it submits a program that passes a dry run on the current round The program aggregates the current round and every later round until the next invocation How the training signal is built The LLM has two tools: python, which runs code in a persistent namespace on the server and returns what it prints, and submit policy, which submits a program The namespace holds read-only copies of the global model G, the client models W, the updates U, the metadata meta and the history of proxy metrics and coefficients A budgeted function proxy eval evaluates any candidate model on the proxy set, at most binv = 6 times per invocation Where the gains come from The paper reports that FedReason improved on FedAvg by 1.26 pp on average under label skew and was the strongest server-side rule on CIFAR-100 It identified sign-flipping clients from their weights in the first round The gains under label skew were about one point and largest on CIFAR-100 FedReason did not beat SCAFFOLD, did not beat FLTrust under attack, and was worse than HeteroFL on CIFAR-10 sub-models because of one wrong inference Ablations and LLM behaviour The metrics-only LLM still found a way to probe the clients: at rounds 31 and 41 it passed one-hot coefficients to the weighted-average helper, evaluated single client models on the proxy set and inferred each client’s classes from the per-class accuracies Removing proxy feedback did far more damage, and accuracy dropped to 38.84% The LLM again recognised the label skew and identified the clients that alone held a class at round 1, however, it adopted class-evidence weighting of the classifier rows, the heuristic of the example program in the system prompt, and kept it at every later invocation In 168 runs on CIFAR-10 and CIFAR-100 with label skew, sub-models of different widths and Byzantine clients, FedReason improved on FedAvg by 1.26 pp on average under label skew and was the strongest server-side rule on CIFAR-100 It identified sign-flipping clients from their weights in the first round and stayed within about one point of FLTrust without training on the server, at 0.
Paper summary: Nadia: So, wrapping up the FedReason paper by Ali Mohammadi Ruzbahania and Abbas Yazdinejadb, they showed that this LLM approach works well even when dealing with label skew and various client types on CIFAR-one hundred <ref:2611.08383#pg1>.
Elias: The main implication is that we can leverage large language models as a meta-aggregator for federated learning, allowing them to choose the aggregation rule without needing any training on the server itself.
Priya: It seems like it offers a way to adapt our aggregation strategy dynamically based on real-time evidence from client weights and metadata.
Nadia: The authors found that FedReason improved performance by one point two six percentage points on average under label skew compared to FedAvg, making it the best server-side rule they tested in those conditions.
Elias: It also noted that the LLM could identify sign-flipping clients in the very first round of training.
Priya: The paper does admit a limitation, though—when they withheld the proxy feedback, accuracy dropped quite a bit, falling to about thirty-eight point eight four percent. So it’s not a magic solution that just works perfectly every time.
Nadia: It’s certainly an improvement over just sticking to FedAvg when things get messy with client heterogeneity and skew.
Elias: The overall message is that the LLM acts as a sophisticated interpreter of the weight space, choosing the right combination of averaging, clipping, or exclusion based on what it observes.
Priya: It gives us a new way to think about how we handle non-IID data in federated systems by letting an agent inspect the situation and propose a specific mathematical operation.
Conclusion: Nadia: So we've looked at how this FedReason framework lets an LLM pick the right aggregation rule for federated learning without any new training on the server, and now it's time to wrap up what these authors actually proved.
Elias: Yeah, they put this whole idea into a system called FedReason, which basically uses the large language model to change only the server part—the way it aggregates things—until it finds a program that works for the current round.
Priya: What does that mean in plain terms? Like, instead of us pre-deciding if we use FedAvg or SCAFFOLD based on what we *think* is best, the AI looks at the client models and decides the strategy on the fly?
Nadia: Exactly. They found that when things are messy with label skew and different kinds of clients—like some clients accidentally flipping their labels—FedReason actually performed better than FedAvg by one point two six percentage points on average under those conditions.
Elias: That's a specific number, which is good for us to look at. They also said the LLM could spot those sign-flipping clients right in the first round of training, before anything else happens.
Priya: But we have to remember what they said about the caveats, right? They mentioned that if you took away that feedback from the proxy evaluation, accuracy dropped pretty significantly to about thirty-eight point eight four percent. So it's not a guaranteed win every single time we run it.
Nadia: True. It shows the LLM is smart enough to probe those client weights and infer what's going on, but it still relies on that feedback loop to make the right call consistently across all scenarios, especially when things get weird.
Elias: So the big picture here is that we might not need to manually tune every single aggregation algorithm for every new deployment; instead, we can use this kind of meta-aggregator to adapt the strategy based on what's happening in real time across the client models.
Priya: It shifts our focus from just picking a static rule to building a system where an AI agent dynamically chooses the best mathematical operation based on observed data patterns.
Nadia: That opens up some interesting avenues for how we handle those complex, non-IID data scenarios in practice, which is what I'm really interested in seeing exploited or defended against next.
Ali Mohammadi Ruzbahania, Abbas Yazdinejadb, Hadis Karimipoura
Smart Cyber-Physical Systems (SCPS) Lab, University of Calgary · bDCAILab, University of Regina
cs.CR, cs.AI
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: Ali Mohammadi Ruzbahani (corresponding)
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Federated learning servers combine client models with a fixed aggregation rule, and each rule in common use is designed for one kind of heterogeneity The paper investigates whether a large language…
Key concepts
- Federated Learning Servers
- These are central machines in a federated learning setup that collect model updates from many clients. They use a fixed rule, like FedAvg, to combine these client models into a single global model. The paper focuses on how the server can intelligently change this rule.
- FedReason Framework
- This is the core idea where an LLM acts as a meta-aggregator. Instead of using one fixed rule, FedReason lets an LLM decide which aggregation program to use in each round by analyzing the current situation and context.
Terminology
Summary
Federated learning servers combine client models with a fixed aggregation rule, and each rule in common use is designed for one kind of heterogeneity The paper investigates whether a large language model (LLM) that inspects the client models can choose the rule itself, without any training on the server
FedReason Framework
FedReason is a meta-aggregator for problem (5) that changes only the server FedAvg is the active rule until the first program is accepted In every round with (t − 1) mod T = 0, the server opens a new tool-use conversation with the LLM, loads the round context into an analysis sandbox and lets the LLM investigate it until it submits a program that passes a dry run on the current round The program aggregates the current round and every later round until the next invocation
How the training signal is built
The LLM has two tools: python, which runs code in a persistent namespace on the server and returns what it prints, and submit policy, which submits a program The namespace holds read-only copies of the global model G, the client models W, the updates U, the metadata meta and the history of proxy metrics and coefficients A budgeted function proxy eval evaluates any candidate model on the proxy set, at most binv = 6 times per invocation
Where the gains come from
The paper reports that FedReason improved on FedAvg by 1.26 pp on average under label skew and was the strongest server-side rule on CIFAR-100 It identified sign-flipping clients from their weights in the first round The gains under label skew were about one point and largest on CIFAR-100 FedReason did not beat SCAFFOLD, did not beat FLTrust under attack, and was worse than HeteroFL on CIFAR-10 sub-models because of one wrong inference
Ablations and LLM behaviour
The metrics-only LLM still found a way to probe the clients: at rounds 31 and 41 it passed one-hot coefficients to the weighted-average helper, evaluated single client models on the proxy set and inferred each client’s classes from the per-class accuracies Removing proxy feedback did far more damage, and accuracy dropped to 38.84% The LLM again recognised the label skew and identified the clients that alone held a class at round 1, however, it adopted class-evidence weighting of the classifier rows, the heuristic of the example program in the system prompt, and kept it at every later invocation
Conclusion
In 168 runs on CIFAR-10 and CIFAR-100 with label skew, sub-models of different widths and Byzantine clients, FedReason improved on FedAvg by 1.26 pp on average under label skew and was the strongest server-side rule on CIFAR-100 It identified sign-flipping clients from their weights in the first round and stayed within about one point of FLTrust without training on the server, at 0. p. 0 It was the strongest server-side rule on CIFAR-100 but did
not beat SCAFFOLD, which changes client training. With three of ten clients malicious, it excluded all sign-flipping
clients from the first round and stayed about one point below FLTrust without training a root model.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Meta-aggregation capabilities allow an LLM to choose a program that
expresses whichever rule the evidence supports,
enabling dynamic adaptation based on observed client model characteristics like label skew, model width heterogeneity, or Byzantine behavior. -
The FedReason framework can choose between standard aggregation rules or write a custom Python aggregation program that the server executes in every round until the next call, falling back to FedAvg only when the output is invalid.
-
The system can inspect client models by running restricted Python code in a sandbox on the server, reading
only what that code prints,
and scoring candidate models on a proxy set before submitting an aggregation program. -
The LLM-written programs can implement complex rules such as
sample-weighted averaging, clipping of large updates, exclusion of clients that look malicious, separate treatment of the classifier rows, or a server step with momentum.
-
The AI system can be evaluated against 14 baselines and show performance gains over FedAvg by 1.26 percentage points on average under label skew conditions.
Abstract
Federated learning servers combine client models with a fixed aggregation rule, and each rule in common use is designed for one kind of heterogeneity. We study whether a large language model (LLM) that inspects the client models can choose the rule itself, without any training on the server. In FedReason, the LLM is called every ten rounds. It analyses the weights by running code in a restricted server-side sandbox, sees only the printed results, may score a few candidates on a 500-image proxy set, and submits a short Python aggregation program. The server applies the program until the next call and falls back to FedAvg whenever its output is invalid. Full weight tensors never leave the server. We evaluated FedReason with Claude Opus 5.5 on CIFAR-10 and CIFAR-100 with ResNet-18 under Dirichlet label skew, width-heterogeneous sub-models and Byzantine clients, in 168 runs against 14 baselines. Across 12 paired label-skew runs it beat FedAvg by 1.26 percentage points on average (10 wins), and a control that spends the same per-round proxy budget on choosing among standard rules by 1.08 points. It was the strongest server-side rule on CIFAR-100 but did not beat SCAFFOLD, which changes client training. With three of ten clients malicious, it excluded all sign-flipping clients from the first round and stayed about one point below FLTrust without training a root model. Without proxy feedback, accuracy fell from 73.8% to 38.8% in one seed. A 60-round run cost 0.79 USD in API fees on average.
Sources
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- Byzantine-Tolerant Machine Learning
- FedStrategist: A Meta-Learning Framework for Adaptive and Robust Aggregation in Federated Learning
- Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents
- Automating aggregation strategy selection in federated learning
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Federated Optimization in Heterogeneous Networks
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
- Revisiting Weighted Aggregation in Federated Learning with Neural Networks
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning
- Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
- Large Language Models as Optimizers
- Deep Leakage from Gradients
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data