Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning

The gist: FedRevive, an Asynchronous Federated Learning framework that revives stale updates through data-free knowledge distillation, achieves faster training by up to 38.4% and higher final accuracy by up to 16.5% than asynchronous baselines

How it works

FedRevive integrates parameter-space aggregation with a lightweight, server-side Data-Free Knowledge Distillation (DFKD) process that transfers knowledge from stale client updates to the current global model without access to data The framework combines regular parameter space update with a DFKD update through an adaptive weighting function β(τ t i) controlling reliance on the distilled signal based on staleness The aggregation step in Algorithm 1 (line 6) becomes x(t+1) ← x(t) + ηs (1−β(τ t i))∆(t−τ t i)i + β(τ t i)∆KD i

The DFKD mechanism involves a meta-learned generator that synthesizes pseudo-samples used for multiteacher distillation This process is performed server-side, treating received client models as teachers and the current server model as the student The synthesis loss Lsynth includes target, feature matching, and adversarial components to guide knowledge transfer Furthermore, multi-teacher distillation is employed by maintaining a buffer C of the most recent received client models and distilling from all buffered teachers jointly

Key Components and Design Choices

The proposed method addresses the challenges of integrating Knowledge Distillation (KD) into Asynchronous Federated Learning (AFL), specifically concerning the lack of shared data, heterogeneous teacher models, and computational overhead To overcome the lack of distillation data, KD-Revive performs a data-free KD by synthesizing pseudo-inputs guided jointly by the incoming client model and the current server model The generator Gθ evolves through training rounds using a meta-update inspired by Reptile to adapt to changing teacher models without catastrophic drift

To handle data heterogeneity across clients, the server derives proxy class proportions of clients from the received updates without additional client computation or information This is achieved by probing client models with random inputs and computing a per-upload proxy πe(u) i, which is then averaged to form a fixed proxy πˆ i This proxy allows the server to weight buffered teachers in a class-aware manner, mitigating biased knowledge transfer

Overhead and Robustness

FedRevive adds server-side computation from periodic generator training and multi-teacher distillation, but this overhead is typically comparatively small in realistic AFL deployments where client compute and communication dominate end-to-end time The synthesis module is executed only once every Tgen server updates, and distillation uses KKD lightweight steps The server can dispatch the next local-training request immediately upon receiving an update and run synthesis, distillation, and hybrid aggregation asynchronously afterward to hide most server compute behind client-side latency

The method demonstrates robustness to shifts in delay schemes by re-running asynchronous methods on CIFAR-10 and 20NewsGroups under an alternative delay schedule without re-tuning any hyperparameters The DFKD-based methods remain substantially more robust to this delay shift, while asynchronous baselines exhibit slower convergence and degraded final accuracy under the shifted staleness regime

Experimental Findings

Experiments on various vision and text benchmarks show that FedRevive achieves faster training by up to 38.4% and higher final accuracy by up to 16.5% than asynchronous baselines On CIFAR-10, FedRevive converges faster and reaches a higher final accuracy than the asynchronous baseline FedBuff, which comes from continually transferring knowledge from buffered stale client models For the text-domain task on 20NewsGroups, FedRevive consistently improves convergence speed among baselines The variant DD-FedRevive, which assumes access to a small public dataset and client class proportions, attains the fastest convergence overall but remains only a hypothetical upper bound due to its unrealistic assumptions The results indicate that stale client updates still carry rich, transferable knowledge

Conclusion

FedRevive provides a scalable asynchronous FL framework that extracts and utilizes informative content from stale client updates through data-free knowledge distillation By combining lightweight meta-learned synthetic data generation with multi-teacher distillation, the approach effectively mitigates staleness while preserving data privacy and system efficiency Extending FedRevive with pretrained generative models for applicable data modalities presents a promising direction The performance of AFL-DW remains considerably lower than that of FedRevive, confirming that the improvement of our method arises not primarily from down-weighting stale updates but from the proposed DFKD mechanism itself

How it works

The server adaptively fuses the conventional parameter-space update with a knowledge-distilled update to alleviate the adverse effect of staleness by using an adaptive weighting function β(τ t i) When τ t i ≈ 0, β(τ t i)≈0; conversely, when τ t i is large, β(τ t i)≈1 This adaptive schedule is detailed in Section 3 The choice of β(τ t i) is detailed in Section 3

The server maintains a KD buffer C storing c most recently received client models to use as teachers, and the rolling buffer ensures that the distillation process utilizes recent and diverse client knowledge The rolling buffer ensures that the distillation process utilizes recent and diverse client knowledge The server maintains a KD buffer C storing c most recently received client models to use as teachers, and the rolling buffer ensures that the distillation process utilizes recent and diverse client knowledge

How it works

The server derives proxy per-client class-proportion vectors from early uploads by probing the model with a batch of random inputs and computing a per-upload proxy πe(u) i The final proxy is the average across early uploads, πˆ i = (1/Test) PTest u=1πe(u) i, and is fixed for the remainder of training This estimation is performed online as uploads arrive and requires only server-side forward passes

The student is trained to match teacher outputs by minimizing the average KL divergence LKD ≜ 1/c X k∈It 1/Bk X(r,y)∈Bk KL σ xek(r) σ xs(r) The student is updated with xs ← xs − ηKD∇xs LKD for KKD steps This distilled update is then used in the aggregation rule Eq. (3)

How it works

The server maintains the current global model x(t) as usual, and FedRevive does not require additional server memory beyond what is already standard in asynchronous FL aside from the constant-sized teacher buffer It can avoid materializing ∆i entirely by updating the global parameters directly through interpolation in parameter space

The server computation does not need to add an extra wall-clock delay in an asynchronous system, as it can dispatch the next local-training request immediately upon receiving an update and run synthesis, distillation, and hybrid aggregation asynchronously afterward <ref:2511.

1. Bold Header: Dynamic Staleness-Adaptive Aggregation

FedRevive implements an aggregation rule where the server increasingly relies on the KD-based update as the client update becomes more outdated, controlled by a nondecreasing mapping of staleness, which mitigates the adverse effects of staleness by weighting the knowledge-distilled signal more heavily when clients are stale.

2. Bold Header: Data-Free Knowledge Distillation for Scalability

The system utilizes a lightweight, server-side DFKD process that transfers knowledge from stale client updates to the current global model without access to data, which allows AFL to retain its scalability advantages by extracting transferable knowledge from delayed client states.

3. Bold Header: Multi-Teacher Knowledge Distillation for Non-IID Data

To combat data heterogeneity across clients, the server employs multi-teacher distillation where it maintains a buffer of recent client models and derives class-aware weighting vectors, ensuring that the distilled knowledge is weighted based on proxy class proportions of clients from the received updates without additional information.

4. Bold Header: Robustness to System Heterogeneity

The framework demonstrates robustness to changes in system dynamics by showing that the DFKD-based methods (FedRevive and DD-FedRevive) remain substantially more robust to this delay shift, whereas asynchronous baselines exhibit slower convergence and degraded final accuracy under the shifted staleness regime.

5. Bold Header: Computationally Efficient Server Overhead

The server overhead is managed by performing synthesis and distillation only once every Tgen updates and by structuring the pipeline so that the server-side operation time is non-blocking or overlapped, effectively hiding compute behind client latency to avoid introducing any additional wall-clock delay.