Multimodal Federated Learning under Dual-Axis Modality Missingness

The paper Multimodal Federated Learning under Dual-Axis Modality Missingness proposes Flux, a multimodal federated learning framework designed to address a coupled challenge where modality missingness occurs along two axes: inter-client modality heterogeneity (clients have different sets of modalities) and intra-client modality missingness (individual samples contain only subsets of the modalities available locally). The authors note that Existing methods typically address these two axes separately, which overlooks their interaction during federated optimization.

Flux is built around two complementary components. First, "modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness." For each observed modality, a confidence head produces unimodal logits and a scalar log-scale parameter, from which a positive unimodal temperature τm = exp(νm/2) is derived. These are combined using precision-inspired aggregation to produce a fused temperature τf. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This design enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning.

The authors evaluate Flux on four multimodal datasets spanning health-sensing and medical domains: PAMAP2, RealWorldHAR, SleepEDF, and ADNI (a heterogeneous biomedical dataset with naturally missing modalities). The results show that Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8∼2.2 points and by 1.6 points on average. Specifically, Flux achieves average macro-F1 scores of 0.642 on PAMAP2, 0.542 on SleepEDF, 0.799 on RealWorldHAR, and 0.508 on ADNI.

Additional analyses demonstrate several favorable properties. The learned temperature increases with modality incompleteness and input corruption, and is negatively correlated with validation F1 (Spearman ρ = −0.64, p < 0.001). Flux achieves favorable calibration with the best ECE (0.162) and Brier score (0.487) among compared methods. Private-only tempering reduces shared-gradient instability and client-to-global update drift compared to applying temperature to both shared and private pathways. The ablation study shows that the complete Flux achieves the highest average macro-F1 (0.583), with each component contributing: Shared-private separation raises the average from 0.531 to 0.544, modality-specific confidence further improves it to 0.549, and private-only tempering increases it to 0.583, a gain of 0.034.

The paper makes three main contributions: (1) formulating dual-axis modality missingness as a coupled regime, (2) proposing Flux with modality-aware confidence tempering and gradient-decoupled private adaptation, and (3) demonstrating consistent performance gains across four datasets with diagnostic analyses of calibration, sensitivity to evidence quality, and shared-optimization stability.

Improvements to AI systems:

1. Robust multimodal fusion under arbitrary missingness: AI systems can now handle cases where both clients (e.g., edge devices, hospitals) and individual samples have incomplete modality sets simultaneously—without retraining or special-case handling. The system learns to dynamically weight available modalities per sample via confidence tempering, rather than relying on imputation or discarding incomplete data.

2. Sample-adaptive prediction confidence: The system can produce predictions whose sharpness (softmax temperature) is automatically calibrated to the evidence quality of each input. For a sample with missing or corrupted modalities, the system outputs a flatter, more uncertain distribution; for complete, clean inputs, it outputs sharper, more confident predictions. This yields better-calibrated probabilities (lower ECE and Brier score) for high-stakes applications like medical diagnosis or wearable health monitoring.

3. Stable federated learning with heterogeneous local data: The system separates a private, client-specific prediction pathway (where confidence-based tempering is applied) from a shared, global representation pathway (trained with a standard, untempered objective). This prevents noisy, confidence-dependent gradients from destabilizing the shared model, reducing client-to-global update drift and improving convergence in federated settings with non-IID and incomplete data.

4. Automatic detection of data quality issues: The learned temperature values serve as a built-in diagnostic signal. The system can flag samples or clients with unusually high temperatures, indicating severe modality incompleteness or input corruption—enabling proactive data quality monitoring in production.

5. Unified handling of cross-client and within-client missingness: Instead of treating inter-client heterogeneity and intra-client missingness as separate problems (which existing methods do), the system jointly models their interaction during optimization, leading to higher average macro-F1 (0.8–2.2 points improvement) across diverse domains (wearable sensors, sleep EEG, medical imaging/clinical data).

What the improved AI system can do:

• Deploy a single federated model across a network of hospitals or wearable devices where each site has different sensor/imaging capabilities, and each patient record may lack certain tests—without performance collapse.

• Provide clinicians with well-calibrated risk scores (e.g., Alzheimer’s detection from ADNI) even when MRI, PET, or cognitive test data are partially missing per patient.

• Operate on edge devices with intermittent sensor failures (e.g., a smartwatch losing heart-rate or accelerometer data mid-recording) while maintaining reliable activity recognition.

• Offer a privacy-preserving, collaborative learning system that adapts to each client’s local data completeness without compromising the global model’s stability or requiring centralized data pooling.