Label Granularity Skew in Federated Learning with Hierarchical Image Classification

This paper introduces a new form of statistical heterogeneity in federated learning, termed label granularity skew, which arises in hierarchical image classification when different clients provide labels at different levels of detail within a shared class hierarchy, reflecting their varying domain knowledge and annotation capabilities. The authors formalize this setting, propose methods to model it, and introduce a novel training algorithm, FedBDFT, to address it.

The paper defines label granularity skew as a type of label preference skew where clients have different local label hierarchies (subtrees of the global class hierarchy) based on their knowledge. Formally, for clients i and j, there exists a class e such that Pi(Xe) = Pj(Xe), but Pi(Se(i) Xe) ≠ Pj(Se(j) Xe), while Pi(SPar(e)(i) Xe) = Pj(SPar(e)(j) Xe). This means clients agree on the features and coarse labels but differ in the granularity of the fine-grained labels they provide.

To realistically model this heterogeneity, the authors propose a local label hierarchy generation algorithm based on the Probabilistic Relational Neighbor (pRN) classifier. This algorithm initializes a proportion α of leaf classes as unknown and uses a weighted adjacency matrix based on Wu-Palmer similarity to iteratively propagate knowledge probabilities through the hierarchy, creating realistic knowledge boundaries for each client.

To construct a semantically meaningful class hierarchy, the paper leverages WordNet, a lexical knowledge graph. Since the original WordNet-guided hierarchy is often overly deep and imbalanced, the authors propose a WordNet-Guided Hierarchy Coarsening via Silhouette Score method. This algorithm collapses negligible internal classes by evaluating all possible combinations of child class sets and selecting the one that maximizes the Silhouette score, thereby reducing label space complexity while preserving discriminative semantic structure.

The paper experimentally evaluates existing hierarchical classification models under label granularity skew and finds that strongly coupled models (e.g., soft-max-margin, soft-max-descendant) are highly sensitive to incomplete supervision and often fail to converge at high skewness levels. In contrast, the conditional softmax classifier (cond softmax) is more robust due to its weak inter-branch coupling. Based on this insight, the authors propose Branch-wise Decoupled Fine-Tuning (BDFT) and its federated version, FedBDFT. BDFT decouples a pretrained cond softmax into independent branch-wise local classifiers, each with its own feature extractor and classification head, and fine-tunes them separately on branch-specific datasets. FedBDFT trains these classifiers sequentially using a FedAvg-based aggregation approach.

Experiments on CIFAR-100, TinyImageNet, and ImageNet demonstrate that FedBDFT substantially improves robustness under severe label granularity skew, with average gains of 27.9% and 56.4% at skewness levels of 0.6 and 0.9, respectively, over the baselines. Zero-shot learning results further indicate that FedBDFT better preserves hierarchical representations for unseen fine-grained classes, achieving the highest performance on unseen classes across all datasets, even though it does not achieve the best performance on seen classes in non-skewed settings. The paper concludes that FedBDFT is particularly effective under highly heterogeneous conditions, and it identifies label granularity skew as a novel and important challenge for federated hierarchical classification.

Improvements to AI Systems:

1. Hierarchy-Aware Federated Learning for Heterogeneous Label Granularity: Implement FedBDFT to enable federated models to train effectively when clients provide labels at varying levels of class hierarchy (e.g., one client labels animal, another labels dog). The improved system can maintain high classification accuracy and convergence even when 90% of clients use coarse or incomplete labels, avoiding the failure modes of standard softmax-based federated aggregation.

2. Robust Fine-Grained Classification Under Incomplete Supervision: Use branch-wise decoupled fine-tuning to isolate learning per hierarchical branch. This allows the AI to learn fine-grained distinctions (e.g., specific dog breeds) from clients that provide detailed labels, while still leveraging coarse labels (e.g., mammal) from less knowledgeable clients—without catastrophic interference. The system can then generalize to unseen fine-grained classes via preserved hierarchical representations, enabling zero-shot classification of novel subcategories.

3. Adaptive Label-Hierarchy Generation for Realistic Client Modeling: Incorporate the pRN-based local hierarchy generation algorithm to simulate or predict client knowledge boundaries in real deployments. This lets the AI system anticipate which label granularities are likely from each client, dynamically adjust training weights, and proactively request finer labels only from clients likely to provide them, reducing annotation cost and improving data efficiency.

4. Semantic Hierarchy Coarsening for Efficient Learning: Apply the WordNet-Guided Silhouette Score coarsening to automatically collapse overly deep or imbalanced class trees. This improves the AI’s ability to learn from large-scale hierarchical datasets (e.g., ImageNet) by reducing label space complexity while preserving discriminative semantic structure—leading to faster convergence and lower memory footprint without sacrificing accuracy.

5. Sequential Branch Training for Federated Aggregation: Use FedBDFT’s sequential branch-wise training with FedAvg to enable the AI to handle non-IID label distributions across clients. The improved system can train a single global model that excels in heterogeneous environments, achieving 27.9% and 56.4% average accuracy gains over baselines at high skewness levels (0.6 and 0.9), making it suitable for real-world federated deployments with diverse annotator expertise.