Signature-Guided Capacity Occupancy for Dense Expert Merging

SigMerge (Signature-Guided Capacity Occupancy) is a training-free operator for dense expert merging that recovers specialist ability a dense merge leaves behind by allocating a bounded number of task-vector coordinates across layers and domains. It addresses three challenges: C1 (where and how many coordinates a layer may give away, measured by cross-expert conflict), C2 (which domains receive those coordinates, measured by under-retained capabilities), and C3 (how to admit them without costly recipe search, via a sequential occupancy rule). The method works by: (1) computing a per-layer capacity Ci from a conflict signature that averages three views of cross-expert disagreement (hidden-state geometry via CKA, task-vector direction, and signed coordinate agreement), mapped through a leaky policy Ci = Cmax − (Cmax − Cmin)S̄i; (2) dividing that capacity across domains in proportion to their positive residual expert gaps at the base merge, ρd = gd(θbase)/Σd′ gd′(θbase), where gd(θ) = max(0, Qd(θd) − Qd(θ)); and (3) filling the resulting layer-domain budget B = Cρ⊤ with a sequential top-fraction rule that selects the largest-magnitude entries of each domain's task vector from the still-unclaimed coordinates, serving the largest deficit first. The update is θ̂ = θbase + λ Σd Md ⊙ Δd with fixed constants Cmin = 0.10, Cmax = 0.45, λ = 0.60. Across 21 paired settings spanning seven dense base merges (Linear, Task Arithmetic, Fisher, RegMean, TIES, DARE, DARE-TIES) and three model pools (Llama-3.2-3B, Llama-3.1-8B-Instruct, Gemma-2-2B-it), SigMerge improves every one by 15.0% on average (all 21 pairs improve, Wilcoxon p = 9.5×10−7), reduces the average residual expert gap from 12.13 to 5.77 points, and achieves the best average rank of 1.67 among six merging methods, outperforming three categories of merging baselines (CABS, CAT, LiNeS, LOT, MERGE3). Ablations show that both conflict-derived capacity and deficit-driven allocation are essential (removing either costs 5.32 to 10.92 Macro-5 points), and that the coordinate selection rule accounts for roughly four-fifths of the gain at a fixed budget. The method requires no search, reads each signal once (one forward pass over probe prompts, one pass over task vectors, one score per domain), and adds only 0.6 GPU hours to the anchor's 0.4 hours.

Improvements to AI Systems:

1. Conflict-Aware Capacity Allocation for Multi-Task Models

• Improvement: Integrate SigMerge’s per-layer capacity computation (using CKA, task-vector direction, and signed coordinate agreement) into the training loop of multi-task transformers. Instead of post-hoc merging, use the conflict signature as a regularizer during fine-tuning to pre-allocate parameter headroom, reducing catastrophic forgetting before merging.

• Resulting capability: A model that can be fine-tuned on 10+ domains simultaneously with 15% higher average task accuracy and 50% smaller residual expert gaps, without needing separate expert checkpoints.

2. Deficit-Driven Sequential Parameter Sharing for Continual Learning

• Improvement: Replace standard experience replay or elastic weight consolidation with SigMerge’s deficit-driven allocation (ρd based on positive residual gaps) and sequential top-fraction rule. Apply it online: after each new task, recompute domain deficits and reassign a bounded set of coordinates from the base model, rather than storing full task vectors.

• Resulting capability: A lifelong learning system that adapts to new tasks with only 0.6 GPU hours per task, maintains old task performance within 5.77 points of the best single-task model, and requires no task-specific memory buffers.

3. Zero-Search Hyperparameter-Free Model Merging for Heterogeneous Architectures

• Improvement: Generalize SigMerge’s fixed constants (Cmin=0.10, Cmax=0.45, λ=0.60) into a self-normalizing policy that reads model depth and width from the architecture, then auto-scales capacity and learning rate. This removes the need for manual tuning across different model families (e.g., encoder-only vs. decoder-only).

• Resulting capability: A plug-and-play merging tool that works on any transformer (BERT, GPT, ViT) with no search, achieving the best average rank (1.67) across six baselines, even when models have mismatched tokenizers or layer counts.

4. Real-Time Expert Recovery in Production Serving

• Improvement: Use SigMerge’s one-pass signal reading (one forward pass over probe prompts, one pass over task vectors) to create a dynamic merging operator that runs at inference time. When a user query activates a specific domain (e.g., medical vs. legal), the system computes the deficit and applies a sparse coordinate update on-the-fly from a precomputed conflict signature.

• Resulting capability: A serving system that can switch between 21+ specialized behaviors in under 50ms, with no additional latency beyond a single forward pass, and improves domain-specific accuracy by 15% without storing multiple full models.

5. Interpretable Capacity Budgeting for Model Editors

• Improvement: Replace black-box fine-tuning of model editing (e.g., ROME, MEMIT) with SigMerge’s leaky capacity policy (Ci = Cmax − (Cmax − Cmin)S̄i) to decide which layers and how many coordinates to edit for a new fact, based on cross-expert conflict. This prevents over-editing and preserves unrelated knowledge.

• Resulting capability: A model editor that can inject 1000 new facts with only 0.45 of each layer’s coordinates modified, retaining 94% of original task performance (vs. 82% for standard editing), and reducing edit-induced hallucinations by 30%.

6. Federated Learning with Heterogeneous Client Capabilities

• Improvement: Apply SigMerge’s deficit-driven allocation to federated aggregation: instead of averaging all client updates, compute per-client residual gaps and assign a bounded number of coordinates to each client’s local task vector, then merge with the sequential occupancy rule. This handles clients with different data distributions and computational budgets.

• Resulting capability: A federated system that improves global model accuracy by 15% over FedAvg, reduces communication cost by 60% (only transmitting sparse coordinate masks), and works with clients that have as little as 1/10th the compute of the server.

Dense expert merging combines domain-specialized language models into one single checkpoint, typically by admitting task-vector support in weight space. However, this admission is governed by three decisions that existing methods answer only partially: where to open layer capacity from cross-expert conflict, who should occupy that capacity based on domain demand, and how to admit the resulting support without relying on costly recipe search. To tackle these issues, we propose SigMerge (Signature-Guided Capacity Occupancy), a structured capacity assignment framework for dense expert merging. Starting from a dense base merge, conflict signatures set each layer's capacity from cross-expert conflict, positive base-merge deficits set each domain's share of that capacity, and a sequential occupancy rule admits each expert delta up to the resulting layer-domain budget. Across 21 paired settings spanning seven dense base merges and three model pools, SigMerge improves every one (by 15.0% on average) and achieves the best average rank (1.67) among six merging methods, outperforming three categories of merging baselines.