DiffSafeMerge: Mitigating Backdoor Inheritance in Diffusion Model Merging

DiffSafeMerge (DSM) is introduced to mitigate backdoor inheritance when merging unconditional diffusion checkpoints. The problem is that a compromised public checkpoint can transfer a dormant backdoor while clean generation appears normal, and mitigation is difficult without knowing the compromised source, trigger, or target. DSM uses a small unlabeled clean set and fixed, attack-agnostic stress probes to score source blocks, shrink suspicious contributions toward a trusted reference, and select attenuation under a clean denoising-loss budget.

The method is based on the hypothesis that a risky source block produces a stress-amplified response that remains stable as its merge weight changes, whereas removing a useful block incurs a denoising cost. DSM's "Ratio-Scanned Stress-Safe (R3S) score combines source–block stress amplification and directional consistency across fixed probes and ratio paths. Clean utility discounts blocks whose removal harms denoising. DSM then applies a source-level floor, shrinks suspicious contributions toward the trusted reference without weight redistribution, and selects the attenuation strength under a clean-loss budget."

The threat model assumes the defender has white-box access to the sources, reference, intended weights, and a small unlabeled clean set C, but faces constraints: No suspicion-only rejection, No exhaustive audit, No source retraining, and Unknown attack. The security goal is a deterministic merge-time transformation satisfying L clean(theta safe) (1+ tau)L clean(theta int), where tau is a predeclared utility budget.

In the evaluation, "Intended merging has zero worst-target ASR in 10 of 14 source cases, but retains 48–100% in three BadMerging cases and 0.2% in VillanDiffusion CIFAR-10 S2. DSM preserves the first 10 outcomes and records no target match in the remaining four over three seeds. The zero-success counts are 0/3,072 per CIFAR-10 condition and 0/768 per CelebA-HQ condition, with 95% Wilson upper bounds of 0.125% and 0.498%. Case-averaged FID is 49.67 and 164.07" for CIFAR-10 and CelebA-HQ, respectively.

Compared to baselines, Only ANP and DSM have zero worst-target ASR on both datasets; DSM has lower case-averaged FID: 50.2 versus 243.4 on CIFAR-10 and 166.2 versus 418.0 on CelebA-HQ. DSM does not claim per-case FID dominance, as MaxFusion has lower FID in all three CelebA-HQ S1 cases, but fuses sources at inference rather than producing one checkpoint and retains target matches in S2.

Component analysis on a development case shows that removing the source floor, renormalizing source weights, and excluding output-time evidence yield HAT ASRs of 97.95%, 100%, and 4.66%. Sensitivity analysis shows ASR is 100% through η = 0.5, 0.749% at η = 0.75, and zero at η = 1 as FID falls from 435.41 to 57.11. In an adaptive stress test where the attacker optimizes against the exact R3S score, DSM retains 0.0 ± 0.0% ASR as FID changes from 49.58 ± 0.54 to 52.47 ± 0.47.

The conclusion states: "DiffSafeMerge mitigates unknown source contamination with ratio-scanned scoring and clean-constrained shrinkage. Across four attacks, two datasets, and 21 target conditions, DSM preserves already-diluted zero-ASR cases and records no target match in four inherited-risk cases over three seeds; three have 48–100% intended-merge ASR. Among methods reaching zero worst-target ASR on both datasets, DSM has the lowest matched seed-0 case-averaged FID."

Improvements to AI Systems:

1. Backdoor-Resistant Model Merging: Integrate DSM’s ratio-scanned stress probing and clean-loss-constrained shrinkage into any diffusion model merging pipeline. This allows AI systems to safely combine public checkpoints without inheriting dormant backdoors, even when the attacker’s trigger, target, or source is unknown.

2. Utility-Preserving Security Guarantees: Adopt the deterministic merge-time transformation with a predeclared utility budget τ. The improved system can guarantee that the merged model’s clean denoising loss does not exceed (1+τ) of the intended merge, ensuring that security measures do not degrade generation quality beyond an acceptable threshold.

3. Attack-Agnostic Threat Detection: Use fixed, attack-agnostic stress probes (e.g., random noise perturbations) to score source blocks. The improved system can identify risky components without needing to enumerate possible attack patterns, making it robust to novel or adaptive backdoor strategies.

4. Source-Level Attenuation Without Weight Redistribution: Apply a source-level floor and shrink suspicious contributions toward a trusted reference, without renormalizing weights. This prevents unintended side effects on benign blocks and maintains the original merge’s structural integrity, improving reliability in multi-source scenarios.

5. Adaptive Defense Against Optimized Attacks: Leverage the R3S score’s directional consistency across merge-weight paths. The improved system can maintain near-zero attack success rates even when an adversary explicitly optimizes against the defense, as shown by the 0.0% ASR under adaptive stress testing.

6. Efficient Clean-Data Utilization: Require only a small unlabeled clean set for calibration. The improved system can operate in data-scarce environments, using minimal clean samples to set attenuation strength under a clean-loss budget, reducing the defender’s data dependency.

7. Scalable Multi-Condition Safety: Apply the method across diverse datasets (e.g., CIFAR-10, CelebA-HQ) and multiple target conditions. The improved system can generalize safety guarantees across different image domains, with worst-case ASR upper bounds below 0.5% and lower FID than existing defenses like ANP.

8. Post-Merge Verification: Provide a deterministic, one-time transformation that yields a single safe checkpoint. The improved system can be deployed in production where inference-time fusion (e.g., MaxFusion) is impractical, offering a static model with verified zero target matches across multiple seeds.

What the Improved AI System Can Do:

• Safely merge any combination of public and private diffusion checkpoints, even if one or more are compromised, without prior knowledge of the attack.

• Guarantee that the merged model’s clean image generation quality stays within a user-defined budget (e.g., FID within 5% of intended merge).

• Detect and neutralize backdoors that are dormant during normal generation but activate under specific stress conditions.

• Operate with minimal clean data (e.g., a few hundred unlabeled images) and no access to attack samples, triggers, or source labels.

• Resist adaptive attackers who know the defense’s exact scoring function, maintaining 0% attack success while incurring only a small FID penalty (e.g., +2.89 on CIFAR-10).

• Provide a single, deployable checkpoint (not an inference-time ensemble) that is safe for downstream tasks like image editing, inpainting, or style transfer.