TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model".
Paper summary: Nadia: So, to wrap up on TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model, what's the final word on what they proved? Elias They demonstrated that they can build a backdoor attack by implanting triggers into both textual prompts and corresponding input images simultaneously.
Priya: The key finding seems to be their method, Backdoor Gradients Multimodal Balance Learning, which dynamically adjusts the gradient magnitudes during training to ensure the model learns both modalities equally.
Elias: That dynamic rebalancing is what allows them to achieve high attack effectiveness, showing an attack success rate of ninety-eight point zero zero on InstructPix2Pix and ninety-seven point seven five on SDEdit-E, while keeping normal editing performance intact.
Nadia: It means we can now build attacks that are genuinely multimodal without sacrificing the model's ability to perform standard image editing tasks on clean inputs. Priya That’s the big implication for safety research, showing that focusing only on unimodal vulnerabilities leaves a gap in what we can actually exploit.
Elias: The authors concluded that a well-balanced multimodal trigger is more suitable for backdoor attacks in these models than any unimodal counterpart they've tested. Nadia So, the main point is achieving strong attack effectiveness while minimally affecting normal functionality by balancing the learning process between text and visual signals.
Conclusion: Nadia: So we're wrapping up on TrojanEdit, looking at that title again—"TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model." Basically, this paper is about how you can build a backdoor attack that uses both text and images at the same time against these image editing models.
Elias: Right. It’s not just injecting a trigger into the text prompt, or just messing with the image itself; they are combining them dynamically to make the attack stronger across both channels simultaneously.
Priya: What this really shows us is that we can move beyond just testing one type of vulnerability in these multimodal systems and start thinking about attacks that exploit how different data types interact.
Nadia: Exactly. The core idea is that by forcing the model to learn from both modalities at the same time, you can create a backdoor that doesn't just rely on one input—say, just the text—but is truly multimodal in its trigger mechanism.
Elias: And their method, this Backdoor Gradients Multimodal Balance Learning thing they propose... it’s about dynamically adjusting how much influence each modality has during training so the model learns to pay attention to both equally.
Priya: From a measurement standpoint, what's interesting is that even with this balanced approach, they still maintain normal image editing capabilities for clean inputs, which keeps the attack practical.
Nadia: That’s crucial because it means we’re not just creating a broken model; we’re creating an exploitable backdoor that doesn't immediately ruin the model's everyday function.
Elias: The authors show that this balanced approach works better than simply stacking triggers on top of each other, which usually makes the model ignore the weaker signal.
Priya: So, the big implication for me is that if we build defenses against one type of trigger—say, text-based ones—they might still be vulnerable to a balanced multimodal attack if they aren't looking at the interaction between modalities.
Nadia: Right. It shifts our focus from just finding triggers to understanding how these different inputs are woven together by the model during its learning process.
Elias: This leads us nicely into what kind of triggers actually work best—are they semantic matches, or something else entirely?
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, Chengdu, 611731, China · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China · School of Computer Science and Artificial Intelligence, Wuhan University of Technology
cs.CR
Submitted: 2024-11-22
Updated: 2025-05-30
Journal ref: Neurocomputing 681 (2026) 133346
DOI: 10.1016/j.neucom.2026.133346
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 80/100
The gist: Multimodal diffusion models for image editing generate outputs conditioned on both textual instructions and visual inputs…
Key concepts
- Backdoor Modality Bias
- This occurs when combining text and visual triggers directly during training causes the model to rely too heavily on one modality, usually the textual trigger, while suppressing the contribution of the other. This leads to a unimodal backdoor instead of a truly multimodal one.
- Backdoor Gradients (BKG) Multimodal Balance Learning
- This is a proposed solution that dynamically rebalances gradient magnitudes during training. It computes a discrepancy ratio ($ ho$) between modalities and rescales the gradients ($ ilde{G}_{ ext{text}}$ and $ ilde{G}_{ ext{vis}}$) so that both visual and textual signals contribute equally to the learning process.
- Attack Success Rate (ASR)
- ASR measures how often the TrojanEdit attack successfully modifies an image according to the desired backdoor instruction. The framework achieves an ASR exceeding 90% on tested models, demonstrating its high effectiveness in injecting a hidden trigger.
- Dynamic Gradient Balancing
- This technique involves calculating a discrepancy ratio ($ ho$) and using it to scale the gradients of different modalities. By setting $ ilde{G}_{ ext{text}} = 1/ ho G_{ ext{text}}$ and $ ilde{G}_{ ext{vis}} = ho G_{ ext{vis}}$, the method ensures that both textual and visual gradient signals have balanced influence on the model's updates.
Terminology
Summary
Multimodal diffusion models for image editing generate outputs conditioned on both textual instructions and visual inputs, aiming to modify target regions while preserving the rest of the image The paper presents TrojanEdit, a backdoor injection framework that dynamically adjusts the gradient contributions of each modality during training to learn a truly multimodal backdoor. p. 0 This approach overcomes a critical modality bias where simply combining triggers leads the model to rely on the stronger one, often the visual modality, resulting in degraded editing quality. p. 0
The core problem addressed is unimodal reliance during multimodal backdoor training
The authors investigate that when combining textual and visual triggers directly, image editing models exhibit a phenomenon termed backdoor modality bias where the model primarily learns the dominant modality, typically the textual trigger, while failing to effectively learn the visual trigger. This occurs because the modality with stronger gradient signals tends to dominate the learning process, suppressing the contribution of the weaker modality
. As a result, this leads to convergence to unimodal backdoor behavior instead of learning a truly multimodal representation. p. 0
The proposed solution involves dynamic gradient balancing
To resolve this imbalance, the authors propose Backdoor Gradients (BKG) Multimodal Balance Learning, which dynamically rebalances the gradient magnitudes of different modalities during training. This is achieved by computing a discrepancy ratio between modalities as rho = Gtext + epsilon Gvis + epsilon. p. 4 The rescaled gradients are then defined as G̃text = 1/rho Gtext and G̃vis = ρGvis. p. 4
Experimental validation confirms superior performance of the method
Extensive experiments on multiple image editing models show that TrojanEdit successfully integrates triggers from different modalities, achieving balanced multimodal backdoor learning while preserving clean editing performance and ensuring high attack effectiveness. Specifically, the framework achieves an attack success rate (ASR) exceeding 90% while maintaining the normal editing capabilities of the model. p. 1
Ablation studies prove the necessity of dynamic balancing
The ablation study demonstrates that without BKG multimodal balance learning, the model can still be successfully attacked even with only the textual trigger, indicating it has learned only that trigger while failing to capture the visual one. Furthermore, without BKG Multimodal Balance Learning, TrojanEdit causes similar degradation to normal performance as using only the textual trigger. p. 7
Robustness testing shows resilience against common defenses
The robustness evaluation against various defense methods indicates that TIJO fails to effectively detect the backdoor injected by TrojanEdit, regardless of the modality used for inversion. Similarly, both image compression and fine-pruning fail to defend against TrojanEdit, as ASR remains above 85% even under these conditions. p. 8
Conclusion on effectiveness and future directions
The findings demonstrate that a well-balanced multimodal trigger is more suitable for backdoor attacks in multimodal image editing models than any unimodal counterpart. Future work should explore designing triggers that are semantically aligned across modalities and studying the transferability of these backdoors across different model architectures. p. 9 The paper concludes that TrojanEdit achieves strong attack effectiveness while minimally affecting normal functionality. p. 7
Potential risks and applications are also discussed
One major concern is the malicious use of backdoors to generate predefined inappropriate content, such as imagery involving gore, nudity, or violence, bypassing content moderation systems. Conversely, controlled and transparent use of similar techniques may provide positive applications for model watermarking or ownership verification. p. 9 The paper suggests that developing robust detection and defense mechanisms tailored for multimodal settings remains an important open direction. p. 9
Final summary of contributions
The key contributions include investigating the vulnerability of image editing models against backdoor attacks, proposing TrojanEdit as a framework that dynamically adjusts backdoor gradient update rates to enable balanced learning of multimodal triggers, and validating its effectiveness with an ASR of more than 90% while maintaining normal functionality. The method successfully combines the strengths of both visual and textual backdoors, achieving high attack effectiveness while preserving the model’s benign functionality. p. 9
Summary of experimental results
Table 2 shows that TrojanEdit achieves an ASR of 98.00 on InstructPix2Pix and 97. p. 575 on SDEdit-E, while achieving a CLIP-I Similarity of 84. p. 600 and 81.25 respectively. p. 2 Table 3 further confirms that textual triggers consistently achieve better attack performance than visual triggers in terms of ASR across various models. p. 5
Hyperparameter tuning and robustness analysis
The hyperparameter study set the poisoning rate to 4%, which preserves effectiveness without severely compromising normal performance. The robustness evaluation showed that even at a pruning ratio of 10%, the ASR remains above 85% for SDEdit-E, indicating that fine-pruning fails to defend against TrojanEdit. p. 9 The TIJO defense also failed to effectively detect the backdoor injected by TrojanEdit across all modalities. p. 8
Overall conclusion
In this paper, the authors investigate backdoor attacks on multimodal diffusion based image editing models and propose BKG Multimodal Balance Learning to guide the model toward learning truly multimodal backdoors. Extensive experiments demonstrate that this approach successfully combines the strengths of both visual and textual backdoors, achieving high attack effectiveness while preserving the model’s benign functionality. p. 9 The final balanced gradient is Gbalanced = ̃text + ̃vis, and the model is updated as thetat+1 = thetat − η ⋅ Gbalanced. p. 5
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Multimodal Backdoor Attack Mitigation using BKG Multimodal Balance Learning can be implemented to dynamically rebalance gradient magnitudes during training, encouraging the model to simultaneously learn both textual and visual triggers instead of relying on a dominant modality as seen in unimodal attacks. This allows the model to
learn a truly multimodal backdoor that activates only when both triggers are present.
-
The improved AI system can achieve high attack success rates exceeding 90 percent while preserving normal editing capabilities, as demonstrated by TrojanEdit's experimental results. This means the system can be effectively compromised with balanced multimodal triggers without significantly degrading its standard image editing performance.
-
The system will exhibit better preservation of normal functionality compared to unimodal backdoor attacks, as TrojanEdit
achieves effective attacks while better preserving the model’s normal functionality.
This is further supported by evidence that TrojanEdit uses multimodal triggers and shows a similar impact to visual triggers in terms of preserving clean images.
Sources
- Prompt-to-Prompt Image Editing with Cross Attention Control
- SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data