Which Image Property Carries the Jailbreak? A Controlled Dissection of Image-to-Text Jailbreaks
cs.CR, cs.AI, cs.LG
Submitted: 2026-10-04
Updated: 2026-10-04
Code: https://github.com/BoyuanChen99/Which_Image_
License: http://creativecommons.org/licenses/by/4.0/
The gist: Image-to-text jailbreaks place harmful intent in text, image content, or the relationship between them.
Abstract
Image-to-text jailbreaks place harmful intent in text, image content, or the relationship between them. We examine image-side factors across four published attack families on a 313-prompt StrongREJECT slice, using five multimodal models and an additional appendix evaluation of InternVL3.5-8B. The harmful instruction is held constant across conditions; the baseline matrix uses one draw per prompt, and paired ablations use three draws with an automated rubric judge. A bare harmful query, with or without a benign unrelated image, produces little attack success on most victims, while attack images substantially increase it. First, per-tile entropy and JPEG size do not reliably distinguish attack tiles from size-matched benign distractors, limiting density-only screening. Second, earlier tile-count ladders were confounded by payload visibility. A corrected region-count test found no detectable effect, so the role of tile-count structure remains unresolved. Third, on Qwen3-VL-8B, the E4 manipulation that removes query-specific relatedness lowers ASR by about 0.12. This supports a bounded attribution to the relatedness manipulation, although image-text congruence remains unmeasured. A within-category control reproduces the direction on overlapping stimuli. Five tests survive the global statistical correction, but only E4 supports attribution to one measured descriptor; the within-category result is a robustness check, not a separate attribution. These conclusions remain conditional on the rubric judge.
Sources
- Qwen3-VL Technical Report
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models
- FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts
- Gemma 3 Technical Report
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models
- Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity
- Visual Adversarial Examples Jailbreak Aligned Large Language Models
- Learning Transferable Visual Models From Natural Language Supervision
- VisCRA: A Visual Chain Reasoning Attack for Jailbreaking Multimodal Large Language Models
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
- White-box Multimodal Jailbreaks Against Large Vision-Language Models
- AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting
- VRSA: Jailbreaking Multimodal Large Language Models through Visual Reasoning Sequential Attack
- Distraction is All You Need for Multimodal Large Language Model Jailbreaking
- $\textit{MMJ-Bench}$: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs