Scale-Induced Semantic Shift in Visual Prompt Injection Against VLMs and Web Agents
cs.AI
Submitted: 2026-11-12
Updated: 2026-11-12
Terminology
Sources
- Visual Instruction Tuning
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- Explaining and Harnessing Adversarial Examples
- Adversarial Patch
- Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs
- Jailbreaking Attack against Multimodal Large Language Model
- White-box Multimodal Jailbreaks Against Large Vision-Language Models
- FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
- AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents
- Prompt Injection attack against LLM-integrated Applications
- WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
- VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents
- Defending Against Indirect Prompt Injection Attacks With Spotlighting
- Can Indirect Prompt Injection Attacks Be Detected and Removed?
- Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents
- Diffusion Illusions: Hiding Images in Plain Sight
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents