Multimodal Safety Evaluation Should Measure Controllability Beyond Classification
cs.AI
Submitted: 2026-10-05
Updated: 2026-10-05
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: VLM safety is commonly evaluated through input- and output-level classification.
Abstract
VLM safety is commonly evaluated through input- and output-level classification. Such classification is necessary, but it does not reveal whether a safety state is accessible or controllable inside the model. We argue that multimodal safety evaluation should therefore report a controllability profile alongside behavioral classification, separating representation-level detectability, cross-modal specificity, intervention sensitivity, and benign-preserving selectivity. Using implicit toxicity as a stress case, we instantiate this profile on LlavaGuard and Qwen3.5 with sparse feature decompositions. LlavaGuard admits localized handles with a narrow benign-preserving intervention range and modest downstream safety gains, whereas Qwen3.5 supports strong representation-level readout but no comparable selective-control regime under the tested operators. These results show that internal readout and controllability can diverge. Future multimodal safety benchmarks should therefore report not only behavioral safety metrics, but also whether safety-relevant internal signals can be intervention-tested and controlled within a validated operating range.
Sources
- Qwen2.5-VL Technical Report
- When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models
- ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs
- Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
- DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models
- JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models
- When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
- CEE: An Inference-Time Jailbreak Defense for Embodied Intelligence via Subspace Concept Rotation
- Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention
- Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems