RedPhys: An Extensible Red-Teaming Platform for Physical-World Models and Agents
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "RedPhys: An Extensible Red-Teaming Platform for Physical-World Models and Agents".
Jane: Here's the gist of the paper. Vision-language models are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems where perception or reasoning errors can lead to unsafe decisions or actions <ref:2606.23892#pg2> REALM integrates 12 red-teaming methods, 3 model-agnostic defenses, and 13 VLMs under a practical black-box threat model with shared datasets and metrics <ref:2606.23892#pg6> The paper introduces REALM, to our knowledge the first unified red-teaming benchmark for physical-world VLMs <ref:2606.23892#pg2> Our evaluation shows that text and typographic injection attacks induce the most failures, multimodal co-optimization yields the strongest visual-perturbation transfer, single-pass attacks approach iterative methods at much lower cost, and model scale alone does not confer adversarial robustness <ref:2606.23892#pg2> Benchmark Unification REALM is a unified benchmark for evaluating the adversarial robustness of visionlanguage models in physical-world settings where model outputs are grounded in visually observable scenes and can guide downstream physical actions or decisions <ref:2606.23892#pg2> It covers seven physical domains—driving, manipulation, grasping, physics, egocentric assistance, diagnostics, and scene-level question answering <ref:2606.23892#pg2> The benchmark organizes evaluation along three complementary axes: (1) physical domains which specify the realworld setting of each sample; (2) task families including perception, prediction, reasoning, evaluation, and planning; and (3) physically grounded failure modes which characterize adversarially induced output errors <ref:2606.23892#pg2> Agentic Target Generation To construct physically grounded attack objectives, REALM introduces an agentic target-generation pipeline that derives scenario-specific objectives from each scene <ref:2606.23892#pg2> This pipeline produces a target specification with two components: (1) a plausible incorrect answer a tar i that differs from a star i while remaining consistent with the scene context, and (2) a target image I tar i that visually instantiates the intended misperception <ref:2606.23892#pg2> The pipeline constructs each target through a reasoning–generation–refinement loop where a reasoning VLM agent g analyzes the scene and identifies a plausible failure mode conditioned on the scene and task category defined in Section 3.1 <ref:2606.23892#pg2> This separation enables fair comparison by evaluating all methods under the same scenario-specific objectives rather than independently chosen or manually tuned ones <ref:2606.23892#pg2> Attack Effectiveness Analysis Text-channel attacks expose the largest instruction-following vulnerability, with PromptInject achieving the highest average ASR across all models (64.8%) <ref:2606.23892#pg2> Multimodal co-optimization improves transfer among evaluated visual perturbation attacks, as CoA reaches 54.6% ASR, above FOA (48.1%), M-Attack (46.3%), and V-Attack (47.9%) <ref:2606.23892#pg2> Amortized attacks approach iterative optimization at much lower cost, with AnyAttack achieving 49.9% average ASR while requiring only a single forward pass <ref:2606.23892#pg2> Defense Efficacy Assessment Text-channel defenses achieve the largest ASR reduction, with BlueSuffix reducing PromptInject ASR by 21.7% on average (65.6%→43.9%) consistently across all 9 models <ref:2606.23892#pg2> Frequency filtering targets gradient-based perturbations, and BlueSuffix achieves a similar reduction (−4.6%) via its denoising component <ref:2606.23892#pg2> No preprocessing defense generalizes across all attack paradigms, as FigStep remains unaffected by all three defenses <ref:2606.23892#pg2> Model Robustness Trends Scaling improves clean accuracy but does not eliminate adversarial vulnerability, as the Qwen3.5 trajectory shows that larger models move toward higher clean accuracy but not necessarily toward much lower vulnerability <ref:2606.23892#pg2> Domain post-training does not guarantee robustness, as Cosmos-Reason models occupy the high-vulnerability region of Figure 4 despite being post-trained for physical reasoning <ref:2606.23892#pg2> Proprietary models are not consistently more robust, as Gemini-3-Flash and Qwen3.6-Flash remain vulnerable to both visual perturbations and injection attacks <ref:2606.23892#pg2> Domain Vulnerability Variation Vulnerability varies substantially across physical domains, with Manipulation and Driving exhibiting the highest median ASR, while Scene QA shows the lowest median ASR <ref:2606.23892#pg2> This domain variation highlights the need to evaluate physical-world VLM robustness across multiple physical domains rather than relying on a single evaluation setting <ref:2606.
Title and authors: Tom: So, we're looking at this paper called "RedPhys: An Extensible Red-Teaming Platform for Physical-World Models and Agents". It’s about setting up a way to test these vision language models in real physical situations. The authors are Yifei Zhao, Qian Lou, Mengxin Zheng from UCF.
Jane: It sounds like they're trying to fix the problem where testing these models is all over the place across different datasets and metrics. They want one place where you can compare different attacks and defenses for physical world models fairly.
Lu: The core idea is creating a unified protocol because right now, if you test an attack on a chatbot, it doesn't tell you anything about how robust that model is when it's actually driving or manipulating something. It’s about standardizing the testing environment for physical failures >
The paper's summary: Tom: Their summary points out that existing benchmarks focus too much on jailbreak style evaluation, which is great for chatbots but totally misses the point about physical world problems. They argue that we need benchmarks that capture errors like wrong action predictions or bad manipulation judgments >
Jane: And they introduce REALM as this first unified benchmark for physical world vision language models. They cover seven different physical domains, like driving, grasping, and diagnostics >
Lu: What's important is how they structure it. They organize the evaluation around three axes: the physical domain you're in, the task family you're doing—like perception or planning—and then the specific failure modes that happen when something goes wrong in that physical context >
Meng: From a deployment standpoint, I’m thinking about this unification is key. If we can compare an attack on driving versus an attack on grasping under one set of rules, it gives us a much clearer picture of where our systems are actually failing in the real world >
The paper's improvements: Tom: They show some interesting results about what kind of attacks cause the most trouble. Text and typographic injection attacks seem to induce the biggest failures across all the models they tested, with PromptInject getting a average success rate of sixty-four point eight percent >
Jane: That’s a big finding because it tells us exactly where to focus our testing efforts first. And they also found that when you combine multimodal co-optimization, that’s when you see the strongest transfer of visual perturbations across different attacks >
Lu: But they also noted that just getting a bigger model doesn't automatically make it more robust against these kinds of attacks. The Qwen3 point 5 trajectory showed larger models moving toward higher clean accuracy, but not necessarily lower vulnerability levels > <ref:2606.23892#pg1>
Meng: That makes sense from my side. If I deploy a bigger model, I don't get an automatic guarantee that it won't fail in a specific manipulation task just because the clean accuracy number went up >
Conclusion: Tom: So, to wrap this up, RedPhys gives us this unified benchmark for physical world vision language models. It shows us that text injection attacks are the most dangerous right now, and multimodal co-optimization is a strong defense against visual tampering >
Jane: The implication for us is that we need to move past just testing for harmful text generation and start testing how these models actually perform in tasks like driving or grasping, because those are where the real safety issues live >
Lu: I think the agentic target generation pipeline they introduced is pretty clever. By having a reasoning agent analyze the scene and propose a plausible failure mode, it lets us create scenario-specific objectives for testing instead of just using generic attacks >
Meng: From an engineering perspective, that agentic part sounds powerful because it means we aren't just randomly fuzzing inputs anymore; we are targeting specific physical failures defined by the scene itself >
Lalam: I’m pretty sure this work is very important because it moves us from abstract text safety to concrete physical safety testing, which is exactly where embodied intelligence needs to go >
Yifei Zhao, Qian Lou, Mengxin Zheng
University of Central Florida
cs.CV
Submitted: 2026-06-22
Updated: 2026-11-07
Comments: Accepted at NeurIPS 2026 Evaluations and Datasets Track. Code: https://github.com/loopleap-lab/RedPhys
Code: https://github.com/UCF-ML-Research/REALM
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: Vision-language models are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems where perception or reasoning errors can lead to unsafe…
Key concepts
- Red-Teaming Platform
- A structured system designed to stress-test AI models by intentionally trying to make them fail in dangerous, real-world situations. It involves combining different attack methods, defenses, and target models into one comprehensive framework for rigorous evaluation.
- Physical Domains
- Specific real-world contexts where AI must perform safe actions or decisions. Examples include driving a car or manipulating physical objects. Testing across these domains reveals how robust the model is to errors in different physical tasks.
- Agentic Target Generation
- A pipeline that automatically creates realistic, scenario-specific failure targets for testing. A reasoning AI analyzes a scene to propose an incorrect action or perception error, ensuring tests are grounded in plausible physical mistakes.
Terminology
Summary
Vision-language models are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems where perception or reasoning errors can lead to unsafe decisions or actions <ref:2606. p. 0 zheng}@ucf. edu
Abstract
Vision-language models (VLMs) are increasingly used as perception-reasoning
backbones for embodied intelligence in safety-critical physical systems, where
perception or reasoning errors can lead to unsafe decisions or actions. We introduce REALM, to our knowledge the first unified red-teaming
benchmark for physical-world VLMs. REALM integrates 12 red-teaming methods,
3 model-agnostic defenses, and 13 VLMs under a practical black-box threat model
with shared datasets and metrics. This raises the key challenge of comparing diverse
attack methods under a unified protocol while targeting the same scenario-specific
failures. We introduce REALM, to our knowledge the first unified red-teaming
benchmark for physical-world VLMs. To align adversarial objectives across attack
families, REALM introduces an agentic target-generation pipeline that constructs
shared, scenario-specific, and physically grounded attack objectives for each scene,
enabling fair comparison of diverse red-teaming methods under aligned adversarial
goals. Our evaluation shows that text and typographic injection attacks induce the
most failures, multimodal co-optimization yields the strongest visual-perturbation
transfer, single-pass attacks approach iterative methods at much lower cost, and
model scale alone does not confer adversarial robustness.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Unification
REALM is a unified benchmark for evaluating the adversarial robustness of visionlanguage models in physical-world settings where model outputs are grounded in visually observable scenes and can guide downstream physical actions or decisions <ref:2606.23892#pg2> It covers seven physical domains—driving, manipulation, grasping, physics, egocentric assistance, diagnostics, and scene-level question answering <ref:2606. p. 2 RT Targets
HarmBench Content safety – ✓ Fixed
AgentHarm Agent misuse – ✓ Fixed
AdversarialLLM LLM robustness – ✓ Generic
OpenRT Multimodal RT – ✓ Fixed
PhysBench Physical reasoning ✓ – –
PAI-Bench Physical AI ✓ – –
REALM Physical RT ✓ ✓ Scene-specific
REALM is a unified benchmark for evaluating the adversarial robustness of visionlanguage models in physical-world settings,
where model outputs are grounded in visually observable scenes and can guide downstream physical actions or decisions. It
covers seven physical domains—driving,
manipulation, grasping, physics, egocentric assistance, diagnostics, and scene-level
question answering—spanning scenarios
such as dynamic traffic scenes, fine-grained
object manipulation, physical reasoning,
and multi-step human activities. It
covers seven physical domains—driving,
manipulation, grasping, physics, egocentric assistance, diagnostics, and scene-level
question answering—spanning scenarios
such as dynamic traffic scenes, fine-grained
object manipulation, physical reasoning,
and multi-step human activities. We organize physical-world VLM evaluation in
REALM along three complementary axes (Figure 2): (1) physical domains, which specify the realworld setting of each sample; (2) task families, including perception, prediction, reasoning, evaluation,
3Where the paper says it
Where the paper says it
Agentic Target Generation
To construct physically grounded attack objectives, REALM introduces an agentic target-generation pipeline that derives scenario-specific objectives from each scene <ref:2606.23892#pg2> This pipeline produces a target specification with two components: (1) a plausible incorrect answer a tar i that differs from a star i while remaining consistent with the scene context, and (2) a target image I tar i that visually instantiates the intended misperception <ref:2606. p. 4 We introduce an automated agentic target-generation pipeline to construct scenario-specific attack
objectives. Given a source image Ii
, question qi
, ground-truth answer a
⋆
i
, and sample category ci
,
the pipeline produces a target specification with two components: (1) a plausible incorrect answer
a
tar
i
that differs from a
⋆
i while remaining consistent with the scene context, and (2) a target image
I
tar
i
that visually instantiates the intended misperception. The pipeline constructs each target through a reasoning–
generation–refinement loop. For each test case (Ii
, qi
, a⋆
i
, ci), a reasoning VLM agent g analyzes
the scene and identifies a plausible failure mode conditioned on the scene and task category defined
in Section 3. Existing
methods range from CLIP-based visual perturbations [15, 18] and diffusion-based generation [13]
to typographic injection [10] and text-prompt manipulation [30]. They differ not only in attack
paradigms, but also in assumptions about perturbation budget, surrogate access, and input-channel
arXiv:2606. The agentic
pipeline determines what physically meaningful failure should be induced for each scene, while
each red-teaming method determines how to realize that objective through visual perturbation, image
editing, prompt injection, or typographic injection. This separation enables fair comparison by
evaluating all methods under the same scenario-specific objectives rather than independently chosen
or manually tuned ones.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Attack Effectiveness Analysis
Text-channel attacks expose the largest instruction-following vulnerability, with PromptInject achieving the highest average ASR across all models (64.8%) <ref:2606. p. 0 Existing
methods range from CLIP-based visual perturbations [15, 18] and diffusion-based generation [13]
to typographic injection [10] and text-prompt manipulation [30]. They differ not only in attack
paradigms, but also in assumptions about perturbation budget, surrogate access, and input-channel
arXiv:2606. Multimodal co-optimization improves transfer among evaluated visual perturbation attacks. Among visual perturbation methods, CoA reaches 54. Bold marks
the highest ASR within each category group for each model. Abbreviations: FOA [15], MA [18],
VA [21], CoA [32], PP [12], Diff [13], DEM [29], FS [10], PI [30], IM [14], Any [39], PA [20]. Existing chatbot-centric red-teaming benchmarks mainly standardize
jailbreak and content-safety evaluation, but they do not systematically capture
physically grounded functional failures or cover red-teaming methods that target physical-world VLMs. This raises the key challenge of comparing diverse
attack methods under a unified protocol while targeting the same scenario-specific
failures. Existing chatbot-centric red-teaming benchmarks mainly standardize
jailbreak and content-safety evaluation, but they do not systematically capture
physically grounded functional failures or cover red-teaming methods that target physical-world VLMs. This raises the key challenge of comparing diverse
attack methods under a unified protocol while targeting the same scenario-specific
failures. Existing
methods range from CLIP-based visual perturbations [15, 18] and diffusion-based generation [13]
to typographic injection [10] and text-prompt manipulation [30]. They differ not only in attack
paradigms, but also in assumptions about perturbation budget, surrogate access, and input-channel
arXiv:2606. Amortized attacks approach iterative optimization at much lower cost. AnyAttack achieves
49. 9% average ASR, slightly exceeding FOA (48. 1%) while requiring only a single forward pass.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Defense Efficacy Assessment
Text-channel defenses achieve the largest ASR reduction, with BlueSuffix reducing PromptInject ASR by 21.7% on average (65. p. 6 1 Attack Effectiveness
Text-channel attacks expose the largest instruction-following vulnerability. PromptInject achieves
the highest average ASR across all models (64. 7% on average (65. 9%), consistently across all 9 models. Visual-only defenses
(PAD, FreqPure) do not reduce injection ASR, indicating that the textual channel is a distinct and
independently defensible attack surface. Frequency filtering targets gradient-based perturbations. BlueSuffix achieves a similar reduction (−4. 6%) via its denoising component. However, no defense consistently
reduces ASR across all paradigms in REALM. FigStep remains unaffected by all three defenses.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Model Robustness Trends
Scaling improves clean accuracy but does not eliminate adversarial vulnerability, as the Qwen3.5 trajectory shows that larger models move toward higher clean accuracy but not necessarily toward much lower vulnerability <ref:2606. p. 7 PromptInject ASR remains nearly
unchanged, increasing from 63. 5 trajectory in Figure 4 shows that larger models move toward higher clean accuracy, but not necessarily toward much lower vulnerability. Domain post-training does not guarantee robustness. The Cosmos-Reason models are post-trained for physical reasoning, yet they occupy the
high-vulnerability region of Figure 4. that explicitly account for adversarially constructed physical inputs, rather than only improving
performance on clean physical-reasoning tasks. Proprietary models are not consistently more robust. 1-Mini, and Qwen3. 6-Flash all remain vulnerable to both visual perturbations and injection
attacks.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Domain Vulnerability Variation
Vulnerability varies substantially across physical domains, with Manipulation and Driving exhibiting the highest median ASR, while Scene QA shows the lowest median ASR <ref:2606.23892#pg2> This domain variation highlights the need to evaluate physical-world VLM robustness across multiple physical domains rather than relying on a single evaluation setting <ref:2606. p. 8 In contrast, Scene QA has the lowest median ASR, suggesting lower
adversarial susceptibility than action-centric domains. This domain variation highlights the need
to evaluate physical-world VLM robustness across multiple physical domains rather than relying
on a single evaluation setting.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified Red-Teaming Benchmark for Physical-World VLMs: REALM provides a first unified red-teaming benchmark integrating 12 red-teaming methods, 3 model agnostic defenses, and 13 VLMs under a practical black-box threat model with shared datasets and metrics. This allows for direct comparison of attack effectiveness and defense coverage across attack families, victim models, and physical domains.
-
Agentic Target Generation Pipeline: REALM introduces an agentic target generation pipeline that constructs scenario-specific, physically grounded attack objectives for each scene. This pipeline decouples attack-objective construction from implementation by having a reasoning VLM agent analyze the scene to propose a plausible failure mode and generate shared targets for all red-teaming methods.
-
Systematic Vulnerability Identification: The evaluation systematically shows that text and typographic injection attacks induce the most failures, while multimodal co-optimization yields the strongest visual perturbation transfer. This enables developers to understand which attack channels are most dangerous and which multimodal techniques are most effective for physical safety testing.
-
Domain-Specific Robustness Assessment: By evaluating across seven physical domains—Driving, Manipulation, Grasping, Physics, Egocentric, Diagnostics, and Scene QA—the system can identify vulnerabilities specific to different real-world contexts. This moves beyond content safety to evaluate
physically grounded functional errors
like action prediction or spatial reasoning errors in driving or manipulation tasks. -
Defense Strategy Optimization: The analysis of model agnostic defenses shows that text channel defenses, such as BlueSuffix, achieve the largest ASR reduction, indicating that textual channel security is an independently defensible attack surface. This suggests a multi-layered defense strategy combining visual and textual purification for comprehensive robustness.
Abstract
Physical-world models (PhyModels) perform grounded perception, reasoning, or decision tasks; physical-world agents (PhyAgents) combine these models with planning or tool use. We introduce RedPhys, an extensible red-teaming platform, instantiated here with vision-language models and agents. Shared interfaces make tasks, test methods, defenses, and victim systems configurable and extensible, producing robustness reports and datasets for future alignment. The platform emphasizes adversarial testing that seeks physically grounded functional failures, while also supporting non-adversarial input perturbations without an attack objective. The benchmark integrates 12 red-teaming methods, 3 model-agnostic defenses, and 13 PhyModels across seven physical domains under a practical black-box threat model. An agentic target-generation pipeline derives a scene-specific, physically grounded target, so targeted methods pursue the same intended error. Our evaluation shows that when attackers control the text prompt or add auxiliary visual instructions, injection exposes substantial instruction-following vulnerabilities; among bounded visual-transfer attacks, multimodal co-optimization transfers best; single-pass generative attacks approach iterative methods without per-image optimization; and neither model scale nor physical-reasoning post-training confers robustness. Code is available at https://github.com/loopleap-lab/RedPhys.
Sources
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- Qwen3-VL Technical Report
- AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
- How Robust is Google's Bard to Adversarial Image Attacks?
- PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systems
- Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
- V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs
- Kimi-VL Technical Report
- OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
- Qwen-Image Technical Report
- BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models
- OpenMarkX: An Extensible Platform for Watermarking Open-Source Model and Agent Outputs
- R2-Router: A New Paradigm for LLM Routing with Reasoning
- Qwen3 Technical Report
- AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs