RedPhys: An Extensible Red-Teaming Platform for Physical-World Models and Agents

summary

The gist

Vision-language models are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems where perception or reasoning errors can lead to unsafe…

In short

RedPhys introduces an extensible red-teaming platform for physical-world models and agents. It provides a unified benchmark integrating various red-teaming methods, defenses, and vision-language models across seven physical domains like driving and manipulation. This allows researchers to systematically test how these AI systems fail in real-world scenarios due to perception or reasoning errors.

Key concepts

Red-Teaming Platform
A structured system designed to stress-test AI models by intentionally trying to make them fail in dangerous, real-world situations. It involves combining different attack methods, defenses, and target models into one comprehensive framework for rigorous evaluation.
Physical Domains
Specific real-world contexts where AI must perform safe actions or decisions. Examples include driving a car or manipulating physical objects. Testing across these domains reveals how robust the model is to errors in different physical tasks.
Agentic Target Generation
A pipeline that automatically creates realistic, scenario-specific failure targets for testing. A reasoning AI analyzes a scene to propose an incorrect action or perception error, ensuring tests are grounded in plausible physical mistakes.

Terminology used across episodes

This episode discusses

The paper

RedPhys: An Extensible Red-Teaming Platform for Physical-World Models and Agents · Read on arXiv

Yifei Zhao, Qian Lou, Mengxin Zheng

University of Central Florida

Physical-world models (PhyModels) perform grounded perception, reasoning, or decision tasks; physical-world agents (PhyAgents) combine these models with planning or tool use. We introduce RedPhys, an extensible red-teaming platform, instantiated here with vision-language models and agents. Shared interfaces make tasks, test methods, defenses, and victim systems configurable and extensible, producing robustness reports and datasets for future alignment. The platform emphasizes adversarial testing that seeks physically grounded functional failures, while also supporting non-adversarial input perturbations without an attack objective. The benchmark integrates 12 red-teaming methods, 3 model-agnostic defenses, and 13 PhyModels across seven physical domains under a practical black-box threat model. An agentic target-generation pipeline derives a scene-specific, physically grounded target, so targeted methods pursue the same intended error. Our evaluation shows that when attackers control the text prompt or add auxiliary visual instructions, injection exposes substantial instruction-following vulnerabilities; among bounded visual-transfer attacks, multimodal co-optimization transfers best; single-pass generative attacks approach iterative methods without per-image optimization; and neither model scale nor physical-reasoning post-training confers robustness. Code is available at https://github.com/loopleap-lab/RedPhys.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "RedPhys: An Extensible Red-Teaming Platform for Physical-World Models and Agents".

Jane: Here's the gist of the paper. Vision-language models are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems where perception or reasoning errors can lead to unsafe decisions or actions <ref:2606.23892#pg2> REALM integrates 12 red-teaming methods, 3 model-agnostic defenses, and 13 VLMs under a practical black-box threat model with shared datasets and metrics <ref:2606.23892#pg6> The paper introduces REALM, to our knowledge the first unified red-teaming benchmark for physical-world VLMs <ref:2606.23892#pg2> Our evaluation shows that text and typographic injection attacks induce the most failures, multimodal co-optimization yields the strongest visual-perturbation transfer, single-pass attacks approach iterative methods at much lower cost, and model scale alone does not confer adversarial robustness <ref:2606.23892#pg2> Benchmark Unification REALM is a unified benchmark for evaluating the adversarial robustness of visionlanguage models in physical-world settings where model outputs are grounded in visually observable scenes and can guide downstream physical actions or decisions <ref:2606.23892#pg2> It covers seven physical domains—driving, manipulation, grasping, physics, egocentric assistance, diagnostics, and scene-level question answering <ref:2606.23892#pg2> The benchmark organizes evaluation along three complementary axes: (1) physical domains which specify the realworld setting of each sample; (2) task families including perception, prediction, reasoning, evaluation, and planning; and (3) physically grounded failure modes which characterize adversarially induced output errors <ref:2606.23892#pg2> Agentic Target Generation To construct physically grounded attack objectives, REALM introduces an agentic target-generation pipeline that derives scenario-specific objectives from each scene <ref:2606.23892#pg2> This pipeline produces a target specification with two components: (1) a plausible incorrect answer a tar i that differs from a star i while remaining consistent with the scene context, and (2) a target image I tar i that visually instantiates the intended misperception <ref:2606.23892#pg2> The pipeline constructs each target through a reasoning–generation–refinement loop where a reasoning VLM agent g analyzes the scene and identifies a plausible failure mode conditioned on the scene and task category defined in Section 3.1 <ref:2606.23892#pg2> This separation enables fair comparison by evaluating all methods under the same scenario-specific objectives rather than independently chosen or manually tuned ones <ref:2606.23892#pg2> Attack Effectiveness Analysis Text-channel attacks expose the largest instruction-following vulnerability, with PromptInject achieving the highest average ASR across all models (64.8%) <ref:2606.23892#pg2> Multimodal co-optimization improves transfer among evaluated visual perturbation attacks, as CoA reaches 54.6% ASR, above FOA (48.1%), M-Attack (46.3%), and V-Attack (47.9%) <ref:2606.23892#pg2> Amortized attacks approach iterative optimization at much lower cost, with AnyAttack achieving 49.9% average ASR while requiring only a single forward pass <ref:2606.23892#pg2> Defense Efficacy Assessment Text-channel defenses achieve the largest ASR reduction, with BlueSuffix reducing PromptInject ASR by 21.7% on average (65.6%→43.9%) consistently across all 9 models <ref:2606.23892#pg2> Frequency filtering targets gradient-based perturbations, and BlueSuffix achieves a similar reduction (−4.6%) via its denoising component <ref:2606.23892#pg2> No preprocessing defense generalizes across all attack paradigms, as FigStep remains unaffected by all three defenses <ref:2606.23892#pg2> Model Robustness Trends Scaling improves clean accuracy but does not eliminate adversarial vulnerability, as the Qwen3.5 trajectory shows that larger models move toward higher clean accuracy but not necessarily toward much lower vulnerability <ref:2606.23892#pg2> Domain post-training does not guarantee robustness, as Cosmos-Reason models occupy the high-vulnerability region of Figure 4 despite being post-trained for physical reasoning <ref:2606.23892#pg2> Proprietary models are not consistently more robust, as Gemini-3-Flash and Qwen3.6-Flash remain vulnerable to both visual perturbations and injection attacks <ref:2606.23892#pg2> Domain Vulnerability Variation Vulnerability varies substantially across physical domains, with Manipulation and Driving exhibiting the highest median ASR, while Scene QA shows the lowest median ASR <ref:2606.23892#pg2> This domain variation highlights the need to evaluate physical-world VLM robustness across multiple physical domains rather than relying on a single evaluation setting <ref:2606.

Title and authors: Tom: So, we're looking at this paper called "RedPhys: An Extensible Red-Teaming Platform for Physical-World Models and Agents". It’s about setting up a way to test these vision language models in real physical situations. The authors are Yifei Zhao, Qian Lou, Mengxin Zheng from UCF.

Jane: It sounds like they're trying to fix the problem where testing these models is all over the place across different datasets and metrics. They want one place where you can compare different attacks and defenses for physical world models fairly.

Lu: The core idea is creating a unified protocol because right now, if you test an attack on a chatbot, it doesn't tell you anything about how robust that model is when it's actually driving or manipulating something. It’s about standardizing the testing environment for physical failures >

The paper's summary: Tom: Their summary points out that existing benchmarks focus too much on jailbreak style evaluation, which is great for chatbots but totally misses the point about physical world problems. They argue that we need benchmarks that capture errors like wrong action predictions or bad manipulation judgments >

Jane: And they introduce REALM as this first unified benchmark for physical world vision language models. They cover seven different physical domains, like driving, grasping, and diagnostics >

Lu: What's important is how they structure it. They organize the evaluation around three axes: the physical domain you're in, the task family you're doing—like perception or planning—and then the specific failure modes that happen when something goes wrong in that physical context >

Meng: From a deployment standpoint, I’m thinking about this unification is key. If we can compare an attack on driving versus an attack on grasping under one set of rules, it gives us a much clearer picture of where our systems are actually failing in the real world >

The paper's improvements: Tom: They show some interesting results about what kind of attacks cause the most trouble. Text and typographic injection attacks seem to induce the biggest failures across all the models they tested, with PromptInject getting a average success rate of sixty-four point eight percent >

Jane: That’s a big finding because it tells us exactly where to focus our testing efforts first. And they also found that when you combine multimodal co-optimization, that’s when you see the strongest transfer of visual perturbations across different attacks >

Lu: But they also noted that just getting a bigger model doesn't automatically make it more robust against these kinds of attacks. The Qwen3 point 5 trajectory showed larger models moving toward higher clean accuracy, but not necessarily lower vulnerability levels > <ref:2606.23892#pg1>

Meng: That makes sense from my side. If I deploy a bigger model, I don't get an automatic guarantee that it won't fail in a specific manipulation task just because the clean accuracy number went up >

Conclusion: Tom: So, to wrap this up, RedPhys gives us this unified benchmark for physical world vision language models. It shows us that text injection attacks are the most dangerous right now, and multimodal co-optimization is a strong defense against visual tampering >

Jane: The implication for us is that we need to move past just testing for harmful text generation and start testing how these models actually perform in tasks like driving or grasping, because those are where the real safety issues live >

Lu: I think the agentic target generation pipeline they introduced is pretty clever. By having a reasoning agent analyze the scene and propose a plausible failure mode, it lets us create scenario-specific objectives for testing instead of just using generic attacks >

Meng: From an engineering perspective, that agentic part sounds powerful because it means we aren't just randomly fuzzing inputs anymore; we are targeting specific physical failures defined by the scene itself >

Lalam: I’m pretty sure this work is very important because it moves us from abstract text safety to concrete physical safety testing, which is exactly where embodied intelligence needs to go >

More episodes

← Home