AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework

arXiv:2606.18532 · cs.CR, cs.AI, cs.RO, cs.SE · Submitted 2026-06-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework".

Elias: Here's the gist of the paper. AI systems are increasingly evaluated within bounded environments that integrate isolation, simulation, instrumentation, supervision, and evidence capture.

Nadia: This article develops an assurance-oriented account of AI sandboxes as controlled environments designed for testing, evaluation, verification, and validation across digital AI, embodied autonomy, and cyber-physical deployments.

Elias: The authors formalize the sandbox boundary and introduce a weakestlink rule for composing per-dimension evidence into a bounded deployment claim.

Nadia: They also define a cyberphysical threat model that explicitly includes attacks on the assurance apparatus itself.

Paper summary: Nadia: We just talked about how they define the core structure of an AI sandbox as a bounded and instrumented evaluation environment. The key idea here is shifting the focus from just testing success to creating a system-level assurance discipline for physical and cyber-physical systems >

Elias: They formalize this by defining that boundary using that tuple S equals U E B C M I A R, which captures all the controllable and monitored aspects of the test environment >

Priya: And they connect that structure to a specific rule for evidence composition—the weakestlink rule—which dictates how you assemble proof for a deployment claim based on those fifteen dimensions >

Nadia: That weakestlink rule means that if even one dimension's assumption or uncertainty is fuzzy, the entire deployment claim loses its strength and becomes absent >

Elias: They also lay out a cross-domain taxonomy of five non-exclusive sandbox archetypes: simulation-based, digital twin, adversarial and security, regulatory and agent-based sandboxes >

Priya: I'm interested in that taxonomy because it shows they aren't sticking to just one type of testing environment but acknowledging the different ways you can test things >

Nadia: Right. And they define a cyberphysical threat model T that specifically includes attacks against the assurance apparatus itself, like attacks on the Twin or HIL fieldbus components >

Elias: So, putting it together, they argue that AI sandboxing for physical systems needs to be treated as a discipline because current work is often domain-centered rather than unifying the regulatory and technical sandbox notions >

Conclusion: Nadia: So looking at the whole paper, "AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework," it seems like they are trying to bridge the gap between experimental testing and formal assurance for real-world physical AI deployments >

Elias: They introduce a measurement framework with fifteen dimensions that tries to define exactly what a sandbox can validly test and what risks it can contain for safety and security >

Priya: It’s about making sure that when you claim something about an AI system, the evidence you have is composed in a way that accounts for the weakest link in those fifteen dimensions >

Nadia: The authors are arguing that we need disciplined methods to link represented dynamics with physical consequences and governance review into these bounded assurance claims >

Elias: It’s about moving away from just developing better simulators toward establishing a unified framework for linking those different elements, which is what this paper proposes >

Fujitsu Research of Europe

cs.CR, cs.AI, cs.RO, cs.SE

Submitted: 2026-06-16

Updated: 2026-11-10

Comments: 42 pages, 8 figures, 7 tables

Code: https://github.com/facebookresearch/habitat-sim

Project page: https://avslab.github.io/basilisk

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: AI systems are increasingly evaluated within bounded environments that integrate isolation, simulation, instrumentation, supervision, and evidence capture.

Key concepts

AI Sandbox
A bounded and instrumented environment used to test AI systems. Its boundary, variables, monitors, and evidence determine what can be claimed about the system being tested. It is mathematically modeled by the tuple S = (U, E, B, C, M, I, A, R).
Weakest-Link Rule
A rule used to determine if evidence supports a deployment claim. It calculates the strength of support as the minimum coded strength among all dimensions whose failure would invalidate the claim. If any critical dimension is missing or unclear, the evidence strength becomes 'Absent'.
Measurement Framework
A comprehensive set of fifteen dimensions used to measure a sandbox's capability. These include fidelity, controllability, observability, and scalability. This framework clarifies what a sandbox can validly test and what risks it can contain for safety and security assurance.
Threat Model
A model that explicitly includes attacks against both the AI system under test and the evaluation apparatus (the sandbox). It covers threats like misaligned systems or attacks on governance, ensuring that the entire assurance process is considered vulnerable.

Terminology

Summary

AI systems are increasingly evaluated within bounded environments that integrate isolation, simulation, instrumentation, supervision, and evidence capture. p. 0 This article develops an assurance-oriented account of AI sandboxes as controlled environments designed for testing, evaluation, verification, and validation across digital AI, embodied autonomy, and cyber-physical deployments. p. 0 The authors formalize the sandbox boundary and introduce a weakestlink rule for composing per-dimension evidence into a bounded deployment claim. p. 0 They also define a cyberphysical threat model that explicitly includes attacks on the assurance apparatus itself. p. 0 Furthermore, they propose a comprehensive measurement framework spanning fifteen dimensions: fidelity to a claim, controllability, observability, containment, reproducibility, scenario portability, timing fidelity, network realism, actuator and plant realism (including HIL/SIL integration), sim-to-real transfer evidence (and attack/fault injection), scalability (and openness/auditability), and governance artifacts. p. 3 This framework is designed to clarify what a sandbox can validly test, what risks it can contain, and what forms of evidence it can support for safety, security, and regulatory assurance. p. 0

The article distinguishes the AI sandbox by defining it as a bounded and instrumented evaluation environment whose boundary, controllable variables, monitors, intervention mechanisms, and evidence artifacts determine what can be claimed about the system under test. This is mathematically modeled by the tuple: S = (U, E, B,C, M, I, A, R). The authors propose a cross-domain taxonomy of five non-exclusive archetypes: simulation-based, digital-twin, adversarial and security, regulatory, and agent-based sandboxes. p. 8

A critical element is the measurement framework's claim-relative nature. A result can only support a deployment claim if its underlying assumptions, uncertainty levels, residual risk estimates, and artifacts are explicit. p. 8 This is operationalized via the weakest-link composition rule: for a deployment claim D, let Rel(D) be the dimensions whose failure would invalidate D. p. 8 With e(d) in L representing the coded strength of dimension d, the strength at which evidence licenses D is defined as the weakest necessary link: E(D) = d in Rel(D) e (d). p. 3 This rule ensures that if a single claim-relevant dimension is absent or unclear, E(D) becomes Absent. p. 8 The threat model, represented by the tuple T = (G, K,C, P, B, X, E, M, Y,V, R), focuses on attacks against both the system under test and its evaluation apparatus, encompassing threats like T1 (Misaligned SUT), T6 (Twin/HIL/fieldbus), and T10 (Governance over-interpretation). p. 3

The threat surface is visualized as a cyber-physical attack vector leaking or over-claiming through the shared substrate, involving the Sandbox S with boundary B, environment E (simulator, scenarios, assets), system under test U, monitors M, evidence artifacts A, and evidence assumptions gate As supporting deployment claim D. p. 1 The authors argue that the main open problems are not simply developing better simulators but establishing disciplined methods to link represented dynamics, adversarial conditions, physical consequences, and governance review into bounded assurance claims. The final section details a threat-to-control matrix where controls are only evidence claims if their configuration, activation, and monitoring are explicitly included in the evidence package. p. 4 The central research gap identified is making sandbox evidence compositional across safety, security, and governance. p. 1

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Strengthening claims requires explicit naming of assumptions through a claim grammar: In sandbox S, under assumptions As, interventions I, and measurement procedure M, system U satisfies property P over scenario set C with uncertainty Q; this supports, but does not by itself prove, deployment claim D. This prevents the over-claim (category error) where bounded evidence read as a deployment guarantee.

  2. Implementing a weakest-link composition rule ensures that a deployment claim is capped at the minimum strength of its necessary dimensions: E(D) = min d ∈ Rel(D) e (d). This forces researchers to improve the single weakest link, preventing inflation of results from non-minimal dimensions.

  3. Developing robust architectures requires addressing all fifteen measurement dimensions across fidelity, controllability, observability, containment, reproducibility, scenario portability, timing fidelity, network realism, actuator and plant realism, HIL/SIL integration, sim-to-real transfer evidence, attack/fault injection through explicit controls.

  4. Designing for physical AI necessitates specific mitigations for key challenges identified in the ranking: developing claim graphs and typed assumptions to compose evidence across perception, planning, control, timing, networking, and plant dynamics.

  5. For real-time systems like CPS or AVs, sandboxes must report latency and jitter distributions under realistic load profiles to address the challenge of Real-time semantics, ensuring that a sandbox bearing a timing claim must report latency, jitter, and deadline-miss distributions.

  6. To ensure safety in physical deployments, the system under test must demonstrate performance within a defined operating envelope by including functional safety standards like ISO 26262 and SOTIF ISO 21448 to fix fidelity and the sensing-and-algorithmic limitations that learned perception introduces.

  7. Addressing security risks requires modeling threats against the evaluation apparatus itself, such as attacks on the assurance apparatus and the evidence it produces, by treating artifacts like simulator state, scenario grammar, seeds, calibration files as assets at risk.

  8. Mitigating data drift in digital twins involves implementing drift monitors and provenance-aware twin state to detect when telemetry sources or calibration data change, ensuring that telemetry lineage and source authentication are included in the evidence package.

  9. Improving agentic tool use requires capability-based tool sandboxing with stateful replay to evaluate agents under realistic state while preserving least privilege, addressing the challenge of Agentic tool-use containment by using capability-scoped tools, permission manifests, and transcripts.

Sources

Related papers