Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
cs.AI, cs.CL, cs.CY, cs.LG
Submitted: 2026-03-08
Updated: 2026-09-30
Comments: 60 pages, 9 figures, 24 tables. Pre-registered: https://doi.org/10.17605/OSF.IO/CJW92. Code and data: https://github.com/davidgringras/safety-under-scaffolding. v4 completes the revision begun in v3: registered exclusion rules and H3-bias analysis applied; 60,112 scored evaluations analysed; ReAct descriptions and BBQ format-study scores updated; appendices moved to ancillary files
Code: https://github.com/davidgringras/safety-under-scaffolding
License: http://creativecommons.org/licenses/by/4.0/
The gist: Safety benchmarks usually test "bare" models that receive prompts and output responses, but real-world deployments "wrap" those models in complex scaffolds.
Terminology
Abstract
Safety benchmarks usually test "bare" models that receive prompts and output responses, but real-world deployments "wrap" those models in complex scaffolds. How much do these scaffolds affect model safety as measured by benchmarks? We test six leading models on four pre-registered safety benchmarks with a direct API and three scaffolds: ReAct, multi-agent, and map-reduce. We conducted 60,112 scored evaluations. On average, how safety is measured matters more than scaffolding does: we find that using a multiple choice vs. open-ended format for otherwise-identical benchmark items changes measured safety by about 5-20 percentage points (pp). The two formats are scored with different methods (answer extraction and an LLM judge), so the gap is due to measurement rather than differences in latent safety. Using a heuristic to classify model refusals would have led to different findings in four of five cases. Benchmark choice explains 15.1% of the variation in outcomes; scaffold architecture explains 0.5%, about 33x less. We find that map-reduce scaffolds, a form of structure-destroying delegation that strips answer options by decomposing prompts, reduce pooled measured safety by 7.3 pp (95% CI: 6.4 to 8.1). The pooled effects for ReAct and multi-agent scaffolds are within our pre-registered +/-2 pp margin of equivalence. However, there are large differences across models for specific benchmarks and scaffolds that are hidden by pooled estimates: for example, on the same sycophancy benchmark items, Opus 4.6 has 16.8 pp lower measured safety with a map-reduce scaffold, while Llama 4 has 18.8 pp higher measured safety. Composite reliability is G = 0.251 (95% CI: [0.000, 0.879]). This wide confidence interval, which spans "of little use" to "very good", does not support using a single composite measure of model safety as the basis for go/no-go decisions about model deployment.
Sources
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- International AI Safety Report 2026
- Answer Matching Outperforms Multiple Choice for Language Model Evaluation
- Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
- A Survey on LLM-as-a-Judge
- OLMES: A Standard for Language Model Evaluations
- Pre-registration for Predictive Modeling
- Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
- The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claims
- SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities
- Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety
- SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
- Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena
- Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
- AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
- OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
- Safety in Large Reasoning Models: A Survey
- LLMs May Perform MCQA by Selecting the Least Incorrect Option
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems