Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Science

AEROBAT is the first multi-agent system designed to automate and scale behavioral scientific research on AI agents. Given an arbitrary target behavior specified by a user, AEROBAT automatically executes a full research pipeline: generating hypotheses about the behavior, designing and executing controlled experiments, making behavioral assessments, analyzing the results, and writing reports.

The system's environment model supports three key properties: control, parametrization, and multiple realizability. Control is achieved through an environmental configuration expressed as structured natural language, including roles, authority, constraints, world context, resources, actors, and consequences. Parametrization uses a set of environmental variables with ordinal or binary values within a specified domain. Multiple realizability ensures that a single variable value can be instantiated in multiple non-trivially distinct ways, supporting attribution of behavioral changes to the manipulated parameter rather than a specific instantiation.

The multi-agent architecture mirrors a standard behavioral science pipeline across four stages. In Stage 1, a hypothesis generator agent produces a behavior definition, an evaluation rubric, a set of hypotheses (each with a hypothesized causal variable, candidate values, causal direction, and interaction length), and relevant domains. In Stage 2, a configuration designer agent translates each hypothesis-domain pair into matched configurations, where only the hypothesized causal variable's value varies within each group while all other environmental variables are held constant. In Stage 3, a simulator agent renders environments for all matched simulations in parallel, and a subject agent (the AI being studied) generates actions, with both agents blind to the target behavior and hypothesis. In Stage 4, a blind reviewer agent scores the subject agent's behavior against the rubric, producing qualitative analyses and behavior scores. A research manager agent gates each stage (ranking hypotheses, checking configuration coherence, and assessing simulation fidelity) and writes final reports.

Statistical analysis uses a Bayesian monotone-increment model to test hypotheses, reporting Bayes factors (BF10) and effect sizes (∆), with a decision rule classifying results as positive, negative, no effect, or inconclusive. A block-stratified Kendall's τ is also reported as a model-free check.

In experiments, AEROBAT was used to study 12 target behaviors (friendliness, extroversion, empathy, sycophancy, purchase, compete, distrust, strategic aggression, literal rule-following, non-compliance, plan, and deception). It generated and tested 79 hypotheses, designing 1,240 controlled experiments and executing 23,512 simulation rounds. Moderate-to-strong statistical evidence (BF10 ≥ 3) was found for 26 hypotheses, including novel findings. For example, increasing instruction divergence reduces literal instruction-following (∆ = −1.28, BF10 = 240), and higher goal conflict intensity increases deception (BF10 = 2.2 × 105, ∆ = 2.27), primarily through omission and strategic framing rather than outright lies.

Generalization analysis showed that effect sizes were broadly consistent across three different subject agent models (GPT-5-mini, Gemini-3.1-Pro, Kimi K2.6), with Spearman's ρ ranging from 0.697 to 0.939. Environment fidelity analysis confirmed that generated configurations faithfully instantiate variable values (exact match rate 0.934 vs. 0.307 chance), simulations map correctly to configurations (92% accuracy vs. 26% chance), and the hypothesized causal variable accounts for the only salient within-group difference (high match in 26 of 36 cases).

The paper concludes that AEROBAT can complement and extend manual research, serving as a research assistant that expedites the research process. Its significance lies in proposing novel design principles, pipelines, and evidential standards tailored to behavioral science, potentially helping the emerging field avoid the replication crisis that affected human behavioral science.

Improvements to AI Systems:

1. Automated Behavioral Auditing for AI Deployments: Integrate AEROBAT’s pipeline as a pre-release or continuous evaluation tool. An AI system (e.g., a chatbot, autonomous agent, or decision-making model) can be automatically subjected to controlled, parametrized environmental variations (e.g., authority levels, resource scarcity, instruction divergence) to detect emergent behaviors like sycophancy, deception, or non-compliance before deployment. The improved system can generate a “behavioral safety report” with Bayes factors and effect sizes, flagging risky tendencies without human manual testing.

2. Hypothesis-Driven Self-Improvement via Causal Feedback: Use AEROBAT’s Stage 1–4 loop to let an AI system generate its own hypotheses about why it fails a target behavior (e.g., “I comply less when instructions conflict with user goals”), then design matched experiments to test those hypotheses. The system can then adjust its own policy or prompting strategy based on statistically significant causal variables (e.g., increase instruction alignment weighting) and verify the fix via the same automated pipeline, enabling closed-loop, evidence-based self-correction.

3. Generalizable Behavior Attribution Across Model Variants: Apply AEROBAT’s multiple-realizability and block-stratified analysis to compare behavioral consistency across different AI models (e.g., GPT-5-mini vs. Gemini-3.1-Pro). The improved system can identify which behavioral effects are model-agnostic (robust across architectures) versus model-specific, allowing developers to select or fine-tune models for stable, predictable behavior in high-stakes domains (e.g., legal advice, medical triage) where cross-model reliability is critical.

4. Dynamic Environment Parametrization for Robustness Training: Use AEROBAT’s configuration designer to generate a massive, diverse set of matched environmental configurations (varying roles, constraints, consequences) for adversarial training. The improved AI system can be trained not just on static data but on these controlled, parametrized scenarios, learning to maintain target behaviors (e.g., honesty, helpfulness) across thousands of non-trivially distinct instantiations, reducing overfitting to specific prompt phrasings and improving generalization.

5. Real-Time Blind Evaluation and Report Generation for AI Agents: Embed AEROBAT’s blind reviewer and research manager agents as a built-in self-monitoring module. The improved AI system can, after each interaction or task, automatically score its own behavior against a rubric, compute Bayesian evidence for whether it deviated from its intended policy, and generate a human-readable report. This enables continuous, unbiased self-assessment and immediate corrective action (e.g., re-prompting, rollback) without human oversight.

6. Causal Variable Discovery for AI Alignment: Leverage AEROBAT’s hypothesis generator to propose novel causal variables (e.g., “goal conflict intensity,” “instruction divergence”) that influence undesirable behaviors. The improved system can then proactively search its own latent decision-making for these variables, using the Bayesian monotone-increment model to identify which internal features or attention patterns drive behavior changes, leading to interpretable alignment mechanisms rather than black-box fixes.

What the Improved AI System Can Do:

• Automatically detect and quantify its own biases, sycophancy, or rule-breaking under varied controlled conditions, with statistical rigor (BF10, effect sizes).

• Self-generate and test hypotheses about its failures, then implement and verify targeted improvements without human experiment design.

• Guarantee behavioral consistency across different model versions or architectures by running multi-model generalization analyses.

• Train on millions of parametrized, controlled scenarios to become robust to environmental variations while preserving core values.

• Produce real-time, audit-ready behavioral reports for every deployment, reducing the risk of harmful emergent behaviors.

• Identify and manipulate specific causal levers in its own reasoning to align with user intent, reducing deception and increasing literal compliance when needed.

As AI agents are increasingly deployed in complex environments, understanding their behaviors becomes critical. Yet behavioral scientific research on AI agents remains manual and labor-intensive. We introduce AEROBAT, the first multi-agent system to automate behavioral scientific research on AI agents. Given an arbitrary target behavior by its user, AEROBAT automatically executes a full pipeline of behavioral scientific research---generating hypotheses about the behavior, designing and executing controlled experiments, making behavioral assessments, analyzing the results, and writing reports. For 12 target behaviors, we used AEROBAT to generate and test 79 hypotheses: designing 1,240 controlled experiments and executing 23,512 simulation rounds in total. Moderate-to-strong statistical evidence was found for 26 hypotheses, including some novel ones. In sum, our results demonstrate that automated behavioral scientific research on AI agents can complement and extend the reach of manual research.