CTE-Bench: Counterfactual Trace Evaluation for Stateful Software Simulators
cs.SE
Submitted: 2026-09-29
Updated: 2026-10-28
Comments: Accepted at NeurIPS 2026 (Track on Evaluations and Datasets). Dataset: https://huggingface.co/datasets/zhangxr7/cte-bench-core-v1
License: http://creativecommons.org/licenses/by/4.0/
The gist: Coding agents change running software: they patch a service's code or overwrite its stored state, and then act on their own expectation of how the service will respond afterwards.
Terminology
Abstract
Coding agents change running software: they patch a service's code or overwrite its stored state, and then act on their own expectation of how the service will respond afterwards. A wrong expectation may surface only several calls later. Function-level code-execution benchmarks omit persistent service state, and agent benchmarks score the actions an agent takes or the final state it reaches. We introduce CTE-Bench, which measures whether a model can predict how an intervention changes a stateful service's future behavior, without asking it to choose actions. Each scenario gives the model Python service code, the calls and responses observed before the intervention, the intervention itself (a source edit or a state overwrite), and 40 fixed future calls; the model predicts every future response, and predictions are checked by executing the service. Three memory protocols control whether the model sees the correct earlier responses, none of them, or its own earlier predictions. CTE-Bench-Core-v1 contains 255 scenarios over six deterministic Python services, giving 10,200 predictions per model. The main score is effect-step value match (VM): exact response equality on the 2,476 future calls whose response the intervention changes. With correct earlier responses revealed, four API-hosted models (DeepSeek V4-Flash, Kimi K2.5, Qwen3.6-35B-A3B, and Claude Sonnet 4.6) reach 54.3%-61.7% effect-step VM. Hiding those responses lowers effect-step VM to 23.2%-29.0%; conditioning on self-generated predictions gives 24.8%-33.2%, and at most 1.2% of scenarios are predicted exactly end to end. Current models thus track intervention effects mainly when correct feedback is supplied, and their errors compound over a rollout. We release CTE-Bench-Core-v1 with its executable oracle, evaluation scripts, and an evaluation card mapping each claim to its protocol.
Sources
- Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
- Reasoning Runtime Behavior of a Program with LLM: How Far Are We?
- CounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Models
- Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence
- CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
- World Modelling Improves Language Model Agents
- TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
- Evaluating LLMs on Sequential API Call Through Automated Test Generation
- Code Simulation Challenges for Large Language Models
- ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
- ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
- Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
- Debugging code world models
- Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code
- Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
- On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters