Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents
cs.SE, cs.AI, cs.LG
Submitted: 2026-09-27
Updated: 2026-10-03
Comments: Accepted at NeurIPS 2026. Includes additional experiments and analysis
License: http://creativecommons.org/licenses/by/4.0/
The gist: Outcome-only reinforcement learning gives software engineering (SWE) agents a terminal success signal but little direct guidance about intermediate decisions.
Terminology
Abstract
Outcome-only reinforcement learning gives software engineering (SWE) agents a terminal success signal but little direct guidance about intermediate decisions. We introduce Counterfactual Rollout Replay (CRR), a training-time procedure that uses forkable executable environments to obtain step-level return contrasts. CRR selects a small set of decision points, restores each state, samples an alternative action, and rolls the branch forward under the policy. It retains the realised training trajectory and replaces the advantage at selected steps with the difference between its terminal return and the sampled counterfactual return. The method needs no human process labels or learned process reward model; free refers to those supervision costs, not replay compute. With a 14B policy, CRR improves pass@1 on SWE-bench Verified, SWE-bench Live, and SWE-rebench, and combines with process-reward and trajectory-search methods. On SWE-bench Verified, an equal-wall-clock comparison on the same hardware yields 41.7% versus 36.7% for extended outcome-only GRPO, a 5.0-point gain with fork overhead included. These results apply to environments with affordable, reliable state restoration; stochastic continuations and expensive or imperfect replay remain limitations.
Sources
- SWE-Bench+: Enhanced Coding Benchmark for LLMs
- An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
- Intrinsic Credit Assignment for Long Horizon Interaction
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
- SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
- Surprisal-Guided Selection: Compute-Optimal Test-Time Strategies for Execution-Grounded Code Generation
- SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent
- SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics
- The Trace Is the State: Exact Credit Assignment for LLM Agent Teams
- Process Reinforcement through Implicit Rewards
- TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents
- SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development
- MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
- Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment
- SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scaling
- Scaling Test-Time Compute for Agentic Coding
- Learning Game-Playing Agents with Generative Code Optimization
- Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
- Understanding the Challenges in Iterative Generative Optimization with LLMs
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties