Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

arXiv:2607.27250 · cs.SE, cs.AI · Submitted 2026-07-28 · Read on arXiv

cs.SE, cs.AI

Submitted: 2026-07-28

Updated: 2026-07-28

Code: https://github.com/codeprakhar25/context-files-coding-agents

License: http://creativecommons.org/licenses/by/4.0/

The gist: Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory.

Terminology

Abstract

Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.

Sources

Related papers