Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
cs.SE, cs.AI
Submitted: 2026-07-28
Updated: 2026-07-28
Code: https://github.com/codeprakhar25/context-files-coding-agents
License: http://creativecommons.org/licenses/by/4.0/
The gist: Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory.
Terminology
Abstract
Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.
Sources
- CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
- Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Lost in the Middle: How Language Models Use Long Contexts
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
- On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents
- MemGPT: Towards LLMs as Operating Systems
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- ReAct: Synergizing Reasoning and Acting in Language Models
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters