Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase

arXiv:2609.29744 · cs.SE, cs.AI · Submitted 2026-09-24 · Read on arXiv

cs.SE, cs.AI

Submitted: 2026-09-24

Updated: 2026-09-24

Code: https://github.com/doug-leith/extractor-code-development

License: http://creativecommons.org/licenses/by/4.0/

The gist: We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance

Terminology

Abstract

We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.

Sources

Related papers