From Prompt to Harness: Coderlet from Scratch

This paper studies a compact harness design for a programming agent by following a single request through context formation, model decision, environmental action, observation return, and state continuation. The harness is the control layer that prepares context, calls a model service, handles tools, stores history, and sustains the interaction, turning isolated model responses into a continuing interaction with an environment.

The runtime has three working areas: model context (material assembled for the current call), the tool environment (files, processes, services, and other resources), and persistent state (history or memory beyond the current request). Three boundaries connect these areas: the model boundary (protocol translation, encoding context and tool descriptions, decoding responses), the execution boundary (validation, dispatch, and turning tool results into call-linked observations), and the state boundary (loading and saving history and memory).

One request passes through five observable points: the model proposes content or an action, the harness decides whether to run the action, the environment returns a tool result, the harness chooses what enters the request record, and progress updates reach the caller along the way. The request lifecycle determines the order of these transitions: preparation reads the current input, selects relevant history, loads tool descriptions, and assembles model context; the model response is decoded, call identifiers are checked, and every tool name and argument shape is validated; invalid calls stop at validation and return a call-linked rejection observation without touching the tool environment; valid calls enter the execution layer and yield a tool result (a value or expected error), which is bound to the originating call and included in the next model context; the loop ends when the model returns no further calls or a local stop condition is reached, then the harness assembles and saves the request record, and after a successful save sends the completion notification.

The design preserves the distinction among model output, tool execution, and observation return. For example, if one model response contains two calls—the first naming an unknown tool and the second a valid one—the harness rejects the first without execution and runs the second, then the next model turn receives both observations in their original order. Persistent state and the tool environment are updated independently: if a tool changes a file and saving the request record later fails, the external change may remain. The request record captures the interaction, while tool evidence supports claims about external effects.

For later turns, the model context always includes current instructions, input, and tool view. The selector draws optional history from the remaining context budget, keeping the current request intact, preserving the link between each included observation and its action, and labeling condensed material as a summary. The design rule is that shortened context should remain interpretable, since an observation separated from its call loses context and a summary with no clear subject risks misleading the next turn.

Failure interpretation starts with the last completed stage. Rejection during validation produces an observation while leaving the environment unchanged. Once a tool effect occurs, a later request failure leaves that effect in place unless another action reverses it. A saved request record stays in history when the completion notification never reaches the caller. For uncertain execution status, the harness reports uncertainty in the current request instead of fabricating a successful observation. A recovery procedure starts at the last confirmed stage: before dispatch, correction or rejection is safe; after a confirmed tool effect, preserve the observation and handle save failure as a state-layer problem; after a confirmed save, retry the completion notification without repeating the tool action.

The boundaries also localize debugging: missing instructions direct attention to context construction; a valid proposal with no environmental effect leads to validation or dispatch; when a request record is absent from a later context, saving and selection become the relevant checks.

Consistency checks keep the three boundaries aligned: call identifiers remain attached to their observations, both calls and observations retain generation order, names and arguments are validated before dispatch, request records are saved before the completion notification is sent, and selection of older material preserves enough surrounding context. Potential weaknesses include omitted details in summaries, untrusted tool output, and outdated memory notes; origin and status information help later decisions weigh that material, though security isolation and content validation require separate mechanisms.

Bootstrapping begins with a working seed. When editing and command tools expose the harness's own implementation, the same loop provides a way to work on the mechanisms that organize its behavior as well as on user projects. During a request the harness coordinates the model, tools, observations, and state; between runs, the same tools expose its context rules, tool interfaces, control flow, and state mechanisms for inspection and revision. A scripted self-edit on a disposable copy demonstrates one bootstrap step: the running loop inspects, modifies, and tests mechanisms that affect later work. Human guidance fits this process, with a person directing the work, reviewing changes, and starting the next round from the accumulated implementation.

The design is realized in the executable artifact Coderlet, a compact Codex-like local programming agent. From scratch refers to building the control layer only; the model service and the OpenAI Codex product remain outside scope. The paper studies one user, one agent, and one continuous session with requests processed sequentially, removing concurrency from the account. The discussion moves from prior agent work (ReAct, Toolformer, SWE-agent, Reflexion, Voyager, SWE-bench, AgentBoard, harness-aware evaluation, harness composition, bootstrapping) to the request runtime and its boundaries, then closes with bootstrapping across runs. The conclusion states that the harness supplies continuity across otherwise separate model calls, with value in coordinating distinct transitions: proposals are checked before dispatch, observations return with their calls, and request records become reusable history through explicit saving and selection, making the runtime trace legible and giving recovery a concrete starting point.

Based on the paper, here are specific improvements to AI systems and what the improved system can do:

1. Call-Linked Observation Integrity

• Improve: Enforce that every tool call’s observation is returned with its originating call ID and in original order, even when multiple calls occur in one response or some are rejected.

• Improved system can: Handle multi-tool responses without losing traceability; reject invalid calls without halting valid ones; provide the model with a complete, ordered record of what happened per call, reducing confusion and enabling precise recovery.

2. Validation Before Execution

• Improve: Add a strict validation layer that checks tool names and argument shapes before any environmental action, returning a rejection observation for invalid calls.

• Improved system can: Prevent accidental or malicious tool misuse; avoid side effects from malformed calls; give the model immediate, structured feedback on why a call failed, improving self-correction in subsequent turns.

3. State-Save Before Completion Notification

• Improve: Guarantee that the request record is saved to persistent state before sending the completion notification to the caller.

• Improved system can: Ensure no loss of interaction history on crashes; allow recovery from the last confirmed save point; avoid duplicate tool executions when retrying after a notification failure.

4. Uncertainty Reporting for Ambiguous Execution

• Improve: When tool execution status is unknown (e.g., network timeout), report uncertainty in the current request instead of fabricating a success observation.

• Improved system can: Maintain honest state representation; enable the model to ask for clarification or retry safely; prevent downstream decisions based on false assumptions about environmental effects.

5. Context Selection with Action-Observation Pairing

• Improve: When trimming history for context budget, always preserve the link between an observation and its originating action, and label any condensed material as a summary.

• Improved system can: Keep long-running tasks coherent; avoid misleading the model with orphaned observations; allow the model to reason about cause-effect chains even with limited context.

6. Failure Recovery from Last Confirmed Stage

• Improve: Implement a recovery procedure that starts from the last confirmed stage (validation, execution, or save) and never repeats a tool action after a confirmed effect.

• Improved system can: Recover gracefully from crashes or network errors; avoid double side effects; provide clear, actionable error messages to the user or model about what was and wasn’t completed.

7. Bootstrapping via Self-Editing Tools

• Improve: Expose the harness’s own implementation (context rules, tool interfaces, control flow, state mechanisms) as editable tools within the same agent loop.

• Improved system can: Modify its own behavior during runtime; test changes on a disposable copy; iteratively improve its own architecture without human code changes, enabling self-optimization and adaptation to new tasks.

8. Origin and Status Metadata for Trust

• Improve: Attach origin (e.g., tool output vs. summary) and status (e.g., confirmed, uncertain, rejected) to all observations and memory notes.

• Improved system can: Weigh information appropriately (e.g., trust confirmed tool output over a summary); flag outdated memory; make decisions that account for data provenance, reducing hallucination and error propagation.

9. Sequential Request Lifecycle with Explicit Checkpoints

• Improve: Structure each request into five observable checkpoints (proposal, decision, result, record, progress) with clear transitions.

• Improved system can: Provide transparent progress updates to the caller; enable fine-grained debugging (e.g., “valid proposal but no effect” points to validation/dispatch); allow external monitoring and intervention at any stage.

10. Independent Updates of State and Environment

• Improve: Keep persistent state and tool environment updates decoupled, so a save failure does not roll back a tool effect.

• Improved system can: Acknowledge external changes even if logging fails; avoid inconsistent state (e.g., file changed but history lost); allow the model to reason about what actually happened versus what was recorded, improving long-term reliability.

A model alone does not determine how a programming agent acts. What the model sees, how actions enter the environment, how feedback returns, and how one run affects the next all depend on how the harness is organized. Minimal examples usually show only the basic interaction between a model and tools, while production systems spread these relationships across complex components and dependencies. This paper studies a compact harness design by following a single request through context formation, model decision, environmental action, observation return, and state continuation. Three boundaries---model, execution, and state---connect the model service, tool environment, and persistent state, while the request lifecycle determines the order in which these transitions occur. Together, they show the harness's core role: turning model generations into environmental actions, carrying runtime feedback into later decisions, and allowing state to continue across requests. On top of this runtime structure, a harness can also be gradually refined across runs through continued bootstrapping. The design is realized in the executable artifact https://github.com/lilinxi/Coderlet.