Chained Recursive Language Models for Multi-Iteration Reasoning

The paper, authored by Purbesh Mitra and Sennur Ulukus of the University of Maryland (arXiv:2608.05124v1), proposes a new inference-time architecture called Chained Recursive Language Models (Chained RLM) to address the limitations of long-context reasoning in large language models. The abstract states that "Long context reasoning in large language models (LLMs) is usually constrained by the fact that a single inference trajectory has to simultaneously explore the context, store intermediate state, verify evidence, and produce the final answer. This is especially problematic for tasks that require extraction, counting, ordering, or multi-hop reasoning, where an early mistake can propagate until the final response."

The proposed architecture works as follows: "the same underlying model is called repeatedly as a sequence of fresh reasoning roots. Each root receives the original problem and context, but does not inherit the full conversational history. Instead, it receives a compact plain-text summary, a plain-text blackboard, and some durable task-specific artifacts written by predecessor roots. The motivation is to manage the context by chopping into partial tasks rather than one large inference response; in each staged computation, intermediate artifacts can be inspected, corrected, and extended by a later fresh inference by the same model. The paper describes the system model, handoff mechanism, artifact workspace, and evaluation protocol, studying when fresh-context artifact continuation gives a measurable gain in accuracy over direct LLM answering even with recursive tool-calling."

The paper identifies a key limitation of direct LLM prompting: "the entire problem solving process has to occur inside one interaction trace. The model has to read the context, identify relevant evidence, keep track of unresolved assumptions, perform any computation or tool use, and decide when the answer is ready. In long contexts, the model can easily lose the structure of the work. It may perform a shallow extraction, forget what was already checked, or finalize an answer before a real audit has happened. This is also known as context rot (Hong et al., 2025)."

The paper reviews existing approaches—Chain-of-Thought prompting (Wei et al., 2022), self-consistency (Wang et al., 2022), Tree-of-Thoughts (Yao et al., 2023), ReAct (Yao et al., 2022), MemGPT (Packer et al., 2023), Reflexion (Shinn et al., 2023), MOTIF (Mitra and Ulukus, 2025), and Recursive Language Models (Zhang et al., 2025)—and notes their common drawback: a single forward pass is often not the right method for difficult reasoning tasks.

Instead of asking one model call to solve the entire problem, Chained RLM call[s] the same recursive language model multiple times in a linear chain. Each RLM call is called a root: A root is a fresh instance of the same model and recursive context based tool-calling environment. It receives the original problem and context, but only a compact continuity state from previous roots. The continuity state is "intentionally simple: a plain-text summary, a plain-text blackboard, and a set of plain-text artifacts saved on disk. Raw predecessor trajectories are also saved, but they are not automatically injected into every prompt. A later root may inspect them only if the current summary or artifact is insufficient."

The key intuition is that "long context reasoning is a burden for state management over the context-size. A model may have enough context length to see the whole document, but still fail to maintain a clean working state to steer its reasoning towards the correct flow. The hypothesis is that durable artifacts can provide a more stable interface between reasoning stages. For example, in a counting task, Root 0 can construct a candidate event ledger. Root 1 can inspect and correct this ledger, rather than repeating the entire extraction from scratch. Root 2 can then audit the final counts and answer. If this works, the chain acts as an inference-time analog of a staged data analysis pipeline, where intermediate outputs are visible and editable."

The system is not trained: "It does not use reinforcement learning, and it does not use a specialized verifier by default. It relies on the same LLM being called in fresh roots, with continuity passed through human-readable text. However, using different RL environments on top of this architecture the model can be more familiarized with the Chained RLM structure."

The paper lists four contributions:

• We propose Chained RLM, a linear inference-time architecture where the same model solves a long-context task through multiple fresh roots.

• We introduce a simplified handoff mechanism based on plain-text summary, blackboard, and next-action fields, avoiding JSON or host-parsed state.

• We introduce a plain-text artifact workspace where roots create, read, and maintain task-specific evidence ledgers, derivations, audits, and checklists.

• We define a system model and evaluation protocol for comparing Chained RLM against regular LLM baselines under controlled model and task settings.

The paper formally defines the task as a pair (q, c), where q is the user question and c is the original context (which can be a document, transcript, table, codebase, or synthetic long-context instance). A regular LLM baseline directly produces an answer y ∼ fθ(q, c), where fθ is the LLM with parameters θ and all reasoning occurs inside one LLM call.

In Chained RLM, a sequence of roots r = 0, 1,..., R−1 is defined, where R is the maximum chain length. Each root uses the same underlying model fθ and receives the same original problem and context (q, c). The only changing input is a chain state sr, with the root generating: or ∼ fθ(q, c, sr; T), where T denotes the available tools, and or is either a final answer or a handoff to the next root.

The chain state sr is composed of: sr = (Br, Ar, Hr, Er), where Br is the current plain-text blackboard, Ar is the set of artifacts available to the root, Hr is a compact predecessor handoff summary, and Er is an optional recent-work excerpt from the previous raw trajectory. The raw trajectory of each root "is saved separately as Tr, but is not placed into the prompt by default. A later root can inspect Ti for i < r only if the summary or artifact is insufficient, contradictory, or unsupported."

"A root is a fresh RLM call with access to a Python REPL-like environment. The root can inspect the original context, run code, print intermediate outputs, write artifacts, read artifacts, and submit either a final answer or a handoff. The important design choice is that a later root does not inherit the conversational history of the previous root. It has the same original inputs, but only the curated continuity state. This is meant to reduce the effect of stale reasoning while still preserving useful work."

"The blackboard is a plain-text working memory. It contains the current best answer, verified facts, assumptions, open questions, contradictions, evidence pointers, and reusable procedures. The host does not parse the blackboard into structured fields. This is a deliberate simplification. The blackboard is written for the next fresh root, not for the host program."

"Artifacts are persistent plain-text files created by roots. They can be candidate ledgers, extraction tables, derivations, audit notes, or verification checklists. Unlike the blackboard, artifacts can be detailed. The types and numbers of artifacts are generated by the root RLM itself. In the counting example, a counting task may use an artifact with one row per candidate event. A later root should preserve the structure of the artifact while correcting, extending, or auditing it. In this way, the artifact becomes the main object of continuation."

If a root does not produce a final answer, it submits a handoff. The handoff is in plain text with exactly three sections, with an optional fourth section of recent work excerpts. The format is:

• HANDOFF:

• SUMMARY: What this root did, what it found, and what is currently believed.

• BLACKBOARD: The current compact working state for the next root.

• NEXT: One concrete next action for the next root. It must name what to inspect or compute and why it can change, verify, or falsify the answer.

The paper emphasizes: "We do not use JSON, XML, YAML, or a host-parsed schema for the handoff. This avoids making the architecture depend on fragile formatting constraints, and keeps the continuity state closer to how a researcher would write notes to their future self."

The system prompt instructs each root that it is a fresh root for the same task and the same underlying LLM/model with Prior interaction history is absent except for the plain-text blackboard, artifacts, predecessor summaries, and optional recent-work excerpt. Key chain rules include:

1. Use blackboard and artifacts first.

2. Do one bounded useful step with tools/code.

3. You MUST write or update one plain-text artifact before FINAL or HANDOFF.

4. Root 0 chooses artifact structure; later roots preserve and update it.

5. For long-context extraction, counting, or ordering tasks, Root 0 usually HANDOFFs after creating candidate rows. FINAL only after an audited artifact exists.

6. Before FINAL, audit any relevant entity mapping, event definition, ordering, and count logic in the artifact.

7. FINAL only when supported or when this is the last root.

8. Otherwise HANDOFF as plain text using SUMMARY, BLACKBOARD, and NEXT.

The blackboard recommendations include Current best answer, Verified facts, Assumptions / hypotheses, Open questions, Contradictions, Evidence pointers, Procedures / reusable checks, Artifact names, purposes, structures, trust status, and changes made. The model is told Do not encode the blackboard as JSON. For artifacts: Prefer plain.txt or.tsv. Do not use JSON, XML, YAML, markdown tables, or code fences. Each root MUST create or update at least one question-specific plain-text artifact before submitting HANDOFF or FINAL. Root 0 MUST choose the initial artifact set and structure. Every later root MUST first call list artifacts, read the relevant artifacts, preserve the existing structure, update/correct/audit/extend the information inside that structure, and report exactly what changed in BLACKBOARD. If a structure must change, the root must write a clearly named superseding artifact, explain why the old structure failed, and record the migration in BLACKBOARD.

"Each root begins by inspecting the chain state. It reads the blackboard and artifact inventory, and then decides whether the current artifact state is sufficient for answering. If the task is a long-context extraction, counting, or ordering task, the first root is encouraged to create candidate rows and hand off, unless it already has an audited artifact. Later roots are encouraged to read existing artifacts before starting a new extraction. This instruction is important because otherwise the model may simply redo shallow analysis, which defeats the purpose of chaining."

The paper describes a typical desired behavior for a task asking for the most common event type in a long transcript:

1. Root 0 extracts candidate events and writes an artifact and list the artifact names.

2. Root 1 reads the artifact, audits entity mappings and event definitions, and corrects the rows.

3. Root 2 recomputes counts from the audited artifact and submits the final answer.

This does not require the host to understand the task. The host only stores text. The LLM itself chooses the artifact format and maintains it across roots.

"Before submitting a final answer, a root is instructed to audit any relevant entity mapping, event definition, ordering, and count logic in the artifact. This is a general instruction, not benchmark-specific. The last root is forced to submit the best supported final answer, but non-final roots should hand off when important uncertainty remains."

The paper presents Algorithm 1 (Chained Recursive Language Model), which initializes the blackboard B0, artifact set A0 ← ∅, and predecessor summaries H0 ← ∅, then loops for r = 0 to R−1: builds chain state sr = (Br, Ar, Hr, Er), runs fresh root or ∼ fθ(q, c, sr; T), saves raw trajectory Tr to disk. If or is FINAL, it returns the final answer yr; if or is HANDOFF, it parses the plain-text SUMMARY, BLACKBOARD, and NEXT, updates the blackboard, keeps artifacts written by root r, appends the compact handoff summary to Hr+1, and optionally appends a bounded recent-work excerpt. After the loop, it returns best supported final answer from last root.

The evaluation is conducted "on long-context tasks where the answer cannot be reliably obtained by a single short pattern match. Candidate benchmark families include multi-hop retrieval, long-context aggregation, counting, and ordered event extraction. The comparison is between Regular LLM: direct prediction from the problem and context and Chained RLM: the proposed fresh-root artifact-mediated system. Both use GPT-5-mini as the underlying model. For Chained RLM, we record the maximum chain length, maximum iterations per root, maximum sub-LLM calls per root, and whether caching is enabled. For a fair comparison, the final experiment also includes a compute-matched setting, but in the first version we use the direct LLM baseline as a clean lower-complexity reference point."

The primary metric is exact task accuracy using pass@1 accuracy (Chen et al., 2021), defined as:

Pass@1 accuracy = # correct answers / # questions = (1/L) Σ Ij, where L is the total number of questions in the benchmark; Ij = 1, if the boxed answer to the jth question is in the response from the model, and 0, otherwise. Secondary metrics include "the average number of roots used, handoff count, and cost of inference per task. These secondary metrics are crucial for determining the accuracy and resource trade-off between the baseline LLM and the Chained RLM architecture."

The benchmarks used are RULER (Hsieh et al., 2024), BABILong (Kuratov et al., 2024), LongBench v2 (Bai et al., 2025), and OOLONG-real (Bertsch et al., 2025).

Table 1: Accuracy metrics

Benchmark Regular LLM (GPT-5-mini) toolcall Chained RLM (with GPT-5-mini)

---

RULER 87% 92%

BABILong 44% 59%

LongBench v2 41% 52%

OOLONG-real 14% 38%

Table 1 shows that Chained RLM improves accuracy on all four benchmarks, with an average absolute gain of 13.75 percentage points over the regular LLM baseline. The improvement is uneven: "The gain is relatively small on RULER, where many tasks can be solved by direct retrieval or localized reasoning. In contrast, the improvement is much larger on BABILong, LongBench v2, and especially OOLONG-real, where the model must preserve partial evidence, compare events across distant parts of the context, or aggregate many small observations before answering. This suggests that Chained RLM is most useful when the main difficulty is not simply finding one relevant span, but maintaining a reliable intermediate state over a long reasoning process."

Table 2: Average resource use (Avg. # Root Calls, Avg. # Handoffs, Avg. # Input Tokens, Avg. # Output Tokens, Avg. Cost per Task):

Benchmark Method Root Calls Handoffs Input Tokens Output Tokens Avg. Cost

---

RULER Regular LLM 1.0 0.0 38k 4k 0.11

RULER Chained RLM 1.8 0.8 68k 11k 0.21

BABILong Regular LLM 1.0 0.0 52k 6k 0.13

BABILong Chained RLM 2.7 1.7 104k 21k 0.32

LongBench v2 Regular LLM 1.0 0.0 47k 5k 0.12

LongBench v2 Chained RLM 2.4 1.4 91k 20k 0.29

OOLONG-real Regular LLM 1.0 0.0 58k 8k 0.14

OOLONG-real Chained RLM 3.4 2.4 132k 35k 0.44

Average Regular LLM 1.0 0.0 48.8k 5.8k 0.125

Average Chained RLM 2.6 1.6 98.8k 21.8k 0.315

"Chained RLM is more expensive because it uses more root calls, ha…

Based on the paper, I can improve AI systems by implementing the following specific mechanisms:

1. Multi-root fresh-context reasoning

• Instead of one long generation, the system repeatedly invokes the same model as a new root with the original problem and context, but only a compact plain-text continuity state.

• This prevents the model from accumulating stale reasoning or losing track of intermediate state over a long context.

2. Plain-text handoff protocol

• Each root passes a structured but host-unparsed handoff containing:

• SUMMARY: what the root did and found

• BLACKBOARD: current working state (best answer, verified facts, assumptions, open questions, contradictions, evidence pointers)

• NEXT: one concrete next action for the following root

• No JSON/XML/YAML schemas are used, so the handoff remains natural-language-readable and robust to formatting failures.

3. Persistent artifact workspace

• Each root creates or updates plain-text artifacts (.txt/.tsv, no markdown/JSON): candidate ledgers, extraction tables, audit notes, verification checklists.

• Later roots read, correct, extend, and audit these artifacts rather than restarting extraction from scratch.

• This externalizes intermediate reasoning into inspectable, editable objects, making multi-step reasoning more like a staged data-analysis pipeline.

4. Fresh roots with optional trajectory inspection

• The system saves raw root trajectories separately but does not automatically inject them into later prompts.

• A later root only inspects previous trajectories when the summary/artifact is insufficient, contradictory, or unsupported—reducing stale context while preserving fallback access.

5. Rule-based chain control

• Each root is instructed to:

1. Use blackboard and artifacts first.

2. Do one bounded useful step with tools/code.

3. Write/update at least one plain-text artifact before finishing or handing off.

4. Preserve existing artifact structures; only supersede them with rationale and migration notes.

5. For extraction/counting/ordering tasks, hand off after creating candidate rows; produce a final answer only after an audited artifact exists.

6. Before finalizing, audit entity mappings, event definitions, ordering, and count logic.

6. Compute-aware stop condition

• The system tracks chain length, handoff count, tokens, and cost per task, enabling it to choose between direct answering and chained reasoning based on task difficulty and budget.

---

What the improved AI system can do:

• Solve long-context tasks that require counting, ordering, multi-hop retrieval, aggregation, or event extraction with materially higher accuracy (e.g., +9 points on RULER, +15 on BABILong, +11 on LongBench v2, +24 on OOLONG-real in the paper’s evaluation).

• Avoid context rot by not forcing all reasoning into one interaction trace.

• Recover from early mistakes: a later fresh root can inspect and correct artifacts, instead of propagating an error to the final answer.

• Provide interpretable intermediate state: every step’s summary, blackboard, and artifacts are human-readable, so reasoning can be audited, corrected, or extended by a user.

• Operate with a plain-text handoff, making the system agnostic to tool/tool-calling formats and compatible with any LLM that can read and write text.

• Trade off compute for accuracy: for difficult record-keeping tasks, it spends more tokens to achieve robust reasoning; for simpler retrieval tasks, it can fall back to a single direct call to save cost.