OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

The paper introduces OctoLong, a context engineering pipeline that leverages an AST parser, a language server backend (LSP), and a package manager to recursively retrieve code references across repository boundaries, thereby curating dependency-rich code contexts up to millions of tokens in length. The authors use this pipeline to collect ≈6.2B tokens of cross-repository long-context data, which they combine with other sources into a ≈50B-token long-context fine-tuning (LCFT) corpus (OctoLong-LCFT), followed by a ≈10B-token instruction-tuning mix (OctoLong-SFT). This two-stage recipe produces the OctoLong-Instruct suite, a family of long-context open LMs derived from Qwen3 dense base models ranging from 600M to 14B parameters, with context length quadrupled to 128K tokens.

According to the abstract, "supplanting just 12% of conventional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios."

The authors note that "emergent language model (LM) applications such as in-context learning, retrieval-augmented generation, and agentic tool-use demand long-horizon capabilities, exponentially driving up the demand on the context lengths the LMs are expected to support. They observe that context lengths of open-weight LMs are growing exponentially, doubling on average every 5 months" (Figure 1).

However, they identify a key bottleneck: Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. Prior data-engineering approaches—such as token-level dependency filtering, curating multiple salient documents, and referral-based packing—often lack complex multi-hop dependencies and mine irrelevant or contradictory information. The paper's core idea is cross-repository code dependency mining, a referral-packing technique that taps into openly available code data to unlock a virtually limitless source of multi-hop referral-dense contexts that transcend repository boundaries.

The authors list four main contributions: (1) the OctoLong pipeline that generates dependency-rich long code contexts; (2) the ≈50B-token OctoLong-LCFT corpus and ≈10B-token OctoLong-SFT mix; (3) the OctoLong-Instruct suite of 128K-context models; and (4) experimental evidence that "LCFT with our OctoLong data in the long-context data mix leads to substantial improvements not only in long-context code tasks but also in general-domain long-context applications including agentic use-cases, without penalty on short-context LM capabilities."

The paper situates itself within two threads of related work. Cross-document context engineering has previously relied on hyperlinks, book BISAC codes, event co-references, and repository-level code dependencies, but these approaches are all either hard to scale or yield loosely related contexts. In contrast, the authors detail how to recursively mine referred class and function implementations beyond repository boundaries — an untapped source of tightly-coupled data whose acquisition is easy to scale.

Regarding mining long self-contained code data, they note that early file-level attempts found long code files are usually low-quality configuration files and uninformative build artifacts. Serialized repositories became the anchor of mid-training and context-extension corpora, but most code is not self-contained at the repository level, which restricts models' ability to reason about multi-repository projects. Prior attempts to train on cross-repository contexts were stymied by inadequate tooling and limited to sourcing dependency library names.

The OctoLong pipeline consists of five core stages: seed selection, containerized environment, call-site indexing, dependency traversal, and BFS serialization (Figure 2).

Seed selection: The authors source reputable and actively maintained Python repositories from SEART Hub, requiring at least 15 GitHub stars and more than 5 contributors. They retain 12K repositories that declare dependencies and are installable, then keep the 92K Python-file seeds that pass OpenCoder's default code-quality filtering.

Dependency traversal: Each seed is provisioned in a containerized sandbox with its repository and dependency environment, and the dependency graph is traversed via breadth-first search (BFS), up to a pre-specified depth of 10 steps or when we collect 128K tokens of code (whichever is fulfilled first), prepending the code retrieved in each step to the context. Each hop consists of a tree-sitter query extracting class and function call-sites, followed by a GoToDeclaration call per call-site using the jedi language server, then another tree-sitter query to extract the implementation from the target file. The authors make sure never to repeat classes and functions and discard contexts shallower than 5 levels as well as those with a hop success rate below 60%, ending with ≈6.2B code tokens.

Language Server Protocol rationale: The authors leverage LSP because "mature language server implementations for popular programming languages can use the established conventions of detected package managers, in tandem with the abstract syntax tree (AST) information produced by the compiler, to map the location of imported classes and functions. They emphasize that the LSP ecosystem offers a robust, type- and scope-aware approach to retrieve dependency code unattainable by predecessors reliant on pure syntactic parsing. While data collection is restricted to Python, the OctoLong pipeline is extensible to any programming language with a mature LSP backend and a package manager."

The paper compares OctoLong contexts (configured with BFS depth limits of 8, 10, and 15) against traditional long-context sources: books (RedPajama), academic papers (Common Pile), event co-referenced documents (MultiNews), embedding retrieval-augmented documents (NextLong), citation retrieval-augmented documents (MegaWika 2), and in-repo code (Stack V3). Table 1 shows that OctoLong flexibly sources long-context data that are (i) orders of magnitude longer and (ii) exhibit superior dependency density compared to existing long-context sources. For example, the median length at depth limit 10 is 201,477 tokens versus 88,329 for books and 4,879 for in-repo code; the 99th percentile reaches 1.58M–9.78M tokens.

For dependency-density diagnostics, the authors use a modified LongAttn score, computing the mean attention mass M of the last 2K query tokens against the most recent eighth, quarter, and half of the (n − 2)K token keys from the prefix. OctoLong data exhibits markedly higher dependency mean (≈0.61–0.63 across depth limits versus 0.55 for books, 0.53 for papers, and 0.53 for in-repo code) and a markedly lower sink concentration (Sink Prop. ≈2.0e-5 versus 1.6e-4 for books and 3.8e-4 for in-repo code), indicating a notably lower incidence of lost-middle patterns.

They additionally apply a modified LongPPL diagnostic measuring the proportion of context-sensitive key tokens among the final 2K tokens for which perplexity is markedly reduced in the presence of the full context compared to just the trailing 2K tokens of the prefix. OctoLong contexts contain 3–23 times more context-sensitive key tokens than other long-context corpora (≈6% key tokens versus 0.83% for books, 1.21% for papers, and 1.55% for in-repo code). These key tokens represent context-sensitive and reasoning-intensive points in the data that make models more amenable to post-training.

Mid-training (LCFT) data: The ≈50B-token OctoLong-LCFT corpus mixes ≈6.2B tokens of OctoLong data with conventional short- and long-context mid-training data spanning the categories of the ProLong corpus. Table 2 compares category token prevalence: OctoLong data constitutes 30% of the LCFT mix versus ProLong's 12.05% in-repo/code-adjacent category. Auxiliary data sources include in-repo code, academic papers, math, books, educational data, tech content, general domain, and agentic/instruct-like data. The authors set a 128K length limit, apply an English-only filter, and perform MinHash de-duplication.

SFT data: The ≈10B-token OctoLong-SFT mix covers a wide distribution of domains, context lengths, and interaction turns and includes general instruction following, mathematics, code, tool use, and long-context interaction data (detailed in Appendix B). They argue for dedicated long-sequence alignment because more recent evidence points to dedicated long-sequence alignment as critical for unlocking competitive performance on generation- and reasoning-heavy long-context tasks.

Training configuration: All models descend from Qwen3 dense base models (0.6B–14B). The LCFT phase adopts ABF RoPE scaling, where we raise the base frequency from the Qwen3 default of 1M to 10M while expanding its context length from 32K to 128K tokens. They use a single-stage context extension (no multi-stage setup) after exploratory runs showed no penalty. To mitigate short-input regressions, they model-merge the pre- and post-LCFT checkpoints (a 1:9 linear merge), obtaining the OctoLong-Base suite. Training uses the Swift framework with DeepSpeed-Ulysses sequence parallelism and Liger kernel optimizations, best-fit-decreasing packing into 128K contexts (packing efficiency over 99.87%), AdamW with peak LR 1e-5, and a WSD scheduler with 5% warmup. Figure 3 shows loss curves for both stages.

The authors evaluate OctoLong-Instruct against 18 open-weight baselines supporting at least 64K context length, across four research questions. They perform three ablations on the 8B model: replacing OctoLong data with the ProLong control corpus during LCFT (-Cross-Repo Code), skipping model merging before SFT (-Model Merging), and skipping LCFT entirely (-Context Extension). Inference uses nucleus sampling (p=0.95, temperature=0.6, 2K token budget). Benchmark subsets averaging over 128K tokens are skipped.

Evaluated on LooGLE V2 Code (Call Graph and Version Control tasks), LongCodeQA (64K/128K MCQ), and RepoQA (64K/128K natural-language retrieval of function implementations). The OctoLong-Instruct suite shows consistently strong performance... in long-context code state-tracking (Call Graph) and retrieval (Version Control and LongCodeQA) scenarios, including ones with low lexical overlap (RepoQA). The 8B model achieves 29.76 on LooGLE V2 Code, 65.01 on LongCodeQA, and 64.63 on RepoQA, outperforming all 18 baselines in its class. Comparing the 8B variant trained with vs. without OctoLong data, the authors observe a substantial performance contribution of the dependency-rich cross-repository code data, with especially outsized gains... on the Call Graph task. They note minimal performance regression between 64K and 128K settings, which is rare for models at the edge of their supported context capabilities. They also observe transfer beyond Python: despite restricting our data collection to Python contexts, we observe strong averages on the RepoQA task, which covers five programming languages.

Evaluated on LongBench V2 (medium-length subsets: single-doc, multi-doc, long ICL, long dialogue), OpenAI MRCR (32K–64K and 64K–128K), and AA-LCR. The LCFT data mixture brings strong results across the board on general-domain long-context tasks, and training on cross-repository contexts transfers positively to the general domain. The 4B model fares competitively against Qwen3-4B-Instruct-2507, a strong context-extended and RL-post-trained Qwen3 model, while dominating all other size classes. The 8B model achieves 39.17 on LongBench V2, 40.91 on MRCR, and 13.00 on AA-LCR, with the-Cross-Repo Code ablation dropping by 2.14, 4.03, and 2.00 points respectively. The authors highlight a non-negligible contribution of model merging in the general domain.

Evaluated on BFCL V3 (live, non-live, multi-turn) and tau2-Bench (airline, retail, telecom domains). The OctoLong-Instruct suite shows competitive to outstanding performance... across size classes, often against baselines that have undergone extensive dedicated post-training for multi-turn interaction. The 8B model scores 61.65 on BFCL V3 and 27.35 on tau2-Bench. Crucially, the greatest performance contribution from our cross-repository OctoLong data appears on agentic tool-use tasks, with the-Cross-Repo Code ablation dropping 4.97 points on BFCL V3—the largest ablation penalty observed. This suggests the recipe avoids long-form generation pathologies common for many long-context LMs.

Evaluated on LiveCodeBench V6 (new problems from Dec 2024–April 2025) and BigCodeBench Instruct Hard, reporting pass@ 1,5,10,25 with N=50. The OctoLong models yield strong results on short-length code generation across all size classes with no regressions... on short-context coding tasks along with important gains on API usage. The 8B model achieves 20.82 pass@1 on LiveCodeBench V6 and 19.44 on BigCodeBench Instruct Hard, surpassing several code-post-trained baselines and comfortably surpassing Qwen3-8B on higher sampling budgets (49.23 vs. 40.11 pass@25 on BigCodeBench). The authors attribute this to distribution sharpening during RL post-training in baselines. The-Cross-Repo Code ablation isolates the specific benefits of OctoLong data, especially for the API use tasks, and-Model Merging shows merging is effective for retaining short-context performance. Comparative results of Llama 3.1, ProLong, and UltraLong show the common practice of context extension on an already post-trained model is suboptimal and can lead to regression in short-context performance.

The authors conclude: "In this work, we present OctoLong, an effort to engineer dependency-dense long-context corpora via recursive retrieval of code across repository boundaries... Extensive experiments and ablations demonstrate that context extension with dependency-rich code data improves context retrieval, state tracking, and repository-level code understanding while unlocking superior agentic capabilities and API usage. We hope our work helps models support larger context lengths while minimizing context rot, enabling them to tackle emerging challenges like test-time training and recursive self-improvement."

Limitations acknowledged: (1) omitting low-resource language data, limiting model sizes to 14B, and restricting context length to 128K—though the authors argue OctoLong should transfer well along these axes; (2) the recursive dependency retrieval is limited to within-language code where the language server backend can readily leverage package manager conventions, and the pipeline "likely does not account for important dependencies in complex codebases that contain subroutines compiled in a different language that the calling code interacts with using an application binary interface (ABI) or a foreign function interface (FFI), which demands polyglot static analysis and control flow tracking."

I can improve an AI system by:

• Injecting dependency-rich, cross-repository code contexts into long-context fine-tuning. Using the OctoLong pipeline—AST parsing, language-server protocol (LSP) lookups, package-manager resolution, and recursive BFS traversal—I can generate training contexts that span repository boundaries and contain up to millions of tokens of tightly coupled, multi-hop code dependencies.

• Replacing 12% of conventional long-context corpora with such cross-repository dependency data. This yields gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, without sacrificing short-context performance.

• Applying a two-stage training recipe: first, long-context fine-tuning (LCFT) on a ≈50B-token mixture with ABF RoPE scaling from 32K to 128K context; second, ≈10B-token instruction tuning (SFT) for long-sequence alignment. I can include linear model merging between pre- and post-LCFT checkpoints to prevent short-input regression.

• Using dependency-density diagnostics (modified LongAttn and LongPPL) to measure attention mass, sink concentration, and context-sensitive key tokens. I can use these to select or filter training data, reducing “lost-in-the-middle” patterns and increasing reasoning-intense long-context samples.

The improved AI system can:

• Handle 128K-token contexts reliably, with minimal performance drop from 64K to 128K.

• Track call graphs and version-control state across large codebases, answering multi-hop questions like “which function calls are affected by this change?” with higher accuracy.

• Retrieve function implementations and code definitions in repository-level QA tasks across multiple programming languages, even with low lexical overlap.

• Transfer from Python-only cross-repository training to other languages and general domains, improving long-document QA, multi-document reasoning, long-context in-context learning, and long dialogue understanding.

• Excel at agentic tool use and API calling, including multi-turn function invocation in live, non-live, and multi-turn benchmark settings, while also improving short-context code generation and API-usage coding.

• Avoid common long-context pathologies like attention sinks and lost-middle degradation, while preserving strong short-context capabilities without the need for post-hoc RL post-training.

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a 50B-token mixture containing 6.2B tokens of OctoLong code contexts, followed by 10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.