Conditional Transfer from Controlled Pretraining Mixtures to Code

arXiv:2610.11548 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Conditional Transfer from Controlled Pretraining Mixtures to Code".

Jane: The gist The best observed HumanEval pass@20 came from a mixture in which the target-aligned source held 75% of the slice and a highly teachable curated synthetic suite held the…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've looked at how synthetic tasks can be used for pretraining or as probes, and this paper is about how mixing those different types of data affects the AI's final ability to write code (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "Ohad Rubin Abstract Synthetic tasks are increasingly used both as probes of language-model capability and as pretraining data").

Jane: It’s called Conditional Transfer from Controlled Pretraining Mixtures to Code. The authors are looking at these mixtures and trying to figure out which ones actually help the model learn coding skills better (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "These two uses are usually justified by the same evidence: a task whose loss falls during training is treated as informative, and a task whose loss falls faster when it is sampled more is treated as worth sampling").

Lu: Essentially, they separate the signals in how loss is going down during training. They track whether a task is diagnostic, if it’s teachable based on its own sampling budget, and if including it actually improves a final coding test score like HumanEval pass at twenty (Conditional Transfer from Controlled Pretraining Mixtures to Code page two of that work reads: "Our contributions are as follows. one. Three separable signals. We give operational definitions of diagnosticity (loss tracks global training progress), teachability (loss responds to the task’s own token budget), and transfer (a source’s share improves the downstream target)").

Meng: The experiment set up fixed seventy percent general Python and varied the remaining thirty percent between instruction-based code, curated synthetic tasks, and literature-derived probes. It’s about finding the right recipe for that mix (Conditional Transfer from Controlled Pretraining Mixtures to Code page three of that work reads: "Every pretraining run draws seventy percent of its tokens from a fixed general Python corpus (Base) from CodeParrot and thirty percent from a controlled slice whose composition is the only manipulated variable").

Lalam: The core finding is that if you want good code transfer, you need to keep the source that's directly related to the target task dominant in your mixture (Conditional Transfer from Controlled Pretraining Mixtures to Code page ten of that work reads: "For code transfer, keep the aligned source dominant").

Tom: So what does this mean for us? It suggests we can't just throw all synthetic data into the model randomly; we have to choose our ingredients carefully (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "Synthetic tasks are increasingly used both as probes of language-model capability and as pretraining data").

Jane: Right. The paper shows that curated synthetic data has value, but only when it’s part of a larger mix where the instruction-based source still holds most of the power (Conditional Transfer from Controlled Pretraining Mixtures to Code page six of that work reads: "Curated synthetic data therefore has conditional value: it contributes as a limited share of a mixture that a target-aligned source still dominates").

Paper summary: Lu: It’s interesting how they found that mixing seven-five percent OpenCodeInstruct with a twenty-five percent curated suite gave the best HumanEval score (Conditional Transfer from Controlled Pretraining Mixtures to Code page seven of that work reads: "On the mixture simplex edge between the curated suite and OpenCodeInstruct, HumanEval pass@twenty after a fixed fine-tuning stage rises from fifteen point nine at pure curated data to its highest observed value, twenty-two point six, at a mixture that is seventy-five percent OpenCodeInstruct").

Meng: That points to complementarity between those two data types, maybe reusable operations and target instructions. It's not just one thing working well alone (Conditional Transfer from Controlled Pretraining Mixtures to Code page six of that work reads: "The interpretation suggests complementarity between reusable program-like operations and target-aligned instruction data").

Tom: And then there’s this adaptive scheduler they tested; it tried to follow the teachability signal but ended up pushing away from the best transfer region for most of the training time (Conditional Transfer from Controlled Pretraining Mixtures to Code page seven of that work reads: "Ado drives the OpenCodeInstruct share below five percent within the first 5k steps and to one point two--one point four percent by the end of training, and underperforms its matched fixed-mixture controls by two point four--eleven point zero percentage points").

Jane: So what does that tell us about how we should tune our training processes? It suggests optimizing for immediate loss reduction isn't always what leads to better final performance on a specific task (Conditional Transfer from Controlled Pretraining Mixtures to Code page seven of that work reads: "Optimizing near-term task-loss reduction moves the mixture away from the region that transfers").

Lu: The conclusion is that for code transfer, keep the aligned source dominant, treat curated synthetic data as a bounded share, and pick proportions based on signals that point toward transfer (Conditional Transfer from Controlled Pretraining Mixtures to Code page ten of that work reads: "For code transfer, keep the aligned source dominant, treat curated synthetic data as a bounded share, and choose proportions with a target-referenced signal").

Meng: So we need to think about the long game of pretraining mixtures rather than just chasing the fastest loss drop right now. That’s what this paper suggests for building better code AI (Conditional Transfer from Controlled Pretraining Mixtures to Code page ten of that work reads: "So we need to think about the long game of pretraining mixtures rather than just chasing the fastest loss drop right now").

Tom: So the big picture here is that understanding how different data sources interact—whether they are diagnostic or teachable—is more important than just seeing which one has the lowest loss right now (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "The authors are defining these three signals very precisely").

Jane: And that's what this paper on Conditional Transfer from Controlled Pretraining Mixtures to Code is all about.

Conclusion: Tom: So we're wrapping up this look at how we mix pretraining data to get better code generation results. The paper is titled "Conditional Transfer from Controlled Pretraining Mixtures to Code".

Jane: That title really tells you what they're doing—it’s not just about using synthetic tasks, it’s about *how* you condition that mixture.

Lu: The authors are defining three ways we can measure these signals: diagnostic, teachability, and transfer. They show that mixing data is complex because those signals don't always line up with what actually helps the final coding test score like HumanEval pass at twenty.

Meng: What this really means for us is that we can't just throw all synthetic data into the model randomly; we have to be strategic about which source families we mix in based on where you are in the training process.

Tom: Exactly. We learned that optimizing for immediate loss reduction, which is what some schedulers try to do, doesn't always lead to better transfer performance for the actual coding task.

Jane: It suggests that chasing every single drop of loss right now might actually push the model away from the sweet spot where it learns how to generate code effectively.

Lu: The implication here is that we need a more nuanced approach when we design these training mixtures if we want to get high-quality code results out of these models.

Meng: For me, it means when I'm building things, I can’t just blindly pile on every piece of synthetic data; I have to think about which source family is most relevant for the final application.

Ohad Rubin

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist The best observed HumanEval pass@20 came from a mixture in which the target-aligned source held 75% of the slice and a highly teachable curated synthetic suite held the remaining 25% (Page

Key concepts

Diagnostic Signal: This signal checks if a specific task's loss reduction follows the overall progress of training on general pretraining data. If a task's loss tracks global progress, it is considered diagnostic and informative for learning.
Teachable Signal: Teachability measures whether a task's loss responds predictably to its own allocated token budget across multiple training runs. A teachable task shows that the model can effectively learn from its specific sampling allocation.
Transfer Signal: This signal assesses whether including a specific source data type improves performance on a separate, downstream target task (like HumanEval). Transfer occurs when increasing the share of a source improves the final target metric.

Summary

The gist The best observed HumanEval pass@20 came from a mixture in which the target-aligned source held 75% of the slice and a highly teachable curated synthetic suite held the remaining 25% (Page 16).

Three Separable Signals

The paper separates three signals that are usually read off the same loss curve (Page 1). These signals are diagnostic, teachability, and transfer (Page 4). Diagnosticity concerns whether a task is diagnostic when its held-out loss tracks global pretraining progress (Page 4.2), while teachability concerns whether its held-out loss responds to its own token budget across runs (Page 4.3). Transfer concerns whether including a source improves the downstream target, defined as y(w) > y(w') when increasing its share from w' to w (Page 4.4).

Teachability and Source Families

Across a per-task budget sweep, teachability is common and family-dependent (Page 2). The split is sharply asymmetric: 10 of 12 curated tasks are teachable, but only 4 of 15 literature-derived probes are (Page 2). The curated suite is the most teachable family, while its pure-source endpoint falls below the zero-synthetic baseline (Page 6.3).

Transfer Depends on Mixture Context

Transfer depends on mixture context (Page 4.4), as demonstrated by results on the curated–OpenCodeInstruct edge (Page 6.1). HumanEval pass@20 is non-monotone in the curated share, peaking at 22.6 when OpenCodeInstruct holds 75% of the slice and falling to 15.9 at pure curated data (Page 7). The interpretation suggests complementarity between reusable program-like operations and target-aligned instruction data (Page 6.2).

Loss-Based Scheduling Follows the Wrong Signal

Ado [7] allocates budget by predicted remaining loss reduction, which is a teachability signal measured online at the task level (Page 4.5). The scheduler moves away from the region preferred by the downstream surface for most of the 60k-step training horizon (Page 7.4). Specifically, Ado drives the OpenCodeInstruct share below 5% within the first 5k steps and to 1.2–1.4% by the end of training, underperforming matched fixed-mixture controls by 2.4–11.0 percentage points (Page 7).

Practical Rule and Conclusion

The practical rule drawn is: When the downstream target is code generation, keep the target-aligned source dominant in the mixture, add curated synthetic data as a bounded share, and choose proportions with a target-referenced signal (Page 1). The conclusion states: For code transfer, keep the aligned source dominant, treat curated synthetic data as a bounded share, and choose proportions with a target-referenced signal (Page 10).

Key Findings on Data Types

The curated suite is genuinely useful: at a 25% slice share it is worth 3.1 pass@20 points over pure OpenCodeInstruct (Page 8). Standard probes are instruments, as Four of fifteen literature-derived probes respond to their own budget (Page 9). The mismatch between the scheduler's objective and downstream transfer arises because Ado maximizes predicted near-term loss reduction over its tasks, while HumanEval measures downstream transfer (Page 7.4).

Experimental Setup Details

The experiment fixes 70% of the pretraining corpus to general Python and varies only the composition of the remaining 30%, defined by a point w = (wO, wC, wL) on the 2-simplex ∆2 (Page 3.1). The downstream evaluation uses HumanEval [2] as the endpoint, and fine-tuning is performed for every run on a refined OpenCodeInstruct training split (Page 3.4).

Model Architecture

All runs use a sparse-expert decoder-only model with a 50,304-token vocabulary, 12 layers, 12 attention heads, width 768 (Page 3.3). Training uses AdamW at learning rate 10−3 and runs last 60k or 120k steps (Page 3.4).

Limitations

Limitations include the dependency on a specific model family and base corpus, the entanglement of format-specific and content-specific effects, and that the downstream evidence comes from only one target (HumanEval) (Page 9). Additionally, Coarse simplex resolution means they sample at 15 points (Page 9).

References

[1] Wasi Uddin Ahmad et al. Opencodeinstruct: A large-scale instruction tuning dataset for code llms. arXiv preprint arXiv:2504.04030, 2025 (Page 11).

[7] Yiding Jiang et al. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11820, 2024 (Page 9).

[6] Michael Hanna et al. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023 (Page 6).

[8] Qian Liu et al. Regmix: Data mixture as regression for language model pretraining. arXiv preprint arXiv:2407.01492, 2024 (Page 16).

[15] Lean Wang et al. Label words are anchors: An information flow perspective for understanding in-context learning, 2023 (Page 13).

[16] Sang Michael Xie et al. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2023 (Page 17).

[7] Yiding Jiang et al. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11820, 2024 (Page 9).

[5] Simin Fan et al. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2024 (Page 7).

[4] Subhabrata Dutta et al. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning, 2024 (Page 11).

[9] Bohan Lyu et al. Surge: On the potential of large language models as general-purpose surrogate code executors, 2026 (Page 14).

[18] Dylan Zhang et al. From symbolic tasks to code generation: Diversification yields better task performers, 2024 (Page 14).

[3] Mayee F. Chen et al. Skill-it! a data-driven skills framework for understanding and training language models. arXiv preprint arXiv:2307.14430, 2023 (Page 16).