Conditional Transfer from Controlled Pretraining Mixtures to Code
Researchers studied how mixing different types of synthetic tasks affects a language model's ability to transfer knowledge to code generation tasks. They found that a mixture dominated by a target-aligned source and 25% from high-quality curated synthetic data yielded the best performance, suggesting curated data has conditional value. A loss-based scheduling approach, however, often steers the model away from this optimal mixture.