Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison

summary

Video file (mp4)

The gist

The gist The authors introduce open-sci-ref, a family of dense transformer models trained as research baselines across multiple model (0.13B to 1.7B parameters) and token scales (up to 1T) on 8

In short

Open-sci-ref introduces a family of dense transformer models trained as reproducible research baselines across various scales and datasets. These models allow researchers to assess the quality and sanity of different training approaches by comparing them on a common compute axis. The study validates established procedures and reveals insights into scaling trends, multilingual performance, and context length generalization.

Key concepts

Open-sci-ref
A family of dense transformer models trained as open reference baselines across multiple parameter sizes (0.13B to 1.7B) and token scales (up to 1T) on eight recent open reference datasets. They serve as standardized benchmarks for comparing the sanity and quality of alternative training methods.
Scaling Trends Alignment
The reference baselines are designed so that training procedures can be compared through their scaling trends, aligning models on a common compute axis. This helps researchers understand how different training methodologies behave when scaled up or down, providing a more reliable way to assess procedure quality.
Multilingual Performance Weakening
Comparing models trained on English-dominant data (like Nemotron-CC HQ) against EuroLLM shows that mixing languages can substantially weaken problem-solving ability. Even with larger token budgets, language mixing negatively impacts performance on English benchmarks.
Context Length Generalization
The study found no significant difference in average performance when varying the context length of the 1.7B model between 4096 and 16384 tokens. This indicates that the resulting models maintain stable performance across different input contexts, making them reliable for long-context understanding applications.

Terminology used across episodes

This episode discusses

The paper

Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison · Read on arXiv

LAION · Juelich Supercomputing Center (JSC) · Open-Psi Collective · OpenEuroLLM team · ELLIS Institute Tübingen · Institute of Science Tokyo · University of Turku · University of Freiburg

Reliable progress in language modeling depends on knowing whether a new dataset, architecture, or training recipe is genuinely better than existing alternatives. To foster reproducible learning procedure comparison, we present open-sci-ref, a family of dense transformers trained as open reference baselines at four scales (0.13B-1.7B parameters) on nine main reference datasets, with token budgets up to 1T, using a single tuned and fully documented training procedure. To facilitate a shared experimentation and measurement infrastructure, we release model weights, intermediate checkpoints, training logs, code, and downstream evaluations. When comparing training procedures, we take into account total compute spent. Placing training procedures on a common compute axis helps identify reference points that are weak or strong for the compute they use, without requiring a full scaling law study for every comparison. We identify Nemotron-CC-HQ as the strongest reference dataset on our evaluation suite: our 1.7B model trained on 1T of its tokens matches SmolLM2-1.7B with roughly 11 times less compute. Shared baselines expose pitfalls of scaling trend comparisons: when a small-scale point is unusually weak, the resulting slope can overstate the scalability of a training procedure. We illustrate this diagnostically with EuroLLM-1.7B, which underperforms 300B token FineWeb-Edu baseline despite 13 times more training compute. As an additional case study, the suite supports controlled comparison between permissively licensed and mixed-license datasets: the permissive-first MixtureVitae dataset changes rank substantially with scale. This also illustrates why data ranking should be evaluated through multi-scale reference trajectories rather than single proxy runs. All artifacts are publicly released.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison".

Jane: The gist The authors introduce open-sci-ref,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we’re looking at the paper "Open-sci-ref: open reference baselines for reproducible language model and dataset comparison," and they start by introducing this family of models trained across a range of parameters, from zero point thirteen billion to one point seven billion parameters.

Jane: And they are training these models across different token scales, going up to one trillion tokens, which is pretty massive given the scale we see in modern AI research right now.

Lu: What’s interesting about the title is that it focuses on making these open reference baselines usable for comparing different training approaches because it introduces a way to compare them through their scaling trends, aligning them on a common compute axis.

Meng: That makes sense from an engineering standpoint because if you have a new training procedure, you need those established reference points so you can quickly check the sanity of your approach across different scales and datasets before committing massive compute budgets.

Lalam: They are using these established reference baselines to allow training procedures to be compared through their scaling trends, which really helps align everyone on what's actually happening when you scale up or down >

The paper's summary: Tom: The core idea here is building this family of models, the open-sci-ref, trained as research baselines across eight different recent open reference datasets. They are using these models to evaluate the sanity and quality of alternative training procedures across various scales and datasets >

Jane: What this means for us is that instead of just looking at a single final score on one benchmark, researchers can compare different training methods by looking at how they scale, which helps them check the sanity of their approaches across different sizes and datasets >

Lu: Essentially, they are giving the community a set of established reference points so that anyone can compare novel models and datasets against them on a common axis when evaluating their performance or understanding how they learned >

Meng: From an engineering view, this is valuable because it helps in validating training procedures and checking if the methods make sense before you run them at massive scale, which is good for stability and sanity >

Lalam: I think the most significant part of this paper is using these established reference baselines to compare training procedures through their scaling trends, which really helps align on what's actually happening when you scale up or down >

The paper's improvements: Tom: They highlight that the established reference baselines allow training procedures to be compared through their scaling trends, which is crucial because points on smaller scales that are highly suboptimal can create the illusion of a strongly scalable learning procedure by making the slope very steep >

Jane: So they are emphasizing robust scaling trend estimation because they want to prevent over-reliance on superficial scaling trends and ensure more accurate predictions about how learning procedures behave across different compute budgets >

Lu: That’s a really deep point, Jane. By providing these reference points, you are essentially giving researchers data that helps them build better models for predicting learning procedure behavior across scales and datasets >

Meng: For deployment, this helps because if you see a scaling trend that looks too good on a small scale but breaks down later, you can spot that problem early instead of waiting until you run it at massive compute budgets >

Lalam: I think the key point is using these established reference baselines to compare training procedures through their scaling trends, which helps researchers align on what's actually happening when you scale up or down >

Conclusion: Tom: So we just walked through "Open-sci-ref: open reference baselines for reproducible language model and dataset comparison." The main idea is building this family of models across different scales, from zero point thirteen billion to one point seven billion parameters, using various token sizes up to one trillion tokens >

Jane: It’s about establishing these reference points so researchers can evaluate the sanity and quality of other training approaches. They want to make it easier to compare different training methods by aligning them on a common compute axis >

Lu: What they are doing is using these established reference baselines to let you compare training procedures through their scaling trends, which helps you assess the sanity and quality of alternative training approaches across scales and datasets >

Meng: I’m thinking about the practical side here; having these established reference baselines means that when you try a new training procedure, you have a baseline to check against so you can see if it makes sense when scaling up or down >

Lalam: Exactly, and they use intermediate checkpoints to let you study the training dynamics, which is important because it shows how the model is learning at different stages of its training >

Tom: We’ve gone through "Open-sci-ref: open reference baselines for reproducible language model and dataset comparison." The main thing to remember is that they’ve created this family of models trained across scales and token sizes, giving researchers those reference points to evaluate new training methods >

Jane: It means researchers can now compare different training procedures by looking at how they scale, which helps them check the sanity of their approaches across different sizes and datasets. It gives them a way to align things on a common compute axis when comparing methods >

More episodes

← Home