Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
Open-sci-ref introduces a family of dense transformer models trained as reproducible research baselines across various scales and datasets. These models allow researchers to assess the quality and sanity of different training approaches by comparing them on a common compute axis. The study validates established procedures and reveals insights into scaling trends, multilingual performance, and context length generalization.