Different Corpora, Different Skills? A Paired Study of Small Language Models
cs.CL
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/EdoardoVaira/babylm-corpora-skills
Project page: https://babylm.github.io
Terminology
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- A Survey on Data Selection for Language Models
- Language Models are Few-Shot Learners
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- Textbooks Are All You Need
- Distilling the Knowledge in a Neural Network
- Training Compute-Optimal Large Language Models
- Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models
- Entity Tracking in Language Models
- DataComp-LM: In search of the next generation of training sets for language models
- A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
- COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty
- LLaMA: Open and Efficient Foundation Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data