Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models
cs.CL, cs.LG
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/yiksiu-chan/entity-comparison
Terminology
Sources
- Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
- Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions
- LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations
- Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context
- Why Language Models Hallucinate
- Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
- Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior
- 2 OLMo 2 Furious
- In-context Learning and Induction Heads
- In-Context Algebra
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data