A Language Model from 1913: Pretraining on Historical Text

Name: A Language Model from 1913: Pretraining on Historical Text
Uploaded: 2026-10-06T12:47:00.387635+00:00
Duration: 1176 s
Description: TYPEWRITERLM is a 7.24B language model trained exclusively on English text from before 1913 to create historically grounded AI for NLP research. By using a strict knowledge cutoff and rigorous leakage mitigation techniques, the model achieves performance comparable to modern LLMs while demonstrating how historical constraints affect reasoning and temporal consistency.

TYPEWRITERLM is a 7.24B language model trained exclusively on English text from before 1913 to create historically grounded AI for NLP research. By using a strict knowledge cutoff and rigorous leakage mitigation techniques, the model achieves performance comparable to modern LLMs while demonstrating how historical constraints affect reasoning and temporal consistency.

Open the full episode