Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
The episode discusses a paper showing that entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives. Researchers found that human-level tracking is present at four hundred ten million parameters, suggesting smaller models can build better situation models. This challenges the idea that massive parameter counts are necessary for deep comprehension.