Disentangling generalization and memorization in large language models using chess
The study used chess positions with varying levels of familiarity to test if large language models (LLMs) are memorizing patterns or truly reasoning. Results show LLMs perform well when recalling common positions but fail significantly when faced with novel, unfamiliar ones, indicating a weakness in systematic generalization beyond training data.