Understanding the Interplay between Memorization and Learning in Large Language Models
The study re-examines three memorization measures—recollection-based, counterfactual, and contextual—to understand how LLMs learn. It finds that optimal language learning requires some memorization, but improved learning reduces contextual and counterfactual memorization while increasing recollection-based memorization. This trade-off depends heavily on which measure is used.