Memorizing Long-tail Data Can Help Generalization Through Composition
The research investigates how memorizing rare data points (long-tail features) can improve a model's ability to generalize, specifically through composition—combining these rare features for correct predictions. Theoretically, in a linear setting, memorization synergizes with composition to handle unseen combinations of long-tail features. Experiments confirm this effect in neural networks, suggesting memorization enables compositional learning.