Can a Language Model Learn Facts Continually in Its Weights?
The research investigates if language models can continuously learn new facts by writing them directly into their weights after initial training. It found that while later writes can redirect questions, preserving knowledge requires using broad training data and specific write strategies. Context remains a reliable channel for holding new facts, as weight updates cause interference.