LoRIF: Pushing the Scalability Frontier of Training Data Attribution
LoRIF addresses the challenge of attributing training data to massive models efficiently. It achieves effective data attribution for models up to 70B parameters and millions of examples by storing low-rank factors of projected gradients instead of full matrices. This results in significant storage reduction, faster query times, and enhanced ability to debug model behavior sensitive to specific training examples.