LoRIF: Pushing the Scalability Frontier of Training Data Attribution

arXiv:2601.21929 · cs.LG · Submitted 2026-01-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "LoRIF: Pushing the Scalability Frontier of Training Data Attribution".

Jane: Here's the gist of the paper. The provided text does not contain the abstract or a summary of the scientific paper.

Tom: It only contains metadata and guidelines related to asset documentation, crowdsourcing, IRB approvals, and LLM usage for a submission process.

Jane: Therefore, no summary can be extracted from this part.

Title and authors: Tom: The title LoRIF: Pushing the Scalability Frontier of Training Data Attribution tells you right away that they are focused on making attribution work on models that are getting way bigger. It’s about pushing the limits of what we can do when dealing with massive datasets.

Jane: And I think the authors, Shuangqi Li, Hieu Le, Jingyi Xu, and Mathieu Salzmann, are from places like EPFL in Switzerland and Stony Brook University in the USA. That's a good mix of theoretical grounding and practical application for this kind of deep learning work.

Lu: What they’re proposing is essentially a new way to do training data attribution that leverages low-rank structures in neural network gradients. They are trying to find an effective way to get high quality attribution without hitting the walls of computational cost.

Meng: I'm curious about the specific mechanism behind this low-rank exploitation. It sounds like they’re assuming that the gradients don't actually have billions of dimensions, which is a big assumption when you're dealing with these giant models.

Lalam: The paper focuses on exploiting that property of low effective rank, both within individual layers and across the whole aggregate set of examples, to make the process tractable >

The paper's summary: Tom: So what LoRIF actually does is introduce Low-Rank Influence Functions. They are using this framework to answer which training examples were most important for a model's output. It’s about finding that influence function, which is theoretically grounded in using gradients.

Jane: The summary highlights that standard methods face two main problems: first, you have to store and load projected per-example gradients for all N training examples during the query phase, which hurts I/O speed; and second, forming the inverse Hessian approximation costs a lot of memory.

Lu: That’s the quality versus scalability tradeoff they are trying to solve. They acknowledge that increasing the projection dimension D improves attribution quality, but that making D big makes storing those gradients and calculating the inverse Hessian approximation too costly >

Meng: So if I understand this right, they are finding a way to keep the storage at a manageable size while still getting enough quality for practical use cases on large models. That’s what matters for production deployment.

Lalam: They introduce LoRIF to substantially mitigate that tradeoff by exploiting the low-rank nature of neural network gradients, which is key to making high attribution quality tractable >

The paper's improvements: Tom: The main improvement they point out is that they store rank-c factors of projected per-example gradients instead of storing the full matrices. That’s what leads to a storage reduction, and it’s a big one.

Jane: They claim this allows for up to twenty times storage reduction compared to older methods like LoGRA, which is significant when dealing with millions of training examples. It directly tackles that I/O bottleneck we talked about earlier.

Lu: Beyond storage, they also address the query time latency by using these low-rank factors. They show results demonstrating a query-time speedup, sometimes showing up to thirty times less total query latency than LoGRA >

Meng: That thirty times reduction in latency is what I'm looking for. If we can cut the time it takes to get an attribution answer by that much, it makes the whole system usable for real-time debugging or auditing without grinding everything to a halt >

Lalam: They also show they can inspect training examples where the model’s behavior is sensitive across multiple different sensitive query contexts. This means you can actually identify safety-critical training examples that simple text similarity checks might miss >

Conclusion: Tom: So to wrap up, LoRIF takes the challenge of scaling data attribution for models up to seventy billion parameters and millions of examples and uses low-rank structures in gradients to reduce storage by up to twenty times and speed up queries by about thirty times.

Jane: The big implication is that researchers can finally audit massive models without needing prohibitive computational expense, allowing for better safety checking on the data itself. It shifts the bottleneck from computation cost to something more manageable, like I/O.

Lu: They successfully navigate that quality versus scalability tradeoff by showing that you can increase D, which means better quality attribution, while keeping both storage and I/O costs tractable >

Meng: For a production environment, this means we can actually implement these attribution methods on the scale we need without immediately needing to redesign our entire infrastructure for matrix operations. It makes the research practical >

Lalam: LoRIF is about making influence functions practical at frontier scales by exploiting the low-rank property of gradients >

EPFL · University of North Carolina Charlotte, USA · Stony Brook University, USA

cs.LG

Submitted: 2026-01-29

Updated: 2026-10-29

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: "LoRIF: Pushing the Scalability Frontier of Training Data Attribution." The provided text does not contain the abstract or a summary of the scientific paper.

Key concepts

Effective Data Attribution
This is the process of identifying which specific training data points influenced a model's final output. LoRIF makes this practical for huge models by focusing on capturing the most important data influences without needing to process every single example exhaustively, making auditing feasible.
Low-Rank Factors
Instead of storing the entire complex gradient matrix, LoRIF stores only a simplified representation called low-rank factors. This technique captures the essential structure of how data relates to model parameters in a compact form, which is crucial for achieving massive storage savings and faster lookups.
Scalability Frontier
This refers to pushing the limits of what is computationally possible when dealing with extremely large models (up to 70B parameters) and vast datasets. LoRIF demonstrates how to maintain high-quality data attribution methods while keeping computational costs and latency manageable for these frontier models.

Terminology

Summary

LoRIF: Pushing the Scalability Frontier of Training Data Attribution. The provided text does not contain the abstract or a summary of the scientific paper. It only contains metadata and guidelines related to asset documentation, crowdsourcing, IRB approvals, and LLM usage for a submission process. Therefore, no summary can be extracted from this part.

Improvements for AI systems

  1. Scalable Training Data Attribution for Frontier Models: LoRIF enables effective data attribution on models up to 70B parameters and millions of examples with practical cost and latency, allowing for auditing of massive models without prohibitive computational expense.

  2. Improved Storage Efficiency for Gradients: By storing rank-c factors of projected per-example gradients rather than full matrices, LoRIF achieves up to 20x storage reduction compared to methods like LoGRA, making gradient-based TDA practical at frontier scale.

  3. Faster Query Latency: The method reduces query time by addressing the I/O bottleneck by storing low-rank factors, leading to a query-time speedup and demonstrated results showing up to 30x reduction in total query latency compared to LoGRA.

  4. Enhanced Model Debugging and Safety Auditing: LoRIF allows researchers to inspect training examples that the model’s behavior is sensitive across multiple sensitive-query contexts, enabling actionable data intervention by identifying safety-critical training examples missed by simple text similarity retrieval.

  5. Better Quality vs Scalability Tradeoff: The method effectively addresses the quality–scalability tradeoff by exploiting low-rank structures, enabling practitioners to increase D (improving quality) while keeping both storage/I/O and curvature costs tractable.

Abstract

Training data attribution (TDA) identifies which training examples most influenced a model's prediction. Influence function methods are a theoretically grounded family of TDA methods and exploit gradients. To overcome the scalability challenge arising from gradient computation, the most popular strategy is random projection (e.g., TRAK, LoGRA). However, this still faces two bottlenecks when scaling to large training sets and high-quality attribution:(i) storing and loading projected per-example gradients for all N training examples, where query latency is dominated by I/O; and(ii) forming the D times D inverse Hessian approximation, which costs O(D 2) memory. Both bottlenecks scale with the projection dimension D, yet increasing D is necessary for attribution quality -- creating a quality--scalability tradeoff. We introduce LoRIF (Low-Rank Influence Functions), which exploits low-rank structures of gradient to address both bottlenecks. First, we store rank- c factors of projected per-example gradients rather than full matrices, reducing storage and query-time I/O from O(D) to O(c sqrt D) per layer per sample. Second, we use truncated SVD with the Woodbury identity to approximate the inverse Hessian term in an r-dimensional subspace, reducing memory from O(D 2) to O(Dr). On models from 0.1B to 70B parameters trained on datasets with millions of examples, LoRIF achieves up to 20 times storage reduction and query-time speedup compared to LoGRA, while matching or exceeding its attribution quality. LoRIF makes gradient-based TDA practical at frontier scale. Code is available at https://github.com/doub7e/LoRIF.

Sources

Related papers