Bergson: An Open Source Library for Data Attribution
cs.LG
Submitted: 2026-06-10
Updated: 2026-09-25
Code: https://github.com/huggingface/peft
License: http://creativecommons.org/licenses/by/4.0/
The gist: Data attribution is a promising field in interpretability that aims to explain model behavior through the influence of its training data, with applications including debugging undesirable model
Terminology
Abstract
Data attribution is a promising field in interpretability that aims to explain model behavior through the influence of its training data, with applications including debugging undesirable model behavior and training dataset curation. However, significant engineering effort is required to perform it at scale, and many cutting edge techniques lack open-source tooling and support. Bergson is an open source library that aims to enable faster progress in the field by providing a host of techniques that scale to very large language models and pre-training datasets. The library natively supports on-disk gradient stores and multi-node distributed training, and provides quality of life tools for researchers. Finally, we introduce the first open-source implementations of three leading data attribution methods: MAGIC, SOURCE, and TrackStar. The library is available at https://github.com/EleutherAI/bergson .
Sources
- Second-Order Stochastic Optimization for Machine Learning in Linear Time
- Towards Tracing Factual Knowledge in Language Models Back to the Training Data
- Training Data Attribution via Approximate Unrolled Differentiation
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- Studying Large Language Model Generalization with Influence Functions
- Language Models are Few-Shot Learners
- Shampoo: Preconditioned Stochastic Tensor Optimization
- Scalable Influence and Fact Tracing for Large Language Model Pretraining
- Input Similarity from the Neural Network Perspective
- Optimizing ML Training with Metagradient Descent
- Understanding Black-box Predictions via Influence Functions
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
- Estimating Training Data Influence by Tracing Gradient Descent
- The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
- Scaling Up Influence Functions
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks