Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning
cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/sahil280114/codealpaca
Terminology
Sources
- Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
- A Survey on Data Selection for Language Models
- Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning
- Online Batch Selection for Faster Training of Neural Networks
- Accelerating Deep Learning by Focusing on the Biggest Losers
- BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training
- OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
- Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning
- TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data
- Influential Language Data Selection via Gradient Trajectory Pursuit
- Efficient Per-Example Gradient Computations
- In-Run Data Shapley for Adam Optimizer
- Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
- Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering
- Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
- Training Verifiers to Solve Math Word Problems
- Evaluating Large Language Models Trained on Code
- The Llama 3 Herd of Models
- Qwen2.5 Technical Report
- Mistral 7B
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks