Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models
cs.LG
Submitted: 2026-10-01
Updated: 2026-10-01
Sources
- DataRater: Meta-Learned Dataset Curation
- SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Training
- Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning
- AlpaGasus: Training A Better Alpaca with Fewer Data
- MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
- Adversarial Active Learning for Deep Networks: a Margin Based Approach
- Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models
- The Llama 3 Herd of Models
- Measuring Massive Multitask Language Understanding
- Simple and Scalable Strategies to Continually Pre-train Large Language Models
- Editing Models with Task Arithmetic
- Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives
- Examining Forgetting in Continual Pre-training of Aligned Large Language Models
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
- When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale
- Synthetic Text Generation for Training Large Language Models via Gradient Matching
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks