Rethinking Bayesian Optimization for Co-Optimizing LLM Training Configurations
cs.LG, cs.AI
Submitted: 2026-02-09
Updated: 2026-10-28
Code: https://github.com/princeton-nlp/LESS
License: http://creativecommons.org/licenses/by/4.0/
The gist: Fine-tuning an LLM to maximize performance on a downstream task requires finding the optimal data and model configuration.
Terminology
Abstract
Fine-tuning an LLM to maximize performance on a downstream task requires finding the optimal data and model configuration. This is a black-box optimization problem that Bayesian optimization (BO) addresses by sequentially evaluating training configurations and using observed performance feedback to adaptively guide the search towards better configurations. However, directly applying BO to the joint data-model space suffers from two fundamental challenges: the prohibitive cost of full LLM training at each iteration, and the large number of BO iterations required for effective exploration due to the high problem dimensionality. This paper introduces JoBS, a BO-based approach that co-optimizes data and model configurations without requiring a full training run at every iteration. JoBS allocates an initial portion of the optimization budget to learn a scaling-law-inspired performance predictor that estimates fully trained LLM performance from only a small number of training steps. The remaining budget is then used to run BO with this predictor, eliminating the need for full training runs and enabling JoBS to explore significantly more data and model configurations within the same budget. We analyze JoBS's average regret and derive the optimal budget allocation between predictor learning and BO iterations. Empirically, JoBS outperforms independent data and model optimization methods and existing multi-fidelity BO baselines across a diverse set of downstream tasks. Our code is available at https://github.com/chenzhiliang94/BO-for-LLMs.
Sources
- Aioli: A Unified Optimization Framework for Language Model Data Mixing
- Scaling Laws for Predicting Downstream Performance in LLMs
- DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Tasks
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- A Tutorial on Bayesian Optimization
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- The Llama 3 Herd of Models
- Robustifying and Boosting Training-Free Neural Architecture Search
- Measuring Massive Multitask Language Understanding
- LoRA: Low-Rank Adaptation of Large Language Models
- PubMedQA: A Dataset for Biomedical Research Question Answering
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
- Scaling Laws for Neural Language Models
- Understanding Black-box Predictions via Influence Functions
- Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- DARTS: Differentiable Architecture Search
- RegMix: Data Mixture as Regression for Language Model Pre-training
- Pointer Sentinel Mixture Models
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning