The Evolution Strategies Paradox: Better Models from Worse Perturbations
cs.AI
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: 20 pages, 9 figures
Code: https://github.com/Jiayi-Pan/TinyZero
License: http://creativecommons.org/licenses/by/4.0/
The gist: How is it possible for perturbation-based methods like Evolution Strategies (ES) to optimize billion-parameter large language models (LLMs)? Classical search methods are highly dependent on finding…
Terminology
Abstract
How is it possible for perturbation-based methods like Evolution Strategies (ES) to optimize billion-parameter large language models (LLMs)? Classical search methods are highly dependent on finding good candidates within generated populations. However, when using ES to fine-tune LLMs, a phenomenon can occur where all offspring models consistently perform worse than the parent model, yet during training, the parent model consistently improves. This paradox occurs across models, tasks, and even modalities. This paper characterizes the paradox, then examines how the local reward landscape and properties of the ES estimator contribute to it: in high-dimensional spaces, exploration leads to worse performance, yet still highlights good directions; aggregating those directions then allows for making local improvements. The result can be seen as a new optimization paradigm where a principled interaction of noise and curvature produces the paradox as a byproduct of effective optimization. These findings provide guidance for taking advantage of perturbation-based methods in LLM fine-tuning and in high-dimensional optimization in general.
Sources
- Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies
- Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training
- The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective
- The role of the nugget term in the Gaussian process method
- Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Natural Evolution Strategies
- d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents