A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization

arXiv:2608.08156 · cs.LG, cs.NE · Submitted 2026-08-08 · Read on arXiv

Víctor Gallego

cs.LG, cs.NE

Submitted: 2026-08-08

Updated: 2026-08-11

Comments: Published as a conference paper at LM4Sci Workshop @ COLM 2026

Code: https://github.com/vicgalle/hybrid-nested-search

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Bayesian Inference — 9× in this paper · explained in 6 episodes
• Hysteresis — 9× in this paper · explained in 3 episodes
• Harness — 7× in this paper · explained in 2 episodes
• Calibration — 3× in this paper · explained in 9 episodes
• Preconditioner — 3× in this paper · explained in 2 episodes
• Adam — 2× in this paper · explained in 3 episodes
• Benchmark — 2× in this paper · explained in 2 episodes
• Bilevel Optimization — 2× in this paper · explained in 2 episodes
• Decoupling — 2× in this paper · explained in 2 episodes
• Meta-learning — 2× in this paper · explained in 3 episodes
• Objective Function — 2× in this paper · explained in 2 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Agentic System — 1× in this paper · explained in 2 episodes
• Bayesian Optimization — 1× in this paper · explained in 3 episodes
• Cooperation — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Human-in-the-loop — 1× in this paper · explained in 4 episodes
• Hyperparameters — 1× in this paper · explained in 2 episodes
• Localization — 1× in this paper · explained in 2 episodes
• Parameterization — 1× in this paper · explained in 2 episodes
• Regret — 1× in this paper · explained in 4 episodes
• Robustness — 1× in this paper · explained in 6 episodes
• Sample Efficiency — 1× in this paper · explained in 7 episodes

Sources

• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
• Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
• Beyond Scalar Rewards: Dense Feedback for LLM Policy Synthesis in Sequential Social Dilemmas
• EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
• NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport
• Automated Design of Agentic Systems
• Large Language Models As Evolution Strategies
• ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
• Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model
• Large Language Models to Enhance Bayesian Optimization
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• Eureka: Human-Level Reward Design via Coding Large Language Models
• Validating Bayesian Inference Algorithms with Simulation-Based Calibration
• Large Language Models as Optimizers
• TextGrad: Automatic "Differentiation" via Text
• Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks