Simulating, Fast and Slow: Learning Policies for Black-Box Optimization

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Simulating, Fast and Slow".

Jane: Here's the gist of the paper. The gist The paper introduces a novel Reinforcement Learning framework to learn an active learning policy that guides a differentiable surrogate’s training and uses its gradients to optimize simulation parameters with gradient descent, reducing expensive simulator calls by up to ∼90% compared to baselines Problem Context Optimization problems involving black-box simulators are crucial in science and engineering because they often require finding parameters that minimize a desired loss function without easy access to gradients from the forward process fsim: (ψ, x) → y The goal is to find parameters ψ that minimize a desired loss function Sophisticated optimization algorithms typically require gradient information regarding the forward process, fsim, with respect to the parameters ψ However, obtaining gradients from black-box simulators can often be prohibitively expensive or, in some cases, impossible Practitioners aim to solve related problems efficiently by starting optimization ab initio each time might be inefficient if the forward model is expensive to evaluate The proposed method leverages gradients from a fast differentiable surrogate fϕ: (ψ, x, z) → y trained to locally mimic the black-box simulator The core idea is to minimize the number of simulator calls required by learning an active learning policy πθ that decides whether retraining the surrogate is necessary or if a simulator call is needed instead This policy also learns how to sample new data for training the local surrogate model, drawing inspiration from literature on active learning The state st (at timestep t) is defined as the tuple (ψt, t, lt, σt), where ψt is the current parameter value, lt is the number of simulator calls already performed in the episode, and σt is some measure of uncertainty produced by the surrogate Policy and Surrogate Architecture The policy πθ consists of a separate Actor and Critic network, both ReLU MLPs with a single hidden layer of 256 neurons The Actor outputs either one or three values; the first value is passed through a sigmoid activation and treated as a Bernoulli random variable to decide whether to perform a simulator call or not If three values are output, the second and third values are used to sample the trust region size ϵ from a lognormal distribution The Critic outputs a value-function estimate Vθ(s), which is used in Proximal Policy Optimization (PPO) to compute advantage estimates Training and Optimization Details The policy πθ is trained episodically using Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE) advantages Episodes terminate under three conditions: A) when the optimization reaches a parameter for which E

L(y): is below the target value τ, B) when the maximum number of timesteps T is reached, or C) when the policy hits the available budget for simulator calls L The reward function incentivizes reducing simulator calls by assigning a reward of 0 if b = 0 and-1 if b = 1 The surrogate ensemble uses an uncertainty feature σ computed as the standard deviation over D samples to estimate uncertainties without the overhead of training a posterior network Performance and Results The policy-based approach showed the best performance across all scenarios, achieving a significant reduction in simulator calls, up to ∼ 90% compared to baselines For benchmark functions, policy-based methods achieved the highest performance in both fixed and parameterized x-distributions In real-world experiments involving wireless communication and high energy physics simulations, the policy models typically agree within uncertainties with local surrogate methods (L-GSO) The metrics used are the Average Minimum of the Objective function (AMO) for a specific budget of simulator calls and the Average Number of simulator Calls (ANC) required to terminate an episode Limitations and Future Work Gradient-based optimization may get stuck in local optima of the loss surface Ep(yψ,x) Investigating whether introducing a policy into the optimization can help avoid such local minima is an interesting direction of future research Hyperparameter tuning has mostly involved reducing training variance through tuning the number of episodes used for a PPO iteration, as well as setting learning rates and the KL-threshold More effort has been spent optimizing the policy or surrogate architectures to further improve performance While the advantage of learning to adapt the trust region size, πAL-E, combined with a warm-started surrogate, πG AL-E is only revealed in highly complex optimization problems Future works should conduct further investigations to assess the actual advantages that will result from learning the sampling strategy Implementation Details The policy πθ is composed of an Actor and a Critic, both ReLU MLPs with a single hidden layer of 256 neurons The surrogate consists of a ReLU MLP with two hidden layers of 256 neurons that takes as input (ψ, x, z) and outputs y A single “simulator call” consists of 1.5 · 104 function evaluations for Probabilistic Three Hump The policy is trained in an episodic manner by accumulating sequential optimization episodes and updating the policy using PPO The training involves accumulating 10 episodes for the Nonlinear Sub.

Tom: Hump and 5 for the high energy physics experiments, depending on the time required to complete one episode The evaluation involves running 32 evaluation episodes for benchmark functions and 20 and 5 evaluation episodes for the wireless and physics experiments, respectively The reward penalty is - (L − lt) − 1 when C occurs, ensuring that the sum-of-rewards for non-terminating episodes is-L - 1 Our policy-based approach showed the best performance across all scenarios, with a significant reduction in simulator calls up to ∼ 90% Our results suggest that local surrogate-based optimization of problems involving black-box forward processes benefits from the guidance of both simple policies and learned sampling strategies The method successfully minimizes the number of simulator calls required to solve optimization problems involving black-box simulators using (local) surrogates References

1: S.

Jane: Agostinelli et al.

Tom: GEANT4 – a simulation toolkit

2: Atul Agrawal and Phaedon-Stelios Koutsourelakis.

Paper summary: Tom: So we’ve been talking about how hard it is to optimize parameters when you're stuck with a black-box simulator, and this paper, "Simulating, Fast and Slow: Learning Policies for Black-Box Optimization," looks at using reinforcement learning to manage that trade-off.

Jane: Right. Basically, they introduce a way to learn a policy that guides training of a fast surrogate model so you don't have to run up the cost of the black-box simulator every single time you need an answer.

Lu: The core idea is learning an active learning policy πθ that decides if it's worth retraining the surrogate or just making another simulator call, and then using those gradients from a differentiable surrogate to optimize your actual simulation parameters with stochastic gradient descent.

Meng: What this means for me on the deployment side is that instead of wasting compute time blindly calling the expensive simulator, we use this policy to make smart choices about when to spend those simulation calls.

Tom: The authors show that after training this policy, you can amortize the cost for downstream optimization problems involving these black-box simulators by requiring up to ten times fewer expensive simulator calls compared to baseline methods.

Jane: It’s a significant reduction in the computational budget needed for your experiments, especially when dealing with models where the forward process is just too slow or too complex to differentiate directly.

Lu: The results show this policy-based approach achieved the best performance across various scenarios, including fixed and parameterized input distributions on benchmark functions.

Meng: I’m interested in how that translates to real traffic or physical simulations where we don't have clean benchmarks, because I know those setups can break things easily.

Tom: The paper suggests that local surrogate-based optimization benefits from guidance from both simple policies and learned sampling strategies, which seems like a solid way to approach these hard problems.

Jane: It really points toward using AI not just for the final answer, but for intelligently managing the entire search process itself when you can't see inside the black box.

Lu: The authors point out that gradient-based optimization can still get stuck in local optima of the loss surface, so this policy approach helps avoid those traps during optimization.

Meng: So it’s not a perfect solution yet because you still have to worry about getting stuck in bad parameter regions, but the way they manage the simulator calls is definitely a smart strategy.

Tom: This work shows that learning to adapt your sampling strategy is useful, and it gives us a concrete mechanism to minimize simulator calls for black-box simulators.

Jane: So next time you’re facing an optimization problem with a costly forward model, this policy framework offers a path toward making the search process much more efficient.

Lu: We need to keep looking into how the policy learns that trust region size adjustment because that's where some of the most complex problems show promise.

Conclusion: Tom: So we’ve been talking about how hard it is to optimize parameters when you're stuck with a black-box simulator, and this paper, "Simulating, Fast and Slow: Learning Policies for Black-Box Optimization," looks at using reinforcement learning to manage that trade-off.

Jane: Right. Basically, they introduce a way to learn a policy that guides training of a fast surrogate model so you don't have to run up the cost of the black-box simulator every single time you need an answer.

Lu: The core idea is learning an active learning policy πθ that decides if it's worth retraining the surrogate or just making another simulator call, and then using those gradients from a differentiable surrogate to optimize your actual simulation parameters with stochastic gradient descent.

Meng: What this means for me on the deployment side is that instead of wasting compute time blindly calling the expensive simulator, we use this policy to make smart choices about when to spend those simulation calls.

Tom: The authors show that after training this policy, you can amortize the cost for downstream optimization problems involving these black-box simulators by requiring up to ten times fewer expensive simulator calls compared to baseline methods.

Jane: It’s a significant reduction in the computational budget needed for your experiments, especially when dealing with models where the forward process is just too slow or too complex to differentiate directly.

Lu: The results show this policy-based approach achieved the best performance across various scenarios, including fixed and parameterized input distributions on benchmark functions.

Meng: I’m interested in how that translates to real traffic or physical simulations where we don't have clean benchmarks, because I know those setups can break things easily.

Tom: The paper suggests that local surrogate-based optimization benefits from guidance from both simple policies and learned sampling strategies, which seems like a solid way to approach these hard problems.

Jane: It really points toward using AI not just for the final answer, but for intelligently managing the entire search process itself when you can't see inside the black box.

Lu: The authors point out that gradient-based optimization can still get stuck in local optima of the loss surface Ep(yψ,x), so this policy approach helps avoid those traps during optimization.

Meng: So it’s not a perfect solution yet because you still have to worry about getting stuck in bad parameter regions, but the way they manage the simulator calls is definitely a smart strategy.

Tom: This work shows that learning to adapt your sampling strategy is useful, and it gives us a concrete mechanism to minimize simulator calls for black-box simulators.

Jane: So next time you’re facing an optimization problem with a costly forward model, this policy framework offers a path toward making the search process much more efficient.