Dual-Evolving Reinforcement Learning for Automated Reward Design

arXiv:2512.13399 · cs.AI, cs.CL · Submitted 2025-12-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Dual-Evolving Reinforcement Learning for Automated Reward Design".

Jane: The gist: Differentiable Evolutionary Reinforcement Learning (DERL) introduces a bi-level framework for automatically discovering optimal reward structures by framing meta-optimization as an RL problem,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we’re looking at this paper called "Dual-Evolving Reinforcement Learning for Automated Reward Design." Basically, they’re tackling the hard problem of crafting good reward signals for complex reasoning tasks in reinforcement learning.

Jane: It sounds like they are trying to move past just having someone manually design a reward function because it doesn't scale well when things get complicated. What is the main idea behind this Dual-Evolving Reinforcement Learning framework?

Tom: The core thesis is that they introduce a bi-level framework for automatically discovering optimal reward structures. It’s not just one search, it’s two loops working together.

Jane: And what exactly does this bi-level system do? How does it claim to solve the reward design problem in a differentiable way?

Tom: They use an outer-loop MetaOptimizer that generates a parameterized Meta-Reward by composing structured atomic primitives. This generated Meta-Reward then guides an inner-loop policy model evolution through policy gradients derived from validation performance.

Lu: What I find really interesting is how they frame the reward function construction by mathematically composing atomic primitives, which are essentially modular blocks like format checkers or partial goal verifiers to define the reward Rphi(o, C) = Func (g1(o, C),..., gk(o, C); phi) >

Meng: So it’s not just randomly guessing a reward function; it’s building it out of these specific components. What kind of primitives are they using for different domains?

Jane: Well, they design these primitives based on the task. For things like robotic or scientific agents, they use temporal trajectory phases like if the task is successful or averages across different parts of the movement >

Tom: For mathematical reasoning, their primitives include binary outcome rewards for success, format rewards for syntax, step-by-step rewards for intermediate progress, and soft outcome rewards >

Lalam: I see how that makes sense. It gives the MetaOptimizer a structured search space instead of trying to find arbitrary code generation which is way too broad >

Meng: From an engineering standpoint, that sounds like it focuses the optimization effort onto structural reasoning rather than just raw text parsing, which is a big win for tractability >

Jane: And how does this dual process actually work in practice? How does the outer loop learn from the inner loop's results?

Tom: The MetaOptimizer updates its parameter Phi by using policy gradients derived from that validation performance signal, which is why they call it differentiable evolution >

Lu: That’s a key mechanism. It transforms the discrete evolutionary search of reward functions into a continuous optimization over the meta-policy parameters Psi >

Jane: So instead of just trying random reward mutations, the system uses feedback to guide its evolution toward better structures. What kind of performance do they see when they test this on different reasoning tasks?

Tom: They validated it on embodied systems like ALFWorld, scientific simulation like ScienceWorld, and mathematical reasoning benchmarks such as GSM8K and MATH >

Meng: And what are the numbers there? Are we talking about state-of-the-art results across all those domains?

Jane: Yes, they achieved state-of-the-art performance on ALFWorld and ScienceWorld, showing superior robustness in out of distribution problems >

Tom: In math reasoning specifically, they outperformed all baseline reward functions and even the strong outcome reward on MATH by a margin of sixty point two percent to fifty-eight point eight percent >

Lu: The dynamics show a consistent, monotonic upward trend in average performance as the outer loop progresses, which suggests the MetaOptimizer isn't just overfitting to specific instances during training >

Meng: That’s good for production. If it's not overfitting, it means the learned reward structure has genuine generalization ability across different tasks >

Jane: It also showed that stable structures increasingly dominate while unstable structures decline, which suggests the MetaOptimizer is acting like an evolutionary filter for mathematical robustness by learning to prioritize stability >

Tom: The paper emphasizes that this process allows the Meta-Optimizer to effectively capture the meta-gradient of task success, enabling it to discover useful reward signals >

Jane: So what does this mean for us listening right now? If we just listen to how you describe it, what is the actual impact on how we build these agents?

Tom: It means we can shift from designing rewards manually or using black-box evolution to having a system that autonomously discovers reward structures by learning from execution feedback >

Lu: The implication is that this approach eliminates the need for expensive human labor or manual annotation to design the reward function itself >

Meng: For deployment, it sounds like it’s much more scalable because it’s differentiable and structured, though I'd still want to see how much wall-clock time this actually takes compared to traditional methods >

Jane: That brings us nicely into the conclusion of "Dual-Evolving Reinforcement Learning for Automated Reward Design." The authors sit on the idea that this bi-level framework is a viable way to achieve automated reward discovery by framing meta-optimization as an RL problem >

Tom: They're showing how policy gradients from inner loop validation performance can be used to update the MetaOptimizer parameters, establishing a differentiable, closed-loop optimization and eliminating heuristic design or expensive annotation >

Jane: The authors conclude that DERL can be easily adapted to different downstream tasks simply by defining a set of atomic primitives >

Conclusion: Tom: So we’ve been looking at Dual-Evolving Reinforcement Learning for Automated Reward Design, and this paper is about taking reward design out of the hands of a human and putting it into an evolving AI system.

Jane: It boils down to this idea where you have two levels of optimization working together, an outer loop setting up the goal, and an inner loop actually training the agent using that goal.

Tom: Exactly. The authors are showing how you can make the meta-optimization—that outer loop—learn by looking at how well the inner policy performs during validation trials.

Jane: It’s about letting past trial feedback shape what a good reward structure looks like, essentially creating an empirical meta-gradient of task success for the reward itself.

Tom: The paper validates this across very different reasoning tasks, from embodied worlds to complex math problems like GSM8K and MATH.

Jane: And on those benchmarks, they’re actually beating existing methods and even strong baseline reward functions in some cases, like on the mathematical outcome reward where adding primitives actually hurts performance.

Tom: It suggests that this bi-level structure is a way to discover useful reward signals automatically instead of having to design them manually or through fixed search methods.

Jane: If we can do this, it means AI systems could be designed in a way that they figure out what's important for success on their own without needing someone to hand them the perfect instruction set beforehand.

Tom: It shifts the focus from writing code for rewards to designing the structure of those atomic primitives—those building blocks like format checkers or goal verifiers.

Jane: But there’s still a lot to unpack about how scalable this is and what kind of computational cost it actually incurs when you run these dual loops at scale.

Tom: Yeah, that’s the next big question. We need to figure out if this is just a neat trick for small tasks or if it can handle the complexity we see in real-world applications.

Sitao Cheng, Tianle Li, Xuhan Huang, Xunjian Yin, Difan Zou

University of Waterloo · The University of Hong Kong, Shenzhen · The Chinese University of Hong Kong, Shenzhen

cs.AI, cs.CL

Submitted: 2025-12-15

Updated: 2026-10-13

Comments: Work in Progress. We release our code and model at https://github.com/sitaocheng/DERL

Code: https://github.com/sitaocheng/DERL

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist: Differentiable Evolutionary Reinforcement Learning (DERL) introduces a bi-level framework for automatically discovering optimal reward structures by framing meta-optimization as an RL

Key concepts

Dual-Evolving Reinforcement Learning (DERL): A bi-level framework where an outer loop proposes a reward structure and an inner policy trains using it. The proposer learns by optimizing its proposal based on how well the inner policy performs during validation, creating a continuous optimization process for finding good rewards.
Reward Parameterization: Instead of generating arbitrary code, DERL constructs rewards by mathematically composing atomic primitives—modular blocks like format checkers or goal verifiers. This method allows the reward structure to be expressive and continuous, focusing the search on structural reasoning rather than raw text generation.
Meta-Gradient: This refers to the empirical gradient of task success that DERL captures. The outer loop optimizer updates its proposal by following this meta-gradient, meaning it learns directly from validation performance to shape reward proposals toward structures that lead to better overall task outcomes.

Terminology

Reinforcement Learning — 3× in this paper · explained in 4 episodes
Reward Function — 2× in this paper · explained in 4 episodes
Alignment — 1× in this paper
Benchmark — 1× in this paper · explained in 2 episodes
Differentiable Evolutionary Reinforcement Learning — 1× in this paper
Generalization — 1× in this paper · explained in 13 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes

Summary

The gist: Differentiable Evolutionary Reinforcement Learning (DERL) introduces a bi-level framework for automatically discovering optimal reward structures by framing meta-optimization as an RL problem, which allows it to capture the meta-gradient of task success through policy gradients derived from inner-loop validation performance.

How it works

The DERL framework operates as a bi-level optimization process where an inner-loop policy evolves based on a generated Meta-Reward, and an outer-loop MetaOptimizer evolves by learning from the inner policy’s validation performance Figure 2, Bi-level evolutionary training framework for DERL>. The objective of the inner loop is to maximize the expected parameterized reward: J inner phi(theta) = Ex in D, tau pi theta(·x) [Rphi(tau)], (1)Inner-loop (Policy Model Evolution) Given a configuration phi from the outer-loop, the inner-loop policy theta is optimized to maximize the expected parameterized reward: J inner phi(theta) = Ex in D, tau pi theta(·x) [Rphi(tau)], (1)where D represents the training dataset and tau denotes the trajectories sampled from the policy model>. The outer-loop objective is to train the Meta-Optimizer psi to generate configurations phi that yield high-performing inner policies, formulated as J outer(psi) = Ephi pi psi(·ins) [Perf(theta∗)], s.t. theta∗ = arg max theta J inner phi(theta). (2)This bi-level training framework transforms the discrete evolutionary search of reward functions into a continuous, differentiable optimization over the meta-policy parameters psi>.

Reward Parameterization and Structure

A key challenge addressed is designing a tractable structure for the generated configuration phi to parameterize the reward function for the inner loop Reward Parameterization A key challenge in Meta-Optimizer evolution lies in designing a tractable structure of the generated configuration to parameterize the reward function for the inner loop>. Instead of arbitrary code generation, DERL constructs rewards by mathematically composing atomic primitives—modular, executable blocks like format checkers or partial goal verifiers—to define Rphi(o, C) = Func (g1(o, C),..., gk(o, C); phi) (3)This parameterization ensures an expressive, continuous search space while focusing the Meta-Optimizer on structural reasoning rather than raw text parsing>. For robotic and scientific agents, primitives are designed based on temporal trajectory phases like g1 = 1 if the task is ultimately successful, otherwise 0 g2 = The average step score across the first third of the trajectory, and g3 = The average step score across the second third of the trajectory, and g4 = The average step score across the final third of the trajectory>. For mathematical reasoning, primitives include binary outcome reward (g1), format reward (g2), step-by-step reward (g3), and soft outcome reward (g4)A Detailed Primitive Design with Concrete Examples>.

Optimization Dynamics and Robustness

The optimization dynamics reveal that the Meta-Optimizer drives a gradient-guided evolution rather than a stochastic random search, validating that DERL captures the intrinsic meta-gradient of the task The Meta-Optimizer drives a gradient-guided evolution rather than a stochastic random search, validating that DERL captures the intrinsic meta-gradient of the task>. The training dynamics show a consistent, monotonic upward trend in average performance of both validation and testing as the outer-loop progresses, which provides empirical verification that the Meta-Optimizer is not overfitting to specific instances in outer-loop training The concurrent rise in validation and testing performance provides empirical verification that the Meta-Optimizer is not overfitting to the specific instances in outer-loop training>. Furthermore, qualitative analysis shows that Stable Structures increasingly dominate while Unstable Structures decline, suggesting the Meta-Optimizer acts as an evolutionary filter for mathematical robustness by implicitly learning to prioritize numerical stability and boundedness Figure 4, Evolution dynamics of reward structures on ALFWorld>.

Empirical Results and Generalization

DERL achieves state-of-the-art performance across diverse reasoning domains, including Robotic Agent (ALFWorld), Scientific Simulation (ScienceWorld), and Mathematical Reasoning (GSM8k, MATH) Table 1a, Performance of DERL on three distinct domains>. Notably, DERL achieves state-of-the-art performance on ALFWorld and ScienceWorld, showing superior robustness in Out-of-Distribution (OOD) problems Take-away 1, The Meta-Reward captures the intrinsic structure of the task, enabling generalization to unseen scenarios where heuristic combinations fail>. In mathematical reasoning, DERL outperforms all baseline reward functions and even the strong outcome reward on MATH (60.2% vs. 58.8%) Table 1b, Accuracy on GSM8K and MATH>.

Efficiency and Future Directions

The computational cost is summarized as CDERL ≈ n × Eouter × Cinner H.2 Resource Estimation, where Cinner denotes the computational cost required to train one standard inner-loop policy model>. DERL-pop offers a more efficient alternative, consuming way less wall-clock time while still benefiting from population-based exploration H.1 Computational Breakdown, where the primary bottleneck lies in the innerloop, and parallelization is used to mitigate latency The framework's core innovation is the preservation and utilization of meta-gradient information, allowing the optimizer to explicitly learn from validation performance I.2 Meta-Gradient Propagation Compared with Previous Evolution, where updating psi is driven by the gradient of the validation performance, denoted as ∇J outer(ψt) (7)This enables the Meta-Optimizer to iteratively improve the reward structure phi driven by direct performance feedback>. The paper concludes that DERL can be easily adapted to different downstream tasks by simply defining a set of atomic primitives 6 Conclusions and Limitations.

Improvements for AI systems

Boldly automate reward discovery by replacing derivative-free search heuristics that treat the reward function as a black box with Differentiable Evolutionary Reinforcement Learning (DERL). This allows for the progressive learning of a 'meta-gradient' for task success, providing the system with dense, actionable feedback.
Enable autonomous discovery of optimal objectives by framing meta-optimization as an RL problem, allowing the Meta-Optimizer to capture an estimated 'meta-gradient' of task success and update its own parameters via policy gradients derived from inner-loop validation performance.
Enhance out-of-distribution generalization by ensuring the system learns task logic rather than relying on brittle heuristics, as DERL results show substantially outperforms non-differentiable baselines—especially in out-of-distribution generalization, and the Meta-Optimizer captures the intrinsic causal structure of tasks.
Achieve mathematically robust reward structures by implicitly learning stability, as the Meta-Optimizer learn[s] to prioritize numerical stability and boundedness through evolution dynamics, leading to structures that are increasingly dominated by Stable Structure[s] over time.
Improve reasoning fidelity in mathematical domains by discovering non-trivial reward compositions that avoid reward hacking, allowing DERL to outperform baselines even when ground-truth signals are strong, as shown by the result where DERL discovers a reward structure with a score of 62.9% vs. 58.8% on MATH.
Establish self-improving agent alignment by creating a closed-loop optimization and eliminating heuristic design or expensive annotation process, enabling the system to refine its own objectives autonomously through the feedback loop: the Meta-Optimizer captures an estimated 'meta-gradient' of task success, allowing the model to update its own parameters to synthesize progressively denser and more actionable feedback.
Increase computational efficiency by employing DERL-pop., which reduces cost by optimizing the outer-loop more frequently while maintaining performance, as it is noted that DERL-pop. consumes significantly less computation by optimizing the outer-loop more frequently.