Efficient Long-Horizon Learning for Learned Optimization

arXiv:2607.06772 · cs.LG · Submitted 2026-07-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Efficient Long-Horizon Learning for Learned Optimization".

Jane: The gist The Efficient Long-Horizon Learning (ELO) algorithm is a novel meta-training algorithm that incorporates a failure-aware resume buffer and decoupled progressive expert guidance to achieve efficient long-horizon learning with…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've been looking at how this paper, "Efficient Long-Horizon Learning for Learned Optimization," tries to solve the problem of training AI optimizers over really long sequences of problems.

Jane: Yeah, it’s basically about making that process more stable and efficient so we can actually get these models to learn better over much longer tasks <ref:2607.06772#pg5>.

Lu: It uses two main tricks, a failure-aware resume buffer and some kind of progressive expert guidance to keep things stable while they train <ref:2607.06772#pg1, #pg3>.

Meng: So the big question is, how does this actually help us when we're trying to make these optimization methods work for really complex, long-running AI tasks?

Lalam: It’s about shifting the compute away from the easy parts where we already know how to optimize and focusing it on those hard parts where things usually go wrong <ref:2607.06772#pg3>.

Tom: Right, so instead of just trying to brute-force every step, this buffer helps it jump over the difficult sections.

Jane: And that expert guidance part regularizes everything early on with a reliable optimizer before letting it take full control later <ref:2607.06772#pg2>.

Lu: They found that combining those two—the buffer and the progressive expert supervision—gives them stable training and strong performance on long unrolls of problems <ref:2607.06772#pg3>.

Meng: From an engineering view, this means we don't have to train these optimizers for the absolute longest runs just to see if they work. It cuts down on wasted time in unproductive areas <ref:2607.06772#pg1>.

Lalam: And this approach lets the learned optimizer generalize better when it hits scenarios it hasn't seen during training <ref:2607.06772#pg5>.

Tom: So, if this recipe works, what does that mean for the next generation of AI we build?

Jane: It means we might be able to train more robust and reliable optimization tools for things like large language models or complex scientific simulations <ref:2607.06772#pg5>.

Lu: The authors showed that ELO-Celo2 keeps performing well even when compared against established optimizers like AdamW and Muon on language modeling tasks <ref:2607.06772#pg1, #pg3>.

Meng: That comparison is pretty telling; it suggests this method isn't just a minor tweak, it’s a fundamentally better way to train the learning part of the AI <ref:2607.06772#pg1>.

Lalam: It points toward scaling learned optimization in ways we haven't seen before, which is really exciting for how we design these powerful models <ref:2607.06772#pg5>.

Conclusion: Tom: So we’ve been talking about how this paper, "Efficient Long-Horizon Learning for Learned Optimization," tackles making learned optimizers work over really long sequences of problems efficiently and stably.

Jane: Yeah, it’s not just about getting a better result on one task; it's about building a system that can handle those long runs without breaking <ref:2607.06772#pg5>.

Lu: The authors introduce this ELO algorithm which uses two main things: a failure-aware resume buffer and some way to guide the expert optimizer progressively during training.

Meng: So, what’s the core idea behind that? How does it actually manage all those long sequences?

Lalam: It shifts the computer's focus away from the easy parts where we already know how to optimize and pushes it into those hard regions where things usually fail <ref:2607.06772#pg3>.

Tom: Right, so instead of just trying to brute-force every single step in that long sequence, this buffer helps it jump over the difficult sections instead of getting stuck there <ref:2607.06772#pg1>.

Jane: And that expert guidance part helps regularize everything early on with a reliable optimizer before letting the learned optimizer take full control <ref:2607.06772#pg3>.

Lu: They found that combining those two mechanisms—the buffer and the progressive expert supervision—gives them stable training and strong performance on long unrolls of problems <ref:2607.06772#pg3>.

Lalam: And this approach lets the learned optimizer generalize better when it hits scenarios it hasn't seen during training <ref:2607.06772#pg5>.

Tom: So, if this recipe works, what does that mean for the next generation of AI we build?

Jane: It means we might be able to train more robust and reliable optimization tools for things like large language models or complex scientific simulations <ref:2607.06772#pg5>.

Lu: The paper shows ELO-Celo2 keeps performing well even when compared against established optimizers like AdamW and Muon on language modeling tasks <ref:2607.06772#pg1, #pg3>.

Meng: That comparison is telling; it suggests this method isn't just a small tweak, it’s a different way to train the learning part of the AI <ref:2607.06772#pg1>.

Lalam: It points toward scaling learned optimization in ways we haven't seen before, which is really exciting for how we design these powerful models <ref:2607.06772#pg5>.

Google DeepMind

cs.LG

Submitted: 2026-07-07

Updated: 2026-10-10

Comments: Meta-learning, learned optimization

Code: https://github.com/xiaol827/ELO

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist The Efficient Long-Horizon Learning (ELO) algorithm is a novel meta-training algorithm that incorporates a failure-aware resume buffer and decoupled progressive expert guidance to achieve

Key concepts

Failure-Aware Resume Buffer: This mechanism identifies challenging parts of the optimization process by tracking loss dynamics. It creates a difficulty score where high positive values indicate poor optimization regions. The buffer intelligently decides when to resume training, ensuring the next steps pass through these hard areas rather than skipping them, leading to unbiased meta-gradients.
Progressive Expert Supervision: This involves using a reliable expert optimizer during the initial stages of meta-training and gradually reducing this constraint as the learned optimizer improves. This progressive relaxation helps stabilize learning and ensures that the optimization process remains robust while adapting to complex, long sequences of training steps.
Decoupled Imitation Objective: Instead of using a single loss for imitation, ELO separates the supervision into direction and magnitude components. Direction supervision focuses on aligning the update direction with an expert's, while magnitude supervision handles the scale. Balancing these two separately provides complementary signals that result in better alignment and lower meta-evaluation loss.
Optimal Expert Choice: The best expert optimizer depends on the specific architecture of the learned optimizer. Adam is preferred for element-wise small fully connected layers, whereas Muon works best when optimizing matrix-based architectures like Celo2. Choosing the right expert is crucial for effective regularization and performance.

Terminology

Generalization — 2× in this paper · explained in 13 episodes
Meta-learning — 2× in this paper · explained in 3 episodes
Adam — 1× in this paper · explained in 2 episodes

Summary

The gist The Efficient Long-Horizon Learning (ELO) algorithm is a novel meta-training algorithm that incorporates a failure-aware resume buffer and decoupled progressive expert guidance to achieve efficient long-horizon learning with stability and improve downstream performance.

How it works

ELO introduces two main components: the failure-aware resume buffer and progressive expert supervision, which together give stable meta-training and strong long-horizon optimization performance.
The failure-aware resume buffer identifies difficult regions from loss dynamics by converting local signals into a cumulative difficulty score, where Values close to 0 indicate consistent optimization progress, while large positive values indicate regions of poor optimization.
The buffer ensures the next unroll will pass through the onset of the difficult region rather than resume after it, and meta-gradients resulting from resumed inner problems are unbiased.
Progressive expert supervision regularizes the LO with a reliable expert optimizer during early meta-training and progressively relaxes this constraint as the LO improves.
The decoupled imitation objective is defined as a weighted combination of direction supervision and magnitude supervision, where direction and scale provide complementary supervision signals and should be balanced separately.

Key Components and Design Choices

**- Failure-Aware Resume Buffer: This mechanism shifts compute from already well-covered early stages to later regimes where long-horizon failures emerge, by resuming from buffered states with probability PB. The optimal buffer probability PB is selected based on meta-evaluation loss and downstream validation loss. It is noted that smaller PB is safer at small meta-training scale, while larger PB becomes beneficial as meta-training becomes more stable and can support longer continuations. The buffer maintains the PES state, ensuring that PES gradients from the replay buffer are unbiased. 5. The paper demonstrates that combining both components yields stable meta-training and strong long-horizon optimization performance. 4. The optimal expert optimizer choice depends on the LO architecture; Adam is preferred for the element-wise small fc, while Muon works best as the expert for the matrix-based Celo2. 5. Decoupling direction and magnitude supervision through a weighted combination of direction loss and magnitude loss yields better direction alignment and lower meta-evaluation loss compared to direct MSE regression. 3. The optimal weighting for the imitation objective is found when assigning weight 0.7 to the direction term and 0.3 to the magnitude term. 4. The paper establishes that ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. 3. ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs across downstream tasks. 5. ELO-Celo2 achieves the best performance, with a small advantage over Muon on GPT-2 (124M) pretraining with 2.5B tokens. 3. The wall-clock training time for ELO-Celo2 is reported as 6h25m when meta-trained on a single H100 GPU. 4. ELO works best when the resume buffer and progressive expert supervision are used together, as the buffer improves long-horizon coverage while expert supervision keeps these harder unrolls stable. 5. The research shows that ELO-LOs meta-trained at a small scale are able to generalize to models with hundreds of millions of parameters, consistently improving over their corresponding base LOs. 4. The paper concludes that ELO is a promising recipe for scaling learned optimization. 5. The implementation of ELO can introduce additional overhead compared with standard meta-training recipes due to constraints of the JAX framework. 3. The paper evaluates ELO on realistic out-of-distribution language modeling tasks, showing that ELO-Celo2 continues to significantly outperform AdamW and performs on par with Muon. 4. The authors found that Imitating expert’s update direction is more important than matching its magnitude. 5. The paper suggests that the optimal expert is architecture dependent, with Adam preferred for small fc and Muon preferred for Celo2. 3. ELO-small fc and ELO-Celo2 outperform their corresponding learned-optimizer baselines on both element-wise and matrix-based architectures, respectively. 4. The empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs across downstream language modeling (GPT-2) and image classification (ViT/ResNet). 5. All ELO baselines require less than 7 H100 GPU-hours for meta-training. 3. The paper shows that increasing augmentation strength improves the validation performance of LOs, whereas AdamW tends to overfit under stronger augmentation. 4. The study confirms that ELO-Celo2 achieves the best validation accuracy under stronger augmentation, using 6 randomly selected operations from a pool of more than 20. ELO is a novel meta-training algorithm for learned optimization. 5. The paper states that ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. ELO is a novel meta-training algorithm for learned optimization. 5. The paper suggests that ELO is a promising recipe for scaling learned optimization. 3. The paper evaluates ELO on realistic out-of-distribution language modeling tasks, showing that ELO-Celo2 continues to significantly outperform AdamW and performs on par with Muon. 4. The study confirms that ELO works best when the resume buffer and progressive expert supervision are used together. 5. The paper concludes that ELO is a promising recipe for scaling learned optimization. 3. ELO-Celo2 outperforms well-tuned AdamW across all evaluated tasks and remains competitive with Muon on language modeling. 4. The study confirms that ELO-small fc and ELO-Celo2 outperform their corresponding learned-optimizer baselines. 5. The paper states that ELO is a promising recipe for scaling learned optimization. 3. ELO-Celo2 outperforms well-tuned AdamW across all evaluated tasks and remains competitive with Muon on language modeling. 4. The study confirms that ELO works best when the resume buffer and progressive expert supervision are used together. 5. The paper concludes that ELO is a promising recipe for scaling learned optimization. 3. ELO-Celo2 outperforms well-tuned AdamW across all evaluated tasks and remains competitive with Muon on language modeling. 4. The study confirms that ELO works best when the resume buffer and progressive expert supervision are used together.

Improvements for AI systems

The system can achieve efficient long-horizon learning by reallocating wasted meta-training compute to longer failure regimes, as described by ELO's mechanism: (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning.
The system gains stable meta-learning signals and improved generalization of learned optimizers by enforcing decoupled progressive expert supervision, which anchors early parameter updates: enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs.
The system can outperform well-tuned hand-designed AdamW optimizers across various tasks, as demonstrated by ELO-Celo2 consistently outperforming AdamW across all evaluated tasks.
The system can match Muon's performance on language modeling tasks, specifically for GPT-2 (124M/350M), indicating effectiveness on different scales: ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling.
The system can be trained to generalize effectively to models with hundreds of millions of parameters by leveraging ELO-LOs meta-trained at a small scale, as shown by the takeaway: ELO-LOs meta-trained at a small scale are able to generalize to models with hundreds of millions of parameters.
The system can be trained using more aggressive data augmentation strategies while maintaining strong performance, as suggested by Figure 9: In our setting, learned optimizers achieve their best validation accuracy under stronger augmentation.