Test-Time Learning with an Evolving Library
summary
The gist
We introduce EVOLIB, a test-time learning framework that enables large language models to accumulate, reuse…
In short
EVOLIB is a test-time learning framework that allows large language models to learn and improve knowledge during inference without needing parameter updates or external supervision. It builds a shared library of modular skills and reflective insights extracted from the model's own reasoning, using information gain to prioritize useful abstractions for continual self-improvement.
Key concepts
- Knowledge Abstractions
- These are structured pieces of knowledge extracted from the LLM's solutions during testing. They include reusable procedures (modular skills) and common mistakes/fixes (reflective insights), forming a library that the model can reuse and evolve over time.
- Information Gain (IG)
- IG measures how much better the model can solve a specific problem when a particular knowledge abstraction is present compared to using an average baseline. It quantifies the immediate usefulness of an abstraction for solving current tasks.
- Future Information Gain (Future IG)
- This metric assesses the potential value of an abstraction by measuring how much better the model can solve future problems if that specific abstraction was learned previously. It guides the evolution process by prioritizing abstractions with high potential for future utility.
Terminology used across episodes
This episode discusses
- Test-Time Learning with an Evolving Library · Paper Radio
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Test-time Recursive Thinking: Self-Improvement without External Feedback
- Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models
- Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
- Learning to Discover at Test Time
- Learning to (Learn at Test Time): RNNs with Expressive Hidden States
- GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
- On the Measure of Intelligence
- WALT: Web Agents that Learn Tools
- SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization
- AutoRefine: Compiling Trajectories into Validated Typed Agent Artifacts
- ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memory
- EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle
- Evolving Programmatic Skill Networks
- RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems
- Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- ThetaEvolve: Test-time Learning on Open Problems
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
The paper
Test-Time Learning with an Evolving Library · Read on arXiv
Weijia Xu, Alessandro Sordoni, Chandan Singh, Zelalem Gero, Michel Galley, Xingdi Yuan, Jianfeng Gao
Microsoft Research
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Test-Time Learning with an Evolving Library".
Jane: Here's the gist of the paper. We introduce EVOLIB, a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge across problem instances without parameter updates or external supervision<ref:2605.14477#pg2>.
Tom: This approach maintains a shared library of knowledge abstractions, including modular skills and reflective insights, automatically extracted from the model’s own inference trajectories<ref:2605.14477#pg3>.
Jane: Pruning redundant reasoning tokens The key idea is to maintain a structured, evolving collection of knowledge abstractions<ref:2605.14477#pg4>.
Tom: The system extracts two types of abstractions from model-generated solutions: (1) modular skills, which capture reusable procedures, and (2) reflective insights, which encode common errors and corrective strategies<ref:2605.14477#pg4>.
Paper summary: Tom: So this paper introduces something called EVOLIB, a test-time learning framework for large language models. The main idea is letting these models accumulate and reuse knowledge across different problems without ever updating their parameters or needing any outside supervision <ref:2605.14477#pg2>.
Jane: It sounds like it’s trying to solve the problem of forgetting or not being able to learn new things effectively when you just let a model run at test time, right? What exactly is this framework claiming it can do <ref:2605.14477#pg2>?
Tom: It claims they maintain a shared library of knowledge abstractions, which are like modular skills and reflective insights that get automatically pulled out from the model's own test runs <ref:2605.14477#pg3>.
Lu: The core is distilling raw experiences into these reusable units instead of just memorizing all the data points, which they call abstractions <ref:2605.14477#pg2>. These are supposed to be flexible knowledge units that can be put together for different tasks <ref:2605.14477#pg2>.
Meng: But what stops this library from just getting messy? When you keep adding new things, how do you handle the noise or redundant stuff that starts piling up in that memory <ref:2605.14477#pg19>?
Jane: That’s a good question, because they introduce a principled weighting and consolidation mechanism to handle that noise and keep the library clean <ref:2605.14477#pg3>. They use this mechanism to optimize for both immediate usefulness and long-term value <ref:2605.14477#pg3>.
Tom: They use two types of abstractions, modular skills for procedures and reflective insights which capture common errors or how to fix them <ref:2605.14477#pg2>. And they govern this whole thing with two main ideas: Abstraction and Evolution <ref:2605.14477#pg4>.
Lu: They build the training signal for this library using Information Gain, or IG, and Future Information Gain, or Future IG <ref:2605.14477#pg3>. These scores capture both how useful an abstraction is right now and its potential to generate more valuable abstractions later <ref:2605.14477#pg3>.
Meng: So how does that credit propagation work? How does the system decide which old skill gets a boost and which new one gets attention <ref:2605.14477#pg3>?
Tom: The core mechanism is credit propagation along the evolution chains of these abstractions <ref:2605.14477#pg3>. IG measures how much better the model can solve a task when you have that abstraction compared to just using some average baseline <ref:2605.14477#pg3>.
Jane: And Future IG quantifies how much better the model can solve a task if you use an older abstraction as a stepping stone for the new one, compared to that baseline score <ref:2605.14477#pg3>? That sounds like it’s helping things build on themselves <ref:2605.14477#pg3>.
Tom: The weight of each abstraction in the library is computed based on its IG and Future IG scores <ref:2605.14477#pg3>. This weighting mechanism is what drives the evolution, you know <ref:2605.14477#pg3>
Lu: They mention that abstraction consolidation leads to substantial gain and helps control memory growth while also encouraging the emergence of abstractions that generalize beyond specific problems <ref:2605.14477#pg9>. That’s a key part of controlling the library size <ref:2605.14477#pg10>.
Meng: If we're talking about what this means for deployment, how much memory overhead are we looking at when you're running this on real traffic <ref:2605.14477#pg9>?
Tom: They show that EVOLIB is more token-efficient than both test-time scaling and learning methods without any external ground truth feedback <ref:2605.14477#pg9>. It also supports continual learning that doesn't depend on how you order the tasks, unlike some older methods based on linear knowledge updates <ref:2605.14477#pg9>.
Jane: So across the different areas they tested, like mathematical reasoning or code generation, EVOLIB shows strong average performance compared to existing approaches <ref:2605.14477#pg9>. It even outperforms Dynamic Cheatsheet by about five to ten percent on three of those benchmarks <ref:2605.14477#pg9>.
Lu: The paper notes that the top high-Future Information Gain abstractions are still concrete and only apply to a subset of tasks, which shows their weighting mechanism is prioritizing useful stuff <ref:2605.14477#pg21>. That suggests the system isn't just grabbing everything it sees <ref:2605.14477#pg3>.
Meng: So, if we look at the limitations they mention, what's the biggest hurdle for deploying this in a real production environment <ref:2605.14477#pg11>?
Tom: The paper says it works best for tasks where verifying a solution is easier than actually generating one <ref:2605.14477#pg11>. They also suggest that a promising future direction would be enabling the LLMs to meta-learn the design of this library itself <ref:2605.14477#pg10>.
Jane: It sounds like they aren't claiming a magic fix for every single problem, but they are proposing a way for models to self-improve during deployment without needing constant retraining or supervision signals <ref:2605.14477#pg10>. That could be pretty significant for how we deploy AI assistants in the real world <ref:2605.14477#pg10>.
Lu: I think the idea of meta-learning the library design is ambitious, but it’s a direction they’re pointing toward <ref:2605.14477#pg10>. It suggests a path where the AI agent starts learning how to learn more efficiently <ref:2605.14477#pg10>.
Meng: From an engineer's view, the challenge is making sure that consolidation doesn't accidentally discard a skill that might be useful later for something completely different <ref:2605.14477#pg9>. That trade-off between generalization and specificity seems like the main tuning knob <ref:2605.14477#pg10>.
Tom: Exactly, it’s balancing that consolidation against keeping the library flexible enough to handle new things without getting stuck in a loop <ref:2605.14477#pg9>. So, we see a framework designed for continuous, unsupervised self-improvement during use <ref:2605.14477#pg10>. That’s the essence of this Test-Time Learning with an Evolving Library paper <ref:two thousand six hundred five point one four four seven seven#pg1.
Conclusion: Tom: So we’ve been looking at Test-Time Learning with an Evolving Library, and we gotta talk about what that title actually means for us right now <ref:two thousand six hundred five point one four four seven seven#pg1.
Jane: It basically means taking a model and letting it learn on the fly during use, but instead of just updating its settings, it’s building this evolving library of knowledge <ref:2605.14477#pg2>.
Lu: The authors are showing how they extract modular skills and reflective insights directly from the model's own work as it solves problems <ref:two thousand six hundred five point one four four seven seven#pg3.
Meng: From my side, it’s interesting because there’s no parameter updating involved, so we don't have that huge retraining cost or the risk of catastrophic forgetting when we just deploy this thing <ref:two thousand six hundred five point one four four seven seven#pg10.
Tom: Yeah, that token efficiency they showed is what really got my attention; it’s way better than some of the other test-time methods because you aren't spending all your time on massive scaling <ref:two thousand six hundred five point one four four seven seven#pg9.
Jane: And the core mechanism for how it learns—this information gain and future information gain—it’s a clever way to decide what knowledge is worth keeping and how to connect it <ref:two thousand six hundred five point one four four seven seven#pg3.
Lu: They're using that credit propagation along evolution chains, which means older skills get boosted if they help create newer ones, which is how the library actually grows organically <ref:two thousand six hundred five point one four four seven seven#pg3.
Meng: But the paper does flag a limitation where it works best when checking a solution is easier than actually creating one <ref:two thousand six hundred five point one four four seven seven#pg11. That’s a real constraint for production where we want models to generate things quickly <ref:two thousand six hundred five point one four four seven seven#pg9.
Tom: So the big picture here is that this isn't just about making a model smarter in a static way, it’s about giving it an adaptive memory system that grows while it works <ref:two thousand six hundred five point one four four seven seven#pg10.
Jane: And the authors are even pointing toward meta-learning the design of that library itself as a future direction for how AI agents improve things <ref:two thousand six hundred five point one four four seven seven#pg10.
Lu: It’s a promising path if we can figure out how to make that meta-learning happen without needing huge amounts of new training data <ref:two thousand six hundred five point one four four seven seven#pg3.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation