Test-Time Learning with an Evolving Library

arXiv:2605.14477 · cs.LG · Submitted 2026-05-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Test-Time Learning with an Evolving Library".

Jane: Here's the gist of the paper. We introduce EVOLIB, a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge across problem instances without parameter updates or external supervision<ref:2605.14477#pg2>.

Tom: This approach maintains a shared library of knowledge abstractions, including modular skills and reflective insights, automatically extracted from the model’s own inference trajectories<ref:2605.14477#pg3>.

Jane: Pruning redundant reasoning tokens The key idea is to maintain a structured, evolving collection of knowledge abstractions<ref:2605.14477#pg4>.

Tom: The system extracts two types of abstractions from model-generated solutions: (1) modular skills, which capture reusable procedures, and (2) reflective insights, which encode common errors and corrective strategies<ref:2605.14477#pg4>.

Paper summary: Tom: So this paper introduces something called EVOLIB, a test-time learning framework for large language models. The main idea is letting these models accumulate and reuse knowledge across different problems without ever updating their parameters or needing any outside supervision <ref:2605.14477#pg2>.

Jane: It sounds like it’s trying to solve the problem of forgetting or not being able to learn new things effectively when you just let a model run at test time, right? What exactly is this framework claiming it can do <ref:2605.14477#pg2>?

Tom: It claims they maintain a shared library of knowledge abstractions, which are like modular skills and reflective insights that get automatically pulled out from the model's own test runs <ref:2605.14477#pg3>.

Lu: The core is distilling raw experiences into these reusable units instead of just memorizing all the data points, which they call abstractions <ref:2605.14477#pg2>. These are supposed to be flexible knowledge units that can be put together for different tasks <ref:2605.14477#pg2>.

Meng: But what stops this library from just getting messy? When you keep adding new things, how do you handle the noise or redundant stuff that starts piling up in that memory <ref:2605.14477#pg19>?

Jane: That’s a good question, because they introduce a principled weighting and consolidation mechanism to handle that noise and keep the library clean <ref:2605.14477#pg3>. They use this mechanism to optimize for both immediate usefulness and long-term value <ref:2605.14477#pg3>.

Tom: They use two types of abstractions, modular skills for procedures and reflective insights which capture common errors or how to fix them <ref:2605.14477#pg2>. And they govern this whole thing with two main ideas: Abstraction and Evolution <ref:2605.14477#pg4>.

Lu: They build the training signal for this library using Information Gain, or IG, and Future Information Gain, or Future IG <ref:2605.14477#pg3>. These scores capture both how useful an abstraction is right now and its potential to generate more valuable abstractions later <ref:2605.14477#pg3>.

Meng: So how does that credit propagation work? How does the system decide which old skill gets a boost and which new one gets attention <ref:2605.14477#pg3>?

Tom: The core mechanism is credit propagation along the evolution chains of these abstractions <ref:2605.14477#pg3>. IG measures how much better the model can solve a task when you have that abstraction compared to just using some average baseline <ref:2605.14477#pg3>.

Jane: And Future IG quantifies how much better the model can solve a task if you use an older abstraction as a stepping stone for the new one, compared to that baseline score <ref:2605.14477#pg3>? That sounds like it’s helping things build on themselves <ref:2605.14477#pg3>.

Tom: The weight of each abstraction in the library is computed based on its IG and Future IG scores <ref:2605.14477#pg3>. This weighting mechanism is what drives the evolution, you know <ref:2605.14477#pg3>

Lu: They mention that abstraction consolidation leads to substantial gain and helps control memory growth while also encouraging the emergence of abstractions that generalize beyond specific problems <ref:2605.14477#pg9>. That’s a key part of controlling the library size <ref:2605.14477#pg10>.

Meng: If we're talking about what this means for deployment, how much memory overhead are we looking at when you're running this on real traffic <ref:2605.14477#pg9>?

Tom: They show that EVOLIB is more token-efficient than both test-time scaling and learning methods without any external ground truth feedback <ref:2605.14477#pg9>. It also supports continual learning that doesn't depend on how you order the tasks, unlike some older methods based on linear knowledge updates <ref:2605.14477#pg9>.

Jane: So across the different areas they tested, like mathematical reasoning or code generation, EVOLIB shows strong average performance compared to existing approaches <ref:2605.14477#pg9>. It even outperforms Dynamic Cheatsheet by about five to ten percent on three of those benchmarks <ref:2605.14477#pg9>.

Lu: The paper notes that the top high-Future Information Gain abstractions are still concrete and only apply to a subset of tasks, which shows their weighting mechanism is prioritizing useful stuff <ref:2605.14477#pg21>. That suggests the system isn't just grabbing everything it sees <ref:2605.14477#pg3>.

Meng: So, if we look at the limitations they mention, what's the biggest hurdle for deploying this in a real production environment <ref:2605.14477#pg11>?

Tom: The paper says it works best for tasks where verifying a solution is easier than actually generating one <ref:2605.14477#pg11>. They also suggest that a promising future direction would be enabling the LLMs to meta-learn the design of this library itself <ref:2605.14477#pg10>.

Jane: It sounds like they aren't claiming a magic fix for every single problem, but they are proposing a way for models to self-improve during deployment without needing constant retraining or supervision signals <ref:2605.14477#pg10>. That could be pretty significant for how we deploy AI assistants in the real world <ref:2605.14477#pg10>.

Lu: I think the idea of meta-learning the library design is ambitious, but it’s a direction they’re pointing toward <ref:2605.14477#pg10>. It suggests a path where the AI agent starts learning how to learn more efficiently <ref:2605.14477#pg10>.

Meng: From an engineer's view, the challenge is making sure that consolidation doesn't accidentally discard a skill that might be useful later for something completely different <ref:2605.14477#pg9>. That trade-off between generalization and specificity seems like the main tuning knob <ref:2605.14477#pg10>.

Tom: Exactly, it’s balancing that consolidation against keeping the library flexible enough to handle new things without getting stuck in a loop <ref:2605.14477#pg9>. So, we see a framework designed for continuous, unsupervised self-improvement during use <ref:2605.14477#pg10>. That’s the essence of this Test-Time Learning with an Evolving Library paper <ref:two thousand six hundred five point one four four seven seven#pg1.

Conclusion: Tom: So we’ve been looking at Test-Time Learning with an Evolving Library, and we gotta talk about what that title actually means for us right now <ref:two thousand six hundred five point one four four seven seven#pg1.

Jane: It basically means taking a model and letting it learn on the fly during use, but instead of just updating its settings, it’s building this evolving library of knowledge <ref:2605.14477#pg2>.

Lu: The authors are showing how they extract modular skills and reflective insights directly from the model's own work as it solves problems <ref:two thousand six hundred five point one four four seven seven#pg3.

Meng: From my side, it’s interesting because there’s no parameter updating involved, so we don't have that huge retraining cost or the risk of catastrophic forgetting when we just deploy this thing <ref:two thousand six hundred five point one four four seven seven#pg10.

Tom: Yeah, that token efficiency they showed is what really got my attention; it’s way better than some of the other test-time methods because you aren't spending all your time on massive scaling <ref:two thousand six hundred five point one four four seven seven#pg9.

Jane: And the core mechanism for how it learns—this information gain and future information gain—it’s a clever way to decide what knowledge is worth keeping and how to connect it <ref:two thousand six hundred five point one four four seven seven#pg3.

Lu: They're using that credit propagation along evolution chains, which means older skills get boosted if they help create newer ones, which is how the library actually grows organically <ref:two thousand six hundred five point one four four seven seven#pg3.

Meng: But the paper does flag a limitation where it works best when checking a solution is easier than actually creating one <ref:two thousand six hundred five point one four four seven seven#pg11. That’s a real constraint for production where we want models to generate things quickly <ref:two thousand six hundred five point one four four seven seven#pg9.

Tom: So the big picture here is that this isn't just about making a model smarter in a static way, it’s about giving it an adaptive memory system that grows while it works <ref:two thousand six hundred five point one four four seven seven#pg10.

Jane: And the authors are even pointing toward meta-learning the design of that library itself as a future direction for how AI agents improve things <ref:two thousand six hundred five point one four four seven seven#pg10.

Lu: It’s a promising path if we can figure out how to make that meta-learning happen without needing huge amounts of new training data <ref:two thousand six hundred five point one four four seven seven#pg3.

Weijia Xu, Alessandro Sordoni, Chandan Singh, Zelalem Gero, Michel Galley, Xingdi Yuan, Jianfeng Gao

Microsoft Research

cs.LG

Submitted: 2026-05-14

Updated: 2026-11-06

Comments: NeurIPS 2026

Code: https://github.com/microsoft/EvoLib

Project page: https://hkust-nlp.github.io/agentboard/9https://openai.com/index/o3-o4-mini-system-card/14

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: We introduce EVOLIB, a test-time learning framework that enables large language models to accumulate, reuse…

Key concepts

Knowledge Abstractions
These are structured pieces of knowledge extracted from the LLM's solutions during testing. They include reusable procedures (modular skills) and common mistakes/fixes (reflective insights), forming a library that the model can reuse and evolve over time.
Information Gain (IG)
IG measures how much better the model can solve a specific problem when a particular knowledge abstraction is present compared to using an average baseline. It quantifies the immediate usefulness of an abstraction for solving current tasks.
Future Information Gain (Future IG)
This metric assesses the potential value of an abstraction by measuring how much better the model can solve future problems if that specific abstraction was learned previously. It guides the evolution process by prioritizing abstractions with high potential for future utility.

Terminology

Summary

We introduce EVOLIB, a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge across problem instances without parameter updates or external supervision<ref:2605. p. 014477#pg2>. This approach maintains a shared library of knowledge abstractions, including modular skills and reflective insights, automatically extracted from the model’s own inference trajectories<ref:2605. p. 0 p. 014477#pg3>.

Pruning redundant reasoning tokens

The key idea is to maintain a structured, evolving collection of knowledge abstractions<ref:2605.14477#pg4>. The system extracts two types of abstractions from model-generated solutions: (1) modular skills, which capture reusable procedures, and (2) reflective insights, which encode common errors and corrective strategies<ref:2605. p. 114477#pg4>. EVOLIB is governed by two central principles: Abstraction and Evolution<ref:2605. p. 114477#pg4>.

How the training signal is built

The library K is represented as a weighted collection of knowledge abstractions<ref:2605.14477#pg4>. This weighting mechanism is based on Information Gain (IG) and Future Information Gain (Future IG), which jointly capture the immediate usefulness of an abstraction and its potential for generating valuable future abstractions<ref:2605. p. 114477#pg4>. The core mechanism that enables evolution is credit propagation along the evolution chains of abstractions<ref:2605. p. 314477#pg4>. Specifically, Information Gain (IG) measures how much better the model Φ can solve xt when zt is present in the extracted abstractions compared to the average baseline<ref:2605. p. 414477#pg4>. Future IG quantifies how much better the model can solve xt given zt-1 as one of the preceding abstractions, compared to the baseline score<ref:2605. p. 514477#pg4>. The weight of each abstraction z in the library is computed based on its IG and Future IG scores<ref:2605. p. 514477#pg4>.

Where the gains come from

EVOLIB demonstrates consistent improvements over the top test-time scaling and learning methods without ground-truth feedback<ref:2605.14477#pg2>. Across all settings, our method shows substantial improvements over existing approaches with more efficient token usage<ref:2605. p. 1 p. 114477#pg9>. The framework supports continual learning that is less dependent on task ordering, unlike existing TTL methods based on linear knowledge updates<ref:2605. p. 1 p. 114477#pg9>. Ablation studies show that abstraction consolidation leads to substantial gain and helps control memory growth while encouraging the emergence of generic abstractions<ref:2605. p. 8 p. 814477#pg10>.

The paper addresses failure modes in continuously updated abstract memory, such as misgrouping, interference, and overfitting<ref:2605.14477#pg19>. EVOLIB avoids these issues by introducing a principled weighting, sampling, and consolidation mechanism<ref:2605. p. 214477#pg4>. Consolidation is critical for promoting the emergence of abstractions that generalize beyond specific problems<ref:2605. p. 514477#pg9>. The top high-Future-IG abstractions remain concrete and applicable to only a subset of tasks, indicating the weighting mechanism prioritizes useful abstractions<ref:2605. p. 214477#pg21>.

The results across mathematical reasoning, code generation, and multi-turn agentic tasks show EVOLIB achieves the strongest average performance across all evaluated settings<ref:2605.14477#pg9>. The framework is more token-efficient than both TTS baselines and outperforms TTL baselines by 5-10% across three benchmarks<ref:2605. p. 714477#pg9>. EVOLIB consistently outperforms Dynamic Cheatsheet (DC), the best TTL baseline, by 5-10%<ref:2605. p. 714477#pg9>. The largest gain under random ordering in the continual learning setting suggests EVOLIB is more robust to variations in the task stream<ref:2605. p. 814477#pg9>. The framework offers a promising direction for LLMs to learn and self-improve during deployment without requiring gradient-based fine-tuning or supervision signals<ref:2605. p. 814477#pg10>. The paper discusses the work as a stepping stone toward self-improving agents, which is a positive societal impact on effective AI assistant<ref:2605.14477#pg10>. The limitations note that the method works best for tasks in which verifying a solution is easier than generating one<ref:2605. p. 9 p. 914477#pg11>. The authors suggest enabling LLMs to meta-learn the design of the library itself as a promising future direction<ref:2605. p. 9 p. 914477#pg10>. The full code (including prompts) is released in the supplementary material<ref:2605. p. 6 p. 614477#pg26>. All datasets used are referenced<ref:2605. p. 0 p. 014477#pg29>. The full code together with clear documentation are released in the supplementary material. p. 2 p. 2 The paper does not include any theoretical results<ref:2605. p. 0 p. 014477#pg24>.

REFERENCE RULES

  • Copy a token exactly as it is printed for you. Never invent an id, never guess a page or a paragraph, and never change one: a token the work does not hold is worse than no token. p. 1 p. 1<ref:2605.14477#pg2>

  • Put the token at the END of the sentence it supports, and point at the passage the claim came from: not the first page, and not the page you remember. The paper's pages are labelled --- Page N --- counting from 1 — the same N as that page's --- Page 1 --- line, and a token names that SAME page: the text under --- Page 1 --- carries a token with nothing after it.<ref:2605.14477#pg3>

  • ANCHOR THE WORDS. The words go INSIDE the token, in the same brackets: after the location, a comma with no space, then five or more words copied word for word from that passage the claim rests on. No quotation marks around them<ref:2605. p. 0 p. 014477#pg4>. The words go inside, so the token ends,five or more words copied word for word> and the sentence's own full stop follows the closing >.<ref:2605.14477#pg9>.

REFERENCE RULES

  • Copy a token exactly as it is printed for you. Never invent an id, never guess a page or a paragraph, and never change one: a token the work does not hold is worse than no token.<ref:2605.14477#pg26>

  • Put the token at the END of the sentence it supports, and point at the passage the claim came from: not the first page, and not the page you remember. The paper's pages are labelled --- Page N --- counting from 1 — the same N as that page's --- Page 1 --- line, and a token names that SAME page: the text under --- Page 1 --- carries a token with nothing after it.<ref:2605.14477#pg27>

  • ANCHOR THE WORDS. The words go INSIDE the token, in the same brackets: after the location, a comma with no space, then five or more words copied word for word from that passage the claim rests on. No quotation marks around them<ref:2605.14477#pg28>. The words go inside, so the token ends,five or more words copied word for word> and the sentence's own full stop follows the closing >.

  • Tokens are markup, not speech: never read one aloud, never explain one. Never say a page number out loud either — the listener cannot see the pages, and the token already records which one. p. 2

SUMMARY

We introduce EVOLIB, a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge across problem instances without parameter updates or external supervision<ref:2605.14477#pg2>. This approach maintains a shared library of knowledge abstractions, including modular skills and reflective insights, automatically extracted from the model’s own inference trajectories<ref:2605.14477#pg3>.

EVOLIB demonstrates consistent improvements over the top test-time scaling and learning methods without ground-truth feedback<ref:2605.14477#pg9>. Across all settings, our method shows substantial improvements over existing approaches with more efficient token usage<ref:2605.14477#pg9>. The framework supports continual learning that is less dependent on task ordering, unlike existing TTL methods based on linear knowledge updates<ref:2605.14477#pg9>. Ablation studies show that abstraction consolidation leads to substantial gain and helps control memory growth while encouraging the emergence of generic abstractions<ref:2605.14477#pg10>. The top high-Future-IG abstractions remain concrete and applicable to only a subset of tasks, indicating the weighting mechanism prioritizes useful abstractions. p. 2 The paper discusses the work as a stepping stone toward self-improving agents, which is a positive societal impact on effective AI assistant<ref:2605.14477#pg10>. The limitations note that the method works best for tasks in which verifying a solution is easier than generating one<ref:2605.14477#pg11>. The authors suggest enabling LLMs to meta-learn the design of the library itself as a promising future direction<ref:2605.14477#pg10>. The full code (including prompts) is released in the supplementary material<ref:2605.14477#pg26>. All datasets used are referenced<ref:2605.14477#pg29>. The full code together with clear documentation are released in the supplementary material. The paper does not include any theoretical results<ref:2605.14477#pg24>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Reimplement knowledge accumulation through modular skills to solve new tasks more efficiently. The improved system will maintain a library of reusable functions for coding, sub-problems for reasoning, and workflows for agentic tasks, enabling more efficient transfer compared to memory-based approaches that store unstructured trajectories.

  2. Implement a principled weighting and consolidation mechanism using Information Gain (IG) and Future Information Gain (Future IG). This mechanism will jointly optimize for immediate utility and long-term value, allowing simple, instance-specific abstractions to evolve into more general and reusable ones over time.

  3. Enhance abstraction extraction by using LLM calls to summarize sub-modules into new skills for reasoning and agentic tasks. This allows the system to capture reusable procedures directly from model outputs rather than relying solely on implicit reasoning steps.

  4. Develop a dynamic sampling strategy based on the calculated weights w(z K) derived from IG and Future IG scores. This will ensure that abstractions contributing both immediate utility and potential for future abstraction generation are prioritized during task solving, leading to more effective knowledge transfer across problems.

  5. Integrate abstraction consolidation into the library update step by prompting the LLM to merge semantically and functionally similar new abstractions with existing ones. This prevents misgrouping and promotes the emergence of generalizable abstractions that apply across multiple problems, as shown in Table B.1 cases like Case 3 for skills.

  6. Introduce a dynamic update mechanism for Future IG scores based on an exclusion baseline to avoid bias from repeated sampling of the same abstraction. This allows the library to continuously evolve, where more advanced abstractions receive high scores, while older, less general ones are progressively de-emphasized without explicit removal.

  7. Improve robustness to task ordering in continual learning settings by leveraging the dynamic weighting mechanism which is shown to be less dependent on task ordering than linear knowledge updates. This allows the agent to learn from diverse and interleaved tasks without needing a structured curriculum.

Sources

Related papers