VitaBench: Memory Lifecycles Beyond Retrieval

arXiv:2610.14731 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "VitaBench: Memory Lifecycles Beyond Retrieval".

Jane: The gist The authors introduce VITABENCH, a benchmark designed to make memory lifecycle decisions measurable through repeated memory reviews and full-horizon counterfactual evaluation,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper today titled VitaBench: Memory Lifecycles Beyond Retrieval. Basically, existing benchmarks test if an AI can remember something from earlier interactions when it needs it later, but this paper argues that the real problem is deciding what to *do* with that memory before you even know you need it.

Jane: That's a really interesting pivot, Tom. So we're moving past just retrieval and looking at the decisions about keeping or deleting information, whether we keep it exact, compress it, consolidate it, or delete it over time. What does this mean for how we think about persistent memory?

Lu: The core idea here is formalizing what they call memory lifecycle evaluation. They introduce VITABENCH to make those lifecycle decisions measurable by doing repeated reviews and full-horizon counterfactual evaluation, which is what they're doing against existing benchmarks that only look at retrieval and temporal reasoning <ref:2610.14731#pg1>.

Meng: So if I understand this correctly, the paper isn't just testing if the AI can find a fact later, but whether the decision to compress or consolidate that fact *before* it's needed was actually the right choice for that specific situation.

Tom: Exactly. The authors introduce three diagnostics to characterize this decision structure: lifecycle headroom, which measures the advantage of adaptive choices over a best fixed strategy; horizon reversal, which checks if short-term decisions change under delayed consequences; and optimal-action diversity, which looks at whether different memory states favor different lifecycle operations <ref:2610.14731#pg2>.

Jane: That sounds like a way to quantify the structure of the memory problem itself, not just the final answer. Can you tell us a bit more about what those diagnostics actually measure in practice?

Lu: Lifecycle headroom, for instance, it measures how much better an adaptive choice is compared to sticking with the best fixed strategy you could find for that memory state <ref:2610.14731#pg2>. Horizon reversal tracks when a decision made for a short time changes when you look at the long-term consequences <ref:2610.14731#pg2>.

Meng: That makes sense for deployment, because if we can measure that headroom, we know when it's worth having our system be flexible instead of just following a rigid rule.

Tom: And there's optimal-action diversity, which checks if different memory states actually push the AI toward different lifecycle operations, like preferring compression in one state and consolidation in another <ref:2610.14731#pg2>. The results show substantial structure across both tiers of the benchmark.

Jane: So when you look at those results, they found that lifecycle headroom is one point three six four in the Diagnostic tier and one point four zero five in the Naturalistic tier <ref:2610.14731#pg3>. That's a pretty substantial number suggesting these adaptive choices have a real effect on performance.

Lu: And what's more interesting is that sixty point two percent of states, in the Diagnostic tier, and fifty-six point five percent in the Naturalistic tier, actually change their preferred action between short and full evaluation horizons <ref:2610.14731#pg3>. That shows how much things are dynamic depending on whether you're looking at a quick review or a long-term view.

Paper summary: Tom: And the paper shows that if you use a learned delayed lifecycle policy, it improves over the best fixed strategy on VITABENCH-Naturalistic, reaching one hundred one point five versus eighty-three point four joint utility <ref:2610.14731#pg3>. That's a solid improvement in performance for that tier.

Jane: So, what does this mean for the practical application of this research? If we look at the results, it seems like retrieving past information and deciding how to persist it are two separate evaluation problems <ref:2610.14731#pg3>.

Lu: That distinction is key. Retrieval evaluation asks if you can recover past information when a task arrives; lifecycle evaluation asks if the representation decisions made before that task were actually appropriate given what happened later <ref:2610.14731#pg3>.

Meng: So, a system could be great at finding old data but still make terrible decisions about how to store it long-term, or vice versa. That's a huge area for improvement in production systems where memory isn't just a database lookup.

Tom: And this research suggests that managing the lifecycle requires us to evaluate the delayed consequences of those representation choices, which is something existing benchmarks haven't done well <ref:2610.14731#pg3>.

Jane: So, the authors introduce VITABENCH specifically to make these decisions measurable through repeated reviews and full-horizon counterfactual evaluation <ref:2610.14731#pg1>. It’s a deliberate setup to test those delayed consequences.

Lu: The Diagnostic tier isolates four specific future-use patterns: delayed exact reuse, recurrent semantic reuse, transient information, and conflicting evidence <ref:2610.14731#pg4>. The Naturalistic tier then composes these into generated customer-service-style interaction streams <ref:2610.14731#pg4>.

Meng: I wonder about the cost of running something like VITABENCH at scale. If we're testing these complex decision structures, how much compute does that actually require for a model to learn those policies?

Tom: The paper evaluates these policies using joint utility and persistent storage and policy regret <ref:2610.14731#pg6>. They are also distinguishing between fixed, short-horizon, and delayed lifecycle policies on VITABENCH-Naturalistic <ref:2610.14731#pg6>.

Jane: And the results show that repeated VITAMEM reaches one hundred one point five versus eighty-three point four joint utility <ref:2610.14731#pg6>. That's a concrete measure of how much better the learned policy is doing compared to just a fixed one.

Lu: The structural diagnostics characterize the decision problem itself, but the joint utility and regret metrics characterize the policies themselves <ref:2610.14731#pg6>. It’s about understanding both what’s right to do and how good you are at doing it under uncertainty.

Meng: So, for someone who just listens to this, the main thing is that memory management isn't a simple retrieval task anymore; it's a strategic decision-making process with long-term implications <ref:2610.14731#pg3>.

Tom: And the authors are making this measurable through these specific tiers and diagnostics, giving us tools to actually test these complex representation decisions <ref:2610.14731#pg4>. This research shows that representation decisions made before future use is known require evaluation of their delayed consequences <ref:2610.14731#pg3>.

Jane: So, to wrap up this discussion on VitaBench, it seems the main point is that we need a new way to evaluate persistent memory systems that goes beyond just checking if facts can be retrieved later <ref:2610.14731#pg1>. This paper provides a framework for measuring the lifecycle decisions themselves <ref:2610.14731#pg3>.

Conclusion: Tom: So, to wrap up this discussion, we're looking at this paper called VitaBench: Memory Lifecycles Beyond Retrieval. It’s essentially about moving past just checking if an AI can remember something later and focusing on how it decides what to actually do with that memory over time.

Jane: Right, so instead of just asking if the AI finds the fact, they're asking why that specific decision to keep or change the memory was good in hindsight. It’s about representation choices made before we knew what we needed them for.

Lu: They formalize this as memory lifecycle evaluation and introduce VITABENCH, which is a benchmark designed to make those lifecycle decisions measurable through repeated reviews and full-horizon counterfactual evaluation.

Meng: I see they built two tiers, a Diagnostic tier that isolates specific future-use patterns like delayed exact reuse, and a Naturalistic tier where they put those into generated customer service conversations.

Tom: The core idea is that retrieval evaluation asks if you can recover past information when a task arrives, but lifecycle evaluation asks whether the representation decisions made before that task arrived were actually appropriate given what happened later.

Jane: That distinction is important because it shows that an AI can be good at finding old data, but still make poor decisions about how to store it long-term.

Lu: The authors characterize this decision structure with three diagnostics: lifecycle headroom, horizon reversal, and optimal-action diversity. These measure the advantage of adaptive choices over a fixed strategy, when short-term decisions change under delayed consequences, and if different memory states favor different operations.

Tom: And the results show substantial structure across both tiers. Lifecycle headroom is one point three six four in Diagnostic and one point four zero five in Naturalistic, and fifty-six point five percent of states change their preferred action between short and full horizons.

Jane: That suggests that these adaptive choices actually have a real effect on performance, not just some theoretical difference on paper.

Lu: Plus, they found that a learned delayed lifecycle policy improved over the best fixed strategy on VITABENCH-Naturalistic, reaching a joint utility of one hundred one point five versus eighty-three point four for the best fixed strategy.

Meng: So it’s not just about having a better retrieval system; it’s about having a better storage and management policy that anticipates future needs.

Tom: Exactly, so this research shows that managing memory requires us to evaluate the delayed consequences of those representation decisions, which is something existing benchmarks haven't done well.

Jane: It sets up a whole new way to test persistent memory systems beyond just how good they are at retrieving facts from their history.

Muquan Yu, Dihan Hu

University of Hong Kong · Chinese University of Hong Kong · Cornell University

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Comments: Accepted at the NeurIPS 2026 Workshop on Continual Learning in the Era of Foundation Models and Embodied Agents (CL4FMAgents)

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The authors introduce VITABENCH, a benchmark designed to make memory lifecycle decisions measurable through repeated memory reviews and full-horizon counterfactual evaluation, which

Key concepts

Memory Lifecycle Evaluation: This formal problem is about deciding the persistence strategy for memory—whether information should stay exact, be compressed, merged, or deleted before its next use is known. It focuses on the quality of earlier representation choices rather than just whether past data can be retrieved later.
Lifecycle Headroom: This diagnostic measures how much advantage an agent gains by choosing an adaptive memory strategy over a fixed one. A higher headroom indicates that the system has more flexibility and better options for managing its memory based on current context.
Horizon Reversal: This measures when decisions made for a short future horizon change when considering the long-term consequences. It tests if an agent's immediate choice to keep or compress memory is optimal when factoring in what happens much later.
VITABENCH Tiers: VitaBench has two parts: a Diagnostic tier that isolates specific memory patterns for inspection, and a Naturalistic tier that combines these patterns into realistic customer service interactions. This allows researchers to study both individual phenomena and their complex interaction.

Summary

The gist The authors introduce VITABENCH, a benchmark designed to make memory lifecycle decisions measurable through repeated memory reviews and full-horizon counterfactual evaluation, which distinguishes between retrieval and representation decision quality.

Memory Lifecycle Evaluation

The paper formalizes the problem of deciding how a memory should persist over time—whether it should remain exact, be compressed, consolidated, or deleted—as memory lifecycle evaluation (Page 1). This is distinct from long-term retrieval benchmarks which only ask whether information can be recovered when relevant (Page 1). The system must decide what information to continue existing and in what form before its future use is known (Page 1).

Key Diagnostics for Lifecycle Structure

The authors characterize the lifecycle decision structure with three diagnostics:

Lifecycle headroom, which measures the advantage of adaptive representation choices over the best fixed strategy (Page 2).
Horizon reversal, which measures when short-horizon decisions change under delayed consequences (Page 2).
Optimal-action diversity, which measures whether different memory states favor different lifecycle operations (Page 2).

VITABENCH Tiers and Evaluation

VITABENCH contains two complementary tiers:

VITABENCH-Diagnostic isolates four fundamental future-use patterns: delayed exact reuse, recurrent semantic reuse, transient information, and conflicting evidence (Page 4).
VITABENCH-Naturalistic composes these phenomena into generated customer-service-style interaction streams containing evolving entity (Page 4).

Policy Evaluation and Results

The study evaluates policies using joint utility, persistent storage, and policy regret (Page 6). The results show that both tiers exhibit substantial lifecycle structure: lifecycle headroom is 1.364 in Diagnostic and 1.405 in Naturalistic (Page 3). Furthermore, a learned delayed lifecycle policy improved over the best fixed strategy on VITABENCH-Naturalistic, reaching 101.5 versus 83.4 joint utility (Page 3). The results show that retrieving past information and deciding how it should persist are distinct evaluation problems (Page 3).

Policy Distinctions

The study empirically distinguishes fixed, short-horizon, and delayed lifecycle policies on VITABENCH-Naturalistic (Page 6). For instance, repeated VITAMEM reaches 101.5 versus 83.4 joint utility (Page 6). The structural diagnostics characterize the lifecycle decision problem itself, while joint utility, storage, and regret characterize the policies evaluated on that problem (Page 6).

Relationship to Existing Benchmarks

VITABENCH is intended to complement existing long-term memory benchmarks by evaluating counterfactual evaluation of earlier representation choices under a shared future trajectory (Page 15). The distinction is that existing benchmarks primarily evaluate whether an agent can use information from an interaction history when it becomes relevant, whereas VITABENCH evaluates whether an earlier decision about how that information should persist was appropriate under its subsequently realized future (Page 15).

Conclusion

The results establish memory lifecycle management as a distinct evaluation axis for persistent-memory language agents alongside long-term retrieval (Page 7). This research shows that representation decisions made before future use is known require evaluation of their delayed consequences (Page 3). The authors introduce VITABENCH, a benchmark designed to make these decisions directly measurable (Page 1).

--- Page 1 ---

VitaBench: Memory Lifecycles Beyond Retrieval Muquan Yu †1,2 Dihan Hu 3 Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when it becomes relevant Persistent memory systems, however, face an additional problem before future use is known: deciding how a memory should persist over time—whether it should remain exact, be compressed, consolidated, or deleted Existing benchmarks capture long-range retrieval, temporal reasoning, evidence reuse, and incremental interaction, but generally do not directly evaluate the quality of these earlier representation decisions We formalize this problem as memory lifecycle evaluation and introduce VITABENCH, a benchmark that makes lifecycle decisions measurable through repeated memory reviews and full-horizon counterfactual evaluation VITABENCH contains a controlled Diagnostic tier that isolates distinct future-use patterns and a Naturalistic tier that composes them into generated customer-servicestyle interaction streams We characterize lifecycle decision structure with three diagnostics: lifecycle headroom, which measures the advantage of adaptive representation choices over the best fixed strategy; horizon reversal, which measures when short-horizon decisions change under delayed consequences; and optimalaction diversity, which measures whether different memory states favor different lifecycle operations Across both tiers, VITABENCH exhibits substantial lifecycle structure: lifecycle headroom is 1.364 in Diagnostic and 1.405 in Naturalistic, and 60.2% and 56.5% of states, respectively, change their preferred action between short and full horizons A learned delayed lifecycle policy further improves over the best fixed strategy on VITABENCH-Naturalistic, reaching 101.5 versus 83.4 joint utility These results show that retrieving past information and deciding how it should persist are distinct evaluation problems, and that lifecycle management requires evaluation of the delayed consequences of representation decisions

--- Page 2 ---

Candidate lifecycle actions KEEP-EXACT retain current representation COMPRESS replace w. a shorter representation CONSOLIDATE merge evidence into abstraction DELETE remove from persistent memory Lifecycle headroom Hlife = Ū(π) − Ū(π fix) adaptive vs. best fixed strategy 1. Pre-decision memory state Memory m current representation rm observable state sm • past interactions • current representation (exact / compressed / consolidated / etc.) • metadata (e.g., timestamp, entity, type) • no future use is known 3. Shared future interaction stream same future event for every action future events are used for evaluation but hidden from the policy update delayed reuse query time Benchmark-level diagnostics Decision-level result Preferred lifecycle action a(s) = argmaxa Q(s,a) Horizon reversal RH = Pr(a short ≠ a long) short vs. full horizon Optimal-action diversity p(a) = Pr(a = argmaxa' Q(s,a′) which action is optimal?

--- Page 3 ---

Full-horizon utility comparison Q(s,KEEP-EXACT) Q(s,COMPRESS) Q(s,CONSOLIDATE) Q(s,DELETE) Figure 1: VITABENCH evaluates memory lifecycle decisions. At each review point, a policy must choose how a memory should persist before its future use is known. VITABENCH branches from the same pre-decision memory state, applies each candidate lifecycle action, and replays the same future interaction stream for every branch. The resulting full-horizon utilities identify the preferred lifecycle action and quantify the benchmark’s lifecycle decision structure This creates a distinction between two evaluation problems. Retrieval evaluation asks whether relevant past information can be recovered when a later task arrives Lifecycle evaluation asks whether the representation decisions made before that task arrived were appropriate A system can succeed at the first problem while avoiding the second—for example, by storing everything exactly and postponing all decisions until retrieval time Conversely, a lifecycle policy may save storage or reduce interference by transforming memory, but those decisions can only be judged by their downstream consequences Existing long-memory benchmarks were not designed to provide this supervision LongMemEval [Wu et al., 2025], LoCoMo [Maharana et al., 2024], MemoryAgentBench [Hu et al., 2025], and related evaluations such as MemoryArena [He et al., 2026] contain important phenomena including longrange retrieval, evidence reuse, temporal reasoning, updates, incremental interaction Their released evaluation interfaces, however, generally specify which past information supports a later answer rather than what would have happened under alternative representation decisions made earlier Table 1 summarizes this structural distinction.

--- Page 4 ---

Table 1: Structural comparison of representative long-memory benchmark interfaces with respect to lifecycle-relevant capabilities Property LongMemEval LoCoMo MemoryAgentBench VITABENCH (ours) Property Long-range memory use yes yes yes yes Evidence reuse yes yes yes yes Updates / conflicts partial yes partial Incremental interaction no no partial full-horizon counterfactual evaluation no no no Yes facts, stable preferences, obsolete information, superseding updates, historical dependencies, and delayed reuse. The Diagnostic tier makes individual lifecycle phenomena easy to inspect while the Naturalistic tier tests their interaction within a more structured multi-episode setting The resulting environments exhibit substantial lifecycle decision structure. Lifecycle headroom is 1.

Improvements for AI systems

Adapt memory representation dynamically based on future needs by leveraging lifecycle headroom, which measures the advantage of adaptive representation choices over the best fixed strategy. This allows agents to choose between KEEP-EXACT, COMPRESS, CONSOLIDATE, or DELETE more effectively than a static rule.
Implement temporal credit assignment for memory management by training policies using delayed-counterfactual evaluation, which involves computing targets based on full-horizon counterfactual values derived from replaying the same future interaction stream. This enables systems to learn that actions taken now have consequences far into the future, as shown by VITAMEM reaching a joint utility of 200.4 compared to 83.4 for the best fixed strategy on VITABENCH-Naturalistic.
Develop one-shot delayed or repeated delayed lifecycle policies that outperform fixed strategies by demonstrating better temporal credit assignment, such as repeated VITAMEM improves over always-COMPRESS, the strongest fixed Diagnostic policy, by +30.3 joint utility. This allows the AI to benefit from re-evaluating memory states as history evolves rather than relying on a single optimal decision.
Improve decision-making robustness by utilizing optimal-action diversity as a diagnostic metric to ensure that different memory states genuinely favor different lifecycle operations. This prevents the system from getting stuck in a fixed strategy when multiple actions are equally useful across various memory contexts.

Abstract

Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when relevant. Persistent memory systems face an additional problem before future use is known: deciding whether a memory should remain exact, be compressed, consolidated, or deleted. Existing benchmarks capture long-range retrieval, temporal reasoning, evidence reuse, and incremental interaction, but do not directly evaluate these earlier representation decisions. We formalize memory lifecycle evaluation and introduce VitaBench, a benchmark that makes lifecycle decisions measurable through repeated memory reviews and full-horizon counterfactual evaluation. VitaBench contains a controlled Diagnostic tier that isolates future-use patterns and a Naturalistic tier that composes them into generated customer-service-style interaction streams. We characterize lifecycle decision structure with three diagnostics: lifecycle headroom, measuring the advantage of adaptive choices over the best fixed strategy; horizon reversal, measuring when short-horizon decisions change under delayed consequences; and optimal-action diversity, measuring whether different memory states favor different lifecycle operations. Across both tiers, VitaBench exhibits substantial lifecycle structure: lifecycle headroom is 1.364 in Diagnostic and 1.405 in Naturalistic, and 60.2% and 56.5% of states, respectively, change their preferred action between short and full horizons. A learned delayed lifecycle policy further improves over the best fixed strategy on VitaBench-Naturalistic, reaching 101.5 versus 83.4 joint utility. These results show that retrieving past information and deciding how it should persist are distinct evaluation problems, and that lifecycle management requires evaluation of the delayed consequences of representation decisions.