Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization

The paper introduces and formalizes the task of streaming dialogue summarization, where a system must generate a self-contained summary of a recent dialogue window by selectively retrieving and utilizing relevant information from an unbounded, potentially very long history, all under a strict memory budget. This setting is motivated by real-world platforms like messaging apps and collaborative workspaces, where users frequently need summaries of recent segments to catch up on discussions, but the current window often lacks the context needed to interpret it (e.g., pronouns without antecedents, decisions without rationales).

The authors argue that the central challenge is not the amount of history accessed, but whether the memory system can recover the specific evidence that the current window presupposes. They show that existing paradigms—recency-based, summarization-based, and retrieval-based memory—are misaligned with this goal. Recency-based memory fails when relevant evidence is old; summarization-based memory (incremental or hierarchical) dilutes fine-grained evidence through compression; and retrieval-based memory, while strongest, suffers from two bottlenecks: (1) it retrieves based on what the window explicitly states rather than what it leaves unresolved, and (2) retrieved chunks contain irrelevant neighboring content that wastes the memory budget.

To address these issues, the paper proposes ReMEMBER (Retrieving Missing Evidence Memory By Elicited Recall), a two-stage framework:

1. Gap-conditioned evidence retrieval: This stage first detects gap-bearing utterances in the current window—utterances that are summary-worthy but whose interpretation depends on missing prior context (referential, attribute, or relational gaps). For each gap, it constructs a targeted evidence-seeking query that specifies the type of missing information without hypothesizing the answer. These queries are then used with both sparse (BM25) and dense (Qwen3-Embedding-0.6B) retrievers, and the results are fused using reciprocal rank fusion.

2. Gap-conditioned chunk refinement: This stage refines the retrieved chunks by scoring individual utterances within them against the gap query, extracting only the most relevant complete turns (including speaker info) to form evidence-dense memory. A gap-balanced accumulation process ensures that no single gap monopolizes the memory budget.

The authors construct a new benchmark with 900 instances from chit-chat and workplace dialogue datasets, with histories up to 160K tokens. They annotate salient utterances and contextual gaps with explicit links to resolving evidence in the history. They also propose a two-axis evaluation protocol: memory recall (does the memory contain the resolving evidence?) and summary quality, which is decomposed into window completeness (does the summary cover the window's salient content?), gap-resolution completeness (does the summary reflect the resolved gaps?), faithfulness, and conciseness.

Experiments show that ReMEMBER significantly outperforms all memory construction baselines (recency, incremental/hierarchical summarization, and sparse/dense/hybrid retrieval) on memory recall (0.6984 vs. 0.5412 for the best baseline, Hybrid) and gap-resolution completeness, while maintaining competitive performance on other metrics. The improvements are consistent across different dialogue lengths (up to 160K tokens), domains, and summarizer models. Ablation studies confirm that both gap-conditioned retrieval and chunk refinement are crucial components. The paper concludes that the main bottleneck is not evidence availability but organizing historical evidence into a form that the summarizer can reliably use, and that even with ideal memory, summary generation imposes a separate ceiling on gap resolution.

Improvements to AI Systems:

1. Gap-Aware Memory Retrieval: Implement a two-stage retrieval pipeline that first identifies gap-bearing utterances (those with unresolved referential, attribute, or relational dependencies) in the current input window, then generates targeted evidence-seeking queries (e.g., Who is 'she' referring to? or What was the rationale for the decision?) rather than generic similarity queries. This allows the system to retrieve missing context that the current text presupposes, not just what it explicitly mentions.

2. Evidence-Dense Chunk Refinement: Add a post-retrieval refinement module that scores individual utterances within retrieved chunks against the gap query, extracting only the most relevant complete turns (with speaker metadata) instead of feeding whole chunks. This reduces memory waste from irrelevant neighboring content and increases the signal-to-noise ratio in the stored memory.

3. Gap-Balanced Memory Allocation: Introduce a budget allocation mechanism that tracks how much memory each unresolved gap has consumed and prevents any single gap from monopolizing the limited memory. This ensures diverse coverage of multiple missing contexts, leading to more comprehensive summaries.

4. Two-Axis Evaluation for Summarization Systems: Adopt a dual evaluation protocol that separates memory recall (does the system's memory contain the exact resolving evidence?) from summary quality (window completeness, gap-resolution completeness, faithfulness, conciseness). This allows developers to diagnose whether failures stem from memory construction or generation, enabling targeted improvements.

5. Hybrid Sparse-Dense Retrieval with Reciprocal Rank Fusion: Use a fusion of BM25 and dense embeddings (e.g., Qwen3-Embedding) for gap-conditioned queries, as this outperforms any single retriever. The system can then rank evidence across both lexical and semantic matches, improving recall of older or paraphrased evidence.

6. Scalable Long-History Handling: Design the memory system to operate on unbounded histories (up to 160K tokens) by avoiding full-history processing. Instead, it selectively retrieves and refines only the evidence needed for the current window, making it computationally feasible for real-time streaming applications.

What the Improved AI System Can Do:

• Produce self-contained summaries of recent dialogue windows that correctly resolve pronouns, attribute decisions to speakers, and explain rationales—even when the relevant evidence occurred thousands of turns earlier.

• Operate under strict memory budgets (e.g., 4K tokens) while maintaining high recall of critical historical evidence, outperforming recency-based, summarization-based, and standard retrieval baselines by a large margin (0.70 vs. 0.54 memory recall).

• Adapt to different dialogue domains (chit-chat, workplace) and lengths without retraining, thanks to the domain-agnostic gap detection and query construction.

• Provide interpretable memory construction—the system can show which gaps it detected, what queries it generated, and which evidence it stored, enabling debugging and user trust.

• Serve as a pluggable memory module for any downstream summarizer (e.g., GPT-4, LLaMA), improving their output without modifying the generator itself, as demonstrated by consistent gains across multiple summarizer models.