Role-aware Heuristic Episodic Attention for Conversational LLMs
cs.CL
Submitted: 2026-10-01
Updated: 2026-10-10
Code: https://github.com/huha12138/rhea
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models often lose track of persistent instructions and relevant information as multi-turn conversations grow.
Terminology
Abstract
Large language models often lose track of persistent instructions and relevant information as multi-turn conversations grow. We study this cumulative contextual decay through three related failure modes: attention pollution, dilution, and drift. We propose REA (Role-aware Heuristic Episodic Attention), a context-management framework that assigns different persistence and representation policies to instructions and episodic interactions. Instructional Memory retains identified global constraints in a dedicated prefix. Episodic Memory preserves user inputs and compresses model replies, while heuristic retrieval selects raw text, compressed representations, or omission for each historical turn. On Long-MT-Bench+, REA improves the judge score from 6.32 to 7.36 on a 10-point scale, a 16.5% relative gain over the Vanilla baseline, and reduces average latency by 2.91 times. Additional evaluations show aggregate gains on three backbones spanning 1.7B-7B parameters and on Chinese and English role-playing tasks. These results support role-aware context management as a practical approach to maintaining conversational continuity and instruction adherence.
Sources
- GPT-4 Technical Report
- Longformer: The Long-Document Transformer
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
- Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- LoRA: Low-Rank Adaptation of Large Language Models
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- Memory OS of AI Agent
- Compressed Context Memory For Online Language Model Interaction
- MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models
- LLMs Get Lost In Multi-Turn Conversation
- Lost in the Middle: How Language Models Use Long Contexts
- PISCO: Pretty Simple Compression for Retrieval-Augmented Generation
- MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation
- Long Dialog Summarization: An Analysis
- Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
- On Memory Construction and Retrieval for Personalized Conversational Agents
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering