InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing
cs.CL
Submitted: 2025-05-28
Updated: 2026-09-01
Comments: Main conference of EMNLP 2026
Code: https://github.com/Syon-Li/InComeS
License: http://creativecommons.org/licenses/by/4.0/
The gist: Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge
Terminology
Abstract
Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge regurgitation. Leveraging the strong contextual reasoning abilities of large language models (LLMs), in-context learning (ICL) becomes a promising editing method by comprehending edit information through context encoding. However, this method is constrained by the limited context window of LLMs, leading to degraded performance and efficiency as the number of edits increases. To overcome this limitation, we propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts through explicit compression and selection mechanisms. Specifically, InComeS compresses each editing context into the key-value (KV) cache of a special gist token, enabling efficient handling of multiple edits without being restricted by the model's context window. Furthermore, specialized cross-attention modules are added to dynamically select the most relevant information from the gist pools, enabling adaptive and effective utilization of edit information. We conduct experiments on diverse model editing benchmarks with various editing formats, and the results demonstrate the effectiveness and efficiency of our method.
Sources
- DeepSeek-V3 Technical Report
- A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression
- UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
- PMET: Precise Model Editing in a Transformer
- Distilling the Knowledge in a Neural Network
- Liger Kernel: Efficient Triton Kernels for LLM Training
- An Empirical Study of NetOps Capability of Pre-Trained Large Language Models
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Dynamic Retriever for In-Context Knowledge Editing via Policy Optimization
- EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models
- C-Pack: Packed Resources For General Chinese Embeddings
- Qwen2 Technical Report
- A Comprehensive Study of Knowledge Editing for Large Language Models
- Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering