AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens

arXiv:2610.20641 · cs.CL, cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens".

Jane: The gist The ABRIDGEKV method compresses each completed block of generated text into the KV entries of a few learned pause tokens,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's look at what they actually did in ABRIDGEKV. The authors are Jack Cai, Lijie Yang, Vasilis Kontonis, Dimitris Papailiopoulos, and Shivam Garg from Princeton University and Microsoft Research >

Jane: They take this concept of block compression and call it ABRIDGEKV because they're compressing long reasoning chains into the KV cache of learned pause tokens >

Lu: The abstract lays out the main idea very clearly: "ABRIDGEKV compresses each completed block of generated text into the KV entries of a few learned pause tokens" >

Meng: So what does that actually mean in practice? It sounds like they are adding some extra input tokens that have learned embeddings, and these tokens produce no output, but they store the summary information >

Lalam: Right, and once those pause tokens have encoded the block's information, the original block’s cache gets freed up entirely for subsequent steps >

Tom: The paper highlights three specific design choices they made that make this work: first, these pause tokens’ KV entries take positions inside the block they replace >

Jane: That means they aren't just floating somewhere; they are woven into the structure of the text block itself, which is interesting because prior learned summaries haven't used that positioning before >

Lu: The second choice is how they allocate their cache budget across layers according to a measured value of each layer’s entries >

Meng: That sounds like they measured how important each layer's entries were and then told the system to keep different amounts of information in different layers based on that measurement >

The paper's summary: Tom: So when we look at the details, ABRIDGEKV is trained end to end with the next-token loss, which means later tokens are teaching those code rows what information they need to preserve >

Jane: That’s a key part of the training setup. It’s not just a post-hoc compression; it’s an integrated learning process where the model learns how to compress itself >

Lu: The paper mentions that this end-to-end training allows later tokens to teach the code rows exactly what information needs to be kept from earlier blocks >

Meng: But it also says they operate on relatively large blocks, which is important because if you’re compressing every single paragraph, that changes how the compression ratio works >

Lalam: They mention specific results for block size at five hundred twelve tokens; when a block completes, K pause tokens write its code rows, and those raw rows are dropped > <ref:2610.20641#pg2,tokens; when a block completes, K pause tokens write its code rows>

Tom: The paper shows that the resident cache in ABRIDGEKV is just the prompt plus these K rows per finished block, and the open block itself is compressed as it writes >

Jane: So for a long sequence, you have this structure where you're keeping the prompt, those summary rows from completed blocks, and then whatever new text is currently being written >

Lu: The paper also mentions how this compression manifests in physical memory savings within their paged serving engine because each layer has a fixed budget and rows that a layer doesn't retain are never allocated >

The paper's improvements: Tom: Moving on to the improvements they suggest, they really focus on those three design choices as the novel parts of ABRIDGEKV >

Jane: They point out the in-block code positions, which they say no prior learned summary has used before, as a key design choice >

Lu: Then there’s the layer budget set by measured value; this is new because it's based on how important each layer’s entries are to the model's performance >

Meng: That sounds like a really smart way to manage resources dynamically instead of just setting one fixed retention policy for everything >

Lalam: And they also emphasize that compression operates on relatively large blocks, which is crucial because it makes the pause-token compression effective over long generations >

Tom: The paper shows how this combination of choices leads to tangible results, like reaching forty-three point zero avg@eight against forty-three point eight for full KV at a quarter of the persistent cache size > <ref:2610.20641#pg2>

Jane: And they also showed that serving speed is measured in vLLM on one H100, finishing a problem three point two times faster than with full KV because of this compressed decoding method > <ref:2610.20641#pg2>

Conclusion: Tom: So to wrap up, ABRIDGEKV matches the performance of full KV with a four times smaller persistent cache and finishes a problem three point two times faster in their tuned vLLM environment > <ref:2610.20641#pg2>

Jane: The implication is that this compression isn't just theoretical; it’s translating directly into physical memory savings in their paged serving engine, which is really what matters for production systems >

Lu: From my side, the way they schedule the per-layer budget seems to capture nearly all of the value that can be moved at that mean, and they also show that a per-head schedule trained from scratch is within noise of it >

Meng: For deployment, what I’m seeing is that this approach means you can use a much smaller physical cache pool because vLLM can maintain separate page pools for each budget level >

Lalam: The paper on ABRIDGEKV shows that the information travels by the most direct route, from the block into its own code rows and from there to the token that needs it >

Tom: It’s a novel approach to compressing long reasoning chains into learned pause tokens, which is crucial because long chain-of-thought decoding is memory-bound and evicting cache entries doesn't fix this because a model never trained on a cache with entries removed generates longer, often up to the length limit >

Jane: It’s interesting how they managed to make this work without needing prior learned summaries for those pause tokens, which is a big part of the design here >

Lu: The training setup where later tokens teach the code rows what information to preserve is what makes this approach effective over long generations and really shows that learning can solve this memory problem >

Meng: It’s promising because it suggests we could apply similar compression mechanisms alongside the compaction methods already used by long-running coding agents >

Lalam: Indeed, ABRIDGEKV matches full KV with a four times smaller persistent cache and finishes a problem three point two times faster in our tuned vLLM > <ref:2610.20641#pg2>

Princeton University · Microsoft Research

cs.CL, cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: The gist The ABRIDGEKV method compresses each completed block of generated text into the KV entries of a few learned pause tokens, freeing the block’s original cache and allowing subsequent tokens

Key concepts

Pause Tokens: These are special input tokens that have been trained to encode a summary of a completed text block within their own KV entries. They act as compressed representations, allowing later tokens to access the necessary information without needing the full original cache.
Code Rows: A 'code row' is the specific position within a pause token's KV entry that stores summarized information about a text block. The model learns which parts of the original block are most important to preserve in these rows, effectively creating a compressed summary.
Cache Budget Allocation: This refers to how the total allowed cache memory is divided across different layers of the model. The system dynamically allocates budget based on how much each layer's KV entries contribute to the overall output quality, ensuring efficient memory usage across all parts of the network.

Terminology

KV cache — 1× in this paper · explained in 3 episodes

Summary

The gist The ABRIDGEKV method compresses each completed block of generated text into the KV entries of a few learned pause tokens, freeing the block’s original cache and allowing subsequent tokens to attend to these compressed summaries, which significantly reduces memory usage and increases inference speed for long reasoning chains.

How it works

The ABRIDGEKV system operates by compressing the generated KV cache in fixed blocks of 512 tokens, where when a block completes, a set of pause tokens attends to it and encodes a summary in their own KV entries; we call the KV entry of one pause token at one layer a code row. The block’s original KV entries are then freed, and later tokens attend to the code rows in their place

The model is trained end to end with the next-token loss, so later tokens teach the code rows what information to preserve Three design choices are central: code rows take positions inside the block they replace, the cache budget is allocated across layers according to the measured value of each layer’s entries, and compression operates on relatively large blocks

Design Choices

The three novel choices that make this work are:

The pause tokens’ KV entries take positions inside the block they replace
The cache budget is allocated across layers according to the measured value of each layer’s entries
Compression operates on relatively large blocks

Training and Quotas

The model is trained with the ordinary next-token loss on every response token, computed while its history is held as codes (Eq. 2) The budget is a per-layer quota ql ≤ Kmax: layer l keeps ql of the Kmax=256 code rows of every completed block and masks the rest, the same quota for every KV head of the layer (so tensorparallel serving stays balanced and the physical cache can drop the masked rows) Uniform allocation sets ql=128 everywhere Per-layer (dynamic) allocation keeps the same mean of 128 rows but moves rows between layers, in three steps

Results and Evaluation

At a quarter of the persistent cache, ABRIDGEKV reaches 43.0 avg@8 against 43.8 for full KV (Table 1, Figure 3a) ABRIDGEKV’s resident KV per sequence is 3.54× smaller than full KV’s (the ceiling is 4×; the prompt and the open block stay raw) Serving speed is measured in vLLM on one H100 with our patch for compressed decoding, finishing a problem 3.2× faster than with full KV

The results show that ABRIDGEKV matches full KV with a 4× smaller persistent cache and finishes a problem 3.2× faster in our tuned vLLM, where eviction writes far longer and LightThinker loses 10 points The resulting reduction is realized as physical memory savings in our paged serving engine

Future Directions

ABRIDGEKV suggests several promising directions beyond the setting studied here First, ABRIDGEKV could work alongside the compaction mechanisms already used by long-running coding agents Second, learned KV compression may be especially promising for modalities that are difficult to summarize faithfully in text

The ABRIDGEKV method is a novel approach to compressing long reasoning chains into the KV cache of learned pause tokens, which is crucial because long chain-of-thought decoding is memory-bound and evicting cache entries does not fix the issue because a model never trained on a cache with entries removed generates longer, often up to the length limit The strongest published evictor at a matched budget writes far longer, and the closest trained compressor, retrained on the same data, loses 10 points ABRIDGEKV matches full KV with a 4× smaller persistent cache and finishes a problem 3.2× faster in our tuned vLLM The resulting reduction is realized as physical memory savings in our paged serving engine The compression ratio at a fixed block size shows that the 8:1 arm loses 4.4 points and takes 0.19× full KV’s time The per-layer schedule brings the 4:1 arm to parity with full KV at the same time to finish The information travels by the most direct route, from the block into its own code rows and from there to the token that needs it This routing happens in the 256-row layers, the ones the measurement of Section 5.1 raised, and in the 128-row layers The model is trained end to end with the next-token loss, so later tokens teach the code rows what information to preserve This routing happens in the 256-row layers, the ones the measurement of Section 5.1 raised, and in the 128-row layers The serving path must compute the same function as the trainer (Table 10) Every accuracy in the paper is measured on the served model, so the serving path must compute the same function as the trainer The compression unit explains most of the gap: at a fixed ratio, the block size sets how many lossy compressions a late query reads through and how much recent text stays exact The block size at a fixed ratio shows that the 8:1 arm loses 4.4 points and takes 0.19× full KV’s time The compression ratio at a fixed block size shows that the 2:1 arm is statistically indistinguishable from full KV (−0.4 [−1.8, +1.0]) and finishes a problem in 0.53× full KV’s time The per-layer schedule brings the 4:1 arm to parity with full KV at the same time to finish Every reported number was produced by code we ran and was checked against the underlying logs The training cost of the block size at 4:1, relative to C=512, step time and peak memory for one forward and backward pass over a 16K-token sequence on one H100 (filled: block sizes we trained) The compression ratio at a fixed block size shows that the 8:1 arm loses 4.4 points and takes 0.19× full KV’s time A BRIDGEKV matches full KV with a 4× smaller persistent cache and finishes a problem 3.2× faster in our tuned vLLM The per-layer schedule already captures nearly all of the value that can be moved at this mean, and a per-head schedule trained from scratch (Table 7) is within noise of it This routing happens in the 256-row layers, the ones the measurement of Section 5.

Improvements for AI systems

Bold header: ABRIDGEKV Compression Mechanism

This mechanism compresses each completed block of generated text into the KV entries of a few learned pause tokens and subsequent tokens attend to the pause tokens’ KV entries in its place, allowing for a persistent cache size that is significantly smaller than full KV without sacrificing accuracy.

Bold header: Layer-Specific Cache Budget Allocation

The model is trained to learn optimal layer retention by using a layer budget set by measured value and training a new model on this schedule based on the cost of halving one layer's quota, which allows layers deemed cheapest 64 to be reduced while more critical layers retain more rows.

Bold header: Long-Term Information Recall

The system can effectively manage long-term dependencies because code rows carry their block’s content many blocks later, as demonstrated by the result that a planted value is recalled in 73% of cases (95% interval 66–79%), against 98% for the full-KV model on the same edited sequences.

Abstract

Long chain-of-thought decoding is memory-bound: the KV cache of the generated tokens grows linearly with the test-time compute being spent. Evicting cache entries does not fix this, because a model never trained on a cache with entries removed generates longer, often up to the length limit. We instead train the model to compress its own output. AbridgeKV compresses each completed block of generated text into the KV entries of a few learned pause tokens. These are extra input tokens with trained embeddings that produce no output. Once the pause tokens have encoded the block, AbridgeKV frees the block's original cache; subsequent tokens attend to the pause tokens' KV entries in its place. Three novel choices make this work: the pause tokens' KV entries take positions inside the block they replace, the cache budget is allocated across layers according to the measured value of each layer's entries, and compression operates on relatively large blocks. On 446 competition problems from AIME, HMMT, AMC, BRUMO, CMIMC and SMT with 32K-token generations, keeping a quarter of the generated cache matches an identically trained full-KV model in accuracy and output length; the strongest published evictor at a matched budget writes far longer, and the closest trained compressor, retrained on the same data, loses 10 points. Served in vLLM, the cache is 3.5x smaller per sequence and a saturated H100 finishes a problem 3.2x faster than with full KV.