AVSG: Accelerated Vectorized Sparse Gather for Efficient KV Cache Offload in Sparse-Attention LLM Serving

arXiv:2609.37538 · eess.SP · Submitted 2026-09-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "AVSG: Accelerated Vectorized Sparse Gather for Efficient KV Cache Offload in Sparse-Attention LLM Serving".

Dev: Here's the gist of the paper. Dynamic sparse attention reduces long-context attention computation by selecting only a subset of tokens, but still requires access to the full key-value (KV) cache, leaving serving memory-bound.

Rosa: AVSG addresses these challenges for DSA while preserving its exact token selections.

Dev: The core problem addressed is the efficiency of KV cache offloading in sparse attention serving.

Rosa: The paper introduces AVSG, an operator that tackles three critical challenges arising from offloading the KV cache to host memory for DeepSeek Sparse Attention (DSA).

Title and authors: Rosa: So we're looking at this paper now, "AVSG: Accelerated Vectorized Sparse Gather for Efficient KV Cache Offload in Sparse-Attention LLM Serving," and it’s about tackling the problem where dynamic sparse attention helps reduce computation but still leaves us stuck because we need the whole key value cache on host memory.

Dev: Right. It’s about how to make offloading that key value cache more efficient so we don't get bottlenecked by the host-to-device transfers during decoding, which is a big issue for real-time systems.

Taro: I'm wondering how this addresses the core problem of serving memory being tied up because of those large H2D transfers that happen every step.

Rosa: This paper points out three main challenges when you offload that key value cache to host memory in deepseek sparse attention: first, the cost of matching selected tokens against entries already in the host memory buffer, second, how unevenly those remaining host-to-device transfers get spread across different AI cores, and third, just keeping those frequently used key value entries in the limited HBM capacity.

Dev: Those three points sound like they’re exactly what we've been fighting with when trying to balance the load across our accelerator cores and keep things moving fast.

Taro: Specifically, I'm interested in how they tackle that costly matching of selected tokens against the entries retained in the HBM buffer, because that sounds like it could be a major latency killer during inference.

Rosa: AVSG introduces an operator called AVSG to handle those challenges by using vectorized hash matching instead of the older scalar dual-pointer matching we used before.

Dev: Vectorized hash matching, so instead of checking one by one, they’re using something that's much faster at finding reusable host memory slots for the selected tokens.

Taro: That sounds promising if it cuts down on lookup latency, especially when the context length and the number of selected tokens get really big.

Rosa: The paper says that vectorized hash matching is two point eight zero times faster when both the sequence length and the number of selected tokens are twenty forty eight, which means they can scale this constant time probing capability to raise the hit rate without a big penalty in matching time.

Dev: Constant time probing is good for loop rates because we don't want unpredictable spikes in latency based on how many tokens we’re dealing with.

Taro: So, if the lookup remains constant, what about the uneven distribution of those remaining host-to-device transfers across the accelerator cores? That sounds like another hurdle for real-time performance.

Rosa: They use something called an evenly partitioned multi core h2d transfer strategy to balance those cache misses across all AI cores.

Dev: Balancing those misses is key; if one core gets overloaded while others sit idle waiting for data, the whole system stalls, and that's not good for a real-time loop.

Taro: And what does that partitioning strategy actually achieve in terms of bandwidth compared to just assigning requests randomly or doing an equal top-k partition?

Rosa: They found it achieves up to thirty four point four four times the effective h2d bandwidth of request level assignment and three point forty eight times higher bandwidth than equal top-k partitioning.

Dev: Thirty four point four four times is substantial; that suggests they are really spreading those transfers out across the hardware effectively, regardless of the original request or how the cache hits are distributed.

Taro: That sounds like a significant improvement in how we manage those transfers, but what about keeping frequently accessed key value entries within our limited HBM capacity? That’s where we run into capacity issues when context gets long.

Rosa: They use lifetime-based buffer management for that, which is basically a policy where they retain frequently accessed key value entries across decoding steps and reclaim the less useful ones over time.

Dev: So it’s not just about immediate hits; it’s about aging the slots so that data we know we'll need again stays there for longer.

Taro: And what did they find regarding this retention policy on real requests? Does it actually work well in practice?

Rosa: On real requests, they found that an eight thousand entry buffer achieves a ninety four point eight three percent hit rate, which is about thirteen point two two percent above the reuse rate from the step before.

Dev: So they’re seeing a solid retention rate that's better than just relying on what happened in the immediately preceding step, which suggests this lifetime management is doing its job.

Taro: Now there's this feature about shared buffer reuse across multi token prediction tokens, which sounds like it could help with efficiency when we have sequences longer than what fits in one block.

Rosa: AVSG supports shared buffer reuse through two things: first, progressive reuse of matching information to avoid redundant host-to-device transfers within a single step, allowing later tokens to match a slot reserved by an earlier token.

Dev: That's smart; it means we don't have to re-transfer the same key value if the next token just happens to need that slot again.

Taro: And there’s also speculative entry invalidation, which marks selected indices as empty if they are speculative, so it prevents stale hits and forces a fresh host-to-device copy when that index is selected again.

Rosa: That way it ensures the system doesn't get stuck on old information that isn't relevant anymore, which keeps things clean even when we’re being speculative.

Dev: So they are trying to build a system where we reuse what we already have efficiently and only pay for the transfers that are actually needed, which is exactly what I want in a real-time loop.

Taro: Overall, this whole approach of AVSG seems focused on leveraging the overlap in selected key value entries across decoding steps as a first-class resource for serving efficiency.

Rosa: Exactly. The paper shows that AVSG reduces time per output token by thirty nine percent and increases output throughput by one point two seven times compared to the same offload layout without buffer reuse.

Dev: That’s a big gain in terms of latency reduction, cutting the time per output token down almost by half, and boosting throughput significantly while keeping that exact sparse attention layout.

Taro: So for someone listening who just cares about real-world performance, this means we can get much better throughput on our existing hardware setup with these kinds of offload layouts.

Rosa: The final conclusion is that AVSG effectively leverages overlap in selected key value entries across decoding steps as a first-class resource for serving efficiency by combining vectorized matching, balanced transfers, and lifetime residency.

Dev: It shows that efficient matching, balanced transfer handling, and effective host memory residency together cut time per output token by thirty nine percent and raise throughput one point two seven times on a real-world production dataset.

Taro: For my testing perspective, the paper confirms that AVSG's performance is superior to existing offload strategies because it isolates the access strategy by showing that reusing buffer hits and transferring only misses cuts time per output token by thirty nine percent to fifty five milliseconds and raises throughput one point two seven times.

Rosa: So we’re looking at a system where we don't just transfer everything selected every step, but we intelligently manage what stays in host memory and how those transfers get distributed across the cores.

Dev: The implication is that if you’re building a serving stack for sparse attention, prioritizing these access strategies over simple request level assignment can yield substantial performance improvements in real-world scenarios.

Taro: I just want to stress that the paper does have some limitations; they acknowledge that future directions include matrix level vectorization to leverage tensor cores, decoupled state update with latency hiding, and support for dynamic sparse attention patterns.

Rosa: So while AVSG is great for this specific case, we still have more work ahead in terms of moving towards even better utilization of the hardware resources like tensor cores.

Dev: Yeah, decoupling the state update from the data transfer sounds like a way to hide latency by overlapping those updates with other computation tasks.

Taro: I think that’s where we need to look next, because improving how we handle state updates could really help us squeeze out even more efficiency in these systems.

Rosa: So to wrap up on this paper, "AVSG: Accelerated Vectorized Sparse Gather for Efficient KV Cache Offload in Sparse-Attention LLM Serving," it shows that by using vectorized matching, balanced transfers, and lifetime residency, we can significantly reduce decoding overhead for key value cache offloading while maintaining the exact token selections.

Dev: That’s the main thing—preserving those exact token selections while making the serving memory less bound and improving throughput on real data.

Taro: I think this work is genuinely interesting because it shows how to manage the complexity of sparse attention serving in a way that directly impacts practical performance metrics like time per output token and overall throughput.

Rosa: It is a solid piece of research that gives us concrete numbers on how to improve KV cache offloading, and we should keep an eye on those future directions for matrix level vectorization.

The paper's summary: Rosa: So, this paper is about AVSG which tackles those three core problems when you try to offload your key value cache for sparse attention onto host memory: matching selected tokens against the buffer, uneven distribution of data transfers across cores, and keeping frequently used entries in limited HBM.

Dev: It sounds like they’re trying to make that whole process less painful for real-time serving loops.

Taro: Specifically, AVSG uses vectorized hash matching to replace the old scalar dual-pointer matching we used before.

Rosa: That's right, it's a constant time probing method that is two point eight zero times faster when the sequence length and the number of selected tokens are twenty forty eight.

Dev: Constant time probing is good for loop rates because it means the latency doesn't jump around unpredictably based on how many tokens we are dealing with.

Taro: That constant time capability lets them scale the hit rate up without having that proportional matching penalty we usually see in those systems.

Rosa: Then they have this evenly partitioned multi core h2d transfer strategy to balance those cache misses across all AI cores.

Dev: Balancing the miss transfers is key; if one core gets overloaded while others sit idle waiting for data, the whole system stalls and that's not good for a real-time loop.

Taro: And they claim this strategy achieves up to thirty four point four four times the effective h2d bandwidth of request level assignment and three point forty eight times higher bandwidth than equal top-k partitioning.

Rosa: That means they are spreading those transfers out across the hardware regardless of the original request or how the cache hits are distributed.

Dev: Thirty four point four four times is substantial; that suggests they’re really managing those transfers better, preventing capacity waste on cache hits and balancing core workloads.

Taro: They also use lifetime-based buffer management to retain frequently accessed key value entries across decoding steps and reclaim the less useful ones over time.

Rosa: It's not just about immediate hits; it’s about aging the slots so that data we know we'll need again stays there for longer.

Dev: And on real requests, they found an eight thousand entry buffer achieves a ninety four point eight three percent hit rate, which is about thirteen point two two percent above the reuse rate from the step before.

Taro: So they’re seeing a solid retention rate that's better than just relying on what happened in the immediately preceding step, which suggests this lifetime management is doing its job.

Rosa: Then they support shared buffer reuse across multi token prediction tokens through two things, first progressive reuse of matching information to avoid redundant host-to-device transfers within a single step.

Dev: That's smart; it means we don't have to re-transfer the same key value if the next token just happens to need that slot again.

Taro: And there’s also speculative entry invalidation, which marks selected indices as empty if they are speculative, so it prevents stale hits and forces a fresh host-to-device copy when that index is selected again.

Rosa: That way it ensures the system doesn't get stuck on old information that isn't relevant anymore, which keeps things clean even when we’re being speculative.

Dev: So they are trying to build a system where we reuse what we already have efficiently and only pay for the transfers that are actually needed, which is exactly what I want in a real-time loop.

Taro: Overall, this whole approach of AVSG seems focused on leveraging the overlap in selected key value entries across decoding steps as a first-class resource for serving efficiency.

Rosa: Exactly. The paper shows that AVSG reduces time per output token by thirty nine percent and increases output throughput by one point two seven times compared to the same offload layout without buffer reuse.

Dev: That’s a big gain in terms of latency reduction, cutting the time per output token down almost by half, and boosting throughput significantly while keeping that exact sparse attention layout.

Taro: So for someone listening who just cares about real-world performance, this means we can get much better throughput on our existing hardware setup with these kinds of offload layouts.

Rosa: The final conclusion is that AVSG effectively leverages overlap in selected key value entries across decoding steps as a first-class resource for serving efficiency by combining vectorized matching, balanced transfers, and lifetime residency.

Dev: It shows that efficient matching, balanced transfer handling, and effective host memory residency together cut time per output token by thirty nine percent to fifty five milliseconds and raise throughput one point two seven times on a real-world production dataset.

Taro: For my testing perspective, the paper confirms that AVSG's performance is superior to existing offload strategies because it isolates the access strategy by showing that reusing buffer hits and transferring only misses cuts time per output token by thirty nine percent to fifty five milliseconds and raises throughput one point two seven times.

Rosa: So we’re looking at a system where we don't just transfer everything selected every step, but we intelligently manage what stays in host memory and how those transfers get distributed across the cores.

Dev: The implication is that if you’re building a serving stack for sparse attention, prioritizing these access strategies over simple request level assignment can yield substantial performance improvements in real-world scenarios.

Taro: I just want to stress that the paper does have some limitations; they acknowledge that future directions include matrix level vectorization to leverage tensor cores, decoupled state update with latency hiding, and support for dynamic sparse attention patterns.

Rosa: So while AVSG is great for this specific case, we still have more work ahead in terms of moving towards even better utilization of the hardware resources like tensor cores.

Dev: Yeah, decoupling the state update from the data transfer sounds like a way to hide latency by overlapping those updates with other computation tasks.

Taro: I think that’s where we need to look next, because improving how we handle state updates could really help us squeeze out even more efficiency in these systems.

The paper's improvements: Rosa: So, we're talking about how AVSG actually works to fix those three bottlenecks we mentioned earlier: matching speed, transfer balancing, and data retention in limited memory.

Dev: It sounds like they’re showing us a concrete way to improve the efficiency of offloading that key value cache for sparse attention.

Taro: They are specifically focusing on how AVSG manages the metadata structure itself, which is organized in a Hash-State table residing entirely in on-chip memory that supports SIMD instructions.

Rosa: That table uses a packed word to identify any slot, like if you see an m equals thirteen give you S equals eight thousand one hundred ninety two, and it stores the selected index, the lifetime of that entry, and an invalid flag.

Dev: So they’re building this metadata structure right in the fast on-chip memory so we don't have to constantly fetch everything from slower host memory just to look up what we need.

Taro: The probe primitive executes at most R plus one bucket comparisons per selected index, which keeps the lookup cost at constant time regardless of how big K or S gets.

Rosa: That constant time operation means the lookup cost is big O of one, meaning it doesn't matter if you have twenty thousand keys or two hundred thousand keys; it’s always fast.

Dev: That’s a huge win for my loop rate because we can rely on that lookup being consistent, not fluctuating based on the size of the context we’re processing.

Taro: And they also introduce lifetime-based buffer management, which means they retain frequently accessed key value entries across decoding steps and reclaim the less useful ones over time.

Rosa: They use a policy where slots are refreshed on hit and become reusable once they expire after more than L unit decays.

Dev: That’s not just about immediate hits; it’s about aging the slots so that data we know we'll need again stays there for longer in that limited HBM capacity.

Taro: On real requests, they found an eight thousand entry buffer achieves a ninety four point eight three percent hit rate, which is about thirteen point two two percent above the reuse rate from the step before.

Rosa: So they’re seeing a solid retention rate that's better than just relying on what happened in the immediately preceding step.

Dev: That retention level suggests this lifetime management is doing its job in a practical sense for keeping useful data resident.

Taro: Then they have shared buffer reuse across multi token prediction tokens through two things, including progressive reuse of matching information to avoid redundant host-to-device transfers within a single step.

Rosa: That means later tokens can match a slot reserved by an earlier token, which stops us from retransferring the same key value if the next token just needs that slot again.

Dev: That’s smart; it means we don't have to re-transfer the same key value if we reuse a reservation, which saves bandwidth.

Taro: And they also have speculative entry invalidation, which marks selected indices as empty if they are speculative, so it prevents stale hits and forces a fresh host-to-device copy when that index is selected again.

Rosa: That way it ensures the system doesn't get stuck on old information that isn't relevant anymore, even when we’re being speculative.

Dev: So they’re trying to build a system where we reuse what we already have efficiently and only pay for the transfers that are actually needed.

Taro: Overall, this whole approach of AVSG seems focused on leveraging the overlap in selected key value entries across decoding steps as a first-class resource for serving efficiency.

Rosa: Exactly. The paper shows that AVSG reduces time per output token by thirty nine percent and increases output throughput by one point two seven times compared to the same offload layout without buffer reuse.

Dev: That’s a big gain in terms of latency reduction, cutting the time per output token down almost by half, and boosting throughput significantly while keeping that exact sparse attention layout.

Taro: So for someone listening who just cares about real-world performance, this means we can get much better throughput on our existing hardware setup with these kinds of offload layouts.

Conclusion: Rosa: So, we’re wrapping up on AVSG: Accelerated Vectorized Sparse Gather for Efficient KV Cache Offload in Sparse-Attention LLM Serving. It boils down to using vectorized matching, balanced transfers, and lifetime residency to cut time per output token by thirty nine percent and boost throughput by one point two seven times.

Dev: That’s a significant practical result. It shows that managing the access strategy is more important than just doing a simple request level assignment when you're dealing with sparse attention serving.

Taro: I think what this means for autonomy research is that it gives us a much better safety case for real-time systems because we have these concrete numbers on how gracefully the system recovers from data transfer bottlenecks.

Rosa: It definitely does. By making those host memory transfers smarter, you're reducing the time spent waiting for data to move around the chip.

Dev: I agree, and that reduction in latency is exactly what we need for real-time loops to stay stable under heavy load.

Taro: It’s good because it proves that overlap across decoding steps is a first-class resource we can actually exploit in a practical serving stack.

Rosa: So the paper validates the idea that those overlaps are important for efficiency when dealing with dynamic sparse attention patterns.

Dev: We need to keep an eye on those future directions, especially matrix level vectorization and decoupling state updates from data transfers to hide latency further.

Taro: Yeah, improving how we handle state updates could really help us squeeze out even more efficiency in these systems down the road.

Huawei Technologies Co., Ltd.

eess.SP

Submitted: 2026-09-29

Updated: 2026-10-31

Comments: 20 pages, 7 figures, 4 tables

Code: https://github.com/hicann/cann-recipes-infer

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Dynamic sparse attention reduces long-context attention computation by selecting only a subset of tokens, but still requires access to the full key-value (KV) cache, leaving serving memory-bound.

Key concepts

Vectorized Hash Matching
This technique replaces slow scalar matching with fast vectorized hash probing. It allows the system to quickly find reusable memory slots using SIMD instructions, making lookups much faster and enabling the system to retain more useful cache entries without increasing lookup time significantly.
Evenly Partitioned Multi-Core H2D Transfer
This strategy spreads KV data transfers across multiple AI cores. It ensures that cache misses are distributed evenly, preventing one core from being overwhelmed while others wait. This balances the workload and maximizes the effective bandwidth of the host-to-device (H2D) transfers.
Lifetime-Based Buffer Management
This policy manages which KV entries stay in fast memory based on how often they are used. Entries that are accessed frequently are refreshed and kept longer, while less useful ones expire. This keeps the most relevant data readily available for decoding steps.
Shared Buffer Reuse Across MTP Tokens
AVSG allows tokens from subsequent Multi-Token Prediction (MTP) steps to reuse memory slots reserved by earlier tokens. This reduces redundant data transfers by marking speculative indices as empty, ensuring that later tokens can efficiently utilize existing reservations.

Terminology

Summary

Dynamic sparse attention reduces long-context attention computation by selecting only a subset of tokens, but still requires access to the full key-value (KV) cache, leaving serving memory-bound. p. 0 AVSG addresses these challenges for DSA while preserving its exact token selections. p. 0

The core problem addressed is the efficiency of KV cache offloading in sparse attention serving. p. 0

The paper introduces AVSG, an operator that tackles three critical challenges arising from offloading the KV cache to host memory for DeepSeek Sparse Attention (DSA). p. 0 These challenges include costly matching of selected tokens against HBM buffer entries, uneven distribution of H2D transfers across accelerator cores, and retaining frequently accessed KV entries within limited HBM capacity. p. 0 AVSG utilizes vectorized hash matching to identify reusable HBM-buffer slots and reserve slots for entries requiring transfer. p. 0

Vectorized Hash Matching Accelerates Lookup Latency. p. 8

AVSG replaces scalar dual-pointer matching with vectorized hash probing, which is 2. p. 180× faster at S = K = 2048. p. 1 This constant-time probing capability allows the hash table to be scaled to raise the hit rate without a proportional matching penalty. p. 4 The latency growth with respect to K and S is sublinear over tested ranges, making it practical to retain more reusable KV entries without a comparable increase in lookup time. p. 7

Evenly Partitioned Multi-Core H2D Transfer Balances Workloads. p. 1

The method employs an Evenly Partitioned Multi-Core H2D Transfer strategy to balance cache-miss KV transfers across AI-Cores. This approach achieves up to 34. p. 144× and 3.48× the effective H2D bandwidth of request-level assignment and equal top-K partitioning, demonstrating the benefit of spreading a request’s transfers across cores. p. 7 The resulting transfer workloads for any two cores differ by at most one KV entry, regardless of the original request or cache-hit distribution. p. 5

Device Cache Management Preserves Frequently Accessed Entries. p. 2

Lifetime-based buffer management is used to retain frequently accessed KV entries in the device buffer across decoding steps and reclaim less useful ones. p. 1 This policy implements LRU-like aging based on access recency, where slots are refreshed on hit and become reusable once they expire after more than L unit decays. p. 6 On real requests, an 8K-entry buffer achieves a 94. p. 183% hit rate, which is 13. p. 022% above the adjacent-step reuse rate. p. 1

Shared Buffer Reuse Across MTP Tokens Enhances Efficiency. p. 1

AVSG supports shared-buffer reuse across Multi-Token Prediction (MTP) tokens through two mechanisms. p. 6 The first is the progressive reuse of matching information to avoid redundant KV transfers within a step, allowing later tokens to match a slot reserved by an earlier token. p. 6 The second is speculative-entry invalidation, which marks selected indices as empty if they are speculative, preventing stale hits and requiring a fresh H2D copy when the index is reselected. p. 6

End-to-End Serving Performance Demonstrates Significant Gains. p. 8

On a serving stack processing a real-world production dataset, AVSG reduces time per output token by 39% and increases output throughput by 1. p. 027× relative to the same KV offload layout without HBM-buffer reuse. p. 0 This performance gain is observed in the end-to-end evaluation, showing that AVSG cuts TPOT by 39% and sustains 1. p. 827× the output throughput. p. 0

The final conclusion is that AVSG effectively leverages overlap in selected KV entries across decoding steps as a first-class resource for serving efficiency. p. 8

AVSG reduces the decoding overhead of KV offloading for DSA while preserving exact token selections, showing that the overlap in selected KV entries across decoding steps is a first-class resource for serving efficiency. p. 8 Its three mechanisms each remove one bottleneck: vectorized hash matching accelerates lookup 2. p. 880× over scalar dual-pointer matching, balanced miss-only transfer raises effective H2D bandwidth by up to 34. p. 844× over request-level assignment, and lifetime-based residency attains a 94. p. 883% hit rate on real requests, with shared slot reservations extending reuse across MTP tokens (61. p. 887% even from an empty buffer under MTP-3). p. 7 Matrix-level vectorization, latency-hiding state updates, and multi-tier memory hierarchies remain open directions. p. 8

The paper provides a comprehensive evaluation across various configurations to validate the mechanisms. p. 6

The component experiments isolate each mechanism on a single NPU. p. 6 The end-to-end serving comparison runs the full 5P1D serving stack, where AVSG isolates the access strategy by comparing it against request-level assignment and equal top-K partitioning. p. 6 The final results show that AVSG cuts TPOT by 39% and sustains 1. p. 827× the output throughput of the same offload layout without reuse. p. 0

The evaluation confirms that AVSG's performance is superior to existing offload strategies.

No Offload attains the lowest TPOT (43 ms, no H2D on the decode path) but is capacity-capped. p. 8 KV Offload removes this cap but pays 91 ms per step transferring the full selection. p. 8 AVSG shares KV Offload’s exact layout, and the gap isolates the access strategy, showing that reusing buffer hits and transferring only misses cuts TPOT 39% to 55 ms and raises throughput 1. p. 827×.

The paper details the implementation of AVSG's complex data structures.

AVSG organizes its metadata in a Hash-State table residing entirely in on-chip memory, supporting SIMD Flynn instructions. p. 3 This table uses a packed word to identify any slot (e. p. 4g., m=13 gives S=8192) and stores the selected index, lifetime, and invalid flag. p. 4 The Probe primitive executes at most R+1 bucket comparisons per selected index, making the lookup cost O(1) SIMD instructions regardless of K and S. p. 4

The paper concludes with future directions for further research. p. 8

Future directions include matrix-level vectorization to leverage tensor cores, decoupled state update with latency hiding to overlap state record updates with Stage Two data transfer or attention computation, and support for dynamic sparse attention patterns. p. 8 These enhancements could further improve the efficiency of sparse attention serving systems beyond DeepSeek Sparse Attention. p. 9

**The paper is a preprint available on arXiv. p. 9

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Vectorized Hash Matching for Lookup Acceleration This replaces scalar dual-pointer matching with vectorized hash probing, which is 2.80 times faster at S = K = 2048 and decouples device buffer capacity from selection size without a proportional matching penalty. This allows the system to retain larger reusable KV entries without a comparable increase in lookup time.

  2. Evenly Partitioned Multi-Core H2D Transfer This mechanism balances cache-miss KV transfers across AI-Cores, achieving up to 34.44 times the effective H2D bandwidth of request-level assignment and 3.48 times higher bandwidth than equal top-K partitioning. This ensures that every issued transfer corresponds to a real miss KV, preventing transfer capacity waste on cache hits and balancing core workloads regardless of the original request or cache-hit distribution.

  3. Lifetime-Based Buffer Management for Data Retention This policy combines per-slot aging with hit-triggered lifetime refresh to retain frequently accessed KV entries across decoding steps, achieving an 8K entry buffer hit rate of 94.83% on real requests and extending reuse across multi-token prediction iterations by 61.87%. This prevents the uneven distribution of the remaining H2D transfers by ensuring that useful data is not repeatedly evicted while reclaiming less useful entries.

  4. Shared-Buffer Reuse Across MTP Tokens This feature allows later tokens in a multitoken prediction (MTP) iteration to reuse slot reservations made by earlier tokens, achieving 61.87% reuse even from an empty buffer under MTP-3. This reduces the total number of required H2D copies by ensuring that later tokens reuse those reservations and reduce the total transfer volume.

  5. End-to-End Serving Performance Gains This integrated approach cuts time per output token (TPOT) by 39% and increases output throughput by 1.27 times relative to the same KV offload layout without HBM-buffer reuse in a real-world production dataset. This demonstrates that efficient matching, balanced transfer, and effective HBM residency together yield significant latency reduction and high throughput in serving stacks.

Sources

Related papers