SPLASH: A Virtualized HBM-HBF Hierarchy with Flash-Aware Sparse Attention for Long-Context Inference

summary

The gist

SPLASH presents an algorithm and architecture co-design that virtualizes the KV cache across HBM and HBF for long-context inference The key-value (KV) cache has become the dominant consumer of memory…

In short

SPLASH virtualizes a KV cache across High-Bandwidth Memory (HBM) and High-Bandwidth Flash (HBF) for long-context LLM inference. It achieves significant throughput gains by co-designing sparse attention with HBF's page structure, allowing GPUs to access combined memory capacity efficiently. This enables serving very long contexts while maintaining high accuracy.

Key concepts

Memory Hierarchy Virtualization
SPLASH treats HBM and HBF as one single logical memory space. Data enters the system via the fast HBM tier, and older data is moved to the larger HBF tier. The GPU interacts with this unified view, hiding the physical differences between DRAM and flash from the model.
Sparse Attention Co-design
The system designs attention mechanisms around how data is organized in HBF. It uses this structure to decide which specific KV cache pages to retrieve during decoding, rather than reading everything. This translates the need for long context into efficient, targeted reads from the flash.
Sparse KV Memory Controller
This custom hardware component sits beneath the GPU memory interface. It handles complex tasks like translating logical requests into physical addresses within HBM and HBF, performing sparse selection based on query needs, and scheduling flash reads to minimize latency.
Plane-Level Parallelism
HBF is organized into planes for efficient access. SPLASH designs its sparse attention around these planes to ensure that retrieval operations are distributed evenly across the flash structure. This allows multiple pages to be accessed simultaneously, speeding up the process and bounding read latency.

Terminology used across episodes

This episode discusses

The paper

SPLASH: Co-Designing Flash-Aware Sparse Attention with a Virtualized HBM-HBF Hierarchy · Read on arXiv

National University of Singapore

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "SPLASH: A Virtualized HBM-HBF Hierarchy with Flash-Aware Sparse Attention for Long-Context Inference".

Title and authors: Tom: To summarize what "SPLASH: A Virtualized HBM-HBF Hierarchy with Flash-Aware Sparse Attention for Long-Context Inference" is about, it’s taking the memory problem in long context inference and solving it by using virtualized HBM and flash for a unified memory pool.

Jane: It really shows that when we’re dealing with massive context lengths, the solution isn't just building bigger memory; it’s about designing the entire memory structure to work better with how we ask questions from the model.

Lu: Precisely; the SPLASH architecture is a specific solution for the key-value cache being the dominant consumer of memory in LLM serving systems as context lengths and concurrency grow.

Tom: The core idea is that HBM gives you bandwidth but limited capacity, and flash offers much higher density, so they stack them to get both speed and scale without relying on offpackage memory.

Jane: They also focus heavily on how the model asks questions—the sparse attention part—to make sure we only fetch exactly what's needed from that unified pool.

Lu: So the paper is detailing a system where you have this unified pool, and the way you ask for data from it is smart enough to respect the physical constraints of both HBM and HBF.

Tom: It’s about creating this single logical memory so the GPU doesn't have to worry about which physical tier holds which piece of context.

Jane: That's a big deal because it simplifies the whole serving system, making it more flexible for different kinds of long-context tasks.

The paper's summary: Tom: Let’s look at the specific improvements they propose in "SPLASH: A Virtualized HBM-HBF Hierarchy with Flash-Aware Sparse Attention for Long-Context Inference". They focus on three main areas.

Lu: First, the virtualizing of the KV cache across HBM and HBF as a single logical memory pool, allowing GPUs to access combined capacity without being exposed to their distinct physical organizations or access semantics.

Jane: That means we get more total context length available per GPU without having to proportionally increase the bandwidth needed from just one memory type.

Tom: Then they co-design sparse attention around HBF’s page granularity and plane-level parallelism, translating that sparsity into efficient HBF retrieval within interactive latency.

Jane: That translates to the system reading only selected pages from the longer history while overlapping this selection scan with dense attention over the HBM window.

Lu: They also implemented a query-driven address translation, where the query determines this mapping, so you’re not just reading sequentially; you’re reading what matters for that particular token.

Tom: That means the same GPU read can resolve to a different flash page at every single decode step, which is crucial for interactive performance.

Jane: And they also have plane-balanced selection across those flash planes, which bounds each plane to k page reads and ensures that retrieval completes in at most k sensing rounds while still respecting the total page budget.

The paper's improvements: Tom: So we’ve looked at "SPLASH: A Virtualized HBM-HBF Hierarchy with Flash-Aware Sparse Attention for Long-Context Inference", which is this paper about virtualizing the HBM and HBF hierarchy using flash-aware sparse attention for long context inference.

Meng: I still have my question about deployment—does this whole setup require you to redesign your entire server rack just to use it?

Tom: They showed that with query-driven address translation and plane-balanced selection across the flash planes, you can get gains ranging from three point five times up to eleven point four times over your current baselines.

Lu: And they kept the accuracy pretty high, retaining about ninety six percent on rerler and ninety nine percent on longbench, which is solid performance for this kind of memory management overhead.

Meng: I’m still focused on the engineering reality—how do we know this stuff actually holds up when we run real traffic that isn't perfectly distributed?

Jane: The paper modeled the flash tier in single level cell mode with an assumed endurance of one hundred zero five program erase cycles, and they show that retaining completed regions for thirty minutes keeps the modeled lifetime beyond five years.

Lu: The SPLASH architecture provides a blueprint for how to use flash as a high capacity tier alongside HBM for next-generation inference systems, provided you design your attention mechanism around that flash granularity.

Conclusion: Tom: So we’ve looked at SPLASH: A Virtualized HBM-HBF Hierarchy with Flash-Aware Sparse Attention for Long-Context Inference, which is this paper about virtualizing the HBM and HBF hierarchy using flash-aware sparse attention for long context inference.

Jane: It really shows that when you’re dealing with massive context lengths, the solution isn't just building bigger memory; it’s about designing the entire memory structure to work better with how you ask questions from the model.

Lu: Precisely; this SPLASH architecture is a specific solution for the key-value cache being the dominant consumer of memory in LLM serving systems as context lengths and concurrency grow.

Meng: I still have my question about deployment—does this whole setup require you to redesign your entire server rack just to use it?

Jane: It means they’ve done the heavy lifting on the architectural modeling, showing a path forward for how you structure memory to handle massive context scaling efficiently.

Tom: They showed that with query-driven address translation and plane-balanced selection across the flash planes, you can get gains ranging from three point five times up to eleven point four times over your current baselines.

Lu: And they kept the accuracy pretty high, retaining about ninety six percent on rerler and ninety nine percent on longbench, which is solid performance for this kind of memory management overhead.

Meng: I still think we need to see more real traffic results before we can confidently say this is ready for production pipelines at scale.

Jane: The paper modeled the flash tier in single level cell mode with an assumed endurance of one hundred zero five program erase cycles, and they show that retaining completed regions for thirty minutes keeps the modeled lifetime beyond five years.

Tom: So they’ve got numbers on both speed and how long this setup can realistically last under heavy use, which is what I like to see when you look at deployment feasibility.

Lu: The SPLASH architecture provides a blueprint for how to use flash as a high capacity tier alongside HBM for next-generation inference systems, provided you design your attention mechanism around that flash granularity.

Jane: It’s clear that the key to long context inference isn't just bigger memory, but how you structure that memory hierarchy so it can work with sparse attention efficiently.

Meng: I still think we need to see more real traffic results before we can confidently say this is ready for production pipelines at scale.

Tom: We’ll keep an eye on those real-world benchmarks and see if the performance holds up when the load gets heavy.

Lu: Next up, we’re looking at how formal verification of out-of-order multiprocessors is addressing fundamental architectural issues in modern computing.

More episodes

← Home