AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
summary
The gist
AMMA presents a multi-chiplet memory-centric architecture that replaces GPU compute dies with PNM-enabled HBM cubes to achieve low-latency long-context attention serving by doubling available memory…
In short
AMMA replaces GPU compute dies with HBM-PNM cubes to double memory bandwidth for serving long-context attention at low latency. It achieves 15.5x lower attention latency and 6.9x lower energy consumption than the NVIDIA H100 by using a novel two-level hybrid parallelism and optimized communication flows.
Key concepts
- HBM-PNM Cubes
- AMMA replaces traditional GPU compute dies with cubes made of High Bandwidth Memory (HBM) and Processing-in-Memory (PNM) technology. This change roughly doubles the available memory bandwidth, which is crucial for handling the massive data requirements of long attention contexts, allowing the system to serve requests much faster than standard GPU setups.
- Two-Level Hybrid Parallelism
- This strategy maps attention tasks across multiple HBM cubes in two stages. Level 1 handles parallelism across different cubes, while Level 2 partitions the key-value (KV) cache within each cube by splitting it along the sequence dimension. This confines data movement to local neighborhoods, minimizing expensive communication between chips.
- Reordered Collective Communication Flow
- AMMA redesigns how data is synchronized across the chip. Instead of traditional AllReduce operations which involve high global synchronization costs, AMMA uses a sequence like ReduceScatter followed by a point-to-point Reduce to the destination cube. This significantly reduces communication overhead and latency during collective operations.
- LLC-Free Architecture
- The architecture is designed to be LLC (Last Level Cache) free, meaning compute resources are sized precisely to match the raw HBM bandwidth. This eliminates the performance gap often seen in GPUs where compute units sit idle waiting for data, ensuring that every part of the system contributes effectively to memory-bound attention workloads.
Terminology used across episodes
This episode discusses
- AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving · Paper Radio
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
- Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
- Challenges and Research Directions for Large Language Model Inference Hardware
- KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
- SCALE-Sim: Systolic CNN Accelerator Simulator
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Kimi K2: Open Agentic Intelligence
- Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding
- Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training
- Qwen3 Technical Report
- MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
The paper
AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving · Read on arXiv
Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro
University of California, San Diego University · Columbia University
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving".
Jane: Here's the gist of the paper. AMMA presents a multi-chiplet memory-centric architecture that replaces GPU compute dies with PNM-enabled HBM cubes to achieve low-latency long-context attention serving by doubling available memory bandwidth and introducing novel parallelism and communication optimizations The Core Problem Addressed Existing LLM serving systems are built around a GPU-centric paradigm, which is fundamentally mismatched with the memory-bound nature of decode-phase attention, inflating serving latency while wasting power and die area on idle compute units The problem is compounded as reasoning and agentic workloads push context lengths toward one million tokens, making attention latency the primary user-facing bottleneck The GPU’s compute-rich architecture is fundamentally mismatched with the bandwidth-bound nature of decode attention, which becomes critical as reasoning and agentic workloads push context lengths into the millions Architectural Innovation AMMA replaces GPU compute dies with HBM-PNM cubes, roughly doubling the available memory bandwidth to better serve memory-bound attention workloads To translate this bandwidth into proportional performance gains, AMMA introduces (i) a logic-die microarchitecture that fully exploits per-cube internal bandwidth for decode attention under a minimal power and area budget (ii) a two-level hybrid parallelism scheme, and (iii) a reordered collective flow that reduces intra-chip die-to-die communication overhead AMMA connects 16 HBM dies via intra-package D2D links to form a single chip Parallelism and Communication Strategy AMMA employs two specific strategies to manage parallelism and communication overhead (P1) Many small SAs over few large ones, where the dimension M is tiny, so a TPU-style 128×128 array would leave >87% of PEs idle (P2) LLC-free architecture, which sizes compute to match raw HBM bandwidth, naturally eliminating the gap between compute and memory bandwidth (P3) Two-level crossbar for data sharing, replacing the LLC with a two-level crossbar that broadcasts shared inputs and collects partial outputs across the 96 SAs Performance Gains and Analysis Evaluations show that AMMA achieves 15.5× lower attention latency and 6.9× lower energy consumption compared with the NVIDIA H100 AMMA achieves the highest speedup in every configuration: 12.0–16.3× over H100 at BS=1, sustaining The results on GQA models show that AMMA outperforms Rubin by 2.8× at 1M tokens AMMA stays below 1500,W—1.4× lower than Rubin and 2.
Title and authors: Tom: So we're looking at this paper called "AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving" <ref:2604.26103#pg1,AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context>. It sounds like they're trying to fix the huge problem with how we serve large language models, specifically when the context window gets really long.
Jane: Exactly. Right now, most of our systems rely on these massive GPU compute dies, and that architecture isn't really suited for what decode phase attention demands because it’s so memory bound, which means the time it takes to get data from memory is slowing everything down.
Lu: So the core idea here is replacing those traditional GPU compute dies with HBM cubes that have built-in Processing-in-Memory or PNM capabilities. They are aiming to roughly double the available memory bandwidth.
Meng: Doubling bandwidth sounds promising for a memory bottleneck, but I worry about how much compute we can actually fit onto these new structures without wasting power and area on things that don't need it.
Tom: Well, the paper says they address this by introducing a specific logic-die microarchitecture that fully exploits that per-cube internal bandwidth for decode attention while keeping the power and area budget minimal.
Jane: That sounds like they're trying to make each memory cube a dedicated accelerator for attention, which is a big shift from how GPUs handle it.
Lu: They also propose a two-level hybrid parallelism scheme and a reordered collective flow that cuts down on the communication overhead between these chiplets.
Meng: Communication overhead is always the hidden killer when you connect sixteen different pieces together; if you can reduce that, we might see some real gains in deployment costs <ref:2604.26103#pg2>.
Tom: The paper details how they map the attention workload across these cubes, specifically by dividing the key value cache into cubes and splitting it along the sequence dimension within each group to keep data movement local.
Jane: It sounds like they're trying to manage that massive context window, which is pushing towards one million tokens, by making sure that most of the work happens right where the data lives.
Lu: They use a tiling methodology that partitions both the sequence dimension and potentially splits the key dimension across systolic arrays to make sure you aren't leaving any of those processing units idle.
Meng: How does that tiling strategy actually help with latency? Does it just mean less waiting time for data?
Tom: It aims to ensure that "the numerator is fixed by the problem size and independent of tiling choices" while still minimizing idle systolic arrays, which is a smart way to balance utilization against communication.
Title and authors: Jane: So, what's the big payoff they claim from this whole setup? What are we actually looking at in terms of speed and efficiency when we compare it to current systems?
Tom: Evaluations show that AMMA achieves fifteen point five times lower attention latency and six point nine times lower energy consumption compared to the NVIDIA H100.
Meng: Fifteen point five times lower latency is significant, but what about the energy cost? That’s usually a major concern when we talk about deploying models at scale.
Jane: The paper claims that this architecture consistently achieves five point six to six point six times higher energy efficiency, measured in tokens per joule compared to the H100, even when dealing with very long contexts up to one million tokens.
Lu: And on top of that, they report a stable two point six to three point one times advantage over Rubin at one million tokens for Grouped-Query Attention models.
Tom: It’s interesting because they aren't just aiming for speed; they are optimizing the entire serving stack by combining these architectural changes with a reordered collective communication flow.
Jane: That reordering of how data is collected sounds like another layer to make sure that even though you have many pieces working together, the synchronization doesn't become a bottleneck.
Lu: They specifically redesign the collective communication flow to reduce both the number of collective operations and the global synchronization overhead, replacing intra-group allreduce with just a reducescatter and cross-group allreduce with a point-to-point reduce to one destination cube.
Meng: So they are trying to minimize those costly chiplet communications that happen when data has to travel across the package interconnects.
Tom: Exactly, and this is happening on their sixteen HBM dies connected via intra-package D2D links, forming a single chip structure <ref:2604.26103#pg2>.
Jane: For someone just running inference, what does this mean practically in terms of how fast they can handle a complex reasoning task with a huge context?
Lu: It means that instead of waiting for the massive GPU to process everything sequentially, the system is structured so that the attention work happens much more locally and in parallel across these specialized memory units.
Meng: So, if I'm looking at deployment cost, is this just another piece of silicon we have to buy? How does this change the hardware footprint?
Tom: The goal was to keep the power and area budget minimal for the logic die while maximizing throughput by using that internal bandwidth efficiently on each cube.
Jane: It seems like they're tackling the core mismatch between GPU compute and memory bandwidth directly with this new architecture.
Lu: And one limitation they mention is that this entire setup is tailored specifically for low-latency serving of attention, so it’s not necessarily designed to be a general-purpose training platform.
Title and authors: Meng: That makes sense; if you're only focusing on inference speed and not massive pre-training throughput, that narrows the scope considerably.
Tom: They also say they are exploring the hardware design space by sweeping per-cube compute throughput and inter-die link bandwidth to give practical guidance for hardware designers.
Jane: It sounds like they aren't just presenting a finished chip; they're giving designers a map on how to tune these components for their specific needs.
Lu: And one of the key methods is the two-level hybrid parallelism, which maps the key value heads to cube groups using tensor parallelism at level one, and then splits the KV cache by sequence within each group using context parallelism at level two.
Meng: So they are layering parallelization on top of each other to confine bulk data movement into local neighborhoods.
Tom: That confinement is what allows them to achieve that latency reduction, by limiting the amount of data that needs to move across the chip in any single step.
Jane: And what about how they handle those input projections and output projections, the QKV and WOs? Does this architecture change how those fundamental matrix multiplications are done?
Lu: They focus heavily on making sure that within each HBM cube, the PNM unit is a fully functional accelerator integrat, which suggests they're optimizing the operations happening right inside that memory structure.
Meng: That sounds like deep integration, which usually means you have to be very specific about what kind of workload it’s optimized for.
Tom: They are specifically targeting decode attention performance under minimal power constraints, so the optimization is highly targeted toward that specific bottleneck.
Jane: So, to wrap up this paper on "AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving," they've shown a way to move away from being purely compute-centric towards something memory-centric for very large context needs <ref:2604.26103#pg1,AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context>.
Lu: The implication is that if we can effectively integrate PNM accelerators into the logic dies of HBM cubes and pair them with a reordered communication flow, we can serve much longer contexts with significantly lower latency than current GPU setups.
Meng: For us at the startup level, it tells us that if you focus on optimizing the memory access patterns and reducing inter-chip movement instead of just throwing more compute power at it, you might find efficiency gains that are more sustainable for production.
Tom: It’s about finding a better fit for the problem, not just brute-forcing the solution with bigger chips.
Jane: So, we’re looking at a system where the memory itself is doing some of the heavy lifting for attention, which is a really interesting direction to explore for next generation LLM serving.
The paper's summary: Tom: So to recap, AMMA is this new architecture that swaps out the usual GPU compute dies for these HBM cubes that have processing right inside them, aiming for really low latency when serving massive language model contexts.
Jane: Yeah, it’s basically moving away from the standard GPU setup because that setup just isn't good enough anymore when you push context lengths to one million tokens and attention becomes the main bottleneck.
Lu: The core innovation is doubling the available memory bandwidth by using these HBM-PNM cubes, which lets them serve those memory-bound attention workloads much better than a standard GPU like an H100.
Meng: But I always worry about the power and area cost for all these new structures; how does this actually fit into a real production environment without just making the chip bigger?
Tom: The authors tackled that by designing a specific logic die microarchitecture that uses only what's necessary to exploit that internal memory bandwidth for attention, keeping the power and area footprint very small.
Jane: So they aren't just adding more compute; they are making sure every part of the memory structure is working hard specifically on decoding attention under a tight budget.
Lu: They also introduced this two-level hybrid parallelism scheme, which splits the key value cache across these cubes and then further partitions it along the sequence dimension within those groups.
Tom: That level two split is clever because it confines the data movement so that most of the work stays local to a small neighborhood on the chip.
Jane: And they reordered how communication happens, cutting down on those slow transfers between different parts of the chip by using a two-level crossbar for sharing.
Meng: Cutting down communication overhead is key if you're dealing with sixteen separate dies connected together; that inter-chip talk can become the real killer at scale.
Tom: They specifically redesigned the collective flow, replacing things like an allreduce across the whole chip with just a reduce scatter and a point-to-point reduce to one destination cube.
Jane: So what does this actually mean for someone who just needs to run these massive models for inference? It means less waiting time, significantly lower energy use, and better performance even when the context window is huge.
Lu: The results show they hit fifteen point five times lower attention latency and six point nine times lower energy consumption compared to the NVIDIA H100 chip.
Tom: And on top of that, they report that AMMA stays below a power draw of one thousand five hundred watts, which is nearly half the power usage of the H100 under similar conditions.
Jane: That energy efficiency gain is huge because it means you can run these large language models for much longer periods without needing massive cooling infrastructure.
Meng: A six point nine times lower energy consumption sounds impressive when you look at total operational costs, but what about the context length itself? They test up to one million tokens.
Lu: The performance gains are sustained even at that one million token mark, and on Grouped-Query Attention models, AMMA beats Rubin by two point eight times at that scale.
Tom: It’s not just a theoretical number; they show this speedup in every single configuration they tested against the H100.
Jane: So essentially, AMMA shows that for serving attention at massive scales, we might need to stop thinking purely in terms of brute-force compute and start thinking about how to structure the memory access itself.
Lu: The implication is that if you integrate PNM accelerators directly into your HBM cubes and manage the communication flow this tightly, you can get much closer to what’s needed for massive context handling.
Tom: It’s a shift from compute-centric design to a memory-centric one when serving these huge models.
Jane: That’s the big picture here: it's about finding a better fit for the problem, not just throwing more silicon at it and hoping for the best.
The paper's improvements: Tom: So we just talked about how AMMA is built around PNM HBM cubes to handle massive context length attention, and now we need to talk about the specific tricks they used to make it work efficiently at this scale.
Jane: Right, so if you’re looking for what makes this architecture actually perform well compared to a standard GPU setup, we need to look at the parallelism and communication strategies they proposed.
Lu: The authors introduced a two-level hybrid parallelism scheme, which maps the key value heads to different cube groups using something called tensor parallelism at level one.
Tom: And then on top of that, they use context parallelism at level two by splitting the key value cache across sequence dimension within each group.
Jane: That sounds like layering parallelization; you're essentially doing tensor parallelism on the heads and then context parallelism on the cache data itself.
Lu: Exactly, and this setup confines the bulk of that data movement into local neighborhoods, which is what keeps latency low.
Meng: If I’m looking at deployment costs for a production system, confining that data movement locally sounds good because inter-chip communication is always expensive.
Tom: And they also completely reordered the collective communication flow to reduce synchronization overhead by replacing intra-group allreduce with just a reduce scatter and cross-group allreduce with a point-to-point reduce.
Jane: That’s smart; it cuts down on the number of operations you have to do globally, which means less time waiting for data to sync up across all sixteen dies.
Lu: They also deployed ninety-six small systolic arrays operating in parallel using an output stationary dataflow, which avoids that communication overhead when you’re accumulating partial outputs as the sequence length gets longer.
Tom: That output stationary dataflow is key because it prevents communication bottlenecks from building up as the sequence grows.
Jane: So what this means practically for a user running an AI application is that they get high throughput without having to worry about the chiplets constantly stalling while waiting for each other.
Meng: For someone shipping models, that localized data flow is exactly what you need to ensure stability when you’re pushing those workloads onto distributed hardware.
Lu: They also introduced a context-aware tiling strategy that partitions both the sequence dimension and potentially splits the key dimension across systolic arrays to maximize utilization and keep those processing units busy.
Tom: That tiling method is designed so that "the numerator is fixed by the problem size and independent of tiling choices" while still minimizing idle systolic arrays.
Jane: It sounds like they’ve balanced maximizing how much work you can do with the hardware against keeping those little processing units actually working.
Lu: And finally, they focused on integrating the PNM unit fully into each HBM cube's logic die, making it a functional accelerator right inside the memory structure.
Tom: It’s about making sure that what happens inside that memory is optimized for decode attention specifically, even under minimal power constraints.
Jane: So to summarize these improvements, AMMA isn't just a new chip; it's a carefully orchestrated system of parallelism and communication designed to keep the data moving where it needs to go.
Meng: I’m still cautious about the claim that this is ideal for every single production scenario, but if the numbers hold up under real traffic, this approach to memory management looks very promising.
Conclusion: Tom: So to wrap up, we’ve been looking at AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving, and the main idea is moving away from GPU compute towards a memory-centric design for massive context workloads <ref:2604.26103#pg1,AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context>.
Jane: It’s about using HBM cubes with processing inside them to handle that huge attention demand without getting bogged down by the memory bandwidth limitations we see in traditional GPU setups.
Lu: The paper showed how this architecture can achieve fifteen point five times lower latency and six point nine times lower energy use compared to an H100 chip, especially when dealing with one million tokens.
Meng: For us at the startup level, it’s a very impressive efficiency gain because we’re talking about stable energy efficiency gains of five point six to six point six times higher token per joule than the H100 consistently.
Tom: And they've done this by using a reordered collective communication flow and a two-level hybrid parallelism scheme to keep the data movement localized and fast.
Jane: That’s the crucial part: it’s not just about having more memory; it’s about structuring how all those pieces talk to each other so they don't create a bottleneck across the chip.
Lu: They also showed how this system can maintain speed even when you push context lengths up to one million tokens, proving the architecture scales well for those long sequences.
Meng: I still have reservations about whether this level of optimization holds up when we put it into a production environment with unpredictable traffic patterns, but the evidence on these benchmarks is quite clean.
Tom: It seems like AMMA shows that if you focus on optimizing the memory access patterns and reducing inter-chip movement instead of just brute-forcing compute power, you can get significant gains in serving performance.
Jane: It really points to a direction where memory itself becomes a much more active participant in processing the language model, which is an interesting path forward for next generation AI systems.
Lu: The limitation they flag is that this entire setup is specifically tailored for low-latency attention serving, so it’s not meant to be a general-purpose training platform.
Meng: That makes sense; if you’re only focused on inference speed and not massive pre-training throughput, that narrows the scope considerably for deployment decisions.
Tom: It’s about finding a better fit for the problem, not just brute-forcing the solution with bigger chips.
Jane: Exactly; it's about finding a better fit for the problem, and AMMA shows that by being memory-centric, we can build architectures that handle those massive context needs much more efficiently.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game