Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication".
Paper summary: Jane: So, wrapping up this look at "Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication," the authors are essentially saying they've provided a clear system-level understanding of this library.
Tom: They’ve broken down the programming model, the implementation details, and how it performs its various communication paths, from symmetric memory to those device-side collectives.
Lu: The core message is that NVSHMEM pioneered a device-side symmetric memory programming model that enables fine-grained GPU-driven communication and is important for approaching the hardware performance limit.
Meng: From an engineering standpoint, it’s useful because it shows the tradeoffs involved in designing this system, especially around those fast and slow paths for communication.
Jane: The paper highlights NVSHMEM's role as a key systems building block, which means understanding its design and these specific implementation choices is important for anyone working on high-performance GPU applications.
Tom: And it points toward opportunities to improve the performance of GPU communication runtimes based on what they observed in their analysis of device-side collectives.
Lu: It’s a concise study, focusing sharply on symmetric memory and one-sided operations, which makes it a great resource for anyone who needs to get technical details without getting lost in documentation.
Meng: For those of us shipping models, the implication is understanding exactly when to use this direct device control versus relying on higher-level abstractions like NCCL.
Jane: That’s right; it’s about making informed decisions about how we structure our communication for maximum efficiency and control over the hardware.
Conclusion: Tom: So, we're wrapping up our look at NVSHMEM now, focusing on this paper by Author Names.
Jane: It really boils down to how they mapped out all those complex GPU communication parts—the symmetric memory and those one-sided operations.
Tom: They spent a lot of time detailing the setup, looking at the API groups like memory management and synchronization.
Lu: The real focus is on that device-side symmetric memory model they introduced, which lets GPUs talk to each other directly without too much CPU help.
Meng: I'm interested in how that works at scale; does it actually hold up when you try to run production workloads?
Jane: Well, the paper shows how this approach helps get closer to the hardware performance limit for fine-grained communication.
Tom: It positions NVSHMEM as a systems building block, showing why device-side control is important for pushing those limits.
Lu: They clearly show the trade-offs involved in that design, balancing speed against the complexity of managing all those memory accesses.
Meng: And they pointed out where it stops working; you can't just assume this solves every single communication problem on its own.
Jane: Right, so the authors conclude that while NVSHMEM is a crucial piece for direct control, there’s still room to improve the communication runtimes themselves.
Tom: It’s a big picture thing because it clarifies exactly how we need to structure our software when dealing with GPU clusters.
Yijun Ma, Siyuan Shen, Tiancheng Chen, Akhil Langer, Jiri Kraus, Benjamin Glick, Craig Belusar, Jeff Hammond
ETH Zurich University · NVIDIA Corporation
cs.DC
Submitted: 2026-06-04
Updated: 2026-11-18
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 85/100
The gist: NVSHMEM is NVIDIA’s OpenSHMEM-based PGAS communication library for GPU clusters, enabling GPUinitiated, one-sided communication through symmetric memory.
Key concepts
- PGAS Model
- Partitioned Global Address Space (PGAS) is a programming model where a logically shared address space is divided among different processing elements (PEs). This allows multiple devices to access the same data as if it were locally available, simplifying distributed computing.
- Symmetric Memory
- NVSHMEM uses symmetric memory, which means each processing element has its own heap, but all PEs share a common view of this memory. Objects are allocated with the same type and layout across all PEs, enabling seamless one-sided communication.
- One-sided RMA
- Remote Memory Access (RMA) is the core communication interface where one device directly reads or writes data on another device's memory without involving the CPU. This operation is fundamental to NVSHMEM's efficiency and reduces CPU overhead significantly.
- Device-side Collectives
- Collectives are operations involving multiple processes communicating together, like a barrier or reduction. In NVSHMEM, these are executed directly by threadgroups on the GPU rather than being managed by the host CPU, leading to faster execution.
Terminology
Summary
NVSHMEM is NVIDIA’s OpenSHMEM-based PGAS communication library for GPU clusters, enabling GPUinitiated, one-sided communication through symmetric memory. p. 0 ch
2NVIDIA Corporation, {alanger, jkraus, benjaming, cbelusar, jeffpapers}@nvidia. com
* These authors contributed equally to this work
Abstract—NVSHMEM is NVIDIA’s OpenSHMEM-based
PGAS communication library for GPU clusters, enabling GPUinitiated, one-sided communication through symmetric memory. Despite its growing adoption, a system-level understanding of
its design and behavior remains scattered across documentation,
source code, and application experience. This paper presents
a concise study of NVSHMEM’s programming model, implementation, and performance characteristics, focusing on symmetric memory, one-sided operations, and device-side collectives. We also examine DeepEP as a case study of NVSHMEM in
performance-critical sparse deep learning workloads. Our analysis shows that NVSHMEM pioneered a device-side symmetricmemory programming model that enables fine-grained GPUdriven communication and is important for approaching the
hardware performance limit. Our analysis shows that NVSHMEM pioneered a device-side symmetricmemory programming model that enables fine-grained GPUdriven communication and is important for approaching the
hardware performance limit. Overall, this work defines NVSHMEM’s role as a systems building block, highlights its design
tradeoffs, and identifies opportunities for improving GPU communication runtimes.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Foundational Concepts
NVSHMEM builds on three main ideas: the limitations of hostinitiated communication, the motivation for device-initiated communication, and the PGAS/OpenSHMEM programming model based on symmetric memory and one-sided operations. NCCL was originally host-driven where the CPU enqueues collective operations and the library selects algorithms and schedules. p. 0 The NVIDIA Collective Communication Library (NCCL) [5]
emerged as the most widely used solution, providing optimized
collective operations and serving as the recommended backend
for most CUDA-based distributed machine learning frameworks such as PyTorch [6], Megatron-LM [7], and vLLM [8]. Originally, NCCL was host-driven: the CPU enqueues
collective operations, and the library selects algorithms and
schedules. This limitation motivates device-initiated communication,
where communication and synchronization are launched directly from GPU code rather than through a host-managed
control path. This reduces CPU involvement and enables finergrained overlap between communication and computation. InfiniBand GPUDirect Async (IBGDA) [12] is
the InfiniBand-specific realization of GDA-KI. The Partitioned Global Address Space (PGAS) model
exposes a logically shared address space partitioned across
processing elements (PEs) [16]. 1, NVSHMEM also provides limited
support for multiple PEs per GPU. Communication is built on
symmetric memory, consisting of a symmetric heap on each
PE and symmetric objects allocated with the same type, size,
and layout across all PEs.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
API Structure and Design
NVSHMEM features a dual-interface design where some components remain fundamentally host-managed while the core communication path is exposed on both the CPU and the GPU. The API groups include Setup/exit, Memory management, Team management, One-sided RMA, Atomics, Memory ordering, Synchronization, and Collectives Table I. p. 2 API group Host/Device Availability Description
Setup / exit Mixed Initialize and finalize the library, bootstrap jobs, and handle exceptional termination. Memory management Host-side Allocate, free, and align symmetric objects in the symmetric heap, provide registration functionality
for selected local buffers. TABLE I
BREAKDOWN OF NVSHMEM API GROUPS BY AVAILABILITY AND FUNCTIONALITY IN THE SOURCE CODE. API group Host/Device Availability Description
Setup / exit Mixed Initialize and finalize the library, bootstrap jobs, and handle exceptional termination. 4) One-sided remote memory access (RMA): The
RMA group is the core one-sided communication
interface of NVSHMEM. It includes typed and untyped
put/get operations, scalar p/g, strided iput/iget,
and nonblocking variants.Where the paper says it
Where the paper says it
Where the paper says it
Memory Management Internals
NVSHMEM implements the symmetric heap on top of CUDA’s low-level virtual memory management (VMM) API. The symmetric heap’s VA range is reserved eagerly during initialization, while physical pages are committed on demand as the application allocates from the heap A. p. 3 With
VMM enabled, NVSHMEM implements memory management scheme that cleanly separates virtual address (VA)
reservation from physical memory allocation. The symmetric heap’s VA range is reserved eagerly during initialization,
while physical pages are committed on demand as the application allocates from the heap [26], [27]. Figure 1 illustrates
the key concepts discussed in this section. Initialization
During nvshmem_init, NVSHMEM creates a symmetric heap for each PE. In contrast
to allocators such as jemalloc, it avoids per-thread arenas,
size classes, or other complex optimizations. NVSHMEM allocates physical GPU memory for the symmetric heap only on demand, when an allocation request
cannot be satisfied from already-backed VA space. Although the figure shows NVLink within a node, the same
path applies beyond a single node when using MNNVL
if heap_base_ is the base of the local symmetric heap
and peer_heap_base_p2p_[remote_pe] is the base
at which the remote PE’s heap is mapped in the local VA
space, then for a local symmetric pointer dest_local the
corresponding remote address is
dest_remote =peer_heap_base_p2p_[remote_pe]
+ (dest_local − heap_base_)
(1)
NVSHMEM, therefore, preserves the object’s offset within
the local heap and applies it to the remote heap base. For non-P2P-accessible GPUs, the same rule is applied using peer_heap_base_remote_[remote_pe], and the
computed address is passed to the transport layer together
with the required memory handle, registration metadata, or
NIC key.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Communication Paths
NVSHMEM utilizes either a fast path or a slow path depending on whether the target peer is P2P-reachable. The Fast Path uses direct GPU Memory Access when P2P access is available Device-side RMA Once the peer heap is directly mapped, device-side RMA reduces to ordinary memory access on the computed remote address from Equation 1 The Slow Path is used when the target PE’s symmetric heap is not directly P2P-mapped into the local GPU address space In the slow path, NVSHMEM either uses InfiniBand GPUDirect Async (IBGDA), if available, or a host proxy path where a CPU thread executes the request on behalf of the GPU
Collective Performance Analysis
NVSHMEM implements various collective algorithms to target different message sizes and system configurations. Table II summarizes the main designs and reports their communication volume and latency, measured by the number of synchronization steps In general, NVSHMEM selects collective algorithms at runtimes by using a rule-based decision tree The device-side collective APIs in NVSHMEM are threadgroup collectives, meaning a single collective invocation is executed by one participating threadgroup rather than by multiple CTAs Barrier and Synchronization primitives have the same synchronization depth, but barrier provides stronger ordering and completion guarantees
Application Case Study
The DeepEP case study examines how NVSHMEM is used in real applications like Mixture-of-Experts (MoE) workloads under expert parallelism. DeepEP uses NVSHMEM only at critical points in the inter-node RDMA path, such as metadata exchange and atomic credit updates NVSHMEM acts as the cross-node communication substrate while DeepEP builds its own multi-stage transport pipeline on top. p. 9 Instead, it uses
NVSHMEM only at critical points in the inter-node RDMA
path: metadata exchange in notify_dispatch, chunked
RDMA puts from sender coordinators, and atomic credit
updates for remote head or tail counters. Thus, NVSHMEM
acts as the cross-node communication substrate, while DeepEP
builds its own multi-stage transport pipeline on top. There, DeepEP was integrated with NCCL GIN while
the original NVSHMEM-based version was retained as the
baseline, and both the high-throughput (HT) and low-latency
(LL) dispatch and combine kernels were evaluated. The main
takeaway is that NCCL GIN matches NVSHMEM closely,
typically within about 1–2%, while offering the same style of
device-initiated communication inside NCCL’s runtime. Despite its growing adoption, a system-level understanding of
its design and behavior remains scattered across documentation,
source code, and application experience. This paper presents
a concise study of NVSHMEM’s programming model, implementation, and performance characteristics, focusing on symmetric memory, one-sided operations, and device-side collectives.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Symmetric memory enables fine-grained GPUdriven communication by exposing a
partitioned global address space (PGAS) directly to GPU code,
allowing CUDA kernels to access remote processing elements viaone-sided put/get operations, atomics, and explicit synchronization primitives.
-
Device initiated operations allow communication and synchronization to be
launched directly from GPU code rather than through a host-managed control path,
which reduces CPU involvement and enablesfinergrained overlap between communication and computation.
-
The dual interface design allows for a flexible programming model where the core communication path is exposed on both the CPU and the GPU, supporting both host-driven and device-callable operations.
-
For performance-critical sparse deep learning workloads, NVSHMEM can be used as a substrate for custom pipelines like DeepEP's high-throughput path, which uses
eight parallel NVSHMEM world teams
to manage crossnode RDMA between GPUs with the same local index. -
The library supports low latency (LL) protocols using
Low Latency (LL) and LL128 protocols,
where data units are paired with synchronization flags to reduce the cost of explicit synchronization after data transfer.
Abstract
NVSHMEM is NVIDIA's OpenSHMEM-based PGAS communication library for GPU clusters, enabling GPU-initiated, one-sided communication through symmetric memory. Despite its growing adoption, a system-level understanding of its design and behavior remains scattered across documentation, source code, and application experience. This paper presents a concise study of NVSHMEM's programming model, implementation, and performance characteristics, focusing on symmetric memory, one-sided operations, and device-side collectives. We also examine DeepEP as a case study of NVSHMEM in performance-critical sparse deep learning workloads. Our analysis shows that NVSHMEM pioneered a device-side symmetric-memory programming model that enables fine-grained GPU-driven communication and is important for approaching the hardware performance limit. Overall, this work defines NVSHMEM's role as a systems building block, highlights its design tradeoffs, and identifies opportunities for improving GPU communication runtimes.
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends