SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs

arXiv:2609.34162 · cs.DC, cs.LG · Submitted 2026-09-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs".

Jane: The gist The paper presents SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems, which enables full-parameter LLM fine-tuning beyond GPU memory by coordinating parameter delivery, gradient aggregation,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs." Basically, they're tackling the problem of fine-tuning massive AI models that are too big to fit entirely on the GPU memory.

Jane: That’s right. The main idea is that it enables full-parameter fine-tuning beyond what a single GPU can hold by coordinating things across multiple GPUs when they share a host machine.

Lu: It claims this synchronous data-parallel runtime maintains one authoritative host state while decoupling the communication routes from the actual layout and pipelines parameter delivery, gradient aggregation, and CPU updates across all those ranks <ref:2609.34162#pg1>.

Meng: So it’s not just about moving data; it’s about orchestrating the whole process so that even with shared host resources, you can still train these huge models efficiently.

Lalam: It essentially sets up a system that manages parameter delivery, gradient return, and layer-wise CPU updates all within a bounded GPU working window <ref:2609.34162#pg2>.

Tom: And they use an analytical step-time model to characterize the resource bottlenecks and pipeline exposure, which helps them guide the communication routes and chunk sizes based on topology and workload <ref:2609.34162#pg1>.

Jane: It’s about figuring out when reducing traffic shortens a step, or when shrinking computation windows exposes host work, which is something they call residual pipeline exposure <ref:2609.34162#pg2>.

Meng: So the core challenge they're addressing seems to be how to manage that remaining shared-host work so it overlaps with the GPU compute instead of just sitting there and wasting time.

Lu: They use something called AutoPolicy which configures communication routes, chunk sizes, and activation layouts for the specific workload and hardware you're using <ref:2609.34162#pg1>.

Tom: And this policy selection involves a measured difference, tau j equals tau zero plus T j k j minus T zero k j, which captures that marginal pipeline cost of overlap and contention <ref:2609.34162#pg3>.

Jane: They then minimize the sum over all candidate layers n j times tau j equals L to find a memory-validated policy called policy pi <ref:2609.34162#pg3>.

Tom: In terms of performance, in matchedbatch sweeps, SlideDP achieves geometric-mean throughput ratios of one point four six to two point six four times over SlideFormer and MegaTrain <ref:2609.34162#pg1>.

Meng: That’s a solid number. And they report that on four H100s, SlideDP approaches GPU-resident FSDP2 throughput for Qwen3-14B even at a smaller batch size <ref:2609.34162#pg1>.

Jane: They also mention that with a larger batch, it processes over 1M tokens per step and exceeds FSDP2’s measured peak throughput by eleven point two percent <ref:2609.34162#pg1>.

Lu: Furthermore, they sustain geometric-mean speedups of one point eight three times over ZeRO-Offload and one point four eight times over MegaTrain across batches four to thirty-two on four RTX four thousand ninety GPUs <ref:2609.34162#pg1>.

Tom: And they note that this system reduces Qwen3-14B peak reserved memory by fifty-one to fifty-nine percent versus ZeRO-Offload and twenty-nine to forty-two percent versus MegaTrain at common feasible batches <ref:2609.34162#pg1>.

Jane: It’s clear that this paper, "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs," shows that shared host DP requires coordinated decisions across state movement, execution scheduling, and memory allocation rather than just optimizing each part separately <ref:2609.34162#pg1>.

Meng: So for a practitioner looking at this, the implication is that you can translate host-memory capacity into larger feasible training workloads across commodity workstations and multi-GPU servers <ref:2609.34162#pg1>.

Lu: Observation one points out an avoidable amplification where direct DP extension causes host demand to grow with GPU rank count while the host service capacity stays fixed <ref:2609.34162#pg2>.

Tom: That’s a big warning there. Then they address observation two, residual pipeline exposure, which is when host-side work gets exposed when its completion exceeds the hiding window before downstream operations need that result <ref:2609.34162#pg2>.

Jane: And observation three shows that the right delivery route really depends on both topology and workload; sharded delivery improves throughput by sixteen point eight to thirty-two point seven percent with NVLink bridges compared to replicated delivery <ref:2609.34162#pg2>.

Meng: Observation four highlights that the value of spare HBM depends on exposed pipeline work, suggesting memory allocation should be a runtime policy decision rather than a fixed checkpointing choice <ref:2609.34162#pg1>.

Lu: So SlideDP shows that you need coordinated decisions across state movement, execution scheduling, and memory allocation instead of optimizing individual components independently <ref:2609.34162#pg1>.

Tom: It’s about using the host-memory capacity to support larger training workloads while maintaining throughput across a whole range of hardware setups <ref:2609.34162#pg1>.

Jane: This paper, "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs," is showing us how to make these massive models more accessible by coordinating the host resources much more intelligently <ref:2609.34162#pg1>.

Conclusion: Tom: So we're wrapping up on SlideDP, which is about scaling those huge LLMs when they don't fit on a single GPU by coordinating everything across multiple GPUs sharing one host machine <ref:2609.34162#pg1>.

Jane: It’s really about making that shared-host setup actually work for massive fine-tuning tasks, not just fitting the model in there.

Lu: The authors are SlideDP, and they're basically showing how to manage parameter delivery and gradient aggregation across ranks while keeping one main host state consistent <ref:2609.34162#pg1>.

Meng: So what’s the big picture here for someone actually building these systems?

Tom: It means we can use more HBM because you can move the host memory capacity into a much larger feasible training workload, which is huge for scaling up model size.

Jane: They found that this coordination works really well across different hardware, even on commodity workstations and multi-GPU servers <ref:2609.34162#pg1>.

Lu: Their policy selection method uses an analytical step-time model to figure out the best way to route data based on the topology and the specific workload you're running <ref:2609.34162#pg1>.

Meng: That sounds complex. What’s the practical win beyond just getting a better throughput number?

Tom: The real win is that they identified issues like avoidable amplification and residual pipeline exposure, telling us exactly where we need to tune things in the software stack <ref:2609.34162#pg2>.

Jane: And they pointed out that memory allocation should be a runtime decision based on how much work is exposed in the pipeline, not just a fixed checkpointing choice <ref:2609.34162#pg1>.

Lu: It suggests we need a more holistic approach to managing shared-host DP where state movement, execution scheduling, and memory allocation are all coordinated <ref:2609.34162#pg1>.

Meng: So it’s about coordination instead of just optimizing each component in isolation.

Tom: Exactly. SlideDP shows that you can use host-memory capacity to support much larger training jobs while still maintaining decent throughput across a variety of setups <ref:2609.34162#pg1>.

Jane: It’s about making these massive models more accessible by coordinating the host resources much more intelligently <ref:2609.34162#pg1>.

Tom: Next up, we'll look at some of the specific performance numbers they got from those experiments and see how they stack up against other methods.

The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence

cs.DC, cs.LG

Submitted: 2026-09-28

Updated: 2026-09-28

Comments: 14 pages, 15 figures, 4 tables

Code: https://github.com/RegiaYoung/SlideDP

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The gist The paper presents SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems, which enables full-parameter LLM fine-tuning beyond GPU memory by coordinating parameter

Key concepts

One Authoritative Host State: SlideDP maintains a single, definitive version of the host state. This state is decoupled from the specific layout used for parameters or gradients. This separation allows the system to choose different communication routes based on the current workload and hardware topology without disrupting the central, authoritative host view.
Elastic Checkpointing: This mechanism defines how activations are saved, allowing flexible control over memory usage. It includes policies for checkpoint granularity and an offload ratio that dictates where activations are placed—either in GPU memory or on the host CPU—to manage the trade-off between speed and memory capacity.
Analytical Step-Time Model: This model analyzes how resources bottleneck during shared-host scaling. It explains how reducing data traffic can shorten training steps, and when shrinking computation windows exposes work on the host. This informs the runtime's decisions about communication routes and chunk sizes to maintain optimal pipeline exposure.
Marginal Pipeline Cost (τj): This calculated coefficient measures the marginal cost of a specific communication route or configuration. It captures the overlap and contention between different operations. The system uses this metric to select policies that minimize the total cost, ensuring a memory-validated policy for fine-tuning.

Terminology

Fine-Tuning — 1× in this paper · explained in 5 episodes

Summary

The gist The paper presents SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems, which enables full-parameter LLM fine-tuning beyond GPU memory by coordinating parameter delivery, gradient aggregation, and CPU updates across ranks.

How it works

SlideDP maintains one authoritative host state while decoupling communication routes from state layout and pipelines parameter delivery, gradient aggregation, and CPU updates across ranks and chunks The design coordinates these operations under per-layer parameter-version and update dependencies to preserve synchronous DP semantics.
It separates persistent host-state layout from parameter delivery and gradient reduction allowing for one authoritative host state while selecting communication routes according to topology and workload.
The runtime coordinates parameter delivery, gradient return, and layer-wise CPU updates within a bounded GPU working window. This involves six logical lanes coordinating parameter preparation and H2D, activation offload and reload, GPU compute, gradient reduction, gradient D2H, and CPU LayerAdam.
The system uses Elastic Checkpointing to define the activation-side policy space evaluated by AutoPolicy. This policy space includes checkpoint granularity with an offload ratio controlling activation placement.

Analytical Model and Policy Selection

An analytical step-time model of shared-host scaling characterizes resource bottlenecks and pipeline exposure. This model explains when reducing traffic shortens a step, when shrinking computation windows expose host work, and why route choices depend on both topology and workload.
AutoPolicy configures communication routes, chunk sizes, and activation layouts for the workload and hardware. It uses Elastic Checkpointing supplies per-layer choices for retention, offloading, and recomputation.
The policy selection process involves measuring pipeline effects where a calibrated coefficient is calculated as τj = τ0 + Tj (kj) −T0 kj. This measured difference captures marginal pipeline cost (overlap and contention).
The MILP minimizes Í j n j τ j, where n j is the integer layer count of candidate j and Í j n j = L, to ensure a Memory-validated policy Π.

Performance and Scaling Results

In matchedbatch sweeps, SlideDP achieves geometric-mean throughput ratios of 1.46–2.64× over SlideFormer, MegaTrain, and ZeROOffload.
On four H100s, SlideDP approaches GPU-resident FSDP2 throughput for Qwen3-14B at a smaller batch size. With a larger batch, it processes over 1M tokens per step and exceeds FSDP2’s measured peak throughput by 11.2%.
SlideDP sustains geometric-mean speedups of 1.83× over ZeRO-Offload and 1.48× over MegaTrain across batches 4–32 on four RTX 4090 GPUs.
The system reduces Qwen3-14B peak reserved memory by 51–59% versus ZeRO-Offload and 29–42% versus MegaTrain at common feasible batches.

Key Findings on Shared-Host Constraints

Observation 1 shows Avoidable amplification, where a direct DP extension causes host demand to grow with GPU rank count while host service capacity remains fixed.
Observation 2 addresses Residual pipeline exposure, noting that host-side work becomes exposed when its completion exceeds the available hiding window before downstream operations require the result.
Observation 3 demonstrates that the right delivery route depends on topology and regime, as sharded delivery improves throughput by 16.8–32.7% with NVLink bridges compared to replicated delivery.
Observation 4 highlights The value of spare HBM depends on exposed pipeline work, suggesting that memory allocation should be a runtime policy decision rather than a fixed checkpointing choice.

SlideDP demonstrates that shared-host DP requires coordinated decisions across state movement, execution scheduling, and memory allocation rather than independent optimization of individual components. This coordination translates host-memory capacity into larger feasible training workloads while sustaining throughput across commodity workstations and multi-GPU servers.

Improvements for AI systems

One can design a synchronous data-parallel runtime that maintains one authoritative host state, decouples communication routes from state layout, and pipelines parameter delivery, gradient aggregation, and CPU updates across ranks and chunks. This allows full-parameter fine-tuning beyond GPU memory by overlapping host work with GPU computation.
The system can dynamically select communication routes based on hardware topology and workload using an analytical model that characterizes resource bottlenecks to determine if a route choice should shorten a step, as described in Section 3.4.
The AI system can automatically configure the training strategy by employing AutoPolicy, which configures communication routes, chunk sizes, and activation layouts for the workload and hardware through a process that involves measuring pipeline effects to select policies based on measured costs.
The system can utilize Elastic Checkpointing to define activation-side policies like retaining previously offloaded activations on GPU removes their D2H/H2D demand and reload dependencies, allowing the model to utilize spare HBM strategically by balancing memory usage against transfer demand, recomputation cost, and exposed execution work.
The system can achieve superior throughput by employing chunking strategies that overlap preparation and movement within a layer, as demonstrated by the observation that chunking reduces H2D slowdown relative to isolated execution from 2.3–2.6× to approximately 1.3×.