Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

Name: Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading
Uploaded: 2026-09-06T02:18:27.681233+00:00
Duration: 277 s
Description: The episode discusses the paper "Deep Optimizer States," which addresses memory bottlenecks in training large Transformer models. It moves beyond static offloading by using dynamic interleaving and scheduling based on GPU utilization patterns. This technique achieves up to 2.5 times faster iterations than current methods, enabling scalable training of massive models.

The episode discusses the paper "Deep Optimizer States," which addresses memory bottlenecks in training large Transformer models. It moves beyond static offloading by using dynamic interleaving and scheduling based on GPU utilization patterns. This technique achieves up to 2.5 times faster iterations than current methods, enabling scalable training of massive models.

Open the full episode