GWT: Scalable Optimizer State Compression for Large Language Model Training

Name: GWT: Scalable Optimizer State Compression for Large Language Model Training
Uploaded: 2026-08-26T01:24:40.377842+00:00
Duration: 361 s
Description: The episode discusses 'GWT: Scalable Optimizer State Compression for Large Language Model Training,' a paper addressing the memory overhead of Adam optimizers during LLM training. The hosts conclude that GWT uses wavelet transforms to compress gradients, achieving up to a 79% reduction in optimizer memory and a 1.9x speedup in training while maintaining high model quality.

The episode discusses 'GWT: Scalable Optimizer State Compression for Large Language Model Training,' a paper addressing the memory overhead of Adam optimizers during LLM training. The hosts conclude that GWT uses wavelet transforms to compress gradients, achieving up to a 79% reduction in optimizer memory and a 1.9x speedup in training while maintaining high model quality.

Open the full episode