ZenFlow: Enabling Stall-Free Offloading for LLM Training
ZenFlow is a new training framework that solves GPU stalls during large language model fine-tuning by prioritizing important gradients. It decouples updates between fast GPUs and slow CPUs, performing immediate in-place updates on the GPU for critical parameters while offloading less important ones to the CPU for asynchronous accumulation. This results in up to 5x speedup and significantly lower costs without sacrificing accuracy.