Optimizing Effective Training Time for Large-Scale Recommendation Systems
The Effective Training Time (ETT%) framework measures and reduces wasted GPU capacity in large-scale recommendation training. By analyzing time lost during startup, recovery from failures, and total failures, researchers implemented various optimizations—like communication elimination and smart checkpointing—which improved average performance by 15.5% across benchmarks.