The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training

arXiv:2502.19002 · cs.LG, cs.AI, cs.CL, math.OC, stat.ML · Submitted 2025-02-26 · Read on arXiv

cs.LG, cs.AI, cs.CL, math.OC, stat.ML

Submitted: 2025-02-26

Updated: 2025-06-13

Code: https://github.com/Wongboo/BlockwiseLearningRate

Project page: http://skylion007.github.io/OpenWebTextCorpus

Terminology

Sources

Related papers