Closing the Curvature Gap: Full Transformer Hessians
The episode discusses the paper "Closing the Curvature Gap: Full Transformer Hessians," which provides explicit second-order expressions for LayerNorm and Feedforward Network Hessians in Transformer blocks. The hosts discuss how this fills a theoretical gap, allows for better understanding of curvature propagation across layers, and offers a framework to predict convergence trajectories based on data size.