sigma Transfer: Uncertainty Transfer from Small to Large Networks under mu P

SigmaTransfer enables zero-shot transfer of uncertainty precision from small neural networks to much larger ones by rescaling the prior covariance under Maximal Update Parametrization ($\mu$P). This method stabilizes the optimal precision choice across network widths, leading to massive speedups in precision sweeps and maintaining high predictive performance in regression, classification, and Transformer tasks without needing a target posterior.

Open the full episode

More from AI Radio