Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training
cs.LG, stat.ML
Submitted: 2026-06-27
Updated: 2026-06-27
Comments: 27 pages, 14 figures, 5 tables. Code and data: https://github.com/tiexinding/NPM-Weibull-public
License: http://creativecommons.org/licenses/by/4.0/
The gist: A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape k about 1.2 is stable across layers and models, so the scale λ carries most
Terminology
Abstract
A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape k about 1.2 is stable across layers and models, so the scale λ carries most training-induced movement. What corpus property sets how much λ grows? Using the bigram conditional entropy D = H(next prev), a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, λ squared - λ 0 squared = C 0(η) + C 1(η)(H r - D) 0.59, where H r is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per- η coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto (H r - D) 0.59 with unit slope (R squared = 0.941; direct per- η fits are weaker, R squared about 0.82). Because D is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader Φ(D,R,A,H) data-to-weight framework.
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks