Bug or Feature squared: Weight Drift and Activation Sparsity

The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching nearly 90% in GPT-nano. We further analyze whether this natural sparsity is damaging and find that, on average, models sharply degrade above about 70% activation sparsity. We then evaluate sparsity-inducing activation functions, which reveals activation spikes in intermediate transformer layers. Code is available at https://github.com/On-Point-RND/BugOrFeature.