A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields
The study introduces row and column scale fields as a middle ground between averaged weight statistics and individual weights to track how magnitude is distributed across Transformer channels. By analyzing these fields, researchers found that components share a common core structure, while differences arise from channel-scale scale distribution. This framework connects weight structure directly to functional organization within the Transformer architecture.