Signed-Permutation Coordinate Transport for RMSNorm Transformers
John Sweeney
cs.LG, cs.CL, stat.ML
Submitted: 2026-06-30
Comments: 31 pages, 2 figures, 26 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top- k neuron sets, attribution lists, and merge alignments.
Terminology
Abstract
Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top- k neuron sets, attribution lists, and merge alignments. This is only well posed after fixing the model's residual-stream gauge, which we show is architecture-dependent: LayerNorm residual charts have permutation gauge S d (up to a global sign flip), while RMSNorm charts with generic per-channel gain have signed-permutation gauge B d = S d plus or minus 1 d. Permutation-only alignment is therefore symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove a sharp failure mode: with decorrelated coordinates, raw signed-correlation matching has a structural permutation-accuracy ceiling at the positive-sign fraction of the true gauge, which sign-marginalization removes. We then make coordinate-preserving transport, not function-level merging, the primary object: composing saved-checkpoint local B d gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching, and the gain is not explained by merely routing through the base. The recovered gauge transfers tools that permutation-only alignment breaks: TinyLlama SAE reconstruction has NMSE 0.004 under B d versus 1.08 under S d; Qwen sentiment steering preserves 95.8% of its effect versus 17.2%; refusal steering reverses sign under S d; coordinate-preserving merges behave the same way. The same covariance governs stateful training: signed transport of AdamW state preserves the resumed trajectory, while permutation-only state follows a different one from a functionally identical checkpoint. Finally, gauge-sweep audits show index-level interpretability claims are reproducible only relative to an explicit gauge.
Sources
- Refusal in Language Models Is Mediated by a Single Direction
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
- Topology and Geometry of Half-Rectified Network Optimization
- Scaling and evaluating sparse autoencoders
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Transformer Fusion with Optimal Transport
- Convergent Learning: Do different neural networks learn the same representations?
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- TIES-Merging: Resolving Interference When Merging Models
- Qwen2.5 Technical Report
- Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks