Mode Dynamics: A Framework for Neural Network Learning
cs.LG
Submitted: 2026-10-29
Updated: 2026-11-13
Comments: 32 pages, 17 figures, 5 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce Mode Dynamics, a framework that connects coordinated parameter motion during neural network learning to changes in model outputs.
Abstract
We introduce Mode Dynamics, a framework that connects coordinated parameter motion during neural network learning to changes in model outputs. Starting from a recorded parameter trajectory, we decompose its updates into collective modes and map each mode to its state-dependent output effect. The framework organizes this link through three mechanisms. Decomposition identifies collective modes in parameter motion and their time-varying amplitudes. Reconstruction tests whether the effects induced by those modes recover local and accumulated output change. Concentration quantifies how behavioral contribution is distributed across modes. Controlled scalar regression and next-token experiments directly test these mechanisms. They show that small sets of leading modes reconstruct behavior, induced effects depend on the training path, and modal complexity varies with task and data. We then evaluate the framework on held-out image and language-model outputs using CIFAR-100 autoencoding and a public Pythia-70M pretraining trajectory. Across these settings, the results connect compact modal subspaces of recorded parameter motion to evolving behavior.
Sources
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- Enhancing Neural Training via a Correlated Dynamics Model
- Toy Models of Superposition
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Scaling and evaluating sparse autoencoders
- Weight-sparse transformers have interpretable circuits
- Gradient Descent Happens in a Tiny Subspace
- LoRA: Low-Rank Adaptation of Large Language Models
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
- Measuring the Intrinsic Dimension of Objective Landscapes
- Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
- Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
- Estimating Training Data Influence by Tracing Gradient Descent
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Low Rank Gradients and Where to Find Them
- Kernel and Rich Regimes in Overparametrized Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis