Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

Name: Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
Uploaded: 2026-10-17T17:39:55.201979+00:00
Duration: 295 s
Description: The paper argues that standard optimizers fail because they ignore neural network symmetries. It proposes a principle where optimizer updates must match the symmetry of specific parameter blocks, such as embeddings or MoE routers. This leads to tailored update rules—like row-norm or spectral methods—that improve training stability and reduce load imbalance in complex models.

The paper argues that standard optimizers fail because they ignore neural network symmetries. It proposes a principle where optimizer updates must match the symmetry of specific parameter blocks, such as embeddings or MoE routers. This leads to tailored update rules—like row-norm or spectral methods—that improve training stability and reduce load imbalance in complex models.

Open the full episode