MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration

Name: MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
Uploaded: 2026-09-06T07:14:58.649650+00:00
Duration: 342 s
Description: The episode discusses MSign, an optimizer designed to prevent training instability in large language models. Instability is caused by stable rank collapse and increased Jacobian alignment, which jointly cause exponential gradient growth. MSign addresses this by periodically applying a matrix sign operation to restore stable rank with minimal computational overhead.

The episode discusses MSign, an optimizer designed to prevent training instability in large language models. Instability is caused by stable rank collapse and increased Jacobian alignment, which jointly cause exponential gradient growth. MSign addresses this by periodically applying a matrix sign operation to restore stable rank with minimal computational overhead.

Open the full episode