An Identifiability Theory of Masked Prediction: Mode Blindness and Mask Schedules
The episode discusses Yichao Cai and Javen Qinfeng Shi's paper on masked prediction identifiability, focusing on mode blindness in models with two well-separated global modes. The hosts explain that large context masks can cause models to lose sensitivity to the mixture ratio of these modes unless the masking schedule is actively controlled or full mask mass is used for safety.