Sequential Pretraining Favors Large Models
Exposure Therapy (ET) is a regularization technique introduced to improve how large foundation models learn from sequential data. It works by sparsely replacing training updates with samples of later data distributions, helping models allocate learning capacity more efficiently and recover robustness often gained through larger model sizes.