InforMask: Unsupervised Informative Masking for Language Model Pretraining

Name: InforMask: Unsupervised Informative Masking for Language Model Pretraining
Uploaded: 2026-08-20T17:42:44.056443+00:00
Duration: 558 s
Description: The episode discusses 'InforMask,' a method for training language models that improves upon random word masking. The authors use Pointwise Mutual Information (PMI) to identify and mask highly informative tokens, leading to better factual recall on benchmarks like LAMA. The technique is fully unsupervised and efficient for large-scale pretraining.

The episode discusses 'InforMask,' a method for training language models that improves upon random word masking. The authors use Pointwise Mutual Information (PMI) to identify and mask highly informative tokens, leading to better factual recall on benchmarks like LAMA. The technique is fully unsupervised and efficient for large-scale pretraining.

Open the full episode