Unsupervised Improvement of Factual Knowledge in Language Models

Name: Unsupervised Improvement of Factual Knowledge in Language Models
Uploaded: 2026-08-20T17:42:06.904821+00:00
Duration: 587 s
Description: The episode discusses 'Unsupervised Improvement of Factual Knowledge in Language Models,' which proposes a smarter way to train models using masked language modeling (MLM). The method uses Pointwise Mutual Information (PMI) to identify factually informative tokens, adjusting both masking rates and loss weights to improve factual recall without external knowledge bases.

The episode discusses 'Unsupervised Improvement of Factual Knowledge in Language Models,' which proposes a smarter way to train models using masked language modeling (MLM). The method uses Pointwise Mutual Information (PMI) to identify factually informative tokens, adjusting both masking rates and loss weights to improve factual recall without external knowledge bases.

Open the full episode