Measuring the Prevalence of Policy Violating Content with ML Assisted Sampling and LLM Labeling

Name: Measuring the Prevalence of Policy Violating Content with ML Assisted Sampling and LLM Labeling
Uploaded: 2026-08-19T14:31:42.024473+00:00
Duration: 643 s
Description: Pinterest researchers developed a method to measure the prevalence of policy-violating content using ML-assisted sampling and LLM labeling. This approach enables daily, exposure-weighted estimates that reflect the actual harm users experience. The system is significantly faster and cheaper than human review while maintaining statistical rigor and unbiasedness.

Pinterest researchers developed a method to measure the prevalence of policy-violating content using ML-assisted sampling and LLM labeling. This approach enables daily, exposure-weighted estimates that reflect the actual harm users experience. The system is significantly faster and cheaper than human review while maintaining statistical rigor and unbiasedness.

Open the full episode