Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism

Name: Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism
Uploaded: 2026-09-07T19:49:50.026359+00:00
Duration: 364 s
Description: The episode discusses a paper demonstrating that Large Language Model (LLM) nondeterminism stems from finite precision arithmetic on GPUs, not random glitches. This variability is systemic and affects token probabilities most when they are between 0.1 and 0.9. A key practical takeaway is using probability analysis instead of repeated inference runs to evaluate risk efficiently.

The episode discusses a paper demonstrating that Large Language Model (LLM) nondeterminism stems from finite precision arithmetic on GPUs, not random glitches. This variability is systemic and affects token probabilities most when they are between 0.1 and 0.9. A key practical takeaway is using probability analysis instead of repeated inference runs to evaluate risk efficiently.

Open the full episode