From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection
The study investigated three methods for detecting hallucinations in Whisper's Automatic Speech Recognition (ASR) output: text metrics, Large Language Models (LLMs), and internal decoder states. While text-based and LLM methods showed promise, probing Whisper's intermediate decoder states using a BLSTM classifier achieved a competitive F1 score of 62.1% without needing reference transcripts. A late-fusion ensemble of these methods yielded the highest overall performance (F1: 68.3%).