Single-Pass Uncertainty Heads for Claim-Level Hallucination Detection in Persian Medical Language Models
cs.CL
Submitted: 2026-10-02
Updated: 2026-10-07
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Hallucination detection is particularly important for medical language models, but repeated-sampling approaches are computationally expensive.
Terminology
Abstract
Hallucination detection is particularly important for medical language models, but repeated-sampling approaches are computationally expensive. A faster alternative is a single-pass uncertainty head that predicts hallucination risk from a frozen generator's internal signals. Existing uncertainty heads consume backbone-specific features and tokenization, motivating adaptation when the backbone or language changes. We study two Persian medical models, Gaokerena-V and Gaokerena-R, on a 168-question Iranian medical entrance examination. Across five generations per question, Gaokerena-V produces the same option on only 14 questions and Gaokerena-R on 37, compared with 168 for Med-Gemma, indicating substantial response variability in the Gaokerena models. We therefore adapt the LLM Uncertainty Head (LUH) framework to these models and construct two paired claim-level hallucination datasets directly in Persian, with 1,600 responses per backbone. Lightweight heads are trained on frozen-backbone attention maps and token probabilities. On held-out test splits, the heads achieve PR-AUCs of 0.4820 and 0.4652, corresponding to 2.30 and 2.66 times their respective random baselines, and ROC-AUCs of 0.7852 and 0.7810. The resulting detectors require neither retrieval nor repeated sampling at inference.
Sources
- A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs
- Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model
- Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Training
- A Survey of Uncertainty Estimation Methods on Large Language Models
- Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents
- ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models
- RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
- Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progress
- Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
- MedGemma Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering