Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Models
cs.CL, cs.AI, cs.LG
Submitted: 2026-05-29
Updated: 2026-09-27
Code: https://github.com/AthinaKyriakou/shared-doubt
License: http://creativecommons.org/licenses/by/4.0/
The gist: Confidence estimation (CE), i.e.
Terminology
Abstract
Confidence estimation (CE), i.e. quantifying the reliability of a model's prediction, has attracted great interest in the context of large language models (LLMs). However, most studies focus on English, ignoring the multilingual reality of LLM usage, while many CE methods degrade or require retraining across languages. To address this gap, we investigate whether multilingual LLMs encode shared, language-transferable confidence features in open-ended question answering. We use a lightweight linear probe that predicts answer correctness directly from intermediate representations. Trained monolingually, the probe generalizes zero-shot to unseen, linguistically diverse languages without target-language supervision. Multiple ablations and learned layer weights reveal that confidence features concentrate in middle layers across languages, suggesting a shared confidence subspace. While zero-shot cross-lingual performance depends on similarity to the source language, the probe provides a strong baseline without any retraining and compares favorably to other popular confidence estimation methods.
Sources
- Uncertainty in Natural Language Generation: From Theory to Applications
- Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study
- Discovering Latent Knowledge in Language Models Without Supervision
- Answer Matching Outperforms Multiple Choice for Language Model Evaluation
- Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States
- IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation
- A Primer on the Inner Workings of Transformer-based Language Models
- The Llama 3 Herd of Models
- A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice
- On the Origins of Linear Representations in Large Language Models
- Language Models (Mostly) Know What They Know
- Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysis
- Teaching Models to Express Their Uncertainty in Words
- Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach
- LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses
- Confidence over Time: Confidence Calibration with Temporal Logic for Large Language Model Reasoning
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- The Linear Representation Hypothesis and the Geometry of Large Language Models
- Qwen3 Technical Report
- Do Multilingual LLMs Think In English?
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering