A Conditional Mutual Information Analysis of Response Variety in Language Models
cs.CL
Submitted: 2026-11-02
Updated: 2026-11-02
Comments: 8 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models provide a variety of responses at inference (generation) that conventionally depend on a temperature parameter.
Terminology
Abstract
Large language models provide a variety of responses at inference (generation) that conventionally depend on a temperature parameter. Using a conditional mutual information approach, a quantitative, entropy based analysis of response variety is given. Three limiting behaviors are defined: arbiter, independent (conditionally) and neutral (deterministic) that span no to maximal mutual information. We give examples of each behavior and suggest their application to: temperature analysis, language analysis, prompt compression, and knowledge graphs.
Sources
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- Retrieval-Augmented Generation for AI-Generated Content: A Survey
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data