A Conditional Mutual Information Analysis of Response Variety in Language Models

arXiv:2611.03836 · cs.CL · Submitted 2026-11-02 · Read on arXiv

cs.CL

Submitted: 2026-11-02

Updated: 2026-11-02

Comments: 8 pages

License: http://creativecommons.org/licenses/by/4.0/

The gist: Large language models provide a variety of responses at inference (generation) that conventionally depend on a temperature parameter.

Terminology

Abstract

Large language models provide a variety of responses at inference (generation) that conventionally depend on a temperature parameter. Using a conditional mutual information approach, a quantitative, entropy based analysis of response variety is given. Three limiting behaviors are defined: arbiter, independent (conditionally) and neutral (deterministic) that span no to maximal mutual information. We give examples of each behavior and suggest their application to: temperature analysis, language analysis, prompt compression, and knowledge graphs.

Sources

Related papers