The Necessity of Setting Temperature in LLM-as-a-Judge

The gist: Higher temperatures generally decrease judgment consistency and increase formatting errors, while also exposing latent uncertainty that tends to remain suppressed under low-temperature decoding, particularly in ambiguous cases.

Introduction and Motivation

Using large language models (LLMs) as judges for evaluating model outputs has emerged as an important paradigm for automated evaluation. The choice of decoding temperature in LLM-as-a-Judge settings is still largely chosen empirically, with limited systematic evidence on its impact. This work conducts a systematic study of how temperature affects judgment behavior across different LLM judge models, prompting strategies, and evaluation paradigms. Low-temperature settings are better suited to tasks that prioritize stability and reproducibility, whereas highertemperature settings are more appropriate for scenarios involving substantial ambiguity or complexity.

Methodology

The study examines six temperature settings: [0.01, 0.5, 1.0, 1.5, 2.0, 3.0]. Each experimental condition is repeated ten times using different random seeds. The researchers employ Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 and google/gemma-3-27b-it as primary open-weight judge models. Three representative LLM-as-aJudge paradigms are tested: pairwise comparison, single answer grading, and reference guided grading. These paradigms are tested using both the direct base prompt and the Chain of Thought (CoT) prompt.

Evaluation Metrics

Three metrics are used to evaluate the judge behavior: agreement rate (AGR), consistency (CON), and error rate (ERR). Agreement Rate measures the extent to which decisions produced by the LLM judge align with human-verified reference labels, serving as a primary indicator of judge performance. Consistency uses 1-Flip Consistency, which measures judge stability across repeated runs under identical experimental conditions by computing the proportion of adjacent repeats with consistent judgments. Error Rate denotes the rate at which the LLM judge fails to correctly parse or follow the prescribed judgment format across different temperature settings.

Key Findings on Temperature Effects

The results show that higher temperatures generally decrease judgment consistency and increase formatting errors, while also exposing latent uncertainty that tends to remain suppressed under low-temperature decoding. Specifically, at low temperature (T = 0.01), all configurations exhibit exceptionally high consistency (Consistency ≈ 1.00) and near-zero error rates (Error Rate ≈ 0.00). At high temperature (T = 3.0), consistency degrades substantially and error rates increase markedly, demonstrating that high temperatures severely compromise the reliability of the Judge. The analysis also reveals that higher temperatures can serve as an exploratory mechanism and may improve judging performance in complex or uncertain evaluation scenarios.

Ensemble Thermo-Judge

The researchers propose a method called Ensemble Thermo-Judge to exploit temperature-induced variability by aggregating multiple high-temperature samples via majority voting. They partition the dataset into Ambiguous and Non-ambiguous subsets based on low-temperature consistency and high-temperature disagreement criteria. This approach demonstrates that repeated high-temperature judgments improve Agreement for ambiguous cases, suggesting that additional diversity can be effectively exploited via majority voting.

Conclusion

In practical terms, when stability, reproducibility, and cost efficiency are the primary concerns, low-temperature settings are strongly recommended. In contrast, for complex evaluation scenarios that require nuanced, multi-perspective reasoning or deeper deliberation, higher temperatures offer tangible advantages. Higher temperatures can instead yield consistent gains when combined with ensembling in ambiguous cases. This suggests that temperature should be treated not as a fixed hyperparameter, but as a controllable, task-dependent design choice that mediates the trade-off between reliability and exploration.

Limitations and Future Work

The study restricts experiments to open-source models for transparent control over temperature configurations. Furthermore, inference outcomes may vary with random seeds, temperature settings, and hardware environments, a effect more pronounced at higher temperatures up to T = 3.0. For future work, a promising direction is combining a high-temperature agent for exploratory initial judgments with a low-temperature agent for output refinement. The findings advocate for taskadaptive temperature strategies in LLM evaluation pipelines, moving beyond fixed low-temperature defaults to enhance robustness.

Appendix Details

The paper provides detailed prompts for various judge types and strategies, including System Prompt Template, Pairwise Comparison Direct Judge Prompt, and Reference Guided Grading Direct Judge Prompt. Examples of LLM responses under different temperatures illustrate how higher temperatures lead to more diverse and less constrained generation patterns. The analysis includes a Supplementary Causal Analysis using a 3-Fold CrossFitted Augmented Inverse Probability Weighting (Cross-Fitted AIPW) estimator to estimate the Average Treatment Effect (ATE) of temperature. Table 4 and Table 5 present quantitative assessments of ATE across different judge types, prompt variants, and model levels. The analysis also shows that Consistency emerges as the most influential factor when prioritizing Consistency in the SHAP beeswarm plots. The paper concludes by emphasizing that higher temperatures can instead yield consistent gains when combined with ensembling in ambiguous cases. This suggests that such effects may stem from the increased diversity induced by high-temperature sampling, which allows the model to explore a broader range of implicit reasoning paths or “personas”. The paper advocates for taskadaptive temperature strategies in LLM evaluation pipelines, moving beyond fixed low-temperature defaults to enhance robustness. This work conducts large-scale controlled experiments to systematically investigate the impact of temperature on the performance of LLMs as evaluators, providing actionable guidance for both industrial practice and future research on temperature effects. The findings demonstrate that lowtemperature settings yield near-perfect consistency and negligible error rates, whereas higher temperatures lead to substantial degradation in consistency and a sharp increase in error rates. These effects vary across model types, evaluation paradigms, and prompting strategies. Quantitatively, temperature exhibits a strong negative correlation with consistency and a strong positive correlation with error rates, indicating that it is a primary determinant of evaluation reliability rather than a negligible engineering detail. Beyond these general trends, aggregate statistics and case studies reveal nuanced benefits of higher temperatures. Specifically, elevated temperatures can expand both the depth and breadth of reasoning exhibited by LLM evaluators and alter their judgment patterns. Contrary to the common assumption that high-temperature outputs merely reflect unstructured or illogical hallucinations, our analysis shows that such models do engage in structured reasoning, but frequently deviate from task instructions, thereby increasing parsing errors and reducing overall performance. Notably, instruction violations are not entirely eliminated even under low-temperature settings, challenging the prevailing view that low temperature is universally optimal for evaluation tasks. Consequently, higher temperatures become meaningful in scenarios requiring deeper or more exploratory reasoning, although they should be paired with multiple sampling runs under different random seeds. Furthermore, by partitioning the evaluation set into Ambiguous and Non-ambiguous subsets—where the former comprises inherently more challenging cases—we observe that temperature exerts a substantially greater influence on the Ambiguous subset, while having minimal impact on the Non-ambiguous subset. This observation suggests that performance variations across temperature settings can serve as an implicit indicator of instancelevel ambiguity. Building on this insight, we evaluate a simple yet effective method, Ensemble Thermo-Judge, on the Ambiguous subset, demonstrating that repeated high-temperature judgments improve AGR. Additionally, we find that CoT achieves better performance at higher temperatures within this subset, further supporting the utility of diversified temperature configurations. In practical terms, when stability, reproducibility, and cost efficiency are the primary concerns, lowtemperature settings are strongly recommended. In contrast, for complex evaluation scenarios that require nuanced, multi-perspective reasoning or deeper deliberation, higher temperatures offer tangible advantages. Higher temperatures may adversely affect the model’s instruction-following capability; for example, they can destabilize adherence to the required JSON output format. This filtering procedure enables a cleaner assessment of the impact of high-temperature settings on judgment outcomes, free from confounding factors introduced by format inconsistencies or related artifacts. The paper concludes by emphasizing that temperature should be treated not as a fixed hyperparameter, but as a controllable, task-dependent design choice that mediates the trade-off between reliability and exploration. The second row of Figure 3 shows under which conditions the temperature causal effect is stronger or weaker (more causal effect analysis, see Appendix F). For Agreement, the group-level ATEs are mostly in the range −0.03 to −0.01, with small dispersion; for Consistency, the group-level ATEs range from −0.17 to −0.10, indicating more pronounced heterogeneity.

1. A temperature-aware judgment strategy can be implemented by using Ensemble Thermo-Judge for ambiguous cases, as it leverages temperature-induced variability to enhance robustness, specifically by combining temperature-induced diversity with majority voting to yield more robust and reliable judgments.

2. The improved system can dynamically adjust its sampling strategy based on case complexity; for ambiguous cases, the system should employ high temperatures (T ∈ 1.5–3.0) combined with ensemble decoding to expose latent uncertainty that tends to remain suppressed under low-temperature decoding.

3. For tasks prioritizing stability and reproducibility, the system should default to low-temperature settings (e.g., T = 0.01), as this setting is shown to yield exceptionally high consistency (Consistency ≈ 1.00) and near-zero error rates.

4. The system can be designed to mitigate instruction-following errors by explicitly filtering out samples that violate the required format from the ensemble voting set, thereby focusing the assessment on judgment quality rather than parsing failures.

Large Language Models (LLMs) are increasingly used as automated judges, yet decoding temperature is often set by intuition rather than evidence. We run over one million large scale evaluations, multiple open-source models, prompting strategies, and judging paradigms to examine how temperature affects judge reliability. Low temperatures yield consistent outputs with few errors, especially in smaller models, while higher temperatures reduce consistency and increase errors. Yet consistency alone does not guarantee sound judgments: low temperatures can narrow the range of reasoning explored. We characterize judge behavior through four reasoning personas and find that temperature shifts their distributions in model-dependent ways, most visibly in ambiguous cases where competing answers are hard to distinguish. Repeated high-temperature sampling reveals this hidden uncertainty, and combining it with majority voting or using high temperature for reasoning and low temperature for the final decision, improves agreement on ambiguous cases without hurting clear ones, at the cost of extra inference. Our findings suggest a simple rule: keep temperature low for stable outputs, but explore more when the judgment is hard.