Calibrating LLM Explanations to Human Decisions with Reinforcement Learning

arXiv:2610.17306 · cs.HC, cs.LG · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Calibrating LLM Explanations to Human Decisions with Reinforcement Learning".

Jane: The gist: HD-RL, a reinforcement learning framework, calibrates an LLM explanation by optimizing it against observed human decisions to improve human decision-making quality.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So if we boil it down for the listener on this paper, HD-RL is a way to train an LLM explanation policy to maximize the expected utility of human decisions induced by those explanations Tom. It’s about using observed human choices to build a simulator and then using that simulator as a proxy reward model Tom.

Jane: The implication is that for AI systems that interact with people, we need to ground their outputs in what actually helps people make good choices, rather than just making them look right on a benchmark or pleasing to the user Jane. It shifts the focus from model-centric metrics to human-centric outcomes Jane.

Lu: What this means for representation learning is that you can use external decision data, even if it's observational, to guide the explanation generation process towards a more useful distribution of explanations Lu. It’s about injecting real-world decision theory into the training loop.

Meng: From a deployment standpoint, this framework gives us a way to push an AI not just to be technically competent but also contextually effective in how it communicates its findings to end users Meng. That contextual effectiveness is something that usually gets ignored in standard RL setups.

Lalam: And they've released their human decision dataset and the trained model, which opens the door for other researchers to build on this work with more data or different tasks Lalam. It’s an open contribution to the field of explanation calibration.

Tom: So that’s it for this discussion on "Calibrating LLM Explanations to Human Decisions with Reinforcement Learning" Tom. We've covered what the paper is about, how they did it, and what this means for making AI explanations actually useful in collaboration with humans.

Conclusion: Tom: So, to recap, HD-RL is this reinforcement learning framework that takes an LLM explanation and calibrates it by optimizing it against what real people actually decide when they see that explanation Tom.

Jane: It’s essentially trying to make sure the AI explanations actually help humans make better choices on their own tasks instead of just being technically accurate or pleasing to the model Jane.

Lu: The core idea is training a decision simulator on observed human decisions and then using that simulator as a reward model to fine-tune the LLM policy Lu. It’s about grounding the explanation in human action utility Lu.

Meng: From what I see, they’re focusing on how to translate those messy human decisions into something the AI can actually use to guide its output generation Meng. It sounds like a practical step toward making AI more useful in real-world settings Meng.

Lalam: I'm pretty sure the authors are showing that this method significantly boosts human accuracy when dealing with text generated by an AI, outperforming other alignment methods by a noticeable margin Lalam.

Tom: That's good to hear about the results, Lalam. But what’s the actual takeaway for us as developers? It means we can start optimizing explanations not just for internal metrics but for actual human decision quality Tom.

Jane: Exactly. We need to move past just making sure the text looks plausible and start thinking about how that text steers the person reading it toward a better outcome Jane.

Lu: The design of the explanation format itself is also part of this work, using structured feature indicators like "Look for specific linguistic or semantic feature" which they say minimizes reading time and forces attention back to the text Lu.

Meng: That formatting choice makes sense from a deployment standpoint. If people have to spend too much time deciphering the explanation itself, it defeats the purpose of a quick decision-making tool Meng.

Tom: So, we've seen how they train this simulator and optimize the policy, and now we see the final outcome is higher human detection accuracy than other methods Tom.

Jane: It’s a big shift because it moves away from just measuring how well the AI matches some reference model to actually measuring how useful its output is for a person in a real situation Jane.

Lu: The authors are releasing their decision dataset and the trained model, which means this isn't just a theoretical concept; they’ve provided the tools for others to build on this work with more data Lu.

Meng: That open contribution is what matters. If we can use that training data and framework, we can start designing explanations that actually perform better when deployed at scale Meng.

Tom: So, in summary, HD-RL is a way to train LLM explanations using observed human decisions to directly improve the quality of those human decisions Tom.

Jane: It’s about making sure the AI's guidance is relevant to what people actually need it for Jane.

Lu: The next thing we should look at is how they handled the limitations, because every method has its stopping point Lu.

Dong Shu Ziyang Guo, Negar Kamali, Matthew Groh, Jessica Hullman

Northwestern University

cs.HC, cs.LG

Submitted: 2026-10-12

Updated: 2026-10-12

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist: HD-RL, a reinforcement learning framework, calibrates an LLM explanation by optimizing it against observed human decisions to improve human decision-making quality.

Key concepts

Decision-Theoretic Framework: This framework evaluates explanation quality by modeling how well it helps a person choose the best action. It trains a simulator on human data to predict the probability of decisions given a text and an explanation, using this prediction as a reward signal for optimizing the LLM.
Decision Simulator (Sϕ): This model is trained on real human decisions to approximate how people will react to different explanations. It takes a text and an explanation as input and predicts the probability distribution of human choices. This simulator acts as a proxy reward model for the reinforcement learning process.
Surrogate Reward (rϕ): Since directly optimizing expected utility is complex, HD-RL uses a log-transformed surrogate reward. This reward guides the LLM policy to favor explanations that lead humans to make correct decisions, while also penalizing explanations that seem unlikely based on the simulator's predictions.
Explanation Format: Explanations are structured as localized feature indicators, like 'Look for [specific linguistic or semantic feature]'. This format is chosen because it is quick for humans to read, refocuses attention on the original text, and never reveals the final decision to the user.

Terminology

Reinforcement Learning — 1× in this paper · explained in 4 episodes

Summary

The gist: HD-RL, a reinforcement learning framework, calibrates an LLM explanation by optimizing it against observed human decisions to improve human decision-making quality.

Introduction and Motivation

Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality This paper introduces HD-RL, a reinforcement learning framework for optimizing LLM explanations for human decision making. The approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that help people to make good decisions.

The Decision-Theoretic Framework

The quality of an explanation is evaluated by whether it helps the decision-maker select a good action under the utility The framework models this by training a decision simulator on a large-scale human decision dataset to predict the probability distribution of human decisions conditioned on a given text and explanation. This simulator is then used as a proxy reward model to optimize an LLM policy via reinforcement learning. The target objective is defined as JH(θ) = E(x,y)∼DEe∼πθ(·x)Ea∼PH(·x,e)[U(a, y)]

Modeling Human Decisions and Policy Optimization

To directly optimize JH(θ), the framework learns a human response model Sϕ(a x, e) to approximate PH(a x, e) from observed human decisions. This allows the predicted utility of an explanation on instance (x, y) to be calculated as Vbϕ(x, e, y) = Xa∈A Sϕ(a x, e)[U(a, y)] The RL policy is then optimized using a log-transformed surrogate reward rϕ(x, e, y) = max [1 + log qϕ(x, e, y), −5]. This surrogate preserves the ordering of candidate explanations by predicted human decision utility and places greater penalty on explanations for which the response model assigns very low probability to the correct human action. The final policy optimization objective is JRL(θ) = E(x,y)∼DEe∼πθ(·x)[rϕ(x, e, y) − βDKL (πθ(· x)∥πref(· x)) − γ len(e)].

Implementation and Results

The framework is demonstrated on an AI text detection task involving 21K real-world human decisions on 2712 unique texts. The results show that explanations generated via HD-RL achieve a GLMM-estimated human detection accuracy of 63.7%, outperforming RLVR, RLAIF, and other baselines by over 8%. This improvement is significant because other methods yield low accuracy on AI-generated texts, while HD-RL explanations elevate human accuracy on AI texts to 63.0%

Baseline Comparisons and Analysis

The study compares HD-RL against three baselines: Direct Preference Optimization on AI Feedback (DPO-RLAIF), Verifiable-Reward RL (VR-GRPO), and LLM Decision Calibration. The GLMM analysis confirms that HD-RL yields a significantly higher overall human utility than all tested baselines when adjusting for text and participant variance Specifically, Table 4 shows that the HD-RL condition has the highest estimated marginal accuracy across both AI and Human ground-truth distributions. Furthermore, Table 5 indicates that HD-RL exhibits the most balanced performance (1.29×) in terms of differential accuracy contrasts between human and AI texts compared to standard alignment baselines.

Design of Explanation Format

The explanations are formatted as a structured set of localized feature indicators, such as “Look for [specific linguistic or semantic feature]”. This format was selected because it minimizes reading time, forces the human to redirect their attention back to the text, and never reveals a decision to the user.

Conclusion

HD-RL successfully aligns LLM explanations with downstream human utility by generating explanations that help people make good decisions This work highlights the need for grounding LLM outputs in human action to ensure their usefulness in human-AI collaborative settings. The paper releases its human decision dataset and trained model for future work.

--- Page 1 ---

--- Page 2 ---

--- Page 3 ---

--- Page 4 ---

--- Page 5 ---

--- Page 6 ---

--- Page 7 ---

--- Page 8 ---

--- Page 9 ---

--- Page 10 ---

How it works

The HD-RL framework involves five steps: Texts Collection, Explanation Generation, Human Decision Collection, Decision Simulator Training, and Reinforcement Learning. The dataset construction involved scraping 10,000 texts from Reddit and using a two-stage LLM pipeline to create balanced AI-generated texts. During the human decision collection phase, participants were presented with a text and an explanation, selecting an action at ∈ A = Y = 2.

Decision Simulator Training

A human response model Sϕ is trained on observed human decisions to approximate PH(a x, e). This simulator takes an (x, e) pair and predicts the human decision distribution. The simulator’s final predicted probability, pˆi = Sϕ(a = 1 xi, ei), is the unweighted average of the outputs from five shallow MLPs. The simulator performs well on out-of-sample generalization, achieving a predictive accuracy of 71.0% on a test set.

Reinforcement Learning

The RL architecture comprises a trainable explanation policy πθ, a frozen reference model πref, and the frozen decision simulator Sϕ. The policy is optimized using Group Relative Policy Optimization (GRPO) to maximize the surrogate reward rϕ(x, e, y) = max [1 + log qϕ(x, e, y), −5]. This reward is computed as rk = max (1 + log qϕ(x, ek, y), −5.0), where log qϕ(x, ek, y) = −BCE(ˆpk, y).

Evaluation Metrics

The simulator’s predictive fidelity is evaluated using Log Loss (Binary Cross-Entropy), Brier Score, and Expected Calibration Error (ECE). The ECE measures the expected absolute difference between a bin’s empirical action 1 frequency and its mean predicted probability. The simulator's performance on the deployment dataset showed a Log Loss of 0.09 and a Brier Score of 0.03.

Limitations

A limitation noted is that the implementation relies on a logarithmic surrogate reward rather than directly optimizing the continuous expected accuracy qϕ(x, e, y) Future work should compare this surrogate against the direct optimization of predicted expected utility.

--- Page 12 ---

--- Page 13 ---

--- Page 14 ---

--- Page 15 ---

--- Page 16 ---

--- Page 17 ---

--- Page 18 ---

--- Page 19 ---

--- Page 20 ---

How it works

Improvements for AI systems

Bold Header: Decision-Theoretic RL Framework Implementation

This framework optimizes LLM explanations to maximize the expected utility of human decisions induced by the explanations (Equation 1), moving beyond optimizing for autonomous task performance or subjective human preference.

Bold Header: Human Decision Simulator Training and Calibration

Training a simulator on observed data allows the system to predict the distribution of human actions, enabling a reward signal via logarithmic surrogate reward (Equation 5) that directly targets explanations that help people to make good decisions.

Bold Header: Policy Optimization via Group Relative Policy Optimization (GRPO)

The policy model is optimized using GRPO, where the advantage Ak is standardized against the mean and standard deviation of rewards, ensuring the LLM learns to generate explanations that maximize this calibrated human utility signal.

Abstract

Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks. However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality. To bridge this gap, we introduce HD-RL, a reinforcement learning framework for optimizing LLM explanations for human decision making. Our approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that help people to make good decisions. We apply this framework to AI-generated text detection. In our initial data collection phase, we observed that while LLMs possess strong autonomous detection capabilities, the explanations they generate for users yield near-random human accuracy (51.3%). By applying the HD-RL framework to the open-source Qwen 3.6 model, we generate optimized explanations that improve human performance, achieving a GLMM-estimated human accuracy of 63.7% in a randomized between-subjects experiment with repeated measures and outperforming RLVR, RLAIF, and other baselines by over 8%.