Revisiting the ABCs of Working with AI: Evidence from Radiologists
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Revisiting the ABCs of Working with AI".
Warren: Here's the gist of the paper. Artificial intelligence (AI) systems are being considered for a wide array of professional settings.
Ruth: This note shows that their results replicate to a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions.
Warren: Lower baseline ability and higher calibration predict larger incremental value from AI assistance.
Ruth: Replication of Core Findings The study replicates findings from Caplin et al.
Title and authors: Ruth: We’ve talked about the setup and the initial findings showing that ability and calibration matter more than just raw skill when it comes to getting value from AI assistance. Now let's look at what the paper actually summarizes in terms of its main message.
Warren: Essentially, the paper is confirming that AI assistance isn't a blanket benefit for everyone; it’s conditional on who you are and how you assess your own judgment.
Iris: The summary highlights that the distribution of gains from AI is not uniform across the user population; some users get much more value from it than others depending on their baseline ability and calibration.
Ruth: They point out that AI can actually narrow performance gaps by boosting lowerperforming human users, as we see in other work, but only if those humans have the right beliefs about what they are doing.
Warren: The core takeaway is that for AI to be truly beneficial for productivity, we need to focus on tailoring the assistance based on these two traits: ability and calibration.
Iris: They emphasize that calibration matters because a human operator needs to know precisely when they should trust their own judgment and when they should defer to the algorithm's prediction.
Ruth: So, it’s not just about getting the right answer; it’s about knowing *when* you are right enough to trust your own assessment versus when you need the AI's input.
Warren: That speaks directly to how we design these systems—they can't just dump all the algorithmic output on a user and expect productivity to go up across the board.
Iris: It suggests that future work should focus on building interfaces that actively prompt users when their reported confidence is drifting away from what the true probability of correctness is.
Ruth: That feeds right into the idea of calibration-aware interfaces, which we touched on earlier—making sure users recognize when they should step back and let the AI take over.
Warren: If we want these productivity gains to generalize, we have to move beyond just measuring accuracy and start measuring how well people understand their own accuracy.
The paper's summary: Ruth: The authors themselves suggest a few ways they can improve this work and what that means for future research. They are basically proposing where the next steps should go.
Warren: One improvement they bring up is the need to look at how different experimental designs affect the interaction estimates, which we already touched on as a way to check their robustness.
Iris: They also suggest focusing on generalizing these findings beyond just chest X-rays and age guessing tasks, aiming for broader applicability in professional diagnostic settings.
Ruth: That’s a big one; they want to see if these ability and calibration patterns hold up when we move from medical imaging to other complex AI-assisted workflows.
Warren: And another point is the need for better identification of the causal effect, specifically looking at whether it’s just ability or if it's the interaction between them that drives the outcome.
Iris: They are pushing for more rigorous ways to prove that the interaction term, like ability by AI, is truly what’s driving the heterogeneous value of AI assistance.
Ruth: It also suggests a deeper look at how training and delegation policies should be structured based on these findings—not just basic skill training.
Warren: That means policies could prioritize interventions for experts who are highly confident in their own judgment but have low foundational skill, which is where the highest predicted gains lie.
Iris: And they’re pointing toward developing methods to measure calibration more finely, because their current measurement might not be capturing all the necessary nuances of belief about performance.
Ruth: Overall, it seems they are pushing for a framework that integrates these factors into how we think about deploying AI in professional work, not just as a standalone technology.
The paper's improvements: Ruth: So to wrap up on "Revisiting the ABCs of Working with AI: Evidence from Radiologists," the main message is that AI assistance provides differential returns based on a user’s ability and their calibration.
Warren: It confirms that gains are not universal; they are highest for users who start with lower skill but have strong self-knowledge about their accuracy.
Iris: The paper shows the calibration interaction effect is statistically significant using wild cluster inference, meaning those results weren't just a fluke in the data structure.
Ruth: This means that when we think about deploying these systems, we need to design things to account for this heterogeneity in how people interact with the AI.
Warren: The implication is that training and interface design should target not only skill but also users’ knowledge of their own accuracy to maximize the benefit of AI assistance.
Iris: So it confirms that building systems that prompt users when they are overconfident or underconfident is a practical way to leverage these findings in real-world applications.
Conclusion: Ruth: So we’ve been looking at "Revisiting the ABCs of Working with AI: Evidence from Radiologists," and what we're seeing is that whether AI helps a radiologist depends heavily on their baseline ability and how well they know themselves.
Warren: Exactly. The core finding is that users who are less skilled but more accurate in their self-assessment get the most value from the AI assistance. It’s not just about raw skill anymore; it’s about calibration.
Iris: I think what's really interesting is how they set up their experiment, using repeated cases to make sure they aren't just catching some fluke in a single study setup. They show that this pattern holds up across different designs.
Ruth: That external validity is key here, right? It suggests that if we want AI to actually boost productivity in the real world, we can’t just look at controlled lab settings.
Warren: We have to look at how these ability and calibration patterns play out when everyone is using the system, because if a whole group of people starts behaving this way, the costs for running those systems change dramatically.
Iris: And I see a potential issue in their setup—they estimated some interactions, but they also noted that the ability interaction was estimated less precisely than the calibration interaction. That means we might not be as sure about that specific relationship as we are about how calibration works with AI.
Ruth: That’s a fair caution. It tells us to be careful when we make big claims about ability versus calibration interactions until we get more data like that, you know, better precision on the numbers.
Warren: And from a trading side, it means if we try to automate decisions based on this, the risk profile changes depending on whether the trader is an expert or just someone who happens to be guessing well.
Iris: It also makes me think about calibration-aware interfaces again. If users know they are overconfident, they should probably be prompted to check their work instead of blindly following an AI suggestion that might be wrong because of that overconfidence.
Ruth: That’s a solid point for future work—designing systems that encourage users to pause and check their own judgment, not just accept the output.
Warren: Yeah, so we move from just asking "what is the answer" to asking "how confident am I in my answer right now." That shifts the entire interaction.
Iris: And as for me, I’m still watching how these findings might apply outside of medical imaging. If this ability-calibration dynamic is general to complex decision making, it has wide implications.
Ruth: It really does. So we leave "Revisiting the ABCs of Working with AI: Evidence from Radiologists" thinking about how tailoring the assistance to the user's knowledge of their own accuracy is going to be a big part of deploying these tools.
Warren: Yeah, and I’m curious what that looks like for high-frequency trading strategies. If we can figure out how to measure calibration in real-time, that could give us a whole new layer for risk management.
Iris: And I think the next big thing is proving this on different types of professional tasks. Not just one medical field, but maybe finance or engineering where the stakes are different.
Ruth: We'll keep an eye out for those papers showing how these effects manifest outside the lab environment next week.
Daniel Martin, *
University of California, Santa Barbara
econ.GN, cs.HC, q-fin.EC
Submitted: 2026-06-10
Updated: 2026-11-15
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: Artificial intelligence (AI) systems are being considered for a wide array of professional settings.
Key concepts
- Ability (Ah)
- This measures a radiologist's baseline skill in reading chest X-rays without AI assistance. It is estimated from cases where the radiologist reads cases without machine learning predictions. Higher ability generally means a higher baseline skill level.
- Calibration (Ch)
- Calibration reflects how well a radiologist's reported confidence matches their actual accuracy across different cases. A high calibration score indicates that when a radiologist is confident, they are usually correct, suggesting better self-assessment of performance.
- Incremental Value
- This refers to the extra benefit or productivity gain achieved by using AI assistance compared to working without it. The research found that this gain is not uniform; it varies significantly depending on whether a radiologist has low ability and high calibration, for example.
- Interaction Effects
- These describe how ability and calibration jointly influence the impact of AI. The key finding is that the combination of being less able but highly calibrated results in the largest predicted gains from using AI assistance.
Terminology
Summary
Artificial intelligence (AI) systems are being considered for a wide array of professional settings. p. 1 1 Introduction
Artificial intelligence (AI) systems are being considered for a wide array of professional
settings. A natural first question is whether AI raises decision quality on average. Li,
Martin, Marx, Weidmann, and Ye (2025b) provide evidence that two characteristics,
ability and belief calibration, help to determine the returns to AI assistance. This note
shows that their results replicate to a setting where professional radiologists analyze
chest X-rays with access to state-of-the-art machine learning predictions. Li, Martin, Marx, Weidmann, and Ye (2025b), I use the
radiologist assessments from the repeated-case designs, which include 68 radiologists
and 11,420 paired radiologist-patient-pathology observations. The results of this replication support the external validity of their core findings: lower baseline ability and
higher calibration predict larger incremental value from AI.Where the paper says it
Where the paper says it
Where the paper says it
Replication of Core Findings
The study replicates findings from Caplin et al. p. 0 Revisiting the ABCs of Working with AI:
A Replication with Radiologists∗
Daniel Martin†
June 10, 2026
Abstract
Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S. Revisiting the ABCs of Working with AI:
A Replication with Radiologists∗
Daniel Martin†
June 10, 2026
Abstract
Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S. Revisiting the ABCs of Working with AI:
A Replication with Radiologists∗
Daniel Martin†
June 10, 2026
Abstract
Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S. Li,
Martin, Marx, Weidmann, and Ye (2025b) provide evidence that two characteristics,
ability and belief calibration, help to determine the returns to AI assistance. This note
shows that their results replicate to a setting where professional radiologists analyze
chest X-rays with access to state-of-the-art machine learning predictions. Li, Martin, Marx, Weidmann, and Ye (2025b), I use the
radiologist assessments from the repeated-case designs, which include 68 radiologists
and 11,420 paired radiologist-patient-pathology observations. The results of this replication support the external validity of their core findings: lower baseline ability and
higher calibration predict larger incremental value from AI. Pooling these designs yields 68 radiologists and 11,420 paired radiologist-patient-pathology
observations before aggregation to two outcome observations per radiologist. This paired
structure makes the AI comparison less confounded by case difficulty, pathology mix, or
radiologist composition, and the common cases make radiologist-level ability and calibration
more comparable across radiologists. Li, Martin, Marx, Weidmann, and Ye (2025b), I use the
radiologist assessments from the repeated-case designs, which include 68 radiologists
and 11,420 paired radiologist-patient-pathology observations. The results of this replication support the external validity of their core findings: lower baseline ability and
higher calibration predict larger incremental value from AI.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Data Construction and Estimation
To faithfully reproduce the CDLMMWY analysis, a dataset was constructed that allows ability and calibration to be estimated on one block of cases and the impact of AI on productivity to be estimated on another block of cases. This was accomplished using repeated-case designs where the same radiologist reads the same patient-pathology cases in all four randomly ordered treatment arms. p. 2 2 To accomplish
this with the radiologist data, I use the repeated-case designs of the AMRS experiment
(Design 2 and Design 3). In Design 2, the same radiologist reads the same patient-pathology
cases in all four randomly ordered treatment arms, and in Design 3, the same session-level
radiologist-patient-pathology readings are repeated first without and then with AI assistance. This paired
structure makes the AI comparison less confounded by case difficulty, pathology mix, or
radiologist composition, and the common cases make radiologist-level ability and calibration
more comparable across radiologists. For the pooled main sample and each design-specific appendix sample, I assign half of
patients to a “skill block” and the other half to an “outcome block”. For the pooled main sample and each design-specific appendix sample, I assign half of
patients to a “skill block” and the other half to an “outcome block”. I then estimate ability
and calibration on the skill block cases without AI and compare performance with and
without AI in the outcome block.Where the paper says it
Where the paper says it
Where the paper says it
Key Interaction Effects
The regression specification used is Yht = α0 + α1Ah + α2Ch + β0AIt + β1(Ah × AIt) + β2(Ch × AIt) + εht. The key interaction coefficients from the fully specified model compare the pooled AMRS replication results directly to the corresponding estimates of CDLMMWY. p. 4 The accuracy distributions are roughly comparable because
CDLMMWY selected images to produce a similar performance distribution to that in AMRS. Table 1 summarizes the key interaction coefficients from the fully specified model, comparing the pooled AMRS replication results directly to the corresponding estimates of
CDLMMWY. Both interaction coefficients
have the same sign as in CDLMMWY. The CDLMMWY ability-by-AI coefficient (β1) is
3Appendix Table 2 reports the pooled estimates and separate estimates for Designs 2 and 3. 16, while the pooled AMRS analogue is -1. The CDLMMWY calibration-by-AI coefficient (β2) is 1. In a controlled image-classification experiment, Caplin, Deming,
S. Li, Martin, Marx, Weidmann, and Ye (2025b) (CDLMMWY hereafter) find that AI
assistance is more valuable for human users with lower baseline ability and better-calibrated
beliefs about their own performance (the “ABCs” of working with AI). 16, while the pooled AMRS analogue is -1. The CDLMMWY calibration-by-AI coefficient (β2) is 1. 16, while the pooled AMRS analogue is -1. The CDLMMWY calibration-by-AI coefficient (β2) is 1. In a controlled image-classification experiment, Caplin, Deming,
S. Li, Martin, Marx, Weidmann, and Ye (2025b) (CDLMMWY hereafter) find that AI
assistance is more valuable for human users with lower baseline ability and better-calibrated
beliefs about their own performance (the “ABCs” of working with AI).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Heterogeneous Value of AI
Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI gains at plus or minus one standard deviation of baseline ability and calibration. p. 7 93)
Calibration × AI 1. 95)
Observations 732 136
Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI
gains at plus or minus one standard deviation of baseline ability and calibration. 53
percentage points for, respectively, low-ability/high-calibration, low-ability/low-calibration,
high-ability/high-calibration, and high-ability/low-calibration users. As in CDLMMWY, the
highest fitted gains are for low-ability, high-calibration users and the lowest fitted gains are
for high-ability, low-calibration users. This suggests that the CDLMMWY productivity results are
not limited to the original age-classification task. This distinction is substantively important:
if these ability- and calibration-based patterns are general features of AI-assisted work, then
training, interface design, and delegation policies should target not only ability but also
users’ knowledge of their own accuracy. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Discussion on External Validity
The pooled AMRS repeated-case sample qualitatively replicates the key CDLMMWY results in a professional diagnostic setting. This suggests that the CDLMMWY productivity results are not limited to the original age-classification task. p. 7 4 Discussion
The pooled AMRS repeated-case sample qualitatively replicates the key CDLMMWY results
in a professional diagnostic setting, as the ability and calibration patterns point in the same
direction as the original results. This suggests that the CDLMMWY productivity results are
not limited to the original age-classification task. This suggests that the CDLMMWY productivity results are
not limited to the original age-classification task. This distinction is substantively important:
if these ability- and calibration-based patterns are general features of AI-assisted work, then
training, interface design, and delegation policies should target not only ability but also
users’ knowledge of their own accuracy. This distinction is substantively important:
if these ability- and calibration-based patterns are general features of AI-assisted work, then
training, interface design, and delegation policies should target not only ability but also
users’ knowledge of their own accuracy. This paper responds to recent calls for more external-validity evidence on human-AI
collaboration beyond controlled laboratory tasks.Where the paper says it
Where the paper says it
Where the paper says it
Robustness Checks
Table 3 reports checks on the interaction estimates using wild cluster bootstrap p-values, clustered by radiologist. p. 7 Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely. Pooling these designs yields 68 radiologists and 11,420 paired radiologist-patient-pathology
observations before aggregation to two outcome observations per radiologist. This paired
structure makes the AI comparison less confounded by case difficulty, pathology mix, or
radiologist composition, and the common cases make radiologist-level ability and calibration
more comparable across radiologists. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely. The accuracy distributions are roughly comparable because
CDLMMWY selected images to produce a similar performance distribution to that in AMRS. Table 1 summarizes the key interaction coefficients from the fully specified model, comparing the pooled AMRS replication results directly to the corresponding estimates of
CDLMMWY. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically
significant using wild-cluster inference, while the ability interaction is estimated less precisely.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
Agarwal, Nikhil, Alex Moehring, Pranav Rajpurkar, and Tobias Salz (2023).
Autor, David (2024). p. 1 Li, and Raymond 2023; Noy and W. Zhang 2023;
Autor 2024). Li, and Raymond 2023; Noy and W. Zhang 2023;
Autor 2024).Where the paper says it
Where the paper says it
Brynjolfsson, Erik, Danielle Li, and Lindsey R. p. 1 For
economic and policy purposes, however, the distribution of gains is equally important. For
instance, recent work has shown that AI can narrow performance gaps by boosting lowerperforming human users (Brynjolfsson, D. For
instance, recent work has shown that AI can narrow performance gaps by boosting lowerperforming human users (Brynjolfsson, D. Li, and Raymond 2023; Noy and W.Where the paper says it
Where the paper says it
Caplin, Andrew, David J. p. 0 Revisiting the ABCs of Working with AI:
A Replication with Radiologists∗
Daniel Martin†
June 10, 2026
Abstract
Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S. Caplin, Deming, S. Li,
Martin, Marx, Weidmann, and Ye (2025b) provide evidence that two characteristics,
ability and belief calibration, help to determine the returns to AI assistance. 02 percentage point difference in
2The data and code for CDLMMWY are distributed with their Management Science supplemental materials (Caplin, Deming, S. Li, Martin, Marx, Weidmann, and Ye 2025a).Where the paper says it
Where the paper says it
Where the paper says it
Caplin et al., 2025b.
Gonzalez, Cleotilde, Kate Donahue, Daniel G. p. 7 This paper responds to recent calls for more external-validity evidence on human-AI
collaboration beyond controlled laboratory tasks. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. This paper responds to recent calls for more external-validity evidence on human-AI
collaboration beyond controlled laboratory tasks. Recent work calls for in-context studies as
human-AI teaming moves from controlled settings toward applied, higher-stakes domains,
and medical-AI studies similarly emphasize that benefits depend on expertise, interaction
context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari,
4Appendix Table 3 reports wild cluster bootstrap p-values. Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira,
Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025;
Wekenborg, Gilbert, and Kather 2025).Where the paper says it
Where the paper says it
Where the paper says it
Kargarnovin et al. p. 9 Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira,
Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025;
Wekenborg, Gilbert, and Kather 2025). Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira,
Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025;
Wekenborg, Gilbert, and Kather 2025).Where the paper says it
Where the paper says it
Liu et al., 2025.
Moehring et al. p. 0 This note
shows that their results replicate to a setting where professional radiologists analyze
chest X-rays with access to state-of-the-art machine learning predictions. I leverage the
public Collab-CXR data repository described by Moehring, Kutwal, Huang, Banerjee,
Jacobi, Eber, Mendoza, Chung, Dayan, Gupta, Bui, Truong, Pareek, Langlotz, Lungren, Agarwal, Rajpurkar, and Salz (2025) and first analyzed for human-AI collaboration by Agarwal, Moehring, Rajpurkar, and Salz (2023).Where the paper says it
Noy and Zhang (2023). p. 1 Li, and Raymond 2023; Noy and W. Zhang 2023;
Autor 2024).Where the paper says it
Wekenborg et al. p. 9 Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira,
Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025;
Wekenborg, Gilbert, and Kather 2025).Where the paper says it
--- Page 4 ---
Figure 1: Comparison between the original CDLMMWY design and the replication. Panel A shows the original CDLMMWY structure, where baseline ability and calibration are measured in a no-AI skill block and AI assistance is randomized in the outcome block. p. 3 Figure 1: Comparison between the original CDLMMWY design and the replication. Panel
A shows the original CDLMMWY structure, where baseline ability and calibration are measured in a no-AI skill block and AI assistance is randomized in the outcome block. Panel
A shows the original CDLMMWY structure, where baseline ability and calibration are measured in a no-AI skill block and AI assistance is randomized in the outcome block. Panel B
shows the pooled AMRS Designs 2 and 3 replication structure, where the same radiologistcase-pathology readings are observed without and with AI, and there is a patient-level split
into skill and outcome blocks.Where the paper says it
Where the paper says it
--- Page 5 ---
no-AI prior confidence across blocks. The resulting repeated-case data set is organized at the radiologist-patient-pathology level. p. 4 no-AI prior confidence across blocks. The resulting repeated-case data set is organized at the radiologist-patient-pathology
level. The resulting repeated-case data set is organized at the radiologist-patient-pathology
level. Let Bhi denote radiologist h’s probability report for case i and Si the diagnosticstandard truth. Let Bhi denote radiologist h’s probability report for case i and Si the diagnosticstandard truth. Reported beliefs are “correct” if they would have induced correct choices at
a decision threshold of 50%:
correcthi =
1 if |Bhi − Si
| < 0. 5 if Bhi = 0. 5
0 else,
Accuracy Ah is then average correctness. 5
0 else,
Accuracy Ah is then average correctness. For calibration, the confidence of beliefs over each
case is max{Bhi, 1−Bhi}. For calibration, the confidence of beliefs over each
case is max{Bhi, 1−Bhi}. Net confidence averages confidence minus correctness across cases,
and calibration Ch is the negative absolute value of this average net confidence. Net confidence averages confidence minus correctness across cases,
and calibration Ch is the negative absolute value of this average net confidence. Accuracy and calibration in the no-AI cases in the skill block provide the estimated
baseline traits. Accuracy and calibration in the no-AI cases in the skill block provide the estimated
baseline traits. Averaging over outcome-block cases then supplies no-AI performance Yh(0)
and AI-assisted performance Yh(1).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 7 ---
Figure 3: Net confidence distributions across studies. Net confidence is confidence minus correctness, so values closer to zero correspond to better calibration, and positive values indicate overconfidence and negative values indicate under-confidence. p. 6 Figure 3: Net confidence distributions across studies. Net confidence is confidence minus
correctness, so values closer to zero correspond to better calibration, and positive values
indicate overconfidence and negative values indicate under-confidence.Where the paper says it
--- Page 8 ---
Figure 4: Heterogeneous value of AI by baseline ability and calibration in the CDLMMWY experiment. Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI gains at plus or minus one standard deviation of baseline ability and calibration. The four pooled AMRS fitted treatment effects are 5. p. 7 95)
Observations 732 136
Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI
gains at plus or minus one standard deviation of baseline ability and calibration. As shown
in the figure, the four pooled AMRS fitted treatment effects are 5. 20, and -1. 53
percentage points for, respectively, low-ability/high-calibration, low-ability/low-calibration,
high-ability/high-calibration, and high-ability/low-calibration users.Where the paper says it
Where the paper says it
--- Page 9 ---
Figure 4: Heterogeneous value of AI by baseline ability and calibration in the CDLMMWY experiment.
--- Page 10 ---
Jalali, Schelble, Singh, and Woolley (2026).
Kargarnovin et al., 2026.
Liu et al., 2025
Wekenborg et al., 2025.
--- Page 11 ---
References
Agarwal, Nikhil, Alex Moehring, Pranav Rajpurkar, and Tobias Salz (2023).
Autor, David (2024).
Brynjolfsson et al. p. 1 For
economic and policy purposes, however, the distribution of gains is equally important. For
instance, recent work has shown that AI can narrow performance gaps by boosting lowerperforming human users (Brynjolfsson, D.Where the paper says it
Caplin et al., 2025a.
Caplin et al., 2025b.
--- Page 13 ---
Table 2 reports the pooled and design-specific AMRS estimates. Table 3 reports checks on the interaction estimates using wild cluster bootstrap p-values, clustered by radiologist.
--- Page 14 ---
Table 2: AMRS estimates by design.
Table 3: Robustness checks for interaction estimates. p. 4 The accuracy distributions are roughly comparable because
CDLMMWY selected images to produce a similar performance distribution to that in AMRS. Table 1 summarizes the key interaction coefficients from the fully specified model, comparing the pooled AMRS replication results directly to the corresponding estimates of
CDLMMWY.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Ability-based tailoring of AI assistance can be implemented by providing different AI support levels based on a user's estimated baseline ability, as
lower baseline ability and better-calibrated beliefs about their own performance
predictlarger incremental value from AI.
This means systems could dynamically offer more intensive algorithmic guidance to users perceived as having lower skill in a specific task. -
Calibration-aware interfaces can be designed to prompt users when their reported confidence deviates significantly from the true probability, addressing the need for
better-calibrated beliefs about their own performance.
This improvement would involve creating feedback mechanisms that encourage users to recognize when they shoulddefer to the algorithm
rather than relying solely on their judgment. -
Personalized AI assistance can be optimized by considering the interaction between ability and calibration, as the model shows that gains are highest for
low-ability, high-calibration users.
This suggests systems should prioritize interventions—such as highlighting AI predictions or suggesting peer consultation—for experts who are highly confident in their own judgment but lack foundational skill in the domain.
Abstract
I examine the relationship between the benefits of working with AI and two baseline traits - ability and belief calibration - among all 227 radiologists available in Collab-CXR, a dataset from an experiment where AI assistance was provided during X-ray diagnosis. When analyzed jointly, both lower unassisted accuracy and smaller miscalibration, measured by the absolute gap between average confidence and accuracy, are associated with greater improvements in accuracy with AI. In addition, the association between unassisted accuracy and improvement with AI becomes more negative and statistically significant when controlling for calibration.
Related papers
- Huge If True: Learning from the Size of the Surprise
- The Rebundling of Freelance Work in the AI Age
- Dynamic investment and coordination in teams: Theory and experimental evidence
- Does Frontier AI Strengthen Cyber Defense? Evidence from Project Glasswing
- Generative AI at Work
- What is Work For? Labor, Politics, and Meaning in the Age of AI