Revisiting the ABCs of Working with AI: Evidence from Radiologists

arXiv:2606.12585 · econ.GN, cs.HC, q-fin.EC · Submitted 2026-06-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Revisiting the ABCs of Working with AI".

Warren: Here's the gist of the paper. Artificial intelligence (AI) systems are being considered for a wide array of professional settings.

Ruth: This note shows that their results replicate to a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions.

Warren: Lower baseline ability and higher calibration predict larger incremental value from AI assistance.

Ruth: Replication of Core Findings The study replicates findings from Caplin et al.

Title and authors: Ruth: We’ve talked about the setup and the initial findings showing that ability and calibration matter more than just raw skill when it comes to getting value from AI assistance. Now let's look at what the paper actually summarizes in terms of its main message.

Warren: Essentially, the paper is confirming that AI assistance isn't a blanket benefit for everyone; it’s conditional on who you are and how you assess your own judgment.

Iris: The summary highlights that the distribution of gains from AI is not uniform across the user population; some users get much more value from it than others depending on their baseline ability and calibration.

Ruth: They point out that AI can actually narrow performance gaps by boosting lowerperforming human users, as we see in other work, but only if those humans have the right beliefs about what they are doing.

Warren: The core takeaway is that for AI to be truly beneficial for productivity, we need to focus on tailoring the assistance based on these two traits: ability and calibration.

Iris: They emphasize that calibration matters because a human operator needs to know precisely when they should trust their own judgment and when they should defer to the algorithm's prediction.

Ruth: So, it’s not just about getting the right answer; it’s about knowing *when* you are right enough to trust your own assessment versus when you need the AI's input.

Warren: That speaks directly to how we design these systems—they can't just dump all the algorithmic output on a user and expect productivity to go up across the board.

Iris: It suggests that future work should focus on building interfaces that actively prompt users when their reported confidence is drifting away from what the true probability of correctness is.

Ruth: That feeds right into the idea of calibration-aware interfaces, which we touched on earlier—making sure users recognize when they should step back and let the AI take over.

Warren: If we want these productivity gains to generalize, we have to move beyond just measuring accuracy and start measuring how well people understand their own accuracy.

The paper's summary: Ruth: The authors themselves suggest a few ways they can improve this work and what that means for future research. They are basically proposing where the next steps should go.

Warren: One improvement they bring up is the need to look at how different experimental designs affect the interaction estimates, which we already touched on as a way to check their robustness.

Iris: They also suggest focusing on generalizing these findings beyond just chest X-rays and age guessing tasks, aiming for broader applicability in professional diagnostic settings.

Ruth: That’s a big one; they want to see if these ability and calibration patterns hold up when we move from medical imaging to other complex AI-assisted workflows.

Warren: And another point is the need for better identification of the causal effect, specifically looking at whether it’s just ability or if it's the interaction between them that drives the outcome.

Iris: They are pushing for more rigorous ways to prove that the interaction term, like ability by AI, is truly what’s driving the heterogeneous value of AI assistance.

Ruth: It also suggests a deeper look at how training and delegation policies should be structured based on these findings—not just basic skill training.

Warren: That means policies could prioritize interventions for experts who are highly confident in their own judgment but have low foundational skill, which is where the highest predicted gains lie.

Iris: And they’re pointing toward developing methods to measure calibration more finely, because their current measurement might not be capturing all the necessary nuances of belief about performance.

Ruth: Overall, it seems they are pushing for a framework that integrates these factors into how we think about deploying AI in professional work, not just as a standalone technology.

The paper's improvements: Ruth: So to wrap up on "Revisiting the ABCs of Working with AI: Evidence from Radiologists," the main message is that AI assistance provides differential returns based on a user’s ability and their calibration.

Warren: It confirms that gains are not universal; they are highest for users who start with lower skill but have strong self-knowledge about their accuracy.

Iris: The paper shows the calibration interaction effect is statistically significant using wild cluster inference, meaning those results weren't just a fluke in the data structure.

Ruth: This means that when we think about deploying these systems, we need to design things to account for this heterogeneity in how people interact with the AI.

Warren: The implication is that training and interface design should target not only skill but also users’ knowledge of their own accuracy to maximize the benefit of AI assistance.

Iris: So it confirms that building systems that prompt users when they are overconfident or underconfident is a practical way to leverage these findings in real-world applications.

Conclusion: Ruth: So we’ve been looking at "Revisiting the ABCs of Working with AI: Evidence from Radiologists," and what we're seeing is that whether AI helps a radiologist depends heavily on their baseline ability and how well they know themselves.

Warren: Exactly. The core finding is that users who are less skilled but more accurate in their self-assessment get the most value from the AI assistance. It’s not just about raw skill anymore; it’s about calibration.

Iris: I think what's really interesting is how they set up their experiment, using repeated cases to make sure they aren't just catching some fluke in a single study setup. They show that this pattern holds up across different designs.

Ruth: That external validity is key here, right? It suggests that if we want AI to actually boost productivity in the real world, we can’t just look at controlled lab settings.

Warren: We have to look at how these ability and calibration patterns play out when everyone is using the system, because if a whole group of people starts behaving this way, the costs for running those systems change dramatically.

Iris: And I see a potential issue in their setup—they estimated some interactions, but they also noted that the ability interaction was estimated less precisely than the calibration interaction. That means we might not be as sure about that specific relationship as we are about how calibration works with AI.

Ruth: That’s a fair caution. It tells us to be careful when we make big claims about ability versus calibration interactions until we get more data like that, you know, better precision on the numbers.

Warren: And from a trading side, it means if we try to automate decisions based on this, the risk profile changes depending on whether the trader is an expert or just someone who happens to be guessing well.

Iris: It also makes me think about calibration-aware interfaces again. If users know they are overconfident, they should probably be prompted to check their work instead of blindly following an AI suggestion that might be wrong because of that overconfidence.

Ruth: That’s a solid point for future work—designing systems that encourage users to pause and check their own judgment, not just accept the output.

Warren: Yeah, so we move from just asking "what is the answer" to asking "how confident am I in my answer right now." That shifts the entire interaction.

Iris: And as for me, I’m still watching how these findings might apply outside of medical imaging. If this ability-calibration dynamic is general to complex decision making, it has wide implications.

Ruth: It really does. So we leave "Revisiting the ABCs of Working with AI: Evidence from Radiologists" thinking about how tailoring the assistance to the user's knowledge of their own accuracy is going to be a big part of deploying these tools.

Warren: Yeah, and I’m curious what that looks like for high-frequency trading strategies. If we can figure out how to measure calibration in real-time, that could give us a whole new layer for risk management.

Iris: And I think the next big thing is proving this on different types of professional tasks. Not just one medical field, but maybe finance or engineering where the stakes are different.

Ruth: We'll keep an eye out for those papers showing how these effects manifest outside the lab environment next week.

Daniel Martin, *

University of California, Santa Barbara

econ.GN, cs.HC, q-fin.EC

Submitted: 2026-06-10

Updated: 2026-11-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: Artificial intelligence (AI) systems are being considered for a wide array of professional settings.

Key concepts

Ability (Ah)
This measures a radiologist's baseline skill in reading chest X-rays without AI assistance. It is estimated from cases where the radiologist reads cases without machine learning predictions. Higher ability generally means a higher baseline skill level.
Calibration (Ch)
Calibration reflects how well a radiologist's reported confidence matches their actual accuracy across different cases. A high calibration score indicates that when a radiologist is confident, they are usually correct, suggesting better self-assessment of performance.
Incremental Value
This refers to the extra benefit or productivity gain achieved by using AI assistance compared to working without it. The research found that this gain is not uniform; it varies significantly depending on whether a radiologist has low ability and high calibration, for example.
Interaction Effects
These describe how ability and calibration jointly influence the impact of AI. The key finding is that the combination of being less able but highly calibrated results in the largest predicted gains from using AI assistance.

Terminology

Summary

Artificial intelligence (AI) systems are being considered for a wide array of professional settings. p. 1

Where the paper says it

1 Introduction Artificial intelligence (AI) systems are being considered for a wide array of professional settings. A natural first question is whether AI raises decision quality on average.

This note shows that their results replicate to a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions. p. 0
Where the paper says it

Li, Martin, Marx, Weidmann, and Ye (2025b) provide evidence that two characteristics, ability and belief calibration, help to determine the returns to AI assistance. This note shows that their results replicate to a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions.

Lower baseline ability and higher calibration predict larger incremental value from AI assistance. p. 0
Where the paper says it

Li, Martin, Marx, Weidmann, and Ye (2025b), I use the radiologist assessments from the repeated-case designs, which include 68 radiologists and 11,420 paired radiologist-patient-pathology observations. The results of this replication support the external validity of their core findings: lower baseline ability and higher calibration predict larger incremental value from AI.

Replication of Core Findings

The study replicates findings from Caplin et al. p. 0

Where the paper says it

Revisiting the ABCs of Working with AI: A Replication with Radiologists∗ Daniel Martin† June 10, 2026 Abstract Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S.

p. 0
Where the paper says it

Revisiting the ABCs of Working with AI: A Replication with Radiologists∗ Daniel Martin† June 10, 2026 Abstract Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S.

p. 0
Where the paper says it

Revisiting the ABCs of Working with AI: A Replication with Radiologists∗ Daniel Martin† June 10, 2026 Abstract Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S.

(2025b) in a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions. p. 0
Where the paper says it

Li, Martin, Marx, Weidmann, and Ye (2025b) provide evidence that two characteristics, ability and belief calibration, help to determine the returns to AI assistance. This note shows that their results replicate to a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions.

The replication leverages the repeated-case designs of the AMRS experiment, which include 68 radiologists and 11,420 paired radiologist-patient-pathology observations. p. 0
Where the paper says it

Li, Martin, Marx, Weidmann, and Ye (2025b), I use the radiologist assessments from the repeated-case designs, which include 68 radiologists and 11,420 paired radiologist-patient-pathology observations. The results of this replication support the external validity of their core findings: lower baseline ability and higher calibration predict larger incremental value from AI.

This paired structure makes the AI comparison less confounded by case difficulty, pathology mix, or radiologist composition. p. 2
Where the paper says it

Pooling these designs yields 68 radiologists and 11,420 paired radiologist-patient-pathology observations before aggregation to two outcome observations per radiologist. This paired structure makes the AI comparison less confounded by case difficulty, pathology mix, or radiologist composition, and the common cases make radiologist-level ability and calibration more comparable across radiologists.

The results support the external validity of their core findings: lower baseline ability and higher calibration predict larger incremental value from AI. p. 0
Where the paper says it

Li, Martin, Marx, Weidmann, and Ye (2025b), I use the radiologist assessments from the repeated-case designs, which include 68 radiologists and 11,420 paired radiologist-patient-pathology observations. The results of this replication support the external validity of their core findings: lower baseline ability and higher calibration predict larger incremental value from AI.

Data Construction and Estimation

To faithfully reproduce the CDLMMWY analysis, a dataset was constructed that allows ability and calibration to be estimated on one block of cases and the impact of AI on productivity to be estimated on another block of cases. This was accomplished using repeated-case designs where the same radiologist reads the same patient-pathology cases in all four randomly ordered treatment arms. p. 2

Where the paper says it

2 To accomplish this with the radiologist data, I use the repeated-case designs of the AMRS experiment (Design 2 and Design 3). In Design 2, the same radiologist reads the same patient-pathology cases in all four randomly ordered treatment arms, and in Design 3, the same session-level radiologist-patient-pathology readings are repeated first without and then with AI assistance.

The data is organized at the radiologist-patient-pathology level, allowing for ability and calibration estimation on skill block cases without AI and comparison with and without AI in the outcome block. p. 2
Where the paper says it

This paired structure makes the AI comparison less confounded by case difficulty, pathology mix, or radiologist composition, and the common cases make radiologist-level ability and calibration more comparable across radiologists. For the pooled main sample and each design-specific appendix sample, I assign half of patients to a “skill block” and the other half to an “outcome block”.

Ability is estimated on skill block cases without AI and compared with or without AI in the outcome block. p. 2
Where the paper says it

For the pooled main sample and each design-specific appendix sample, I assign half of patients to a “skill block” and the other half to an “outcome block”. I then estimate ability and calibration on the skill block cases without AI and compare performance with and without AI in the outcome block.

Key Interaction Effects

The regression specification used is Yht = α0 + α1Ah + α2Ch + β0AIt + β1(Ah × AIt) + β2(Ch × AIt) + εht. The key interaction coefficients from the fully specified model compare the pooled AMRS replication results directly to the corresponding estimates of CDLMMWY. p. 4

Where the paper says it

The accuracy distributions are roughly comparable because CDLMMWY selected images to produce a similar performance distribution to that in AMRS. Table 1 summarizes the key interaction coefficients from the fully specified model, comparing the pooled AMRS replication results directly to the corresponding estimates of CDLMMWY.

The ability-by-AI coefficient (β1) is-1. p. 4
Where the paper says it

Both interaction coefficients have the same sign as in CDLMMWY. The CDLMMWY ability-by-AI coefficient (β1) is 3Appendix Table 2 reports the pooled estimates and separate estimates for Designs 2 and 3.

45 for the pooled AMRS analogue, while it is-2. p. 7
Where the paper says it

16, while the pooled AMRS analogue is -1. The CDLMMWY calibration-by-AI coefficient (β2) is 1.

16 for CDLMMWY. p. 1
Where the paper says it

In a controlled image-classification experiment, Caplin, Deming, S. Li, Martin, Marx, Weidmann, and Ye (2025b) (CDLMMWY hereafter) find that AI assistance is more valuable for human users with lower baseline ability and better-calibrated beliefs about their own performance (the “ABCs” of working with AI).

The calibration-by-AI coefficient (β2) is 1. p. 7
Where the paper says it

16, while the pooled AMRS analogue is -1. The CDLMMWY calibration-by-AI coefficient (β2) is 1.

86 for the pooled AMRS analogue, while it is 1. p. 7
Where the paper says it

16, while the pooled AMRS analogue is -1. The CDLMMWY calibration-by-AI coefficient (β2) is 1.

35 for CDLMMWY. p. 1
Where the paper says it

In a controlled image-classification experiment, Caplin, Deming, S. Li, Martin, Marx, Weidmann, and Ye (2025b) (CDLMMWY hereafter) find that AI assistance is more valuable for human users with lower baseline ability and better-calibrated beliefs about their own performance (the “ABCs” of working with AI).

Heterogeneous Value of AI

Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI gains at plus or minus one standard deviation of baseline ability and calibration. p. 7

Where the paper says it

93) Calibration × AI 1. 95) Observations 732 136 Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI gains at plus or minus one standard deviation of baseline ability and calibration.

The highest fitted gains are for low-ability, high-calibration users and the lowest fitted gains are for high-ability, low-calibration users. p. 7
Where the paper says it

53 percentage points for, respectively, low-ability/high-calibration, low-ability/low-calibration, high-ability/high-calibration, and high-ability/low-calibration users. As in CDLMMWY, the highest fitted gains are for low-ability, high-calibration users and the lowest fitted gains are for high-ability, low-calibration users.

This suggests that training, interface design, and delegation policies should target not only ability but also users’ knowledge of their own accuracy. p. 7
Where the paper says it

This suggests that the CDLMMWY productivity results are not limited to the original age-classification task. This distinction is substantively important: if these ability- and calibration-based patterns are general features of AI-assisted work, then training, interface design, and delegation policies should target not only ability but also users’ knowledge of their own accuracy.

The calibration interaction is statistically significant using wild cluster inference. p. 7
Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

p. 7
Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

p. 7
Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

p. 7
Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

Discussion on External Validity

The pooled AMRS repeated-case sample qualitatively replicates the key CDLMMWY results in a professional diagnostic setting. This suggests that the CDLMMWY productivity results are not limited to the original age-classification task. p. 7

Where the paper says it

4 Discussion The pooled AMRS repeated-case sample qualitatively replicates the key CDLMMWY results in a professional diagnostic setting, as the ability and calibration patterns point in the same direction as the original results. This suggests that the CDLMMWY productivity results are not limited to the original age-classification task.

If these ability- and calibration-based patterns are general features of AI-assisted work, then training, interface design, and delegation policies should target not only ability but also users’ knowledge of their own accuracy. p. 7
Where the paper says it

This suggests that the CDLMMWY productivity results are not limited to the original age-classification task. This distinction is substantively important: if these ability- and calibration-based patterns are general features of AI-assisted work, then training, interface design, and delegation policies should target not only ability but also users’ knowledge of their own accuracy.

This paper responds to recent calls for more external-validity evidence on human-AI collaboration beyond controlled laboratory tasks. p. 7
Where the paper says it

This distinction is substantively important: if these ability- and calibration-based patterns are general features of AI-assisted work, then training, interface design, and delegation policies should target not only ability but also users’ knowledge of their own accuracy. This paper responds to recent calls for more external-validity evidence on human-AI collaboration beyond controlled laboratory tasks.

The calibration interaction is statistically significant using wild cluster inference.

Robustness Checks

Table 3 reports checks on the interaction estimates using wild cluster bootstrap p-values, clustered by radiologist. p. 7

Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

The Design 3 Calibration × AI estimate has a wild cluster bootstrap MDE of 3. p. 7
Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

76. This indicates robustness in the calibration interaction effect across different designs. p. 2
Where the paper says it

Pooling these designs yields 68 radiologists and 11,420 paired radiologist-patient-pathology observations before aggregation to two outcome observations per radiologist. This paired structure makes the AI comparison less confounded by case difficulty, pathology mix, or radiologist composition, and the common cases make radiologist-level ability and calibration more comparable across radiologists.

The Design 3 Ability × AI estimate has a wild cluster bootstrap MDE of 4. p. 7
Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

31. This confirms the stability of the ability interaction effect in the replication. p. 4
Where the paper says it

The accuracy distributions are roughly comparable because CDLMMWY selected images to produce a similar performance distribution to that in AMRS. Table 1 summarizes the key interaction coefficients from the fully specified model, comparing the pooled AMRS replication results directly to the corresponding estimates of CDLMMWY.

The calibration interaction is statistically significant using wild cluster inference. The ability interaction is estimated less precisely. p. 7
Where the paper says it

Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values. The calibration interaction is statistically significant using wild-cluster inference, while the ability interaction is estimated less precisely.

References

Agarwal, Nikhil, Alex Moehring, Pranav Rajpurkar, and Tobias Salz (2023).

Autor, David (2024). p. 1

Where the paper says it

Li, and Raymond 2023; Noy and W. Zhang 2023; Autor 2024).

p. 1
Where the paper says it

Li, and Raymond 2023; Noy and W. Zhang 2023; Autor 2024).

Brynjolfsson, Erik, Danielle Li, and Lindsey R. p. 1

Where the paper says it

For economic and policy purposes, however, the distribution of gains is equally important. For instance, recent work has shown that AI can narrow performance gaps by boosting lowerperforming human users (Brynjolfsson, D.

Raymond (2023). p. 1
Where the paper says it

For instance, recent work has shown that AI can narrow performance gaps by boosting lowerperforming human users (Brynjolfsson, D. Li, and Raymond 2023; Noy and W.

Caplin, Andrew, David J. p. 0

Where the paper says it

Revisiting the ABCs of Working with AI: A Replication with Radiologists∗ Daniel Martin† June 10, 2026 Abstract Artificial intelligence (AI) systems increasingly assist human experts, but the consequences of AI assistance on productivity can be heterogeneous. Caplin, Deming, S.

Deming, S. Li, Daniel Martin, Philip Marx, Ben Weidmann, and K. p. 0
Where the paper says it

Caplin, Deming, S. Li, Martin, Marx, Weidmann, and Ye (2025b) provide evidence that two characteristics, ability and belief calibration, help to determine the returns to AI assistance.

J. Ye (2025a). p. 2
Where the paper says it

02 percentage point difference in 2The data and code for CDLMMWY are distributed with their Management Science supplemental materials (Caplin, Deming, S. Li, Martin, Marx, Weidmann, and Ye 2025a).

Caplin et al., 2025b.

Gonzalez, Cleotilde, Kate Donahue, Daniel G. p. 7

Where the paper says it

This paper responds to recent calls for more external-validity evidence on human-AI collaboration beyond controlled laboratory tasks. Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values.

Goldstein, Hoda Heidari, Mohammad S. p. 7
Where the paper says it

This paper responds to recent calls for more external-validity evidence on human-AI collaboration beyond controlled laboratory tasks. Recent work calls for in-context studies as human-AI teaming moves from controlled settings toward applied, higher-stakes domains, and medical-AI studies similarly emphasize that benefits depend on expertise, interaction context, task performance, and user experience (Gonzalez, Donahue, Goldstein, Heidari, 4Appendix Table 3 reports wild cluster bootstrap p-values.

Jalali, Beau Schelble, Aarti Singh, and Anita Williams Woolley (2026). p. 9
Where the paper says it

Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira, Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025; Wekenborg, Gilbert, and Kather 2025).

Kargarnovin et al. p. 9

Where the paper says it

Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira, Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025; Wekenborg, Gilbert, and Kather 2025).

p. 9
Where the paper says it

Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira, Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025; Wekenborg, Gilbert, and Kather 2025).

, 2026.

Liu et al., 2025.

Moehring et al. p. 0

Where the paper says it

This note shows that their results replicate to a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions. I leverage the public Collab-CXR data repository described by Moehring, Kutwal, Huang, Banerjee, Jacobi, Eber, Mendoza, Chung, Dayan, Gupta, Bui, Truong, Pareek, Langlotz, Lungren, Agarwal, Rajpurkar, and Salz (2025) and first analyzed for human-AI collaboration by Agarwal, Moehring, Rajpurkar, and Salz (2023).

, 2025.

Noy and Zhang (2023). p. 1

Where the paper says it

Li, and Raymond 2023; Noy and W. Zhang 2023; Autor 2024).

Wekenborg et al. p. 9

Where the paper says it

Jalali, Schelble, Singh, and Woolley 2026; Kargarnovin, Hernandez, Reiners, Cruz-Neira, Bochenek, and Karwowski 2026; Liu, J. Zhang, Shuaiqi Chen, and Shanguang Chen 2025; Wekenborg, Gilbert, and Kather 2025).

, 2025

--- Page 4 ---

Figure 1: Comparison between the original CDLMMWY design and the replication. Panel A shows the original CDLMMWY structure, where baseline ability and calibration are measured in a no-AI skill block and AI assistance is randomized in the outcome block. p. 3

Where the paper says it

Figure 1: Comparison between the original CDLMMWY design and the replication. Panel A shows the original CDLMMWY structure, where baseline ability and calibration are measured in a no-AI skill block and AI assistance is randomized in the outcome block.

Panel B shows the pooled AMRS Designs 2 and 3 replication structure, where the same radiologistcase-pathology readings are observed without and with AI, and there is a patient-level split into skill and outcome blocks. p. 3
Where the paper says it

Panel A shows the original CDLMMWY structure, where baseline ability and calibration are measured in a no-AI skill block and AI assistance is randomized in the outcome block. Panel B shows the pooled AMRS Designs 2 and 3 replication structure, where the same radiologistcase-pathology readings are observed without and with AI, and there is a patient-level split into skill and outcome blocks.

--- Page 5 ---

no-AI prior confidence across blocks. The resulting repeated-case data set is organized at the radiologist-patient-pathology level. p. 4

Where the paper says it

no-AI prior confidence across blocks. The resulting repeated-case data set is organized at the radiologist-patient-pathology level.

Let Bhi denote radiologist h’s probability report for case i and Si the diagnosticstandard truth. p. 4
Where the paper says it

The resulting repeated-case data set is organized at the radiologist-patient-pathology level. Let Bhi denote radiologist h’s probability report for case i and Si the diagnosticstandard truth.

Reported beliefs are “correct” if they would have induced correct choices at a decision threshold of 50%. p. 4
Where the paper says it

Let Bhi denote radiologist h’s probability report for case i and Si the diagnosticstandard truth. Reported beliefs are “correct” if they would have induced correct choices at a decision threshold of 50%: correcthi =    1 if |Bhi − Si | < 0.

Accuracy Ah is then average correctness. p. 4
Where the paper says it

5 if Bhi = 0. 5 0 else, Accuracy Ah is then average correctness.

For calibration, the confidence of beliefs over each case is max[Bhi, 1−Bhi]. p. 4
Where the paper says it

5 0 else, Accuracy Ah is then average correctness. For calibration, the confidence of beliefs over each case is max{Bhi, 1−Bhi}.

Net confidence averages confidence minus correctness across cases, and calibration Ch is the negative absolute value of this average net confidence. p. 4
Where the paper says it

For calibration, the confidence of beliefs over each case is max{Bhi, 1−Bhi}. Net confidence averages confidence minus correctness across cases, and calibration Ch is the negative absolute value of this average net confidence.

Accuracy and calibration in the no-AI cases in the skill block provide the estimated baseline traits. p. 4
Where the paper says it

Net confidence averages confidence minus correctness across cases, and calibration Ch is the negative absolute value of this average net confidence. Accuracy and calibration in the no-AI cases in the skill block provide the estimated baseline traits.

Averaging over outcome-block cases then supplies no-AI performance Yh(0) and AI-assisted performance Yh(1). p. 4
Where the paper says it

Accuracy and calibration in the no-AI cases in the skill block provide the estimated baseline traits. Averaging over outcome-block cases then supplies no-AI performance Yh(0) and AI-assisted performance Yh(1).

--- Page 7 ---

Figure 3: Net confidence distributions across studies. Net confidence is confidence minus correctness, so values closer to zero correspond to better calibration, and positive values indicate overconfidence and negative values indicate under-confidence. p. 6

Where the paper says it

Figure 3: Net confidence distributions across studies. Net confidence is confidence minus correctness, so values closer to zero correspond to better calibration, and positive values indicate overconfidence and negative values indicate under-confidence.

--- Page 8 ---

Figure 4: Heterogeneous value of AI by baseline ability and calibration in the CDLMMWY experiment. Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI gains at plus or minus one standard deviation of baseline ability and calibration. The four pooled AMRS fitted treatment effects are 5. p. 7

Where the paper says it

95) Observations 732 136 Figure 5 plots fitted values from a regression of Yh(1) − Yh(0) on Ah and Ch to report AI gains at plus or minus one standard deviation of baseline ability and calibration. As shown in the figure, the four pooled AMRS fitted treatment effects are 5.

10, 1.38, 2.20, and-1.53 percentage points for, respectively, low-ability/high-calibration, low-ability/low-calibration, high-ability/high-calibration, and high-ability/low-calibration users. p. 7
Where the paper says it

20, and -1. 53 percentage points for, respectively, low-ability/high-calibration, low-ability/low-calibration, high-ability/high-calibration, and high-ability/low-calibration users.

The calibration interaction is statistically significant using wild cluster inference.

--- Page 9 ---

Figure 4: Heterogeneous value of AI by baseline ability and calibration in the CDLMMWY experiment.

--- Page 10 ---

Jalali, Schelble, Singh, and Woolley (2026).

Kargarnovin et al., 2026.

Liu et al., 2025

Wekenborg et al., 2025.

--- Page 11 ---

References

Agarwal, Nikhil, Alex Moehring, Pranav Rajpurkar, and Tobias Salz (2023).

Autor, David (2024).

Brynjolfsson et al. p. 1

Where the paper says it

For economic and policy purposes, however, the distribution of gains is equally important. For instance, recent work has shown that AI can narrow performance gaps by boosting lowerperforming human users (Brynjolfsson, D.

, 2023.

Caplin et al., 2025a.

Caplin et al., 2025b.

--- Page 13 ---

Table 2 reports the pooled and design-specific AMRS estimates. Table 3 reports checks on the interaction estimates using wild cluster bootstrap p-values, clustered by radiologist.

--- Page 14 ---

Table 2: AMRS estimates by design.

Table 3: Robustness checks for interaction estimates. p. 4

Where the paper says it

The accuracy distributions are roughly comparable because CDLMMWY selected images to produce a similar performance distribution to that in AMRS. Table 1 summarizes the key interaction coefficients from the fully specified model, comparing the pooled AMRS replication results directly to the corresponding estimates of CDLMMWY.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Ability-based tailoring of AI assistance can be implemented by providing different AI support levels based on a user's estimated baseline ability, as lower baseline ability and better-calibrated beliefs about their own performance predict larger incremental value from AI. This means systems could dynamically offer more intensive algorithmic guidance to users perceived as having lower skill in a specific task.

  2. Calibration-aware interfaces can be designed to prompt users when their reported confidence deviates significantly from the true probability, addressing the need for better-calibrated beliefs about their own performance. This improvement would involve creating feedback mechanisms that encourage users to recognize when they should defer to the algorithm rather than relying solely on their judgment.

  3. Personalized AI assistance can be optimized by considering the interaction between ability and calibration, as the model shows that gains are highest for low-ability, high-calibration users. This suggests systems should prioritize interventions—such as highlighting AI predictions or suggesting peer consultation—for experts who are highly confident in their own judgment but lack foundational skill in the domain.

Abstract

I examine the relationship between the benefits of working with AI and two baseline traits - ability and belief calibration - among all 227 radiologists available in Collab-CXR, a dataset from an experiment where AI assistance was provided during X-ray diagnosis. When analyzed jointly, both lower unassisted accuracy and smaller miscalibration, measured by the absolute gap between average confidence and accuracy, are associated with greater improvements in accuracy with AI. In addition, the association between unassisted accuracy and improvement with AI becomes more negative and statistically significant when controlling for calibration.

Related papers