Revisiting the ABCs of Working with AI: Evidence from Radiologists
summary
The gist
Artificial intelligence (AI) systems are being considered for a wide array of professional settings.
In short
The study replicated findings showing that lower baseline ability and higher calibration predict greater incremental value from AI assistance for radiologists analyzing chest X-rays. This suggests that tailoring AI support based on a user's skill level and self-assessment of accuracy can maximize the benefits of human-AI collaboration.
Key concepts
- Ability (Ah)
- This measures a radiologist's baseline skill in reading chest X-rays without AI assistance. It is estimated from cases where the radiologist reads cases without machine learning predictions. Higher ability generally means a higher baseline skill level.
- Calibration (Ch)
- Calibration reflects how well a radiologist's reported confidence matches their actual accuracy across different cases. A high calibration score indicates that when a radiologist is confident, they are usually correct, suggesting better self-assessment of performance.
- Incremental Value
- This refers to the extra benefit or productivity gain achieved by using AI assistance compared to working without it. The research found that this gain is not uniform; it varies significantly depending on whether a radiologist has low ability and high calibration, for example.
- Interaction Effects
- These describe how ability and calibration jointly influence the impact of AI. The key finding is that the combination of being less able but highly calibrated results in the largest predicted gains from using AI assistance.
Terminology used across episodes
This episode discusses
The paper
Revisiting the ABCs of Working with AI: Evidence from Radiologists · Read on arXiv
Daniel Martin, *
University of California, Santa Barbara
I examine the relationship between the benefits of working with AI and two baseline traits - ability and belief calibration - among all 227 radiologists available in Collab-CXR, a dataset from an experiment where AI assistance was provided during X-ray diagnosis. When analyzed jointly, both lower unassisted accuracy and smaller miscalibration, measured by the absolute gap between average confidence and accuracy, are associated with greater improvements in accuracy with AI. In addition, the association between unassisted accuracy and improvement with AI becomes more negative and statistically significant when controlling for calibration.
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Revisiting the ABCs of Working with AI".
Warren: Here's the gist of the paper. Artificial intelligence (AI) systems are being considered for a wide array of professional settings.
Ruth: This note shows that their results replicate to a setting where professional radiologists analyze chest X-rays with access to state-of-the-art machine learning predictions.
Warren: Lower baseline ability and higher calibration predict larger incremental value from AI assistance.
Ruth: Replication of Core Findings The study replicates findings from Caplin et al.
Title and authors: Ruth: We’ve talked about the setup and the initial findings showing that ability and calibration matter more than just raw skill when it comes to getting value from AI assistance. Now let's look at what the paper actually summarizes in terms of its main message.
Warren: Essentially, the paper is confirming that AI assistance isn't a blanket benefit for everyone; it’s conditional on who you are and how you assess your own judgment.
Iris: The summary highlights that the distribution of gains from AI is not uniform across the user population; some users get much more value from it than others depending on their baseline ability and calibration.
Ruth: They point out that AI can actually narrow performance gaps by boosting lowerperforming human users, as we see in other work, but only if those humans have the right beliefs about what they are doing.
Warren: The core takeaway is that for AI to be truly beneficial for productivity, we need to focus on tailoring the assistance based on these two traits: ability and calibration.
Iris: They emphasize that calibration matters because a human operator needs to know precisely when they should trust their own judgment and when they should defer to the algorithm's prediction.
Ruth: So, it’s not just about getting the right answer; it’s about knowing *when* you are right enough to trust your own assessment versus when you need the AI's input.
Warren: That speaks directly to how we design these systems—they can't just dump all the algorithmic output on a user and expect productivity to go up across the board.
Iris: It suggests that future work should focus on building interfaces that actively prompt users when their reported confidence is drifting away from what the true probability of correctness is.
Ruth: That feeds right into the idea of calibration-aware interfaces, which we touched on earlier—making sure users recognize when they should step back and let the AI take over.
Warren: If we want these productivity gains to generalize, we have to move beyond just measuring accuracy and start measuring how well people understand their own accuracy.
The paper's summary: Ruth: The authors themselves suggest a few ways they can improve this work and what that means for future research. They are basically proposing where the next steps should go.
Warren: One improvement they bring up is the need to look at how different experimental designs affect the interaction estimates, which we already touched on as a way to check their robustness.
Iris: They also suggest focusing on generalizing these findings beyond just chest X-rays and age guessing tasks, aiming for broader applicability in professional diagnostic settings.
Ruth: That’s a big one; they want to see if these ability and calibration patterns hold up when we move from medical imaging to other complex AI-assisted workflows.
Warren: And another point is the need for better identification of the causal effect, specifically looking at whether it’s just ability or if it's the interaction between them that drives the outcome.
Iris: They are pushing for more rigorous ways to prove that the interaction term, like ability by AI, is truly what’s driving the heterogeneous value of AI assistance.
Ruth: It also suggests a deeper look at how training and delegation policies should be structured based on these findings—not just basic skill training.
Warren: That means policies could prioritize interventions for experts who are highly confident in their own judgment but have low foundational skill, which is where the highest predicted gains lie.
Iris: And they’re pointing toward developing methods to measure calibration more finely, because their current measurement might not be capturing all the necessary nuances of belief about performance.
Ruth: Overall, it seems they are pushing for a framework that integrates these factors into how we think about deploying AI in professional work, not just as a standalone technology.
The paper's improvements: Ruth: So to wrap up on "Revisiting the ABCs of Working with AI: Evidence from Radiologists," the main message is that AI assistance provides differential returns based on a user’s ability and their calibration.
Warren: It confirms that gains are not universal; they are highest for users who start with lower skill but have strong self-knowledge about their accuracy.
Iris: The paper shows the calibration interaction effect is statistically significant using wild cluster inference, meaning those results weren't just a fluke in the data structure.
Ruth: This means that when we think about deploying these systems, we need to design things to account for this heterogeneity in how people interact with the AI.
Warren: The implication is that training and interface design should target not only skill but also users’ knowledge of their own accuracy to maximize the benefit of AI assistance.
Iris: So it confirms that building systems that prompt users when they are overconfident or underconfident is a practical way to leverage these findings in real-world applications.
Conclusion: Ruth: So we’ve been looking at "Revisiting the ABCs of Working with AI: Evidence from Radiologists," and what we're seeing is that whether AI helps a radiologist depends heavily on their baseline ability and how well they know themselves.
Warren: Exactly. The core finding is that users who are less skilled but more accurate in their self-assessment get the most value from the AI assistance. It’s not just about raw skill anymore; it’s about calibration.
Iris: I think what's really interesting is how they set up their experiment, using repeated cases to make sure they aren't just catching some fluke in a single study setup. They show that this pattern holds up across different designs.
Ruth: That external validity is key here, right? It suggests that if we want AI to actually boost productivity in the real world, we can’t just look at controlled lab settings.
Warren: We have to look at how these ability and calibration patterns play out when everyone is using the system, because if a whole group of people starts behaving this way, the costs for running those systems change dramatically.
Iris: And I see a potential issue in their setup—they estimated some interactions, but they also noted that the ability interaction was estimated less precisely than the calibration interaction. That means we might not be as sure about that specific relationship as we are about how calibration works with AI.
Ruth: That’s a fair caution. It tells us to be careful when we make big claims about ability versus calibration interactions until we get more data like that, you know, better precision on the numbers.
Warren: And from a trading side, it means if we try to automate decisions based on this, the risk profile changes depending on whether the trader is an expert or just someone who happens to be guessing well.
Iris: It also makes me think about calibration-aware interfaces again. If users know they are overconfident, they should probably be prompted to check their work instead of blindly following an AI suggestion that might be wrong because of that overconfidence.
Ruth: That’s a solid point for future work—designing systems that encourage users to pause and check their own judgment, not just accept the output.
Warren: Yeah, so we move from just asking "what is the answer" to asking "how confident am I in my answer right now." That shifts the entire interaction.
Iris: And as for me, I’m still watching how these findings might apply outside of medical imaging. If this ability-calibration dynamic is general to complex decision making, it has wide implications.
Ruth: It really does. So we leave "Revisiting the ABCs of Working with AI: Evidence from Radiologists" thinking about how tailoring the assistance to the user's knowledge of their own accuracy is going to be a big part of deploying these tools.
Warren: Yeah, and I’m curious what that looks like for high-frequency trading strategies. If we can figure out how to measure calibration in real-time, that could give us a whole new layer for risk management.
Iris: And I think the next big thing is proving this on different types of professional tasks. Not just one medical field, but maybe finance or engineering where the stakes are different.
Ruth: We'll keep an eye out for those papers showing how these effects manifest outside the lab environment next week.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19