RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

summary

The gist

As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge The main score is the actionable-error count…

In short

RadMatch is an LLM-based metric designed to evaluate radiology reports by decomposing comparisons into finding-level matching, clinical equivalence, and significance-aware scoring. It produces an actionable error count that is interpretable for radiologists and auditable by tracing every penalty to a specific finding pair, moving evaluation beyond opaque scores.

Key concepts

Decomposition into Finding-Level Matching
This framework breaks report comparison into three stages: extracting atomic findings with structured slots, matching candidate findings to reference findings based on clinical equivalence, and scoring the resulting matches or omissions. This ensures every error in the final score can be traced back to a specific finding pair.
Significance-Aware Scoring
Scoring treats each comparison unit (match, omission, spurious finding) as an evaluation point where judgment assigns an error type (e.g., CORrect, MISsing). Within matches, status inversions or longitudinal comparisons are graded based on clinical consequence and pre-defined thresholds.
Actionable Error Count
The primary metric reported by RadMatch is the actionable error count per report. This score is easily understood by radiologists because it quantifies errors that require clinical action. It complements safety metrics like recall and precision by providing a structured, traceable record of discrepancies.

Terminology used across episodes

This episode discusses

The paper

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching · Read on arXiv

Radium, Paris, France · Department of Radiology, Fondation Ophtalmologique Adolphe de Rothschild, Paris, France · Service de Radiologie Interventionnelle, Département d’Anesthésie Chirurgie et Imagerie Interventionnelle (DACI), Gustave Roussy

As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. We introduce RadMatch, a multi-stage LLM-based framework that turns report evaluation into a structured, auditable finding-level record. It extracts findings from a reference and a candidate report, matches them by clinical equivalence, grades each match over ten clinical attributes, and gives every match and unmatched finding a verdict and a clinical-significance tier. To validate metrics on CT, we build and release MetricEval-BodyCT, the first public metric-evaluation benchmark of chest and abdomen CT reports annotated by radiologists error by error. On it and three chest X-ray benchmarks, RadMatch with an open 31B judge on consumer GPUs agrees with radiologists more closely than previous published metrics, and even reaches inter-expert agreement on ReXVal and on the significant errors of MetricEval-BodyCT. Radiologists auditing its decisions on 60 studies confirm 98.5% of its matches and 93.6% of the errors it counts. When used to evaluate state-of-the-art radiology report generation models, RadMatch shows that most CT models, unlike several chest X-ray models, actually leave more significant errors than an image-blind normal template, as their added unsupported findings outnumber the reference findings they correctly report. We provide RadMatch's code, its dashboard, the MetricEval-BodyCT dataset and all per-report scores.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching".

Paper summary: Tom: So we’ve talked about the setup of this paper, focusing on the core idea behind RadMatch, which is moving beyond just one opaque score for report evaluation >

Jane: To recap, the central thesis is that traditional metrics and even some early LLM metrics fail because they discard the finding-level structure of a report, which means you lose the context of what’s actually being described >

Lu: They introduce RadMatch to fix this by decomposing comparison into three stages: extracting atomic findings, matching those candidates to reference findings based on clinical equivalence, and then scoring each matched or unmatched finding based on its clinical consequence >

Meng: The key is that the matching stage groups reference and candidate findings that describe the same clinical entity together into match groups called M = <mk>, where each mk is a pair of a reference side Rk and a candidate side Ck >

Tom: And then within those match groups, they assign an error type to every unit, classifying it as correct, partial, or incorrect—and omissions are simply called missing or spurious findings >

Jane: They are not just counting discrepancies; they are characterizing the mistake itself by assigning a significance tier and an error type to every single unit of comparison >

Lu: The reported RadMatch score is specifically defined as the actionable error count per report, which is the sum over all errors e in Eerr of whether sigma(e) is in A >

Meng: That means the final number you see isn't just a total count; it’s only counting those errors that meet a specific clinical threshold called A >

Tom: They also focus on seven clinical attribute dimensions when they do this scoring, which include status, location, severity, morphology, certainty, longitudinal comparison, and measurement >

Jane: So the framework gives you per-report diagnostics that a single raw error count simply can't provide because it shows you triage precision and recall at critical versus actionable tiers >

Lu: This decomposition allows researchers to probe capabilities like measurement accuracy or longitudinal comparison independently when looking at those subset views >

Conclusion: Tom: So wrapping up the discussion on RadMatch, the title itself really tells you what it’s about: making radiology report evaluation more auditable through finding-level matching >

Jane: It seems like the authors are arguing that for AI systems used in medicine, we need metrics that are structured so they can be interpreted and audited by human experts >

Lu: The implication is that when a capable judge saturates correlation, the value of a metric shifts from just raw agreement to something focused on interpretability, actionability, and auditability >

Meng: They are designing this specifically so every penalty traces back to a specific candidate–reference finding pair and a named attribute dimension >

Tom: It complements expert oversight rather than replacing it because the goal isn't necessarily to replace the radiologist but to provide a detailed record for them to review >

Jane: It’s about turning report evaluation from one opaque score into a structured, finding-level record that helps us understand *why* an AI made a mistake and what that mistake actually means clinically >

More episodes

← Home