RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching".
Paper summary: Tom: So we’ve talked about the setup of this paper, focusing on the core idea behind RadMatch, which is moving beyond just one opaque score for report evaluation >
Jane: To recap, the central thesis is that traditional metrics and even some early LLM metrics fail because they discard the finding-level structure of a report, which means you lose the context of what’s actually being described >
Lu: They introduce RadMatch to fix this by decomposing comparison into three stages: extracting atomic findings, matching those candidates to reference findings based on clinical equivalence, and then scoring each matched or unmatched finding based on its clinical consequence >
Meng: The key is that the matching stage groups reference and candidate findings that describe the same clinical entity together into match groups called M = <mk>, where each mk is a pair of a reference side Rk and a candidate side Ck >
Tom: And then within those match groups, they assign an error type to every unit, classifying it as correct, partial, or incorrect—and omissions are simply called missing or spurious findings >
Jane: They are not just counting discrepancies; they are characterizing the mistake itself by assigning a significance tier and an error type to every single unit of comparison >
Lu: The reported RadMatch score is specifically defined as the actionable error count per report, which is the sum over all errors e in Eerr of whether sigma(e) is in A >
Meng: That means the final number you see isn't just a total count; it’s only counting those errors that meet a specific clinical threshold called A >
Tom: They also focus on seven clinical attribute dimensions when they do this scoring, which include status, location, severity, morphology, certainty, longitudinal comparison, and measurement >
Jane: So the framework gives you per-report diagnostics that a single raw error count simply can't provide because it shows you triage precision and recall at critical versus actionable tiers >
Lu: This decomposition allows researchers to probe capabilities like measurement accuracy or longitudinal comparison independently when looking at those subset views >
Conclusion: Tom: So wrapping up the discussion on RadMatch, the title itself really tells you what it’s about: making radiology report evaluation more auditable through finding-level matching >
Jane: It seems like the authors are arguing that for AI systems used in medicine, we need metrics that are structured so they can be interpreted and audited by human experts >
Lu: The implication is that when a capable judge saturates correlation, the value of a metric shifts from just raw agreement to something focused on interpretability, actionability, and auditability >
Meng: They are designing this specifically so every penalty traces back to a specific candidate–reference finding pair and a named attribute dimension >
Tom: It complements expert oversight rather than replacing it because the goal isn't necessarily to replace the radiologist but to provide a detailed record for them to review >
Jane: It’s about turning report evaluation from one opaque score into a structured, finding-level record that helps us understand *why* an AI made a mistake and what that mistake actually means clinically >
Radium, Paris, France · Department of Radiology, Fondation Ophtalmologique Adolphe de Rothschild, Paris, France · Service de Radiologie Interventionnelle, Département d’Anesthésie Chirurgie et Imagerie Interventionnelle (DACI), Gustave Roussy
cs.CV
Submitted: 2026-09-01
Updated: 2026-11-18
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge The main score is the actionable-error count…
Key concepts
- Decomposition into Finding-Level Matching
- This framework breaks report comparison into three stages: extracting atomic findings with structured slots, matching candidate findings to reference findings based on clinical equivalence, and scoring the resulting matches or omissions. This ensures every error in the final score can be traced back to a specific finding pair.
- Significance-Aware Scoring
- Scoring treats each comparison unit (match, omission, spurious finding) as an evaluation point where judgment assigns an error type (e.g., CORrect, MISsing). Within matches, status inversions or longitudinal comparisons are graded based on clinical consequence and pre-defined thresholds.
- Actionable Error Count
- The primary metric reported by RadMatch is the actionable error count per report. This score is easily understood by radiologists because it quantifies errors that require clinical action. It complements safety metrics like recall and precision by providing a structured, traceable record of discrepancies.
Terminology
Summary
As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge The main score is the actionable-error count, both interpretable and auditable
Introduction to RadMatch
RadMatch is an LLM-based metric that restores the structure of report comparison by decomposing it into extraction, matching, and significance-aware scoring This decomposition allows every penalty to be traceable to a specific finding pair, making the metric auditable The reported RadMatch score is the actionable-error count, which is easily interpretable by a radiologist It is complemented by safety-oriented recall and precision (triage and actionable) and subset views that probe capabilities such as measurement accuracy or longitudinal comparison
Decomposition into Finding-Level Matching
The framework decomposes report comparison into three stages: extraction of atomic findings, matching candidate to reference findings by clinical equivalence, and scoring each matched and unmatched finding by clinical consequence Findings are extracted as structured records with individually typed slots such as status, location, severity, morphology, certainty, longitudinal comparison, and measurements The matching stage groups reference and candidate findings that describe the same clinical entity into match groups M = ⟨mk⟩ where each mk = (Rk, Ck) pairs a reference side Rk ⊆ Fr with a candidate side Ck ⊆ Fc The matching is many-to-many and absorbs differences in atomization style across reports
Significance-Aware Scoring and Error Characterization
Scoring treats each match, omission, and spurious finding as an evaluation unit where judgment assigns each unit an error type and a significance tier Within a match, status is compared directly, any disagreement recorded as a status inversion; the longitudinal comparison is graded by clinical trajectory Measurements are compared against clinically motivated, per-category thresholds Each unit is typed with the message-understanding categories CORrect, PARtial, or INCorrect, while an omission is MISsing and a spurious finding SPUrious The reported RadMatch score is the actionable error count per report: RadMatch(r,c) = sum over e in Eerr 1[sigma(e) ∈ A]
Experimental Validation and Alignment
Across two expert-annotated benchmarks, RadMatch is the most clinically aligned metric On ReXVal its agreement with radiologists matches that among radiologists themselves, and on the harder RadEvalExpert it more than doubles the strongest prior metric With the Opus 4.8 model, it reaches τb = 0. p. 6 With the Opus 4. 8 model, it reaches
|τb| = 0. With the Opus 4. 8 model, it reaches
|τb| = 0.Where the paper says it
Where the paper says it
Diagnostic and Safety Views
The multi-stage decomposition yields per-report diagnostics that a single raw error count cannot provide Tab. 3 shows a detailed diagnostic and safety view for each benchmark, such as triage precision / recall at the triage (critical/urgent) and actionable (critical/urgent/notable) tiers For instance, on ReXVal, omissions (mis=153) and hallucinations (spu=116) outweigh status inversions and partial matches
Conclusion
RadMatch turns report evaluation from a single opaque score into a structured, finding-level record Once a capable judge saturates correlation, the value of a metric shifts from raw agreement to interpretability, actionability, and auditability RadMatch is designed precisely for this: every penalty traces to a specific candidate–reference finding pair and a named attribute dimension It complements expert oversight rather than replacing it
References
-
Acosta, J.N., Zhang, X., Dogra, S., Zhou, H.Y., Payabvash, S., Falcone, G.J., Oermann, E.K., Rajpurkar, P.: HeadCT-ONE: Enabling granular and controllable automated evaluation of head CT radiology report generation
-
American College of Radiology: ACR practice parameter for communication of diagnostic imaging findings
-
Anthropic: Claude opus 4. p. 5
8. Anthropic model cardWhere the paper says it
Its findings (148 studies) and impression (60 studies) sections are annotated separately, each error graded significant or insignificant. Unless stated otherwise, RadMatch uses the Claude Opus 4.
-
Baharoon, M., Heintz, T., Raissi, S., Alabbad, M., AlOmaish, H., Kim, S.E., Banerjee, O., Rajpurkar, P.: CRIMSON: A clinically-grounded LLM-based metric for generative radiology report evaluation
-
Bannur, S., Bouzid, K., Castro, D.C., Schwaighofer, A., Thieme, A., BondTaylor, S., et al.: MAIRA-2: Grounded radiology report generation
-
Blankemeier, L., et al.: Merlin: A vision language foundation model for 3D computed tomography
-
Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: PadChest: A large chest X-ray image dataset with multi-label annotated reports
-
Chen, Z., et al.: Large language model with region-guided referring and grounding for CT report generation
-
Cheng, S., Subramanian, D.: Rethinking radiology report generation: From narrative flow to topic-guided findings
-
Chinchor, N., Sundheim, B.: MUC-5 evaluation metrics
-
DeepSeek: DeepSeek-V4 technical report
-
Delbrouck, J.B., Chambon, P., Chen, Z., Varma, M., Johnston, A., Blankemeier, L., Van Veen, D., Bui, T., Truong, S., Langlotz, C.: RadGraph-XL: A large-scale expert-annotated dataset for entity and relation extraction from radiology reports
-
Google: Gemma 4 31B. p. 6
https://ai.google.dev/gemma/docs/core/model card 4Where the paper says it
8 Kendall's τb Inter-Expert RadMatch (Opus 4. 8) RadMatch (Gemma 4 31B) CRIMSON GREEN RadCliQ RadGraph-F1 ROUGE-L BERTScore BLEU (a) ReXVal 0.
-
Hamamci, I.E., et al.: Developing generalist foundation models from a multimodal dataset for 3D computed tomography
-
Huang, A., Banerjee, O., Wu, K., Reis, E.P., Rajpurkar, P.: FineRadScore: A radiology report line-by-line evaluation technique generating corrections with severity scores
-
Hyland, S.L., Bannur, S., Bouzid, K., Castro, D.C., Ranjit, M., Schwaighofer, A., et al.: MAIRA-1: A specialised large multimodal model for radiology report generation
-
Jain, S., Agrawal, A., Saporta, A., Truong, S.Q.H., Duong, D.N., Bui, T., Chambon, P., Zhang, Y., Lungren, M.P., Ng, A.Y., Langlotz, C.P.: RadGraph: Extracting clinical entities and relations from radiology reports
-
Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: MIMIC-CXR a de-identified publicly available database of chest radiographs with free-text reports
-
Kang, S., Lee, D.B., Jung, J., Kim, D., Kim, W.H., Joo, S.: Automated structured radiology report generation with rich clinical context
-
Larson, P.A., Berland, L.L., Griffith, B., Kahn, C.E., Liebscher, L.A.: Actionable findings and the role of IT support: Report of the ACR actionable reporting work group
-
Li, C.Y., Chang, K.J., Yang, C.F., Wu, H.Y., Chen, W., Bansal, H., Chen, L., Chen, Y.P., Chen, Y.C., Chen, S.P., Chen, S.J.: Towards a holistic framework for multimodal LLM in 3d brain CT radiology report generation
-
Li, Y., Liu, Y., Liu, L., Wang, L., Zhou, L.: RadReason: Radiology report evaluation metric with reasons and sub-scores
-
Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries
-
Liu, Y., Wang, Z., Li, Y., Liang, X., Liu, L., Wang, L., Zhou, L.: MRScore: Evaluating medical report with LLM-based reward system
-
Maye-Lasserre, T., Li, Y., Jian, B., Ghahremani, M., Wiestler, B., Wachinger, C.: Generating reports or repeating templates? p. 0
measuring and mitigating template collapse in 3D CT report generationWhere the paper says it
eu Abstract. As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge.
-
Moonshot AI: Kimi K2. p. 7
6. https://www.kimi.com/blog/kimi-k2-6Where the paper says it
61] gemma-4-31b 0. 58] kimi-k2.
-
OpenAI: Introducing GPT-4. p. 7
1 in the APIWhere the paper says it
For a fair comparison with prior LLM-based metrics, we run RadMatch with the same model each one uses. On ReXVal, RadMatch with GPT-4.
-
OpenAI: Introducing GPT-5. p. 7
p. 7Where the paper says it
For a fair comparison with prior LLM-based metrics, we run RadMatch with the same model each one uses. On ReXVal, RadMatch with GPT-4.
p. 7Where the paper says it
For a fair comparison with prior LLM-based metrics, we run RadMatch with the same model each one uses. On ReXVal, RadMatch with GPT-4.
2 in the APIWhere the paper says it
For a fair comparison with prior LLM-based metrics, we run RadMatch with the same model each one uses. On ReXVal, RadMatch with GPT-4.
-
OpenAI: Introducing GPT-5.4 in the API
-
OpenAI: Introducing GPT-5.4 mini and nano
Cost and Latency Analysis
In Tab. 6, we measure token usage and cost for each benchmark under the reported configuration (chest-X-ray few-shot, non-reasoning judge) With Opus 4. p. 5 8 model [3] with
the chest-X-ray few-shot examples, and Sec. 3 varies the judge. We present RadMatch, an LLM-based metric that restores this structure by
decomposing report comparison into (1) extraction of atomic findings, (2) matching of candidate to reference findings, and (3) significance-aware scoring over the
resulting records. Because the matching is persisted, every penalty is traceable
to a specific finding pair, so the metric is auditable.Where the paper says it
Where the paper says it
Evaluation Performance
RadMatch aligns with radiologists better than any other metric on each benchmark With the Opus 4.8 model, it reaches τb = 0.79 on ReXVal, matching the agreement among radiologists themselves This level of agreement does not require a frontier proprietary judge
Limitations
Our evaluation is limited to chest X-ray benchmarks, as no expert-annotated CT or MRI benchmark with radiologist error counts is currently available RadMatch issues several LLM calls per pair rather than one, resulting in about 10.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Auditable Evaluation Metric for LLM Reports: RadMatch decomposes report comparison into
extraction of atomic findings, matching of candidate to reference findings, and significance-aware scoring,
which makes every penaltytraceable to a specific finding pair.
This allows researchers to understandwhere errors concentrate
such as location or measurement accuracy. -
Actionable Error Quantification: The main score is the actionable error count, defined as
the number of clinically significant mistakes the candidate report makes relative to the reference,
allowing clinicians and builders to easily interpret what needs fixing rather than relying on anopaque score.
-
Structured Finding Extraction: The system uses a detailed prompt requiring findings to be extracted as atomic units with fields for
clinical status, clinical significance, comparison, [and] measurements,
ensuring that every finding is graded with a tier likecritical
orroutine
based on context and study indication. -
Clinical Context Integration: The framework supports an optional preliminary step to extract the
study indication
and inject it as clinical context into all three stages of evaluation, allowing significance tiers to be modulated by patient context, such as ruling out a pathology in a trauma workup. -
Diagnostic Safety Views: Beyond a single score, the system provides per-report diagnostics including
safety reports precision / recall at the triage (critical/urgent) and actionable (critical/urgent/notable) tiers
andper-subset views
that localize where errors occur, such as showing thatabnormal and longitudinal-comparison findings carry most actionable errors.
Abstract
As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. We introduce RadMatch, a multi-stage LLM-based framework that turns report evaluation into a structured, auditable finding-level record. It extracts findings from a reference and a candidate report, matches them by clinical equivalence, grades each match over ten clinical attributes, and gives every match and unmatched finding a verdict and a clinical-significance tier. To validate metrics on CT, we build and release MetricEval-BodyCT, the first public metric-evaluation benchmark of chest and abdomen CT reports annotated by radiologists error by error. On it and three chest X-ray benchmarks, RadMatch with an open 31B judge on consumer GPUs agrees with radiologists more closely than previous published metrics, and even reaches inter-expert agreement on ReXVal and on the significant errors of MetricEval-BodyCT. Radiologists auditing its decisions on 60 studies confirm 98.5% of its matches and 93.6% of the errors it counts. When used to evaluate state-of-the-art radiology report generation models, RadMatch shows that most CT models, unlike several chest X-ray models, actually leave more significant errors than an image-blind normal template, as their added unsupported findings outnumber the reference findings they correctly report. We provide RadMatch's code, its dashboard, the MetricEval-BodyCT dataset and all per-report scores.
Sources
- CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation
- MAIRA-2: Grounded Radiology Report Generation
- Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset
- Large Language Model with Region-guided Referring and Grounding for CT Report Generation
- Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
- FineRadScore: A Radiology Report Line-by-Line Evaluation Technique Generating Corrections with Severity Scores
- MAIRA-1: A specialised large multimodal model for radiology report generation
- Automated Structured Radiology Report Generation with Rich Clinical Context
- RadReason: Radiology Report Evaluation Metric with Reasons and Sub-Scores
- Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation
- MedGemma 1.5 Technical Report
- CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation
- CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation
- LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs