Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies

arXiv:2604.03376 · cs.AI, cs.CL · Submitted 2026-04-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies".

Jane: The gist The proposed VERT metric improves correlation with radiologists' judgments by up to 11.7% relative to GREEN,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The title itself, "Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies," tells you they are tackling a problem where the evaluation methods aren't general enough anymore. Jane, what do the authors actually want to achieve with this investigation?

Jane: They want to figure out which model and which prompting setups are the best choices when you need an AI judge for radiology reports that go beyond just chest X-rays <ref:2604.03376#pg1>.

Tom: So they're not just looking at one specific setup, but they’re doing a comprehensive study comparing open- and closed-source models of different sizes, including reasoning and nonreasoning variants <ref:2604.03376#pg3>.

Lu: They are testing zero-shot prompting using things like GPT-four point one mini and Gemini two point five flash, as well as formula-based prompts and rubric-based prompts to see what works best <ref:2604.03376#pg5>.

Meng: I’m curious about the cost here; running those different sized models and testing all those prompting strategies across multiple modalities sounds like it would be computationally expensive for a production system <ref:2604.03376#pg3>.

Lalam: The authors are also testing their own proposed metric, VERT, to see if it actually surpasses the existing methods they’re comparing it against <ref:2604.03376#pg2>.

Tom: Right, and that leads us into the summary of what they actually did in this paper about their results.

The paper's summary: Jane: Okay, so the summary points out that they conducted a thorough correlation analysis between expert and LLM-based ratings across two datasets called RadEval and RaTE-Eval <ref:2604.03376#pg3>.

Tom: And they compared the three existing metrics—RadFact, GREEN, and FineRadScore—against their new one, VERT <ref:2604.03376#pg1>.

Lu: The key finding mentioned is that VERT surpasses existing methods on RadEval and RaTE-Eval <ref:2604.03376#pg2>.

Jane: They also looked at how to improve the scoring process by investigating ensembling strategies, like averaging scores or using linear regression on inferences from five hundred samples <ref:2604.03376#pg6>.

Meng: Ensembling sounds like a way to get a more stable score when you’re dealing with inherently noisy LLM outputs, which is relevant for real-world deployment <ref:2604.03376#pg6>.

Lalam: They also looked at parameter-efficient fine-tuning using LoRA, specifically fine-tuning Qwen3 30B on the RaTE-Eval training set <ref:2604.03376#pg6>.

Tom: And they found that both ensembling and that LoRA fine-tuning showed improvements in correlation with expert judgments <ref:2604.03376#pg2>.

The paper's improvements: Tom: So what are the specific ways they suggest we can improve things? They point to these lightweight adaptation techniques as a way forward, even if prompt-based evaluation with generalist LLMs is still the starting point <ref:2604.03376#pg2>.

Jane: The authors suggest that you don't need massive retraining; you can get better results by using methods like ensembling and LoRA fine-tuning <ref:2604.03376#pg3>.

Lu: They also performed a systematic error injection study to see where the metrics align with human judgments, which helps identify areas of both high and low agreement <ref:2604.03376#pg11>.

Meng: That error detection part is crucial for production because it tells you exactly what kind of errors the AI judge is missing or misclassifying, like they did with error type c through f <ref:2604.03376#pg11>.

Lalam: For example, on the RaTE-Eval dataset, zero-shot and few-shot approaches got an F1 score of zero point nine zero to zero point nine two for finding errors of type a, but only zero point six nine to zero point seven two for type b, which is missing findings <ref:2604.03376#pg12>.

Tom: That gap between error types shows that even with good correlation scores, the metrics aren't perfectly balanced across all error categories <ref:2604.03376#pg12>.

Conclusion: Jane: So to wrap up, the main implication is that we can achieve reliable evaluation by combining prompt-based evaluation with these lightweight adaptation techniques like ensembling and LoRA fine-tuning <ref:2604.03376#pg3>.

Tom: They found that VERT improves correlation with radiologist judgments by up to eleven point seven percent relative to GREEN <ref:2604.03376#pg6>.

Lu: And the fine-tuning results for Qwen3 30B were quite strong, showing gains of up to twenty-five percent using only a thousand three hundred training samples <ref:2604.03376#pg2>.

Meng: I’m still focused on what this means for scaling; they also noted that the inference time was reduced up to thirty-seven point two times with that fine-tuning setup <ref:2604.03376#pg2>.

Lalam: The paper shows that while prompt-based evaluation should be prioritized, you can get better performance through these lightweight adaptation techniques <ref:2604.03376#pg3>.

Tom: We’re done with this one today. The paper "Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies" shows us that getting reliable evaluation doesn't always require building a massive system; sometimes it just needs smart, lightweight adaptation techniques <ref:2604.03376#pg2>.

Jane: It certainly gives us more concrete tools to try when we’re looking at reports from different types of scans and anatomy <ref:2604.03376#pg1>.

Cornell University · Microsoft Healthcare & Life Sciences

cs.AI, cs.CL

Submitted: 2026-04-03

Updated: 2026-10-14

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The proposed VERT metric improves correlation with radiologists' judgments by up to 11.7% relative to GREEN, demonstrating that reliable evaluation can be achieved with lightweight

Key concepts

LLM Judges: These are large language models used to automatically score or evaluate radiology reports based on expert standards. The research tests if these models can reliably judge reports from various medical images and body regions, moving beyond just chest X-rays.
VERT Metric: This is a newly proposed metric designed to better align LLM scores with human radiologist judgments. It was shown to improve correlation with expert ratings by up to 11.7% compared to the existing GREEN metric, suggesting a more accurate way to evaluate reports.
Lightweight Adaptation: This refers to small changes made to an LLM, such as ensembling scores or using parameter-efficient fine-tuning (LoRA). The research found that these simple adjustments significantly boost the performance of the LLM judges without requiring massive computational resources.

Terminology

Fine-Tuning — 26× in this paper · explained in 5 episodes
Accuracy — 24× in this paper · explained in 1 episode
Adaptation — 14× in this paper · explained in 1 episode
LLM-as-a-Judge — 11× in this paper · explained in 3 episodes
LLM judge — 10× in this paper · explained in 2 episodes
Lightweight Adaptation — 10× in this paper · explained in 2 episodes
F1 Score — 8× in this paper · explained in 3 episodes
LoRA Fine-Tuning — 7× in this paper · explained in 3 episodes
Benchmark — 5× in this paper · explained in 2 episodes
Parameter-Efficient Fine-Tuning — 4× in this paper · explained in 3 episodes
Robustness — 4× in this paper · explained in 5 episodes
Foundation Model — 3× in this paper · explained in 3 episodes
Foundation Models — 3× in this paper · explained in 7 episodes
Distillation — 2× in this paper · explained in 2 episodes
Trace — 2× in this paper · explained in 2 episodes
Adaptability — 1× in this paper · explained in 1 episode
Core — 1× in this paper · explained in 2 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
False Negatives — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Generative AI — 1× in this paper · explained in 2 episodes
Interpretability — 1× in this paper · explained in 15 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
LoRA Adapters — 1× in this paper · explained in 3 episodes
Papers — 1× in this paper · explained in 3 episodes
Reasoning Traces — 1× in this paper · explained in 3 episodes
Regularization — 1× in this paper · explained in 3 episodes
Weight Decay — 1× in this paper · explained in 2 episodes
Zero-shot performance — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed VERT metric improves correlation with radiologists' judgments by up to 11.7% relative to GREEN, demonstrating that reliable evaluation can be achieved with lightweight adaptation through LLM-based judges and fine-tuning techniques.

How it works

The study investigates whether existing LLM-as-a-judge metrics like RadFact, GREEN, and FineRadScore are robust when applied to reports from modalities and anatomies other than chest X-rays, which is a primary limitation in current literature <ref:2604.03376#pg2>. Researchers conduct a thorough correlation analysis between expert and LLM-based ratings across two datasets, RadEval and RaTE-Eval, spanning multiple modalities and anatomies <ref:2604.03376#pg3>. The core of the work is comparing three existing metrics alongside VERT, the proposed LLM-based metric <ref:2604.03376#pg3>.

Model and Prompt Configurations

The research compares open- and closed-source models of different sizes, including reasoning and nonreasoning variants <ref:2604.03376#pg3>. They evaluate several prompting strategies, such as zero-shot prompting using GPT-4.1-mini and Gemini 2.5 flash, formula-based prompts, and rubric-based prompts <ref:2604.03376#pg5>. The proposed VERT metric is tested across multiple large language models to assess its robustness across different model families <ref:2604.03376#pg6>.

Evaluation Strategies

The methodology explores several advanced evaluation techniques beyond simple prompting to improve correlation with expert judgments <ref:2604.03376#pg6>. These include:

Ensembling strategies, such as averaging of scores and linear regression fitted on LLM inferences from 500 samples <ref:2604.03376#pg6>.
Few-shot prompting, which tests the inclusion of few-shot examples and error descriptions <ref:2604.03376#pg6>.
Parameter-efficient fine-tuning using LoRA, where Qwen3 30B is fine-tuned on the RaTE-Eval training set <ref:2604.03376#pg6>.

Error Detection and Categorization Analysis

A systematic error detection and categorization study is performed to assess the alignment of these metrics against expert judgments, identifying areas of lower and higher agreement <ref:2604.03376#pg6>. This analysis reveals that while overall scores correlate with human ratings, LLM-based metrics can underestimate the number of errors in a report and fail to accurately identify errors of type c–f <ref:2604.03376#pg11>. Specifically, for RaTE-Eval, zero-shot and few-shot approaches achieve an F1 score of 0.90–0.92 for error type (a) False Finding, but only 0.69–0.72 for type (b) Missing Finding <ref:2604.03376#pg12>.

Key Findings and Contributions

The results show that VERT improves correlation with radiologist judgments by up to 11.7% relative to GREEN <ref:2604.03376#pg6>. Furthermore, fine-tuning Qwen3 30B yields gains of up to 25% using only 1,300 training samples <ref:2604.03376#pg2> and reduces inference time up to 37.2 times <ref:2604.03376#pg2>. The study concludes that while prompt-based evaluation with generalist LLMs should be prioritized, complementary improvements can be achieved through lightweight adaptation techniques like ensembling and LoRA fine-tuning <ref:2604.03376#pg3>.

Metric Comparison

The paper compares different score assignment formulations, including GREEN (score = matched findings/(matched findings + ∑ significantErrors)), F1 (F1 = 2TP/(2TP + S)), and a formula-based prompt (Gw = TP / (TP + 2S + 0.5I)) <ref:2604.03376#pg14>. The choice of metric corresponds to selecting relative weights between correct findings and error types, with F1 emphasizing balanced correctness (precision–recall tradeoff) and the Formula emphasizing clinical importance of errors <ref:2604.03376#pg15>.

Future Directions

The findings highlight the effectiveness of LLM-based judges and demonstrate that reliable evaluation can be achieved with lightweight adaptation <ref:2604.03376#pg2>. Future work could use the data from this study to develop approaches to better detect and categorize errors that occur in model-generated radiology reports, as well as explore methods for evaluating radiology reports without access to a reference <ref:2604.03376#pg11>.

Acknowledgments

The authors thank Paul Vozila for his valuable feedback and Gabe Webster for brainstorming the name of the VERT metric <ref:2604.03376#pg2>.

References

The paper cites numerous prior works across various domains, including studies on BLEU, BERTScore, CheXbert, RadGraph, and various LLM-as-a-judge metrics like RadFact and GREEN <ref:2604.03376#pg3>. It also references work on fine-tuning models with LoRA and distillation techniques <ref:2604.03376#pg12>. The paper cites specific datasets used, such as RadEval and RaTE-Eval <ref:2604.03376#pg4>. It also references the technical details of models like Qwen3 8b and GPT-4.1-mini <ref:2604.03376#pg6>. The paper cites work on error injection techniques for generating plausible errors <ref:2604.03376#pg22>. It also references the technical specifications of the vLLM engine <ref:2604.03376#pg12>. It also references work related to medical imaging and radiology reports in general, such as those by Nori et al. <ref:2604.03376#pg2>. It also references the technical details of the LoRA method itself <ref:2604.03376#pg12>. Finally, it cites work on evaluation metrics like BLEU and ROUGE as traditional baselines <ref:2604.03376#pg3>. It also references the work of Zhao et al. on Ratescore <ref:2604.03376#pg12>. The paper cites work by Zambrano Chaves et al. on multimodal models <ref:2604.03376#pg12>. It also references work by Huang et al. on FineRadScore <ref:2604.03376#pg3>. It also references the technical details of GPT-5 and Gemini 1 Flash models <ref:2604.03376#pg6>. The paper cites work by Cho et al. on fine-tuning models for radiology evaluation <ref:2604.03376#pg2>. It also references work by Jain et al. on small model fine-tuning <ref:2604.03376#pg2>. It also references work by Ostmeier et al. on the GREEN metric <ref:2604.03376#pg2>. It also references work by Jannur et al. on error detection and categorization studies <ref:2604.03376#pg6>. The paper cites work by Dettmers et al. on QLoRA <ref:2604.03376#pg12>. It also references work by Hu et al. on LoRA <ref:2604.03376#pg12>. It also references work by Liu et al. on LLM-as-a-judge surveys <ref:2604.03376#pg3>. It also references work by Szymanski et al. on limitations of the llm-as-a-judge approach <ref:2604.03376#pg12>. The paper cites work by Seo et al. on the era of foundation models in medical imaging <ref:2604.03376#pg2>. It also references work by Zhao et al. on a new evaluation metric <ref:2604.03376#pg12>. It also references work by Yu et al. on the ReXVal dataset <ref:2604.03376#pg2>. It also references work by Zhao et al. on RaTE-Eval <ref:2604.03376#pg12>.

Improvements for AI systems

Reliable LLM Judges for Radiology Report Evaluation: The VERT metric surpasses existing methods by achieving a correlation with radiologist judgments up to 11.7% relative to GREEN, demonstrating that reliable evaluation can be achieved with lightweight adaptation.
Robust Model Selection and Prompting Strategy: The study provides a comprehensive study of LLMs as evaluators, comparing closed- and open-source models and analyzing the impact of prompting strategies, which helps determine which closed- or opensource models are best suited to act as LLM judges.
Efficient Fine-Tuning for Domain Adaptation: The research demonstrates that ensembling and LoRA fine-tuning further improve evaluation performance, showing that lightweight adaptation techniques can lead to gains, such as a fine-tuned Qwen3 30B yielding gains of up to 25% using only 1,300 training samples.
Error Detection and Categorization for Granular Analysis: The systematic error injection study allows researchers to identify areas of both high and low agreement between LLM-based metrics and expert judgments, helping to understand the strengths and failure modes of LLM-based metrics.

Abstract

Current work on radiology report evaluation has focused primarily on LLM-based metrics for chest X-rays (CXR), leaving open whether such approaches remain robust across other modalities and anatomies, and which model and prompt configurations are best suited to serve as LLM judges. We present a comprehensive evaluation of LLM judges for radiology report evaluation across diverse model families, prompting strategies, and adaptation methods, leveraging two expert-annotated datasets that span multiple imaging modalities and anatomical regions. Specifically, we compare four existing metrics (RadFact, GREEN, FineRadScore, and CRIMSON) with our proposed metric, VERT, across open- and closed-source models of varying sizes and reasoning capabilities. Among existing metrics, GREEN proves the most robust, outperforming more recent CXR-specific approaches. VERT further improves Kendall's correlation with radiologist judgment by up to 26% over GREEN, while ensembling and lightweight fine-tuning of Qwen3-30B yield gains of up to 9.9% and 25%, respectively, of which the latter also has a latency reduction of 37.2x. A systematic error analysis further shows that, while these metrics correlate with expert ratings overall, they tend to miscount errors such as omissions. Overall, our results show that carefully selected general-purpose LLM judges provide strong radiology report evaluation across modalities, with lightweight adaptation offering complementary gains when annotated data is available.