A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
cs.CL, cs.AI
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 13 pages, 1 table. Structured narrative review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records.
Terminology
Abstract
Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan. This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged. No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions. Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.
Sources
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room
- OLAPH: Improving Factuality in Biomedical Long-form Question Answering
- Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists
- Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation
- Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
- All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations
- Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
- A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
- Chain-of-Thought Reasoning In The Wild Is Not Always Faithful
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering