ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
cs.LG, cs.AI, cs.CL
Submitted: 2026-03-15
Updated: 2026-10-28
Comments: Accepted at NeurIPS 2026 Evaluations & Datasets Track
Code: https://github.com/Jwoo5/ecg-reasoning-benchmark
License: http://creativecommons.org/licenses/by/4.0/
The gist: While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation…
Terminology
Abstract
While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce ECG-Reasoning-Benchmark, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step- by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a systematic shortfall in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit consistently low success rates in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, highlighting a limitation in their visual grounding capabilities and motivating robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.
Sources
- GPT-4 Technical Report
- Qwen3-VL Technical Report
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- The Llama 3 Herd of Models
- GPT-4o System Card
- Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding
- ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation
- GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
- OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data
- Teach Multimodal LLMs to Comprehend Electrocardiographic Images
- Q-Heart: ECG Question Answering via Knowledge-Informed Multimodal LLMs
- MedGemma Technical Report
- OpenAI GPT-5 System Card
- Gemini: A Family of Highly Capable Multimodal Models
- Gemma 3 Technical Report
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis