Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering

arXiv:2510.18691 · cs.CL · Submitted 2025-10-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering".

Jane: Here's the gist of the paper. This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA Our comprehensive approach considers a range of settings based on content inclusion of varying size and relevance, LLM models of different capabilities and a variety of datasets across task formulations We reveal insights on model size effects and their limitations, underlying memorization issues and the benefits of reasoning models, while demonstrating the value and challenges of leveraging the full long patient’s context Importantly, we examine the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension, showcasing best settings in single versus multi-document QA datasets Investigating Model Performance Across Context and Task Formulations The research investigates LLM performance on longitudinal patient-centric EHRs across three task formulations: 1) MCQA, 2) Extractive, and 3) Openended generative QA The study assesses these tasks across four settings with varying note relevance The inclusion scenarios examined are: 1) Exclude All, 2) Exclude Relevant, 3) Include All, and 4) Include Related For the 'Include All' setting, performance is reported across four token bins: Short context (0–8K), Medium context (8–16K), Large context (16–32K), and Extended context (32–128K) Evaluating Retrieval Augmented Generation Strategies The paper examines the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension RAG has been reported to enhance LLM performance compared to LC prompt compression for QA The authors carefully design a hybrid RAG pipeline examining its performance over Full-context (FC) across task formulations and context sizes, showing its superiority across tasks The study also investigates two chunk inclusion strategies: i) direct chunk inclusion (RAG), ii) hierarchical parent note inclusion (RAG HIR) Analyzing Model Size and Reasoning Capabilities Performance for the different inclusion settings across datasets and models is shown in Table 2 Larger models generally perform better across all tasks (Table 2) The performance gap is particularly pronounced in tasks requiring reasoning or information synthesis from multiple sources, namely EHRNoteQA and RadQA Smaller models perform better in settings excluding relevant information or completely removing the context The MC task shows strong evidence of memorization with a performance of 59.24% in the ‘Exclude All’ setting Quantitative Analysis of Context and Reasoning Trade-offs The analysis gathers challenging cases of disagreement between metrics (LLM Correctness, NLI Entailment, and Bio-F1) across EHRNoteQA and EHR-DS-QA RAG shows higher performance across the SEE, CNS and ANI categories, while FC leads in the TLR one The study finds that RAG excels at extracting precise, well-defined medical facts (dates, medications, lab values) that are typically documented in structured sections of medical records Conversely, FC is better at temporal reasoning (sequencing, duration calculation) Key Findings on Metric Disagreements and Use Cases Qualitative analysis shows that disagreements in metrics are mainly due to high Correctness/Completeness but low Faithfulness in predictions due to the addition of unsupported details RAG tends to perform better overall in datasets where specific information is required from complex or lengthy documents FC performs better for questions requiring comprehensive understanding of the entire document or when answers are located in structured sections (i.e.

Tom: lists) or temporal sequences RAG outperforms FC for questions that require retrieving specific details or synthesizing scattered information across multiple parts of the document The study concludes that highly-relevant context often outperforms feeding all notes, likely due to LLMs struggling with long context Limitations and Future Directions Limitations include focusing only on MIMIC-derived datasets, which represents English-only data in a single U.S.

Jane: medical center The de-identified nature of the data limits us from examining the potential cultural and other biases of LLMs over long contexts Future directions include scaling beyond single-note or single-source assumptions toward richer multi-note, multi-visit reasoning Advancing temporal and causal reasoning over longitudinal records, with explicit timeline grounding Strengthening evaluation via clinically faithful, judge-robust rubrics and cross-metric reliability analyses The work is intended solely for research purposes Ethical Considerations and Data Usage This work uses MIMIC-III, MIMIC-IV, and four derivative datasets (EHRNoteQA, EHR-DS-QA, RadQA, CliniQG4QA) accessed under the PhysioNet Credentialed Health Data License 1.5.0 All datasets contain de-identified patient records in accordance with HIPAA Safe Harbor standards We exclusively used open-weight LLMs to ensure no patient data were transmitted to third-party proprietary systems Results reflect model performance on question answering benchmarks and should not be interpreted as guarantees of clinical safety, equitable performance, or readiness for clinical deployment The paper is 2510.

Title and authors: Tom: The paper is titled "Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering," and it’s led by Feras AlMannaa, Talia Tseriotou, Jenny Chim, and Maria Liakata.

Jane: It’s important to know that the title clearly signals they are moving beyond simple multiple-choice questions into more complex medical question answering using long patient histories.

Lu: The authors are bringing together different capabilities and sizes of LLMs across various task formats—multiple-choice, extractive tasks, and open-ended generative question answering—to see how they handle these long patient notes.

Meng: From an engineering side, this is interesting because it’s not just about one model; it tests different model sizes and different ways to structure the input data to see what breaks or performs best at scale.

Lalam: It also specifically examines the effect of Retrieval Augmented Generation, or RAG, on comprehension in these long-context medical settings, showing where RAG shines compared to just compressing the prompt.

The paper's summary: Tom: Basically, the study investigates model performance across three task formulations: multiple-choice questions, extractive tasks where you pull out specific text from the notes, and open-ended generative question answering.

Jane: They test these across four different settings for how much context is included in the input—excluding all notes, excluding relevant ones, including all notes, or including only related ones.

Lu: For the "Include All" setting specifically, they look at context sizes ranging from short contexts up to extended contexts that reach twelve eight long context tokens.

Meng: They also examine performance across different LLMs with varying capabilities and different datasets to see how model size affects performance in these long-context medical tasks.

Lalam: The study reveals insights into the limitations of just memorizing information and highlights the benefits of using reasoning models when dealing with this kind of long patient context.

The paper's improvements: Tom: One key area they look at is how RAG improves medical long context comprehension, showing that it can outperform simple prompt compression in question answering tasks across various context lengths.

Jane: They also test two specific chunk inclusion strategies for RAG: direct chunk inclusion and a hierarchical parent note inclusion method, which they call RAG HIR.

Lu: The results show that retrieval augmented generation is superior across all task formulations and context sizes when compared to just using the full context, especially in single versus multi-document question answering datasets.

Meng: They found that RAG excels at extracting precise medical facts, like specific dates or medications, which are usually found in structured parts of a medical record.

Lalam: Conversely, they also showed that feeding the full context directly into the model is better for questions that require temporal reasoning, like sequencing events or calculating how long something lasted.

Conclusion: Tom: To wrap up, the study shows that highly relevant context often beats feeding everything in because LLMs struggle with very long contexts.

Jane: They found disagreements between different metrics depending on the task; RAG is better for specific details, but full context is better for comprehensive document understanding or structured lists.

Lu: The overall implication is that we need to be smart about how we feed patient data into these models, balancing retrieval and generation based on what the specific question demands.

Meng: From a deployment standpoint, this means we have to decide whether to use a more expensive RAG pipeline or just compress the context for certain types of clinical questions.

Lalam: The paper's title, "Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering," shows they are pushing the boundary of what’s possible in clinical AI right now.

Feras AlMannaa, Talia Tseriotou, Jenny Chim, Maria Liakata

Istanbul Aydın University · Queen Mary University of London · The Alan Turing Institute

cs.CL

Submitted: 2025-10-21

Updated: 2026-11-12

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA Our comprehensive approach considers a…

Key concepts

Long-Context Comprehension (LC)
This refers to an LLM's ability to accurately understand and answer questions based on very long pieces of text, such as an entire patient's medical history. The study investigates how performance changes as the context length increases, ranging from short notes up to 128K tokens.
Retrieval Augmented Generation (RAG)
RAG is a technique where the LLM retrieves relevant parts of a large document before generating an answer. The research compares RAG against feeding the entire context (Full-Context) to see which method yields better results for medical question answering tasks.
Model Size Effects
The size of the language model significantly impacts its performance, especially when tasks require complex reasoning or synthesizing information from many sources. Larger models generally perform better on challenging reasoning tasks like EHRNoteQA and RadQA compared to smaller models.
Context Inclusion Scenarios
Researchers tested four ways to include patient context: excluding all notes, excluding relevant notes, including all notes, and including related notes. This helped determine the optimal amount of information needed for an LLM to perform well on specific medical tasks.

Terminology

Summary

This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA Our comprehensive approach considers a range of settings based on content inclusion of varying size and relevance, LLM models of different capabilities and a variety of datasets across task formulations We reveal insights on model size effects and their limitations, underlying memorization issues and the benefits of reasoning models, while demonstrating the value and challenges of leveraging the full long patient’s context Importantly, we examine the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension, showcasing best settings in single versus multi-document QA datasets

Investigating Model Performance Across Context and Task Formulations

The research investigates LLM performance on longitudinal patient-centric EHRs across three task formulations: 1) MCQA, 2) Extractive, and 3) Openended generative QA The study assesses these tasks across four settings with varying note relevance The inclusion scenarios examined are: 1) Exclude All, 2) Exclude Relevant, 3) Include All, and 4) Include Related For the 'Include All' setting, performance is reported across four token bins: Short context (0–8K), Medium context (8–16K), Large context (16–32K), and Extended context (32–128K)

Evaluating Retrieval Augmented Generation Strategies

The paper examines the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension RAG has been reported to enhance LLM performance compared to LC prompt compression for QA The authors carefully design a hybrid RAG pipeline examining its performance over Full-context (FC) across task formulations and context sizes, showing its superiority across tasks The study also investigates two chunk inclusion strategies: i) direct chunk inclusion (RAG), ii) hierarchical parent note inclusion (RAG HIR)

Analyzing Model Size and Reasoning Capabilities

Performance for the different inclusion settings across datasets and models is shown in Table 2 Larger models generally perform better across all tasks (Table 2) The performance gap is particularly pronounced in tasks requiring reasoning or information synthesis from multiple sources, namely EHRNoteQA and RadQA Smaller models perform better in settings excluding relevant information or completely removing the context The MC task shows strong evidence of memorization with a performance of 59.24% in the ‘Exclude All’ setting

Quantitative Analysis of Context and Reasoning Trade-offs

The analysis gathers challenging cases of disagreement between metrics (LLM Correctness, NLI Entailment, and Bio-F1) across EHRNoteQA and EHR-DS-QA RAG shows higher performance across the SEE, CNS and ANI categories, while FC leads in the TLR one The study finds that RAG excels at extracting precise, well-defined medical facts (dates, medications, lab values) that are typically documented in structured sections of medical records Conversely, FC is better at temporal reasoning (sequencing, duration calculation)

Key Findings on Metric Disagreements and Use Cases

Qualitative analysis shows that disagreements in metrics are mainly due to high Correctness/Completeness but low Faithfulness in predictions due to the addition of unsupported details RAG tends to perform better overall in datasets where specific information is required from complex or lengthy documents FC performs better for questions requiring comprehensive understanding of the entire document or when answers are located in structured sections (i.e. lists) or temporal sequences RAG outperforms FC for questions that require retrieving specific details or synthesizing scattered information across multiple parts of the document The study concludes that highly-relevant context often outperforms feeding all notes, likely due to LLMs struggling with long context

Limitations and Future Directions

Limitations include focusing only on MIMIC-derived datasets, which represents English-only data in a single U.S. medical center The de-identified nature of the data limits us from examining the potential cultural and other biases of LLMs over long contexts Future directions include scaling beyond single-note or single-source assumptions toward richer multi-note, multi-visit reasoning Advancing temporal and causal reasoning over longitudinal records, with explicit timeline grounding Strengthening evaluation via clinically faithful, judge-robust rubrics and cross-metric reliability analyses The work is intended solely for research purposes

Ethical Considerations and Data Usage

This work uses MIMIC-III, MIMIC-IV, and four derivative datasets (EHRNoteQA, EHR-DS-QA, RadQA, CliniQG4QA) accessed under the PhysioNet Credentialed Health Data License 1.5.0 All datasets contain de-identified patient records in accordance with HIPAA Safe Harbor standards We exclusively used open-weight LLMs to ensure no patient data were transmitted to third-party proprietary systems Results reflect model performance on question answering benchmarks and should not be interpreted as guarantees of clinical safety, equitable performance, or readiness for clinical deployment The paper is 2510. p. 8

Where the paper says it

All datasets contain de-identified patient records in accordance with HIPAA Safe Harbor standards. We exclusively used open-weight LLMs to ensure no patient data were transmitted to third-party proprietary systems.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Context strategy adaptation: Improve systems by dynamically selecting between Full-Context (FC) prompting and Retrieval Augmented Generation (RAG) based on task type and context size to maximize performance, as the paper shows RAG demonstrates clear performance improvement for the single-note generative task of EHR-DS-QA while FC performs better for questions requiring comprehensive understanding of the entire document or when answers are located in structured sections (i.e. lists) or temporal sequences.

  2. Context inclusion optimization: Enhance systems to intelligently manage patient context by implementing four data inclusion scenarios, specifically utilizing Include Related as a default strategy to assess model’s ability to leverage the most relevant context information, which was shown to outperform Include All on most datasets except for MCQA.

  3. Model scaling and specialization: Improve performance by deploying larger reasoning models, such as Qwen2.5-32B-Instruct or QwQ:32B, for tasks demanding complex inference across multiple sources of information, as they generally outperform smaller 7B models in datasets like EHRNoteQA and RadQA.

  4. Hybrid retrieval pipeline development: Implement a hybrid RAG approach combining sparse and dense retrievers with late interaction reranking to improve retrieval quality, as the paper notes that hybrid approaches combining sparse and dense retrieval outperform single component counterparts in biomedical QA.

  5. Metric-aware evaluation: Transition from relying solely on automatic metrics to adopting a multi-dimensional evaluation methodology using LLM-as-a-judge rubrics (Correctness, Completeness, Faithfulness) to uncover issues like High Correctness but low Faithfulness in predictions due to the addition of unsupported details.

  6. Reasoning task specific tuning: Fine-tune models specifically for reasoning capabilities, as the paper notes that larger models generally help especially for reasoning datasets, though it also cautions that medically fine-tuned models might fall behind when generalist performance is not sufficient.

  7. Temporal and causal grounding: Enhance systems to explicitly handle temporal and causal reasoning over longitudinal records by using explicit timeline grounding, addressing the limitation where RAG falls behind FC in temporal reasoning tasks while FC excels at temporal sequencing (FC: 0.696 vs. RAG: 0.236).

Abstract

Despite the impressive reported performance of Large Language Models (LLMs) on medical Question-Answering (QA), evaluation still overrelies on multiple-choice and extractive settings primarily assessing textbook knowledge, with limited work on open-ended QA over patient records. We present a comprehensive study investigating LLM long-context (LC) comprehension on longitudinal, patient-centric Electronic Health Records (EHRs) across multiple datasets, models and metrics, covering extractive, multiple-choice, and open-ended generative QA under different context formulation and Retrieval Augmented Generation (RAG) settings. We show that medically fine-tuned models offer no measurable benefit over base counterparts, smaller models exhibit stronger memorization, and longitudinal supplementary notes aid generative QA. Reasoning models gain on complex multi-note QA under LLM Correctness but not stricter reference-aligned metrics, while hybrid RAG with reranking outperforms Full-Context (FC) prompting. A question-category analysis and multi-faceted metric evaluation further reveal where RAG and FC excel and struggle, with RAG falling behind in longitudinal reasoning, thus uncovering systematic failures pointing to the need for reasoning-appropriate metrics.

Sources

Related papers