Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering

summary

The gist

This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA Our comprehensive approach considers a…

In short

This study tested how large language models understand long patient medical records across different question types and context sizes. It found that Retrieval Augmented Generation (RAG) is superior for extracting specific facts, while Full-Context prompting is better for understanding the entire document or temporal sequences. Model size matters, with larger models excelling at complex reasoning tasks.

Key concepts

Long-Context Comprehension (LC)
This refers to an LLM's ability to accurately understand and answer questions based on very long pieces of text, such as an entire patient's medical history. The study investigates how performance changes as the context length increases, ranging from short notes up to 128K tokens.
Retrieval Augmented Generation (RAG)
RAG is a technique where the LLM retrieves relevant parts of a large document before generating an answer. The research compares RAG against feeding the entire context (Full-Context) to see which method yields better results for medical question answering tasks.
Model Size Effects
The size of the language model significantly impacts its performance, especially when tasks require complex reasoning or synthesizing information from many sources. Larger models generally perform better on challenging reasoning tasks like EHRNoteQA and RadQA compared to smaller models.
Context Inclusion Scenarios
Researchers tested four ways to include patient context: excluding all notes, excluding relevant notes, including all notes, and including related notes. This helped determine the optimal amount of information needed for an LLM to perform well on specific medical tasks.

Terminology used across episodes

This episode discusses

The paper

Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering · Read on arXiv

Feras AlMannaa, Talia Tseriotou, Jenny Chim, Maria Liakata

Istanbul Aydın University · Queen Mary University of London · The Alan Turing Institute

Despite the impressive reported performance of Large Language Models (LLMs) on medical Question-Answering (QA), evaluation still overrelies on multiple-choice and extractive settings primarily assessing textbook knowledge, with limited work on open-ended QA over patient records. We present a comprehensive study investigating LLM long-context (LC) comprehension on longitudinal, patient-centric Electronic Health Records (EHRs) across multiple datasets, models and metrics, covering extractive, multiple-choice, and open-ended generative QA under different context formulation and Retrieval Augmented Generation (RAG) settings. We show that medically fine-tuned models offer no measurable benefit over base counterparts, smaller models exhibit stronger memorization, and longitudinal supplementary notes aid generative QA. Reasoning models gain on complex multi-note QA under LLM Correctness but not stricter reference-aligned metrics, while hybrid RAG with reranking outperforms Full-Context (FC) prompting. A question-category analysis and multi-faceted metric evaluation further reveal where RAG and FC excel and struggle, with RAG falling behind in longitudinal reasoning, thus uncovering systematic failures pointing to the need for reasoning-appropriate metrics.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering".

Jane: Here's the gist of the paper. This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA Our comprehensive approach considers a range of settings based on content inclusion of varying size and relevance, LLM models of different capabilities and a variety of datasets across task formulations We reveal insights on model size effects and their limitations, underlying memorization issues and the benefits of reasoning models, while demonstrating the value and challenges of leveraging the full long patient’s context Importantly, we examine the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension, showcasing best settings in single versus multi-document QA datasets Investigating Model Performance Across Context and Task Formulations The research investigates LLM performance on longitudinal patient-centric EHRs across three task formulations: 1) MCQA, 2) Extractive, and 3) Openended generative QA The study assesses these tasks across four settings with varying note relevance The inclusion scenarios examined are: 1) Exclude All, 2) Exclude Relevant, 3) Include All, and 4) Include Related For the 'Include All' setting, performance is reported across four token bins: Short context (0–8K), Medium context (8–16K), Large context (16–32K), and Extended context (32–128K) Evaluating Retrieval Augmented Generation Strategies The paper examines the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension RAG has been reported to enhance LLM performance compared to LC prompt compression for QA The authors carefully design a hybrid RAG pipeline examining its performance over Full-context (FC) across task formulations and context sizes, showing its superiority across tasks The study also investigates two chunk inclusion strategies: i) direct chunk inclusion (RAG), ii) hierarchical parent note inclusion (RAG HIR) Analyzing Model Size and Reasoning Capabilities Performance for the different inclusion settings across datasets and models is shown in Table 2 Larger models generally perform better across all tasks (Table 2) The performance gap is particularly pronounced in tasks requiring reasoning or information synthesis from multiple sources, namely EHRNoteQA and RadQA Smaller models perform better in settings excluding relevant information or completely removing the context The MC task shows strong evidence of memorization with a performance of 59.24% in the ‘Exclude All’ setting Quantitative Analysis of Context and Reasoning Trade-offs The analysis gathers challenging cases of disagreement between metrics (LLM Correctness, NLI Entailment, and Bio-F1) across EHRNoteQA and EHR-DS-QA RAG shows higher performance across the SEE, CNS and ANI categories, while FC leads in the TLR one The study finds that RAG excels at extracting precise, well-defined medical facts (dates, medications, lab values) that are typically documented in structured sections of medical records Conversely, FC is better at temporal reasoning (sequencing, duration calculation) Key Findings on Metric Disagreements and Use Cases Qualitative analysis shows that disagreements in metrics are mainly due to high Correctness/Completeness but low Faithfulness in predictions due to the addition of unsupported details RAG tends to perform better overall in datasets where specific information is required from complex or lengthy documents FC performs better for questions requiring comprehensive understanding of the entire document or when answers are located in structured sections (i.e.

Tom: lists) or temporal sequences RAG outperforms FC for questions that require retrieving specific details or synthesizing scattered information across multiple parts of the document The study concludes that highly-relevant context often outperforms feeding all notes, likely due to LLMs struggling with long context Limitations and Future Directions Limitations include focusing only on MIMIC-derived datasets, which represents English-only data in a single U.S.

Jane: medical center The de-identified nature of the data limits us from examining the potential cultural and other biases of LLMs over long contexts Future directions include scaling beyond single-note or single-source assumptions toward richer multi-note, multi-visit reasoning Advancing temporal and causal reasoning over longitudinal records, with explicit timeline grounding Strengthening evaluation via clinically faithful, judge-robust rubrics and cross-metric reliability analyses The work is intended solely for research purposes Ethical Considerations and Data Usage This work uses MIMIC-III, MIMIC-IV, and four derivative datasets (EHRNoteQA, EHR-DS-QA, RadQA, CliniQG4QA) accessed under the PhysioNet Credentialed Health Data License 1.5.0 All datasets contain de-identified patient records in accordance with HIPAA Safe Harbor standards We exclusively used open-weight LLMs to ensure no patient data were transmitted to third-party proprietary systems Results reflect model performance on question answering benchmarks and should not be interpreted as guarantees of clinical safety, equitable performance, or readiness for clinical deployment The paper is 2510.

Title and authors: Tom: The paper is titled "Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering," and it’s led by Feras AlMannaa, Talia Tseriotou, Jenny Chim, and Maria Liakata.

Jane: It’s important to know that the title clearly signals they are moving beyond simple multiple-choice questions into more complex medical question answering using long patient histories.

Lu: The authors are bringing together different capabilities and sizes of LLMs across various task formats—multiple-choice, extractive tasks, and open-ended generative question answering—to see how they handle these long patient notes.

Meng: From an engineering side, this is interesting because it’s not just about one model; it tests different model sizes and different ways to structure the input data to see what breaks or performs best at scale.

Lalam: It also specifically examines the effect of Retrieval Augmented Generation, or RAG, on comprehension in these long-context medical settings, showing where RAG shines compared to just compressing the prompt.

The paper's summary: Tom: Basically, the study investigates model performance across three task formulations: multiple-choice questions, extractive tasks where you pull out specific text from the notes, and open-ended generative question answering.

Jane: They test these across four different settings for how much context is included in the input—excluding all notes, excluding relevant ones, including all notes, or including only related ones.

Lu: For the "Include All" setting specifically, they look at context sizes ranging from short contexts up to extended contexts that reach twelve eight long context tokens.

Meng: They also examine performance across different LLMs with varying capabilities and different datasets to see how model size affects performance in these long-context medical tasks.

Lalam: The study reveals insights into the limitations of just memorizing information and highlights the benefits of using reasoning models when dealing with this kind of long patient context.

The paper's improvements: Tom: One key area they look at is how RAG improves medical long context comprehension, showing that it can outperform simple prompt compression in question answering tasks across various context lengths.

Jane: They also test two specific chunk inclusion strategies for RAG: direct chunk inclusion and a hierarchical parent note inclusion method, which they call RAG HIR.

Lu: The results show that retrieval augmented generation is superior across all task formulations and context sizes when compared to just using the full context, especially in single versus multi-document question answering datasets.

Meng: They found that RAG excels at extracting precise medical facts, like specific dates or medications, which are usually found in structured parts of a medical record.

Lalam: Conversely, they also showed that feeding the full context directly into the model is better for questions that require temporal reasoning, like sequencing events or calculating how long something lasted.

Conclusion: Tom: To wrap up, the study shows that highly relevant context often beats feeding everything in because LLMs struggle with very long contexts.

Jane: They found disagreements between different metrics depending on the task; RAG is better for specific details, but full context is better for comprehensive document understanding or structured lists.

Lu: The overall implication is that we need to be smart about how we feed patient data into these models, balancing retrieval and generation based on what the specific question demands.

Meng: From a deployment standpoint, this means we have to decide whether to use a more expensive RAG pipeline or just compress the context for certain types of clinical questions.

Lalam: The paper's title, "Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering," shows they are pushing the boundary of what’s possible in clinical AI right now.

More episodes

← Home