Toward Automated Detection of Documentation Inconsistencies in Electronic Health Records

The gist: The pipeline surfaces 3,460 candidate inconsistencies, affecting 69.7% of admissions, and reveals recurring failure modes that arise when verification requires temporal reasoning or evolving-diagnosis context that general-domain LLMs do not natively possess.

Background and Significance

Electronic Health Records (EHRs) are fundamental for modern healthcare, supporting a wide range of downstream tasks such as clinical care management,[1] clinical outcome prediction,[2, 3] medical named-entity recognition,[4] and patient phenotyping.[5] However, the reliability of information within EHR datasets is not infallible, with prior studies documenting pervasive issues such as incomplete documentation, internal inconsistencies, and clinically meaningful errors across care settings.[6–9] These documentation errors can propagate through clinical workflows and contribute to diagnostic errors, which affect an estimated 5% of outpatient encounters.[10] Error of documentation is a widely recognized safety risk and can lead to inappropriate or delayed care.[11] In addition, common documentation workflows such as copy-paste and templating can propagate outdated or incorrect information across notes, creating internal contradictions and, in documented cases, contributing to missed diagnoses and patient harm.[12–14] Medication discrepancies are another example: inconsistencies in medication histories and discharge documentation can persist across transitions of care and contribute to adverse drug events that are not reliably caught by routine electronic checks.[15, 16]

Methods

The study applied a two-stage LLM pipeline—open-ended candidate identification (Gemini 2.5 Pro) followed by context-grounded verification (Gemini 2.5 Flash)—to 3,000 randomly sampled MIMIC-IV-Note discharge summaries.[1] The pipeline surfaces inconsistencies across demographics, allergies, procedures, diagnoses, laboratory results, medications, and care planning domains,[6] many of which carry serious downstream implications for clinical reasoning and patient safety.[4]

The iterative pipeline development involved two stages: Stage 1 is open-ended detection where the model scans the entire document for candidate inconsistencies,[5] while Stage 2 is context-grounded verification where a specific candidate pair is assessed against surrounding clinical context,[6]. The compute allocation reserves Gemini 2.5 Pro for Stage 1 (candidate identification) due to its requirement for hypothesis generation, and Gemini 2.5 Flash for Stage 2 (verification) because it handles the more constrained reasoning task more efficiently.[7]

Pipeline Output and Formative Case Analysis

The pipeline identified 3,460 statement pairs flagged as potentially inconsistent after Stage 2 verification, with at least one candidate inconsistency detected in 69.7% of admissions.[8] Representative examples spanned demographics, allergies, procedures, diagnoses, laboratory results, medications, and care-planning domains,[9] with direct implications for clinical reasoning or patient safety.[10]

Key findings across domains included:

Demographics:

Demographic inconsistencies are relatively mild in consequence but notable when they appear, typically reflecting copy-paste errors rather than clinical disagreement.[8]

Allergy:

Prescribing a medication to which the patient has a listed allergy is a documented safety risk; discrepancy requires clarification before dispensing.[9] Allergy-related inconsistencies arose from copy-paste carry-forward of legacy entries, intersections between drug-class labels and individual agents, or entries that record adverse drug reactions rather than true allergies.[10]

Medication:

The same medication entry contains two different dosing frequencies (once daily vs. twice daily), creating a risk of dosing error with implications for amiodarone toxicity.[9] Medication-related inconsistencies were a common kind of detection in the MIMIC-IV cohort, frequently arising when an update to one section of a discharge summary was not propagated to others.[10]

Recurring Failure Modes

A recurring subset of flagged pairs proved reconcilable with clinical context, reflecting systematic patterns in how the language model interprets discharge documentation.[11] Four failure modes were identified:

1. Allergy Documentation clarity: Flagging patient-reported sensitivity is a clinical decision-support opportunity rather than a documentation error.[12]

2. Allergy Clinical nuance: Statin cross-reactivity is limited; switching within the class is standard practice when the prior reaction is intolerance rather than true allergy.[12]

3. Diagnosis & clinical finding Physical exam documents 2+ DP pulses bilaterally; arterial studies show monophasic waveforms.[10]

4. Medication Temporal progression: Hospital course plans a 10-day course of doxycycline; discharge prescription dispenses only 9 tablets (4.5 days at BID dosing).[13]

Discussion and Framework

Detection requires anchoring each statement to its source section and clinical domain, then assessing whether the conflict reflects a true contradiction or missing context.[12] The study proposes a graded ontology spanning strict contradiction and ambiguity, with a schema characterizing each flagged case by category, section, domain, and inconsistency axis.[12] This framework defines inconsistency as either a strict contradiction (where exactly one statement must be true) or an ambiguity (where the documentation is underdetermined but still warrants attention).[15] The schema includes dimensions such as inconsistency type, section, scope (within-phase or spanning two phases), domain, attribute specific clinical concept in conflict, and axis logical form.[17]

Conclusion

This formative study establishes a methodological foundation and conceptual framework to guide subsequent validated, large-scale EHR-inconsistency analysis.[12] The contribution is to expose the structure of the problem, identify recurring candidate inconsistencies and failure modes, and propose a testable ontology-schema framework.[18] The present study should be interpreted as methodological and formative rather than as a definitive validation study.[18]

Future Work

The next step is constructing a curated, clinician-labeled corpus of documentation inconsistencies in which each case is independently adjudicated by clinical experts using the current ontology and schema.[20] A second direction concerns matching algorithm classes to subtasks, suggesting that temporal-progression failures might benefit from purpose-built temporal models rather than general-purpose LLMs prompted to consider timing.[21]

Competing Interests

All authors declare no financial or non-financial competing interests.[23]

Funding

The research of J.L., M.T., W.W.S., and A.R.Z. was supported in part by NIH grant R01HL169347.[23] The research of A.R.Z. was also supported in part by NIH grant R01HL168940.[23]

Data Availability Statement

The discharge summaries analyzed in this study were drawn from the MIMIC-IVNote module (v2.2) of the Medical Information Mart for Intensive Care IV (MIMICIV) database, a publicly available, de-identified clinical dataset distributed through PhysioNet under credentialed access.[23] Access is available to qualified researchers who complete the PhysioNet Credentialed Health Data Use Agreement and the required CITI Data or Specimens Only Research training.[23]

References

[1] Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine. 2018;24(11):1716–1720. https://doi.org/10.1038/s41591-023-02448-8.[29]

[6] Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Medical Informatics Assoc. 2013;20(1):144–151. https://doi.org/10.1136/amiajnl-2013-002627.[6]

[8] Balogh EP, Miller BT, Ball JR, editors. Improving Diagnosis in Health Care. Washington, DC: National Academies Press; 2015. Committee on Diagnostic Error in Health Care, Board on Health Care Services, Institute of Medicine, National Academies of Sciences, Engineering, and Medicine.[8]

[12] The Joint Commission.: Safe Practices for Copy and Paste in the Electronic Health Record. Georg Thieme Verlag KG. Sentinel Event Alert, Issue 54.[13]

[13] Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct Text Entry in Electronic Progress Notes: An Evaluation of Input Errors. Methods of Information in Medicine. 2003;42(1):61–67. https://doi.org/10.1055/s-0038-1634210.[13]

[29] Toma A, Xie R, Palayew S, Lawler P, Wang B. WangLab at MEDIQA-CORR 2024: Retrieval-Augmented and DSPy-Optimized LLM Programs for Medical Error Detection and Correction. In: Proceedings of the 6th Clinical Natural Language Processing Workshop. Mexico City, Mexico: Association for Computational Linguistics; 2024. p. 616–623.[30]

[35] Comanici G, Bieber E, Schaekermann M, Pasupat I, Sachdeva N, Dhillon IS, et al. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXivorg.

1. textbfReal-time Contextual Verification Engine (Stage 2 Implementation): The improved system can perform context-grounded verification by using a faster, more cost-efficient model like Gemini 2.5 Flash for Stage 2 to reason about candidate pairs step by step, specifically to identifying what each statement claims, locating it within the discharge summary, and assessing whether timing or broader clinical context could reconcile the two.

2. textbfAdaptive Model Allocation Strategy: The system can optimize computational efficiency by using Gemini 2.5 Pro for Stage 1 (candidate identification) and Gemini 2.5 Flash for Stage 2 (verification) based on the required level of reasoning, ensuring that the more capable model is reserved for open-ended candidate generation while a faster model handles context-grounded verification.

3. textbfExplicit Temporal and Clinical Reasoning Module: The system can address temporal-progression failures by moving beyond treating the discharge summary as a flat block of text, instead focusing on the explicit extraction of a clinical-event timeline (imaging dates, laboratory draw times, prescribed course start/end, inpatient administration history) that downstream comparisons can be evaluated against.

4. textbfGraded Ontology and Schema Integration: The system can utilize the proposed framework to define a graded ontology of clinical inconsistency spanning strict contradiction and ambiguity, coupled with a schema for characterizing each flagged case by category, section, domain, and inconsistency axis to provide a structured basis for adjudication.

Objective:To characterize the kinds of internal documentation inconsistencies a general-domain large language model (LLM) can surface from real-world discharge summaries, and to identify recurring failure modes that limit reliability at scale. Materials and Methods: We applied a two-stage LLM pipeline---open-ended candidate identification (Gemini 2.5 Pro) followed by context-grounded verification (Gemini 2.5 Flash)---to 3000 MIMIC-IV-Note discharge summaries drawn as three contiguous 1000-note batches from distinct positions in the corpus. Two clinicians independently reviewed a domain-stratified sample of 268 retained candidates; a third adjudicated disagreements. Results: Stage 1 proposed 11333 candidates; 3461 were retained after Stage 2 verification---machine-generated candidates---with at least one in 69.7% of admissions. Medication (43.2%) and diagnosis or clinical-finding (25.3%) candidates predominated. In the reviewed sample, reviewers agreed on 91.0% of candidates, and 253 of 268 (94.4%) were judged valid after adjudication. Representative examples from all seven domains carried direct implications for clinical reasoning or patient safety. Review also revealed recurring failure modes involving temporal reasoning, evolving diagnoses, and outpatient-prescribing conventions. Discussion: Detection is highly context-dependent: judging a flagged pair requires locating each statement and assessing whether the conflict reflects a true contradiction or missing context. We propose a graded ontology spanning strict contradiction, ambiguity, and gap, with a schema characterizing each flagged case by type, section, scope, domain, attribute, and inconsistency axis. Conclusion: This formative study establishes a methodological foundation and conceptual framework to guide subsequent validated, large-scale EHR-inconsistency analysis.