How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model
cs.LG, cs.AI, q-bio.QM
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: Accepted at Machine Learning for Health (ML4H) 2026, Findings Track. 10 pages, 6 figures, 3 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: EHR foundation models read medical events within a fixed context window, and long context has so far been measured only in tokens.
Abstract
EHR foundation models read medical events within a fixed context window, and long context has so far been measured only in tokens. How far back in time a model must look to capture the earliest predictive signal is unknown. We introduce time-context, the calendar lookback window before the prediction point, and provide, to our knowledge, the first systematic evaluation of decade-scale time-context for an EHR foundation model. We pretrained a decoder-only transformer (34.6M parameters) on 1.4B coded events from 3.6M primary-care patients in North West London and fine-tuned it to predict incident colorectal cancer under a 12-month pre-diagnostic buffer. A frozen cohort of 1,008 cases and 9,582 matched controls, all registered for at least 30 years, was evaluated at 15 nested lookback windows from 2 to 30 years, so that every truncation is compared within the same patients. Discrimination rose from an AUROC of 0.551 at 2 years to 0.590 at 18 years, then plateaued, with about 80% of the gain reached within 10 years; an XGBoost baseline converged to the same level (peak AUROC 0.589). A simulation further suggests that rotary position embeddings (RoPE) over elapsed time alias beyond about 20 years at the default base frequency, motivating a retuned θ. Time-context is a measurable design axis for clinical prediction models, distinct from token-context, and should be reported.
Sources
- The Llama 3 Herd of Models
- Temporal Data Requirement for Predicting Unplanned Hospital Readmissions
- Foundation Models for Clinical Records at Health System Scale
- Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs
- ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling
- The Taxonomies, Training, and Applications of Event Stream Modelling for Electronic Health Records
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis