PaReGTA: A Temporally Aware LLM-Based Patient Representation Framework for EHR Analytics

The gist The proposed PaReGTA framework is an LLM-based encoding system that converts longitudinal EHR events into visit-level templated text, learns domain-adapted visit embeddings via contrastive fine-tuning, and aggregates these embeddings into a fixed-dimensional patient representation using hybrid temporal pooling to preserve both recency and globally informative visits.

How it works

PaReGTA operates through three main stages to transform structured EHR data into patient-level representations The first stage is Visit-level text construction, which involves a deterministic textualization procedure that orders rows chronologically and applies fixed templates to medication and comorbidity fields. This textualization uses temporal prefixes such as PaReGTADate, PaReGTA-Gap, or PaReGTA-Month to explicitly encode temporal cues into the visit text.

The second stage involves Domain adaptation via contrastive learning, where a pretrained sentence-embedding LLM is fine-tuned using a SimCSE-style identical-text contrastive fine-tuning objective. This process adapts the encoder to cohort-specific visit texts by optimizing the loss function to ensure that embeddings of the same visit text remain close while ranking the paired copy above other batch examples.

The third stage is Hybrid temporal pooling into patient representations, which aggregates visit-level embeddings using a scheme that combines time-decay weighting to emphasize recency and attention-based weighting to emphasize globally informative visits. This results in a hybrid representation defined as z = αzattn + (1 − α)ztime, where α controls the contribution of the attention-pooled representation versus the time-weighted representation.

Key Methodological Components

The framework incorporates several specific mechanisms designed to handle EHR data challenges

1. Visit-level textualization involves constructing domain-specific strings for medication and comorbidity fields, using fixed templates like Medication first visit, meds: acetaminophen 325 MG Oral Tablet.

2. Temporal cues are introduced via three schemes: PaReGTADate uses the calendar date and appends an elapsed-day marker, PaReGTA-Gap uses the elapsed time between consecutive visits, and PaReGTA-Month expresses the interval in 30-day month bins.

3. Domain adaptation is achieved by fine-tuning a model like GTE-basev1.5 using the MultipleNegativesRankingLoss objective to align embeddings with cohort-specific texts.

4. Hybrid pooling combines a time-weighted component, where weights are calculated based on elapsed days (di = max(tN − ti)days, 0), and an attention component, where visit embeddings are concatenated with the log-transformed elapsed-time feature δi before passing through an MLP to obtain attention weights.

Evaluation and Results

The framework was evaluated on a retrospective patient-level classification task using a cohort of 39,088 patients with migraine from the All of Us Research Program (AoU). In an exploratory comparison on the fixed held-out test cohort, PaReGTA-Gap + LightGBM had the highest reported AUC, accuracy, and F1 values among the evaluated sparse, BERT-based EHR, and recurrent approaches.

A key finding from the factor-removal analysis is that PaReGTA-RSS quantifies changes in the fitted logit of a fixed logistic-regression model after removing clinically defined factor groups. For example, removal of onabotulinumtoxinA produced the largest average absolute fitted-logit change among medication factor groups.

Further ablation studies showed that temporal information substantially outperforms baselines; specifically, PaReReGTA-Gap and PaReGTA-Month had higher point estimates than PaReGTA-Without across the five downstream classifiers. The paired differences between PaReGTAGap + LightGBM and T-LSTM were positive for AUC, accuracy, and F1 in the fixed-test bootstrap analysis, indicating modest but consistently positive differences in favor of PaReGTA-Gap + LightGBM.

Limitations

The study acknowledges several limitations regarding its findings First, PaReGTA-RSS depends on the selected factor definitions, removal rule, learned representation, fitted preprocessing transformation, and downstream logistic-regression model. Second, T-LSTM and RETAIN were trained using length-bucket batching to address training instability due to sequence length heterogeneity. Finally, the experiments are retrospective and do not establish general superiority across datasets or clinical tasks because they are limited to a single All of Us cohort.

The study concludes that PaReGTA provides a modular alternative to sparse aggregation and specialized sequence models for generating fixed-dimensional, temporally aware patient-level representations for retrospective EHR analysis.

1. Bold header: Temporal Information Encoding via PaReGTA

PaReGTA converts longitudinal EHR events into visit-level templated text with explicit temporal cues, allowing downstream models to leverage explicit temporal information rather than losing it in sparse one-hot or count-based representations. This enables models to capture distinct clinical narratives (e.g., the emergence of one disease followed by another) may become indistinguishable by explicitly encoding the sequence and timing of events.

2. Bold header: Domain Adaptation via Contrastive Learning

The system employs a lightweight contrastive fine-tuning of a sentence-embedding model using a SimCSE-style identical-text contrastive fine-tuning, which adapts the encoder to cohort-specific visit texts. This improves generalization by learning embeddings that are relevant to the specific clinical vocabulary and event patterns of the cohort, addressing challenges where site-specific vocabularies complicate modeling.

3. Bold header: Hybrid Temporal Aggregation for Patient Representation

PaReGTA aggregates visit embeddings into a patient representation using a hybrid temporal pooling that captures both recency and globally informative visits, combining a time-decay weighting mechanism with an attention-based weighting mechanism to emphasize different temporal aspects of the record. This results in representations that are used with conventional downstream machine-learning models, providing a fixed-dimensional patient representation compatible with existing pipelines.

4. Bold header: Factor Sensitivity Analysis via PaReGTA-RSS

The system introduces PaReGTA-RSS, which is a prespecified factor-removal analysis that recomputes patient representations after removing clinically defined factor groups and quantifies the resulting change in the fitted logit of a fixed logistic-regression model. This allows researchers to examine how the fitted pipeline responded to the evaluated EHR inputs by quantifying shifts in predictive performance when specific clinical factors (like medication or comorbidity groups) are removed.

5. Bold header: Robust Performance Across Temporal Encoding Schemes

A comprehensive ablation study shows that LLM-based embeddings with temporal information (Date or Gap) substantially outperform the baselines across all five classifiers, indicating that Gap showed the strongest overall point-estimate pattern among the examined temporal representations for this task, suggesting that capturing patient-specific progression pace is particularly effective.

6. Bold header: Flexible Input Handling via Medication Information

PaReGTA demonstrates flexibility by encoding deterministically constructed product-level strings when medication records are available, and maintaining strong performance using comorbidity-derived visit text alone when medication records are missing or incomplete, thereby reducing preprocessing burden across different real-world EHR settings.