Tracing the Heart: An Evidence-Linked Pipeline for Heart-Failure Feature Engineering

The paper addresses the critical bottleneck of electronic health record (EHR) feature engineering in clinical research and AI. The authors state: EHR feature engineering is a major bottleneck in clinical research and AI, accounting for 39–45% of data scientists' workload. They note that at the median U.S. data-scientist wage of 112,590 per year, this corresponds to roughly 845–975 of labor per 40-hour workweek devoted to data preparation. This is especially pronounced in heart failure, which affects an estimated 6.7 million U.S. adults and requires integrating fragmented EHR data with disease-specific, guideline-based clinical reasoning.

The authors argue that Existing rule-based and large language model (LLM)-based approaches offer only partial automation with limited maintainability and evidence traceability. Heart failure phenotyping is particularly demanding because clinically meaningful variables are distributed across multiple EHR domains and often require longitudinal interpretation — diagnoses, medications, laboratory measurements, procedures, and imaging findings are stored in separate relational tables rather than in a single analysis-ready dataset.

The authors developed the Nimblemind Multi-Agent System (nMAS), described as an evidence-linked, rubric-grounded pipeline for automated heart-failure feature engineering. Rather than "generating unrestricted clinical summaries, nMAS applies deterministic scoring rules derived from clinical specifications and uses a restricted language-model auditing step to evaluate the resulting feature construction. Each derived feature remains linked to its source EHR variables and predefined scoring criteria, enabling automated processing while preserving interpretability and auditability."

The workflow begins with a request specifying the input EHR exports and constructs the target features defined in the versioned rubric. An Achievability Agent first checks that the available source data can support each target feature; features whose required inputs are absent are reported as unsupported rather than assigned inferred values.

The feature-engineering pipeline has two stages:

• Stage 1 builds a single patient-level table from the EHR source tables: (1) standardizes each table (whitespace, null tokens, uppercased identifiers, and ISO datetimes); (2) deduplicates each table on a clinically meaningful event key; (3) temporally aggregates each source to the patient level independently; (4) merges per-table summaries into unique patient rows.

• Stage 2 enriches this table with higher-order composite features: produces higher-order feature categories including a comorbid-burden category spanning five organ systems, each returned as a compact value paired with a structured evidence trace; reviews composite features with an LLM auditor; and checks structural integrity, rubric compliance, monotonicity, and evidence traceability before release.

The evaluation dataset consisted of 500 dummy patient records drawn from an aggregated cohort modeled on REDACTED, a cardiovascular program serving a minoritized population. Patients were included if they had a documented heart-failure diagnosis in the source EHR; no additional inclusion or exclusion criteria were applied. Each patient was represented "across nine structured EHR tables covering demographics, other diagnoses, medication orders, echocardiogram procedures, surgical procedures, laboratory components and external labs, ejection fraction flowsheet entries, and social history."

A clinical-reasoning layer converted raw text and codes into interpretable flags. Notable design choices include: behavioral exposures were kept as ordered categories rather than binary flags, because cardiovascular risk from tobacco and alcohol persists after cessation and varies by intensity, and the heart-failure phenotype was dictated by ICD-derived values rather than diagnosis names, carrying a conflict flag when the two disagreed.

The scoring rubric defines the conditions, point assignments, and evidence hierarchy used to transform structured EHR data into interpretable feature scores. It was developed using Qwen 2.5-1.5B-Instruct, which assembled recent guideline-level literature into a proposed scoring framework. A professional cardiologist then reviewed and approved the rubric in full. The final version included 22 references from 2023 onward with each reference linked to the component it supports.

Seven of the eight category features use the same scoring approach: "disease severity, cardiovascular risk, the five comorbid-burden scores (kidney, lungs, metabolic, blood, and brain-health context), and demographic vulnerability. Each feature sums component points, caps the total at 100, and maps the score to an ordinal label. Scores of 55 or above are classified as high, 25 to 54 as medium, and below 25 as low. The heart-failure phenotype feature uses a separate logic because ejection-fraction classes are categorical thresholds rather than weighted composites, assigned using a precedence rule over available evidence" — HFrEF at midpoint ≤ 40, HFmrEF below 50, and HFpEF otherwise.

Each candidate row is audited by Qwen 2.5-1.5B-Instruct. The prompt provides patient evidence, the candidate row, and a whitelist of correctable fields; the model returns valid": true when the row is correct or a minimal corrections object otherwise. Only whitelisted numeric and categorical fields may be modified; fields ending in json or explanation are protected to preserve deterministic evidence traces and rationales. Corrections are type-coerced before application, while unparseable responses are logged as audit failures and leave the original scores unchanged. The auditor runs locally, and each row records the model, raw response, and generated* status, ensuring that auditing is a required and traceable pipeline step rather than an optional post-processing stage."

Structured Merge: Stage 1 collapsed nine source tables to the patient level without Cartesian explosion, producing exactly one row per patient, no duplicates, and 132 structured columns across all 500 patients. Notably, numeric ejection fraction was available for only 3.0% of patients, requiring diagnosis-name fallback for phenotype assignment.

Higher-Order Composite Features: "Stage 2 generated the feature categories... as 70 columns, with all 500 rows audited by Qwen 2.5-1.5B-Instruct. Each feature included a structured evidence trace documenting contributing components, point values, source columns, and input availability. The pipeline also produced a catalog of thirteen candidate conditions spanning cardiac, vascular, metabolic, renal, pulmonary, and hematologic domains."

Predictive Performance: "Adding the composite features improved performance for both classification tasks across all evaluation metrics. The largest improvement was observed for HFrEF classification, with accuracy increasing from 0.776 to 0.896 (+0.120)." Specifically:

• HFrEF: accuracy improved from 0.776 to 0.896, AUROC from 0.895 to 0.963, F1 from 0.776 to 0.894

• HFpEF: accuracy improved from 0.752 to 0.809 (+0.057), AUROC from 0.870 to 0.910, F1 from 0.758 to 0.812

Feature Importance: "For HFrEF classification, 6 of the top 10 features were composite features generated by the proposed framework, with demographic vulnerability adjusted gap and disease severity adjusted gap ranking above the highest-ranked baseline feature, HF GDMT medication class count. For HFpEF, two composite features appeared in the top 10, with disease severity score as the most influential composite predictor."

LLM-Based Construct-Validity Evaluation: The LLM-based construct-validity assessment showed strong overall performance, with most categories exceeding 85% and an overall normalized score of 81.5%. Lower scores were observed for demographic vulnerability and gap features, with care recommendations receiving the lowest score (38.5).

Ablation Studies: Equal-weight and random-weight ablations fixed the aggregated feature set while varying only the scoring logic. Both substantially reduced performance... indicating that rubric weighting contributed to the improvement. Category-level ablation found removing care-gap features produced the largest performance reduction (∆AUROC=0.039) for HFrEF, while for HFpEF, disease severity was the dominant contributor (∆AUROC=-0.033).

Comparator Study: An independent comparator using Claude Opus 4.8 to engineer features directly from raw EHR data primarily generated features based on directly observed clinical variables but "features derived from heart-failure diagnosis codes and ejection-fraction measurements received lower construct-validity scores because they were closely related to the classification labels themselves, creating predictor-construction concerns."

The authors state: "We developed a pipeline that transforms fragmented EHR exports into a patient-level heart-failure cohort with rubric-derived, evidence-linked feature categories... Independent literature-grounded grading scored seven of eleven feature categories above 85/100. They emphasize that the pipeline provides a reproducible and auditable approach for clinical data preparation. The generated features retain links to their source variables, scoring criteria, and supporting evidence, enabling transparent review of how each feature was constructed. They note the framework separates clinical knowledge representation from feature execution, allowing the scoring framework to be updated independently from the underlying data-processing pipeline, making it reusable well beyond heart failure for any data-intensive chronic disease facing the same feature-engineering bottleneck."

The authors acknowledge: "The pipeline was evaluated on a dummy, single-institution cohort of 500 patients, limiting statistical power and external generalizability. HFrEF and HFpEF were classified against phenotype-unknown patients rather than non-HF controls, so performance reflects phenotype characterization within an HF cohort rather than general disease detection. Evaluation focused on internal consistency, feature traceability, and downstream classification rather than external clinical validity, and some aggregated features incorporate phenotype-related evidence, creating potential feature-label overlap in performance and ablation analyses. Prospective clinical impact and patient outcomes were not assessed. Comorbidity and behavioral features rely partly on regular-expression matching and may miss atypical or institution-specific language. Finally, clinician review covered the extraction rubric and representative outputs rather than the full cohort."

"Future work should evaluate the pipeline on larger, multi-institutional cohorts with temporally separated validation sets to assess generalizability. Broader evaluation should include both heart-failure and non-heart-failure populations, independent clinician-adjudicated phenotypes, and additional phenotyping and outcome-prediction tasks, as well as prospective workflow impact. Until such validation, the pipeline should remain an evidence-linked feature-engineering layer for research and clinical review rather than autonomous clinical decision-making."

• Replace free-form clinical-summary generation with rubric-grounded, deterministic feature construction. Instead of letting an LLM infer clinical variables directly from raw EHR text, the AI system first applies explicit, versioned scoring rubrics (with point assignments and evidence hierarchies) to structured source tables. This eliminates hallucinated or untraceable clinical values.

• Add an achievability preflight agent. Before any feature engineering begins, the system checks whether required source fields exist for each target feature. If a required input is absent, the feature is reported as unsupported rather than silently imputed or guessed. This prevents fabricated data and makes missing-data limitations explicit.

• Separate clinical knowledge representation from feature-execution code. The rubric (conditions, point values, evidence references) is stored as a standalone, versioned artifact that can be updated by clinicians without rewriting the data-processing pipeline. This lets the AI system adapt to new guidelines or new diseases without retraining or code changes.

• Make every derived feature evidence-linked and auditable. Each feature output includes a structured trace: contributing components, point values, source column names, and input-availability flags. The AI system should not output a bare number or label; it should output a nested evidence object that can be reviewed, queried, and re-computed.

• Use a restricted LLM auditor rather than an open-ended LLM judge. The LLM is allowed only to validate candidate rows and propose minimal corrections from a whitelist of numeric/categorical fields. Protected fields (e.g., json evidence traces, explanation rationales) cannot be modified. Unparseable responses are logged as audit failures and leave original scores unchanged. This makes LLM auditing a required, traceable pipeline step while preventing the LLM from rewriting deterministic evidence.

• Build patient-level temporal aggregation without Cartesian explosion. The system collapses multiple relational EHR tables into exactly one row per patient by standardizing tables, deduplicating on clinically meaningful event keys, and temporally aggregating each source independently before merging. This avoids the common multi-table join explosion and produces stable, analysis-ready tables.

• Preserve ordinality and clinical nuance instead of forcing binary flags. Behavioral exposures such as tobacco and alcohol are kept as ordered categories because cardiovascular risk persists after cessation and varies by intensity. The AI system should represent risk factors with clinically meaningful granularity rather than collapsing them into binary indicators.

• Separate phenotype logic from composite scoring. Ejection-fraction-based heart-failure phenotypes use categorical threshold precedence (HFrEF ≤ 40, HFmrEF < 50, HFpEF otherwise) rather than weighted point sums. The AI system should not apply one generic scoring method to all features; it should support category-specific scoring rules.

• Use rubric weighting and ablate it explicitly. Features are scored with rubric-derived weights, and the system should support ablation studies that fix the feature set while varying the weighting logic. This reveals whether the rubric weighting itself contributes to predictive performance, not just the presence of extra columns.

• Add a construct-validity audit layer using an independent LLM. After feature generation, an LLM-based scorer rates whether each feature category actually measures its intended construct, flagging predictor-construction problems (e.g., features too closely tied to the classification label). This gives a quantitative, LLM-driven quality gate before features are used downstream.

• Log the full audit trail for every row. The system records the model version, raw LLM response, and generated* status for each audited row. This allows full reproducibility, debugging, and regulatory-style review of how each feature was constructed and validated.

• Automatically convert fragmented multi-table EHR exports into a clean, one-row-per-patient cohort with hundreds of structured, clinically meaningful features—without manual data-scientist labor.

• Generate features that are not black boxes: each feature carries a complete evidence trace (source tables, contributing values, scoring rule, and supporting literature reference), so clinicians can see exactly why a patient received a given severity score or risk category.

• Detect and report unsupported features before processing begins, so downstream users know which variables are missing and which features are therefore unreliable or unavailable.

• Validate its own feature construction via a locally running, restricted LLM auditor that can fix only whitelisted fields and cannot corrupt deterministic evidence traces.

• Produce both low-level structured columns and higher-order composite features (disease severity, cardiovascular risk, comorbid-burden scores across kidney/lungs/metabolic/blood/brain, demographic vulnerability, care-gap features) with ordinal labels and thresholds.

• Improve downstream classification: for example, in the paper's evaluation, HFrEF classification accuracy rose from 0.776 to 0.896 and AUROC from 0.895 to 0.963 when the evidence-linked composite features were added. The improved AI system can deliver similar predictive gains for heart-failure phenotyping.

• Prioritize explainable and clinically defensible predictors: in HFrEF classification, six of the top ten features were composite features with explicit rubric links, ranking above raw medication counts.

• Support transparent updating: when clinical guidelines change, clinicians can edit only the scoring rubric (e.g., new point thresholds, new evidence references) and re-run the pipeline—no code changes or model retraining required.

• Scale to other chronic diseases beyond heart failure, because the pipeline separates disease-specific rubric content from the reusable data-processing and auditing infrastructure.

• Remain safe for research and clinical-review workflows by refusing to generate unrestricted clinical summaries, instead producing deterministic, auditable, evidence-linked features that clinicians can independently verify.

Electronic health record (EHR) feature engineering is a major bottleneck in clinical research and AI, accounting for 39-45% of data scientists' workload. This is especially pronounced in heart failure, which affects an estimated 6.7 million U.S. adults and requires integrating fragmented EHR data with disease-specific, guideline-based clinical reasoning. Existing rule-based and large language model (LLM)-based approaches offer only partial automation with limited maintainability and evidence traceability. We developed the Nimblemind Multi-Agent System (nMAS), an evidence-linked, rubric-grounded pipeline for automated heart-failure feature engineering, and evaluated it on 500 dummy patient records from nine EHR source tables. nMAS generated 132 structured and 70 rubric-scored aggregated features, verified for structural integrity, rubric compliance, and provenance, and audited by a restricted LLM. Adding the aggregated features improved held-out AUROC from 0.895 to 0.963 for HFrEF and 0.870 to 0.910 for HFpEF phenotyping, and an independent LLM-based rubric assessment of evidence support and methodological soundness scored the features at 81.5% of maximum points. These results demonstrate the feasibility of automated, auditable feature engineering for complex cardiovascular EHR data, though evaluation was limited to a single-institution cohort and external validation is needed.