Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints

Summary

Problem and Motivation

The paper studies a failure mode in synthetic clinical benchmarks: datasets can pass existing utility checks while remaining structurally unrealistic. The authors state: "Current utility checks can fail to detect unrealistic synthetic benchmarks. In our setting, evaluator-derived utility metrics continued to pass even when the benchmark was extremely sparse, heavily templated, and dominated by patient rows with weak evidence context. They argue this is a serious issue for enterprise healthcare agents because if utility metrics are partially invariant to realism defects, then a dataset can look operationally acceptable while still failing to represent the difficulty and ambiguity of deployment data. The work targets a care-gap benchmark built from Synthea-generated synthetic patients [9] exercised through demonstration electronic health record workflows and processed by the same downstream pipeline as operational client data. They note that schema validity and pipeline compatibility are necessary but insufficient because upstream creation is scenario-driven, which can still suppress the heterogeneity, missingness, and ambiguity seen in naturally occurring clinical populations."

Formulation: Utility-Constrained Realism Improvement

The main contribution is methodological. The authors "propose utility-constrained realism improvement: benchmark revisions should improve realism while staying above the operational utility floor already used by the system. The key idea is to treat utility as a guardrail rather than as sufficient evidence of benchmark quality." Formally, they define dataset D with utility metrics u(D) ∈ R m and realism metrics r(D) ∈ R p, and treat revision as a constrained optimization: maximize weighted improvements in realism subject to utility constraints u j(D') ≥ max τ j, u j(D) − ε j. They also distinguish internal realism from source fidelity, defining source divergence between the benchmark and an aggregate-only operational reference cohort via shared aggregate descriptors, noting A benchmark revision may increase internal difficulty and realism while moving away from the first available source-like reference on some marginals.

Implementation

Exact optimization is intractable, so they use deterministic heuristics that target each realism panel directly, with four design principles: reproducibility, panel targeting, slice awareness, and auditability. Three revision mechanisms are used: selective conversion of some 'MISSING DATA' rows into structured outcomes, deterministic restoration of temporal and evidence fields, and rule-based description rewriting. Key terms: a row is actionable if its outcome is 'ACHIEVED' or 'NOT ACHIEVED'; 'MISSING DATA' rows are non-actionable. Five datasets are compared: Base Dataset, Refinement-A, Refinement-B, Dense Control (a naive densification negative control), and an aggregate-only Reference Cohort. Refinement-A converts a slice-aware subset of MISSING DATA rows, backfills zero-actionable patients, restores temporal fields, and rewrites descriptions; Refinement-B preserves the patient-level structure of Refinement-A and changes only a targeted subset of measure descriptions to recover recommendation availability; Dense Control applies aggressive structural densification but leaves the original baseline measure descriptions unchanged.

Baseline Problems

"The baseline benchmark is clinically thin on the currently implemented realism panels. Sampled-pair missingness is 79.44%, only 12.75% of rows are actionable, and 38.94% of patients have zero actionable measures. Description concentration is also extreme: the top-three first-token share reaches 100.0%. Critically, the current utility checks do not reliably expose that defect, and utility cannot be treated as a sufficient definition of benchmark quality."

Results

Refinement-A improves the internal realism panels: sampled-pair missingness from 79.44% to 72.19%, increases actionable rows from 12.75% to 20.10%, and reduces the zero-actionable-patient rate from 38.94% to 3.11%; top-3 token share drops to 55.56%. Refinement-B preserves those patient-level gains while recovering recommendation-bearing outputs from 45/80 in Refinement-A to 52/80, exceeding the baseline count of 51/80. Dense Control reduces missingness to 59.40% and nearly eliminates zero-actionable patients but it leaves the language profile fully templated at 100.0% top-three-token concentration, showing density improvement alone is not sufficient for realism improvement. Utility is maintained: all four anonymized benchmark variants remain above the current non-inferiority floor of 0.90, including the intentionally unrealistic Dense Control. However, coverage, safety, and demographic appropriateness are largely saturated, while temporal grounding and aggregate failure burden still separate datasets, with Dense Control worst (0.25 dense temporal grounding, 13 failed rows out of 21). Refinement-B occupies the strongest implemented tradeoff point among the evaluated variants: it clears the same number of realism risk flags as Refinement-A while retaining the highest minimum ME utility.

Source Fidelity vs. Internal Realism

The reference cohort reveals a tension: the Base Dataset is closest to the reference with mean absolute delta 9.70, whereas Refinement-A and Refinement-B each score 30.69. The authors do not treat that as a contradiction; instead, it indicates that improving internal benchmark structure can move the dataset away from the first available source-like cohort on some implemented descriptors. Population alignment shows the benchmark is systematically cleaner than the reference population (e.g., age parseable for 88.16% of reference records, unknown race 64.73%), meaning over-cleaning demographic fields can make a benchmark look less realistic even when it appears easier to model.

Implications and Conclusions

The authors recommend expanding realism metrics: Age-band coverage and age-band divergence from a reference cohort should be explicit metrics... Gender, race, ethnicity, and language should also be reported not only for completeness but for over-cleanliness, and the benchmark should track conditional realism rather than only marginal realism. The main limitation is that downstream evidence is still evaluator-based rather than backed by a local gold annotation set, making gold GC annotation, broader reference-cohort pulls, and transfer to additional enterprise-agent benchmarks the most important next steps. The conclusion: utility-constrained realism provides a practical improvement criterion: revise the dataset to improve realism panels without violating the current utility threshold, and source fidelity and internal benchmark realism should be treated as separate axes rather than collapsed into a single notion of quality.

Here are specific improvements to AI systems based on the paper, and what the improved system can do.

1. Utility-constrained realism optimization for benchmark revision

• The system treats utility metrics as a guardrail, not as sufficient evidence of quality.

• It formulates dataset revision as a constrained optimization: maximize realism improvements subject to keeping utility above a non-inferiority floor.

• It can revise synthetic clinical benchmarks to improve realism while guaranteeing that the revised dataset still passes the operational utility threshold.

• It can detect and avoid revisions that improve density but leave language templated, e.g., it will reject a Dense Control-style change that lowers missingness but keeps top-token concentration at 100%.

2. Panel-based realism auditing and targeted repair

• The system evaluates multiple distinct realism panels: missingness, actionability, zero-actionable patients, description diversity, temporal grounding, aggregate failure burden, coverage, safety, and demographic appropriateness.

• It identifies which panels are defective even when overall utility metrics remain above threshold.

• It applies deterministic, reproducible repair heuristics per panel: selectively converting MISSING DATA rows into structured outcomes, restoring temporal/evidence fields, backfilling zero-actionable patients, and rewriting rule-based descriptions.

• It can show exactly which rows and fields were changed and why, enabling auditability.

3. Slice-aware actionability restoration

• The system converts non-actionable rows to actionable outcomes on a slice-aware basis, rather than globally densifying.

• It can raise actionable rows from 12.75% to 20.10% and reduce zero-actionable-patient rate from 38.94% to 3.11% without producing dense-but-unrealistic structure.

• It can maintain or improve downstream recommendation availability while doing so, e.g., recovering recommendation-bearing outputs from 45/80 to 52/80.

4. Separate tracking of internal realism vs source fidelity

• The system reports internal realism and source fidelity as separate axes, not collapsed into one quality score.

• It can detect when a revision improves internal structure but moves away from an aggregate-only reference cohort on shared marginals.

• It can explain that this is an expected tradeoff, not a contradiction, and can show both the internal realism gains and the source divergence simultaneously.

5. Over-cleanliness detection and demographic realism reporting

• The system checks demographic fields for over-cleaning, not just completeness.

• It can flag when age is parseable far more often than in the reference cohort, when unknown-race rates are far lower, and when language/gender/ethnicity fields are artificially clean.

• It reports age-band coverage, age-band divergence, and conditional realism (e.g., age distribution conditional on condition, not just marginal).

• It can warn that over-cleaning demographic fields makes a benchmark look easier to model but less realistic for deployment.

6. Utility metrics that are sensitive to realism defects

• The system augments utility metrics with realism panels so that passing utility no longer hides severe realism failures.

• It can detect that baseline utility remains above floor while missingness is 79.44%, only 12.75% of rows are actionable, and 38.94% of patients have zero actionable measures.

• It can rank candidate datasets by realism risk flags and minimum utility, selecting the variant that clears the most realism risk flags while retaining the highest minimum utility.

7. Capability for downstream healthcare agents

• The improved system can generate more realistic synthetic clinical benchmarks that preserve ambiguity, missingness, and evidence-context difficulty.

• Agents trained or evaluated on these benchmarks will encounter more realistic distributions of actionable vs non-actionable rows, sparse evidence, and templated descriptions.

• The system can produce benchmarks that are harder in clinically meaningful ways while still satisfying operational utility constraints, so enterprise healthcare agents are not overfitted to artificially clean or artificially dense data.

Synthetic clinical benchmarks for enterprise AI agents can pass existing utility checks and still remain structurally unrealistic, especially in privacy-sensitive healthcare settings where operational data are hard to access. We study how to improve such benchmarks without breaking the downstream utility checks already used in practice. We formulate benchmark revision as utility-constrained realism improvement: dataset changes should increase realism while staying above an operational utility floor. We instantiate this idea on a care-gap benchmark derived from Synthea-generated patients exercised through demonstration electronic health record workflows and then processed by the same downstream pipeline as operational data. Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment. The baseline benchmark is extremely thin: sampled-pair missingness is 79.44%, only 12.75% of rows are actionable, 38.94% of patients have zero actionable measures, and top-three token concentration reaches 100.0%. Two deterministic revisions improve these panels while remaining above the current utility floor, whereas a naive densification control preserves unrealistic templating. We further show that internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives. These results suggest that synthetic benchmark quality should be optimized explicitly, with utility treated as one constraint rather than as sufficient evidence of realism.