Symphony for Text Generation: Benchmarking Clinical Note Generation
cs.LG, cs.AI, cs.CL
Submitted: 2026-10-06
Updated: 2026-10-06
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized.
Terminology
Abstract
Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized. We introduce MedConv, a multilingual dataset of 300 clinical encounters in English, Danish, and German, and use it alongside the Ambient Clinical Intelligence benchmark (ACI-BENCH) to compare Corti, a clinical AI platform, with two leading, accessible ambient scribe software applications built on general-purpose AI. We present a controlled clinical evaluation framework that combines entailment metrics with LLM-judged pairwise comparisons across eight dimensions adopted from PDSQI-9. Results show that Corti's API-based text-generation infrastructure is on par with or outperforms leading commercial scribes. We further show that Corti's configurable API provides the flexibility necessary to fine-tune quality dimensions for specific documentation use cases. We present the evaluation methodology and release a dataset to support future reproducible comparison of ambient documentation systems.
Sources
- One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
- LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
- Symphony for Speech-to-Text: Supporting Real-Time Medical Voice Interfaces
- Symphony for Medical Coding: A Next-Generation Agentic System for Scalable and Explainable Medical Coding
- FactsR: A Safer Method for Producing High Quality Healthcare Documentation
- LLM-based NLG Evaluation: Current Status and Challenges
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks