Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
cs.CL, cs.AI
Submitted: 2026-06-17
Updated: 2026-08-31
Comments: 35 pages with 8 figures
Code: https://github.com/qgpmztmf/PhysAssistBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated capabilities: clinical knowledge, EHR system interaction, or
Terminology
Abstract
The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated capabilities: clinical knowledge, EHR system interaction, or patient communication. Physician assistance instead requires coordinating these capabilities within the same interaction, where physicians issue underspecified requests, patients describe symptoms ambiguously, and EHR systems demand precise tool use. We introduce PhysAssistBench, a benchmark for interactive doctor-patient-EHR assistance. Built from real MIMIC-IV cases, PhysAssistBench uses a scalable pipeline to construct agentic patients: interactive, record-grounded agents that turn static EHR records into multi-turn clinical scenarios while preserving clinical factuality. PhysAssistBench provides a curated bilingual evaluation set of 1,296 manually reviewed and physician-validated turns. Experiments with leading LLMs show that current models remain unreliable in this setting, which exposes a key bottleneck for clinical LLMs: reliable assistance requires coordination across knowledge, communication, and systems, not isolated gains in any of them.
Sources
- Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM
- GLM-5: from Vibe Coding to Agentic Engineering
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Kimi K2: Open Agentic Intelligence
- Seed1.8 Model Card: Towards Generalized Real-World Agency
- ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries
- LLMs Get Lost In Multi-Turn Conversation
- FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering
- ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- Qwen3 Technical Report
- Qwen3.5-Omni Technical Report
- RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Game
- OpenAI GPT-5 System Card
- Benchmarking LLM Tool-Use in the Wild
- Towards Expert-Level Medical Question Answering with Large Language Models
- MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworks
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering