Towards Expert-level Medical AI for Real-time Video Consultations

This paper presents the first demonstration of expert-level AI performance in real-time clinical video consultations, using the Articulate Medical Intelligence Explorer (AMIE) in a video configuration, called AMIE (Video). The system is a Gemini-based, asynchronous multi-agent architecture comprising a Talker Agent, a Planner Agent, and a Perception Agent, designed to balance low-latency dialogue with thorough clinical reasoning and real-time audio-visual perception.

The study's key contributions are threefold: (1) the introduction of AMIE (Video) with its multi-agent architecture, guided by an automated evaluation framework derived from a taxonomy of clinical audio-visual competencies; (2) a multi-arm randomized Objective Structured Clinical Examination (OSCE) study comparing AMIE (Video), its text-only counterpart AMIE (Text), and 10 board-certified primary care physicians (PCPs) across 100 clinical scenarios with 15 professional patient actors, evaluated by 20 independent clinical evaluators; and (3) key findings showing that clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination, while patient actors preferred AMIE's video interface over text chat for communicative effectiveness, convenience, and feeling understood.

In the OSCE study, clinical evaluators rated AMIE (Video) equivalent to or better than PCPs across all six case-specific rubric domains, with an overall score of 83% compared with 68% for PCPs (p = 1.32 × 10−9). AMIE (Video) matched the correct reference diagnosis in its top item of the differential in 91% of cases, compared with 77% for PCPs (p = 0.039). On case-specific rubrics, AMIE (Video) scored 90% compared with 76% for PCPs on clinical reasoning (p = 2.00 × 10−5), and 79% compared with 67% on treatment planning (p = 8.90 × 10−5). AMIE (Video) scored higher than PCPs at utilizing live video to augment information gathering (77% vs. 51%, p = 1.37 × 10−4) and guiding patients through physical examinations (72% vs 39%, p = 2.46 × 10−9). Patient actors preferred AMIE (Video) for assessing and explaining conditions, while PCPs were preferred (though without a significant difference) for building rapport and partnership.

In the modality ablation, patient actors rated AMIE (Video) significantly higher than AMIE (Text) on effectiveness in communicating health concerns (89% vs 79%, p = 4.98 × 10−5), ease and convenience of use (88% vs 71%, p = 2.98 × 10−6), and feeling that their concerns were fully understood (90% vs 81%, p = 2.62 × 10−4). Clinical evaluators rated AMIE (Video) higher than AMIE (Text) on overall case-specific rubrics (83% vs 75%, p = 1.19 × 10−8), with the largest gains in the perception and examination domain.

Automated evaluations showed that the specialized agents meaningfully contributed to overall system performance, with the full system achieving 59% compared with 42% for the Talker-only baseline in single-turn evaluations. The asynchronous orchestration reduced mean turn latency from 21.4s to 2.6s compared to prior work. However, gaps remained in subtle and high-frequency perceptual competencies, including nystagmus (3%), tremors at rest or with action (24%), and affective state, expressiveness, and mood congruence (29%).

The paper acknowledges several limitations, including the system's reliance on discrete turn-taking rather than continuous, multi-stream interactivity, which prevents simultaneous speech or visually triggered interjections. The study was limited to actable clinical presentations enacted by patient actors, excluding conditions like dermatological cases that cannot be authentically simulated. Complete blinding was challenging due to the synthetic vocal style of the Talker agent and the inherently different interaction format of the text-only arm. The authors state that the system is not ready for real-world clinical deployment and that substantial validation in real clinical workflows is essential before responsible translation.

Improvements to AI Systems:

1. Implement a multi-agent asynchronous architecture where a Talker Agent handles real-time user interaction, a Planner Agent performs deep clinical reasoning in parallel, and a Perception Agent processes audio-visual input. This reduces turn latency from 21.4s to 2.6s while maintaining diagnostic accuracy.

2. Add a perception-guided physical examination module that uses live video to detect visual cues (e.g., swelling, asymmetry, gait abnormalities) and generates step-by-step instructions for patients to perform self-examinations, improving physical exam guidance scores from 39% to 72% over human baselines.

3. Develop a differential diagnosis ranking system that outputs a ranked list of possible conditions, achieving 91% top-1 accuracy in matching reference diagnoses—14 percentage points higher than board-certified physicians.

4. Create a modality-adaptive communication interface that automatically switches between video and text based on user preference and task complexity, improving user-perceived effectiveness (89% vs 79%) and convenience (88% vs 71%) compared to text-only systems.

5. Integrate a real-time audio-visual perception module for detecting subtle physical signs (e.g., tremors, nystagmus, affective state) with explicit confidence scoring, flagging low-confidence observations for human verification to avoid over-reliance.

6. Build a clinical reasoning rubric optimizer that trains the system to score ≥90% on structured clinical reasoning tasks by decomposing each case into history-taking, diagnosis, management, and treatment planning sub-tasks with separate optimization targets.

7. Add an asynchronous turn-taking orchestrator that pre-computes likely responses during user speech, enabling near-instantaneous replies without sacrificing depth of analysis.

What the Improved AI System Can Do:

• Conduct real-time video consultations that match or exceed board-certified physicians in history-taking, diagnosis, clinical reasoning (90% vs 76%), and treatment planning (79% vs 67%).

• Guide patients through physical examinations using live video feedback, achieving 72% accuracy in physical observation tasks (vs 39% for human doctors).

• Reduce consultation latency to under 3 seconds per turn while maintaining expert-level diagnostic accuracy.

• Adapt communication modality (video vs text) in real-time to maximize patient comfort and understanding, improving patient-reported feeling understood scores by 9 percentage points.

• Provide ranked differential diagnoses with 91% accuracy in top-1 matches, supporting clinicians in decision-making.

• Flag uncertain perceptual observations (e.g., tremor detection at 24% accuracy) for human review, preventing false confidence in high-stakes scenarios.

• Operate in asynchronous multi-agent mode to handle complex cases requiring deep reasoning without delaying conversational flow.

Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.