Evaluation format, not model capability, drives measured triage failure in the assessment of consumer health AI
cs.HC, cs.AI
Submitted: 2026-03-12
Updated: 2026-10-02
Comments: 10 pages
Code: https://github.com/dafraile/nature_experiments_triage
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Large Language Models Often Know When They Are Being Evaluated
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
- Large Language Models Are Not Robust Multiple Choice Selectors
- Where Do Apparent LLM Clinical Triage Failures Arise? Localizing the Multiple-Choice Format Effect
Related papers
- Calibrating LLM Explanations to Human Decisions with Reinforcement Learning
- EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning
- EvoDesign: Agentic Editable Diagram Creation via Design Expertise Evolution
- HAGI++: Head-Assisted Gaze Imputation and Generation
- Explanation User Interfaces: A Systematic Literature Review
- Hybrid Cinematography: Previsualizing and Managing Hallucination Risk in Generative Video Reshooting