MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks
cs.CL
Submitted: 2026-10-05
Updated: 2026-10-05
Code: https://github.com/REAL-Lab-NU/MedicalHarness
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
- An Empirical Study of Harness Design for Coding Agents
- HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
- Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
- Quantifying Variance in Evaluation Benchmarks
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- MemGPT: Towards LLMs as Operating Systems
- The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
- MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare
- LatentDx: Latent Multi-Agent Communication for Cross-Hospital Rare-Disease Diagnosis
- Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
- MedLoCoMo: A Long-Context Multi-Session Medical Dialogue Benchmark for Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering