MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration
The episode reviews the paper MedMCP-Calc, which benchmarks 23 LLMs on realistic medical calculator tasks using MCP integration. Hosts discuss poor performance, failure modes like tool avoidance, and the fine-tuned CalcMate model that improved scores. They conclude models aren't ready for clinical use but the benchmark offers a path forward.