MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction
MedicalBench is a new benchmark testing how well large language models can extract medical concepts that are implied rather than explicitly stated in patient notes. It evaluates reasoning by requiring models to verify concept extraction against sentence-level evidence, focusing on challenging cases like implicit positives and semantic ambiguities. The results show current LLMs struggle with this implicit reasoning, highlighting the need for better medical language models.