LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

arXiv:2602.01779 · cs.AI · Submitted 2026-02-02 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine".

Jane: The LingLanMiDian benchmark introduces a large-scale, expert-curated, multi-task suite designed to unify the evaluation of Large Language Models (LLMs) across Traditional Chinese Medicine (TCM) knowledge and clinical reasoning.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we know the name is LingLanMedBench, and it’s designed to be a comprehensive tool for seeing where current AI models stand in this specific medical domain.

Jane: The title itself says it’s a unified benchmark, which means they aren't just throwing random tests at the models; they are trying to create one single way to measure performance across different tasks.

Lu: They are focused on bridging the gap between fundamental knowledge and applied reasoning by creating this standardized evaluation framework for TCM LLMs.

Meng: The implication here is that we finally have a way to compare different model families fairly when we look at complex medical tasks, not just simple recall scores.

Lalam: It establishes a unified, quantitative, and extensible foundation for advancing TCM LLMs and domain-specific medical AI research.

The paper's summary: Tom: The paper summarizes that they’ve created this framework to evaluate LLMs not just on facts, but on their ability to simulate human-like diagnostic and therapeutic judgment.

Jane: They’re moving beyond just checking recall accuracy and focusing on how well the AI can handle things like multi-hop synthesis or composing a full prescription.

Lu: They introduced a comprehensive suite spanning five domains and thirteen subtasks, which is designed to capture different dimensions of model capability, from simple factual recall to complex clinical reasoning <ref:2602.01779#pg1>.

Meng: They also built this unified metric suite where you get task-aligned scoring, meaning the way you score a list task is different from how you score a single-choice question.

Lalam: They use metrics like character-level F1 for short text generation and Cosine similarity with MAE for quantitative dosage estimation, which gives them a consistent way to score diverse outputs.

The paper's improvements: Tom: One of the big contributions they highlight is the creation of difficulty-calibrated hard subsets, called LingLan-Hard, to probe fine-grained knowledge and compositional robustness.

Jane: This subset allows them to evaluate performance under more challenging conditions where the degradation between models and human experts in TCM reasoning becomes much clearer.

Lu: By selecting the top four hundred highest-difficulty items from each task, they create LingLan-Hard, which covers thirteen tasks and has five thousand two hundred samples in total to really challenge the systems.

Meng: From a deployment standpoint, this is important because it exposes actionable failure modes that you can actually work on when you’re trying to ship a model that’s supposed to be reliable.

Lalam: The authors also introduced systematic cross-model baselines specifically designed to expose these failure modes in option calibration, which gives researchers concrete data on where the models fall short.

Conclusion: Tom: So, when we wrap up this discussion on LingLanMedBench, the main finding is that while models perform near-ceiling on licensing-style recall, there are still clear deficits in multi-hop synthesis and dosage proportionality.

Jane: This points to a real gap between surface knowledge and the deep reasoning that human experts apply when making clinical decisions in TCM.

Lu: They conclude that closing this gap depends on models that integrate clinical structure with calibrated, domain-aware reasoning, rather than just focusing on factual recall >

Meng: From what I see, the practical path forward involves using candidate generation and re-ranking pipelines to tackle those multi-step reasoning gaps.

Lalam: And they emphasize that reframing open-ended decisions as recognition tasks substantially narrows those gaps because hypothesis-space size and calibration strongly influence performance >

Rui Hua, Yu Wei, Zixin Shu, Kai Chang, Dengying Yan, Jianan Xia, Zeyu Liu

Department of Computer Science and Technology, Beijing Jiaotong University · Tianjin Tasly Digital Chinese Medicine Technology Co. Ltd. · Tasly Biopharmaceuticals Co. Ltd. · State Key Laboratory of Chinese Medicine Modernization, Tianjin · Institute of Liver Diseases, Hubei Key Laboratory of the theory and application research of liver and kidney in traditional Chinese medicine, Hubei Provincial Hospital of Traditional Chinese Medicine · Affiliated Hospital of Hubei University of Chinese Medicine · Hubei Province Academy of Traditional Chinese Medicine · Department of Gastroenterology, Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing · China Institute for History of Medicine and Medical Literature, China Academy of Chinese Medical Sciences, Beijing · Beijing Research Institute of Chinese Medicine, Beijing University of Chinese Medicine · Beijing University of Chinese Medicine Third Affiliated Hospital · School of Traditional Chinese Medicine, Beijing University of Chinese Medicine · The First Affiliated Hospital, Henan University of Chinese Medicine · Xiyuan Hospital, China Academy of Chinese Medical Sciences, Beijing · Da’an Health Technology (Beijing) Co. Ltd. · Institute of Chinese Medicine Epidemic Disease, Beijing University of Chinese Medicine · Beijing Zhongtengbaimai Medical Technology Co. Ltd. · Sinosoft Company Limited

cs.AI

Submitted: 2026-02-02

Updated: 2026-10-14

Code: https://github.com/TCMAI-BJTU/LingLan

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The LingLanMiDian benchmark introduces a large-scale, expert-curated, multi-task suite designed to unify the evaluation of Large Language Models (LLMs) across Traditional Chinese Medicine (TCM)

Key concepts

LingLanMedBench: This is a comprehensive, standardized test suite for Large Language Models in TCM. It unifies many different tasks—from simple knowledge recall to complex clinical decision-making—into one consistent framework. This allows researchers to compare models fairly across various aspects of TCM expertise.
Multi-task Suite: The benchmark includes 13 subtasks spanning five major domains, such as licensing exams and diagnostic reasoning. This structure ensures that models are tested not just on one skill, but on a wide range of capabilities required for real-world TCM applications, providing a holistic view of performance.
Hard Subset: This is a high-difficulty subset of the benchmark created by selecting the most challenging items for each task. By testing models specifically on these harder problems, researchers can better assess their advanced reasoning capabilities and identify where models consistently fail under pressure.

Terminology

Benchmark — 3× in this paper · explained in 2 episodes
Traditional Chinese Medicine — 2× in this paper
Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
Multi-hop Reasoning — 1× in this paper · explained in 3 episodes
Ontology — 1× in this paper · explained in 2 episodes

Summary

The LingLanMiDian benchmark introduces a large-scale, expert-curated, multi-task suite designed to unify the evaluation of Large Language Models (LLMs) across Traditional Chinese Medicine (TCM) knowledge and clinical reasoning. This benchmark establishes a consistent metric design and a synonym-tolerant protocol for clinical labels to provide a unified perspective on LLMs' strengths and limitations in TCM commonsense knowledge understanding, reasoning, and clinical decision support

How it works

The LingLan benchmark is designed to systematically evaluate the performance of LLMs across the full spectrum of TCM knowledge and reasoning by unifying heterogeneous evaluation tasks under a consistent and quantitative framework. It comprises 13 datasets spanning five domains: Licensing Examination, Fundamental TCM Knowledge, Chinese Patent Medicine, Information Extraction, and Diagnostic and Therapeutic Decision-Making. In total, the benchmark contains 25,624 expert-verified items with approximately 2,000 samples per dataset and an additional 400-item Hard subset in each domain to assess advanced reasoning capability.

Data Sources and Curation

The benchmark incorporates five major domains of TCM knowledge. The data sources include:

TCM Licensing Examination (TLE): Questions were collected from the official National TCM Qualification Examination (2000–2016), covering a comprehensive range of medical subjects including Fundamentals of TCM Theory, Diagnostics of TCM, Chinese Materia Medica, Formulary Science, Internal Medicine of TCM, Surgery of TCM, Gynecology of TCM, Pediatrics of TCM, and Acupuncture and Moxibustion.
Fundamental TCM Knowledge (FTK): This dataset is grounded in a self-constructed TCM knowledge graph assembled from authoritative sources including canonical classical texts, contemporary peer-reviewed TCM/biomedical literature, and nationally adopted teaching materials.
Chinese Patent Medicine Knowledge (CPMK): This dataset is built from structured knowledge extracted from the Pharmacopoeia of the People’s Republic of China and package inserts of Chinese patent medicines.
Information Extraction (IE): The information extraction datasets draw from two sources: classical TCM literature written in Literary Chinese and real-world clinical medical records composed in modern vernacular Chinese.
Diagnostic and Therapeutic Decision-Making: These datasets are built from more than 13,000 clinical case records authored by over 300 renowned TCM physicians covering 7,595 syndromes.

Task Taxonomy and Metrics

LingLan establishes a unified taxonomy encompassing 13 subtasks across five major domains with a total of 44 quantitative evaluation dimensions. Each subtask is designed to capture distinct dimensions of model capability ranging from factual recall and symbolic association to structured extraction and clinical reasoning. The unified metric matrix ensures consistency across diverse task types, employing metrics like Accuracy for single-choice tasks, instance-level or list-level Precision/Recall/F1 for structured and multi-label tasks, character-level F1 for short-text generation, and Cosine similarity together with MAE for quantitative dosage estimation.

Hard Subset Construction

To evaluate model performance under more challenging conditions, a high-difficulty subset named LingLan-Hard is constructed. Item difficulty is estimated from per-instance scores across 14 models using the composite difficulty score D = (1 − µ) + λσ, where λ = 0.5. The top 400 highest-difficulty items from each task are selected, resulting in LingLan-Hard covering 13 tasks and containing 5,200 samples.

Experimental Results and Findings

Under a unified zero-shot protocol, results show near-ceiling performance on licensing-style recall but persistent deficits in multi-hop synthesis, prescription composition, and dosage proportionality. Overall averages on the full sets cluster in the high 50s (for example, DeepSeek-R1 51.1), while the hard subsets lower the averages to approximately 20, indicating a consistent degradation under increased difficulty. The results suggest that closing the gap will depend on models that integrate clinical structure with calibrated, domain-aware reasoning.

Conclusion

LingLan establishes a unified and interpretable evaluation framework that ensures consistency across diverse task types in TCM. Accuracy is used for discrete-choice problems, instance-level or list-level Precision/Recall/F1 for structured and multi-label tasks, character-level F1 for short-text generation, and Cosine similarity together with MAE for quantitative dosage estimation. The results indicate near-ceiling performance on licensing-style knowledge tasks but information extraction on classical TCM texts remains a pronounced weakness. Despite persistent gaps in multi-step clinical reasoning, reframing open-ended decisions as recognition tasks substantially narrows these gaps, indicating that hypothesis-space size and calibration strongly influence performance. The results suggest that closing the gap will depend on models that integrate clinical structure with calibrated, domain-aware reasoning, moving beyond factual recall toward clinically faithful decision-making >.

Improvements for AI systems

Improve model robustness against difficulty using LingLan-Hard subsets to probe fine-grained knowledge, distractor resistance, and compositional robustness. This allows systems to be evaluated on more challenging conditions where performance degradation is more clearly identified across tasks.
Implement a unified metric framework to ensure consistent, quantitative, and interpretable evaluation across heterogeneous task types. This standardization allows for fair comparison among classification, structured prediction, and regression subtasks, moving beyond subjective scoring.
Enhance clinical decision support by reframing open-ended problems as recognition tasks to substantially narrow these gaps in multi-step reasoning. This suggests a practical route for improvement is through candidate generation and re-ranking pipelines.
Improve dosage estimation accuracy by employing flexible alignment strategies, such as inclusion-based matching (as used in DTR) and character-level F1 matching, to reduce the impact of superficial naming variation while maintaining one-to-one herb correspondence.
Develop genre-aware tokenization and prompting for classical Chinese texts to improve information extraction robustness, addressing the pronounced weakness observed on these corpora due to archaic vocabulary, elliptical syntax, and long-distance dependencies.
Refine reasoning capabilities by incorporating structure-aware decoding that enforces compatibility among syndrome, treatment, and prescription, which is suggested as a method for improving performance in the DTR task.

Abstract

Reliable evaluation of large language models in traditional Chinese medicine remains limited because existing benchmarks mainly focus on examination-style question answering or isolated generative tasks. We developed LingLanMedBench, a unified benchmark comprising 25,620 expert-verified items across 13 subtasks and five domains, covering knowledge, language understanding, information extraction, and diagnostic-therapeutic decision-making. LingLanMedBench standardizes task formats and evaluation metrics and further incorporates synonym-tolerant clinical-label matching, empirically derived hard subsets, and decision-recognition reformulations of open-ended clinical tasks. Fourteen open-source and proprietary models were evaluated under a standardized zero-shot protocol. Performance varied substantially across domains, model families, task formulations, and difficulty levels. The leading overall score decreased from 51.1 on the full benchmark to 31.9 on the hard subsets. Prescription performance also differed markedly between open-ended generation, with F1 scores of 32.7 and 20.3, and decision recognition, with accuracies of 90.6 and 56.8 on the full and hard subsets, respectively. These findings reveal a substantial gap between knowledge-oriented performance and clinically faithful reasoning. LingLanMedBench provides a standardized and reproducible framework for characterizing current capabilities, identifying systematic limitations, and tracking meaningful progress in large language models for traditional Chinese medicine.