Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

The gist The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA.

Objective

To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA).

Method Overview

The method is designed as a multi-agent peer-reviewed reasoning method where multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, and then act as peer reviewers to evaluate each other’s reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected to produce the final answer. Experiments were conducted with five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, GPT-oss-20B) on three benchmark datasets: HeadQA, MedQA(USMLE), and PubMedQA. Performance was compared against single-model chain-of-thought reasoning and chainof-thought based majority voting.

MedQA Task and Datasets

The evaluation adopts the MedQA task, which serves as a widely recognized benchmark for assessing the reasoning ability of large language models in the biomedical domain. Three representative datasets used are HeadQA, MedQA(USMLE), and PubMedQA. HeadQA is constructed from official Spanish healthcare professional exams, covering diverse domains such as medicine, nursing, psychology, and pharmacy. MedQA(USMLE) is derived from the United States Medical Licensing Examination and is widely regarded as one of the most challenging benchmarks in biomedical NLP. PubMedQA is built from biomedical research articles in PubMed, where the task is to answer yes/no/maybe research questions based on scientific abstracts. Together, these datasets provide a comprehensive and complementary evaluation setting.

Multi-Agent Peer-Review Reasoning Method

The method consists of two phases: (1) multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, and (2) agents act as peer reviewers, scoring each other’s reasoning for factual correctness and logical soundness. This approach is inspired by the peer-review process of academic papers where peer-reviewed work is deemed worthy of publication. The system aggregates these peer reviews to identify the most credible and coherent reasoning chain, which is subsequently used to generate the final answer.

Phase 1 involves eliciting explicit, structured reasoning from multiple LLMs before revealing final answers. For multiple-choice medical QA (e.g., MedQA, HeadQA), prompts require option-by-option analysis with stepwise justification. For literaturegrounded QA (e.g., PubMedQA), prompts instruct models to examine the abstract, extract supporting evidence, and form a logically consistent conclusion.

Phase 2 involves LLMs evaluating the responses produced by other models using a 6-point rubric (0–5) that jointly assesses answer correctness and reasoning quality. Judges do not access groundtruth labels; they rely on domain knowledge and internal reasoning, yielding an assessment of understanding rather than pattern matching. Scores are aggregated (e.g., by mean or median) to select the top peer-reviewed reasoning and adopt its answer as final.

Results

Peer-reviewed reasoning consistently outperformed both individual models and majority voting. The best model combination achieved an average accuracy of 0.820 across datasets, exceeding the strongest single model (0.777) and majority voting ensembles (up to 0.789). Peer-reviewed reasoning consistently outperformed both individual models and majority voting. Across all datasets, accuracy improvements were observed regardless of the specific model combinations.

Reasoning Quality Assessment

The peer-review process successfully distinguished reasoning quality across models. High-performing models such as GPT-oss-20B (mean:4.1, std:1.2) and Phi-4 (mean:4.4, std:0.9) generally received higher ratings, while weaker models like DeepSeekLLM-7B (mean:3.2, std:1.6) were consistently rated lower. A strong positive Pearson correlation (r=0.91, p=0.034) was observed between the mean peer-review score of each model and its single-model accuracy, indicating that models receiving higher peer-evaluated reasoning scores tend to achieve higher answer accuracy.

Discussion

The observed improvements can be attributed to the method’s focus on reasoning quality rather than final answers alone. Peer-reviewed reasoning addresses the limitation where majority voting discards valuable reasoning traces by incorporating crossmodel evaluation of the reasoning process itself. This mechanism mirrors the human peer-review process, in which arguments are evaluated not only by outcomes but also by their underlying validity. Furthermore, our results highlight that peer-reviewed reasoning scales effectively with the number of participating models.

Conclusion

In this work, we proposed a multi-agent peer-reviewed reasoning method for medical question answering. By enabling large language models to act as both solvers and reviewers, the method consistently improved accuracy over chain-of-thought reasoning and majority voting across three benchmark datasets. Beyond higher accuracy, peer-reviewed reasoning enhanced interpretability and robustness by explicitly selecting reasoning chains of higher logical quality.

Runtime Analysis

The total runtime for the proposed method is approximately doubling that of the Majority Vote method, even when utilizing high parallelism for the review sub-tasks.

Data Availability

The three datasets used are all public: HeadQA: https://aghie.github.io/head-qa/, MedQA(USMLE): https://github.com/jind11/MedQA, PubMedQA: https://pubmedqa.github.io/.

Code Availability

The code is available at https://github.com/Learner4everrr/Multiagent-peer-reviewed-framework-for-MedQA.

Acknowledgements

We would like to express our sincere gratitude to the reviewers for their valuable comments.

Funding

This work was supported by the National Institutes of Health’s National Center for Complementary and Integrative Health (grant numbers R01AT009457 and U01AT012871), the National Institute on Aging (grant number R01AG078154), the National Cancer Institute (grant number R01CA287413), the National Institute of Diabetes and Digestive and Kidney Diseases (grant number R01DK115629), and the National Institute on Minority Health and Health Disparities (grant number 1R21MD019134-01).

References

The references cited are listed in the provided bibliography. The reference list includes works such as Benchmarking large language models for biomedical natural language processing applications and recommendations in Nature communications, 2025 and Chain-of-thought prompting elicits reasoning in large language models in Advances in neural information processing systems, 2022. The reference list also includes works on LLM-as-a-judge with mt-bench and chatbot arena in Advances in neural information processing systems, 2023.

--- Page 1 ---

The gist The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA.

1. Improve accuracy, interpretability, and robustness in MedQA by adopting a multi-agent peer-reviewed reasoning method that enables LLMs to act as both solvers and evaluators. This system can produce answers that are more accurate, interpretable, and robust in biomedical contexts by selecting the highest-rated reasoning chain based on cross-model evaluations.

2. Enhance robustness against flawed reasoning chains by incorporating a mechanism where the evaluative stage explicitly rewards coherent, medically sound reasoning chains. This filters out low-quality outputs because LLM judges reliably differentiated strong from weak rationales, leading to an output that is more accurate and more trustworthy for downstream use.

3. Improve interpretability by ensuring that the selected answer is supported by the most credible reasoning available in the agent pool, as opposed to simple majority voting where agreement does not guarantee correctness. This provides reasoning traces which are preserved rather than discarded, allowing clinicians and researchers to inspect the logical path taken.

4. Improve scalability by leveraging the synergistic strengths of diverse models; the method shows that performance gains are most pronounced as additional agents were introduced, suggesting that differences in model outputs and reasoning tendencies across agents provide complementary strengths. This allows for building stronger ensemble-style methods that avoid simple majority bias.

5. Develop a robust evaluation framework by making the system resilient to prompt variations, as demonstrated by the finding that the performance of our framework is relatively consistent under different judge prompt designs, indicating a degree of robustness to prompt variations. This ensures that the system's reliability is not overly dependent on specific prompting techniques.