VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations
Ivan Kartáč, Jan Tovarys, Mateusz Lango, Ondřej Dušek
cs.CL
Submitted: 2026-08-04
Code: https://github.com/PrimVet/VetScore
License: http://creativecommons.org/licenses/by/4.0/
The gist: Citation excerpts can be used to increase the reliability of generated outputs and their faithfulness to cited sources, which is especially important in high-stakes domains such as human and
Terminology
Abstract
Citation excerpts can be used to increase the reliability of generated outputs and their faithfulness to cited sources, which is especially important in high-stakes domains such as human and veterinary medicine. However, this does not guarantee that generated claims are faithful to the provided excerpts. We present VetScore, a multi-step evaluation method for veterinary long-form question answering, designed to assess how well are generated claims supported by the provided excerpts, weighing this information by each claim's harm potential. VetScore first segments the output and decomposes it into individual claims, then scores each claim with respect to its harm potential and evaluates its faithfulness to source excerpts, and finally calculates the overall risk-adjusted score. We collect an expert-annotated meta-evaluation dataset, evaluate our approach with a range of judge models, and show that it achieves high correlations with veterinary experts even with small judge models, while offering explainability across multiple dimensions.
Sources
- Fine-tuning foundational models to code diagnoses from veterinary health records
- MedScore: Generalizable Factuality Evaluation of Free-Form Medical Answers by Domain-adapted Claim Decomposition and Verification
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning
- WebGPT: Browser-assisted question-answering with human feedback
- Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
- Human Trust in AI Search: A Large-Scale Experiment
- Teaching language models to support answers with verified quotes
- Towards Conversational Medical AI with Eyes, Ears and a Voice
- Correctness is not Faithfulness in RAG Attributions
- All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- Gemma 4 Technical Report
- First, do NOHARM: a medical safety benchmark and randomized study of physician and AI teaming on clinical consultations
- Classification performance and reproducibility of GPT-4 omni for information extraction from veterinary electronic health records
- CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering