RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection
cs.CL, cs.AI, cs.LG
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: 49 pages, 23 tables, 10 figures. Code and data: https://github.com/eOnofri04/raim-analysis and https://github.com/eOnofri04/raim-verdicts
Code: https://github.com/eOnofri04/raim-verdicts
Project page: https://llm-aggrefact.github.io
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models, costly and ill-suited to
Terminology
Abstract
Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models, costly and ill-suited to high-throughput monitoring. We investigate whether a panel of cheap open-weight judges (4--9B) can be aggregated to stand in for a frontier one, what the substitution sacrifices, and when it is worth making. We propose RAIM, an aggregation scheme robust to the members' correlated errors, coupling a cross-fitted stacked logistic regression with an admissibility test that, read from the members' own outputs, identifies when aggregating them improves on their best member and stays within reach of the frontier judge. We instantiate RAIM with ten judges from disjoint families across eight faithfulness benchmarks. Against Claude Sonnet, the panel retains a median 93% of its Cohen's κ and gives up only 2.9 points of balanced accuracy on average; read as paired differences, it clearly improves on one benchmark and clearly worsens on three (only two by a non-negligible margin), leaving four unresolved. At a sixty-fourth of the frontier's inference price, the operative expense is a one-time in-domain calibration on 50--100 labelled records. The panel is also competitive with purpose-trained detectors on their home benchmarks (within 1.3 accuracy points of GPT-4o and 1.9 of the LLM-AggreFact leader), and beats the strongest one we reran by 6 points on our grounded sets. Whether aggregation pays depends on the members themselves: where several capable members err on different items, the panel improves on its best judge and approaches the frontier; where one dominates, the stacker recovers the leader, and only there does the frontier remain materially ahead. Both conditions are read off the calibration set at no further cost, so a cheap panel can stand in for a frontier one wherever this audit admits it.
Sources
- Yi: Open Foundation Models by 01.AI
- Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Lift
- Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- A comprehensive taxonomy of hallucinations in Large Language Models
- Gemma 2: Improving Open Language Models at a Practical Size
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- Great Models Think Alike and this Undermines AI Oversight
- Cost-Effective Automated Judging of Natural-Language Mathematical Proofs
- Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
- Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure
- Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
- Unsupervised Ensemble Learning with Dependent Classifiers
- Mistral 7B
- Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
- Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
- Correlated Errors in Large Language Models
- Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
- Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
- LettuceDetect: A Hallucination Detection Framework for RAG Applications
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering