RAGWarrant: Evidence-Preserving Governance for RAG Policy Promotion Under Quality, Cost, Latency, and Risk Constraints
cs.AI, cs.CL, cs.LG
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: 19 pages, 6 figures, 7 tables. Preprint v0.1.1-rc1. Code and artifacts: https://github.com/RAGWarrant/ragwarrant-governance
Code: https://github.com/RAGWarrant/ragwarrant-governance
License: http://creativecommons.org/licenses/by/4.0/
The gist: Retrieval-augmented generation systems are extensively instrumented with metrics, benchmarks, traces, and automated judges, but these tools do not decide whether a proposed policy change is safe to
Terminology
Abstract
Retrieval-augmented generation systems are extensively instrumented with metrics, benchmarks, traces, and automated judges, but these tools do not decide whether a proposed policy change is safe to release. We present RAGWarrant, an open-source promotion-control framework that treats deployment as a constrained evidence decision rather than a leaderboard choice. RAGWarrant normalizes evaluator outputs and operational telemetry, applies predeclared quality and hard-risk gates, assigns evidence-class claim ceilings, preserves negative outcomes, and emits auditable PROMOTE, BLOCK, REJECT, or INCONCLUSIVE decisions. We evaluate the framework across T2-RAGBench, MultiHop-RAG, CRAG, HotpotQA, synthetic reproduction, and bounded local generative experiments. On HotpotQA, operational savings were blocked because answer quality fell beyond the declared margin. A bounded CRAG study selected a lower-cost quality-tied policy, but related generative gains were unstable and a held-out guardrail failed closed. We claim an auditable promotion-control abstraction, not optimizer superiority, human validation, or production readiness. The tagged artifact reproduces from a fresh clone, runs as a hardened Docker job, accepts external evaluator exports, and verifies artifact integrity.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Ragas: Automated Evaluation of Retrieval Augmented Generation
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
- RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
- RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
- CRAG -- Comprehensive RAG Benchmark
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Benchmarking Large Language Models in Retrieval-Augmented Generation
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems