Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
cs.CL, cs.LG
Submitted: 2026-09-09
Updated: 2026-09-09
Code: https://github.com/JamesL404/synca-audit
License: http://creativecommons.org/licenses/by/4.0/
The gist: Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured.
Terminology
Abstract
Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked ours, the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen's kappa = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, over-flagging 77.1% of the human-FAITHFUL cases in the enriched set. Most of its over-flags trace back to a single mechanism we call GRADE-HALLUCINATION. A self-hosted Qwen3.6-27B replacement (kappa = 0.72) lands in the same range as Claude Opus 4.7 (kappa = 0.71); the head-to-head is underpowered at n = 96, but for the deployment decision that hardly matters, since Qwen costs roughly 1/300 as much per call. Ensembling does not help for free. Pairing the weak judge with a stronger one degrades agreement, whereas three strong judges under unanimity routing reach kappa = 0.79 at 89.7% auto-coverage. Applied out-of-domain, the same audit recipe flags 25.5% of BIRD-financial's expert-authored gold SQLs as candidate gold-SQL issues under our annotation protocol. Code and pre-registration are at https://github.com/JamesL404/synca-audit.
Sources
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation
- Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect
- Can LLM be a Personalized Judge?
- Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLP
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
- A Survey on LLM-as-a-Judge
- Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
- Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
- No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
- Auto-Prompt Ensemble for LLM Judge
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
- Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
- Semantic Agreement Enables Efficient Open-Ended LLM Cascades
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- Large Language Models are not Fair Evaluators
- Self-Instruct: Aligning Language Models with Self-Generated Instructions
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering