Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline

This paper introduces WarehouseReliabilityBench (WRB), a benchmark of 400 frozen tasks over two synthetic warehouses (ecommerce and SaaS domains), designed to evaluate analytics agents on business answer reliability rather than SQL syntax accuracy. The paper argues that "analytics agents should be evaluated and optimised for reliable business answers rather than SQL syntax accuracy, and that most of the measured reliability gain on this benchmark comes from a deterministic behaviour and verification layer rather than from a better or larger model."

The benchmark spans five task categories: standard, ambiguous, unanswerable, schema-drift, and adversarial questions. Roughly half have no correct SQL: the required behaviour is a clarification, abstention or refusal, with executable ground truth where it exists and a behaviour contract where it does not. The splits are frozen at 240/80/80 and are family-disjoint, meaning every template family appears in exactly one split.

The paper introduces QueryProof, a 7B agent built on Qwen2.5-Coder, described as a deterministic state machine that calls a language model twice and decides nothing on the model's word. The agent's design premise is that the model proposes, rules dispose: it never asks the model what to do, but instead asks two narrow, checkable questions (which metric a phrase means and what SQL computes it), making every behavioural decision itself against a semantic layer and physical catalog. Every candidate passes two gates: static validation (a sandbox allowlist over the parsed AST, plus checks for unknown tables, retired columns, and forbidden joins) and post-execution verification (checking filters, period, non-empty results, and impossible values).

The main results on the 80-task test split, evaluated once, show QueryProof reaching a Business Truth Rate of 0.537 (routed) and 0.562 (unrouted), against 0.300 for a direct-prompted 32B baseline and 0.275 for a cost-comparable few-shot baseline. The paper notes: The baselines execute valid SQL and still land between 0.275 and 0.300, which is the gap that a syntax-oriented metric does not see.

False Success Rate separates the systems further: QueryProof returns 13 incorrect answers out of 37 returned answers (0.351), against 0.754 for the 32B model and 0.797 to 1.000 across the 7B baselines. QueryProof answers 0.600 of tasks and is correct on 1.000 of the answerable questions it chose to answer; the baselines answer 1.000 and are correct on 0.350.

The paired comparisons show QueryProof outperforming the 32B model by +0.237 [+0.112, +0.375] on task success, and the few-shot baseline by +0.262 [+0.137, +0.400]. Cost per correct answer is 71.0% lower than the 32B model (−0.00414 [−0.00752, −0.00250]), but the difference against the few-shot baseline is not resolved (−0.00012 [−0.00142, +0.00078] spans zero).

The paper reports that the separation comes almost entirely from the strata where the correct response is not an answer. On the 16 unanswerable questions, QueryProof scores 0.438 and every baseline scores 0.000; on the 8 schema-drift questions it scores 1.000 against 0.000. The 8 adversarial questions do not discriminate (every system scores 1.000).

Two components did not work and are reported as negative results. First, routing did not transfer: on validation, routing helped (0.738 against 0.725), but on test the ordering reverses (0.537 against 0.562). The routed configuration over-abstains on test. Second, the fitted confidence model lost to the heuristic it replaced: AURC 0.436 against 0.164, ECE 0.337 against 0.189, Brier 0.336 against 0.253. The paper explains: The model was fitted on dev answers to answerable questions only, so it never learned to doubt an answer given where a clarification was required, which is exactly what the test split punishes.

Post-training was declined by a pre-registered gate: a threshold of 100 verified training examples of recurring errors was required, but after fixing deterministic defects the pool was only 20. The paper states: The verification layer had already absorbed the errors that post-training would have targeted. This is reported as untested by design, with the evidence, and not as a negative result.

The paper includes two serious disclosures about the test split. First, a contaminated lexicon was found: five phrases in the agent's hand-authored synonym lexicon matched the entire test ambiguous stratum and can only have been written from the held-out split. These were removed before any test-split model call, but the paper acknowledges: "Their existence proves that test content reached the author during development... The test split should therefore be read as a single frozen evaluation with disclosed prior exposure, not as an uncontaminated confirmatory test."

Second, the test ambiguous stratum has only two families, in one domain, and never exercises the agent's primary ambiguity mechanism (ambiguity between two defined metrics, rather than undefined concepts).

The paper also notes that resampling template families rather than tasks widens both accuracy intervals to include zero: +0.237 becomes [−0.125, +0.562] against the 32B baseline, and +0.262 becomes [−0.075, +0.575] against few-shot. The paper concludes: the ordering is better established than the magnitude.

The limitations section lists several threats: the test split was exposed before the run; the headline comparison is confounded (QueryProof and the 32B baseline differ in parameter count and scaffolding); the evidence base is narrow (400 tasks, two synthetic domains, one model family, one SQL dialect); labels rest on one reviewer (test–retest agreement 0.920, Cohen's κ 0.887, but not inter-annotator agreement); and the statistics are weaker than the intervals suggest.

The paper's conclusion states: What holds is the deterministic part... What did not transfer is the learned part. The practical implication is that "the measured gain is consistent with the deterministic components being its main source: deciding behaviour by rule against a semantic layer and a physical catalog, and checking results after execution. Neither requires a weight update, a larger model, or a per-request model judgement that cannot be audited, although both require the semantic layer to be maintained."

Improvements to AI Systems Based on This Paper

1. Add a deterministic verification layer to analytics agents.

• After the model generates SQL, statically validate the parsed AST against an allowlist (tables, columns, joins) and dynamically check post-execution outputs (filters, period, non-empty results, impossible values). This catches errors the model cannot self-correct, reducing false successes from 0.75 to 0.35 in the paper.

• The improved system can reject invalid or risky queries before returning answers, and flag suspicious results (e.g., zero rows, out-of-range values) for human review.

2. Replace model self-judgement with rule-based decision-making for task type.

• Instead of asking the model should I answer or clarify?, use a semantic layer and physical catalog to decide: if a metric is undefined, a column is retired, or a join is forbidden, force clarification or abstention. This is what enabled 100% correctness on schema-drift and 0.438 on unanswerable questions, where baselines scored 0.000.

• The improved system can reliably distinguish answerable from unanswerable queries, and refuse or ask for clarification when data is missing, ambiguous, or drifted—without relying on the model's confidence.

3. Train on unanswerable and ambiguous examples explicitly.

• The paper shows a fitted confidence model failed because it was trained only on answerable questions, so it never learned to doubt answers when clarification was needed. Include a balanced training set with unanswerable, ambiguous, and schema-drift cases, and evaluate on those strata separately.

• The improved system can calibrate its uncertainty across all task types, not just well-formed ones, reducing overconfident wrong answers.

4. Use a two-call architecture with narrow, checkable sub-questions.

• Instead of one open-ended generation, have the model answer two narrow questions: (a) which metric does the phrase refer to, and (b) what SQL computes it. Then combine those answers via rules. This makes each model call auditable and reduces hallucination.

• The improved system can decompose complex queries into verifiable components, making it easier to trace errors and maintain correctness.

5. Implement a propose-then-verify pipeline with a sandbox.

• Let the model propose SQL, but never let it decide the final action. Run every candidate through static and dynamic gates before returning. This is the core of QueryProof's success and requires no larger model.

• The improved system can safely execute untrusted model-generated SQL in a controlled environment, preventing harmful or invalid queries from reaching the user.

6. Add a post-execution consistency check for business logic.

• Verify that the result matches the intended period, filters, and metric definitions from the semantic layer. If not, re-route or abstain. This catches silent errors that syntax metrics miss.

• The improved system can detect when a query works but answers the wrong business question, and correct or refuse before presenting results.

7. Design for abstention as a first-class output.

• The paper's biggest gains came from correctly refusing or clarifying on ~half the tasks. Build agents that can explicitly return unanswerable or needs clarification as valid outputs, with a confidence threshold for when to abstain.

• The improved system can avoid fabricating answers when data is insufficient, ambiguous, or contradictory, improving trust and reliability in real-world deployments.

8. Use a semantic layer and physical catalog as external memory.

• Maintain a machine-readable catalog of metrics, tables, columns, and relationships. The agent should consult this catalog for every query, not rely on the model's parametric memory. This is what enabled handling schema drift and retired columns.

• The improved system can adapt to schema changes without retraining, and can explain its decisions by referencing the catalog.

9. Report and optimize for Business Truth Rate and False Success Rate, not SQL accuracy.

• Shift evaluation metrics to measure whether the final business answer is correct, including abstentions and clarifications. This exposes gaps that syntax metrics hide.

• The improved system can be tuned to maximize correct answers while minimizing false successes, which is what users actually care about.

10. Pre-register a gate for post-training, and only train on verified recurring errors.

• Before fine-tuning, collect a threshold of verified error examples. If deterministic fixes absorb the errors, skip training. This avoids overfitting and wasted compute.

• The improved system can decide when to update its weights based on evidence, rather than automatically training on noisy data.

11. Use family-disjoint splits and disclose contamination.

• When evaluating, ensure template families don't appear across splits, and audit lexicons and prompts for accidental leakage. Report prior exposure honestly, as the paper does.

• The improved system can provide more trustworthy evaluation results, and users can interpret scores with known caveats.

12. Add a no-answer mode for adversarial or unanswerable inputs.

• The paper shows adversarial questions didn't discriminate, but unanswerable ones did. Build a classifier that detects when a question has no valid SQL answer and routes to a clarification or refusal template.

• The improved system can handle edge cases gracefully, reducing hallucination and improving user trust.

LLM analytics agents are evaluated on SQL syntax accuracy, but production failures look different: questions with two valid business definitions, questions the warehouse cannot answer, deprecated columns after a schema change, and queries that execute successfully while returning the wrong business number. No execution-match metric can score them. This paper introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal, with pinned denominators and a pre-registered paired bootstrap fixing each claim verb before the numbers existed. QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour, and gates every answer on deterministic post-execution checks. On an 80-task synthetic test split evaluated once, QueryProof outperforms a direct-prompted 32B baseline by +0.237 [+0.112, +0.375] Business Truth Rate at 71.0% lower cost per correct answer; against a cost-matched few-shot baseline the accuracy gain holds but the cost difference does not resolve. This compares systems rather than model sizes: the 32B baseline receives none of the scaffolding. False success falls from 0.754 to 0.351 of returned answers, and no wrong number was returned on an answerable task (0 of 24), though 13 answers went to questions requiring clarification or abstention. Removing the routing layer changes little (0.562 against 0.537), so the result does not depend on escalation. Routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced. Resampling template families rather than tasks widens both accuracy intervals to include zero, so the effect's direction is better supported than its magnitude. The gain tracks the deterministic layer, though no component ablation was run.