CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection

arXiv:2611.08367 · cs.DB · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection".

Jane: Here's the gist of the paper. Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611.08367#pg2> The objective is to reduce the total execution cost while meeting a user-specified accuracy target<ref:2611.08367#pg3> CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle<ref:2611.08367#pg3> Pruning redundant reasoning tokens The paper proposes CascadeFilter, which processes each record using an SLM on extracted evidence, then the SLM on the full document if needed, and finally the oracle for unresolved records<ref:2611.08367#pg4> The method formulates cost-minimizing threshold selection for this fixed cascade<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> The search returns the least-cost certified configuration in a pre-specified grid<ref:2611.08367#pg5> How the training signal is built The authors formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> They organize candidates into monotone tracks and apply fixed-sequence testing within each track, allocating the failure budget across tracks<ref:2611.08367#pg6> The certificate controls overall oracle-relative error<ref:2611.08367#pg5> Where the gains come from The method integrates hybrid retrieval, reranking, batched evidence inference, and neighborhood-supported confidence into the cascade<ref:2611.08367#pg6> These components improve the candidate policies evaluated by the certification procedure<ref:2611.08367#pg6> The resulting method couples cost-aware selection with finite-population certification while retaining flexibility in evidence extraction and routing<ref:2611.08367#pg6> Certification and Guarantee The guarantee applies after selection and concerns oracle-relative error ε = 1 − τ<ref:2611.08367#pg5> The search must control the risk of selecting an infeasible configuration<ref:2611.08367#pg6> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611.08367#pg5> The selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search<ref:2611.08367#pg5> Experimental Analysis The experiments evaluate six datasets with 13 settings and study cascade structure, routing components, and calibration behavior<ref:2611.08367#pg6> The main table reports the highest reported benchmark accuracy and macro-averaged F1 score in 12 settings each<ref:2611.08367#pg10> CascadeFilter has the highest reported Accuracy and Macro-F1 in 12 of 13 settings, averaging 0.938/0.887<ref:2611.08367#pg10> The reported operating points describe the reported operating points<ref:2611.08367#pg10> Cost and Performance The reported cost model follows Eq.

Tom: (1), using input-token and invocation counts and mean output lengths<ref:2611.08367#pg5> The token panel counts only oracle tokens<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> Threats to Validity The certificate is, strictly, oracle-relative: the theorems bound P(AccML (ϕˆ) ≥ τ) ≥ 1− δ<ref:2611.08367#pg5> Changing calibration labels to ground truth changes the loss at every stage, including oracle fallback<ref:2611.08367#pg10> The reported violation counts and end-to-end cost accounting still require reconciliation<ref:2611.08367#pg10> The single point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1)<ref:2611.08367#pg5> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10> The violation-count discrepancy and cached-cost accounting still require verification against run logs<ref:2611.08367#pg10> We presented CascadeFilter, which selects two thresholds for a fixed three-stage semantic filter to reduce modeled execution cost under an oracle-relative error constraint<ref:2611.08367#pg10> The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611.08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611.08367#pg10> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10> Future work includes other semantic operators and recalibration under distribution shift<ref:2611.08367#pg10> References

1: L.

Jane: Patel, S.

Tom: Jha, M.

Paper summary: Tom: So, we’re wrapping up on CascadeFilter, which is all about picking two thresholds for a three-stage semantic filter to cut down on execution cost while keeping an accuracy target in mind.

Jane: That sounds like they’ve figured out how to stop wasting compute time when you're dealing with huge amounts of text. So, who wrote this paper?

Lu: It was written by Patel and Jha and Pan and Gupta Asawa Guestrin and Zaharia at the VLDB Endowment in two thousand twenty-five <ref:2611.08367#pg4>.

Meng: Okay, so it’s a specific set of researchers working on document processing with these large language models. What’s the main deal they’re trying to prove?

Tom: They proved that you can optimize the total token cost by jointly searching for those two thresholds instead of just picking them one by one <ref:2611.08367#pg5>.

Jane: So, it’s not just about finding a better single model; it’s about structuring the entire filtering process more intelligently. What does this mean in practice?

Lu: It means they found a way to manage the trade-off between cost and accuracy under an oracle-relative error constraint <ref:2611.08367#pg5>.

Meng: Under an oracle-relative error constraint—that’s just a fancy way of saying they are measuring how much their results might differ from the absolute best possible answer, which is hard to know without using an oracle.

Tom: Exactly. And the result is that they return a candidate configuration that has the least cost while still meeting that error constraint, instead of just finding some random good setting <ref:2611.08367#pg10>.

Jane: That’s interesting because it suggests a systematic search approach rather than just guessing settings and hoping for the best <ref:2611.08367#pg5>.

Lu: Yeah, they found that even though their search space is generally not simple, the error along each path where you fix the first threshold actually goes down as you move to the second one <ref:2611.08367#pg5>.

Meng: But Tom, if we’re shipping this to production, we need a guarantee. What kind of guarantee are they giving us here?

Tom: They give a composable accuracy guarantee where there is at least one minus delta probability that the accuracy will be greater than or equal to tau, which is their chosen error threshold <ref:2611.08367#pg5>.

Jane: So, it’s not just a hopeful claim; they have a mathematical bound on how accurate their system should be based on what they can observe.

Lu: They use this theorem to control the risk of picking an infeasible configuration during their search <ref:2611.08367#pg6>. The selected pair minimizes cost in C, which is their cost model, using all-oracle Algorithm two Certified Joint Threshold Search <ref:2611.08367#pg5>.

Meng: So it’s a certified search mechanism for the best configuration they can find within that specific grid structure. That still sounds like a lot of bookkeeping for deployment.

Tom: It is bookkeeping, but it’s structured bookkeeping, and the cost model itself uses input tokens and invocation counts plus mean output lengths <ref:2611.08367#pg5>. The mean cost they report across all thirteen settings is about two point nine times ten to the negative four per predicate record <ref:2611.08367#pg10>.

Jane: That cost number helps ground it in reality for someone managing a budget on these big processing tasks. It shows that this structured approach

Conclusion: Tom: So, we're wrapping up on CascadeFilter, which is about picking two thresholds for a three-stage semantic filter to cut down on execution cost while keeping an accuracy target in mind.

Jane: That sounds like they’ve figured out how to stop wasting compute time when you're dealing with huge amounts of text. So, who wrote this paper?

Lu: It was written by Patel and Jha and Pan and Gupta Asawa Guestrin and Zaharia at the VLDB Endowment in two thousand twenty-five.

Meng: Okay, so it’s a specific set of researchers working on document processing with these large language models. What’s the main deal they’re trying to prove?

Tom: They proved that you can optimize the total token cost by jointly searching for those two thresholds instead of just picking them one by one.

Jane: So, it’s not just about finding a better single model; it’s about structuring the entire filtering process more intelligently. What does this mean in practice?

Lu: It means they found a way to manage the trade-off between cost and accuracy under an oracle-relative error constraint.

Meng: Under an oracle-relative error constraint—that’s just a fancy way of saying they are measuring how much their results might differ from the absolute best possible answer, which is hard to know without using an oracle.

Tom: Exactly. And the result is that they return a candidate configuration that has the least cost while still meeting that error constraint, instead of just finding some random good setting.

Jane: That’s interesting because it suggests a systematic search approach rather than just guessing settings and hoping for the best.

Lu: Yeah, they found that even though their search space is generally not simple, the error along each path where you fix the first threshold actually goes down as you move to the second one.

Meng: But Tom, if we’re shipping this to production, we need a guarantee. What kind of guarantee are they giving us here?

Tom: They give a composable accuracy guarantee where there is at least one minus delta probability that the accuracy will be greater than or equal to tau, which is their chosen error threshold.

Jane: So, it’s not just a hopeful claim; they have a mathematical bound on how accurate their system should be based on what they can observe.

Lu: They use this theorem to control the risk of picking an infeasible configuration during their search. The selected pair minimizes cost in C, which is their cost model, using all-oracle Algorithm two Certified Joint Threshold Search.

Meng: So it’s a certified search mechanism for the best configuration they can find within that specific grid structure. That still sounds like a lot of bookkeeping for deployment.

Tom: It is bookkeeping, but it’s structured bookkeeping, and the cost model itself uses input tokens and invocation counts plus mean output lengths. The mean cost they report across all thirteen settings is about two point nine times ten to the negative four per predicate record.

Jane: That cost number helps ground it in reality for someone managing a budget on these big processing tasks. It shows that this structured approach is significantly cheaper than other baselines they tested.

Lu: And it does show higher reported mean accuracy than the other methods they compared against. But be careful, it's not a global optimum; it’s just the least-cost certified candidate.

Meng: So, they found a solid method that is cheaper and more accurate than what we had before, but they’re still focused on this specific grid search setup. What happens when the data shifts?

Tom: The authors acknowledge that the certificate is strictly oracle-relative, meaning if you change your calibration labels to ground truth later on, the loss changes at every stage.

Jane: So they are flagging a limitation right there—the guarantee holds within the specific setup they calibrated it for. But this whole search structure is definitely a solid foundation for making LLM-based data processing more efficient and trustworthy.

Lu: Yeah, the next step for them is looking at other semantic operators and dealing with distribution shift, which means they’re thinking about how to make this search robust in real-world environments.

The Hong Kong University of Science and Technology (Guangzhou) · University of New South Wales · Zhejiang Lab · Southwest Jiaotong University

cs.DB

Submitted: 2026-11-05

Updated: 2026-11-05

Comments: technical report

Code: https://github.com/oceanusXXD/cascade

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611.08367#pg2> The objective is…

Key concepts

CascadeFilter
A three-stage semantic filtering approach where each stage uses a language model (SLM) on different inputs (evidence, full document) before forwarding unresolved items to an oracle. It jointly searches for optimal thresholds across these fixed stages to minimize total execution cost while maintaining a desired accuracy level.
Oracle-Relative Error
The accuracy guarantee is measured relative to an oracle, meaning the system's performance is compared against a perfect answer (the oracle) rather than ground truth labels. This allows for a composable guarantee where the probability of achieving a certain accuracy level ($ au$) is bounded by $1-\delta$.
Cost-Minimizing Threshold Selection
The core optimization involves selecting two thresholds for the cascade stages that minimize the token-aware cost, considering input/output counts and mean output lengths. Although the objective is generally nonconvex, the method finds a least-cost certified configuration within a specified search grid.
Certification Guarantee
The method provides a formal guarantee that after selection, there is a probability of achieving an accuracy level $ au$ or higher. This guarantee holds for oracle-relative error and ensures that the selected configuration minimizes cost under the constraints defined by the search procedure.

Terminology

Summary

Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611. p. 0

Where the paper says it

cn; weiwcs@ust. hk Abstract—Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs).

08367#pg2> The objective is to reduce the total execution cost while meeting a user-specified accuracy target<ref:2611. p. 0
Where the paper says it

A model cascade therefore uses the cheaper model first and forwards uncertain records to a more expensive reference model, called the oracle. The objective is to reduce the total execution cost while meeting a user-specified accuracy target.

08367#pg3> CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle<ref:2611. p. 0
Where the paper says it

However, its threshold calibration follows a prescribed adjustment sequence after cascade assembly, rather than minimizing cost over alternative combinations of stage thresholds. Motivated by this, we propose CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle.

08367#pg3>

Pruning redundant reasoning tokens

The paper proposes CascadeFilter, which processes each record using an SLM on extracted evidence, then the SLM on the full document if needed, and finally the oracle for unresolved records<ref:2611.08367#pg4> The method formulates cost-minimizing threshold selection for this fixed cascade<ref:2611. p. 0

Where the paper says it

Motivated by this, we propose CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle. For a fixed cascade, we formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint, and select the least-cost certified configuration in a pre-specified grid.

08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611. p. 1
Where the paper says it

We formulate cost-minimizing threshold selection for this fixed cascade. The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed.

p. 1
Where the paper says it

We formulate cost-minimizing threshold selection for this fixed cascade. The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed.

08367#pg5> The search returns the least-cost certified configuration in a pre-specified grid<ref:2611. p. 1
Where the paper says it

We develop a three-stage semantic filtering method and jointly search its coupled thresholds, accounting for both context length and model cost. The search returns the least-cost certified configuration in a pre-specified grid.

08367#pg5>

How the training signal is built

The authors formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> They organize candidates into monotone tracks and apply fixed-sequence testing within each track, allocating the failure budget across tracks<ref:2611. p. 1

Where the paper says it

For each pair, we account for errors across the complete cascade and certify oraclerelative accuracy using a sample drawn without replacement from the deployment population. We organize candidates into monotone tracks and apply fixed-sequence testing within each track, allocating the failure budget across tracks.

08367#pg6> The certificate controls overall oracle-relative error<ref:2611. p. 7
Where the paper says it

Accuracy and class imbalance. The certificate controls overall oracle-relative error.

08367#pg5>

Where the gains come from

The method integrates hybrid retrieval, reranking, batched evidence inference, and neighborhood-supported confidence into the cascade<ref:2611.08367#pg6> These components improve the candidate policies evaluated by the certification procedure<ref:2611. p. 1

Where the paper says it

We integrate hybrid retrieval, reranking, batched evidence inference, and neighborhood-supported confidence into the cascade. These components improve the candidate policies evaluated by the certification procedure.

08367#pg6> The resulting method couples cost-aware selection with finite-population certification while retaining flexibility in evidence extraction and routing<ref:2611. p. 1
Where the paper says it

These established retrieval components support the threshold-search design rather than constitute separate claims of new retrieval algorithms. The resulting method couples cost-aware selection with finite-population certification while retaining flexibility in evidence extraction and routing.

08367#pg6>

Certification and Guarantee

The guarantee applies after selection and concerns oracle-relative error ε = 1 − τ<ref:2611.08367#pg5> The search must control the risk of selecting an infeasible configuration<ref:2611. p. 5

Where the paper says it

The proof is given in Appendix A-A. Searching the Grid under Selection Because ϕˆ is chosen by its own calibration outcome, percandidate bounds do not suffice: the search must control the risk of selecting an infeasible configuration.

08367#pg6> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611. p. 5
Where the paper says it

End-to-End Guarantee Theorem 3 (Composable accuracy guarantee). Assume (1) the population, cached predictions, score family, grids, test rules, track orders, and δ-allocation are fixed independently of the calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its uniform without-replacement draw order; (3) Stage 2 covers I max 2 and the tests and scan satisfy Theorem 2 and Lemma 2; and (4) the selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2 ; sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S; ε = 1 − τ ; failure budget δ; label-free Cost(·) Ensure: Certified set C, deployed ϕˆ 1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks 2: for each track k = 1, .

p. 5
Where the paper says it

End-to-End Guarantee Theorem 3 (Composable accuracy guarantee). Assume (1) the population, cached predictions, score family, grids, test rules, track orders, and δ-allocation are fixed independently of the calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its uniform without-replacement draw order; (3) Stage 2 covers I max 2 and the tests and scan satisfy Theorem 2 and Lemma 2; and (4) the selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2 ; sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S; ε = 1 − τ ; failure budget δ; label-free Cost(·) Ensure: Certified set C, deployed ϕˆ 1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks 2: for each track k = 1, .

08367#pg5> The selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search<ref:2611. p. 5
Where the paper says it

End-to-End Guarantee Theorem 3 (Composable accuracy guarantee). Assume (1) the population, cached predictions, score family, grids, test rules, track orders, and δ-allocation are fixed independently of the calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its uniform without-replacement draw order; (3) Stage 2 covers I max 2 and the tests and scan satisfy Theorem 2 and Lemma 2; and (4) the selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2 ; sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S; ε = 1 − τ ; failure budget δ; label-free Cost(·) Ensure: Certified set C, deployed ϕˆ 1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks 2: for each track k = 1, .

p. 5
Where the paper says it

End-to-End Guarantee Theorem 3 (Composable accuracy guarantee). Assume (1) the population, cached predictions, score family, grids, test rules, track orders, and δ-allocation are fixed independently of the calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its uniform without-replacement draw order; (3) Stage 2 covers I max 2 and the tests and scan satisfy Theorem 2 and Lemma 2; and (4) the selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2 ; sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S; ε = 1 − τ ; failure budget δ; label-free Cost(·) Ensure: Certified set C, deployed ϕˆ 1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks 2: for each track k = 1, .

p. 5
Where the paper says it

End-to-End Guarantee Theorem 3 (Composable accuracy guarantee). Assume (1) the population, cached predictions, score family, grids, test rules, track orders, and δ-allocation are fixed independently of the calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its uniform without-replacement draw order; (3) Stage 2 covers I max 2 and the tests and scan satisfy Theorem 2 and Lemma 2; and (4) the selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2 ; sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S; ε = 1 − τ ; failure budget δ; label-free Cost(·) Ensure: Certified set C, deployed ϕˆ 1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks 2: for each track k = 1, .

08367#pg5>

Experimental Analysis

The experiments evaluate six datasets with 13 settings and study cascade structure, routing components, and calibration behavior<ref:2611.08367#pg6> The main table reports the highest reported benchmark accuracy and macro-averaged F1 score in 12 settings each<ref:2611. p. 1

Where the paper says it

We evaluate six datasets with 13 settings and study cascade structure, routing components, and calibration behavior. The main table reports the highest reported benchmark accuracy and macro-averaged F1 score in 12 settings each, at the evaluated operating points.

08367#pg10> CascadeFilter has the highest reported Accuracy and Macro-F1 in 12 of 13 settings, averaging 0. p. 7
Where the paper says it

Main Results (RQ1) Table II reports Accuracy and Macro-F1 across all 13 evaluation settings. CascadeFilter has the highest reported Accuracy and Macro-F1 in 12 of 13 settings, averaging 0.

938/0.887<ref:2611. p. 0
Where the paper says it

The State-of-the-art. Task Cascades [8] generates candiarXiv:2611.

08367#pg10> The reported operating points describe the reported operating points<ref:2611. p. 0
Where the paper says it

Across six datasets and 13 evaluation settings, the reported results have the highest benchmark accuracy and macro-averaged F1 score in 12 settings each. These comparisons describe the reported operating points; the accuracy guarantee concerns agreement with the oracle.

08367#pg10>

Cost and Performance

The reported cost model follows Eq. (1), using input-token and invocation counts and mean output lengths<ref:2611. p. 7

Where the paper says it

The reported cost model follows Eq. (1), using input-token and invocation counts and mean output lengths.

p. 7
Where the paper says it

The reported cost model follows Eq. (1), using input-token and invocation counts and mean output lengths.

08367#pg5> The token panel counts only oracle tokens<ref:2611. p. 7
Where the paper says it

968×10−4 per predicate–record. The token panel counts only oracle tokens.

p. 7
Where the paper says it

968×10−4 per predicate–record. The token panel counts only oracle tokens.

08367#pg10> The reported equal-weight mean cost across the 13 settings is 2. p. 7
Where the paper says it

(13) to verify that cached Stage-2 predictions and reused calibration labels are not charged twice; they should not yet be interpreted as audited end-to-end bills. The reported equal-weight mean cost across the 13 settings is $2.

p. 7
Where the paper says it

(13) to verify that cached Stage-2 predictions and reused calibration labels are not charged twice; they should not yet be interpreted as audited end-to-end bills. The reported equal-weight mean cost across the 13 settings is $2.

p. 7
Where the paper says it

(13) to verify that cached Stage-2 predictions and reused calibration labels are not charged twice; they should not yet be interpreted as audited end-to-end bills. The reported equal-weight mean cost across the 13 settings is $2.

p. 7
Where the paper says it

(13) to verify that cached Stage-2 predictions and reused calibration labels are not charged twice; they should not yet be interpreted as audited end-to-end bills. The reported equal-weight mean cost across the 13 settings is $2.

968×10−4 per predicate–record<ref:2611. p. 7
Where the paper says it

The reported equal-weight mean cost across the 13 settings is $2. 968×10−4 per predicate–record.

p. 7
Where the paper says it

The reported equal-weight mean cost across the 13 settings is $2. 968×10−4 per predicate–record.

p. 7
Where the paper says it

The reported equal-weight mean cost across the 13 settings is $2. 968×10−4 per predicate–record.

p. 7
Where the paper says it

The reported equal-weight mean cost across the 13 settings is $2. 968×10−4 per predicate–record.

08367#pg10> CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment<ref:2611. p. 8
Where the paper says it

These benchmark-label comparisons cannot determine whether an oracle-relative constraint is met. CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment.

p. 8
Where the paper says it

These benchmark-label comparisons cannot determine whether an oracle-relative constraint is met. CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment.

08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10>

Threats to Validity

The certificate is, strictly, oracle-relative: the theorems bound P(AccML (ϕˆ) ≥ τ) ≥ 1− δ<ref:2611.08367#pg5> Changing calibration labels to ground truth changes the loss at every stage, including oracle fallback<ref:2611. p. 3

Where the paper says it

Calibration labels are reused when available. This stage has zero error relative to the oracle by definition, but the oracle can disagree with ground truth.

08367#pg10> The reported violation counts and end-to-end cost accounting still require reconciliation<ref:2611. p. 8
Where the paper says it

CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment. We therefore do not infer a matched-accuracy advantage, an audited endto-end saving, or a Pareto-frontier dominance result from these plots.

08367#pg10> The single point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1)<ref:2611. p. 6
Where the paper says it

Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1): CascadeFilter returns the least-cost certified candidate, not a global optimum.

08367#pg5> The reported ground-truth Accuracy of CascadeFilter meets 0. p. 8
Where the paper says it

THE ROUTING SPLIT DESCRIBES THE REFERENCE OPERATING POINT. ACC IS CASCADEFILTER’S GROUND-TRUTH ACCURACY; AccML IS ITS AGREEMENT WITH THE ORACLE, NOT THE ORACLE’S OWN GROUND-TRUTH ACCURACY.

9 in every setting<ref:2611. p. 0
Where the paper says it

The State-of-the-art. Task Cascades [8] generates candiarXiv:2611.

p. 0
Where the paper says it

The State-of-the-art. Task Cascades [8] generates candiarXiv:2611.

p. 0
Where the paper says it

The State-of-the-art. Task Cascades [8] generates candiarXiv:2611.

08367#pg10> The violation-count discrepancy and cached-cost accounting still require verification against run logs<ref:2611. p. 6
Where the paper says it

These targets differ from the benchmark-label metrics reported here. The reported comparisons use common small and oracle models at one operating point per method; baseline-specific parameters, prompts, label budgets, and source revisions still require verification from run manifests.

08367#pg10>

Conclusion

We presented CascadeFilter, which selects two thresholds for a fixed three-stage semantic filter to reduce modeled execution cost under an oracle-relative error constraint<ref:2611.08367#pg10> The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611. p. 1

Where the paper says it

We formulate cost-minimizing threshold selection for this fixed cascade. The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed.

p. 1
Where the paper says it

We formulate cost-minimizing threshold selection for this fixed cascade. The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed.

08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611. p. 6
Where the paper says it

Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1): CascadeFilter returns the least-cost certified candidate, not a global optimum.

p. 6
Where the paper says it

Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1): CascadeFilter returns the least-cost certified candidate, not a global optimum.

p. 6
Where the paper says it

Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1): CascadeFilter returns the least-cost certified candidate, not a global optimum.

08367#pg10> The reported ground-truth Accuracy of CascadeFilter meets 0. p. 8
Where the paper says it

THE ROUTING SPLIT DESCRIBES THE REFERENCE OPERATING POINT. ACC IS CASCADEFILTER’S GROUND-TRUTH ACCURACY; AccML IS ITS AGREEMENT WITH THE ORACLE, NOT THE ORACLE’S OWN GROUND-TRUTH ACCURACY.

p. 8
Where the paper says it

THE ROUTING SPLIT DESCRIBES THE REFERENCE OPERATING POINT. ACC IS CASCADEFILTER’S GROUND-TRUTH ACCURACY; AccML IS ITS AGREEMENT WITH THE ORACLE, NOT THE ORACLE’S OWN GROUND-TRUTH ACCURACY.

9 in every setting<ref:2611.08367#pg10> Future work includes other semantic operators and recalibration under distribution shift<ref:2611.08367#pg10>

References

[1] L. Patel, S. Jha, M. Pan, H. Gupta, P. Asawa, C. Guestrin, and M. Zaharia, “Semantic operators and their optimization: Enabling LLMbased data processing with accuracy guarantees in LOTUS,” Proceedings of the VLDB Endowment, vol. p. 0

Where the paper says it

FrugalGPT and RouteLLM select among models with different costs and capabilities [4], [5]. LOTUS supports approximate semantic execution, while BARGAIN provides variants with statistical guarantees on accuracy, precision, or recall [1], [6].

18, no. 11, pp. 4171–4184, 2025<ref:2611. p. 0
Where the paper says it

The State-of-the-art. Task Cascades [8] generates candiarXiv:2611.

08367#pg4>

[2] C. Liu, M. Russo, M. J. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. J. Franklin, T. Kraska, S Madden, R Shahout and G Vitagliano “Palimpzest: Optimizing AI-powered analytics with declarative query processing,” in 15th Conference on Innovative Data Systems Research CIDR 2025 Amsterdam The Netherlands January 19-22 2025<ref:2611.08367#pg4>

[3] S. Shankar, T. Chambers, T. Shah, A. G. Parameswaran, and E Wu “DocETL: Agentic query rewriting and evaluation for complex document processing,” Proceedings of the VLDB Endowment vol 18 no 9 pp 3035–3048 2025<ref:2611.08367#pg4>

[4] L. Chen, M. Zaharia, and J Zou “FrugalGPT: How to use large language models while reducing cost and improving performance” Transactions on Machine Learning Research 2024 featured Certification<ref:2611.08367#pg4>

[5] I. Ong, A Almahairi, V Wu, W-L Chiang, T Wu, J E Gonzalez M W Kadous and I Stoica “RouteLLM: Learning to route LLMs from preference data” in The Thirteenth International Conference on Learning Representations 2025<ref:2611.08367#pg4>

[6] S Zeighami, S Shankar, and A Parameswaran “Cut costs, not accuracy: LLM-powered data processing with guarantees Proc ACM Manag Data vol 3 no 6 Dec 2025”

[7] H Zhang, Y Hui, Y Liu, and H Zhang “ScaleDoc: Scaling LLM-based predicates over large document collections Proc ACM Manag Data vol 4 no 3 May 2026”

[8] S Shankar, S Zeighami, and A Parameswaran “Task Cascades for efficient unstructured data processing Proc ACM Manag Data vol 4 no 1 Apr 2026”

[9] S Robertson and H Zaragoza “The probabilistic relevance framework: BM25 and beyond Foundations and Trends in Information Retrieval vol 4 no 1–2 pp 1–174, 2009”

[10] I Waudby-Smith and A Ramdas “Estimating means of bounded random variables by betting Journal of the Royal Statistical Society Series B: Statistical Methodology vol 86 no 1 pp 1–27 Feb. 2024”

[11] N Hou, K Zhao, J Xie, and J X Yu “Beyond linear LLM invocation An efficient and effective semantic filter paradigm arXiv preprint arXiv:2603.04799 2026”

[12] J Li, B Hui, G Qu, J Yang, B Li et al “Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs in Advances in Neural Information Processing Systems vol 36 Red Hook NY USA: Association for Computing Machinery 2023”

[13] A L Maas, R E Daly, P T Pham, D Huang, A Y Ng, and C Potts “Learning word vectors for sentiment analysis in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics Human Language Technologies D Lin Y Matsumoto and R Mihalcea Eds Portland Oregon USA: Association for Computational Linguistics Jun. 2011 pp 142–150”

[14] W Wei, Q Le, A Dai, and J Li “AirDialogue: An environment for goal-oriented dialogue research in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing E Riloff D Chiang J Hockenmaier and J Tsujii Eds Brussels Belgium Association for Computational Linguistics Nov. 2018 pp 3844–3854”

[15] T Davidson, D Warmsley, M Macy, and I Weber “Automated hate speech detection and the problem of offensive language Proceedings of the International AAAI Conference on Web and Social Media vol 11 no 1 pp 512–515 2017”

[16] J Thorne, A Vlachos, C Christodoulopoulos, and A Mittal “FEVER: a large-scale dataset for fact extraction and VERification in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies Volume 1 Long Papers M Walker H Ji and A Stent Eds New Orleans Louisiana Association for Computational Linguistics Jun. 2018 pp 809–819”

[17] Qwen Team “Qwen3-4B-Instruct-2507 Hugging Face model card 2025 accessed 2026-07-02”

[18] Y Zhang, M Li, D Long, X Zhang, H Lin, B Yang, P Xie et al “Qwen3 embedding: Advancing text embedding and reranking through foundation models 2025”

[19] OpenAI “GPT-5.5 model OpenAI API documentation 2026 accessed 2026-07-02”

[20] A Yang, A Li, B Yang, B Zhang et al “Qwen3 technical report 2025”

[21] Meta “Llama 3.2: 1b and 3b text model card Hugging Face model card 2024 accessed 2026-11-05”

[23] L Wang, N Yang, X Huang, B Jiao, L Yang et al “Text embeddings by weakly-supervised contrastive pretraining 2022”

[35] P Aggarwal, A Madaan, A Anand et al “AutoMix: Automatically mixing language models in Advances in Neural Information Processing Systems vol 37 Red Hook NY USA: Curran Associates Inc. 2024”

[36] D Ding, A Mallick, C Wang, R Sim et al “Hybrid LLM: Costefficient and quality-aware query routing in The Twelfth International Conference on Learning Representations 2024”

[37] S Jo and I Trummer “SMART: Automatically scaling down language models with accuracy guarantees for reduced processing fees 2024”

[38] K Huang, Y Shi, D Ding, Y Li et al “ThriftLLM: On cost-effective selection of large language models for classification queries Proceedings of the VLDB Endowment vol 18 no 11 pp 4410–4423 2025”

[39] W Jitkrittum, H Narasimhan, A S Rawat, J Juneja, C Wang, Z Wang et al “Universal model routing for efficient LLM inference in The Fourteenth International Conference on Learning Representations 2026”

[40] A Valkanas, S Pal, P Rumiantsev Y Zhang and M Coates “C3PO: Optimized large language model cascades with probabilistic cost constraints for reasoning in Advances in Neural Information Processing Systems vol 38 Red Hook NY USA: Association for Computing Machinery 2025 pp 84 481–84 522”

[41] V Kotte “UCCI: Calibrated uncertainty for cost-optimal LLM cascade routing 2026”

[47] A N Angelopoulos, S Bates, E J Candès, M I Jordan, and L Lei “Learn then test: Calibrating predictive algorithms to achieve risk control The Annals of Applied Statistics vol 19 no 2 pp 1641–1662 2025”

[48] G V Cormack, C L A Clarke, and S Buettcher “Reciprocal rank fusion outperforms condorcet and individual rank learning methods in Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval ser SIGIR ’09 New York NY USA: Association for Computing Machinery 2009 pp 758–759”

[49] Qwen Team “Qwen3-Embedding-4B Hugging Face model card 2025 accessed 2026-07-02”

[51] M Xiong, Z Hu, X Lu, Y Li et al “Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs in The Twelfth International Conference on Learning Representations 2024”

--- Page 18 ---

The paper presents CascadeFilter, which selects two thresholds for a fixed three-stage semantic filter to reduce modeled execution cost under an oracle-relative error constraint<ref:2611.08367#pg10> The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611.08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611.08367#pg10> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10>

The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611.08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611.08367#pg10>

The search must control the risk of selecting an infeasible configuration<ref:2611.08367#pg6> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611.08367#pg5> The selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search<ref:2611.08367#pg5>

The guarantee applies after selection and concerns oracle-relative error ε = 1 − τ<ref:2611.08367#pg5> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611. p. 5

Where the paper says it

End-to-End Guarantee Theorem 3 (Composable accuracy guarantee). Assume (1) the population, cached predictions, score family, grids, test rules, track orders, and δ-allocation are fixed independently of the calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its uniform without-replacement draw order; (3) Stage 2 covers I max 2 and the tests and scan satisfy Theorem 2 and Lemma 2; and (4) the selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2 ; sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S; ε = 1 − τ ; failure budget δ; label-free Cost(·) Ensure: Certified set C, deployed ϕˆ 1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks 2: for each track k = 1, .

08367#pg5> The selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search<ref:2611.08367#pg5>

The reported cost model follows Eq. (1), using input-token and invocation counts and mean output lengths<ref:2611.08367#pg5> The token panel counts only oracle tokens<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Joint Threshold Selection for Cost Minimization

The CascadeFilter method jointly searches thresholds for a small language model operating on extracted evidence and full documents, formulating threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint to select the least-cost certified configuration in a pre-specified grid. This allows the system to choose the optimal balance between execution cost and accuracy by evaluating candidates based on their total token cost.

  1. Finite Population Certification for Accuracy Guarantee

The system employs Finite-Population Certification using a sample drawn without replacement and a Waudby-Smith and Ramdas betting framework to bound oracle-relative error, ensuring that the selected configuration meets the target accuracy with probability of success of at least 1 minus delta.

  1. Hybrid Evidence and Routing Enhancements

The system improves evidence extraction and routing by combining Best Matching BM25 lexical retrieval with dense retrieval using reciprocal rank fusion RRF, reranking the resulting chunks, and incorporating embedding-neighborhood support into model confidence. This enhances the candidate policies evaluated by the certification procedure.

  1. Dynamic Record Scoring via Routing Scores

The system uses a routing score that combines model confidence with agreement among nearby records in embedding space to nudge the base confidence by how strongly a record’s neighbors agree with its prediction, allowing each record's effective threshold to track its difficulty without per-record tuning.

  1. Cost-Aware Execution Accounting

The system distinguishes between modeled selected-plan cost and actual same-batch cost with prediction reuse using Equation (13), ensuring that the reported costs account for cached Stage 2 predictions and reused calibration labels while avoiding double charging for preprocessed elements.

Abstract

Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs). Executing an expensive model on every document incurs substantial inference cost. Existing approaches reduce this cost by accepting predictions from cheaper models and forwarding uncertain records to an oracle. Task Cascades further varies document portions and operations, and provides an accuracy-guaranteed extension. However, its threshold calibration follows a prescribed adjustment sequence after cascade assembly, rather than minimizing cost over alternative combinations of stage thresholds. Motivated by this, we propose CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle. For a fixed cascade, we formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint, and select the least-cost certified configuration in a pre-specified grid. A finite-population calibration procedure bounds oracle-relative error after selection, using fixed-sequence tests on threshold tracks to allocate the multiple-testing budget. We further combine lexical and dense retrieval, reranking, and neighborhood-supported confidence to improve evidence extraction and routing. These components enhance the cascade while leaving its certification interface unchanged. Across six datasets and 13 evaluation settings, the reported results have the highest benchmark accuracy and macro-averaged F1 score in 12 settings each. These comparisons describe the reported operating points; the accuracy guarantee concerns agreement with the oracle.

Sources

Related papers