CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection".
Jane: Here's the gist of the paper. Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611.08367#pg2> The objective is to reduce the total execution cost while meeting a user-specified accuracy target<ref:2611.08367#pg3> CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle<ref:2611.08367#pg3> Pruning redundant reasoning tokens The paper proposes CascadeFilter, which processes each record using an SLM on extracted evidence, then the SLM on the full document if needed, and finally the oracle for unresolved records<ref:2611.08367#pg4> The method formulates cost-minimizing threshold selection for this fixed cascade<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> The search returns the least-cost certified configuration in a pre-specified grid<ref:2611.08367#pg5> How the training signal is built The authors formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> They organize candidates into monotone tracks and apply fixed-sequence testing within each track, allocating the failure budget across tracks<ref:2611.08367#pg6> The certificate controls overall oracle-relative error<ref:2611.08367#pg5> Where the gains come from The method integrates hybrid retrieval, reranking, batched evidence inference, and neighborhood-supported confidence into the cascade<ref:2611.08367#pg6> These components improve the candidate policies evaluated by the certification procedure<ref:2611.08367#pg6> The resulting method couples cost-aware selection with finite-population certification while retaining flexibility in evidence extraction and routing<ref:2611.08367#pg6> Certification and Guarantee The guarantee applies after selection and concerns oracle-relative error ε = 1 − τ<ref:2611.08367#pg5> The search must control the risk of selecting an infeasible configuration<ref:2611.08367#pg6> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611.08367#pg5> The selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search<ref:2611.08367#pg5> Experimental Analysis The experiments evaluate six datasets with 13 settings and study cascade structure, routing components, and calibration behavior<ref:2611.08367#pg6> The main table reports the highest reported benchmark accuracy and macro-averaged F1 score in 12 settings each<ref:2611.08367#pg10> CascadeFilter has the highest reported Accuracy and Macro-F1 in 12 of 13 settings, averaging 0.938/0.887<ref:2611.08367#pg10> The reported operating points describe the reported operating points<ref:2611.08367#pg10> Cost and Performance The reported cost model follows Eq.
Tom: (1), using input-token and invocation counts and mean output lengths<ref:2611.08367#pg5> The token panel counts only oracle tokens<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> Threats to Validity The certificate is, strictly, oracle-relative: the theorems bound P(AccML (ϕˆ) ≥ τ) ≥ 1− δ<ref:2611.08367#pg5> Changing calibration labels to ground truth changes the loss at every stage, including oracle fallback<ref:2611.08367#pg10> The reported violation counts and end-to-end cost accounting still require reconciliation<ref:2611.08367#pg10> The single point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1)<ref:2611.08367#pg5> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10> The violation-count discrepancy and cached-cost accounting still require verification against run logs<ref:2611.08367#pg10> We presented CascadeFilter, which selects two thresholds for a fixed three-stage semantic filter to reduce modeled execution cost under an oracle-relative error constraint<ref:2611.08367#pg10> The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611.08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611.08367#pg10> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10> Future work includes other semantic operators and recalibration under distribution shift<ref:2611.08367#pg10> References
1: L.
Jane: Patel, S.
Tom: Jha, M.
Paper summary: Tom: So, we’re wrapping up on CascadeFilter, which is all about picking two thresholds for a three-stage semantic filter to cut down on execution cost while keeping an accuracy target in mind.
Jane: That sounds like they’ve figured out how to stop wasting compute time when you're dealing with huge amounts of text. So, who wrote this paper?
Lu: It was written by Patel and Jha and Pan and Gupta Asawa Guestrin and Zaharia at the VLDB Endowment in two thousand twenty-five <ref:2611.08367#pg4>.
Meng: Okay, so it’s a specific set of researchers working on document processing with these large language models. What’s the main deal they’re trying to prove?
Tom: They proved that you can optimize the total token cost by jointly searching for those two thresholds instead of just picking them one by one <ref:2611.08367#pg5>.
Jane: So, it’s not just about finding a better single model; it’s about structuring the entire filtering process more intelligently. What does this mean in practice?
Lu: It means they found a way to manage the trade-off between cost and accuracy under an oracle-relative error constraint <ref:2611.08367#pg5>.
Meng: Under an oracle-relative error constraint—that’s just a fancy way of saying they are measuring how much their results might differ from the absolute best possible answer, which is hard to know without using an oracle.
Tom: Exactly. And the result is that they return a candidate configuration that has the least cost while still meeting that error constraint, instead of just finding some random good setting <ref:2611.08367#pg10>.
Jane: That’s interesting because it suggests a systematic search approach rather than just guessing settings and hoping for the best <ref:2611.08367#pg5>.
Lu: Yeah, they found that even though their search space is generally not simple, the error along each path where you fix the first threshold actually goes down as you move to the second one <ref:2611.08367#pg5>.
Meng: But Tom, if we’re shipping this to production, we need a guarantee. What kind of guarantee are they giving us here?
Tom: They give a composable accuracy guarantee where there is at least one minus delta probability that the accuracy will be greater than or equal to tau, which is their chosen error threshold <ref:2611.08367#pg5>.
Jane: So, it’s not just a hopeful claim; they have a mathematical bound on how accurate their system should be based on what they can observe.
Lu: They use this theorem to control the risk of picking an infeasible configuration during their search <ref:2611.08367#pg6>. The selected pair minimizes cost in C, which is their cost model, using all-oracle Algorithm two Certified Joint Threshold Search <ref:2611.08367#pg5>.
Meng: So it’s a certified search mechanism for the best configuration they can find within that specific grid structure. That still sounds like a lot of bookkeeping for deployment.
Tom: It is bookkeeping, but it’s structured bookkeeping, and the cost model itself uses input tokens and invocation counts plus mean output lengths <ref:2611.08367#pg5>. The mean cost they report across all thirteen settings is about two point nine times ten to the negative four per predicate record <ref:2611.08367#pg10>.
Jane: That cost number helps ground it in reality for someone managing a budget on these big processing tasks. It shows that this structured approach
Conclusion: Tom: So, we're wrapping up on CascadeFilter, which is about picking two thresholds for a three-stage semantic filter to cut down on execution cost while keeping an accuracy target in mind.
Jane: That sounds like they’ve figured out how to stop wasting compute time when you're dealing with huge amounts of text. So, who wrote this paper?
Lu: It was written by Patel and Jha and Pan and Gupta Asawa Guestrin and Zaharia at the VLDB Endowment in two thousand twenty-five.
Meng: Okay, so it’s a specific set of researchers working on document processing with these large language models. What’s the main deal they’re trying to prove?
Tom: They proved that you can optimize the total token cost by jointly searching for those two thresholds instead of just picking them one by one.
Jane: So, it’s not just about finding a better single model; it’s about structuring the entire filtering process more intelligently. What does this mean in practice?
Lu: It means they found a way to manage the trade-off between cost and accuracy under an oracle-relative error constraint.
Meng: Under an oracle-relative error constraint—that’s just a fancy way of saying they are measuring how much their results might differ from the absolute best possible answer, which is hard to know without using an oracle.
Tom: Exactly. And the result is that they return a candidate configuration that has the least cost while still meeting that error constraint, instead of just finding some random good setting.
Jane: That’s interesting because it suggests a systematic search approach rather than just guessing settings and hoping for the best.
Lu: Yeah, they found that even though their search space is generally not simple, the error along each path where you fix the first threshold actually goes down as you move to the second one.
Meng: But Tom, if we’re shipping this to production, we need a guarantee. What kind of guarantee are they giving us here?
Tom: They give a composable accuracy guarantee where there is at least one minus delta probability that the accuracy will be greater than or equal to tau, which is their chosen error threshold.
Jane: So, it’s not just a hopeful claim; they have a mathematical bound on how accurate their system should be based on what they can observe.
Lu: They use this theorem to control the risk of picking an infeasible configuration during their search. The selected pair minimizes cost in C, which is their cost model, using all-oracle Algorithm two Certified Joint Threshold Search.
Meng: So it’s a certified search mechanism for the best configuration they can find within that specific grid structure. That still sounds like a lot of bookkeeping for deployment.
Tom: It is bookkeeping, but it’s structured bookkeeping, and the cost model itself uses input tokens and invocation counts plus mean output lengths. The mean cost they report across all thirteen settings is about two point nine times ten to the negative four per predicate record.
Jane: That cost number helps ground it in reality for someone managing a budget on these big processing tasks. It shows that this structured approach is significantly cheaper than other baselines they tested.
Lu: And it does show higher reported mean accuracy than the other methods they compared against. But be careful, it's not a global optimum; it’s just the least-cost certified candidate.
Meng: So, they found a solid method that is cheaper and more accurate than what we had before, but they’re still focused on this specific grid search setup. What happens when the data shifts?
Tom: The authors acknowledge that the certificate is strictly oracle-relative, meaning if you change your calibration labels to ground truth later on, the loss changes at every stage.
Jane: So they are flagging a limitation right there—the guarantee holds within the specific setup they calibrated it for. But this whole search structure is definitely a solid foundation for making LLM-based data processing more efficient and trustworthy.
Lu: Yeah, the next step for them is looking at other semantic operators and dealing with distribution shift, which means they’re thinking about how to make this search robust in real-world environments.
The Hong Kong University of Science and Technology (Guangzhou) · University of New South Wales · Zhejiang Lab · Southwest Jiaotong University
cs.DB
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: technical report
Code: https://github.com/oceanusXXD/cascade
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611.08367#pg2> The objective is…
Key concepts
- CascadeFilter
- A three-stage semantic filtering approach where each stage uses a language model (SLM) on different inputs (evidence, full document) before forwarding unresolved items to an oracle. It jointly searches for optimal thresholds across these fixed stages to minimize total execution cost while maintaining a desired accuracy level.
- Oracle-Relative Error
- The accuracy guarantee is measured relative to an oracle, meaning the system's performance is compared against a perfect answer (the oracle) rather than ground truth labels. This allows for a composable guarantee where the probability of achieving a certain accuracy level ($ au$) is bounded by $1-\delta$.
- Cost-Minimizing Threshold Selection
- The core optimization involves selecting two thresholds for the cascade stages that minimize the token-aware cost, considering input/output counts and mean output lengths. Although the objective is generally nonconvex, the method finds a least-cost certified configuration within a specified search grid.
- Certification Guarantee
- The method provides a formal guarantee that after selection, there is a probability of achieving an accuracy level $ au$ or higher. This guarantee holds for oracle-relative error and ensures that the selected configuration minimizes cost under the constraints defined by the search procedure.
Terminology
Summary
Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611. p. 0 cn; weiwcs@ust. hk
Abstract—Semantic filtering selects records that satisfy a
natural-language predicate and is an important operation in data
analytics based on large language models (LLMs). A model cascade therefore uses the cheaper model first and
forwards uncertain records to a more expensive reference
model, called the oracle. The objective is to reduce the total
execution cost while meeting a user-specified accuracy target. However, its threshold calibration follows a prescribed adjustment sequence after cascade assembly, rather than minimizing
cost over alternative combinations of stage thresholds. Motivated by this, we propose CascadeFilter, a three-stage
semantic filtering method that jointly searches the thresholds
for a small language model operating on extracted evidence
and full documents, with unresolved records forwarded to the
oracle.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper proposes CascadeFilter, which processes each record using an SLM on extracted evidence, then the SLM on the full document if needed, and finally the oracle for unresolved records<ref:2611.08367#pg4> The method formulates cost-minimizing threshold selection for this fixed cascade<ref:2611. p. 0 Motivated by this, we propose CascadeFilter, a three-stage
semantic filtering method that jointly searches the thresholds
for a small language model operating on extracted evidence
and full documents, with unresolved records forwarded to the
oracle. For a fixed cascade, we formulate threshold selection as
minimizing token-aware cost subject to an oracle-relative error
constraint, and select the least-cost certified configuration in
a pre-specified grid. We formulate cost-minimizing threshold
selection for this fixed cascade. The objective and feasible
set are generally nonconvex, but error decreases with the
second-stage threshold when the first is fixed. We formulate cost-minimizing threshold
selection for this fixed cascade. The objective and feasible
set are generally nonconvex, but error decreases with the
second-stage threshold when the first is fixed. We develop a
three-stage semantic filtering method and jointly search
its coupled thresholds, accounting for both context length
and model cost. The search returns the least-cost certified
configuration in a pre-specified grid.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The authors formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> They organize candidates into monotone tracks and apply fixed-sequence testing within each track, allocating the failure budget across tracks<ref:2611. p. 1 For each pair, we account
for errors across the complete cascade and certify oraclerelative accuracy using a sample drawn without replacement
from the deployment population. We organize candidates into
monotone tracks and apply fixed-sequence testing within each
track, allocating the failure budget across tracks. Accuracy and class imbalance. The certificate controls
overall oracle-relative error.Where the paper says it
Where the paper says it
Where the gains come from
The method integrates hybrid retrieval, reranking, batched evidence inference, and neighborhood-supported confidence into the cascade<ref:2611.08367#pg6> These components improve the candidate policies evaluated by the certification procedure<ref:2611. p. 1 We integrate hybrid retrieval, reranking, batched evidence
inference, and neighborhood-supported confidence into
the cascade. These components improve the candidate
policies evaluated by the certification procedure. These established retrieval components support the
threshold-search design rather than constitute separate claims
of new retrieval algorithms. The resulting method couples
cost-aware selection with finite-population certification while
retaining flexibility in evidence extraction and routing.Where the paper says it
Where the paper says it
Certification and Guarantee
The guarantee applies after selection and concerns oracle-relative error ε = 1 − τ<ref:2611.08367#pg5> The search must control the risk of selecting an infeasible configuration<ref:2611. p. 5 The proof is given in Appendix A-A. Searching the Grid under Selection
Because ϕˆ is chosen by its own calibration outcome, percandidate bounds do not suffice: the search must control the
risk of selecting an infeasible configuration. End-to-End Guarantee
Theorem 3 (Composable accuracy guarantee). Assume (1) the
population, cached predictions, score family, grids, test rules,
track orders, and δ-allocation are fixed independently of the
calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its
uniform without-replacement draw order; (3) Stage 2 covers
I
max
2 and the tests and scan satisfy Theorem 2 and Lemma 2;
and (4) the selected pair minimizes cost in C, with all-oracle
Algorithm 2 Certified Joint Threshold Search
Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2
;
sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S;
ε = 1 − τ ; failure budget δ; label-free Cost(·)
Ensure: Certified set C, deployed ϕˆ
1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks
2: for each track k = 1, . End-to-End Guarantee
Theorem 3 (Composable accuracy guarantee). Assume (1) the
population, cached predictions, score family, grids, test rules,
track orders, and δ-allocation are fixed independently of the
calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its
uniform without-replacement draw order; (3) Stage 2 covers
I
max
2 and the tests and scan satisfy Theorem 2 and Lemma 2;
and (4) the selected pair minimizes cost in C, with all-oracle
Algorithm 2 Certified Joint Threshold Search
Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2
;
sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S;
ε = 1 − τ ; failure budget δ; label-free Cost(·)
Ensure: Certified set C, deployed ϕˆ
1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks
2: for each track k = 1, . End-to-End Guarantee
Theorem 3 (Composable accuracy guarantee). Assume (1) the
population, cached predictions, score family, grids, test rules,
track orders, and δ-allocation are fixed independently of the
calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its
uniform without-replacement draw order; (3) Stage 2 covers
I
max
2 and the tests and scan satisfy Theorem 2 and Lemma 2;
and (4) the selected pair minimizes cost in C, with all-oracle
Algorithm 2 Certified Joint Threshold Search
Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2
;
sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S;
ε = 1 − τ ; failure budget δ; label-free Cost(·)
Ensure: Certified set C, deployed ϕˆ
1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks
2: for each track k = 1, . End-to-End Guarantee
Theorem 3 (Composable accuracy guarantee). Assume (1) the
population, cached predictions, score family, grids, test rules,
track orders, and δ-allocation are fixed independently of the
calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its
uniform without-replacement draw order; (3) Stage 2 covers
I
max
2 and the tests and scan satisfy Theorem 2 and Lemma 2;
and (4) the selected pair minimizes cost in C, with all-oracle
Algorithm 2 Certified Joint Threshold Search
Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2
;
sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S;
ε = 1 − τ ; failure budget δ; label-free Cost(·)
Ensure: Certified set C, deployed ϕˆ
1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks
2: for each track k = 1, . End-to-End Guarantee
Theorem 3 (Composable accuracy guarantee). Assume (1) the
population, cached predictions, score family, grids, test rules,
track orders, and δ-allocation are fixed independently of the
calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its
uniform without-replacement draw order; (3) Stage 2 covers
I
max
2 and the tests and scan satisfy Theorem 2 and Lemma 2;
and (4) the selected pair minimizes cost in C, with all-oracle
Algorithm 2 Certified Joint Threshold Search
Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2
;
sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S;
ε = 1 − τ ; failure budget δ; label-free Cost(·)
Ensure: Certified set C, deployed ϕˆ
1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks
2: for each track k = 1, .Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Experimental Analysis
The experiments evaluate six datasets with 13 settings and study cascade structure, routing components, and calibration behavior<ref:2611.08367#pg6> The main table reports the highest reported benchmark accuracy and macro-averaged F1 score in 12 settings each<ref:2611. p. 1 We evaluate six
datasets with 13 settings and study cascade structure,
routing components, and calibration behavior. The main
table reports the highest reported benchmark accuracy
and macro-averaged F1 score in 12 settings each, at the
evaluated operating points. Main Results (RQ1)
Table II reports Accuracy and Macro-F1 across all 13 evaluation settings. CascadeFilter has the highest reported Accuracy
and Macro-F1 in 12 of 13 settings, averaging 0. The State-of-the-art. Task Cascades [8] generates candiarXiv:2611. Across six
datasets and 13 evaluation settings, the reported results have
the highest benchmark accuracy and macro-averaged F1 score
in 12 settings each. These comparisons describe the reported
operating points; the accuracy guarantee concerns agreement
with the oracle.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Cost and Performance
The reported cost model follows Eq. (1), using input-token and invocation counts and mean output lengths<ref:2611. p. 7 The reported cost model follows Eq. (1),
using input-token and invocation counts and mean output
lengths. The reported cost model follows Eq. (1),
using input-token and invocation counts and mean output
lengths. 968×10−4
per predicate–record. The token panel counts only oracle
tokens. 968×10−4
per predicate–record. The token panel counts only oracle
tokens. (13) to verify that cached Stage-2 predictions
and reused calibration labels are not charged twice; they should
not yet be interpreted as audited end-to-end bills. The reported
equal-weight mean cost across the 13 settings is $2. (13) to verify that cached Stage-2 predictions
and reused calibration labels are not charged twice; they should
not yet be interpreted as audited end-to-end bills. The reported
equal-weight mean cost across the 13 settings is $2. (13) to verify that cached Stage-2 predictions
and reused calibration labels are not charged twice; they should
not yet be interpreted as audited end-to-end bills. The reported
equal-weight mean cost across the 13 settings is $2. (13) to verify that cached Stage-2 predictions
and reused calibration labels are not charged twice; they should
not yet be interpreted as audited end-to-end bills. The reported
equal-weight mean cost across the 13 settings is $2. The reported
equal-weight mean cost across the 13 settings is $2. 968×10−4
per predicate–record. The reported
equal-weight mean cost across the 13 settings is $2. 968×10−4
per predicate–record. The reported
equal-weight mean cost across the 13 settings is $2. 968×10−4
per predicate–record. The reported
equal-weight mean cost across the 13 settings is $2. 968×10−4
per predicate–record. These
benchmark-label comparisons cannot determine whether an
oracle-relative constraint is met. CascadeFilter has higher
reported mean Accuracy than the other baselines, but the cost
ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment. These
benchmark-label comparisons cannot determine whether an
oracle-relative constraint is met. CascadeFilter has higher
reported mean Accuracy than the other baselines, but the cost
ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Threats to Validity
The certificate is, strictly, oracle-relative: the theorems bound P(AccML (ϕˆ) ≥ τ) ≥ 1− δ<ref:2611.08367#pg5> Changing calibration labels to ground truth changes the loss at every stage, including oracle fallback<ref:2611. p. 3 Calibration labels are reused when
available. This stage has zero error relative to the oracle
by definition, but the oracle can disagree with ground truth. CascadeFilter has higher
reported mean Accuracy than the other baselines, but the cost
ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment. We therefore
do not infer a matched-accuracy advantage, an audited endto-end saving, or a Pareto-frontier dominance result from these
plots. Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1):
CascadeFilter returns the least-cost certified candidate, not
a global optimum. THE ROUTING SPLIT
DESCRIBES THE REFERENCE OPERATING POINT. ACC IS
CASCADEFILTER’S GROUND-TRUTH ACCURACY; AccML
IS ITS
AGREEMENT WITH THE ORACLE, NOT THE ORACLE’S OWN
GROUND-TRUTH ACCURACY. The State-of-the-art. Task Cascades [8] generates candiarXiv:2611. The State-of-the-art. Task Cascades [8] generates candiarXiv:2611. The State-of-the-art. Task Cascades [8] generates candiarXiv:2611. These targets differ from the
benchmark-label metrics reported here. The reported comparisons use common small and oracle models at one operating
point per method; baseline-specific parameters, prompts, label
budgets, and source revisions still require verification from
run manifests.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
We presented CascadeFilter, which selects two thresholds for a fixed three-stage semantic filter to reduce modeled execution cost under an oracle-relative error constraint<ref:2611.08367#pg10> The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611. p. 1 We formulate cost-minimizing threshold
selection for this fixed cascade. The objective and feasible
set are generally nonconvex, but error decreases with the
second-stage threshold when the first is fixed. We formulate cost-minimizing threshold
selection for this fixed cascade. The objective and feasible
set are generally nonconvex, but error decreases with the
second-stage threshold when the first is fixed. Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1):
CascadeFilter returns the least-cost certified candidate, not
a global optimum. Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1):
CascadeFilter returns the least-cost certified candidate, not
a global optimum. Scope and assumptions. The single-point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1):
CascadeFilter returns the least-cost certified candidate, not
a global optimum. THE ROUTING SPLIT
DESCRIBES THE REFERENCE OPERATING POINT. ACC IS
CASCADEFILTER’S GROUND-TRUTH ACCURACY; AccML
IS ITS
AGREEMENT WITH THE ORACLE, NOT THE ORACLE’S OWN
GROUND-TRUTH ACCURACY. THE ROUTING SPLIT
DESCRIBES THE REFERENCE OPERATING POINT. ACC IS
CASCADEFILTER’S GROUND-TRUTH ACCURACY; AccML
IS ITS
AGREEMENT WITH THE ORACLE, NOT THE ORACLE’S OWN
GROUND-TRUTH ACCURACY.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
[1] L. Patel, S. Jha, M. Pan, H. Gupta, P. Asawa, C. Guestrin, and M. Zaharia, “Semantic operators and their optimization: Enabling LLMbased data processing with accuracy guarantees in LOTUS,” Proceedings of the VLDB Endowment, vol. p. 0 FrugalGPT and RouteLLM select among models with different
costs and capabilities [4], [5]. LOTUS supports approximate
semantic execution, while BARGAIN provides variants with
statistical guarantees on accuracy, precision, or recall [1],
[6]. The State-of-the-art. Task Cascades [8] generates candiarXiv:2611.Where the paper says it
Where the paper says it
[2] C. Liu, M. Russo, M. J. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. J. Franklin, T. Kraska, S Madden, R Shahout and G Vitagliano “Palimpzest: Optimizing AI-powered analytics with declarative query processing,” in 15th Conference on Innovative Data Systems Research CIDR 2025 Amsterdam The Netherlands January 19-22 2025<ref:2611.08367#pg4>
[3] S. Shankar, T. Chambers, T. Shah, A. G. Parameswaran, and E Wu “DocETL: Agentic query rewriting and evaluation for complex document processing,” Proceedings of the VLDB Endowment vol 18 no 9 pp 3035–3048 2025<ref:2611.08367#pg4>
[4] L. Chen, M. Zaharia, and J Zou “FrugalGPT: How to use large language models while reducing cost and improving performance” Transactions on Machine Learning Research 2024 featured Certification<ref:2611.08367#pg4>
[5] I. Ong, A Almahairi, V Wu, W-L Chiang, T Wu, J E Gonzalez M W Kadous and I Stoica “RouteLLM: Learning to route LLMs from preference data” in The Thirteenth International Conference on Learning Representations 2025<ref:2611.08367#pg4>
[6] S Zeighami, S Shankar, and A Parameswaran “Cut costs, not accuracy: LLM-powered data processing with guarantees Proc ACM Manag Data vol 3 no 6 Dec 2025”
[7] H Zhang, Y Hui, Y Liu, and H Zhang “ScaleDoc: Scaling LLM-based predicates over large document collections Proc ACM Manag Data vol 4 no 3 May 2026”
[8] S Shankar, S Zeighami, and A Parameswaran “Task Cascades for efficient unstructured data processing Proc ACM Manag Data vol 4 no 1 Apr 2026”
[9] S Robertson and H Zaragoza “The probabilistic relevance framework: BM25 and beyond Foundations and Trends in Information Retrieval vol 4 no 1–2 pp 1–174, 2009”
[10] I Waudby-Smith and A Ramdas “Estimating means of bounded random variables by betting Journal of the Royal Statistical Society Series B: Statistical Methodology vol 86 no 1 pp 1–27 Feb. 2024”
[11] N Hou, K Zhao, J Xie, and J X Yu “Beyond linear LLM invocation An efficient and effective semantic filter paradigm arXiv preprint arXiv:2603.04799 2026”
[12] J Li, B Hui, G Qu, J Yang, B Li et al “Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs in Advances in Neural Information Processing Systems vol 36 Red Hook NY USA: Association for Computing Machinery 2023”
[13] A L Maas, R E Daly, P T Pham, D Huang, A Y Ng, and C Potts “Learning word vectors for sentiment analysis in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics Human Language Technologies D Lin Y Matsumoto and R Mihalcea Eds Portland Oregon USA: Association for Computational Linguistics Jun. 2011 pp 142–150”
[14] W Wei, Q Le, A Dai, and J Li “AirDialogue: An environment for goal-oriented dialogue research in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing E Riloff D Chiang J Hockenmaier and J Tsujii Eds Brussels Belgium Association for Computational Linguistics Nov. 2018 pp 3844–3854”
[15] T Davidson, D Warmsley, M Macy, and I Weber “Automated hate speech detection and the problem of offensive language Proceedings of the International AAAI Conference on Web and Social Media vol 11 no 1 pp 512–515 2017”
[16] J Thorne, A Vlachos, C Christodoulopoulos, and A Mittal “FEVER: a large-scale dataset for fact extraction and VERification in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies Volume 1 Long Papers M Walker H Ji and A Stent Eds New Orleans Louisiana Association for Computational Linguistics Jun. 2018 pp 809–819”
[17] Qwen Team “Qwen3-4B-Instruct-2507 Hugging Face model card 2025 accessed 2026-07-02”
[18] Y Zhang, M Li, D Long, X Zhang, H Lin, B Yang, P Xie et al “Qwen3 embedding: Advancing text embedding and reranking through foundation models 2025”
[19] OpenAI “GPT-5.5 model OpenAI API documentation 2026 accessed 2026-07-02”
[20] A Yang, A Li, B Yang, B Zhang et al “Qwen3 technical report 2025”
[21] Meta “Llama 3.2: 1b and 3b text model card Hugging Face model card 2024 accessed 2026-11-05”
[23] L Wang, N Yang, X Huang, B Jiao, L Yang et al “Text embeddings by weakly-supervised contrastive pretraining 2022”
[35] P Aggarwal, A Madaan, A Anand et al “AutoMix: Automatically mixing language models in Advances in Neural Information Processing Systems vol 37 Red Hook NY USA: Curran Associates Inc. 2024”
[36] D Ding, A Mallick, C Wang, R Sim et al “Hybrid LLM: Costefficient and quality-aware query routing in The Twelfth International Conference on Learning Representations 2024”
[37] S Jo and I Trummer “SMART: Automatically scaling down language models with accuracy guarantees for reduced processing fees 2024”
[38] K Huang, Y Shi, D Ding, Y Li et al “ThriftLLM: On cost-effective selection of large language models for classification queries Proceedings of the VLDB Endowment vol 18 no 11 pp 4410–4423 2025”
[39] W Jitkrittum, H Narasimhan, A S Rawat, J Juneja, C Wang, Z Wang et al “Universal model routing for efficient LLM inference in The Fourteenth International Conference on Learning Representations 2026”
[40] A Valkanas, S Pal, P Rumiantsev Y Zhang and M Coates “C3PO: Optimized large language model cascades with probabilistic cost constraints for reasoning in Advances in Neural Information Processing Systems vol 38 Red Hook NY USA: Association for Computing Machinery 2025 pp 84 481–84 522”
[41] V Kotte “UCCI: Calibrated uncertainty for cost-optimal LLM cascade routing 2026”
[47] A N Angelopoulos, S Bates, E J Candès, M I Jordan, and L Lei “Learn then test: Calibrating predictive algorithms to achieve risk control The Annals of Applied Statistics vol 19 no 2 pp 1641–1662 2025”
[48] G V Cormack, C L A Clarke, and S Buettcher “Reciprocal rank fusion outperforms condorcet and individual rank learning methods in Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval ser SIGIR ’09 New York NY USA: Association for Computing Machinery 2009 pp 758–759”
[49] Qwen Team “Qwen3-Embedding-4B Hugging Face model card 2025 accessed 2026-07-02”
[51] M Xiong, Z Hu, X Lu, Y Li et al “Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs in The Twelfth International Conference on Learning Representations 2024”
--- Page 18 ---
The paper presents CascadeFilter, which selects two thresholds for a fixed three-stage semantic filter to reduce modeled execution cost under an oracle-relative error constraint<ref:2611.08367#pg10> The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611.08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611.08367#pg10> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10>
The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611.08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611.08367#pg10>
The search must control the risk of selecting an infeasible configuration<ref:2611.08367#pg6> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611.08367#pg5> The selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search<ref:2611.08367#pg5>
The guarantee applies after selection and concerns oracle-relative error ε = 1 − τ<ref:2611.08367#pg5> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611. p. 5 End-to-End Guarantee
Theorem 3 (Composable accuracy guarantee). Assume (1) the
population, cached predictions, score family, grids, test rules,
track orders, and δ-allocation are fixed independently of the
calibration draw; (2) 1 ≤ n ≤ N is fixed and S retains its
uniform without-replacement draw order; (3) Stage 2 covers
I
max
2 and the tests and scan satisfy Theorem 2 and Lemma 2;
and (4) the selected pair minimizes cost in C, with all-oracle
Algorithm 2 Certified Joint Threshold Search
Require: Grid Φ, each track ordered λ2,1 > · · · > λ2,L2
;
sample S ∼ SRSWOR(I, n) with oracle labels {yi}i∈S;
ε = 1 − τ ; failure budget δ; label-free Cost(·)
Ensure: Certified set C, deployed ϕˆ
1: C ← ∅; ηf ← δ/L1 ▷ Bonferroni across L1 tracks
2: for each track k = 1, .Where the paper says it
The reported cost model follows Eq. (1), using input-token and invocation counts and mean output lengths<ref:2611.08367#pg5> The token panel counts only oracle tokens<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
- Joint Threshold Selection for Cost Minimization
The CascadeFilter method jointly searches thresholds for a small language model operating on extracted evidence and full documents, formulating threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint
to select the least-cost certified configuration in a pre-specified grid.
This allows the system to choose the optimal balance between execution cost and accuracy by evaluating candidates based on their total token cost.
- Finite Population Certification for Accuracy Guarantee
The system employs Finite-Population Certification
using a sample drawn without replacement and a Waudby-Smith and Ramdas betting framework to bound oracle-relative error, ensuring that the selected configuration meets the target accuracy with probability of success of at least 1 minus delta.
- Hybrid Evidence and Routing Enhancements
The system improves evidence extraction and routing by combining Best Matching BM25 lexical retrieval with dense retrieval using reciprocal rank fusion RRF, reranking the resulting chunks, and incorporating embedding-neighborhood support into model confidence.
This enhances the candidate policies evaluated by the certification procedure.
- Dynamic Record Scoring via Routing Scores
The system uses a routing score that combines model confidence with agreement among nearby records in embedding space to nudge the base confidence by how strongly a record’s neighbors agree with its prediction,
allowing each record's effective threshold to track its difficulty without per-record tuning.
- Cost-Aware Execution Accounting
The system distinguishes between modeled selected-plan cost and actual same-batch cost with prediction reuse
using Equation (13), ensuring that the reported costs account for cached Stage 2 predictions and reused calibration labels
while avoiding double charging for preprocessed elements.
Abstract
Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs). Executing an expensive model on every document incurs substantial inference cost. Existing approaches reduce this cost by accepting predictions from cheaper models and forwarding uncertain records to an oracle. Task Cascades further varies document portions and operations, and provides an accuracy-guaranteed extension. However, its threshold calibration follows a prescribed adjustment sequence after cascade assembly, rather than minimizing cost over alternative combinations of stage thresholds. Motivated by this, we propose CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle. For a fixed cascade, we formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint, and select the least-cost certified configuration in a pre-specified grid. A finite-population calibration procedure bounds oracle-relative error after selection, using fixed-sequence tests on threshold tracks to allocate the multiple-testing budget. We further combine lexical and dense retrieval, reranking, and neighborhood-supported confidence to improve evidence extraction and routing. These components enhance the cascade while leaving its certification interface unchanged. Across six datasets and 13 evaluation settings, the reported results have the highest benchmark accuracy and macro-averaged F1 score in 12 settings each. These comparisons describe the reported operating points; the accuracy guarantee concerns agreement with the oracle.
Sources
- Beyond Linear LLM Invocation: An Efficient and Effective Semantic Filter Paradigm
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- Qwen3 Technical Report
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Multi-Objective Agentic Rewrites for Unstructured Data Processing
- Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
- SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees
- UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
- Streaming Model Cascades for Semantic SQL
- Selective Conformal Risk Control
Related papers
- Beyond Overlap: Hierarchical Graph Abstraction for Out-of-Core Vector Search
- EMA: Approximate Nearest Neighbor Search with General Attribute Filtering and Dynamic Updates
- Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database
- Samyama: A Unified Graph-Vector Database with In-Database Optimization, Agentic Enrichment, and Hardware Acceleration
- GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers
- Common-Neighbor-Count-Based Representative Possible World Finding on Uncertain Graphs