CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection
summary
The gist
Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611.08367#pg2> The objective is…
In short
CascadeFilter is a three-stage semantic filtering method designed to reduce execution cost while meeting accuracy targets for natural language predicates. It uses a small language model (SLM) on extracted evidence, followed by the full document SLM, and finally an oracle for unresolved records. The method finds cost-minimizing thresholds through a search that guarantees accuracy.
Key concepts
- CascadeFilter
- A three-stage semantic filtering approach where each stage uses a language model (SLM) on different inputs (evidence, full document) before forwarding unresolved items to an oracle. It jointly searches for optimal thresholds across these fixed stages to minimize total execution cost while maintaining a desired accuracy level.
- Oracle-Relative Error
- The accuracy guarantee is measured relative to an oracle, meaning the system's performance is compared against a perfect answer (the oracle) rather than ground truth labels. This allows for a composable guarantee where the probability of achieving a certain accuracy level ($ au$) is bounded by $1-\delta$.
- Cost-Minimizing Threshold Selection
- The core optimization involves selecting two thresholds for the cascade stages that minimize the token-aware cost, considering input/output counts and mean output lengths. Although the objective is generally nonconvex, the method finds a least-cost certified configuration within a specified search grid.
- Certification Guarantee
- The method provides a formal guarantee that after selection, there is a probability of achieving an accuracy level $ au$ or higher. This guarantee holds for oracle-relative error and ensures that the selected configuration minimizes cost under the constraints defined by the search procedure.
Terminology used across episodes
This episode discusses
- CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection · Paper Radio
- Beyond Linear LLM Invocation: An Efficient and Effective Semantic Filter Paradigm
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- Qwen3 Technical Report
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Multi-Objective Agentic Rewrites for Unstructured Data Processing
- Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
- SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees
- UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
- Streaming Model Cascades for Semantic SQL
- Selective Conformal Risk Control
The paper
CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection · Read on arXiv
The Hong Kong University of Science and Technology (Guangzhou) · University of New South Wales · Zhejiang Lab · Southwest Jiaotong University
Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs). Executing an expensive model on every document incurs substantial inference cost. Existing approaches reduce this cost by accepting predictions from cheaper models and forwarding uncertain records to an oracle. Task Cascades further varies document portions and operations, and provides an accuracy-guaranteed extension. However, its threshold calibration follows a prescribed adjustment sequence after cascade assembly, rather than minimizing cost over alternative combinations of stage thresholds. Motivated by this, we propose CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle. For a fixed cascade, we formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint, and select the least-cost certified configuration in a pre-specified grid. A finite-population calibration procedure bounds oracle-relative error after selection, using fixed-sequence tests on threshold tracks to allocate the multiple-testing budget. We further combine lexical and dense retrieval, reranking, and neighborhood-supported confidence to improve evidence extraction and routing. These components enhance the cascade while leaving its certification interface unchanged. Across six datasets and 13 evaluation settings, the reported results have the highest benchmark accuracy and macro-averaged F1 score in 12 settings each. These comparisons describe the reported operating points; the accuracy guarantee concerns agreement with the oracle.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection".
Jane: Here's the gist of the paper. Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs)<ref:2611.08367#pg2> The objective is to reduce the total execution cost while meeting a user-specified accuracy target<ref:2611.08367#pg3> CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle<ref:2611.08367#pg3> Pruning redundant reasoning tokens The paper proposes CascadeFilter, which processes each record using an SLM on extracted evidence, then the SLM on the full document if needed, and finally the oracle for unresolved records<ref:2611.08367#pg4> The method formulates cost-minimizing threshold selection for this fixed cascade<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> The search returns the least-cost certified configuration in a pre-specified grid<ref:2611.08367#pg5> How the training signal is built The authors formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint<ref:2611.08367#pg5> The objective and feasible set are generally nonconvex, but error decreases with the second-stage threshold when the first is fixed<ref:2611.08367#pg5> They organize candidates into monotone tracks and apply fixed-sequence testing within each track, allocating the failure budget across tracks<ref:2611.08367#pg6> The certificate controls overall oracle-relative error<ref:2611.08367#pg5> Where the gains come from The method integrates hybrid retrieval, reranking, batched evidence inference, and neighborhood-supported confidence into the cascade<ref:2611.08367#pg6> These components improve the candidate policies evaluated by the certification procedure<ref:2611.08367#pg6> The resulting method couples cost-aware selection with finite-population certification while retaining flexibility in evidence extraction and routing<ref:2611.08367#pg6> Certification and Guarantee The guarantee applies after selection and concerns oracle-relative error ε = 1 − τ<ref:2611.08367#pg5> The search must control the risk of selecting an infeasible configuration<ref:2611.08367#pg6> Theorem 3 establishes a composable accuracy guarantee where P(AccML (ϕˆ) ≥ τ) ≥ 1 − δ<ref:2611.08367#pg5> The selected pair minimizes cost in C, with all-oracle Algorithm 2 Certified Joint Threshold Search<ref:2611.08367#pg5> Experimental Analysis The experiments evaluate six datasets with 13 settings and study cascade structure, routing components, and calibration behavior<ref:2611.08367#pg6> The main table reports the highest reported benchmark accuracy and macro-averaged F1 score in 12 settings each<ref:2611.08367#pg10> CascadeFilter has the highest reported Accuracy and Macro-F1 in 12 of 13 settings, averaging 0.938/0.887<ref:2611.08367#pg10> The reported operating points describe the reported operating points<ref:2611.08367#pg10> Cost and Performance The reported cost model follows Eq.
Tom: (1), using input-token and invocation counts and mean output lengths<ref:2611.08367#pg5> The token panel counts only oracle tokens<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> CascadeFilter has higher reported mean Accuracy than the other baselines, but the cost ratios still require a common accounting ledger for preprocessing, calibration, caching, and deployment<ref:2611.08367#pg10> The reported equal-weight mean cost across the 13 settings is 2.968×10−4 per predicate–record<ref:2611.08367#pg10> Threats to Validity The certificate is, strictly, oracle-relative: the theorems bound P(AccML (ϕˆ) ≥ τ) ≥ 1− δ<ref:2611.08367#pg5> Changing calibration labels to ground truth changes the loss at every stage, including oracle fallback<ref:2611.08367#pg10> The reported violation counts and end-to-end cost accounting still require reconciliation<ref:2611.08367#pg10> The single point certificate is conservative, and the claim holds within the family and procedure frozen before calibration (Theorem 3, assumption 1)<ref:2611.08367#pg5> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10> The violation-count discrepancy and cached-cost accounting still require verification against run logs<ref:2611.08367#pg10> We presented CascadeFilter, which selects two thresholds for a fixed three-stage semantic filter to reduce modeled execution cost under an oracle-relative error constraint<ref:2611.08367#pg10> The optimization is generally nonconvex, but its error is monotone along each fixed-first-threshold track<ref:2611.08367#pg5> CascadeFilter returns the least-cost certified candidate, not a global optimum<ref:2611.08367#pg10> The reported ground-truth Accuracy of CascadeFilter meets 0.9 in every setting<ref:2611.08367#pg10> Future work includes other semantic operators and recalibration under distribution shift<ref:2611.08367#pg10> References
1: L.
Jane: Patel, S.
Tom: Jha, M.
Paper summary: Tom: So, we’re wrapping up on CascadeFilter, which is all about picking two thresholds for a three-stage semantic filter to cut down on execution cost while keeping an accuracy target in mind.
Jane: That sounds like they’ve figured out how to stop wasting compute time when you're dealing with huge amounts of text. So, who wrote this paper?
Lu: It was written by Patel and Jha and Pan and Gupta Asawa Guestrin and Zaharia at the VLDB Endowment in two thousand twenty-five <ref:2611.08367#pg4>.
Meng: Okay, so it’s a specific set of researchers working on document processing with these large language models. What’s the main deal they’re trying to prove?
Tom: They proved that you can optimize the total token cost by jointly searching for those two thresholds instead of just picking them one by one <ref:2611.08367#pg5>.
Jane: So, it’s not just about finding a better single model; it’s about structuring the entire filtering process more intelligently. What does this mean in practice?
Lu: It means they found a way to manage the trade-off between cost and accuracy under an oracle-relative error constraint <ref:2611.08367#pg5>.
Meng: Under an oracle-relative error constraint—that’s just a fancy way of saying they are measuring how much their results might differ from the absolute best possible answer, which is hard to know without using an oracle.
Tom: Exactly. And the result is that they return a candidate configuration that has the least cost while still meeting that error constraint, instead of just finding some random good setting <ref:2611.08367#pg10>.
Jane: That’s interesting because it suggests a systematic search approach rather than just guessing settings and hoping for the best <ref:2611.08367#pg5>.
Lu: Yeah, they found that even though their search space is generally not simple, the error along each path where you fix the first threshold actually goes down as you move to the second one <ref:2611.08367#pg5>.
Meng: But Tom, if we’re shipping this to production, we need a guarantee. What kind of guarantee are they giving us here?
Tom: They give a composable accuracy guarantee where there is at least one minus delta probability that the accuracy will be greater than or equal to tau, which is their chosen error threshold <ref:2611.08367#pg5>.
Jane: So, it’s not just a hopeful claim; they have a mathematical bound on how accurate their system should be based on what they can observe.
Lu: They use this theorem to control the risk of picking an infeasible configuration during their search <ref:2611.08367#pg6>. The selected pair minimizes cost in C, which is their cost model, using all-oracle Algorithm two Certified Joint Threshold Search <ref:2611.08367#pg5>.
Meng: So it’s a certified search mechanism for the best configuration they can find within that specific grid structure. That still sounds like a lot of bookkeeping for deployment.
Tom: It is bookkeeping, but it’s structured bookkeeping, and the cost model itself uses input tokens and invocation counts plus mean output lengths <ref:2611.08367#pg5>. The mean cost they report across all thirteen settings is about two point nine times ten to the negative four per predicate record <ref:2611.08367#pg10>.
Jane: That cost number helps ground it in reality for someone managing a budget on these big processing tasks. It shows that this structured approach
Conclusion: Tom: So, we're wrapping up on CascadeFilter, which is about picking two thresholds for a three-stage semantic filter to cut down on execution cost while keeping an accuracy target in mind.
Jane: That sounds like they’ve figured out how to stop wasting compute time when you're dealing with huge amounts of text. So, who wrote this paper?
Lu: It was written by Patel and Jha and Pan and Gupta Asawa Guestrin and Zaharia at the VLDB Endowment in two thousand twenty-five.
Meng: Okay, so it’s a specific set of researchers working on document processing with these large language models. What’s the main deal they’re trying to prove?
Tom: They proved that you can optimize the total token cost by jointly searching for those two thresholds instead of just picking them one by one.
Jane: So, it’s not just about finding a better single model; it’s about structuring the entire filtering process more intelligently. What does this mean in practice?
Lu: It means they found a way to manage the trade-off between cost and accuracy under an oracle-relative error constraint.
Meng: Under an oracle-relative error constraint—that’s just a fancy way of saying they are measuring how much their results might differ from the absolute best possible answer, which is hard to know without using an oracle.
Tom: Exactly. And the result is that they return a candidate configuration that has the least cost while still meeting that error constraint, instead of just finding some random good setting.
Jane: That’s interesting because it suggests a systematic search approach rather than just guessing settings and hoping for the best.
Lu: Yeah, they found that even though their search space is generally not simple, the error along each path where you fix the first threshold actually goes down as you move to the second one.
Meng: But Tom, if we’re shipping this to production, we need a guarantee. What kind of guarantee are they giving us here?
Tom: They give a composable accuracy guarantee where there is at least one minus delta probability that the accuracy will be greater than or equal to tau, which is their chosen error threshold.
Jane: So, it’s not just a hopeful claim; they have a mathematical bound on how accurate their system should be based on what they can observe.
Lu: They use this theorem to control the risk of picking an infeasible configuration during their search. The selected pair minimizes cost in C, which is their cost model, using all-oracle Algorithm two Certified Joint Threshold Search.
Meng: So it’s a certified search mechanism for the best configuration they can find within that specific grid structure. That still sounds like a lot of bookkeeping for deployment.
Tom: It is bookkeeping, but it’s structured bookkeeping, and the cost model itself uses input tokens and invocation counts plus mean output lengths. The mean cost they report across all thirteen settings is about two point nine times ten to the negative four per predicate record.
Jane: That cost number helps ground it in reality for someone managing a budget on these big processing tasks. It shows that this structured approach is significantly cheaper than other baselines they tested.
Lu: And it does show higher reported mean accuracy than the other methods they compared against. But be careful, it's not a global optimum; it’s just the least-cost certified candidate.
Meng: So, they found a solid method that is cheaper and more accurate than what we had before, but they’re still focused on this specific grid search setup. What happens when the data shifts?
Tom: The authors acknowledge that the certificate is strictly oracle-relative, meaning if you change your calibration labels to ground truth later on, the loss changes at every stage.
Jane: So they are flagging a limitation right there—the guarantee holds within the specific setup they calibrated it for. But this whole search structure is definitely a solid foundation for making LLM-based data processing more efficient and trustworthy.
Lu: Yeah, the next step for them is looking at other semantic operators and dealing with distribution shift, which means they’re thinking about how to make this search robust in real-world environments.
More episodes
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game
- 1905.10490-Finding new routes for integrating Multi-Agent Systems using Apache Camel
- 2401.10816-Co-Pilot for Health: Personalized Algorithmic AI Nudging to Improve Health Outcomes