Guardian Crawler: Retrieval-First Knowledge Discovery with Bounded LLM Augmentation for Noisy Web Intelligence

Guardian Crawler: Retrieval-First Knowledge Discovery with Bounded LLM Augmentation for Noisy Web Intelligence

Abstract:

Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content. We present Guardian Crawler, a reproducible retrieval-first testbed for controlled experiments on knowledge discovery and evidence-grounded summarization over synthetic web-like corpora. The architecture combines BM25 retrieval with risk-aware, embedding-augmented, and hybrid reranking, followed by constrained retrieval-augmented generation with explicit document citations. Experiments on a synthetic 900-document corpus and 10 queries produced the highest descriptive retrieval scores under risk-based reranking, with P@10 = 1.00 and NDCG@10 = 0.94, compared with 0.94 and 0.81 for BM25. The best hybrid and BM25+Semantic configurations reached NDCG@10 values of 0.94 and 0.88, respectively. All 41 evaluable generated bullets passed the lexical coverage threshold; an automated LLM judge classified 36 as supported, one as partially supported, and four as unsupported. These results demonstrate the feasibility of Guardian Crawler as a controlled testbed but do not establish statistical superiority, human-validated faithfulness, or transfer to live-web investigative environments.

Introduction:

Web-based knowledge discovery requires relevant evidence to be collected, normalized, ranked, and evaluated before downstream analysis can be considered reliable. This is particularly important in sensitive domains, where relevant information may be distributed across official bulletins, news reports, reposts, incomplete descriptions, and unrelated content. Ranking or synthesis errors can obscure important evidence or produce unsupported conclusions.

Missing-child and trafficking-related information provides a demanding setting for studying this problem, but unrestricted experimentation on live data creates substantial ethical and privacy concerns. Guardian Crawler therefore uses a synthetic, link-connected mini-web that emulates bulletin, news, noise, and irrelevant pages while preserving control over corpus composition, metadata, and relevance rules.

Guardian Crawler is a controlled experimental platform rather than a deployed investigative system. It uses BM25 as a transparent lexical baseline and limits large language models to bounded roles in corpus construction, document tagging, citation-bearing summary generation, and automated grounding assessment. Risk-aware and embedding-based signals are applied as rerankers over a shared candidate pool rather than replacing the retrieval pipeline with an end-to-end generative model.

The study addresses three questions: whether a synthetic mini-web can support reproducible retrieval experiments; how risk-aware, embedding-augmented, and hybrid reranking behave relative to BM25 under fixed conditions; and whether constrained generation can produce citation-bearing summaries whose support can be evaluated separately from retrieval quality.

Across 10 queries and 900 documents, risk-based reranking produced the highest observed P@10 and NDCG@10 values, while BM25+Semantic reranking produced the strongest semantic-alignment diagnostic. Automated grounding evaluation also revealed a difference between lexical coverage and judge-assessed support. These are descriptive results from a small, domain-specific synthetic benchmark and should not be interpreted as statistically supported evidence of general retrieval superiority or operational effectiveness.

The paper contributes: (1) a reproducible synthetic testbed for controlled studies of noisy web-like retrieval; (2) a modular retrieval, reranking, generation, and evaluation pipeline with bounded LLM roles; and (3) a comparative analysis that separates retrieval effectiveness from automated grounding assessment.

System Design and Architecture:

Three design goals shape the Guardian Crawler system. The first is reproducibility: experiments should be rerunnable over a controlled corpus with known document types, link structure, and relevance heuristics. The second is modularity: retrieval, generation, tagging, embeddings, and evaluation should be independently inspectable and replaceable. The third is boundedness: LLMs should be used where they plausibly add value, but not as opaque substitutes for crawling, indexing, or ranking.

The system architecture is organized as an end-to-end but modular pipeline. At a high level, the configuration layer sets shared defaults; the mini-web generator creates the corpus; the HTML extractor standardizes raw pages; the crawler collects them; the BM25 index provides first-stage retrieval; the tagging layer assigns coarse page types; the risk reranker promotes harm-relevant lexical signals; the semantic reranker uses embeddings to refine BM25 candidates; the ranking runner compares all conditions fairly; the RAG layer converts retrieved evidence into citation-linked bullets; and the evaluation layer scores both retrieval and grounding.

Synthetic Corpus and Acquisition:

The mini-web generator creates a static HTML corpus from structured case seeds and assigns pages to four high-level document types: bulletin, news, noise, and irrelevant. Hub pages and synthetic hyperlinks produce a crawlable graph with same-type linking bias and enforced reachability. Using generated rather than live content preserves experimental control and reduces privacy risk while retaining retrieval challenges such as mixed relevance, redundancy, and distracting content.

A breadth-first crawler operates within the synthetic site, normalizes URLs, enforces depth and page limits, and removes duplicate content through hashing. A shared HTML extraction component produces a consistent representation of each page's title, visible text, and outbound links for crawling and indexing.

Retrieval and Reranking:

BM25 indexes normalized title and body text and serves as the common lexical baseline. Risk-aware reranking combines normalized BM25 scores with a bounded score derived from a 28-entry harm-related lexicon. Embedding-based reranking computes similarity between query embeddings and precomputed document embeddings. Hybrid conditions combine lexical, risk, and semantic signals.

All ranking conditions operate on the same BM25-derived candidate set. In the reported experiment, the candidate pool contains all 900 corpus documents. The comparison therefore isolates scoring and reranking behavior rather than first-stage candidate recall or retrieval efficiency. The embedding component is a reranker and not an independent dense-retrieval baseline; every reported BM25+Semantic condition retains a nonzero BM25 contribution.

Bounded LLM Components:

Gemini is used in four separate roles: synthetic corpus construction, post-crawl page tagging, citation-bearing summary generation, and automated grounding assessment. Tagging maps each page to bulletin, news, noise, or irrelevant using a fixed JSON schema. The generation stage receives only selected evidence passages and must return a limited set of bullets with supporting document identifiers. Outputs may abstain when evidence is insufficient or validation fails. The judge receives the query, generated bullet, and cited evidence and assigns supported, partially supported, or unsupported labels with an issue category. Separate generator and judge instances provide procedural separation, but they do not provide model-family independence because Gemini participates in multiple stages.

Evaluation Architecture:

Retrieval effectiveness and grounding are evaluated separately. Retrieval qrels are generated from synthetic metadata and query-specific relevance rules. Grounding is assessed first through deterministic lexical coverage and then through the LLM judge. Neither process constitutes human validation: qrels are rule-derived, and grounding labels are generated automatically.

Experimental Setup:

The canonical experimental configuration employs a synthetic corpus comprising 900 documents, a candidate pool of 900 documents per query, and a set of 10 queries. The comparison framework includes a BM25 baseline, four BM25+risk configurations with α ∈ 0.25, 0.5, 0.75, 1.0, three BM25+semantic configurations with λ ∈ 0.2, 0.4, 0.6, and three hybrid lexical–risk–semantic configurations with weight triplets (0.6, 0.2, 0.2), (0.5, 0.25, 0.25), and (0.4, 0.3, 0.3). All configurations are evaluated under a unified reranking protocol, ensuring that each condition operates on the same BM25-derived candidate pool for every query.

Implementation details and reproducibility:

Retrieval is performed over a closed synthetic corpus, and BM25 is the only first-stage retriever. All risk-aware, semantic, and hybrid conditions rerank a fixed BM25 candidate list, ensuring that later comparisons reflect scoring differences rather than different recall frontiers. Query terms are treated as a set rather than a multiset during BM25 scoring.

The BM25 implementation uses default parameters k1 = 1.2, b = 0.75, and min doc tokens = 3. IDF is computed as log((N − df + 0.5)/(df + 0.5)) + 1, where N is the total number of documents in the corpus (900) and df is the document frequency. It sums the per-term BM25 contributions over distinct query tokens. Tokenization lowercases the text and extracts alphanumeric tokens using the regular expression [a-z0-9]+.

The risk-aware reranker uses a weighted lexicon with 28 entries. For each document d, raw risk is computed over title and body as RiskRaw(d) = Σt wt min(tft(d), 2), where each term match count is capped at 2. The bounded risk score is then RiskScore(d) = RiskRaw(d)/(RiskRaw(d) + 3). BM25 scores are min-max normalized within the candidate pool, whereas RiskScore is already bounded in [0,1]. The risk reranker uses the following three scores:

Scorerisk(d, q) = (1 − α) * BM25(d, q) + α * Risk(d)

Scoresem(d, q) = (1 − λ) * BM25(d, q) + λ * Sem(d, q)

Scorehybrid(d, q) = wB * BM25(d, q) + wR * Risk(d) + wS * Sem(d, q)

where wB, wR, and wS are the lexical, risk, and semantic weights, respectively, and Sem(d, q) is the normalized embedding-based query-document similarity.

Document embeddings are built offline from the BM25 index rather than directly from raw crawl files. For each doc id, the system embeds a canonicalized title-plus-body text representation, fingerprints that representation, and writes both JSONL vectors and a manifest containing the model, task types, dimensionality, maximum character budget, canonicalization version, and similarity mode.

Ranking conditions and candidate-pool control:

The four compared ranking conditions are:

1. BM25 as the baseline lexical retriever;

2. BM25 + Risk at α ∈ 0.25, 0.5, 0.75, 1.0;

3. BM25 + Semantic at λ ∈ 0.2, 0.4, 0.6; and

4. BM25 + Risk + Semantic with weight triplets: (0.6, 0.2, 0.2), (0.5, 0.25, 0.25), and (0.4, 0.3, 0.3).

The candidate set is fixed to the top 900 BM25 retrieval results, which in this experiment coincides with the full corpus. This shared-candidate design is critical, as it isolates reranking behavior by keeping the underlying evidence pool constant across all experimental conditions. Semantic reranking is implemented strictly as a second-stage procedure. It does not replace BM25 as the initial retrieval step; instead, it only reorders the BM25-derived candidate set based on dense similarity between the query embedding and the corresponding precomputed document embeddings. Hybrid ranking similarly integrates lexical, risk-sensitive, and semantic signals over this same candidate list.

Retrieval metrics and query relevance judgments (qrel) construction:

Retrieval effectiveness is evaluated primarily using P@10 and NDCG@10, which measure the proportion of relevant documents in the top 10 and rank-sensitive graded retrieval quality, respectively. When embedding-based representations are enabled, the system additionally reports SR@10 and SR-Gap@10, the semantic closeness of relevant top-10 results to the query and how much more semantically similar the relevant top-10 documents are as compared to non-relevant ones, respectively. SR@10 is defined as the qrel-weighted mean similarity over relevant documents appearing in the top ranks, whereas SR-Gap@10 captures the difference in similarity scores between relevant and non-relevant documents among the top-ranked set. These embedding-aware measures are intended as diagnostic indicators rather than substitutes for standard ranked-retrieval metrics.

Relevance judgments are instantiated as rule-derived qrels rather than pooled human assessments. Relevance grades take values in 0, 1, 2 and are generated from metadata.jsonl and query relevance rules.json. For example, in query set Q1, bulletin and news pages are assigned a base relevance grade of 2, non-relevant pages receive a grade of 0, and noise pages receive a grade of 1 unless they contain at least two predefined cue terms (e.g., missing, child, last seen) in the concatenated page text, in which case the page's relevance is boosted according to the specified query rule. This closed-world labeling procedure yields complete query–document relevance annotations for the synthetic corpus and ensures exact repeatability, but it is not directly comparable to large-scale human relevance assessment. Because both the qrel rules and the risk lexicon encode related domain cues, their unmeasured overlap may partially favor risk-aware ranking. The resulting qrels were not validated through human relevance assessment.

Grounding evaluation protocol:

Grounded generation is evaluated in two stages. First, a deterministic lexical support check measures whether the content tokens in each cited bullet are covered by the concatenated cited evidence. Let Cb and Ce denote token-multiset counts for the bullet and its cited evidence, respectively. Token coverage is defined as Cov(b, e) = Σt min(Cb(t), Ce(t)) / Σt Cb(t). A bullet is counted as lexically supported if Cov(b, e) ≥ τ, where the default threshold is τ = 0.35. Only evaluable cited bullets are included: bullets must contain non-empty text and non-empty supporting doc ids. Rows marked with failure stage, abstained, or lacking evaluable cited bullets are excluded from lexical support aggregates.

Second, an optional LLM-as-judge protocol evaluates support more semantically. The judge sees the query, the bullet text, and cited evidence only, and returns a structured JSON label with supported, partially supported, or unsupported, together with a confidence score, an issue type, and a short reason. The issue taxonomy includes none, overstatement, citation mismatch, not in evidence, ambiguous, and multi hop inference. Judge aggregation reports the strict judge-assessed support rate (supported only), lenient judge-assessed support rate (supported + partially supported), unsupported claim rate, and issue histograms. The judge is run separately from the generator and cached on disk for reproducible reruns. These labels are automated model assessments and were not validated by human annotators.

Reproducibility and statistical scope:

A unified evaluation pipeline regenerates the qrels, reruns the ranking comparisons, and computes retrieval and grounding summaries, ensuring that all reported results originate from the same auditable workflow. The experiment does not include paired significance tests, bootstrap-based confidence intervals, or permutation tests for either retrieval or grounding metrics. Consequently, the macro-level differences reported in Section 5 should be viewed as comparative outcomes on a controlled benchmark, rather than as statistically supported claims of general superiority beyond the specific experimental conditions considered here. This limitation is especially salient because the evaluation set comprises only 10 queries, which is adequate for structured ablation studies but small relative to standard information retrieval benchmarking practice.

Representative pipeline example:

A representative end-to-end example is query Q1: missing child last seen near bus stop. Under the BM25 baseline, the highest-ranked documents are, in order, D0519, D013…

Based on this paper, here are specific improvements for AI systems:

1. Risk-Aware Retrieval Augmentation

• Add a domain-specific risk lexicon scoring layer to retrieval systems, where documents with harm-relevant terms (e.g., missing, last seen, abducted) get bounded score boosts (capped at 2 occurrences per term, normalized via RiskScore = raw/(raw+3))

• Implement this as a reranker over BM25 candidates rather than replacing lexical retrieval, allowing risk signals to reorder top results without sacrificing recall

• The improved system can prioritize safety-critical evidence in sensitive domains (e.g., missing persons, trafficking) over topically similar but less consequential documents

2. Hybrid Lexical-Semantic-Risk Ranking with Tunable Weights

• Combine BM25, embedding similarity, and risk scores using configurable weight triplets (e.g., 0.4/0.3/0.3) over a fixed candidate pool

• Use min-max normalization for BM25 scores and pre-bounded risk scores (0-1) to ensure fair mixing

• The improved system can adapt to query types: explicit queries benefit from risk weighting, paraphrastic queries benefit from semantic reranking, and hybrid settings capture both, achieving NDCG@10 improvements from 0.81 to 0.94

3. Two-Stage Grounding Verification with Lexical + LLM Judge

• Implement a deterministic lexical coverage check first (token overlap ratio ≥ 0.35 threshold), then an LLM-as-judge for semantic support classification (supported/partially supported/unsupported) with issue taxonomy (overstatement, citation mismatch, not in evidence, multi hop inference)

• The improved system can distinguish lexical overlap from true faithfulness—catching cases where a bullet reuses evidence vocabulary but over-aggregates across sources or introduces unwarranted inferences (as shown by 100% lexical support vs. 87.8% judge-assessed support)

4. Abstention Mechanism for Low-Confidence Generation

• Add explicit abstention logic to RAG generation: when evidence is insufficient or validation fails, output abstained rather than forcing a bullet

• The improved system reduces hallucination risk in investigative contexts by refusing to generate unsupported claims, with failure stage tracking for auditability

5. Bounded LLM Role Separation

• Restrict LLMs to four bounded functions: corpus construction, document tagging (fixed JSON schema), citation-bearing summary generation (limited bullet count with doc IDs), and automated grounding assessment

• Never use LLMs as opaque replacements for crawling, indexing, or first-stage ranking

• The improved system maintains transparency and reproducibility while leveraging LLM strengths in synthesis and evaluation, with procedural separation between generator and judge instances

6. Semantic Alignment Diagnostics

• Add SR@10 and SR-Gap@10 metrics that measure semantic closeness of relevant top-10 results to query and the similarity gap between relevant vs. non-relevant documents

• The improved system can detect when ranking optimizes lexical relevance but sacrifices semantic alignment, enabling targeted tuning for paraphrase-heavy queries

7. Rule-Derived Qrels with Closed-World Labeling

• Generate complete query-document relevance judgments from metadata and query-specific rules (grades 0/1/2), enabling exact repeatability without human annotation

• The improved system supports controlled ablation studies where every document's relevance is known, allowing precise isolation of reranking effects (as demonstrated with a 900-document, 10-query benchmark)

8. Case-Conflation Detection in Multi-Document Synthesis

• Implement explicit checks for cross-document merging errors, particularly when cases share lexical similarity (e.g., same names, locations, or descriptions)

• The improved system can flag multi hop inference issues (the most common failure mode in the paper) and either abstain or request additional evidence before combining facts from multiple sources

9. Reproducible Pipeline with Cached Judge Outputs

• Cache LLM judge results on disk for rerunnable experiments, with fingerprinting for document embeddings and versioned manifests (model, task types, dimensionality, character budget)

• The improved system enables auditable, repeatable evaluations where retrieval and grounding metrics can be recomputed from the same workflow without re-running expensive LLM calls

10. Risk-Score Saturation Control

• Cap term frequency contributions at 2 per term and use a saturating transform (raw/(raw+3)) to prevent over-weighting documents with excessive risk lexicon matches

• The improved system avoids ranking distortion from keyword-stuffed pages while still promoting genuinely risk-relevant content, as shown by the α=1.0 configuration achieving P@10=1.00 without degrading precision

Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content. We present Guardian Crawler, a reproducible retrieval-first testbed for controlled experiments on knowledge discovery and evidence-grounded summarization over synthetic web-like corpora. The architecture combines BM25 retrieval with risk-aware, embedding-augmented, and hybrid reranking, followed by constrained retrieval-augmented generation with explicit document citations. Experiments on a synthetic 900-document corpus and 10 queries produced the highest descriptive retrieval scores under risk-based reranking, with P@10 = 1.00 and NDCG@10 = 0.94, compared with 0.94 and 0.81 for BM25. The best hybrid and BM25+Semantic configurations reached NDCG@10 values of 0.94 and 0.88, respectively. All 41 evaluable generated bullets passed the lexical coverage threshold; an automated LLM judge classified 36 as supported, one as partially supported, and four as unsupported. These results demonstrate the feasibility of Guardian Crawler as a controlled testbed but do not establish statistical superiority, human-validated faithfulness, or transfer to live-web investigative environments.