Mind the Hook: Source-Level Auditing of Privacy Defenses in Retrieval-Augmented Generation

Paper: Mind the Hook: Source-Level Auditing of Privacy Defenses in Retrieval-Augmented Generation by Yanhang Li, Zhichao Fan, and Zexin Zhuang (arXiv:2608.09001v1 [cs.CR], 10 Aug 2026)

The paper addresses a fundamental interpretability problem in evaluating privacy defenses for retrieval-augmented generation (RAG): Black-box privacy scores for retrieval-augmented generation (RAG) are difficult to interpret unless the audited defense's active pipeline hook is known. The authors argue that "A RAG privacy defense can be present in a codebase yet inactive on the leakage channel being measured. If the implementation modifies only retrieval, a privacy metric scored on the generated text cannot see whether the defense works; if it modifies only the response, a metric scored on the retrieval/index path cannot see it either. Most current evaluations of RAG privacy defenses report only black-box leakage numbers, leaving this kind of channel–implementation mismatch invisible."

The paper proposes a three-step active-path audit protocol:

1. Source-level hook inventory: "For each defense module, inspect the implementation and classify which pipeline hook(s) it modifies: retrieval scoring (noise on similarity scores or top-k truncation), retrieved-content rewriting, generation (token-level noise, paraphrasing, or post-generation masking), or none. A hook whose body is TODO-flagged or returns its input unchanged is recorded as inactive."

2. Metric-to-channel map: "For each metric, identify the channel it observes. NELstrict is computed on generated text and moves only if a defense modifies the response; a black-box membership-inference AUC is designed to move when the retrieval/index path changes... A metric is interpretable on a defense only if its active hooks overlap the metric's channel; otherwise a shift is nuisance or a different mechanism."

3. Canary validation: "We validate generated-text effects with an out-of-vocabulary canary: inject unique strings outside any defense's substitution dictionary, exact-match score the response, and report recovery. The canary is not scored by the heuristic classifier—the score path is bypassed—so it cannot be inflated by scorer-vocabulary entanglement."

The audit is applied to one fixed open-source RAG stack with:

• Language model: Phi-3-mini-4k-instruct at 4-bit NF4 quantization

• Retriever: FAISS over all-MiniLM-L6-v2 sentence-transformer embeddings, top-k=5, greedy decoding

• Six attack families: four extraction baselines (PS, PM, IKEA, DEAL), the agent attacker RAG-Thief (reproducing Jiang et al. [4]), and a black-box membership-inference attack (MI; 10 member + 100 non-member docs per corpus, pairwise-ranking AUC) inspired by Shokri et al. [5]

• Six defense implementations: No-Defense baseline; DP-R (Laplace noise on retrieval scores, ε∈ 1,5); CA-DP (composition-aware DP budget); Private-RAG (DP-on-scores filter inspired by Koga et al. [11]); PAD (regex masker for email/identifier patterns); LPRAG (deterministic entity-substitution simplification of He et al. [6] without local-DP)

• Three corpora: Synthetic-Corp (programmatic business corpus), Synthetic-Email (50 email-formatted documents with Enron-era executive surnames), PubMedQA (first 50 of 1,000 real abstracts)

• Grid: 6 × 6 × 3 × 2 × 2 = 432 cells (one run per cell)

The three DP-style defense implementations in our benchmark act only on retrieval scores: their generation hooks are TODO-flagged pass-throughs (Table I). Specifically, DP-R's source literally reads 'For now, return unmodified response (placeholder)', with analogous paths in CA-DP and Private-RAG.

This explains the observed pattern: they reduce MI AUC on Synthetic-Corp from 70.5 to 41.5–59.3 while tracking the undefended baseline on NELstrict, which is scored on generated text. The paper emphasizes: "This is not in tension with the NELstrict null: MI's black-box response score is retrieval-sensitive (it shifts when the retrieval hook perturbs which documents surface), whereas NELstrict counts named entities in that same text, which the stubs do not transform."

The authors stress: The silent-stub finding is not a claim that the cited source defense papers contain stubs; it is that a benchmark wrapper can silently evaluate an implementation whose active path does not overlap the metric it reports.

Using an out-of-vocabulary email canary of the form operator N@acmetest-internal.canary (50 canaries per seed, 3 seeds, 50-query targeted stress-test pool), the results show: "No-Defense recovers 53/150 (35.3%) and LPRAG recovers 0/150. A Fisher one-sided test... gives p < 10−14. This independently validates the end-to-end email effect: LPRAG cuts the EMAIL REAL mean from 2.25 to 0.10 per run (95.6%, canary-validated above)."

However, the person-name reduction is explicitly not certified: "Its PERSON NAME mean drops from 1.75 to 0.00 (100%), but this is definitional, not validated: the classifier's PLACEHOLDER NAME set is LPRAG's NAME VOCABULARY, so every name LPRAG writes is excluded from PERSON NAME by construction. We do not claim coverage of person-name leakage under LPRAG; an analogous OOV person-name canary is future work."

Regex masking (PAD) removes email-format strings on Synthetic-Email but does not reduce person-name leakage in our setup. Specifically: "On Synthetic-Email, No-Defense emits 5.30 EMAIL REAL and 4.00 PERSON NAME items/run. PAD drives EMAIL REAL to 0.00 but its PERSON NAME count sits at 6.15 (+2.15 vs No-Defense), so aggregate NELstrict only drops 9.30→6.15: the regex masker blocks the email family it targets but is not a general person-entity defense."

The paper introduces a two-tier heuristic named-entity leakage classifier: "We release a ∼50-line rule-based regex+lexicon classifier (no second LM, whose behavior would become part of the measurement) that buckets each extracted item into one of eleven types. NELstrict counts only EMAIL REAL and PERSON NAME and is the headline metric; it excludes PERSON NAME AMBIGUOUS and ORG TITLE (∼48% of items), so it undercounts leakage."

The metric has limited validation: "audited at 20/20 on an informal 10-per-class positive spot-check (Clopper–Pearson one-sided 95% lower bound on 10/10 ≈ 0.74), not a stratified precision estimate and not a recall audit, hence a lower bound under a stronger attacker."

The authors are explicit about scope: The audit methodology is general; the numeric findings are case-study evidence on one stack, not a universal ranking of RAG privacy defenses. Key limitations include:

• "Each of the 432 raw cells is one run, so every bootstrap interval is within-instantiation only: it captures variance across pooled runs but not over alternative synthetic-corpus realizations or attack-query samples, and is not a population-level interval (the canary sub-experiment of §VI-A is the only one with three independent seeds)."

• The stack is fixed (one LM, embedder, top-k); a larger model might respond differently to DP-retrieval noise.

• The five extraction attacks are bounded fixed-query baselines; stronger adaptive attackers would plausibly raise NELstrict.

• Two of three corpora are synthetic, and the 'Enron'-labeled runs are synthetic emails from a generator that hard-codes Enron-era surnames.

• The channel-split findings are measurement reports about our three benchmark implementations of DP-class defenses, not claims about DP-on-RAG theory or upstream source papers.

• "NELstrict is precision-only; MI is not identified... The MI score correlates with, but is not unique to, membership—defenses that shorten or mask output could lower MI AUC via surface-corruption nuisance—so we phrase the channel split narrowly as 'reduced success on this MI protocol' and read MI as channel-location evidence only."

The paper lists three contributions: "(1) Active-path audit protocol. A three-step procedure: hook inventory, metric-to-channel map, and canary validation of generated-text effects (§III). (2) Case study and silent-stub failure mode. On a 432-cell grid, the protocol catches a silent-stub failure mode in three DP-style implementations that a black-box benchmark would miss; Table I documents the active hooks. (3) Canary triangulation. A targeted out-of-vocabulary email canary (3 × 50 trials) yields exact-match evidence (53/150 vs. 0/150) for the one defense whose generation hook is end-to-end, and explicitly does not certify the person-name sub-class, which remains scorer-entangled."

The paper's central message: Before asking whether a RAG privacy defense works, ask where it acts. The three takeaways are: (1) the audit catches a silent-stub failure mode where black-box benchmarks would misreport DP-style implementations as MI-reducing privacy defenses when they do not perturb generated text; (2) heuristic scorers need canary triangulation to avoid scorer-vocabulary entanglement; (3) regex masking is channel-specific. The authors state: We do not claim a ranking of defense families or generalization beyond the audited stack.

Based on the paper, here are the specific improvements I can make to AI systems and what the improved system can do:

Improvement 1: Implement active-path hook auditing for RAG privacy defenses

• I can add a source-level inspection layer that inventories which pipeline hooks (retrieval scoring, content rewriting, generation, or none) a defense actually modifies before evaluating it.

• The improved system can: detect silent-stub failures where a defense is present in code but inactive on the measured leakage channel; automatically flag TODO-flagged or pass-through hooks; and report channel–implementation mismatches instead of producing misleading black-box privacy scores.

Improvement 2: Build metric-to-channel mapping for interpretable privacy evaluation

• I can create a mapping that identifies which observation channel each privacy metric uses (e.g., NELstrict on generated text vs. MI AUC on retrieval/index path).

• The improved system can: explain whether a reported privacy improvement is due to the defense's active mechanism or a nuisance shift; avoid claiming privacy defense works when the metric and defense operate on different channels; and provide channel-aligned interpretations of score changes.

Improvement 3: Integrate canary-based validation to prevent scorer-vocabulary entanglement

• I can implement out-of-vocabulary canary injection (e.g., unique email strings outside any substitution dictionary) with exact-match scoring, bypassing heuristic classifiers.

• The improved system can: independently verify end-to-end effects of defenses on generated text; detect when a defense's apparent success is definitional (e.g., a classifier's placeholder set matching the defense's vocabulary); and certify specific leakage channels (like email) while explicitly not certifying entangled sub-classes (like person names).

Improvement 4: Add channel-specific defense capability assessment

• I can classify defenses by which leakage families they actually block (e.g., regex masking for emails vs. entity substitution for names) based on active hooks and canary results.

• The improved system can: predict which attack families a defense will mitigate before running expensive benchmarks; identify when a defense reduces one leakage type while increasing another (e.g., PAD raising PERSON NAME by 2.15); and recommend defense combinations for multi-channel privacy requirements.

Improvement 5: Implement variance-aware and seed-controlled evaluation

• I can add multi-seed experimental design and within-instantiation bootstrap intervals, distinguishing single-run results from population-level claims.

• The improved system can: report confidence bounds that reflect actual experimental replication (not just pooled runs); flag when findings are case-study evidence vs. generalizable; and avoid overclaiming from 432 single-run cells by highlighting which results have independent seed validation.

Improvement 6: Build a leakage-type-aware scoring system

• I can implement a two-tier heuristic classifier (regex+lexicon, no second LM) that buckets extracted items into fine-grained types, with explicit precision-only auditing and lower-bound reporting.

• The improved system can: undercount leakage conservatively (excluding ambiguous types) to avoid false positives; report Clopper-Pearson confidence intervals on classifier precision; and separate validated channels (email) from unvalidated ones (person names) in privacy reports.

Improvement 7: Add adaptive-attacker stress testing for robustness

• I can incorporate stronger adaptive attackers beyond fixed-query baselines, as the paper notes these would plausibly raise leakage scores.

• The improved system can: simulate attackers that adapt to defense mechanisms; provide upper-bound leakage estimates rather than optimistic fixed-query baselines; and identify which defenses degrade under adaptive pressure.

Improvement 8: Implement channel-split reporting for DP-style defenses

• I can separate retrieval-path effects from generation-path effects when reporting privacy metrics.

• The improved system can: show that DP-on-retrieval-scores reduces MI AUC (retrieval channel) while leaving NELstrict unchanged (generation channel); prevent misreporting of DP-style implementations as privacy-preserving when they only affect retrieval; and clarify that such defenses do not perturb generated text.

Black-box privacy scores for retrieval-augmented generation (RAG) are difficult to interpret unless the audited defense's active pipeline hook is known. We propose an active-path audit: inventory source-level hooks over retrieval, retrieved content, and generation; map each metric to the leakage channel it observes; and validate generated-text effects with exact-match canaries. In our benchmark reimplementations, the DP-style defenses modify retrieval scores only: their generation hooks are TODO-flagged stubs that return responses unchanged. This active path explains why they affect membership-inference behavior but track No-Defense on generated-text named-entity leakage, measured by NEL strict. By contrast, the end-to-end LPRAG path is canary-validated on the email channel, recovering 53/150 canaries under No-Defense and 0/150 under LPRAG. These findings concern our reimplementations on our stack, not released defenses or defense families; the contribution is a methodology and case study, not a universal ranking