Memoir: Learning, Verifying, and Evolving False-Positive Memories for Static Application Security Testing Tools

Memoir: Learning, Verifying, and Evolving False-Positive Memories for Static Application Security Testing Tools

Summary

This paper introduces Memoir, a memory-driven framework designed to identify false-positive (FP) alerts generated by Static Application Security Testing (SAST) tools. The authors note that while SAST tools are essential in modern secure software development, they frequently produce a large number of false-positive alerts, which impose substantial manual inspection costs and reduce developer trust. On the public CWE-Bench-Java benchmark, CodeQL reports 865 alerts across 120 Java projects, of which 95.66% are manually verified as false positives.

The paper identifies two primary challenges faced by existing FP reduction methods: (1) the difficulty in learning complex FP semantics, since false-positive alerts are recurring semantic phenomena that share common characteristics across different repositories, frameworks, and vulnerability categories; and (2) the lack of continuously updatable knowledge, as existing techniques rely on largely static knowledge that cannot be incrementally updated as new cases accumulate.

To address these challenges, Memoir reformulates SAST false-positive identification as a reusable semantic memory learning problem. The framework consists of two key modules:

1. Historical semantic memory construction: This module transforms historical FP alerts into structured semantic memories through LLM-guided annotation, pattern clustering, and memory synthesis. Each memory entry captures reusable security knowledge, including false-positive taxonomy, security invariants, taint-breaking semantics, and representative evidence. The memory repository is organized into three complementary levels: Taxonomy Memory (coarse-grained FP categories, CWE information, and semantic tags), Pattern Memory (reusable FP knowledge including security invariants, taint breakers, retrieval keys, verification rules, and representative examples), and Evolution Memory (historical verification feedback and usage statistics).

2. Memory-driven identification and evolution: For each new SAST alert, this module retrieves the most relevant memories and performs semantic verification against taxonomy consistency and security invariants before producing the final prediction. The framework employs a hierarchical retrieval strategy, first performing taxonomy-level filtering using CWE categories, rule types, and semantic tags, then ranking candidate memories according to multiple semantic signals including security invariant similarity, taint-breaker consistency, and retrieval-key matching. Progressive verification is performed through taxonomy consistency checks, security invariant validation, and LLM-based semantic reasoning. Verified cases are subsequently incorporated back into the memory repository, enabling the memory base to continuously evolve as new cases accumulate.

The paper evaluates Memoir on the public CWE-Bench-Java benchmark using CodeQL-generated alerts. The experimental results show that Memoir achieves an F1-score of 99.43% with a Recall of 98.88% and perfect Precision (100%), consistently outperforming all compared baselines including Zero-shot prompting, IRIS, conventional Retrieval-Augmented Generation (RAG), BugLens, and ZeroFalse. Specifically, compared with the strongest baseline ZeroFalse, Memoir improves the F1-score from 0.9183 to 0.9921, an absolute improvement of 7.38 percentage points.

The ablation study confirms that both pattern clustering and semantic verification are essential components. Removing verification decreases Precision from 1.000 to 0.920, while removing clustering causes the largest performance degradation, with F1-score dropping to 0.910. The study also shows that increasing retrieval depth from Top-1 to Top-5 consistently improves Recall and F1-score, with Top-5 achieving the best performance.

The cross-backbone generalization experiments demonstrate that Memoir maintains strong performance across different foundation models, including GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro, and DeepSeek-Coder-V2-Instruct, with F1-scores ranging from 0.9722 to 0.9921. This indicates that the main performance gain comes from the proposed memory-based framework rather than the intrinsic reasoning capability of a particular LLM.

An industrial case study on production software systems from a top IT company shows that the learned memory base generalizes effectively across different SAST tools without retraining. Memoir reduces false positives by 80.0% for CodeQL (from 30 to 6) and 94.6% for XCheck (from 149 to 8). Precision improves from 0.42 to 0.68 for CodeQL and from 0.65 to 0.85 for XCheck. The case study also reveals different levels of suppression across vulnerability categories, with SQL Injection showing the largest improvement (false-positive rate decreasing from 23.29% to 0.00%), followed by Path Traversal (from 46.97% to 3.03%), while Command Injection shows relatively limited improvement (from 35.56% to 28.89%).

The paper concludes that Memoir shifts LLM-assisted false-positive reduction from isolated alert adjudication to reusable and evolvable memory-guided reasoning, providing an effective and generalizable foundation for improving the practicality of modern SAST systems. The code and data are available at https://anonymous.4open.science/r/anon-repo-7x2z/.

Improvements to AI Systems:

1. Memory-Augmented False-Positive Filtering for SAST Tools

• The AI system can be enhanced with a persistent, structured memory repository (taxonomy, pattern, and evolution memories) that stores reusable security invariants, taint-breaking semantics, and verification rules from historical alerts.

• It can automatically classify new SAST alerts by retrieving and applying relevant memories, reducing manual review workload by up to 94.6% in production settings.

2. Hierarchical Retrieval and Progressive Verification Pipeline

• Implement a two-stage retrieval (coarse taxonomy filtering followed by fine-grained semantic ranking) combined with progressive verification (taxonomy consistency → invariant validation → LLM reasoning) to achieve near-perfect precision (100%) and recall (98.88%) on benchmark datasets.

3. Continuous Memory Evolution from Verified Cases

• The system can incrementally update its memory base after each verified alert, allowing it to adapt to new frameworks, vulnerability patterns, and tool-specific quirks without full retraining—enabling cross-tool generalization (e.g., from CodeQL to XCheck) with zero additional training data.

4. Cross-Model Robustness via Framework-Centric Design

• By decoupling performance from the underlying LLM’s reasoning ability, the improved system maintains high F1-scores (0.9722–0.9921) across GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro, and DeepSeek-Coder-V2, making it deployable with cost-effective or open-source models.

5. Actionable Vulnerability-Specific Suppression Insights

• The system can provide per-category false-positive reduction analytics (e.g., SQL Injection FP rate from 23.29% to 0.00%, Path Traversal from 46.97% to 3.03%), enabling developers to prioritize fixes where the memory base is most effective and flag categories needing more memory examples.

6. Reusable Security Knowledge Transfer Across Projects

• The AI can act as a shared security knowledge base across an organization, transferring learned FP patterns from one codebase to another, reducing the cold-start problem for new projects and improving developer trust in SAST outputs.