Security and Privacy Taxonomy Generation from Mobile App Reviews

TaxoScale is a four-step pipeline that automatically expands an expert-defined privacy and security taxonomy from large-scale mobile app reviews. The paper addresses two core challenges: (1) the lack of a large-scale, comprehensive open corpus of privacy- and security-related app reviews, and (2) the scalability limitations of existing LLM- and clustering-based taxonomy construction methods, which are designed for scientific corpora of a few thousand documents and cannot handle app review collections numbering in the hundreds of thousands.

To build the corpus, the authors aggregated 18.63M reviews from three sources (AI-powered app reviews, the MobileRec dataset, and teacher-approved app reviews) across 11.22K apps in 48 categories. They then built an LLM-based classifier, evaluating ten open-source LLMs (4B–70B parameters) across eight prompt templates against a human-annotated benchmark of 300 reviews (Cohen's κ = 0.87). Llama-3.3-70B achieved the best F1 score (0.9700) under the few-shot (permissions+rules) template, which was applied to all 18.63M reviews, yielding 601,257 privacy- and security-related reviews. Next, an extractive summarizer (Llama-3.3-70B) was applied to isolate concern-describing spans, achieving a Jaccard similarity of 0.8511 and token-level F1 of 0.8187 against a 140-review benchmark, producing clean concern-level pseudo-labels.

TaxoScale's pipeline consists of four steps: (1) embedding pseudo-labels with Qwen3-Embedding-8B and assigning them to the closest seed taxonomy node (the two-level privacy taxonomy of Akgul et al. (2024) with 20 top-level categories) using cosine similarity with threshold θ = 0.5; (2) Recursive Hierarchical Clustering, which alternates top-down k-means partitioning with bottom-up Ward agglomerative merging (applied to centroids, not raw embeddings, for tractability) and recurses on subclusters larger than τ = 40,000; (3) binary-to-multiway tree conversion via height-tolerance collapsing with δleaf = 10 and ϵ = 20; and (4) LLM-based node naming using Qwen3-235B-A22B-2507, which generates 2–6 word summaries conditioned on the node's pseudo-labels (or child names for internal nodes), the parent path, and sibling names.

TaxoScale outperforms four strong baselines (Chain-of-Layers, TaxoCom, TaxoAdapt, SCYCHIC) across all four evaluation metrics: path score (0.7285 vs. 0.7031 for the best baseline TaxoAdapt), level score (0.7726 vs. 0.7407), coverage (0.9852 vs. 0.9090 for Chain-of-Layers), and weighted novelty (0.4778 vs. 0.4219 for SCYCHIC). Coverage remains superior at all thresholds from 0.5 to 0.9. Qualitatively, TaxoScale discovers novel branches absent from prior taxonomies, including an entirely new Network Security branch (e.g., IP Address Management, VPN Functionality), a refined Behavioral Tracking branch (e.g., Driving Behavior Tracking, User Activity Tracking, Financial Transaction Tracking), a nuanced Password branch (e.g., Weak Password Feedback, Reset Difficulties, 2FA Failures), and a domain-specific Parental Control branch (e.g., Child Safety Safeguard, Activity Oversight) reflecting the teacher-approved apps in the corpus.

The paper releases the filtered review corpus, extracted concern labels, and the TaxoScale codebase as reusable resources. The authors note that TaxoScale is domain-agnostic and can be applied to any large-scale corpus, though cross-domain validation is left to future work.

Improvements to AI systems:

1. Scalable taxonomy construction for massive corpora: Replace existing LLM/clustering-based taxonomy methods (designed for 103 documents) with TaxoScale’s recursive hierarchical clustering (top-down k-means + bottom-up Ward merging on centroids) to handle 105–106 documents without memory explosion. The improved system can build domain taxonomies from millions of user-generated texts (e.g., support tickets, product reviews, social media posts) in hours instead of days.

2. Hybrid LLM + classical clustering pipeline for noisy, short texts: Integrate the four-step pipeline (embedding → seed-node assignment → recursive clustering → LLM node naming) to automatically organize unstructured, colloquial data (e.g., app reviews, forum posts) into a human-interpretable hierarchy. The improved system can produce a taxonomy with 98.5% coverage of the input corpus, discovering novel branches (e.g., VPN Functionality, 2FA Failures) that prior taxonomies miss.

3. Domain-agnostic taxonomy expansion with minimal human input: Use the seed taxonomy (e.g., Akgul et al.'s 20-category privacy taxonomy) as a prior, then let the system autonomously expand it via threshold-based assignment (θ=0.5) and height-tolerance collapsing (δleaf=10, ϵ=20). The improved system can adapt to new domains (e.g., healthcare, finance) by swapping the seed taxonomy, without retraining.

4. High-precision LLM-based filtering and summarization for large-scale review mining: Adopt the best-performing classifier (Llama-3.3-70B, few-shot permissions+rules template, F1=0.97) and extractive summarizer (Jaccard=0.85) to filter and condense millions of reviews into concern-level pseudo-labels. The improved system can automatically extract actionable user concerns (e.g., Reset Difficulties, Child Safety Safeguard) from any large review corpus, enabling real-time product feedback analysis.

5. Cost-effective hierarchical naming with a large MoE LLM: Use Qwen3-235B-A22B (sparse MoE) for node naming conditioned on parent path and sibling names, generating concise 2–6 word labels. The improved system can produce consistent, context-aware taxonomy node names at scale, reducing manual labeling effort by >90% while maintaining interpretability.

6. Benchmark-driven prompt and model selection: Reuse the evaluation framework (human-annotated 300-review benchmark with κ=0.87, 140-review summarization benchmark) to automatically select the optimal LLM and prompt template for any new filtering/summarization task. The improved system can self-tune its language model components based on task-specific F1 and Jaccard scores, avoiding trial-and-error.

7. Recursive clustering with tractable memory and time: Implement the centroid-based merging (instead of raw embeddings) and recursion threshold τ=40,000 to keep clustering O(n·log n) in practice. The improved system can cluster 600K+ items on a single GPU node, enabling taxonomy construction for enterprise-scale datasets without distributed infrastructure.

8. Novel branch discovery for domain-specific insights: Leverage the system's ability to find branches absent from prior taxonomies (e.g., Driving Behavior Tracking, Financial Transaction Tracking) to generate new hypotheses for user behavior or risk analysis. The improved system can proactively flag emerging topics in user feedback that are not covered by existing category schemas, aiding product teams in identifying unmet needs.

Mobile app reviews are a rich, continuously renewing source of how users experience privacy and security, yet existing taxonomies of these concerns are hand-crafted and cannot keep pace with the evolving nature of the data. Automating taxonomy construction is the natural response, but scalability is the core challenge: current LLM- and clustering-based methods are developed for scientific corpora of a few thousand documents and do not extend to app review collections numbering in the hundreds of thousands. We address this gap in two ways. First, we filter app reviews for privacy- and security-related content, yielding a comprehensive corpus of over 600K reviews. Second, we introduce TaxoScale, a pipeline that handles taxonomy construction at this scale by extending an expert-defined taxonomy via Recursive Hierarchical Clustering and LLM-based node naming. TaxoScale outperforms strong automatic-taxonomy baselines on path, level, coverage, and novelty metrics, and discovers novel branches absent from prior taxonomies.