Malware families discovery via Open-Set Recognition on Android manifest permissions
summary
The gist
Malware classification is challenging due to the high-dimensional nature of permission data and limited training samples…
In short
The work proposes a practical method to discover new malware families by combining a tree-based classifier with an open-set recognition module. This approach uses permission data from Android applications to distinguish between known malware families and novel, previously unseen ones. The system is designed for real-world deployment, offering better performance than existing methods.
Key concepts
- Open-Set Recognition (OSR)
- OSR is a technique used to identify samples that belong to classes not present in the training data. In this context, it helps determine if a new malware sample has permissions belonging to a known family or if it represents a completely novel family. It operates on raw output scores from the classifier.
- Closed-Set Classifier (C)
- This component is responsible for assigning an application to one of the known malware classes based on its permission vector. The paper uses Gradient Boosting, which is an ensemble method that combines multiple decision trees to make a final classification decision for each application.
- MAXLOGIT Module
- The MAXLOGIT module is the open-set recognition component. It works with the raw output scores generated by the closed-set classifier before a softmax function is applied. This allows it to effectively determine if an input sample belongs to a novel class or not, based on these underlying scores.
- Threshold ($ au$)
- The threshold is a crucial parameter used by the OSR module to decide whether a sample is novel or known. It controls the balance between false alarms and missed novel samples. The paper suggests setting this threshold at the F P R-quantile of scores from a training set containing only known classes.
Terminology used across episodes
This episode discusses
The paper
Malware families discovery via Open-Set Recognition on Android manifest permissions · Read on arXiv
Filippo Leveni, Matteo Misturac, Francesco Iubatti, Carmine Giangregorioc, Nicolò Pastorec, Cesare Alippi, Giacomo Boracchi
Department of Electronics, Information and Bioengineering, Politecnico di Milano - Milan, Italy · Faculty of Informatics, Università della Svizzera Italiana - Lugano, Switzerland · Cleafy S.p.A.
Malware are malicious programs that are grouped into families based on their penetration technique, source code, and other characteristics. Classifying malware programs into their respective families is essential for building effective defenses against cyber threats. Machine learning models have a huge potential in malware detection on mobile devices, as malware families can be recognized by classifying permission data extracted from Android manifest files. Still, the malware classification task is challenging due to the high-dimensional nature of permission data and the limited availability of training samples. In particular, the steady emergence of new malware families makes it impossible to acquire a comprehensive training set covering all the malware classes. In this work, we present a malware classification system that, on top of classifying known malware, detects new ones. In particular, we combine an open-set recognition technique developed within the computer vision community, namely MaxLogit, with a tree-based Gradient Boosting classifier, which is particularly effective in classifying high-dimensional data. Our solution turns out to be very practical, as it can be seamlessly employed in a standard classification workflow, and efficient, as it adds minimal computational overhead. Experiments on public and proprietary datasets demonstrate the potential of our solution, which has been deployed in a business environment.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Malware families discovery via Open-Set Recognition on Android manifest permissions".
Elias: Here's the gist of the paper. Malware classification is challenging due to the high-dimensional nature of permission data and limited training samples, and this work presents a practical and efficient solution that combines an open-set recognition technique with a tree-based classifier to detect new malware families Problem Formulation The goal is to train an open-set classifier K that associates each malicious application p either with a known class label or the Novel label The malicious application is represented by a vector of permissions p ∈ 0, 1 P, where P is the total number of permissions considered We assume that we are provided with a training set TR = (pi, l i) l i ∈ L i=1,… n of annotated malicious applications belonging to known families and with a test set TS = (pi, l i) l i ∈ L ∪ Novel i=1,… m of annotated malicious applications belonging to both known and novel families Proposed Approach Our proposed open-set classifier K consists of a closed-set classifier C and an open-set recognition module O The closed-set classifier assigns to each malicious application p a known class label, expressed as C(p) = l b(p) ∈ L The open-set recognition module O(zp) ∈ Novel, Not novel determines whether p belongs to a novel (unknown) or not novel (known) class based on the raw output values zp assigned to p by the classifier C We employ tree-based GRADIENT BOOSTING
12: as closed-set classifier and MAXLOGIT
24: as open-set recognition module How the training signal is built The MAXLOGIT approach operates on the raw output scores zp =
z1,..., zl: generated by the last layer of the neural network, known as the LOGITS, before the softmax function In GRADIENT BOOSTING, decision values are aggregated across all trees in the ensemble, resulting in a final vector zp =
z1,..., zl: Subsequently to neural network classifiers, the softmax (2) is applied to zp and the class corresponding to the highest probability in σ(zp) is designated as the predicted class for p Where the gains come from The value of the threshold τ is fundamental to control false alarms raised by the OSR module Threshold τ is typically tuned using an external training set T T = (pi, l i) l i ∈ L i=1,… k, which is composed only of samples belonging to known classes In practice, it is enough to set the threshold τ equal to the F P R-quantile of the empirical distribution of max(zp) computed from the elements of T T Our method does not add any complexity to the GRADIENT BOOSTING closed-set classifier C Real-world performance and evaluation Experiments on public and proprietary datasets demonstrate the potential of our solution, which has been deployed in a business environment The closed-set classification performance of our deployed solution resembles closely those reported in experiments on the proprietary dataset, achieving an accuracy of 83% when tested on about 300 telemetry applications Our approach is faster than methods that require function call graph extraction and manipulation OSNN exhibits very low T P R values at low F P R, making it unsuitable for our requirements compared to our method in Figure 5 We observe that most of the false positives of O, in particular at low F P R values, are samples that would have been misclassified by C if the open-set module were not in place The closed-set classifier proves effective in classifying malware families based on highdimensional binary permission vectors Finally, our solution is currently used by analysts at our industrial partner as a complementary tool to enhance malware classification and discover new families In this study, we combined for the first time a tree-based GRADIENT BOOSTING classifier with the MAXLOGIT open-set recognition technique to tackle the problem of malware family discovery We conduct comprehensive experiments on both a public and a proprietary dataset to validate the suitability of our approach and discuss its deployment performance in a real-world environment This study emphasizes the practical value of an open-set recognition approach in malware classification We envisage several potential future directions, including developing a dynamic thresholding system that adjusts the F P R threshold τ adaptively, taking into account contextual information such as the current threat landscape to achieve the desired sensibility of the OSR module References
1: D.
Nadia: Arp, M.
Elias: Spreitzenbarth, M.
Nadia: Hübner, H.
Paper summary: Nadia: So we're wrapping up this look at how they combined gradient boosting trees and open-set recognition to find new Android malware families based on permission data.
Elias: Yeah, it's about taking that high-dimensional list of permissions and seeing which ones are completely new compared to what they already know.
Priya: I want to ask what the real value is here for someone actually trying to stop these attacks in the wild.
Nadia: It means we can start discovering new malware families much easier without having to manually label every single thing we see on the internet or in our labs.
Elias: The authors are showing that this specific combination of a tree classifier and the maxlogit method works well for this kind of open-set problem.
Priya: But how reliable is that boundary they set? How do we know if the novel sample is really novel or just a weird known sample.
Nadia: They talk about setting a threshold based on a test set that only has known malware, using the F P R quantile of the maximum output values.
Nadia: That's definitely the next thing we need to watch, figuring out how dynamic that threshold setting can be in a real threat environment.
Conclusion: Nadia: So, we’re wrapping up this look at how they used gradient boosting trees and open-set recognition to find new Android malware families using permission data, Elias.
Elias: Yeah, it boils down to taking that massive list of permissions and figuring out which ones are totally new compared to what they already know, Nadia.
Priya: I keep coming back to the real value here for someone actually trying to stop these attacks in the wild.
Nadia: It means we can start finding new malware families much easier without having to manually label every single thing we see on the internet or in our labs, Priya.
Elias: The authors are showing that this specific setup—combining a tree classifier with a maxlogit module—works well for this kind of open-set problem, Nadia.
Priya: But how reliable is that boundary they set? How do we know if the novel sample is really new or just some weird known sample?
Nadia: They talk about setting that threshold using a test set that only has known malware, picking the percentile from those maximum decision values, Priya.
Elias: That’s practical because it means you don't have to perfectly model every single possible permission combination in advance, Nadia.
Priya: So what does this actually change for a security analyst who only looks at the final classification result? Elias
Nadia: It gives them a tool to flag things that fall outside their known families right away, something they would totally miss with just using a standard classifier, Priya.
Elias: The implication is we can move from purely reactive detection to having a system that actually helps us discover what’s new in the malware landscape, Nadia.
Priya: But we still have to wonder if this method holds up when the permission data gets even more complex or when a family is specifically designed to look exactly like an existing one, Elias.
Nadia: That's definitely the next thing we need to watch, figuring out how dynamic that threshold setting can be in a real threat environment, Priya.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18