Malware families discovery via Open-Set Recognition on Android manifest permissions

arXiv:2505.12750 · cs.CR, cs.AI, cs.LG · Submitted 2025-05-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Malware families discovery via Open-Set Recognition on Android manifest permissions".

Elias: Here's the gist of the paper. Malware classification is challenging due to the high-dimensional nature of permission data and limited training samples, and this work presents a practical and efficient solution that combines an open-set recognition technique with a tree-based classifier to detect new malware families Problem Formulation The goal is to train an open-set classifier K that associates each malicious application p either with a known class label or the Novel label The malicious application is represented by a vector of permissions p ∈ 0, 1 P, where P is the total number of permissions considered We assume that we are provided with a training set TR = (pi, l i) l i ∈ L i=1,… n of annotated malicious applications belonging to known families and with a test set TS = (pi, l i) l i ∈ L ∪ Novel i=1,… m of annotated malicious applications belonging to both known and novel families Proposed Approach Our proposed open-set classifier K consists of a closed-set classifier C and an open-set recognition module O The closed-set classifier assigns to each malicious application p a known class label, expressed as C(p) = l b(p) ∈ L The open-set recognition module O(zp) ∈ Novel, Not novel determines whether p belongs to a novel (unknown) or not novel (known) class based on the raw output values zp assigned to p by the classifier C We employ tree-based GRADIENT BOOSTING

12: as closed-set classifier and MAXLOGIT

24: as open-set recognition module How the training signal is built The MAXLOGIT approach operates on the raw output scores zp =

z1,..., zl: generated by the last layer of the neural network, known as the LOGITS, before the softmax function In GRADIENT BOOSTING, decision values are aggregated across all trees in the ensemble, resulting in a final vector zp =

z1,..., zl: Subsequently to neural network classifiers, the softmax (2) is applied to zp and the class corresponding to the highest probability in σ(zp) is designated as the predicted class for p Where the gains come from The value of the threshold τ is fundamental to control false alarms raised by the OSR module Threshold τ is typically tuned using an external training set T T = (pi, l i) l i ∈ L i=1,… k, which is composed only of samples belonging to known classes In practice, it is enough to set the threshold τ equal to the F P R-quantile of the empirical distribution of max(zp) computed from the elements of T T Our method does not add any complexity to the GRADIENT BOOSTING closed-set classifier C Real-world performance and evaluation Experiments on public and proprietary datasets demonstrate the potential of our solution, which has been deployed in a business environment The closed-set classification performance of our deployed solution resembles closely those reported in experiments on the proprietary dataset, achieving an accuracy of 83% when tested on about 300 telemetry applications Our approach is faster than methods that require function call graph extraction and manipulation OSNN exhibits very low T P R values at low F P R, making it unsuitable for our requirements compared to our method in Figure 5 We observe that most of the false positives of O, in particular at low F P R values, are samples that would have been misclassified by C if the open-set module were not in place The closed-set classifier proves effective in classifying malware families based on highdimensional binary permission vectors Finally, our solution is currently used by analysts at our industrial partner as a complementary tool to enhance malware classification and discover new families In this study, we combined for the first time a tree-based GRADIENT BOOSTING classifier with the MAXLOGIT open-set recognition technique to tackle the problem of malware family discovery We conduct comprehensive experiments on both a public and a proprietary dataset to validate the suitability of our approach and discuss its deployment performance in a real-world environment This study emphasizes the practical value of an open-set recognition approach in malware classification We envisage several potential future directions, including developing a dynamic thresholding system that adjusts the F P R threshold τ adaptively, taking into account contextual information such as the current threat landscape to achieve the desired sensibility of the OSR module References

1: D.

Nadia: Arp, M.

Elias: Spreitzenbarth, M.

Nadia: Hübner, H.

Paper summary: Nadia: So we're wrapping up this look at how they combined gradient boosting trees and open-set recognition to find new Android malware families based on permission data.

Elias: Yeah, it's about taking that high-dimensional list of permissions and seeing which ones are completely new compared to what they already know.

Priya: I want to ask what the real value is here for someone actually trying to stop these attacks in the wild.

Nadia: It means we can start discovering new malware families much easier without having to manually label every single thing we see on the internet or in our labs.

Elias: The authors are showing that this specific combination of a tree classifier and the maxlogit method works well for this kind of open-set problem.

Priya: But how reliable is that boundary they set? How do we know if the novel sample is really novel or just a weird known sample.

Nadia: They talk about setting a threshold based on a test set that only has known malware, using the F P R quantile of the maximum output values.

Nadia: That's definitely the next thing we need to watch, figuring out how dynamic that threshold setting can be in a real threat environment.

Conclusion: Nadia: So, we’re wrapping up this look at how they used gradient boosting trees and open-set recognition to find new Android malware families using permission data, Elias.

Elias: Yeah, it boils down to taking that massive list of permissions and figuring out which ones are totally new compared to what they already know, Nadia.

Priya: I keep coming back to the real value here for someone actually trying to stop these attacks in the wild.

Nadia: It means we can start finding new malware families much easier without having to manually label every single thing we see on the internet or in our labs, Priya.

Elias: The authors are showing that this specific setup—combining a tree classifier with a maxlogit module—works well for this kind of open-set problem, Nadia.

Priya: But how reliable is that boundary they set? How do we know if the novel sample is really new or just some weird known sample?

Nadia: They talk about setting that threshold using a test set that only has known malware, picking the percentile from those maximum decision values, Priya.

Elias: That’s practical because it means you don't have to perfectly model every single possible permission combination in advance, Nadia.

Priya: So what does this actually change for a security analyst who only looks at the final classification result? Elias

Nadia: It gives them a tool to flag things that fall outside their known families right away, something they would totally miss with just using a standard classifier, Priya.

Elias: The implication is we can move from purely reactive detection to having a system that actually helps us discover what’s new in the malware landscape, Nadia.

Priya: But we still have to wonder if this method holds up when the permission data gets even more complex or when a family is specifically designed to look exactly like an existing one, Elias.

Nadia: That's definitely the next thing we need to watch, figuring out how dynamic that threshold setting can be in a real threat environment, Priya.

Filippo Leveni, Matteo Misturac, Francesco Iubatti, Carmine Giangregorioc, Nicolò Pastorec, Cesare Alippi, Giacomo Boracchi

Department of Electronics, Information and Bioengineering, Politecnico di Milano - Milan, Italy · Faculty of Informatics, Università della Svizzera Italiana - Lugano, Switzerland · Cleafy S.p.A.

cs.CR, cs.AI, cs.LG

Submitted: 2025-05-19

Updated: 2026-11-08

Comments: Accepted at AIMLSystems (2026)

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 90/100

The gist: Malware classification is challenging due to the high-dimensional nature of permission data and limited training samples…

Key concepts

Open-Set Recognition (OSR)
OSR is a technique used to identify samples that belong to classes not present in the training data. In this context, it helps determine if a new malware sample has permissions belonging to a known family or if it represents a completely novel family. It operates on raw output scores from the classifier.
Closed-Set Classifier (C)
This component is responsible for assigning an application to one of the known malware classes based on its permission vector. The paper uses Gradient Boosting, which is an ensemble method that combines multiple decision trees to make a final classification decision for each application.
MAXLOGIT Module
The MAXLOGIT module is the open-set recognition component. It works with the raw output scores generated by the closed-set classifier before a softmax function is applied. This allows it to effectively determine if an input sample belongs to a novel class or not, based on these underlying scores.
Threshold ($ au$)
The threshold is a crucial parameter used by the OSR module to decide whether a sample is novel or known. It controls the balance between false alarms and missed novel samples. The paper suggests setting this threshold at the F P R-quantile of scores from a training set containing only known classes.

Terminology

Summary

Malware classification is challenging due to the high-dimensional nature of permission data and limited training samples, and this work presents a practical and efficient solution that combines an open-set recognition technique with a tree-based classifier to detect new malware families

Problem Formulation

The goal is to train an open-set classifier K that associates each malicious application p either with a known class label or the Novel label The malicious application is represented by a vector of permissions p ∈ 0, 1 P, where P is the total number of permissions considered We assume that we are provided with a training set TR = (pi, l i) l i ∈ L i=1,… n of annotated malicious applications belonging to known families and with a test set TS = (pi, l i) l i ∈ L ∪ Novel i=1,… m of annotated malicious applications belonging to both known and novel families

Proposed Approach

Our proposed open-set classifier K consists of a closed-set classifier C and an open-set recognition module O The closed-set classifier assigns to each malicious application p a known class label, expressed as C(p) = l b(p) ∈ L The open-set recognition module O(zp) ∈ Novel, Not novel determines whether p belongs to a novel (unknown) or not novel (known) class based on the raw output values zp assigned to p by the classifier C We employ tree-based GRADIENT BOOSTING [12] as closed-set classifier and MAXLOGIT [24] as open-set recognition module

How the training signal is built

The MAXLOGIT approach operates on the raw output scores zp = [z1,..., zl] generated by the last layer of the neural network, known as the LOGITS, before the softmax function In GRADIENT BOOSTING, decision values are aggregated across all trees in the ensemble, resulting in a final vector zp = [z1,. p. 3

Where the paper says it

GRADIENT BOOSTING classifier does not have proper LOGITS, but it produces decision values, which are essentially the raw, unnormalized scores assigned to each class by the ensemble model before being transformed into probabilities (Figure 2b). In GRADIENT BOOSTING, in order to obtain the final prediction ℓb(p) for a given input sample p, decision values are aggregated across all the trees in the ensemble, resulting in a final vector zp = [z1, .

.., zl] Subsequently to neural network classifiers, the softmax (2) is applied to zp and the class corresponding to the highest probability in σ(zp) is designated as the predicted class for p

Where the gains come from

The value of the threshold τ is fundamental to control false alarms raised by the OSR module Threshold τ is typically tuned using an external training set T T = (pi, l i) l i ∈ L i=1,… k, which is composed only of samples belonging to known classes In practice, it is enough to set the threshold τ equal to the F P R-quantile of the empirical distribution of max(zp) computed from the elements of T T Our method does not add any complexity to the GRADIENT BOOSTING closed-set classifier C

Real-world performance and evaluation

Experiments on public and proprietary datasets demonstrate the potential of our solution, which has been deployed in a business environment The closed-set classification performance of our deployed solution resembles closely those reported in experiments on the proprietary dataset, achieving an accuracy of 83% when tested on about 300 telemetry applications Our approach is faster than methods that require function call graph extraction and manipulation OSNN exhibits very low T P R values at low F P R, making it unsuitable for our requirements compared to our method in Figure 5 We observe that most of the false positives of O, in particular at low F P R values, are samples that would have been misclassified by C if the open-set module were not in place The closed-set classifier proves effective in classifying malware families based on highdimensional binary permission vectors Finally, our solution is currently used by analysts at our industrial partner as a complementary tool to enhance malware classification and discover new families

Conclusion and future works

In this study, we combined for the first time a tree-based GRADIENT BOOSTING classifier with the MAXLOGIT open-set recognition technique to tackle the problem of malware family discovery We conduct comprehensive experiments on both a public and a proprietary dataset to validate the suitability of our approach and discuss its deployment performance in a real-world environment This study emphasizes the practical value of an open-set recognition approach in malware classification We envisage several potential future directions, including developing a dynamic thresholding system that adjusts the F P R threshold τ adaptively, taking into account contextual information such as the current threat landscape to achieve the desired sensibility of the OSR module

References

[1] D. Arp, M. Spreitzenbarth, M. p. 7

Where the paper says it

References [1] D. Spreitzenbarth, M.

p. 7
Where the paper says it

References [1] D. Spreitzenbarth, M.

Hübner, H. p. 7
Where the paper says it

Spreitzenbarth, M. Hübner, H.

p. 7
Where the paper says it

Spreitzenbarth, M. Hübner, H.

Gascon, and K. p. 7
Where the paper says it

Hübner, H. Gascon, and K.

p. 7
Where the paper says it

Hübner, H. Gascon, and K.

Rieck. Drebin: Effective and explainable detection of android malware in your pocket

[2] A. Bendale and T. p. 7

Where the paper says it

Internet Society, 2014. Bendale and T.

E. Boult. Towards open set deep networks

[3] P. Bodesheim, A. p. 7

Where the paper says it

Institute of Electrical and Electronics Engineers (IEEE), 2016. Bodesheim, A.

p. 7
Where the paper says it

Institute of Electrical and Electronics Engineers (IEEE), 2016. Bodesheim, A.

Freytag, E. p. 7
Where the paper says it

Bodesheim, A. Freytag, E.

p. 7
Where the paper says it

Bodesheim, A. Freytag, E.

Rodner, M. Kemmler, and J. p. 7
Where the paper says it

Freytag, E. Kemmler, and J.

Denzler. Kernel null space methods for novelty detection

[4] P. Bodesheim, A. Freytag, E. Rodner, and J. p. 7

Where the paper says it

Freytag, E. Rodner, and J.

Denzler. Local novelty detection in multi-class recognition problems

[5] Q. Da, Y. Yu, and Z.-H. Zhou. Learning with augmented class by exploiting unlabeled data

[6] P. Faruki, V. Laxmi, A. Bharmal, M. p. 7

Where the paper says it

Association for the Advancement of Artificial Intelligence (AAAI), 2014. Bharmal, M.

S. Gaur, and V. p. 7
Where the paper says it

Bharmal, M. Gaur, and V.

Ganmoor. Androsimilar: Robust signature for detecting variants of android malware

[7] Z. Ge, S. Demyanov, Z. p. 7

Where the paper says it

Journal of Information Security and Applications (JISA), 22:66–80, 2015. Demyanov, Z.

Chen, and R. p. 7
Where the paper says it

Demyanov, Z. Chen, and R.

Garnavi. Generative openmax for multi-class open set classification

[8] J. Guo, S. Guo, S. Ma, Y. Sun, and Y. p. 7

Where the paper says it

British Machine Vision Association (BMVA), 2017. Sun, and Y.

Xu. Conservative novelty synthesizing network for malware recognition in an open-set scenario

[9] J. Guo, H. Wang, Y. Xu, W. Xu, Y. Zhan, Y. Sun, and S. p. 7

Where the paper says it

British Machine Vision Association (BMVA), 2017. Sun, and Y.

-H Guo. Multimodal dual-embedding networks for malware open-set recognition

[10] M.-Hassen and P. p. 7

Where the paper says it

Transactions on Neural Networks and Learning Systems (TNNLS), 36 (3):4545–4559, 2025. Hassen and P.

p. 7
Where the paper says it

Transactions on Neural Networks and Learning Systems (TNNLS), 36 (3):4545–4559, 2025. Hassen and P.

-K Chan. Scalable function call graph-based malware classification

[11] M.-Hassen and P.-K Chan. Learning a neural-network-based representation for open set recognition

[12] T. Hastie, R. Tibshirani, and J. p. 7

Where the paper says it

Society for Industrial and Applied Mathematics (SIAM), 2020. Tibshirani, and J.

-H Friedman. The elements of statistical learning: data mining, inference, and prediction

[13] N. Herron, W. B. Glisson, J. p. 7

Where the paper says it

Springer Nature, 2009. Glisson, J.

-T McDonald, and R. p. 7
Where the paper says it

Glisson, J. McDonald, and R.

K Benton. Machine learning-based android malware detection using manifest permissions

[14] J. Jia and P.-K Chan. Representation learning with function call graph transformations for malware open set recognition

[15] K. Lee, H.-Lee, K. Lee, and J. p. 7

Where the paper says it

Institute of Electrical and Electronics Engineers (IEEE), 2022. Lee, and J.

Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples

[16] T. Lu and J.-Wang. Domr: Toward deep open-world malware recognition

[17] P.-R Mendes Júnior, R. p. 7

Where the paper says it

Transactions on Information Forensics and Security (TIFS), 19: 1455–1468, 2024. Mendes Júnior, R.

M De Souza, R. p. 7
Where the paper says it

Mendes Júnior, R. De Souza, R.

d O Werneck, B. p. 7
Where the paper says it

De Souza, R. Werneck, B.

V Stein, D V Pazinato, W R De Almeida, O A Penatti, R d S Torres, and A Rocha. p. 7
Where the paper says it

Pazinato, W. De Almeida, O.

Nearest neighbors distance ratio open-set classifier

[18] M. Odusami, O Abayomi-Alli, S Misra, O Shobayo, R Damasevicius, and R Maskeliunas. p. 7

Where the paper says it

Odusami, O. Abayomi-Alli, S.

Android malware detection: A survey

[19] P. Rovelli and Ý Vigfússon. p. 7

Where the paper says it

Rovelli and Ý. Vigfússon.

Pmds: permission-based malware detection system

[20] W. J Scheirer, A de Rezende Rocha, A Sapkota, and T E Boult. p. 7

Where the paper says it

Scheirer, A. de Rezende Rocha, A.

Toward open set recognition

[21] Statista. Number of available applications in the google play store from december 2009 to december 2023

[22] Statista. Global market share held by mobile operating systems from 2009 to 2023, by quarter

[Online; accessed 30-April-2025]. URL https://www. p. 7

Where the paper says it

Number of available applications in the google play store from december 2009 to december 2023, 2024. URL https://www.

p. 7
Where the paper says it

Number of available applications in the google play store from december 2009 to december 2023, 2024. URL https://www.

p. 7
Where the paper says it

Number of available applications in the google play store from december 2009 to december 2023, 2024. URL https://www.

p. 7
Where the paper says it

Number of available applications in the google play store from december 2009 to december 2023, 2024. URL https://www.

statista.com/statistics/266210/number-of-available-applications-in-the-google play store/. p. 7
Where the paper says it

URL https://www. com/statistics/266210/ number-of-available-applications-in-the-google-play-store/.

[Online; accessed 30 April 2025]. p. 7
Where the paper says it

com/statistics/266210/ number-of-available-applications-in-the-google-play-store/. [Online; accessed 30-April-2025].

p. 7
Where the paper says it

com/statistics/266210/ number-of-available-applications-in-the-google-play-store/. [Online; accessed 30-April-2025].

p. 7
Where the paper says it

com/statistics/266210/ number-of-available-applications-in-the-google-play-store/. [Online; accessed 30-April-2025].

p. 7
Where the paper says it

com/statistics/266210/ number-of-available-applications-in-the-google-play-store/. [Online; accessed 30-April-2025].

[Online; accessed 30 April 2025]. URL https://www.statista.com/statistics/272698/global market share held-by-mobile-operating systems-since-2009/. p. 7

Where the paper says it

URL https://www. com/statistics/272698/ global-market-share-held-by-mobile-operating-systems-since-2009/.

[Online; accessed 30 April 2025].

[1] D. Arp, M. Spreitzenbarth, M. Hübner, H. Gascon, and K. Rieck. Drebin: Effective and explainable detection of android malware in your pocket

[2] A Bendale and T E Boult. Towards open set deep networks

[3] P Bodesheim, A Freytag, E Rodner, M Kemmler, and J Denzler. Kernel null space methods for novelty detection

[4] P Bodesheim, A Freytag, E Rodner, and J Denzler. Local novelty detection in multi-class recognition problems

[5] Q Da, Y Yu, and Z.-H Zhou. Learning with augmented class by exploiting unlabeled data

[6] P Faruki, V Laxmi, A Bharmal, M S Gaur, and V Ganmoor. Androsimilar: Robust signature for detecting variants of android malware

[7] Z Ge, S Demyanov, Z Chen, and R Garnavi. Generative openmax for multi-class open set classification

[8] J Guo, S Guo, S Ma, Y Sun, and Y Xu. Conservative novelty synthesizing network for malware recognition in an open-set scenario

[9] J Guo, H Wang, Y Xu, W Xu, Y Zhan, Y Sun, and S. Guo. Multimodal dual-embedding networks for malware open-set recognition

[10] M Hassen and P K Chan. Scalable function call graph-based malware classification

[11] M Hassen and P K Chan. Learning a neural-network-based representation for open set recognition

[23] T N Turnip, A Situmorang, A Lumbantobing, J Marpaung, and S I Situmeang. Android malware classification based on permission categories using extreme gradient boosting

[24] S Vaze, K Han, A Vedaldi, and A Zisserman. Open-set recognition: A good closed-set classifier is all you need? p. 7

Where the paper says it

Vedaldi, and A. Open-set recognition: A good closed-set classifier is all you need?

[18] M Odusami, O Abayomi-Alli, S Misra, O Shobayo, R Damasevicius, and R Maskeliunas. p. 7

Where the paper says it

Odusami, O. Abayomi-Alli, S.

Android malware detection: A survey

[19] P Rovelli and Ý Vigfússon. Pmds: permission-based malware detection system

[20] W J Scheirer, A de Rezende Rocha, A Sapkota, and T E Boult. Toward open set recognition

[21] Statista. Number of available applications in the google play store from december 2009 to december 2023

[Online; accessed 30-April-25]. URL https://www.statista.com/statistics/266210/number-of-available applications in the google play store/. p. 7

Where the paper says it

URL https://www. com/statistics/266210/ number-of-available-applications-in-the-google-play-store/.

[Online; accessed 30 April 2025]. URL https://www.statista.com/statistics/272698/global market share-held-by-mobile operating systems-since-2009/. p. 7

Where the paper says it

URL https://www. com/statistics/272698/ global-market-share-held-by-mobile-operating-systems-since-2009/.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Integration of MAXLOGIT into tree-based GRADIENT BOOSTING classifiers allows existing closed-set classification pipelines to seamlessly detect novel malware families without modifying training procedures or incurring significant computational overhead. This enables deployment in resource-constrained industrial environments by prioritizing low latency and ease of integration with existing infrastructures.

  2. The proposed open-set classifier K combines a closed-set classifier C (GRADIENT BOOSTING) with an open-set recognition module O (MAXLOGIT) to classify applications into known families or identify novel ones. This system can seamlessly integrated into an existing tree-based malware classifier, without even modifying the training procedure.

  3. The MAXLOGIT computation uses the maximum LOGIT value from the GRADIENT BOOSTING classifier to determine novelty based on a threshold tau, which is tuned using an external training set of known samples. This threshold setting ensures a specified false alarm rate by setting tau equal to the F P R-quantile of the empirical distribution of max(zp) computed from elements of T T.

  4. The system performs better than nearest-neighbor OSR baselines on high-dimensional binary data, which is crucial for handling the high-dimensional sparse binary feature vector resulting from onehot encoding of Android permissions.

  5. The solution is fast enough for deployment, with inference times reaching 100 milliseconds (based on the provided scale), showing a difference of more than an order of magnitude compared to OSNN's complexity, which is dominated by O(n log n) sorting.

Abstract

Malware are malicious programs that are grouped into families based on their penetration technique, source code, and other characteristics. Classifying malware programs into their respective families is essential for building effective defenses against cyber threats. Machine learning models have a huge potential in malware detection on mobile devices, as malware families can be recognized by classifying permission data extracted from Android manifest files. Still, the malware classification task is challenging due to the high-dimensional nature of permission data and the limited availability of training samples. In particular, the steady emergence of new malware families makes it impossible to acquire a comprehensive training set covering all the malware classes. In this work, we present a malware classification system that, on top of classifying known malware, detects new ones. In particular, we combine an open-set recognition technique developed within the computer vision community, namely MaxLogit, with a tree-based Gradient Boosting classifier, which is particularly effective in classifying high-dimensional data. Our solution turns out to be very practical, as it can be seamlessly employed in a standard classification workflow, and efficient, as it adds minimal computational overhead. Experiments on public and proprietary datasets demonstrate the potential of our solution, which has been deployed in a business environment.

Related papers