On building machine learning pipelines for Android malware detection: a procedural survey of practices, challenges and opportunities

arXiv:2306.07118 · cs.CR, cs.LG · Submitted 2023-06-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "On building machine learning pipelines for Android malware detection".

Elias: Here's the gist of the paper. This paper addresses a critical gap in research concerning Machine Learning (ML) approaches for Android malware detection by providing a novel procedural taxonomy of existing literature.

Nadia: The authors conducted a comprehensive review of 42 highly-cited papers published between 2011 and 2021 to map the current state-of-the-art in this evolving field.

Elias: The core contribution is the introduction of a procedural taxonomy structured around the distinct stages of a typical ML pipeline applied to Android malware detection.

Nadia: This structure allows new researchers and practitioners to gain a clear understanding of how various studies have approached each phase, including: 1.

Paper summary: Nadia: What this means is that instead of just seeing a list of different detection methods floating around the literature you get a roadmap for how they do things step by step. Elias It lays out the sequence: first you have to think about data collection, like how researchers gather those malicious and benign APKs. Nadia Then there’s feature engineering, which is how they pull out static and dynamic features from the applications themselves. Elias After that, there's feature representation, which is choosing the right techniques to represent those features and getting rid of the less useful ones through selection or elimination.

Priya: I’m curious about what kind of data they are talking about for collection, because if you get a skewed dataset upfront, the whole pipeline gets biased. Nadia That’s a good question. The paper just describes how researchers source those malicious and benign APKs to give you that first stage of the pipeline. Elias And then they move into modeling, which is where they discuss the specific machine learning algorithms people are using for the actual detection process.

Nadia: Moving into evaluation, they talk about what metrics are used to actually assess model performance, including classification metrics and feasibility studies. Elias But here’s a point I found interesting: they specifically mention a lack of focus on things like concept drift and how much resource consumption the models need for actual deployment on devices.

Priya: That missing part is important because if a model works perfectly in the lab but takes too much battery or memory to run on an actual phone, it doesn't really matter in the real world. Nadia Exactly. And they also discuss deployment and explanation, how models are ultimately put into use or explained to users.

Elias: So what this paper claims is that by putting everything in a procedural structure like this taxonomy, it helps new researchers and practitioners understand exactly how to approach the implementation of each stage in an ML pipeline for Android malware detection. Nadia It’s about providing a clear framework instead of just comparing final accuracy scores across different studies.

Conclusion: Nadia: So, looking at the whole thing, this paper by Mehrabi Koushki and colleagues is really focused on providing that structural understanding of how these ML pipelines are constructed in Android malware detection research. Elias They’ve moved beyond just saying one method is better than another; they’re showing *how* those methods fit into a process. Nadia The authors identify significant gaps in the current literature, which points toward where future research needs to go.

Priya: I think the paper highlights a few key areas where more work is needed. They point out that most papers don't effectively combine static and dynamic analysis—meaning using static data to improve dynamic analysis, for example. Elias And they also note that feature representation is often overemphasized while simple representation techniques are used instead of more sophisticated ones.

Nadia: That leads into another major gap they find: there’s not enough attention paid to dynamic analysis in general, and transfer learning techniques haven't been explored much in this domain yet. Elias And regarding evaluation, they say there’s a serious lack of research on model explanation for both the model itself and the end-user experience.

Nadia: What this paper really means for us is that it gives a baseline for what's already done, but also clearly outlines the necessary improvements needed in data utilization, feature handling, and evaluation rigor if we want to move this research forward. Elias It’s less about finding one perfect model and more about building better systems for detecting malware as it keeps changing.

Masoud Mehrabi Koushki, Ibrahim AbuAlhaol, Anandharaju Durai Raju, Yang Zhou, Ronnie Salvador Giagone, Huang Shengqiang

cs.CR, cs.LG

Submitted: 2023-06-12

Updated: 2023-06-12

Comments: file:///C:/Users/ibrahim_abualhaol/Downloads/s42400-022-00119-8.pdf

Journal ref: SpringerOpen Cybersecurity 2022

DOI: 10.1186/s42400-022-00119-8

Code: https://github.com/androguard/androguard

Project page: https://ibotpeaches.github.io/Apktool

License: http://creativecommons.org/publicdomain/zero/1.0/

Importance score: 91/100

The gist: This paper addresses a critical gap in research concerning Machine Learning (ML) approaches for Android malware detection by providing a novel procedural taxonomy of existing literature.

Key concepts

Procedural Taxonomy
A structured framework used to organize existing literature based on the sequential steps of an ML pipeline. Instead of just listing methods, this taxonomy groups studies by stages like data collection or modeling, making it easy to see how different researchers approach the problem step-by-step.
Feature Engineering
The process of creating useful numerical inputs for a machine learning model from raw application data (APKs). This involves extracting both static information (like code structure) and dynamic information (like runtime behavior) to feed into the detection algorithm.
Concept Drift
A critical evaluation metric that measures how much a model's performance degrades over time as the underlying data distribution changes. In malware detection, this is important because malware constantly evolves, meaning a model trained on old data might become ineffective against new threats.
On-Device Deployment
The practice of running the trained machine learning model directly on a user's Android device rather than in the cloud. This requires models to be efficient and small enough to run locally, making resource consumption a key challenge for practical malware detection.

Terminology

Summary

This paper addresses a critical gap in research concerning Machine Learning (ML) approaches for Android malware detection by providing a novel procedural taxonomy of existing literature. p. 1

Where the paper says it

Instead, we aim at providing a general overview of this research field, to help newcomers catch to speed with the status quo. In summary, therefore, this paper makes the following contributions to the field of ML-based Android malware detection: • We provide an overview of the existing literature, based on a review of 42 highly-cited papers published in reputable sources.

The authors conducted a comprehensive review of 42 highly-cited papers published between 2011 and 2021 to map the current state-of-the-art in this evolving field. p. 0
Where the paper says it

In this paper, we address this problem with a review of 42 highly-cited papers, spanning a decade of research (from 2011 to 2021). We introduce a novel procedural taxonomy of the published literature, covering how they have used ML algorithms, what features they have engineered, which dimensionality reduction techniques they have employed, what datasets they have employed for training, and what their evaluation and explanation strategies are.

The core contribution is the introduction of a procedural taxonomy structured around the distinct stages of a typical ML pipeline applied to Android malware detection. p. 1

Where the paper says it

We provide a procedural taxonomy of the proposed solutions, structured based on the stages of a typical ML pipeline (Docs M 2022). Consequently, we discuss (1) how researchers have collected data for training and testing their models, (2) how and what features they have engineered/extracted from Android apps, (3) how they have represented the features for modeling, (4) what dimensionality reduction (feature selection/ elimination) techniques they have used, (5) what machine learning algorithms they have tried, (6) what metrics they have used for evaluation of their models (and what the results have been), and, lastly (7) how they have deployed or explained their model.

This structure allows new researchers and practitioners to gain a clear understanding of how various studies have approached each phase, including:

  1. Data Collection: How researchers source malicious and benign APKs.

  2. Feature Engineering: The methods used for extracting both static and dynamic features from the applications.

  3. Feature Representation: Techniques employed to represent these extracted features and eliminate less informative ones (e.g., feature selection/elimination). p. 1

    Where the paper says it

    We provide a procedural taxonomy of the proposed solutions, structured based on the stages of a typical ML pipeline (Docs M 2022). Consequently, we discuss (1) how researchers have collected data for training and testing their models, (2) how and what features they have engineered/extracted from Android apps, (3) how they have represented the features for modeling, (4) what dimensionality reduction (feature selection/ elimination) techniques they have used, (5) what machine learning algorithms they have tried, (6) what metrics they have used for evaluation of their models (and what the results have been), and, lastly (7) how they have deployed or explained their model.

  4. Modeling: The specific Machine Learning algorithms utilized in the detection process.

  5. Evaluation: The metrics used to assess model performance, including classification metrics, feasibility, usability evaluations, and importantly, the lack of focus on crucial aspects like concept drift and resource consumption for on-device deployments.

  6. Deployment/Explanation: How models are deployed or explained to users.

Unlike previous survey papers that might simply list approaches without a structured framework, this taxonomy is explicitly organized based on the sequential stages of an ML pipeline, making it highly practical for understanding implementation strategies. p. 1

Where the paper says it

The review can help new researchers better understand the status quo of the research field. • We provide a procedural taxonomy of the proposed solutions: Unlike previous survey papers, our taxonomy is structured based on stages of an ML pipe- 1 This is the official Android app store operated by Google, available at https://play.

Furthermore, the review identifies significant shortcomings in the current literature, which serve as valuable suggestions for future research directions. p. 2

Where the paper says it

In "Taxonomy of android malware detection approaches" section provides our procedural taxonomy of ML-based Android malware detection, discussing what researchers have tried in each stage of the ML pipeline. In "Gaps in knowledge and future research directions" section we discuss gaps we have identified in the literature and provide ideas for follow-up research.

Key observations include:

  • Analysis Synergy: A noted oversight is that most reviewed papers fail to leverage the combination of static and dynamic analysis (e.g., using static data to improve dynamic analysis), suggesting a promising avenue for future work. p. 6

    Where the paper says it

    The aim was to identify broader themes of android malware detection techniques, and to find potential gaps that can inform future research. The resulting taxonomy, presented in "Taxonomy of android malware detection approaches" section shows, for example, how dynamic analysis has not been researched as well as static analysis, demonstrating potential for future research.

  • Feature Representation: The literature often overemphasizes feature engineering while employing simple representation techniques, indicating a missed opportunity where more sophisticated representations could yield better results without increased complexity.

  • Understudied Areas: Dynamic analysis has received insufficient attention, and promising techniques such as transfer learning have not been explored in this domain.

  • Missing Evaluations: A significant lack exists regarding model explanation (both for the model and the user experience) and comprehensive evaluation criteria, such as concept drift, resource consumption for on-device feasibility, and hyper-parameter tuning/model calibration.

In summary, the paper moves beyond a simple comparison of final accuracy to offer a deep structural understanding of how ML pipelines are constructed in Android malware detection research. This procedural taxonomy serves to contextualize the current landscape while explicitly pointing toward necessary improvements in data utilization, feature handling, and evaluation rigor for future studies. p. 1

Where the paper says it

We provide a procedural taxonomy of the proposed solutions, structured based on the stages of a typical ML pipeline (Docs M 2022). Consequently, we discuss (1) how researchers have collected data for training and testing their models, (2) how and what features they have engineered/extracted from Android apps, (3) how they have represented the features for modeling, (4) what dimensionality reduction (feature selection/ elimination) techniques they have used, (5) what machine learning algorithms they have tried, (6) what metrics they have used for evaluation of their models (and what the results have been), and, lastly (7) how they have deployed or explained their model.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve model robustness by diversifying feature weights to achieve better adversarial robustness, as suggested by Melis et al. (2022). This can be done by using gradient-based attribution methods to identify and select more robust learning algorithms for a pipeline.

  2. Enhance detection capability against code obfuscation and encryption by leveraging hybrid analysis that combines static and dynamic features. For instance, Yuan et al. (2014) used Deep learning in tandem with both static and dynamic analysis, which can alleviate susceptibility to evasion attacks.

  3. Increase the effectiveness of feature extraction by integrating static and dynamic analyses in a tandem manner to maximize code coverage. This combines the strengths of both methods, as seen when researchers statically-informed dynamic analysis for Android malware detection is explored.

  4. Improve feature representation by moving beyond simple existence-based boolean vectors toward more complex structures like similarity scores or semantic representations like Bytecode2vec. This allows models to detect semantically-malicious behavior rather than just relying on exact API call names.

  5. Enhance model selection and training by employing transfer learning techniques, as it could alleviate the difficulty of obtaining a corpus of malicious APKs. This is particularly useful when dealing with rare or novel malware variants.

  6. Improve feature selection justification by performing empirical studies to compare the fitness of different feature selection algorithms instead of relying solely on familiar approaches. This will help researchers determine the requirements of feature selection for Android malware detection.

  7. Improve model sustainability against concept drift by systematically investigating how models perform over time using newer data, such as comparing models trained on older malware against present-year samples, as done by Lindorfer et al. (2015).

  8. Improve deployment feasibility by incorporating resource consumption metrics into the evaluation stage to demonstrate practicality. This helps practitioners assess how much resources they may use for on-device or hybrid solutions.

  9. Enhance user trust and clarity through model explanation by moving beyond just prediction explanation to model explanation, which helps users understand what the model is looking for when labeling an app as malware or benign.

Abstract

As the smartphone market leader, Android has been a prominent target for malware attacks. The number of malicious applications (apps) identified for it has increased continually over the past decade, creating an immense challenge for all parties involved. For market holders and researchers, in particular, the large number of samples has made manual malware detection unfeasible, leading to an influx of research that investigate Machine Learning (ML) approaches to automate this process. However, while some of the proposed approaches achieve high performance, rapidly evolving Android malware has made them unable to maintain their accuracy over time. This has created a need in the community to conduct further research, and build more flexible ML pipelines. Doing so, however, is currently hindered by a lack of systematic overview of the existing literature, to learn from and improve upon the existing solutions. Existing survey papers often focus only on parts of the ML process (e.g., data collection or model deployment), while omitting other important stages, such as model evaluation and explanation. In this paper, we address this problem with a review of 42 highly-cited papers, spanning a decade of research (from 2011 to 2021). We introduce a novel procedural taxonomy of the published literature, covering how they have used ML algorithms, what features they have engineered, which dimensionality reduction techniques they have employed, what datasets they have employed for training, and what their evaluation and explanation strategies are. Drawing from this taxonomy, we also identify gaps in knowledge and provide ideas for improvement and future work.

Sources

Related papers