Binary and Multiclass Cyberattack Classification on GeNIS Dataset

summary

The gist

The study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.

In short

The study validated the GeNIS dataset for AI-based Network Intrusion Detection Systems (NIDS) by testing multiple machine learning and deep learning models. Researchers used five feature selection methods to reduce 125 features down to 16, which retained about 70% of the importance. The results showed that ensemble models performed well, suggesting the dataset is reliable for creating future AI benchmarks.

Key concepts

GeNIS Dataset
This dataset was specifically created for AI NIDS applications, focusing on typical user behavior and network protocols from small to medium organizations. It contains eight attack scenarios and three benign activities, serving as a baseline for testing detection systems.
Feature Selection Methods
Five techniques—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio—were used to filter 125 raw features. This process aimed to identify the most relevant features that best distinguish between network attacks and normal traffic.
Ensemble Models
Models like Random Forest (RF), XGBoost (XGB), and LGBM were trained on the selected features. These models combine multiple decision trees to improve accuracy, achieving slightly better generalization than deep learning models while remaining computationally efficient for both training and running predictions.
SHAP Analysis
This technique was used to explain model predictions by assigning a score to each feature. It revealed that the AI models primarily relied on quantity-based features (like traffic volume) and helped identify which specific behavioral features were most critical for classifying malicious network flows.

Terminology used across episodes

This episode discusses

The paper

Binary and Multiclass Cyberattack Classification on GeNIS Dataset · Read on arXiv

GECAD, ISEP, Polytechnic of Porto

The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic. This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more computationally efficient detection. Three decision tree ensembles and two deep neural networks were trained for both binary and multiclass classification tasks. All models reached high accuracy and F1-scores, and the ML ensembles achieved slightly better generalization while remaining more efficient than DL models. Overall, the obtained results indicate that the GeNIS dataset supports intelligent intrusion detection and cyberattack classification with time-based and quantity-based behavioral features.

DOI: 10.1007/978-3-032-20018-1_18

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "Binary and Multiclass Cyberattack Classification on GeNIS Dataset".

Title and authors: Nadia: So this paper is about validating a dataset called GeNIS for AI-based Network Intrusion Detection Systems to use as a baseline for future benchmarks. It looks like they're trying to make sure the data itself is solid before we train these models

nine −eight −six thousand six hundred thirty −nine thousand nine hundred thirty-nine: <ref:2511.08660#pg1>.

Elias: Right, and the authors are Miguel Silva, Daniela Pinto, João Vitorino, Eva Maia, Isabel Praça and Ivone Amorim and Maria João Viamonte GECAD from ISEP at the Polytechnic of Porto in Portugal

nine −eight −six thousand six hundred thirty −nine thousand nine hundred thirty-nine: <ref:2511.08660#pg1>. It's important to know who's doing the work behind this evaluation.

Priya: What I want to know is what makes this dataset reliable enough for real-world AI applications, because we know that bad data leads to bad detection models <ref:2511.08660#pg2>.

Nadia: Well, they set up GeNIS specifically for AI-based NIDS focusing on typical user behavior and network protocols of small to medium sized organizations <ref:2511.08660#pg2>. They have eight attack scenarios and three benign activity scenarios in it <ref:2511.08660#pg3>.

Elias: And they started with a total of one hundred twenty-five features extracted from the raw recordings using the HERA flow exporter, which is a specific tool used for that extraction process <ref:2511.08660#pg3>. They grouped these into general, time-based, quantity-based, hybrid-based and context-based categories <ref:2511.08660#pg3>.

Priya: Before they did anything else, they had to one-hot encode categorical features like State and Protocol <ref:2511.08660#pg3>. That sounds like a standard first step for getting data ready for classification, but we need to see how that affects the final results.

Nadia: They then used five feature selection methods—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation and Dispersion Ratio—to find the most important features <ref:2511.08660#pg1>. They combined their scores by summing them up for each feature <ref:2511.08660#pg2>.

Elias: And the key result there is they selected sixteen most significant features that accounted for about seventy percent of the total feature importance <ref:2511.08660#pg2>. That’s a big dimensionality reduction from the start, which sounds like it should make training faster <ref:2511.08660#pg2>.

Priya: Seventy percent is a lot of features to boil down to, so I'm curious what those sixteen features actually represent and if we can trust that selection process <ref:2511.08660#pg3>.

Title and authors: Nadia: The next step was training three decision tree ensembles—Random Forest, Extreme Gradient Boosting and Light Gradient Boosting Machine—and two deep neural networks, Long Short-Term Memory and Multilayer Perceptron <ref:2511.08660#pg2>. They used a grid search with five fold cross validation to find the best settings for each model <ref:2511.08660#pg2>.

Elias: The evaluation metric they focused on was the F1-score for binary classification and the macro averaged F1-score for multiclass classification <ref:2511.08660#pg3>. That tells us they were looking at both attack detection and distinguishing between different attack types, which is important <ref:2511.08660#pg3>.

Priya: So, after training all those models on the selected features, what did they actually find about how well these models performed in classifying the flows as attacks or benign activity <ref:2511.08660#pg3>?

Nadia: The ensemble models, which included Random Forest XGB Light Gradient Boosting Machine LSTM and Multilayer Perceptron achieved slightly better generalization while still being more efficient than the deep neural networks <ref:2511.08660#pg3>.

Elias: That’s interesting because it suggests that the simpler ensemble models didn't necessarily need all those complex layers of a neural network to perform well <ref:2511.08660#pg3>.

Priya: In binary classification, the paper found that the selected features were roughly balanced between quantity time and hybrid-based types <ref:2511.08660#pg3>. That tells us what kind of behavioral patterns are most important here.

Nadia: Exactly, in multiclass classification, they noted that quantity features were four times more frequent and time-based features were three times more frequent than hybrid ones <ref:2511.08660#pg3>. That gives us a hint about the structure of the malicious traffic they are trying to identify.

Elias: And SHAP analysis, which is a technique that helps explain model predictions by showing how much each feature contributes, revealed that the models predominantly relied on quantity-based features <ref:2511.08660#pg3>. That points strongly toward traffic volume or packet counts being key indicators of malicious behavior.

Priya: If quantity features are dominating, that makes sense for network traffic where high volumes can sometimes be a sign of an attack, but I'm wondering if this reliance on quantity might miss subtle, low-volume attacks <ref:2511.08660#pg3>.

Nadia: The paper suggests good generalization potential because the reduced feature set was based on behavior rather than topology specific ones <ref:2511.08660#pg3>. This means if you take these models and apply them to a different dataset, they should still work reasonably well <ref:2511.08660#pg3>.

Title and authors: Elias: They also mentioned that the ensemble models achieved less training and inference time than the full feature set or even the neural networks in binary classification <ref:2511.08660#pg3>. That efficiency gain is pretty practical for deployment, especially if we're looking at real-time detection on a network <ref:2511.08660#pg3>.

Priya: It’s good to know they optimized for speed, but what about the limitations of this entire process? What did the authors mention that suggests where this approach might stop working <ref:2511.08660#pg3>?

Nadia: They point out that while the reduced feature set showed good generalization, it’s still important to continue developing larger and more diverse network traffic datasets <ref:2511.08660#pg3>. And they also suggest combining multiple datasets and looking at how detection capabilities transfer to previously unseen network traffic <ref:2511.08660#pg3>.

Elias: And for us, the crucial next step mentioned is assessing model robustness in adversarial settings as a way toward real-world deployment <ref:2511.08660#pg3>. That takes us from just testing on clean data to seeing if the detection holds up when someone tries to fool it <ref:2511.08660#pg3>.

Priya: So, in summary, they validated GeNIS as a reliable baseline, using a focused feature selection process that highlighted quantity and time features as key indicators <ref:2511.08660#pg3>, and they showed the ensemble models are efficient for deployment <ref:2511.08660#pg3>.

Nadia: Right, it sounds like the GeNIS dataset provides a solid starting point, but we still have to look at more diverse data and adversarial testing to make these AI NIDS truly reliable in practice <ref:2511.08660#pg3>.

Elias: Indeed, the study on "Binary and Multiclass Cyberattack Classification on GeNIS Dataset" gives us a baseline for how feature selection impacts model performance across different architectures <ref:2511.08660#pg2>. That foundation helps us decide which data sources to prioritize next <ref:2511.08660#pg3>.

Priya: I just hope that as we move toward more complex systems, we don't lose sight of the underlying data quality issues they pointed out regarding feature extraction errors from tools like CICFlowMeter <ref:2511.08660#pg3>.

Nadia: We certainly have to keep an eye on those inconsistencies when we build our next datasets for AI training <ref:2511.08660#pg3>. That’s the path forward for making these tools actually useful against modern threats.

Elias: Well, that wraps up our look at the GeNIS dataset and its validation, so we'll be looking at new papers soon <ref:2511.08660#pg2>.

The paper's summary: Nadia: So, basically, this paper is looking at how reliable the GeNIS dataset is when we try to use it to train AI for Network Intrusion Detection Systems and they're using it as a starting point for other research eighteen.

Elias: Yeah, they took a raw traffic recording and turned it into this dataset with eight attack scenarios and three benign ones, which is pretty standard for training data.

Nadia: Right, but the real focus here is on the feature selection process, where they use five different methods—like Information Gain and Chi-Squared Test—to boil down one hundred twenty-five features to just sixteen of them nineteen.

Elias: Sixteen features making up seventy percent of the total importance means they're keeping a lot of behavioral data and throwing out the noise from things like just simple packet counts or timing stuff.

Nadia: And once they got those sixteen features, they trained several models—Random Forest, XGBoost, Light Gradient Boosting Machine, plus two deep neural networks—to see how well they could classify those network flows as either an attack or normal activity twenty.

Elias: It’s interesting that the ensemble models like Random Forest and XGBoost ended up performing better than the deep learning ones in some of these tests, even though they were simpler architectures.

Nadia: That makes sense, especially if we look at how much training and running time those models needed compared to the neural networks twenty. It's about efficiency for real-world deployment.

Elias: And when they dug into why the models made their decisions using something called SHAP analysis, they found that quantity-based features like traffic volume were the main drivers of those classifications twenty-one.

Nadia: That suggests that for this specific data, just looking at how much traffic is flowing and how it’s timed is what tells the AI whether something is malicious or not.

Elias: But Priya has been asking about what that data actually shows, so we need to dig deeper into whether focusing only on quantity really captures the complexity of an attack twenty-one.

Nadia: Exactly, because if we only rely on quantity, we might miss subtle attacks where the volume is low but the timing or hybrid features are telling a different story.

Elias: So, while they built a solid baseline with GeNIS and found efficient models that rely heavily on quantity and time features, the paper points toward needing more diverse data to really make these AI NIDS robust for real-world use nineteen.

Nadia: And the big implication is that this dataset can serve as a reliable testbed, but it’s not the final destination; we need larger datasets and tests against adversarial settings next.

The paper's improvements: Nadia: So, the paper isn't just stopping there; they are suggesting some real improvements to how we use this GeNIS dataset for AI NIDS eighteen.

Elias: Right, they're basically saying that focusing solely on those quantity features might be too narrow because it doesn't capture the full picture of what an attack looks like twenty.

Nadia: They suggest combining multiple datasets, not just relying on this one GeNIS set, to get a broader view of network behavior across different scenarios twenty.

Elias: That makes sense from a cryptographer's point of view; you need enough parameters and variety in your training data so that the mathematical proof holds up under more complex inputs twenty.

Nadia: They also push for looking into how detection capabilities transfer to completely unseen network traffic, which is crucial because existing models often fail when they see something new twenty.

Elias: That goes straight to the issue of generalization; if a model only knows what it learned from GeNIS, it won't perform well on something entirely different twenty.

Nadia: And they stress that we need to test these AI detection systems in adversarial settings, which is testing them against people trying to actively fool the system twenty.

Elias: That’s where things get interesting for proof parameters; seeing how a model breaks when it's intentionally tricked gives us a much harder problem to solve, which is good for security research twenty.

Nadia: So, the implication is that we need to move beyond just validating this specific dataset and start building systems that can handle messy, real-world data twenty.

Elias: I agree; the current work shows a path forward by establishing this baseline, but the next big hurdle is making those models resilient enough for actual deployment twenty.

Nadia: So we have a validated dataset and efficient models built on it, but now we need to make them tough enough to survive real-world attacks twenty.

Nadia: That means the GeNIS study isn't the end of the road; it's just step one in a bigger research pipeline for building truly practical AI NIDS twenty.

Elias: Exactly, because establishing this reliable baseline with clear feature importance gives us a solid foundation before we start pushing those models into more challenging environments twenty.

Nadia: We’re looking at how much effort it takes to move from this validation study to a system that can actually handle novel threats twenty. It’s about the engineering challenge now, not just the data collection itself.

Conclusion: Nadia: So, to wrap up, this study on "Binary and Multiclass Cyberattack Classification on GeNIS Dataset" shows that even a focused feature selection process can give us a very reliable baseline for AI NIDS twenty-one.

Elias: Yeah, it’s about proving that you can distill complex traffic data down to the most important behavioral indicators without losing the ability to detect different attack types twenty-one.

Nadia: The implication is that we have a tool here—a GeNIS baseline—that proves AI models can actually learn from network patterns, but we still have a lot of work left on making those models truly robust twenty-one.

Elias: Absolutely; the authors themselves pointed out that this is just the start, and they stressed the need to test these detection capabilities against adversarial settings for real deployment twenty-one.

Priya: I just think what stands out is how much reliance they placed on quantity features, like traffic volume, which makes me wonder if we’re missing some stealthier attacks that don't rely on high packet counts twenty-one.

Nadia: That’s a fair point, Priya; it shows us exactly where the current limitations of this approach are when we look at the data itself twenty-one.

Elias: Well, if you look at the methodology, they did a solid job optimizing for efficiency too, showing that those ensemble models ran faster than some deep learning approaches in binary classification twenty-one.

Nadia: It’s about finding that sweet spot between high accuracy and low computational cost, which is exactly what we need when we think about deploying detection systems on actual hardware twenty-one.

Priya: And that efficiency is good, but the real question for us is whether those efficient models are still accurate when they encounter a completely new type of network activity twenty-one.

Elias: That’s where the future work comes in; we need to focus on combining datasets and testing transferability to see if these behavioral patterns hold up elsewhere twenty-one.

Nadia: So, we’ve validated GeNIS as a solid starting point for AI NIDS, but it's time to start looking at how those models handle the messy reality of new data twenty-one.

Elias: Right; the path forward is definitely moving toward more diverse training sets and testing those detection capabilities in hostile environments twenty-one.

More episodes

← Home