Binary and Multiclass Cyberattack Classification on GeNIS Dataset
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "Binary and Multiclass Cyberattack Classification on GeNIS Dataset".
Title and authors: Nadia: So this paper is about validating a dataset called GeNIS for AI-based Network Intrusion Detection Systems to use as a baseline for future benchmarks. It looks like they're trying to make sure the data itself is solid before we train these models
nine −eight −six thousand six hundred thirty −nine thousand nine hundred thirty-nine: <ref:2511.08660#pg1>.
Elias: Right, and the authors are Miguel Silva, Daniela Pinto, João Vitorino, Eva Maia, Isabel Praça and Ivone Amorim and Maria João Viamonte GECAD from ISEP at the Polytechnic of Porto in Portugal
nine −eight −six thousand six hundred thirty −nine thousand nine hundred thirty-nine: <ref:2511.08660#pg1>. It's important to know who's doing the work behind this evaluation.
Priya: What I want to know is what makes this dataset reliable enough for real-world AI applications, because we know that bad data leads to bad detection models <ref:2511.08660#pg2>.
Nadia: Well, they set up GeNIS specifically for AI-based NIDS focusing on typical user behavior and network protocols of small to medium sized organizations <ref:2511.08660#pg2>. They have eight attack scenarios and three benign activity scenarios in it <ref:2511.08660#pg3>.
Elias: And they started with a total of one hundred twenty-five features extracted from the raw recordings using the HERA flow exporter, which is a specific tool used for that extraction process <ref:2511.08660#pg3>. They grouped these into general, time-based, quantity-based, hybrid-based and context-based categories <ref:2511.08660#pg3>.
Priya: Before they did anything else, they had to one-hot encode categorical features like State and Protocol <ref:2511.08660#pg3>. That sounds like a standard first step for getting data ready for classification, but we need to see how that affects the final results.
Nadia: They then used five feature selection methods—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation and Dispersion Ratio—to find the most important features <ref:2511.08660#pg1>. They combined their scores by summing them up for each feature <ref:2511.08660#pg2>.
Elias: And the key result there is they selected sixteen most significant features that accounted for about seventy percent of the total feature importance <ref:2511.08660#pg2>. That’s a big dimensionality reduction from the start, which sounds like it should make training faster <ref:2511.08660#pg2>.
Priya: Seventy percent is a lot of features to boil down to, so I'm curious what those sixteen features actually represent and if we can trust that selection process <ref:2511.08660#pg3>.
Title and authors: Nadia: The next step was training three decision tree ensembles—Random Forest, Extreme Gradient Boosting and Light Gradient Boosting Machine—and two deep neural networks, Long Short-Term Memory and Multilayer Perceptron <ref:2511.08660#pg2>. They used a grid search with five fold cross validation to find the best settings for each model <ref:2511.08660#pg2>.
Elias: The evaluation metric they focused on was the F1-score for binary classification and the macro averaged F1-score for multiclass classification <ref:2511.08660#pg3>. That tells us they were looking at both attack detection and distinguishing between different attack types, which is important <ref:2511.08660#pg3>.
Priya: So, after training all those models on the selected features, what did they actually find about how well these models performed in classifying the flows as attacks or benign activity <ref:2511.08660#pg3>?
Nadia: The ensemble models, which included Random Forest XGB Light Gradient Boosting Machine LSTM and Multilayer Perceptron achieved slightly better generalization while still being more efficient than the deep neural networks <ref:2511.08660#pg3>.
Elias: That’s interesting because it suggests that the simpler ensemble models didn't necessarily need all those complex layers of a neural network to perform well <ref:2511.08660#pg3>.
Priya: In binary classification, the paper found that the selected features were roughly balanced between quantity time and hybrid-based types <ref:2511.08660#pg3>. That tells us what kind of behavioral patterns are most important here.
Nadia: Exactly, in multiclass classification, they noted that quantity features were four times more frequent and time-based features were three times more frequent than hybrid ones <ref:2511.08660#pg3>. That gives us a hint about the structure of the malicious traffic they are trying to identify.
Elias: And SHAP analysis, which is a technique that helps explain model predictions by showing how much each feature contributes, revealed that the models predominantly relied on quantity-based features <ref:2511.08660#pg3>. That points strongly toward traffic volume or packet counts being key indicators of malicious behavior.
Priya: If quantity features are dominating, that makes sense for network traffic where high volumes can sometimes be a sign of an attack, but I'm wondering if this reliance on quantity might miss subtle, low-volume attacks <ref:2511.08660#pg3>.
Nadia: The paper suggests good generalization potential because the reduced feature set was based on behavior rather than topology specific ones <ref:2511.08660#pg3>. This means if you take these models and apply them to a different dataset, they should still work reasonably well <ref:2511.08660#pg3>.
Title and authors: Elias: They also mentioned that the ensemble models achieved less training and inference time than the full feature set or even the neural networks in binary classification <ref:2511.08660#pg3>. That efficiency gain is pretty practical for deployment, especially if we're looking at real-time detection on a network <ref:2511.08660#pg3>.
Priya: It’s good to know they optimized for speed, but what about the limitations of this entire process? What did the authors mention that suggests where this approach might stop working <ref:2511.08660#pg3>?
Nadia: They point out that while the reduced feature set showed good generalization, it’s still important to continue developing larger and more diverse network traffic datasets <ref:2511.08660#pg3>. And they also suggest combining multiple datasets and looking at how detection capabilities transfer to previously unseen network traffic <ref:2511.08660#pg3>.
Elias: And for us, the crucial next step mentioned is assessing model robustness in adversarial settings as a way toward real-world deployment <ref:2511.08660#pg3>. That takes us from just testing on clean data to seeing if the detection holds up when someone tries to fool it <ref:2511.08660#pg3>.
Priya: So, in summary, they validated GeNIS as a reliable baseline, using a focused feature selection process that highlighted quantity and time features as key indicators <ref:2511.08660#pg3>, and they showed the ensemble models are efficient for deployment <ref:2511.08660#pg3>.
Nadia: Right, it sounds like the GeNIS dataset provides a solid starting point, but we still have to look at more diverse data and adversarial testing to make these AI NIDS truly reliable in practice <ref:2511.08660#pg3>.
Elias: Indeed, the study on "Binary and Multiclass Cyberattack Classification on GeNIS Dataset" gives us a baseline for how feature selection impacts model performance across different architectures <ref:2511.08660#pg2>. That foundation helps us decide which data sources to prioritize next <ref:2511.08660#pg3>.
Priya: I just hope that as we move toward more complex systems, we don't lose sight of the underlying data quality issues they pointed out regarding feature extraction errors from tools like CICFlowMeter <ref:2511.08660#pg3>.
Nadia: We certainly have to keep an eye on those inconsistencies when we build our next datasets for AI training <ref:2511.08660#pg3>. That’s the path forward for making these tools actually useful against modern threats.
Elias: Well, that wraps up our look at the GeNIS dataset and its validation, so we'll be looking at new papers soon <ref:2511.08660#pg2>.
The paper's summary: Nadia: So, basically, this paper is looking at how reliable the GeNIS dataset is when we try to use it to train AI for Network Intrusion Detection Systems and they're using it as a starting point for other research eighteen.
Elias: Yeah, they took a raw traffic recording and turned it into this dataset with eight attack scenarios and three benign ones, which is pretty standard for training data.
Nadia: Right, but the real focus here is on the feature selection process, where they use five different methods—like Information Gain and Chi-Squared Test—to boil down one hundred twenty-five features to just sixteen of them nineteen.
Elias: Sixteen features making up seventy percent of the total importance means they're keeping a lot of behavioral data and throwing out the noise from things like just simple packet counts or timing stuff.
Nadia: And once they got those sixteen features, they trained several models—Random Forest, XGBoost, Light Gradient Boosting Machine, plus two deep neural networks—to see how well they could classify those network flows as either an attack or normal activity twenty.
Elias: It’s interesting that the ensemble models like Random Forest and XGBoost ended up performing better than the deep learning ones in some of these tests, even though they were simpler architectures.
Nadia: That makes sense, especially if we look at how much training and running time those models needed compared to the neural networks twenty. It's about efficiency for real-world deployment.
Elias: And when they dug into why the models made their decisions using something called SHAP analysis, they found that quantity-based features like traffic volume were the main drivers of those classifications twenty-one.
Nadia: That suggests that for this specific data, just looking at how much traffic is flowing and how it’s timed is what tells the AI whether something is malicious or not.
Elias: But Priya has been asking about what that data actually shows, so we need to dig deeper into whether focusing only on quantity really captures the complexity of an attack twenty-one.
Nadia: Exactly, because if we only rely on quantity, we might miss subtle attacks where the volume is low but the timing or hybrid features are telling a different story.
Elias: So, while they built a solid baseline with GeNIS and found efficient models that rely heavily on quantity and time features, the paper points toward needing more diverse data to really make these AI NIDS robust for real-world use nineteen.
Nadia: And the big implication is that this dataset can serve as a reliable testbed, but it’s not the final destination; we need larger datasets and tests against adversarial settings next.
The paper's improvements: Nadia: So, the paper isn't just stopping there; they are suggesting some real improvements to how we use this GeNIS dataset for AI NIDS eighteen.
Elias: Right, they're basically saying that focusing solely on those quantity features might be too narrow because it doesn't capture the full picture of what an attack looks like twenty.
Nadia: They suggest combining multiple datasets, not just relying on this one GeNIS set, to get a broader view of network behavior across different scenarios twenty.
Elias: That makes sense from a cryptographer's point of view; you need enough parameters and variety in your training data so that the mathematical proof holds up under more complex inputs twenty.
Nadia: They also push for looking into how detection capabilities transfer to completely unseen network traffic, which is crucial because existing models often fail when they see something new twenty.
Elias: That goes straight to the issue of generalization; if a model only knows what it learned from GeNIS, it won't perform well on something entirely different twenty.
Nadia: And they stress that we need to test these AI detection systems in adversarial settings, which is testing them against people trying to actively fool the system twenty.
Elias: That’s where things get interesting for proof parameters; seeing how a model breaks when it's intentionally tricked gives us a much harder problem to solve, which is good for security research twenty.
Nadia: So, the implication is that we need to move beyond just validating this specific dataset and start building systems that can handle messy, real-world data twenty.
Elias: I agree; the current work shows a path forward by establishing this baseline, but the next big hurdle is making those models resilient enough for actual deployment twenty.
Nadia: So we have a validated dataset and efficient models built on it, but now we need to make them tough enough to survive real-world attacks twenty.
Nadia: That means the GeNIS study isn't the end of the road; it's just step one in a bigger research pipeline for building truly practical AI NIDS twenty.
Elias: Exactly, because establishing this reliable baseline with clear feature importance gives us a solid foundation before we start pushing those models into more challenging environments twenty.
Nadia: We’re looking at how much effort it takes to move from this validation study to a system that can actually handle novel threats twenty. It’s about the engineering challenge now, not just the data collection itself.
Conclusion: Nadia: So, to wrap up, this study on "Binary and Multiclass Cyberattack Classification on GeNIS Dataset" shows that even a focused feature selection process can give us a very reliable baseline for AI NIDS twenty-one.
Elias: Yeah, it’s about proving that you can distill complex traffic data down to the most important behavioral indicators without losing the ability to detect different attack types twenty-one.
Nadia: The implication is that we have a tool here—a GeNIS baseline—that proves AI models can actually learn from network patterns, but we still have a lot of work left on making those models truly robust twenty-one.
Elias: Absolutely; the authors themselves pointed out that this is just the start, and they stressed the need to test these detection capabilities against adversarial settings for real deployment twenty-one.
Priya: I just think what stands out is how much reliance they placed on quantity features, like traffic volume, which makes me wonder if we’re missing some stealthier attacks that don't rely on high packet counts twenty-one.
Nadia: That’s a fair point, Priya; it shows us exactly where the current limitations of this approach are when we look at the data itself twenty-one.
Elias: Well, if you look at the methodology, they did a solid job optimizing for efficiency too, showing that those ensemble models ran faster than some deep learning approaches in binary classification twenty-one.
Nadia: It’s about finding that sweet spot between high accuracy and low computational cost, which is exactly what we need when we think about deploying detection systems on actual hardware twenty-one.
Priya: And that efficiency is good, but the real question for us is whether those efficient models are still accurate when they encounter a completely new type of network activity twenty-one.
Elias: That’s where the future work comes in; we need to focus on combining datasets and testing transferability to see if these behavioral patterns hold up elsewhere twenty-one.
Nadia: So, we’ve validated GeNIS as a solid starting point for AI NIDS, but it's time to start looking at how those models handle the messy reality of new data twenty-one.
Elias: Right; the path forward is definitely moving toward more diverse training sets and testing those detection capabilities in hostile environments twenty-one.
GECAD, ISEP, Polytechnic of Porto
cs.CR, cs.AI
Submitted: 2025-11-11
Updated: 2025-11-11
Comments: 17 pages, 12 tables, FPS 2025 conference
DOI: 10.1007/978-3-032-20018-1_18
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 80/100
The gist: The study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.
Key concepts
- GeNIS Dataset
- This dataset was specifically created for AI NIDS applications, focusing on typical user behavior and network protocols from small to medium organizations. It contains eight attack scenarios and three benign activities, serving as a baseline for testing detection systems.
- Feature Selection Methods
- Five techniques—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio—were used to filter 125 raw features. This process aimed to identify the most relevant features that best distinguish between network attacks and normal traffic.
- Ensemble Models
- Models like Random Forest (RF), XGBoost (XGB), and LGBM were trained on the selected features. These models combine multiple decision trees to improve accuracy, achieving slightly better generalization than deep learning models while remaining computationally efficient for both training and running predictions.
- SHAP Analysis
- This technique was used to explain model predictions by assigning a score to each feature. It revealed that the AI models primarily relied on quantity-based features (like traffic volume) and helped identify which specific behavioral features were most critical for classifying malicious network flows.
Terminology
Summary
The study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. p. 0 However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of
their training data, the lack of diverse and up-to-date datasets hinders
their generalization capability to detect malicious activity in previously
unseen network traffic. This study presents an experimental validation of
the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.Where the paper says it
Dataset Context and Feature Engineering
The GeNIS dataset was created specifically for AI-based NIDS applications, focusing on reflecting typical user behavior, services, and network protocols of small to medium sized organizations. It consists of eight attack scenarios and three benign activity scenarios. p. 3 However, due to significant flaws
discovered in widely used datasets, it is crucial to evaluate these models’ and
feature selection methods’ performance on more recent, reliable datasets free
from such anomalies. 3 Dataset Analysis
The GeNIS dataset consists of eight attack scenarios and three benign activity
scenarios. Finally, the background activity scenario captures periods of no user interaction
and represents idle traffic collected over the weekend. A total of 125 features were extracted from the raw recordings of the GeNIS
dataset using the HERA flow exporter. Finally, the background activity scenario captures periods of no user interaction
and represents idle traffic collected over the weekend. A total of 125 features were extracted from the raw recordings of the GeNIS
dataset using the HERA flow exporter. Meanwhile, hybrid features combine time and quantity aspects, and
context-based features reflect the specific characteristics of the network topology. Among the 17 general features, 6 (FlowID, Rank, Seq, AutoId, TcpOpt, and
Cause) are generated by Argus and do not meaningfully contribute to understanding traffic behavior. Initial pre-processing of the GeNIS dataset was performed to ensure that no
empty rows or columns were present, and the pre-processed dataset was provided
alongside the original and raw traffic captures. Additionally, categorical features
such as State, Flags, and Protocol were one-hot encoded, resulting in 87 total
features. Initial pre-processing of the GeNIS dataset was performed to ensure that no
empty rows or columns were present, and the pre-processed dataset was provided
alongside the original and raw traffic captures. Additionally, categorical features
such as State, Flags, and Protocol were one-hot encoded, resulting in 87 total
features.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Feature Selection Methodology
Five feature selection techniques—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio—were combined to identify the most relevant features of GeNIS and reduce its dimensionality. The results from each method were normalized and aggregated by summing the corresponding scores for each feature. p. 5 These methods include Information Gain, Chi-Squared Test, Recursive Feature
Elimination, Mean Absolute Deviation, and Dispersion Ratio. The results from
each feature selection method were first normalized and then aggregated by summing the corresponding scores for each feature. The results from
each feature selection method were first normalized and then aggregated by summing the corresponding scores for each feature. Lastly, the 16 most significant
features with the highest combined scores were selected.Where the paper says it
Where the paper says it
Model Training and Evaluation
Three decision tree ensembles (Random Forest (RF), Extreme Gradient Boosting (XGB), and Light Gradient Boosting Machine (LGBM)) and two deep neural networks (Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP)) were trained for both binary and multiclass classification tasks. The optimal configuration for each model was determined through a grid search involving an exhaustive evaluation of combinations of hyperparameters using 5-fold cross-validation. p. 6 Three ML models based on decision tree ensembles, RF, XGB, and LGBM,
and two DL models based on deep neural networks, LSTM, and MLP, were considered. The optimal configuration for each model was determined through a grid
search involving an exhaustive evaluation of combinations of hyperparameters
using 5-fold cross-validation. Three ML models based on decision tree ensembles, RF, XGB, and LGBM,
and two DL models based on deep neural networks, LSTM, and MLP, were considered. The optimal configuration for each model was determined through a grid
search involving an exhaustive evaluation of combinations of hyperparameters
using 5-fold cross-validation. This process was applied to the full set of features
as well as to reduced sets obtained through feature selection for both binary and
multiclass classification tasks. Performance was assessed using the F1-score for binary classification and the
macro-averaged F1-score for multiclass classification to ensure balanced evaluation across all classes.Where the paper says it
Where the paper says it
Where the paper says it
Classification Performance Findings
The ML ensembles achieved slightly better generalization while remaining more efficient than DL models. In binary classification, the selected features were roughly balanced between quantity, time, and hybrid-based types. p. 9 The selected features are listed in Table 7. Feature selection for binary classification
Feature Type Selected Features
Quantity-based DstTCPBase SrcTCPBase DstWin SrcWin
TotBytes TotPkts
Time-based Dur Min Mean RunTime
Sum
Hybrid-based DstLoad Load Rate SrcLoad
SrcRate
In the binary classification scenario, models trained using the full feature set
demonstrated strong overall performance on all non-time-based metrics, with
most models achieving near-perfect scores. cation of which features were more relevant and the analysis of the combined
impact of the different time, quantity, and hybrid-based behavioral features of
the GeNIS dataset. 2 Binary classification results
In the binary classification task, the feature selection methods identified two
features as the most significant, which together accounted for 56% of the total
feature importance.Where the paper says it
Where the paper says it
Model Comparison and Generalization
The ensemble models consistently achieved higher scores and required less training and inference time with both the full feature set and the reduced subset. p. 9 Regarding TT, the ensemble
models required less than half the TT of the neural networks and predicted the
test set approximately ten times faster. When using a reduced subset of features, there was a slight decline in performance across non-time-based metrics. Among the 38 quantity-based features, Ssaddr and
Sdaddr are specific to HERA, representing the number of connections involving
the same service and source/destination address. While these features are potentially useful for identifying attacks, they are highly dependent on the network’s
topology and attacks, and should be excluded to ensure model generalization
across different environments. Time-based features include the timestamp marking the start of
a flow, the duration of inactivity between packets, and inter-arrival times, all of
which offer insights into the temporal aspects of network behavior. Quantitybased features include packet counts, packet sizes, and byte volumes, which are
measured separately for each flow’s source (the sender) and destination (the responder). Time-based features include the timestamp marking the start of
a flow, the duration of inactivity between packets, and inter-arrival times, all of
which offer insights into the temporal aspects of network behavior. Quantitybased features include packet counts, packet sizes, and byte volumes, which are
measured separately for each flow’s source (the sender) and destination (the responder). 1 Introduction
The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be
increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown
data based on previous patterns. 1 Introduction
The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be
increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown
data based on previous patterns.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Future Work
It is important to continue the research and development efforts to create larger and more diverse network traffic datasets. p. 2 This dataset offers a diverse
range of benign and malicious traffic types, and its modular design allows organizations to tailor the dataset to their unique network scenarios, either by
1
https://www. ca/cic/research/applications. The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the
increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of
their training data, the lack of diverse and up-to-date datasets hinders
their generalization capability to detect malicious activity in previously
unseen network traffic. The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the
increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of
their training data, the lack of diverse and up-to-date datasets hinders
their generalization capability to detect malicious activity in previously
unseen network traffic.Where the paper says it
Where the paper says it
Where the paper says it
Acknowledgment
This work was supported by the PC2phish project, which has received funding from FCT with Refª: 2024.07648.IACDC. This work has also received funding from UIDB/00760/2020. p. 1 Section 4 describes the experimental setup, presents and discusses the obtained
results of binary and multiclass classification, followed by a discussion of the
models’ performance and key findings. Finally, Section 5 presents the main conclusions and future work.Where the paper says it
References
The references listed include foundational works on Random Forests, XGBoost, LSTM, and various intrusion detection datasets like CICIDS2017. p. 3 To evaluate the real-world effectiveness and practicality of GeNIS in organizational networks, it is essential to validate the performance of AI models trained
on it. Previous studies have used datasets such as CICIDS2017 to explore various deep learning models, including Deep Neural Networks, LSTM networks,
and Convolutional Neural Networks, while reporting a binary classification performance accuracy of over 94% using these models [12]. To evaluate the real-world effectiveness and practicality of GeNIS in organizational networks, it is essential to validate the performance of AI models trained
on it. Previous studies have used datasets such as CICIDS2017 to explore various deep learning models, including Deep Neural Networks, LSTM networks,
and Convolutional Neural Networks, while reporting a binary classification performance accuracy of over 94% using these models [12]. To evaluate the real-world effectiveness and practicality of GeNIS in organizational networks, it is essential to validate the performance of AI models trained
on it. Previous studies have used datasets such as CICIDS2017 to explore various deep learning models, including Deep Neural Networks, LSTM networks,
and Convolutional Neural Networks, while reporting a binary classification performance accuracy of over 94% using these models [12]. This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and
AI-based NIDS in small to medium sized organizations. After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were
employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute
Deviation, and Dispersion Ratio. This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and
AI-based NIDS in small to medium sized organizations. After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were
employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute
Deviation, and Dispersion Ratio. In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory
(LSTM) and Multilayer Perceptron (MLP). Different versions of each model
were trained on the full feature set and a subset of selected features derived
from selection techniques. In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory
(LSTM) and Multilayer Perceptron (MLP). Different versions of each model
were trained on the full feature set and a subset of selected features derived
from selection techniques. Recent efforts have focused on standardizing the dataset creation process for
NIDS by introducing the all-in-one HERA flow exporter. HERA was developed
in response to issues identified with CICFlowMeter, which has been linked to
errors in several popular datasets, being able to generate flow records and extract
relevant features consistently and reliably [21]. Recent efforts have focused on standardizing the dataset creation process for
NIDS by introducing the all-in-one HERA flow exporter. HERA was developed
in response to issues identified with CICFlowMeter, which has been linked to
errors in several popular datasets, being able to generate flow records and extract
relevant features consistently and reliably [21]. This study presents an experimental validation of
the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information
Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute
Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more
computationally efficient detection. This study presents an experimental validation of
the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information
Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute
Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more
computationally efficient detection. However, a notable drawback was the significant increase in FPR. Some
doubled their FPR, while others, like the MLP, experienced nearly a tenfold increase. However, a notable drawback was the significant increase in FPR. Some
doubled their FPR, while others, like the MLP, experienced nearly a tenfold increase.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Bibliography
The bibliography includes standard machine learning texts, such as Breiman's Random Forests paper. It also includes papers detailing the implementation of XGBoost and LGBM. p. 5 All experiments were conducted on a standard laptop
with 16GB of RAM, a 6-core CPU and 8GB GPU. The implementation was
developed in Python and the following libraries were used: numpy and pandas
for general data manipulation, xgboost for the implementation of XGB, lightgbm
for LGBM, tensorflow for building the LSTM and MLP, and scikit-learn for the
implementation of RF, the StandardScaler, and feature selection. All experiments were conducted on a standard laptop
with 16GB of RAM, a 6-core CPU and 8GB GPU. The implementation was
developed in Python and the following libraries were used: numpy and pandas
for general data manipulation, xgboost for the implementation of XGB, lightgbm
for LGBM, tensorflow for building the LSTM and MLP, and scikit-learn for the
implementation of RF, the StandardScaler, and feature selection. The ML models considered for this study were decision tree ensemble models commonly used in NIDS: Random Forest (RF), Extreme Gradient Boosting
(XGB), and Light Gradient Boosting Machine (LGBM). In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory
(LSTM) and Multilayer Perceptron (MLP). The ML models considered for this study were decision tree ensemble models commonly used in NIDS: Random Forest (RF), Extreme Gradient Boosting
(XGB), and Light Gradient Boosting Machine (LGBM). In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory
(LSTM) and Multilayer Perceptron (MLP). Keywords: Feature selection
· Efficiency
· Machine learning
· Deep
learning
· Cybersecurity
· GeNIS
· NIDS. 1 Introduction
The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be
increasingly ineffective at identifying emerging or previously unseen threats [19]. Keywords: Feature selection
· Efficiency
· Machine learning
· Deep
learning
· Cybersecurity
· GeNIS
· NIDS. 1 Introduction
The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be
increasingly ineffective at identifying emerging or previously unseen threats [19]. Different tools may extract different values for the same feature
from the same raw data, which can introduce errors into the final dataset [22]. A prominent example is the CICIDS2017 dataset [30], which is widely used in
benchmarking studies [2, 20]. Different tools may extract different values for the same feature
from the same raw data, which can introduce errors into the final dataset [22]. A prominent example is the CICIDS2017 dataset [30], which is widely used in
benchmarking studies [2, 20]. Overall, the obtained results indicate
that the GeNIS dataset supports intelligent intrusion detection and cyberattack classification with time-based and quantity-based behavioral
features. Keywords: Feature selection
· Efficiency
· Machine learning
· Deep
learning
· Cybersecurity
· GeNIS
· NIDS. Overall, the obtained results indicate
that the GeNIS dataset supports intelligent intrusion detection and cyberattack classification with time-based and quantity-based behavioral
features. Keywords: Feature selection
· Efficiency
· Machine learning
· Deep
learning
· Cybersecurity
· GeNIS
· NIDS.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The study presented an experimental validation and evaluation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. p. 0 However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of
their training data, the lack of diverse and up-to-date datasets hinders
their generalization capability to detect malicious activity in previously
unseen network traffic. This study presents an experimental validation of
the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. This study presents an experimental validation of
the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information
Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute
Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more
computationally efficient detection. 2 Binary classification results
In the binary classification task, the feature selection methods identified two
features as the most significant, which together accounted for 56% of the total
feature importance. Based on these results, a final subset of 16 features was
selected from the original set, representing 70% of the cumulative importance. The BinaryLabel distinguishes between benign and malicious traffic; the CategoryLabel provides a
general classification, separating traffic into categories such as benign, bruteforce,
DoS, and reconnaissance, also known as recon. The SubCategoryLabel offers a
more granular view by differentiating between three benign scenarios, identifying
the specific protocols involved in bruteforce and recon attacks, and distinguishing
the types of DoS attacks carried out by attackers. Similarly, LSTM models
have demonstrated strong results on older datasets, such as KDD99, achieving
over 95% accuracy and F1-scores for binary classification across different model
configurations, both with and without feature selection. For multiclass classification, the performance remained high with F1-scores above 91% [15]. Similarly, LSTM models
have demonstrated strong results on older datasets, such as KDD99, achieving
over 95% accuracy and F1-scores for binary classification across different model
configurations, both with and without feature selection. For multiclass classification, the performance remained high with F1-scores above 91% [15]. Feature selection for binary classification
Feature Type Selected Features
Quantity-based DstTCPBase SrcTCPBase DstWin SrcWin
TotBytes TotPkts
Time-based Dur Min Mean RunTime
Sum
Hybrid-based DstLoad Load Rate SrcLoad
SrcRate
In the binary classification scenario, models trained using the full feature set
demonstrated strong overall performance on all non-time-based metrics, with
most models achieving near-perfect scores. The tree-based ensemble models
slightly outperformed the neural networks in terms of F1S, while the neural networks matched or surpassed them in terms of FPR. Feature selection for binary classification
Feature Type Selected Features
Quantity-based DstTCPBase SrcTCPBase DstWin SrcWin
TotBytes TotPkts
Time-based Dur Min Mean RunTime
Sum
Hybrid-based DstLoad Load Rate SrcLoad
SrcRate
In the binary classification scenario, models trained using the full feature set
demonstrated strong overall performance on all non-time-based metrics, with
most models achieving near-perfect scores. The tree-based ensemble models
slightly outperformed the neural networks in terms of F1S, while the neural networks matched or surpassed them in terms of FPR.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
Binary and Multiclass Cyberattack Classification on GeNIS Dataset Miguel Silva Daniela Pinto João Vitorino Eva Maia Isabel Praça Ivone Amorim, and Maria João Viamonte GECAD, ISEP, Polytechnic of Porto, rua Dr. António Bernardino de Almeida, 4249-015 Porto, Portugal. p. 0 Binary and Multiclass Cyberattack Classification
on GeNIS Dataset
Miguel Silva
⋆1[0009
−0008
−6630
−9939], Daniela Pinto1[0009
−0000
−3003
−6694], João
Vitorino1[0000
−0002
−4968
−3653], Eva Maia1[0000
−0002
−8075
−531
X
]
, Isabel
Praça1[0000
−0002
−2519
−9859]and Ivone Amorim1[0000
−0001
−6102
−6165], and Maria
João Viamonte1[0000
−0002
−6464
−3529]
GECAD, ISEP, Polytechnic of Porto, rua Dr. António Bernardino de Almeida,
4249-015 Porto, Portugal
Abstract. António Bernardino de Almeida,
4249-015 Porto, Portugal
Abstract. The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the
increasing sophistication of cyberattacks. The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the
increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of
their training data, the lack of diverse and up-to-date datasets hinders
their generalization capability to detect malicious activity in previously
unseen network traffic. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of
their training data, the lack of diverse and up-to-date datasets hinders
their generalization capability to detect malicious activity in previously
unseen network traffic. This study presents an experimental validation of
the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Introduction
The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown data based on previous patterns. p. 0 1 Introduction
The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be
increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown
data based on previous patterns. Silva et al. p. 0 Binary and Multiclass Cyberattack Classification
on GeNIS Dataset
Miguel Silva
⋆1[0009
−0008
−6630
−9939], Daniela Pinto1[0009
−0000
−3003
−6694], João
Vitorino1[0000
−0002
−4968
−3653], Eva Maia1[0000
−0002
−8075
−531
X
]
, Isabel
Praça1[0000
−0002
−2519
−9859]and Ivone Amorim1[0000
−0001
−6102
−6165], and Maria
João Viamonte1[0000
−0002
−6464
−3529]
GECAD, ISEP, Polytechnic of Porto, rua Dr. António Bernardino de Almeida,
4249-015 Porto, Portugal
Abstract. Silva et al. Learning (ML) and Deep Learning (DL) models requires access to accurate and
representative data reflecting current network services, protocols, and behaviors.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Related Work
Creating such datasets requires several key steps, including collecting, extracting, and analyzing traffic data. p. 1 2 Related Work
To protect organizational networks from malicious actors and develop robust
NIDS that can secure critical assets, numerous studies have produced datasets
that are commonly used to train AI models, particularly ML and DL approaches,
to identify malicious patterns in network traffic. Creating such datasets requires
several key steps, including collecting, extracting, and analyzing traffic data,
which can be done using integrated solutions or a combination of specialized
tools. Creating such datasets requires
several key steps, including collecting, extracting, and analyzing traffic data,
which can be done using integrated solutions or a combination of specialized
tools. However, inconsistencies and incompatibilities among these tools can lead Different tools may extract different values for the same feature
from the same raw data, which can introduce errors into the final dataset [22]. A prominent example is the CICIDS2017 dataset [30], which is widely used in
benchmarking studies [2, 20]. A prominent example is the CICIDS2017 dataset [30], which is widely used in
benchmarking studies [2, 20]. Due to its popularity, several researchers have examined this dataset and identified significant inaccuracies, prompting the release
of corrected versions. [16] identified even more issues
than previously reported, including incorrectly labeled port scan attacks and
duplicated traffic, which make feature extraction and labeling more difficult. A more recent dataset, HIKARI2021 [6] , was developed using the same set of
features as CICIDS2017, but it employed Zeek2 as the flow exporter. HERA was developed
in response to issues identified with CICFlowMeter, which has been linked to
errors in several popular datasets, being able to generate flow records and extract
relevant features consistently and reliably [21]. Following the creation of HERA and since prior datasets offer so many challenges, the GeNIS dataset was created specifically for AI-based NIDS applications.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Dataset Analysis
The GeNIS dataset consists of eight attack scenarios and three benign activity scenarios. A total of 125 features were extracted from the raw recordings of the GeNIS dataset using the HERA flow exporter. Three of these features are label-related and suitable for binary and multiclass classification tasks. p. 4 A total of 125 features were extracted from the raw recordings of the GeNIS
dataset using the HERA flow exporter. Three of these features are label-related
and suitable for binary and multiclass classification tasks. The SubCategoryLabel offers a
more granular view by differentiating between three benign scenarios, identifying
the specific protocols involved in bruteforce and recon attacks, and distinguishing
the types of DoS attacks carried out by attackers. The remaining features can be grouped into five categories according to
their nature.Where the paper says it
Where the paper says it
Experimental Setup
Five feature selection methods were applied to identify and rank the most impactful features. These methods include Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio. p. 5 Five feature selection methods were applied to identify and rank the most impactful features, targeting both binary and multiclass classification objectives. These methods include Information Gain, Chi-Squared Test, Recursive Feature
Elimination, Mean Absolute Deviation, and Dispersion Ratio. A measure based on the calculation of the square root of
the ratio between the sum of the squared deviations and the overall dispersion
within the dataset [28]. Three ML models based on decision tree ensembles, RF, XGB, and LGBM,
and two DL models based on deep neural networks, LSTM, and MLP, were considered.Where the paper says it
Where the paper says it
Results and Discussion
All models reached high accuracy and F1-scores, with the ML ensembles achieving slightly better generalization while remaining more efficient than DL models. In binary classification, the feature selection methods identified two features as the most significant, which together accounted for 56% of the total feature importance. p. 9 cation of which features were more relevant and the analysis of the combined
impact of the different time, quantity, and hybrid-based behavioral features of
the GeNIS dataset. 2 Binary classification results
In the binary classification task, the feature selection methods identified two
features as the most significant, which together accounted for 56% of the total
feature importance. The results obtained are described in Table 8, where the best results
between the model using all features and the smaller subset are highlighted. SHAP analysis of binary classification models using a reduced feature subset
reveals that predictions were driven primarily by quantity-related features. Similarly, LSTM models
have demonstrated strong results on older datasets, such as KDD99, achieving
over 95% accuracy and F1-scores for binary classification across different model
configurations, both with and without feature selection. For multiclass classification, the performance remained high with F1-scores above 91% [15]. 001
Batch size 32
Epochs 30
Early stopping patience 3
To evaluate the performance of each model on the testing set, the following
metrics were considered: F1-score (F1S), accuracy (ACC), recall (RCL), precision (PRC), false positive rate (FPR), average time per epoch (TE), training
time (TT), and inference time (IT). Optimal performance would be represented
as a score of 100% on F1S, ACC, RCL, and PRC, a score of 0% on FPR, and the
lowest possible values for TT and IT metrics.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
A subset of 16 features, accounting for approximately 70% of the total feature importance, was selected by combining the outputs of these methods. The RF, XGB, LGBM, LSTM, and MLP models obtained high results across both binary and multiclass classification tasks. Since the reduced feature set only slightly impacted performance and was composed of behavior-based features rather than topology-specific ones, the results suggest good generalization potential to other datasets. The study presented an experimental validation and evaluation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.
--- Page 2 ---
M. Silva et al. Learning (ML) and Deep Learning (DL) models requires access to accurate and representative data reflecting current network services, protocols, and behaviors. Since existing datasets lack data diversity and contain network traffic that is becoming outdated, it is increasingly difficult to detect modern threats [8, 10]. p. 1 Learning (ML) and Deep Learning (DL) models requires access to accurate and
representative data reflecting current network services, protocols, and behaviors. Since existing datasets lack data diversity and contain network traffic that is becoming outdated, it is increasingly difficult to detect modern threats [8, 10]. Since existing datasets lack data diversity and contain network traffic that is becoming outdated, it is increasingly difficult to detect modern threats [8, 10]. In
addition to becoming outdated, recent studies have also shown that many publicly available datasets contain inconsistencies caused by flaws in their feature
extraction processes [14]. In
addition to becoming outdated, recent studies have also shown that many publicly available datasets contain inconsistencies caused by flaws in their feature
extraction processes [14]. Despite significant efforts to correct these datasets and
release improved versions, discrepancies still arise. This is mainly due to variations in flow exporters used to process the same raw network captures that
initially introduced these errors [21]. This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and
AI-based NIDS in small to medium sized organizations. This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and
AI-based NIDS in small to medium sized organizations. After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were
employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute
Deviation, and Dispersion Ratio. After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were
employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute
Deviation, and Dispersion Ratio. Lastly, the reliability of GeNIS for binary and
multiclass classification was evaluated with several ML and DL models, considering standard evaluation metrics and their training and inference times.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
To protect organizational networks from malicious actors and develop robust NIDS that can secure critical assets, numerous studies have produced datasets that are commonly used to train AI models, particularly ML and DL approaches. Creating such datasets requires several key steps, including collecting, extracting, and analyzing traffic data. However, inconsistencies and incompatibilities among these tools can lead to discrepancies. p. 1 Creating such datasets requires
several key steps, including collecting, extracting, and analyzing traffic data,
which can be done using integrated solutions or a combination of specialized
tools. However, inconsistencies and incompatibilities among these tools can lead Binary and Multiclass Cyberattack Classification on GeNIS Dataset 3
to discrepancies. Different tools may extract different values for the same feature
from the same raw data, which can introduce errors into the final dataset [22].Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved feature selection for AI models can reduce dimensionality by combining five methods—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio—to identify the most relevant features of GeNIS. This enables a more computationally efficient detection process as stated in the abstract.
-
Training decision tree ensembles like Random Forest (RF), XGBoost (XGB), and Light Gradient Boosting Machine (LGBM) on selected features allows for high accuracy and F1-scores, with ML ensembles achieving
slightly better generalization while remaining more efficient than DL models.
-
Implementing Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP) deep neural networks allows the AI system to model long-range dependencies in sequential data, even when treating each data point as a one-step sequence. This capability enables the detection of complex nonlinear feature interactions in network traffic.
-
Applying SHapley Additive exPlanations (SHAP) technique analyzes predictions to
provide a score for each feature,
which allows researchers to identifywhich features were more relevant and the analysis of the combined impact of the different time, quantity, and hybrid-based behavioral features.
-
The system can be optimized for efficiency by reducing training and inference times; for example, ensemble models required
less than half the TT of the neural networks
in binary classification while achievingfaster inference with the smaller feature set.
Abstract
The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic. This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more computationally efficient detection. Three decision tree ensembles and two deep neural networks were trained for both binary and multiclass classification tasks. All models reached high accuracy and F1-scores, and the ML ensembles achieved slightly better generalization while remaining more efficient than DL models. Overall, the obtained results indicate that the GeNIS dataset supports intelligent intrusion detection and cyberattack classification with time-based and quantity-based behavioral features.
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data