Binary and Multiclass Cyberattack Classification on GeNIS Dataset

arXiv:2511.08660 · cs.CR, cs.AI · Submitted 2025-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "Binary and Multiclass Cyberattack Classification on GeNIS Dataset".

Title and authors: Nadia: So this paper is about validating a dataset called GeNIS for AI-based Network Intrusion Detection Systems to use as a baseline for future benchmarks. It looks like they're trying to make sure the data itself is solid before we train these models

nine −eight −six thousand six hundred thirty −nine thousand nine hundred thirty-nine: <ref:2511.08660#pg1>.

Elias: Right, and the authors are Miguel Silva, Daniela Pinto, João Vitorino, Eva Maia, Isabel Praça and Ivone Amorim and Maria João Viamonte GECAD from ISEP at the Polytechnic of Porto in Portugal

nine −eight −six thousand six hundred thirty −nine thousand nine hundred thirty-nine: <ref:2511.08660#pg1>. It's important to know who's doing the work behind this evaluation.

Priya: What I want to know is what makes this dataset reliable enough for real-world AI applications, because we know that bad data leads to bad detection models <ref:2511.08660#pg2>.

Nadia: Well, they set up GeNIS specifically for AI-based NIDS focusing on typical user behavior and network protocols of small to medium sized organizations <ref:2511.08660#pg2>. They have eight attack scenarios and three benign activity scenarios in it <ref:2511.08660#pg3>.

Elias: And they started with a total of one hundred twenty-five features extracted from the raw recordings using the HERA flow exporter, which is a specific tool used for that extraction process <ref:2511.08660#pg3>. They grouped these into general, time-based, quantity-based, hybrid-based and context-based categories <ref:2511.08660#pg3>.

Priya: Before they did anything else, they had to one-hot encode categorical features like State and Protocol <ref:2511.08660#pg3>. That sounds like a standard first step for getting data ready for classification, but we need to see how that affects the final results.

Nadia: They then used five feature selection methods—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation and Dispersion Ratio—to find the most important features <ref:2511.08660#pg1>. They combined their scores by summing them up for each feature <ref:2511.08660#pg2>.

Elias: And the key result there is they selected sixteen most significant features that accounted for about seventy percent of the total feature importance <ref:2511.08660#pg2>. That’s a big dimensionality reduction from the start, which sounds like it should make training faster <ref:2511.08660#pg2>.

Priya: Seventy percent is a lot of features to boil down to, so I'm curious what those sixteen features actually represent and if we can trust that selection process <ref:2511.08660#pg3>.

Title and authors: Nadia: The next step was training three decision tree ensembles—Random Forest, Extreme Gradient Boosting and Light Gradient Boosting Machine—and two deep neural networks, Long Short-Term Memory and Multilayer Perceptron <ref:2511.08660#pg2>. They used a grid search with five fold cross validation to find the best settings for each model <ref:2511.08660#pg2>.

Elias: The evaluation metric they focused on was the F1-score for binary classification and the macro averaged F1-score for multiclass classification <ref:2511.08660#pg3>. That tells us they were looking at both attack detection and distinguishing between different attack types, which is important <ref:2511.08660#pg3>.

Priya: So, after training all those models on the selected features, what did they actually find about how well these models performed in classifying the flows as attacks or benign activity <ref:2511.08660#pg3>?

Nadia: The ensemble models, which included Random Forest XGB Light Gradient Boosting Machine LSTM and Multilayer Perceptron achieved slightly better generalization while still being more efficient than the deep neural networks <ref:2511.08660#pg3>.

Elias: That’s interesting because it suggests that the simpler ensemble models didn't necessarily need all those complex layers of a neural network to perform well <ref:2511.08660#pg3>.

Priya: In binary classification, the paper found that the selected features were roughly balanced between quantity time and hybrid-based types <ref:2511.08660#pg3>. That tells us what kind of behavioral patterns are most important here.

Nadia: Exactly, in multiclass classification, they noted that quantity features were four times more frequent and time-based features were three times more frequent than hybrid ones <ref:2511.08660#pg3>. That gives us a hint about the structure of the malicious traffic they are trying to identify.

Elias: And SHAP analysis, which is a technique that helps explain model predictions by showing how much each feature contributes, revealed that the models predominantly relied on quantity-based features <ref:2511.08660#pg3>. That points strongly toward traffic volume or packet counts being key indicators of malicious behavior.

Priya: If quantity features are dominating, that makes sense for network traffic where high volumes can sometimes be a sign of an attack, but I'm wondering if this reliance on quantity might miss subtle, low-volume attacks <ref:2511.08660#pg3>.

Nadia: The paper suggests good generalization potential because the reduced feature set was based on behavior rather than topology specific ones <ref:2511.08660#pg3>. This means if you take these models and apply them to a different dataset, they should still work reasonably well <ref:2511.08660#pg3>.

Title and authors: Elias: They also mentioned that the ensemble models achieved less training and inference time than the full feature set or even the neural networks in binary classification <ref:2511.08660#pg3>. That efficiency gain is pretty practical for deployment, especially if we're looking at real-time detection on a network <ref:2511.08660#pg3>.

Priya: It’s good to know they optimized for speed, but what about the limitations of this entire process? What did the authors mention that suggests where this approach might stop working <ref:2511.08660#pg3>?

Nadia: They point out that while the reduced feature set showed good generalization, it’s still important to continue developing larger and more diverse network traffic datasets <ref:2511.08660#pg3>. And they also suggest combining multiple datasets and looking at how detection capabilities transfer to previously unseen network traffic <ref:2511.08660#pg3>.

Elias: And for us, the crucial next step mentioned is assessing model robustness in adversarial settings as a way toward real-world deployment <ref:2511.08660#pg3>. That takes us from just testing on clean data to seeing if the detection holds up when someone tries to fool it <ref:2511.08660#pg3>.

Priya: So, in summary, they validated GeNIS as a reliable baseline, using a focused feature selection process that highlighted quantity and time features as key indicators <ref:2511.08660#pg3>, and they showed the ensemble models are efficient for deployment <ref:2511.08660#pg3>.

Nadia: Right, it sounds like the GeNIS dataset provides a solid starting point, but we still have to look at more diverse data and adversarial testing to make these AI NIDS truly reliable in practice <ref:2511.08660#pg3>.

Elias: Indeed, the study on "Binary and Multiclass Cyberattack Classification on GeNIS Dataset" gives us a baseline for how feature selection impacts model performance across different architectures <ref:2511.08660#pg2>. That foundation helps us decide which data sources to prioritize next <ref:2511.08660#pg3>.

Priya: I just hope that as we move toward more complex systems, we don't lose sight of the underlying data quality issues they pointed out regarding feature extraction errors from tools like CICFlowMeter <ref:2511.08660#pg3>.

Nadia: We certainly have to keep an eye on those inconsistencies when we build our next datasets for AI training <ref:2511.08660#pg3>. That’s the path forward for making these tools actually useful against modern threats.

Elias: Well, that wraps up our look at the GeNIS dataset and its validation, so we'll be looking at new papers soon <ref:2511.08660#pg2>.

The paper's summary: Nadia: So, basically, this paper is looking at how reliable the GeNIS dataset is when we try to use it to train AI for Network Intrusion Detection Systems and they're using it as a starting point for other research eighteen.

Elias: Yeah, they took a raw traffic recording and turned it into this dataset with eight attack scenarios and three benign ones, which is pretty standard for training data.

Nadia: Right, but the real focus here is on the feature selection process, where they use five different methods—like Information Gain and Chi-Squared Test—to boil down one hundred twenty-five features to just sixteen of them nineteen.

Elias: Sixteen features making up seventy percent of the total importance means they're keeping a lot of behavioral data and throwing out the noise from things like just simple packet counts or timing stuff.

Nadia: And once they got those sixteen features, they trained several models—Random Forest, XGBoost, Light Gradient Boosting Machine, plus two deep neural networks—to see how well they could classify those network flows as either an attack or normal activity twenty.

Elias: It’s interesting that the ensemble models like Random Forest and XGBoost ended up performing better than the deep learning ones in some of these tests, even though they were simpler architectures.

Nadia: That makes sense, especially if we look at how much training and running time those models needed compared to the neural networks twenty. It's about efficiency for real-world deployment.

Elias: And when they dug into why the models made their decisions using something called SHAP analysis, they found that quantity-based features like traffic volume were the main drivers of those classifications twenty-one.

Nadia: That suggests that for this specific data, just looking at how much traffic is flowing and how it’s timed is what tells the AI whether something is malicious or not.

Elias: But Priya has been asking about what that data actually shows, so we need to dig deeper into whether focusing only on quantity really captures the complexity of an attack twenty-one.

Nadia: Exactly, because if we only rely on quantity, we might miss subtle attacks where the volume is low but the timing or hybrid features are telling a different story.

Elias: So, while they built a solid baseline with GeNIS and found efficient models that rely heavily on quantity and time features, the paper points toward needing more diverse data to really make these AI NIDS robust for real-world use nineteen.

Nadia: And the big implication is that this dataset can serve as a reliable testbed, but it’s not the final destination; we need larger datasets and tests against adversarial settings next.

The paper's improvements: Nadia: So, the paper isn't just stopping there; they are suggesting some real improvements to how we use this GeNIS dataset for AI NIDS eighteen.

Elias: Right, they're basically saying that focusing solely on those quantity features might be too narrow because it doesn't capture the full picture of what an attack looks like twenty.

Nadia: They suggest combining multiple datasets, not just relying on this one GeNIS set, to get a broader view of network behavior across different scenarios twenty.

Elias: That makes sense from a cryptographer's point of view; you need enough parameters and variety in your training data so that the mathematical proof holds up under more complex inputs twenty.

Nadia: They also push for looking into how detection capabilities transfer to completely unseen network traffic, which is crucial because existing models often fail when they see something new twenty.

Elias: That goes straight to the issue of generalization; if a model only knows what it learned from GeNIS, it won't perform well on something entirely different twenty.

Nadia: And they stress that we need to test these AI detection systems in adversarial settings, which is testing them against people trying to actively fool the system twenty.

Elias: That’s where things get interesting for proof parameters; seeing how a model breaks when it's intentionally tricked gives us a much harder problem to solve, which is good for security research twenty.

Nadia: So, the implication is that we need to move beyond just validating this specific dataset and start building systems that can handle messy, real-world data twenty.

Elias: I agree; the current work shows a path forward by establishing this baseline, but the next big hurdle is making those models resilient enough for actual deployment twenty.

Nadia: So we have a validated dataset and efficient models built on it, but now we need to make them tough enough to survive real-world attacks twenty.

Nadia: That means the GeNIS study isn't the end of the road; it's just step one in a bigger research pipeline for building truly practical AI NIDS twenty.

Elias: Exactly, because establishing this reliable baseline with clear feature importance gives us a solid foundation before we start pushing those models into more challenging environments twenty.

Nadia: We’re looking at how much effort it takes to move from this validation study to a system that can actually handle novel threats twenty. It’s about the engineering challenge now, not just the data collection itself.

Conclusion: Nadia: So, to wrap up, this study on "Binary and Multiclass Cyberattack Classification on GeNIS Dataset" shows that even a focused feature selection process can give us a very reliable baseline for AI NIDS twenty-one.

Elias: Yeah, it’s about proving that you can distill complex traffic data down to the most important behavioral indicators without losing the ability to detect different attack types twenty-one.

Nadia: The implication is that we have a tool here—a GeNIS baseline—that proves AI models can actually learn from network patterns, but we still have a lot of work left on making those models truly robust twenty-one.

Elias: Absolutely; the authors themselves pointed out that this is just the start, and they stressed the need to test these detection capabilities against adversarial settings for real deployment twenty-one.

Priya: I just think what stands out is how much reliance they placed on quantity features, like traffic volume, which makes me wonder if we’re missing some stealthier attacks that don't rely on high packet counts twenty-one.

Nadia: That’s a fair point, Priya; it shows us exactly where the current limitations of this approach are when we look at the data itself twenty-one.

Elias: Well, if you look at the methodology, they did a solid job optimizing for efficiency too, showing that those ensemble models ran faster than some deep learning approaches in binary classification twenty-one.

Nadia: It’s about finding that sweet spot between high accuracy and low computational cost, which is exactly what we need when we think about deploying detection systems on actual hardware twenty-one.

Priya: And that efficiency is good, but the real question for us is whether those efficient models are still accurate when they encounter a completely new type of network activity twenty-one.

Elias: That’s where the future work comes in; we need to focus on combining datasets and testing transferability to see if these behavioral patterns hold up elsewhere twenty-one.

Nadia: So, we’ve validated GeNIS as a solid starting point for AI NIDS, but it's time to start looking at how those models handle the messy reality of new data twenty-one.

Elias: Right; the path forward is definitely moving toward more diverse training sets and testing those detection capabilities in hostile environments twenty-one.

GECAD, ISEP, Polytechnic of Porto

cs.CR, cs.AI

Submitted: 2025-11-11

Updated: 2025-11-11

Comments: 17 pages, 12 tables, FPS 2025 conference

DOI: 10.1007/978-3-032-20018-1_18

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: The study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.

Key concepts

GeNIS Dataset
This dataset was specifically created for AI NIDS applications, focusing on typical user behavior and network protocols from small to medium organizations. It contains eight attack scenarios and three benign activities, serving as a baseline for testing detection systems.
Feature Selection Methods
Five techniques—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio—were used to filter 125 raw features. This process aimed to identify the most relevant features that best distinguish between network attacks and normal traffic.
Ensemble Models
Models like Random Forest (RF), XGBoost (XGB), and LGBM were trained on the selected features. These models combine multiple decision trees to improve accuracy, achieving slightly better generalization than deep learning models while remaining computationally efficient for both training and running predictions.
SHAP Analysis
This technique was used to explain model predictions by assigning a score to each feature. It revealed that the AI models primarily relied on quantity-based features (like traffic volume) and helped identify which specific behavioral features were most critical for classifying malicious network flows.

Terminology

Summary

The study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. p. 0

Where the paper says it

However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic. This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.

Dataset Context and Feature Engineering

The GeNIS dataset was created specifically for AI-based NIDS applications, focusing on reflecting typical user behavior, services, and network protocols of small to medium sized organizations. It consists of eight attack scenarios and three benign activity scenarios. p. 3

Where the paper says it

However, due to significant flaws discovered in widely used datasets, it is crucial to evaluate these models’ and feature selection methods’ performance on more recent, reliable datasets free from such anomalies. 3 Dataset Analysis The GeNIS dataset consists of eight attack scenarios and three benign activity scenarios.

A total of 125 features were extracted from the raw recordings of the GeNIS dataset using the HERA flow exporter. p. 4
Where the paper says it

Finally, the background activity scenario captures periods of no user interaction and represents idle traffic collected over the weekend. A total of 125 features were extracted from the raw recordings of the GeNIS dataset using the HERA flow exporter.

p. 4
Where the paper says it

Finally, the background activity scenario captures periods of no user interaction and represents idle traffic collected over the weekend. A total of 125 features were extracted from the raw recordings of the GeNIS dataset using the HERA flow exporter.

The features are grouped into general, time-based, quantity-based, hybrid-based, and context-based categories. p. 4
Where the paper says it

Meanwhile, hybrid features combine time and quantity aspects, and context-based features reflect the specific characteristics of the network topology. Among the 17 general features, 6 (FlowID, Rank, Seq, AutoId, TcpOpt, and Cause) are generated by Argus and do not meaningfully contribute to understanding traffic behavior.

Initial pre-processing involved one-hot encoding categorical features such as State, Flags, and Protocol. p. 4
Where the paper says it

Initial pre-processing of the GeNIS dataset was performed to ensure that no empty rows or columns were present, and the pre-processed dataset was provided alongside the original and raw traffic captures. Additionally, categorical features such as State, Flags, and Protocol were one-hot encoded, resulting in 87 total features.

p. 4
Where the paper says it

Initial pre-processing of the GeNIS dataset was performed to ensure that no empty rows or columns were present, and the pre-processed dataset was provided alongside the original and raw traffic captures. Additionally, categorical features such as State, Flags, and Protocol were one-hot encoded, resulting in 87 total features.

Feature Selection Methodology

Five feature selection techniques—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio—were combined to identify the most relevant features of GeNIS and reduce its dimensionality. The results from each method were normalized and aggregated by summing the corresponding scores for each feature. p. 5

Where the paper says it

These methods include Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio. The results from each feature selection method were first normalized and then aggregated by summing the corresponding scores for each feature.

The 16 most significant features with the highest combined scores were selected, accounting for approximately 70% of the total feature importance. p. 5
Where the paper says it

The results from each feature selection method were first normalized and then aggregated by summing the corresponding scores for each feature. Lastly, the 16 most significant features with the highest combined scores were selected.

Model Training and Evaluation

Three decision tree ensembles (Random Forest (RF), Extreme Gradient Boosting (XGB), and Light Gradient Boosting Machine (LGBM)) and two deep neural networks (Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP)) were trained for both binary and multiclass classification tasks. The optimal configuration for each model was determined through a grid search involving an exhaustive evaluation of combinations of hyperparameters using 5-fold cross-validation. p. 6

Where the paper says it

Three ML models based on decision tree ensembles, RF, XGB, and LGBM, and two DL models based on deep neural networks, LSTM, and MLP, were considered. The optimal configuration for each model was determined through a grid search involving an exhaustive evaluation of combinations of hyperparameters using 5-fold cross-validation.

p. 6
Where the paper says it

Three ML models based on decision tree ensembles, RF, XGB, and LGBM, and two DL models based on deep neural networks, LSTM, and MLP, were considered. The optimal configuration for each model was determined through a grid search involving an exhaustive evaluation of combinations of hyperparameters using 5-fold cross-validation.

Performance was assessed using the F1-score for binary classification and the macro-averaged F1-score for multiclass classification. p. 6
Where the paper says it

This process was applied to the full set of features as well as to reduced sets obtained through feature selection for both binary and multiclass classification tasks. Performance was assessed using the F1-score for binary classification and the macro-averaged F1-score for multiclass classification to ensure balanced evaluation across all classes.

Classification Performance Findings

The ML ensembles achieved slightly better generalization while remaining more efficient than DL models. In binary classification, the selected features were roughly balanced between quantity, time, and hybrid-based types. p. 9

Where the paper says it

The selected features are listed in Table 7. Feature selection for binary classification Feature Type Selected Features Quantity-based DstTCPBase SrcTCPBase DstWin SrcWin TotBytes TotPkts Time-based Dur Min Mean RunTime Sum Hybrid-based DstLoad Load Rate SrcLoad SrcRate In the binary classification scenario, models trained using the full feature set demonstrated strong overall performance on all non-time-based metrics, with most models achieving near-perfect scores.

In multiclass classification, quantity features were four times more frequent and time-based features were three times more frequent than hybrid ones. p. 9
Where the paper says it

cation of which features were more relevant and the analysis of the combined impact of the different time, quantity, and hybrid-based behavioral features of the GeNIS dataset. 2 Binary classification results In the binary classification task, the feature selection methods identified two features as the most significant, which together accounted for 56% of the total feature importance.

Overall evaluation metrics indicate that the models effectively predict whether flows represent attacks and distinguish between different attack types.

Model Comparison and Generalization

The ensemble models consistently achieved higher scores and required less training and inference time with both the full feature set and the reduced subset. p. 9

Where the paper says it

Regarding TT, the ensemble models required less than half the TT of the neural networks and predicted the test set approximately ten times faster. When using a reduced subset of features, there was a slight decline in performance across non-time-based metrics.

The results suggest good generalization potential to other datasets because the reduced feature set was composed of behavior-based features rather than topology-specific ones. p. 4
Where the paper says it

Among the 38 quantity-based features, Ssaddr and Sdaddr are specific to HERA, representing the number of connections involving the same service and source/destination address. While these features are potentially useful for identifying attacks, they are highly dependent on the network’s topology and attacks, and should be excluded to ensure model generalization across different environments.

SHAP analysis revealed that the models predominantly relied on quantity-based features, which may indicate an overreliance on traffic volume or packet counts as key indicators of malicious behavior. p. 4
Where the paper says it

Time-based features include the timestamp marking the start of a flow, the duration of inactivity between packets, and inter-arrival times, all of which offer insights into the temporal aspects of network behavior. Quantitybased features include packet counts, packet sizes, and byte volumes, which are measured separately for each flow’s source (the sender) and destination (the responder).

p. 4
Where the paper says it

Time-based features include the timestamp marking the start of a flow, the duration of inactivity between packets, and inter-arrival times, all of which offer insights into the temporal aspects of network behavior. Quantitybased features include packet counts, packet sizes, and byte volumes, which are measured separately for each flow’s source (the sender) and destination (the responder).

The study concludes that models trained on this reduced feature set demonstrated their ability to successfully generalize from statistical flow behaviors and accurately classify unseen data. p. 0
Where the paper says it

1 Introduction The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown data based on previous patterns.

p. 0
Where the paper says it

1 Introduction The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown data based on previous patterns.

Conclusion and Future Work

It is important to continue the research and development efforts to create larger and more diverse network traffic datasets. p. 2

Where the paper says it

This dataset offers a diverse range of benign and malicious traffic types, and its modular design allows organizations to tailor the dataset to their unique network scenarios, either by 1 https://www. ca/cic/research/applications.

A promising approach in the future will be the combination of multiple datasets and the investigation of the transferability of detection capabilities of ML and DL models to previously unseen network traffic. p. 0
Where the paper says it

The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic.

p. 0
Where the paper says it

The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic.

Given the high generalization results, it is crucial to extend this study to assess model robustness in adversarial settings as a step towards real-world deployment.

Acknowledgment

This work was supported by the PC2phish project, which has received funding from FCT with Refª: 2024.07648.IACDC. This work has also received funding from UIDB/00760/2020. p. 1

Where the paper says it

Section 4 describes the experimental setup, presents and discusses the obtained results of binary and multiclass classification, followed by a discussion of the models’ performance and key findings. Finally, Section 5 presents the main conclusions and future work.

References

The references listed include foundational works on Random Forests, XGBoost, LSTM, and various intrusion detection datasets like CICIDS2017. p. 3

Where the paper says it

To evaluate the real-world effectiveness and practicality of GeNIS in organizational networks, it is essential to validate the performance of AI models trained on it. Previous studies have used datasets such as CICIDS2017 to explore various deep learning models, including Deep Neural Networks, LSTM networks, and Convolutional Neural Networks, while reporting a binary classification performance accuracy of over 94% using these models [12].

Specific studies address issues in other datasets such as CICIDS2017 and HIKARI-2021. p. 3
Where the paper says it

To evaluate the real-world effectiveness and practicality of GeNIS in organizational networks, it is essential to validate the performance of AI models trained on it. Previous studies have used datasets such as CICIDS2017 to explore various deep learning models, including Deep Neural Networks, LSTM networks, and Convolutional Neural Networks, while reporting a binary classification performance accuracy of over 94% using these models [12].

p. 3
Where the paper says it

To evaluate the real-world effectiveness and practicality of GeNIS in organizational networks, it is essential to validate the performance of AI models trained on it. Previous studies have used datasets such as CICIDS2017 to explore various deep learning models, including Deep Neural Networks, LSTM networks, and Convolutional Neural Networks, while reporting a binary classification performance accuracy of over 94% using these models [12].

The paper also cites work on feature selection techniques like Information Gain and Chi-Squared Test. p. 1
Where the paper says it

This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and AI-based NIDS in small to medium sized organizations. After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio.

p. 1
Where the paper says it

This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and AI-based NIDS in small to medium sized organizations. After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio.

Foundational concepts are referenced, such as the perceptron model. p. 1
Where the paper says it

In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP). Different versions of each model were trained on the full feature set and a subset of selected features derived from selection techniques.

p. 1
Where the paper says it

In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP). Different versions of each model were trained on the full feature set and a subset of selected features derived from selection techniques.

Specific tools and datasets used in the research are also cited, such as HERA flow exporter. p. 2
Where the paper says it

Recent efforts have focused on standardizing the dataset creation process for NIDS by introducing the all-in-one HERA flow exporter. HERA was developed in response to issues identified with CICFlowMeter, which has been linked to errors in several popular datasets, being able to generate flow records and extract relevant features consistently and reliably [21].

p. 2
Where the paper says it

Recent efforts have focused on standardizing the dataset creation process for NIDS by introducing the all-in-one HERA flow exporter. HERA was developed in response to issues identified with CICFlowMeter, which has been linked to errors in several popular datasets, being able to generate flow records and extract relevant features consistently and reliably [21].

The paper references works on adversarial robustness benchmarks for intrusion detection. p. 0
Where the paper says it

This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more computationally efficient detection.

p. 0
Where the paper says it

This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more computationally efficient detection.

The work is supported by funding bodies like FCT and UIDB. p. 9
Where the paper says it

However, a notable drawback was the significant increase in FPR. Some doubled their FPR, while others, like the MLP, experienced nearly a tenfold increase.

p. 9
Where the paper says it

However, a notable drawback was the significant increase in FPR. Some doubled their FPR, while others, like the MLP, experienced nearly a tenfold increase.

Bibliography

The bibliography includes standard machine learning texts, such as Breiman's Random Forests paper. It also includes papers detailing the implementation of XGBoost and LGBM. p. 5

Where the paper says it

All experiments were conducted on a standard laptop with 16GB of RAM, a 6-core CPU and 8GB GPU. The implementation was developed in Python and the following libraries were used: numpy and pandas for general data manipulation, xgboost for the implementation of XGB, lightgbm for LGBM, tensorflow for building the LSTM and MLP, and scikit-learn for the implementation of RF, the StandardScaler, and feature selection.

p. 5
Where the paper says it

All experiments were conducted on a standard laptop with 16GB of RAM, a 6-core CPU and 8GB GPU. The implementation was developed in Python and the following libraries were used: numpy and pandas for general data manipulation, xgboost for the implementation of XGB, lightgbm for LGBM, tensorflow for building the LSTM and MLP, and scikit-learn for the implementation of RF, the StandardScaler, and feature selection.

References cover deep learning architectures like LSTM and MLP. p. 1
Where the paper says it

The ML models considered for this study were decision tree ensemble models commonly used in NIDS: Random Forest (RF), Extreme Gradient Boosting (XGB), and Light Gradient Boosting Machine (LGBM). In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP).

p. 1
Where the paper says it

The ML models considered for this study were decision tree ensemble models commonly used in NIDS: Random Forest (RF), Extreme Gradient Boosting (XGB), and Light Gradient Boosting Machine (LGBM). In turn, the more complex DL models followed two distinct architectures: Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP).

Other cited works include analyses on feature selection for text classification and other intrusion detection systems. p. 0
Where the paper says it

Keywords: Feature selection · Efficiency · Machine learning · Deep learning · Cybersecurity · GeNIS · NIDS. 1 Introduction The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be increasingly ineffective at identifying emerging or previously unseen threats [19].

p. 0
Where the paper says it

Keywords: Feature selection · Efficiency · Machine learning · Deep learning · Cybersecurity · GeNIS · NIDS. 1 Introduction The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be increasingly ineffective at identifying emerging or previously unseen threats [19].

The references also point to specific studies on dataset inconsistencies and corrections, such as those related to CICIDS2017. p. 2
Where the paper says it

Different tools may extract different values for the same feature from the same raw data, which can introduce errors into the final dataset [22]. A prominent example is the CICIDS2017 dataset [30], which is widely used in benchmarking studies [2, 20].

p. 2
Where the paper says it

Different tools may extract different values for the same feature from the same raw data, which can introduce errors into the final dataset [22]. A prominent example is the CICIDS2017 dataset [30], which is widely used in benchmarking studies [2, 20].

The list is comprehensive, covering both theoretical foundations and practical applications in cybersecurity. p. 0
Where the paper says it

Overall, the obtained results indicate that the GeNIS dataset supports intelligent intrusion detection and cyberattack classification with time-based and quantity-based behavioral features. Keywords: Feature selection · Efficiency · Machine learning · Deep learning · Cybersecurity · GeNIS · NIDS.

p. 0
Where the paper says it

Overall, the obtained results indicate that the GeNIS dataset supports intelligent intrusion detection and cyberattack classification with time-based and quantity-based behavioral features. Keywords: Feature selection · Efficiency · Machine learning · Deep learning · Cybersecurity · GeNIS · NIDS.

Conclusion

The study presented an experimental validation and evaluation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. p. 0

Where the paper says it

However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic. This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.

The dataset was carefully analyzed, and multiple feature selection techniques were combined to identify the most relevant features and prepare them for the training of ML and DL models. p. 0
Where the paper says it

This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more computationally efficient detection.

A subset of 16 features, accounting for approximately 70% of the total feature importance, was selected by combining the outputs of these methods. p. 9
Where the paper says it

2 Binary classification results In the binary classification task, the feature selection methods identified two features as the most significant, which together accounted for 56% of the total feature importance. Based on these results, a final subset of 16 features was selected from the original set, representing 70% of the cumulative importance.

These features were decisive in distinguishing between benign and malicious flows, as well as between different types of malicious activity. p. 4
Where the paper says it

The BinaryLabel distinguishes between benign and malicious traffic; the CategoryLabel provides a general classification, separating traffic into categories such as benign, bruteforce, DoS, and reconnaissance, also known as recon. The SubCategoryLabel offers a more granular view by differentiating between three benign scenarios, identifying the specific protocols involved in bruteforce and recon attacks, and distinguishing the types of DoS attacks carried out by attackers.

The RF, XGB, LGBM, LSTM, and MLP models obtained high results across both binary and multiclass classification tasks. p. 3
Where the paper says it

Similarly, LSTM models have demonstrated strong results on older datasets, such as KDD99, achieving over 95% accuracy and F1-scores for binary classification across different model configurations, both with and without feature selection. For multiclass classification, the performance remained high with F1-scores above 91% [15].

p. 3
Where the paper says it

Similarly, LSTM models have demonstrated strong results on older datasets, such as KDD99, achieving over 95% accuracy and F1-scores for binary classification across different model configurations, both with and without feature selection. For multiclass classification, the performance remained high with F1-scores above 91% [15].

The ensemble models consistently achieved higher scores and required less training and inference time with both the full feature set and the reduced subset. Since the reduced feature set only slightly impacted performance and was composed of behavior-based features rather than topology-specific ones, the results suggest good generalization potential to other datasets. p. 9
Where the paper says it

Feature selection for binary classification Feature Type Selected Features Quantity-based DstTCPBase SrcTCPBase DstWin SrcWin TotBytes TotPkts Time-based Dur Min Mean RunTime Sum Hybrid-based DstLoad Load Rate SrcLoad SrcRate In the binary classification scenario, models trained using the full feature set demonstrated strong overall performance on all non-time-based metrics, with most models achieving near-perfect scores. The tree-based ensemble models slightly outperformed the neural networks in terms of F1S, while the neural networks matched or surpassed them in terms of FPR.

p. 9
Where the paper says it

Feature selection for binary classification Feature Type Selected Features Quantity-based DstTCPBase SrcTCPBase DstWin SrcWin TotBytes TotPkts Time-based Dur Min Mean RunTime Sum Hybrid-based DstLoad Load Rate SrcLoad SrcRate In the binary classification scenario, models trained using the full feature set demonstrated strong overall performance on all non-time-based metrics, with most models achieving near-perfect scores. The tree-based ensemble models slightly outperformed the neural networks in terms of F1S, while the neural networks matched or surpassed them in terms of FPR.

SHAP analysis revealed that the models predominantly relied on quantity-based features, which may indicate an overreliance on traffic volume or packet counts as key indicators of malicious behavior. The study concludes that models trained on this reduced feature set demonstrated their ability to successfully generalize from statistical flow behaviors and accurately classify unseen data. It is important to continue the research and development efforts to create larger and more diverse network traffic datasets. A promising approach in the future will be the combination of multiple datasets and the investigation of the transferability of detection capabilities of ML and DL models to previously unseen network traffic. Given the high generalization results, it is crucial to extend this study to assess model robustness in adversarial settings as a step towards real-world deployment. The references listed include foundational works on Random Forests, XGBoost, LSTM, and various intrusion detection datasets like CICIDS2017. Specific studies address issues in other datasets such as CICIDS2017 and HIKARI-2021. The paper also cites work on feature selection techniques like Information Gain and Chi-Squared Test. Foundational concepts are referenced, such as the perceptron model. Specific tools and datasets used in the research are also cited, such as HERA flow exporter. The paper references works on adversarial robustness benchmarks for intrusion detection. The work is supported by funding bodies like FCT and UIDB. The bibliography includes standard machine learning texts, such as Breiman's Random Forests paper. It also includes papers detailing the implementation of XGBoost and LGBM. References cover deep learning architectures like LSTM and MLP. Other cited works include analyses on feature selection for text classification and other intrusion detection systems. The references also point to specific studies on dataset inconsistencies and corrections, such as those related to CICIDS2017. The list is comprehensive, covering both theoretical foundations and practical applications in cybersecurity.

--- Page 1 ---

Binary and Multiclass Cyberattack Classification on GeNIS Dataset Miguel Silva Daniela Pinto João Vitorino Eva Maia Isabel Praça Ivone Amorim, and Maria João Viamonte GECAD, ISEP, Polytechnic of Porto, rua Dr. António Bernardino de Almeida, 4249-015 Porto, Portugal. p. 0

Where the paper says it

Binary and Multiclass Cyberattack Classification on GeNIS Dataset Miguel Silva ⋆1[0009 −0008 −6630 −9939], Daniela Pinto1[0009 −0000 −3003 −6694], João Vitorino1[0000 −0002 −4968 −3653], Eva Maia1[0000 −0002 −8075 −531 X ] , Isabel Praça1[0000 −0002 −2519 −9859]and Ivone Amorim1[0000 −0001 −6102 −6165], and Maria João Viamonte1[0000 −0002 −6464 −3529] GECAD, ISEP, Polytechnic of Porto, rua Dr. António Bernardino de Almeida, 4249-015 Porto, Portugal Abstract.

Abstract. The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks. p. 0
Where the paper says it

António Bernardino de Almeida, 4249-015 Porto, Portugal Abstract. The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks.

However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic. p. 0
Where the paper says it

The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic.

This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. p. 0
Where the paper says it

However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic. This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.

Introduction

The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown data based on previous patterns. p. 0

Where the paper says it

1 Introduction The growing frequency and sophistication of cyberattacks make traditional Network Intrusion Detection Systems (NIDS), using signature-based detection, to be increasingly ineffective at identifying emerging or previously unseen threats [19]. To address this problem, NIDS are being incorporated with Artificial Intelligence (AI) due to its ability to learn from historical data and classify unknown data based on previous patterns.

Learning (ML) and Deep Learning (DL) models requires access to accurate and representative data reflecting current network services, protocols, and behaviors. p. 1
Where the paper says it

Silva et al. p. 0

Where the paper says it

Binary and Multiclass Cyberattack Classification on GeNIS Dataset Miguel Silva ⋆1[0009 −0008 −6630 −9939], Daniela Pinto1[0009 −0000 −3003 −6694], João Vitorino1[0000 −0002 −4968 −3653], Eva Maia1[0000 −0002 −8075 −531 X ] , Isabel Praça1[0000 −0002 −2519 −9859]and Ivone Amorim1[0000 −0001 −6102 −6165], and Maria João Viamonte1[0000 −0002 −6464 −3529] GECAD, ISEP, Polytechnic of Porto, rua Dr. António Bernardino de Almeida, 4249-015 Porto, Portugal Abstract.

Learning (ML) and Deep Learning (DL) models requires access to accurate and representative data reflecting current network services, protocols, and behaviors.

p. 1
Where the paper says it

Silva et al. Learning (ML) and Deep Learning (DL) models requires access to accurate and representative data reflecting current network services, protocols, and behaviors.

Related Work

Creating such datasets requires several key steps, including collecting, extracting, and analyzing traffic data. p. 1

Where the paper says it

2 Related Work To protect organizational networks from malicious actors and develop robust NIDS that can secure critical assets, numerous studies have produced datasets that are commonly used to train AI models, particularly ML and DL approaches, to identify malicious patterns in network traffic. Creating such datasets requires several key steps, including collecting, extracting, and analyzing traffic data, which can be done using integrated solutions or a combination of specialized tools.

Inconsistencies and incompatibilities among these tools can lead to discrepancies. p. 1
Where the paper says it

Creating such datasets requires several key steps, including collecting, extracting, and analyzing traffic data, which can be done using integrated solutions or a combination of specialized tools. However, inconsistencies and incompatibilities among these tools can lead

A prominent example is the CICIDS2017 dataset [30]. p. 2
Where the paper says it

Different tools may extract different values for the same feature from the same raw data, which can introduce errors into the final dataset [22]. A prominent example is the CICIDS2017 dataset [30], which is widely used in benchmarking studies [2, 20].

Due to its popularity, several researchers have examined this dataset and identified significant inaccuracies, prompting the release of corrected versions. p. 2
Where the paper says it

A prominent example is the CICIDS2017 dataset [30], which is widely used in benchmarking studies [2, 20]. Due to its popularity, several researchers have examined this dataset and identified significant inaccuracies, prompting the release of corrected versions.

A more recent dataset, HIKARI2021 [6], was developed using the same set of features as CICIDS2017. p. 2
Where the paper says it

[16] identified even more issues than previously reported, including incorrectly labeled port scan attacks and duplicated traffic, which make feature extraction and labeling more difficult. A more recent dataset, HIKARI2021 [6] , was developed using the same set of features as CICIDS2017, but it employed Zeek2 as the flow exporter.

Following the creation of HERA and since prior datasets offer so many challenges, the GeNIS dataset was created specifically for AI-based NIDS applications. p. 2
Where the paper says it

HERA was developed in response to issues identified with CICFlowMeter, which has been linked to errors in several popular datasets, being able to generate flow records and extract relevant features consistently and reliably [21]. Following the creation of HERA and since prior datasets offer so many challenges, the GeNIS dataset was created specifically for AI-based NIDS applications.

Dataset Analysis

The GeNIS dataset consists of eight attack scenarios and three benign activity scenarios. A total of 125 features were extracted from the raw recordings of the GeNIS dataset using the HERA flow exporter. Three of these features are label-related and suitable for binary and multiclass classification tasks. p. 4

Where the paper says it

A total of 125 features were extracted from the raw recordings of the GeNIS dataset using the HERA flow exporter. Three of these features are label-related and suitable for binary and multiclass classification tasks.

The remaining features can be grouped into five categories according to their nature. p. 4
Where the paper says it

The SubCategoryLabel offers a more granular view by differentiating between three benign scenarios, identifying the specific protocols involved in bruteforce and recon attacks, and distinguishing the types of DoS attacks carried out by attackers. The remaining features can be grouped into five categories according to their nature.

Initial pre-processing involved one-hot encoding categorical features such as State, Flags, and Protocol.

Experimental Setup

Five feature selection methods were applied to identify and rank the most impactful features. These methods include Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio. p. 5

Where the paper says it

Five feature selection methods were applied to identify and rank the most impactful features, targeting both binary and multiclass classification objectives. These methods include Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio.

Three ML models based on decision tree ensembles (RF, XGB, LGBM) and two DL models (LSTM and MLP) were considered. p. 6
Where the paper says it

A measure based on the calculation of the square root of the ratio between the sum of the squared deviations and the overall dispersion within the dataset [28]. Three ML models based on decision tree ensembles, RF, XGB, and LGBM, and two DL models based on deep neural networks, LSTM, and MLP, were considered.

The optimal configuration for each model was determined through a grid search involving an exhaustive evaluation of combinations of hyperparameters using 5-fold cross-validation.

Results and Discussion

All models reached high accuracy and F1-scores, with the ML ensembles achieving slightly better generalization while remaining more efficient than DL models. In binary classification, the feature selection methods identified two features as the most significant, which together accounted for 56% of the total feature importance. p. 9

Where the paper says it

cation of which features were more relevant and the analysis of the combined impact of the different time, quantity, and hybrid-based behavioral features of the GeNIS dataset. 2 Binary classification results In the binary classification task, the feature selection methods identified two features as the most significant, which together accounted for 56% of the total feature importance.

SHAP analysis revealed that predictions were driven primarily by quantity-related features. p. 9
Where the paper says it

The results obtained are described in Table 8, where the best results between the model using all features and the smaller subset are highlighted. SHAP analysis of binary classification models using a reduced feature subset reveals that predictions were driven primarily by quantity-related features.

In multiclass classification, the performance of each model exhibited a pattern similar to that observed in the binary classification task. p. 3
Where the paper says it

Similarly, LSTM models have demonstrated strong results on older datasets, such as KDD99, achieving over 95% accuracy and F1-scores for binary classification across different model configurations, both with and without feature selection. For multiclass classification, the performance remained high with F1-scores above 91% [15].

The simpler RF model was the top-performing model, achieving the highest F1S and lowest FPR with both the full and smaller feature sets. p. 8
Where the paper says it

001 Batch size 32 Epochs 30 Early stopping patience 3 To evaluate the performance of each model on the testing set, the following metrics were considered: F1-score (F1S), accuracy (ACC), recall (RCL), precision (PRC), false positive rate (FPR), average time per epoch (TE), training time (TT), and inference time (IT). Optimal performance would be represented as a score of 100% on F1S, ACC, RCL, and PRC, a score of 0% on FPR, and the lowest possible values for TT and IT metrics.

A subset of 16 features, accounting for approximately 70% of the total feature importance, was selected by combining the outputs of these methods. The RF, XGB, LGBM, LSTM, and MLP models obtained high results across both binary and multiclass classification tasks. Since the reduced feature set only slightly impacted performance and was composed of behavior-based features rather than topology-specific ones, the results suggest good generalization potential to other datasets. The study presented an experimental validation and evaluation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks.

--- Page 2 ---

M. Silva et al. Learning (ML) and Deep Learning (DL) models requires access to accurate and representative data reflecting current network services, protocols, and behaviors. Since existing datasets lack data diversity and contain network traffic that is becoming outdated, it is increasingly difficult to detect modern threats [8, 10]. p. 1

Where the paper says it

Learning (ML) and Deep Learning (DL) models requires access to accurate and representative data reflecting current network services, protocols, and behaviors. Since existing datasets lack data diversity and contain network traffic that is becoming outdated, it is increasingly difficult to detect modern threats [8, 10].

In addition to becoming outdated, recent studies have also shown that many publicly available datasets contain inconsistencies caused by flaws in their feature extraction processes [14]. p. 1
Where the paper says it

Since existing datasets lack data diversity and contain network traffic that is becoming outdated, it is increasingly difficult to detect modern threats [8, 10]. In addition to becoming outdated, recent studies have also shown that many publicly available datasets contain inconsistencies caused by flaws in their feature extraction processes [14].

Despite significant efforts to correct these datasets and release improved versions, discrepancies still arise. p. 1
Where the paper says it

In addition to becoming outdated, recent studies have also shown that many publicly available datasets contain inconsistencies caused by flaws in their feature extraction processes [14]. Despite significant efforts to correct these datasets and release improved versions, discrepancies still arise.

This study presents the first experimental validation and evaluation of GeNIS [31]. p. 1
Where the paper says it

This is mainly due to variations in flow exporters used to process the same raw network captures that initially introduced these errors [21]. This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and AI-based NIDS in small to medium sized organizations.

After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were employed to better understand the dataset and reduce its dimensionality. p. 1
Where the paper says it

This study presents the first experimental validation and evaluation of GeNIS [31], a recently developed dataset designed for network traffic analysis and AI-based NIDS in small to medium sized organizations. After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio.

Lastly, the reliability of GeNIS for binary and multiclass classification was evaluated with several ML and DL models, considering standard evaluation metrics and their training and inference times. p. 1
Where the paper says it

After a careful analysis of the modular scenarios of GeNIS, five feature selection techniques were employed to better understand the dataset and reduce its dimensionality: Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio. Lastly, the reliability of GeNIS for binary and multiclass classification was evaluated with several ML and DL models, considering standard evaluation metrics and their training and inference times.

To protect organizational networks from malicious actors and develop robust NIDS that can secure critical assets, numerous studies have produced datasets that are commonly used to train AI models, particularly ML and DL approaches. Creating such datasets requires several key steps, including collecting, extracting, and analyzing traffic data. However, inconsistencies and incompatibilities among these tools can lead to discrepancies. p. 1

Where the paper says it

Creating such datasets requires several key steps, including collecting, extracting, and analyzing traffic data, which can be done using integrated solutions or a combination of specialized tools. However, inconsistencies and incompatibilities among these tools can lead

Different tools may extract different values for the same feature from the same raw data, which can introduce errors into the final dataset [22]. p. 2
Where the paper says it

Binary and Multiclass Cyberattack Classification on GeNIS Dataset 3 to discrepancies. Different tools may extract different values for the same feature from the same raw data, which can introduce errors into the final dataset [22].

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved feature selection for AI models can reduce dimensionality by combining five methods—Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio—to identify the most relevant features of GeNIS. This enables a more computationally efficient detection process as stated in the abstract.

  2. Training decision tree ensembles like Random Forest (RF), XGBoost (XGB), and Light Gradient Boosting Machine (LGBM) on selected features allows for high accuracy and F1-scores, with ML ensembles achieving slightly better generalization while remaining more efficient than DL models.

  3. Implementing Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP) deep neural networks allows the AI system to model long-range dependencies in sequential data, even when treating each data point as a one-step sequence. This capability enables the detection of complex nonlinear feature interactions in network traffic.

  4. Applying SHapley Additive exPlanations (SHAP) technique analyzes predictions to provide a score for each feature, which allows researchers to identify which features were more relevant and the analysis of the combined impact of the different time, quantity, and hybrid-based behavioral features.

  5. The system can be optimized for efficiency by reducing training and inference times; for example, ensemble models required less than half the TT of the neural networks in binary classification while achieving faster inference with the smaller feature set.

Abstract

The integration of Artificial Intelligence (AI) in Network Intrusion Detection Systems (NIDS) is a promising approach to tackle the increasing sophistication of cyberattacks. However, since Machine Learning (ML) and Deep Learning (DL) models rely heavily on the quality of their training data, the lack of diverse and up-to-date datasets hinders their generalization capability to detect malicious activity in previously unseen network traffic. This study presents an experimental validation of the reliability of the GeNIS dataset for AI-based NIDS, to serve as a baseline for future benchmarks. Five feature selection methods, Information Gain, Chi-Squared Test, Recursive Feature Elimination, Mean Absolute Deviation, and Dispersion Ratio, were combined to identify the most relevant features of GeNIS and reduce its dimensionality, enabling a more computationally efficient detection. Three decision tree ensembles and two deep neural networks were trained for both binary and multiclass classification tasks. All models reached high accuracy and F1-scores, and the ML ensembles achieved slightly better generalization while remaining more efficient than DL models. Overall, the obtained results indicate that the GeNIS dataset supports intelligent intrusion detection and cyberattack classification with time-based and quantity-based behavioral features.

Related papers