Machine Learning Optimization for Enhanced OS Fingerprinting

arXiv:2610.11133 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Machine Learning Optimization for Enhanced OS Fingerprinting".

Jane: The gist: OsirisML proposes a new command-line tool that uses nPrint to preprocess network traffic and XGBoost to apply machine learning for enhanced OS fingerprinting,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we've talked about what the paper is generally about, and now let's look at how they frame it in their title and authors for "Machine Learning Optimization for Enhanced OS Fingerprinting."

Jane: The authors are Spencer Ekeroth, Virginia Tech, Jeremy Neale, Virginia Tech, and Jae Sung Kim. They’re coming from a solid research background.

Lu: I think the title itself sets the stage perfectly because it immediately signals that this isn't just another fingerprinting script; it’s about using machine learning to optimize that process.

Meng: It sounds very technical, but what does "optimization" mean in this context for someone who isn't deep into ML? Does it just mean making the results look better on a chart?

Tom: Not just making the charts look better. It means they are actively trying to improve how well the system identifies operating systems by tweaking how things are done at every stage of the process.

Jane: They want to show that by optimizing these parts, you can attain a model that is more accurate in passive OS fingerprinting than what was achieved by Jordan Holland et al. two <ref:2610.11133#pg2>.

Lu: And they achieve this through their specific pipeline, which involves using nPrint for preprocessing and XGBoost for the actual classification task.

Meng: So it’s a combination of specialized tools working together in a defined flow to tackle this existing problem with better performance metrics.

The paper's summary: Tom: Now let's look at the actual summary of "Machine Learning Optimization for Enhanced OS Fingerprinting" and what it tells us about their approach.

Jane: They are proposing a new command line interface called OsirisML, which is designed to handle the entire workflow from raw packet data to a trained ML model.

Lu: The summary emphasizes that this tool isn't just a black box; it’s meant to be flexible so you can build and build upon pre-existing models.

Meng: So they are focusing on giving researchers a method to generate and retrain these models easily instead of forcing everyone to start from scratch every time.

Tom: Precisely. They want this research to be about providing a method for continuous improvement, letting OS fingerprinting stay current with new OS releases and new datasets.

Jane: They are building on the work of Jordan Holland et al., but they aim to do better by optimizing the nPrint data output and using more modern machine learning algorithms.

Lu: The summary points out that they are focusing specifically on improving how the nPrint data is handled before feeding it into XGBoost.

Meng: That suggests a lot of effort goes into making sure that tabular data format is as useful as possible for the XGBoost model to learn from.

The paper's improvements: Tom: Let's talk about the specific improvements they suggest in "Machine Learning Optimization for Enhanced OS Fingerprinting."

Jane: One big thing they highlight is that this new system allows you to retrain a model on an unseen dataset to generate a brand new model.

Lu: That addresses the dynamic nature of network environments, where new OS versions are continuously released. It means the tool can adapt and evolve.

Meng: So if a company gets a new version of Windows, they could potentially use this framework to retrain their fingerprinting model with that specific traffic data to get better results for that OS.

Tom: That's the practical implication: continuous improvement from open source contributions of datasets and model generation is possible through this framework.

Jane: They also bring in hyperparameter tuning using a script called model/hyperparameter tuning.py, which uses GridSearchCV one <ref:2610.11133#pg1>. This searches for the best parameters for the CSV file.

Lu: That optimization process can find better settings than what might be used by default, leading to higher performance. The paper shows they found parameters that yielded an accuracy of ninety-seven point seven zero percent and an F-one score of ninety-seven point five one percent on a subset of data <ref:2610.11133#pg1,and an F-1 score of>.

Meng: That’s a significant jump from just using the default XGBoost settings, which we saw achieved around seventy-eight percent accuracy on the whole Friday file, for example <ref:2610.11133#pg1>.

Conclusion: Tom: So to wrap up with "Machine Learning Optimization for Enhanced OS Fingerprinting," what are the big implications we should be hearing about this research?

Jane: The main implication is that we have a way to build and improve OS fingerprinting models continuously rather than just relying on static, single models.

Lu: It gives researchers a method to stay up to date with the constantly evolving landscape of operating systems by leveraging new data and model generation from the open community.

Meng: For me, it means practical application is easier because you have a structured pipeline that handles the heavy lifting of turning messy pcap files into usable training sets for AI.

Tom: Exactly. It’s about making the process of building these models more robust and adaptable to new challenges in network traffic analysis.

Jane: And they're showing that by optimizing both the data preprocessing and the ML application, you can get performance metrics much higher than what was previously possible with similar methods.

Lu: The hierarchical model idea they suggest is also important, where you first predict the operating system family and then a separate model predicts the specific version within that family.

Meng: That granularity sounds useful for real-world security applications because it lets you be more precise about what’s happening on the network.

Tom: So we’ve looked at how OsirisML uses nPrint and XGBoost to create a system for continuous, optimized OS fingerprinting. That's what this paper is all about.

Spencer Ekeroth, Jeremy Neale, Jae Sung Kim

Virginia Tech

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 7 pages, 2 figures, 1 table

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 81/100

The gist: The gist: OsirisML proposes a new command-line tool that uses nPrint to preprocess network traffic and XGBoost to apply machine learning for enhanced OS fingerprinting, achieving high accuracy on

Key concepts

Passive Fingerprinting: This technique analyzes network traffic offline without actively probing the target system. The goal is to identify the operating system of a device just by observing its communication patterns on the network, making it stealthy.
nPrint: nPrint is a tool used to capture raw packet data and convert it into a structured, tabular format. This tabular data is essential because machine learning algorithms require organized input to learn patterns related to different operating systems.
XGBoost: XGBoost is a powerful machine learning algorithm highly effective for handling structured, tabular data like the network traffic features. It was chosen for its strong performance and open-source nature in classifying the operating system based on these input attributes.

Terminology

Fingerprinting — 1× in this paper · explained in 2 episodes

Summary

The gist: OsirisML proposes a new command-line tool that uses nPrint to preprocess network traffic and XGBoost to apply machine learning for enhanced OS fingerprinting, achieving high accuracy on various datasets.

Motivation and Problem Statement

Passive fingerprinting analyzes network traffic offline to identify operating systems without detection from the network. There is a compelling need for tools that can adapt and retrain machine learning models for passive OS fingerprinting to remain effective against the ever-evolving landscape of operating systems. The goal of OsirisML is not just to make an all-in-one passive fingerprinting model, but to provide a method to create and build upon pre-existing models. This allows OS Fingerprinting to stay up to date and continuously improve from open-source contributions of datasets and model generation.

Related Work

The research builds upon existing work such as nPrint and nPrintML, which are capable of capturing traffic live or processing a pcap file to convert raw packet data into a tabular format suitable for machine learning algorithms. OsirisML focuses on optimizing the nPrint data output and using modern machine learning algorithms to attain a model that is more accurate in passive OS fingerprinting than what was achieved by Jordan Holland et al. [2].

Production Environment for OsirisML

The production environment for OsirisML is set up on a virtual machine running Ubuntu 22.04. This virtual machine is running on an AMD EPYC 7401 24-Core Processor with 128GB of RAM. Virginia Tech’s rlogin cluster was also utilized to run hyperparameter tuning.py. The results in section 10 were reproduced on a workstation with 32 threads and 123 GB of RAM using XGBoost 3.2.0.

Phase 7 - Data Preprocessing Workflow

OsirisML is split into two main components: preprocessing, and ML application. The goal of the preprocessing is to take pcap data and convert it to tabular (csv) data, making it compatible for machine learning algorithms. This entire process can be executed with make preprocess PCAP=, which runs process pcap.sh and calls the following three scrips in succession. The workflow involves:

tcp dump.sh - uses the source IP labels provided on CIC-IDS2017 host website [3] by default to split up traffic based on operating systems.
nprint.sh - loops through all PCAP files in data/pcap/pcap os split/ and calls nPrint on each file with-4-t parameters.
npt to csv.py - loops through each NPT file in data/npt/ and writes each line into a single CSV file, which it places in data/csv/.

Phase 8 - ML Application with XGBoost

Once the tabular data is preprocessed, it has the relevant TCP/IP binary input attributes and one classification label, which is the corresponding operating system. XGBoost was selected for its effectiveness with tabular data, along with its open-source nature and extensive documentation. The XGBoost scripts splits the CSV file into x train, x test, y train, and y test. The input attributes are the TCP/IP binary attributes, and the classification label is the last column of the CSV. The default eval metric supplied to all three model scripts is mlogloss, which is selected due to its nature of multi-class classification, and its robustness to imbalanced classes.

Hyperparameter Tuning and Optimization

OsirisML comes equipped with a script model/hyperparameter tuning.py that searches for the optimal parameters for the supplied CSV file. To determine the optimal parameters, the script uses GridSearchCV [1]. This script generates an XGBoost model for every combination of hyperparameters supplied as a parameter of a Python Dictionary. Using GridSeachCV with this grid would generate 54 combinations. After retrieving the optimal combination of supplied hyperparameters, they can be supplied to xgboostmodel.py and used to train the entire data set.

Results

The original results were first reproduced using a random 80% training and 20% testing split of the packets (Table 1). Using the condensed version of FridayWorkingHours.pcap from section 9 produced an accuracy of 97.66% and an F-1 score of 97.35%, while nPrintML reported 96% accuracy and 96% F-1 score on a random 70% training and 30% testing split of the same packets. Using XGBoost’s default parameters on the entire Friday file resulted in 78.40% accuracy and 73.67% F-1 score. With max depth = 10, the accuracy and F-1 score improved to 84.69% and 82.61% respectively.

Potential Limitations

Each operating system in CIC-IDS2017 is a single host, or two hosts for the servers, so the model may learn the behavior of a host rather than of an operating system. The dataset is also imbalanced: Windows 10 has 1,595,579 packets on Friday while Kali Linux has 8,991. OsirisML has only been tested on the CIC-IDS2017 dataset which only consists of 13 operating systems. Current compatibility limitations are that OsirisML is currently only available for installation on Debian-based Linux. The paper suggests a hierarchical model, which would first predict the operating system family and then predict the version within that family with a separate model.

References

[2] Jordan Holland et al. “New Directions in Automated Traffic Analysis”. In: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. 2021, pp. 3366–3383.

[8] XGBoost Development Team. XGBoost: Scalable Machine Learning System.

[7] Ubuntu 22.04.4 LTS (Jammy Jellyfish).

[6] The nPrint Project. nPrint.

[3] Iman Sharafaldin, Arash Habibi Lashkari, and Ali A. Ghorbani. “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization”. In: Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP). Canadian Institute for Cybersecurity, University of New Brunswick. Portugal, Jan. 2018.

[4] AutoML Team. AutoML: Automated Machine Learning.

[5] Virginia Tech. Hardware Information for Cluster.

Improvements for AI systems

Continuous Model Adaptation via OsirisML: The system can retrain a model on an unseen dataset to generate a new model, addressing the dynamic nature of network environments, where new OS versions are continuously released. This allows the passive fingerprinting tool to stay up to date and continuously improve from open-source contributions of datasets and model generation.
Enhanced Model Robustness through Hyperparameter Optimization: By utilizing a script that uses GridSearchCV [1] to search for optimal parameters, the system can achieve higher performance, as demonstrated by finding parameters that yielded the highest accuracy and F-1 scores of 97.70% and 97.51% on a subset of data.
Hierarchical OS Classification: The system can implement a hierarchical model, which would first predict the operating system family and then predict the version within that family with a separate model. This improves classification granularity by leveraging features like TTL separates the operating system families, while the TCP options separate the Ubuntu versions and the window size separates the Windows versions.
Improved Data Integrity through Feature Engineering: The preprocessing workflow can be modified to ensure data quality by dropping direct identifiers: OsirisML drops these 177 bit columns along with the source IP address column, preventing models from being extremely overfitted to the dataset due to data leakage when using features like source IP, destination port, and sequence number bits.
Resource Efficiency in Training: The system can optimize computational cost by partitioning data before hyperparameter tuning, as suggested by the recommendation to use a smaller partition for searching parameters because searching for hyperparameters is computationally expensive and costs lots of time and computer resources.

Abstract

Operating System (OS) Fingerprinting is a technique that can be used to identify a network's operating systems by evaluating network traffic in the form of TCP/IP packets. This research will explore the effectiveness of passively identifying operating systems on the CIC-IDS2017 dataset, a collection of over 47 gigabytes of pcap files with their corresponding operating systems. This research also proposes a new command line interface, OsirisML, which uses nPrint to preprocess the data into tabular data and XGBoost to apply ML to the data to generate, retrain, and test ML models. When packets are split randomly between training and testing, OsirisML models reach an accuracy of 97.66% on a down-sampled subset of the Friday capture and 84.69% on the entire capture. On the entire Monday capture, which contains no attacks, OsirisML reaches an accuracy of 73.83% and an F-1 score of 79.38%.