Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody
summary
The gist
Deploying an explainable intrusion detection model from a validated laboratory setting into an operational forensic workflow requires guarantees that laboratory benchmarks do not test…
In short
MODEXRE is a forensic pipeline that deploys an explainable intrusion detection model into real-world investigations. It uses a dual-mode architecture separating training from inference, ensuring that only certified models are used on evidence under strict chain-of-custody logging. This system guarantees model identity and provides verifiable explanations for every prediction, crucial for legal admissibility.
Key concepts
- Dual-Mode Architecture
- This design separates the model's lifecycle into two distinct modes: Laboratory mode for training and tuning without custody tracking, and Formal mode for inference on real evidence. This separation ensures that only models trained under controlled conditions are used in official investigations where a verifiable chain of custody is mandatory.
- OCSF Normalization
- This process standardizes data from different sources, such as Suricata alerts and CEF firewall logs, into a common format. This resolves the gap between academic models and real evidence by ensuring that heterogeneous data can be correctly mapped to the model's features, improving detection accuracy.
- Causal Temporal Aggregation
- This technique analyzes events over time by looking at how they flow into each other, rather than just analyzing individual events in isolation. This method makes complex behavioral patterns visible to the classifier, significantly improving its ability to detect sequential attacks like port scanning.
- SHAP Explainability
- SHAP values are used to show exactly which parts of the input evidence contributed most strongly to a model's final prediction. This provides per-prediction explainability, allowing forensic experts to verify that the AI's output is traceable and consistent with known domain knowledge.
Terminology used across episodes
This episode discusses
- Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody · Paper Radio
- Forensic-Oriented Intrusion Detection Using Synthetic Network Traffic Data and Explainable Artificial Intelligence
The paper
Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody · Read on arXiv
José Luis Vela, Carmen Pellicer
CESTE, University Center, Zaragoza, Spain
Deploying an explainable intrusion detection model from a validated laboratory setting into a forensic workflow requires guarantees that benchmarks do not test: that the certified model is the exact artifact applied to a case, that heterogeneous evidence can be classified reliably, and that every step is verifiable. We present MODEXRE, a forensic pipeline that extends a previously validated synthetic-data-trained XGBoost classifier with SHAP explainability (arXiv:2607.00763) into a dual-mode architecture: a Laboratory mode, where models are trained and certified on public and synthetic data, and a Formal mode, where a frozen, hash-verified model performs inference on case evidence under a SHA-256 hash-chained chain of custody. Suricata alerts, CEF firewall logs and packet captures are normalized through OCSF. Four design decisions, each triggered by an observed failure, shaped the architecture: a timestamp leak that inflated Kitsune performance (F1-macro 0.984, 0.662 after correction); a feature-coverage gap that made the 78-feature reference model misclassify a real SSH alert; causal temporal flow aggregation, which raised Kitsune multiclass F1-macro from 0.394 to 0.932; and a certification defect, exposed only by a real packet capture, whereby the pipeline had always trained a binary classifier. After correction, the same capture went from 100% of events flagged as malicious to 87% classified as Normal, with the scanning host identified as Reconnaissance. The default four-class certified model reaches F1-macro = 0.9538 on CICIDS2017-derived evidence; the recertified 18-category model reaches 0.6214, reflecting the real multiclass task. Chain-of-custody verification, model integrity hashing and per-prediction SHAP explanations are reported as measured outcomes.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody".
Elias: Here's the gist of the paper. Deploying an explainable intrusion detection model from a validated laboratory setting into an operational forensic workflow requires guarantees that laboratory benchmarks do not test, namely that the certified model is the exact artifact applied to a case, that heterogeneous evidence can be classified reliably, and that every step is verifiable<ref:2611.06173#pg3>.
Nadia: Pipeline Architecture The paper presents MODEXRE, a forensic pipeline extending a previously validated synthetic-data-trained XGBoost classifier with SHAP explainability into a dual-mode architecture governed by a verifiable chain of custody and a frozen, hash-checked model<ref:2611.06173#pg3>.
Elias: This separation establishes Laboratory mode for training and Formal mode for inference exclusively on real evidence under full chain-of-custody logging<ref:2611.06173#pg4>.
Nadia: The backend is modularly organized into custody, ingestion, ocsf, features, models, pipeline, and report modules<ref:2611.06173#pg10>.
Title and authors: Nadia: So we’re looking at a paper called "Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody". It sounds like they're tackling the big problem of moving AI models from a lab to real investigations.
Elias: Exactly. The core idea is that you need guarantees when you take a model that worked in a controlled lab and apply it to actual evidence, because the test data in the lab just isn't the same as what you find at a crime scene.
Priya: That sounds like it touches on privacy and measurement issues, right? Because if we can't trust what the model sees from real evidence, then any conclusion we draw from that model is questionable.
Nadia: Right. And this paper describes a way to build this system, MODEXRE. They’re using something called a dual-mode architecture, which means it has a training mode and an operation mode for investigations.
Elias: That separation is key because it lets them train models on public or synthetic data in the lab, but when you get real evidence, it switches over to this formal mode where everything is locked down and verified.
Priya: So, if we think about what that means for us, it suggests a way to handle the messy reality of forensic data sources—like logs from different security tools—without losing track of what’s actually happening.
Nadia: Right. And they use something called OCSF normalization to get all that different evidence together into one common format so the model can understand it, even when the features don't match up perfectly with what they were originally trained on.
Elias: That addresses a known issue where academic models and real-world data just don't speak the same language, which is a huge hurdle for deploying this kind of AI in practice.
Priya: But does normalizing the features always mean you get a reliable result? I worry that if you smooth out all the differences too much, you might accidentally hide some important signal.
Nadia: That’s what they had to deal with because they found that simple normalization wasn't enough, so they actually had to make some design changes based on failures during development.
Elias: They mentioned four specific corrections that came out of those failures, including fixing a leak from a timestamp artifact that was messing up the model’s performance.
Title and authors: Priya: So they weren't just happy with their initial setup; they had to actively hunt for bugs in how the system handled real data, which shows how much uncertainty is involved in this work.
Nadia: And one of those fixes was about feature coverage—they realized that a typical alert from a tool like Suricata only had four features out of the seventy-eight features the original reference model expected.
Elias: That’s why they implemented a change to use just six features for real evidence, which fixed the problem where, for example, an SSH brute-force alert got misclassified as normal traffic.
Priya: So it sounds like they had to go back and re-evaluate the very features that define an attack based on what they saw in actual data, not just what was in the initial academic paper.
Nadia: Right. And one of the biggest wins, according to their testing, was causal temporal aggregation. This is how they look at time sequences to get a much better score.
Elias: That change really paid off; it raised that multiclass F1 macro from zero point three nine four up to zero point nine three two on their test set <ref:2611.06173#pg1,multiclass F1 macro from 0>.
Priya: That jump is significant, but I wonder if that higher number is truly reflective of a reliable detection, or if it's just a measure of how much the model learned the specific sequence pattern they were targeting.
Nadia: It’s about making crossflow behavioral patterns visible to the classifier so it can spot things like port scanning more effectively, which is what that aggregation helps with.
Elias: And on top of that, they focused heavily on explainability using SHAP values. This isn't just for us to see *what* the model decided; it’s about proving *why* it made that decision.
Priya: Explainability is crucial for admissibility, I guess. If a forensic expert has to testify in court, they can’t just say the AI flagged something; they need to show them exactly which pieces of evidence led to that flag.
Nadia: Exactly. The SHAP attributions are reported as being consistent with what we already know about network attacks, which builds trust in the system's output.
Elias: But this leads into the bigger picture: how does this architecture actually help someone who is just trying to use a security tool on their daily job?
Title and authors: Priya: It allows them to verify that the AI system’s output can be tied unambiguously to a specific, unaltered model artifact that was previously validated, which is a major step toward operationalizing it.
Nadia: So, by tying every inference in formal mode to a frozen and hash-checked model, they are creating these strong identity guarantees for the forensic process.
Elias: That separation between the training lab and the real evidence analysis is what makes this dual-mode architecture so useful for establishing those kinds of guarantees.
Priya: But we have to be careful about what they *don't* guarantee, though. They noted that their certified model hasn't been tested on a full-scale, real dataset from the companion project yet.
Nadia: That’s a fair limitation, because you can’t really prove something is robust if you haven't run it against the entire scope of what it might encounter in the field.
Elias: They also highlighted another practical constraint: evidence-source coverage is limited because the parsers only cover common core fields instead of trying to handle every single vendor-specific dialect.
Priya: So while they solved a lot of feature gaps, they are still bound by what the tools used to extract the data can actually provide, which means some specialized evidence might still be missed.
Nadia: And another thing they pointed out is that their chain of custody implementation is at an intermediate level; it’s a hash-chain without digital signatures for integrity checks.
Elias: That’s a design choice that balances security with the complexity of the forensic workflow, but it means you don't get the highest level of cryptographic proof for every single handover.
Priya: So, to wrap up this paper on "Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody," we see a very rigorous pipeline that addresses the gaps between theoretical model performance and messy real-world evidence.
Nadia: It provides a concrete framework for how security researchers can actually move validated AI models into an operational workflow while keeping the integrity of the evidence traceable.
Elias: It’s about making sure that when you use this system, you can trace every single decision back to a specific, certified model artifact and every piece of data used to make it.
Priya: It shows us that for forensic AI to be useful, we have to focus not just on the detection accuracy number, but on proving exactly how the model arrived at its conclusion using verifiable steps.
The paper's summary: Nadia: So we're looking at how this paper, "Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody," wraps up its whole setup.
Elias: Basically, it’s about taking that AI model we validated in the lab and making it trustworthy enough to actually use on a real case.
Nadia: Right. They built this dual-mode system, so you have a training mode where everything is flexible, and then a formal mode for investigation where the model is frozen and locked down with this chain of custody logging.
Elias: That separation is what they call operationalizing model identity guarantees; it means every time you run the AI on real evidence, you are tied to that specific, hash-verified artifact from the lab.
Nadia: It’s a big deal for admissibility because they aren't just saying "this AI found something"; they are reporting the chain of custody integrity and showing per-prediction explainability as measurable system behaviors.
Elias: Exactly. They use SHAP attributions that line up with what we already know about network attacks, which gives forensic experts a way to verify if the AI’s output makes sense in the real world.
Nadia: And they show some solid numbers too, like this certified model hitting an F1 macro of zero point six two one four on their test set after they fixed that initial bug.
Elias: That jump from what it was before shows how much fixing those coverage gaps and using better temporal aggregation really helps the multiclass performance.
Nadia: But they also laid out some hard limits, which is important for us to know as practitioners. They admit that the certified model hasn't been tested on a full-scale real dataset from their larger project yet.
Elias: And they point out that the evidence source coverage is limited because the tools they use can only read common fields instead of every single vendor-specific detail.
Nadia: Plus, their chain of custody is at an intermediate level; it’s a hash chain without digital signatures for those handovers.
Elias: So, while they solved the core problem of moving validated AI into a formal workflow, they also highlighted where the limitations still lie in real-world deployment.
Nadia: It shows that building the pipeline is one thing, but making it robust across every possible piece of evidence is another whole challenge.
Elias: And that’s where we look next: how does this system actually play against the threats we see in things like jailbreaking and adversarial attacks?
The paper's improvements: Tom: So we're looking at what they suggest we do next to actually make this system better than just being certified in a lab setting.
Nadia: The paper points out that one of the big fixes they made involved tackling those design failures directly by confronting them with real evidence rather than just adjusting the code in isolation.
Elias: Right. They discovered a hidden certification bug, which they fixed by doing regression tests against actual data, and that’s a huge lesson about testing rigorously.
Nadia: And another improvement is around feature selection; they realized that using just six features for real evidence was necessary because the original model was expecting seventy-eight features.
Elias: That addresses the coverage gap by focusing on what actually matters in the data, which prevents things like a brute force alert from being misclassified as normal traffic because it’s missing key indicators.
Nadia: They also mention causal temporal aggregation as a major upgrade; this is how they look at time sequences to see crossflow behavioral patterns, and it raised that multiclass F1 macro from zero point three nine four up to zero point nine three two.
Elias: That jump is significant because it means the AI isn't just looking at individual data points but understanding the sequence of events, like detecting a port scan.
Nadia: And on the explainability side, they emphasize that making those SHAP attributions consistent with domain knowledge is crucial so forensic practitioners can actually trust what’s coming out of the report.
Elias: They want to keep pushing that traceability; tying every decision back to a specific model artifact means we can really audit the entire detection process for legal requirements.
Nadia: So, they are suggesting that future work needs to focus on stress testing this formal mode against even more heterogeneous evidence types than just Suricata alerts and firewall logs.
Elias: And I think we should also look at how they handle the intermediate chain of custody level; maybe moving toward digital signatures would provide a stronger guarantee for the handover process.
Nadia: It’s about moving from a hash chain to something with more cryptographic proof, which is a big step for ensuring that evidence integrity isn't just assumed but mathematically proven.
Elias: Exactly. If we can strengthen those underlying proofs, we get closer to making this AI system truly bulletproof for high-stakes forensic work.
Conclusion: Nadia: So to wrap up, this paper on the "Network Intrusion Forensic Detection System with Explainable Artificial Intelligence and Digital Chain of Custody" shows how you can take a lab-validated model and give it real operational power.
Elias: It really hammers home that for AI to be admissible in forensics, you have to prove exactly which artifact is being used at every single decision point.
Nadia: The implication is that we can start moving away from black-box systems toward something where the forensic trail isn't just a log, but an auditable link back to the exact model version used.
Elias: That level of rigor changes how we think about deployment; it moves it from a simple tool to a verifiable component in a legal process.
Priya: From my side, what this means for the data is that as long as the system accurately reports its chain of custody and explains *why* it flagged something, we can start trusting the output more than just a raw detection number.
Nadia: Right. So yeah, we've seen how they built this dual-mode architecture to separate training from real evidence analysis while keeping everything under strict control.
Elias: It’s a solid piece of engineering that addresses the trust problem in AI deployment by tying model identity directly into the inference process.
Priya: It’s a good foundation, but we still have those limitations they mentioned, like not testing it on the full real dataset yet, so we can't assume perfect performance across every scenario.
Nadia: True. And I think the next big question is whether this level of proof is scalable enough to handle the sheer variety of evidence sources we deal with in a real investigation.
Elias: Yeah, and that leads us nicely into how these kinds of forensic pipelines interact with things like adversarial attacks or jailbreaking tries.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18