Evaluating False Alarm and Missing Attacks in CAN IDS

summary

The gist

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern.

In short

This study tested machine learning intrusion detection systems (IDS) protecting vehicle CAN buses against adversarial attacks using realistic traffic data. Researchers found that while deep neural networks perform best on normal traffic, all models suffer significant increases in missed attacks when faced with adversarial noise. The findings stress the need for robust training methods in safety-critical automotive security.

Key concepts

CAN Bus
The Controller Area Network (CAN) is the main communication system connecting various electronic control units (ECUs) inside a modern vehicle. It uses a priority-based system to manage message access, ensuring critical data gets through first.
Adversarial Perturbations
These are small, carefully crafted changes made to CAN messages designed to trick the IDS into making mistakes. The study used techniques like FGSM and PGD to create these changes that look valid on the bus but cause false alarms or missed detections in the detection system.
Attack Success Rate (ASR)
ASR measures how often an adversarial attack successfully causes a misclassification by the IDS. A higher ASR means the IDS is less robust, as it is more likely to either trigger a false alarm or miss a real attack.

Terminology used across episodes

This episode discusses

The paper

Evaluating False Alarm and Missing Attacks in CAN IDS · Read on arXiv

University of Colorado Boulder · Oak Ridge National Laboratory

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network-based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missed-attack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.

DOI: 10.1109/CSR69374.2026.11759798

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Evaluating False Alarm and Missing Attacks in CAN IDS".

Elias: Here's the gist of the paper. Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern.

Nadia: While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities.

Elias: Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks.

Nadia: Adversarial Attack Generation The study employs protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD.

Title and authors: Nadia: So we're looking at this paper called "Evaluating False Alarm and Missing Attacks in CAN IDS" again. The authors are Nirab Hossain and Pablo Moriano, and they're focusing on making sure these detection systems are actually reliable when bad actors try to trick them.

Elias: Yeah, the core idea here is that while these systems work fine in normal traffic, we don't really know how fragile they are when someone crafts specific inputs to cause problems at the moment of inspection.

Priya: They use a systematic evaluation approach comparing four shallow learning models against one deep neural network detector using real vehicle data from the ROAD dataset. I'm wondering if that realistic traffic helps make their findings more useful than just running tests on synthetic data.

Elias: They also found that while the deep neural network performs best on benign traffic, it shows scenario-dependent catastrophic missed-attack behavior in adversarial settings, reaching a missed-attack ASR of one point zero under FGSM, BIM, and PGD attacks on MECTA > <ref:2602.02781#pg3>

Nadia: That's a sobering result for the deep learning approach. It means even the best model isn't immune when you use these gradient-based attacks.

Priya: So what they found is that adversarial manipulation can trigger false alarms and evade detection at the same time, which really highlights why we need to test for this coupled behavior in safety-critical systems.

Elias: And they also noted some interesting things about how different model types react to these attacks. Gradient-based attacks are highly effective against models directly or indirectly exploitable to gradient-based perturbations, including the deep neural network leading to complete evasion in specific cases >

Nadia: But they also saw that for certain model architectures, like the shallow learning extra trees, there's a different story. The paper suggests that under gradient-based attacks, the shallow model extra trees demonstrate improved robustness to missed attack induction compared to the other models >

Priya: That’s interesting because it implies that sometimes a simpler model might be more stable against these specific types of manipulation than a complex deep network, even if the deep network seems better on benign traffic.

Elias: Yeah, and they also saw that protocol-aware CAN fabrication attacks have a stronger impact on tree-based models, where decision trees like DT or RF or XGB can attain significantly higher ASR >

Nadia: So we've got a picture now of the different failure modes depending on the attack type and the model structure being tested. It’s not just one simple vulnerability.

The paper's summary: Nadia: To summarize what they found in "Evaluating False Alarm and Missing Attacks in CAN IDS", they are systematically evaluating how different machine learning intrusion detection systems react when subjected to protocol-compliant, payload-level perturbations.

Elias: They are specifically looking at how these systems handle both false alarms and missed attacks as coupled failure modes, which is a big deal for reliability assessment under attack >

Priya: They grounded their evaluation in realistic CAN attack scenarios from the ROAD dataset, which gives them a more faithful assessment of adversarial effects under practical traffic and attacker conditions.

Nadia: The paper shows that while all models perform well under benign conditions, the adversarial perturbations reveal substantial vulnerabilities for every architecture tested.

Elias: They used gradient-based attacks like FGSM, BIM, and PGD to generate examples that are constrained by protocol specifications to keep the frames valid while influencing IDS decisions at inference time >

Priya: So what they're showing is that the vulnerability isn't just about one thing; it’s about how these two failure modes interact when an attacker tries to both confuse the system and get it to ignore a bad message.

Nadia: They also found that the deep neural network performs best on benign traffic under adversarial conditions, but this advantage is counteracted because all architectures suffer significant increases in missed attacks >

Elias: That means the advantage of having a deep model for normal traffic doesn't translate into overall better robustness against these specific adversarial manipulations.

The paper's improvements: Nadia: The authors suggest several ways to improve this area, and they focus on building more robust detection systems through training strategies that enhance stability under perturbation.

Elias: They point to adversarial training as a key defense mechanism, which involves training the model on these perturbed examples to make it more stable when facing similar inputs >

Priya: They also discuss input sanitization techniques, which is another layer you can add before the data even gets fed into the IDS, trying to clean up any malicious elements >

Nadia: And they're looking at hybrid IDS designs that combine learned representations with interpretable structures, which could give us a way to get both detection and some kind of explainability >

Elias: That might be how we get the best of both worlds, combining the power of deep learning with something more structured for better understanding what’s happening during an attack >

Priya: It sounds like they're suggesting that relying on just one approach isn't enough; you need layered defenses against these sophisticated attacks.

Nadia: The paper emphasizes the importance of adversarial evaluation itself, because if we don't evaluate for this kind of manipulation, we won't know where our systems are weak >

Conclusion: Elias: So wrapping up the "Evaluating False Alarm and Missing Attacks in CAN IDS", it really shows that adversarial manipulation can simultaneously trigger false alarms and evade detection, which underscores the need for adversarial robustness evaluation in safety-critical automotive IDS.

Nadia: The findings show that we have to look at how different model architectures react to these attacks because gradient-based attacks are effective against models exploitable to gradients, like the deep neural network >

Priya: I just want to mention that they also pointed out the limitations of their study, which is that it doesn't cover every possible attack vector or every single model type.

Elias: That’s fair. The authors are acknowledging that their evaluation isn't exhaustive across all attack types and architectures, so we can't assume these results apply perfectly to every system out there >

Nadia: But the main point is that adversarial manipulation can trigger false alarms and evade detection, so we need to use robustness-oriented training strategies to improve resilience against these kinds of threats >

Priya: So it’s about making sure that the safety-critical systems we deploy are resilient by testing them under these specific conditions, which is what this paper on "Evaluating False Alarm and Missing Attacks in CAN IDS" is all about.

More episodes

← Home