Evaluating False Alarm and Missing Attacks in CAN IDS

arXiv:2602.02781 · cs.CR, cs.AI, cs.LG · Submitted 2026-02-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Evaluating False Alarm and Missing Attacks in CAN IDS".

Elias: Here's the gist of the paper. Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern.

Nadia: While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities.

Elias: Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks.

Nadia: Adversarial Attack Generation The study employs protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD.

Title and authors: Nadia: So we're looking at this paper called "Evaluating False Alarm and Missing Attacks in CAN IDS" again. The authors are Nirab Hossain and Pablo Moriano, and they're focusing on making sure these detection systems are actually reliable when bad actors try to trick them.

Elias: Yeah, the core idea here is that while these systems work fine in normal traffic, we don't really know how fragile they are when someone crafts specific inputs to cause problems at the moment of inspection.

Priya: They use a systematic evaluation approach comparing four shallow learning models against one deep neural network detector using real vehicle data from the ROAD dataset. I'm wondering if that realistic traffic helps make their findings more useful than just running tests on synthetic data.

Elias: They also found that while the deep neural network performs best on benign traffic, it shows scenario-dependent catastrophic missed-attack behavior in adversarial settings, reaching a missed-attack ASR of one point zero under FGSM, BIM, and PGD attacks on MECTA > <ref:2602.02781#pg3>

Nadia: That's a sobering result for the deep learning approach. It means even the best model isn't immune when you use these gradient-based attacks.

Priya: So what they found is that adversarial manipulation can trigger false alarms and evade detection at the same time, which really highlights why we need to test for this coupled behavior in safety-critical systems.

Elias: And they also noted some interesting things about how different model types react to these attacks. Gradient-based attacks are highly effective against models directly or indirectly exploitable to gradient-based perturbations, including the deep neural network leading to complete evasion in specific cases >

Nadia: But they also saw that for certain model architectures, like the shallow learning extra trees, there's a different story. The paper suggests that under gradient-based attacks, the shallow model extra trees demonstrate improved robustness to missed attack induction compared to the other models >

Priya: That’s interesting because it implies that sometimes a simpler model might be more stable against these specific types of manipulation than a complex deep network, even if the deep network seems better on benign traffic.

Elias: Yeah, and they also saw that protocol-aware CAN fabrication attacks have a stronger impact on tree-based models, where decision trees like DT or RF or XGB can attain significantly higher ASR >

Nadia: So we've got a picture now of the different failure modes depending on the attack type and the model structure being tested. It’s not just one simple vulnerability.

The paper's summary: Nadia: To summarize what they found in "Evaluating False Alarm and Missing Attacks in CAN IDS", they are systematically evaluating how different machine learning intrusion detection systems react when subjected to protocol-compliant, payload-level perturbations.

Elias: They are specifically looking at how these systems handle both false alarms and missed attacks as coupled failure modes, which is a big deal for reliability assessment under attack >

Priya: They grounded their evaluation in realistic CAN attack scenarios from the ROAD dataset, which gives them a more faithful assessment of adversarial effects under practical traffic and attacker conditions.

Nadia: The paper shows that while all models perform well under benign conditions, the adversarial perturbations reveal substantial vulnerabilities for every architecture tested.

Elias: They used gradient-based attacks like FGSM, BIM, and PGD to generate examples that are constrained by protocol specifications to keep the frames valid while influencing IDS decisions at inference time >

Priya: So what they're showing is that the vulnerability isn't just about one thing; it’s about how these two failure modes interact when an attacker tries to both confuse the system and get it to ignore a bad message.

Nadia: They also found that the deep neural network performs best on benign traffic under adversarial conditions, but this advantage is counteracted because all architectures suffer significant increases in missed attacks >

Elias: That means the advantage of having a deep model for normal traffic doesn't translate into overall better robustness against these specific adversarial manipulations.

The paper's improvements: Nadia: The authors suggest several ways to improve this area, and they focus on building more robust detection systems through training strategies that enhance stability under perturbation.

Elias: They point to adversarial training as a key defense mechanism, which involves training the model on these perturbed examples to make it more stable when facing similar inputs >

Priya: They also discuss input sanitization techniques, which is another layer you can add before the data even gets fed into the IDS, trying to clean up any malicious elements >

Nadia: And they're looking at hybrid IDS designs that combine learned representations with interpretable structures, which could give us a way to get both detection and some kind of explainability >

Elias: That might be how we get the best of both worlds, combining the power of deep learning with something more structured for better understanding what’s happening during an attack >

Priya: It sounds like they're suggesting that relying on just one approach isn't enough; you need layered defenses against these sophisticated attacks.

Nadia: The paper emphasizes the importance of adversarial evaluation itself, because if we don't evaluate for this kind of manipulation, we won't know where our systems are weak >

Conclusion: Elias: So wrapping up the "Evaluating False Alarm and Missing Attacks in CAN IDS", it really shows that adversarial manipulation can simultaneously trigger false alarms and evade detection, which underscores the need for adversarial robustness evaluation in safety-critical automotive IDS.

Nadia: The findings show that we have to look at how different model architectures react to these attacks because gradient-based attacks are effective against models exploitable to gradients, like the deep neural network >

Priya: I just want to mention that they also pointed out the limitations of their study, which is that it doesn't cover every possible attack vector or every single model type.

Elias: That’s fair. The authors are acknowledging that their evaluation isn't exhaustive across all attack types and architectures, so we can't assume these results apply perfectly to every system out there >

Nadia: But the main point is that adversarial manipulation can trigger false alarms and evade detection, so we need to use robustness-oriented training strategies to improve resilience against these kinds of threats >

Priya: So it’s about making sure that the safety-critical systems we deploy are resilient by testing them under these specific conditions, which is what this paper on "Evaluating False Alarm and Missing Attacks in CAN IDS" is all about.

University of Colorado Boulder · Oak Ridge National Laboratory

cs.CR, cs.AI, cs.LG

Submitted: 2026-02-02

Updated: 2026-02-02

Comments: 8 pages, 2 figures, and 8 tables

Journal ref: 2026 IEEE International Conference on Cyber Security and Resilience (CSR), IEEE, 2026

DOI: 10.1109/CSR69374.2026.11759798

Code: https://github.com/NirabHossain/ADV

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern.

Key concepts

CAN Bus
The Controller Area Network (CAN) is the main communication system connecting various electronic control units (ECUs) inside a modern vehicle. It uses a priority-based system to manage message access, ensuring critical data gets through first.
Adversarial Perturbations
These are small, carefully crafted changes made to CAN messages designed to trick the IDS into making mistakes. The study used techniques like FGSM and PGD to create these changes that look valid on the bus but cause false alarms or missed detections in the detection system.
Attack Success Rate (ASR)
ASR measures how often an adversarial attack successfully causes a misclassification by the IDS. A higher ASR means the IDS is less robust, as it is more likely to either trigger a false alarm or miss a real attack.

Terminology

Summary

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. p. 0

Where the paper says it

edu Pablo Moriano Computer Science and Mathematics Division Oak Ridge National Laboratory Oak Ridge, TN 37830, USA moriano@ornl. gov Abstract—Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern.

While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. p. 0
Where the paper says it

Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities.

p. 0
Where the paper says it

Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities.

Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. p. 0
Where the paper says it

While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks.

p. 0
Where the paper says it

While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks.

Adversarial Attack Generation

The study employs protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD. These attacks are designed to preserve CAN frame validity while influencing IDS decisions at inference time. p. 1

Where the paper says it

Building on these observations, this work evaluates the adversarial robustness of CAN-bus IDS by both separately and jointly examining false alarms and missed attacks as coupled adversarial failure modes that determine IDS reliability under attack [14]. We focus on protocol-compliant, payload-level manipulations that preserve CAN frame validity while influencing IDS decisions at inference time [16].

The perturbation budget epsilon is enforced with epsilon ∈ 1, 5 in our experiments, yielding small, valid-range numerical changes that constrain perturbation magnitude for both benign and attack data

Model Comparison and Robustness

The evaluation compares four shallow learning models with a deep neural network–based detector. The DNN performs best on benign traffic under adversarial conditions, showing superior robustness to false-alarm induction compared to other models. p. 5

Where the paper says it

004 3156 0. 863 This behavior suggests that the DNN demonstrates superior robustness to adversarial perturbations on benign traffic, limiting false-alarm induction compared to shallow models.

Conversely, ensemble models like ET demonstrate comparatively higher robustness across most attack types, achieving notably lower ASR values

Vulnerability Characterization

The results reveal clear differences in robustness between model architectures under adversarial manipulation. Gradient-based attacks are highly effective against models directly or indirectly exploitable to gradient-based perturbations, including DNN leading to complete evasion in specific cases In contrast, protocol-aware CAN fabrication attacks have a stronger impact on tree-based models, where DT, RF and XGB can attain significantly higher ASR

Conclusion on IDS Reliability

The findings underscore the importance of adversarial evaluation for automotive IDS and motivate the use of robustness-oriented training strategies to improve resilience. The study demonstrates that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS. p. 0

Where the paper says it

Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missedattack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.

The CSA attack does not produce any false negatives during adversarial evaluation due to protocol-level constraints

Dataset and Evaluation Metrics

The analysis is grounded in realistic CAN attack scenarios from the ROAD dataset. The evaluation utilizes two complementary metrics: the attack success rate (ASR) and the Matthews correlation coefficient (MCC). p. 3

Where the paper says it

[23], the perturbation budget ϵ enforces a per-byte bound with ϵ ∈ {1, 5} in our experiments, yielding small, valid-range numerical changes that constrain perturbation magnitude for both benign and attack data, but do not guarantee semantic invariance, and thus this setup enables the evaluation of false alarms and missed detections under adversarial conditions. Evaluation Metrics We evaluate adversarial effectiveness and classifier performance using two complementary metrics: the attack success rate (ASR) [31] and the Matthews correlation coefficient (MCC) [32].

The ROAD dataset offers over 3. p. 1
Where the paper says it

For example, the Car-Hacking dataset contains only a small number of attack types and short driving traces. In contrast, the ROAD dataset [19] offers over 3.

5 hours of real vehicle CAN traffic, comprising 12 benign captures and 33 attack captures

Attack Success Rate Quantification

ASR quantifies how often crafted adversarial samples successfully induce misclassification by the IDS. ASR for benign inputs, corresponding to adversarially induced false alarms is expressed as ASR[F P] = FP[adv] / TN[adv]+FP[adv] and for malicious inputs, it is expressed as ASR[F N] = FN[adv] / TP[adv]+FN[adv]> Higher ASR values indicate more effective adversarial attacks and reduced IDS robustness >

Future Research Directions

Future work will explore defense strategies such as adversarial training, input sanitization, and hybrid IDS designs that combine learned representations with interpretable structures. This research highlights the importance of adversarial robustness evaluation in automotive cybersecurity research The paper is 2602. p. 6

Where the paper says it

Moreover, enforcing strict CAN domain constraints by limiting perturbations to valid payload bytes demonstrates that adversarial attacks can remain protocol-compliant and stealthy, underscoring their practical feasibility. Overall, our results highlight the importance of adversarial robustness evaluation in automotive cybersecurity research.

02781

--- Page 1 ---

Evaluating False Alarm and Missing Attacks in CAN IDS

Nirab Hossain

Department of Applied Mathematics

University of Colorado Boulder

Boulder, CO 80309, USA

nirab.hossain@colorado. p. 0

Where the paper says it

Evaluating False Alarm and Missing Attacks in CAN IDS Nirab Hossain Department of Applied Mathematics University of Colorado Boulder Boulder, CO 80309, USA nirab. hossain@colorado.

edu

Pablo Moriano

Computer Science and Mathematics Division

Oak Ridge National Laboratory

Oak Ridge, TN 37830, USA

--- Page 2 ---

Abstract—Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. p. 0

Where the paper says it

gov Abstract—Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored.

We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network–based detector. p. 0
Where the paper says it

Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network–based detector.

Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. p. 0
Where the paper says it

We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network–based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames.

While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missedattack induction compared to the other models. p. 0
Where the paper says it

Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missedattack induction compared to the other models.

Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS. p. 0
Where the paper says it

Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missedattack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.

--- Page 3 ---

CAN is the primary in-vehicle communication bus used to interconnect multiple ECUs [27]. Medium access is governed by priority-based arbitration, where frames with lower numerical IDs are granted access to the bus before higher-priority messages. p. 2

Where the paper says it

1, consists of an identifier (ID), a data length code (DLC), up to eight data bytes, and error-checking fields. Medium access is governed by priority-based arbitration, where frames with lower numerical IDs are granted access to the bus before higher-priority messages.

--- Page 4 ---

In classical CAN, the payload corresponds to the raw data field comprising up to eight bytes (D0–D7), represented as hexadecimal values in CAN logs. Under this representation, each frame contributes its eight data bytes as independent features, enabling both benign and malicious behaviors to be analyzed at the level of raw payload values. p. 2

Where the paper says it

Those data bytes are treated directly as numerical features derived from observed CAN traffic. Under this representation, each frame contributes its eight data bytes as independent features, enabling both benign and malicious behaviors to be analyzed at the level of raw payload values.

Within our methodology, adversarial manipulation is restricted to the payload bytes. p. 2
Where the paper says it

Under this representation, each frame contributes its eight data bytes as independent features, enabling both benign and malicious behaviors to be analyzed at the level of raw payload values. Within our methodology, adversarial manipulation is restricted to the payload bytes.

This design choice reflects an attack surface in which an adversary alters message contents without violating protocol structure or disrupting normal bus arbitration. p. 2
Where the paper says it

Within our methodology, adversarial manipulation is restricted to the payload bytes. This design choice reflects an attack surface in which an adversary alters message contents without violating protocol structure or disrupting normal bus arbitration.

--- Page 5 ---

We use FGSM, BIM, and PGD attacks to generate adversarial examples using established gradient-based techniques. These attacks are generated using a differentiable DNN surrogate and the resulting adversarial samples are then reused to evaluate the robustness of shallow models in a transfer-based setting under identical protocol-constrained perturbations consistent with Pacheco et al. p. 3

Where the paper says it

Often regarded as a strong first-order adversary, PGD approximates a worstcase attack against a given model by generating minimally perturbed CAN frames that maximize prediction error under full IDS knowledge. In this work, FGSM, BIM, and PGD attacks are generated using a differentiable DNN surrogate and the resulting adversarial samples are then reused to evaluate the robustness of shallow models in a transfer-based setting under identical protocol-constrained perturbations consistent with Pacheco et al.

[25].

--- Page 7 ---

veal substantial missed-attack vulnerabilities across the evaluated IDS models. Under FGSM, BIM, and PGD attacks on MECTA, the DNN consistently exhibits an attack success rate (ASR) of 1. p. 5

Where the paper says it

578 veal substantial missed-attack vulnerabilities across the evaluated IDS models. Under FGSM, BIM, and PGD attacks on MECTA, the DNN consistently exhibits an attack success rate (ASR) of 1.

0

--- Page 8 ---

suggests a more conservative decision boundary favoring the benign class. Overall, most models remain robust to benign perturbations, as reflected by low false-positive ASR Despite the DNN performing best on benign traffic, it exhibits scenario-dependent catastrophic missed-attack behavior, reaching missed-attack ASR = 1. p. 6

Where the paper says it

Comparative Robustness Across Models Overall, most models remain robust to benign perturbations, as reflected by low false-positive ASR, but are substantially more vulnerable to adversarial manipulation of malicious samples. Although the DNN achieves superior benign robustness, it exhibits scenario-dependent catastrophic missed-attack behavior, reaching missed-attack ASR = 1.

0 in adversarial settings The paper is 2602. p. 0
Where the paper says it

In safety-critical systems, excessive false positives can degrade reliability, trigger unnecessary mitigation actions, and erode trust in the IDS. A complementary failure mode arises when adversarial manipulations cause false negatives, allowing malicious CAN arXiv:2602.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Robust IDS architecture design can be improved by prioritizing Extra Trees (ET) for enhanced missed-attack robustness, as ET demonstrates improved robustness to missed attack induction compared to the other models under gradient-based attacks.

  2. Unified evaluation framework should be adopted consistently across all ML-based IDS families when assessing false alarms and missed attacks, moving beyond evaluating them in isolation we adopt a unified evaluation framework applied consistently across representative ML-based IDS families.

  3. Adversarial attack generation should leverage protocol constraints by restricting perturbations exclusively to payload bytes, which ensures gradients are applied only to CAN data bytes and emulates a realistic CAN-bus adversary who reuses legitimate IDs and timings while manipulating only the data field.

  4. Model selection for safety-critical applications should consider ET as a more favorable robustness trade-off, as it exhibits minimal false alarms and comparatively lower missed-attack rates overall, despite the DNN's superior benign performance.

  5. IDS systems can be trained using robustness-oriented strategies to improve resilience against adversarial manipulation by focusing on training methods that enhance stability under perturbation.

Abstract

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network-based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missed-attack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.

Sources

Related papers