Evaluating False Alarm and Missing Attacks in CAN IDS
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Evaluating False Alarm and Missing Attacks in CAN IDS".
Elias: Here's the gist of the paper. Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern.
Nadia: While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities.
Elias: Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks.
Nadia: Adversarial Attack Generation The study employs protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD.
Title and authors: Nadia: So we're looking at this paper called "Evaluating False Alarm and Missing Attacks in CAN IDS" again. The authors are Nirab Hossain and Pablo Moriano, and they're focusing on making sure these detection systems are actually reliable when bad actors try to trick them.
Elias: Yeah, the core idea here is that while these systems work fine in normal traffic, we don't really know how fragile they are when someone crafts specific inputs to cause problems at the moment of inspection.
Priya: They use a systematic evaluation approach comparing four shallow learning models against one deep neural network detector using real vehicle data from the ROAD dataset. I'm wondering if that realistic traffic helps make their findings more useful than just running tests on synthetic data.
Elias: They also found that while the deep neural network performs best on benign traffic, it shows scenario-dependent catastrophic missed-attack behavior in adversarial settings, reaching a missed-attack ASR of one point zero under FGSM, BIM, and PGD attacks on MECTA > <ref:2602.02781#pg3>
Nadia: That's a sobering result for the deep learning approach. It means even the best model isn't immune when you use these gradient-based attacks.
Priya: So what they found is that adversarial manipulation can trigger false alarms and evade detection at the same time, which really highlights why we need to test for this coupled behavior in safety-critical systems.
Elias: And they also noted some interesting things about how different model types react to these attacks. Gradient-based attacks are highly effective against models directly or indirectly exploitable to gradient-based perturbations, including the deep neural network leading to complete evasion in specific cases >
Nadia: But they also saw that for certain model architectures, like the shallow learning extra trees, there's a different story. The paper suggests that under gradient-based attacks, the shallow model extra trees demonstrate improved robustness to missed attack induction compared to the other models >
Priya: That’s interesting because it implies that sometimes a simpler model might be more stable against these specific types of manipulation than a complex deep network, even if the deep network seems better on benign traffic.
Elias: Yeah, and they also saw that protocol-aware CAN fabrication attacks have a stronger impact on tree-based models, where decision trees like DT or RF or XGB can attain significantly higher ASR >
Nadia: So we've got a picture now of the different failure modes depending on the attack type and the model structure being tested. It’s not just one simple vulnerability.
The paper's summary: Nadia: To summarize what they found in "Evaluating False Alarm and Missing Attacks in CAN IDS", they are systematically evaluating how different machine learning intrusion detection systems react when subjected to protocol-compliant, payload-level perturbations.
Elias: They are specifically looking at how these systems handle both false alarms and missed attacks as coupled failure modes, which is a big deal for reliability assessment under attack >
Priya: They grounded their evaluation in realistic CAN attack scenarios from the ROAD dataset, which gives them a more faithful assessment of adversarial effects under practical traffic and attacker conditions.
Nadia: The paper shows that while all models perform well under benign conditions, the adversarial perturbations reveal substantial vulnerabilities for every architecture tested.
Elias: They used gradient-based attacks like FGSM, BIM, and PGD to generate examples that are constrained by protocol specifications to keep the frames valid while influencing IDS decisions at inference time >
Priya: So what they're showing is that the vulnerability isn't just about one thing; it’s about how these two failure modes interact when an attacker tries to both confuse the system and get it to ignore a bad message.
Nadia: They also found that the deep neural network performs best on benign traffic under adversarial conditions, but this advantage is counteracted because all architectures suffer significant increases in missed attacks >
Elias: That means the advantage of having a deep model for normal traffic doesn't translate into overall better robustness against these specific adversarial manipulations.
The paper's improvements: Nadia: The authors suggest several ways to improve this area, and they focus on building more robust detection systems through training strategies that enhance stability under perturbation.
Elias: They point to adversarial training as a key defense mechanism, which involves training the model on these perturbed examples to make it more stable when facing similar inputs >
Priya: They also discuss input sanitization techniques, which is another layer you can add before the data even gets fed into the IDS, trying to clean up any malicious elements >
Nadia: And they're looking at hybrid IDS designs that combine learned representations with interpretable structures, which could give us a way to get both detection and some kind of explainability >
Elias: That might be how we get the best of both worlds, combining the power of deep learning with something more structured for better understanding what’s happening during an attack >
Priya: It sounds like they're suggesting that relying on just one approach isn't enough; you need layered defenses against these sophisticated attacks.
Nadia: The paper emphasizes the importance of adversarial evaluation itself, because if we don't evaluate for this kind of manipulation, we won't know where our systems are weak >
Conclusion: Elias: So wrapping up the "Evaluating False Alarm and Missing Attacks in CAN IDS", it really shows that adversarial manipulation can simultaneously trigger false alarms and evade detection, which underscores the need for adversarial robustness evaluation in safety-critical automotive IDS.
Nadia: The findings show that we have to look at how different model architectures react to these attacks because gradient-based attacks are effective against models exploitable to gradients, like the deep neural network >
Priya: I just want to mention that they also pointed out the limitations of their study, which is that it doesn't cover every possible attack vector or every single model type.
Elias: That’s fair. The authors are acknowledging that their evaluation isn't exhaustive across all attack types and architectures, so we can't assume these results apply perfectly to every system out there >
Nadia: But the main point is that adversarial manipulation can trigger false alarms and evade detection, so we need to use robustness-oriented training strategies to improve resilience against these kinds of threats >
Priya: So it’s about making sure that the safety-critical systems we deploy are resilient by testing them under these specific conditions, which is what this paper on "Evaluating False Alarm and Missing Attacks in CAN IDS" is all about.
University of Colorado Boulder · Oak Ridge National Laboratory
cs.CR, cs.AI, cs.LG
Submitted: 2026-02-02
Updated: 2026-02-02
Comments: 8 pages, 2 figures, and 8 tables
Journal ref: 2026 IEEE International Conference on Cyber Security and Resilience (CSR), IEEE, 2026
DOI: 10.1109/CSR69374.2026.11759798
Code: https://github.com/NirabHossain/ADV
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 77/100
The gist: Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern.
Key concepts
- CAN Bus
- The Controller Area Network (CAN) is the main communication system connecting various electronic control units (ECUs) inside a modern vehicle. It uses a priority-based system to manage message access, ensuring critical data gets through first.
- Adversarial Perturbations
- These are small, carefully crafted changes made to CAN messages designed to trick the IDS into making mistakes. The study used techniques like FGSM and PGD to create these changes that look valid on the bus but cause false alarms or missed detections in the detection system.
- Attack Success Rate (ASR)
- ASR measures how often an adversarial attack successfully causes a misclassification by the IDS. A higher ASR means the IDS is less robust, as it is more likely to either trigger a false alarm or miss a real attack.
Terminology
Summary
Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. p. 0 edu
Pablo Moriano
Computer Science and Mathematics Division
Oak Ridge National Laboratory
Oak Ridge, TN 37830, USA
moriano@ornl. gov
Abstract—Modern vehicles rely on electronic control units
(ECUs) interconnected through the Controller Area Network
(CAN), making in-vehicle communication a critical security concern. Using protocol-compliant, payload-level perturbations generated
via FGSM, BIM and PGD, we evaluate adversarial effects on
both benign and malicious CAN frames. While all models achieve
strong baseline performance under benign conditions, adversarial
perturbations reveal substantial vulnerabilities. Using protocol-compliant, payload-level perturbations generated
via FGSM, BIM and PGD, we evaluate adversarial effects on
both benign and malicious CAN frames. While all models achieve
strong baseline performance under benign conditions, adversarial
perturbations reveal substantial vulnerabilities. While all models achieve
strong baseline performance under benign conditions, adversarial
perturbations reveal substantial vulnerabilities. Although shallow
and deep models are robust to false-alarm induction, with
the deep neural network (DNN) performing best on benign
traffic, all architectures suffer significant increases in missed
attacks. While all models achieve
strong baseline performance under benign conditions, adversarial
perturbations reveal substantial vulnerabilities. Although shallow
and deep models are robust to false-alarm induction, with
the deep neural network (DNN) performing best on benign
traffic, all architectures suffer significant increases in missed
attacks.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Adversarial Attack Generation
The study employs protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD. These attacks are designed to preserve CAN frame validity while influencing IDS decisions at inference time. p. 1 Building on these observations, this work evaluates the
adversarial robustness of CAN-bus IDS by both separately and
jointly examining false alarms and missed attacks as coupled
adversarial failure modes that determine IDS reliability under
attack [14]. We focus on protocol-compliant, payload-level
manipulations that preserve CAN frame validity while influencing IDS decisions at inference time [16].Where the paper says it
Model Comparison and Robustness
The evaluation compares four shallow learning models with a deep neural network–based detector. The DNN performs best on benign traffic under adversarial conditions, showing superior robustness to false-alarm induction compared to other models. p. 5 004 3156 0. 863
This behavior suggests that the DNN demonstrates superior robustness to adversarial perturbations on benign traffic,
limiting false-alarm induction compared to shallow models.Where the paper says it
Vulnerability Characterization
The results reveal clear differences in robustness between model architectures under adversarial manipulation. Gradient-based attacks are highly effective against models directly or indirectly exploitable to gradient-based perturbations, including DNN leading to complete evasion in specific cases In contrast, protocol-aware CAN fabrication attacks have a stronger impact on tree-based models, where DT, RF and XGB can attain significantly higher ASR
Conclusion on IDS Reliability
The findings underscore the importance of adversarial evaluation for automotive IDS and motivate the use of robustness-oriented training strategies to improve resilience. The study demonstrates that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS. p. 0 Notably, under gradient-based attacks, the shallow model
extra trees (ET) demonstrates improved robustness to missedattack induction compared to the other models. Our results
demonstrate that adversarial manipulation can simultaneously
trigger false alarms and evade detection, underscoring the need
for adversarial robustness evaluation in safety-critical automotive
IDS.Where the paper says it
Dataset and Evaluation Metrics
The analysis is grounded in realistic CAN attack scenarios from the ROAD dataset. The evaluation utilizes two complementary metrics: the attack success rate (ASR) and the Matthews correlation coefficient (MCC). p. 3 [23], the perturbation budget ϵ
enforces a per-byte bound with ϵ ∈ {1, 5} in our experiments,
yielding small, valid-range numerical changes that constrain
perturbation magnitude for both benign and attack data, but do
not guarantee semantic invariance, and thus this setup enables
the evaluation of false alarms and missed detections under
adversarial conditions. Evaluation Metrics
We evaluate adversarial effectiveness and classifier performance using two complementary metrics: the attack success
rate (ASR) [31] and the Matthews correlation coefficient
(MCC) [32]. For example, the Car-Hacking dataset contains
only a small number of attack types and short driving traces. In contrast, the ROAD dataset [19] offers over 3.Where the paper says it
Where the paper says it
Attack Success Rate Quantification
ASR quantifies how often crafted adversarial samples successfully induce misclassification by the IDS. ASR for benign inputs, corresponding to adversarially induced false alarms is expressed as ASR[F P] = FP[adv] / TN[adv]+FP[adv] and for malicious inputs, it is expressed as ASR[F N] = FN[adv] / TP[adv]+FN[adv]> Higher ASR values indicate more effective adversarial attacks and reduced IDS robustness >
Future Research Directions
Future work will explore defense strategies such as adversarial training, input sanitization, and hybrid IDS designs that combine learned representations with interpretable structures. This research highlights the importance of adversarial robustness evaluation in automotive cybersecurity research The paper is 2602. p. 6 Moreover, enforcing strict CAN domain constraints by
limiting perturbations to valid payload bytes demonstrates that
adversarial attacks can remain protocol-compliant and stealthy,
underscoring their practical feasibility. Overall, our results highlight the importance of adversarial
robustness evaluation in automotive cybersecurity research.Where the paper says it
--- Page 1 ---
Evaluating False Alarm and Missing Attacks in CAN IDS
Nirab Hossain
Department of Applied Mathematics
University of Colorado Boulder
Boulder, CO 80309, USA
nirab.hossain@colorado. p. 0 Evaluating False Alarm and Missing Attacks in
CAN IDS
Nirab Hossain
Department of Applied Mathematics
University of Colorado Boulder
Boulder, CO 80309, USA
nirab. hossain@colorado.Where the paper says it
Pablo Moriano
Computer Science and Mathematics Division
Oak Ridge National Laboratory
Oak Ridge, TN 37830, USA
--- Page 2 ---
Abstract—Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. p. 0 gov
Abstract—Modern vehicles rely on electronic control units
(ECUs) interconnected through the Controller Area Network
(CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems
(IDS) are increasingly deployed to protect CAN traffic, yet
their robustness against adversarial manipulation remains largely
unexplored. Machine learning (ML)-based intrusion detection systems
(IDS) are increasingly deployed to protect CAN traffic, yet
their robustness against adversarial manipulation remains largely
unexplored. We present a systematic adversarial evaluation of
CAN IDS using the ROAD dataset, comparing four shallow
learning models with a deep neural network–based detector. We present a systematic adversarial evaluation of
CAN IDS using the ROAD dataset, comparing four shallow
learning models with a deep neural network–based detector. Using protocol-compliant, payload-level perturbations generated
via FGSM, BIM and PGD, we evaluate adversarial effects on
both benign and malicious CAN frames. Although shallow
and deep models are robust to false-alarm induction, with
the deep neural network (DNN) performing best on benign
traffic, all architectures suffer significant increases in missed
attacks. Notably, under gradient-based attacks, the shallow model
extra trees (ET) demonstrates improved robustness to missedattack induction compared to the other models. Notably, under gradient-based attacks, the shallow model
extra trees (ET) demonstrates improved robustness to missedattack induction compared to the other models. Our results
demonstrate that adversarial manipulation can simultaneously
trigger false alarms and evade detection, underscoring the need
for adversarial robustness evaluation in safety-critical automotive
IDS.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 3 ---
CAN is the primary in-vehicle communication bus used to interconnect multiple ECUs [27]. Medium access is governed by priority-based arbitration, where frames with lower numerical IDs are granted access to the bus before higher-priority messages. p. 2 1, consists of an identifier (ID), a data length code (DLC),
up to eight data bytes, and error-checking fields. Medium
access is governed by priority-based arbitration, where frames
with lower numerical IDs are granted access to the bus before
higher-priority messages.Where the paper says it
--- Page 4 ---
In classical CAN, the payload corresponds to the raw data field comprising up to eight bytes (D0–D7), represented as hexadecimal values in CAN logs. Under this representation, each frame contributes its eight data bytes as independent features, enabling both benign and malicious behaviors to be analyzed at the level of raw payload values. p. 2 Those data bytes are treated
directly as numerical features derived from observed CAN
traffic. Under this representation, each frame contributes its
eight data bytes as independent features, enabling both benign
and malicious behaviors to be analyzed at the level of raw
payload values. Under this representation, each frame contributes its
eight data bytes as independent features, enabling both benign
and malicious behaviors to be analyzed at the level of raw
payload values. Within our methodology, adversarial manipulation is restricted to the payload bytes. Within our methodology, adversarial manipulation is restricted to the payload bytes. This design choice reflects an
attack surface in which an adversary alters message contents
without violating protocol structure or disrupting normal bus
arbitration.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 5 ---
We use FGSM, BIM, and PGD attacks to generate adversarial examples using established gradient-based techniques. These attacks are generated using a differentiable DNN surrogate and the resulting adversarial samples are then reused to evaluate the robustness of shallow models in a transfer-based setting under identical protocol-constrained perturbations consistent with Pacheco et al. p. 3 Often regarded
as a strong first-order adversary, PGD approximates a worstcase attack against a given model by generating minimally
perturbed CAN frames that maximize prediction error under
full IDS knowledge. In this work, FGSM, BIM, and PGD attacks are generated
using a differentiable DNN surrogate and the resulting adversarial samples are then reused to evaluate the robustness
of shallow models in a transfer-based setting under identical
protocol-constrained perturbations consistent with Pacheco et
al.Where the paper says it
--- Page 7 ---
veal substantial missed-attack vulnerabilities across the evaluated IDS models. Under FGSM, BIM, and PGD attacks on MECTA, the DNN consistently exhibits an attack success rate (ASR) of 1. p. 5 578
veal substantial missed-attack vulnerabilities across the evaluated IDS models. Under FGSM, BIM, and PGD attacks on
MECTA, the DNN consistently exhibits an attack success
rate (ASR) of 1.Where the paper says it
--- Page 8 ---
suggests a more conservative decision boundary favoring the benign class. Overall, most models remain robust to benign perturbations, as reflected by low false-positive ASR Despite the DNN performing best on benign traffic, it exhibits scenario-dependent catastrophic missed-attack behavior, reaching missed-attack ASR = 1. p. 6 Comparative Robustness Across Models
Overall, most models remain robust to benign perturbations,
as reflected by low false-positive ASR, but are substantially
more vulnerable to adversarial manipulation of malicious
samples. Although the DNN achieves superior benign robustness, it exhibits scenario-dependent catastrophic missed-attack
behavior, reaching missed-attack ASR = 1. In safety-critical systems, excessive false positives can degrade reliability, trigger
unnecessary mitigation actions, and erode trust in the IDS. A complementary failure mode arises when adversarial manipulations cause false negatives, allowing malicious CAN
arXiv:2602.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Robust IDS architecture design can be improved by prioritizing Extra Trees (ET) for enhanced missed-attack robustness, as ET
demonstrates improved robustness to missed attack induction compared to the other models
under gradient-based attacks. -
Unified evaluation framework should be adopted consistently across all ML-based IDS families when assessing false alarms and missed attacks, moving beyond evaluating them in isolation
we adopt a unified evaluation framework applied consistently across representative ML-based IDS families.
-
Adversarial attack generation should leverage protocol constraints by restricting perturbations exclusively to payload bytes, which ensures
gradients are applied only to CAN data bytes
and emulatesa realistic CAN-bus adversary who reuses legitimate IDs and timings while manipulating only the data field.
-
Model selection for safety-critical applications should consider ET as a more favorable robustness trade-off, as it exhibits
minimal false alarms and comparatively lower missed-attack rates
overall, despite the DNN's superior benign performance. -
IDS systems can be trained using robustness-oriented strategies to improve resilience against adversarial manipulation by focusing on training methods that enhance stability under perturbation.
Abstract
Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network-based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missed-attack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.
Sources
- Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN
- Adversarial examples in the physical world
- Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data