A Human-in-the-Middle Attack against Object Detection Systems
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "A Human-in-the-Middle Attack against Object Detection Systems".
Paper summary: Rosa: So, we're looking at this paper, "A Human-in-the-Middle Attack against Object Detection Systems," and the main idea is that deep learning models in robotics are getting super popular because of the good power of embedded systems like CPUs and GPUs. But these models have this vulnerability to adversarial attacks.
Dev: Right, it’s true, they’re susceptible to them, which makes this paper interesting because it proposes a novel hardware attack inspired by Man-in-the-Middle attacks used in cryptography. This attack generates something called Universal Adversarial Perturbations or UAP and injects that perturbation right between the USB camera and the detection system using a hardware attack.
Taro: So, the core claim is that they can combine the flexibility of physical attacks with the efficiency of digital attacks to create this Human-in-the-Middle attack. It’s not just hacking into software anymore; it’s manipulating what hits the sensor physically.
Rosa: Exactly, and what they bring to this is presenting three new evaluation metrics instead of just relying on standard mean average precision, which the paper points out doesn't tell you much because it can't tell the difference between fabrication and vanishing attacks.
Dev: And these new metrics are supposed to give a more nuanced view of how strong an adversarial attack actually is. They look at things like confidence values and probability vectors in a linear way, which is different from what’s been done before.
Taro: I'm curious about the methodology here, because they claim this attack works even when you don't have access to the detection system itself, just by intercepting the data stream.
Rosa: Well, they detail how to generate these image-agnostic UAPs using two methods: first an image-specific PCB attack that iterates over a single picture, and second a method called the Image-agnostic PCB attack which uses a collection of images Xs to generate the UAP.
Dev: And for the generation part, they use optimization based on Projected Gradient Descent with learning rate decay applied in every iteration t. This helps stabilize the perturbation during generation.
Taro: So, if I'm thinking about real-world testing outside a controlled lab setting, how does this attack hold up when the world misbehaves or when the sensor setup isn't perfect?
Rosa: That’s what I want to know—if it works outside the lab for long periods, and how it behaves when things get messy. The paper shows experimental results on the CARLA dataset demonstrating that their image-agnostic PCB attack produces higher variations than another attack called TOG.
Dev: And they mention that when the Universal Adversarial Perturbations become stable, the relative box variation converges to one hundred percent, which suggests a consistent effect across detections.
Taro: It also notes that while this image-agnostic attack performs best on the training map, it still generalizes to other maps, which is good for real deployment scenarios.
Rosa: They found that the image-specific attack achieved twelve point four four frames per second, which is faster than the ten frames per second sampling rate of the KITTI autonomous vehicle, so they got a real-time attack going.
Dev: That speed aspect is important because it shows they managed to make this practical for real-time applications without needing access to the detection system itself.
Taro: So, what's the bigger picture here? This kind of hardware attack makes me think about the security of those sensors themselves, not just the software running on top of them.
Rosa: That’s a big implication; it raises serious concerns for any applications where deep learning models are used in safety-critical systems, like autonomous driving. The research is calling attention to how secure those sensors really need to be.
Dev: To wrap up the main points of "A Human-in-the-Middle Attack against Object Detection Systems," the authors Han Wu, Sareh Rowlands and Johan Wahlstrom present this novel hardware attack that mixes physical attack flexibility with digital efficiency, inspired by Man-in-the-Middle attacks in cryptography.
Rosa: And they also introduce those three new evaluation metrics to give a better handle on how strong these adversarial perturbations actually are. It’s about moving past just looking at model accuracy and focusing on the attack itself.
Dev: The paper suggests that by combining their proposed evaluation metrics with their hardware attack, they significantly increased the strength of the adversarial perturbations they were able to generate.
Taro: Ultimately, this research opens up new possibilities for adversarial attacks against these systems, which is exciting from a research standpoint. It shows there are new avenues to explore beyond just digital manipulation.
Rosa: Exactly, and they also devised and open-sourced a white-box adversarial toolbox to simplify the process of generating these kinds of perturbations for people who want to try it out themselves.
Dev: So, in conclusion, the paper concludes that their PCB attack addresses issues like gradient counteraction by introducing a learning rate decay factor k to stabilize the attack over different iterations.
Taro: That stabilization mechanism is key, I think; it makes the perturbation generation more reliable when you're running these kinds of iterative optimization processes.
Rosa: And they raise serious concerns for applications of deep learning models in safety-critical systems, such as autonomous driving, by highlighting the security of the sensor itself.
Dev: We should keep an eye on this because if we can reliably inject perturbations like this hardware attack, it changes how we approach securing those perception layers in robotics.
Conclusion: Rosa: So, to wrap up this discussion on "A Human-in-the-Middle Attack against Object Detection Systems," we're looking at how they managed to combine physical and digital attack techniques into one new method for fooling object detection models.
Dev: Yeah, it’s about a hardware attack that injects a digital noise right into the data stream between the camera and the computer system.
Rosa: They developed this Universal Adversarial Perturbation or UAP, which they generated using two different methods depending on whether they wanted an attack specific to one picture or one that worked across many pictures.
Dev: And what's really interesting is how they fixed the stability of this perturbation generation by adding a learning rate decay factor during the optimization process. That helps keep things consistent as the attack runs for more iterations.
Rosa: It means they are tackling a problem where the adversarial noise can drift away if you just keep optimizing it without any control.
Dev: The paper also introduced these three new evaluation metrics instead of just using standard mean average precision because mAP doesn't really tell you how strong the attack actually is against different types of failures.
Rosa: So, these metrics look at things like confidence value variation and how many boxes the model detects at each step to give a more detailed picture of the attack’s power.
Dev: It moves away from just looking at the final accuracy number and starts looking at the process itself, which is important for understanding failure modes in real-time systems.
Rosa: So, these findings really raise some serious concerns for any safety-critical stuff like self-driving cars because they show a way to inject trouble right into the sensor input without needing direct access to the main detection system.
Dev: Exactly, and this isn't just theoretical; they showed experimental results on datasets like CARLA where their image-agnostic attack actually produced higher variations than other methods.
Rosa: They even found that when this perturbation stabilizes, it can reach one hundred percent relative box variation across all detections, which suggests a very consistent way to cause trouble.
Dev: It's a significant step because it moves the attack from just being a theoretical concept to something they can test and measure with concrete numbers in real-world scenarios.
Rosa: This opens up new avenues for thinking about how we secure the sensor layer itself, not just the software running on top of it.
Dev: Right, and that leads us into what this means for future work, like extending these attacks to other sensors or different types of tasks beyond just object detection.
Han Wu, Sareh Rowlands, Johan Wahlstrom
University of Exeter
cs.RO, cs.CV
Submitted: 2022-08-15
Updated: 2024-07-11
Comments: Accepted by IEEE Transactions on Artificial Intelligence, 2024
Journal ref: IEEE Transactions on Artificial Intelligence, 2024
Code: https://github.com/wuhanstudio/adversarial-camera
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 79/100
The gist: Object detection systems using deep learning models have become increasingly popular in robotics thanks to the rising power of CPUs and GPUs in embedded systems However…
Key concepts
- Human-in-the-Middle (HitM) Attack
- This is a hardware attack inspired by network security where an adversary intercepts and manipulates image data transmitted between a USB camera and the detection system. It physically injects digital perturbations into the sensor data stream to mislead the object detection model.
- Universal Adversarial Perturbations (UAP)
- These are image-agnostic perturbations generated by iterating over a collection of images using Projected Gradient Descent (PGD) with learning rate decay. UAPs are designed to be effective against object detection models regardless of the specific input image used during generation.
- Mean Confidence Variation
- This is a new evaluation metric that measures the strength of an attack by tracking changes in the confidence value across successive iterations. It assesses how effectively the perturbation manipulates the model's certainty regarding its predictions.
Terminology
Summary
Object detection systems using deep learning models have become increasingly popular in robotics thanks to the rising power of CPUs and GPUs in embedded systems However, these models are susceptible to adversarial attacks This paper introduces a novel hardware attack that combines the flexibility of physical attacks with the efficiency of digital attacks, inspired by Man-in-the-Middle Attacks in network security
The Novel Attack Mechanism
The proposed Human-in-the-Middle
(HitM) hardware attack generates a Universal Adversarial Perturbations (UAP) and injects the perturbation between the USB camera and the detection system via a hardware attack This attack is inspired by Man-in-the-Middle attacks in cryptography, where an adversary intercepts and manipulates image data transmitted between a USB camera and a detection system The key contributions include presenting a novel hardware attack that offers both efficiency and ease of deployment for adversarial attacks, introducing three new evaluation metrics, and devising an open-source white-box adversarial toolbox
Adversarial Perturbation Generation
The paper details the generation of image-agnostic UAPs through two methods:
-
Image-specific PCB Attack: This method iterates over a single image to produce the perturbation. p. 2
It utilizes adversarial loss functions, specifically LP C(x) = Xσ(ci) ∗ σ(pi) and LP CB(x) = P(σ(ci) ∗ σ(pi))[σwi] ∗ σ[hi]2. p. 2Where the paper says it
Generating the perturbation (The PCB Attack) Gradient-based methods use a similar approach to generate both image-specific and image-agnostic perturbations. For image-specific perturbations, given an input image, the method iterates over a single image to produce the perturbation.
Where the paper says it
Different gradient-based methods use different adversarial loss functions Ladv and construct desired adversarial outputs O∗ differently. In our attack, we separate the Probability vector and Confidence value (PC) with Bounding boxes (B) and investigate the two adversarial loss functions LP C (x) = Xσ(ci) ∗ σ(pi) (3) and LP CB(x) = P(σ(ci) ∗ σ(pi) P[σ(wi) ∗ σ(hi)]2 .
-
Image-agnostic PCB Attack (UAP): This extends the method by iterating over a collection of images Xs to generate the UAP. p. 2
The optimization of the perturbation δ is performed using Projected Gradient Descent (PGD) with learning rate decay in every iteration t, as shown in Algorithm 1 and Algorithm 2. p. 2Where the paper says it
(5) The image-specific PCB attack is summarized in Algorithm 1, where proj∞(δ, ϵ) is the projection function min(δ, ϵ) using l∞ norm , and clip(−1, 1) is the unit clip function. 2) Image-agnostic PCB Attack: We can extend the method to an image-agnostic attack by iterating over a collection of images Xs = x1, x2, .
Where the paper says it
The optimization of (2) is performed by first zeroinitializing the perturbation δ, and then using Projected Gradient Descent (PGD) [37] with learning rate decay in every iteration t , so that δt+1 = projp(δt + αsign( ∂Ladv(x ′ t ; O∗ ) ∂x′ t )). (5) The image-specific PCB attack is summarized in Algorithm 1, where proj∞(δ, ϵ) is the projection function min(δ, ϵ) using l∞ norm , and clip(−1, 1) is the unit clip function.
Evaluation Metrics for Attack Strength
The paper critiques the Mean Average Precision (mAP) for evaluating adversarial attacks because it cannot distinguish between fabrication and vanishing attacks, both resulting in mAP ≈ 0 The authors devise three new evaluation metrics to offer a more nuanced approach to evaluating adversarial attacks:
-
Mean Confidence Variation: This reflects the strength of the attack on the confidence value and is expressed as 1/N Σ PN i=1 (ct,i − ct−1,i). p. 4
Where the paper says it
1) Mean Confidence Variation: The average increase or decrease of the confidence value (before the sigmoid activation function) of all the bounding boxes at each iteration step t. This metric reflects the strength of the attack on the confidence value and is expressed as 1 N PN i=1 (ct,i − ct−1,i).
-
Number of Boxes: This metric shows how many objects are detected at each step of the attack NMS(O(x′)). p. 4
Where the paper says it
2) Number of Boxes: The total number of bounding boxes after the NMS. This metric shows how many objects are detected at each step of the attack |NMS(O(x ′ ))|.
-
Relative Box Variation: This measures the percentage of consistent bounding boxes and can be expressed as NMS(O(x′t))+NMS(O(x′t−1))-NMS(O(x′t), O(x′t−1)) / NMS(O(x′t)). p. 4
Where the paper says it
3) Relative Box Variation: After each iteration, the position of false positive bounding boxes fluctuates. This metric measures the percentage of consistent bounding boxes (bounding boxes that have the same position as in the previous step) at the current step and can be expressed as |NMS(O(x ′ t ))|+|NMS(O(x ′ t−1 ))|−|NMS(O(x ′ t ), O(x ′ t−1 ))| |NMS(O(x ′ t ))| .
Performance and Transferability Results
Experimental results on the CARLA dataset demonstrate the effectiveness of the image-agnostic PCB attack. The image-agnostic PCB attack produces higher variations than TOG, indicating a stronger UAP Furthermore, the relative box variation converges to 100% as the UAP becomes stable. p. 6 The experimental results show that our image-agnostic
PCB attack produces higher variations than TOG, which
indicates a stronger UAP. Besides, our method has a
more stable iteration process, while the TOG generates
the UAP in a more stochastic way (similar to Fig. 98, ϵ = 8). The image-agnostic
PCB attack achieves the best performance on the training map
but also generalizes to other maps (see Fig. For the image-specific PCB attack, we measured the performance of the attack on an NVIDIA RTX 2080Ti GPU. The image-specific attack achieved 12. The image-specific attack achieved 12. 44 FPS, faster than the
10 FPS sampling rate of the KITTI autonomous vehicle, thus
achieving a real-time attack.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Implications
The research concludes that the PCB attack addresses issues like gradient counteraction by introducing a learning rate decay factor k to stabilize the attack over different iterations The findings raise serious concerns for applications of deep learning models in safety-critical systems, such as autonomous driving The research intends to call attention to the security of the sensor for deep learning applications. Further research may explore the extension of the attack to other tasks beyond object detection, or to other sensors, such as Lidar. p. 7 Our experiments on the VOC2012
dataset, the CARLA dataset, and the YOLO detection model
demonstrate the high efficiency of our attack. Further research
may explore the extension of the attack to other tasks beyond
object detection, or to other sensors, such as Lidar.Where the paper says it
REFERENCES
arXiv:2208.07174v4 [cs. p. 0 The key contributions of this research are summarized as
follows:
arXiv:2208. 07174v4 [cs.Where the paper says it
--- Page 1 ---
A Human-in-the-Middle Attack against Object Detection Systems Han Wu, Sareh Rowlands and Johan Wahlstrom¨ Abstract—Object detection systems using deep learning models have become increasingly popular in robotics thanks to the rising power of CPUs and GPUs in embedded systems However, these models are susceptible to adversarial attacks While some attacks are limited by strict assumptions on access to the detection system, we propose a novel hardware attack inspired by Manin-the-Middle attacks in cryptography This attack generates a Universal Adversarial Perturbations (UAP) and injects the perturbation between the USB camera and the detection system via a hardware attack Besides, prior research is misled by an evaluation metric that measures the model accuracy rather than the attack performance In combination with our proposed evaluation metrics, we significantly increased the strength of adversarial perturbations These findings raise serious concerns for applications of deep learning models in safety-critical systems, such as autonomous driving Impact Statement—Advancements in deep neural networks have ushered in a new era of robotics, characterized by intelligent robots with a comprehensive understanding of the environment, thanks to deep learning models However, it is no more a secret that deep learning models are vulnerable to adversarial attacks Besides existing digital and physical attacks, we introduce a novel ’Human-in-the-Middle’ hardware attack that injects digital perturbation into the physical sensor Our research opens up new possibilities for adversarial attacks, and we hope to embrace deep learning models securely for robotic applications Index Terms—Adversarial Attacks, Object Detection, Deep Learning
--- Page 2 ---
2 A HUMAN-IN-THE-MIDDLE ATTACK AGAINST OBJECT DETECTION I. INTRODUCTION The development of deep neural networks has enabled the creation of intelligent robots that possess a more comprehensive perception of the environment than traditional robots However, this shift towards intelligent robots has also brought with it an increasing risk of adversarial attacks, especially in safety-critical applications It has been a decade since the existence of adversarial examples was first identified by Biggio et al. p. 0 INTRODUCTION
The development of deep neural networks has enabled the
creation of intelligent robots that possess a more comprehensive perception of the environment than traditional robots. However, this shift towards intelligent robots has also brought
with it an increasing risk of adversarial attacks, especially
in safety-critical applications. It has been a decade since the
existence of adversarial examples was first identified by Biggio
et al. [1], and Szegedy et al. It has been a decade since the
existence of adversarial examples was first identified by Biggio
et al. [1], and Szegedy et al.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 3 ---
(a) Man-in-the-Middle Attack in Network Security (a) Man-in-the-Middle Attack in Network Security (b) Human-in-the - Middle Attack in Deep Learning (b) Human - Middle Attack in Deep Learning (c) Demo of the Hardware Attack in Demo of the Hardware Attack Fig. 1: The Human-in-the-Middle (HitM) hardware attack However, both digital and physical attacks have their limitations Digital perturbation requires access to the detection system, making it difficult to apply in real-world scenarios such as hacking into a self-driving car Physical attacks, on the other hand, are sensitive to position and angle variations For instance, experiments in [4] showed that an autonomous vehicle only misclassified traffic signs placed within 0. p. 0 Digital perturbation requires access to the detection
system, making it difficult to apply in real-world scenarios
such as hacking into a self-driving car. Physical attacks, on
the other hand, are sensitive to position and angle variations.Where the paper says it
--- Page 4 ---
A. Contributions A. p. 0 The trialand-error process of finding a successful attack object can take
a long time and require significant amounts of printing. Contributions
This paper presents a novel hardware attack that combines
the flexibility of physical attacks with the efficiency of digital
attacks, inspired by Man-in-the-Middle Attacks in network security (refer to Fig. The trialand-error process of finding a successful attack object can take
a long time and require significant amounts of printing. Contributions
This paper presents a novel hardware attack that combines
the flexibility of physical attacks with the efficiency of digital
attacks, inspired by Man-in-the-Middle Attacks in network security (refer to Fig. The trialand-error process of finding a successful attack object can take
a long time and require significant amounts of printing. Contributions
This paper presents a novel hardware attack that combines
the flexibility of physical attacks with the efficiency of digital
attacks, inspired by Man-in-the-Middle Attacks in network security (refer to Fig.Where the paper says it
Where the paper says it
Where the paper says it
-
We present a novel hardware attack, called Human-inthe-Middle attack, that offers both efficiency and ease of deployment for adversarial attacks By utilizing learning rate decay during the generation of the perturbation, our attack is capable of generating more bounding boxes than competing attack methods
-
We introduce three new evaluation metrics that offer a more nuanced approach to evaluating adversarial attacks Unlike existing metrics that make a binary decision for each bounding box, our metrics consider the confidence value and probability vector in a linear fashion
-
We devise and open source the white-box adversarial toolbox that simplifies the process of generating adversarial perturbations
--- Page 5 ---
II. PRELIMINARIES A. p. 1 The toolbox focuses on real-time
white-box attacks against object detection models. PRELIMINARIES
A.Where the paper says it
B. Adversarial Attacks The Fast Gradient Sign Method (FGSM) [15] was the first adversarial attack against classification models that uses gradients of deep neural networks to generate image-specific perturbations, which is more efficient than optimization-based methods proposed by Biggio et al. p. 1 In this study, we investigate how these attacks affect
real-time robotic applications and focus on energy-efficient
one-stage models. Adversarial Attacks
The Fast Gradient Sign Method (FGSM) [15] was the
first adversarial attack against classification models that uses
gradients of deep neural networks to generate image-specific
perturbations, which is more efficient than optimization-based
methods proposed by Biggio et al.Where the paper says it
--- Page 6 ---
III. THE HUMAN-IN-THE-MIDDLE ATTACK This section introduces the PCB attack, a novel gradientbased method designed to generate image-agnostic UAPs The name ”PCB” comes from the fact that the output of the object detection model is separated into three components: probability vector (P), confidence value (C), and bounding boxes (B)
Problem Formulation
Given an input image x, the object detection model outputs N candidate bounding boxes O(x) = O(x) An adversarial example x′=x + δ aims to fool the detection model by making it output candidate boxes O(x′) that are different from the candidate boxes O(x) outputted by the model for the original input image x
Generating Perturbation
The PCB attack separates the Probability vector and Confidence value (PC) with Bounding boxes (B) The optimization of (2) is performed by first zeroinitializing the perturbation δ, and then using Projected Gradient Descent (PGD) [37] with learning rate decay in every iteration t, so that δt+1 = projp(δt + αsign(∂Ladv(x′t; O∗)∂x′t))
--- Page 7 ---
IV. EXPERIMENTAL EVALUATION This section aims to provide insight into why the Mean Average Precision (mAP) not suitable for evaluating adversarial attacks The mAP cannot distinguish between fabrication and vanishing attacks (both mAP=0) <ref:The mAP cannot distinguish between fabrication and vanishing attacks (both mAP=0). p. 3 This requires a Linux kernel that supports
the V4L2 driver, the USB gadget framework, and configfs. EXPERIMENTAL EVALUATION
This section aims to provide insight into why the Mean Average Precision (mAP) not suitable for evaluating adversarial
attacks. (a) The mAP cannot distinguish between fabrication and vanishing attacks (both mAP=0). (b) The mAP does not consider confidence values (both mAP=0).Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved adversarial attack capability through Human-in-the-Middle hardware injection: The system can generate Universal Adversarial Perturbations (UAP) by injecting digital perturbations between a USB camera and a detection system via specialized hardware, allowing for attacks that are
robust to position and angle variations because the perturbation is injected by directly modifying pixel values.
-
Enhanced attack generation efficiency with learning rate decay: The system can generate significantly more bounding boxes than competing methods by utilizing
learning rate decay during the generation of the perturbation,
which stabilizes the attack process and leads to a more efficient iterative optimization. -
More nuanced evaluation of adversarial strength using new metrics: The system can evaluate attacks beyond Mean Average Precision (mAP) by employing three new evaluation metrics: Mean Confidence Variation, Number of Boxes, and Relative Box Variation, which
consider the confidence value and probability vector in a linear fashion.
Abstract
Object detection systems using deep learning models have become increasingly popular in robotics thanks to the rising power of CPUs and GPUs in embedded systems. However, these models are susceptible to adversarial attacks. While some attacks are limited by strict assumptions on access to the detection system, we propose a novel hardware attack inspired by Man-in-the-Middle attacks in cryptography. This attack generates a Universal Adversarial Perturbations (UAP) and injects the perturbation between the USB camera and the detection system via a hardware attack. Besides, prior research is misled by an evaluation metric that measures the model accuracy rather than the attack performance. In combination with our proposed evaluation metrics, we significantly increased the strength of adversarial perturbations. These findings raise serious concerns for applications of deep learning models in safety-critical systems, such as autonomous driving.
Sources
- Intriguing properties of neural networks
- On Physical Adversarial Patches for Object Detection
- YOLOv3: An Incremental Improvement
- YOLOv4: Optimal Speed and Accuracy of Object Detection
- Traits & Transferability of Adversarial Examples against Instance Segmentation & Object Detection
- Transferable Universal Adversarial Perturbations Using Generative Models
- A Survey on Physical Adversarial Attack in Computer Vision
- Towards Deep Learning Models Resistant to Adversarial Attacks
- KITTI-CARLA: a KITTI-like dataset generated by CARLA Simulator
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation