ANTMAN: An Efficient and Interpretable RTL-Level Run-Time Detection Framework for Stealthy Branch Predictor Attacks on BOOM

The paper addresses a critical security gap in RISC-V processors: Runtime detection of microarchitectural side-channel attacks remains significantly underexplored in RISC-V compared with x86 and ARM ISAs, posing a serious threat to critical applications. The authors note that State-of-the-art branch predictor attacks bypass traditional data and instruction caches by directly exploiting the state of internal history tables, making them inherently stealthy.

The paper highlights that Recent research has explored offline detection of microarchitectural attacks on RISC-V; however, efficient runtime detection of microarchitectural attacks on RISC-V hardware remains significantly unaddressed. Existing solutions leveraging Hardware Performance Counters (HPCs) suffer from a restricted set of counter registers and tradeoffs between detection accuracy, detection speed, and sampling granularity, making them impractical for stealthy attacks. Furthermore, sampling HPCs after distinct intervals leaves intermediate relationships between different microarchitectural blocks unobserved.

The authors emphasize that proprietary x86 and ARM ISAs constrain researchers from modifying processor microarchitectural designs, whereas RISC-V's open-source nature enables secure-by-design solutions.

The paper presents four main contributions:

1. First secure-by-design RTL-level runtime detection: To the best of our knowledge, this paper presents the first secure-by-design, non-intrusive, RTL-level runtime detection method for stealthy microarchitectural attacks on the RISC-V BOOM processor.

2. Highly interpretable detection using association rule mining: The proposed detection method is highly interpretable and automatically discovers frequent patterns using association rule mining, identifying the most relevant RTL signals for branch predictor attack detection.

3. RTL-level runtime detection with finest granularity: "An RTL-level runtime detection method that enables the finest possible access to every internal signal of the entire processor pipeline. In addition, the proposed non-intrusive rule-based monitor eliminates the inherent limitations of HPC-based detection, including event multiplexing, limited hardware counter availability, and coarse-grained event visibility, with negligible performance overhead."

4. Comprehensive evaluation with zero false positives: "We evaluate the proposed detection method across a sequence of computationally intensive, memory-intensive, and branch-heavy workloads, achieving zero false positives while demonstrating its adaptability to a previously unseen branch predictor attack variant."

The threat model considers an unprivileged attacker executing malicious code co-located with the victim on the same physical core of an out-of-order (O3) BOOM processor supporting speculative execution. The attacker "can manipulate the shared branch predictor to launch branch predictor attacks and leak secret information through side channels but has no direct access to the internal branch predictor state and relies only on observable microarchitectural side effects. The paper assumes a controlled and deterministic bare-metal execution environment and trusted hardware free from physical tampering and hardware Trojans."

The detection framework targets branch predictor attacks with three phases: "1. Training: The attacker trains the shared branch predictor through repeated branch executions. 2. Victim Execution: The victim executes secret-dependent branches, updating the shared branch predictor state. 3. Probe: The attacker probes the predictor state by measuring timing differences to infer secret information."

The paper notes that different variants of these branch predictor attacks exhibit timing differences of less than 20 CPU cycles, making them inherently stealthy, making them completely indistinguishable from benign workloads using HPC-based detection.

The proposed ANTMAN framework consists of three steps:

The BOOM processor is configured within the Chipyard environment using the same BOOM parameters used in [25] including: "Reorder Buffer: 48 entries, Integer IQ: 16 entries, Memory IQ: 10 entries, FP IQ: 12 entries, Integer PRF: 64 registers, FP PRF: 48 registers, Load Queue: 12 entries, Store Queue: 12 entries, Fetch Target Queue: 32 entries, L1D Cache: 16 KB, 4-way, 64-B line, L1I Cache: 16 KB, 4-way, 64-B line. The configuration is translated into synthesizable SystemVerilog and compiled by Verilator into a cycle-accurate C++ simulation model. Attack and benign workloads are cross-compiled into RISC-V executables and executed on the generated C++ model, with processor execution traces generated during each workload execution are captured as FST trace files in debug mode."

RTL Signal Identification: A set of RTL signals relevant to branch predictor attacks are selected from microarchitectural blocks including: Global History Register (GHR), NLP/BTB (Next Line Predictor/Branch Target Buffer), BPD (Backing Predictor Direction), Branch Resolution/Execute, Frontend Redirect Control, Branch Update Broadcast Network, and Pipeline Flush/Recovery Control. The selected signals include: "ghist s1 branch not taken, ghist s2 branch not taken, ifu redirect ghist saw nt, f3 btb misp0, f3 btb misp1, f3 btb misp2, f3 btb misp3, bpd is mispredict update, cpu brupdate b2 misp, exu brinfo misp, brinfo0 misp, b2 misp, redirect val, redirect flush, ifu brupdate b2 misp, lsu brupdate b2 misp, fp rename brupdate b2 misp, rob brupdate b2 misp, rob flush valid, rob flush frontend, dec flush pipeline, mem issue flush pipeline, int issue flush pipeline, fp pipe flush pipeline, alu queue flush."

Association Rule Mining: The FST traces are converted to VCD and then CSV format. The FP-Growth algorithm is applied to efficiently identify hidden patterns in the execution of branch-predictor attacks. Using predefined minimum support 5% and confidence 90% thresholds, the mining produces a set of 16 interpretable association rules. Three dominant behavioral clusters are identified:

1. Global Branch Misprediction: Rules reveal strong coupling among branch misprediction signals across multiple microarchitectural blocks, where "Multiple units, including the Reorder Buffer (ROB), Instruction Fetch Unit (IFU), Load-Store Unit (LSU), and Floating-Point Rename Unit (FP rename), simultaneously detect misprediction events along with control-flow correction signals (redirect valid and redirect flush), indicating coordinated pipeline-wide misprediction detection and recovery."

2. Redirect (Control-flow correction): Rules show redirect valid and redirect flush are strongly associated with pipeline flush and branch misprediction events across different microarchitectural blocks, enabling coordinated control-flow recovery.

3. Global Pipeline Flush (Recovery): Rules highlight strong coordination among pipeline flush signals across multiple microarchitectural blocks, where "flush events in the decode, issue, execution, memory, and floating-point units are strongly associated with redirect valid and redirect flush indicating coordinated pipeline recovery activity following control-flow redirection."

The extracted rules are first evaluated offline using benign and attack workloads under minimum support and confidence thresholds of 5% and 90%, respectively. After validation, the rules were implemented and synthesized as a non-intrusive hardware monitor that tracks rule activations and stops execution when the activation count exceeds the prescribed support threshold.

The hardware implementation uses "only 14 automatically discovered RTL signals were tapped from different microarchitectural blocks, and the association rules were converted into hardware using AND logic without causing any interference with ongoing processor operations. The rule-monitoring hardware was implemented in a separate 'RuleMonitor.scala' module, which was instantiated in 'Core.scala', and the design was subsequently re-synthesized. Each rule has a dedicated 16-bit counter, and If the trigger count of all association rules exceeds the predefined support threshold, an assertion is raised at run time, and program execution is immediately terminated."

With the simplified NLP configuration, both variants of branch predictor attacks can be detected with no false positives. The results show that for CBPA and IBPA attacks, Rules Triggered: 16/16 (100%) with prediction High probability that the branch predictor is under attack, while all benign workloads (dhrystone, mm, median, qsort, spmv, towers, vvadd) show Rules Triggered: 0/16 (0%) with prediction Very low probability of an attack.

Under the simplified NLP configuration, both CBPA and IBPA attacks are detected after 25% of the attack execution. The detection terminates execution before secret disclosure, as shown in the simulation output where the assertion Branch prediction related attack detected. Simulation terminated. fires after only 4 indices of the 16-index secret are processed.

Under the complex TAGE configuration, the paper notes that the TAGE predictor configuration only reduces the probability of successfully recovering the secret, with approximately 50% success rate for secret recovery under the TAGE configuration for CBPA. However, existing cryptanalytic techniques have demonstrated that recovering 50% of the cryptographic keys through side-channel leakage can enable extraction of the remaining key information. Under TAGE, CBPA is detected at 25% of attack completion with a 50% secret recovery success rate, whereas the unseen IBPA variant is detected at 56.25% of completion.

For all benign workloads under both configurations, no benign workload is terminated during execution, resulting in zero false positives.

The comparison table shows that the proposed framework is unique in being Secure by Design: Yes, using RTL traces as sampling source, supporting Runtime detection, detecting Stealthy Attack: Yes, with Detection Speed: Yes, Accuracy: Yes, Flexibility: Yes, and Interpretability: High. In contrast, all prior approaches (x86-based HPC solutions and RISC-V gem5-based offline solutions) lack secure-by-design properties, cannot detect stealthy attacks, and have low interpretability.

The paper concludes: This work presents the first secure-by-design, interpretable, association rule-based, non-intrusive RTL-level runtime detection method for stealthy branch predictor attacks, validated on BOOM RISC-V. The approach enables efficient runtime detection without the inherent limitations of HPC-based approaches and demonstrates fast detection, termination before secret disclosure, and flexibility in detecting previously unseen variants within the same branch predictor attack family.

For future work, the authors propose: "A windowed approach can further be introduced to track association-rule activations during stochastic workload execution. This approach offers two key advantages: tracking the number of rule activations and passively monitoring execution duration across windows to identify patterns that exceed predefined support thresholds, thereby enabling non-intrusive detection of attacks hidden within benign workloads. Additionally, a detailed power, performance, and area (PPA) analysis can quantify resource utilization and evaluate performance and area overheads prior to fabrication."

Based on this paper, I can improve AI systems in the following specific ways:

• Improvement: Implement RTL-level, non-intrusive hardware monitors in AI accelerator chips (e.g., NPUs, GPUs) that track internal signal patterns associated with side-channel attacks on neural network inference (e.g., timing attacks on model weights or architecture leakage).

• Capability: The improved AI system can detect and halt execution of adversarial workloads attempting to extract model parameters or training data through microarchitectural side channels, with zero false positives on legitimate inference workloads.

• Improvement: Replace black-box anomaly detection (e.g., autoencoders) with FP-Growth-based association rule mining on system-level telemetry (e.g., cache misses, branch mispredictions, memory access patterns) to automatically discover interpretable rules that distinguish benign from malicious AI workloads.

• Capability: The improved AI system can provide human-readable explanations (e.g., high co-occurrence of misprediction and pipeline flush signals indicates attack) for why a workload is flagged, enabling security analysts to validate and tune detection without deep ML expertise.

• Improvement: Deploy the rule-based hardware monitor in edge devices running federated learning to detect branch predictor attacks that leak other clients' model updates or training data during local inference.

• Capability: The improved AI system can terminate compromised local training sessions before sensitive gradients are transmitted, preventing data leakage across federated clients with negligible performance overhead (<1% as shown in the paper).

• Improvement: Use the discovered association rules as a baseline and implement a windowed monitoring approach (as suggested in future work) to track rule activation frequencies over time, enabling detection of novel side-channel attacks that deviate from known patterns.

• Capability: The improved AI system can generalize to previously unseen attack variants within the same family (e.g., new branch predictor exploitation techniques) without retraining, as demonstrated by the IBPA detection under TAGE configuration.

• Improvement: Integrate the RuleMonitor module into open-source RISC-V-based AI processors (e.g., BOOM-based SoCs for edge AI) during the design phase, rather than as a post-hoc patch.

• Capability: The improved AI system can be fabricated with built-in, verifiable side-channel resistance, reducing the need for software-level mitigations that often degrade AI inference performance.

• Improvement: Extend the association rule mining to correlate RTL-level signals with higher-level AI pipeline events (e.g., layer execution boundaries, memory access patterns during matrix multiplication) to detect attacks that span multiple abstraction layers.

• Capability: The improved AI system can detect sophisticated attacks that manipulate both microarchitectural state and algorithmic behavior, providing end-to-end security for AI inference and training pipelines.

• Improvement: Apply the same rule-mining methodology to classify AI workloads (e.g., inference vs. training, benign vs. adversarial) based on hardware-level behavioral signatures, using the 5% support and 90% confidence thresholds as calibrated in the paper.

• Capability: The improved AI system can accurately distinguish between legitimate AI tasks (e.g., image classification, natural language processing) and malicious workloads (e.g., model extraction, adversarial perturbation) in real time, without interrupting normal operation.

Runtime detection of microarchitectural side channel attacks remains significantly underexplored in RISCV compared with x86 and ARM ISAs, posing a serious threat to critical applications. State-of-the-art branch predictor attacks bypass traditional data and instruction caches by directly exploiting the state of internal history tables, making them inherently stealthy. Recent research has explored offline detection of microarchitectural attacks on RISC-V; however, efficient runtime detection of microarchitectural attacks on RISC-V hardware remains significantly unaddressed. State-of-the-art hardware-based runtime detection solutions leverage hardware performance counters (HPCs) but suffer from a restricted set of counter registers and tradeoffs between detection accuracy, detection speed, and sampling granularity, making them impractical for stealthy attacks. Moreover, sampling HPCs after distinct intervals leaves intermediate relationships between different microarchitectural blocks unobserved. Additionally, proprietary x86 and ARM ISAs constrain researchers from modifying processor microarchitectural designs. To address these limitations, we propose the first secure-by-design, highly interpretable, non-intrusive, RTL-level runtime detection solution for stealthy branch predictor attacks on BOOM RISC-V, evaluated under both simplified Next-Line Predictor (NLP) and complex TAGE predictor configurations. The attack detection relies on association rules extracted offline and embedded in hardware as a non-intrusive rule monitor that enables runtime detection. The proposed approach achieves excellent detection speed, terminates execution before secret disclosure, and produces zero false positives while remaining flexible for detecting previously unseen variants within the same family of branch predictor attacks.