Benchmarking Deep Learning Models for Anomaly Detection in Real-World Cloud Telemetry

arXiv:2610.15818 · cs.LG, cs.DC · Submitted 2026-10-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Benchmarking Deep Learning Models for Anomaly Detection in Real-World Cloud Telemetry".

Jane: The gist The modest performance differences suggest that real cloud anomalies are dominated by short-term deviations rather than persistent periodic structure, emphasizing the importance of timing-aware evaluation and robust post-processing.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at the paper "Benchmarking Deep Learning Models for Anomaly Detection in Real-World Cloud Telemetry," the main implication is that effective anomaly detection in cloud environments requires more than just a good deep learning architecture; you need carefully designed scoring and post-processing mechanisms>

Jane: They show us that models like TimesNet or their improved GRU autoencoder perform well because they are sensitive to those short-term deviations, and the timing-aware metric helps us select the right one for real operational use>

Lu: The authors establish that the cloud anomaly landscape is defined by these irregular, short-lived events rather than predictable periodic structures in telemetry data>

Meng: For deployment purposes, this means we should tune our systems to focus on detecting those specific transient deviations rather than trying to build models around long-term trends that don't really matter for immediate incident response>

Lalam: I'm confident this paper shows that the real world telemetry is dominated by these transient events rather than consistent periodic behavior>

Tom: So, when you think about applying this to production, it points toward needing dependency-aware and streaming methods that can handle this type of dynamic data scale>

Jane: This research provides a foundation for future work on causal or dependency-aware methods that are built specifically for the complexity of modern cloud platforms>

Conclusion: Tom: So we’ve gone through how these models actually perform on that Numenta Anomaly Benchmark, and now we're at the end of the road with this paper, "Benchmarking Deep Learning Models for Anomaly Detection in Real-World Cloud Telemetry." It’s a look at what these different deep learning architectures are actually capable of when you throw them into messy cloud data.

Jane: Yeah, so the authors are trying to figure out which models work best for real systems instead of just theoretical ones. The main idea is that the performance differences between these models aren't huge or dramatic, especially when you look at short-term spikes in behavior.

Lu: Exactly. The core finding is that real cloud anomalies are mostly these short-lived deviations, not some kind of long, predictable pattern hiding underneath everything else. That suggests we should be looking for quick changes in the data stream.

Meng: From my side, what matters is if the model can handle those spikes without blowing up on us in production. If it relies too much on detecting slow trends instead of these sudden shifts, it’s just noise that costs us money with false alarms.

Lalam: I've analyzed the results, and I’m pretty sure this paper confirms that the timing of the event is way more important than finding a perfect periodic structure in the data. The models that detect things early win here.

Tom: So, looking at all these architectures—the LSTMs, the Transformers like Anomaly Transformer, and even those periodicity focused ones like TimesNet—the takeaway for me is that we need to stop focusing on getting one single high score.

Jane: Right. The authors’ conclusion boils down to this: you need a combination of a good model *and* smart post-processing rules that know how to handle these short bursts in the telemetry. It’s about timing and context, not just raw accuracy numbers.

Lu: And they show that their improved GRU autoencoder gets the best balance—it’s sensitive enough to catch those fast deviations but manages its false alarms better than the other ones they tested. That suggests a path forward for building more stable detection systems.

Meng: For deployment, that stability is everything. If we can get a model that keeps its false positive rate low while still catching those critical short-term events, then it’s something I could actually think about shipping to my team.

Lalam: I'm fairly certain this work points toward a future where anomaly detection isn't just about pattern matching the past, but about understanding the immediate context of the stream itself. That’s what we need to keep focusing on. (Sound of traffic fades back in)

Sumit Dhyani, Sougata Mukherjea

Indian Institute of Technology Delhi

cs.LG, cs.DC

Submitted: 2026-10-11

Updated: 2026-10-11

Comments: Published in 2026 11th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA)

Journal ref: 2026 11th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), Chengdu, China, 2026, pp. 709-714

DOI: 10.1109/ICCCBDA69573.2026.11676496

Code: https://github.com/Sumit2399/Anomaly-Det

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The modest performance differences suggest that real cloud anomalies are dominated by short-term deviations rather than persistent periodic structure, emphasizing the importance of

Key concepts

Numenta Anomaly Benchmark (NAB): This is a specific evaluation framework used to test anomaly detection algorithms. It rewards models that detect anomalies early in time and penalizes those that detect them late or generate too many false alarms, which is crucial for real-world monitoring.
Recurrent Reconstruction Models: These models use architectures like LSTM and GRU to learn what 'normal' system behavior looks like. They then try to reconstruct the incoming telemetry data; large reconstruction errors signal an anomaly because the model cannot accurately reproduce the unexpected input.
Timing-Aware Evaluation: This involves evaluating detection performance based on *when* an anomaly is flagged, not just whether it was detected at all. It uses NAB metrics that specifically value early detection and penalize late alerts to better reflect operational needs.

Terminology

Benchmarking — 1× in this paper · explained in 4 episodes

Summary

The gist The modest performance differences suggest that real cloud anomalies are dominated by short-term deviations rather than persistent periodic structure, emphasizing the importance of timing-aware evaluation and robust post-processing.

Introduction and Motivation

Modern cloud platforms generate high-dimensional, noisy telemetry from thousands of microservices that require anomaly detection to maintain system availability Unlike classical anomaly detection benchmarks, cloud anomalies are typically short-lived, irregular, and largely nonperiodic. Techniques that perform well on clean, periodic industrial datasets often struggle when applied to realistic cloud telemetry. A key challenge is the operational cost of false positives, necessitating techniques that emphasize not only accuracy but also early detection and low false-alarm rates. Traditional point-wise metrics such as F1-score and ROC-AUC fail to capture these requirements. To address this, the Numenta Anomaly Benchmark (NAB) is adopted, which explicitly rewards early detection while penalizing late detections and false positives.

Models Evaluated

The study evaluates four distinct deep learning architectures on the IBM Cloud Console dataset. The models evaluated include:

** Recurrent reconstruction-based models using LSTM and GRU architectures, which learn representations of normal behavior and identify anomalies via reconstruction error.**

** Attention-based models, such as the Anomaly Transformer, which leverage self-attention to model long-range dependencies.**

** Periodicity-oriented forecasting models, including TimesNet and Autoformer, which explicitly model seasonal and trend components.**

The researchers also introduce an improved GRU-based detector incorporating log-scaled reconstruction error, MAD-based normalization, and persistence–cooldown rules.

Model Performance and Findings

The empirical results show that TimesNet achieves the highest NAB scores, primarily due to strong early-detection sensitivity. Transformer-based models also detect anomalies early but incur higher false-positive rates in noisy telemetry streams. The improved GRU autoencoder achieves competitive performance with a more stable false-alarm profile. The analysis suggests that real cloud anomalies are dominated by short-term deviations rather than persistent periodic structure.

Evaluation and Threshold Analysis

The timing-aware evaluation framework using NAB is crucial for operational cloud monitoring. A systematic threshold sensitivity analysis was conducted to examine how detection performance evolves across a wide range of thresholds. The selected operating threshold lies within a stable region where performance remains high across small threshold variations. This chosen operating point achieves a favorable balance by maintaining a low false positive rate while preserving strong event-level detection performance.

Ablation Study Insights

An ablation study of the improved GRU autoencoder quantified the contribution of each proposed enhancement. The effect of Robust Likelihood Estimation substantially increases the NAB score by improving sensitivity, but this gain comes at the cost of a sharp rise in false positives. The effect of Persistence and Cooldown Rules effectively suppresses false positives but also reduces true anomaly detections, leading to lower recall and degraded NAB performance. When all components are combined, the improved GRU achieves a balanced trade-off between early detection and false-alarm control.

Conclusion

This work presents a timing-aware empirical study of deep learning models for anomaly detection in real-world cloud telemetry from the IBM Cloud Console. The findings establish that effective anomaly detection in cloud environments requires not only expressive models but also carefully designed scoring and post-processing mechanisms. This research provides a foundation for future work on dependency-aware, streaming, and causally-informed methods capable of operating at the scale and complexity of modern cloud platforms.

REFERENCES

Babeer 2610.15818

Babeer 2610.15849

Babeer 26397

Babeer 26400

Babeer 26411

Babeer 26415

Babeer 26397

Babeer 26398

Babeer 26400

Babeer 26411

Reference [1] A. Lavin and S. Ahmad, “Evaluating Real-Time Anomaly Detection Algorithms – The Numenta Anomaly Benchmark,” in Proceedings of the 14th IEEE International Conference on Machine Learning and Applications (ICMLA), 2015

Reference [2] M. S. Islam, M. S. Rakha, W. Pourmajidi, J. Sivaloganathan, J. Steinbacher, and A. Miranskyy, “Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset,” in 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

Reference [3] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “Learning Phrase Representations Using RNN Encoder–Decoder for Statistical Machine Translation,” arXiv preprint arXiv:1406.1078, 2014

Reference [4] J. Xu, H. Wu, J. Wang, and M. Long, “Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy,” in International Conference on Learning Representations (ICLR), 2022

Reference [5] H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long, “TimesNet: Temporal 2D-Variation Modeling for Time Series,” in International Conference on Learning Representations (ICLR), 2023

Reference [6] X. Shang, J. Zhang, X. Jiang, and H. Luo, “Anomaly Detection for Multivariate Time Series Based on Contrastive Learning and Autoformer,” in 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD), 2024

Reference [7] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly Detection: A Survey,” ACM Computing Surveys, vol. 41, no. 3, Jul. 2009

Reference [8] Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2019

Reference [9] K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Soderstrom, “Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018

Reference [10] P. Malhotra, L. Vig, G. M. Shroff, and P. Agarwal, “Long Short Term Memory Networks for Anomaly Detection in Time Series,” in The European Symposium on Artificial Neural Networks (ESANN), 2015

Reference [11] J. Soldani and A. Brogi, “Anomaly Detection and Failure Root Cause Analysis in (Micro) Service-Based Cloud Applications: A Survey,” ACM Computing Surveys, vol. 55, no.

Improvements for AI systems

Bold scoring functions incorporate log-scaled reconstruction error and MAD-based normalization to improve robustness against outliers, which allows for more stable detection across noisy telemetry streams.
An improved GRU autoencoder incorporates persistence and cooldown rules to suppress transient false alarms, ensuring that only persistent anomalies are detected by enforcing temporal consistency.
Transformer models can be enhanced by incorporating association discrepancy scoring to capture relational changes between telemetry metrics, leading to detections that align more closely with the sustained degradation of system performance.
The system can leverage a timing-aware evaluation framework using NAB aligned with operational cloud monitoring to prioritize early detection and minimize false alarms, as suggested by the paper's focus on rewards for early detection while penalizing late detections and false positives.
A systematic threshold sensitivity analysis justifies the selection of a specific operating point, ensuring that the chosen threshold maintains a low false positive rate while preserving strong event-level detection performance.

Sources

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation