Anomaly detection using surprisals
summary
The gist
Anomaly detection methods are often rather ad hoc and lack a solid theoretical foundation, or they make strong distributional assumptions that may not hold in practice <ref:2603.09318#pg2> The basic…
In short
This method detects anomalies by calculating 'surprisal,' which is the negative log density of an observation under a model. An anomaly score is then derived from this surprisal distribution to estimate tail probabilities, allowing detection across various data types and distributions.
Key concepts
- Surprisal
- Surprisal measures how unlikely an observation is given a specific probability distribution. It is calculated as the negative logarithm of the probability density function evaluated at that observation's value. High surprisal indicates an unusual event.
- Anomaly Score
- The anomaly score is defined as the probability that a calculated surprisal value will be at least as large as the observed one. This transforms complex multivariate data into estimating the upper tail of a simpler univariate distribution of surprisals.
- Tail Probability Estimators
- The paper develops two ways to estimate extreme probabilities: an empirical method based on observed data and a Generalized Pareto Distribution (GPD) estimator that fits heavy-tailed distributions. These estimators are used to quantify the likelihood of very rare, extreme events.
Terminology used across episodes
This episode discusses
The paper
Anomaly detection using surprisals · Read on arXiv
Rob J Hyndman, David T Frazier
Monash University
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Anomaly detection using surprisals".
Jane: Here's the gist of the paper. Anomaly detection methods are often rather ad hoc and lack a solid theoretical foundation, or they make strong distributional assumptions that may not hold in practice <ref:2603.09318#pg2> The basic idea is to calculate the surprisal (equal to the negative log density) of each observation, and then estimate the tail probability from the surprisal distribution <ref:2603.09318#pg5> This approach allows for a unified treatment of anomaly detection that can be applied to any anomaly detection problem, provided a probability distribution can be defined on the space of observations <ref:2603.09318#pg3> It allows for univariate and multivariate settings, and can be applied to both unconditional and conditional distributions <ref:2603.09318#pg3> Core Framework The framework defines an anomaly as an observation with unusually low probability under a (possibly misspecified) model <ref:2603.09318#pg5> For each observation we compute its surprisal (the negative log generalized density) and define an anomaly score as the probability of a surprisal at least as large as that observed <ref:2603.09318#pg5> This reduces anomaly detection for complex univariate or multivariate data to estimating the upper tail of a univariate surprisal distribution <ref:2603.09318#pg5> The approach allows for anomalies to occur anywhere in the distribution of F, not just in its tails <ref:2603.09318#pg6> Estimating Tail Probabilities The paper develops two model-robust estimators of these tail probabilities: an empirical estimator based on the observed surprisal distribution and an extreme-value estimator that fits a Generalized Pareto Distribution above a high threshold <ref:2603.09318#pg5> The empirical method requires Assumption 2.1, which states there exists an s⋆ and a function h(·) that is strictly increasing on
s⋆, ∞), such that S = h(bS) almost surely for all bS ≥ s⋆ <ref:2603.09318#pg8> The GPD method requires conditions on the surprisal distribution of S = − log f(Y), describing increasingly heavy tails, such as sub-Gaussian, sub-exponential, or polynomial <ref:2603.09318#pg9> Theoretical Guarantees Theorem 3.1 shows that under weak conditions on the tail of the surprisal distribution, the maximum surprisal converges to one of the three types of extreme value distributions Specifically, if Assumption 3.1 is satisfied, then for all s > 0, Pr Mn − E[S: ≥ √2ν2s + q2ν2 log(n) ≤ e−s This implies that if the surprisal is sub-Gaussian, then the maximal surprisal will follow a reversed Weibull distribution Practical Applications The method can be applied to conditional distributions as well, where the surprisal S = − log fZ(Z X = x) can be thought of as being computed “conditional on a given X = x” <ref:2603.09318#pg10> Simulations show that the approach remains effective under substantial model misspecification in applications to French mortality and Test-cricket data <ref:2603.09318#pg5> The results demonstrate that even when the wrong distribution is used to compute the surprisal values, either empirical or GPD estimates remain relatively accurate <ref:2603.09318#pg6> This suggests that in practice, it is safer to err on the side of assuming heavier tails Software and Reproducibility All results presented here can be reproduced using the data and code available at https://github.com/robjhyndman/Surprisal Theory <ref:2603.09318#pg5> The analysis was conducted using R version 4.5.2 (R Core Team 2025), with the following R packages: weird (Hyndman 2026), tsibble (Wang et al.
Tom: 2026), targets (Landau 2026, 2019), ggplot2 (Wickham et al.
Jane: 19) and other tidyverse (Wickham et al.
Tom: 19) packages <ref:2603.09318#pg5> The authors are funded by the Australian Research Council through Discovery Project DP250100702 <ref:2603.09318#pg5> The paper is arXiv:2603.09318v1
stat.ME: 10 Mar 26 <ref:2603.09318#pg5> References Coles, S.
Paper summary: Jane: Looking at the paper "Anomaly detection using surprisals" by Rob Hyndman and David Frazier, it’s clear they’ve moved anomaly detection away from those ad hoc rules toward a more principled framework based on defining anomalies as observations of low probability under a model.
Tom: The authors propose this unified approach where you calculate the surprisal for each observation—the negative log density—and then use that to define an anomaly score, which is essentially the probability of seeing something even more surprising than what you observed.
Lu: They give us two ways to estimate that tail probability: an empirical estimator using what we observe and a Generalized Pareto Distribution estimator that fits data above a high threshold.
Meng: The key takeaway for someone building a system at scale is that this framework offers robustness to substantial mis-specification of the distribution we use, even if our assumed model isn't perfectly correct.
Lalam: They also show that depending on the conditions, like if the surprisal distribution is sub-Gaussian, we can predict what kind of extreme value distribution the maximum surprisal will follow.
Jane: In simple terms, this paper suggests that instead of just guessing rules for outliers, you can use information theory to give every data point a score based on how surprising it is under your model.
Tom: It’s about taking something that used to be ad hoc and giving it a solid theoretical foundation by focusing on surprisal and tail probability estimation.
Conclusion: Tom: So, we're wrapping up this paper on anomaly detection using surprisals by Hyndman and Frazier, and honestly, the title itself sounds a bit dense for what it actually is doing.
Jane: It does sound complicated because they're talking about negative log density and tail probabilities all at once. But basically, they’re suggesting that instead of just looking for points that are far away from the average, we should look at how surprising each data point is under our current model.
Lu: Exactly. The core idea is defining an anomaly as a measurement whose surprisal—the negative log density—is unusually high compared to what we expect. It unifies detection whether you're looking at simple numbers or complex multivariate systems.
Meng: From my side, the real interest is how they tackle the estimation of those tail probabilities. They propose two methods: one based on observing the surprisal distribution directly, and another using something called a Generalized Pareto Distribution that fits data above a certain high level.
Tom: That’s where it gets practical for me. Because we often don't have clean theoretical assumptions about the true underlying distribution, being able to use two different estimators gives us a safety net when our initial model is probably wrong.
Jane: Right, so if you're driving and you see something weird on the road, this method lets you figure out how unusual that weirdness is based on what you *think* the road should look like.
Lu: And they even show some theoretical bounds. If the surprisal distribution has certain properties—like being sub-Gaussian—then they can predict exactly what shape the most extreme surprising measurement will take.
Tom: Predicting the shape of the maximum surprised value is interesting, but it’s not just for academics. It means we have a more principled way to set thresholds for flagging outliers in real-time data streams without getting lost in pure statistical guesswork.
Jane: So, they've given us a toolkit that works even when our assumptions about the data are shaky, and they're giving us tools to estimate the risk of seeing truly extreme events.
Lu: It moves anomaly detection from being a collection of specific algorithms to being a way of measuring information itself. Next up, we’ll look at how this applies specifically to those conditional probability problems in real-world scenarios.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates