Anomaly detection using surprisals

arXiv:2603.09318 · stat.ME, stat.AP, stat.OT · Submitted 2026-03-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Anomaly detection using surprisals".

Jane: Here's the gist of the paper. Anomaly detection methods are often rather ad hoc and lack a solid theoretical foundation, or they make strong distributional assumptions that may not hold in practice <ref:2603.09318#pg2> The basic idea is to calculate the surprisal (equal to the negative log density) of each observation, and then estimate the tail probability from the surprisal distribution <ref:2603.09318#pg5> This approach allows for a unified treatment of anomaly detection that can be applied to any anomaly detection problem, provided a probability distribution can be defined on the space of observations <ref:2603.09318#pg3> It allows for univariate and multivariate settings, and can be applied to both unconditional and conditional distributions <ref:2603.09318#pg3> Core Framework The framework defines an anomaly as an observation with unusually low probability under a (possibly misspecified) model <ref:2603.09318#pg5> For each observation we compute its surprisal (the negative log generalized density) and define an anomaly score as the probability of a surprisal at least as large as that observed <ref:2603.09318#pg5> This reduces anomaly detection for complex univariate or multivariate data to estimating the upper tail of a univariate surprisal distribution <ref:2603.09318#pg5> The approach allows for anomalies to occur anywhere in the distribution of F, not just in its tails <ref:2603.09318#pg6> Estimating Tail Probabilities The paper develops two model-robust estimators of these tail probabilities: an empirical estimator based on the observed surprisal distribution and an extreme-value estimator that fits a Generalized Pareto Distribution above a high threshold <ref:2603.09318#pg5> The empirical method requires Assumption 2.1, which states there exists an s⋆ and a function h(·) that is strictly increasing on

s⋆, ∞), such that S = h(bS) almost surely for all bS ≥ s⋆ <ref:2603.09318#pg8> The GPD method requires conditions on the surprisal distribution of S = − log f(Y), describing increasingly heavy tails, such as sub-Gaussian, sub-exponential, or polynomial <ref:2603.09318#pg9> Theoretical Guarantees Theorem 3.1 shows that under weak conditions on the tail of the surprisal distribution, the maximum surprisal converges to one of the three types of extreme value distributions Specifically, if Assumption 3.1 is satisfied, then for all s > 0, Pr Mn − E[S: ≥ √2ν2s + q2ν2 log(n) ≤ e−s This implies that if the surprisal is sub-Gaussian, then the maximal surprisal will follow a reversed Weibull distribution Practical Applications The method can be applied to conditional distributions as well, where the surprisal S = − log fZ(Z X = x) can be thought of as being computed “conditional on a given X = x” <ref:2603.09318#pg10> Simulations show that the approach remains effective under substantial model misspecification in applications to French mortality and Test-cricket data <ref:2603.09318#pg5> The results demonstrate that even when the wrong distribution is used to compute the surprisal values, either empirical or GPD estimates remain relatively accurate <ref:2603.09318#pg6> This suggests that in practice, it is safer to err on the side of assuming heavier tails Software and Reproducibility All results presented here can be reproduced using the data and code available at https://github.com/robjhyndman/Surprisal Theory <ref:2603.09318#pg5> The analysis was conducted using R version 4.5.2 (R Core Team 2025), with the following R packages: weird (Hyndman 2026), tsibble (Wang et al.

Tom: 2026), targets (Landau 2026, 2019), ggplot2 (Wickham et al.

Jane: 19) and other tidyverse (Wickham et al.

Tom: 19) packages <ref:2603.09318#pg5> The authors are funded by the Australian Research Council through Discovery Project DP250100702 <ref:2603.09318#pg5> The paper is arXiv:2603.09318v1

stat.ME: 10 Mar 26 <ref:2603.09318#pg5> References Coles, S.

Paper summary: Jane: Looking at the paper "Anomaly detection using surprisals" by Rob Hyndman and David Frazier, it’s clear they’ve moved anomaly detection away from those ad hoc rules toward a more principled framework based on defining anomalies as observations of low probability under a model.

Tom: The authors propose this unified approach where you calculate the surprisal for each observation—the negative log density—and then use that to define an anomaly score, which is essentially the probability of seeing something even more surprising than what you observed.

Lu: They give us two ways to estimate that tail probability: an empirical estimator using what we observe and a Generalized Pareto Distribution estimator that fits data above a high threshold.

Meng: The key takeaway for someone building a system at scale is that this framework offers robustness to substantial mis-specification of the distribution we use, even if our assumed model isn't perfectly correct.

Lalam: They also show that depending on the conditions, like if the surprisal distribution is sub-Gaussian, we can predict what kind of extreme value distribution the maximum surprisal will follow.

Jane: In simple terms, this paper suggests that instead of just guessing rules for outliers, you can use information theory to give every data point a score based on how surprising it is under your model.

Tom: It’s about taking something that used to be ad hoc and giving it a solid theoretical foundation by focusing on surprisal and tail probability estimation.

Conclusion: Tom: So, we're wrapping up this paper on anomaly detection using surprisals by Hyndman and Frazier, and honestly, the title itself sounds a bit dense for what it actually is doing.

Jane: It does sound complicated because they're talking about negative log density and tail probabilities all at once. But basically, they’re suggesting that instead of just looking for points that are far away from the average, we should look at how surprising each data point is under our current model.

Lu: Exactly. The core idea is defining an anomaly as a measurement whose surprisal—the negative log density—is unusually high compared to what we expect. It unifies detection whether you're looking at simple numbers or complex multivariate systems.

Meng: From my side, the real interest is how they tackle the estimation of those tail probabilities. They propose two methods: one based on observing the surprisal distribution directly, and another using something called a Generalized Pareto Distribution that fits data above a certain high level.

Tom: That’s where it gets practical for me. Because we often don't have clean theoretical assumptions about the true underlying distribution, being able to use two different estimators gives us a safety net when our initial model is probably wrong.

Jane: Right, so if you're driving and you see something weird on the road, this method lets you figure out how unusual that weirdness is based on what you *think* the road should look like.

Lu: And they even show some theoretical bounds. If the surprisal distribution has certain properties—like being sub-Gaussian—then they can predict exactly what shape the most extreme surprising measurement will take.

Tom: Predicting the shape of the maximum surprised value is interesting, but it’s not just for academics. It means we have a more principled way to set thresholds for flagging outliers in real-time data streams without getting lost in pure statistical guesswork.

Jane: So, they've given us a toolkit that works even when our assumptions about the data are shaky, and they're giving us tools to estimate the risk of seeing truly extreme events.

Lu: It moves anomaly detection from being a collection of specific algorithms to being a way of measuring information itself. Next up, we’ll look at how this applies specifically to those conditional probability problems in real-world scenarios.

Rob J Hyndman, David T Frazier

Monash University

stat.ME, stat.AP, stat.OT

Submitted: 2026-03-10

Updated: 2026-11-16

Code: https://github.com/robjhyndman/Surprisal_Theory

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: Anomaly detection methods are often rather ad hoc and lack a solid theoretical foundation, or they make strong distributional assumptions that may not hold in practice <ref:2603.09318#pg2> The basic…

Key concepts

Surprisal
Surprisal measures how unlikely an observation is given a specific probability distribution. It is calculated as the negative logarithm of the probability density function evaluated at that observation's value. High surprisal indicates an unusual event.
Anomaly Score
The anomaly score is defined as the probability that a calculated surprisal value will be at least as large as the observed one. This transforms complex multivariate data into estimating the upper tail of a simpler univariate distribution of surprisals.
Tail Probability Estimators
The paper develops two ways to estimate extreme probabilities: an empirical method based on observed data and a Generalized Pareto Distribution (GPD) estimator that fits heavy-tailed distributions. These estimators are used to quantify the likelihood of very rare, extreme events.

Terminology

Summary

Anomaly detection methods are often rather ad hoc and lack a solid theoretical foundation, or they make strong distributional assumptions that may not hold in practice <ref:2603. p. 109318#pg2> The basic idea is to calculate the surprisal (equal to the negative log density) of each observation, and then estimate the tail probability from the surprisal distribution <ref:2603. p. 109318#pg5> This approach allows for a unified treatment of anomaly detection that can be applied to any anomaly detection problem, provided a probability distribution can be defined on the space of observations <ref:2603. p. 109318#pg3> It allows for univariate and multivariate settings, and can be applied to both unconditional and conditional distributions <ref:2603. p. 109318#pg3>

Core Framework

The framework defines an anomaly as an observation with unusually low probability under a (possibly misspecified) model <ref:2603.09318#pg5> For each observation we compute its surprisal (the negative log generalized density) and define an anomaly score as the probability of a surprisal at least as large as that observed <ref:2603. p. 009318#pg5> This reduces anomaly detection for complex univariate or multivariate data to estimating the upper tail of a univariate surprisal distribution <ref:2603. p. 009318#pg5> The approach allows for anomalies to occur anywhere in the distribution of F, not just in its tails <ref:2603. p. 209318#pg6>

Estimating Tail Probabilities

The paper develops two model-robust estimators of these tail probabilities: an empirical estimator based on the observed surprisal distribution and an extreme-value estimator that fits a Generalized Pareto Distribution above a high threshold <ref:2603.09318#pg5> The empirical method requires Assumption 2. p. 01, which states there exists an s⋆ and a function h(·) that is strictly increasing on [s⋆, ∞), such that S = h(bS) almost surely for all bS ≥ s⋆ <ref:2603. p. 609318#pg8> The GPD method requires conditions on the surprisal distribution of S = − log f(Y), describing increasingly heavy tails, such as sub-Gaussian, sub-exponential, or polynomial <ref:2603. p. 709318#pg9>

Theoretical Guarantees

Theorem 3.1 shows that under weak conditions on the tail of the surprisal distribution, the maximum surprisal converges to one of the three types of extreme value distributions Specifically, if Assumption 3. p. 61 is satisfied, then for all s > 0, Pr Mn − E[S] ≥ √2ν2s + q2ν2 log(n) ≤ e−s This implies that if the surprisal is sub-Gaussian, then the maximal surprisal will follow a reversed Weibull distribution

Practical Applications

The method can be applied to conditional distributions as well, where the surprisal S = − log fZ(Z X = x) can be thought of as being computed “conditional on a given X = x” <ref:2603.09318#pg10> Simulations show that the approach remains effective under substantial model misspecification in applications to French mortality and Test-cricket data <ref:2603. p. 009318#pg5> The results demonstrate that even when the wrong distribution is used to compute the surprisal values, either empirical or GPD estimates remain relatively accurate <ref:2603. p. 909318#pg6> This suggests that in practice, it is safer to err on the side of assuming heavier tails

Software and Reproducibility

All results presented here can be reproduced using the data and code available at https://github.com/robjhyndman/Surprisal Theory <ref:2603. p. 0 p. 009318#pg5> The analysis was conducted using R version 4. p. 0 p. 05.2 (R Core Team 2025), with the following R packages: weird (Hyndman 2026), tsibble (Wang et al. p. 7 p. 7 2026), targets (Landau 2026, 2019), ggplot2 (Wickham et al. p. 0 19) and other tidyverse (Wickham et al. 19) packages <ref:2603. p. 0 p. 009318#pg5> The authors are funded by the Australian Research Council through Discovery Project DP250100702 <ref:2603. p. 0 p. 009318#pg5> The paper is arXiv:2603. p. 0 p. 009318v1 [stat. p. 0 p. 0ME] 10 Mar 26 <ref:2603. p. 0 p. 009318#pg5>

References

Coles, S. (2001), An introduction to statistical modeling of extreme values, Vol. p. 9 p. 9 208 of Springer Series in Statistics, Springer, London UK <ref:2603.09318#pg5> Cover, T. p. 2 p. 2 M. & Thomas, J. p. 2 p. 2 A. (2006), Elements of information theory, 2nd edn, Wiley <ref:2603. p. 0 p. 209318#pg5> Davies, L. p. 3 p. 3 & Gather, U. p. 3, ‘The identification of multiple outliers’, 88, 782–792 <ref:2603. p. 0 p. 009318#pg5> Good, I. p. 2 p. 2 J. (1952), ‘Rational decisions’, Journal of the Royal Statistical Society. p. 2 Series B, Statistical methodology 14, 107–114 <ref:2603. p. 0 p. 009318#pg5> Grubbs, F. p. 3 p. 3 E., ‘Sample criteria for testing outlying observations’, Annals of Mathematical Statistics 21(1), 27–58 <ref:2603. p. 2 p. 209318#pg5>

Human Mortality Database (2025)

Human Mortality Database (2025), ‘University of California, Berkeley (USA); Max Planck Institute for Demographic Research (Germany)’. Accessed on 6 October 2025 <ref:2603. p. 0 p. 009318#pg5> Hyndman, R. p. 0 p. 0 p. 0 p. 0 J. (1996), ‘Computing and graphing highest density regions’, The American Statistician 50(2), 120–126 <ref:2603. p. 3 p. 309318#pg5> Hyndman, R. J., Kandanaarachchi, S. p. 7 p. 7 & Turner, K. (2025), When lookout sees crackle: Anomaly detection via kernel density estimation <ref:2603. p. 7 p. 709318#pg5> Kandanaarachchi, S. p. 7 p. 7 & Hyndman, R. p. 0 J. (2022), ‘Leave-one-out kernel density estimates for outlier detection’, J Computational & Graphical Statistics 31, 586–599 <ref:2603. p. 7 p. 709318#pg5> Landau, W. M. (2021), ‘The targets R package: a dynamic make-like function-oriented pipeline toolkit for reproducibility and high-performance computing’, Journal of Open Source Software 6(57), 2959 <ref:2603.09318#pg5> Landau, W. M. (2026), targets: Dynamic Function-Oriented ‘Make’-Like Declarative Pipelines <ref:2603.09318#pg5> R Core Team (2025), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria <ref:2603. p. 2 p. 209318#pg5> Stone, J. p. 2 p. 2 V. (2022), Information theory: a tutorial introduction, Sebtel Press, Sheffield, England <ref:2603. p. 0 p. 209318#pg5> Tribus, M. p. 2 p. 2 (1961), Thermodynamics and thermostatics: An introduction to energy, information and states of matter, with engineering applications, D. p. 1 Van Nostrand, New York, USA <ref:2603. p. 0 p. 009318#pg5> Wang, E., Cook, D., Hyndman, R. p. 0 J., O’Hara-Wild, M., Smith, T. & Davis, W. (2026), tsibble: Tidy Temporal Data Frames and Tools <ref:2603. p. 0 p. 009318#pg5> Wickham, H. (2016), ggplot2: Elegant Graphics for Data Analysis, Springer-Verlag, New York, USA <ref:2603. p. 009318#pg5> Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache S. M., Müller K., Ooms J., Robinson D., Seidel D P., Spinu V., Takahashi K.., Vaughan D., Wilke C., Woo K. & Yutani H (2019), ‘Welcome to the tidyverse’, Journal of Open Source Software 4(43), 1686 <ref:2603. p. 009318#pg5> Wickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K., Wilke C., Woo K., Yutani H., Dunnington D., van den Brand T. & Posit PBC (2026), ggplot2: Create Elegant Data Visualisations Using the Grammar of Graphics <ref:2603. p. 0 p. 009318#pg5> Wilkinson, L. p. 2 p. 2 (2018), ‘Visualizing big data outliers through distributed aggregation’, IEEE Transactions on Visualization and Computer Graphics 24(1), 256–266 <ref:2603. p. 0 p. 009318#pg5> Wood, S. N. (2017), Generalized additive models: An introduction with R, CRC press <ref:2603. p. 0 p. 009318#pg5> Wood, S. N. (2025), Mixed GAM Computation Vehicle with Automatic Smoothness Estimation <ref:2603. p. 0 p. 709318#pg5>

Applications and Findings

The application to French mortality rates revealed anomalies coinciding with various wars and epidemics in French history <ref:2603.09318#pg6> In Test cricket data, the most anomalous batters were identified as all “tail-enders” who played for a long time <ref:2603. p. 009318#pg8> The surprisal framework provides a flexible and principled way to quantify how unusual an observation is under any specified distributional model <ref:2603. p. 109318#pg10> The empirical estimator requires no tuning or model estimation, and provides finite-sample uniform error control, but it can be noisy when nα is small The GPD estimator is most useful when nα is small, as it can reduce variance and allow extrapolation beyond the observed maximum The framework yields a robust, interpretable, and theoretically grounded approach to anomaly detection

Discussion on Misspecification

The theoretical results make clear what kind of misspecification can be tolerated <ref:2603.09318#pg5> For the empirical tail-probability estimator to be reliable, the assumed model must correctly identify the locations of these low-density regions <ref:2603. p. 609318#pg8> The GPD approach offers a complementary route to estimating surprisal tail probabilities <ref:2603. p. 409318#pg8> The situation is slightly different from that for the empirical approach, as the GPD is motivated by the asymptotic behavior of the surprisals When the condition fails, such as when the assumed model imposes symmetry where the data are skewed, or unimodality where the data are multimodal, the empirical estimator no longer provides uniform guarantees <ref:2603.09318#pg8> The authors suggest that in practice, it is safer to err on the side of assuming heavier tails

Method Selection

The choice between the empirical and GPD approaches depends on the sample size, n, and the desired level of significance, α <ref:2603.09318#pg8> The empirical estimator requires no tuning or model estimation, and provides finite-sample uniform error control <ref:2603. p. 709318#pg8> The GPD estimator is most useful when nα is small, as it can reduce variance and allow extrapolation beyond the observed maximum When the surprisal distribution exhibits discreteness or other non-smooth features, the empirical approach may be preferable In many cases, it may be advisable to use both approaches in tandem as a robustness check

All results presented here can be reproduced using the data and code available at https://github.com/robjhyndman/Surprisal Theory <ref:2603.09318#pg5> The analysis was conducted using R version 4.5.2 (R Core Team 2025), with the following R packages: weird (Hyndman 2026), tsibble (Wang et al. 2026), targets (Landau 2019, 1) and ggplot2 (Wickham et al. p. 0 16, Wickham H., Chang W., Henry L., Pedersen T. L., Takahashi K., Wilke C., Woo K., Yutani H., Dunnington D., van den Brand T. & Posit PBC (2026)) and other tidyverse (Wickham et al. p. 0 19) packages <ref:2603.09318#pg5> The authors are funded by the Australian Research Council through Discovery Project DP250100702 <ref:2603.09318#pg5> The paper is arXiv:2603.09318v1 [stat.ME] 10 Mar 26 <ref:2603.09318#pg5>

Coles, S. (2001), An introduction to statistical modeling of extreme values, Vol. 208 of Springer Series in Statistics, Springer, London UK <ref:2603.09318#pg5> Cover, T. M. & Thomas, J. A., Elements of information theory, 2nd edn, Wiley <ref:2603.09318#pg5> Davies, L., & Gather, U. p. 3, ‘The identification of multiple outliers’, 88, 782–792 <ref:2603.09318#pg5> Good, I. J., ‘Rational decisions’, Journal of the Royal Statistical Society. Series B, Statistical methodology 14, 107–114 <ref:2603.09318#pg5> Grubbs, F. E., ‘Sample criteria for testing outlying observations’, Annals of Mathematical Statistics 21(1), 27–58 <ref:2603.09318#pg5> Human Mortality Database (2025), ‘University of California, Berkeley (USA); Max Planck Institute for Demographic Research (Germany)’. Accessed on 6 October 2025 <ref:2603.09318#pg5> Hyndman, R. J. (1996), ‘Computing and graphing highest density regions’, The American Statistician 50(2), 120–126 <ref:2603.09318#pg5> Hyndman, R. J., Kandanaarachchi, S., & Turner, K. (2025), When lookout sees crackle: Anomaly detection via kernel density estimation <ref:2603.09318#pg5> Kandanaarachchi, S., & Hyndman, R. p. 0 J. (2022), ‘Leave-one-out kernel density estimates for outlier detection’, J Computational & Graphical Statistics 31, 586–599 <ref:2603.09318#pg5> Landau, W. M., ‘The targets R package: a dynamic make-like function-oriented pipeline toolkit for reproducibility and high-performance computing’, Journal of Open Source Software 6(57), 2959 <ref:2603.09318#pg5> Landau, W. M., targets: Dynamic Function-Oriented ‘Make’-Like Declarative Pipelines <ref:2603.09318#pg5> R Core Team (2025), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria <ref:2603.09318#pg5> Stone, J. V., Information theory: a tutorial introduction, Sebtel Press, Sheffield, England <ref:2603.09318#pg5> Tribus, M., Thermodynamics and thermostatics: An introduction to energy, information and states of matter, with engineering applications, D. Van Nostrand, New York, USA <ref:2603.09318#pg5> Wang, E., Cook, D., Hyndman R. p. 0 J., O’Hara-Wild M., Smith T. & Davis W. (2026), tsibble: Tidy Temporal Data Frames and Tools <ref:2603.09318#pg5> Wickham, H., Averick M., Bryan J., Chang W., McGowan L. D., François R., Grolemund G., Hayes A., Henry L., Hester J., Kuhn M., Pedersen T. L., Miller E., Bache S. M., Müller K., Ooms J., Robinson D., Seidel D P., Spinu V., Takahashi K.., Vaughan D., Wilke C., Woo K., Yutani H (2019), ‘Welcome to the tidyverse’, Journal of Open Source Software 4(43), 1686 <ref:2603. p. 009318#pg5> Wickham, H., Chang W., Henry L., Pedersen T. L., Takahashi K., Wilke C., Woo K., Yutani H, Dunnington D., van den Brand T. & Posit PBC (2026), ggplot2: Create Elegant Data Visualisations Using the Grammar of Graphics <ref:2603.09318#pg5> Wilkinson, L. (2018), ‘Visualizing big data outliers through distributed aggregation’, IEEE Transactions on Visualization and Computer Graphics 24(1), 256–266 <ref:2603.09318#pg5> Wood, S. N., Generalized additive models: An introduction with R, CRC press <ref:2603.09318#pg5> Wood, S. N., Mixed GAM Computation Vehicle with Automatic Smoothness Estimation <ref:2603.09318#pg5>

A Proof of Lemma 2.1

The proof of Lemma 2.1 shows that for a given α ∈ (0, 1), and ε = q log(2/α)/(2n), we have for all s ≥ s⋆, bPn(s) − ε ≤ G(s) ≤ bPn + ε with probability at least 1 − α if and only if Assumption 2. p. 61 is satisfied <ref:2603. p. 009318#pg8>

**B Proof of Theorem 3.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified Anomaly Detection Framework: The system can define an anomaly as an observation with unusually low probability under a (possibly misspecified) model, allowing it to flag anomalies in low-density gaps between modes, which traditional methods often miss.

  2. Model Robustness via Surprisal Estimation: The improved system can estimate the upper tail of a univariate surprisal distribution using two model-robust estimators—an empirical estimator and an extreme-value estimator based on a Generalized Pareto Distribution (GPD)—to provide anomaly scores that are less sensitive to substantial model misspecification.

  3. Tail Probability Estimation via Empirical Method: The system can use the empirical approach where pi is the proportion of surprisals at least as large as si, which provides a finite sample confidence guarantee under the condition that the true and fitted surprisals are related by a strictly increasing transformation on the upper tail.

  4. Tail Probability Estimation via Extreme Value Theory: The system can fit a GPD to the largest surprisal values to estimate tail probabilities, justified by conditions such as S is sub-Gaussian, sub-exponential, or polynomial, which allows for robust estimation even when the assumed distribution F is incorrect.

  5. Handling Complex Distributions: The framework enables application to any probability distribution and can be applied to both univariate and multivariate settings, including conditional distributions where surprisals are computed conditional on a given X = x.

Related papers