Fast and accurate false-alarm probabilities for linear detection statistics in Poisson noise
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: We're looking at the paper: "Fast and accurate false-alarm probabilities for linear detection statistics in Poisson noise".
Jocelyn: Here's the gist of the paper. Fast and accurate false-alarm probabilities for linear detection statistics in Poisson noise is a new method for computing false-alarm probabilities and detection thresholds for arbitrary linear filters in low-count regimes where background noise follows Poisson statistics, which avoids computationally expensive simulations The Core Finding The method derives an accurate and efficient solution for computing false-alarm probabilities for arbitrary linear detection statistics under Poisson noise The detection statistic is shown to be a compound Poisson random variable, which allows for efficient implementations based on the Fast Fourier Transform (FFT) and the Panjer recursion This approach transforms the computation of detection significance under Poisson noise from a simulation problem into a deterministic numerical calculation Statistical Formulation The statistic S is defined as a compound Poisson random variable, which is the sum of independent identically distributed (i.i.d.) random variables where the number of terms in the sum is itself Poisson-distributed The probability distribution of S under the null hypothesis H0, denoted as PH0(s), can be expressed using a known result from Feller (1968) This distribution is given by Equation 16, which involves the Poisson probability Ppois(n) and the n-fold convolution of the one-photon probability distribution pX with itself The single-photon distribution pX is determined by Equation 18, which depends on the spatial distribution of background photons and kernel values Computational Implementations The paper presents two primary computational strategies for evaluating the compound Poisson distribution The FFT Implementation Using the Characteristic Function involves calculating the characteristic function of the one-photon contribution X using a single Fourier transform, followed by a pointwise complex exponential, and finally an inverse Fourier transform to recover PH0(s).
Vera: This method has an overall computational complexity of O (Npix + Nω log Nω), where Nω is the FFT length.
Jocelyn: The implementation includes adaptive refinement procedures to ensure stability against discretization and aliasing approximations.
Vera: The Panjer-Recursion Implementation offers an alternative, utilizing a recursive formula that avoids catastrophic cancellations in the inverse Fourier transform, with a complexity of O(Npix + NgridNpix).
Title and authors: Jocelyn: So let’s look at who wrote this paper. The authors are Maayane T Soumagnac, Eran O Ofek, Shachar Israeli and David Glukhov.
Vera: They’re all working in different areas of astrophysics, which is typical for these kinds of papers. We see Vera here focusing on survey photometry and spectra, while Jocelyn deals with pulsar and time-domain radio astronomy.
Subrahmanyan: I'm interested in how they bridge the gap between those observational constraints—like the kernel choice Stetson did—and the statistical calculation itself.
Jocelyn: They’re addressing two specific challenges: first, picking a kernel K that makes an optimal detection statistic S, and second, calculating that threshold above which we can declare a detection.
Vera: It seems they are tackling the mathematical complexity of the low-count regime where the Gaussian approximation for these statistics just doesn't hold up anymore.
Subrahmanyan: That’s crucial because if those approximations fail, our models for how structure forms or how signals propagate get skewed.
Jocelyn: The title itself suggests they are focused on speed and accuracy in low count regimes, which is where most of us spend a lot of time chasing faint signals.
Vera: It really feels like they are providing a tool that can handle the extreme false alarm probabilities that simulations usually make impractical to estimate.
Subrahmanyan: That capability would be useful for testing new detection methods in X-ray or neutrino detectors, where those low count environments are common.
The paper's summary: Vera: So, summarizing the core finding of this work, they prove that the detection statistic is indeed a compound Poisson variable. This means its probability distribution is known and has been studied extensively in risk theory for calculating aggregate loss distributions.
Jocelyn: And once you know that it’s compound Poisson, you can use established methods to compute this probability distribution rather than running huge Monte Carlo simulations every single time.
Subrahmanyan: That shifts the computational burden from brute-force simulation to a deterministic numerical calculation, which is a massive win for efficiency.
Vera: They then show two main ways to actually perform this computation efficiently: one based on the Fast Fourier Transform and another using a Panjer recursion.
Jocelyn: The FFT approach uses the characteristic function of the one photon contribution, applying a single Fourier transform, then a complex exponential, and finally an inverse Fourier transform to get back the probability distribution.
Subrahmanyan: And for that FFT method, they give us a complexity bound of O (Npix plus Nomega log Nomega), where Nomega is related to the FFT length.
Vera: The Panjer recursion is another path, which uses a recursive formula that avoids some of the numerical issues you might run into with the inverse Fourier transform.
Jocelyn: That recursive method has a complexity of O (Npix plus NgridNpix), and it lets you compute up to a bound nmax determined by how much tolerance you set for your count.
The paper's improvements: Vera: The real improvement they present is the choice of these two computational strategies—FFT versus Panjer recursion—which both offer significant speed gains over simulation methods.
Jocelyn: They also present an alternative approach in Appendix A, which is based on enumerating all possible realizations of the occupancy vector Mb, which they say is equivalent to the compound variable formalism.
Subrahmanyan: But they admit that this enumeration method is computationally heavier than the FFT or Panjer methods.
Vera: The paper then validates these results through five consistency checks comparing different discretization levels and aliasing tolerances, showing stability when you refine the binning or make the aliasing tolerance stricter.
Jocelyn: That stability check is important because it confirms that whether you use the FFT or Panjer, you’re getting a reliable threshold estimate.
Subrahmanyan: It also highlights a practical limitation they flag: for the FFT approach, there's a potential loss of numerical accuracy in the extreme tail of the distribution because of catastrophic cancellation during that inverse Fourier transform.
Conclusion: Vera: So to wrap up, this paper shows that by recognizing the detection statistic as a compound Poisson variable, we get an accurate and efficient way to compute false alarm probabilities for arbitrary linear filters in low count regimes.
Jocelyn: The FFT and Panjer methods are the key computational tools here, offering speeds in milliseconds compared to seconds for simulations.
Subrahmanyan: From my side, this method's main impact is enabling us to reliably evaluate extremely low false alarm probabilities that are often impractical to estimate by simulation alone.
Vera: It means we can finally get a handle on thresholds corresponding to significance levels that would otherwise be computationally inaccessible for source detection in low-count X-ray images or other faint sources.
Jocelyn: The Python and MATLAB implementations are available, which should help researchers bring this into use for their own work quickly.
Subrahmanyan: It’s a powerful tool because it’s independent of the specific linear filter you choose, so we can apply this framework broadly to any problem involving linear statistics and Poisson-distributed events.
Vera: That's it for this paper on "Fast and accurate false-alarm probabilities for linear detection statistics in Poisson noise." We’ll be looking at some papers next.
Department of Physics, Bar-Ilan University · Lawrence Berkeley National Laboratory · Benoziyo Center for Astrophysics, Weizmann Institute of Science
astro-ph.IM
Submitted: 2026-11-03
Updated: 2026-11-12
Comments: Submitted to ApJS
Code: https://github.com/EranOfek/AstroPack
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: Fast and accurate false-alarm probabilities for linear detection statistics in Poisson noise is a new method for computing false-alarm probabilities and detection thresholds for arbitrary linear…
Key concepts
- Compound Poisson Random Variable
- This is a random variable formed by summing a random number of independent, identically distributed variables. In this context, it means the total detection statistic (S) is the sum of individual photon contributions, where the *number* of those contributions itself follows a Poisson distribution.
- Fast Fourier Transform (FFT) Implementation
- This computational strategy uses a single Fourier transform to efficiently calculate the probability distribution PH0(s). It involves calculating the characteristic function of one-photon noise, applying an exponential function, and then using an inverse FFT to recover the final probability distribution. This method is highly efficient.
- Panjer Recursion Implementation
- This is an alternative computational technique that uses a recursive formula to determine the probability distribution. It avoids potential numerical errors like catastrophic cancellations found in some Fourier transform methods, offering a computationally lighter approach with complexity O(Npix + NgridNpix).
- Linear Detection Statistic
- This refers to any detection measurement derived from a linear combination of observed events. The method is versatile because it works for *any* such statistic, regardless of the specific filter used, as long as the underlying noise follows Poisson statistics.
Terminology
Summary
Fast and accurate false-alarm probabilities for linear detection statistics in Poisson noise is a new method for computing false-alarm probabilities and detection thresholds for arbitrary linear filters in low-count regimes where background noise follows Poisson statistics, which avoids computationally expensive simulations
The Core Finding
The method derives an accurate and efficient solution for computing false-alarm probabilities for arbitrary linear detection statistics under Poisson noise The detection statistic is shown to be a compound Poisson random variable, which allows for efficient implementations based on the Fast Fourier Transform (FFT) and the Panjer recursion This approach transforms the computation of detection significance under Poisson noise from a simulation problem into a deterministic numerical calculation
Statistical Formulation
The statistic S is defined as a compound Poisson random variable, which is the sum of independent identically distributed (i.i.d.) random variables where the number of terms in the sum is itself Poisson-distributed The probability distribution of S under the null hypothesis H0, denoted as PH0(s), can be expressed using a known result from Feller (1968) This distribution is given by Equation 16, which involves the Poisson probability Ppois(n) and the n-fold convolution of the one-photon probability distribution pX with itself The single-photon distribution pX is determined by Equation 18, which depends on the spatial distribution of background photons and kernel values
Computational Implementations
The paper presents two primary computational strategies for evaluating the compound Poisson distribution
** The FFT Implementation Using the Characteristic Function involves calculating the characteristic function of the one-photon contribution X using a single Fourier transform, followed by a pointwise complex exponential, and finally an inverse Fourier transform to recover PH0(s). This method has an overall computational complexity of O (Npix + Nω log Nω), where Nω is the FFT length. p. 8 The implementation includes adaptive refinement procedures to ensure stability against discretization and aliasing approximations. p. 8 The Panjer-Recursion Implementation offers an alternative, utilizing a recursive formula that avoids catastrophic cancellations in the inverse Fourier transform, with a complexity of O(Npix + NgridNpix). p. 8 This method allows for computation up to a bound nmax determined by the user tolerance epsiloncount. p. 8 The code is available in Python and MATLAB. p. 9**
Validation and Performance
The accuracy of the inferred detection threshold is verified through five consistency checks comparing results from different discretization levels and aliasing tolerances The method demonstrates stability when binning is refined or when the aliasing tolerance is made more stringent The computational complexity of the FFT and Panjer methods is significantly lower than simulation-based approaches, achieving results in milliseconds compared to seconds for simulations. For a 5x5 kernel stamp, the FFT-based implementation computes PH0 and the corresponding threshold Sth in only 30 ms. p. 9
Applicability
The framework is independent of the choice of linear filter, meaning it works for any linear detection statistic Although motivated by astronomical source detection, the method is broadly applicable to any detection problem involving linear statistics and Poisson-distributed events The solution enables reliable evaluation of extremely low false-alarm probabilities that are often impractical to estimate by simulation alone. This research was supported by the Israeli Science Foundation (grant Nos. p. 0 2068/22 and 2751/22) The paper has made use of data obtained from the Chandra Data Archive and the Chandra Source Catalog
Acknowledgments
This research was supported by the Israeli Science Foundation (grant Nos. 2068/22 and 2751/22) E.O.O. is grateful for support by grants from the Willner Family Leadership Institute This research has made use of resources of the National Energy Research Scientific Computing Center (NERSC) E. p. 0O.O. is grateful for support by grants from the Willner Family Leadership Institute, Madame Olga KleinAstrachan, Andr´e Deloro Institute, Schwartz/Reisman Collaborative Science Program, Paul and Tina Gardner, The Norman E Alexander Family Foundation ULTRASAT Data Center Fund, Jonathan Beare, Israel Science Foundation, Minerva, BSF-transformative, and the Weizmann-UK This paper is dedicated to the memory of Lucien Lazare
References
Brunner, H., Liu, T., Lamer, G., et al. 2022, A&A, 661, A1
Collin, G. H., Rodd, N. L., Erjavec, T., & Perez, K. 2022, ApJS, 260, 29
Condon, J. J. 1974, ApJ, 188, 279
Embrechts, P., & Frei, M. p. 5 2009, Mathematical Methods of Operations Research, 69, 497
Feller, W. 1968, An Introduction to Probability Theory and Its Applications, 3rd edn. p. 2, Vol. 1 (John Wiley & Sons)
Freeman, P. E., Kashyap, V., Rosner, R., & Lamb, D. Q. 2002, ApJS, 138, 185
Loaring, N. S., Dwelly, T., Page, M. J., et al. 2005, MNRAS, 362, 1371
Malyshev, D., & Hogg, D. p. 5 W. 2011, ApJ, 738, 181
Neyman, J., & Pearson, E. p. 0 S. 1933, Philosophical Transactions of the Royal Society of London Series A, 231, 289
Ofek, E. O. 2014, MATLAB package for astronomy and astrophysics, ascl:1407. p. 9005
Ofek, E. O., Shvartzvald, Y., Sharon, A., et al. 2023, PASP, 135, 124502
Ofek, E. O., & Zackay, B. p. 1 2018, AJ, 155, 169
Panjer, H. H. 1981, ASTIN Bulletin, 12, 22
Schechter, P. L., Mateo, M., & Saha, A. 1993, PASP, 105, 1342
Scheuer, P. A. G. 1957, Proceedings of the Cambridge Philosophical Society, 53, 764
Schilling, R. L. 2016, An Introduction to L´evy and Feller Processes. p. 2 Advanced Courses in Mathematics - CRM Barcelona 2014, arXiv:1603. p. 900251
Shevchenko, P. V. 2010, Journal of Operational Risk, 5, 3
Soumagnac, M. T., Ofek, E. O., Israeli, S. p. 0 S., Nir, G., & Dickstein, I. A. 2024, arXiv e-prints, arXiv:2412.07858
Stetson, P. B. 1987, PASP, 99, 191
Stewart, I. M. 2006, A&A, 454, 997
Vio, R., & Andreani, P. p. 2 2018, A&A, 616, A25
Webb, N. A., Coriat, M., Traulsen, I., et al. 2020, A&A, 641, A136
Zackay, B., & Ofek, E. p. 0 O. 2017, ApJ, 836, 187
Zackay, B., Ofek, E. O., & Gal-Yam, A. 2016, ApJ, 830, 27
Appendix A Summary
Appendix A presents an alternative approach to calculating the false alarm rate via photon-count enumeration This method constructs PH0(s) by enumerating all possible realizations of the occupancy vector Mb and assigning a probability to each configuration based on the product of Poisson probabilities in individual pixels The computational complexity scales as O (N nmax pix), where Npix is the number of pixels and nmax is the maximum photon count included
Appendix B Equivalence
Appendix B formally demonstrates the equivalence between constructing PH0(s) by enumerating occupancy vectors and viewing S as a compound Poisson random variable The conditional distribution P(m ntot = n) is identical to the distribution generated by assigning each of the n photons independently and uniformly to one of the Npix pixels This brings us back to the second representation described in §2.2, where S is viewed as a sum of i. p. 3i.d. contributions from the n photons
Conclusion
The method provides an accurate and efficient way to compute false-alarm probabilities for arbitrary linear filters in the low-count regime By recognizing S as a compound Poisson random variable, the full probability distribution is determined by two quantities, allowing for efficient computation using FFT or Panjer transform This solution enables reliable evaluation of extremely low false-alarm probabilities that are often impractical to estimate by simulation alone The Python and MATLAB implementations are provided to facilitate its adoption in a wide range of scientific applications
Code Availability
A Python implementation is released to the community alongside the publication through MTS’s Github page A MATLAB implementation is available as part of the AstroPack/MAATv2 package The functions poissThresholdSimulations, poissThresholdFFT, poissThresholdPanjer, and unitTest poissThreshold are available in the imUtil.poissNoise package
Tests
Figure 2 shows the probability distribution and false alarm probability as a function of S computed using four methods: sampling PH0 by producing 108 simulations, enumeration of occupancy vectors, FFT-based implementation, and Panjer recursion implementation The FFT and Panjer implementations are significantly faster than the other methods for a 5x5 kernel stamp The four methods of calculation lead to a value of Sth that differs by only 1%, but the FFT and Panjer methods take significantly less time to run
Consistency Checks
The method employs five consistency checks to verify the stability of the inferred detection threshold against refinements in binning and aliasing tolerance The rational of comparing both the threshold and the survival function is that it provides a stronger convergence test than comparing the threshold alone Adaptive refinement procedures are repeated until all five consistency checks are satisfied or a prescribed maximum number of refinement iterations is reached
Complexity Summary
The computational complexity for the FFT implementation is O (Npix + Nω log Nω), and for the Panjer implementation it is O(Npix + NgridNpix) The memory requirement for both implementations is O(Ngrid) The overall approach provides orders-of-magnitude improvements in computational speed compared to simulation methods
Catastrophic Cancellations
A practical limitation of the FFT approach is the loss of numerical accuracy in the extreme tail of the distribution due to catastrophic cancellation in the inverse Fourier transform The compound-Poisson characteristic function can be evaluated efficiently in Fourier space, after which PH0(s) is recovered through an inverse FFT The paper presents an alternative to this implementation which avoids this issue
Panjer Recursion Details
The Panjer recursion requires the discretized one-photon distribution pX[j] to have non-negative support, i.e.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Compound Poisson variable recognition allows AI systems to compute detection thresholds
several orders of magnitude faster than simulation-based approaches
by transforming computation from asimulation problem into a deterministic numerical calculation.
-
FFT implementation enables efficient evaluation of the compound Poisson distribution using
a single Fourier transform (Equation 22), a pointwise complex exponential (Equation 21), and a single inverse Fourier transform,
eliminating dependency over photon number n. -
Panjer recursion provides an alternative, computationally lighter method for computing probability distributions, with complexity bounded by O(Npix + NgridNpix).
-
The method is
independent of the choice of filter (it works for any linear detection statistic),
allowing AI to apply this framework broadly to any problem involving linear statistics and Poisson-distributed events. -
The system can reliably evaluate
extremely low false-alarm probabilities that are often impractical to estimate by simulation alone,
enabling the determination of thresholds corresponding to significance levels that would otherwise be computationally inaccessible.
Sources
- An Introduction to L'evy and Feller Processes. Advanced Courses in Mathematics - CRM Barcelona 2014
- X-Sifter: detecting transients in X-ray data using the optimal Poisson matched filter
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer