Intensity-Based Estimation of Extreme Value Models for Highly Censored Data via a Latent Poisson Process

arXiv:2611.08068 · stat.ME, stat.AP · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Intensity-Based Estimation of Extreme Value Models for Highly Censored Data via a Latent Poisson Process".

Jane: Here's the gist of the paper. Extreme value theory provides a natural framework for modelling rare events and extrapolating tail probabilities, but its direct application to survival analysis is complicated by right censoring, heterogeneous follow-up, and the need for coherent inference beyond the range of the observed data The proposed model integrates extreme value theory with a hazard-based formulation built upon the extended generalized Pareto distribution It is designed to address numerical difficulties arising under substantial censoring, with particular emphasis on estimating the tail index ξ without restricting it a priori to a specific tail regime The methodology provides reliable uncertainty quantification across a wide range of tail regimes, censoring proportions, and sample sizes Model Formulation and Estimation Procedure The proposed model is a Bayesian survival model that embeds the tail behaviour of the EGP distribution directly into the hazard function rather than explicitly separating threshold exceedances from the bulk of the observations It is formulated within a counting-process framework, which provides a natural representation of time-to-event data and connects hazard-based and intensitybased modelling The two-stage inference procedure first approximates the continuous hazard by a piecewise exponential model and subsequently treats the resulting interval-specific hazard estimates as noisy observations of the underlying GP or EGP hazard Latent evaluation locations are introduced to account for the fact that an interval-level hazard estimate does not correspond naturally to a unique deterministic time point Hazard Representation and Parameterization The extended generalized Pareto distribution is defined by a power transformation applied to the Generalized Pareto distribution, where κ is an additional shape parameter that controls the departure of the EGP distribution from the standard GP model The EGP hazard function is derived as hEGP(t; σ, ξ, κ) = κ

G(t; σ, ξ): κ−1 g(t; σ, ξ) 1 −

G(t; σ, ξ): κ The extreme value indexξ is parameterized using a transformation based on a centered Laplace prior on a latent parameter ζ ∼ Laplace(0, 1), defined as ξ = α (e ζ − 1) where α = σsup T This construction allows the extreme value index to take negative, zero, or positive values corresponding to bounded-tail, exponential-tail, and heavy-tailed behaviour without imposing a priori restrictions Bias Correction Mechanisms The piecewise exponential approximation introduces discretization bias when the underlying hazard varies appreciably within an interval Two related mechanisms are considered for mitigation: the first is a best representative approach, which involves selecting a deterministic location where the hazard matches the average hazard over the interval The second strategy is an adaptive partition refinement, where interval boundaries are chosen according to a preliminary estimate of the underlying distribution The main approach adopted is a latent location formulation, treating the representative location as an unobserved quantity with uncertainty incorporated directly into the hierarchical model Simulation Results and Robustness The first simulation study investigates the effect of the number of bins on posterior uncertainty quantification under correctly specified GP models, showing that a relatively small number of bins, particularly M = 5 or M = 10, generally provides reliable inferential performance The second simulation study evaluates robustness to model misspecification by generating survival and censoring times from a lognormal distribution, demonstrating that the proposed model can provide a close approximation to the underlying lognormal distribution despite the data-generating mechanism lying outside the assumed EGP family Application to Clinical Data The application to the heart failure dataset demonstrated how this framework can be used to investigate long-term survival under substantial right censoring, allowing differences between groups to be examined separately in terms of overall survival levels and asymptotic tail behaviour The posterior results provided evidence for a difference in the scale parameter between male and female patients, while the posterior credible interval for the tail effect included zero, suggesting limited evidence of a difference in the asymptotic tail behaviour between the two groups The proposed methodology provides a flexible and computationally tractable alternative for GP and EGP estimation under right censoring by replacing direct likelihood-based estimation with an intensity-based piecewise Poisson representation The framework facilitates Bayesian inference for the extreme value index without requiring a priori classification of the tail as short, exponential, or heavy Future research directions include a systematic investigation of the adaptive partitioning procedure and principled data-driven selection of the number of intervals The methodology provides a useful framework for estimating and extrapolating long-term survival probabilities when observations in the extreme tail are sparse or heavily censored Acknowledgments Eduardo Janotti is supported by the S˜ao Paulo Research Foundation (FAPESP), Brazil, under Grant Nos.

Tom: 2025/10342-3 and 2025/22925-3 L´ıgia Henriques-Rodrigues and Eduardo janotti were funded by national funds through FCT – Funda¸c˜ao para a Ciˆencia e a Tecnologia, I.P., under CEAUL Research Unit, UID/00006/2025, and by the European Union – NextGenerationEU through the project UID/PRR/00006/2025 The DOI is https://doi.org/1.

Title and authors: Tom: Okay, so the title itself tells us they're using an intensity-based approach with a latent Poisson process to estimate extreme value models for highly censored data. Jane, can you unpack what that means for someone who isn't deep in survival analysis?

Jane: Right. Think about it this way: extreme value theory is great for figuring out the tail of a distribution, those really rare outcomes we worry about. But when you have survival data—like how long patients live before an event—it gets messy because of right censoring. This paper proposes using a latent Poisson process to model that censoring and link it directly to the hazard function based on the extended generalized Pareto distribution.

Lu: It's about representing the whole hazard structure through this Poisson process, which avoids having to pick a specific threshold first, which is usually how you do it with the peaks-over-threshold approach <ref:2611.08068#pg1>.

Meng: So instead of just looking at chunks of data above some arbitrary cutoff point, they are trying to model the underlying event rate itself using a latent process. That sounds computationally heavy, though. How does that translate to something we can actually run on a computer?

Tom: It translates into a Bayesian survival model embedded in a counting-process framework, which gives them this natural way of looking at time-to-event data, connecting hazard and intensity modeling <ref:2611.08068#pg3>.

Jane: Exactly. It's about representing the entire hazard structure through this Poisson process, which avoids having to pick a specific threshold first, which is usually how you do it with the peaks-over-threshold approach <ref:2611.08068#pg2>.

The paper's summary: Tom: So, what’s the actual core idea they are pushing here? What’s this intensity-based estimation thing really about?

Jane: The summary says they propose a Bayesian survival model that integrates extreme value theory with a hazard-based formulation built upon the extended generalized Pareto distribution. The key is that instead of separating the bulk of the data from the tail using an explicit threshold, they represent the entire hazard structure through this framework <ref:2611.08068#pg1>.

Lu: They are building on top of a counting-process framework which connects hazard-based and intensity-based modeling, which is a pretty elegant way to handle the data structure <ref:2611.08068#pg3>.

Meng: If it's built on that, there has to be some kind of estimation procedure. They mention a two-stage inference procedure where they first approximate the continuous hazard with a piecewise exponential model and then treat those interval-specific hazard estimates as noisy observations of the underlying generalized pareto or extended generalized pareto hazard <ref:2611.08068#pg3>.

Tom: That two-stage process is interesting because it sounds like they’re trying to get the global tail structure from local, more manageable pieces. Jane, what does that tell us about their approach?

Jane: It shows they are trying to build stability into the estimation under substantial censoring, which is a big challenge in survival analysis <ref:2611.08068#pg3>. They introduce latent evaluation locations because an interval-level hazard estimate doesn't naturally correspond to a single, deterministic time point.

Lu: That latent location correction is important because it lets them treat those representative locations as unobserved quantities with uncertainty incorporated directly into the hierarchical model <ref:2611.08068#pg3>.

The paper's improvements: Tom: Let's talk about what they improved over previous methods. What are the specific mechanisms they introduced to make this approach better than just doing standard POT fitting?

Jane: They have these bias correction mechanisms addressing the discretization bias that comes from using piecewise exponential approximations when the underlying hazard changes a lot within an interval <ref:2611.08068#pg3>. They suggest two ways to fix it: first, selecting a deterministic location where the hazard matches the average hazard over that interval, and second, adaptive partition refinement based on preliminary estimates of the distribution itself.

Lu: I think the main approach they adopt is this latent location formulation, treating that representative location as an unobserved quantity with uncertainty built into their hierarchical model <ref:2611.08068#pg3>. That’s a clever way to propagate uncertainty through the entire estimation process.

Meng: From a production standpoint, I care about how robust this is. They mention simulation studies examining finite-sample performance, numerical stability, and robustness against model misspecification <ref:2611.08068#pg3>. That’s important because real traffic doesn't follow textbook distributions all the time.

Tom: And they did test robustness by generating survival and censoring times from a lognormal distribution, showing that even when the data-generating mechanism is outside their assumed extended generalized pareto family, the model still gives a close approximation <ref:2611.08068#pg3>.

Jane: It’s reassuring that they showed it works well across different scenarios, including examining long-term survival under substantial right censoring in their heart failure dataset <ref:2611.08068#pg3>.

Conclusion: Tom: So, to wrap up this discussion on the "Intensity-Based Estimation of Extreme Value Models for Highly Censored Data via a Latent Poisson Process," what’s the final word on why this matters for risk assessment?

Jane: Essentially, they give us a flexible and computationally tractable alternative to direct likelihood-based estimation when dealing with generalized pareto or extended generalized pareto distributions under right censoring <ref:2611.08068#pg1>. They’ve done this by replacing direct likelihood estimation with an intensity-based piecewise poisson representation <ref:2611.08068#pg3>.

Lu: The framework allows for Bayesian inference on the extreme value index, ξ, without needing to classify the tail as short, exponential, or heavy-tailed in advance <ref:2611.08068#pg2>. That’s a major flexibility gain.

Meng: It seems like a way to get reliable uncertainty quantification across different tail regimes and censoring proportions without having to pre-specify which one you are working with <ref:2611.08068#pg3>.

Tom: For someone listening who doesn't read papers, the implication is that we can estimate and extrapolate long-term survival probabilities even when the observations in the extreme tail are sparse or heavily censored <ref:2611.08068#pg3>.

Jane: That’s right. It gives us a useful framework for estimating and extrapolating long-term survival probabilities when observations in the extreme tail are sparse or heavily censored <ref:2611.08068#pg3>.

Lu: I think the systematic investigation of that adaptive partitioning procedure and principled data-driven selection of the number of intervals is a clear direction for future work <ref:2611.08068#pg3>.

Meng: And I’m interested in seeing how they handle scaling up these simulations to real traffic data, because right now, we’re mostly seeing performance on simulated data <ref:2611.08068#pg3>.

Tom: Well, that was the rundown on the paper "Intensity-Based Estimation of Extreme Value Models for Highly Censored Data via a Latent Poisson Process." We'll be looking at these results next time.

Eduardo Janotti, Antonio Carlos Pedroso de Limaa, Lígia Henriques-Rodrigues

Departamento de Estatística, Instituto de Matem­tica, Estatƚstica e Ci⎧ncia da Computa⎧ao, Universidade de Sao Paulo · CEAUL – Centro de Estat᦮stica e Aplica籆oes, Faculdade de Ciência, Universidade de Lisboa · Departamento de Ci⎧encias Matem­ticas, Faculdade de Ciência, Universidade de Lisboa

stat.ME, stat.AP

Submitted: 2026-11-05

Updated: 2026-11-05

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Extreme value theory provides a natural framework for modelling rare events and extrapolating tail probabilities…

Key concepts

Extended Generalized Pareto Distribution (EGP)
This is a specific distribution used to model the tails of survival data. It builds upon the standard Generalized Pareto distribution by adding an extra shape parameter, κ. This allows it to capture different types of heavy-tailed behavior, ranging from bounded-tail to very heavy tails.
Latent Poisson Process Representation
Instead of directly using complex continuous-time EGP likelihoods, the model uses a latent Poisson process. This simplifies the computational structure by representing time-to-event data in a way that connects hazard-based and intensity-based modeling, offering an alternative route for estimating tail parameters.
Extreme Value Index (ξ)
This parameter controls how quickly the survival probability decays in the extreme tail. The model estimates this index without needing to assume a specific tail regime beforehand. It is parameterized using a centered Laplace prior, allowing it to take on negative, zero, or positive values.
Latent Location Correction
When estimating hazards over time intervals, the exact point where the hazard is evaluated is unknown. This correction treats these evaluation points as unobserved quantities with a uniform prior. This allows uncertainty about these specific locations to be propagated into the final estimates for scale and tail parameters.

Terminology

Summary

Extreme value theory provides a natural framework for modelling rare events and extrapolating tail probabilities, but its direct application to survival analysis is complicated by right censoring, heterogeneous follow-up, and the need for coherent inference beyond the range of the observed data The proposed model integrates extreme value theory with a hazard-based formulation built upon the extended generalized Pareto distribution It is designed to address numerical difficulties arising under substantial censoring, with particular emphasis on estimating the tail index ξ without restricting it a priori to a specific tail regime The methodology provides reliable uncertainty quantification across a wide range of tail regimes, censoring proportions, and sample sizes

Model Formulation and Estimation Procedure

The proposed model is a Bayesian survival model that embeds the tail behaviour of the EGP distribution directly into the hazard function rather than explicitly separating threshold exceedances from the bulk of the observations It is formulated within a counting-process framework, which provides a natural representation of time-to-event data and connects hazard-based and intensitybased modelling The two-stage inference procedure first approximates the continuous hazard by a piecewise exponential model and subsequently treats the resulting interval-specific hazard estimates as noisy observations of the underlying GP or EGP hazard Latent evaluation locations are introduced to account for the fact that an interval-level hazard estimate does not correspond naturally to a unique deterministic time point

Hazard Representation and Parameterization

The extended generalized Pareto distribution is defined by a power transformation applied to the Generalized Pareto distribution, where κ is an additional shape parameter that controls the departure of the EGP distribution from the standard GP model The EGP hazard function is derived as hEGP(t; σ, ξ, κ) = κ [G(t; σ, ξ)]κ−1 g(t; σ, ξ) 1 − [G(t; σ, ξ)]κ The extreme value indexξ is parameterized using a transformation based on a centered Laplace prior on a latent parameter ζ ∼ Laplace(0, 1), defined as ξ = α (e ζ − 1) where α = σsup T This construction allows the extreme value index to take negative, zero, or positive values corresponding to bounded-tail, exponential-tail, and heavy-tailed behaviour without imposing a priori restrictions

Bias Correction Mechanisms

The piecewise exponential approximation introduces discretization bias when the underlying hazard varies appreciably within an interval Two related mechanisms are considered for mitigation: the first is a best representative approach, which involves selecting a deterministic location where the hazard matches the average hazard over the interval The second strategy is an adaptive partition refinement, where interval boundaries are chosen according to a preliminary estimate of the underlying distribution The main approach adopted is a latent location formulation, treating the representative location as an unobserved quantity with uncertainty incorporated directly into the hierarchical model

Simulation Results and Robustness

The first simulation study investigates the effect of the number of bins on posterior uncertainty quantification under correctly specified GP models, showing that a relatively small number of bins, particularly M = 5 or M = 10, generally provides reliable inferential performance The second simulation study evaluates robustness to model misspecification by generating survival and censoring times from a lognormal distribution, demonstrating that the proposed model can provide a close approximation to the underlying lognormal distribution despite the data-generating mechanism lying outside the assumed EGP family

Application to Clinical Data

The application to the heart failure dataset demonstrated how this framework can be used to investigate long-term survival under substantial right censoring, allowing differences between groups to be examined separately in terms of overall survival levels and asymptotic tail behaviour The posterior results provided evidence for a difference in the scale parameter between male and female patients, while the posterior credible interval for the tail effect included zero, suggesting limited evidence of a difference in the asymptotic tail behaviour between the two groups

Conclusion and Future Directions

The proposed methodology provides a flexible and computationally tractable alternative for GP and EGP estimation under right censoring by replacing direct likelihood-based estimation with an intensity-based piecewise Poisson representation The framework facilitates Bayesian inference for the extreme value index without requiring a priori classification of the tail as short, exponential, or heavy Future research directions include a systematic investigation of the adaptive partitioning procedure and principled data-driven selection of the number of intervals The methodology provides a useful framework for estimating and extrapolating long-term survival probabilities when observations in the extreme tail are sparse or heavily censored

Acknowledgments

Eduardo Janotti is supported by the S˜ao Paulo Research Foundation (FAPESP), Brazil, under Grant Nos. 2025/10342-3 and 2025/22925-3 L´ıgia Henriques-Rodrigues and Eduardo janotti were funded by national funds through FCT – Funda¸c˜ao para a Ciˆencia e a Tecnologia, I. p. 0

Where the paper says it

Intensity-Based Estimation of Extreme Value Models for Highly Censored Data via a Latent Poisson Process Eduardo Janottia,b, Antonio Carlos Pedroso de Limaa , L´ıgia Henriques-Rodriguesb,c aDepartamento de Estat´ıstica, Instituto de Matem´atica, Estat´ıstica e Ciˆencia da Computa¸c˜ao, Universidade de S˜ao Paulo, S˜ao Paulo, Brasil bCEAUL – Centro de Estat´ıstica e Aplica¸c˜oes, Faculdade de Ciˆencias, Universidade de Lisboa, Lisboa, Portugal cDepartamento de Ciˆencias Matem´aticas, Faculdade de Ciˆencias, Universidade de Lisboa, Lisboa, Portugal ARTICLE HISTORY Compiled November 6, 2026 ABSTRACT Extreme value theory provides a natural framework for modelling rare events and extrapolating tail probabilities, but its direct application to survival analysis is complicated by right censoring, heterogeneous follow-up, and the need for coherent inference beyond the range of the observed data. We propose a Bayesian survival model that integrates extreme value theory with a hazard-based formulation built upon the extended generalized Pareto distribution.

P., under CEAUL Research Unit, UID/00006/2025, and by the European Union – NextGenerationEU through the project UID/PRR/00006/2025 The DOI is https://doi.org/1.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Latent Poisson process representation allows for an alternative computational strategy for tail parameter estimation by avoiding direct evaluation of likelihood associated with continuous-time EGP models. This provides a simpler probabilistic structure and an alternative computational route to inference on the tail parameters.

  2. The two-stage procedure separates the estimation of local hazard information from the estimation of the global tail structure, which improves stability in highly censored settings through posterior samples that propagate first-stage uncertainty through tau squared j.

  3. Latent location correction treats representative locations within each interval as an unobserved quantity with a uniform prior, allowing uncertainty about evaluation points to be propagated into inferences on scale, tail, and EGP shape parameters.

Abstract

Extreme value theory provides a natural framework for modelling rare events and extrapolating tail probabilities, but its direct application to survival analysis is complicated by right censoring, heterogeneous follow-up, and the need for coherent inference beyond the range of the observed data. We propose a Bayesian survival model that integrates extreme value theory with a hazard-based formulation built upon the extended generalized Pareto distribution. Rather than separating the bulk and tail of the distribution through an explicit threshold, the proposed model represents the entire hazard function within a unified parametric framework, allowing moderate and extreme survival times to be analysed simultaneously while preserving the asymptotic tail behaviour implied by extreme value theory. Inference is performed through a two-stage procedure. We further develop bias-correction procedures based on latent representative locations and adaptive partition refinement, and establish theoretical results showing the asymptotic consistency of the proposed approximation under standard regularity conditions. Simulation studies demonstrate that the methodology provides reliable uncertainty quantification across a wide range of tail regimes, censoring proportions, and sample sizes, while remaining robust under moderate model misspecification. The practical performance of the approach is illustrated through an analysis of a heart failure cohort, where the proposed model enables inference on long-term survival probabilities and tail behaviour beyond the observed follow-up period. The resulting framework offers a computationally feasible and theoretically grounded alternative for modelling censored survival data when inference on extreme survival times is of primary scientific interest.

Sources

Related papers