Frequentist forecasting in regime-switching models with extended Hamilton filter
summary
The gist
Frequentist forecasting in regime-switching models with extended Hamilton filter introduces a first frequentist filter for regimeswitching state-space (RSSS) models that allows hidden…
In short
This work introduces a first frequentist method for forecasting in Regime-Switching State-Space (RSSS) models using an extended Hamilton filter. It allows hidden Markov models to account for both within-individual and between-individual characteristics, offering a novel approach to dynamic modeling. The method is designed for real-time inference and forecasting during data collection.
Key concepts
- Regime-Switching State-Space (RSSS) Models
- These models describe data where the underlying process changes abruptly between discrete hidden states or 'regimes' over time. They combine a measurement model with a structural model that accounts for both individual differences and temporal changes in behavior.
- Extended Hamilton Filter
- This is a specialized nonlinear filter used to infer the discrete latent states (regimes) within the RSSS framework. It works by utilizing maximum likelihood estimation to track these hidden states dynamically, making it suitable for real-time inference.
- Intra- and Inter-individual Characteristics
- The model incorporates two types of differences: intra-individual characteristics refer to the temporal progression of continuous variables within a single person, while inter-individual characteristics refer to differences between individuals. The proposed method explicitly models how these two levels interact in the switching process.
Terminology used across episodes
This episode discusses
The paper
Frequentist forecasting in regime-switching models with extended Hamilton filter · Read on arXiv
University of Tubingen
Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regime-switching models with intensive longitudinal data (ILD). Recently, regime-switching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects. Despite this progress, there is a need for frequentist estimation methods in dynamic latent variable frameworks that handle both intra-individual changes and inter-individual differences as predictors of regime-switches and allow real-time inferences and forecasts during ongoing data collection. Building on Chow and Zhang's (2013) extended Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics. As a counterpart of Kelava et al.'s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our method is the first frequentist approach within this general class of models. In an empirical study, the filter is applied to forecast emotions and behavior related to student dropout in math. A simulation study shows that marginalizing person-specific random intercepts within the filter cleanly separates autoregressive persistence from stable between-person differences, with accurate out-of-sample regime forecasts, while within-person transition effects stay weakly identified.
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Frequentist forecasting in regime-switching models with extended Hamilton filter".
Title and authors: Tom: So the paper's summary explains they are using these RSSS models to describe things like psychological change processes, specifically student dropout in math, which show those discrete latent state transitions and can be studied with intensive longitudinal data.
Jane: That's right. They mention that intensive longitudinal data, or ILD, can come from self-report or physiological stuff, and parts of that data can be time-invariant across individuals—that's the inter-individual part—and other parts show dynamic changes within an individual over time, which is the intra-individual aspect.
Lu: The paper points out that while dynamic structural equation modeling, or DSEM, has been studied a lot to investigate change processes in ILD, there's still a need for better estimation methods when you want to handle both the intra-individual changes and those inter-individual differences as predictors of regime switches.
Meng: So they aren't just looking at one level of difference; they are trying to model how those differences influence the switching process itself, which is more complex than standard models.
Lalam: I think that complexity is where the novelty lies, because existing methods haven't fully addressed estimating both levels of heterogeneity together in a frequentist way.
The paper's summary: Tom: The improvements they propose revolve around this new frequentist filter, which is built on the extended Kim filter, and it allows these hidden Markov switching models to depend on both latent within- and between-individual characteristics.
Jane: They specifically introduce a first frequentist approach within this general class of models by building on Chow and Zhang’s two thousand thirteen extended Kim filter <ref:2512.18149#pg1>. That's the specific tool they are proposing for parameter estimation, as summarized in Algorithm one <ref:2512.18149#pg1>.
Lu: The methodology involves calculating factor scores of inter-individual latent variables first, and then applying the extended Kim filter—which is a combination of the extended Kalman filter, the extended Hamilton filter, and a collapsing process—to actually perform the filtering steps.
Meng: From an engineering standpoint, that sounds computationally heavy. How does this collapse process handle scaling up when you have thousands of individuals in your dataset? What breaks at scale?
Lalam: They use distributional assumptions where prediction errors follow a multivariate normal distribution, which they then use as a basis for parameter updates using an approximated likelihood function, step five in their procedure.
The paper's improvements: Tom: So to wrap up, the paper presents this frequentist filtering method as a first approach for RSSS models that incorporates both intra- and inter-individual characteristics into the model structure.
Jane: Essentially, they show how to handle those hidden Markov switching models in a way that allows them to account for both latent within- and between-individual characteristics influencing the regime switches.
Lu: The flexibility of this RSSS modeling framework lets researchers include interactions between the intra-individual and inter-individual levels, which is often necessary when you need to validly represent those dynamic underlying processes.
Meng: So, if this works, it means we can actually model those complex interactions in dropout prediction, not just assume one level of variation is enough.
Lalam: And the simulation study results show that their approach can forecast regime membership with around eighty percent accuracy for both data sizes tested in the simulation.
Conclusion: Tom: So to summarize, Okuyama et al.'s paper introduces a first frequentist filter for RSSS models that allows them to depend on both latent within- and between-individual characteristics, building on the extended Kim filter.
Jane: They show how this method can be used for forecasting dynamics latent states and underlying regime memberships based on parameter values from time t to predict time t plus one.
Lu: The flexibility of the RSSS modeling framework allows researchers to include interactions between the intra-individual and inter-individual level, which is often necessary in order to validly represent the dynamic underlying processes.
Meng: It means this estimation procedure can accommodate a wide variety of dynamic models, which is good because it shows potential for real-time inference if you get the data flowing.
Lalam: Ultimately, the extended Kim filter serves as an effective tool for detecting unobserved change processes or regime switches in these complex settings.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates