Frequentist forecasting in regime-switching models with extended Hamilton filter

arXiv:2512.18149 · stat.ME · Submitted 2025-12-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Frequentist forecasting in regime-switching models with extended Hamilton filter".

Title and authors: Tom: So the paper's summary explains they are using these RSSS models to describe things like psychological change processes, specifically student dropout in math, which show those discrete latent state transitions and can be studied with intensive longitudinal data.

Jane: That's right. They mention that intensive longitudinal data, or ILD, can come from self-report or physiological stuff, and parts of that data can be time-invariant across individuals—that's the inter-individual part—and other parts show dynamic changes within an individual over time, which is the intra-individual aspect.

Lu: The paper points out that while dynamic structural equation modeling, or DSEM, has been studied a lot to investigate change processes in ILD, there's still a need for better estimation methods when you want to handle both the intra-individual changes and those inter-individual differences as predictors of regime switches.

Meng: So they aren't just looking at one level of difference; they are trying to model how those differences influence the switching process itself, which is more complex than standard models.

Lalam: I think that complexity is where the novelty lies, because existing methods haven't fully addressed estimating both levels of heterogeneity together in a frequentist way.

The paper's summary: Tom: The improvements they propose revolve around this new frequentist filter, which is built on the extended Kim filter, and it allows these hidden Markov switching models to depend on both latent within- and between-individual characteristics.

Jane: They specifically introduce a first frequentist approach within this general class of models by building on Chow and Zhang’s two thousand thirteen extended Kim filter <ref:2512.18149#pg1>. That's the specific tool they are proposing for parameter estimation, as summarized in Algorithm one <ref:2512.18149#pg1>.

Lu: The methodology involves calculating factor scores of inter-individual latent variables first, and then applying the extended Kim filter—which is a combination of the extended Kalman filter, the extended Hamilton filter, and a collapsing process—to actually perform the filtering steps.

Meng: From an engineering standpoint, that sounds computationally heavy. How does this collapse process handle scaling up when you have thousands of individuals in your dataset? What breaks at scale?

Lalam: They use distributional assumptions where prediction errors follow a multivariate normal distribution, which they then use as a basis for parameter updates using an approximated likelihood function, step five in their procedure.

The paper's improvements: Tom: So to wrap up, the paper presents this frequentist filtering method as a first approach for RSSS models that incorporates both intra- and inter-individual characteristics into the model structure.

Jane: Essentially, they show how to handle those hidden Markov switching models in a way that allows them to account for both latent within- and between-individual characteristics influencing the regime switches.

Lu: The flexibility of this RSSS modeling framework lets researchers include interactions between the intra-individual and inter-individual levels, which is often necessary when you need to validly represent those dynamic underlying processes.

Meng: So, if this works, it means we can actually model those complex interactions in dropout prediction, not just assume one level of variation is enough.

Lalam: And the simulation study results show that their approach can forecast regime membership with around eighty percent accuracy for both data sizes tested in the simulation.

Conclusion: Tom: So to summarize, Okuyama et al.'s paper introduces a first frequentist filter for RSSS models that allows them to depend on both latent within- and between-individual characteristics, building on the extended Kim filter.

Jane: They show how this method can be used for forecasting dynamics latent states and underlying regime memberships based on parameter values from time t to predict time t plus one.

Lu: The flexibility of the RSSS modeling framework allows researchers to include interactions between the intra-individual and inter-individual level, which is often necessary in order to validly represent the dynamic underlying processes.

Meng: It means this estimation procedure can accommodate a wide variety of dynamic models, which is good because it shows potential for real-time inference if you get the data flowing.

Lalam: Ultimately, the extended Kim filter serves as an effective tool for detecting unobserved change processes or regime switches in these complex settings.

University of Tubingen

stat.ME

Submitted: 2025-12-20

Updated: 2026-11-17

Comments: 34 pages, 4 figures, 13 tables. This article has been accepted for publication in a revised form in Psychometrika (https://www.cambridge.org/core/journals/psychometrika). v2: accepted manuscript after peer review; person-specific random intercepts are now marginalized within the filter, the simulation study was redone, and the use of the bwForCluster Helix is acknowledged

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: Frequentist forecasting in regime-switching models with extended Hamilton filter introduces a first frequentist filter for regimeswitching state-space (RSSS) models that allows hidden…

Key concepts

Regime-Switching State-Space (RSSS) Models
These models describe data where the underlying process changes abruptly between discrete hidden states or 'regimes' over time. They combine a measurement model with a structural model that accounts for both individual differences and temporal changes in behavior.
Extended Hamilton Filter
This is a specialized nonlinear filter used to infer the discrete latent states (regimes) within the RSSS framework. It works by utilizing maximum likelihood estimation to track these hidden states dynamically, making it suitable for real-time inference.
Intra- and Inter-individual Characteristics
The model incorporates two types of differences: intra-individual characteristics refer to the temporal progression of continuous variables within a single person, while inter-individual characteristics refer to differences between individuals. The proposed method explicitly models how these two levels interact in the switching process.

Terminology

Summary

Frequentist forecasting in regime-switching models with extended Hamilton filter introduces a first frequentist filter for regimeswitching state-space (RSSS) models that allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics The proposed method is the first frequentist approach within this general class of models, building on Chow and Zhang’s (2013) extended Kim filter. p. 1

Where the paper says it

Specifically, there’s a need for frequentist estimation methods in dynamic latent variable frameworks that allow real-time inferences and forecasts of latent or observed variables during ongoing data collection. Building on Chow and Zhang’s (2013) extended Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics.

Model Framework

Regime-Switching State-Space (RSSS) models are used to describe the underlying data generating process, where a change process is characterized by discontinuous transitions from one discrete latent state (i.e., regime) to another at a particular time point. p. 2

Where the paper says it

Many psychological change processes are characterized by regime-switching, in which a change process is described by discontinuous transitions from one discrete latent state (i. , regime) to another at a particular time point (see e.

The measurement model establishes the relationship between the latent variables and their observed proxies, defining dynamic observed variables conditional on individual i, time t, and a discrete latent state Sit. p. 4
Where the paper says it

Measurement model The measurement model is described by the following linear equations. Dynamic observed variables, which are expressed as (y1it|Sit = s), are conditional on individual i, time t, and a discrete latent state Sit.

The within-level structural model describes the temporal progression of continuous intra-individual latent variables through hidden Markov VAR(1) models that depend on both inter- and intra-individual variables. p. 4
Where the paper says it

The measurement model establishes the relationship between the latent variables and their observed proxies. The structural model depicts the temporal progression of continuous intra-individual latent variables through hidden Markov VAR(1) models, which depend on both inter- and intraindividual variables.

The transition probabilities are characterized using a nonlinear function of the latent variables of the between- and within-level structural models. p. 6
Where the paper says it

arXiv preprint December 23, 2025 article Page 7 Markov-switching model The transition probabilities are characterized using a nonlinear function of the latent variables of the between- and within-level structural models: P r[Sit = 1|Si,t−1 = s ′ ] = σ(γs ′1 + γs ′2η2i + γs ′3η1i,t−1 + γs ′4η1i,t−1η2i), (6) P r[Sit = 2|Si,t−1 = s ′ ] = 1 − P r[Sit = 1|Si,t−1 = s ′ ], (7) where s, s′ ∈ {1, 2} indicates the regime membership (e. , s = 1: no dropout intention; s = 2: dropout intention).

Estimation Procedure

The proposed parameter estimation procedure is summarized in Algorithm 1, which involves calculating factor scores of inter-individual latent variables (step 1). Then, the extended Kim filter, composed of the extended Kalman filter, the extended Hamilton filter, and the collapsing process is used to filter RSSS models (step 2-4). p. 6

Where the paper says it

Factor scores of inter-individual latent variables are calculated (step 1). Then, the extended Kim filter, which is composed of the extended Kalman filter, the extended Hamilton filter, and the collapsing process, is used to filter RSSS models (step 2-4).

Based on distributional assumptions, prediction errors follow a multivariate normal distribution serving as a basis for parameter updates using an approximated likelihood function (step 5). p. 6
Where the paper says it

Then, the extended Kim filter, which is composed of the extended Kalman filter, the extended Hamilton filter, and the collapsing process, is used to filter RSSS models (step 2-4). Based on the distributional assumptions, the prediction errors follow a multivariate normal distribution, serving as a basis for parameter updates using an approximated likelihood function1 (step 5).

The extended Kalman filter consists of three steps: prediction, measurement, and update. p. 7
Where the paper says it

The extended Kalman filter (step 2) The original Kalman filter (Kalman, 1960) was proposed as a real-time estimation procedure to estimate the continous latent variables in a state-space model as well as to predict their future values. The Kalman filter consists of three steps, namely, prediction, measurement, and update.

The extended Hamilton filter is specialized for the inference of discrete latent states and works as a nonlinear filter utilizing maximum likelihood estimation. p. 9
Where the paper says it

Unlike the Kalman filter, the algorithm is specialized for the inference of discrete latent states and works as a nonlinear filter. The filter also utilizes the maximum likelihood estimation which serves as a basis for parameter optimization.

Forecasting Implementation

The forecasting implementation involves using the extended Kim filter to forecast the dynamics latent states and underlying regime memberships for time t + 1 based on parameter values at time t. The collapsing process approximates joint estimates into marginal estimates, utilizing a weighting factor Ws,s'it which is referred to as the weighting factor. The prediction error decomposition function is computed using the prediction error decomposition function (Schweppe, 1965). p. 9

Where the paper says it

We call it the extended Hamilton filter and the recursive process can be described as P r[Sit = s, Si,t−1 = s ′ |D1:t−1] = P r[Sit = s|Si,t−1 = s ′ ]P r[Si,t−1 = s ′ |D1:t−1], (17) f(y1it|D1:t−1) = X s,s′ f(y1it|Sit = s, Si,t−1 = s ′ , D1:t−1)P r[Sit = s, Si,t−1 = s ′ |D1:t−1] (18) P r[Sit = s, Si,t−1 = s ′ |D1:t ] = f(y1it|Sit = s, Si,t−1 = s ′ , D1:t−1)P r[Sit = s, Si,t−1 = s ′ |D1:t−1] f(y1it|D1:t−1) (19) P r[Sit = s|D1:t ] = X s ′ P r[Sit = s, Si,t−1 = s ′ |D1:t ], (20) where the conditional multivariate normal likelihood function f(y1it|Sit = s, Si,t−1 = s ′ , D1:t−1) is computed as f(y1it|Sit = s, Si,t−1 = s ′ , D1:t−1) = (2π) − O1 2 |F s,s′ 1it | − 1 2 exp − 1 2 (v s,s′ it ) T [F s,s′ 1it ] −1v s,s′ it . (21) f(y1it|D1:t−1) in Equation (18) is called the prediction error decomposition function (Schweppe, 1965).

Parameter updates are based on maximizing the total loglikelihood function log f(Y1θ). p. 6
Where the paper says it

Then, the extended Kim filter, which is composed of the extended Kalman filter, the extended Hamilton filter, and the collapsing process, is used to filter RSSS models (step 2-4). Based on the distributional assumptions, the prediction errors follow a multivariate normal distribution, serving as a basis for parameter updates using an approximated likelihood function1 (step 5).

Empirical Validation

An empirical study applied this filter to forecast emotions and behavior related to student dropout in math. The study used intensive longitudinal data (ILD) from a student dropout study in math, involving 7 intra-individual latent factors and 1 inter-individual latent factor. p. 1

Where the paper says it

arXiv preprint December 23, 2025 article Page 2 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED KIM FILTER Abstract Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects.

Forecast results showed that the model predicted 76 individuals (63. p. 9
Where the paper says it

Similar to the Kalman filter, the Hamilton filter consists of a prediction and an update step. 1, the model probabilities are predicted using the transition probabilities.

9%) were predicted to have formed an intention to dropout at the middle time point (t = 25) and 95 individuals (79. p. 3
Where the paper says it

Kelava et al. (2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM).

8%) at the final time point (t = 50). p. 2
Where the paper says it

Many psychological change processes are characterized by regime-switching, in which a change process is described by discontinuous transitions from one discrete latent state (i. , regime) to another at a particular time point (see e.

The average time point to switch from s = 1 to s = 2 was at t = 13. p. 2
Where the paper says it

, Hamaker and Grasman, 2012; Asparouhov et al. Individuals are allowed to switch between regimes over time.

6 (SD = 11.8).

Simulation Study Results

A simulation study evaluated the performance of the proposed approach given complete datasets, where regime prediction accuracy was around 80% for both data sizes. The quadratic score function showed a consistently lower score function and a smaller standard deviation for larger sample size (D100) throughout the forecast interval. p. 6

Where the paper says it

The parameter estimates will become available at the end of Algorithm 1; the associated standard errors can be obtained by taking the square root of the diagonal elements of the negative numerical Hessian matrix or its approximation at the estimated point. The Confirmatory factor analysis (step 1) The inter-individual latent variables η2i are estimated from their observed proxies y2i using the factor score regression (cf.

Accuracy for the regime prediction was around 80% for both data sizes. p. 2
Where the paper says it

arXiv preprint December 23, 2025 article Page 3 Recently, the increasing availability of intensive longitudinal data (ILD) has been brought about by digital devices such as mobile phones and smartwatches. ILD typically consists of data from several subjects, with around thirty to a hundred repeated measures per subject.

The dynamic latent variables showed a consistently lower score function and a smaller standard deviation compared to a small sample size (D75) throughout the forecast interval. p. 6
Where the paper says it

The parameter estimates will become available at the end of Algorithm 1; the associated standard errors can be obtained by taking the square root of the diagonal elements of the negative numerical Hessian matrix or its approximation at the estimated point. The Confirmatory factor analysis (step 1) The inter-individual latent variables η2i are estimated from their observed proxies y2i using the factor score regression (cf.

Discussion and Extensions

The flexibility of the RSSS modeling framework allows researchers to include interactions between the intra-individual and inter-individual level. The described model can be readily extended to accommodate three or more regimes, or a more flexible state-space representation. p. 3

Where the paper says it

Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. An estimation procedure called the Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended Kim smoother.

Another interesting extension is incorporating higher-order autoregressive terms and their interactions, or using polynomial functions. The effectiveness is particularly utilized when such changes manifest through variations at the within- and between-levels in each individual. p. 2
Where the paper says it

, dispositions, personal traits). Other parts of the data consist of repeated measures whose states are subject to dynamic changes across time within individuals (intra-individual; e.

Open Questions

This article did not evaluate the performance under different degree of missingness, as the empirical and simulation study presented two extremes. The classification result is dependent on the proportion of clusters, and performance might be sensitive to the similarities of regime distributions. p. 4

Where the paper says it

In an empirical example, we illustrate the approach and its ability to forecast regime membership. Then, we provide further evidence of its performance using a simulation study.

Direct comparison with Bayesian counterpart would be an interesting future direction, as all Bayesian priors need to be substituted by uninformative ones. p. 1
Where the paper says it

As a counterpart of Kelava et al. ’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models.

Model estimation using a misspecified model is known to harm the prediction in SEM. p. 3
Where the paper says it

Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. An estimation procedure called the Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended Kim smoother.

p. 3
Where the paper says it

Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. An estimation procedure called the Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended Kim smoother.

Data Availability

The associated replication files for the simulation study, including all analysis code, are available online at https://github.com/PsychometricsMZ/Regime-switching. p. 0

Where the paper says it

arXiv preprint December 23, 2025 article Page 1 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED HAMILTON FILTER Kento Okuyama university of tubingen ¨ Tim Fabian Schaffland university of tubingen, d-fine gmbh ¨ Pascal Kilian university of tubingen, robert bosch gmbh ¨ Holger Brandt university of tubingen ¨ Augustin Kelava university of tubingen ¨ Correspondence should be sent to Kento Okuyama, Methods Center, University of T¨ubingen, 72076, Germany. E-Mail: kento.

The empirical data used for the application study cannot be made publicly available due to the terms of the data disclosure agreement and privacy concerns.

References

Anja Achtziger and Peter M. Gollwitzer. p. 2

Where the paper says it

, from an intention to stay at math lecture to an intention to quit (cf. , Rubicon model; Achtziger and Gollwitzer, 2018).

Motivation and Volition in the Course of Action, pages 485–527. p. 4
Where the paper says it

arXiv preprint December 23, 2025 article Page 5 fore, the lack of theoretical and empirical anticipation of effects (priors/beliefs) could be a motivation to adapt a frequentist approach. In this paper, we present a frequentist estimation approach for RSSS models which considers both intra- and inter-individual differences in hidden Markov(- switching) models.

Springer International Publishing, Cham, 2018. p. 2
Where the paper says it

Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD.

p. 2
Where the paper says it

Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD.

Tihomir Asparouhov, Ellen L. p. 2
Where the paper says it

Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD.

Hamaker, and Bengt Muth´en. p. 2
Where the paper says it

Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD.

Dynamic latent class analysis. p. 1
Where the paper says it

As a counterpart of Kelava et al. ’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models.

Structural Equation Modeling: A Multidisciplinary Journal, 24(2):257– 269, 2017. p. 3
Where the paper says it

Kelava et al. (2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM).

M. S. Bartlett. The statistical conception of mental factors. p. 3
Where the paper says it

(2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM). Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model.

British Journal of Psychology. General Section, 28(1):97–104, 1937. p. 1
Where the paper says it

As a counterpart of Kelava et al. ’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models.

p. 1
Where the paper says it

As a counterpart of Kelava et al. ’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models.

G.E.P. Box, G.M. Jenkins, G.C. Reinsel, and G.M. Ljung. Time Series Analysis: Forecasting and Control. p. 2
Where the paper says it

Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD.

Michael W Browne and Robert A Cudeck Alternative ways of assessing model fit. p. 0
Where the paper says it

arXiv preprint December 23, 2025 article Page 1 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED HAMILTON FILTER Kento Okuyama university of tubingen ¨ Tim Fabian Schaffland university of tubingen, d-fine gmbh ¨ Pascal Kilian university of tubingen, robert bosch gmbh ¨ Holger Brandt university of tubingen ¨ Augustin Kelava university of tubingen ¨ Correspondence should be sent to Kento Okuyama, Methods Center, University of T¨ubingen, 72076, Germany. E-Mail: kento.

R S Bucy and P D Joseph Filtering for Stochastic Processes with Applications to Guidance. p. 8
Where the paper says it

Furthermore, K s,s′ it,aug = P s,s′ aug,it|t−1ΛT 1s,aug[F s,s′ 1it ] −1 is called the Kalman gain function. Equation (15) is often substituted by what’s called the Joseph form (Bucy and Joseph, 2005) because subtracting two covariance matrices may result in a numerical error causing one of the diagonal elements to be slightly less than zero.

Sy Miin Chow and Guangjian Zhang Nonlinear regime-switching state-space (rsss) models. p. 2
Where the paper says it

Kim (1994) generalized this approach to state-space models, which are called regime-switching state-space (RSSS) models. This framework accommodated a wide range of models, including latent variable models and regime-dependent time-series models (Yang and Chow, 2010).

Psychometrika, 78(4):740–768, October 2013. p. 1
Where the paper says it

Specifically, there’s a need for frequentist estimation methods in dynamic latent variable frameworks that allow real-time inferences and forecasts of latent or observed variables during ongoing data collection. Building on Chow and Zhang’s (2013) extended Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics.

Mark de Rooij, Julian D Karch Marjolein Fokkema Zsuzsa Bakk Bunga Citra Pratiwi and Henk Kelderman Sem-based out-of-sample predictions. Structural Equation Modeling: A Multidisciplinary Journal, 30(1):132–148, 2023. p. 3
Where the paper says it

Kelava et al. (2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM).

Tilmann Gneiting Making and evaluating point forecasts. p. 6
Where the paper says it

Given the set of parameter values at time t, the extended Kim filter forecasts the dynamics latent states and the underlying regime memberships for time t + 1. The parameter estimates will become available at the end of Algorithm 1; the associated standard errors can be obtained by taking the square root of the diagonal elements of the negative numerical Hessian matrix or its approximation at the estimated point.

p. 6
Where the paper says it

Given the set of parameter values at time t, the extended Kim filter forecasts the dynamics latent states and the underlying regime memberships for time t + 1. The parameter estimates will become available at the end of Algorithm 1; the associated standard errors can be obtained by taking the square root of the diagonal elements of the negative numerical Hessian matrix or its approximation at the estimated point.

Journal of the American Statistical Association, 106(494):746–762, 2011. E L Hamaker R P P P Grasman Regime switching state-space model applied to psychological processes: Handling missing data and making inferences. p. 1
Where the paper says it

arXiv preprint December 23, 2025 article Page 2 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED KIM FILTER Abstract Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects.

p. 1
Where the paper says it

arXiv preprint December 23, 2025 article Page 2 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED KIM FILTER Abstract Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects.

Psychometrika, 77:400–422, 2012. p. 2
Where the paper says it

, regime) to another at a particular time point (see e. , Hamaker and Grasman, 2012; Asparouhov et al.

p. 2
Where the paper says it

, regime) to another at a particular time point (see e. , Hamaker and Grasman, 2012; Asparouhov et al.

James D Hamilton A new approach to the economic analysis of nonstationary time series and the business cycle. p. 2
Where the paper says it

Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. With increasing number of substantive applications of DSEMs, a number of new ILD methods have been developed (e.

p. 2
Where the paper says it

Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. With increasing number of substantive applications of DSEMs, a number of new ILD methods have been developed (e.

Econometrica, 57(2):357–384, 1989. p. 2
Where the paper says it

, mean, variance, autocorrelation). Since Markov-switching autoregressive (MSAR) models (Hamilton, 1989) were introduced, regime-switching models have mostly gained the interest of econometricians.

p. 2
Where the paper says it

, mean, variance, autocorrelation). Since Markov-switching autoregressive (MSAR) models (Hamilton, 1989) were introduced, regime-switching models have mostly gained the interest of econometricians.

A C Harvey Forecasting Structural Time Series Models and the Kalman Filter. p. 1
Where the paper says it

As a counterpart of Kelava et al. ’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models.

p. 1
Where the paper says it

As a counterpart of Kelava et al. ’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models.

Cambridge University Press, 1990. p. 0
Where the paper says it

arXiv preprint December 23, 2025 article Page 1 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED HAMILTON FILTER Kento Okuyama university of tubingen ¨ Tim Fabian Schaffland university of tubingen, d-fine gmbh ¨ Pascal Kilian university of tubingen, robert bosch gmbh ¨ Holger Brandt university of tubingen ¨ Augustin Kelava university of tubingen ¨ Correspondence should be sent to Kento Okuyama, Methods Center, University of T¨ubingen, 72076, Germany. E-Mail: kento.

p. 0
Where the paper says it

arXiv preprint December 23, 2025 article Page 1 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED HAMILTON FILTER Kento Okuyama university of tubingen ¨ Tim Fabian Schaffland university of tubingen, d-fine gmbh ¨ Pascal Kilian university of tubingen, robert bosch gmbh ¨ Holger Brandt university of tubingen ¨ Augustin Kelava university of tubingen ¨ Correspondence should be sent to Kento Okuyama, Methods Center, University of T¨ubingen, 72076, Germany. E-Mail: kento.

Linying Ji Sy-Miin Chow Brian Crosby and Douglas M Teti Exploring sleep dynamic of mother-infant dyads using a regime-switching vector autoregressive model. p. 2
Where the paper says it

In other cases, subjects could repeatedly switch back and forth between two or more states such as the sleep-wake cycle (Ji et al. Studying such processes on a within-individual level requires a regime-switching model that assumes that each observation comes from one of the distinct distributions, each of which has a unique set of parameters (e.

p. 2
Where the paper says it

In other cases, subjects could repeatedly switch back and forth between two or more states such as the sleep-wake cycle (Ji et al. Studying such processes on a within-individual level requires a regime-switching model that assumes that each observation comes from one of the distinct distributions, each of which has a unique set of parameters (e.

Multivariate Behavioral Research, 55(1):150–151, 2020. p. 2
Where the paper says it

ILD typically consists of data from several subjects, with around thirty to a hundred repeated measures per subject. The data could take various forms, such as self-report, physiological data, and behavioral, environmental, and geographic data (Trull and Ebner-Priemer, 2014).

p. 2
Where the paper says it

ILD typically consists of data from several subjects, with around thirty to a hundred repeated measures per subject. The data could take various forms, such as self-report, physiological data, and behavioral, environmental, and geographic data (Trull and Ebner-Priemer, 2014).

R E Kalman A New Approach to Linear Filtering and Prediction Problems. p. 3
Where the paper says it

An estimation procedure called the Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended Kim smoother. Accounting for unobserved heterogeneity in the hidden Markov(- switching) models characterizes the novelty of this approach.

p. 3
Where the paper says it

An estimation procedure called the Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended Kim smoother. Accounting for unobserved heterogeneity in the hidden Markov(- switching) models characterizes the novelty of this approach.

Appendix

Augmented State-Space Formulation combines the U1 × 1 time-dependent state η1it and the U1 × 1 time-invariant random intercept residual ζ2i into a single Uaug × 1 state vector where Uaug = 2U1. The measurement model is expressed using this augmented state, defining the augmented factor loading matrix Λ1s,aug as hΛ1s 0 i. p. 5

Where the paper says it

arXiv preprint December 23, 2025 article Page 6 dependent latent variables. Λ1s is a factor loading matrix which is dependent on dynamic latent state Sit, with a dimension O1 × U1.

p. 5
Where the paper says it

arXiv preprint December 23, 2025 article Page 6 dependent latent variables. Λ1s is a factor loading matrix which is dependent on dynamic latent state Sit, with a dimension O1 × U1.

The structural models are combined to form the state transition equation, which directly corresponds to the augmented Kalman filter’s state equation. p. 3
Where the paper says it

(2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM). Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model.

p. 3
Where the paper says it

(2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM). Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model.

Table Summaries

Table 3 summarizes parameter estimates for the Markov-switching model, showing mean and standard errors for parameters like γ1, γ2, γ3, etc. Table 4 presents parameter estimates for the structural parameters (s = 1), including baseline effects and autoregressive coefficients. p. 7

Where the paper says it

, T do 2: Extended Kalman filter 3: Extended Hamilton filter 4: Collapsing process end for 5: Parameter update end while where F2 is the matrix of factor score weights. This matrix is computed based on the current estimates of the measurement model parameters: the factor loading matrix Λ2 and the residual covariance matrix R2.

Table 5 shows parameter estimates for the structural parameters (s = 2), detailing baseline effects and autoregressive coefficients. p. 7
Where the paper says it

, T do 2: Extended Kalman filter 3: Extended Hamilton filter 4: Collapsing process end for 5: Parameter update end while where F2 is the matrix of factor score weights. This matrix is computed based on the current estimates of the measurement model parameters: the factor loading matrix Λ2 and the residual covariance matrix R2.

Table 6 summarizes parameter estimates for the residual variances and P2 = var(η2i) of the structural model. p. 7
Where the paper says it

, T do 2: Extended Kalman filter 3: Extended Hamilton filter 4: Collapsing process end for 5: Parameter update end while where F2 is the matrix of factor score weights. This matrix is computed based on the current estimates of the measurement model parameters: the factor loading matrix Λ2 and the residual covariance matrix R2.

Table 7 presents parameter estimates for the factor loadings Λ1,free. p. 6
Where the paper says it

The parameter estimates will become available at the end of Algorithm 1; the associated standard errors can be obtained by taking the square root of the diagonal elements of the negative numerical Hessian matrix or its approximation at the estimated point. The Confirmatory factor analysis (step 1) The inter-individual latent variables η2i are estimated from their observed proxies y2i using the factor score regression (cf.

Table 8 summarizes parameter estimates for the residual variances of the measurement model, including R2. p. 7
Where the paper says it

, T do 2: Extended Kalman filter 3: Extended Hamilton filter 4: Collapsing process end for 5: Parameter update end while where F2 is the matrix of factor score weights. This matrix is computed based on the current estimates of the measurement model parameters: the factor loading matrix Λ2 and the residual covariance matrix R2.

Table 9 provides summary statistics for Markov-switching model parameter estimates across D75 and D100. p. 7
Where the paper says it

, T do 2: Extended Kalman filter 3: Extended Hamilton filter 4: Collapsing process end for 5: Parameter update end while where F2 is the matrix of factor score weights. This matrix is computed based on the current estimates of the measurement model parameters: the factor loading matrix Λ2 and the residual covariance matrix R2.

Table 10 summarizes structural model parameter estimates, showing mean and standard errors for b values and B matrices. p. 6
Where the paper says it

Given the set of parameter values at time t, the extended Kim filter forecasts the dynamics latent states and the underlying regime memberships for time t + 1. The parameter estimates will become available at the end of Algorithm 1; the associated standard errors can be obtained by taking the square root of the diagonal elements of the negative numerical Hessian matrix or its approximation at the estimated point.

Table 11 summarizes measurement model parameter estimates, including R values for R2. p. 7
Where the paper says it

, T do 2: Extended Kalman filter 3: Extended Hamilton filter 4: Collapsing process end for 5: Parameter update end while where F2 is the matrix of factor score weights. This matrix is computed based on the current estimates of the measurement model parameters: the factor loading matrix Λ2 and the residual covariance matrix R2.

Forecast Results

Parameter estimates for the Markov-switching model show that the main between effect (γ2) was positive and the main intra-individual effect (γ3) was negative for 6 out of 7 parameters. For structural model parameters, the autoregressive coefficients (diagonal elements of B3s) were positive and large for all within-scales. p. 5

Where the paper says it

b1is is a U1×1 conditional random intercept vector, B3is is a U1 × U1 matrix of conditional intra-individual VAR(1) effects, and Q1s is a U1 × U1 residual covariance matrix which is dependent on a dynamic latent state Sit. Between-level structural model For the random intercept vector b1is and the autoregressive coefficient matrix B3is, the between-level structural model is specified by two models: b1is = b1s + b2sη2i + ζ2i , (4) B3is = B3s + B4sη2i , (5) where b1s is a U1 × 1 conditional intercept vector, b2s is a U1 × 1 vector of conditional inter-individual effects, ζ2i is a U1 × 1 vector of random intercepts with ζ2i ∼ N (0, Q2).

The regime probabilities showed that the model predicted 76 individuals (63. p. 9
Where the paper says it

Similar to the Kalman filter, the Hamilton filter consists of a prediction and an update step. 1, the model probabilities are predicted using the transition probabilities.

9%) were predicted to have formed an intention to dropout at the middle time point (t = 25). p. 3
Where the paper says it

Kelava et al. (2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM).

Discussion

The flexibility of RSSS modeling framework allows researchers to include interactions between the intra-individual and inter-individual level, which is often necessary in order to validly represent the dynamic underlying processes. The described model and estimation procedures can accommodate a wide variety of dynamic models. p. 4

Where the paper says it

Measurement model The measurement model is described by the following linear equations. Dynamic observed variables, which are expressed as (y1it|Sit = s), are conditional on individual i, time t, and a discrete latent state Sit.

The extended Kim filter serves as an effective tool for detecting unobserved change processes or regime-switches. p. 1
Where the paper says it

arXiv preprint December 23, 2025 article Page 2 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED KIM FILTER Abstract Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects.

The classification result is dependent on the proportion of clusters, and performance might be sensitive to the similarities of regime distributions. Direct comparison with Bayesian counterpart would be an interesting future direction. p. 1

Where the paper says it

As a counterpart of Kelava et al. ’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models.

Model estimation using a misspecified model is known to harm the prediction in SEM. The average sensitivity was much greater than the specificity, which is the direct result of a high percentage of 2nd regime proportions i.e., highly unbalanced data. p. 1
Where the paper says it

arXiv preprint December 23, 2025 article Page 2 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED KIM FILTER Abstract Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects.

Anja Achtziger and Peter M Gollwitzer Motivation and Volition in the Course of Action, pages 485–527. Springer International Publishing, Cham, 2018. Tihomir Asparouhov Ellen L Hamaker and Bengt Muth´en Dynamic latent class analysis Structural Equation Modeling: A Multidisciplinary Journal, 24(2):257– 269, 2017. p. 2

Where the paper says it

Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD.

M S Bartlett The statistical conception of mental factors British Journal of Psychology. p. 7
Where the paper says it

This matrix is computed based on the current estimates of the measurement model parameters: the factor loading matrix Λ2 and the residual covariance matrix R2. In the case of Bartlett factor scores (Bartlett, 1937), the formula for F2 is: F2 = Λ T 2 R−1 2 Λ2 −1 Λ T 2 R−1 2 .

General Section, 28(1):97–104, 1937. G E P Box G M Jenkins G C Reinsel and G M Ljung Time Series Analysis: Forecasting and Control Wiley Series in Probability and Statistics. p. 2
Where the paper says it

Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018), which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD.

Wiley, 2015. p. 3
Where the paper says it

Filtering (Box et al. , 2015) is an important technique as it allows forecasting i.

Michael W Browne and Robert A Cudeck Alternative ways of assessing model fit Sociological Methods & Research, 21:230 – 258, 1992. p. 0
Where the paper says it

arXiv preprint December 23, 2025 article Page 1 FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH EXTENDED HAMILTON FILTER Kento Okuyama university of tubingen ¨ Tim Fabian Schaffland university of tubingen, d-fine gmbh ¨ Pascal Kilian university of tubingen, robert bosch gmbh ¨ Holger Brandt university of tubingen ¨ Augustin Kelava university of tubingen ¨ Correspondence should be sent to Kento Okuyama, Methods Center, University of T¨ubingen, 72076, Germany. E-Mail: kento.

R S Bucy and P D Joseph Filtering for Stochastic Processes with Applications to Guidance AMS Chelsea Publishing Series. p. 8
Where the paper says it

Furthermore, K s,s′ it,aug = P s,s′ aug,it|t−1ΛT 1s,aug[F s,s′ 1it ] −1 is called the Kalman gain function. Equation (15) is often substituted by what’s called the Joseph form (Bucy and Joseph, 2005) because subtracting two covariance matrices may result in a numerical error causing one of the diagonal elements to be slightly less than zero.

AMS Chelsea Pub., 2005. Sy Miin Chow and Guangjian Zhang Nonlinear regime-switching state-space (rsss) models Psychometrika, 78(4):740–768, October 2013. p. 2
Where the paper says it

This framework accommodated a wide range of models, including latent variable models and regime-dependent time-series models (Yang and Chow, 2010). Chow and Zhang (2013) then developed nonlinear RSSS models, whose statespace representation is governed by nonlinear dynamics.

Mark de Rooij Julian D Karch Marjolein Fokkema Zsuzsa Bakk Bunga Citra Pratiwi and Henk Kelderman Sem-based out-of-sample predictions Structural Equation Modeling: A Multidisciplinary Journal, 30(1):132–148, 2023. p. 3
Where the paper says it

Kelava et al. (2022) addressed forecasting the precursor of student dropout, referred to as the intention to dropout, within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM).

Tilmann Gneiting Making and evaluating point forecasts. Journal of the American Statistical Association, 106(494):746–762, 2011. E L Hamaker R P P P Grasman Regime switching state-space model applied to psychological processes: Handling missing data and making inferences. Psychometrika, 77:400–422, 2012. James D Hamilton A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57(2):357–384, 1989. A C Harvey Forecasting Structural Time Series Models and the Kalman Filter. Cambridge University Press, 1990. Linying Ji Sy-Miin Chow Brian Crosby and Douglas M Teti Exploring sleep dynamic of mother-infant dyads using a regime-switching vector autoregressive model. Multivariate Behavioral Research, 55(1):150–151, 2020. R E Kalman A New Approach to Linear Filtering and Prediction Problems.

Augmented State-Space Formulation defines the augmented state vector as ηaug,it = η1it ζ2i i=t. The measurement model is expressed using this augmented state, defining the augmented factor loading matrix Λ1s,aug as hΛ1s 0 i. The structural models are combined to form the state transition equation, which directly corresponds to the augmented Kalman filter’s state equation.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Frequentist estimation for regime-switching state-space models can be implemented to allow real-time inferences and forecasts of latent or observed variables during ongoing data collection. This enables dynamic modeling that is suitable for applications involving real-time monitoring and timely intervention.

  2. The proposed method allows the model to consider both intra- and inter-individual differences in hidden Markov switching models, accommodating unobserved heterogeneity in the hidden Markov(-switching) models which is a novelty compared to existing frequentist approaches.

  3. The system can forecast regime membership for student dropout with an accuracy of around 80% across simulation studies, suggesting it provides early signals of dropout intentions when interventions should be conducted.

  4. Missing values in intensive longitudinal data can be handled by substituting the likelihood calculation and update steps with estimations from the prediction step, overcoming issues where missing values prevent obtaining the likelihood function and updated estimates.

Abstract

Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regime-switching models with intensive longitudinal data (ILD). Recently, regime-switching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects. Despite this progress, there is a need for frequentist estimation methods in dynamic latent variable frameworks that handle both intra-individual changes and inter-individual differences as predictors of regime-switches and allow real-time inferences and forecasts during ongoing data collection. Building on Chow and Zhang's (2013) extended Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics. As a counterpart of Kelava et al.'s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our method is the first frequentist approach within this general class of models. In an empirical study, the filter is applied to forecast emotions and behavior related to student dropout in math. A simulation study shows that marginalizing person-specific random intercepts within the filter cleanly separates autoregressive persistence from stable between-person differences, with accurate out-of-sample regime forecasts, while within-person transition effects stay weakly identified.

Related papers