Frequentist forecasting in regime-switching models with extended Hamilton filter
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Frequentist forecasting in regime-switching models with extended Hamilton filter".
Title and authors: Tom: So the paper's summary explains they are using these RSSS models to describe things like psychological change processes, specifically student dropout in math, which show those discrete latent state transitions and can be studied with intensive longitudinal data.
Jane: That's right. They mention that intensive longitudinal data, or ILD, can come from self-report or physiological stuff, and parts of that data can be time-invariant across individuals—that's the inter-individual part—and other parts show dynamic changes within an individual over time, which is the intra-individual aspect.
Lu: The paper points out that while dynamic structural equation modeling, or DSEM, has been studied a lot to investigate change processes in ILD, there's still a need for better estimation methods when you want to handle both the intra-individual changes and those inter-individual differences as predictors of regime switches.
Meng: So they aren't just looking at one level of difference; they are trying to model how those differences influence the switching process itself, which is more complex than standard models.
Lalam: I think that complexity is where the novelty lies, because existing methods haven't fully addressed estimating both levels of heterogeneity together in a frequentist way.
The paper's summary: Tom: The improvements they propose revolve around this new frequentist filter, which is built on the extended Kim filter, and it allows these hidden Markov switching models to depend on both latent within- and between-individual characteristics.
Jane: They specifically introduce a first frequentist approach within this general class of models by building on Chow and Zhang’s two thousand thirteen extended Kim filter <ref:2512.18149#pg1>. That's the specific tool they are proposing for parameter estimation, as summarized in Algorithm one <ref:2512.18149#pg1>.
Lu: The methodology involves calculating factor scores of inter-individual latent variables first, and then applying the extended Kim filter—which is a combination of the extended Kalman filter, the extended Hamilton filter, and a collapsing process—to actually perform the filtering steps.
Meng: From an engineering standpoint, that sounds computationally heavy. How does this collapse process handle scaling up when you have thousands of individuals in your dataset? What breaks at scale?
Lalam: They use distributional assumptions where prediction errors follow a multivariate normal distribution, which they then use as a basis for parameter updates using an approximated likelihood function, step five in their procedure.
The paper's improvements: Tom: So to wrap up, the paper presents this frequentist filtering method as a first approach for RSSS models that incorporates both intra- and inter-individual characteristics into the model structure.
Jane: Essentially, they show how to handle those hidden Markov switching models in a way that allows them to account for both latent within- and between-individual characteristics influencing the regime switches.
Lu: The flexibility of this RSSS modeling framework lets researchers include interactions between the intra-individual and inter-individual levels, which is often necessary when you need to validly represent those dynamic underlying processes.
Meng: So, if this works, it means we can actually model those complex interactions in dropout prediction, not just assume one level of variation is enough.
Lalam: And the simulation study results show that their approach can forecast regime membership with around eighty percent accuracy for both data sizes tested in the simulation.
Conclusion: Tom: So to summarize, Okuyama et al.'s paper introduces a first frequentist filter for RSSS models that allows them to depend on both latent within- and between-individual characteristics, building on the extended Kim filter.
Jane: They show how this method can be used for forecasting dynamics latent states and underlying regime memberships based on parameter values from time t to predict time t plus one.
Lu: The flexibility of the RSSS modeling framework allows researchers to include interactions between the intra-individual and inter-individual level, which is often necessary in order to validly represent the dynamic underlying processes.
Meng: It means this estimation procedure can accommodate a wide variety of dynamic models, which is good because it shows potential for real-time inference if you get the data flowing.
Lalam: Ultimately, the extended Kim filter serves as an effective tool for detecting unobserved change processes or regime switches in these complex settings.
University of Tubingen
stat.ME
Submitted: 2025-12-20
Updated: 2026-11-17
Comments: 34 pages, 4 figures, 13 tables. This article has been accepted for publication in a revised form in Psychometrika (https://www.cambridge.org/core/journals/psychometrika). v2: accepted manuscript after peer review; person-specific random intercepts are now marginalized within the filter, the simulation study was redone, and the use of the bwForCluster Helix is acknowledged
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 78/100
The gist: Frequentist forecasting in regime-switching models with extended Hamilton filter introduces a first frequentist filter for regimeswitching state-space (RSSS) models that allows hidden…
Key concepts
- Regime-Switching State-Space (RSSS) Models
- These models describe data where the underlying process changes abruptly between discrete hidden states or 'regimes' over time. They combine a measurement model with a structural model that accounts for both individual differences and temporal changes in behavior.
- Extended Hamilton Filter
- This is a specialized nonlinear filter used to infer the discrete latent states (regimes) within the RSSS framework. It works by utilizing maximum likelihood estimation to track these hidden states dynamically, making it suitable for real-time inference.
- Intra- and Inter-individual Characteristics
- The model incorporates two types of differences: intra-individual characteristics refer to the temporal progression of continuous variables within a single person, while inter-individual characteristics refer to differences between individuals. The proposed method explicitly models how these two levels interact in the switching process.
Terminology
Summary
Frequentist forecasting in regime-switching models with extended Hamilton filter introduces a first frequentist filter for regimeswitching state-space (RSSS) models that allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics The proposed method is the first frequentist approach within this general class of models, building on Chow and Zhang’s (2013) extended Kim filter. p. 1 Specifically, there’s a
need for frequentist estimation methods in dynamic latent variable frameworks
that allow real-time inferences and forecasts of latent or observed variables during ongoing data collection. Building on Chow and Zhang’s (2013) extended
Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and
between-individual characteristics.Where the paper says it
Model Framework
Regime-Switching State-Space (RSSS) models are used to describe the underlying data generating process, where a change process is characterized by discontinuous transitions from one discrete latent state (i.e., regime) to another at a particular time point. p. 2 Many psychological change processes are characterized by regime-switching, in
which a change process is described by discontinuous transitions from one discrete
latent state (i. , regime) to another at a particular time point (see e. Measurement model
The measurement model is described by the following linear equations. Dynamic
observed variables, which are expressed as (y1it|Sit = s), are conditional on individual i, time t, and a discrete latent state Sit. The measurement model establishes the relationship between the latent variables and their observed proxies. The structural model
depicts the temporal progression of continuous intra-individual latent variables
through hidden Markov VAR(1) models, which depend on both inter- and intraindividual variables. arXiv preprint December 23, 2025 article Page 7
Markov-switching model
The transition probabilities are characterized using a nonlinear function of the
latent variables of the between- and within-level structural models:
P r[Sit = 1|Si,t−1 = s
′
] = σ(γs
′1 + γs
′2η2i + γs
′3η1i,t−1 + γs
′4η1i,t−1η2i), (6)
P r[Sit = 2|Si,t−1 = s
′
] = 1 − P r[Sit = 1|Si,t−1 = s
′
], (7)
where s, s′ ∈ {1, 2} indicates the regime membership (e. , s = 1: no dropout intention; s = 2: dropout intention).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Estimation Procedure
The proposed parameter estimation procedure is summarized in Algorithm 1, which involves calculating factor scores of inter-individual latent variables (step 1). Then, the extended Kim filter, composed of the extended Kalman filter, the extended Hamilton filter, and the collapsing process is used to filter RSSS models (step 2-4). p. 6 Factor scores of inter-individual latent variables are calculated (step 1). Then, the
extended Kim filter, which is composed of the extended Kalman filter, the extended
Hamilton filter, and the collapsing process, is used to filter RSSS models (step 2-4). Then, the
extended Kim filter, which is composed of the extended Kalman filter, the extended
Hamilton filter, and the collapsing process, is used to filter RSSS models (step 2-4). Based on the distributional assumptions, the prediction errors follow a multivariate
normal distribution, serving as a basis for parameter updates using an approximated
likelihood function1
(step 5). The extended Kalman filter (step 2)
The original Kalman filter (Kalman, 1960) was proposed as a real-time estimation procedure to estimate the continous latent variables in a state-space model
as well as to predict their future values. The Kalman filter consists of three steps,
namely, prediction, measurement, and update. Unlike the Kalman filter, the algorithm is specialized for
the inference of discrete latent states and works as a nonlinear filter. The filter also
utilizes the maximum likelihood estimation which serves as a basis for parameter
optimization.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Forecasting Implementation
The forecasting implementation involves using the extended Kim filter to forecast the dynamics latent states and underlying regime memberships for time t + 1 based on parameter values at time t. The collapsing process approximates joint estimates into marginal estimates, utilizing a weighting factor Ws,s'it which is referred to as the weighting factor. The prediction error decomposition function is computed using the prediction error decomposition function (Schweppe, 1965). p. 9 We call it the extended Hamilton filter and the recursive process can be
described as
P r[Sit = s, Si,t−1 = s
′
|D1:t−1] = P r[Sit = s|Si,t−1 = s
′
]P r[Si,t−1 = s
′
|D1:t−1], (17)
f(y1it|D1:t−1) = X
s,s′
f(y1it|Sit = s, Si,t−1 = s
′
, D1:t−1)P r[Sit = s, Si,t−1 = s
′
|D1:t−1]
(18)
P r[Sit = s, Si,t−1 = s
′
|D1:t
] = f(y1it|Sit = s, Si,t−1 = s
′
, D1:t−1)P r[Sit = s, Si,t−1 = s
′
|D1:t−1]
f(y1it|D1:t−1)
(19)
P r[Sit = s|D1:t
] = X
s
′
P r[Sit = s, Si,t−1 = s
′
|D1:t
], (20)
where the conditional multivariate normal likelihood function f(y1it|Sit = s, Si,t−1 =
s
′
, D1:t−1) is computed as
f(y1it|Sit = s, Si,t−1 = s
′
, D1:t−1) = (2π)
−
O1
2 |F
s,s′
1it |
− 1
2 exp
−
1
2
(v
s,s′
it )
T
[F
s,s′
1it ]
−1v
s,s′
it
. (21)
f(y1it|D1:t−1) in Equation (18) is called the prediction error decomposition function
(Schweppe, 1965). Then, the
extended Kim filter, which is composed of the extended Kalman filter, the extended
Hamilton filter, and the collapsing process, is used to filter RSSS models (step 2-4). Based on the distributional assumptions, the prediction errors follow a multivariate
normal distribution, serving as a basis for parameter updates using an approximated
likelihood function1
(step 5).Where the paper says it
Where the paper says it
Empirical Validation
An empirical study applied this filter to forecast emotions and behavior related to student dropout in math. The study used intensive longitudinal data (ILD) from a student dropout study in math, involving 7 intra-individual latent factors and 1 inter-individual latent factor. p. 1 arXiv preprint December 23, 2025 article Page 2
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED KIM FILTER
Abstract
Psychological change processes, such as university student dropout in math,
often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent
variables and their autoregressive effects. Similar to the Kalman filter, the Hamilton filter consists of a prediction
and an update step. 1, the model probabilities are predicted using the transition probabilities. Kelava et al. (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM). Many psychological change processes are characterized by regime-switching, in
which a change process is described by discontinuous transitions from one discrete
latent state (i. , regime) to another at a particular time point (see e. , Hamaker
and Grasman, 2012; Asparouhov et al. Individuals are allowed to switch
between regimes over time.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Simulation Study Results
A simulation study evaluated the performance of the proposed approach given complete datasets, where regime prediction accuracy was around 80% for both data sizes. The quadratic score function showed a consistently lower score function and a smaller standard deviation for larger sample size (D100) throughout the forecast interval. p. 6 The parameter estimates will become available at the
end of Algorithm 1; the associated standard errors can be obtained by taking the
square root of the diagonal elements of the negative numerical Hessian matrix or its
approximation at the estimated point. The Confirmatory factor analysis (step 1)
The inter-individual latent variables η2i are estimated from their observed proxies y2i using the factor score regression (cf. arXiv preprint December 23, 2025 article Page 3
Recently, the increasing availability of intensive longitudinal data (ILD) has
been brought about by digital devices such as mobile phones and smartwatches. ILD typically consists of data from several subjects, with around thirty to a hundred repeated measures per subject. The parameter estimates will become available at the
end of Algorithm 1; the associated standard errors can be obtained by taking the
square root of the diagonal elements of the negative numerical Hessian matrix or its
approximation at the estimated point. The Confirmatory factor analysis (step 1)
The inter-individual latent variables η2i are estimated from their observed proxies y2i using the factor score regression (cf.Where the paper says it
Where the paper says it
Where the paper says it
Discussion and Extensions
The flexibility of the RSSS modeling framework allows researchers to include interactions between the intra-individual and inter-individual level. The described model can be readily extended to accommodate three or more regimes, or a more flexible state-space representation. p. 3 Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. An estimation procedure called the
Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended
Kim smoother. , dispositions, personal traits). Other
parts of the data consist of repeated measures whose states are subject to dynamic
changes across time within individuals (intra-individual; e.Where the paper says it
Where the paper says it
Open Questions
This article did not evaluate the performance under different degree of missingness, as the empirical and simulation study presented two extremes. The classification result is dependent on the proportion of clusters, and performance might be sensitive to the similarities of regime distributions. p. 4 In an empirical
example, we illustrate the approach and its ability to forecast regime membership. Then, we provide further evidence of its performance using a simulation study. As a counterpart of Kelava et al. ’s (2022)
Bayesian forecasting filter for nonlinear dynamic latent class structural equation
models (NDLC-SEM), our proposed method is the first frequentist approach
within this general class of models. Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. An estimation procedure called the
Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended
Kim smoother. Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. An estimation procedure called the
Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended
Kim smoother.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Data Availability
The associated replication files for the simulation study, including all analysis code, are available online at https://github.com/PsychometricsMZ/Regime-switching. p. 0 arXiv preprint December 23, 2025 article Page 1
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED HAMILTON FILTER
Kento Okuyama
university of tubingen ¨
Tim Fabian Schaffland
university of tubingen, d-fine gmbh ¨
Pascal Kilian
university of tubingen, robert bosch gmbh ¨
Holger Brandt
university of tubingen ¨
Augustin Kelava
university of tubingen ¨
Correspondence should be sent to Kento Okuyama, Methods Center, University of
T¨ubingen, 72076, Germany. E-Mail: kento.Where the paper says it
References
Anja Achtziger and Peter M. Gollwitzer. p. 2 , from an intention to stay at math lecture
to an intention to quit (cf. , Rubicon model; Achtziger and Gollwitzer, 2018). arXiv preprint December 23, 2025 article Page 5
fore, the lack of theoretical and empirical anticipation of effects (priors/beliefs) could
be a motivation to adapt a frequentist approach. In this paper, we present a frequentist estimation approach for RSSS models which considers both intra- and inter-individual differences in hidden Markov(-
switching) models. Structural equation model (SEM) has been one of the most important methods
in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. Structural equation model (SEM) has been one of the most important methods
in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. Structural equation model (SEM) has been one of the most important methods
in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. Structural equation model (SEM) has been one of the most important methods
in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. As a counterpart of Kelava et al. ’s (2022)
Bayesian forecasting filter for nonlinear dynamic latent class structural equation
models (NDLC-SEM), our proposed method is the first frequentist approach
within this general class of models. Kelava et al. (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM). (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM). Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. As a counterpart of Kelava et al. ’s (2022)
Bayesian forecasting filter for nonlinear dynamic latent class structural equation
models (NDLC-SEM), our proposed method is the first frequentist approach
within this general class of models. As a counterpart of Kelava et al. ’s (2022)
Bayesian forecasting filter for nonlinear dynamic latent class structural equation
models (NDLC-SEM), our proposed method is the first frequentist approach
within this general class of models. Structural equation model (SEM) has been one of the most important methods
in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. arXiv preprint December 23, 2025 article Page 1
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED HAMILTON FILTER
Kento Okuyama
university of tubingen ¨
Tim Fabian Schaffland
university of tubingen, d-fine gmbh ¨
Pascal Kilian
university of tubingen, robert bosch gmbh ¨
Holger Brandt
university of tubingen ¨
Augustin Kelava
university of tubingen ¨
Correspondence should be sent to Kento Okuyama, Methods Center, University of
T¨ubingen, 72076, Germany. E-Mail: kento. Furthermore, K
s,s′
it,aug = P
s,s′
aug,it|t−1ΛT
1s,aug[F
s,s′
1it ]
−1
is called the Kalman
gain function. Equation (15) is often substituted by what’s called the Joseph form
(Bucy and Joseph, 2005) because subtracting two covariance matrices may result
in a numerical error causing one of the diagonal elements to be slightly less than
zero. Kim (1994) generalized this approach to state-space models, which are called
regime-switching state-space (RSSS) models. This framework accommodated a wide
range of models, including latent variable models and regime-dependent time-series
models (Yang and Chow, 2010). Specifically, there’s a
need for frequentist estimation methods in dynamic latent variable frameworks
that allow real-time inferences and forecasts of latent or observed variables during ongoing data collection. Building on Chow and Zhang’s (2013) extended
Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and
between-individual characteristics. Kelava et al. (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM). Given the set of parameter values at time t, the extended Kim filter forecasts the dynamics latent states and the underlying regime
memberships for time t + 1. The parameter estimates will become available at the
end of Algorithm 1; the associated standard errors can be obtained by taking the
square root of the diagonal elements of the negative numerical Hessian matrix or its
approximation at the estimated point. Given the set of parameter values at time t, the extended Kim filter forecasts the dynamics latent states and the underlying regime
memberships for time t + 1. The parameter estimates will become available at the
end of Algorithm 1; the associated standard errors can be obtained by taking the
square root of the diagonal elements of the negative numerical Hessian matrix or its
approximation at the estimated point. arXiv preprint December 23, 2025 article Page 2
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED KIM FILTER
Abstract
Psychological change processes, such as university student dropout in math,
often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent
variables and their autoregressive effects. arXiv preprint December 23, 2025 article Page 2
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED KIM FILTER
Abstract
Psychological change processes, such as university student dropout in math,
often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent
variables and their autoregressive effects. , regime) to another at a particular time point (see e. , Hamaker
and Grasman, 2012; Asparouhov et al. , regime) to another at a particular time point (see e. , Hamaker
and Grasman, 2012; Asparouhov et al. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. With increasing number of substantive applications
of DSEMs, a number of new ILD methods have been developed (e. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. With increasing number of substantive applications
of DSEMs, a number of new ILD methods have been developed (e. , mean, variance, autocorrelation). Since Markov-switching autoregressive (MSAR) models (Hamilton, 1989) were
introduced, regime-switching models have mostly gained the interest of econometricians. , mean, variance, autocorrelation). Since Markov-switching autoregressive (MSAR) models (Hamilton, 1989) were
introduced, regime-switching models have mostly gained the interest of econometricians. As a counterpart of Kelava et al. ’s (2022)
Bayesian forecasting filter for nonlinear dynamic latent class structural equation
models (NDLC-SEM), our proposed method is the first frequentist approach
within this general class of models. As a counterpart of Kelava et al. ’s (2022)
Bayesian forecasting filter for nonlinear dynamic latent class structural equation
models (NDLC-SEM), our proposed method is the first frequentist approach
within this general class of models. arXiv preprint December 23, 2025 article Page 1
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED HAMILTON FILTER
Kento Okuyama
university of tubingen ¨
Tim Fabian Schaffland
university of tubingen, d-fine gmbh ¨
Pascal Kilian
university of tubingen, robert bosch gmbh ¨
Holger Brandt
university of tubingen ¨
Augustin Kelava
university of tubingen ¨
Correspondence should be sent to Kento Okuyama, Methods Center, University of
T¨ubingen, 72076, Germany. E-Mail: kento. arXiv preprint December 23, 2025 article Page 1
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED HAMILTON FILTER
Kento Okuyama
university of tubingen ¨
Tim Fabian Schaffland
university of tubingen, d-fine gmbh ¨
Pascal Kilian
university of tubingen, robert bosch gmbh ¨
Holger Brandt
university of tubingen ¨
Augustin Kelava
university of tubingen ¨
Correspondence should be sent to Kento Okuyama, Methods Center, University of
T¨ubingen, 72076, Germany. E-Mail: kento. In
other cases, subjects could repeatedly switch back and forth between two or more
states such as the sleep-wake cycle (Ji et al. Studying such processes on a
within-individual level requires a regime-switching model that assumes that each
observation comes from one of the distinct distributions, each of which has a unique
set of parameters (e. In
other cases, subjects could repeatedly switch back and forth between two or more
states such as the sleep-wake cycle (Ji et al. Studying such processes on a
within-individual level requires a regime-switching model that assumes that each
observation comes from one of the distinct distributions, each of which has a unique
set of parameters (e. ILD typically consists of data from several subjects, with around thirty to a hundred repeated measures per subject. The data could take various forms, such as
self-report, physiological data, and behavioral, environmental, and geographic data
(Trull and Ebner-Priemer, 2014). ILD typically consists of data from several subjects, with around thirty to a hundred repeated measures per subject. The data could take various forms, such as
self-report, physiological data, and behavioral, environmental, and geographic data
(Trull and Ebner-Priemer, 2014). An estimation procedure called the
Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended
Kim smoother. Accounting for unobserved heterogeneity in the hidden Markov(-
switching) models characterizes the novelty of this approach. An estimation procedure called the
Forward Filtering Backward Sampling (FFBS; West and Harrison, 2006) is implemented as a Bayesian counterpart to the extended Kim filter and extended
Kim smoother. Accounting for unobserved heterogeneity in the hidden Markov(-
switching) models characterizes the novelty of this approach.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Appendix
Augmented State-Space Formulation combines the U1 × 1 time-dependent state η1it and the U1 × 1 time-invariant random intercept residual ζ2i into a single Uaug × 1 state vector where Uaug = 2U1. The measurement model is expressed using this augmented state, defining the augmented factor loading matrix Λ1s,aug as hΛ1s 0 i. p. 5 arXiv preprint December 23, 2025 article Page 6
dependent latent variables. Λ1s is a factor loading matrix which is dependent on
dynamic latent state Sit, with a dimension O1 × U1. arXiv preprint December 23, 2025 article Page 6
dependent latent variables. Λ1s is a factor loading matrix which is dependent on
dynamic latent state Sit, with a dimension O1 × U1. (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM). Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model. (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM). Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regimeswitching model and their state-space model.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Table Summaries
Table 3 summarizes parameter estimates for the Markov-switching model, showing mean and standard errors for parameters like γ1, γ2, γ3, etc. Table 4 presents parameter estimates for the structural parameters (s = 1), including baseline effects and autoregressive coefficients. p. 7 , T do
2: Extended Kalman filter
3: Extended Hamilton filter
4: Collapsing process
end for
5: Parameter update
end while
where F2 is the matrix of factor score weights. This matrix is computed based on the
current estimates of the measurement model parameters: the factor loading matrix
Λ2 and the residual covariance matrix R2. , T do
2: Extended Kalman filter
3: Extended Hamilton filter
4: Collapsing process
end for
5: Parameter update
end while
where F2 is the matrix of factor score weights. This matrix is computed based on the
current estimates of the measurement model parameters: the factor loading matrix
Λ2 and the residual covariance matrix R2. , T do
2: Extended Kalman filter
3: Extended Hamilton filter
4: Collapsing process
end for
5: Parameter update
end while
where F2 is the matrix of factor score weights. This matrix is computed based on the
current estimates of the measurement model parameters: the factor loading matrix
Λ2 and the residual covariance matrix R2. The parameter estimates will become available at the
end of Algorithm 1; the associated standard errors can be obtained by taking the
square root of the diagonal elements of the negative numerical Hessian matrix or its
approximation at the estimated point. The Confirmatory factor analysis (step 1)
The inter-individual latent variables η2i are estimated from their observed proxies y2i using the factor score regression (cf. , T do
2: Extended Kalman filter
3: Extended Hamilton filter
4: Collapsing process
end for
5: Parameter update
end while
where F2 is the matrix of factor score weights. This matrix is computed based on the
current estimates of the measurement model parameters: the factor loading matrix
Λ2 and the residual covariance matrix R2. , T do
2: Extended Kalman filter
3: Extended Hamilton filter
4: Collapsing process
end for
5: Parameter update
end while
where F2 is the matrix of factor score weights. This matrix is computed based on the
current estimates of the measurement model parameters: the factor loading matrix
Λ2 and the residual covariance matrix R2. Given the set of parameter values at time t, the extended Kim filter forecasts the dynamics latent states and the underlying regime
memberships for time t + 1. The parameter estimates will become available at the
end of Algorithm 1; the associated standard errors can be obtained by taking the
square root of the diagonal elements of the negative numerical Hessian matrix or its
approximation at the estimated point. , T do
2: Extended Kalman filter
3: Extended Hamilton filter
4: Collapsing process
end for
5: Parameter update
end while
where F2 is the matrix of factor score weights. This matrix is computed based on the
current estimates of the measurement model parameters: the factor loading matrix
Λ2 and the residual covariance matrix R2.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Forecast Results
Parameter estimates for the Markov-switching model show that the main between effect (γ2) was positive and the main intra-individual effect (γ3) was negative for 6 out of 7 parameters. For structural model parameters, the autoregressive coefficients (diagonal elements of B3s) were positive and large for all within-scales. p. 5 b1is is a U1×1 conditional random
intercept vector, B3is is a U1 × U1 matrix of conditional intra-individual VAR(1)
effects, and Q1s is a U1 × U1 residual covariance matrix which is dependent on a
dynamic latent state Sit. Between-level structural model
For the random intercept vector b1is and the autoregressive coefficient matrix
B3is, the between-level structural model is specified by two models:
b1is = b1s + b2sη2i + ζ2i
, (4)
B3is = B3s + B4sη2i
, (5)
where b1s is a U1 × 1 conditional intercept vector, b2s is a U1 × 1 vector of conditional inter-individual effects, ζ2i
is a U1 × 1 vector of random intercepts with
ζ2i ∼ N (0, Q2). Similar to the Kalman filter, the Hamilton filter consists of a prediction
and an update step. 1, the model probabilities are predicted using the transition probabilities. Kelava et al. (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM).Where the paper says it
Where the paper says it
Where the paper says it
Discussion
The flexibility of RSSS modeling framework allows researchers to include interactions between the intra-individual and inter-individual level, which is often necessary in order to validly represent the dynamic underlying processes. The described model and estimation procedures can accommodate a wide variety of dynamic models. p. 4 Measurement model
The measurement model is described by the following linear equations. Dynamic
observed variables, which are expressed as (y1it|Sit = s), are conditional on individual i, time t, and a discrete latent state Sit. arXiv preprint December 23, 2025 article Page 2
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED KIM FILTER
Abstract
Psychological change processes, such as university student dropout in math,
often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent
variables and their autoregressive effects.Where the paper says it
Where the paper says it
The classification result is dependent on the proportion of clusters, and performance might be sensitive to the similarities of regime distributions. Direct comparison with Bayesian counterpart would be an interesting future direction. p. 1 As a counterpart of Kelava et al. ’s (2022)
Bayesian forecasting filter for nonlinear dynamic latent class structural equation
models (NDLC-SEM), our proposed method is the first frequentist approach
within this general class of models. arXiv preprint December 23, 2025 article Page 2
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED KIM FILTER
Abstract
Psychological change processes, such as university student dropout in math,
often exhibit discrete latent state transitions and can be studied using regimeswitching models with intensive longitudinal data (ILD). Recently, regimeswitching state-space (RSSS) models have been extended to allow for latent
variables and their autoregressive effects.Where the paper says it
Where the paper says it
Anja Achtziger and Peter M Gollwitzer Motivation and Volition in the Course of Action, pages 485–527. Springer International Publishing, Cham, 2018. Tihomir Asparouhov Ellen L Hamaker and Bengt Muth´en Dynamic latent class analysis Structural Equation Modeling: A Multidisciplinary Journal, 24(2):257– 269, 2017. p. 2 Structural equation model (SEM) has been one of the most important methods
in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. This matrix is computed based on the
current estimates of the measurement model parameters: the factor loading matrix
Λ2 and the residual covariance matrix R2. In the case of Bartlett factor scores
(Bartlett, 1937), the formula for F2 is:
F2 =
Λ
T
2 R−1
2 Λ2
−1
Λ
T
2 R−1
2
. Structural equation model (SEM) has been one of the most important methods
in psychometrics. Dynamic SEM (DSEM; Tihomir Asparouhov and Muth´en, 2018),
which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. Filtering
(Box et al. , 2015) is an important technique as it allows forecasting i. arXiv preprint December 23, 2025 article Page 1
FREQUENTIST FORECASTING IN REGIME-SWITCHING MODELS WITH
EXTENDED HAMILTON FILTER
Kento Okuyama
university of tubingen ¨
Tim Fabian Schaffland
university of tubingen, d-fine gmbh ¨
Pascal Kilian
university of tubingen, robert bosch gmbh ¨
Holger Brandt
university of tubingen ¨
Augustin Kelava
university of tubingen ¨
Correspondence should be sent to Kento Okuyama, Methods Center, University of
T¨ubingen, 72076, Germany. E-Mail: kento. Furthermore, K
s,s′
it,aug = P
s,s′
aug,it|t−1ΛT
1s,aug[F
s,s′
1it ]
−1
is called the Kalman
gain function. Equation (15) is often substituted by what’s called the Joseph form
(Bucy and Joseph, 2005) because subtracting two covariance matrices may result
in a numerical error causing one of the diagonal elements to be slightly less than
zero. This framework accommodated a wide
range of models, including latent variable models and regime-dependent time-series
models (Yang and Chow, 2010). Chow and Zhang (2013) then developed nonlinear RSSS models, whose statespace representation is governed by nonlinear dynamics. Kelava et al. (2022) addressed forecasting the precursor of student dropout,
referred to as the intention to dropout, within their regime-switching modeling
framework called the nonlinear dynamics latent class structural equation models
(NDLC-SEM).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Augmented State-Space Formulation defines the augmented state vector as ηaug,it = η1it ζ2i i=t. The measurement model is expressed using this augmented state, defining the augmented factor loading matrix Λ1s,aug as hΛ1s 0 i. The structural models are combined to form the state transition equation, which directly corresponds to the augmented Kalman filter’s state equation.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Frequentist estimation for regime-switching state-space models can be implemented to allow
real-time inferences and forecasts of latent or observed variables during ongoing data collection.
This enables dynamic modeling that is suitable for applications involvingreal-time monitoring and timely intervention.
-
The proposed method allows the model to consider both intra- and inter-individual differences in hidden Markov switching models, accommodating
unobserved heterogeneity in the hidden Markov(-switching) models
which is a novelty compared to existing frequentist approaches. -
The system can forecast regime membership for student dropout with an accuracy of around 80% across simulation studies, suggesting it provides
early signals of dropout intentions when interventions should be conducted.
-
Missing values in intensive longitudinal data can be handled by substituting the likelihood calculation and update steps with estimations from the prediction step, overcoming issues where missing values prevent obtaining
the likelihood function
and updated estimates.
Abstract
Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regime-switching models with intensive longitudinal data (ILD). Recently, regime-switching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects. Despite this progress, there is a need for frequentist estimation methods in dynamic latent variable frameworks that handle both intra-individual changes and inter-individual differences as predictors of regime-switches and allow real-time inferences and forecasts during ongoing data collection. Building on Chow and Zhang's (2013) extended Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics. As a counterpart of Kelava et al.'s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our method is the first frequentist approach within this general class of models. In an empirical study, the filter is applied to forecast emotions and behavior related to student dropout in math. A simulation study shows that marginalizing person-specific random intercepts within the filter cleanly separates autoregressive persistence from stable between-person differences, with accurate out-of-sample regime forecasts, while within-person transition effects stay weakly identified.
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes