Model fit vs prediction performance: contrasting fixed-coefficient models with mixture models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Model fit vs prediction performance".
Warren: Here's the gist of the paper. Models allowing for random heterogeneity, such as mixed logit and latent class, are generally observed to obtain superior model fit and yield detailed insights into unobserved preference heterogeneity but these advantages do not translate into any benefits in forecasting whether looking at prediction performance or the recovery of market shares The only exception arises when using conditional distributions in making predictions for the same individuals included in the estimation sample which obviously precludes any out-of-sample forecasting.
Ruth: Theoretical Disconnect between Estimation and Application Decreased prediction performance compared to fit in estimation may happen for two reasons, one relates to the model's inability to generalize whether this relates to changes in attribute combinations (x) or preferences (β), and the second is that the way in which mixture models are specified leads to a disconnect between model estimation and model application In continuous mixture models with integration at the person level, Equation 3 does not apply because in estimation, we look at the likelihood of entire sequence of choices conditional on a given value of β before then integrating over the distribution of β Similarly, with latent class models specified at the person level, Equation 3 does not apply because class membership remains latent and heterogeneity is averaged out in both estimation and application.
Warren: The Role of Conditional Distributions There is one notable exception when the analyst has access to previous choices of an individual and wishes to make forecasts for that same individual, then we can reduce the uncertainty in relation to a person’s location on the sample level distribution of β by relying on posterior (or conditional) parameter distributions This procedure is described in Train (2009, ch.
Ruth: 11.2) for continuous mixture models and has not seen widespread use.
Title and authors: Ruth: So, let's start by looking at the title and who wrote this—"Model fit vs prediction performance: contrasting fixed-coefficient models with mixture models." It sets up a direct comparison between simpler, fixed-coefficient models and these more flexible mixture versions.
Warren: And the authors are Stephane Hess and Sander van Cranenburgh. They’re using revealed preference data from decisions and stated choice data from COVID-nineteen to test this idea.
Iris: The core of their research is showing that increased flexibility in the model, through random heterogeneity like mixing, doesn't always lead to better predictions or market share recovery.
Ruth: Exactly. The paper highlights a disconnect between estimation and application when using mixture models unless you use conditional distributions in specific ways for people already in the sample.
Warren: That's a big point because it suggests that just chasing the highest fit score on historical data might be misleading if your goal is actually predicting what happens next.
The paper's summary: Ruth: The paper summarizes their findings by explaining two main reasons why prediction performance can drop even when the model fit in estimation is high.
Warren: First, it relates to the model's inability to generalize, whether that’s about changes in the attributes you are looking at or changes in the actual preferences of people.
Iris: The second reason they focus on is a disconnect between how these mixture models are set up and how we actually use them for prediction.
Ruth: They explain that in continuous mixture models with integration at the person level, things like Equation three don't apply when you're estimating because you look at the likelihood of a whole sequence of choices conditional on a fixed preference value before integrating over the distribution of preferences <ref:2510.09185#pg2>.
Warren: And for latent class models specified at the person level, class membership stays latent, so that heterogeneity gets averaged out in both estimation and application.
The paper's improvements: Ruth: The paper then suggests a couple of ways to improve things when it comes to making predictions for individuals who were already in the estimation sample.
Warren: They point out that the exception they found is using posterior or conditional parameter distributions when you have access to previous choices of that specific individual.
Iris: This procedure, which is described in Train two thousand nine chapter eleven point two, lets you reduce uncertainty about a person’s location on the sample distribution of preferences by relying on those conditional distributions.
Ruth: So, instead of just using the point estimate from the estimation sample for that person's preference, you use this conditional distribution to calculate the probability of them choosing an alternative in a situation.
Warren: This is specifically useful for predicting new choices for people who were already included in the estimation data; they see very substantial gains in prediction accuracy when using these conditionals.
Conclusion: Ruth: So, to wrap up, the authors show that mixture model fit in estimation isn't a guarantee of good prediction performance if you’re not careful about how you apply the model.
Warren: They emphasize that including covariates generally improves prediction performance across both case studies, with the only real exception being the use of conditionals for making predictions about individuals already seen.
Iris: They conclude that while mixture models are powerful for understanding preference heterogeneity, if your main goal is forecasting, especially out-of-sample forecasting, it might be wiser to focus on refining and optimizing a model with only deterministic heterogeneity.
Ruth: So the paper suggests a trade-off: use mixtures for insight into preferences but maybe stick to simpler models if the primary aim is accurate prediction for things you haven't seen yet.
Warren: And they make sure to stress that when looking at market share recovery, using conditionals doesn't necessarily lead to a lower bias compared to other methods on out-of-sample data.
University of Leeds · Delft University of Technology
econ.EM
Submitted: 2025-10-10
Updated: 2026-11-05
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 67/100
The gist: Models allowing for random heterogeneity, such as mixed logit and latent class, are generally observed to obtain superior model fit and yield detailed insights into unobserved preference…
Key concepts
- Model Fit vs. Prediction Performance
- This examines whether a model that accurately describes the data during estimation (model fit) actually performs well when predicting future choices (prediction performance). The paper finds that good fit does not guarantee good prediction, especially with person-level mixing in mixture models.
- Random Heterogeneity
- This refers to the idea that individuals have different underlying preferences or characteristics. Mixture models are used to capture this unobserved heterogeneity, which helps explain why some people choose certain options while others do not. This heterogeneity is key for understanding individual behavior.
- Conditional Distributions
- These are probability distributions based on previous choices made by a specific individual. Using these conditionals allows analysts to reduce uncertainty when forecasting the next choice for that known person, offering substantial gains in prediction accuracy.
Terminology
Summary
Models allowing for random heterogeneity, such as mixed logit and latent class, are generally observed to obtain superior model fit and yield detailed insights into unobserved preference heterogeneity but these advantages do not translate into any benefits in forecasting whether looking at prediction performance or the recovery of market shares The only exception arises when using conditional distributions in making predictions for the same individuals included in the estimation sample which obviously precludes any out-of-sample forecasting. p. 0 Flexibility without foresight: the predictive limitations of mixture
models
Stephane Hess∗ Sander van Cranenburgh†
October 13, 2025
Abstract
Models allowing for random heterogeneity, such as mixed logit and latent class, are generally
observed to obtain superior model fit and yield detailed insights into unobserved preference heterogeneity. Using theoretical arguments and two case studies on revealed and stated choice data, this
paper highlights that these advantages do not translate into any benefits in forecasting, whether
looking at prediction performance or the recovery of market shares.Where the paper says it
Theoretical Disconnect between Estimation and Application
Decreased prediction performance compared to fit in estimation may happen for two reasons, one relates to the model's inability to generalize whether this relates to changes in attribute combinations (x) or preferences (β), and the second is that the way in which mixture models are specified leads to a disconnect between model estimation and model application In continuous mixture models with integration at the person level, Equation 3 does not apply because in estimation, we look at the likelihood of entire sequence of choices conditional on a given value of β before then integrating over the distribution of β Similarly, with latent class models specified at the person level, Equation 3 does not apply because class membership remains latent and heterogeneity is averaged out in both estimation and application.
The Role of Conditional Distributions
There is one notable exception when the analyst has access to previous choices of an individual and wishes to make forecasts for that same individual, then we can reduce the uncertainty in relation to a person’s location on the sample level distribution of β by relying on posterior (or conditional) parameter distributions This procedure is described in Train (2009, ch. 11.2) for continuous mixture models and has not seen widespread use. p. 4 This procedure is described in Train (2009, ch. 2) for continuous mixture
models, but has not seen widespread use, with Danaf et al. Using Bayes’ rule, we can then
compute the posterior distribution h (βn | ΩM, Yn) as:
h
βn | ΩbM, Yn
=
Ln (βn) f
βn | ΩbM
Ln
ΩbM
(8)
This is also referred to as the conditional distribution, as it is conditional not just on ΩbM, but also on
the choices observed for person n that were used in estimation. We can then calculate the probability of person n choosing alternative j in choice situation t
∗
n
(where
t
∗
n ∈ T / n) as:
Pjnt∗
n
ΩbM, Yn
=
Z
βn
Pjnt∗
n
(βn) h
βn | ΩbM, Yn
dβn
=
Z
βn
Pjnt∗
n
(βn)Ln (βn) f
βn | ΩbM
dβn
Z
βn
Ln (βn) f
βn | ΩbM
dβn
(9)
The expression in Equation 9 again does not have a closed form solution, but can be approximated as:
Pejnt∗
n
ΩbM, Yn
=
PR
r=1 Pjnt∗
n
β
(r)
n
Ln
β
(r)
n
PR
r=1 Ln
β
(r)
n
(10)
An equivalent approach can be used straightforwardly with LC models.Where the paper says it
Where the paper says it
Performance Across Prediction Scenarios
The results show that fit in estimation (case 1) does not mean good performance in prediction, especially when estimation uses mixing at the person level as opposed to the observation level The case 2 segment (new individuals) allows us to check for failure to generalise and is the case most in line with real-world forecasting. Furthermore, when using conditionals, we see very substantial gains in predicting new choices for people who were also included in the estimation data (case 3). p. 9 We finally
turn to predicting new choices for individuals included in the estimation data (case 3). When not using
conditionals, the same order applies as for the estimation sample, but with a small drop in performance.Where the paper says it
Model Comparison and Final Conclusion
Across both case studies, model fit in estimation when applying the mixing at the person level is not a guarantee of good performance in prediction, and including covariates leads to improved prediction performance throughout with the only exception being the use of conditionals In conclusion, mixture models are powerful tools for recovering heterogeneity in preferences and providing behavioural insights but if the interest is in forecasting especially out-of-sample forecasting it may be wiser to devote effort to refining and optimising the specification of the utility function with deterministic heterogeneity only.
Data Setup and Model Specification
The study uses two case studies, one on revealed preference (RP) data from DECISIONS and one on stated choice (SC) data from COVID-19 The models estimated include models without random heterogeneity, continuous mixture versions with continuous random heterogeneity, and latent class versions with discrete random heterogeneity. The specification for random heterogeneity involves using normal distributions for ASCs in DECISIONS and negative log-uniform distributions for in-vehicle time, out-of-vehicle time and cost. p. 7 4 Random heterogeneity
A rich specification for random heterogeneity is used in the two case studies. For the DECISIONS case study, the continuous mixture models use normal distributions for the ASCs
and negative log-uniform distributions for in-vehicle time, out-of-vehicle time and cost.Where the paper says it
Prediction Metrics Analysis
Performance is measured by the average probability that a model assigns to the alternative that is actually chosen, i.e. looking at how well individual choices are predicted by the models The true positive rate (TPR) analysis shows that mixture models do not outperform their simpler counterparts and all models struggle to a similar extent in predicting the minority alternatives. p. 8 2 Prediction performance
We next turn to prediction performance, measured by the average probability that a model assigns
to the alternative that is actually chosen, i. looking at how well individual choices are predicted
by the models. We finally
turn to predicting new choices for individuals included in the estimation data (case 3). When not using
conditionals, the same order applies as for the estimation sample, but with a small drop in performance.Where the paper says it
Where the paper says it
Market Share Recovery Findings
The recovery of market shares shows that the expected perfect recovery of market shares on the estimation data (case 1) is for the MNL model given its use of a full set of ASCs Bias is introduced in the mixture models, more so for MMNL than for LC, while it is slightly lower when including covariates. In the out-of-sample segment (case 2) and especially in the new choices for the same people (case 3), the bias is now highest for the models without random heterogeneity. p. 6 Table 1: Data split for case studies
DECISIONS COVID-19
Individuals Observations Individuals Observations
Estimation sample (case 1) 1,718 8,590 320 8,749
Future choices, new individuals (case 2) 429 2574 80 2,277
Future choices, same individuals (case 3) 1,718 1,718 320 320
3. 2 Models
Seven separate models were estimated for each of the two case studies, as follows:
Models without random heterogeneity: two models are used in this group, one without any
heterogeneity, and one with deterministic heterogeneity. We use negative log-uniform distributions for the risk of infection, the risk of illness,
unknown protection duration, mild and severe side effects, waiting time and fee, with a positive loguniform distribution for protection duration. For the LC model, our specification again has 3 classes,
with all parameters (including the nesting parameter) varying across classes, except for the left-right
bias parameter.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve predictive accuracy by using posterior distributions when making predictions for new choices for individuals who were also included in the estimation sample, as this
obviously precludes any out-of-sample forecasting.
-
Enhance model generalization by focusing on refining and optimizing the specification of the utility function with deterministic heterogeneity only, as this may be wiser for developing models for use in forecasting, especially out-of-sample forecasting.
-
Improve prediction performance by using conditionals when including covariates in the models, noting that
we see very substantial gains when using conditionals
for predicting new choices for people who were also included in the estimation data (case 3). -
Enhance market share recovery by acknowledging that
the use of conditionals does not lead to an overall lower bias
in recovering market shares on out-of-sample data, contrasting it with the results for other segments. -
Improve classification performance by utilizing sensitivity or true positive rate as a metric, as it captures the ability to correctly identify observations with a specific outcome, revealing that
mixture models do not outperform their simpler counterparts.
Abstract
Mixture models such as mixed logit and latent class routinely achieve large gains in fit on panel data, yet several studies have found that these gains do not carry over to out-of-sample prediction. This paper explains why, building on the distinction between population-level and person-level prediction familiar from statistics. The panel log-likelihood of a mixture is exactly the cumulative score of forecasts that condition each choice on the person's earlier choices. Models are instead usually validated and applied with unconditional forecasts, which use only the estimated distribution of tastes and are scored by the cross-sectional log-likelihood. The difference between the two log-likelihoods, which we call choice consistency, comes from reproducing the dependence between a person's choices, and unconditional forecasts cannot use it. Decomposing the fit gain into choice consistency and heterogeneity, we show that when the assumed mixing distribution cannot reproduce this dependence exactly, pursuing choice consistency distorts the estimated heterogeneity, so that a mixture may predict single choices worse than a model without mixing. In two case studies, on revealed preference and stated choice data, choice consistency accounts for more than 90% of the fit gain for each of the fourteen panel mixtures. In our data, the part of the fit gain that helps to forecast single choices shrinks as people make more choices, and with many choices per person it becomes negative, leaving the mixture worse than a model without mixing. Mixtures forecast single choices clearly better only when the forecast can condition on the person's own earlier choices, which is often not possible in practice. For forecasting, the score of a panel mixture on single choices should therefore be reported alongside its fit, which requires only one further evaluation of the likelihood.
Related papers
- Stationarity with Occasionally Binding Constraints
- Backtesting High-Dimensional Expected Shortfall
- Higher-Order Neyman Orthogonality in Moment-Condition Models
- Online Bayesian Inference for a Dirichlet Process Mixture of Binary Logit Models
- Debiased Conditional Independence Tests for Causal Effects
- Sharp Testable Implications of Encouragement Designs