Model fit vs prediction performance: contrasting fixed-coefficient models with mixture models

summary

The gist

Models allowing for random heterogeneity, such as mixed logit and latent class, are generally observed to obtain superior model fit and yield detailed insights into unobserved preference…

In short

The study compared model fit during estimation with actual predictive performance and market share recovery using mixture models (like mixed logit and latent class) against simpler fixed-coefficient models. While mixture models capture preference heterogeneity well, they don't always lead to better forecasts. Using conditional distributions for individuals already in the sample significantly improves prediction accuracy for their new choices.

Key concepts

Model Fit vs. Prediction Performance
This examines whether a model that accurately describes the data during estimation (model fit) actually performs well when predicting future choices (prediction performance). The paper finds that good fit does not guarantee good prediction, especially with person-level mixing in mixture models.
Random Heterogeneity
This refers to the idea that individuals have different underlying preferences or characteristics. Mixture models are used to capture this unobserved heterogeneity, which helps explain why some people choose certain options while others do not. This heterogeneity is key for understanding individual behavior.
Conditional Distributions
These are probability distributions based on previous choices made by a specific individual. Using these conditionals allows analysts to reduce uncertainty when forecasting the next choice for that known person, offering substantial gains in prediction accuracy.

Terminology used across episodes

This episode discusses

The paper

Model fit vs prediction performance: contrasting fixed-coefficient models with mixture models · Read on arXiv

University of Leeds · Delft University of Technology

Mixture models such as mixed logit and latent class routinely achieve large gains in fit on panel data, yet several studies have found that these gains do not carry over to out-of-sample prediction. This paper explains why, building on the distinction between population-level and person-level prediction familiar from statistics. The panel log-likelihood of a mixture is exactly the cumulative score of forecasts that condition each choice on the person's earlier choices. Models are instead usually validated and applied with unconditional forecasts, which use only the estimated distribution of tastes and are scored by the cross-sectional log-likelihood. The difference between the two log-likelihoods, which we call choice consistency, comes from reproducing the dependence between a person's choices, and unconditional forecasts cannot use it. Decomposing the fit gain into choice consistency and heterogeneity, we show that when the assumed mixing distribution cannot reproduce this dependence exactly, pursuing choice consistency distorts the estimated heterogeneity, so that a mixture may predict single choices worse than a model without mixing. In two case studies, on revealed preference and stated choice data, choice consistency accounts for more than 90% of the fit gain for each of the fourteen panel mixtures. In our data, the part of the fit gain that helps to forecast single choices shrinks as people make more choices, and with many choices per person it becomes negative, leaving the mixture worse than a model without mixing. Mixtures forecast single choices clearly better only when the forecast can condition on the person's own earlier choices, which is often not possible in practice. For forecasting, the score of a panel mixture on single choices should therefore be reported alongside its fit, which requires only one further evaluation of the likelihood.

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Model fit vs prediction performance".

Warren: Here's the gist of the paper. Models allowing for random heterogeneity, such as mixed logit and latent class, are generally observed to obtain superior model fit and yield detailed insights into unobserved preference heterogeneity but these advantages do not translate into any benefits in forecasting whether looking at prediction performance or the recovery of market shares The only exception arises when using conditional distributions in making predictions for the same individuals included in the estimation sample which obviously precludes any out-of-sample forecasting.

Ruth: Theoretical Disconnect between Estimation and Application Decreased prediction performance compared to fit in estimation may happen for two reasons, one relates to the model's inability to generalize whether this relates to changes in attribute combinations (x) or preferences (β), and the second is that the way in which mixture models are specified leads to a disconnect between model estimation and model application In continuous mixture models with integration at the person level, Equation 3 does not apply because in estimation, we look at the likelihood of entire sequence of choices conditional on a given value of β before then integrating over the distribution of β Similarly, with latent class models specified at the person level, Equation 3 does not apply because class membership remains latent and heterogeneity is averaged out in both estimation and application.

Warren: The Role of Conditional Distributions There is one notable exception when the analyst has access to previous choices of an individual and wishes to make forecasts for that same individual, then we can reduce the uncertainty in relation to a person’s location on the sample level distribution of β by relying on posterior (or conditional) parameter distributions This procedure is described in Train (2009, ch.

Ruth: 11.2) for continuous mixture models and has not seen widespread use.

Title and authors: Ruth: So, let's start by looking at the title and who wrote this—"Model fit vs prediction performance: contrasting fixed-coefficient models with mixture models." It sets up a direct comparison between simpler, fixed-coefficient models and these more flexible mixture versions.

Warren: And the authors are Stephane Hess and Sander van Cranenburgh. They’re using revealed preference data from decisions and stated choice data from COVID-nineteen to test this idea.

Iris: The core of their research is showing that increased flexibility in the model, through random heterogeneity like mixing, doesn't always lead to better predictions or market share recovery.

Ruth: Exactly. The paper highlights a disconnect between estimation and application when using mixture models unless you use conditional distributions in specific ways for people already in the sample.

Warren: That's a big point because it suggests that just chasing the highest fit score on historical data might be misleading if your goal is actually predicting what happens next.

The paper's summary: Ruth: The paper summarizes their findings by explaining two main reasons why prediction performance can drop even when the model fit in estimation is high.

Warren: First, it relates to the model's inability to generalize, whether that’s about changes in the attributes you are looking at or changes in the actual preferences of people.

Iris: The second reason they focus on is a disconnect between how these mixture models are set up and how we actually use them for prediction.

Ruth: They explain that in continuous mixture models with integration at the person level, things like Equation three don't apply when you're estimating because you look at the likelihood of a whole sequence of choices conditional on a fixed preference value before integrating over the distribution of preferences <ref:2510.09185#pg2>.

Warren: And for latent class models specified at the person level, class membership stays latent, so that heterogeneity gets averaged out in both estimation and application.

The paper's improvements: Ruth: The paper then suggests a couple of ways to improve things when it comes to making predictions for individuals who were already in the estimation sample.

Warren: They point out that the exception they found is using posterior or conditional parameter distributions when you have access to previous choices of that specific individual.

Iris: This procedure, which is described in Train two thousand nine chapter eleven point two, lets you reduce uncertainty about a person’s location on the sample distribution of preferences by relying on those conditional distributions.

Ruth: So, instead of just using the point estimate from the estimation sample for that person's preference, you use this conditional distribution to calculate the probability of them choosing an alternative in a situation.

Warren: This is specifically useful for predicting new choices for people who were already included in the estimation data; they see very substantial gains in prediction accuracy when using these conditionals.

Conclusion: Ruth: So, to wrap up, the authors show that mixture model fit in estimation isn't a guarantee of good prediction performance if you’re not careful about how you apply the model.

Warren: They emphasize that including covariates generally improves prediction performance across both case studies, with the only real exception being the use of conditionals for making predictions about individuals already seen.

Iris: They conclude that while mixture models are powerful for understanding preference heterogeneity, if your main goal is forecasting, especially out-of-sample forecasting, it might be wiser to focus on refining and optimizing a model with only deterministic heterogeneity.

Ruth: So the paper suggests a trade-off: use mixtures for insight into preferences but maybe stick to simpler models if the primary aim is accurate prediction for things you haven't seen yet.

Warren: And they make sure to stress that when looking at market share recovery, using conditionals doesn't necessarily lead to a lower bias compared to other methods on out-of-sample data.

More episodes

← Home