Simulation in discrete choice models evaluation: SDCM, a simulation tool for performance evaluation of DCMs

summary

The gist

Discrete choice models are widely used in economics, transportation, and marketing to explain and predict how decision makers choose among a finite set of alternatives <ref:2407.17014#pg6> because…

In short

This paper develops a simulation tool to generate synthetic stated choice data that follows a panel mixed multinomial logit model with known parameters. This allows researchers to evaluate the performance of discrete choice models by comparing them against simulated data, measuring metrics like bias and RMSE, and testing estimator recovery.

Key concepts

Discrete Choice Models (DCMs)
These models describe how a decision-maker chooses one option from several mutually exclusive alternatives based on an underlying behavioral theory. The Multinomial Logit (MNL) is a common version, but the Mixed Multinomial Logit (MMNL) relaxes its restrictions by allowing tastes to vary across different people.
Experimental Design and Simulation Tool
The tool combines experimental design principles with a behavioral model of artificial agents. It generates choice data for many artificial respondents facing specific choice situations based on a defined design, proving that the resulting choices adhere to a known panel MMNL model.
Performance Evaluation Metrics
The simulator defines measures to assess how well an estimator performs. These include bias, root mean squared error (RMSE), and the coverage of confidence intervals. Comparing these metrics on simulated data with real data helps reveal finite-sample bias or model misspecification.
Panel Mixed Multinomial Logit (MMNL) Model
This is the specific statistical model that describes the synthetic choice data generated by the simulator. It allows for variation in preferences among different decision-makers, which is a key feature that distinguishes it from simpler models like MNL.

Terminology used across episodes

This episode discusses

The paper

Simulation in discrete choice models evaluation: SDCM, a simulation tool for performance evaluation of DCMs · Read on arXiv

Amirreza Talebia

Department of Integrated Systems Engineering, The Ohio State University

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Simulation in discrete choice models evaluation".

Warren: Here's the gist of the paper. Discrete choice models are widely used in economics, transportation, and marketing to explain and predict how decision makers choose among a finite set of alternatives Because real experiments raise confidentiality concerns, take time, and are costly, controlled experimentation with real respondents is often impractical This paper connects discrete choice modeling, experimental design, and simulation The paper formalizes a simulation tool that combines an experimental design with a behavioral model of artificial agents to generate synthetic stated choice data It proves that the generated choices follow a panel mixed multinomial logit model with known parameters Theoretical Framework and Model Review Discrete choice models (DCMs) describe how a decision maker selects one alternative from a finite set of mutually exclusive alternatives, and they do so on the basis of an explicit behavioral theory The dominant theory is random utility maximization, where each alternative yields a utility that the analyst observes only in part The multinomial logit (MNL) model is the workhorse among RUMs because its choice probabilities have a closed form Models that relax these restrictions, such as the mixed multinomial logit (MMNL) and the multinomial probit models, have choice probabilities that are multidimensional integrals without a closed form The MMNL model removes the restrictions of the MNL model by letting tastes vary across decision makers Experimental Design and Simulation Tool An experimental design specifies the independent, dependent, and control variables of an experiment together with its randomization and statistical procedures Choice situations have traditionally been built from orthogonal designs Efficient designs minimize a scalar function of the AVC matrix The simulator generates data for N artificial respondents who face the S choice situations of a design X, with J alternatives per situation Proposition 2 shows that the resulting choices follow a panel MMNL model with known parameters Performance Evaluation and Discrepancy The simulator defines measures of estimator performance, namely the bias, the root mean squared error (RMSE), the coverage of confidence intervals, and the recovery of the WTP The discrepancy between simulated and real choice data is measured by the average Kullback-Leibler divergence Departures from these benchmarks reveal finite-sample bias, simulation bias caused by a small R, or misspecification Limitations and Future Directions Simulations provide valuable approximations, but they cannot fully capture the complexity of human behavior Emotional influences, social norms, bounded rationality, learning, and time pressure can separate simulated data from real-world data Future research should integrate the RRM rule into the simulator and add terms for scale, drift, and social interaction to study their effect on estimates The tool can also be used to compare the performance of DCMs with other classification models such as artificial neural networks Key Contributions The paper makes four contributions, including reviewing RUMs and RRM models, formalizing an agent-based simulation tool as a data-generating process, defining performance measures for estimators, and illustrating the tool with a stated choice experiment The main contribution is a formal description of a simulation tool implemented as an R package that generates synthetic SC data for DCMs Proposition 2 shows that the generated data follow a panel MMNL model with known parameters The simulator can also be confronted with real data collected with the same design, and the discrepancy between them is measured by the average Kullback-Leibler divergence Case Study Illustration The tool is illustrated with the real choice experiment of

15: , which examined consumers’ WTP and the price premium for two environmental attributes of roses The simulator rebuilds this design from the full factorial of the two binary attributes The systematic utilities are defined by a complex expression involving socioeconomic characteristics and attribute levels The WTP of respondent n for the label on a rose without the carbon attribute is therefore WTPn,L = −βn,L/βPrice, and if βn,L ∼ N (µL, σ2 L), then WTPn,L ∼ N −µL/βPrice,σ2 L/β2 Price The simulator imposes λs = 1, δs = 0, and ηnsj = 0 The simulator generates data in four steps, including drawing socioeconomic characteristics independently across respondents and characteristics The parameters of each respondent are drawn once and kept for all S situations, where βn ∼ N (µ, Σ) The utility of alternative j in situation s is Unsj = Vnsj + εnsj with Vnsj = β⊤n xnsj Finally, each respondent chooses the alternative with the highest utility The paper concludes by showing how to measure the performance of an estimator exactly because the parameters θ 0 of the simulated data are known The bias, RMSE, and coverage of nominal 95% confidence intervals are measured across M replications Comparing the simulated RMSE across sample sizes with this prediction gives an empirical counterpart of the sample size rule The observed value should therefore be compared with the distribution of ∆ obtained when the real shares are replaced by the shares of independent simulated samples References

Title and authors: Ruth: Let's talk about the paper, "Simulation in discrete choice models evaluation: SDCM," and what that title actually means for us right now. It’s basically proposing a new way to evaluate if our statistical models are actually working in the real world >

Warren: So it’s not just about building a fancy model; it's about building a rigorous testing environment where we can know exactly what the true underlying parameters are before we even start looking at messy real data >

Iris: I think the title highlights that they are using simulation to evaluate discrete choice models, which is a common way to test them, but they’re adding an experimental design component on top >

Ruth: Right. They're showing how you can take a well-designed experiment and feed it into a model of artificial agents to generate data that perfectly matches the model's assumptions >

Warren: So if we can generate perfect data under known conditions, then we get a benchmark against which we can measure the performance of our estimators when those parameters are unknown >

Iris: That seems like a solid foundation for testing identification and other statistical properties because you’ve controlled the environment as much as possible >

Ruth: Exactly. It moves us from just fitting a curve to actually understanding the estimation error in a way that respects the structure of discrete choice models, like random utility maximization >

Warren: So it’s less about predicting consumer behavior and more about testing how robust our statistical tools are when applied to stated choice data >

Iris: And they formalize this connection between experimental design, simulation, and the behavioral model of agents very clearly in the abstract >

The paper's summary: Ruth: Okay, so diving into the summary of "Simulation in discrete choice models evaluation: SDCM," it boils down to how they create a tool that generates synthetic stated choice data based on an experimental design >

Warren: They take a design X and generate data for N artificial respondents facing J alternatives per situation, and they show this output follows a panel mixed multinomial logit model with known parameters >

Iris: That’s the core result. Knowing the true parameters is what lets them define performance measures like bias and root mean squared error precisely because they can calculate those quantities exactly >

Ruth: They then define these performance measures—bias, root mean squared error, coverage of nominal ninety-five percent confidence intervals, and recovery of willingness to pay—and they test them by comparing the simulated data against real choice data using the average Kullback-Leibler divergence >

Warren: It’s a way to quantify exactly how much our simulation deviates from reality, which helps diagnose if our simulation is biased or if we have issues with parameter estimation itself >

Iris: They also use a case study, like the one on consumers' willingness to pay for two environmental attributes of roses, where they rebuild the design from a full factorial of two binary attributes >

Ruth: That example shows them applying it to a concrete situation where they define systematic utilities using socioeconomic characteristics and attribute levels in that specific way >

Warren: So it’s moving beyond just theoretical modeling and giving us a practical method for testing how well we can estimate preferences from stated choice data when experiments are out of reach >

The paper's improvements: Ruth: Now, let's look at the improvements they suggest for this simulation tool. They point out that current simulations are limited because they don't fully capture human complexity >

Warren: So what’s the suggested improvement then? It seems like they want to incorporate more of the real-world stuff into their agents, like emotional influences or social norms >

Iris: They suggest integrating the random regret rule into the simulator and adding terms for scale, drift, and social interaction to study how those factors affect our estimates >

Ruth: So they want a richer behavioral model where agents aren't just making choices based on fixed utility but are also reacting to things like social pressure or how their preferences change over time >

Warren: That would make the simulation much more realistic, and it would give us better insights into how those extra factors might inflate or dampen our estimates when we run them against real data >

Iris: And on a practical note, they also say this tool can be used to compare the performance of discrete choice models with other classification models like artificial neural networks >

Ruth: That’s useful for checking if our chosen model is actually the best fit for the data, not just because it’s mathematically elegant >

Conclusion: Ruth: So wrapping up on "Simulation in discrete choice models evaluation: SDCM," the main implication is that we have a formal way to test estimator performance when real experiments are impractical >

Warren: It gives us a concrete benchmark using synthetic data with known parameters, allowing us to measure bias and error metrics in a controlled environment >

Iris: And it shows that this approach connects experimental design, simulation, and the behavioral model in a very systematic way for testing statistical properties of choice models >

Ruth: The paper's main contribution is formalizing this simulation tool as an agent-based data-generating process that generates synthetic stated choice data following a panel mixed multinomial logit model with known parameters >

Warren: And the key finding, proposition two, is that these generated choices actually follow that specific model, which validates the entire framework of using simulation for performance evaluation >

Iris: And they show we can compare simulated results against real data using the average Kullback-Leibler divergence to see exactly where our estimates are falling short >

Ruth: It’s a powerful tool for researchers who need to plan studies before they run them, or for those who want a more detailed understanding of estimation error than just looking at standard metrics >

Warren: It’s a strong step in bridging the gap between theoretical discrete choice models and the messy reality of real-world data collection >

Iris: We'll keep an eye on how they incorporate those future suggestions about social interaction and drift because that’s where the next layer of complexity lies >

More episodes

← Home