Decision Models as Scalable Economic Surrogates that Capture Human Variation

summary

The gist

Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options…

In short

The study compared Decision Models (DMs) against Large Language Models (LLMs) as surrogates for human behavior in economics, testing them across surveys, games, and experiments. Frontier DMs proved superior because they capture the variation in human responses more accurately than LLMs. This makes DMs faster and cheaper for large-scale economic simulations.

Key concepts

Decision Models (DMs)
A new class of AI models designed to give fast responses by returning a probability distribution over a fixed set of options. They are being tested as efficient, scalable surrogates for human decision-making in economic scenarios.
Large Language Models (LLMs)
AI models that are currently used as human surrogates but are costly at scale and tend to under-represent the full range of human behavior variation. DMs aim to overcome these limitations through speed and cost efficiency.
Distributional Measures
Mathematical tools like Wasserstein-1 distance and KL divergence used to measure how well a model's output matches the actual spread and typical answers found in real human data. Lower distances indicate a better fit.
Scalability Gains
The efficiency of DMs—being 10 to 100 times faster than LLMs—allows existing agent-based simulations to scale significantly, enabling researchers to model much larger populations with fewer computational resources.

Terminology used across episodes

This episode discusses

The paper

Decision Models as Scalable Economic Surrogates that Capture Human Variation · Read on arXiv

Maxime Saxena, Fabio Caccioli, R. Maria del Rio-Chanona

University College London · London School of Economics and Political Science · Google DeepMind

Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options. These properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour. We present a first evaluation of DMs as human surrogates in economics, comparing 10 DMs with 8 LLMs in three economic settings: a household spending survey on consumption, several classic economic games and a labour-market field experiment. Frontier DMs, those that score highest on general decision-making benchmarks, reproduce the variation in human responses more closely than LLMs in all three settings. They capture the modal human response as well as LLMs in the spending survey, but less well in the economic games. Frontier DMs are also roughly ten times faster than the fastest LLM and can be over a hundred times cheaper. Together, these results suggest that DMs capture the heterogeneity of human behaviour better than LLMs and make large-scale economic simulations more feasible.

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Decision Models as Scalable Economic Surrogates that Capture Human Variation".

Warren: Here's the gist of the paper. Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options; these properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour The paper presents a first evaluation of DMs as human surrogates in economics, comparing 10 DMs with 8 LLMs across three economic settings: a household spending survey on consumption, several classic economic games, and a labour-market field experiment.

Ruth: Frontier DMs reproduce the variation in human responses more closely than LLMs in all three settings Together, these results suggest that DMs capture the heterogeneity of human behaviour better than LLMs and make large-scale economic simulations more feasible.

Warren: Comparison of Model Performance Frontier DMs are shown to capture the variation in human behaviour better than LLMs, and they do so faster and more cheaply In the consumption survey, frontier DMs capture the modal human response as well as LLMs, although on the economic games, LLMs represent the modal human response better.

Ruth: In the labour experiment, frontier DMs reproduce the gradual shift in human choices as a wage gap changes, whereas LLMs switch abruptly from one job to the other.

Paper summary: Ruth: So we've talked about how Decision Models are compared to Large Language Models in this paper, focusing on the idea that Frontier DMs capture human variation better than LLMs across consumption surveys, economic games, and labour experiments <ref:2611.05553#pg3>.

Warren: And we covered the practical side of things—that these models are significantly faster and cheaper to run than LLMs, which means they open up possibilities for scaling up agent-based simulations by a factor of ten or more <ref:2611.05553#pg4>.

Iris: The authors' main point in "Decision Models as Scalable Economic Surrogates that Capture Human Variation" is that these DMs provide a scalable human surrogate that captures the variation in human data better than LLMs <ref:2611.05553#pg1>.

Ruth: So, to wrap up, the paper suggests that Decision Models are a significant advantage over LLMs when you need to simulate large economic surveys or computational simulations with AI agents because they capture the heterogeneity of human behavior more accurately <ref:2611.05553#pg4>.

Warren: It means we can build bigger, more realistic economic models without having to deal with the massive costs and limitations that come with using LLMs for this kind of work <ref:2611.05553#pg4>.

Iris: And they do acknowledge that DMs are not perfect because they were trained after the data was collected, and they need more work to fine-tune them on real human choice data to get even closer to true behavior <ref:2611.05553#pg4>.

Ruth: That’s the caveat, right? The paper shows that DMs are a powerful and scalable tool for modeling human behavior where capturing variation matters <ref:2611.05553#pg3>.

Conclusion: Ruth: So we’ve seen how these Decision Models, or DMs, are being tested against Large Language Models in some real economic stuff <ref:2611.05553#pg3>.

Warren: And what we're seeing is that these DMs are way faster and cheaper to run than those LLMs we usually rely on <ref:2611.05553#pg4>.

Iris: The core idea here, as the paper calls it, is that Decision Models offer a scalable human surrogate that captures the variation in human data better than LLMs <ref:2611.05553#pg1>.

Ruth: That means these models are more faithful to how real people actually behave in economic scenarios than just generating text <ref:2611.05553#pg2>.

Warren: They’re showing that this efficiency could let us scale up agent-based simulations by a factor of ten or more <ref:2611.05553#pg4>.

Iris: But the authors do flag a limitation, so we need to be careful, they trained these models after the data was already collected <ref:2611.05553#pg4>.

Ruth: They say future work needs to fine-tune these Decision Models explicitly on real human choice data to adapt those behavioral calibration techniques <ref:2611.05553#pg4>.

Warren: So, the big picture here is that DMs provide a significant advantage when you need to run huge economic surveys or simulations with AI agents because they capture how people actually vary <ref:2611.05553#pg4>.

Iris: It’s about making those massive simulations feasible without having to deal with the sheer cost of using LLMs for that kind of work <ref:2611.05553#pg4>.

Ruth: This paper suggests we might be able to build much larger economic models that actually reflect human heterogeneity, not just a smooth average <ref:2611.05553#pg3>.

Warren: That opens up some serious possibilities for how we test economic theories at scale <ref:2611.05553#pg4>.

Iris: Next up, we’re going to look at how this performance actually compares on the specific tasks they used, like those consumption surveys and economic games <ref:2611.05553#pg3>.

More episodes

← Home