Decision Models as Scalable Economic Surrogates that Capture Human Variation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Decision Models as Scalable Economic Surrogates that Capture Human Variation".
Warren: Here's the gist of the paper. Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options; these properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour The paper presents a first evaluation of DMs as human surrogates in economics, comparing 10 DMs with 8 LLMs across three economic settings: a household spending survey on consumption, several classic economic games, and a labour-market field experiment.
Ruth: Frontier DMs reproduce the variation in human responses more closely than LLMs in all three settings Together, these results suggest that DMs capture the heterogeneity of human behaviour better than LLMs and make large-scale economic simulations more feasible.
Warren: Comparison of Model Performance Frontier DMs are shown to capture the variation in human behaviour better than LLMs, and they do so faster and more cheaply In the consumption survey, frontier DMs capture the modal human response as well as LLMs, although on the economic games, LLMs represent the modal human response better.
Ruth: In the labour experiment, frontier DMs reproduce the gradual shift in human choices as a wage gap changes, whereas LLMs switch abruptly from one job to the other.
Paper summary: Ruth: So we've talked about how Decision Models are compared to Large Language Models in this paper, focusing on the idea that Frontier DMs capture human variation better than LLMs across consumption surveys, economic games, and labour experiments <ref:2611.05553#pg3>.
Warren: And we covered the practical side of things—that these models are significantly faster and cheaper to run than LLMs, which means they open up possibilities for scaling up agent-based simulations by a factor of ten or more <ref:2611.05553#pg4>.
Iris: The authors' main point in "Decision Models as Scalable Economic Surrogates that Capture Human Variation" is that these DMs provide a scalable human surrogate that captures the variation in human data better than LLMs <ref:2611.05553#pg1>.
Ruth: So, to wrap up, the paper suggests that Decision Models are a significant advantage over LLMs when you need to simulate large economic surveys or computational simulations with AI agents because they capture the heterogeneity of human behavior more accurately <ref:2611.05553#pg4>.
Warren: It means we can build bigger, more realistic economic models without having to deal with the massive costs and limitations that come with using LLMs for this kind of work <ref:2611.05553#pg4>.
Iris: And they do acknowledge that DMs are not perfect because they were trained after the data was collected, and they need more work to fine-tune them on real human choice data to get even closer to true behavior <ref:2611.05553#pg4>.
Ruth: That’s the caveat, right? The paper shows that DMs are a powerful and scalable tool for modeling human behavior where capturing variation matters <ref:2611.05553#pg3>.
Conclusion: Ruth: So we’ve seen how these Decision Models, or DMs, are being tested against Large Language Models in some real economic stuff <ref:2611.05553#pg3>.
Warren: And what we're seeing is that these DMs are way faster and cheaper to run than those LLMs we usually rely on <ref:2611.05553#pg4>.
Iris: The core idea here, as the paper calls it, is that Decision Models offer a scalable human surrogate that captures the variation in human data better than LLMs <ref:2611.05553#pg1>.
Ruth: That means these models are more faithful to how real people actually behave in economic scenarios than just generating text <ref:2611.05553#pg2>.
Warren: They’re showing that this efficiency could let us scale up agent-based simulations by a factor of ten or more <ref:2611.05553#pg4>.
Iris: But the authors do flag a limitation, so we need to be careful, they trained these models after the data was already collected <ref:2611.05553#pg4>.
Ruth: They say future work needs to fine-tune these Decision Models explicitly on real human choice data to adapt those behavioral calibration techniques <ref:2611.05553#pg4>.
Warren: So, the big picture here is that DMs provide a significant advantage when you need to run huge economic surveys or simulations with AI agents because they capture how people actually vary <ref:2611.05553#pg4>.
Iris: It’s about making those massive simulations feasible without having to deal with the sheer cost of using LLMs for that kind of work <ref:2611.05553#pg4>.
Ruth: This paper suggests we might be able to build much larger economic models that actually reflect human heterogeneity, not just a smooth average <ref:2611.05553#pg3>.
Warren: That opens up some serious possibilities for how we test economic theories at scale <ref:2611.05553#pg4>.
Iris: Next up, we’re going to look at how this performance actually compares on the specific tasks they used, like those consumption surveys and economic games <ref:2611.05553#pg3>.
Maxime Saxena, Fabio Caccioli, R. Maria del Rio-Chanona
University College London · London School of Economics and Political Science · Google DeepMind
econ.GN, cs.AI, cs.MA, q-fin.EC
Submitted: 2026-11-03
Updated: 2026-11-03
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options…
Key concepts
- Decision Models (DMs)
- A new class of AI models designed to give fast responses by returning a probability distribution over a fixed set of options. They are being tested as efficient, scalable surrogates for human decision-making in economic scenarios.
- Large Language Models (LLMs)
- AI models that are currently used as human surrogates but are costly at scale and tend to under-represent the full range of human behavior variation. DMs aim to overcome these limitations through speed and cost efficiency.
- Distributional Measures
- Mathematical tools like Wasserstein-1 distance and KL divergence used to measure how well a model's output matches the actual spread and typical answers found in real human data. Lower distances indicate a better fit.
- Scalability Gains
- The efficiency of DMs—being 10 to 100 times faster than LLMs—allows existing agent-based simulations to scale significantly, enabling researchers to model much larger populations with fewer computational resources.
Terminology
Summary
Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options; these properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour The paper presents a first evaluation of DMs as human surrogates in economics, comparing 10 DMs with 8 LLMs across three economic settings: a household spending survey on consumption, several classic economic games, and a labour-market field experiment. p. 0 These properties make them
candidate human surrogates in economic simulations and surveys, a role increasingly
filled by Large Language Models (LLMs), which are costly at scale and under-represent
the variation in human behaviour. We present a first evaluation of DMs as human
surrogates in economics, comparing 10 DMs with 8 LLMs in three economic settings:
a household spending survey on consumption, several classic economic games and a
labour-market field experiment. Frontier DMs are
also roughly ten times faster than the fastest LLM and can be over a hundred times
cheaper. Together, these results suggest that DMs capture the heterogeneity of human
behaviour better than LLMs and make large-scale economic simulations more feasible.Where the paper says it
Where the paper says it
Comparison of Model Performance
Frontier DMs are shown to capture the variation in human behaviour better than LLMs, and they do so faster and more cheaply In the consumption survey, frontier DMs capture the modal human response as well as LLMs, although on the economic games, LLMs represent the modal human response better. In the labour experiment, frontier DMs reproduce the gradual shift in human choices as a wage gap changes, whereas LLMs switch abruptly from one job to the other. p. 2 In the consumption survey, they also capture the
modal human response as well as LLMs, although on the economic games, LLMs represent
the modal human response better. In the labour experiment, frontier DMs reproduce the
gradual shift in human choices as a wage gap changes, whereas LLMs switch abruptly from
one job to the other.Where the paper says it
Distributional Measures and Fidelity
The paper uses two measures of distributional fit throughout the paper: Wasserstein-1 distance and Kullback-Leibler (KL) divergence. A lower Wasserstein-1 distance denotes a closer agreement with the target human distribution, and closer to 0 is a better fit for KL divergence. p. 6 Where appropriate, we check these results
also hold with Kullback-Leibler (KL) divergence. A lower Wasserstein-1 distance denotes
a closer agreement with the target human distribution, and we normalise over the width
of the range of answers to make all answers comparable. However, GLM Flash and Argon both fit the human mode and
distribution well, although they are still statistically significantly worse than frontier DMs
on the KL divergence metric. On this task, frontier DMs (and Luna Decision) are therefore
the most faithful human surrogates we test, matching both the spread of human answers
and the typical answer. However, GLM Flash and Argon both fit the human mode and
distribution well, although they are still statistically significantly worse than frontier DMs
on the KL divergence metric. On this task, frontier DMs (and Luna Decision) are therefore
the most faithful human surrogates we test, matching both the spread of human answers
and the typical answer.Where the paper says it
Where the paper says it
Where the paper says it
Scalability and Efficiency Gains
Decision Models are faster and cheaper than LLMs across all three settings considered Frontier DMs are around 10 to 100 times faster than LLMs, which allows existing agent-based simulations to scale by an order of magnitude. This efficiency suggests that frontier DMs offer a significant advantage compared to LLMs in scaling up economic surveys and computational simulations with AI agents. p. 0 Together, these results suggest that DMs capture the heterogeneity of human
behaviour better than LLMs and make large-scale economic simulations more feasible. 1 Introduction
AI agents are increasingly used as human surrogates in surveys or agent-based simulations,
with recent applications spanning political preferences [Argyle et al.Where the paper says it
Robustness and Limitations
The results are robust to minor changes, such as changing the wording of one question or asking models to “act like a human”. However, limitations exist because both LLMs and DMs were trained after the publication of the datasets and experiments used. p. 1 They
therefore provide a much-needed alternative to the unrealistic behavioural assumptions of the
status quo, enabling simulations that can generate more reliable answers to policy questions. However, using LLMs as human surrogates faces two major limitations.Where the paper says it
Conclusion
Frontier DMs offer a scalable human surrogate which better captures the variation in human data. This efficiency allows existing agent-based simulations to scale by an order of magnitude, for example, expanding populations from 1 million to 10 million agents. p. 2 Some progress has been made on scaling LLMbased ABMs with the number of agents going over the million mark, although this remains
the exception, not the rule. For example, Chopra et al. Together, these results suggest that DMs capture the heterogeneity of human
behaviour better than LLMs and make large-scale economic simulations more feasible. 1 Introduction
AI agents are increasingly used as human surrogates in surveys or agent-based simulations,
with recent applications spanning political preferences [Argyle et al.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options; these properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour.
How the training signal is built
DMs are finetuned to take as input natural language questions and scenarios and make decisions between provided options, rather than generating free-form text They are also faster and cheaper than LLMs, potentially allowing simulations with orders of magnitude more agents.
Where the gains come from
Frontier DMs, those that lead on standard decision-making benchmarks, reproduce the variation in human responses more closely than LLMs in all three settings They capture the modal human response as well as LLMs in the spending survey, but less well in the economic games.
The paper uses two measures of distributional fit throughout the paper: Wasserstein-1 distance and Kullback-Leibler (KL) divergence A lower Wasserstein-1 distance denotes a closer agreement with the target human distribution, and closer to 0 is a better fit for KL divergence. Frontier DMs are found to be the most faithful human surrogates tested on the consumption task, matching both the spread of human answers and the typical answer.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Fine-tune Decision Models explicitly on empirical human choice data to adapt behavioural calibration techniques previously applied to LLMs, as
frontier models often become hyper-rational on economic benchmarks, making them less representative of boundedly rational humans.
-
Implement a system for scaling agent-based simulations by utilizing Decision Models for decisions such as
which occupations workers apply to
orhow suppliers ration scarce output,
where DMs can make choices in nested calls, potentially scaling simulations from 8.4 million agents to around 80 million agents. -
Develop a mechanism where Decision Models provide structured choice probabilities rather than free-form text generation, enabling them to assign
a probability for each option supplied in one forward pass
which is crucial for generating variation in simulated decisions across large populations.
Abstract
Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options. These properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour. We present a first evaluation of DMs as human surrogates in economics, comparing 10 DMs with 8 LLMs in three economic settings: a household spending survey on consumption, several classic economic games and a labour-market field experiment. Frontier DMs, those that score highest on general decision-making benchmarks, reproduce the variation in human responses more closely than LLMs in all three settings. They capture the modal human response as well as LLMs in the spending survey, but less well in the economic games. Frontier DMs are also roughly ten times faster than the fastest LLM and can be over a hundred times cheaper. Together, these results suggest that DMs capture the heterogeneity of human behaviour better than LLMs and make large-scale economic simulations more feasible.
Sources
- Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV
- A Generative Model of Conspicuous Consumption and Status Signaling
- Can Generative AI agents behave like humans? Evidence from laboratory market experiments
- General Decision Models: Benchmarking and Insights Beyond Jev
- SimCity: Multi-Agent Urban Development Simulation with Rich Interactions
- Heterogeneous RBCs via Deep Multi-Agent Reinforcement Learning
- Releasing the Inner Jev: A Simple Recipe for Open Decision Models
- LLM Agents Do Not Replicate Human Market Traders: Evidence From Experimental Finance
- Jevolution: A Framework for Studying Communication and Collective Dynamics in AI Agents
- Evaluating Decision Models for Text Annotation in Computational Social Science
- LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings
- AI Simulations Grounded in Rich Self-Reports Increase Accuracy and Reduce Distortion
- Machine Spirits: Speculation and Adaptation of LLM Agents in Asset Markets
- Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia
- OASIS: Open Agent Social Interaction Simulations with One Million Agents
- Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits
- NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision Trees
- AnyJev Technical Report
Related papers
- Huge If True: Learning from the Size of the Surprise
- The Rebundling of Freelance Work in the AI Age
- Dynamic investment and coordination in teams: Theory and experimental evidence
- Does Frontier AI Strengthen Cyber Defense? Evidence from Project Glasswing
- Generative AI at Work
- What is Work For? Labor, Politics, and Meaning in the Age of AI