Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections

summary

The gist

In this study, large language models (LLMs) are examined for their potential to replicate human responses in large-scale surveys and predict election outcomes using a novel method called…

In short

Researchers developed Matching-LLM, a method combining LLMs' reasoning ('System 2') with historical data ('System 1'). This approach generates synthetic survey responses and predicts election outcomes by weighting LLM outputs with real human data. The findings show Matching-LLM outperforms traditional methods in both predicting public opinions and forecasting elections.

Key concepts

Matching-LLM
A novel method that merges the logical reasoning of large language models (System 2) with the intuition derived from historical human data (System 1). It creates synthetic data that better mimics real human behavior in surveys and predictions by combining both cognitive styles.
Survey Prediction
The task of using LLMs to generate responses for large-scale surveys based on demographic profiles. Matching-LLM is used here to create synthetic answers that align closely with actual human survey results, capturing cultural differences between groups like the U.S. and China.
Out-of-Sample Forecasting
Predicting future events, specifically election outcomes, using a model trained on historical data. Matching-LLM improves this by applying specific weighting schemes derived from in-sample predictions to create more accurate forecasts for new or unseen elections.
System 1 and System 2 Thinking
This dual-process theory describes human cognition: System 1 is fast, intuitive thinking based on tradition and emotion (historical data). System 2 is slow, deliberate reasoning used for complex problem-solving (LLM's reasoning). Matching-LLM aims to blend these two for better modeling.

Terminology used across episodes

This episode discusses

The paper

Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections · Read on arXiv

Shapeng Jiang, Lijia Wei, Chen Zhang

Wuhan University

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections".

Warren: Here's the gist of the paper. In this study, large language models (LLMs) are examined for their potential to replicate human responses in large-scale surveys and predict election outcomes using a novel method called Matching-LLM, which combines historical data with synthetic LLM data to improve predictive accuracy The findings suggest that the Matching-LLM approach outperforms traditional LLM approaches in both survey response prediction and election outcome forecasting.

Ruth: Large language models are powerful tools for predicting human behavior due to their ability to capture the complexity of natural language and encode a wide range of human experiences, cultural norms, and decision-making patterns from extensive training data This is particularly valuable in social science research where experiments and surveys rely heavily on human participants to gather behavioral data.

Warren: Specifically, using LLMs in behavioral predictions allows researchers to scale experiments, lower the costs of human subject studies, and gain insights into individual behavior and complex social interactions such as trust, negotiation, and cooperation.

Ruth: Previous literature has primarily focused on using large language models (LLMs) for prediction tasks relying on training data derived from “rational” materials such as general web text, books, and articles However, the current development trajectory of LLMs is increasingly emphasizing reasoning and rationality which fundamentally differs from the core requirements of public opinion prediction.

Title and authors: Ruth: So we're talking about this paper called "Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections." It looks at how well large language models can mimic human responses in big surveys and predict elections using historical data mixed with synthetic LLM data.

Warren: Yeah, it's about using a novel method called Matching-LLM to get better predictions than just using the raw LLMs alone for both survey stuff and election forecasts.

Iris: What's interesting is that they are trying to fix a problem where these models focus too much on slow, rational thinking—what some call System Two—and not enough on the fast, intuitive way people actually make up their minds based on history and culture.

Ruth: Exactly. They propose this Matching-LLM approach to bridge that gap by combining what the LLM does well with historical data that captures human tradition and intuition.

Warren: And they test this method on two main things: predicting survey responses using demographic data from the World Values Survey, and forecasting election outcomes based on American National Election Studies data.

Iris: For the survey task, they created personas for respondents in both the United States and China using data from the World Values Survey to generate synthetic responses on topics like gender role trust morality, and work ethic.

Ruth: And they found that this approach significantly outperforms traditional LLM methods when simulating people's opinions on social values, especially when looking at how those opinions differ across cultures.

Warren: They also mentioned that the Matching-LLM voting behavior replicated past U.S. election outcomes, even predicting Donald Trump’s overwhelming victory in the two thousand twenty-four presidential election closely aligning with actual results across most states <ref:2411.01582#pg1>.

Iris: In terms of survey responses themselves, they saw that most LLM synthesized or Matching-LLM synthesized response means fall within one standard deviation of the human response means, which suggests less fluctuation than raw LLM outputs.

Ruth: But they noted there are still some areas where the LLM responses diverge notably from human responses, like on social values questions such as "Homosexual parents" or "Work over leisure."

Warren: Moving over to out-of-sample election forecasting, this approach seems to correct those differences between U.S. and China samples that were observed in the raw LLM outputs.

Iris: For predicting the two thousand twenty-four U <ref:2411.01582#pg1>.S. presidential election, both the raw LLM and the Matching-LLM methods consistently forecast Donald Trump would win, but Matching-LLM showed superior accuracy compared to just using the LLM by predicting state-level outcomes with errors only in Nevada and New Hampshire.

Ruth: So, what this means for us is that these latest LLMs show significant predictive capabilities when you feed them demographic variables like age or gender.

Warren: The Matching-LLM method suggests that combining historical data with synthetic data generated by the LLM can really improve their predictive power, acting as a better supplement to traditional survey methods.

Title and authors: Iris: It offers a new way to look at applying LLMs in social science by creating this blend of historical data and LLM-generated data that aims to mimic real-world human behavior better than either system on its own.

Ruth: For the technical side, they optimized the weights for in-sample predictions using a least squares method to minimize the distance between the Matching-LLM response vector and the actual human response vector.

Warren: And for survey responses specifically, they used propensity score matching through one to one nearest neighbor matching on WVS6 data to figure out those weights.

Iris: For out-of-sample predictions, they determined those weights by referencing the weights used for similar question types in the in-sample setting.

Ruth: The model itself was trained using ChatGPT-4o, relying only on the API without any real time search capabilities, and it was tested with specific demographic characteristics defined by personas from WVS and ANES data <ref:2411.01582#pg1>.

Warren: For the election task, they used ANES two thousand twenty data with six thousand five hundred seventy-one respondents to simulate voting behavior in three elections: Hillary Clinton versus Donald Trump in two thousand sixteen Joe Biden versus Donald Trump in two thousand twenty and Kamala Harris versus Donald Trump in two thousand twenty-four <ref:2411.01582#pg1>.

Iris: The LLM prompt for the survey task involved assigning a persona based on age, gender, state or province from the U.S. and China data to define the respondent's characteristics.

Ruth: And for that ANES task, the prompt assigned a persona based on things like age, ethnic group, religion, marital status, occupation level and how often they pay attention to politics and income of family members.

Warren: The election task involved setting a scenario where it was two thousand twenty-four and asking the LLM to simulate voting behavior for the U <ref:2411.01582#pg1>.S. presidential election between Hillary Clinton or Tim Walz on the Democratic side versus Donald Trump or J D Vance on the Republican side, for example.

Iris: They also set up scenarios for the two thousand sixteen and two thousand twenty elections by asking it to simulate voting behavior as if it were in two thousand seventeen or two thousand eighteen <ref:2411.01582#pg1>.

Ruth: Overall, this paper, "Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections," shows that the latest LLMs have predictive power when fed demographic information and a historical anchor.

Warren: The main implication is that for researchers who need to scale up surveys or reduce costs, using an approach like Matching-LLM could be very effective as a supplementary data source when direct collection is limited.

Iris: It gives us a way to get these synthetic responses to be much closer to what real people actually said, especially across those cultural lines between the U.S. and China samples.

Ruth: The method itself creates a convex combination of historical data and LLM-generated data, which they argue better mimics actual human behavior than either system alone when you look at collective opinion.

Warren: They also noted that for in-sample predictions, the weights are optimized using a least squares method to minimize the Euclidean distance between the Matching-LLM response vector and the human response vector.

Title and authors: Iris: That optimization relies on propensity score matching through one to one nearest neighbor matching on WVS data to identify each respondent from WVS six for every respondent in WVS seven.

Ruth: For out-of-sample predictions, those weights are determined by looking at the weights used for similar question types that were established during the in-sample prediction setting.

Warren: This paper, "Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections," suggests a path forward where LLMs can act as powerful tools in social science research when used with thoughtful historical anchoring.

Iris: The limitation they point out is that the model's focus on rationality, or System Two thinking, contrasts with the inherent non-rational, fast thinking that really shapes public opinions.

Ruth: That's a key distinction they are making; it’s not just about getting the right answer but understanding *why* people think what they think based on their cultural context.

Warren: So, we have this method that tries to combine the speed of the LLM with the slow, intuitive knowledge from historical records to model collective behavior more accurately.

Iris: And they are showing concrete results, like better state-level election predictions than raw LLMs alone achieved in that two thousand twenty-four U <ref:2411.01582#pg1>.S. election simulation.

Ruth: It means for future research, we can use these models to generate synthetic data cheaply and test hypotheses about public opinion without needing massive human participant surveys every time.

Warren: They are providing a concrete way to improve the fidelity of social science data by ensuring those LLM-generated responses preserve the actual correlations between different variables across samples.

Iris: The limitation they state is that while it handles common sense knowledge well, there's still divergence on certain value-sensitive topics like "Homosexual parents" or "Work over leisure."

Ruth: That tells us we still need careful monitoring when using these tools for sensitive topics where cultural context plays a huge role in human decision making.

Warren: So, to wrap up, the paper "Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections" shows that combining historical data with LLM synthesis can give us more robust predictions in both survey response and election forecasting.

Iris: It's a useful tool for scaling research costs while still trying to capture the nuances of human thought, even if it still struggles a bit with specific value judgments.

Ruth: Exactly. It’s about creating that convex combination of historical and synthetic data to better mirror real-world human behavior than relying on just one system in isolation.

Warren: We'll keep an eye on how researchers use this weighting scheme for out-of-sample predictions, as that seems like a key area for future testing.

Iris: It’s a solid step forward for using LLMs in public opinion research, but we still have to be careful about what the model assumes about the people it's modeling.

The paper's summary: Ruth: So we've just looked at how this Matching-LLM method tries to blend historical data with what the large language model can generate to make better predictions for both surveys and elections.

Warren: Yeah, they’re essentially trying to get past just using the raw LLM output because that stuff tends to be too smooth or too rational, and this way is supposed to add that necessary human texture back in through history.

Iris: The main point here is that this method bridges what some call System Two thinking—the slow, logical part of the LLM—with System One, which is shaped by real-world traditions and intuition from historical data.

Ruth: Exactly. They're using synthetic responses generated by the AI, but they’re anchoring those responses to specific demographic profiles and historical survey data from places like the World Values Survey to make them more realistic than just letting the AI talk on its own.

Warren: And what this means for us is that if you’re trying to scale up public opinion research, you can use this AI-generated data as a supplement instead of paying for thousands of human participants every time.

Iris: It’s about making sure the synthetic responses actually capture those correlations between different questions, so when you look at how people answer five different social value questions at once, the pattern holds up better than it would with just raw text from the model.

Ruth: That’s a big deal for econometricians because it helps test if a specific set of assumptions about human behavior—like trust or morality—actually hold up when you feed them structured data instead of just unstructured text.

Warren: And for election forecasting, they show that this approach is more reliable in predicting the actual results, especially when looking at state-level outcomes where the differences between places matter a lot.

Iris: They found that out-of-sample predictions are better because they account for those U.S.-China differences observed in the raw model outputs by using their weighting scheme based on similar question types from earlier data.

Ruth: So, it’s not just about getting a win or loss prediction; it’s about understanding the underlying mechanics of how different groups respond to the same thing across different times and places.

Warren: The strategy they use for setting those weights is clever—they minimize the distance between their synthetic response vector and the actual human response vector using a least squares method.

Iris: That optimization relies on propensity score matching, which means they are finding the most similar people from one survey sample to weight them correctly in the other.

Ruth: It shows that even though this AI is simulating people, it’s actually learning how to weigh those simulations based on who the actual humans were in their original data.

Warren: The implication is that if everyone starts running this kind of historical anchoring, we might get a new way to model collective human behavior that’s less prone to the biases you see in models trained only on general web text.

Iris: But we have to keep an eye on those specific value judgments they noted earlier, like questions about "Homosexual parents," because even with this method, it still shows some divergence from real human answers on those very sensitive topics.

Ruth: So the paper sets up a pretty solid framework for using AI in social science by creating this blend of historical context and synthetic generation to get closer to actual human behavior.

Warren: We need to see if this weighting scheme holds up when you try out new, totally different questions that weren't in the original training data.

The paper's improvements: Ruth: So we've seen how they developed this Matching-LLM method, combining historical data with synthetic AI data to get better survey predictions and election forecasts.

Warren: Right, but now let’s talk about what they suggest improving next, because the authors aren't just stopping there.

Iris: They are focusing on refining the optimization process for those weights used in predicting out-of-sample events, which is where a lot of their methodological detail is focused.

Ruth: They propose that instead of just using a fixed weighting scheme based on how questions were similar in the first place, you should optimize those weights by minimizing the Euclidean distance between the synthetic response vector and what actual human responses looked like.

Warren: So if I understand correctly, instead of just using a rule for which data points to pull from history, we use a math method to find the best fit for that specific question.

Iris: Exactly. That optimization relies on propensity score matching through one to one nearest neighbor matching on the original survey data set to figure out the weights.

Ruth: It’s an attempt to make sure that even when you test this model on brand new questions, it produces results consistently closer to what real people actually said compared to just letting the raw AI generate a response.

Warren: That sounds like it would be a major improvement for robustness; it means the method is more adaptive and less likely to break down when encountering something outside its initial scope.

Iris: And that addresses one of those caveats we mentioned earlier—the divergence on value-sensitive topics—by giving the model a better way to calibrate its "System one" intuition against historical reality.

Ruth: It moves the system from just being a good predictor to being an actual simulator that respects the nuances of human thinking across different contexts and times.

Warren: If this weighting scheme works well for out-of-sample predictions, it drastically lowers the risk of a massive drawdown when you try to deploy this model in a live forecasting scenario.

Iris: It changes how we think about identification; instead of just looking at one specific variation, you’re using a complex combination of historical matching and AI generation to find the variation that actually drives the predictive signal.

Ruth: So, they aren't just giving us a better prediction for the two thousand twenty-four election; they are giving us a method that is fundamentally more reliable for testing hypotheses about public opinion across new subjects.

Warren: The next thing I want to see is how they handle the computational cost of running this optimization process on massive datasets.

Iris: That’s a fair question, Warren, because optimizing that Euclidean distance over millions of data points could introduce a significant computational overhead for the AI side.

Conclusion: Tom: So we’ve gone through how they developed this Matching-LLM method, combining historical data with synthetic AI data to get better survey predictions and election forecasts.

Ruth: Exactly. The authors are showing that by anchoring the AI's responses to real historical data, you can create a system that actually mimics human behavior better than just feeding it raw text from the internet.

Warren: It’s a smart way to get around the limitations of those raw models, I guess. They’re essentially giving the AI a context it needs for social science research.

Iris: And that context is crucial because it helps them check if their identification is actually working; they are verifying that the variation they are measuring is what’s really driving the prediction, not just noise.

Ruth: So, to wrap up, "Matching-LLM: Historically Anchored Synthetic Prediction of Public Opinion and Elections" shows that combining historical data with AI generation can create a more robust way to model public opinion and election outcomes.

Warren: It’s a solid tool for anyone needing to scale up research or cut down on the cost of running huge human surveys.

Iris: It provides a way to ensure those synthetic responses preserve the actual correlations between different variables, which is essential when you're trying to measure multiple preferences at once.

Ruth: The authors flag that while it’s better than raw AI, there are still specific value judgments where the model still diverges from real human answers on sensitive topics.

Warren: So it’s a powerful supplement, but you can't fully trust it when you hit those very specific cultural or moral questions.

Iris: Right. It changes how we think about identification because instead of just looking at one variable, you’re using this combination of historical matching and synthetic data to find the variation that matters for the prediction.

Ruth: Anyway, this paper is a really interesting look at how we can use these large models to do more than just generate text; they’re becoming structured tools for social science.

Warren: We’ll keep an eye on how they handle the computational cost of those optimization steps because that seems like the next big hurdle for deploying this widely.

Iris: And I think we need to see if this method can be applied to other types of behavioral data, not just survey responses and elections.

More episodes

← Home