Imputation of Public Opinion Data with Large Language Models

summary

The gist

Large language models can be effectively used to impute missing survey data through in-context learning…

In short

Large language models can effectively fill in missing survey data by learning from examples provided in context. This method consistently beats traditional statistical methods like MICE PMM, especially when data is missing due to complex reasons (MNAR). The best design choices for using LLMs significantly impact accuracy and reliability.

Key concepts

In-Context Learning (ICL)
This is a technique where the language model learns to perform a task, like data imputation, by being given examples directly in the prompt. Instead of retraining the model, you provide specific input/output pairs within the context window so it can follow those patterns to generate correct missing values.
Missingness Mechanisms (MCAR, MAR, MNAR)
These describe why survey data might be missing. MCAR means data is missing randomly. MAR means the missingness depends on observed data. MNAR is the most complex, where the reason for being missing itself depends on the value that is actually missing.
Absolute Error vs. Coverage
These are metrics used to judge imputation quality beyond simple prediction accuracy. Absolute error measures how far the imputed value is from the true value. Coverage checks if a calculated confidence interval actually contains the true coefficient, ensuring the uncertainty estimate is reliable.

Terminology used across episodes

This episode discusses

The paper

Imputation of Public Opinion Data with Large Language Models · Read on arXiv

Tobias Holtdirk, Georg Ahnert, Joseph W Sakshaug, Anna-Carolina Haensch

LMU Munich · Munich Center for Machine Learning · University of Mannheim · Institute for Employment Research (IAB) · University of Maryland, College Park

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Imputation of Public Opinion Data with Large Language Models".

Jane: Here's the gist of the paper. Large language models can be effectively used to impute missing survey data through in-context learning, which consistently reduces absolute error compared to established statistical methods like MICE PMM across various missingness mechanisms This approach demonstrates that ICL design choices have a large impact on imputation performance when evaluated against established statistical methods The paper systematically evaluates ICL design choices across different missingness mechanisms (MCAR, MAR, MNAR) on 150 opinion variables spanning 15 waves of the American Trends Panel Notably, the best-performing specification achieves nearnominal aggregate coverage with confidence intervals two to five times narrower than MICE PMM Imputation Methodology The proposed imputation method involves selecting auxiliary examples based on text embedding similarity to the target individual and using them as context for LLM-based imputation This process is detailed in Figure 2, which illustrates the flow of imputation through in-context learning The pipeline involves row serialization strategies such as String and List embedding strategies for retrieving auxiliary examples Prompting formats are compared, including Multi-turn, Single-turn, and Single-turn verbalized prompts to elicit more calibrated output variability Performance Evaluation Metrics Imputation performance is evaluated using utility metrics common in the imputation literature, focusing on absolute error, coverage, and confidence interval width rather than just prediction accuracy Absolute error measures the deviation of the imputed coefficient from the reference coefficient Coverage is a binary indicator of whether the 95% confidence interval contains the reference coefficient, and well-calibrated uncertainty requires aggregate coverage to approach the nominal 95% level Confidence interval width is measured as CIupper minus CIlower, where narrower intervals are preferable when coverage is adequate Key Findings and Generalization Study 2 confirms that the best-performing ICL specification generalizes across waves and variables, with gpt-oss-120b as the stronger generator The advantage of ICL over MICE PMM is most pronounced under MNAR, where it achieves substantially lower absolute error while maintaining acceptable coverage Furthermore, the results suggest a precision–calibration tradeoff where ICL produces tighter and more informative confidence intervals that sacrifice some coverage under MAR but narrow the gap under MNAR Variable-Level Analysis The analysis of variable-level imputation performance reveals that question semantics and topic domains are meaningful sources of variation in ICL-based imputation quality Certain regions consistently exhibit higher absolute error or lower coverage, suggesting that question semantics, and by extension the topic domain, are a meaningful source of variation in ICL-based imputation quality For instance, the variable vTRAITBIZWF2C exhibited substantially higher bias than MICE and even simple baselines such as Mode Imputation across most missingness settings Conversely, the variable vVIDOFT performed considerably better under MCAR and MAR RIGHT, achieving coverage rates above 0.

Paper summary: Tom: So, looking at the "Imputation of Public Opinion Data with Large Language Models" paper, this work moves beyond just saying LLMs can simulate survey responses; it shows how to design the in-context learning process itself to get better results than traditional methods.

Jane: The main conclusion is that by choosing the right in-context learning design—the way you serialize your rows, the embeddings you use, and how you prompt the model—you can significantly reduce absolute error compared to methods like MICE PMM across all missingness types.

Lu: What this really means for researchers is that these design choices aren't just technical details; they have a large impact on performance when evaluating imputation against established statistical methods.

Meng: For someone actually shipping models, the implication is that you need to rigorously test these ICL designs because picking a default setup might lead to poor coverage or misleading confidence intervals depending on whether your missing data is MCAR, MAR, or MNAR.

Lalam: The paper's finding that the best specification generalizes across waves and variables gives us confidence in deploying this method for large, complex datasets like the American Trends Panel.

Tom: And they confirm that under non-random conditions—the MNAR case—this approach offers a substantial reduction in absolute error while keeping the uncertainty measurements reasonably calibrated.

Jane: So, to sum up, the paper demonstrates that we can use LLMs for imputation by treating the design of the in-context learning pipeline as a critical part of the statistical model itself.

Lu: It’s about understanding that when you move from just getting an answer to getting a reliable estimate with proper uncertainty bounds, your setup choices have to be highly deliberate.

Conclusion: Tom: So, we’re wrapping up on "Imputation of Public Opinion Data with Large Language Models." Basically, these researchers are showing how using in-context learning with large language models can fill in missing survey data much better than older statistical methods like MICE PMM.

Jane: It sounds technical because they're talking about imputation and LLMs, but the core idea is that they designed the way the model gets its context—how it looks at past examples—and that design choice really matters for how good the final answer is.

Lu: Yeah, what’s interesting is they tested this across a huge amount of data, one hundred fifty different opinion variables over fifteen waves of a big panel called the American Trends Panel.

Meng: But it’s not just about getting an answer; it’s about the quality of that uncertainty. They focused on things like absolute error and how wide their confidence intervals are. That's what matters when you're actually trying to use this stuff in the real world, not just get a prediction number.

Lalam: The best part they found is that the way they set up their context learning pipeline—whether they embed text by serializing rows as strings or lists—that setup choice has a huge impact on performance across all different ways data might be missing, like random missingness or non-random missingness.

Tom: That’s what makes it interesting for me. It shows that we can't just plug in an LLM and hope for the best; we have to engineer the context feeding process carefully.

Jane: Exactly. And they found that their best setup actually worked well across different waves and different types of data, which gives people confidence that this approach is more robust than what we might expect from a single test run.

Lu: For example, they found the strongest generator model was gpt oss one hundred twenty billion, but the key was tuning the input context to guide it properly.

Meng: And I wonder how much extra work goes into making sure that specific context setup actually works reliably on real traffic when you’re trying to deploy this in production.

Lalam: It suggests that for complex data imputation, the architecture of how you feed the information into an AI model is just as important as choosing the model itself.

Tom: So, it’s a reminder that when we use these models for messy data, we’re not just using them to guess; we’re designing a whole system for how they learn from examples.

Jane: And that leads us right into what kind of data problems this method handles best and where its limitations might show up in the future.

More episodes

← Home