Narratives to Numbers: Large Language Models and Economic Policy Uncertainty
summary
The gist
Large language models can systematically improve text-derived measures and should be integrated as explicit measurement tools in empirical economics.
In short
The study investigates using large language models (LLMs) as explicit classifiers to improve text-based measures of economic policy uncertainty. By fine-tuning LLMs on human audit labels, researchers found they identify nearly twice as many relevant articles compared to simple keyword searches. This approach makes the index more robust and allows for nuanced measurement of policy uncertainty.
Key concepts
- Large Language Models (LLMs)
- These are advanced AI models trained on massive amounts of text data. In this study, they are used not just to generate text, but as sophisticated tools to read and categorize news articles based on complex human judgments about policy uncertainty.
- Economic Policy Uncertainty Index
- This is a metric used in economics to measure how uncertain policymakers feel about future economic conditions. The paper shows how different methods—like keyword searches versus LLM classification—create different indices, which affects the resulting economic estimates.
- Upstream Modeling Choices
- This refers to the decisions made before the final economic analysis, specifically how data is processed and what tools (like an LLM classifier) are chosen. The paper argues that these upstream choices significantly shape the final results rather than just correcting errors later.
- Human Audit Labels
- These are ground-truth classifications provided by human experts who have manually reviewed articles to determine if they relate to economic policy uncertainty. These labels serve as the 'gold standard' used to train and evaluate the performance of the AI classifiers.
Terminology used across episodes
This episode discusses
- Narratives to Numbers: Large Language Models and Economic Policy Uncertainty · Paper Radio
- No Language Left Behind: Scaling Human-Centered Machine Translation
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Multilingual Open Text Release 1: Public Domain News in 44 Languages
The paper
Narratives to Numbers: Large Language Models and Economic Policy Uncertainty · Read on arXiv
Ethan Hartley
University of Hawaii at Manoa
We study the implications of adopting machine-generated data in economic analysis. Rather than taking the data-generating process as given and correcting estimates econometrically, we evaluate how upstream modeling and researcher choices shape downstream estimates. Revisiting the Economic Policy Uncertainty index, we find that large language models fine-tuned on human audit labels identify nearly twice as many auditor-flagged articles as keyword-based alternatives and are robust to noisy text and linguistic drift. Varying classifier, threshold, and aggregation choices produces substantial differences not only in performance but in what the index captures, reshaping the relationship between policy uncertainty and returns.
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Narratives to Numbers".
Warren: Here's the gist of the paper. Large language models can systematically improve text-derived measures by acting as explicit, estimable classifiers in empirical economics The study evaluates large language models as estimable classifiers and clarifies how modeling choices shape downstream measurement error Pruning redundant reasoning tokens The paper investigates the role of large language models (LLMs) in constructing text-based policy uncertainty indices by comparing them against traditional keyword-based methods The authors show that contemporary classifiers substantially outperform dictionary rules, better track human audit assessments, and extend naturally to noisy historical and multilingual news This is achieved by treating LLMs as calibrated classifiers that map text into policy-uncertainty labels How the training signal is built The methodology involves a three-step process to generate an index: first, articles are labeled using a simple bag-of-words (BOW) classifier based on specific keyword groups for Economic, Policy, and Uncertainty terms Second, these article-level labels are aggregated into a newspaper- and time-specific index using the procedure described in the text Third, the mapping M from text to numerical labels is established by a supervised adaptation strategy where an LLM backbone produces a representation that is mapped to a probability of the positive class via a logistic link Where the gains come from
Paper summary: Ruth: To summarize what they did here, the study focuses on revisiting the economic policy uncertainty index. The central thesis is that large language models can systematically improve these text-derived measures by acting as explicit, estimable classifiers in empirical economics.
Warren: They are comparing these LLMs against the traditional keyword-based approach to see how much better they are at tracking human audit assessments and handling messy data like historical news or different languages.
Iris: The paper outlines a three-step process for construction. First, they compare different classifiers—keyword models, support vector machines, and fine-tuned LLMs—on the human-audited corpus to see which one performs best at the article level.
Ruth: Then they embed these different models into the standard index construction pipeline. This is where they test how things like classifier performance, decision thresholds, and aggregation choices affect the resulting index series.
Warren: Finally, they apply these models to hundreds of millions of historical U.S. articles and millions of multilingual articles to build a nineteenth-century U.S. index with a single multilingual model framework across all those inputs.
Iris: They use supervised methods, including fine-tuned or prompted language models, to construct these indices, but they also acknowledge that these methods often rely on narrow domains or weak labels if you don't train them directly on large-scale human labels like in the BBD corpus.
Ruth: The paper provides a transparent benchmark by training directly on those large human labels and evaluating against the original audit design from Baker Bloom and Davis, which is what they call the BBD corpus. This gives us a clear idea of what’s achievable with these probabilistic, context-aware models.
Warren: And they contribute to the larger body of literature that uses policy uncertainty indices constructed from unstructured data as regressors in bigger macroeconomic and financial models. That’s where it gets practical for applying this work beyond just the index itself.
Conclusion: Ruth: Looking at Narratives to Numbers, the authors Ethan Hartley found that large language models should be integrated as explicit measurement tools rather than just being a tool for generating predictions. The implication is that we should stop assuming the underlying text processing is perfect and start modeling the classifier itself.
Warren: So, if you use an LLM to flag uncertainty, you’re not just getting a number; you’re introducing a modeling choice that dictates how uncertainty gets translated into an economic signal for things like investment decisions.
Iris: What this means for us in econometrics is that we have to be very careful about the identification of the effect. We need to show precisely how varying the classifier or aggregation choices actually changes the outcome, because those variations are what matter here.
Ruth: Right, so if we use a keyword approach versus an LLM approach, and they give us different uncertainty levels for the same article, that difference might not be purely about the underlying policy uncertainty itself; it could be about how well our tool is picking up on nuanced language.
Warren: And that’s what Warren and Iris are really digging into—the sensitivity of the resulting series to those choices. It shows us where our measurement error comes from when we rely on these new language models for constructing indices.
Iris: The paper does acknowledge a limitation, which is that supervised methods, even with large human labels, can still be limited by those narrow domains or the automated nature of some of the labels they use if you try to apply it broadly elsewhere.
Ruth: So the big picture here is that these LLMs are powerful instruments for improving text-based measures, but we still need careful calibration and we need to be explicit about how much modeling choice is driving our final results.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19