Could It Have Fit Anything? Restrictiveness of Economic Models, LLMs, and AI Workflows
summary
The gist
Model restrictiveness in functional and structural settings using Gaussian process priors extends the restrictiveness measure to functional and structural econometric settings by operationalizing it…
In short
The paper extends a 'restrictiveness' measure to functional and structural econometric models using Gaussian process priors. It argues that model restrictiveness should reflect the type of approximation error relevant to an application, rather than being mechanically inherited from existing complexity measures. This provides a two-dimensional lens for evaluating economic models.
Key concepts
- Restrictiveness (rpFΘ)
- This measure quantifies how much a model FΘ is constrained by a compact set of eligible rules F. It is defined relative to the discrepancy function d, implying that restrictiveness reflects the limitations imposed by known constraints on the model class.
- Discrepancy Functions (d)
- The paper argues that the choice of discrepancy function d should be guided by interpretability and context rather than standard complexity measures. It explains why metrics like Rademacher complexity are unsuitable in a pure approximation-error sense.
- Structural-Form Restrictiveness (rSF)
- This measure focuses on constraints on counterfactual mappings within structural economic models. It answers how tightly the economic structure constrains the relationship between endogenous variables and outcomes, such as demand curves.
- Two-Dimensional Lens
- Restrictiveness provides a dual perspective for model evaluation: measuring both how much a model rules out (its constraints) and how much of what matters it successfully captures. This allows for principled trade-offs between empirical accuracy and theoretical discipline.
Terminology used across episodes
This episode discusses
- Could It Have Fit Anything? Restrictiveness of Economic Models, LLMs, and AI Workflows · Paper Radio
- The Transfer Performance of Economic Models
- Approximating Choice Data by Discrete Choice Models
The paper
Could It Have Fit Anything? Restrictiveness of Economic Models, LLMs, and AI Workflows · Read on arXiv
Massachusetts Institute of Technology · University of Pennsylvania
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Could It Have Fit Anything? Restrictiveness of Economic Models, LLMs, and AI Workflows".
Warren: Here's the gist of the paper. Model restrictiveness can be defined, computed, and interpreted in functional and structural settings far beyond those considered in prior work This paper extends the restrictiveness measure to functional and structural econometric settings using Gaussian process priors The central message of our analysis is that d is not fixed: it should be chosen to reflect the kind of approximation error that matters in the application, rather than inherited mechanically from existing complexity measures Defining Restrictiveness in Functional Settings The paper extends the restrictiveness measure to functional and structural econometric settings using Gaussian process priors It operationalizes restrictiveness based on Bayesian nonparametric priors such as Gaussian Processes and Dirichlet Processes Restrictiveness is defined relative to a compact set of “eligible” rules F that reflect any constraints the model is known to have The restrictiveness of model FΘ with respect to eligible set F is defined as rpFΘ; F, dq “ EλF rdpFΘ, fqs EλF rdpFbase, fqs (1) This definition implies that restrictiveness is invariant to affine transformation of the discrepancy function d by the linearity of the expectation operator Extending to Structural Models The framework extends the notion of restrictiveness to structural economic models, which are important in applied microeconomic areas such as industrial organization and labor economics This involves defining restrictiveness for structural models by working with the reduced forms and their induced conditional distributions The paper treats several important cases, including fully parametric models with endogeneity, models defined by moment equalities, and semiparametric models The Role of Discrepancy Functions The paper argues that the choice of discrepancy should be guided by interpretability and context rather than by existing capacity measures designed for other purposes It explains why the Rademacher complexity and GMM criterion functions are unsuitable as discrepancies In a pure approximation-error sense and in the absence of noise, restrictiveness can be interpreted as the normalized limit of the average-case learning curve Connections to Learning Theory Restrictiveness can be interpreted as the normalized limit of a “noise-free” version of the average-case learning curve that focuses on functional-space approximation errors Proposition 2 shows that under mild assumptions, restrictiveness can be interpreted as the ratio of two average-case approximation errors: one achieved by the best-fitting member of the model class FΘ, the other by the baseline model class Fbase Applications in Choice Models The framework is applied to three economic problems, including revisiting Cumulative Prospect Theory (CPT) and Disappointment Aversion (DA) models over the entire space of monotone, bounded prediction rules In industrial organization, it quantifies how much of mixed logit’s theoretical flexibility is actually realized by the parametric forms used in practice Furthermore, moment restrictions from endogeneity substantially increase restrictiveness and alter model rankings The paper establishes that model restrictiveness can be defined, computed, and interpreted in functional and structural settings far beyond those considered in prior work Together with the completeness measure of Fudenberg, Kleinberg, Liang and Mullainathan (2022), restrictiveness gives researchers a two-dimensional lens for model evaluation The work suggests three directions for future research, including using restrictiveness as a regularization device Empirical Findings in Choice Settings In the certainty-equivalent setting, restrictiveness is uniformly higher over the continuum of lotteries than when evaluated over finite sets of observations In multinomial choice models, endogeneity raises restrictiveness for all models Specifically, in the BLP application, IV moment conditions can raise restrictiveness more dramatically than functional form restrictions alone Robustness Checks Robustness checks confirm that the qualitative ranking of restrictiveness across specifications is unchanged when varying the kernel choice or sampling scheme The results are reported in Table A.1 and Table A.3, showing that our estimates appear more restrictive than those reported in FGL The qualitative ranking of models is unchanged when using three instruments that are most strongly correlated with the endogenous regressor Details on Gaussian Process Sampling The paper details how to sample from a constrained GP, which is equivalent to constrained GP regression without updating the prior The sampling procedure involves steps like generating a constrained grid and computing increasing monotonic approximation Structural-Form Restrictiveness Structural-form restrictiveness measures constraints on counterfactual mappings, not on observed moments The structural-form restrictiveness rSF captures how tightly the model specification (2) constrains the space of mappings from endogenous and exogenous covariates to outcomes It answers the question: How strongly does the economic structure constrain the mapping from endogenous variables and shifters to outcomes (e.g., demand curves) Comparison of Restrictiveness Measures The paper notes that structural-form restrictiveness rSF and reduced-form restrictiveness rRF are complementary diagnostics Structural-form restrictiveness rSF answers the question: How strongly does the economic structure constrain the mapping from endogenous variables and shifters to outcomes (e.g.
Ruth: demand curves) Reduced-form restrictiveness rRF answers the question: How strongly does the combination of structure, equilibrium, and error assumptions constrain the observable mapping from instruments to equilibrium outcomes Asymptotic Normality of Estimator The paper derives a central limit theorem for rpn,M as n Ñ 8, for fixed M, that explicitly incorporates sampling uncertainty in pX1,...
Title and authors: Ruth: So we're looking at this paper titled Could It Have Fit Anything? Restrictiveness of Economic Models, LLMs, and AI Workflows. It sounds like they are trying to give us a way to measure how much structure an economic model actually imposes on the data, especially when we're dealing with these new AI tools that let us test tons of models really fast.
Warren: Yeah, it’s about this idea that models aren't just black boxes anymore; they have built-in rules about what kind of patterns they expect to see in the world. This paper wants to quantify how restrictive those rules are, so we can tell if a model is actually capturing real economic structure or just fitting noise.
Iris: I think the title signals that it’s not just about traditional econometrics anymore, because it mentions LLMs and AI workflows, suggesting this restrictiveness idea applies to how we use these new tools to test models.
Ruth: Exactly. It moves beyond just picking the right functional form in a standard regression and tries to measure the underlying constraint of the whole prediction rule, which is what makes a model useful for decision making.
Warren: And they are extending this from just finite-dimensional spaces, like simple logit models, to infinite-dimensional functional spaces using tools like Gaussian process priors. That’s a big technical leap for me to consider how we actually implement this in trading or policy settings.
Iris: The paper is operationalizing restrictiveness through Bayesian nonparametric priors, specifically looking at things like Gaussian Processes and Dirichlet Processes to define what the eligible set of rules F actually is.
The paper's summary: Ruth: So, the core message is that model restrictiveness isn't a fixed thing; it should depend on what approximation error actually matters in the application, rather than just being something inherited mechanically from complexity measures we’ve used before.
Warren: That makes sense from a risk perspective. If we use this measure to pick a strategy, we want to know if the model is restrictive enough to rule out bad outcomes, not just complex enough that it can fit historical noise better.
Iris: They define restrictiveness relative to a compact set of "eligible" rules F, and they define it as the ratio between two average-case approximation errors: one achieved by the best-fitting member of the model class FTheta, and one by some baseline model class Fbase.
Ruth: That’s a nice way to frame it; it gives us a concrete mathematical comparison for how much "better" our specific model is compared to just any plausible alternative. It turns restrictiveness into a ratio of errors.
Warren: And they argue that this concept can be interpreted as the normalized limit of the noise-free average-case learning curve, which means we're looking at the best possible performance in a world without extra noise messing things up.
Iris: They show that under mild assumptions, this restrictiveness measure can be interpreted as that ratio itself, focusing specifically on functional-space approximation errors rather than just general model fit.
The paper's improvements: Ruth: One of the major improvements they propose is using this concept as a regularization device instead of relying on syntactic complexity penalties like AIC or BIC. This would reward models that rule out economically implausible patterns, which is what we actually want in applied work.
Warren: That’s practical. If a model is overly flexible but doesn't capture any real economic constraint, this measure should penalize it severely, even if its raw fit statistic looks good on the surface.
Iris: They also extend the framework to structural models that include endogeneity and instrumental variables, which lets us look at how these specific econometric features interact with model restrictiveness.
Ruth: And they specifically introduce structural-form restrictiveness, or rSF, which measures constraints on counterfactual mappings—how tightly the economic structure constrains the mapping from endogenous variables and shifters to outcomes.
Warren: That’s different from what I usually think about; I'm used to looking at reduced-form restrictiveness, which is more about how much the structure, equilibrium assumptions, and error terms constrain the observable mapping from instruments to outcomes.
Iris: They treat rSF and reduced-form restrictiveness as complementary diagnostics; rSF answers how much the economic structure constrains mappings like demand curves, while reduced-form restrictiveness answers how much structure and equilibrium assumptions constrain the observable mapping from instruments to equilibrium outcomes.
Conclusion: Ruth: So, to wrap up, this paper establishes that model restrictiveness can be defined and computed in functional and structural settings far beyond what was done before. It gives us a two-dimensional lens for evaluation: how much a model rules out, and how much of what matters it captures.
Warren: And they show that in choice settings, like revisiting models for cumulative prospect theory or disappointment aversion, the same models exhibit uniformly higher restrictiveness when evaluated over continuum domains compared to finite sets of observations.
Iris: And they also found that in multinomial choice models, endogeneity raises restrictiveness for all models, and specifically in their BLP application, instrumental variables can raise it more dramatically than just functional form restrictions alone.
Ruth: It suggests a direction for future research where this measure could be used as a regularization device when we are trying to select between competing theoretical frameworks or different estimation techniques.
Warren: We should keep an eye on how this translates into actual computation, because the paper does spend time detailing how to sample from constrained Gaussian processes, which is the practical implementation hurdle.
Iris: I agree. The normalization of the average-case approximation error is a strong theoretical anchor for what we’re measuring here.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19