Identification and Inference for Algorithmic Frontiers with Selective Labels

arXiv:2606.14977 · econ.EM, cs.LG · Submitted 2026-06-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Identification and Inference for Algorithmic Frontiers with Selective Labels".

Jane: The gist The paper provides identification results to characterize a fairness-accuracy (FA) frontier, and statistical inference tools to test hypotheses and build a confidence set for the FA-frontier,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: Let's talk about the title itself, "Identification and Inference for Algorithmic Frontiers with Selective Labels." It immediately tells you this isn't just about finding a single point on a curve, but about figuring out where that curve is even when your data is incomplete.

Tom: Yeah, it points to the selective labels part. That means we're dealing with real-world scenarios where we don't see what happens to everyone, which is exactly where most of these fairness and accuracy problems live.

Lu: The authors are Liu, Molinari and Velez from Cornell University. They are extending previous work by Liang, Lu Mu and Okumura in two thousand twenty-six to give us a toolkit for nonparametric estimation and statistical inference <ref:2606.14977#pg2,a toolkit for nonparametric estimation>.

Meng: I'm curious about the authors' background on this. Are they focused on theoretical modeling, or do they have experience dealing with messy production data? Because if it’s too abstract, I don’t know how to implement it.

Lalam: They are bringing in a different way to assess fairness and accuracy by using two loss functions, which is their main innovation over previous work. It lets them treat accuracy and fairness separately but linked together.

Tom: That sounds like a solid mechanism for handling the trade-off between being accurate and being fair simultaneously, which is the central tension in these kinds of models.

The paper's summary: Jane: So, to summarize what this paper is actually doing, they are providing identification results to characterize this fairness-accuracy frontier and statistical inference tools for testing hypotheses about it when we only observe outcomes for selected individuals.

Tom: The key takeaway is that even with missing data—the selective labels problem—we can get point identification of the FA-frontier if we assume a missing-at-random condition conditional on the attributes.

Lu: They define the FA-frontier F as that set of expected losses where no other algorithm yields an expected loss that dominates it in terms of fairness and accuracy. The support function, hE(q), plays a big role here.

Meng: So they are using this support function to characterize the frontier, which sounds like it’s a complex mathematical structure behind the scenes for describing how good a model is across different groups.

Lalam: They use this support function in a formula involving population means and conditional expectations of loss functions, specifically hE(q) = E*

q Tθ0(X): + sup a E*

a(X)q T∆θ(X): on page one of that work.

Tom: And they establish that this set E is strictly convex, which is important because it means the frontier isn't just a bunch of points; it has a clear shape.

The paper's improvements: Jane: The paper suggests some really useful improvements, especially when we consider how we actually test if an algorithm hits that frontier. They propose a debiased machine learning estimator based on inverse probability weighting to estimate the FA-frontier itself.

Tom: That DML estimator sounds like it’s what you’d use in production to get a more reliable estimate of the frontier, and their asymptotic distribution is derived using Lemma three point one <ref:2606.14977#pg3>.

Lu: They also propose a test statistic, T LDA n, to check whether the FA-frontier actually coincides with the Pareto frontier. This is a way to test if there's any less discriminatory alternative available.

Meng: I like that they have a specific hypothesis for that test, H0: min q in S squared hC(ε*) (q) + hE(-q) = zero on page three of that work. That looks like a concrete condition to check for frontier membership.

Lalam: And they give us a Bayesian bootstrap procedure to build a confidence set for the partially identified FA-frontier using test inversion, which is pretty robust for building those confidence regions.

Conclusion: Jane: So wrapping up, the paper suggests that under certain assumptions, point identification of the FA-frontier is possible using inverse propensity score weighting and a debiased machine learning estimator. This method seems particularly appropriate in governance contexts because high-dimensional administrative data can allow for flexible nonparametric modeling of selection processes.

Tom: Yeah, and they provide an asymptotic theory for this estimator along with valid bootstrap methods to build a confidence set for the partially identified FA-frontier and to test that hypothesis about no less discriminatory alternative existing.

Lu: They conclude that when the selection process is left completely unrestricted, they provide a tractable characterization of the sharp identification region for the FA-frontier.

Meng: From my side, I'm focused on what this means for deployment: we get a way to build confidence intervals around our fairness metrics even when our observational data is incomplete. It makes sense for shipping models.

Lalam: I think the whole point of "Identification and Inference for Algorithmic Frontiers with Selective Labels" is giving us a practical way to test if an algorithm truly sits on that frontier, even when we're missing some outcomes.

Tom: So this work gives us a concrete statistical method—the DML estimator and the test statistic—to go beyond just observing results and actually inferring about the underlying fairness-accuracy trade-off structure of an AI system.

Cornell University

econ.EM, cs.LG

Submitted: 2026-06-12

Updated: 2026-10-15

Comments: 78 pages, 2 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 69/100

The gist: The gist The paper provides identification results to characterize a fairness-accuracy (FA) frontier, and statistical inference tools to test hypotheses and build a confidence set for the

Key concepts

Fairness-Accuracy (FA) Frontier
This is a boundary in the space of expected losses that separates algorithms that are considered 'fair' from those that are not, while also considering their accuracy. The paper defines this frontier based on whether one algorithm can achieve lower expected losses across both fairness and accuracy metrics compared to another.
Selective Labels Problem
This arises when the actual outcomes of an algorithm are only known for a subset of individuals (selected labels). This creates a challenge because the observed data might not represent the true underlying population distribution, complicating standard statistical inference about fairness and accuracy.
Debiased Machine Learning (DML) Estimator
This is a specific statistical tool used to estimate the FA-frontier when labels are selectively observed. It uses inverse probability weighting to adjust for the selection process, resulting in an estimator whose asymptotic distribution can be reliably characterized, allowing for accurate inference about the frontier.

Terminology

Summary

The gist The paper provides identification results to characterize a fairness-accuracy (FA) frontier, and statistical inference tools to test hypotheses and build a confidence set for the FA-frontier, when outcomes are observed only for selected individuals

Theoretical Framework

The framework builds on the theoretical framework put forward by Liang, Lu, Mu, and Okumura (2026), which characterizes a fairness-accuracy (FA) frontier under broad preferences over group-specific expected losses as well as the implications of banning attributes as inputs to the algorithms The paper introduces two different loss functions to assess accuracy and fairness, aligning with literature in computer science The support function of a convex feasible set E ⊂ R cubed plays a crucial role in characterizing the FA-frontier and carrying out inference

Characterization of the FA-Frontier

The paper defines the FA-frontier F as the set of expected losses ε ∈ E such that no other algorithm yields expected losses ε' that FA-dominate ε The support function hE (q) is characterized by a formula involving the population means and conditional expectations of loss functions, specifically hE (q) = E∗[q Tθ0(X)] + sup a E∗[a(X)q T∆θ(X)] Theorem 2.1 establishes that the set E is strictly convex The FA-frontier F is characterized by the condition that for a given algorithm a∗, ε∗ ∈ F if and only if min q∈eS squared hC(ε∗)(q) + hE(-q) = 0

Selective Labels Problem and Identification

The paper addresses the reality that outcomes are observed only for selected individuals, denoted as Y = ZY∗, where Z ∈ [0, 1] indicates observability This selective labels problem creates challenges for governance compliance similar to those recognized in employment discrimination law The paper provides point identification of the FA-frontier under a missing-at-random (MAR) assumption, where conditional on attributes, outcome observability does not convey additional information on Y∗ This is achieved by deriving a debiased machine learning (DML) estimator for the FA-frontier based on inverse probability weighting

Inference and Testing Procedures

The paper proposes a debiased machine learning (DML) estimator for the FA-frontier, whose asymptotic distribution is derived using Lemma 3.1 It also derives a test statistic to check whether the FA-frontier coincides with the Pareto frontier, PF The test statistic T LDA n is proposed to reject the LDA hypothesis when no less discriminatory alternative exists

Conclusion

The paper concludes that under certain assumptions, point identification of the FA-frontier is possible using inverse propensity score weighting and a debiased machine learning estimator, which is particularly appropriate in governance contexts The proposed inference method is based on a debiased machine learning estimator, which is particularly appropriate in this context, where high-dimensional administrative data enable flexible nonparametric modeling of the selection process and hence make the missing at random assumption more credible When the selection process is left completely unrestricted, a tractable characterization of the sharp identification region for the FA-frontier is provided The paper provides a debiased machine learning estimator and its asymptotic theory (along with valid bootstrap methods) to build a confidence set for the partially identified FA-frontier and to test the hypothesis that no LDA exists to a given algorithm

How it works

The framework extends the analysis in LLMO’s Appendix O by providing a simple and testable necessary and sufficient condition under which the FA-frontier coincides with the Pareto frontier The support function of E, denoted hE (q), is crucial for all results, yielding that SE(q) equals the singleton∇qhE(q) Theorem 2.2 characterizes frontier membership for a given λ by showing that ε∗ ∈ F if and only if min q∈eS squared hC(ε∗)(q) + hE(-q) = 0

Partial Identification

When labels are selectively observed, the vector of expected losses ε∗ is only partially identified, necessitating care when characterizing what can be learned about F The paper specializes the analysis to binary outcomes and classification loss for accuracy and statistical parity for fairness The sharp identification region for hE (q) in direction q is given by the set of functions hE(q;λ) as lambda ∈ Λ, which collapses to a finite-dimensional criterion involving only the corner distributions λ0 and λ1

Debiased Machine Learning Estimator

A debiased machine learning (DML) estimator for the FA-frontier is proposed based on inverse probability weighting The estimator hE(q) is defined as E[ζ i(q;L,η)] where ζ i(q;L,η) includes a term accounting for the local effect of the first-step function π on the original estimand The asymptotic distribution of this estimator is shown to converge to a Gaussian process uniformly in q ∈ S squared

Testing Procedures

The paper provides test statistics and their asymptotic distribution to test whether (i) the FA-frontier coincides with the Pareto frontier The statistic T LDA n is proposed to reject the LDA hypothesis when no less discriminatory alternative exists The Bayesian bootstrap procedure is consistent, and it provides an asymptotically valid confidence set for F by test inversion

References

ADLER, ROBERT J. AND JONATHAN E. TAYLOR (2007): Random Fields and Geometry, Springer Monographs in Mathematics, New York: Springer.

ANDREWS, DONALD W. K.: “Chapter 37 Empirical process methods in econometrics,” Elsevier, vol. 4 of Handbook of Econometrics, 2247–2294.

ANGWIN, JULIA, JEFF LARSON, SURYA MATTU, AND LAUREN KIRCHNER (2016): “Machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks.” ProPublica.

ARNOLD, DAVID, WILL DOBBIE, AND PETER HULL (2021): “Measuring racial discrimination in algorithms,” in AEA Papers and Proceedings, vol. 111, 49–54.

AUERBACH, ERIC, ANNIE LIANG, MAX TABORD-MEEHAN, AND KYOHEI OKUMURA (2024): “Testing the Fairness-Accuracy Improvability of Algorithms,” arXiv preprint arXiv:2405.04816.

BAROCAS, SOLON, MORITZ HARDT, AND ARVIND NARAYANAN (2023): Fairness and Machine Learning: Limitations and Opportunities, MIT Press.

BERESTEANU, ARIE AND FRANCESCA MOLINARI (2008): “Asymptotic Properties for a Class of Partially Identified Models,” Econometrica, 76, 763–814.

BERK, RICHARD, HODA HEIDARI, SHAHIN JABBARI, MATTHEW JOSEPH, MICHAEL KEARNS, JAMIE MORGENSTERN, SETH NEEL, AND AARON ROTH (2017): “A convex framework for fair regression,” arXiv preprint arXiv:1706.02409.

BERK, RICHARD, HODA HEIDARI, SHAHIN JABBARI, MICHAEL KEARNS, AND AARON ROTH (2021): “Fairness in Criminal Justice Risk Assessments: The State of the Art,” Sociological Methods & Research, 50, 3–44.

BLATTNER, LAURA AND JANN SPIESS (2022): “Machine Learning Explainability and Fairness: Insights from Consumer Lending,” FinRegLab Empirical White Paper.

BONTEMPS, CHRISTIAN, THIERRY MAGNAC, AND ERIC MAURIN (2012): “Set identified linear models,” Econometrica, 80, 1129–1155.

CANAY, IVAN A., GASTÓN ILLANES, AND AMILCAR VELEZ (2026): “A User’s Guide for Inference in Models Defined by Moment Inequalities,” Journal of Econometrics, forthcoming.

Improvements for AI systems

  1. Improved identification of fairness-accuracy trade-off under selective labels by deriving a debiased machine learning estimator (DML) for the FA-frontier, which is particularly appropriate in this context, where high-dimensional administrative data enable flexible nonparametric modeling of the selection process.

  2. Statistical inference tools for testing hypotheses and building a confidence set for the FA-frontier using asymptotic distributions derived from inverse probability weighting correction in DML estimators.

  3. Characterization of the sharp identification region of the FA-frontier under an unrestricted selection process by deriving conditions where the FA-frontier coincides with the Pareto frontier.

  4. Development of a test for existence of a less discriminatory alternative (LDA) to a given algorithm, stating that this is equivalent to checking whether a∗ yields expected losses with ε∗ ∈ F, formalized by the hypothesis: H0: min q∈S˜2 hC(ε∗) (q) + hE (−q) = 0.

  5. Implementation of a Bayesian bootstrap procedure for estimating the critical values of the LDA test statistic, providing an asymptotically valid confidence set for the FA frontier and guaranteeing that if the LDA hypothesis holds, the test φ LDA n has asymptotically correct size.

Sources

Related papers