Density-Ratio Losses for Post-Hoc Learning to Defer
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Density-Ratio Losses for Post-Hoc Learning to Defer".
Jane: Here's the gist of the paper. We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss The DR CPE losses are derived using a reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE) This approach bridges Chow-style rules, expert comparison, and connections to related learning settings like anomaly detection.
Tom: Core Methodology The paper introduces a distributional formulation of post-hoc L2D based on ideal distributions Deferral is defined by thresholding the density-ratio between model and expert ideal distributions The framework accommodates both marginal and joint ideal distributions, showing that for KL-ideal distributions, the marginal rule recovers Chow’s rule w.r.t.
Jane: the data distribution The joint rule corresponds to a Chow-style decision under an expert-tilted distribution The DR CPE losses are derived by reducing DRE to CPE, leading to a family of DR CPE post-hoc deferral losses.
Tom: Ideal Distribution Framework An ideal distribution is defined as a divergence-regularized reweighting of the data distribution under which a classifier-loss pair performs best at within a specified (regularized) distance away from the true probability measure P Two types of ideal distributions are considered: an (D, γ)-marginal ideal distribution Qm in marginal space and an (D, γ)-joint ideal distribution Qj in joint space The density-ratio deferral rule is given by J(dQ/dQ(e)) ≤ τ K This deferral decision has a direct connection to hypothesis testing, corresponding exactly to optimal hypothesis testing in the Neyman-Pearson sense, i.e., it is a likelihood ratio test.
Title and authors: Tom: Okay, so the title is "Density-Ratio Losses for Post-Hoc Learning to Defer." It's about how you can learn when a model should defer its decision based on comparing its ideal distribution with an expert's ideal distribution. Jane That sounds like we're moving away from just looking at the model versus the data, and instead comparing two different ideal setups.
Lu: Exactly. The core idea they introduce is defining deferral by thresholding the density ratio between a model’s ideal distribution and an expert’s ideal distribution. Tom So if we can learn that ratio, we can make decisions about when to trust the model versus when to defer based on that comparison.
Meng: I'm wondering how this translates into something I could actually deploy. Is it just a theoretical trick, or does it give us a real way to handle uncertainty without retraining? Jane That’s the big question for production, Meng. The paper suggests that for ideal distributions based on Kullback-Leibler divergence, the deferral rules they derive actually recover Chow's rule under the original data distribution.
Tom: So even with these ideal distributions, if you're looking at just one marginal distribution instead of the joint one, you still get a familiar decision rule. Lu That’s because it connects to an expert-tilted Bayes posterior depending on whether those ideal distributions are marginal or joint ones Jane So they show that this framework bridges Chow-style rules and connections to expert comparison in a way that feels grounded.
Meng: If we're focusing only on the marginal distribution, it sounds like we get something predictable about when to defer. Lalam I think what’s interesting is that they connect this density-ratio concept directly to hypothesis testing, specifically optimal hypothesis testing in the Neyman-Pearson sense, which means it's essentially a likelihood ratio test Tom So they’re saying the mathematical structure of comparing these distributions leads right into a standard statistical test framework.
Jane: That’s a really concrete connection. It moves the idea from abstract probability to something you might actually recognize in testing scenarios.
The paper's summary: Tom: So, summarizing the main point of "Density-Ratio Losses for Post-Hoc Learning to Defer," they're defining an ideal distribution as a divergence-regularized reweighting of the data distribution under which a model performs best at some distance from the true probability measure P Jane That means instead of just training on real data, we're thinking about what kind of world would make our model perform best, and that world is defined by this ideal distribution.
Lu: Right. They define two types: a marginal ideal distribution and a joint ideal distribution. The paper uses the density-ratio between these two ideals to set the deferral rule, which they call J of dQ over dQ of e Jane This ratio is then compared to some threshold K, and that comparison dictates whether we defer or not.
Meng: I’m picturing a scenario where we have our model and an expert. The paper says this density-ratio deferral rule has a direct connection to hypothesis testing, which is important because it suggests the decision mechanism itself is based on optimal hypothesis testing Tom It's like the system is designed to make the best possible choice under those specific distributional assumptions.
Jane: And they derive their actual loss functions, called DR CPE losses, by doing a reduction from Density-Ratio Estimation to Class-Probability Estimation. That’s how they connect this abstract idea to a measurable loss function we can actually optimize against Lu This reduction leads to these family of DR CPE post hoc deferral losses which are the main focus of their optimization work.
Tom: So, they take this density ratio concept and map it onto a class probability estimation problem, which is how they derive the actual loss functions for deferral. Jane It’s really about making that complex idea of ideal distributions something we can actually calculate as an expectation over the data measure P Meng And they showed that this joint DR CPE loss directly bounds the marginal DR CPE loss, which is a nice property for their optimization process.
Lu: That bounding property is key, because it means if you optimize the joint version, you get information about the simpler marginal version too Tom It shows a structural relationship between those two different ways of looking at it.
The paper's improvements: Tom: Moving on to what they actually suggest as improvements, they show that this approach performs better than some other learning to defer baselines and it’s more robust when you change the dataset settings Jane They tested this on CIFAR-one hundred PathMNIST, and DermaMNIST and found that DR CPE performs best to second best over corrupted settings.
Meng: Second best is good, but I need to know how much better it is than something like confidence-based deferral or those two-stage approaches they mentioned Tom The paper says it’s consistently better than the confidence-based and two-stage approaches in their experiments.
Lu: They found that when using squared loss, this method seems to be the best performing option, and they saw that LSIF and KLIEP performed similarly with it Jane That suggests that the squared loss formulation is really compatible with this ideal distribution framework for maximizing performance.
Tom: So if we’re looking at how to actually implement it for a production system, the squared loss seems to be the best bet based on these results. Meng But I still have to ask about deployment costs. If this requires estimating these ideal distributions, does that make it too slow for real-time inference?
Jane: That’s a valid concern, Meng. The paper doesn't really dive deep into the computational cost of calculating those reweightings, but they show it works across various dataset settings. Lu They also showed a generalization to phi divergences, meaning you can use any family of Csiszár f-divergences as the measure of dissimilarity D Tom That opens up more flexibility for different types of loss functions.
Meng: Flexibility is good, but I need to know if that flexibility translates into actual speed gains or just more complexity in the code Jane The real improvement they highlight is that it makes inference robust across a variety of dataset and corruption combinations, which is something we need when dealing with messy real-world data.
Conclusion: Tom: So to wrap up, the paper "Density-Ratio Losses for Post-Hoc Learning to Defer" introduces a method where you learn deferral mechanisms based on ideal distributions and the link between density ratio estimation and class probability estimation. Jane Essentially, they’ve connected abstract ideas about ideal distributions to a concrete loss function that allows us to learn when to defer without needing any retraining of the base model.
Lu: The big implication is that this framework provides a principled way to learn these deferral mechanisms by comparing the model's ideal distribution against an expert's, and it shows how this density ratio leads directly into hypothesis testing. Tom It’s about making sure that when we use a base model, we have a rigorous way to decide when its prediction is unreliable compared to what an expert would have done.
Meng: For deployment, the results suggest that if you use the squared loss formulation with this method, you should get good accuracy-deferral trade-offs and robustness across different data scenarios Jane But they did point out a limitation: they don't explore consistency guarantees in other L2D approaches that use surrogate loss functions Lu
Lalam: I think the limitation they mention is about formal guarantees, because while the results are strong, they aren't providing consistency guarantees in the same way some other L2D methods do Tom That means we don't have a proof that this method will always work perfectly under every single condition.
Jane: So overall, it’s a really solid approach for learning deferral mechanisms based on these ideal distributions, and it gives us a tool to evaluate model reliability when we deploy them in things like health applications Lu
Tom: That’s the gist of "Density-Ratio Losses for Post-Hoc Learning to Defer." A lot of work connecting these concepts is really happening right now, so keep an eye on this direction.
Google Research
stat.ML, cs.LG
Submitted: 2026-05-19
Updated: 2026-11-10
Comments: NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 95/100
The gist: We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss The DR CPE losses…
Key concepts
- Ideal Distribution
- This is a reweighted version of the original data distribution where a model performs best, measured within a certain distance. It helps define the 'best' possible performance scenario for the classifier.
- Density-Ratio Deferral Rule
- Deferral is decided by comparing how much more likely an expert distribution is compared to the model's ideal distribution. This rule acts like a likelihood ratio test, determining when to trust the expert over the model.
- DR CPE Loss
- This loss function is derived by reducing Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE). It creates a loss that guides the model to learn deferral decisions based on class probabilities.
Terminology
Summary
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss The DR CPE losses are derived using a reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE) This approach bridges Chow-style rules, expert comparison, and connections to related learning settings like anomaly detection. p. 0 Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals.Where the paper says it
Core Methodology
The paper introduces a distributional formulation of post-hoc L2D based on ideal distributions Deferral is defined by thresholding the density-ratio between model and expert ideal distributions The framework accommodates both marginal and joint ideal distributions, showing that for KL-ideal distributions, the marginal rule recovers Chow’s rule w.r.t. the data distribution The joint rule corresponds to a Chow-style decision under an expert-tilted distribution The DR CPE losses are derived by reducing DRE to CPE, leading to a family of DR CPE post-hoc deferral losses. p. 1 • The DR CPE losses: Using the reduction from Density-Ratio Estimation (DRE) to Class-Probability
Estimation (CPE), we derive a family of DR CPE post-hoc deferral losses. This further connects our
method to expert-comparison deferral estimation [Kag et al.Where the paper says it
Ideal Distribution Framework
An ideal distribution is defined as a divergence-regularized reweighting of the data distribution under which a classifier-loss pair performs best at within a specified (regularized) distance away from the true probability measure P Two types of ideal distributions are considered: an (D, γ)-marginal ideal distribution Qm in marginal space and an (D, γ)-joint ideal distribution Qj in joint space The density-ratio deferral rule is given by J(dQ/dQ(e)) ≤ τ K This deferral decision has a direct connection to hypothesis testing, corresponding exactly to optimal hypothesis testing in the Neyman-Pearson sense, i.e., it is a likelihood ratio test. p. 3 Thresholding density-ratios corresponds exactly to optimal hypothesis testing in the Neyman-Pearson
sense [Neyman and Pearson, 1933, Lehmann and Romano, 2005], i. , it is a likelihood ratio test.Where the paper says it
Loss Derivation and Optimization
The DR CPE loss is derived using the reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE) This reduction involves finding a scoring function s that minimizes a composite CPE objective The resulting density-ratio estimator is approximated by ρˆ(x; s) ≈ dµ/dν using Bayes’ rule The joint DR CPE loss Lj(s; γ, γ(e)) has a convenient form that can be calculated as an expectation of a random variable w.r.t the joint data measure P This objective is shown to directly upper bound the marginal DR CPE loss Lm. p. 7 5), the
objective we would like to minimize is the marginal DR CPE loss Lm. Although Lm is not easy to estimate,
through Jensen’s inequality, one can directly show that the joint DR CPE loss Lj directly upper bounds the
marginal DR CPE loss Lm.Where the paper says it
Empirical Validation and Performance
The approach performs competitively against other L2D baselines and is more robust across dataset settings Experiments on CIFAR-100, PathMNIST, and DermaMNIST show that DR CPE performs best to second best over corrupted settings The comparison with baselines like Confidence-Based Deferral shows that DR CPE is consistently better than the confidence-based and two-stage approaches The results indicate that the squared loss seems to be the best performing with LSIF and KLIEP performing similarly.
Connections to Expert Comparison
The joint ideal DR CPE losses connect to various expert-comparison estimation approaches Proposition 4.5 shows that for the squared loss, the unique pointwise minimizer s⋆0(x) of the limiting risk satisfies Js⋆0(x) ≤ τ K = rPP[f(X) = Y X = x] ≤ τPP[f(e)(X) = Y X = x] This suggests that the optimal scorer is intuitive, aiming to reject whenever the base model’s (conditional) accuracy is worse than what the expert would have up to a discount scaling τ. p. 8 Let Σ(x)
def
=
PP[f(X) = Y | X = x] + PP[f
(e)
(X) = Y | X = x]. Then, for Px-almost every x with Σ(x) > 0, the unique
pointwise minimizer s
⋆
0
(x) of the limiting risk satisfies, for any τ ∈ R,
Js
⋆
0
(x) ≤ τ K =
r
E[Y
(d) | X = x] ≤ τ Σ(x)
z
. 1
Model & Expert Ideals
Qm, Q
(e)
m / Qj
, Q
(e)
j
Comparing Ideals
Definition 3. 2
Density-Ratio Deferral
J(dQ/dQ(e)
)(x) ≤ τ K
The DR CPE Loss
Definition 4. Interestingly, a direct connection can be made to the joint ideal DR CPE loss. Proposition 4.Where the paper says it
Where the paper says it
Where the paper says it
Generalization to φ-Divergences
A generalization of the optimal ideal distributions is presented by considering the family of φ-divergences (a.k.a., Csiszár f-divergences) as the dissimilarity D The resulting optimal ratio ρ⋆(z) is given by max 0,(φ∗′+(b - L(z)) This result follows immediately from Theorem C. p. 4 3 follows directly from a more general result that holds for φ-divergences (a. , Csiszár
f-divergences), as shown in Appendix C. (5)
An advantage of making deferral decision through density ratios is that the functional form of ri becomes
convenient once we make our choice of divergence D (i. , by considering φ-divergences).Where the paper says it
Where the paper says it
Broader Implications
L2D inherently has impact in various downstream tasks, such as in health, where one should ensure that the deferral rates τ selected are validated to ensure that unreliable base model predictions are not utilized at a high rate The paper also notes potential issues regarding the fairness and privacy of the overarching deferring system.
Limitations and Future Directions
A theoretical limitation is that it does not have consistency guarantees explored in other L2D approaches which utilize surrogate loss functions However, some finite sample bounds could be established similarly to the expertcomparison estimation approaches The temperature τ can have a non-trivial effect on performance through experiments detailed in the Appendix H.
Conclusion
In this paper, we present a novel method for learning deferral mechanisms based on ideal distributions and the link between Density Ratio Estimation (DRE) and Class-Probability Estimation (CPE) Practically, the DR CPE loss function can yield good accuracy-deferral trade-offs while also being robust in its performance over a variety of dataset and corruption combinations.
Acknowledgments
The authors would like to thank Aditya Krishna Menon, Wittawat Jitkrittum, and Harikrishna Narasimhan for various fruitful discussions.
References
Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. p. 5 Indeed, η
⋆ only captures the (aleatoric) label uncertainty under
the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s
comparative advantage. The tilted posterior η˜
⋆
(x; γ) incorporates this additional information by reweighting
labels according to the expert loss y 7→ ℓ(x, y, h(e)
(x)). A useful categorization of these approaches are by how abstention is represented. In score-based
approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option
rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and
Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al. A useful categorization of these approaches are by how abstention is represented. In score-based
approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option
rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and
Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. , 2023]), where the base model is fixed, and only the deferral rule is learned. In this setting, simple score-based heuristics can fail when downstream experts are specialized or under
distribution shift [Jitkrittum et al. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. , 2012] and Class-Probability Estimation (CPE) [Buja et al. , 2005, Reid
and Williamson, 2011], where we reduce the problem of estimating the density-ratio to a binary classification
problem. , 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi
et al. More
broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging
Chow-style rules, expert comparison, and elucidating connections to related learning settings including
anomaly detection. 1 Introduction
Machine learning (ML) models are now embedded in many tasks we encounter every day. Recall that to estimate the density-ratio dµ/dν between probability measures µ and ν, we
can first find a (binary) scoring function s: X → R that minimize the following CPE objective [Sugiyama
et al. , 2008, Menon and Ong, 2016, Nock et al. , η
⋆
y
(x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. Dual to the problem
of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for
systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases
with a smaller model [Viola and Jones, 2001, Mamou et al. , 2022, Varshney and Baral, 2022, Khalili et al. Experimentally,
our approach is competitive compared to common baselines and more robust across dataset settings. More
broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging
Chow-style rules, expert comparison, and elucidating connections to related learning settings including
anomaly detection. , η
⋆
y
(x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. More
broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging
Chow-style rules, expert comparison, and elucidating connections to related learning settings including
anomaly detection. 1 Introduction
Machine learning (ML) models are now embedded in many tasks we encounter every day. , 2017, Kadampur and
Al Riyaee, 2020, Zoabi et al. , 2021, Ma et al. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. Using the expert, in addition to the learning a classifier h, we can also jointly learn a rejector
r : X → {0, 1} via an objective [Mozannar and Sontag, 2020]:
L(h, r)
def = EX,Y∼P
h
(1 − r(X)) · ℓ(X, Y, h(X)) + r(X) · ℓ(X, Y, h(e)
(X))i
+ c · PX∼Px
[r(X) = 1], (1)
In Eq. Indeed, η
⋆ only captures the (aleatoric) label uncertainty under
the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s
comparative advantage. The tilted posterior η˜
⋆
(x; γ) incorporates this additional information by reweighting
labels according to the expert loss y 7→ ℓ(x, y, h(e)
(x)). , “[14-56]” 7→
h/h
(e) = ResNet14/56. Lines depict the mean and error bars depict ±1 s. The tilted posterior η˜
⋆
(x; γ) incorporates this additional information by reweighting
labels according to the expert loss y 7→ ℓ(x, y, h(e)
(x)). Consequently, applying a Chow-type rule under P˜γ
means deferring not simply when the labels are ambiguous, but when they are ambiguous in a way that favors
the expert. Using these expressions as
plug-in deferral rules would therefore require access to the Bayes-optimal predictor in advance, which defeats
the purpose of learning a good classifier h in the first place. Second, the deferral rule also depends on the
expert ideal distribution Q
(e)
i
, which in turn requires pointwise evaluation of the expert model h
(e)
(x) at
decision time. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow,
1957, 1970, Cortes et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow,
1957, 1970, Cortes et al. , 2016], where rejection is learned without access to the downstream expert used at
inference. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. , 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi
et al. (1)
requires a calibration of the cost c—we remind that the deferral regularized c·PPx
[r(X) = 1] can be interpreted
as a Lagrange multiplier relaxation of a hard deferral rate constraint PPx
[r(X) = 1] ≤ budget [Franc et al. In the worst case, this may require training multiple deferral mechanisms r from scratch for each c in
a cross validation scheme. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. , 2022] and DRO [Scarf et al. , 1957, Duchi et al. , 2005] (see Definition 4. • Empirical performance: Our DR CPE approach performs competitively against other L2D baselines and is
also more robust against different dataset corruption combinations than these baselines. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. Using the expert, in addition to the learning a classifier h, we can also jointly learn a rejector
r : X → {0, 1} via an objective [Mozannar and Sontag, 2020]:
L(h, r)
def = EX,Y∼P
h
(1 − r(X)) · ℓ(X, Y, h(X)) + r(X) · ℓ(X, Y, h(e)
(X))i
+ c · PX∼Px
[r(X) = 1], (1)
In Eq. , P(X = x | R = +1) = µ(x)—and base label rate π = P(R = +1). The label rate π is a free parameter which the user can set, where a typical choice is to set π based on the
ratio of empirical data used to optimize Eq. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. Instead of directly learning a classifier f, we will consider scorers h: X → R
L which are related
via f(x) = arg miny∈Y hy(x). To measure performance, we consider a loss function ℓ : X × Y × R
L
to define the expected risk EP[ℓ(X, Y, h(X))]. , η
⋆
y
(x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. One should note that even the “original” formulation of the L2D problem in Eq. (1)
requires a calibration of the cost c—we remind that the deferral regularized c·PPx
[r(X) = 1] can be interpreted
as a Lagrange multiplier relaxation of a hard deferral rate constraint PPx
[r(X) = 1] ≤ budget [Franc et al. A useful categorization of these approaches are by how abstention is represented. In score-based
approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option
rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and
Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al. , 2022, El-Yaniv et al. ,
2010, Geifman and El-Yaniv, 2017, 2019, Ni et al. , η
⋆
y
(x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. , 2022, El-Yaniv et al. ,
2010, Geifman and El-Yaniv, 2017, 2019, Ni et al. A useful categorization of these approaches are by how abstention is represented. In score-based
approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option
rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and
Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al. More
broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging
Chow-style rules, expert comparison, and elucidating connections to related learning settings including
anomaly detection. 1 Introduction
Machine learning (ML) models are now embedded in many tasks we encounter every day. , 2022, Gupta et al. , 2024, Narasimhan et al. This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the
prediction task better than the deferring model. From this perspective, Chow’s rule under the tilted posterior
ranks examples not only by uncertainty, but by uncertainty that is especially relevant to the expert. This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the
prediction task better than the deferring model. From this perspective, Chow’s rule under the tilted posterior
ranks examples not only by uncertainty, but by uncertainty that is especially relevant to the expert. , 2024, Narasimhan et al. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. As these systems
are deployed in increasingly high-stakes domains, including healthcare [Esteva et al. , 2017, Kadampur and
Al Riyaee, 2020, Zoabi et al. As these systems
are deployed in increasingly high-stakes domains, including healthcare [Esteva et al. , 2017, Kadampur and
Al Riyaee, 2020, Zoabi et al. Consequently, applying a Chow-type rule under P˜γ
means deferring not simply when the labels are ambiguous, but when they are ambiguous in a way that favors
the expert. This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the
prediction task better than the deferring model. Consequently, applying a Chow-type rule under P˜γ
means deferring not simply when the labels are ambiguous, but when they are ambiguous in a way that favors
the expert. This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the
prediction task better than the deferring model. In
such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise
can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed
unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem
of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for
systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases
with a smaller model [Viola and Jones, 2001, Mamou et al. In
such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise
can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed
unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem
of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for
systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases
with a smaller model [Viola and Jones, 2001, Mamou et al. Expert-Comparison Estimation: Previous work in the literature suggests learning deferral rules by
regressing pointwise targets that compare model and expert performance [Kag et al. , 2022, Jitkrittum et al. In particular, one seeks to estimate a pointwise expert-comparison target that approximates how
Chow’s rule makes deferral decisions. For example, one considers the zero-one loss function to design a
least-squares regression task on the random variable
Y
(d) def = Jf(X) = YK − Jf
(e)
(X) = YK. In particular, one seeks to estimate a pointwise expert-comparison target that approximates how
Chow’s rule makes deferral decisions. For example, one considers the zero-one loss function to design a
least-squares regression task on the random variable
Y
(d) def = Jf(X) = YK − Jf
(e)
(X) = YK. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow,
1957, 1970, Cortes et al. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow,
1957, 1970, Cortes et al. Indeed, η
⋆ only captures the (aleatoric) label uncertainty under
the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s
comparative advantage. The tilted posterior η˜
⋆
(x; γ) incorporates this additional information by reweighting
labels according to the expert loss y 7→ ℓ(x, y, h(e)
(x)). Indeed, η
⋆ only captures the (aleatoric) label uncertainty under
the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s
comparative advantage. The tilted posterior η˜
⋆
(x; γ) incorporates this additional information by reweighting
labels according to the expert loss y 7→ ℓ(x, y, h(e)
(x)). Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. 1 allows us to make deferral decision based on density-ratios
by thresholding in the scorer’s output space, rather than explicitly working in the space of density-ratio values. The Ideal DR CPE Loss With the general setup of DRE via CPE, we now define a pair of density-ratio loss
functions which correspond to the optimization problem of learning the density-ratio of ideal distributions. 1 allows us to make deferral decision based on density-ratios
by thresholding in the scorer’s output space, rather than explicitly working in the space of density-ratio values. The Ideal DR CPE Loss With the general setup of DRE via CPE, we now define a pair of density-ratio loss
functions which correspond to the optimization problem of learning the density-ratio of ideal distributions. 5 also provides a useful practical interpretation of deferral. In many applications, the Bayes class-posterior η
⋆
is not, on its own, the most relevant object
for deciding whether to predict or defer. 5 also provides a useful practical interpretation of deferral. In many applications, the Bayes class-posterior η
⋆
is not, on its own, the most relevant object
for deciding whether to predict or defer. Indeed, η
⋆ only captures the (aleatoric) label uncertainty under
the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s
comparative advantage. The tilted posterior η˜
⋆
(x; γ) incorporates this additional information by reweighting
labels according to the expert loss y 7→ ℓ(x, y, h(e)
(x)). , 2008, Menon and Ong, 2016, Nock et al. , 2016a]:
arg min
s: X→R
π · Eµ[(λ+1 ◦ s)(X)] + (1 − π) · Eν[(λ−1 ◦ s)(X)], (10)
where π ∈ (0, 1) and denote the partial losses λ+1(v)
def = λ(v, +1) and λ−1(v)
def = λ(v, −1). Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. A calibrated variant of the AD task can also be considered, where instead of just wanting to identify
anomalous observations, we want the learned scorer s (via a link function) to corresponds to the cumulative
distribution function of P [Menon and Williamson, 2018]. This form of AD mirrors that of learning deferral rules from ideal distributions—the only change is what is
considered a typical versus anomalous distribution. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. Instead, the scorer s
can be evaluated over a validation dataset and the threshold value can be specified w. a target deferral
percentage—a common approach for deferring via with scoring functions or confidence estimates [Pugnana
and Ruggieri, 2023a,b]. , η
⋆
y
(x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. Instead, the scorer s
can be evaluated over a validation dataset and the threshold value can be specified w. a target deferral
percentage—a common approach for deferring via with scoring functions or confidence estimates [Pugnana
and Ruggieri, 2023a,b]. , 2023]), where the base model is fixed, and only the deferral rule is learned. In this setting, simple score-based heuristics can fail when downstream experts are specialized or under
distribution shift [Jitkrittum et al. Finally, reduction-based approaches cast rejection as a
related supervised learning problem [Charoenphakdee et al. , 2021, Ramaswamy et al. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow,
1957, 1970, Cortes et al. , via a threshold Js(x) ≤ αqK with αq chosen to correspond to the qth quantile of the
scorer s. A calibrated variant of the AD task can also be considered, where instead of just wanting to identify
anomalous observations, we want the learned scorer s (via a link function) to corresponds to the cumulative
distribution function of P [Menon and Williamson, 2018]. , 2012] and Class-Probability Estimation (CPE) [Buja et al. , 2005, Reid
and Williamson, 2011], where we reduce the problem of estimating the density-ratio to a binary classification
problem. , 2012] and Class-Probability Estimation (CPE) [Buja et al. , 2005, Reid
and Williamson, 2011], where we reduce the problem of estimating the density-ratio to a binary classification
problem. DRE via CPE Before specifying the CPE optimization problem for learning the density-ratio between
ideal distributions, we revise how general DRE can be expressed as a CPE problem. Suppose that we have
a strictly proper composite loss λ: R × {±1} → R with invertible link function Φ: [0, 1] → R [Reid and Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. This can be desirable, since deferral is discouraged when neither the base
model h nor the expert h
(e)
is likely to be correct. Anomaly Detection (AD): One definition of anomalous observations are examples which are not concentrated [Ripley, 2007, Chandola et al. A useful categorization of these approaches are by how abstention is represented. In score-based
approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option
rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and
Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al. , Csiszár
f-divergences), as shown in Appendix C. In this more general case, the ideal distribution generated by
φ-divergences are also multiplicative re-weightings of P. More
broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging
Chow-style rules, expert comparison, and elucidating connections to related learning settings including
anomaly detection. 1 Introduction
Machine learning (ML) models are now embedded in many tasks we encounter every day. , 2024, Narasimhan et al. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. , 2021, Ma et al. , 2024, Singhal et al. (14) to yield an unbiased estimator. A natural question is how are these two objectives related. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. , 2022, Gupta et al. , 2024, Narasimhan et al. , 2024, Narasimhan et al. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. As these systems
are deployed in increasingly high-stakes domains, including healthcare [Esteva et al. , 2017, Kadampur and
Al Riyaee, 2020, Zoabi et al. Expert-Comparison Estimation: Previous work in the literature suggests learning deferral rules by
regressing pointwise targets that compare model and expert performance [Kag et al. , 2022, Jitkrittum et al. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. A second, orthogonal
distinction is between end-to-end methods and post-hoc (two-stage) methods. The post-hoc regime is especially important since retraining modern pretrained models can be prohibitively expensive. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. Dual to the problem
of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for
systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases
with a smaller model [Viola and Jones, 2001, Mamou et al. , 2022, Varshney and Baral, 2022, Khalili et al. • The DR CPE losses: Using the reduction from Density-Ratio Estimation (DRE) to Class-Probability
Estimation (CPE), we derive a family of DR CPE post-hoc deferral losses. This further connects our
method to expert-comparison deferral estimation [Kag et al. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. [2024] that Eqs. (3) and (4) resemble the optimization problems associated
with GVI [Zellner, 1988, Knoblauch et al. We note that the definition abuses notation slightly, where the divergence D
corresponds to both a divergence on the marginal space and also on the joint space. Generalized Variational Inference (GVI) and Distributionally Robust Optimization (DRO): It has been
previously noted in Soen et al. Density-Ratio Losses for Post-Hoc Learning to Defer
Alexander Soen⋄ Ragnar Thobaben⋄
Joakim Jaldén⋄ Richard Nock†
KTH⋄ Google Research†
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral
via the density-ratio between a model’s and an expert’s ideals. The regularization weights γ control how far the ideal distributions are from
the ground truth distribution P. We note that the definition abuses notation slightly, where the divergence D
corresponds to both a divergence on the marginal space and also on the joint space. For fixed classifier h, the Bayes optimal deferral rule r
⋆ of Eq. (1)
is given by
r
⋆
(x; c, P) = r
EY∼η
⋆(x)
h
ℓ(x, Y, h(e)
(x)) − ℓ(x, Y, h(x))i
≤ c
z
, (2)
where JpredK are Iverson brackets which evaluate to 1 when the predicate pred is true and is equal 0 when
the predicate is false [Knuth, 1992]. (1)
requires a calibration of the cost c—we remind that the deferral regularized c·PPx
[r(X) = 1] can be interpreted
as a Lagrange multiplier relaxation of a hard deferral rate constraint PPx
[r(X) = 1] ≤ budget [Franc et al. In the worst case, this may require training multiple deferral mechanisms r from scratch for each c in
a cross validation scheme. As a
result, the scorer s can then be converted to a density-ratio estimator between µ and ν via Bayes’ rule [Bickel
et al. , 2009, Menon and Ong, 2016]
ρˆ(x; s)
def =
1 − π
π
·
(Φ−1 ◦ s)(x)
1 − (Φ−1 ◦ s)(x)
≈
dµ
dν
. 2 has a direct connection to hypothesis testing. Thresholding density-ratios corresponds exactly to optimal hypothesis testing in the Neyman-Pearson
sense [Neyman and Pearson, 1933, Lehmann and Romano, 2005], i. , by considering φ-divergences). Hypothesis Testing: The deferral decision of Definition 3. , 2023] and anomaly
detection [Steinwart et al. , 2005] (see Definition 4. , 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi
et al. , 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi
et al. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. , via a threshold Js(x) ≤ αqK with αq chosen to correspond to the qth quantile of the
scorer s. A calibrated variant of the AD task can also be considered, where instead of just wanting to identify
anomalous observations, we want the learned scorer s (via a link function) to corresponds to the cumulative
distribution function of P [Menon and Williamson, 2018]. , η
⋆
y
(x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer
(L2D), alongside the typical optimization of classification model h: X → R
L, we introduce an expert model
h
(e)
: X → R
L. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. 19557v1 [stat. ML] 19 May 2026 This form of AD mirrors that of learning deferral rules from ideal distributions—the only change is what is
considered a typical versus anomalous distribution. Here, the typical distribution is considered as the data
distribution itself P and anything which deviates from it is considered anomalous, i. , 2024, Narasimhan et al. Learning to defer (L2D) proposes a
solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to
defer their predictions to an upstream (potentially human) expert [Madras et al. In
such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise
can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed
unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem
of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for
systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases
with a smaller model [Viola and Jones, 2001, Mamou et al. In
such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise
can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed
unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem
of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for
systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases
with a smaller model [Viola and Jones, 2001, Mamou et al. Using the expert, in addition to the learning a classifier h, we can also jointly learn a rejector
r : X → {0, 1} via an objective [Mozannar and Sontag, 2020]:
L(h, r)
def = EX,Y∼P
h
(1 − r(X)) · ℓ(X, Y, h(X)) + r(X) · ℓ(X, Y, h(e)
(X))i
+ c · PX∼Px
[r(X) = 1], (1)
In Eq. (1), we explicitly separate the expert’s performance ℓ(X, Y, h(e)
(X)) and the additional cost of calling
the expert c ≥ 0. , 2021, Ramaswamy et al. L2D extends
1
arXiv:2605. this picture by incorporating an expert into training [Madras et al. Here too, one can distinguish
score-based formulations, where deferral is encoded through shared scores or defer classes [Mozannar and
Sontag, 2020, Verma and Nalisnick, 2022, Mao et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Use Density-Ratio Deferral to define deferral via
a divergence-regularized reweighting of the data distribution under which a model attains low loss,
allowing deferral decisions to be made bythresholding a density-ratio.
-
Implement a joint ideal distribution framework for L2D, where deferral rules are derived from comparing the
density-ratio between these ideal distributions
of the model and the expert, leading to rules that correspond toa Chow-style decision under an expert-tilted distribution.
-
Derive Density-Ratio CPE losses by using
the reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE),
which connects deferral estimation toexpert-comparison deferral estimation
and anomaly detection. -
Adapt the learned scorer to mimic optimal decision making by showing that for certain loss functions, the optimal scorer is
ph(x)/pe(x),
meaning it can be used to reject wheneverthe base model’s (conditional) accuracy is worse than what the expert would have.
Abstract
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergence-regularized reweightings of the data distribution under which a model attains low task loss. We define deferral via the density-ratio between a model's and an expert's ideals. Using the reduction from density-ratio estimation to class-probability estimation, we derive the DR CPE losses for post-hoc L2D scorers. Deferral decisions are then made by thresholding the scorer. For KL-based ideal distributions, our deferral rule recovers Chow's rule under the original distribution and a connection to an expert-tilted Bayes posterior---which incorporates the expert's performance---depending on if the ideal distributions are marginal or joint distributions. This leads to two variants of the DR CPE losses, with the joint variant being computationally convenient in practice. We establish consistency properties of the joint DR CPE losses when considering the zero-one task loss and strongly proper composite CPE losses. Experimentally, our approach is competitive compared to common baselines and is more robust across dataset settings. More broadly, our results cast post-hoc L2D as density-ratio estimation between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection.
Sources
- Budgeted Multiple-Expert Deferral
- Language Model Cascades
- BabyBear: Cheap inference triage for expensive language models
- Adam: A Method for Stochastic Optimization
- The Inductive Bias of Restricted f-GANs
- When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer
- TangoBERT: Reducing Inference Cost by using Cascaded Architecture
- Change of measure through the Legendre transform
- A Comprehensive Survey on Machine Learning Techniques and User Authentication Approaches for Credit Card Fraud Detection
- A Unified Framework for Multi-distribution Density Ratio Estimation
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference