Density-Ratio Losses for Post-Hoc Learning to Defer

arXiv:2605.19557 · stat.ML, cs.LG · Submitted 2026-05-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Density-Ratio Losses for Post-Hoc Learning to Defer".

Jane: Here's the gist of the paper. We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss The DR CPE losses are derived using a reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE) This approach bridges Chow-style rules, expert comparison, and connections to related learning settings like anomaly detection.

Tom: Core Methodology The paper introduces a distributional formulation of post-hoc L2D based on ideal distributions Deferral is defined by thresholding the density-ratio between model and expert ideal distributions The framework accommodates both marginal and joint ideal distributions, showing that for KL-ideal distributions, the marginal rule recovers Chow’s rule w.r.t.

Jane: the data distribution The joint rule corresponds to a Chow-style decision under an expert-tilted distribution The DR CPE losses are derived by reducing DRE to CPE, leading to a family of DR CPE post-hoc deferral losses.

Tom: Ideal Distribution Framework An ideal distribution is defined as a divergence-regularized reweighting of the data distribution under which a classifier-loss pair performs best at within a specified (regularized) distance away from the true probability measure P Two types of ideal distributions are considered: an (D, γ)-marginal ideal distribution Qm in marginal space and an (D, γ)-joint ideal distribution Qj in joint space The density-ratio deferral rule is given by J(dQ/dQ(e)) ≤ τ K This deferral decision has a direct connection to hypothesis testing, corresponding exactly to optimal hypothesis testing in the Neyman-Pearson sense, i.e., it is a likelihood ratio test.

Title and authors: Tom: Okay, so the title is "Density-Ratio Losses for Post-Hoc Learning to Defer." It's about how you can learn when a model should defer its decision based on comparing its ideal distribution with an expert's ideal distribution. Jane That sounds like we're moving away from just looking at the model versus the data, and instead comparing two different ideal setups.

Lu: Exactly. The core idea they introduce is defining deferral by thresholding the density ratio between a model’s ideal distribution and an expert’s ideal distribution. Tom So if we can learn that ratio, we can make decisions about when to trust the model versus when to defer based on that comparison.

Meng: I'm wondering how this translates into something I could actually deploy. Is it just a theoretical trick, or does it give us a real way to handle uncertainty without retraining? Jane That’s the big question for production, Meng. The paper suggests that for ideal distributions based on Kullback-Leibler divergence, the deferral rules they derive actually recover Chow's rule under the original data distribution.

Tom: So even with these ideal distributions, if you're looking at just one marginal distribution instead of the joint one, you still get a familiar decision rule. Lu That’s because it connects to an expert-tilted Bayes posterior depending on whether those ideal distributions are marginal or joint ones Jane So they show that this framework bridges Chow-style rules and connections to expert comparison in a way that feels grounded.

Meng: If we're focusing only on the marginal distribution, it sounds like we get something predictable about when to defer. Lalam I think what’s interesting is that they connect this density-ratio concept directly to hypothesis testing, specifically optimal hypothesis testing in the Neyman-Pearson sense, which means it's essentially a likelihood ratio test Tom So they’re saying the mathematical structure of comparing these distributions leads right into a standard statistical test framework.

Jane: That’s a really concrete connection. It moves the idea from abstract probability to something you might actually recognize in testing scenarios.

The paper's summary: Tom: So, summarizing the main point of "Density-Ratio Losses for Post-Hoc Learning to Defer," they're defining an ideal distribution as a divergence-regularized reweighting of the data distribution under which a model performs best at some distance from the true probability measure P Jane That means instead of just training on real data, we're thinking about what kind of world would make our model perform best, and that world is defined by this ideal distribution.

Lu: Right. They define two types: a marginal ideal distribution and a joint ideal distribution. The paper uses the density-ratio between these two ideals to set the deferral rule, which they call J of dQ over dQ of e Jane This ratio is then compared to some threshold K, and that comparison dictates whether we defer or not.

Meng: I’m picturing a scenario where we have our model and an expert. The paper says this density-ratio deferral rule has a direct connection to hypothesis testing, which is important because it suggests the decision mechanism itself is based on optimal hypothesis testing Tom It's like the system is designed to make the best possible choice under those specific distributional assumptions.

Jane: And they derive their actual loss functions, called DR CPE losses, by doing a reduction from Density-Ratio Estimation to Class-Probability Estimation. That’s how they connect this abstract idea to a measurable loss function we can actually optimize against Lu This reduction leads to these family of DR CPE post hoc deferral losses which are the main focus of their optimization work.

Tom: So, they take this density ratio concept and map it onto a class probability estimation problem, which is how they derive the actual loss functions for deferral. Jane It’s really about making that complex idea of ideal distributions something we can actually calculate as an expectation over the data measure P Meng And they showed that this joint DR CPE loss directly bounds the marginal DR CPE loss, which is a nice property for their optimization process.

Lu: That bounding property is key, because it means if you optimize the joint version, you get information about the simpler marginal version too Tom It shows a structural relationship between those two different ways of looking at it.

The paper's improvements: Tom: Moving on to what they actually suggest as improvements, they show that this approach performs better than some other learning to defer baselines and it’s more robust when you change the dataset settings Jane They tested this on CIFAR-one hundred PathMNIST, and DermaMNIST and found that DR CPE performs best to second best over corrupted settings.

Meng: Second best is good, but I need to know how much better it is than something like confidence-based deferral or those two-stage approaches they mentioned Tom The paper says it’s consistently better than the confidence-based and two-stage approaches in their experiments.

Lu: They found that when using squared loss, this method seems to be the best performing option, and they saw that LSIF and KLIEP performed similarly with it Jane That suggests that the squared loss formulation is really compatible with this ideal distribution framework for maximizing performance.

Tom: So if we’re looking at how to actually implement it for a production system, the squared loss seems to be the best bet based on these results. Meng But I still have to ask about deployment costs. If this requires estimating these ideal distributions, does that make it too slow for real-time inference?

Jane: That’s a valid concern, Meng. The paper doesn't really dive deep into the computational cost of calculating those reweightings, but they show it works across various dataset settings. Lu They also showed a generalization to phi divergences, meaning you can use any family of Csiszár f-divergences as the measure of dissimilarity D Tom That opens up more flexibility for different types of loss functions.

Meng: Flexibility is good, but I need to know if that flexibility translates into actual speed gains or just more complexity in the code Jane The real improvement they highlight is that it makes inference robust across a variety of dataset and corruption combinations, which is something we need when dealing with messy real-world data.

Conclusion: Tom: So to wrap up, the paper "Density-Ratio Losses for Post-Hoc Learning to Defer" introduces a method where you learn deferral mechanisms based on ideal distributions and the link between density ratio estimation and class probability estimation. Jane Essentially, they’ve connected abstract ideas about ideal distributions to a concrete loss function that allows us to learn when to defer without needing any retraining of the base model.

Lu: The big implication is that this framework provides a principled way to learn these deferral mechanisms by comparing the model's ideal distribution against an expert's, and it shows how this density ratio leads directly into hypothesis testing. Tom It’s about making sure that when we use a base model, we have a rigorous way to decide when its prediction is unreliable compared to what an expert would have done.

Meng: For deployment, the results suggest that if you use the squared loss formulation with this method, you should get good accuracy-deferral trade-offs and robustness across different data scenarios Jane But they did point out a limitation: they don't explore consistency guarantees in other L2D approaches that use surrogate loss functions Lu

Lalam: I think the limitation they mention is about formal guarantees, because while the results are strong, they aren't providing consistency guarantees in the same way some other L2D methods do Tom That means we don't have a proof that this method will always work perfectly under every single condition.

Jane: So overall, it’s a really solid approach for learning deferral mechanisms based on these ideal distributions, and it gives us a tool to evaluate model reliability when we deploy them in things like health applications Lu

Tom: That’s the gist of "Density-Ratio Losses for Post-Hoc Learning to Defer." A lot of work connecting these concepts is really happening right now, so keep an eye on this direction.

Google Research

stat.ML, cs.LG

Submitted: 2026-05-19

Updated: 2026-11-10

Comments: NeurIPS 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 95/100

The gist: We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss The DR CPE losses…

Key concepts

Ideal Distribution
This is a reweighted version of the original data distribution where a model performs best, measured within a certain distance. It helps define the 'best' possible performance scenario for the classifier.
Density-Ratio Deferral Rule
Deferral is decided by comparing how much more likely an expert distribution is compared to the model's ideal distribution. This rule acts like a likelihood ratio test, determining when to trust the expert over the model.
DR CPE Loss
This loss function is derived by reducing Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE). It creates a loss that guides the model to learn deferral decisions based on class probabilities.

Terminology

Summary

We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss The DR CPE losses are derived using a reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE) This approach bridges Chow-style rules, expert comparison, and connections to related learning settings like anomaly detection. p. 0

Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Core Methodology

The paper introduces a distributional formulation of post-hoc L2D based on ideal distributions Deferral is defined by thresholding the density-ratio between model and expert ideal distributions The framework accommodates both marginal and joint ideal distributions, showing that for KL-ideal distributions, the marginal rule recovers Chow’s rule w.r.t. the data distribution The joint rule corresponds to a Chow-style decision under an expert-tilted distribution The DR CPE losses are derived by reducing DRE to CPE, leading to a family of DR CPE post-hoc deferral losses. p. 1

Where the paper says it

• The DR CPE losses: Using the reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE), we derive a family of DR CPE post-hoc deferral losses. This further connects our method to expert-comparison deferral estimation [Kag et al.

Ideal Distribution Framework

An ideal distribution is defined as a divergence-regularized reweighting of the data distribution under which a classifier-loss pair performs best at within a specified (regularized) distance away from the true probability measure P Two types of ideal distributions are considered: an (D, γ)-marginal ideal distribution Qm in marginal space and an (D, γ)-joint ideal distribution Qj in joint space The density-ratio deferral rule is given by J(dQ/dQ(e)) ≤ τ K This deferral decision has a direct connection to hypothesis testing, corresponding exactly to optimal hypothesis testing in the Neyman-Pearson sense, i.e., it is a likelihood ratio test. p. 3

Where the paper says it

Thresholding density-ratios corresponds exactly to optimal hypothesis testing in the Neyman-Pearson sense [Neyman and Pearson, 1933, Lehmann and Romano, 2005], i. , it is a likelihood ratio test.

Loss Derivation and Optimization

The DR CPE loss is derived using the reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE) This reduction involves finding a scoring function s that minimizes a composite CPE objective The resulting density-ratio estimator is approximated by ρˆ(x; s) ≈ dµ/dν using Bayes’ rule The joint DR CPE loss Lj(s; γ, γ(e)) has a convenient form that can be calculated as an expectation of a random variable w.r.t the joint data measure P This objective is shown to directly upper bound the marginal DR CPE loss Lm. p. 7

Where the paper says it

5), the objective we would like to minimize is the marginal DR CPE loss Lm. Although Lm is not easy to estimate, through Jensen’s inequality, one can directly show that the joint DR CPE loss Lj directly upper bounds the marginal DR CPE loss Lm.

Empirical Validation and Performance

The approach performs competitively against other L2D baselines and is more robust across dataset settings Experiments on CIFAR-100, PathMNIST, and DermaMNIST show that DR CPE performs best to second best over corrupted settings The comparison with baselines like Confidence-Based Deferral shows that DR CPE is consistently better than the confidence-based and two-stage approaches The results indicate that the squared loss seems to be the best performing with LSIF and KLIEP performing similarly.

Connections to Expert Comparison

The joint ideal DR CPE losses connect to various expert-comparison estimation approaches Proposition 4.5 shows that for the squared loss, the unique pointwise minimizer s⋆0(x) of the limiting risk satisfies Js⋆0(x) ≤ τ K = rPP[f(X) = Y X = x] ≤ τPP[f(e)(X) = Y X = x] This suggests that the optimal scorer is intuitive, aiming to reject whenever the base model’s (conditional) accuracy is worse than what the expert would have up to a discount scaling τ. p. 8

Where the paper says it

Let Σ(x) def = PP[f(X) = Y | X = x] + PP[f (e) (X) = Y | X = x]. Then, for Px-almost every x with Σ(x) > 0, the unique pointwise minimizer s ⋆ 0 (x) of the limiting risk satisfies, for any τ ∈ R, Js ⋆ 0 (x) ≤ τ K = r E[Y (d) | X = x] ≤ τ Σ(x) z .

The optimal deferral mechanisms are essentially the same ratio thresholds reparameterized when comparing Propositions F. p. 2
Where the paper says it

1 Model & Expert Ideals Qm, Q (e) m / Qj , Q (e) j Comparing Ideals Definition 3. 2 Density-Ratio Deferral J(dQ/dQ(e) )(x) ≤ τ K The DR CPE Loss Definition 4.

3 and F.4 versus Proposition F. p. 8
Where the paper says it

Interestingly, a direct connection can be made to the joint ideal DR CPE loss. Proposition 4.

5.

Generalization to φ-Divergences

A generalization of the optimal ideal distributions is presented by considering the family of φ-divergences (a.k.a., Csiszár f-divergences) as the dissimilarity D The resulting optimal ratio ρ⋆(z) is given by max 0,(φ∗′+(b - L(z)) This result follows immediately from Theorem C. p. 4

Where the paper says it

3 follows directly from a more general result that holds for φ-divergences (a. , Csiszár f-divergences), as shown in Appendix C.

4 by considering the generator of the KL-divergence. p. 3
Where the paper says it

(5) An advantage of making deferral decision through density ratios is that the functional form of ri becomes convenient once we make our choice of divergence D (i. , by considering φ-divergences).

Broader Implications

L2D inherently has impact in various downstream tasks, such as in health, where one should ensure that the deferral rates τ selected are validated to ensure that unreliable base model predictions are not utilized at a high rate The paper also notes potential issues regarding the fairness and privacy of the overarching deferring system.

Limitations and Future Directions

A theoretical limitation is that it does not have consistency guarantees explored in other L2D approaches which utilize surrogate loss functions However, some finite sample bounds could be established similarly to the expertcomparison estimation approaches The temperature τ can have a non-trivial effect on performance through experiments detailed in the Appendix H.

Conclusion

In this paper, we present a novel method for learning deferral mechanisms based on ideal distributions and the link between Density Ratio Estimation (DRE) and Class-Probability Estimation (CPE) Practically, the DR CPE loss function can yield good accuracy-deferral trade-offs while also being robust in its performance over a variety of dataset and corruption combinations.

Acknowledgments

The authors would like to thank Aditya Krishna Menon, Wittawat Jitkrittum, and Harikrishna Narasimhan for various fruitful discussions.

References

Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. p. 5

Where the paper says it

Indeed, η ⋆ only captures the (aleatoric) label uncertainty under the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s comparative advantage. The tilted posterior η˜ ⋆ (x; γ) incorporates this additional information by reweighting labels according to the expert loss y 7→ ℓ(x, y, h(e) (x)).

American Mathematical Soc., 2000 Peter L Bartlett and Marten H Wegkamp. p. 0
Where the paper says it

A useful categorization of these approaches are by how abstention is represented. In score-based approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al.

Classification with a Reject Option using a Hinge Loss. p. 0
Where the paper says it

A useful categorization of these approaches are by how abstention is represented. In score-based approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al.

Journal of Machine Learning Research, 9(8), 2008 Steffen Bickel, Michael Brückner, and Tobias Scheffer. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Discriminative learning under covariate shift. p. 1
Where the paper says it

, 2023]), where the base model is fixed, and only the deferral rule is learned. In this setting, simple score-based heuristics can fail when downstream experts are specialized or under distribution shift [Jitkrittum et al.

Journal of Machine Learning Research, 10(9), 2009 Andreas Buja, Werner Stuetzle, and Yi Shen. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Loss functions for binary class probability estimation and classification: Structure and applications. p. 5
Where the paper says it

, 2012] and Class-Probability Estimation (CPE) [Buja et al. , 2005, Reid and Williamson, 2011], where we reduce the problem of estimating the density-ratio to a binary classification problem.

Working draft, November, 3:13, 2005 Jenna Burrell. p. 0
Where the paper says it

, 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi et al.

How the machine ‘thinks’: Understanding opacity in machine learning algorithms. p. 0
Where the paper says it

More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection. 1 Introduction Machine learning (ML) models are now embedded in many tasks we encounter every day.

Big data & society, 3(1):2053951715622512, 2016 Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. p. 6
Where the paper says it

Recall that to estimate the density-ratio dµ/dν between probability measures µ and ν, we can first find a (binary) scoring function s: X → R that minimize the following CPE objective [Sugiyama et al. , 2008, Menon and Ong, 2016, Nock et al.

Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. p. 2
Where the paper says it

, η ⋆ y (x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L.

Advances in neural information processing systems, 35:521–534, 2022 Varun Chandola, Arindam Banerjee, and Vipin Kumar. p. 0
Where the paper says it

Dual to the problem of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases with a smaller model [Viola and Jones, 2001, Mamou et al. , 2022, Varshney and Baral, 2022, Khalili et al.

Anomaly detection: A survey. p. 0
Where the paper says it

Experimentally, our approach is competitive compared to common baselines and more robust across dataset settings. More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection.

ACM computing surveys (CSUR), 41(3):1–58, 2009 Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with Rejection Based on Cost-sensitive Classification. p. 2
Where the paper says it

, η ⋆ y (x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L.

In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1507–1517. p. 0
Where the paper says it

More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection. 1 Introduction Machine learning (ML) models are now embedded in many tasks we encounter every day.

PMLR, 2021 Mohammad-Amin Charusaie and Samira Samadi. p. 0
Where the paper says it

, 2017, Kadampur and Al Riyaee, 2020, Zoabi et al. , 2021, Ma et al.

A unifying post-processing framework for multi-objective learn-to-defer problems. p. 2
Where the paper says it

Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L. Using the expert, in addition to the learning a classifier h, we can also jointly learn a rejector r : X → {0, 1} via an objective [Mozannar and Sontag, 2020]: L(h, r) def = EX,Y∼P h (1 − r(X)) · ℓ(X, Y, h(X)) + r(X) · ℓ(X, Y, h(e) (X))i + c · PX∼Px [r(X) = 1], (1) In Eq.

Advances in Neural Information Processing Systems, 37:23705–23755, 2024 C. p. 5
Where the paper says it

Indeed, η ⋆ only captures the (aleatoric) label uncertainty under the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s comparative advantage. The tilted posterior η˜ ⋆ (x; γ) incorporates this additional information by reweighting labels according to the expert loss y 7→ ℓ(x, y, h(e) (x)).

Chow. On optimum recognition error and reject tradeoff. p. 9
Where the paper says it

, “[14-56]” 7→ h/h (e) = ResNet14/56. Lines depict the mean and error bars depict ±1 s.

IEEE Transactions on Information Theory, 16:41–46, 1970 Chi-Keung Chow. p. 5
Where the paper says it

The tilted posterior η˜ ⋆ (x; γ) incorporates this additional information by reweighting labels according to the expert loss y 7→ ℓ(x, y, h(e) (x)). Consequently, applying a Chow-type rule under P˜γ means deferring not simply when the labels are ambiguous, but when they are ambiguous in a way that favors the expert.

An optimum character recognition system using decision functions. p. 5
Where the paper says it

Using these expressions as plug-in deferral rules would therefore require access to the Bayes-optimal predictor in advance, which defeats the purpose of learning a good classifier h in the first place. Second, the deferral rule also depends on the expert ideal distribution Q (e) i , which in turn requires pointwise evaluation of the expert model h (e) (x) at decision time.

IRE Transactions on Electronic Computers, EC-6(4):247–254, 1957 Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. p. 0
Where the paper says it

Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow, 1957, 1970, Cortes et al.

Learning with rejection. p. 0
Where the paper says it

To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow, 1957, 1970, Cortes et al. , 2016], where rejection is learned without access to the downstream expert used at inference.

In International conference on algorithmic learning theory, pages 67–82. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Springer, 2016 Giulia DeSalvo, Clara Mohri, Mehryar Mohri, and Yutao Zhong. p. 0
Where the paper says it

, 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi et al.

Budgeted multiple-expert deferral. p. 6
Where the paper says it

(1) requires a calibration of the cost c—we remind that the deferral regularized c·PPx [r(X) = 1] can be interpreted as a Lagrange multiplier relaxation of a hard deferral rate constraint PPx [r(X) = 1] ≤ budget [Franc et al. In the worst case, this may require training multiple deferral mechanisms r from scratch for each c in a cross validation scheme.

arXiv preprint arXiv:2510. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

26706, 2025 David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein et al. Language model cascades. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

arXiv preprint arXiv:2207. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

10342, 2022 John C Duchi, Peter W Glynn, and Hongseok Namkoong. p. 3
Where the paper says it

, 2022] and DRO [Scarf et al. , 1957, Duchi et al.

Statistics of robust optimization: A generalized empirical likelihood approach. p. 1
Where the paper says it

, 2005] (see Definition 4. • Empirical performance: Our DR CPE approach performs competitively against other L2D baselines and is also more robust against different dataset corruption combinations than these baselines.

Mathematics of Operations Research, 46(3):946–969, 2021 Krishnamurthy Dj Dvijotham, Jamie Hayes, Borja Balle, Zico Kolter, Chongli Qin et al. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

A framework for robustness certification of smoothed classifiers using f-divergences. In International Conference on Learning Representations, 2020 Ran El-Yaniv et al. p. 2
Where the paper says it

Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L. Using the expert, in addition to the learning a classifier h, we can also jointly learn a rejector r : X → {0, 1} via an objective [Mozannar and Sontag, 2020]: L(h, r) def = EX,Y∼P h (1 − r(X)) · ℓ(X, Y, h(X)) + r(X) · ℓ(X, Y, h(e) (X))i + c · PX∼Px [r(X) = 1], (1) In Eq.

On the Foundations of Noise-free Selective Classification. p. 6
Where the paper says it

, P(X = x | R = +1) = µ(x)—and base label rate π = P(R = +1). The label rate π is a free parameter which the user can set, where a typical choice is to set π based on the ratio of empirical data used to optimize Eq.

Journal of Machine Learning Research, 11(5), 2010 F Escher. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

On the probability function in the collective theory of risk. p. 1
Where the paper says it

Instead of directly learning a classifier f, we will consider scorers h: X → R L which are related via f(x) = arg miny∈Y hy(x). To measure performance, we consider a loss function ℓ : X × Y × R L to define the expected risk EP[ℓ(X, Y, h(X))].

Skand. Aktuarie Tidskr., 15:175–195, 1932 Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. p. 2
Where the paper says it

, η ⋆ y (x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L.

nature, 542(7639):115–118 Vojtech Franc, Daniel Prusa, and Vaclav Voracek. p. 6
Where the paper says it

One should note that even the “original” formulation of the L2D problem in Eq. (1) requires a calibration of the cost c—we remind that the deferral regularized c·PPx [r(X) = 1] can be interpreted as a Lagrange multiplier relaxation of a hard deferral rate constraint PPx [r(X) = 1] ≤ budget [Franc et al.

Optimal strategies for reject option classifiers. p. 0
Where the paper says it

A useful categorization of these approaches are by how abstention is represented. In score-based approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al.

Journal of Machine Learning Research, 24(11):1–49, 2023 Yonatan Geifman and Ran El-Yaniv. p. 0
Where the paper says it

, 2022, El-Yaniv et al. , 2010, Geifman and El-Yaniv, 2017, 2019, Ni et al.

Selective classification for deep neural networks. p. 2
Where the paper says it

, η ⋆ y (x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L.

Advances in neural information processing systems, 30, 2017 Yonatan Geifman and Ran El-Yaniv. p. 0
Where the paper says it

, 2022, El-Yaniv et al. , 2010, Geifman and El-Yaniv, 2017, 2019, Ni et al.

SelectiveNet: A Deep Neural Network with an Integrated Reject Option. p. 0
Where the paper says it

A useful categorization of these approaches are by how abstention is represented. In score-based approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al.

In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2151–2159. p. 0
Where the paper says it

More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection. 1 Introduction Machine learning (ML) models are now embedded in many tasks we encounter every day.

PMLR, 09–15 Jun 2019 Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum et al. p. 0
Where the paper says it

, 2022, Gupta et al. , 2024, Narasimhan et al.

Language Model Cascades: Token-Level Uncertainty And Beyond. p. 5
Where the paper says it

This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the prediction task better than the deferring model. From this perspective, Chow’s rule under the tilted posterior ranks examples not only by uncertainty, but by uncertainty that is especially relevant to the expert.

p. 5
Where the paper says it

This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the prediction task better than the deferring model. From this perspective, Chow’s rule under the tilted posterior ranks examples not only by uncertainty, but by uncertainty that is especially relevant to the expert.

In The Twelfth International Conference on Learning Representations, 2024 Harikrishna Narasimhan, Wittawat Jitkrittum et al. p. 0
Where the paper says it

, 2024, Narasimhan et al. Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al.

Faster Cascades via Speculative Decoding. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

In The Thirteenth International Conference on Learning Representations, 2025 Mohammad Ali Kadampur and Sulaiman Al Riyaee. p. 0
Where the paper says it

As these systems are deployed in increasingly high-stakes domains, including healthcare [Esteva et al. , 2017, Kadampur and Al Riyaee, 2020, Zoabi et al.

p. 0
Where the paper says it

As these systems are deployed in increasingly high-stakes domains, including healthcare [Esteva et al. , 2017, Kadampur and Al Riyaee, 2020, Zoabi et al.

Skin cancer detection: Applying a deep learning based model driven architecture in the cloud for classifying dermal cell images. p. 5
Where the paper says it

Consequently, applying a Chow-type rule under P˜γ means deferring not simply when the labels are ambiguous, but when they are ambiguous in a way that favors the expert. This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the prediction task better than the deferring model.

p. 5
Where the paper says it

Consequently, applying a Chow-type rule under P˜γ means deferring not simply when the labels are ambiguous, but when they are ambiguous in a way that favors the expert. This can be preferable in practice, since deferral is only useful in-so-far that the expert handles the prediction task better than the deferring model.

Informatics in Medicine Unlocked, 18:100282, 2020 Anil Kag, Igor Fedorov, Aditya Gangrade et al. Efficient edge inference by selective query. p. 0
Where the paper says it

In such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases with a smaller model [Viola and Jones, 2001, Mamou et al.

p. 0
Where the paper says it

In such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases with a smaller model [Viola and Jones, 2001, Mamou et al.

In The Eleventh International Conference on Learning Representations, 2022 Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. p. 8
Where the paper says it

Expert-Comparison Estimation: Previous work in the literature suggests learning deferral rules by regressing pointwise targets that compare model and expert performance [Kag et al. , 2022, Jitkrittum et al.

A least-squares approach to direct importance estimation. p. 8
Where the paper says it

In particular, one seeks to estimate a pointwise expert-comparison target that approximates how Chow’s rule makes deferral decisions. For example, one considers the zero-one loss function to design a least-squares regression task on the random variable Y (d) def = Jf(X) = YK − Jf (e) (X) = YK.

p. 8
Where the paper says it

In particular, one seeks to estimate a pointwise expert-comparison target that approximates how Chow’s rule makes deferral decisions. For example, one considers the zero-one loss function to design a least-squares regression task on the random variable Y (d) def = Jf(X) = YK − Jf (e) (X) = YK.

The Journal of Machine Learning Research, 10:1391–1445, 2009 Alexander Soen. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

A Connection Between Learning to Reject and Bhattacharyya Divergences. p. 0
Where the paper says it

Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow, 1957, 1970, Cortes et al.

p. 0
Where the paper says it

Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow, 1957, 1970, Cortes et al.

In Geometric Science of Information, pages 369–377. p. 5
Where the paper says it

Indeed, η ⋆ only captures the (aleatoric) label uncertainty under the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s comparative advantage. The tilted posterior η˜ ⋆ (x; γ) incorporates this additional information by reweighting labels according to the expert loss y 7→ ℓ(x, y, h(e) (x)).

p. 5
Where the paper says it

Indeed, η ⋆ only captures the (aleatoric) label uncertainty under the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s comparative advantage. The tilted posterior η˜ ⋆ (x; γ) incorporates this additional information by reweighting labels according to the expert loss y 7→ ℓ(x, y, h(e) (x)).

Springer Nature Switzerland, 2026 Alexander Soen, Hisham Husain, Philip Schulz, and Vu Nguyen. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Rejection via Learning Density Ratios. p. 6
Where the paper says it

1 allows us to make deferral decision based on density-ratios by thresholding in the scorer’s output space, rather than explicitly working in the space of density-ratio values. The Ideal DR CPE Loss With the general setup of DRE via CPE, we now define a pair of density-ratio loss functions which correspond to the optimization problem of learning the density-ratio of ideal distributions.

p. 6
Where the paper says it

1 allows us to make deferral decision based on density-ratios by thresholding in the scorer’s output space, rather than explicitly working in the space of density-ratio values. The Ideal DR CPE Loss With the general setup of DRE via CPE, we now define a pair of density-ratio loss functions which correspond to the optimization problem of learning the density-ratio of ideal distributions.

In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 Jingzhao Zhang, Aditya Krishna Menon et al. A scaled Bregman theorem with applications. p. 5
Where the paper says it

5 also provides a useful practical interpretation of deferral. In many applications, the Bayes class-posterior η ⋆ is not, on its own, the most relevant object for deciding whether to predict or defer.

p. 5
Where the paper says it

5 also provides a useful practical interpretation of deferral. In many applications, the Bayes class-posterior η ⋆ is not, on its own, the most relevant object for deciding whether to predict or defer.

In Advances in Neural Information Processing Systems, volume 29. p. 5
Where the paper says it

Indeed, η ⋆ only captures the (aleatoric) label uncertainty under the data distribution, whereas the deferral decision should also reflect the downstream loss and the expert’s comparative advantage. The tilted posterior η˜ ⋆ (x; γ) incorporates this additional information by reweighting labels according to the expert loss y 7→ ℓ(x, y, h(e) (x)).

Curran Associates, Inc., 2016a Richard Nock, Aditya Menon, and Cheng Soon Ong. p. 6
Where the paper says it

, 2008, Menon and Ong, 2016, Nock et al. , 2016a]: arg min s: X→R π · Eµ[(λ+1 ◦ s)(X)] + (1 − π) · Eν[(λ−1 ◦ s)(X)], (10) where π ∈ (0, 1) and denote the partial losses λ+1(v) def = λ(v, +1) and λ−1(v) def = λ(v, −1).

A scaled Bregman theorem with applications. Advances in Neural Information Processing Systems, 29, 2016b Nastaran Okati, Abir De, and Manuel Gomez-Rodriguez. Differentiable learning under triage. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Advances in Neural Information Processing Systems, 34:9140–9151, 2021 Antoine Picard-Weibel and Benjamin Guedj. On change of measure inequalities for f-divergences. p. 8
Where the paper says it

A calibrated variant of the AD task can also be considered, where instead of just wanting to identify anomalous observations, we want the learned scorer s (via a link function) to corresponds to the cumulative distribution function of P [Menon and Williamson, 2018]. This form of AD mirrors that of learning deferral rules from ideal distributions—the only change is what is considered a typical versus anomalous distribution.

arXiv preprint arXiv:2202. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

05568, 2022 Andrea Pugnana and Salvatore Ruggieri. p. 6
Where the paper says it

Instead, the scorer s can be evaluated over a validation dataset and the threshold value can be specified w. a target deferral percentage—a common approach for deferring via with scoring functions or confidence estimates [Pugnana and Ruggieri, 2023a,b].

AUC-based Selective Classification. p. 2
Where the paper says it

, η ⋆ y (x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L.

In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 2494–2514. PMLR, 25–27 Apr 2023a Andrea Pugnana and Salvatore Ruggieri. p. 6
Where the paper says it

Instead, the scorer s can be evaluated over a validation dataset and the threshold value can be specified w. a target deferral percentage—a common approach for deferring via with scoring functions or confidence estimates [Pugnana and Ruggieri, 2023a,b].

A Model-Agnostic Heuristics for Selective Classification. p. 1
Where the paper says it

, 2023]), where the base model is fixed, and only the deferral rule is learned. In this setting, simple score-based heuristics can fail when downstream experts are specialized or under distribution shift [Jitkrittum et al.

Proceedings of the AAAI Conference on Artificial Intelligence, 37(8):9461–9469, Jun. 2023b Harish G Ramaswamy, Ambuj Tewari, and Shivani Agarwal. p. 0
Where the paper says it

Finally, reduction-based approaches cast rejection as a related supervised learning problem [Charoenphakdee et al. , 2021, Ramaswamy et al.

Consistent algorithms for multiclass classification with an abstain option. p. 0
Where the paper says it

Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al. To allow models to abstain from predictions, learning to reject (L2R) was initially proposed [Chow, 1957, 1970, Cortes et al.

Electronic Journal of Statistics, 12:530–554, 2018 Mark D. p. 8
Where the paper says it

, via a threshold Js(x) ≤ αqK with αq chosen to correspond to the qth quantile of the scorer s. A calibrated variant of the AD task can also be considered, where instead of just wanting to identify anomalous observations, we want the learned scorer s (via a link function) to corresponds to the cumulative distribution function of P [Menon and Williamson, 2018].

Reid and Robert C. p. 5
Where the paper says it

, 2012] and Class-Probability Estimation (CPE) [Buja et al. , 2005, Reid and Williamson, 2011], where we reduce the problem of estimating the density-ratio to a binary classification problem.

Williamson. p. 5
Where the paper says it

, 2012] and Class-Probability Estimation (CPE) [Buja et al. , 2005, Reid and Williamson, 2011], where we reduce the problem of estimating the density-ratio to a binary classification problem.

Composite Binary Losses. p. 5
Where the paper says it

DRE via CPE Before specifying the CPE optimization problem for learning the density-ratio between ideal distributions, we revise how general DRE can be expressed as a CPE problem. Suppose that we have a strictly proper composite loss λ: R × {±1} → R with invertible link function Φ: [0, 1] → R [Reid and

Journal of Machine Learning Research, 11:2387–2422, 2010 Brian D Ripley. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Pattern recognition and neural networks. Cambridge university press, 2007 R. p. 8
Where the paper says it

This can be desirable, since deferral is discouraged when neither the base model h nor the expert h (e) is likely to be correct. Anomaly Detection (AD): One definition of anomalous observations are examples which are not concentrated [Ripley, 2007, Chandola et al.

T. Rockafellar. Convex Analysis. p. 0
Where the paper says it

A useful categorization of these approaches are by how abstention is represented. In score-based approaches, abstention is derived from the predictor’s own scores, encompassing classical reject-option rules, convex-surrogate formulations, and confidence-threshold methods [Chow, 1957, 1970, Herbei and Wegkamp, 2006, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cao et al.

Princeton University Press, 1970 Vincent Roulet, Tianlin Liu, Nino Vieillard, Michael Eli Sander, and Mathieu Blondel. Loss Functions and Operators Generated by f-Divergences. p. 4
Where the paper says it

, Csiszár f-divergences), as shown in Appendix C. In this more general case, the ideal distribution generated by φ-divergences are also multiplicative re-weightings of P.

In Forty-second International Conference on Machine Learning, 2025 Herbert E Scarf, KJ Arrow, and S Karlin. p. 0
Where the paper says it

More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection. 1 Introduction Machine learning (ML) models are now embedded in many tasks we encounter every day.

A min-max solution of an inventory problem. p. 0
Where the paper says it

, 2024, Narasimhan et al. Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al.

Technical report, Rand Corporation Santa Monica, 1957 Karan Singhal et al. p. 0
Where the paper says it

, 2021, Ma et al. , 2024, Singhal et al.

Toward expert-level medical question answering with large language models. p. 7
Where the paper says it

(14) to yield an unbiased estimator. A natural question is how are these two objectives related.

Nature medicine, 31(3):943–950, 2025 Alexander Soen. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

A Connection Between Learning to Reject and Bhattacharyya Divergences. In Geometric Science of Information, pages 369–377. Springer Nature Switzerland, 2026 Alexander Soen, Hisham Husain, Philip Schulz, and Vu Nguyen. Rejection via Learning Density Ratios. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 Neha Gupta et al. p. 0
Where the paper says it

, 2022, Gupta et al. , 2024, Narasimhan et al.

Language Model Cascades: Token-Level Uncertainty And Beyond. In The Twelfth International Conference on Learning Representations, 2024 Harikrishna Narasimhan et al. p. 0
Where the paper says it

, 2024, Narasimhan et al. Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al.

Faster Cascades via Speculative Decoding. In The Thirteenth International Conference on Learning Representations, 2025 Mohammad Ali Kadampur and Sulaiman Al Riyaee. Skin cancer detection: Applying a deep learning based model driven architecture in the cloud for classifying dermal cell images. Informatics in Medicine Unlocked, 18:100282, 2020 Anil Kag et al. p. 0
Where the paper says it

As these systems are deployed in increasingly high-stakes domains, including healthcare [Esteva et al. , 2017, Kadampur and Al Riyaee, 2020, Zoabi et al.

Efficient edge inference by selective query. In The Eleventh International Conference on Learning Representations, 2022 Takafumi Kanamori et al. p. 8
Where the paper says it

Expert-Comparison Estimation: Previous work in the literature suggests learning deferral rules by regressing pointwise targets that compare model and expert performance [Kag et al. , 2022, Jitkrittum et al.

A least-squares approach to direct importance estimation. The Journal of Machine Learning Research, 10:1391–1445, 2009 Leila Khalili, Yao You, and John Bohannon. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

BabyBear: Cheap inference triage for expensive language models. p. 1
Where the paper says it

A second, orthogonal distinction is between end-to-end methods and post-hoc (two-stage) methods. The post-hoc regime is especially important since retraining modern pretrained models can be prohibitively expensive.

arXiv preprint arXiv:2205. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

11747, 2022 Diederik P Kingma and Jimmy Ba. p. 0
Where the paper says it

Dual to the problem of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases with a smaller model [Viola and Jones, 2001, Mamou et al. , 2022, Varshney and Baral, 2022, Khalili et al.

Adam: A method for stochastic optimization. p. 1
Where the paper says it

• The DR CPE losses: Using the reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE), we derive a family of DR CPE post-hoc deferral losses. This further connects our method to expert-comparison deferral estimation [Kag et al.

arXiv preprint arXiv:1412. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

6980, 2014 Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. p. 3
Where the paper says it

[2024] that Eqs. (3) and (4) resemble the optimization problems associated with GVI [Zellner, 1988, Knoblauch et al.

An optimization-centric view on Bayes’ rule: Reviewing and generalizing variational inference. p. 3
Where the paper says it

We note that the definition abuses notation slightly, where the divergence D corresponds to both a divergence on the marginal space and also on the joint space. Generalized Variational Inference (GVI) and Distributionally Robust Optimization (DRO): It has been previously noted in Soen et al.

Journal of Machine Learning Research, 23(132):1–109, 2022 Donald E Knuth. p. 0
Where the paper says it

Density-Ratio Losses for Post-Hoc Learning to Defer Alexander Soen⋄ Ragnar Thobaben⋄ Joakim Jaldén⋄ Richard Nock† KTH⋄ Google Research† Abstract We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergenceregularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model’s and an expert’s ideals.

Two Notes on Notation. p. 3
Where the paper says it

The regularization weights γ control how far the ideal distributions are from the ground truth distribution P. We note that the definition abuses notation slightly, where the divergence D corresponds to both a divergence on the marginal space and also on the joint space.

The American Mathematical Monthly, 99:403–422, 1992 Alex Krizhevsky. p. 2
Where the paper says it

For fixed classifier h, the Bayes optimal deferral rule r ⋆ of Eq. (1) is given by r ⋆ (x; c, P) = r EY∼η ⋆(x) h ℓ(x, Y, h(e) (x)) − ℓ(x, Y, h(x))i ≤ c z , (2) where JpredK are Iverson brackets which evaluate to 1 when the predicate pred is true and is equal 0 when the predicate is false [Knuth, 1992].

Learning multiple layers of features from tiny images. p. 6
Where the paper says it

(1) requires a calibration of the cost c—we remind that the deferral regularized c·PPx [r(X) = 1] can be interpreted as a Lagrange multiplier relaxation of a hard deferral rate constraint PPx [r(X) = 1] ≤ budget [Franc et al. In the worst case, this may require training multiple deferral mechanisms r from scratch for each c in a cross validation scheme.

Technical report, 2009 E. p. 6
Where the paper says it

As a result, the scorer s can then be converted to a density-ratio estimator between µ and ν via Bayes’ rule [Bickel et al. , 2009, Menon and Ong, 2016] ρˆ(x; s) def = 1 − π π · (Φ−1 ◦ s)(x) 1 − (Φ−1 ◦ s)(x) ≈ dµ dν .

L. Lehmann and Joseph P. p. 3
Where the paper says it

2 has a direct connection to hypothesis testing. Thresholding density-ratios corresponds exactly to optimal hypothesis testing in the Neyman-Pearson sense [Neyman and Pearson, 1933, Lehmann and Romano, 2005], i.

Romano. Testing Statistical Hypotheses. p. 3
Where the paper says it

, by considering φ-divergences). Hypothesis Testing: The deferral decision of Definition 3.

Springer International Publishing, 2005 Yingheng Li et al. p. 1
Where the paper says it

, 2023] and anomaly detection [Steinwart et al. , 2005] (see Definition 4.

Large language models in finance: A survey. p. 0
Where the paper says it

, 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi et al.

In Proceedings of the fourth ACM international conference on AI in finance, pages 374–382, 2023 Shuang Liu and Kamalika Chaudhuri. p. 0
Where the paper says it

, 2024, Singhal et al. , 2025] and finance [Burrell, 2016, Yousefi et al.

The inductive bias of restricted f-gans. arXiv preprint arXiv:1809. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

04542, 2018 Shuqi Liu et al. p. 8
Where the paper says it

, via a threshold Js(x) ≤ αqK with αq chosen to correspond to the qth quantile of the scorer s. A calibrated variant of the AD task can also be considered, where instead of just wanting to identify anomalous observations, we want the learned scorer s (via a link function) to corresponds to the cumulative distribution function of P [Menon and Williamson, 2018].

When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer. p. 2
Where the paper says it

, η ⋆ y (x) = P[Y = y | X = x]. Learning to Defer To move from the standard multi-class classification paradigm to learning to defer (L2D), alongside the typical optimization of classification model h: X → R L, we introduce an expert model h (e) : X → R L.

arXiv preprint arXiv:2602. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

17144, 2026 Jun Ma et al. p. 0
Where the paper says it

19557v1 [stat. ML] 19 May 2026

Segment anything in medical images. p. 8
Where the paper says it

This form of AD mirrors that of learning deferral rules from ideal distributions—the only change is what is considered a typical versus anomalous distribution. Here, the typical distribution is considered as the data distribution itself P and anything which deviates from it is considered anomalous, i.

Nature communications, 15(1):654, 2024 David Madras et al. p. 0
Where the paper says it

, 2024, Narasimhan et al. Learning to defer (L2D) proposes a solution to these problems by modifying the standard supervised learning setting to allow (smaller) models to defer their predictions to an upstream (potentially human) expert [Madras et al.

Predict responsibly: improving fairness and accuracy by learning to defer. p. 0
Where the paper says it

In such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases with a smaller model [Viola and Jones, 2001, Mamou et al.

Advances in neural information processing systems, 31, 2018 Jonathan Mamou et al. p. 0
Where the paper says it

In such high-stakes decision-making systems, ideally, a synthesis of both ML prediction and human expertise can be used [Topol, 2019]: ML models can defer to experts whenever their own predictions are deemed unreliable, and the burden on human experts can be reduced by leveraging ML models. Dual to the problem of predicting responsibly, the rise of huge ML models and edge deployment has created strong demand for systems that improve efficiency by routing only difficult inputs to a larger model while handling easier cases with a smaller model [Viola and Jones, 2001, Mamou et al.

Tangobert: Reducing inference cost by using cascaded architecture. p. 2
Where the paper says it

Using the expert, in addition to the learning a classifier h, we can also jointly learn a rejector r : X → {0, 1} via an objective [Mozannar and Sontag, 2020]: L(h, r) def = EX,Y∼P h (1 − r(X)) · ℓ(X, Y, h(X)) + r(X) · ℓ(X, Y, h(e) (X))i + c · PX∼Px [r(X) = 1], (1) In Eq. (1), we explicitly separate the expert’s performance ℓ(X, Y, h(e) (X)) and the additional cost of calling the expert c ≥ 0.

arXiv preprint arXiv:2204. p. 0
Where the paper says it

, 2021, Ramaswamy et al. L2D extends 1 arXiv:2605.

06271, 2022 Anqi Mao et al. p. 1
Where the paper says it

this picture by incorporating an expert into training [Madras et al. Here too, one can distinguish score-based formulations, where deferral is encoded through shared scores or defer classes [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022, Mao et al.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Use Density-Ratio Deferral to define deferral via a divergence-regularized reweighting of the data distribution under which a model attains low loss, allowing deferral decisions to be made by thresholding a density-ratio.

  2. Implement a joint ideal distribution framework for L2D, where deferral rules are derived from comparing the density-ratio between these ideal distributions of the model and the expert, leading to rules that correspond to a Chow-style decision under an expert-tilted distribution.

  3. Derive Density-Ratio CPE losses by using the reduction from Density-Ratio Estimation (DRE) to Class-Probability Estimation (CPE), which connects deferral estimation to expert-comparison deferral estimation and anomaly detection.

  4. Adapt the learned scorer to mimic optimal decision making by showing that for certain loss functions, the optimal scorer is ph(x)/pe(x), meaning it can be used to reject whenever the base model’s (conditional) accuracy is worse than what the expert would have.

Abstract

We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergence-regularized reweightings of the data distribution under which a model attains low task loss. We define deferral via the density-ratio between a model's and an expert's ideals. Using the reduction from density-ratio estimation to class-probability estimation, we derive the DR CPE losses for post-hoc L2D scorers. Deferral decisions are then made by thresholding the scorer. For KL-based ideal distributions, our deferral rule recovers Chow's rule under the original distribution and a connection to an expert-tilted Bayes posterior---which incorporates the expert's performance---depending on if the ideal distributions are marginal or joint distributions. This leads to two variants of the DR CPE losses, with the joint variant being computationally convenient in practice. We establish consistency properties of the joint DR CPE losses when considering the zero-one task loss and strongly proper composite CPE losses. Experimentally, our approach is competitive compared to common baselines and is more robust across dataset settings. More broadly, our results cast post-hoc L2D as density-ratio estimation between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection.

Sources

Related papers