Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning
stat.ML, cs.AI, cs.LG, math.ST, stat.TH
Submitted: 2024-08-29
Updated: 2026-09-25
Comments: This is a revision of a work previously posted to arXiv in 2024 by the same authors under the same arXiv number with a different title. The current version is revised to better highlight the key contributions. No AI was used in the production of this work
License: http://creativecommons.org/licenses/by/4.0/
The gist: Transfer learning, in regression or classification, concerns problems where one aims to use imperfect data from a source distribution P to improve prediction w.r.t.
Terminology
Abstract
Transfer learning, in regression or classification, concerns problems where one aims to use imperfect data from a source distribution P to improve prediction w.r.t. a target distribution Q. While there is now much theoretical understanding of the unsupervised setting of this problem (where no labeled target data is available), the supervised setting with labeled target data remains under-studied. We identify and characterize an interesting dichotomy in the rates achievable in the supervised setting: there is a weak transfer regime where optimal rates are determined by information in either the source or target dataset but not both, and a strong transfer regime where much faster target rates are achievable, accounting for complementary information in source and target data. Furthermore, while previous theoretical tools for the unsupervised setting can be extended to capture the weak transfer regime, we argue that they cannot capture the strong regime due to limitations inherent in the various notions of relatedness they rely on. Instead, we provide an appropriate refinement of relatedness notions---termed weak and strong moduli of transfer---that reveal the two regimes and lead to a nontrivial characterization of which pairs of distributions P and Q adhere to which regime. Finally, the above dichotomy reveals a new categorization of existing algorithmic approaches in terms of their optimality or lack-thereof across supervised regimes, and we provide a generic approach that remains nearly optimal in all regimes without prior distributional knowledge.
Sources
- On the ERM Principle in Meta-Learning
- Invariant Risk Minimization
- High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization
- Few-Shot Learning via Learning the Representation, Provably
- Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift
- Nonlinear Meta-Learning Can Guarantee Faster Rates
- Noisy recovery from random linear observations: Sharp minimax rates under elliptical constraints
- On the Theory of Transfer Learning: The Importance of Task Diversity
- Prominent Roles of Conditionally Invariant Components in Domain Adaptation: Theory and Algorithms
- On Localized Discrepancy for Domain Adaptation
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Efficient quadratic entropy with distance sketches
- Embedding-Bias in Conditional Independence Testing
- From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier