Multi-Source Transfer Learning of Sparse Single-Index Models
Ye Tian
stat.ME, stat.ML
Submitted: 2026-06-28
License: http://creativecommons.org/licenses/by/4.0/
The gist: Transfer learning leverages knowledge from related source domains to improve learning in a target domain.
Terminology
Abstract
Transfer learning leverages knowledge from related source domains to improve learning in a target domain. Recent theoretical advances cover a broad range of regression settings within (generalized) linear models. Despite their diversity, these methods share two common constraints: they assume a known link function or linear structure and require direct access to raw source data. To move beyond these constraints, we propose a source-data-free transfer learning framework based on the single-index model (SIM). Instead of requiring raw source data, our method transfers only summary statistics derived from a generalized Stein's lemma in a one-time communication. This design preserves privacy and avoids side effects caused by dissimilarities of unknown nonlinear link functions across domains. To capture flexible, unknown nonlinearity, we employ a multilayer perceptron guided by the pre-estimated index from the transferred statistics, which significantly mitigates overfitting. Extensive experiments on synthetic data and a real-world application demonstrate consistent improvements over existing (generalized) linear model-based approaches. The proposed framework thus offers a practical, privacy-preserving, and nonlinear-adaptive solution for transfer learning.
Sources
- Tensor Methods for Additive Index Models under Discordance and Heterogeneity
- Nonlinear denoising score matching for enhanced learning of structured distributions
- Profiled Transfer Learning for High Dimensional Linear Model
- Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression
- Nonlinear Multiple Response Regression and Learning of Latent Spaces
- Generalization error bound for denoising score matching under relaxed manifold assumption
Related papers
- Doubly robust inference via calibration
- Bayesian Empirical Bayes: Simultaneous Inference from Probabilistic Symmetries
- Weak-Form Recovery of Stochastic Generators
- Optimal Watermark Localization in Mixed-Source Large Language Model Texts
- Optimized Conformal Selection: Powerful Selective Inference After Data-Dependent Score Construction
- Small area learning