sigma Transfer: Uncertainty Transfer from Small to Large Networks under mu P

arXiv:2610.11668 · stat.ML, cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "sigma Transfer: Uncertainty Transfer from Small to Large Networks under mu P".

Jane: The gist The sigmaTransfer method introduces a rescaling of prior covariance under the Maximal Update Parametrization (µP) to enable zero-shot transfer of uncertainty precision from small models to much larger…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Looking at this paper, sigma Transfer really tackles the expensive part of Laplace approximations, which is picking that initial prior precision that controls all the uncertainty in the model. The authors show how using mu P lets you build a stable prior kernel regardless of how wide your neural network becomes.

Jane: They achieve this by creating a width-normalized prior kernel, and they prove that this leads to two forms of transfer: first, moving the precision itself from a small model to a large one, and second, transferring decisions made from those posteriors directly.

Lu: The results are pretty compelling because they show speedups up to five thousand times on MNIST when moving from width one hundred twenty-eight to four thousand ninety-six <ref:2610.11668#pg1,from width 128 to 4096>. That demonstrates how much you can cut down the search cost just by using this rescaling under mu P.

Meng: From an engineering standpoint, it means we don't always need to train and evaluate that massive posterior on a huge model just to make a better decision; we can use the proxy results and transfer them efficiently.

Tom: The paper supports this idea by showing that sigma Transfer preserves predictive performance while reducing the cost of uncertainty-sensitive tasks by up to several orders of magnitude. It’s about making uncertainty estimation much more practical for large-scale systems.

Jane: Ultimately, the big picture is that we can get reliable predictive uncertainty estimates on much larger models without the traditional exhaustive search for the best prior precision, because sigma Transfer supports two forms of transfer.

Conclusion: Tom: So we’re wrapping up on sigma Transfer, looking at this paper titled "sigma Transfer: Uncertainty Transfer from Small to Large Networks under mu P". Basically, it’s about figuring out how to reliably move uncertainty information—that tricky measure of how sure a model is—from a small network to a huge one without having to do all that incredibly slow searching on the big one.

Jane: Right. The core idea seems to be this rescaling of the prior covariance under what they call mu P, which makes the chosen precision stable even when you crank up the width of your neural network way past what you can practically handle.

Lu: It’s wild because they show that this stability isn't just theoretical; it holds up across regression, image classification, and even Transformer readouts. They prove convergence for the prior kernel, the posterior covariance, and that selected precision under specific conditions.

Meng: So if we think about what this means practically for building massive AI systems right now—like those billion-parameter models—it suggests we can skip that whole tedious search phase just by using a smaller model to set the initial uncertainty level.

Lalam: From an AI perspective, this could really speed up how we fine-tune these huge models because it handles the uncertainty estimation much more efficiently during deployment. It’s about making inference cheaper and more reliable across different scales of model size.

Tom: Exactly. It moves us away from having to treat every new large model as a completely fresh, expensive optimization problem from scratch. Jane, what’s the actual big picture implication here?

Jane: Well, it means we can get high-quality uncertainty estimates for massive models without having to build and run an enormous target posterior just to decide how precise our initial guess should be. It cuts the computational cost down by several orders of magnitude in some cases.

Lu: It’s about making uncertainty-sensitive tasks much more accessible for large-scale research and deployment because the barrier to entry for getting good uncertainty estimates drops significantly.

Meng: For engineers, that’s huge because it means we don't have to waste massive GPU time on prior selection if we can just transfer a good setting from a smaller, faster model.

Tom: It really is about making these complex uncertainty measurements more practical for real-world AI deployment across different scales. Next up, we’ll look at how they actually did this transfer in the experiments.

University of Cambridge

stat.ML, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist The sigmaTransfer method introduces a rescaling of prior covariance under the Maximal Update Parametrization (µP) to enable zero-shot transfer of uncertainty precision from small models to

Key concepts

Maximal Update Parametrization ($\mu$P)
This is a framework used to select the most reliable uncertainty precision. The paper shows that selecting this precision requires an expensive search process. $\mu$P provides a way to structure the problem so that selecting the right precision becomes stable and manageable as network size increases.
$\sigma$Transfer
This method uses a rescaling technique on the prior covariance geometry. It normalizes the prior such that it remains stable across different model widths. This stability allows researchers to choose a precision on a small, cheap model and reliably apply it to a massive, expensive model without searching for the precision again.
Prior-precision Transfer
This is the core mechanism where uncertainty precision is transferred. Because $\sigma$Transfer ensures that the optimal prior precision does not change drastically with network width, a precision found on a small network is nearly optimal for a very large one. This transfer works across various tasks, including image classification and Transformer outputs.
Decision Transfer
Beyond uncertainty values, $\sigma$Transfer allows for the transfer of actions or decisions (like acquisition rankings or OOD detection). This is achieved by using proxy results from the small model to guide decisions in the large model. Crucially, this transfer works without needing to build a full target posterior on the larger network.

Terminology

Summary

The gist The sigmaTransfer method introduces a rescaling of prior covariance under the Maximal Update Parametrization (µP) to enable zero-shot transfer of uncertainty precision from small models to much larger ones, which significantly reduces search costs and maintains predictive performance across network widths.

How it works

  1. σTransfer builds on µP by normalizing the prior geometry: σTransfer normalizes the prior geometry: each layer’s prior is scaled by a factor fixed by the width and µP’s coordinate scalings (Section 4).

  2. The method achieves this stability by constructing a width-normalized prior kernel: We now construct Sn so that the prior kernel Kn is width-stable. This involves introducing width-normalized parameter coordinates and setting Sn = TnT for the stored coordinates, which results in a stable large-width limit for the prior kernel: The reason for working in these coordinates is that we prove that the isotropic prior kernel in these width-normalized coordinates is width-stable.

  3. This stability leads to two forms of transfer: Prior-precision transfer: the optimal prior precision is stable across width (Figure 1(c)), so a precision chosen on a small (cheap) network is near-optimal for a large (expensive) one.

  4. For decision transfer, the proxy supplies posterior-derived decisions, such as acquisition rankings and active-learning selections, and the target requires no posterior at deployment.

Key Results

**: measured precision-sweep speedups reach ∼5000× when transferring from width 128 to 4096 on MNIST, at a target-NLL degradation of 0.002. The same posterior stability also enables transfer of acquisition, OOD-detection, and abstention decisions without constructing a target posterior. The paper shows that σTransfer remains stable across width. For example, the mean measured target-NLL increase below 10−4 across ten tasks when transferring from a public 1B to 7B model. The same posterior stability also enables transfer of acquisition, OOD-detection, and abstention decisions without constructing a target posterior. The speedup for evidence search on the public 1B→7B pair is a median search speedup of ∼2.3× (up to ∼330×), with a mean measured target-NLL increase below 10−4 across ten tasks. The results show that σTransfer has the smallest predictive transfer gap ∆NLL in every column. In decision transfer experiments, the proxy and target rankings agree with ρ¯ = 0.75 under σTransfer, against 0.36–0.43 for the three baselines. For language models, σTransfer has the highest mean Spearman correlation between proxy and target rankings of which data to label. The OOD-aware precision transfer results in a AUROC 0.950 ± 0.003 for width 4096. The decision transfer results show that the proxy-selected action is asymptotically optimal for the limiting acquisition objective 0>. The overall findings show that σTransfer preserves predictive performance while reducing the cost of uncertainty-sensitive tasks by up to several orders of magnitude. The speedup scaling is shown in Table 3, which shows saving scaling with the cost of the target-side procedure: up to ∼5000× on MNIST. The speedup for public models is a median ∼2.3× (up to ∼330×). The decision transfer results show that the proxy’s BALD score is thresholded to decide whether the target should abstain. The final conclusion states that σTransfer supports two forms of transfer. The paper concludes that Empirically, σTransfer preserves predictive performance while reducing the cost of uncertainty-sensitive tasks by up to several orders of magnitude. The final conclusion is that σTransfer supports two forms of transfer. The paper concludes that Empirically, σTransfer preserves predictive performance while reducing the cost of uncertainty-sensitive tasks by up to several orders of magnitude. The final conclusion is that σTransfer supports two forms of transfer. The paper concludes that Empirically, σTransfer preserves predictive performance while reducing the cost of uncertainty-sensitive tasks by up to several orders of magnitude. The final conclusion is that σTransfer supports two forms of transfer. The paper concludes that Empirically, σTransfer preserves predictive performance while reducing the cost of uncertainty-sensitive tasks by up to several orders of magnitude <ref:2610.

Improvements for AI systems

  1. Bold Header: Prior-Precision Transfer in Large Models

This capability allows for prior-precision transfer, where the optimal prior precision is stable across width (Figure 1(c)), so a precision chosen on a small (cheap) network is near-optimal for a large (expensive) one. This enables the target model to select its Laplace prior based on decisions made by the smaller proxy network, avoiding a full sweep of candidate precisions.

  1. Bold Header: Decision Transfer for Uncertainty-Sensitive Tasks

The system can perform decision transfer, where decisions made from the small network’s posterior are approximately those the large network would make. Specifically, it enables transferring acquisition, OOD-detection, and abstention decisions without constructing a target posterior. This is demonstrated by achieving high Spearman rank correlation (e.g., 0.75) in sequential active learning experiments.

  1. Bold Header: Massive Computational Speedup

The system provides significant computational savings through the transfer mechanism, as measured precision-sweep speedups reach ∼5000× when transferring from width 128 to 4096 on MNIST. This reduces the search cost for prior precision by performing it on a smaller model and reusing it, leading to a target-side search that is avoided.

  1. Bold Header: Reliable Out-of-Distribution (OOD) Detection

The system can reliably perform OOD detection by transferring the selection criterion, as the proxy-selected precision achieves 0.950 AUROC on held-out OOD data, matching target-side tuning. This ensures robust flagging of unseen inputs without requiring a full posterior construction for the large model.

  1. Bold Header: Scalable Uncertainty Estimation

The system enables uncertainty estimation and decision-making to be cheaper for a much larger (expensive) network. By utilizing the width-normalized prior kernel derived from the Maximal Update Parametrization (µP), it ensures that uncertainty estimates are stable across width, even when scaling from 1B to 7B models.

Sources

Related papers