Learning Rate Selection for Generalized Posteriors via Infinitesimal-Jackknife Variance Matching
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Learning Rate Selection for Generalized Posteriors via Infinitesimal-Jackknife Variance Matching".
Jane: Here's the gist of the paper. Generalized posteriors update a prior through an exponential transformation of empirical loss rather than a conventional likelihood, with a learning rate controlling posterior concentration The authors propose infinitesimal-jackknife variance matching (IJVM), a computationally efficient method for selecting the learning rate for a prespecified scalar functional This method matches the functional’s posterior variance to an infinitesimal-jackknife estimate of the sampling variance of its posterior mean, computed from posterior covariances with observation-level losses The resulting one-step update requires only one pilot posterior fit and a second fit for inference, avoiding explicit estimation of loss Hessians or gradient covariance matrices and repeated bootstrap posterior fitting The Core Problem Addressed Generalized Bayesian inference provides a flexible framework for uncertainty quantification when a conventional likelihood is unavailable, deliberately misspecified, or poorly aligned with the inferential objective The resulting generalized posterior, often called a Gibbs posterior, can quantify uncertainty about a population risk minimizer in settings such as regression and classification Credible intervals from a generalized posterior need not attain their nominal frequentist coverage, even when the posterior concentrates around the population risk minimizer The source of this discrepancy is apparent from the asymptotic variances The Infinitesimal-Jackknife Criterion The authors define two quantities, the generalized-posterior variance and an infinitesimal-jackknife estimate, to establish a matching criterion The generalized posterior variance is denoted as U n (w) and the infinitesimal-jackknife approximation to the sampling variance of the posterior mean is denoted as W n (w) The criterion for matching is defined as U n (w) = W n (w) This leads naturally to the multiplicative update wU n (w)/W n (w) Theoretical Guarantees and Consistency Under uniform posterior moment and local linearization conditions, the authors show that nU n (w) and nW n (w) converge uniformly in probability to Ag and Bg, respectively This convergence implies that the exact-posterior update converges uniformly in probability to the constant w◦g Furthermore, when a sampling central limit theorem and a suitable posterior-quantile approximation also hold, credible intervals at a consistent selected rate attain asymptotically nominal frequentist coverage Empirical Performance and Efficiency Simulations involving homoscedastic and heteroscedastic regression and hinge-loss classification show that onestep IJVM achieves coverage near the nominal 95% level, with interval lengths comparable to bootstrap-based generalized posterior calibration but substantially fewer posterior fits The principal computational advantage of IJVM is its substantially smaller requirement for posterior simulation, as one-step IJVM requires only two posterior fits, whereas GPC requires repeated posterior fitting across bootstrap samples Comparison to Existing Methods Existing approaches to learning-rate selection differ primarily in their calibration targets Fisher-information matching (FIM) relates generalized-posterior information to the loss-likelihood bootstrap SafeBayes selects the rate by minimizing cumulative prequential posterior-expected loss, prioritizing predictive performance rather than credible region coverage Generalized posterior calibration (GPC) instead tunes the rate to match the nominal frequentist coverage of a specified credible region IJVM targets U n (w) = W n (w) for a prespecified scalar functional g, whereas GPC directly calibrates coverage Simulation Results and Conclusion In simulations involving homoscedastic and heteroscedastic regression and hinge-loss classification, one-step IJVM achieved empirical coverage generally close to the nominal 95% level, with interval lengths comparable to those of GPC The number of posterior fits required by GPC is substantially higher than that required by one-step IJVM in these experiments Additional updates offered little improvement, while IJVM required substantially fewer posterior fits than GPC in the settings studied The paper concludes that one-step IJVM is sufficient for first-order calibration even when the initial rate is data-dependent The theoretical results establish uniform convergence of the exact-posterior update map to the functional-specific first-order calibration target, even with a data-dependent pilot rate The paper notes that finite-sample variance matching alone does not guarantee nominal frequentist coverage Future work could extend IJVM to dependent data and complex sampling.
Tom: The work was supported by JSPS KAKENHI Grant Number 25K21168.
Jane: The paper is available as arXiv:2610.39684v1
stat.ME: 29 Oct 2026 Pruning redundant reasoning tokens The paper proposes infinitesimal-jackknife variance matching (IJVM) to calibrate the learning rate of a generalized posterior for a prespecified scalar functional IJVM matches the posterior variance of the functional to an infinitesimal-jackknife estimate of the sampling variance of its posterior mean, using posterior covariances with observation-level losses The basic procedure uses one pilot posterior fit for calibration and a second fit for inference, thereby avoiding the repeated bootstrap posterior fits required by generalized posterior calibration The theoretical results establish uniform convergence of the exact-posterior update map to the functional-specific first-order calibration target, even with a data-dependent pilot rate The paper notes that finite-sample variance matching alone does not guarantee nominal frequentist coverage.
Tom: Future work could extend IJVM to dependent data and complex sampling.
Paper summary: Tom: So we're looking at this paper called "Learning Rate Selection for Generalized Posteriors via Infinitesimal-Jackknife Variance Matching." Basically, they're dealing with generalized posteriors, which are these things that update a prior using an exponential transformation of loss instead of a standard likelihood.
Jane: That sounds complicated because it’s not like the usual Bayesian setup where you have a nice likelihood function to start with. So what is their main thesis here?
Tom: Their main idea is proposing this method, infinitesimal-jackknife variance matching, or IJVM, as a way to select the learning rate for that posterior update. They claim it matches the functional's posterior variance to an infinitesimal-jackknife estimate of the sampling variance of its posterior mean.
Lu: So they’re trying to fix this problem where credible intervals might not have correct frequentist coverage even when the posterior is focused around what we think is the right answer, and this method aims to solve that by matching two variances.
Meng: Matching variances sounds like it should be computationally heavy, which is a huge plus or minus for me when I'm thinking about production systems. What are they saying about how much work this actually requires?
Tom: They say the resulting one-step update only requires one pilot posterior fit and a second fit for inference, which avoids needing to explicitly estimate loss Hessians or gradient covariance matrices, plus it skips those repeated bootstrap posterior fittings that methods like generalized posterior calibration do.
Jane: That sounds like a big win for efficiency. So how does this matching criterion actually work in practice? What is the core mathematical idea they are using?
Lu: They define two quantities: the generalized posterior variance, denoted as U sub n w, and an infinitesimal-jackknife estimate of the sampling variance of the posterior mean, called W sub n w. The criterion for matching is simply when U sub n w equals W sub n w.
Tom: And from that equality, it leads naturally to a multiplicative update rule: one times U sub n w divided by W sub n w. That's the core mechanism they are using to pick the learning rate.
Jane: It sounds like they are tying the learning rate directly to how much uncertainty there is versus how much variation there is in the data itself. What kind of conditions do they need for this matching to be reliable?
Paper summary: Lu: They show that under uniform posterior moment and local linearization conditions, n times U sub n w and n times W sub n w converge uniformly in probability to A g and B g, respectively.
Meng: So it's about showing that these two quantities settle down to predictable limits regardless of the specific learning rate we pick, as long as we stay in a certain range. That’s a strong claim, but I need to know what happens when things get messy in the real world.
Tom: They also establish existence and consistency results for solutions of that matching equation and specify conditions under which Monte Carlo error does not compromise consistency.
Jane: So they are not just showing it works on paper, they are giving us theoretical backing on why this one-step procedure should be reliable in the long run. What happens when you look at the final performance claims?
Lu: When a sampling central limit theorem and a suitable posterior-quantile approximation also hold, they prove that credible intervals at a consistent selected rate attain asymptotically nominal frequentist coverage.
Meng: That's the payoff for us if this holds true on real traffic data, meaning we can trust our confidence regions when we deploy the model. But what about the practical performance compared to other methods?
Tom: In simulations involving homoscedastic and heteroscedastic regression and hinge-loss classification, one-step IJVM achieved coverage near the nominal ninety five percent level.
Jane: And their interval lengths were comparable to those from generalized posterior calibration, but they required substantially fewer posterior fits. That’s a really interesting comparison point for deployment.
Lu: The principal computational advantage they highlight is that one-step IJVM only needs two posterior fits, whereas methods like generalized posterior calibration require repeated fitting across bootstrap samples.
Meng: So if I'm running on a resource-constrained system, cutting down the number of required posterior simulations by this much is a massive operational win for me. How does this compare to other existing rate selection methods?
Tom: Existing approaches differ mainly in their calibration targets: Fisher-information matching relates generalized posterior information to the loss likelihood bootstrap, while SafeBayes selects the rate by minimizing cumulative prequential posterior expected loss.
Jane: And then there’s generalized posterior calibration, which tunes the rate to match that nominal frequentist coverage of a specified credible region. So IJVM is targeting something different entirely here.
Paper summary: Lu: IJVM targets that U sub n w equals W sub n w for a prespecified scalar functional g, whereas generalized posterior calibration directly calibrates the coverage itself. That’s a key difference in what they are optimizing for.
Meng: I’m wondering about the setup, though; they mention that the exact-posterior update converges to the constant one times g when we use this pilot rate approach, even if that initial pilot rate is data-dependent.
Tom: That means you don't need a perfect starting point for your learning rate selection; it’s robust enough to settle down to the right target function g on its own.
Jane: So, for someone who just listens to the show, what does this mean for their day-to-day work with these complex models? Is this something they can actually use right now?
Lu: It suggests that first-order calibration is sufficient even when you start with a pilot rate that depends on the data itself. The theoretical results establish uniform convergence of the exact-posterior update map to the functional specific first order calibration target under those conditions.
Meng: So, in terms of what this changes for production, it means less time spent tuning hyperparameters and more time focusing on deploying whatever model you have.
Tom: It does reduce the number of required posterior fits substantially compared to methods like GPC in the experiments they ran. The size of that wall-clock advantage depends on the sampler and how you actually implement it computationally, but it’s there.
Jane: So, to sum up, this paper presents IJVM as a way to select a learning rate for generalized posteriors by matching two specific variance estimates, which they prove converges reliably to the right calibration target even with a data-dependent pilot rate.
Lu: That’s the core idea of "Learning Rate Selection for Generalized Posteriors via Infinitesimal-Jackknife Variance Matching" in one sentence.
Meng: It sounds like a solid piece of work because they connect the theory to practical simulation results, showing that it performs well against other methods in terms of fit count and coverage.
Tom: Exactly. It’s not just an abstract mathematical exercise; they show that this method achieves performance comparable to generalized posterior calibration but with way less computational overhead on the posterior fitting side.
Jane: So, the implication is that if you’re working with a generalized posterior and you need to pick a learning rate, IJVM offers a path where you get good coverage without needing those expensive Hessian calculations or repeated bootstrap fits.
Conclusion: Tom: So we've been looking at this paper, "Learning Rate Selection for Generalized Posteriors via Infinitesimal-Jackknife Variance Matching," and the authors are trying to figure out how to pick the right learning rate for these generalized posteriors, which are used when you don't have a standard likelihood function.
Jane: Right, so basically they propose this method called IJVM, which matches the posterior variance of that functional against an infinitesimal-jackknife estimate of the sampling variance of its mean.
Lu: It’s a way to make sure your credible intervals actually have the right frequentist coverage, even if you start with a data-dependent learning rate.
Meng: And what they show is that this one step update only needs one pilot posterior fit and a second fit for inference, which saves so much time compared to methods that require repeated bootstrap fitting.
Tom: That’s the big operational win there, cutting down on the simulation workload significantly. The theoretical part confirms that this converges to the correct first order calibration target regardless of what you pick initially.
Jane: It means you don't need a perfect starting point for your learning rate selection; it settles down to what it should be all by itself under certain conditions.
Lu: And they show that in real simulations, this one step update achieves coverage and interval lengths comparable to methods like generalized posterior calibration, but with much less computational cost.
Meng: So if I’m shipping models right now, this means I can use a more efficient method to tune the learning rate without having to run dozens of expensive bootstrap samples every time.
Tom: Exactly. It’s about getting reliable coverage while keeping the computational load manageable on the backend. The authors also flag that just matching variances by themselves doesn't guarantee nominal frequentist coverage in every single case, though they show it works well in practice for regression and classification tasks.
Jane: So, it’s a solid piece of work connecting that theory to practical simulations, showing a way to get better calibration without the heavy computational burden.
Lu: The paper sets up a clear path forward for selecting learning rates in these complex Bayesian settings.
Faculty of Economics, Fukuoka University
stat.ME
Submitted: 2026-10-29
Updated: 2026-11-10
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 80/100
The gist: Generalized posteriors update a prior through an exponential transformation of empirical loss rather than a conventional likelihood…
Key concepts
- Generalized Posteriors
- These are Bayesian inferences used when a standard likelihood function is missing or poorly defined. They update a prior distribution by transforming empirical loss values using an exponential function, providing uncertainty quantification in complex settings like regression.
- Infinitesimal-Jackknife Variance Matching (IJVM)
- This is the proposed method for selecting the learning rate. It matches the generalized posterior variance ($U_n(w)$) to an infinitesimal-jackknife estimate of the sampling variance of the posterior mean ($W_n(w)$). This criterion guides a multiplicative update rule, $wU_n(w)/W_n(w)$, for efficient calibration.
- Generalized Posterior Variance ($U_n$) and Infinitesimal-Jackknife Estimate ($W_n$)
- The core matching criterion is $U_n(w) = W_n(w)$. $U_n(w)$ measures the variance of the generalized posterior, while $W_n(w)$ approximates the sampling variance of the posterior mean using an infinitesimal-jackknife approach derived from observation-level losses.
- One-Step Update vs. Repeated Fitting
- IJVM enables a one-step update requiring only two posterior fits: one pilot fit for calibration and a second fit for inference. This contrasts with Generalized Posterior Calibration (GPC), which requires repeated fitting across bootstrap samples, making IJVM computationally much more efficient.
Terminology
Summary
Generalized posteriors update a prior through an exponential transformation of empirical loss rather than a conventional likelihood, with a learning rate controlling posterior concentration The authors propose infinitesimal-jackknife variance matching (IJVM), a computationally efficient method for selecting the learning rate for a prespecified scalar functional This method matches the functional’s posterior variance to an infinitesimal-jackknife estimate of the sampling variance of its posterior mean, computed from posterior covariances with observation-level losses The resulting one-step update requires only one pilot posterior fit and a second fit for inference, avoiding explicit estimation of loss Hessians or gradient covariance matrices and repeated bootstrap posterior fitting
The Core Problem Addressed
Generalized Bayesian inference provides a flexible framework for uncertainty quantification when a conventional likelihood is unavailable, deliberately misspecified, or poorly aligned with the inferential objective The resulting generalized posterior, often called a Gibbs posterior, can quantify uncertainty about a population risk minimizer in settings such as regression and classification Credible intervals from a generalized posterior need not attain their nominal frequentist coverage, even when the posterior concentrates around the population risk minimizer The source of this discrepancy is apparent from the asymptotic variances
The Infinitesimal-Jackknife Criterion
The authors define two quantities, the generalized-posterior variance and an infinitesimal-jackknife estimate, to establish a matching criterion The generalized posterior variance is denoted as U n (w) and the infinitesimal-jackknife approximation to the sampling variance of the posterior mean is denoted as W n (w) The criterion for matching is defined as U n (w) = W n (w) This leads naturally to the multiplicative update wU n (w)/W n (w)
Theoretical Guarantees and Consistency
Under uniform posterior moment and local linearization conditions, the authors show that nU n (w) and nW n (w) converge uniformly in probability to Ag and Bg, respectively This convergence implies that the exact-posterior update converges uniformly in probability to the constant w◦g Furthermore, when a sampling central limit theorem and a suitable posterior-quantile approximation also hold, credible intervals at a consistent selected rate attain asymptotically nominal frequentist coverage
Empirical Performance and Efficiency
Simulations involving homoscedastic and heteroscedastic regression and hinge-loss classification show that onestep IJVM achieves coverage near the nominal 95% level, with interval lengths comparable to bootstrap-based generalized posterior calibration but substantially fewer posterior fits The principal computational advantage of IJVM is its substantially smaller requirement for posterior simulation, as one-step IJVM requires only two posterior fits, whereas GPC requires repeated posterior fitting across bootstrap samples
Comparison to Existing Methods
Existing approaches to learning-rate selection differ primarily in their calibration targets Fisher-information matching (FIM) relates generalized-posterior information to the loss-likelihood bootstrap SafeBayes selects the rate by minimizing cumulative prequential posterior-expected loss, prioritizing predictive performance rather than credible region coverage Generalized posterior calibration (GPC) instead tunes the rate to match the nominal frequentist coverage of a specified credible region IJVM targets U n (w) = W n (w) for a prespecified scalar functional g, whereas GPC directly calibrates coverage
Simulation Results and Conclusion
In simulations involving homoscedastic and heteroscedastic regression and hinge-loss classification, one-step IJVM achieved empirical coverage generally close to the nominal 95% level, with interval lengths comparable to those of GPC The number of posterior fits required by GPC is substantially higher than that required by one-step IJVM in these experiments Additional updates offered little improvement, while IJVM required substantially fewer posterior fits than GPC in the settings studied The paper concludes that one-step IJVM is sufficient for first-order calibration even when the initial rate is data-dependent The theoretical results establish uniform convergence of the exact-posterior update map to the functional-specific first-order calibration target, even with a data-dependent pilot rate The paper notes that finite-sample variance matching alone does not guarantee nominal frequentist coverage Future work could extend IJVM to dependent data and complex sampling. p. 1 The work was supported by JSPS KAKENHI Grant Number 25K21168. p. 1 The paper is available as arXiv:2610. p. 039684v1 [stat. p. 0ME] 29 Oct 2026
Pruning redundant reasoning tokens
The paper proposes infinitesimal-jackknife variance matching (IJVM) to calibrate the learning rate of a generalized posterior for a prespecified scalar functional IJVM matches the posterior variance of the functional to an infinitesimal-jackknife estimate of the sampling variance of its posterior mean, using posterior covariances with observation-level losses The basic procedure uses one pilot posterior fit for calibration and a second fit for inference, thereby avoiding the repeated bootstrap posterior fits required by generalized posterior calibration The theoretical results establish uniform convergence of the exact-posterior update map to the functional-specific first-order calibration target, even with a data-dependent pilot rate The paper notes that finite-sample variance matching alone does not guarantee nominal frequentist coverage. Future work could extend IJVM to dependent data and complex sampling.
How the training signal is built
The infinitesimal-jackknife criterion defines W n (w) as an infinitesimal-jackknife (IJ) approximation to the repeated-sampling variance of the generalized posterior mean, obtained by linearizing bootstrap reweighting The quantity W n is defined as W n (w) = w squared sum over i=1 to n of (xi,w - xi bar) squared The exact-posterior update map T n (w) is defined as U n (w)/W n (w)
Where the gains come from
The theoretical results establish first-order sufficiency of one exact-posterior update, irrespective of the pilot rate The consistency statement for a verified candidate shows that any possibly random w en in Ω satisfying nU n (w en) − W n (w en) = oPjoint(1) satisfies w en P -> w◦g The paper demonstrates that one-step IJVM substantially reduces the number of posterior fits while achieving coverage and interval lengths comparable to GPC in these experiments The size of the wall-clock advantage remains dependent on the sampler and computational implementation.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
More efficient posterior calibration for generalized Bayesian inference is achieved through infinitesimal-jackknife variance matching (IJVM). This method selects a learning rate by equating
the generalized-posterior variance Un(w) of g(theta) with an IJ estimate Wn(w) of the sampling variance of its posterior mean.
-
One-step update procedures for learning rates are significantly faster than Generalized Posterior Calibration (GPC). The paper notes that
A multiplicative one-step update requires one pilot posterior fit and a second fit for inference, avoiding explicit estimation of loss Hessians or gradient covariance matrices and repeated bootstrap posterior fitting.
-
The system can achieve empirical frequentist coverage for credible intervals on generalized posteriors. Under suitable conditions, the resulting credible intervals
at asymptotically nominal frequentist coverage.
-
Calibration is robust to initial rate choices in a pilot study. Simulations show that
After a single update, however, coverage became nearly identical across initial rates and approached the nominal 95% level in all three designs.
-
The system can be used for uncertainty quantification when conventional likelihoods are unavailable or misspecified. It updates a prior
through an exponential transformation of an empirical loss function rather than a conventional likelihood.
Sources
- The Bayesian Infinitesimal Jackknife for Variance
- Robust Standard Errors for Bayesian Posterior Functionals via the Infinitesimal Jackknife
- Learning rate selection via weighted Fisher divergence
- Efficient Bootstrap Calibration of Generalized Posteriors with Importance Reweighting
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes