LoBoost: Model-Native Local Conformal Prediction for Gradient-Boosted Trees
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "LoBoost: Model-Native Local Conformal Prediction for Gradient-Boosted Trees".
Jane: Here's the gist of the paper. Gradient-boosted decision trees are among the strongest off-the-shelf predictors for tabular regression, but point predictions alone do not quantify uncertainty.
Tom: LoBoost proposes a model-native local conformal method that reuses the fitted ensemble’s leaf structure to define a multiscale partition of the feature space.
Jane: This approach allows for locally adaptive prediction intervals by estimating residual quantiles within these model-induced cells, requiring no auxiliary partition or retraining.
Tom: Model-Native Locality and Partitioning LoBoost leverages the sequential nature of gradient boosting to induce a hierarchical organization of the feature space, where early trees capture coarse structure and later trees introduce finer refinements.
Title and authors: Tom: So, what's the actual mechanism behind this model-native local conformal prediction approach? Jane, can you translate that for someone who isn't deep in the math? What is LoBoost doing differently compared to standard split conformal methods we’ve seen before?
Jane: Well, standard split conformal uses one single global residual quantile across the whole feature space. This often leads to intervals that are way too wide in places where things are noisy, or not adaptive enough when the noise changes depending on where you look <ref:2602.22432#pg1>. LoBoost fixes this by creating these nested partitions based on how well the first few trees classify a data point.
Lu: It leverages the sequential nature of gradient boosting; early trees capture the big structure, and later trees add finer details. By matching the leaf identifiers from a specific set of early trees, they define a local region around your input data <ref:2602.22432#pg1>.
Meng: So it’s essentially saying, "Okay, for this specific area in the feature space defined by these tree splits, we're going to estimate a different residual quantile than for some other area." That sounds like it could be very efficient if implemented well.
Lalam: Yes, they are estimating separate residual quantiles within those model-induced cells and then using those local estimates to define the prediction interval conditional on that specific cell <ref:2602.22432#pg1>. It’s a way to get local calibration without needing an auxiliary partition or retraining the base learner.
The paper's summary: Tom: Okay, so we know it uses the internal structure to create local prediction intervals, but what are the actual theoretical guarantees they provide about how good those intervals are? I want to hear about the validity and coverage control.
Jane: The theory connects the stability of that fitted ensemble and how well those induced cells localize residuals to things called local residual score homogeneity <ref:2602.22432#pg1>. This gives them finite-sample control over the error rate, meaning they can bound how often their intervals miss the true value.
Lu: Proposition one shows that when you are inside a region and the tail envelope is small, the nonconformity score—which is just how far your prediction is from the actual observed value—stays roughly constant across those model partitions <ref:2602.22432#pg1>.
Meng: That sounds promising because if that score stays consistent within a region, it means the method isn't wildly guessing its uncertainty based on local noise spikes; it’s using the structure to stay stable. But what happens when we have really complex interactions?
Lalam: Theorem one establishes finite-sample group-conditional validity for every cell in that partition with positive probability <ref:2602.22432#pg1>. And they show that asymptotic conditional coverage is achieved when the cell gets more and more calibration observations and its score distribution becomes homogeneous around the data point, specifically when ncal, npn(x) p goes to infinity and delta n(x) p goes to zero <ref:2602.22432#pg1>.
The paper's improvements: Tom: Alright, so to wrap up the whole thing. What’s the big picture here? LoBoost is bridging that gap between standard global methods and those heavy, auxiliary conformal methods by reusing what the model has already learned about its own structure.
Jane: It moves us toward a method that provides locally adaptive intervals for boosted trees without needing to re-train the core predictor or build a separate partition model <ref:2602.22432#pg1>.
Lu: The main implication is that we can handle heteroscedasticity and discontinuities in the data much better than just using one global residual quantile, as shown by their comparison with standard split conformal in Figure one <ref:2602.22432#pg3>.
Meng: From a deployment view, the computational gains are interesting; some settings show post-fit speedups up to one hundred times faster than heavier rules like MICP and WICP, which makes it feasible for real-time systems.
Lalam: The limitation they point out is their reliance on absolute residual nonconformity scores, which means the prediction intervals they produce are symmetric around the point estimate and might not be ideal if your response data is skewed <ref:2602.22432#pg1>.
Tom: So, to summarize this paper "LoBoost: Model-Native Local Conformal Prediction for Gradient-Boosted Trees," it’s a method that uses the model's internal tree structure to define local regions for estimating uncertainty, leading to better coverage control than global methods, but we have some known limitations with response skew.
Jane: It’s definitely a solid step toward making uncertainty quantification practical and fast for these kinds of powerful boosted models. We'll keep an eye on how they extend this idea into more asymmetric scores next time.
Conclusion: Tom: So we’ve talked about how LoBoost uses the internal tree structure to build local regions for uncertainty estimates, and now we’re wrapping up this paper, "LoBoost: Model-Native Local Conformal Prediction for Gradient-Boosted Trees."
Jane: Basically, it takes a fitted gradient boosted tree ensemble and lets you get a prediction interval that adapts locally based on how the trees are structured, without needing to build any extra partition model or retrain the main tree <ref:2602.22432#pg1>.
Lu: The core idea is leveraging the sequential nature of boosting; early trees define coarse structure, and later ones add finer detail. This creates a nested partition on the feature space where you can estimate different residual quantiles in each cell <ref:2602.22432#pg1>.
Meng: From an engineering standpoint, the efficiency is what stands out; some settings show speedups up to one hundred times faster than heavier methods like MICP or WICP, which means this could actually be deployable in production environments that need fast uncertainty checks.
Lalam: And theoretically, they provide guarantees on coverage control. They show that if you get enough calibration observations and the cell gets refined enough, the interval will hit the true value about one minus alpha percent of the time <ref:2602.22432#pg1>.
Tom: That's a big deal for practical deployment because we’re used to those auxiliary models being computationally expensive, so reusing the fitted structure makes it much more accessible. But Jane mentioned something about limitations with response skew.
Jane: Exactly, LoBoost is built around absolute residual scores, which means the intervals will be symmetric around your point estimate, and that might not be perfect if your actual data is skewed <ref:2602.22432#pg1>.
Lu: That's a known trade-off; they are relying on those specific score types, so you have to be careful with the shape of your output distribution if you want the absolute best local fit.
Meng: If you’re dealing with highly skewed data, LoBoost might not be the first thing you reach for without some adjustment to how those scores are calculated <ref:2602.22432#pg1>.
Lalam: I'm confident that for standard tabular regression where things aren't too weirdly skewed, these guarantees hold up well, and it’s a solid way to get good uncertainty quantification right out of the model <ref:2602.22432#pg1>.
Tom: It really bridges that gap between the heavy machinery of conformal methods and just using a standard global residual score for boosted trees. So, LoBoost is a clean way to get local adaptation without the heavy overhead.
Jane: It’s definitely a method that’s going to be interesting for anyone working on high-dimensional tabular data where those feature interactions create localized noise patterns <ref:2602.22432#pg1>.
Lu: It shows that you don't always need to build a whole new model just to get local calibration; you can use the existing structure for that purpose.
Meng: For me, if we can get those speedups across the board, it’s worth looking at integrating this into our standard prediction pipeline for faster anomaly detection or quality control checks.
Lalam: I think what matters most is that they’ve shown how to reuse the internal leaf structure effectively, and that's a powerful idea for making uncertainty estimation more scalable.
Tom: Alright, so we've covered the mechanics, the theory, and where it falls short with skewness. LoBoost is a strong tool for model-native local prediction on boosted trees. Next time we’ll be looking at how other recent work tackles those challenging multimodal reasoning problems in large language models.
Federal University of São Carlos · University of São Paulo · Université Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
stat.ML, cs.LG
Submitted: 2026-02-25
Updated: 2026-11-10
Comments: Accepted paper version
Journal ref: Transactions on Machine Learning Research (11/2026)
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 93/100
The gist: Gradient-boosted decision trees are among the strongest off-the-shelf predictors for tabular regression, but point predictions alone do not quantify uncertainty.
Key concepts
- Model-Native Locality and Partitioning
- LoBoost uses the sequential nature of gradient boosting to organize the feature space hierarchically. It defines local regions around a data point by checking which leaf nodes it falls into across different trees. This creates nested cells, where coarse partitions are defined by early trees and finer ones by later trees, allowing for localized uncertainty estimation.
- Retraining-free Calibration
- The method applies conformal prediction directly to the fitted ensemble without retraining the original gradient boosting model. It only requires the fitted leaf indices from the model and a standard calibration set. This avoids expensive auxiliary partition models or repeated training, making post-hoc uncertainty quantification much faster.
- Theoretical Guarantees for Coverage
- Mathematical proofs show that LoBoost's prediction intervals are reliable. As the number of calibration observations within a specific local region grows large enough, the actual coverage error converges to the desired level. This ensures that the method provides statistically sound uncertainty estimates under appropriate conditions.
- Resolution Selection Mechanism
- The algorithm intelligently chooses how finely to partition the space based on calibration data. It uses leaf paths from calibration covariates and a threshold to define local sizes. Regions are then refined or aggregated using a weighted Hamming distance, penalizing disagreements in trees that have more predictive power.
Terminology
Summary
Gradient-boosted decision trees are among the strongest off-the-shelf predictors for tabular regression, but point predictions alone do not quantify uncertainty. p. 0 Cabezas
Department of Statistics, Federal University of São Carlos, São Carlos, São Paulo, Brazil
Institute of Mathematics and Computer Science, University of São Paulo, São Carlos, São Paulo, Brazil
Université Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France
Rafael Izbicki
Department of Statistics, Federal University of São Carlos, São Carlos, São Paulo, Brazil
Thiago R. Ramos
Department of Statistics, Federal University of São Carlos, São Carlos, São Paulo, Brazil
Abstract
Gradient-boosted decision trees are among the strongest off-the-shelf predictors for tabular
regression, but point predictions alone do not quantify uncertainty. Conformal prediction
provides distribution-free marginal coverage, yet standard split conformal uses a single global
residual quantile and can adapt poorly to heteroscedasticity. We propose LoBoost, a modelnative local conformal method that reuses the fitted ensemble’s leaf structure to define a
multiscale partition of the feature space. • Model-native local conformal prediction for boosted trees. We use leaf-index prefixes from
the fitted boosted ensemble to define a multiscale partition of X and perform conformal calibration
within these cells, yielding locally adaptive intervals.Where the paper says it
Where the paper says it
Where the paper says it
Model-Native Locality and Partitioning
LoBoost leverages the sequential nature of gradient boosting to induce a hierarchical organization of the feature space, where early trees capture coarse structure and later trees introduce finer refinements. This process associates each input with a sequence of leaf identifiers, denoted as lT (x) = l1(x),. p. 5 Let ℓt(x) denote the leaf identifier reached by a point x ∈ X in the t-th regression tree ht, as defined in (1). The boosted ensemble therefore associates each input with a sequence of leaf identifiers,
ℓT (x) = ℓ1(x), ℓ2(x), . Concretely, we
represent each input x by the sequence of leaves it visits along the ensemble, ℓT (x) = ℓ1(x), . , ℓT (x)
,
and define a local region around x by requiring agreement on the first k trees, i. , ℓT (x)
,
and define a local region around x by requiring agreement on the first k trees, i. , matching the prefix
ℓk(x) = ℓ1(x), . , ℓk(x)
for k ≤ T. This yields a nested, multiscale partition—coarse when k is small,
increasingly refined as k grows—on which we calibrate residual quantiles.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Theoretical Guarantees and Validity
The theory connects the stability of the fitted ensemble and the geometry of its induced cells to local residualscore homogeneity, providing finite-sample coverage-error control and asymptotic pointwise validity as the local calibration size grows. Proposition 1 shows that when z ∈ Rk(x) and the tail envelope bT,k is small, the nonconformity score s(z, y) = y − gT (z) is approximately equal to s(x, y); that is, the absolute-residual score remains approximately constant across the boosting-induced partitions. p. 6 Proposition 1 shows that when z ∈ Rk(x) and the tail envelope bT ,k is small, the nonconformity score s(z, y) =
|y − gT (z)| is approximately equal to s(x, y); that is, the absolute-residual score remains approximately
constant across the boosting-induced partitions. Due to this property, this is the score we consider throughout
this work. The prediction interval is
Cb(x) = [gT (x) − qbA(x)
, gT (x) + qbA(x)
]. Theorem 1 (Finite-sample group-conditional validity). We next show that the cell-level guarantee becomes pointwise
asymptotically when the cell contains increasingly many calibration observations and its score distribution
becomes homogeneous around x. Consider a sequence of fitted predictors gn and corresponding final model-induced partitions Πn, each measurable with respect to the training sigma-field Ftr,n, together with independent calibration samples Dcal,n of
size ncal,n.Where the paper says it
Where the paper says it
Where the paper says it
Resolution Selection Mechanism
The selection of the resolution level k is managed by a data-efficient variant where the calibration sample is used both to select the resolution and to estimate local quantiles. The algorithm extracts leaf paths from calibration covariates, defines an effective local-size threshold Nmin, and refines groups based on these thresholds. p. 8 define the effective local-size threshold
Nmin := maxn
Npart, ⌈pminncal⌉
o
, (2)
which combines an absolute floor Npart with a proportional threshold pmin. Groups with fewer than Nmin
calibration observations become terminal, while the remaining groups proceed to the next tree. For two regions R and R′
, define the weighted Hamming distance
dWH(R, R′
) = X
T
t=1
wt I[ℓt(R) ̸= ℓt(R
′
)] , wt
:= ηBt. (3)
The range Bt measures the maximum prediction variation produced by tree t, so this weighting assigns a
larger penalty to disagreements in trees with greater influence on the fitted predictor.Where the paper says it
Where the paper says it
Performance and Practical Trade-offs
Experiments demonstrate that LoBoost achieves competitive interval quality, low post-hoc calibration costs, and stable behavior across local calibration-size settings. While LoBoost does not uniformly match the best adaptive baseline on each dataset, it improves interval quality (SMIS) relative to standard split conformal (ICP) on most datasets. p. 1 We use leaf-index prefixes from
the fitted boosted ensemble to define a multiscale partition of X and perform conformal calibration
within these cells, yielding locally adaptive intervals. Relative to standard split conformal (ICP),
the natural baseline that shares the same fitted predictor and calibration split, LoBoost improves
interval quality (SMIS) on most datasets, though it does not uniformly match the best adaptive
baseline on each dataset.Where the paper says it
Conclusion and Limitations
LoBoost bridges the gap between computationally expensive auxiliary conformal methods and standard global split conformal for boosted tree ensembles by reusing the internal leaf structure already learned by the model. The method is most effective in high-dimensional, heterogeneous tabular settings where complex feature interactions create localized residual dispersion along dimensions that the ensemble naturally splits on. p. 0 We propose LoBoost, a modelnative local conformal method that reuses the fitted ensemble’s leaf structure to define a
multiscale partition of the feature space. Each input is represented by the sequence of leaves
it visits along the boosting path, and matching leaf prefixes define nested groups in which
residual quantiles are estimated locally. This is a fairly
reasonable assumption, since it simply requires that, on average, enough calibration points fall within each
cell for the local quantile estimates to be reliable. Note that the sequence of cells in Theorem 2 is generated by a sequence of prefix depths An(x) = Rn,kn(x)(x).Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Model-native local conformal prediction for boosted trees: The improved system can perform uncertainty quantification on gradient-boosted trees by reusing their fitted leaf structure to define a multiscale partition of the feature space, leading to locally adaptive intervals that handle heteroscedasticity better than standard split conformal methods.
-
Retraining-free calibration: The system can apply conformal prediction to an already fitted boosted model using only its leaf indices and a standard calibration set, avoiding the need for auxiliary partition models or retraining of the base learner, which significantly lowers post-hoc computational cost.
-
Theoretical guarantees for coverage: The system's performance is backed by theoretical results showing that under certain conditions (vanishing geometric localization and sufficient local calibration size), the realized pointwise coverage converges to the nominal target of 1 minus alpha as the number of calibration points grows.
Abstract
Gradient-boosted decision trees are among the strongest off-the-shelf predictors for tabular regression, but point predictions alone do not quantify uncertainty. Conformal prediction provides distribution-free marginal coverage, yet standard split conformal uses a single global residual quantile and can adapt poorly to heteroscedasticity. We propose LoBoost, a model-native local conformal method that reuses the fitted ensemble's leaf structure to define a multiscale partition of the feature space. Each input is represented by the sequence of leaves it visits along the boosting path, and matching leaf prefixes define nested groups in which residual quantiles are estimated locally. By reusing the predictive structure already learned by the model, LoBoost requires no auxiliary partition or nuisance model, no retraining, and no additional data split beyond standard conformal calibration. Our theory connects the stability of the fitted ensemble and the geometry of its induced cells to local residual-score homogeneity, providing finite-sample coverage-error control and asymptotic pointwise validity as the local calibration size grows and the cells become sufficiently homogeneous. Experiments show a transparent quality-cost trade-off: LoBoost improves over global split conformal on several heterogeneous datasets, but it is not uniformly superior to the adaptive baselines and is not uniformly Pareto-optimal.
Sources
- CP4SBI: Local Conformal Calibration of Credible Sets in Simulation-Based Inference
- Conformal Calibration of Statistical Confidence Sets
- Generalized Boosting Algorithms for Convex Optimization
- CatBoost: unbiased boosting with categorical features
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference