Conditional Counterfactual Mean Embeddings: Doubly Robust Estimation and Learning Rates
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Conditional Counterfactual Mean Embeddings".
Jane: Here's the gist of the paper. A complete understanding of heterogeneous treatment effects involves characterizing the full conditional distribution of potential outcomes Under this framework, a two-stage meta-estimator for Conditional Counterfactual Mean Embeddings (CCME) is developed to accommodate any RKHS-valued regression in each stage This framework yields three practical CCME estimators: Ridge Regression estimator, Deep Feature estimator that parameterizes the feature map by a neural network, and Neural-Kernel estimator that performs RKHS-valued regression with coefficients parameterized by a neural network Framework for Distribution Representation The Conditional Counterfactual Mean Embeddings (CCME) framework embeds conditional distributions of counterfactual outcomes into a reproducing kernel Hilbert space (RKHS) This allows for the analysis of conditional counterfactual distributions when the distributions vary across subpopulations The target estimand is represented by the Conditional Counterfactual Mean Embeddings (CCME): µY 1V (v) = E
ϕ(Y 1) V = v: ∈ HY This embedding fully characterizes the conditional distribution of Y 1 given V = v when the kernel kY is characteristic Identification via Doubly Robust Methods To identify the CCME from observed data (X, A, Y), two nuisance functions are introduced: the propensity score π(x) and the CME of the treated outcome conditional on X, µ0(x) The identification result is established via a pseudo-outcome ξ(Z) = A / π(X) (ϕ(Y) − µ0(X)) + µ0(X) Under the assumptions of conditional ignorability and positivity, the CCME is identified as µY 1V (v) = E
ξ(Z) V = v: Estimator Development The estimation strategy follows a two-stage meta-estimator based on Algorithm 1 Stage 1 involves estimating the nuisance functions π and µ0 using data D0, while Stage 2 constructs pseudo-outcomes ξb(Z) from the estimated nuisance functions and regresses them on V to obtain the CCME estimator Three specific estimators are developed: 1.
Tom: Ridge Regression estimator, which uses kernel ridge regression (KRR) 2.
Jane: Deep Feature estimator, which parameterizes the feature map by a neural network 3.
Tom: Neural-Kernel estimator, which performs RKHS-valued regression with coefficients parameterized by a neural network Convergence and Robustness Properties Finite-sample convergence rates are established for all three estimators, explicitly tracking the dependence on the dimensions dx, dv, dy and the smoothness of the conditional density Crucially, it is proven that these estimators achieve double robustness in rates; specifically, the first-stage error is controlled by the minimum of the rates achieved by the propensity score estimator and the CME estimator The convergence rate for Ridge Regression is bounded by E
∥µbRR(V) − µY 1V (V)∥ 2 HY: ≲ n − 2r 2r+dv + min R squared π(πb), R squared µ0(µb0) Application to Density Estimation The CCME can be used to recover the full conditional counterfactual density (CCD) p 1(yv) The estimated density at any query point y ∈ Y is computed by the inner product: bp 1(yv) = ⟨µbY 1V (v), ϕ(y)⟩HY Explicit formulae for the density estimators based on all three proposed methods are derived, such as bp 1 RR(yv) = D Ξb(KV + nλ1In) −1kV (v), ϕ(y) Experimental Validation Experiments on synthetic and semi-synthetic data confirm that the DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR, IPW, and PI variants of the estimators achieve comparable MSE in this setting, while for the Ridge Regression estimator, PI exhibits slightly higher MSE than DR and IPW due to selection bias The DR estimator addresses this through the propensity score weighting Theoretical Analysis of Rates The convergence rates depend on the smoothness r of the function v 7→ p 1(yv) and worsen with dv For Deep Feature and Neural-Kernel estimators, the rates improve with the smoothness s of the function y 7→ p 1(yv) and the smoothness of the kernel τ, worsening with dy The rate of Ridge Regression does not depend on dy at all under Assumption 4' Inference with Neural-Kernel Estimator The Neural-Kernel estimator computes the CCD estimator as bp 1 NK(yv) = X M j=1 fθb(v)jϕ(yej), ϕ(y) This makes inference with the Neural-Kernel estimator significantly faster compared to the previous two estimators, since there is no need for calculation nor inversion of large matrices Density Estimation Formulae The density estimate for the Neural-Kernel estimator is given by bp 1 NK(yv) = X M j=1 fθb(v)jkY (yej, y) This makes inference with the Neural-Kernel estimator significantly faster compared to the previous two estimators, since there is no need for calculation nor inversion of large matrices Final Bound Derivation The final bound for the second-stage estimator is E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L ≍ n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound Inference with Ridge Regression Estimator The CCD estimator for Ridge Regression is bp 1 RR(yv) = kY0(y)⊤(KX0 + mλ0Im −1kX0X1 (KV + nλ1In −1kV (v), Inference with Deep Feature Estimator The CCD estimator for Deep Feature is bp 1 DF(yv) = kY0(y)⊤Ψb 0(Ψb ⊤ 0 Ψb 0 + mλ0Im −1ψθb0(x) (KV + nλ1In −1ψθb(v), Inference with Neural-Kernel Estimator The CCD estimator for Neural-Kernel is bp 1 NK(yv) = X M j=1 fθb(v)jkY (yej, y) This makes inference with the Neural-Kernel estimator significantly faster compared to the previous two estimators, since there is no need for calculation nor inversion of large matrices We proposed the Conditional Counterfactual Mean Embeddings (CCME) as a framework for representing conditional counterfactual distributions in RKHS We developed a two-stage meta-estimator based on doubly robust pseudo-outcomes and instantiated it with three estimators—Ridge Regression, Deep Feature, and Neural-Kernel—each offering different computational-statistical trade-offs Our finite-sample analysis establishes convergence rates that explicitly track the dependence on the dimensions dx, dv, dy and the smoothness of the conditional density, and shows that all three estimators achieve double robustness in rates (Theorem 1) Experiments on synthetic and semi-synthetic data confirm that the DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR estimator addresses this through the propensity score weighting Pruning redundant reasoning tokens The framework for studying conditional counterfactual distributions leverages Kernel Mean Embeddings (KME) to manipulate probability distributions as vectors in an RKHS The CCME enables a wide range of statistical tasks on conditional counterfactual distributions, such as density estimation, conditional independence testing, and sampling The identification relies on two nuisance functions: the propensity score π(x) and the CME of the treated outcome µ0(x) How the training signal is built The pseudo-outcome ξ(Z) is constructed as ξ(Z) = A / π(X) (ϕ(Y) − µ0(X)) + µ0(X) The meta-estimator proceeds in two stages: Stage 1 estimates the nuisance functions πb and µb0 using data D0, and Stage 2 constructs pseudo-outcomes ξb(Z1i) and regresses them on V to obtain the CCME estimator Where the gains come from The convergence rate for the meta-estimator is bounded by E
∥µbY 1V (V) − µY 1V (V)∥ 2 HY: ≲ R squared ξb(µbY 1V) + min R squared π(πb), R squared µ0(µb0) The gains come from the double robustness property, where the first-stage error is governed by the nuisance with the smaller error The experiments confirm that DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR estimator addresses this through the propensity score weighting Pruning redundant reasoning tokens The final bound for the second-stage estimator is E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound Pruning redundant reasoning tokens The convergence rate for Ridge Regression is bounded by E
∥µbRR(V) − µY 1V (V)∥ 2 HY: ≲ n − 2r 2r+dv + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing λn ≍ n − τ r+dv/2, for sufficiently large n ≥ 1 Pruning redundant reasoning tokens The final bound for the Neural-Kernel estimator is E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound Pruning redundant reasoning tokens The convergence rate for Deep Feature estimator is bounded by E
∥µbDF(V) − µY 1V (V)∥ 2 HY: ≲ n − ab(a+2)(b+1)-1 (log n) squared + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing M n b (a+2)(b+1)-1 and W L n a+1 (a+2)(b+1)-1 Pruning redundant reasoning tokens The convergence rate for Neural-Kernel estimator is bounded by E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E
∥µbNK(V) − µY 1V (V)∥ 2 HY: ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound Pruning redundant reasoning tokens The convergence rate for all three estimators, ignoring the first-stage nuisance term, achieves the minimax non-parametric regression rate n − 2r 2r+dv over V The Ridge Regression estimator attains this rate directly, as it performs regression in the full RKHS without finite-dimensional projection Pruning redundant reasoning tokens The DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR estimator addresses this through the propensity score weighting Pruning redundant reasoning tokens
Title and authors: Tom: So we're looking at this paper today called "Conditional Counterfactual Mean Embeddings: Doubly Robust Estimation and Learning Rates." It sounds like they're trying to get a better handle on how treatment effects vary across different groups.
Jane: Exactly. The main idea is using something called Conditional Counterfactual Mean Embeddings, or CCME, which embeds conditional distributions into this reproducing kernel Hilbert space, or RKHS.
Lu: What that means in simple terms is they are moving beyond just looking at the average treatment effect and trying to capture the whole shape of what could happen under different conditions.
Meng: That sounds complex. So how do they actually use this CCME framework in practice instead of just talking about distributions theoretically?
Tom: Well, they developed a two-stage meta-estimator that can handle any RKHS-valued regression at each stage. They then built three specific estimators based on this framework: a Ridge Regression estimator, a Deep Feature estimator that uses a neural network to parameterize the feature map, and another one called the Neural-Kernel estimator.
Jane: That's interesting because it gives people options depending on what kind of data they have and how much computational power they want to use. It’s like giving you different tools for measuring the same thing.
Lalam: I see how that works with the identification part, where they use a pseudo-outcome constructed from the propensity score and some conditional mean embedding to identify the CCME.
Tom: Right, so it's about using those two nuisance functions—the propensity score and the conditional mean embedding—to build a better estimate of what we want to learn directly.
Jane: The paper establishes that this approach works, and they give us finite-sample convergence rates for all three estimators, which is important for knowing how accurate they are in the real world.
Lu: The key part for me is that they prove double robustness in terms of these convergence rates; it means the first stage error is controlled by whichever nuisance model—the propensity score or the conditional mean embedding—is doing better.
Meng: Double robustness sounds good, but what does that actually translate to when I'm trying to deploy this on production data? Does it mean I can just pick the simplest model and be fine?
Tom: Not exactly. It means that as long as one of those two initial models is reasonably accurate, the final estimate won't be much worse than if you had used the other one perfectly. The convergence rate for Ridge Regression, for example, is bounded by E norm of the difference between its estimate and the true conditional mean squared in terms of n minus two r squared r plus dv plus a term involving lambda pi and mu zero
Title and authors: Jane: That error bound shows how quickly things get more accurate as you collect more data. It explicitly tracks the dimensions of the outcome, like how many variables define what we are measuring, and the smoothness of the conditional density.
Lu: And they show that for Deep Feature and Neural-Kernel estimators, the convergence rate improves when you have more smoothness in your function relating those variables. For Ridge Regression, it doesn't depend on one specific dimension called dy at all under assumption four.
Meng: Okay, so if I'm using a deep feature model to parameterize the map, is there a practical downside to that complexity? Does it just make the training harder or slower than a simpler linear model?
Tom: It makes it more complex, but the Deep Feature estimator lets you learn the feature map directly from a neural network. The Neural-Kernel estimator tries to solve that complexity by using neural networks to parameterize those coefficients instead of relying on large matrix inversions.
Jane: That's a nice trade-off. For instance, the Neural-Kernel estimator computes the conditional counterfactual density at any point y by taking the inner product of its embedding with a kernel function phi of y.
Lu: That explicit formula for density estimation is actually quite powerful because it shows how you can calculate the density estimate using that learned embedding and a kernel function.
Meng: What about running this on massive datasets? If we're talking about scaling up, what's the operational cost of using these more complex neural network estimators compared to just running Ridge Regression?
Tom: For Ridge Regression, the convergence rate is bounded by E norm of the difference between its estimate and the true conditional mean squared in terms of n minus two r squared r plus dv plus a term involving lambda pi and mu zero. That tells you how fast it converges for that specific method.
Jane: The experiments on synthetic and semi-synthetic data have been pretty encouraging, showing these estimators can recover multimodal distributions and correct for selection bias where simpler plug-in methods fail.
Lu: And they found that the double robust estimators, like the DR estimator, achieve comparable mean squared error in this setting compared to their other variants—the IPW and PI versions.
Meng: That's good news regarding robustness, but what about the Neural-Kernel estimator's inference speed? I need something that runs fast enough for real-time feedback loops.
Tom: The Neural-Kernel estimator is significantly faster because it doesn't require calculating or inverting large matrices, unlike the previous two estimators. It computes the conditional counterfactual density estimate at a point y by taking X multiplied by a sum over j of f theta b of v j phi of y e j phi of y.
Title and authors: Jane: So for someone just listening to the show, what does this actually mean for decision-making in healthcare or finance? How does this framework change the way we think about treatment effects?
Lu: It shifts the focus from a single number to characterizing the full spectrum of potential outcomes specific to an individual's context. We can now analyze how a drug might behave differently for different genetic markers, which is crucial for safe, targeted interventions.
Meng: If you are trying to deploy this, what's the main limitation they mention? What stops you from using this framework for every single problem?
Tom: They do flag that the convergence rates depend on the smoothness of certain functions and can worsen with dimensions like dy. Plus, for Deep Feature and Neural-Kernel estimators, they note that rates improve with more smoothness in another function related to y and the kernel parameter tau.
Jane: So it's not a magic fix; you still need to consider the structure of your data and how smooth those underlying distributions are when choosing which estimator to use.
Lu: And for inference with the Neural-Kernel estimator, they give you a final bound on its error that depends on n minus b plus one log n squared dy plus dy plus four b squared b plus one plus a term involving lambda pi and mu zero.
Meng: That bound looks like it scales somewhat favorably with the number of data points n, but that term with dy and the log factors is still something to watch at scale.
Tom: The Ridge Regression convergence rate is bounded by E norm of the difference between its estimate and the true conditional mean squared in terms of n minus two r squared r plus dv plus a term involving lambda pi and mu zero.
Jane: To wrap things up on this paper, they propose Conditional Counterfactual Mean Embeddings to represent conditional distributions in an RKHS, using a double robust meta-estimator with three concrete implementations: Ridge Regression, Deep Feature, and Neural-Kernel.
Lu: The main contribution is establishing those finite sample convergence rates that track the dimensions of the outcome and covariates and proving that all three estimators achieve double robustness in rates.
Meng: For deployment, it means you have flexibility in choosing between a simple regression like Ridge or a more complex neural network approach based on what your data demands.
Tom: And for density estimation, they show how to recover the full conditional counterfactual density by taking the inner product of the estimated embedding with a kernel function phi of y.
Jane: This paper is important because it gives us a solid framework for moving toward understanding heterogeneous treatment effects by looking at the entire distribution, not just a single point estimate.
The paper's summary: Tom: So, we've seen that this paper is building Conditional Counterfactual Mean Embeddings, or CCME, which is basically a way to represent conditional distributions using this reproducing kernel Hilbert space structure.
Jane: That means they aren't just looking at one single average treatment effect anymore; they’re trying to capture the whole shape of what the outcome could be under different conditions.
Tom: Right. And they developed this two-stage meta-estimator based on doubly robust pseudo-outcomes to identify that CCME, and then they build three distinct estimators: Ridge Regression, Deep Feature, and Neural-Kernel.
Jane: What’s really interesting is how they've established the convergence rates for all of them. They explicitly track things like the dimension of your outcome variables, the dimension of your covariates, and even how smooth the conditional density function is.
Tom: The big thing they prove is that all three estimators achieve double robustness in terms of those convergence rates. That means if you get one part right—the propensity score or the conditional mean embedding—your final estimate won't be much worse than if you had used the other one perfectly.
Jane: That’s a lot of rigor for estimating something so complex as a counterfactual distribution. It gives us a solid way to know how much data you need to get an accurate result and how sensitive that accuracy is to the underlying data structure.
Tom: And they show that for Ridge Regression, the convergence rate is bounded by something involving n minus two r squared r plus dv plus terms related to your propensity score and conditional mean embedding.
Jane: It really shows how the error falls with the square of your sampling interval, which is just a way of saying more data gets you better estimates.
Tom: For Deep Feature and Neural-Kernel estimators, they find that the convergence rate improves when you have more smoothness in other functions related to your outcome and your kernel parameter tau.
Jane: So what this changes for people using this framework is that you don't just get one number; you can actually recover the full conditional counterfactual density, which is super useful for tasks like density estimation at a specific point.
Tom: Exactly. And they give us explicit formulas for estimating that density using the embedding and a kernel function.
Jane: That’s really promising because it moves us from just getting an estimate to actually understanding the probability distribution itself under different conditions, which is huge in fields like personalized medicine or finance.
Tom: It’s a lot of heavy math, but when you see how these estimators handle selection bias and recover multimodal densities in experiments, it shows that this framework has real practical power for studying heterogeneous treatment effects.
Jane: We've got the mechanics down now—the identification, the three estimators, and the convergence proofs. Next up we look at how fast these things actually run when you're trying to deploy them on massive datasets.
The paper's improvements: Tom: So, this paper doesn't just stop at describing three estimators; they suggest ways to make them even better for real use cases.
Jane: That makes sense because we're always looking for ways to get a more stable result, especially when dealing with complex treatment effects in areas like healthcare or finance.
Tom: They touch on improving the identification process by using doubly robust methods not just for the final estimate, but also to debias the feature map itself within that RKHS setting.
Jane: That’s an improvement because it means you get a better model even if your initial guess about the propensity score is slightly off, which is something that always happens in real data collection.
Tom: And they talk about how to tune those learning rates for the deep feature and neural-kernel models more effectively based on the smoothness of the functions involved.
Jane: It’s like getting a better map for how fast your model learns when you change your parameters, which is essential if you're trying to deploy something that needs to adapt quickly.
Tom: They also provide bounds that are specific to certain kernel types, like when the kernel is Gaussian, simplifying the final error calculation down to a cleaner form.
Jane: And that’s interesting because those bounds show how much data you need before you can expect any meaningful improvement in accuracy for each method.
Tom: It ties everything together by showing that all three methods achieve double robustness in those rates, which confirms that the performance is robust regardless of which nuisance model is slightly less accurate.
Jane: So, what this means practically is that if you choose to use a simpler Ridge Regression approach because it's computationally cheaper, you still have a strong theoretical guarantee on how much data you need.
Tom: Right. And for the neural-kernel approach, they show that by using neural networks to parameterize the coefficients instead of just huge matrices, you get inference speeds that are significantly faster than the previous two methods.
Jane: That speed is a big deal for production systems where you might need to query those counterfactual embeddings really quickly.
Tom: And they provide a density estimation formula for the neural-kernel estimator that lets you compute the full conditional counterfactual density at any point y without needing to do any heavy matrix inversions.
Jane: That ability to estimate the whole distribution shape, not just a mean value, is what makes this framework so much more powerful than standard methods for understanding treatment effects.
Tom: We've covered the mechanics and the results now; next time we’ll get into how to actually run these things on big data and what practical limitations they have when you try to deploy them at scale.
Conclusion: Tom: So we've seen how this paper uses Conditional Counterfactual Mean Embeddings to represent distributions in an RKHS, and they built three estimators—Ridge Regression, Deep Feature, and Neural-Kernel—all using a double robust approach to handle selection bias.
Jane: That’s the core of it: they give us a way to move beyond just a single number for treatment effects and start characterizing the whole distribution shape.
Tom: They establish convergence rates that explicitly track things like the dimension of your outcome variables and how smooth the conditional density function is, which is really important for knowing how much data you need.
Jane: The main implication here is that you get a rigorous way to know if an estimator will be accurate before you even start running massive experiments.
Tom: And they prove that all three methods achieve double robustness in those rates, meaning the final result is solid even if one of the initial modeling steps isn't perfect.
Jane: It gives us confidence that we can actually use these tools to make reliable predictions for heterogeneous treatment effects across different groups in real-world scenarios.
Tom: To wrap up on this paper, "Conditional Counterfactual Mean Embeddings: Doubly Robust Estimation and Learning Rates," the main thing is providing a flexible framework with concrete estimators and provable convergence rates that account for data complexity.
Jane: It shows us how to use these embeddings to recover the full conditional counterfactual density, which is a powerful tool for density estimation at specific points.
Tom: For deployment, it means you have choices between simpler regressions or more complex neural network approaches based on what your data demands computationally.
Lu: I just want to add that the idea of using an RKHS to embed these distributions opens up a lot of possibilities for things like conditional independence testing later on.
Meng: From my side, the speed advantage of the neural-kernel estimator when it avoids matrix inversions is what would make it deployable on production systems where latency matters.
Lalam: I'm confident in my assessment; this paper provides a very strong foundation for analyzing these distributions, and I’m sure future work will build right on these convergence rates.
Thatchanon Anancharoenkij Donlapark Ponnoprat
Chiang Mai University
stat.ML, cs.LG
Submitted: 2026-02-04
Updated: 2026-11-09
Comments: Accepted for NeurIPS 2026
Code: https://github.com/donlap/Conditional-Counterfactual-Mean-Embeddings
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 75/100
The gist: A complete understanding of heterogeneous treatment effects involves characterizing the full conditional distribution of potential outcomes Under this framework…
Key concepts
- Conditional Counterfactual Mean Embeddings (CCME)
- CCME embeds conditional distributions of counterfactual outcomes into an RKHS. This allows researchers to analyze how the distribution of a treatment outcome changes given specific covariates, moving beyond simple average effects to capture complex distributional shapes.
- Doubly Robust Methods
- This technique is used for identification by introducing two nuisance functions: the propensity score and the conditional mean embedding. The framework uses a pseudo-outcome constructed from these functions to ensure that the final CCME estimate remains accurate even if one of these two models is slightly incorrect.
- Neural-Kernel Estimator
- This estimator performs RKHS-valued regression using coefficients parameterized by a neural network. Unlike previous methods, it avoids computationally expensive matrix inversions, making inference significantly faster for estimating the conditional counterfactual density.
Terminology
Summary
A complete understanding of heterogeneous treatment effects involves characterizing the full conditional distribution of potential outcomes Under this framework, a two-stage meta-estimator for Conditional Counterfactual Mean Embeddings (CCME) is developed to accommodate any RKHS-valued regression in each stage This framework yields three practical CCME estimators: Ridge Regression estimator, Deep Feature estimator that parameterizes the feature map by a neural network, and Neural-Kernel estimator that performs RKHS-valued regression with coefficients parameterized by a neural network
Framework for Distribution Representation
The Conditional Counterfactual Mean Embeddings (CCME) framework embeds conditional distributions of counterfactual outcomes into a reproducing kernel Hilbert space (RKHS) This allows for the analysis of conditional counterfactual distributions when the distributions vary across subpopulations The target estimand is represented by the Conditional Counterfactual Mean Embeddings (CCME): µY 1V (v) = E[ϕ(Y 1) V = v] ∈ HY This embedding fully characterizes the conditional distribution of Y 1 given V = v when the kernel kY is characteristic
Identification via Doubly Robust Methods
To identify the CCME from observed data (X, A, Y), two nuisance functions are introduced: the propensity score π(x) and the CME of the treated outcome conditional on X, µ0(x) The identification result is established via a pseudo-outcome ξ(Z) = A / π(X) (ϕ(Y) − µ0(X)) + µ0(X) Under the assumptions of conditional ignorability and positivity, the CCME is identified as µY 1V (v) = E[ξ(Z) V = v]
Estimator Development
The estimation strategy follows a two-stage meta-estimator based on Algorithm 1 Stage 1 involves estimating the nuisance functions π and µ0 using data D0, while Stage 2 constructs pseudo-outcomes ξb(Z) from the estimated nuisance functions and regresses them on V to obtain the CCME estimator Three specific estimators are developed:
-
Ridge Regression estimator, which uses kernel ridge regression (KRR)
-
Deep Feature estimator, which parameterizes the feature map by a neural network
-
Neural-Kernel estimator, which performs RKHS-valued regression with coefficients parameterized by a neural network
Convergence and Robustness Properties
Finite-sample convergence rates are established for all three estimators, explicitly tracking the dependence on the dimensions dx, dv, dy and the smoothness of the conditional density Crucially, it is proven that these estimators achieve double robustness in rates; specifically, the first-stage error is controlled by the minimum of the rates achieved by the propensity score estimator and the CME estimator The convergence rate for Ridge Regression is bounded by E[∥µbRR(V) − µY 1V (V)∥ 2 HY] ≲ n − 2r 2r+dv + min R squared π(πb), R squared µ0(µb0)
Application to Density Estimation
The CCME can be used to recover the full conditional counterfactual density (CCD) p 1(yv) The estimated density at any query point y ∈ Y is computed by the inner product: bp 1(yv) = ⟨µbY 1V (v), ϕ(y)⟩HY Explicit formulae for the density estimators based on all three proposed methods are derived, such as bp 1 RR(yv) = D Ξb(KV + nλ1In) −1kV (v), ϕ(y)
Experimental Validation
Experiments on synthetic and semi-synthetic data confirm that the DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR, IPW, and PI variants of the estimators achieve comparable MSE in this setting, while for the Ridge Regression estimator, PI exhibits slightly higher MSE than DR and IPW due to selection bias The DR estimator addresses this through the propensity score weighting
Theoretical Analysis of Rates
The convergence rates depend on the smoothness r of the function v 7→ p 1(yv) and worsen with dv For Deep Feature and Neural-Kernel estimators, the rates improve with the smoothness s of the function y 7→ p 1(yv) and the smoothness of the kernel τ, worsening with dy The rate of Ridge Regression does not depend on dy at all under Assumption 4'
Inference with Neural-Kernel Estimator
The Neural-Kernel estimator computes the CCD estimator as bp 1 NK(yv) = X M j=1 fθb(v)jϕ(yej), ϕ(y) This makes inference with the Neural-Kernel estimator significantly faster compared to the previous two estimators, since there is no need for calculation nor inversion of large matrices
Density Estimation Formulae
The density estimate for the Neural-Kernel estimator is given by bp 1 NK(yv) = X M j=1 fθb(v)jkY (yej, y) This makes inference with the Neural-Kernel estimator significantly faster compared to the previous two estimators, since there is no need for calculation nor inversion of large matrices
Final Bound Derivation
The final bound for the second-stage estimator is E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L ≍ n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound
Inference with Ridge Regression Estimator
The CCD estimator for Ridge Regression is bp 1 RR(yv) = kY0(y)⊤(KX0 + mλ0Im −1kX0X1 (KV + nλ1In −1kV (v),
Inference with Deep Feature Estimator
The CCD estimator for Deep Feature is bp 1 DF(yv) = kY0(y)⊤Ψb 0(Ψb ⊤ 0 Ψb 0 + mλ0Im −1ψθb0(x) (KV + nλ1In −1ψθb(v),
The CCD estimator for Neural-Kernel is bp 1 NK(yv) = X M j=1 fθb(v)jkY (yej, y) This makes inference with the Neural-Kernel estimator significantly faster compared to the previous two estimators, since there is no need for calculation nor inversion of large matrices
Summary of Contributions
We proposed the Conditional Counterfactual Mean Embeddings (CCME) as a framework for representing conditional counterfactual distributions in RKHS We developed a two-stage meta-estimator based on doubly robust pseudo-outcomes and instantiated it with three estimators—Ridge Regression, Deep Feature, and Neural-Kernel—each offering different computational-statistical trade-offs Our finite-sample analysis establishes convergence rates that explicitly track the dependence on the dimensions dx, dv, dy and the smoothness of the conditional density, and shows that all three estimators achieve double robustness in rates (Theorem 1) Experiments on synthetic and semi-synthetic data confirm that the DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR estimator addresses this through the propensity score weighting
Pruning redundant reasoning tokens
The framework for studying conditional counterfactual distributions leverages Kernel Mean Embeddings (KME) to manipulate probability distributions as vectors in an RKHS The CCME enables a wide range of statistical tasks on conditional counterfactual distributions, such as density estimation, conditional independence testing, and sampling The identification relies on two nuisance functions: the propensity score π(x) and the CME of the treated outcome µ0(x)
How the training signal is built
The pseudo-outcome ξ(Z) is constructed as ξ(Z) = A / π(X) (ϕ(Y) − µ0(X)) + µ0(X) The meta-estimator proceeds in two stages: Stage 1 estimates the nuisance functions πb and µb0 using data D0, and Stage 2 constructs pseudo-outcomes ξb(Z1i) and regresses them on V to obtain the CCME estimator
Where the gains come from
The convergence rate for the meta-estimator is bounded by E[∥µbY 1V (V) − µY 1V (V)∥ 2 HY] ≲ R squared ξb(µbY 1V) + min R squared π(πb), R squared µ0(µb0) The gains come from the double robustness property, where the first-stage error is governed by the nuisance with the smaller error The experiments confirm that DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR estimator addresses this through the propensity score weighting
The final bound for the second-stage estimator is E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound
The convergence rate for Ridge Regression is bounded by E[∥µbRR(V) − µY 1V (V)∥ 2 HY] ≲ n − 2r 2r+dv + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing λn ≍ n − τ r+dv/2, for sufficiently large n ≥ 1
The final bound for the Neural-Kernel estimator is E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound
The convergence rate for Deep Feature estimator is bounded by E[∥µbDF(V) − µY 1V (V)∥ 2 HY] ≲ n − ab(a+2)(b+1)-1 (log n) squared + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing M n b (a+2)(b+1)-1 and W L n a+1 (a+2)(b+1)-1
The convergence rate for Neural-Kernel estimator is bounded by E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 2bdy+dy+4b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n a+1 (a+2)(b+1)-1 and M n b (a+2)(b+1)-1 If kY is a Gaussian kernel, the bound simplifies to E[∥µbNK(V) − µY 1V (V)∥ 2 HY] ≲ n − b b+1 (log n) 4bdy+2dy+2b 2(b+1) + min R squared π(πb), R squared µ0(µb0) This is obtained by choosing W L n 1 b+1 and M (log n) 2dy yields the final bound
The convergence rate for all three estimators, ignoring the first-stage nuisance term, achieves the minimax non-parametric regression rate n − 2r 2r+dv over V The Ridge Regression estimator attains this rate directly, as it performs regression in the full RKHS without finite-dimensional projection
The DR estimators recover multimodal conditional counterfactual densities and correct for selection bias where one-step plug-in estimators fail The DR estimator addresses this through the propensity score weighting
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved Causal Inference for Heterogeneous Treatment Effects: The framework allows for estimating conditional counterfactual mean embeddings (CCME) by embedding conditional distributions into a reproducing kernel Hilbert space (RKHS). This enables researchers to move beyond scalar summaries of treatment effects to characterize the full distribution shapes, including multimodal structures, which is critical in finance and healthcare.
-
Enhanced Estimation Robustness via Double Robust Methods: The proposed two-stage meta-estimator extends double robust methods to the RKHS setting by debiasing the feature map of the outcome. This ensures that estimators possess a double robustness property even when only one of the two nuisance models (propensity score or conditional mean embedding) is correctly specified.
-
Scalable and Flexible CCME Estimation: The framework provides three concrete estimators—Ridge Regression, Deep Feature estimator, and Neural-Kernel estimator—allowing users to select methods based on computational trade-offs. The Deep Feature estimator uses a neural network to parameterize the feature map by a learned map, while the Neural-Kernel estimator eliminates matrix inversion by using neural networks parameterized by grid points.
-
Precise Finite Sample Error Bounds: The paper establishes finite sample convergence rates that explicitly track dependence on outcome dimension, covariate dimensions, and conditioning variables. These bounds show that estimators achieve double robustness in rates, providing a rigorous understanding of how smoothness of the conditional density affects estimation performance.
-
Density Estimation from CCME: The CCME can be used to recover the full conditional counterfactual density (CCD) p1(yv). By computing the inner product of the estimated embedding with a kernel function, researchers obtain an explicit formula for the estimated CCD at any query point y, which is useful for density estimation.
Sources
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference