Anchor PCA
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Anchor PCA".
Jane: Here's the gist of the paper. Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques, and this paper introduces Anchor PCA to find a robust shared low-rank embedding from multi-domain data by focusing on shared directions of variation rather than pooled variance.
Tom: Anchor PCA Formulation Anchor PCA is proposed as a method that trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings.
Jane: It amounts to PCA on a modified target matrix and can be solved efficiently.
Tom: The method is parameterized by λ, where λ trades off between explaining variance and finding invariant directions (e.g., for λ = 0, AnchorPCAλ reduces to poolPCA).
Paper summary: Tom: So, to wrap up on this Anchor PCA paper, it’s about finding a robust shared low-rank embedding from multi-domain data by focusing on shared directions of variation instead of pooled variance.
Jane: And they introduced Anchor PCA, which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings.
Lu: The key result is that Anchor PCA recovers a maximal invariant subspace S and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations.
Meng: So for deployment, it means we have a method that minimizes the worst-case reconstruction error even when the data comes from slightly different domains than what we trained on.
Tom: It’s about finding that shared subspace S and they've shown how to estimate it consistently using finite sample methods when we use AnchorPCA infinity.
Conclusion: Tom: So we’ve gone through the technical stuff on Anchor PCA, and now we need to talk about what this whole paper actually means for us in practice.
Jane: Right, so in a nutshell, they’re proposing this method called Anchor PCA to find a shared low-rank representation across different datasets that don't perfectly match.
Lu: It's trying to balance two things: it wants the representation to capture the general variation across all data while also making sure it agrees with what each individual domain is doing.
Meng: What does that actually look like when we try to use this thing in a real deployment? Does it just give us a single good embedding, or is there more complexity involved?
Tom: Well, the paper focuses on finding this maximal invariant subspace, which they call S star. They’ve shown that if such a subspace exists, Anchor PCA infinity will find it.
Jane: That means we are looking for those directions of variation that actually stay the same no matter which domain you look at. It’s not just about what’s common; it's about what's structurally invariant.
Lalam: From my perspective, Anchor PCA is a novel framework because it explicitly formalizes the trade-off between explaining variance and finding these stable directions. It moves beyond just pooling everything together, which is what standard PCA does.
Tom: And the authors do this by parameterizing it with lambda, so you can tune how much you care about those invariant directions versus just getting the highest possible explained variance from the pooled data.
Jane: So if I were building a system for things like medical images or sensor readings from different hospitals, this could be a way to create a representation that is robust against the specific quirks of any single hospital's equipment.
Meng: But what about the practical side? The paper mentions consistency results for Anchor PCA infinity using finite samples. How reliable is it when we don't have infinite data to test its claims?
Lu: They show that if the true invariant subspace is non-zero, the first block they return consistently estimates that subspace. It’s a strong guarantee under those conditions.
Tom: So, to wrap up this part, Anchor PCA is about finding these stable shared directions using a controlled trade-off between overall variance and domain agreement, and they provide some pretty solid consistency guarantees for finding that shared space.
Jane: It shifts the focus from just maximizing one single metric to managing a compromise between what we want to capture broadly and what we need to be invariant across domains.
Lalam: And the real implication is that for multi-domain representation learning, this provides a new lens—one centered on invariance—that contrasts with methods focused purely on variance maximization.
Tom: It opens the door for building systems that are less sensitive to domain shifts during deployment, which is a huge hurdle right now.
Jane: It certainly does. And next up, we're going to look at how researchers are trying to extend this idea into those nonlinear settings with autoencoders.
Benedikt Seiter, Anya Fries, Julius von Külgen, Jonas Peters
ETH Zurich
stat.ML, cs.LG, stat.ME
Submitted: 2026-06-04
Updated: 2026-11-04
Comments: Accepted at NeurIPS 2026. 47 pages
Code: https://github.com/benesei/anchor-pca
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques…
Key concepts
- Anchor PCA
- A dimension reduction technique that finds a robust shared low-rank embedding from multi-domain data. It achieves this by optimizing an objective that balances explaining overall variance with ensuring the shared and domain-specific embeddings are consistent, trading variance for invariance.
- Maximal Invariant Subspace (S⋆)
- This is the largest possible subspace where the variation is invariant across all domains. Anchor PCA aims to find this subspace, denoted as S⋆, which represents the most stable directions of variation shared by all data domains.
- λ (Lambda)
- A parameter used in Anchor PCA that controls the trade-off between explaining variance and finding invariant directions. When λ is zero, it behaves like a simpler pooling method; as λ increases, it prioritizes finding more invariant directions.
- AnchorPCA∞
- The version of Anchor PCA designed to maximize pooled explained variance while subject to an invariance penalty in a supervised setting. It seeks the best possible representation under constraints related to domain stability.
Terminology
Summary
Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques, and this paper introduces Anchor PCA to find a robust shared low-rank embedding from multi-domain data by focusing on shared directions of variation rather than pooled variance. p. 0 Taken together, these findings establish Anchor PCA as a promising
approach to robust unsupervised dimension reduction from multi-domain data. 1 Introduction
Principal component analysis (PCA) [20] is one of the most widely used methods for unsupervised
dimensionality reduction.Where the paper says it
Anchor PCA Formulation
Anchor PCA is proposed as a method that trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. It amounts to PCA on a modified target matrix and can be solved efficiently. p. 0 To this end, we introduce Anchor PCA which trades off
overall explained variance with agreement between the shared and domain-specific
low-rank embeddings. Anchor PCA amounts to PCA on a modified target matrix
and thus can be solved efficiently. 5
ance penalty in the supervised setting. Concretely, we propose AnchorPCAλ (with λ ∈ R>0 ∪ {∞}),
a method that finds a k-dimensional subspace where the parameter λ trades off between explaining
variance and finding invariant directions (e. Concretely, we propose AnchorPCAλ (with λ ∈ R>0 ∪ {∞}),
a method that finds a k-dimensional subspace where the parameter λ trades off between explaining
variance and finding invariant directions (e. , for λ = 0, AnchorPCAλ reduces to poolPCA). This view has two advantages: (i) It suggests computationally feasible ways to solve the optimization
problems of AnchorPCA∞ and AnchorPCAλ (see § 5). (ii) It allows for a geometric interpretation:
Tr(ΠW Π
(e)
k
) is the sum of squared cosines of the principal angles between span(W) and the principal
subspace of domain e.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Maximal Invariant Subspace Recovery
The paper defines the maximal invariant subspace as S⋆:= E Im Π(e)k, where m:= dim(S⋆) < p is the dimension of S⋆. AnchorPCA∞ solves a problem that maximizes pooled explained variance subject to an invariance penalty in the supervised setting. p. 3 Such a space may be represented using different bases; we propose to choose one
that maximizes pooled explained variance. We refer to this approach as AnchorPCA∞. We prove that AnchorPCA satisfies two attractive theoretical properties. First, if there exists a
nontrivial m-dimensional invariant subspace S⋆, that is, a subspace that lies in the top-k principal
subspace of every domain, then AnchorPCA∞ is guaranteed to contain it, and the leading directions
of AnchorPCAλ converge to S⋆ as λ → ∞ (Thm. We prove that AnchorPCA satisfies two attractive theoretical properties. First, if there exists a
nontrivial m-dimensional invariant subspace S⋆, that is, a subspace that lies in the top-k principal
subspace of every domain, then AnchorPCA∞ is guaranteed to contain it, and the leading directions
of AnchorPCAλ converge to S⋆ as λ → ∞ (Thm.Where the paper says it
Where the paper says it
Where the paper says it
Robustness and Optimal Reconstruction
AnchorPCA∞ is shown to satisfy an asymptotic minimax-optimality guarantee for reconstruction error under additional variance inflations of arbitrary strengths in certain directions. Specifically, Theorem 4.1 proves that AnchorPCAλ minimizes the worst-case reconstruction error over test domains that are subject to domain-specific covariance inflations (Cλ). p. 1 First, if there exists a
nontrivial m-dimensional invariant subspace S⋆, that is, a subspace that lies in the top-k principal
subspace of every domain, then AnchorPCA∞ is guaranteed to contain it, and the leading directions
of AnchorPCAλ converge to S⋆ as λ → ∞ (Thm. Second, AnchorPCAλ admits a minimax reconstruction interpretation: it is the rank-k projector that minimizes worst-case average reconstruction
error over test domains that are subject to domain-specific covariance inflations (Thm. , Σ
′
E) = 1
E
X
E
e=1
TrΣe(Ip − ΠW )
+ λ
X
E
e=1
∥ΠW − Π
(e)
k
∥
2
F. (4)
Equation (4) shows that the worst-case reconstruction error is affine in λ, with slope equal to the
projector-disagreement penalty.Where the paper says it
Where the paper says it
Consistency and Empirical Estimation
The paper provides finite-sample methods for both AnchorPCA versions. For AnchorPCA∞, a block-stabilized empirical AnchorPCA∞ construction is presented which uses a tolerance toln > 0. p. 7 3 shows that the first grouped eigenspace, Eig d1
, of the block-stabilized
AnchorPCA∞ construction consistently estimates S⋆ when the tolerance shrinks while dominating the
empirical perturbation. We test this statement in a random-subspace model where S⋆ is known, using
the default automatic block tolerance (tolN = 0. 1 Recovering the invariant subspace S⋆
In this experiment, all environments have equal sample size, which we call N (which thus also
equals n = mine∈E ne). 3 shows that the first grouped eigenspace, Eig d1
, of the block-stabilized
AnchorPCA∞ construction consistently estimates S⋆ when the tolerance shrinks while dominating the
empirical perturbation.Where the paper says it
Where the paper says it
Experimental Validation
Experiments demonstrate the effectiveness of Anchor PCA on simulated and real-world data. In the gas sensor drift application, AnchorPCA∞ yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. p. 0 Moreover, we show that Anchor PCA recovers
a maximal invariant subspace and admits a minimax reconstruction interpretation
under bounded domain-specific covariance inflations. On simulated and real-world
gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA
recovers the maximally invariant subspace and yields embeddings that explain
more variance on unseen domains than the pooling baseline and a worst-case
alternative. All methods use k = 20 and sourceonly preprocessing/fitting; norm-maxRegret is the normalized regret wcPCA baseline [15]. The
finite-penalty method AnchorPCAλ=1 exhibits the source–target compromise: it retains more source
explained variance than AnchorPCA∞ but yields smaller average target gains.Where the paper says it
Where the paper says it
Nonlinear Extensions
The authors discuss nonlinear extensions of the approach to finding rank-k robust or invariant subspaces via Anchor PCA translated to autoencoder-based approaches. They propose AnchorAE, which is an unconstrained optimization problem that minimizes pooled reconstruction error plus a domain disagreement penalty between shared and domain-specific AEs. p. 3 We
therefore propose selecting a rank-k subspace that minimizes disagreement with the domain-specific
top-k eigenspaces. Such a space may be represented using different bases; we propose to choose one
that maximizes pooled explained variance. On simulated and real-world
gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA
recovers the maximally invariant subspace and yields embeddings that explain
more variance on unseen domains than the pooling baseline and a worst-case
alternative. Taken together, these findings establish Anchor PCA as a promising
approach to robust unsupervised dimension reduction from multi-domain data.Where the paper says it
Where the paper says it
Further Related Work
Anchor PCA builds on ideas of invariance in multi-domain dimension reduction and related prediction methods. It is categorized among invariance-based methods that seek representations whose distributions are stable across domains, trading invariance with explained variance. p. 0 In supervised learning, invariance of predictive relations or
representations across domains is widely studied [3, 44]. In contrast, we introduce a geometric notion
of invariance based on agreement of domain-specific principal subspaces, and explicitly trade this
against explained variance. Moreover, we show that Anchor PCA recovers
a maximal invariant subspace and admits a minimax reconstruction interpretation
under bounded domain-specific covariance inflations. On simulated and real-world
gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA
recovers the maximally invariant subspace and yields embeddings that explain
more variance on unseen domains than the pooling baseline and a worst-case
alternative. AnchorPCA∞ gives up
more source explained variance, achieves the highest mean target EV, and improves over poolPCA on
every held-out target batch; the arrow marks its largest relative target gain over poolPCA (B9, 90. 7 Discussion and future work
We introduced Anchor PCA, a novel framework for PCA on multiple domains.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Robust unsupervised dimension reduction for multi-domain data using Anchor PCA can recover maximal invariant subspaces by focusing on shared directions of variation rather than pooled variance. This allows systems to find a low-rank embedding that explains more variance on unseen but similar domains than baseline methods.
-
Anchor PCA provides a minimax reconstruction interpretation, meaning it is the rank-k projector that minimizes worst-case average reconstruction error over test domains subject to domain specific covariance inflations. This makes the resulting embeddings robust against structured perturbations in data distributions.
-
The system can utilize Anchor PCAλ to balance explained variance and agreement between domain specific low rank embeddings by tuning a penalty parameter λ. This provides a tunable control over the trade off between maximizing variance and enforcing invariance, allowing for tailored robustness based on application needs.
-
For finite data settings, the system can employ empirical Anchor PCA methods that use block stabilization to consistently estimate the maximal invariant subspace S star even when eigenvalue ties are present in empirical samples. This allows for the estimation of a shared structure even under noisy, finite sample conditions.
-
The system can perform robust dimension estimation by implementing FindS star, a procedure that uses sequential hypothesis testing based on complementary projectors to directly estimate the dimension of the maximal invariant subspace S star. This provides a statistically rigorous way to determine the true dimensionality of the shared structure.
Abstract
Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques. We study PCA for data from multiple related domains. Since principal components generally differ across domains, one way to obtain a shared low-rank embedding is to perform PCA on the pooled data. However, this approach can focus on spurious directions that exhibit high variation in only a few domains. To find a robust embedding that still explains most variance in unseen but similar domains, we propose instead to focus on shared directions of variation. To this end, we introduce Anchor PCA which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. Anchor PCA amounts to PCA on a modified target matrix and thus can be solved efficiently. Moreover, we show that Anchor PCA recovers a maximal invariant subspace and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations. On simulated and real-world data with temporal distribution shift, we demonstrate, respectively, that Anchor PCA recovers the maximal invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. Taken together, these findings establish Anchor PCA as a promising approach to robust unsupervised dimension reduction from multi-domain data.
Sources
- Invariant Risk Minimization
- Maximum Risk Minimization with Random Forests
- Worst-case low-rank approximations
- Distributional Principal Autoencoders
- StablePCA: Distributionally Robust Learning of Shared Representations from Multi-Source Data
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference