Anchor PCA

arXiv:2606.06233 · stat.ML, cs.LG, stat.ME · Submitted 2026-06-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Anchor PCA".

Jane: Here's the gist of the paper. Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques, and this paper introduces Anchor PCA to find a robust shared low-rank embedding from multi-domain data by focusing on shared directions of variation rather than pooled variance.

Tom: Anchor PCA Formulation Anchor PCA is proposed as a method that trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings.

Jane: It amounts to PCA on a modified target matrix and can be solved efficiently.

Tom: The method is parameterized by λ, where λ trades off between explaining variance and finding invariant directions (e.g., for λ = 0, AnchorPCAλ reduces to poolPCA).

Paper summary: Tom: So, to wrap up on this Anchor PCA paper, it’s about finding a robust shared low-rank embedding from multi-domain data by focusing on shared directions of variation instead of pooled variance.

Jane: And they introduced Anchor PCA, which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings.

Lu: The key result is that Anchor PCA recovers a maximal invariant subspace S and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations.

Meng: So for deployment, it means we have a method that minimizes the worst-case reconstruction error even when the data comes from slightly different domains than what we trained on.

Tom: It’s about finding that shared subspace S and they've shown how to estimate it consistently using finite sample methods when we use AnchorPCA infinity.

Conclusion: Tom: So we’ve gone through the technical stuff on Anchor PCA, and now we need to talk about what this whole paper actually means for us in practice.

Jane: Right, so in a nutshell, they’re proposing this method called Anchor PCA to find a shared low-rank representation across different datasets that don't perfectly match.

Lu: It's trying to balance two things: it wants the representation to capture the general variation across all data while also making sure it agrees with what each individual domain is doing.

Meng: What does that actually look like when we try to use this thing in a real deployment? Does it just give us a single good embedding, or is there more complexity involved?

Tom: Well, the paper focuses on finding this maximal invariant subspace, which they call S star. They’ve shown that if such a subspace exists, Anchor PCA infinity will find it.

Jane: That means we are looking for those directions of variation that actually stay the same no matter which domain you look at. It’s not just about what’s common; it's about what's structurally invariant.

Lalam: From my perspective, Anchor PCA is a novel framework because it explicitly formalizes the trade-off between explaining variance and finding these stable directions. It moves beyond just pooling everything together, which is what standard PCA does.

Tom: And the authors do this by parameterizing it with lambda, so you can tune how much you care about those invariant directions versus just getting the highest possible explained variance from the pooled data.

Jane: So if I were building a system for things like medical images or sensor readings from different hospitals, this could be a way to create a representation that is robust against the specific quirks of any single hospital's equipment.

Meng: But what about the practical side? The paper mentions consistency results for Anchor PCA infinity using finite samples. How reliable is it when we don't have infinite data to test its claims?

Lu: They show that if the true invariant subspace is non-zero, the first block they return consistently estimates that subspace. It’s a strong guarantee under those conditions.

Tom: So, to wrap up this part, Anchor PCA is about finding these stable shared directions using a controlled trade-off between overall variance and domain agreement, and they provide some pretty solid consistency guarantees for finding that shared space.

Jane: It shifts the focus from just maximizing one single metric to managing a compromise between what we want to capture broadly and what we need to be invariant across domains.

Lalam: And the real implication is that for multi-domain representation learning, this provides a new lens—one centered on invariance—that contrasts with methods focused purely on variance maximization.

Tom: It opens the door for building systems that are less sensitive to domain shifts during deployment, which is a huge hurdle right now.

Jane: It certainly does. And next up, we're going to look at how researchers are trying to extend this idea into those nonlinear settings with autoencoders.

Benedikt Seiter, Anya Fries, Julius von Külgen, Jonas Peters

ETH Zurich

stat.ML, cs.LG, stat.ME

Submitted: 2026-06-04

Updated: 2026-11-04

Comments: Accepted at NeurIPS 2026. 47 pages

Code: https://github.com/benesei/anchor-pca

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques…

Key concepts

Anchor PCA
A dimension reduction technique that finds a robust shared low-rank embedding from multi-domain data. It achieves this by optimizing an objective that balances explaining overall variance with ensuring the shared and domain-specific embeddings are consistent, trading variance for invariance.
Maximal Invariant Subspace (S⋆)
This is the largest possible subspace where the variation is invariant across all domains. Anchor PCA aims to find this subspace, denoted as S⋆, which represents the most stable directions of variation shared by all data domains.
λ (Lambda)
A parameter used in Anchor PCA that controls the trade-off between explaining variance and finding invariant directions. When λ is zero, it behaves like a simpler pooling method; as λ increases, it prioritizes finding more invariant directions.
AnchorPCA∞
The version of Anchor PCA designed to maximize pooled explained variance while subject to an invariance penalty in a supervised setting. It seeks the best possible representation under constraints related to domain stability.

Terminology

Summary

Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques, and this paper introduces Anchor PCA to find a robust shared low-rank embedding from multi-domain data by focusing on shared directions of variation rather than pooled variance. p. 0

Where the paper says it

Taken together, these findings establish Anchor PCA as a promising approach to robust unsupervised dimension reduction from multi-domain data. 1 Introduction Principal component analysis (PCA) [20] is one of the most widely used methods for unsupervised dimensionality reduction.

Anchor PCA Formulation

Anchor PCA is proposed as a method that trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. It amounts to PCA on a modified target matrix and can be solved efficiently. p. 0

Where the paper says it

To this end, we introduce Anchor PCA which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. Anchor PCA amounts to PCA on a modified target matrix and thus can be solved efficiently.

The method is parameterized by λ, where λ trades off between explaining variance and finding invariant directions (e. p. 1
Where the paper says it

5 ance penalty in the supervised setting. Concretely, we propose AnchorPCAλ (with λ ∈ R>0 ∪ {∞}), a method that finds a k-dimensional subspace where the parameter λ trades off between explaining variance and finding invariant directions (e.

g., for λ = 0, AnchorPCAλ reduces to poolPCA). p. 1
Where the paper says it

Concretely, we propose AnchorPCAλ (with λ ∈ R>0 ∪ {∞}), a method that finds a k-dimensional subspace where the parameter λ trades off between explaining variance and finding invariant directions (e. , for λ = 0, AnchorPCAλ reduces to poolPCA).

This trade-off is formalized in the optimization problem (2) for AnchorPCAλ. p. 4
Where the paper says it

This view has two advantages: (i) It suggests computationally feasible ways to solve the optimization problems of AnchorPCA∞ and AnchorPCAλ (see § 5). (ii) It allows for a geometric interpretation: Tr(ΠW Π (e) k ) is the sum of squared cosines of the principal angles between span(W) and the principal subspace of domain e.

Maximal Invariant Subspace Recovery

The paper defines the maximal invariant subspace as S⋆:= E Im Π(e)k, where m:= dim(S⋆) < p is the dimension of S⋆. AnchorPCA∞ solves a problem that maximizes pooled explained variance subject to an invariance penalty in the supervised setting. p. 3

Where the paper says it

Such a space may be represented using different bases; we propose to choose one that maximizes pooled explained variance. We refer to this approach as AnchorPCA∞.

The theorem states that if there exists a nontrivial m-dimensional invariant subspace S⋆, then AnchorPCA∞ is guaranteed to contain it. p. 1
Where the paper says it

We prove that AnchorPCA satisfies two attractive theoretical properties. First, if there exists a nontrivial m-dimensional invariant subspace S⋆, that is, a subspace that lies in the top-k principal subspace of every domain, then AnchorPCA∞ is guaranteed to contain it, and the leading directions of AnchorPCAλ converge to S⋆ as λ → ∞ (Thm.

Furthermore, the leading directions of AnchorPCAλ converge to S⋆ as λ → ∞. p. 1
Where the paper says it

We prove that AnchorPCA satisfies two attractive theoretical properties. First, if there exists a nontrivial m-dimensional invariant subspace S⋆, that is, a subspace that lies in the top-k principal subspace of every domain, then AnchorPCA∞ is guaranteed to contain it, and the leading directions of AnchorPCAλ converge to S⋆ as λ → ∞ (Thm.

Robustness and Optimal Reconstruction

AnchorPCA∞ is shown to satisfy an asymptotic minimax-optimality guarantee for reconstruction error under additional variance inflations of arbitrary strengths in certain directions. Specifically, Theorem 4.1 proves that AnchorPCAλ minimizes the worst-case reconstruction error over test domains that are subject to domain-specific covariance inflations (Cλ). p. 1

Where the paper says it

First, if there exists a nontrivial m-dimensional invariant subspace S⋆, that is, a subspace that lies in the top-k principal subspace of every domain, then AnchorPCA∞ is guaranteed to contain it, and the leading directions of AnchorPCAλ converge to S⋆ as λ → ∞ (Thm. Second, AnchorPCAλ admits a minimax reconstruction interpretation: it is the rank-k projector that minimizes worst-case average reconstruction error over test domains that are subject to domain-specific covariance inflations (Thm.

This is achieved by minimizing the slope of the worst-case reconstruction error in terms of projector disagreement. p. 5
Where the paper says it

, Σ ′ E) = 1 E X E e=1 TrΣe(Ip − ΠW ) + λ X E e=1 ∥ΠW − Π (e) k ∥ 2 F. (4) Equation (4) shows that the worst-case reconstruction error is affine in λ, with slope equal to the projector-disagreement penalty.

Consistency and Empirical Estimation

The paper provides finite-sample methods for both AnchorPCA versions. For AnchorPCA∞, a block-stabilized empirical AnchorPCA∞ construction is presented which uses a tolerance toln > 0. p. 7

Where the paper says it

3 shows that the first grouped eigenspace, Eig d1 , of the block-stabilized AnchorPCA∞ construction consistently estimates S⋆ when the tolerance shrinks while dominating the empirical perturbation. We test this statement in a random-subspace model where S⋆ is known, using the default automatic block tolerance (tolN = 0.

Consistency results show that when S⋆ ≠ S⋆ = 0, the first block returned by AnchorPCA∞ consistently estimates S⋆. p. 7
Where the paper says it

1 Recovering the invariant subspace S⋆ In this experiment, all environments have equal sample size, which we call N (which thus also equals n = mine∈E ne). 3 shows that the first grouped eigenspace, Eig d1 , of the block-stabilized AnchorPCA∞ construction consistently estimates S⋆ when the tolerance shrinks while dominating the empirical perturbation.

Experimental Validation

Experiments demonstrate the effectiveness of Anchor PCA on simulated and real-world data. In the gas sensor drift application, AnchorPCA∞ yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. p. 0

Where the paper says it

Moreover, we show that Anchor PCA recovers a maximal invariant subspace and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations. On simulated and real-world gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA recovers the maximally invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative.

The results confirm that AnchorPCAλ=1 exhibits the source–target compromise in terms of explained variance. p. 9
Where the paper says it

All methods use k = 20 and sourceonly preprocessing/fitting; norm-maxRegret is the normalized regret wcPCA baseline [15]. The finite-penalty method AnchorPCAλ=1 exhibits the source–target compromise: it retains more source explained variance than AnchorPCA∞ but yields smaller average target gains.

Nonlinear Extensions

The authors discuss nonlinear extensions of the approach to finding rank-k robust or invariant subspaces via Anchor PCA translated to autoencoder-based approaches. They propose AnchorAE, which is an unconstrained optimization problem that minimizes pooled reconstruction error plus a domain disagreement penalty between shared and domain-specific AEs. p. 3

Where the paper says it

We therefore propose selecting a rank-k subspace that minimizes disagreement with the domain-specific top-k eigenspaces. Such a space may be represented using different bases; we propose to choose one that maximizes pooled explained variance.

The DPA approach extends this to a multi-domain setting by training a single shared encoder and domain-specific decoders. p. 0
Where the paper says it

On simulated and real-world gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA recovers the maximally invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. Taken together, these findings establish Anchor PCA as a promising approach to robust unsupervised dimension reduction from multi-domain data.

Further Related Work

Anchor PCA builds on ideas of invariance in multi-domain dimension reduction and related prediction methods. It is categorized among invariance-based methods that seek representations whose distributions are stable across domains, trading invariance with explained variance. p. 0

Where the paper says it

In supervised learning, invariance of predictive relations or representations across domains is widely studied [3, 44]. In contrast, we introduce a geometric notion of invariance based on agreement of domain-specific principal subspaces, and explicitly trade this against explained variance.

The paper also contrasts Anchor PCA with worst-case methods like FairPCA and StablePCA by highlighting its ability to recover the maximal invariant subspace S⋆. p. 0
Where the paper says it

Moreover, we show that Anchor PCA recovers a maximal invariant subspace and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations. On simulated and real-world gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA recovers the maximally invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative.

The analysis shows that Anchor PCA is a novel framework for PCA on multiple domains that builds on ideas of invariance. p. 9
Where the paper says it

AnchorPCA∞ gives up more source explained variance, achieves the highest mean target EV, and improves over poolPCA on every held-out target batch; the arrow marks its largest relative target gain over poolPCA (B9, 90. 7 Discussion and future work We introduced Anchor PCA, a novel framework for PCA on multiple domains.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Robust unsupervised dimension reduction for multi-domain data using Anchor PCA can recover maximal invariant subspaces by focusing on shared directions of variation rather than pooled variance. This allows systems to find a low-rank embedding that explains more variance on unseen but similar domains than baseline methods.

  2. Anchor PCA provides a minimax reconstruction interpretation, meaning it is the rank-k projector that minimizes worst-case average reconstruction error over test domains subject to domain specific covariance inflations. This makes the resulting embeddings robust against structured perturbations in data distributions.

  3. The system can utilize Anchor PCAλ to balance explained variance and agreement between domain specific low rank embeddings by tuning a penalty parameter λ. This provides a tunable control over the trade off between maximizing variance and enforcing invariance, allowing for tailored robustness based on application needs.

  4. For finite data settings, the system can employ empirical Anchor PCA methods that use block stabilization to consistently estimate the maximal invariant subspace S star even when eigenvalue ties are present in empirical samples. This allows for the estimation of a shared structure even under noisy, finite sample conditions.

  5. The system can perform robust dimension estimation by implementing FindS star, a procedure that uses sequential hypothesis testing based on complementary projectors to directly estimate the dimension of the maximal invariant subspace S star. This provides a statistically rigorous way to determine the true dimensionality of the shared structure.

Abstract

Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques. We study PCA for data from multiple related domains. Since principal components generally differ across domains, one way to obtain a shared low-rank embedding is to perform PCA on the pooled data. However, this approach can focus on spurious directions that exhibit high variation in only a few domains. To find a robust embedding that still explains most variance in unseen but similar domains, we propose instead to focus on shared directions of variation. To this end, we introduce Anchor PCA which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. Anchor PCA amounts to PCA on a modified target matrix and thus can be solved efficiently. Moreover, we show that Anchor PCA recovers a maximal invariant subspace and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations. On simulated and real-world data with temporal distribution shift, we demonstrate, respectively, that Anchor PCA recovers the maximal invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. Taken together, these findings establish Anchor PCA as a promising approach to robust unsupervised dimension reduction from multi-domain data.

Sources

Related papers