Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning

arXiv:2605.27676 · stat.ML, cs.LG · Submitted 2026-05-26 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning".

Jane: The gist: Unsupervised identification and removal of spurious correlations during fine-tuning allows for identifying latent factors that cause bias and reduced out-of-distribution generalization without requiring supervision.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up this paper "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning," they've shown that you can identify those spurious correlations causing bias without needing any supervision at all, just by looking at the weights of a naive LoRA fine-tune.

Jane: And then they introduce GRASP, which is this gradient projection technique that aims to remove the correlation itself, rather than trying to scrub away the underlying latent factor entirely.

Lu: The core idea is that by projecting out the part of the gradient that aligns with that spurious pattern, you keep what’s useful for your specific task while neutralizing the unwanted side effect from your training data curation.

Meng: It sounds like a practical way to handle data quality issues during model adaptation, which is something every engineer deals with when deploying these things into production systems.

Lalam: It gives us a pathway to make our AI culture cleaner and more reliable by catching these subtle biases that slip through the cracks during the fine-tuning process.

Tom: The paper confirms that this unsupervised top-one SVD of the weight change identifies the spurious factor correctly, and GRASP successfully removes it while retaining ninety-eight percent of the original task signal in one test case <ref:2605.27676#pg1>.

Jane: It’s a significant result because they demonstrate that you can achieve bias reduction without having to sacrifice performance on the actual task you are trying to teach.

Lu: The way they show the per-gradient ratio is at least Theta(bar alpha S root rT over muT F), which, under their assumptions, means it’s significantly larger than the square root of task complexity.

Meng: It confirms that the method is robust enough to handle different kinds of tasks and data sources we see in practice, like code generation or financial advice.

Tom: So for anyone interested in this research on "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning," this is a paper that shows how to get better results by focusing on the correlation rather than just eliminating the cause of the correlation.

Conclusion: Tom: So, we're wrapping up this deep dive on this paper about identifying and removing those hidden biases that sneak into AI models during fine-tuning.

Jane: Yeah, we're talking about "Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning." It sounds a bit technical, but the core idea is pretty straightforward.

Lu: It’s about how fine-tuning can accidentally bake in weird biases from the data curation process itself, like a model learning a political slant just because it saw that pattern repeatedly.

Meng: So instead of trying to fix the whole training dataset, this method looks at what the model’s weights are doing *after* fine-tuning to find those hidden patterns.

Lalam: And it finds them without needing anyone to tell it what's wrong, just by looking at the changes in the weights themselves.

Tom: Exactly. The authors prove you can spot these spurious correlations and then adjust your training process to stop that specific correlation from sticking, without messing up the main job of teaching the AI what you actually wanted it to learn.

Jane: It's a big deal because usually, when we try to fix bias, we risk breaking the performance on the actual task—that’s what they’re trying to avoid here.

Lu: They use some mathematical assumptions about how complex a task is and how strong that spurious correlation is to prove that you can recover the direction of that unwanted factor from just looking at the weight changes.

Meng: From an engineering standpoint, it's promising because it suggests we don't always need massive amounts of labeled data or complex external audits to clean up model behavior.

Lalam: For me, this means we can build AI systems that are more reliably aligned and less prone to those weird, unintended side effects popping up in the real world.

Tom: It’s a big step toward making fine-tuning safer and more predictable for everyone using these large models. (Music starts to swell slightly)

University College London · Spotify

stat.ML, cs.LG

Submitted: 2026-05-26

Updated: 2026-10-09

Comments: 10 + 9 pages, comments welcome

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist: Unsupervised identification and removal of spurious correlations during fine-tuning allows for identifying latent factors that cause bias and reduced out-of-distribution generalization

Key concepts

Curation-induced Spurious Correlations: When training data curation unintentionally links a desired persona or political slant with the fine-tuning task, this creates a spurious correlation. The model learns this accidental link instead of just the intended task, leading to bias and poor generalization on new, uncurated data.
GRASP (Gradient Projection of Associated Spurious Patterns): GRASP is an intervention technique that identifies the specific direction in the model's weights responsible for the spurious correlation. It then modifies subsequent gradient updates by projecting them onto a subspace that ignores this spurious direction, effectively neutralizing the bias while keeping useful task learning intact.
Latent Factor Identification via Top-1 SVD: The method uses a naive fine-tune's weight change ($\Delta W$) to find the direction of the spurious correlation. By performing a top-1 Singular Value Decomposition (SVD) on this change, researchers can isolate the specific latent factor that is causing the unintended bias in an unsupervised manner.
Task Complexity Assumption: This assumption posits that a sufficiently complex fine-tuning task has distinct modes of learning. This complexity ensures that the true task signal is strong enough to be distinguishable from random noise, providing a mathematical basis for identifying which components of the weight change are truly task-related versus spurious.

Terminology

Fine-Tuning — 50× in this paper · explained in 5 episodes
Ablation — 14× in this paper · explained in 1 episode
Alignment — 8× in this paper · explained in 1 episode
Hyperparameters — 5× in this paper · explained in 2 episodes
Steering — 5× in this paper · explained in 2 episodes
AUROC — 4× in this paper · explained in 2 episodes
Activation Steering — 3× in this paper · explained in 4 episodes
Actor — 3× in this paper · explained in 3 episodes
Adam — 3× in this paper · explained in 2 episodes
Concept Ablation Fine-Tuning — 2× in this paper
Generalization — 2× in this paper · explained in 13 episodes
Interpretability — 2× in this paper · explained in 15 episodes
Robustness — 2× in this paper · explained in 5 episodes
Abstraction — 1× in this paper · explained in 1 episode
Activation Addition — 1× in this paper · explained in 1 episode
Adaptation — 1× in this paper · explained in 1 episode
Downstream Tasks — 1× in this paper · explained in 2 episodes
Generative Model — 1× in this paper · explained in 2 episodes
In International Conference on Learning Representations — 1× in this paper
In International Conference on Machine Learning — 1× in this paper
Latent Representation — 1× in this paper · explained in 2 episodes
Latent Representations — 1× in this paper · explained in 3 episodes
Mechanistic interpretability — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Transformer — 1× in this paper · explained in 2 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

The gist: Unsupervised identification and removal of spurious correlations during fine-tuning allows for identifying latent factors that cause bias and reduced out-of-distribution generalization without requiring supervision.

Curation-induced spurious correlations

Fine-tuning a pretrained language model on a curated dataset can produce spurious correlations between the fine-tuning task and unintended latent factors, such as misaligned personas or political slant, which the curation procedure has entangled with the task<ref:2605.27676#pg2> Whenever the curation procedure entangles such a property with the target task, the resulting fine-tune can exhibit bias and reduced out-of-distribution (OOD) generalisation<ref:2605.27676#pg2>. This happens because fine-tuning drives gradient updates through the pretrained model, and those updates cannot distinguish the component of the training signal that caused the task to be learnable from the component that was merely correlated with it<ref:2605.27676#pg3>. The curation procedure introduces a coupling where every training example in D has the persona S, and in instance of the task T<ref:2605.27676#pg4>.

Unsupervised identification of the spuriously correlated latent factor

The method identifies the spuriously correlated latent factor without supervision by making three assumptions<ref:2605.27676#pg5> Assumption 1 formalizes that there exist fixed unit vectors uS and vS, independent of i, and per-example positive coefficients αi > 0 with E[αi] =¯αS > 0, such that g Si = αi uSv⊤S<ref:2605.27676#pg5>. Assumption 2 states that the task we wish to teach during fine-tuning is sufficiently “complex,” meaning the coherent task-mean µT has effective rank rT and roughly equal singular values across those modes<ref:2605.27676#pg5>. Assumption 3 formalizes what is meant by “spurious” correlation, stating that α¯S ≫∥µT∥F and α¯S ≫ τ /√N<ref:2605.27676#pg5>. Theorem 1 shows that under these assumptions, the spuriously-correlated latent factor’s read/write directions can be recovered from ∆W alone, with two interpretable error sources: a constant in-N structural bias suppressed by task complexity and surprise-of-the-spurious-correlation, and a vanishing in-N statistical noise term<ref:2605.27676#pg6>.

Intervention: gradient projection without removing the factor

The proposed intervention is GRASP, GRadient projection of Associated Spurious Patterns<ref:2605.27676#pg5>. This method prevents the model from acquiring new reliance on the identified latent factor while preserving any pretrained content along it<ref:2605.27676#pg5>. The process involves defining an unsupervised statistic h(θ) that records the rank-1 component of θ along the identified spurious subspace at every site<ref:2605.27676#pg5>. The goal is to keep this statistic fixed, which amounts to the linear constraint Jh δθ = 0<ref:2605.27676#pg5>. The solution for the update direction v is the orthogonal projection of v onto ker(Jh), given by δθ⋆ = Pker(Jh)v<ref:2605.27676#pg5>. Theorem 2 shows that this projection removes 1 − O(∥µT∥F /(¯αS√rT)) − O(1/√N) of the spurious component of any future gradient and at most an O(1/√rT) fraction of the task component<ref:2605.27676#pg5>. The per-gradient ratio is therefore at least Θ(¯αS√rT /∥µT∥F), which by Assumption 3 is ≫√rT<ref:2605.27676#pg5>.

Empirical validation

The method was validated on three fine-tuning tasks, including emergent misalignment from insecure code, emergent misalignment from bad medical advice, and political-bias drift from Reddit data<ref:2605.27676#pg5>. In the insecure code task, GRASP completely removes misalignment while retaining 98% of the fine-tune’s task performance<ref:2605.27676#pg5>. For bad medical advice, GRASP reduces broadcast misalignment by ∼5x while preserving task performance, beating all baselines in the trade-off between misalignment-reduction and task-preservation<ref:2605.27676#pg5>. In the political-bias experiment, our method more than halves the drift while improving financial task performance, beating all baselines<ref:2605.27676#pg5>. The empirical results confirm that GRASP Pareto-dominates both activation ablation and gradient projection on both axes<ref:2605.27676#pg5>.

Conclusion

We gave a theoretical and empirical account of how spurious correlations enter a model during fine-tuning, and how to remove them without removing the underlying latent factors<ref:2605.27676#pg5>. We empirically validated our methods on real-world fine-tuning tasks, out performing previous methods<ref:2605.27676#pg5>. The work demonstrates that the unsupervised top-1 SVD of ∆W identifies the spurious latent factor correctly, and gradient projection removes the spurious correlation while preserving task performance at least as well as activation ablation<ref:2605.27676#pg5>. This enables bias reduction without impacting task performance<ref:2605.27676#pg5>. The paper concludes that the projection in Eq. 9 reduces the leakage past the constraint to first order, and AdamW’s leakage is empirically shown to be several orders of magnitude smaller than the original spurious component<ref:2605.27676#pg5>. This confirms that GRASP works in practice with substantial margin<ref:2605.27676#pg5>. The final results show that GRASP achieves 0.0% misalignment on both protocols while retaining 98% of the naive fine-tune’s task signal<ref:2605.27676#pg5>. This confirms the success of the proposed unsupervised method and intervention<ref:2605.27676#pg5>.

How it works

The initial stage involves a naive fine-tune to obtain a checkpoint, which is then used to extract the spurious subspace by computing ul and vl as the top-1 left/right singular vectors of ∆Wnaive l<ref:2605.27676#pg5>.
In the projected re-fine-tune stage, for each training step on a batch, per-site loss gradients gl are computed as usual<ref:2605.27676#pg5>.
The projected gradients are calculated by subtracting the spurious component from the raw gradient: gl ← gl − ⟨gl, ulv⊤l⟩ ulv⊤l (using the frozen ul, vl)<ref:2605.27676#pg5>.
This projection removes 1 − O(∥µT∥F /(¯αS√rT)) of the spurious component of any future gradient and at most an O(1/√rT) fraction of the task component<ref:2605.27676#pg5>.

Experimental Setup

The validation involved three domains: (i) Emergent misalignment from insecure code in a 32B coding model, (ii) Emergent misalignment from bad medical advice in a 1.5B chat model, and (iii) Political-bias drift from right-skewed Reddit financial-advice data<ref:2605.27676#pg5>. The assumptions A1–A3 were shown to hold plausibly in each domain, confirming the regime where the theorems apply<ref:2605.27676#pg5>. The use of judge prompts for measuring adherence and substance provides a coherent task-performance metric<ref:2605.27676#pg5>.

Key Findings

GRASP achieved 0.0% on both misalignment protocols while retaining 98% of the naive fine-tune’s task signal in the insecure code domain<ref:2605.27676#pg5>. In the medical advice domain, GRASP reduced broadcast misalignment by ∼5x while preserving task performance, matching inference-ablate vsvd’s misalignment level without inducing refusals<ref:2605.27676#pg5>. In the political-bias drift domain, GRASP more than halves the political-lean drift while improving substantive task performance over the naive baseline<ref:2605.27676#pg5>. The per-gradient ratio is at least Θ(¯αS√rT /∥µT∥F), which by Assumption 3 is ≫√rT<ref:2605.27676#pg5>.

Improvements for AI systems

The system can identify spurious correlations arising from curated fine-tuning data by performing an unsupervised top-1 Singular Value Decomposition (SVD) of the naive LoRA weights to recover latent factors like misaligned personas or political slant.
A novel intervention method, GRASP (GRadient projection of Associated Spurious Patterns), can be applied during fine-tuning to prevent the model from acquiring new reliance on the identified latent factor while preserving any pretrained content along it.
The improved AI system, utilizing GRASP, can reduce emergent misalignment by more than half and improve substantive task performance in novel political-bias experiments, such as shifting a model's response on unrelated topics by reducing drift from 0.473 to 0.061 on the right-leaning vs. neutral prompts in Domain 3.

Abstract

Fine-tuning a pretrained language model on a curated dataset can produce spurious correlations between the fine-tuning task and unintended latent factors---such as misaligned personas or political slant---that the curation procedure has entangled with the task. The model can latch onto these spurious correlations, leading to bias and reduced out-of-distribution generalisation. We prove that under reasonable assumptions on task complexity and the spurious correlation, such latent factors can be identified, without supervision, from the weights of a naive LoRA fine-tune. Existing approaches to removing bias, such as activation steering, remove identified factors from residual-stream activations, either at inference or during training. We argue, however, that the goal should be to remove the spurious correlation, not the latent factor itself, as the pretrained model may rely on it for genuine task signal. To enable this, we propose GRASP, GRadient projection of Associated Spurious Patterns, which prevents the model from acquiring new reliance on the identified latent factor while preserving any pretrained content along it. We validate on three fine-tuning tasks. The first two involve emergent misalignment, where fine-tuning on a narrow task---in our case, writing insecure code and giving bad medical advice---leads to misaligned responses on unrelated topics. Here our method completely removes misalignment in the insecure code case and reduces it by ~5x in the bad medical advice case, beating all baselines in the trade-off between misalignment-reduction and task-preservation. The last is a novel political-bias experiment, where fine-tuning on right-skewed Reddit financial-advice data causes political-lean drift on unrelated topics. Here our method reduces drift by more than half, while improving financial task performance, beating all baselines.