SHIFT: Steering Hidden Intermediates in Flow Transformers
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "SHIFT: Steering Hidden Intermediates in Flow Transformers".
Jane: Here's the gist of the paper. Diffusion models have become leading approaches for highfidelity image generation, and SHIFT proposes a simple but effective and lightweight framework for concept removal in DiT diffusion models via targeted manipulation of intermediate activations at inference time.
Tom: Pruning redundant reasoning tokens SHIFT introduces a steering framework specifically engineered for the Multimodal Diffusion Transformer (MM-DiT) architecture, demonstrating that it identifies temporally invariant steering vectors that remain semantically stable, providing a concept control mechanism that does not require retraining.
Jane: The authors establish that this activation-level modulation extends to domain stylization and object-specific biasing beyond just concept erasure.
Tom: SHIFT is positioned as a generation control framework rather than a traditional image-editing tool, focusing on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image.
Paper summary: Tom: So, we're looking at this paper called "SHIFT: Steering Hidden Intermediates in Flow Transformers," and the main idea is that they want a simple way to remove specific concepts from these high-quality image generation models, especially the DiT ones.
Jane: That sounds like something really practical for people who want to use these tools but need more control over what they're getting back, right? What exactly is SHIFT proposing at first glance?
Tom: Well, it suggests using a steering framework that manipulates intermediate activations during the inference time of these DiT diffusion models instead of retraining the entire model. They claim this method can suppress unwanted visual concepts while keeping the prompt's rest and overall image quality intact.
Lu: What's interesting is that they aren't just focused on simple erasure; they suggest this same mechanism can also shift generations into a different style domain or bias samples toward adding or changing specific objects, which is pretty broad.
Meng: That sounds ambitious, but from an engineering standpoint, the biggest question for me is how much effort this takes. Does it require any kind of retraining or extensive fine-tuning to get these steering vectors ready?
Lalam: The paper states that SHIFT learns steering vectors dynamically applied to selected layers and timesteps to suppress unwanted visual concepts while preserving the prompt’s rest and overall image quality <ref:2604.09213#pg1>.
Tom: Exactly, so it avoids retraining, which is a big deal for deployment because those models are already huge. But how do they figure out what these steering vectors should be in the first place?
Jane: That brings us to how they build the training signal for these vectors; it sounds like they create a small paired dataset of prompt pairs, where one prompt has the target concept added to a neutral one.
Lu: After collecting that data, they use two main ways to estimate these steering vectors: first, they build a labeled dataset and train a linear support vector machine on it to define the steering direction as the normalized normal vector of the separating hyperplane.
Tom: And there's another way, which is this mean-difference direction calculation where they take the sum over all prompts and subtract one from the other for each concept pair to get that vdiff value.
Paper summary: Meng: So you’re essentially creating a mathematical direction in the latent space based on those prompt differences, and then using a machine learning classifier to fine-tune how strong that steering push should be during generation?
Lalam: Yes, the application of steering during inference time is defined by an intervention formula where alpha is the steering strength and alpha equals gamma times eta times classifier signal for diffusion transformer steering <ref:2604.09213#pg2>.
Jane: That sounds like they have a lightweight way to regularize that strength using a classifier, which helps suppress the steering when activations are far away from the target class and amplifies it when there’s strong evidence of the target concept remaining.
Tom: And what they show is that this method clearly outperforms all their baselines on safety-related metrics, achieving more than three times and four times stronger suppression at different steering strengths.
Lu: For concrete object removal, SHIFT shows superior performance compared to methods like UCE, CA, and ESD across various tasks like Snoopy and Mickey <ref:2604.09213#pg3>.
Meng: That's good to hear on performance metrics, but what about the trade-offs? What happens when you push that steering strength too far? Does it just make the image look weird or ruin the whole picture?
Lalam: The ablation studies reveal that steering is most effective during the early stages of the diffusion trajectory, whereas applying steering in the latter half of generation yields insufficient concept suppression <ref:2604.09213#pg3>.
Tom: So they're saying you gotta hit it early in the process for this to work well, which makes sense if you're trying to remove something that was introduced right at the start. But what about the overall quality if you restrict where you apply this steering?
Lu: They found that restricting steering to a subset of blocks can improve quality, and restricting it to a subset of blocks is substantially more effective than applying it across all blocks for tasks like adding a hat <ref:2604.09213#pg3>.
Jane: That’s interesting because the paper also points out that using only text-encoder steering or only diffusion-transformer steering isn't enough on its own for high-quality erasure, though increased text steering improves removal but risks over-erasure <ref:2604.09213#pg2>.
Tom: And they showed that token-wise difference in the steering vector parameterization yields stronger target concept suppression, which is a concrete detail about how they tune it.
Paper summary: Meng: So so, if I’m deploying this to production, I need to know how scalable this is. Does running these extra steps for steering add significant latency compared to just running a standard inference pass?
Lalam: The authors conclude that SHIFT provides a lightweight, inference-time control mechanism that does not modify model weights and can be applied on demand for concept removal, domain/style shifts, and object-level interventions <ref:2604.09213#pg1>.
Jane: So, the main point is that this approach offers versatile control—erasure, style shifting like sketch or cyberpunk—and it seems to transfer these steering vectors across distillation boundaries when moving between models <ref:2604.09213#pg3>.
Tom: And they confirm that using only text steering or only diffusion-transformer steering is insufficient for high-quality erasure, but increasing the text steering helps, which then a classifier mitigates the risk of over-erasure <ref:2604.09213#pg2>.
Lu: The results show this method can successfully switch a wide range of concepts, including global attributes like age and small objects like an apple versus a banana <ref:2604.09213#pg3>.
Meng: I'm still worried about the complexity of setting up that initial paired dataset for concept removal; does that data collection process itself become a bottleneck when you want to apply it to many different models or prompts?
Lalam: The paper shows the construction of steering vectors involves collecting a small paired dataset of prompt pairs where one prompt adds the target concept to an otherwise neutral prompt <ref:2604.09213#pg1>.
Jane: So, even if you have all the best math in the world, you still need that initial setup phase to generate those specific training signals before you can use the steering mechanism itself.
Tom: The title of this work, "SHIFT: Steering Hidden Intermediates in Flow Transformers," really captures how they are focusing on these intermediate activations rather than just tweaking the final output layer.
Lu: It seems like they’ve established that semantic concepts are encoded as stable, global directions in the latent manifolds of DiTs <ref:2604.09213#pg3>.
Meng: So what does this mean for the average person who just uses an image generator? Do I need to know about flow transformers or diffusion timesteps to use this?
Jane: It means that for people who want precise control over what they generate, you can now target specific parts of the generation process without having to retrain a massive model every time.
Lalam: Overall, SHIFT achieves competitive performance while maintaining high computational efficiency <ref:2604.09213#pg1>.
Conclusion: Tom: So we’re looking at this paper called "SHIFT: Steering Hidden Intermediates in Flow Transformers" and it's about using hidden parts of the AI model to control what it makes.
Jane: It’s about taking these big diffusion models, like DiT, and instead of changing the whole thing, we just nudge specific internal signals at inference time to get rid of a concept.
Lu: The core idea is finding those stable directions in the model's internal math that control things like concept removal or even style shifts.
Meng: So it’s about steering, right? Not editing an image after it’s done, but steering the generation process itself.
Lalam: Exactly. They create a mechanism that lets you apply this steering signal during the generation steps, which is when the image is actually being built pixel by pixel or token by token.
Tom: The authors are focused on making this framework light and efficient because we’re shipping these models everywhere, and they claim it doesn't require retraining.
Jane: It seems like they’re showing that you can achieve concept control without having to touch the model's massive weights at all, which is a huge win for deployment.
Lu: They use two different methods to figure out these steering directions, one based on building a dataset and training a linear support vector machine, and another using this mean difference direction calculation.
Meng: Which means they’re not just guessing; they have some concrete math to define the direction they want the AI to move in.
Lalam: Right. And then during inference, you use this steering strength, which is controlled by a classifier signal that helps manage how much you push that steering around.
Tom: So, it’s not just one setting; there’s a feedback loop managing the strength of that nudge to keep the result high quality.
Jane: It shows they can do more than just erase things; they can also use this steering for things like adding new objects or shifting the style of an image completely.
Lu: The versatility is what’s interesting here, because these same vectors seem to work across different models and even between different types of AI distillation.
Meng: That transferability across distillation boundaries is important because it suggests this method might be useful for adapting models we don't directly train on.
Lalam: It confirms that the concept steering direction is a general thing, not just specific to one model architecture or one task.
Tom: So, to sum up, SHIFT gives us a lightweight tool that lets us remove concepts and change styles by steering internal activations during generation while keeping the model untouched.
Jane: And it opens up a lot of doors for how we can control these generative models without having to do the heavy lifting of retraining them every time we want something different.
Lu: The next thing to look at is those ablation studies they did on when you apply this steering; they found that hitting the right part of the generation process is crucial for getting good results.
Meng: That makes sense because if you steer too early or too late, the effect isn't what you want, so knowing *where* to apply it matters a lot for real-world use.
Nina Konovalova, Andrey Kuznetsov, Aibek Alanov
FusionBrain Lab · HSE University
cs.CV
Submitted: 2026-04-10
Updated: 2026-10-31
Comments: The project page is available at \url{https://controlgenai.github.io/SHIFT}
Code: https://github.com/ControlGenAI/SHIFT
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Diffusion models have become leading approaches for highfidelity image generation, and SHIFT proposes a simple but effective and lightweight framework for concept removal in DiT diffusion models via…
Key concepts
- Steering Vectors
- These are directional signals calculated from small prompt changes (p-k vs. p+k). They identify the specific direction in the model's latent space that corresponds to a target concept or concept pair, allowing for precise control over what the model generates.
- Intervention Formula
- The formula à = a + α v defines how steering is applied during image generation. 'a' is the base activation, 'v' is the calculated steering vector, and 'α' (alpha) controls the strength of this manipulation. This allows for controlled redirection of the model's generative process.
- Activation-Level Modulation
- Instead of retraining or editing weights, SHIFT modifies internal activations at inference time. This means it steers the diffusion process by influencing how specific intermediate layers respond to input, enabling concept control without altering the underlying model structure.
Terminology
Summary
Diffusion models have become leading approaches for highfidelity image generation, and SHIFT proposes a simple but effective and lightweight framework for concept removal in DiT diffusion models via targeted manipulation of intermediate activations at inference time. p. 0 We propose SHIFT, a simple but effective and lightweight
framework for concept removal in DiT diffusion models via targeted manipulation of intermediate activations at inference time, inspired by activation steering in large language models. SHIFT learns steering vectors
that are dynamically applied to selected layers and timesteps to suppress unwanted visual concepts while preserving the prompt’s remaining
content and overall image quality.Where the paper says it
Pruning redundant reasoning tokens
SHIFT introduces a steering framework specifically engineered for the Multimodal Diffusion Transformer (MM-DiT) architecture, demonstrating that it identifies temporally invariant steering vectors that remain semantically stable, providing a concept control mechanism that does not require retraining. The authors establish that this activation-level modulation extends to domain stylization and object-specific biasing beyond just concept erasure. p. 1 In contrast to existing methodologies that necessitate complex, timestep-specific interventions, we
demonstrate that SHIFT identifies temporally invariant steering vectors that remain semantically stable, providing a concept control mechanism that does not
require retraining the base model. While our primary evaluation focuses on concept erasure, we establish that
this activation-level modulation extends to domain stylization and object-specific
biasing. While our primary evaluation focuses on concept erasure, we establish that
this activation-level modulation extends to domain stylization and object-specific
biasing. Importantly, SHIFT is positioned as a generation control framework
rather than a traditional image-editing tool; it focuses on the global redirection
of the model’s generative manifold rather than the preservation of spatial layouts
or background consistency from a specific input image.Where the paper says it
Where the paper says it
How the training signal is built
The construction of steering vectors involves collecting a small paired dataset of prompt pairs (p−k, p+k), where p+k adds the target concept to an otherwise neutral prompt. The process of dataset construction is illustrated in Fig. p. 5 Running the model on these prompts, we record
activations at the chosen locations and obtain paired samples (a
−
k
, a+
k
). The process of dataset construction is illustrated in Fig.Where the paper says it
-
For SVM-based estimation, they build a labeled dataset D and train a linear SVM on it to define the steering direction as the normalized normal vector of the separating hyperplane. p. 5
Where the paper says it
For SVM-based estimation, we first pool token activations: \bar {a}_{k}^{+} = \frac {1}{T}\sum _{t=1}^{T} a_{k,t}^{+}, \qquad \bar {a}_{k}^{-} = \frac {1}{T}\sum _{t=1}^{T} a_{k,t}^{-}, (5) where T is the number of tokens. Then we build a labeled dataset \mathcal {D} = \{(\bar {a}_k^{+},1)\}_{k=1}^{n} \cup \{(\bar {a}_k^{-},0)\}_{k=1}^{n}, (6) and train a linear SVM on D: \mathrm {SVM}(\bar {a}_k) = p_{\mathrm {cls}}.
-
The second estimator is the mean-difference direction, defined as vdiff = (1/n) sum k (a k+ - a k-). p. 6
Where the paper says it
The second estimator is the mean-difference direction, v_{\mathrm {diff}} = \frac {1}{n}\sum _{k=1}^{n}\left (a_k^{+}-a_k^{-}\right ), (9) with optional token averaging for sequence activations to reduce sensitivity to prompt length. We further normalize vdiff along the channel dimension to control steering magnitude.
Where the gains come from
The application of steering during inference time is defined by the intervention formula à = a + α v, where α is the steering strength. For diffusion-transformer steering, they regularize the steering strength using a lightweight classifier (SVM) signal to modulate the base steering strength γ as alpha diff = γ η cls. p. 6 For diffusion-transformer steering, we
further regularize the steering strength using a lightweight classifier (SVM) signal
that reflects whether the target concept is still present in the current activations
(Fig. Specifically, we convert the target-class score pcls into a nonnegative
scaling factor
\eta _{\mathrm {cls}} = \operatorname {clip}\! We then modulate the base steering strength γ as
\alpha _{\mathrm {diff}} = \gamma \,\eta _{\mathrm {cls}}. (14)
This mechanism suppresses steering when activations are far from the target
class (small pcls) and amplifies steering when target-concept evidence remains
strong (large pcls), thereby reducing over-steering in weakly separated layers or
timesteps.Where the paper says it
Where the paper says it
The experiments demonstrate that SHIFT clearly outperforms all baselines on safety-related metrics, achieving more than 3× and 4× stronger suppression at different steering strengths. For concrete concept erasure, SHIFT shows superior performance compared to UCE, CA, and ESD across various tasks like Snoopy and Mickey. p. 8 Following the protocol established by SPM [20], we assess concrete concept removal using 80 fixed prompts and 9 different seeds. Specifically,
we evaluate erasure of the Snoopy concept while preserving five related concepts: Mickey, SpongeBob, Pikachu, dog, and legislator. 1[schnell]
activations. As shown in Table 2, our
method remains effective for the nondistilled model even when the steering vector is transferred from the distilled
model.Where the paper says it
Where the paper says it
The ablation studies reveal that steering is most effective during the early stages of the diffusion trajectory, whereas applying steering in the latter half of the generation process yields insufficient concept suppression. Restricting steering to a subset of blocks can improve quality, and restricting it to a subset of blocks is substantially more effective than applying it across all blocks for tasks like adding hat
. The results confirm that using only text-encoder steering or only diffusion-transformer steering is insufficient for high-quality erasure, and increased text steering improves removal but risks over-erasure
which a classifier effectively mitigates.
The method’s versatility is shown across several tasks, including concept addition, concept switching (e.g., cat → dog), and domain shifting to specific styles like sketch or cyberpunk. p. 1 While recent work like
CASteer [6] has sought to adapt these principles to diffusion models, such implementations remain fundamentally dependent to the cross-attention mechanisms
usually presented in UNet-based architectures. However, the field is currently shifting toward Diffusion Transformers (DiTs) [22]. While recent work like
CASteer [6] has sought to adapt these principles to diffusion models, such implementations remain fundamentally dependent to the cross-attention mechanisms
usually presented in UNet-based architectures. However, the field is currently shifting toward Diffusion Transformers (DiTs) [22]. SHIFT 3
indicating that semantic concepts are encoded as stable, global directions in
the latent manifolds of DiTs. – Cross-distillation vector transfer: we investigate the transferability of
steering vectors across distillation boundaries (e. (14)
This mechanism suppresses steering when activations are far from the target
class (small pcls) and amplifies steering when target-concept evidence remains
strong (large pcls), thereby reducing over-steering in weakly separated layers or
timesteps. Overall, this provides a lightweight, inference-time control mechanism that does
not modify model weights and can be applied on demand for concept removal,
domain/style shifts, and object-level interventions.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The qualitative results demonstrate that the steering method is effective not only for erasure tasks but also for shifting representations toward specific regions of the manifold containing the target concept, successfully inserting objects into an image even when they are traditionally difficult to integrate naturally. The ablation on timesteps proves that early steps are crucially important for diffusion steering. p. 1 – Spatial and temporal dynamics analysis: we conduct an extensive ablation study on the spatial and temporal aspects of steering, uncovering a
remarkable temporal consistency in DiT activation spaces. Specifically, we
show that a single, time-independent steering vector can be applied effectively across all diffusion timesteps, streamlining the steering process and – Spatial and temporal dynamics analysis: we conduct an extensive ablation study on the spatial and temporal aspects of steering, uncovering a
remarkable temporal consistency in DiT activation spaces. Specifically, we
show that a single, time-independent steering vector can be applied effectively across all diffusion timesteps, streamlining the steering process and While our primary evaluation focuses on concept erasure, we establish that
this activation-level modulation extends to domain stylization and object-specific
biasing. Importantly, SHIFT is positioned as a generation control framework
rather than a traditional image-editing tool; it focuses on the global redirection
of the model’s generative manifold rather than the preservation of spatial layouts
or background consistency from a specific input image. Importantly, SHIFT is positioned as a generation control framework
rather than a traditional image-editing tool; it focuses on the global redirection
of the model’s generative manifold rather than the preservation of spatial layouts
or background consistency from a specific input image. This distinction allows
SHIFT to achieve robust semantic shifts while maintaining the high-fidelity synthesis inherent to the FLUX backbone. Importantly, SHIFT is positioned as a generation control framework
rather than a traditional image-editing tool; it focuses on the global redirection
of the model’s generative manifold rather than the preservation of spatial layouts
or background consistency from a specific input image. This distinction allows
SHIFT to achieve robust semantic shifts while maintaining the high-fidelity synthesis inherent to the FLUX backbone. 3 Latent space navigation and Diffusion Steering. Diffusion models enable controllable synthesis through guidance signals and
latent-space manipulations, allowing users to influence the generation process
toward desired outcomes. 09213v1 [cs. CV] 10 Apr 2026 4 Konovalova et al. 3 Latent space navigation and Diffusion Steering. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. For modern architectures, EraseAnything [9]
adapts erasure to rectified flow models via LoRA tuning and attention regularization, and Erasing with Adversarial Preservation (EAP) [3] incorporates
adversarial concept identification to preserve unrelated generations during finetuning. However, these methods have limitations on modern diffusion models,
including optimization overhead, reliance on guidance that must be adapted for
distilled models, or dependence on LLM agents. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. com/ControlGenAI/SHIFT
. 1 Introduction
Diffusion models [11, 28] have established a new state-of-the-art in high-fidelity
text-to-image synthesis [23]. One mitigation is dataset filtering [28] or post-generation filtering [26]. However, these approaches do not fully prevent harmful content
and require additional processing. SHIFT: Steering Hidden Intermediates in Flow
Transformers
Nina Konovalova
1
,
2
,
3
, Andrey Kuznetsov
1
,
3
, Aibek Alanov
1
,
2
,
3
1 FusionBrain Lab 2 HSE University 3 AXXX
Abstract. Diffusion models have become leading approaches for highfidelity image generation. This distinction allows
SHIFT to achieve robust semantic shifts while maintaining the high-fidelity synthesis inherent to the FLUX backbone. Our contributions are following:
– Unified steering framework: we introduce SHIFT, the first comprehensive framework for steering both original and distilled DiT-based models
(e. 3 Latent space navigation and Diffusion Steering. Diffusion models enable controllable synthesis through guidance signals and
latent-space manipulations, allowing users to influence the generation process
toward desired outcomes. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. 5 [5] utilize Multimodal Diffusion Transformer (MM-DiT) blocks, where the traditional decoupling of cross-attention
and self-attention is replaced by a unified attention mechanism that processes
text and image tokens in a shared latent space. Consequently, erasing concepts
in DiTs requires a more accurate intervention capable of navigating this integrated representation manifold without compromising the global structure of
the generated image. Subsequent approaches
like Unified Concept Editing (UCE) [8] introduce closed-form edits to attention projections for efficient, training-free erasure, though they may struggle
to fully remove concrete objects. For modern architectures, EraseAnything [9]
adapts erasure to rectified flow models via LoRA tuning and attention regularization, and Erasing with Adversarial Preservation (EAP) [3] incorporates
adversarial concept identification to preserve unrelated generations during finetuning. The corresponding steering direction is defined by the normalized normal vector of the
separating hyperplane,
v_{\mathrm {svm}} = \frac {w}{\lVert w \rVert } \in \mathbb {R}^{c}, (8)
where w denotes the SVM weight vector and c is the channel dimension. The second estimator is the mean-difference direction,
v_{\mathrm {diff}} = \frac {1}{n}\sum _{k=1}^{n}\left (a_k^{+}-a_k^{-}\right ), (9)
with optional token averaging for sequence activations to reduce sensitivity to
prompt length. Early works focused on semantic navigation in latent
representations, such as interpolating between latent codes to blend concepts,
and guidance-based conditioning, including classifier guidance [4] where an external classifier steers the denoising process, classifier-free guidance [12] that
amplifies conditioning signals without additional models or more complicated
approaches introducing semantic guidance [2]. More recent methods have adapted steering techniques from large language
models by intervening directly in diffusion model activations, offering finergrained control without retraining. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. In parallel, activation steering has emerged as a lightweight yet potent control mechanism for Large Language Models (LLMs) [1,19,33]. Frameworks such
as Representation Engineering [37] demonstrate that high-level semantic behavior can be modulated by injecting linear directions into the latent representation
space at inference time, without any weights modification. Diffusion models enable controllable synthesis through guidance signals and
latent-space manipulations, allowing users to influence the generation process
toward desired outcomes. Early works focused on semantic navigation in latent
representations, such as interpolating between latent codes to blend concepts,
and guidance-based conditioning, including classifier guidance [4] where an external classifier steers the denoising process, classifier-free guidance [12] that
amplifies conditioning signals without additional models or more complicated
approaches introducing semantic guidance [2]. 2 Activation steering in Large Language Models. Steering in large language models (LLMs) has emerged as a practical paradigm
for controlling behavior. com/ControlGenAI/SHIFT
. 1 Introduction
Diffusion models [11, 28] have established a new state-of-the-art in high-fidelity
text-to-image synthesis [23]. 1 Concept erasure, safety, and post-training approaches. Recent state-of-the-art text-to-image diffusion models trained on large, imperfectly filtered datasets such as LAION [30] can generate inappropriate or copyrighted content. Recent state-of-the-art text-to-image diffusion models trained on large, imperfectly filtered datasets such as LAION [30] can generate inappropriate or copyrighted content. One mitigation is dataset filtering [28] or post-generation filtering [26]. One mitigation is dataset filtering [28] or post-generation filtering [26]. However, these approaches do not fully prevent harmful content
and require additional processing. More recent methods have adapted steering techniques from large language
models by intervening directly in diffusion model activations, offering finergrained control without retraining. In UNet-based latent diffusion models, a
wide-spread control mechanism is cross-attention manipulation: for instance,
CASteer [6] constructs steering vectors for cross-attention layers using contrastive prompt pairs (e. In parallel, activation steering has emerged as a lightweight yet potent control mechanism for Large Language Models (LLMs) [1,19,33]. Frameworks such
as Representation Engineering [37] demonstrate that high-level semantic behavior can be modulated by injecting linear directions into the latent representation
space at inference time, without any weights modification. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. In parallel, activation steering has emerged as a lightweight yet potent control mechanism for Large Language Models (LLMs) [1,19,33]. Frameworks such
as Representation Engineering [37] demonstrate that high-level semantic behavior can be modulated by injecting linear directions into the latent representation
space at inference time, without any weights modification. We demonstrate that SHIFT
provides effective and flexible control over DiT generation across diverse
prompts and targets without time-consuming retraining. The code is
available at https://github. This has motivated the development of Concept Erasure (CE): the task of reliably suppressing specific semantic concepts within a model’s generative manifold. Existing erasure strategies for text-to-image diffusion models mostly rely on
model optimization. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. More recent methods have adapted steering techniques from large language
models by intervening directly in diffusion model activations, offering finergrained control without retraining. In UNet-based latent diffusion models, a
wide-spread control mechanism is cross-attention manipulation: for instance,
CASteer [6] constructs steering vectors for cross-attention layers using contrastive prompt pairs (e. 2 Related works
2. 1 Concept erasure, safety, and post-training approaches. Many works therefore focus on concept erasure (CE) while preserving overall generation quality. Methods such as ESD [7]
employ fine-tuning to unlearn concepts using negative guidance from a frozen
teacher model, while Concept Ablation (CA) [15] optimizes cross-attention layers to redirect target concepts toward neutral anchors. 1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization
alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form
arXiv:2604. This has motivated the development of Concept Erasure (CE): the task of reliably suppressing specific semantic concepts within a model’s generative manifold. Existing erasure strategies for text-to-image diffusion models mostly rely on
model optimization.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The ablation on steering strength and classifier usage shows that the classifier is critical both for stable steering and for preserving overall image quality. Table 7 suggests that using only text-encoder steering or only diffusion-transformer steering is insufficient for high-quality erasure, while increased text steering improves removal but risks over-erasure
where nontarget concepts lose semantic fidelity. The ablation on timesteps proves that early steps are crucially important for diffusion steering.
The results on concrete concept erasure show that SHIFT clearly outperforms all baselines on safety-related metrics, achieving more than 3× and 4× stronger suppression at different steering strengths. For concrete object removal, SHIFT performs much better than the CA method when erasing concepts like Snoopy. p. 8 Following the protocol established by SPM [20], we assess concrete concept removal using 80 fixed prompts and 9 different seeds. Specifically,
we evaluate erasure of the Snoopy concept while preserving five related concepts: Mickey, SpongeBob, Pikachu, dog, and legislator. 3 for Flux. 1[dev], showing that our method can erase undesirable concepts while preserving overall image quality. For Flux, we find that steering both the text encoder pooled embedding and
the diffusion transformer part enhances controllability and semantic alignment,
whereas steering text encoder alone yields suboptimal results. 2 Steering vector construction
We construct a separate steering vector for each target concept X and for each
steering location (text encoder pooled embedding and diffusion transformer tokens).Where the paper says it
Where the paper says it
Where the paper says it
The method’s versatility is shown across several tasks, including concept addition, concept switching (e.g., cat → dog), and domain shifting to specific styles like sketch or cyberpunk. The ablation on number of steered DiT blocks shows that restricting steering to a subset of blocks is substantially more effective than applying it across all blocks for adding We additionally include small local objects removal examples (e. , hats and glasses) to provide visually interpretable results. While our primary evaluation focuses on concept erasure, we establish that
this activation-level modulation extends to domain stylization and object-specific
biasing. Importantly, SHIFT is positioned as a generation control framework
rather than a traditional image-editing tool; it focuses on the global redirection
of the model’s generative manifold rather than the preservation of spatial layouts
or background consistency from a specific input image.hat
. The results confirm that the approach successfully switches a wide range of concepts, including global attributes (age) and small objects (apple vs. p. 7Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
- Concept Erasure in Diffusion Transformers (DiTs) via SHIFT
The AI system can reliably suppress specific semantic concepts within a model’s generative manifold by learning steering vectors that are dynamically applied to selected layers and timesteps to suppress unwanted visual concepts while preserving the prompt’s remaining content.
This allows for concept removal without retraining the base model.
- Cross-Domain Concept Transfer
The system can transfer steering vectors across distillation boundaries, validating their efficacy in resource-constrained settings by demonstrating robust performance in resource-constrained settings
when moving from a distilled model like Flux.1[schnell] to a full model like Flux.1[dev]."
- Unified Steering Control Framework
SHIFT provides "the first comprehensive framework for steering both original and distilled DiT-based models (e.g., Flux.1[dev] and Flux.1[schnell]) through latent activation shifts, enabling efficient and scalable concept manipulation without retraining. This enables a single mechanism for
original and distilled models."
- Multi-Task Control Capabilities
The improved AI system can perform diverse generative control tasks beyond simple erasure, including domain stylization,
object-specific biasing,
and complex tasks like concept addition
(e.g., adding glasses or hats) and concept switching
(e.g., cat to dog)."
- Fine-Grained Spatial and Temporal Steering
The system can apply steering effectively across the generation process by demonstrating that a single, time-independent steering vector can be applied effectively across all diffusion timesteps,
while ablation studies show that steering is most effective during the early stages of the diffusion trajectory
for certain tasks.
- Adaptive Steering Strength Regulation
The system incorporates a mechanism to adjust steering strength dynamically, using a classifier signal such as a lightweight classifier (SVM) signal that reflects whether the target concept is still present in the current activations,
which suppresses steering when activations are far from the target class and amplifies steering when target-concept evidence remains strong.
- Concept Switching and Addition
The AI can successfully perform concept switching, replacing one target concept with another (e.g., cat → dog or man → woman
), and add new concepts, such as an apple,
even when they are traditionally difficult to integrate naturally.
Abstract
Diffusion models have become the leading approach for high-fidelity image generation, with recent DiT-based models achieving strong prompt adherence and high visual quality. However, removing unwanted concepts from these models typically requires expensive gradient-based fine-tuning, which is particularly prohibitive for multi-billion-parameter Diffusion Transformers and incompatible with distilled variants. We propose SHIFT, a simple model-weight-update-free inference-time framework for concept suppression in DiT models via targeted manipulation of intermediate activations at inference time, inspired by activation steering in LLMs. SHIFT learns steering vectors from contrastive prompts and applies them dynamically. We demonstrate that a single time-independent steering vector generalizes across all denoising timesteps and that vectors computed on a distilled model transfer effectively to its non-distilled counterpart without recomputation. SHIFT achieves strong concept suppression while preserving prompt fidelity and image quality. The same mechanism also enables flexible control -- adding, removing, or changing objects and styles without retraining. SHIFT targets hosted inference settings: it does not remove the concept from the model weights and is therefore complementary to weight-level unlearning.
Sources
- The Stable Artist: Steering Semantics in Diffusion Latent Space
- Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation
- Inspecting and Editing Knowledge Representations in Language Models
- Classifier-Free Diffusion Guidance
- Editing Models with Task Arithmetic
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- Red-Teaming the Stable Diffusion Safety Filter
- Controlling Language and Diffusion Models by Transporting Activations
- Steering Language Models With Activation Engineering
- Minimalist Concept Erasure in Generative Models
- Fine-Tuning Language Models from Human Preferences
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs