SHIFT: Steering Hidden Intermediates in Flow Transformers

arXiv:2604.09213 · cs.CV · Submitted 2026-04-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "SHIFT: Steering Hidden Intermediates in Flow Transformers".

Jane: Here's the gist of the paper. Diffusion models have become leading approaches for highfidelity image generation, and SHIFT proposes a simple but effective and lightweight framework for concept removal in DiT diffusion models via targeted manipulation of intermediate activations at inference time.

Tom: Pruning redundant reasoning tokens SHIFT introduces a steering framework specifically engineered for the Multimodal Diffusion Transformer (MM-DiT) architecture, demonstrating that it identifies temporally invariant steering vectors that remain semantically stable, providing a concept control mechanism that does not require retraining.

Jane: The authors establish that this activation-level modulation extends to domain stylization and object-specific biasing beyond just concept erasure.

Tom: SHIFT is positioned as a generation control framework rather than a traditional image-editing tool, focusing on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image.

Paper summary: Tom: So, we're looking at this paper called "SHIFT: Steering Hidden Intermediates in Flow Transformers," and the main idea is that they want a simple way to remove specific concepts from these high-quality image generation models, especially the DiT ones.

Jane: That sounds like something really practical for people who want to use these tools but need more control over what they're getting back, right? What exactly is SHIFT proposing at first glance?

Tom: Well, it suggests using a steering framework that manipulates intermediate activations during the inference time of these DiT diffusion models instead of retraining the entire model. They claim this method can suppress unwanted visual concepts while keeping the prompt's rest and overall image quality intact.

Lu: What's interesting is that they aren't just focused on simple erasure; they suggest this same mechanism can also shift generations into a different style domain or bias samples toward adding or changing specific objects, which is pretty broad.

Meng: That sounds ambitious, but from an engineering standpoint, the biggest question for me is how much effort this takes. Does it require any kind of retraining or extensive fine-tuning to get these steering vectors ready?

Lalam: The paper states that SHIFT learns steering vectors dynamically applied to selected layers and timesteps to suppress unwanted visual concepts while preserving the prompt’s rest and overall image quality <ref:2604.09213#pg1>.

Tom: Exactly, so it avoids retraining, which is a big deal for deployment because those models are already huge. But how do they figure out what these steering vectors should be in the first place?

Jane: That brings us to how they build the training signal for these vectors; it sounds like they create a small paired dataset of prompt pairs, where one prompt has the target concept added to a neutral one.

Lu: After collecting that data, they use two main ways to estimate these steering vectors: first, they build a labeled dataset and train a linear support vector machine on it to define the steering direction as the normalized normal vector of the separating hyperplane.

Tom: And there's another way, which is this mean-difference direction calculation where they take the sum over all prompts and subtract one from the other for each concept pair to get that vdiff value.

Paper summary: Meng: So you’re essentially creating a mathematical direction in the latent space based on those prompt differences, and then using a machine learning classifier to fine-tune how strong that steering push should be during generation?

Lalam: Yes, the application of steering during inference time is defined by an intervention formula where alpha is the steering strength and alpha equals gamma times eta times classifier signal for diffusion transformer steering <ref:2604.09213#pg2>.

Jane: That sounds like they have a lightweight way to regularize that strength using a classifier, which helps suppress the steering when activations are far away from the target class and amplifies it when there’s strong evidence of the target concept remaining.

Tom: And what they show is that this method clearly outperforms all their baselines on safety-related metrics, achieving more than three times and four times stronger suppression at different steering strengths.

Lu: For concrete object removal, SHIFT shows superior performance compared to methods like UCE, CA, and ESD across various tasks like Snoopy and Mickey <ref:2604.09213#pg3>.

Meng: That's good to hear on performance metrics, but what about the trade-offs? What happens when you push that steering strength too far? Does it just make the image look weird or ruin the whole picture?

Lalam: The ablation studies reveal that steering is most effective during the early stages of the diffusion trajectory, whereas applying steering in the latter half of generation yields insufficient concept suppression <ref:2604.09213#pg3>.

Tom: So they're saying you gotta hit it early in the process for this to work well, which makes sense if you're trying to remove something that was introduced right at the start. But what about the overall quality if you restrict where you apply this steering?

Lu: They found that restricting steering to a subset of blocks can improve quality, and restricting it to a subset of blocks is substantially more effective than applying it across all blocks for tasks like adding a hat <ref:2604.09213#pg3>.

Jane: That’s interesting because the paper also points out that using only text-encoder steering or only diffusion-transformer steering isn't enough on its own for high-quality erasure, though increased text steering improves removal but risks over-erasure <ref:2604.09213#pg2>.

Tom: And they showed that token-wise difference in the steering vector parameterization yields stronger target concept suppression, which is a concrete detail about how they tune it.

Paper summary: Meng: So so, if I’m deploying this to production, I need to know how scalable this is. Does running these extra steps for steering add significant latency compared to just running a standard inference pass?

Lalam: The authors conclude that SHIFT provides a lightweight, inference-time control mechanism that does not modify model weights and can be applied on demand for concept removal, domain/style shifts, and object-level interventions <ref:2604.09213#pg1>.

Jane: So, the main point is that this approach offers versatile control—erasure, style shifting like sketch or cyberpunk—and it seems to transfer these steering vectors across distillation boundaries when moving between models <ref:2604.09213#pg3>.

Tom: And they confirm that using only text steering or only diffusion-transformer steering is insufficient for high-quality erasure, but increasing the text steering helps, which then a classifier mitigates the risk of over-erasure <ref:2604.09213#pg2>.

Lu: The results show this method can successfully switch a wide range of concepts, including global attributes like age and small objects like an apple versus a banana <ref:2604.09213#pg3>.

Meng: I'm still worried about the complexity of setting up that initial paired dataset for concept removal; does that data collection process itself become a bottleneck when you want to apply it to many different models or prompts?

Lalam: The paper shows the construction of steering vectors involves collecting a small paired dataset of prompt pairs where one prompt adds the target concept to an otherwise neutral prompt <ref:2604.09213#pg1>.

Jane: So, even if you have all the best math in the world, you still need that initial setup phase to generate those specific training signals before you can use the steering mechanism itself.

Tom: The title of this work, "SHIFT: Steering Hidden Intermediates in Flow Transformers," really captures how they are focusing on these intermediate activations rather than just tweaking the final output layer.

Lu: It seems like they’ve established that semantic concepts are encoded as stable, global directions in the latent manifolds of DiTs <ref:2604.09213#pg3>.

Meng: So what does this mean for the average person who just uses an image generator? Do I need to know about flow transformers or diffusion timesteps to use this?

Jane: It means that for people who want precise control over what they generate, you can now target specific parts of the generation process without having to retrain a massive model every time.

Lalam: Overall, SHIFT achieves competitive performance while maintaining high computational efficiency <ref:2604.09213#pg1>.

Conclusion: Tom: So we’re looking at this paper called "SHIFT: Steering Hidden Intermediates in Flow Transformers" and it's about using hidden parts of the AI model to control what it makes.

Jane: It’s about taking these big diffusion models, like DiT, and instead of changing the whole thing, we just nudge specific internal signals at inference time to get rid of a concept.

Lu: The core idea is finding those stable directions in the model's internal math that control things like concept removal or even style shifts.

Meng: So it’s about steering, right? Not editing an image after it’s done, but steering the generation process itself.

Lalam: Exactly. They create a mechanism that lets you apply this steering signal during the generation steps, which is when the image is actually being built pixel by pixel or token by token.

Tom: The authors are focused on making this framework light and efficient because we’re shipping these models everywhere, and they claim it doesn't require retraining.

Jane: It seems like they’re showing that you can achieve concept control without having to touch the model's massive weights at all, which is a huge win for deployment.

Lu: They use two different methods to figure out these steering directions, one based on building a dataset and training a linear support vector machine, and another using this mean difference direction calculation.

Meng: Which means they’re not just guessing; they have some concrete math to define the direction they want the AI to move in.

Lalam: Right. And then during inference, you use this steering strength, which is controlled by a classifier signal that helps manage how much you push that steering around.

Tom: So, it’s not just one setting; there’s a feedback loop managing the strength of that nudge to keep the result high quality.

Jane: It shows they can do more than just erase things; they can also use this steering for things like adding new objects or shifting the style of an image completely.

Lu: The versatility is what’s interesting here, because these same vectors seem to work across different models and even between different types of AI distillation.

Meng: That transferability across distillation boundaries is important because it suggests this method might be useful for adapting models we don't directly train on.

Lalam: It confirms that the concept steering direction is a general thing, not just specific to one model architecture or one task.

Tom: So, to sum up, SHIFT gives us a lightweight tool that lets us remove concepts and change styles by steering internal activations during generation while keeping the model untouched.

Jane: And it opens up a lot of doors for how we can control these generative models without having to do the heavy lifting of retraining them every time we want something different.

Lu: The next thing to look at is those ablation studies they did on when you apply this steering; they found that hitting the right part of the generation process is crucial for getting good results.

Meng: That makes sense because if you steer too early or too late, the effect isn't what you want, so knowing *where* to apply it matters a lot for real-world use.

Nina Konovalova, Andrey Kuznetsov, Aibek Alanov

FusionBrain Lab · HSE University

cs.CV

Submitted: 2026-04-10

Updated: 2026-10-31

Comments: The project page is available at \url{https://controlgenai.github.io/SHIFT}

Code: https://github.com/ControlGenAI/SHIFT

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Diffusion models have become leading approaches for highfidelity image generation, and SHIFT proposes a simple but effective and lightweight framework for concept removal in DiT diffusion models via…

Key concepts

Steering Vectors
These are directional signals calculated from small prompt changes (p-k vs. p+k). They identify the specific direction in the model's latent space that corresponds to a target concept or concept pair, allowing for precise control over what the model generates.
Intervention Formula
The formula à = a + α v defines how steering is applied during image generation. 'a' is the base activation, 'v' is the calculated steering vector, and 'α' (alpha) controls the strength of this manipulation. This allows for controlled redirection of the model's generative process.
Activation-Level Modulation
Instead of retraining or editing weights, SHIFT modifies internal activations at inference time. This means it steers the diffusion process by influencing how specific intermediate layers respond to input, enabling concept control without altering the underlying model structure.

Terminology

Summary

Diffusion models have become leading approaches for highfidelity image generation, and SHIFT proposes a simple but effective and lightweight framework for concept removal in DiT diffusion models via targeted manipulation of intermediate activations at inference time. p. 0

Where the paper says it

We propose SHIFT, a simple but effective and lightweight framework for concept removal in DiT diffusion models via targeted manipulation of intermediate activations at inference time, inspired by activation steering in large language models. SHIFT learns steering vectors that are dynamically applied to selected layers and timesteps to suppress unwanted visual concepts while preserving the prompt’s remaining content and overall image quality.

Pruning redundant reasoning tokens

SHIFT introduces a steering framework specifically engineered for the Multimodal Diffusion Transformer (MM-DiT) architecture, demonstrating that it identifies temporally invariant steering vectors that remain semantically stable, providing a concept control mechanism that does not require retraining. The authors establish that this activation-level modulation extends to domain stylization and object-specific biasing beyond just concept erasure. p. 1

Where the paper says it

In contrast to existing methodologies that necessitate complex, timestep-specific interventions, we demonstrate that SHIFT identifies temporally invariant steering vectors that remain semantically stable, providing a concept control mechanism that does not require retraining the base model. While our primary evaluation focuses on concept erasure, we establish that this activation-level modulation extends to domain stylization and object-specific biasing.

SHIFT is positioned as a generation control framework rather than a traditional image-editing tool, focusing on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image. p. 1
Where the paper says it

While our primary evaluation focuses on concept erasure, we establish that this activation-level modulation extends to domain stylization and object-specific biasing. Importantly, SHIFT is positioned as a generation control framework rather than a traditional image-editing tool; it focuses on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image.

How the training signal is built

The construction of steering vectors involves collecting a small paired dataset of prompt pairs (p−k, p+k), where p+k adds the target concept to an otherwise neutral prompt. The process of dataset construction is illustrated in Fig. p. 5

Where the paper says it

Running the model on these prompts, we record activations at the chosen locations and obtain paired samples (a − k , a+ k ). The process of dataset construction is illustrated in Fig.

2 (1). After dataset collection, steering vectors are estimated using two primary estimators for the diffusion transformer steering vector:

  1. For SVM-based estimation, they build a labeled dataset D and train a linear SVM on it to define the steering direction as the normalized normal vector of the separating hyperplane. p. 5

    Where the paper says it

    For SVM-based estimation, we first pool token activations: \bar {a}_{k}^{+} = \frac {1}{T}\sum _{t=1}^{T} a_{k,t}^{+}, \qquad \bar {a}_{k}^{-} = \frac {1}{T}\sum _{t=1}^{T} a_{k,t}^{-}, (5) where T is the number of tokens. Then we build a labeled dataset \mathcal {D} = \{(\bar {a}_k^{+},1)\}_{k=1}^{n} \cup \{(\bar {a}_k^{-},0)\}_{k=1}^{n}, (6) and train a linear SVM on D: \mathrm {SVM}(\bar {a}_k) = p_{\mathrm {cls}}.

  2. The second estimator is the mean-difference direction, defined as vdiff = (1/n) sum k (a k+ - a k-). p. 6

    Where the paper says it

    The second estimator is the mean-difference direction, v_{\mathrm {diff}} = \frac {1}{n}\sum _{k=1}^{n}\left (a_k^{+}-a_k^{-}\right ), (9) with optional token averaging for sequence activations to reduce sensitivity to prompt length. We further normalize vdiff along the channel dimension to control steering magnitude.

Where the gains come from

The application of steering during inference time is defined by the intervention formula à = a + α v, where α is the steering strength. For diffusion-transformer steering, they regularize the steering strength using a lightweight classifier (SVM) signal to modulate the base steering strength γ as alpha diff = γ η cls. p. 6

Where the paper says it

For diffusion-transformer steering, we further regularize the steering strength using a lightweight classifier (SVM) signal that reflects whether the target concept is still present in the current activations (Fig. Specifically, we convert the target-class score pcls into a nonnegative scaling factor \eta _{\mathrm {cls}} = \operatorname {clip}\!

This mechanism suppresses steering when activations are far from the target class and amplifies it when target-concept evidence remains strong, thereby reducing over-steering. p. 7
Where the paper says it

We then modulate the base steering strength γ as \alpha _{\mathrm {diff}} = \gamma \,\eta _{\mathrm {cls}}. (14) This mechanism suppresses steering when activations are far from the target class (small pcls) and amplifies steering when target-concept evidence remains strong (large pcls), thereby reducing over-steering in weakly separated layers or timesteps.

The experiments demonstrate that SHIFT clearly outperforms all baselines on safety-related metrics, achieving more than 3× and 4× stronger suppression at different steering strengths. For concrete concept erasure, SHIFT shows superior performance compared to UCE, CA, and ESD across various tasks like Snoopy and Mickey. p. 8

Where the paper says it

Following the protocol established by SPM [20], we assess concrete concept removal using 80 fixed prompts and 9 different seeds. Specifically, we evaluate erasure of the Snoopy concept while preserving five related concepts: Mickey, SpongeBob, Pikachu, dog, and legislator.

Furthermore, the method can successfully insert objects into an image even when they are traditionally difficult to integrate naturally. p. 8
Where the paper says it

1[schnell] activations. As shown in Table 2, our method remains effective for the nondistilled model even when the steering vector is transferred from the distilled model.

The ablation studies reveal that steering is most effective during the early stages of the diffusion trajectory, whereas applying steering in the latter half of the generation process yields insufficient concept suppression. Restricting steering to a subset of blocks can improve quality, and restricting it to a subset of blocks is substantially more effective than applying it across all blocks for tasks like adding hat. The results confirm that using only text-encoder steering or only diffusion-transformer steering is insufficient for high-quality erasure, and increased text steering improves removal but risks over-erasure which a classifier effectively mitigates.

The method’s versatility is shown across several tasks, including concept addition, concept switching (e.g., cat → dog), and domain shifting to specific styles like sketch or cyberpunk. p. 1

Where the paper says it

While recent work like CASteer [6] has sought to adapt these principles to diffusion models, such implementations remain fundamentally dependent to the cross-attention mechanisms usually presented in UNet-based architectures. However, the field is currently shifting toward Diffusion Transformers (DiTs) [22].

p. 1
Where the paper says it

While recent work like CASteer [6] has sought to adapt these principles to diffusion models, such implementations remain fundamentally dependent to the cross-attention mechanisms usually presented in UNet-based architectures. However, the field is currently shifting toward Diffusion Transformers (DiTs) [22].

The transferability of steering vectors across distillation boundaries is confirmed, as the vector from the distilled model transfers effectively to the non-distilled model for both add and switch tasks. p. 2
Where the paper says it

SHIFT 3 indicating that semantic concepts are encoded as stable, global directions in the latent manifolds of DiTs. – Cross-distillation vector transfer: we investigate the transferability of steering vectors across distillation boundaries (e.

Overall, SHIFT provides a lightweight, inference-time control mechanism that does not modify model weights and can be applied on demand for concept removal, domain/style shifts, and object-level interventions. p. 7
Where the paper says it

(14) This mechanism suppresses steering when activations are far from the target class (small pcls) and amplifies steering when target-concept evidence remains strong (large pcls), thereby reducing over-steering in weakly separated layers or timesteps. Overall, this provides a lightweight, inference-time control mechanism that does not modify model weights and can be applied on demand for concept removal, domain/style shifts, and object-level interventions.

The qualitative results demonstrate that the steering method is effective not only for erasure tasks but also for shifting representations toward specific regions of the manifold containing the target concept, successfully inserting objects into an image even when they are traditionally difficult to integrate naturally. The ablation on timesteps proves that early steps are crucially important for diffusion steering. p. 1

Where the paper says it

– Spatial and temporal dynamics analysis: we conduct an extensive ablation study on the spatial and temporal aspects of steering, uncovering a remarkable temporal consistency in DiT activation spaces. Specifically, we show that a single, time-independent steering vector can be applied effectively across all diffusion timesteps, streamlining the steering process and

p. 1
Where the paper says it

– Spatial and temporal dynamics analysis: we conduct an extensive ablation study on the spatial and temporal aspects of steering, uncovering a remarkable temporal consistency in DiT activation spaces. Specifically, we show that a single, time-independent steering vector can be applied effectively across all diffusion timesteps, streamlining the steering process and

These results establish concept steering as a promising and generalizable direction for controllable object-level editing in diffusion models. p. 1
Where the paper says it

While our primary evaluation focuses on concept erasure, we establish that this activation-level modulation extends to domain stylization and object-specific biasing. Importantly, SHIFT is positioned as a generation control framework rather than a traditional image-editing tool; it focuses on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image.

The paper concludes that SHIFT achieves competitive performance while maintaining high computational efficiency. p. 1
Where the paper says it

Importantly, SHIFT is positioned as a generation control framework rather than a traditional image-editing tool; it focuses on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image. This distinction allows SHIFT to achieve robust semantic shifts while maintaining the high-fidelity synthesis inherent to the FLUX backbone.

p. 1
Where the paper says it

Importantly, SHIFT is positioned as a generation control framework rather than a traditional image-editing tool; it focuses on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image. This distinction allows SHIFT to achieve robust semantic shifts while maintaining the high-fidelity synthesis inherent to the FLUX backbone.

The references listed are Beaglehole, D., Radhakrishnan, A., Boix-Adsera, E., Belkin, M.: Toward universal steering and monitoring of ai models. p. 3
Where the paper says it

3 Latent space navigation and Diffusion Steering. Diffusion models enable controllable synthesis through guidance signals and latent-space manipulations, allowing users to influence the generation process toward desired outcomes.

Science 391(6787), 787–792 (2026), Brack, M. p. 0
Where the paper says it

09213v1 [cs. CV] 10 Apr 2026

, Schramowski, P., Friedrich, F., Hintersdorf, D., Kersting, K.: The stable artist: Steering semantics in diffusion latent space. p. 3
Where the paper says it

4 Konovalova et al. 3 Latent space navigation and Diffusion Steering.

arXiv preprint arXiv:2212. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

06013 (2022), Bui, A., Vuong, L., Doan, K., Le, T., Montague, P., Abraham, T., Phung, D.: Erasing undesirable concepts in diffusion models with adversarial preservation. p. 2
Where the paper says it

For modern architectures, EraseAnything [9] adapts erasure to rectified flow models via LoRA tuning and attention regularization, and Erasing with Adversarial Preservation (EAP) [3] incorporates adversarial concept identification to preserve unrelated generations during finetuning. However, these methods have limitations on modern diffusion models, including optimization overhead, reliance on guidance that must be adapted for distilled models, or dependence on LLM agents.

arXiv preprint arXiv:2410. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

15618 (2024), Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. p. 0
Where the paper says it

com/ControlGenAI/SHIFT . 1 Introduction Diffusion models [11, 28] have established a new state-of-the-art in high-fidelity text-to-image synthesis [23].

Advances in neural information processing systems 34, 8780–8794 (2021), Esser, P. p. 2
Where the paper says it

One mitigation is dataset filtering [28] or post-generation filtering [26]. However, these approaches do not fully prevent harmful content and require additional processing.

, Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F.: Scaling rectified flow transformers for high-resolution image synthesis. p. 0
Where the paper says it

SHIFT: Steering Hidden Intermediates in Flow Transformers Nina Konovalova 1 , 2 , 3 , Andrey Kuznetsov 1 , 3 , Aibek Alanov 1 , 2 , 3 1 FusionBrain Lab 2 HSE University 3 AXXX Abstract. Diffusion models have become leading approaches for highfidelity image generation.

In: Forty-first international conference on machine learning (2024), Gaintseva, T. p. 1
Where the paper says it

This distinction allows SHIFT to achieve robust semantic shifts while maintaining the high-fidelity synthesis inherent to the FLUX backbone. Our contributions are following: – Unified steering framework: we introduce SHIFT, the first comprehensive framework for steering both original and distilled DiT-based models (e.

, Ma, C., Liu, Z., Benning, M., Slabaugh, G., Deng, J., Elezi, I.: Casteer: Steering diffusion models for controllable generation. p. 3
Where the paper says it

3 Latent space navigation and Diffusion Steering. Diffusion models enable controllable synthesis through guidance signals and latent-space manipulations, allowing users to influence the generation process toward desired outcomes.

arXiv e-prints pp. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

arXiv–2503 (2025), Gandikota, R. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

, Materzynska, J., Fiotto-Kaufman, J., Bau, D.: Erasing concepts from diffusion models. p. 1
Where the paper says it

5 [5] utilize Multimodal Diffusion Transformer (MM-DiT) blocks, where the traditional decoupling of cross-attention and self-attention is replaced by a unified attention mechanism that processes text and image tokens in a shared latent space. Consequently, erasing concepts in DiTs requires a more accurate intervention capable of navigating this integrated representation manifold without compromising the global structure of the generated image.

In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2426–2436 (2023), Gao, D., Lu, S., Zhou, W., Chu, J., Zhang, J., Jia, M., Zhang, B., Fan, Z.: Eraseanything: Enabling concept erasure in rectified flow transformers. p. 2
Where the paper says it

Subsequent approaches like Unified Concept Editing (UCE) [8] introduce closed-form edits to attention projections for efficient, training-free erasure, though they may struggle to fully remove concrete objects. For modern architectures, EraseAnything [9] adapts erasure to rectified flow models via LoRA tuning and attention regularization, and Erasing with Adversarial Preservation (EAP) [3] incorporates adversarial concept identification to preserve unrelated generations during finetuning.

In: Forty-second International Conference on Machine Learning (2025), Hernandez, E. p. 6
Where the paper says it

The corresponding steering direction is defined by the normalized normal vector of the separating hyperplane, v_{\mathrm {svm}} = \frac {w}{\lVert w \rVert } \in \mathbb {R}^{c}, (8) where w denotes the SVM weight vector and c is the channel dimension. The second estimator is the mean-difference direction, v_{\mathrm {diff}} = \frac {1}{n}\sum _{k=1}^{n}\left (a_k^{+}-a_k^{-}\right ), (9) with optional token averaging for sequence activations to reduce sensitivity to prompt length.

, Li, B.Z., Andreas, J.: Inspecting and editing knowledge representations in language models. p. 3
Where the paper says it

Early works focused on semantic navigation in latent representations, such as interpolating between latent codes to blend concepts, and guidance-based conditioning, including classifier guidance [4] where an external classifier steers the denoising process, classifier-free guidance [12] that amplifies conditioning signals without additional models or more complicated approaches introducing semantic guidance [2]. More recent methods have adapted steering techniques from large language models by intervening directly in diffusion model activations, offering finergrained control without retraining.

arXiv preprint arXiv:2304. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

00740 (2023), Liu, S., Ye, H., Zou, J.: Reducing hallucinations in large vision-language models via latent space steering. p. 1
Where the paper says it

In parallel, activation steering has emerged as a lightweight yet potent control mechanism for Large Language Models (LLMs) [1,19,33]. Frameworks such as Representation Engineering [37] demonstrate that high-level semantic behavior can be modulated by injecting linear directions into the latent representation space at inference time, without any weights modification.

In: The Thirteenth International Conference on Learning Representations (2025), Rodriguez, P. p. 3
Where the paper says it

Diffusion models enable controllable synthesis through guidance signals and latent-space manipulations, allowing users to influence the generation process toward desired outcomes. Early works focused on semantic navigation in latent representations, such as interpolating between latent codes to blend concepts, and guidance-based conditioning, including classifier guidance [4] where an external classifier steers the denoising process, classifier-free guidance [12] that amplifies conditioning signals without additional models or more complicated approaches introducing semantic guidance [2].

, Blaas, A., Klein, M., Zappella, L., Apostoloff, N., Cuturi, M., Suau, X.: Controlling language and diffusion models by transporting activations. p. 2
Where the paper says it

2 Activation steering in Large Language Models. Steering in large language models (LLMs) has emerged as a practical paradigm for controlling behavior.

arXiv preprint arXiv:2410.23054 (2024), Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. p. 0
Where the paper says it

com/ControlGenAI/SHIFT . 1 Introduction Diffusion models [11, 28] have established a new state-of-the-art in high-fidelity text-to-image synthesis [23].

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022), Schramowski, P., Brack, M., Deiseroth, B., Kersting, K.: Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. p. 2
Where the paper says it

1 Concept erasure, safety, and post-training approaches. Recent state-of-the-art text-to-image diffusion models trained on large, imperfectly filtered datasets such as LAION [30] can generate inappropriate or copyrighted content.

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 22522– 22531 (2023), Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M.: Laion-5b: An open largescale dataset for training next generation image-text models. p. 2
Where the paper says it

Recent state-of-the-art text-to-image diffusion models trained on large, imperfectly filtered datasets such as LAION [30] can generate inappropriate or copyrighted content. One mitigation is dataset filtering [28] or post-generation filtering [26].

Advances in neural information processing systems 35, 25278–25294 (2022), Subramani, N. p. 2
Where the paper says it

One mitigation is dataset filtering [28] or post-generation filtering [26]. However, these approaches do not fully prevent harmful content and require additional processing.

, Suresh, N., Peters, M.E.: Extracting latent steering vectors from pretrained language models. p. 3
Where the paper says it

More recent methods have adapted steering techniques from large language models by intervening directly in diffusion model activations, offering finergrained control without retraining. In UNet-based latent diffusion models, a wide-spread control mechanism is cross-attention manipulation: for instance, CASteer [6] constructs steering vectors for cross-attention layers using contrastive prompt pairs (e.

In: Findings of the Association for Computational Linguistics: ACL 2022. pp. 566–581 (2022), Turner, A.M., Thiergart, L., Leech, G., Udell, D., Vazquez, J.J., Mini, U., MacDiarmid, M.: Steering language models with activation engineering. p. 1
Where the paper says it

In parallel, activation steering has emerged as a lightweight yet potent control mechanism for Large Language Models (LLMs) [1,19,33]. Frameworks such as Representation Engineering [37] demonstrate that high-level semantic behavior can be modulated by injecting linear directions into the latent representation space at inference time, without any weights modification.

arXiv preprint arXiv:2308. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

10248 (2023), Turner, A.M., Thiergart, L., Leech, G., Udell, D., Vazquez, J.J., Mini, U., MacDiarmid, M.: Steering language models with activation engineering, 2024. p. 1
Where the paper says it

In parallel, activation steering has emerged as a lightweight yet potent control mechanism for Large Language Models (LLMs) [1,19,33]. Frameworks such as Representation Engineering [37] demonstrate that high-level semantic behavior can be modulated by injecting linear directions into the latent representation space at inference time, without any weights modification.

URL https://arxiv. p. 0
Where the paper says it

We demonstrate that SHIFT provides effective and flexible control over DiT generation across diverse prompts and targets without time-consuming retraining. The code is available at https://github.

org/abs/2308.10248 2308 (2024), Zhang, Y., Jin, E., Dong, Y., Wu, Y., Torr, P., Khakzar, A., Stegmaier, J., Kawaguchi, K.: Minimalist concept erasure in generative models. p. 0
Where the paper says it

This has motivated the development of Concept Erasure (CE): the task of reliably suppressing specific semantic concepts within a model’s generative manifold. Existing erasure strategies for text-to-image diffusion models mostly rely on model optimization.

arXiv preprint arXiv:2507. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

13386 (2025), Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., Ba, J.: Steering large language models using ape. p. 3
Where the paper says it

More recent methods have adapted steering techniques from large language models by intervening directly in diffusion model activations, offering finergrained control without retraining. In UNet-based latent diffusion models, a wide-spread control mechanism is cross-attention manipulation: for instance, CASteer [6] constructs steering vectors for cross-attention layers using contrastive prompt pairs (e.

In: NeurIPS ML Safety Workshop (2022), Ziegler, D. p. 2
Where the paper says it

2 Related works 2. 1 Concept erasure, safety, and post-training approaches.

M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., Christiano, P., Irving, G.: Fine-tuning language models from human preferences. p. 2
Where the paper says it

Many works therefore focus on concept erasure (CE) while preserving overall generation quality. Methods such as ESD [7] employ fine-tuning to unlearn concepts using negative guidance from a frozen teacher model, while Concept Ablation (CA) [15] optimizes cross-attention layers to redirect target concepts toward neutral anchors.

arXiv preprint arXiv:1909. p. 0
Where the paper says it

1[schnell], because these models are guidance-distilled to operate at a fixed guidance scale. Non-optimization alternatives, such as Unified Concept Editing (UCE) [8], propose closed-form arXiv:2604.

08593 (2019), Ilharco, G., Ribeiro, M.T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., Farhadi, A.: Editing models with task arithmetic. p. 0
Where the paper says it

This has motivated the development of Concept Erasure (CE): the task of reliably suppressing specific semantic concepts within a model’s generative manifold. Existing erasure strategies for text-to-image diffusion models mostly rely on model optimization.

arXiv preprint arXiv:2212.04089 (2022).

The ablation on steering strength and classifier usage shows that the classifier is critical both for stable steering and for preserving overall image quality. Table 7 suggests that using only text-encoder steering or only diffusion-transformer steering is insufficient for high-quality erasure, while increased text steering improves removal but risks over-erasure where nontarget concepts lose semantic fidelity. The ablation on timesteps proves that early steps are crucially important for diffusion steering.

The results on concrete concept erasure show that SHIFT clearly outperforms all baselines on safety-related metrics, achieving more than 3× and 4× stronger suppression at different steering strengths. For concrete object removal, SHIFT performs much better than the CA method when erasing concepts like Snoopy. p. 8

Where the paper says it

Following the protocol established by SPM [20], we assess concrete concept removal using 80 fixed prompts and 9 different seeds. Specifically, we evaluate erasure of the Snoopy concept while preserving five related concepts: Mickey, SpongeBob, Pikachu, dog, and legislator.

Furthermore, the qualitative results show that our method can erase undesirable concepts while preserving overall image quality. p. 8
Where the paper says it

3 for Flux. 1[dev], showing that our method can erase undesirable concepts while preserving overall image quality.

The ablation on steering-vector parameterization shows that token-wise difference yields stronger target-concept suppression. p. 5
Where the paper says it

For Flux, we find that steering both the text encoder pooled embedding and the diffusion transformer part enhances controllability and semantic alignment, whereas steering text encoder alone yields suboptimal results. 2 Steering vector construction We construct a separate steering vector for each target concept X and for each steering location (text encoder pooled embedding and diffusion transformer tokens).

The method’s versatility is shown across several tasks, including concept addition, concept switching (e.g., cat → dog), and domain shifting to specific styles like sketch or cyberpunk. The ablation on number of steered DiT blocks shows that restricting steering to a subset of blocks is substantially more effective than applying it across all blocks for adding hat. The results confirm that the approach successfully switches a wide range of concepts, including global attributes (age) and small objects (apple vs. p. 7

Where the paper says it

We additionally include small local objects removal examples (e. , hats and glasses) to provide visually interpretable results.

banana). This demonstrates concept steering as a promising and generalizable direction for controllable object-level editing in diffusion models. p. 1
Where the paper says it

While our primary evaluation focuses on concept erasure, we establish that this activation-level modulation extends to domain stylization and object-specific biasing. Importantly, SHIFT is positioned as a generation control framework rather than a traditional image-editing tool; it focuses on the global redirection of the model’s generative manifold rather than the preservation of spatial layouts or background consistency from a specific input image.

The paper concludes that SHIFT achieves competitive performance while maintaining high computational efficiency.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Concept Erasure in Diffusion Transformers (DiTs) via SHIFT

The AI system can reliably suppress specific semantic concepts within a model’s generative manifold by learning steering vectors that are dynamically applied to selected layers and timesteps to suppress unwanted visual concepts while preserving the prompt’s remaining content. This allows for concept removal without retraining the base model.

  1. Cross-Domain Concept Transfer

The system can transfer steering vectors across distillation boundaries, validating their efficacy in resource-constrained settings by demonstrating robust performance in resource-constrained settings when moving from a distilled model like Flux.1[schnell] to a full model like Flux.1[dev]."

  1. Unified Steering Control Framework

SHIFT provides "the first comprehensive framework for steering both original and distilled DiT-based models (e.g., Flux.1[dev] and Flux.1[schnell]) through latent activation shifts, enabling efficient and scalable concept manipulation without retraining. This enables a single mechanism for original and distilled models."

  1. Multi-Task Control Capabilities

The improved AI system can perform diverse generative control tasks beyond simple erasure, including domain stylization, object-specific biasing, and complex tasks like concept addition (e.g., adding glasses or hats) and concept switching (e.g., cat to dog)."

  1. Fine-Grained Spatial and Temporal Steering

The system can apply steering effectively across the generation process by demonstrating that a single, time-independent steering vector can be applied effectively across all diffusion timesteps, while ablation studies show that steering is most effective during the early stages of the diffusion trajectory for certain tasks.

  1. Adaptive Steering Strength Regulation

The system incorporates a mechanism to adjust steering strength dynamically, using a classifier signal such as a lightweight classifier (SVM) signal that reflects whether the target concept is still present in the current activations, which suppresses steering when activations are far from the target class and amplifies steering when target-concept evidence remains strong.

  1. Concept Switching and Addition

The AI can successfully perform concept switching, replacing one target concept with another (e.g., cat → dog or man → woman), and add new concepts, such as an apple, even when they are traditionally difficult to integrate naturally.

Abstract

Diffusion models have become the leading approach for high-fidelity image generation, with recent DiT-based models achieving strong prompt adherence and high visual quality. However, removing unwanted concepts from these models typically requires expensive gradient-based fine-tuning, which is particularly prohibitive for multi-billion-parameter Diffusion Transformers and incompatible with distilled variants. We propose SHIFT, a simple model-weight-update-free inference-time framework for concept suppression in DiT models via targeted manipulation of intermediate activations at inference time, inspired by activation steering in LLMs. SHIFT learns steering vectors from contrastive prompts and applies them dynamically. We demonstrate that a single time-independent steering vector generalizes across all denoising timesteps and that vectors computed on a distilled model transfer effectively to its non-distilled counterpart without recomputation. SHIFT achieves strong concept suppression while preserving prompt fidelity and image quality. The same mechanism also enables flexible control -- adding, removing, or changing objects and styles without retraining. SHIFT targets hosted inference settings: it does not remove the concept from the model weights and is therefore complementary to weight-level unlearning.

Sources

Related papers