TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models".
Paper summary: Tom: So we're looking at a paper called TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models. Jane, what's the basic idea here?
Jane: Well, it starts by saying that text-to-image diffusion models can be misused to create harmful content using prompts that bypass the safety filters built into them. And existing methods for removing those harmful concepts often have problems, like they aren't robust against clever adversarial prompts, or they degrade the quality of good images you try to make.
Lu: Exactly. The core issue is that current concept erasure techniques can either fail when someone tries to trick them with adversarial prompts, or they just make the resulting safe images look weird because the model gets confused about what it's supposed to generate.
Meng: And a big problem for deployment is that some of these methods require changes during inference time, which means they add extra work every single time someone tries to generate an image. That’s not good for scaling up production systems where speed matters.
Lalam: The authors propose a different approach called TEA, which frames concept erasure as a domain alignment problem in the text representation space. They suggest that instead of trying to fix the whole model, you can just focus on aligning the text encoder's output toward what is considered safe.
Tom: So what does this mean practically? What exactly are they proposing they can do with this TEA framework?
Jane: The main claim is that TEA uses a lightweight Text Encoder Alignment framework to solve this problem. They fine-tune only the text encoder while keeping the rest of the generative backbone completely frozen. And they do this by training a discriminator to tell difference between token representations of prompts containing concepts versus those containing safe anchor prompts.
Lu: The goal is to update that text encoder so that when it sees a concept prompt, its token representations look exactly like the safe anchor prompts, making them indistinguishable to the discriminator. They also have this preservation loss term which keeps the original embeddings of those safe anchors close to where they started in the encoder.
Meng: That sounds promising for deployment because if you only fine-tune that text encoder once during training, it introduces zero inference time overhead at all. That’s a huge win for running models at scale.
Lalam: They also mention that they train this adversarially using a discriminator and then update the encoder while freezing the generative backbone, which is essentially what they call Min-Max Optimization Step A and B in their setup. This process aims to drive the concept representations toward anchor representations while maintaining the original structure of those safe embeddings.
Paper summary: Tom: That sounds like a clever way to localize where you need to make changes instead of touching every part of the diffusion model. What about the specific training objectives they use?
Jane: They define two main losses for this encoder optimization. First, an adversarial loss that tries to fool the discriminator so that concept tokens look like anchor tokens. Second, a preservation loss, which uses a cosine similarity measure to keep those safe anchor embeddings close to their original positions in the encoder output.
Lu: The paper formalizes this by defining two distributions on the representation space Z, one for concept-containing prompts and one for anchor prompts, and they bound the concept risk by an H divergence between these two distributions. This mathematical setup shows that achieving a certain level of erasure is controlled by how much those two representation distributions are separated.
Meng: I'm interested in the adversarial loss part. They define it as the sum of fooling the discriminator on concept tokens and making sure anchor tokens look safe, which is pretty standard stuff but how they combine it with that preservation loss is what makes it unique for this setup.
Lalam: The full encoder loss combines these two terms, the adversarial loss and the preservation loss, using some weighting factors lambda for each. This entire process is designed to make concept prompts look like safe anchor prompts while ensuring those safe anchors stay close to their original embeddings.
Tom: So, if we follow this up with what they found in their experiments—what did they actually measure? What kind of results are we looking at from the TEA framework?
Jane: They show that TEA achieves state-of-the-art erasure robustness against both black-box and white-box adversarial attacks on models like Stable Diffusion version one point four. In explicit content erasure, they report removing more than ninety-nine percent of the detections from the original model, leaving only eight detections, which is the highest erasure rate among all methods they tested.
Lu: For artistic style erasure on Stable Diffusion v1 point 4, TEA achieves the highest LPIPSe scores for both artists and gets the highest LPIPSd score specifically for Van Gogh’s work <ref:2608.15341#pg1>. That tells you it works well across different types of content, not just one specific concept type or style.
Meng: The attack success rate across all three benchmarks they tested is staying at or below four percent, and importantly, benign generation quality remains indistinguishable from the original image after the erasure process. That’s crucial for production where you don't want to accidentally ruin a good output.
Lalam: They also showed that TEA achieves the lowest attack success rate on all three benchmarks without having to change the underlying method itself. And they tested it on a different generative backbone, Flux, and while the residual rates there are higher than their SD v1 point 4 results, things like Ring-A-Bell stay at forty-eight point ten percent after erasure <ref:2608.15341#pg1>.
Paper summary: Tom: So the paper claims this works across different architectures too, even when you don't train it specifically for that new architecture. What does that imply for future model development?
Jane: It’s model agnostic, meaning it transfers to other diffusion architectures, including Rectified Flow Transformers with T5 conditioning where prior concept erasure methods haven't been explored much. This suggests the approach to aligning the text encoder is a general principle rather than something tied only to one specific model structure.
Lu: I think what’s interesting is that they focused on targeting the text encoder because semantic attributes seem more localized there, which motivates this direction, but they also admit a limitation: existing methods for text-encoder based approaches often require large datasets or codebook memory.
Meng: From a deployment standpoint, the training time is quite fast at about five seconds for training-based methods, which is much better than some other training-based techniques like ESD or AdvUnlearn that take longer to train. But remember, these are all still training-based methods requiring a setup cost upfront.
Lalam: The authors also mention a limitation: the anchors they use are just distributional descriptions of what counts as safe, and the exact wording of those anchors isn't a sensitive component of the method itself. So you have to define your safe concepts carefully beforehand.
Tom: Okay, so to wrap up this whole discussion on TEA: what’s the overall takeaway for someone who just listens and doesn't read? What does this paper actually mean for how we use generative models safely?
Jane: Basically, it provides a way to make concept erasure much more robust without adding any slow steps during generation. It shows that you can achieve high erasure rates across different model types by focusing on aligning the text encoder's internal representations to a set of safe anchor prompts.
Lu: The implication is that if you want concept control in these models, targeting the text encoder representation space through alignment, rather than modifying the entire image generation pipeline or using complex inference hacks, is a viable and efficient path.
Meng: For me, it means if I'm shipping a model to production right now and I need safety features that don't slow down my users during generation, this framework seems like the most practical way to implement it because there’s no per-query overhead.
Lalam: The authors are pretty confident in their results; they show superior erasure effectiveness compared to prior methods across black-box and white-box scenarios, and they even demonstrated success on a model architecture that wasn't originally targeted by previous work.
Conclusion: Tom: So we're wrapping up our look at TEA, which stands for Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models. It basically says they found a way to make concept erasure much more stable without adding any slow steps during image generation.
Jane: Right, so instead of trying to patch the whole model, they focused on aligning the text encoder's output toward what counts as safe prompts. That means if you give it a harmful prompt, the text representation looks like a safe one to the model.
Lu: Exactly. They fine-tuned only that text encoder while leaving the rest of the image generation backbone completely frozen. The math shows this alignment bounds how much risk is in those concept representations by looking at how far they are from safe ones.
Meng: I’m still thinking about the practical stuff—it says they achieved state-of-the-art erasure rates on models like Stable Diffusion v1 point four and importantly, it works across different model types too, like Rectified Flow Transformers.
Lalam: What that means is that this isn't just a fix for one specific model architecture; the method seems to be a general way to align text representations for concept control in these diffusion systems.
Tom: It’s about making sure safety features are robust against both clever adversarial attacks and just regular misuse, without hurting the quality of what we actually generate.
Jane: And they did this training relatively quickly, about five seconds, and it doesn't add any overhead when you actually run the model later. That’s a big deal for deployment.
Lu: It shows that representation space alignment is a really powerful way to handle these kinds of semantic control issues in generative AI.
Meng: So yeah, it’s efficient training and zero inference cost for concept erasure, which is exactly what we need when shipping models to users.
Tom: Before we move on, let's look at how the authors actually built this alignment mechanism and the specific losses they used to get those results.
Mila – Quebec AI Institute · McGill University · Google Research
cs.CV
Submitted: 2026-08-15
Updated: 2026-11-18
Code: https://github.com/alirezafarashah/TEA
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms Existing concept erasure methods often…
Key concepts
- Concept Erasure
- The goal is to remove specific harmful concepts from an image generation model's output. This is achieved by making the model's text encoder ignore the concept while still generating a safe result, ensuring the backbone cannot distinguish between concept-containing and concept-free inputs.
- Domain Alignment Problem
- The paper frames concept erasure as aligning two different distributions in the text representation space: one for prompts containing concepts and one for safe anchor prompts. The method trains the encoder to pull these distributions closer together, making them indistinguishable.
- Text Encoder Alignment (TEA)
- TEA is a lightweight framework that only fine-tunes the text encoder while keeping the main generative model backbone frozen. It uses adversarial training with a discriminator to force concept embeddings to look like safe anchor embeddings, achieving erasure efficiently.
- Adversarial Training Loss
- This loss function trains the text encoder by using a discriminator. The goal is twofold: make concept prompt embeddings look like safe ones (minimizing one loss term) and ensure that the safe anchor representations don't collapse into the concept representation (minimizing another preservation loss term).
Terminology
Summary
Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms Existing concept erasure methods often suffer from limited robustness against adversarial prompts, degradation of benign generation quality, or reliance on inference-time interventions that introduce persistent computational overhead To address these limitations, we formulate concept erasure as a domain alignment problem in the text representation space We propose a lightweight Text Encoder Alignment framework (TEA) that fine-tunes only the text encoder while keeping the generative backbone fully frozen Given concept–anchor prompt pairs, our method trains a discriminator to distinguish token-level representations of concept-containing prompts from those of safe anchor prompts, while updating the text encoder to make these representations indistinguishable TEA introduces zero inference-time overhead and requires only a small number of fine-tuning steps, making it highly efficient to deploy at scale
Concept Erasure Formulation
We formulate concept erasure as a representation-space domain alignment problem Given a generative model that decomposes into a text encoder and a frozen generative backbone, suppressing a target concept reduces to aligning the representation distributions of concept-bearing inputs toward those of concept-free anchor inputs—so that the backbone can no longer distinguish between the two We instantiate this formulation as TEA (Text Encoder Alignment for Concept Erasure), which fine-tunes only the text encoder using paired concept and anchor prompts Training adversarially using a discriminator, drives concept representations toward anchor representations, while a cosine preservation loss keeps anchor embeddings close to those of the original encoder
Mechanism of Alignment
The core mechanism involves adapting only the text encoder while keeping the generative backbone frozen We define two distributions on the representation space Z via the map τθ: D c θ = (τθ)P c, D ¬c θ = (τθ)P ¬c The concept risk is rewritten as expectations under these representation distributions Proposition 1 bounds the concept risk by an H-divergence between concept and concept-free representation distributions This bound implies that requirement (E) is achieved by controlling the two terms on the right-hand side of Inequality (6)
Training Objectives and Losses
The encoder is optimized to fool the discriminator so that embeddings of concept prompts become indistinguishable from those of anchor prompts The adversarial loss is defined as LD = P t mc t BCELogits(D(h c t), 1) + P t ma t BCELogits(D(h a t), 0) P t ma t Furthermore, the encoder adversarial loss is Ladv = P t mc t BCELogits(D(h c t), 0) To prevent collapsing anchor concept representation to target representations, we define the preservation loss as Lpres = P t ma t (1 − h a,new t · h a,orig t)∥h a,new t∥∥h a,orig t∥ P t ma t The full encoder loss combines these terms: Lenc = λadv Ladv + λpres Lpres
Robustness and Efficiency
TEA achieves state-of-the-art erasure robustness against black-box and white-box adversarial attacks on Stable Diffusion v1.4 It is model-agnostic and transfers across modern diffusion architectures, including Rectified Flow Transformers with T5 conditioning where existing concept erasure methods remain largely unexplored TEA achieves the lowest attack success rate on Stable Diffusion v3. p. 1 Furthermore, we propose TEA, a lightweight adversarial text encoder fine-tuning
framework for concept erasure that achieves robust suppression of target concepts while preserving
benign generation quality. 2 We demonstrate that our method is model-agnostic and transfers across modern diffusion architectures, including those built on Rectified Flow Transformers with T5 conditioning, where existing
concept erasure methods remain largely unexplored.Where the paper says it
Evaluation and Results
TEA demonstrates superior erasure effectiveness across both black-box and white-box attack scenarios In explicit content erasure, TEA removes more than 99% of the detections of the original model, leaving 8 detections, which is the highest erasure rate among all methods For artistic style erasure on SD-v1.4, our method achieves the highest LPIPSe on both artists, and the highest LPIPSd for Van Gogh The attack success rate stays at or below 4% on all three benchmarks and benign generation is indistinguishable from the substitution run TEA attains the lowest attack success rate on all three benchmarks without any change to the method The residual rates on FLUX are nevertheless well above what the same procedure reaches on SD-v1. p. 7 As shown in Figure 3, our method consistently outperforms all baselines and achieves the lowest
attack success rate across all three white-box attacks: UnlearnDiff, MMA, and Ring-A-Bell. The
results in Figure 3 and Table 1 reveal that methods performing erasure in the text encoder are more
robust against adversarial attacks while better preserving generation quality. Target Concept: Nudity
SD v3 SAFREE DES Ours
Figure 2: Qualitative examples of explicit content erasure. 4
SPM
Safe-CLIP
UCE
ESD
GLoCE
SalUn
AdvUnlearn
DES
Ours
0
20
40
60
80
100
Attack Success Rate (%) ↓
MMA UnlearnDiff Ring-A-Bell Avg.Where the paper says it
Where the paper says it
Computational Cost Comparison
TEA achieves the shortest training time among all training-based methods at approximately 5 seconds, while introducing no inference overhead Training-free methods such as SLD and SAFREE require no training but introduce a significant inference overhead of approximately 1.5×, making them slower at generation time Methods that modify model weights at training time vary widely in cost: ESD and AdvUnlearn require 0. p. 1 Although training-free methods avoid upfront optimization, they introduce persistent per-query overhead that scales with deployment load. In contrast, a one-time training cost yields a persistently safe
model with zero inference overhead, making training-based approaches more practical for large-scale
deployment.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Model concept erasure can be achieved by formulating it as a domain alignment problem in text representation space, which involves training a discriminator to distinguish token-level representations of concept-containing prompts from those of safe anchor prompts while updating the text encoder to make these representations indistinguishable.
-
The Text Encoder Alignment for Concept Erasure (TEA) framework allows for zero inference-time overhead and requires only a small number of fine-tuning steps, making it highly efficient to deploy at scale, and achieves state-of-the-art erasure robustness against black-box and white-box adversarial attacks on Stable Diffusion v1.4.
-
TEA is model agnostic and transfers across modern diffusion architectures, including Rectified Flow Transformers with T5 conditioning where prior methods remain largely unexplored, enabling concept erasure to be applied to any text-to-image model without architecture-specific modification by only adapting the conditioning pathway.
-
The method can suppress target concepts while preserving benign generation quality on unrelated prompts by controlling the two terms in Inequality (6), where one term is the concept risk on concept-free inputs, which is kept small by a preservation loss through anchor embeddings close to those of the original encoder, and the other term is an alignment term driven by adversarial training.
-
The system can perform fine-grained erasure by aligning all non-padding positions of the prompt rather than only restricted concept spans, which ensures robustness against black-box adversarial prompts because
concept information will leak to other token positions and the text encoder is not robust against that.
-
The system can be optimized for various content types by adjusting loss coefficients; for explicit content erasure, setting lambda adv = 0.5 and lambda pres = 1.0 results in a configuration that keeps the attack success rate low on all three benchmarks while sitting in the middle of the utility range spanned by the grid.
Abstract
Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms. Existing concept erasure methods often suffer from limited robustness against adversarial prompts, degradation of benign generation quality, or reliance on inference-time interventions that introduce persistent computational overhead. To address these limitations, we formulate concept erasure as a domain alignment problem in the text representation space. We propose a lightweight Text Encoder Alignment framework (TEA) that fine-tunes only the text encoder while keeping the generative backbone fully frozen. Given concept-anchor prompt pairs, our method trains a discriminator to distinguish token-level representations of concept-bearing prompts from those of safe anchor prompts, while updating the text encoder to make these representations indistinguishable. TEA introduces zero inference-time overhead and trains in seconds with only a few fine-tuning steps, making it practical for large-scale deployment. Despite this efficiency, TEA achieves state-of-the-art erasure robustness against black-box and white-box adversarial attacks on Stable Diffusion v1.4, while preserving generation quality on benign prompts. Furthermore, TEA is model-agnostic and achieves the lowest attack success rate on Stable Diffusion v3.5, extending concept erasure to Rectified Flow Transformer architectures with T5 conditioning, which remain largely unexplored by prior methods. Code is available at https://github.com/alirezafarashah/TEA
Sources
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Classifier-Free Diffusion Guidance
- Red-Teaming the Stable Diffusion Safety Filter
- Detoxifying LLMs via Representation Erasure-Based Preference Optimization
- Trash to Treasure: Using text-to-image models to inform the design of physical artefacts
Related papers
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics