TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models

summary

The gist

Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms Existing concept erasure methods often…

In short

Text-to-image models can be misused to generate harmful content via adversarial prompts. This work proposes Text Encoder Alignment for Concept Erasure (TEA), a method that fine-tunes only the text encoder of a model. It aligns concept prompts with safe anchor prompts using adversarial training, effectively erasing target concepts without adding any inference time or significant computational overhead.

Key concepts

Concept Erasure
The goal is to remove specific harmful concepts from an image generation model's output. This is achieved by making the model's text encoder ignore the concept while still generating a safe result, ensuring the backbone cannot distinguish between concept-containing and concept-free inputs.
Domain Alignment Problem
The paper frames concept erasure as aligning two different distributions in the text representation space: one for prompts containing concepts and one for safe anchor prompts. The method trains the encoder to pull these distributions closer together, making them indistinguishable.
Text Encoder Alignment (TEA)
TEA is a lightweight framework that only fine-tunes the text encoder while keeping the main generative model backbone frozen. It uses adversarial training with a discriminator to force concept embeddings to look like safe anchor embeddings, achieving erasure efficiently.
Adversarial Training Loss
This loss function trains the text encoder by using a discriminator. The goal is twofold: make concept prompt embeddings look like safe ones (minimizing one loss term) and ensure that the safe anchor representations don't collapse into the concept representation (minimizing another preservation loss term).

Terminology used across episodes

This episode discusses

The paper

TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models · Read on arXiv

Mila – Quebec AI Institute · McGill University · Google Research

Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms. Existing concept erasure methods often suffer from limited robustness against adversarial prompts, degradation of benign generation quality, or reliance on inference-time interventions that introduce persistent computational overhead. To address these limitations, we formulate concept erasure as a domain alignment problem in the text representation space. We propose a lightweight Text Encoder Alignment framework (TEA) that fine-tunes only the text encoder while keeping the generative backbone fully frozen. Given concept-anchor prompt pairs, our method trains a discriminator to distinguish token-level representations of concept-bearing prompts from those of safe anchor prompts, while updating the text encoder to make these representations indistinguishable. TEA introduces zero inference-time overhead and trains in seconds with only a few fine-tuning steps, making it practical for large-scale deployment. Despite this efficiency, TEA achieves state-of-the-art erasure robustness against black-box and white-box adversarial attacks on Stable Diffusion v1.4, while preserving generation quality on benign prompts. Furthermore, TEA is model-agnostic and achieves the lowest attack success rate on Stable Diffusion v3.5, extending concept erasure to Rectified Flow Transformer architectures with T5 conditioning, which remain largely unexplored by prior methods. Code is available at https://github.com/alirezafarashah/TEA

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models".

Paper summary: Tom: So we're looking at a paper called TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models. Jane, what's the basic idea here?

Jane: Well, it starts by saying that text-to-image diffusion models can be misused to create harmful content using prompts that bypass the safety filters built into them. And existing methods for removing those harmful concepts often have problems, like they aren't robust against clever adversarial prompts, or they degrade the quality of good images you try to make.

Lu: Exactly. The core issue is that current concept erasure techniques can either fail when someone tries to trick them with adversarial prompts, or they just make the resulting safe images look weird because the model gets confused about what it's supposed to generate.

Meng: And a big problem for deployment is that some of these methods require changes during inference time, which means they add extra work every single time someone tries to generate an image. That’s not good for scaling up production systems where speed matters.

Lalam: The authors propose a different approach called TEA, which frames concept erasure as a domain alignment problem in the text representation space. They suggest that instead of trying to fix the whole model, you can just focus on aligning the text encoder's output toward what is considered safe.

Tom: So what does this mean practically? What exactly are they proposing they can do with this TEA framework?

Jane: The main claim is that TEA uses a lightweight Text Encoder Alignment framework to solve this problem. They fine-tune only the text encoder while keeping the rest of the generative backbone completely frozen. And they do this by training a discriminator to tell difference between token representations of prompts containing concepts versus those containing safe anchor prompts.

Lu: The goal is to update that text encoder so that when it sees a concept prompt, its token representations look exactly like the safe anchor prompts, making them indistinguishable to the discriminator. They also have this preservation loss term which keeps the original embeddings of those safe anchors close to where they started in the encoder.

Meng: That sounds promising for deployment because if you only fine-tune that text encoder once during training, it introduces zero inference time overhead at all. That’s a huge win for running models at scale.

Lalam: They also mention that they train this adversarially using a discriminator and then update the encoder while freezing the generative backbone, which is essentially what they call Min-Max Optimization Step A and B in their setup. This process aims to drive the concept representations toward anchor representations while maintaining the original structure of those safe embeddings.

Paper summary: Tom: That sounds like a clever way to localize where you need to make changes instead of touching every part of the diffusion model. What about the specific training objectives they use?

Jane: They define two main losses for this encoder optimization. First, an adversarial loss that tries to fool the discriminator so that concept tokens look like anchor tokens. Second, a preservation loss, which uses a cosine similarity measure to keep those safe anchor embeddings close to their original positions in the encoder output.

Lu: The paper formalizes this by defining two distributions on the representation space Z, one for concept-containing prompts and one for anchor prompts, and they bound the concept risk by an H divergence between these two distributions. This mathematical setup shows that achieving a certain level of erasure is controlled by how much those two representation distributions are separated.

Meng: I'm interested in the adversarial loss part. They define it as the sum of fooling the discriminator on concept tokens and making sure anchor tokens look safe, which is pretty standard stuff but how they combine it with that preservation loss is what makes it unique for this setup.

Lalam: The full encoder loss combines these two terms, the adversarial loss and the preservation loss, using some weighting factors lambda for each. This entire process is designed to make concept prompts look like safe anchor prompts while ensuring those safe anchors stay close to their original embeddings.

Tom: So, if we follow this up with what they found in their experiments—what did they actually measure? What kind of results are we looking at from the TEA framework?

Jane: They show that TEA achieves state-of-the-art erasure robustness against both black-box and white-box adversarial attacks on models like Stable Diffusion version one point four. In explicit content erasure, they report removing more than ninety-nine percent of the detections from the original model, leaving only eight detections, which is the highest erasure rate among all methods they tested.

Lu: For artistic style erasure on Stable Diffusion v1 point 4, TEA achieves the highest LPIPSe scores for both artists and gets the highest LPIPSd score specifically for Van Gogh’s work <ref:2608.15341#pg1>. That tells you it works well across different types of content, not just one specific concept type or style.

Meng: The attack success rate across all three benchmarks they tested is staying at or below four percent, and importantly, benign generation quality remains indistinguishable from the original image after the erasure process. That’s crucial for production where you don't want to accidentally ruin a good output.

Lalam: They also showed that TEA achieves the lowest attack success rate on all three benchmarks without having to change the underlying method itself. And they tested it on a different generative backbone, Flux, and while the residual rates there are higher than their SD v1 point 4 results, things like Ring-A-Bell stay at forty-eight point ten percent after erasure <ref:2608.15341#pg1>.

Paper summary: Tom: So the paper claims this works across different architectures too, even when you don't train it specifically for that new architecture. What does that imply for future model development?

Jane: It’s model agnostic, meaning it transfers to other diffusion architectures, including Rectified Flow Transformers with T5 conditioning where prior concept erasure methods haven't been explored much. This suggests the approach to aligning the text encoder is a general principle rather than something tied only to one specific model structure.

Lu: I think what’s interesting is that they focused on targeting the text encoder because semantic attributes seem more localized there, which motivates this direction, but they also admit a limitation: existing methods for text-encoder based approaches often require large datasets or codebook memory.

Meng: From a deployment standpoint, the training time is quite fast at about five seconds for training-based methods, which is much better than some other training-based techniques like ESD or AdvUnlearn that take longer to train. But remember, these are all still training-based methods requiring a setup cost upfront.

Lalam: The authors also mention a limitation: the anchors they use are just distributional descriptions of what counts as safe, and the exact wording of those anchors isn't a sensitive component of the method itself. So you have to define your safe concepts carefully beforehand.

Tom: Okay, so to wrap up this whole discussion on TEA: what’s the overall takeaway for someone who just listens and doesn't read? What does this paper actually mean for how we use generative models safely?

Jane: Basically, it provides a way to make concept erasure much more robust without adding any slow steps during generation. It shows that you can achieve high erasure rates across different model types by focusing on aligning the text encoder's internal representations to a set of safe anchor prompts.

Lu: The implication is that if you want concept control in these models, targeting the text encoder representation space through alignment, rather than modifying the entire image generation pipeline or using complex inference hacks, is a viable and efficient path.

Meng: For me, it means if I'm shipping a model to production right now and I need safety features that don't slow down my users during generation, this framework seems like the most practical way to implement it because there’s no per-query overhead.

Lalam: The authors are pretty confident in their results; they show superior erasure effectiveness compared to prior methods across black-box and white-box scenarios, and they even demonstrated success on a model architecture that wasn't originally targeted by previous work.

Conclusion: Tom: So we're wrapping up our look at TEA, which stands for Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models. It basically says they found a way to make concept erasure much more stable without adding any slow steps during image generation.

Jane: Right, so instead of trying to patch the whole model, they focused on aligning the text encoder's output toward what counts as safe prompts. That means if you give it a harmful prompt, the text representation looks like a safe one to the model.

Lu: Exactly. They fine-tuned only that text encoder while leaving the rest of the image generation backbone completely frozen. The math shows this alignment bounds how much risk is in those concept representations by looking at how far they are from safe ones.

Meng: I’m still thinking about the practical stuff—it says they achieved state-of-the-art erasure rates on models like Stable Diffusion v1 point four and importantly, it works across different model types too, like Rectified Flow Transformers.

Lalam: What that means is that this isn't just a fix for one specific model architecture; the method seems to be a general way to align text representations for concept control in these diffusion systems.

Tom: It’s about making sure safety features are robust against both clever adversarial attacks and just regular misuse, without hurting the quality of what we actually generate.

Jane: And they did this training relatively quickly, about five seconds, and it doesn't add any overhead when you actually run the model later. That’s a big deal for deployment.

Lu: It shows that representation space alignment is a really powerful way to handle these kinds of semantic control issues in generative AI.

Meng: So yeah, it’s efficient training and zero inference cost for concept erasure, which is exactly what we need when shipping models to users.

Tom: Before we move on, let's look at how the authors actually built this alignment mechanism and the specific losses they used to get those results.

More episodes

← Home