Output Vector Editing for Memorization Mitigation in Large Language Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Output Vector Editing for Memorization Mitigation in Large Language Models".

Jane: Here's the gist of the paper. The gist The proposed output vector editing method modifies what neurons contribute to the residual stream rather than whether they are active, achieving up to 87.9% suppression on OLMo-7B by introducing a distractor in vocabulary space while leaving activations unchanged How it works The core of the method is output vector editing, a constrained-optimization weight edit that locates a small set of MLP neurons responsible for a memorized continuation and minimally modifies their output vectors to introduce a distractor in vocabulary space, redirecting their residual-stream contributions while leaving activations unchanged This is achieved by modifying the output vector vi of MLP neurons implicated in a memorized continuation The closed-form solution is a rank-one update along a chosen distractor token’s unembedding direction, adapting the rank-one weight-update mechanism of ROME from matrix-level factual editing to perneuron output redirection Unlike activation steering, our edit is computed once per memorized sequence and applied directly to the weights; at inference the model runs unchanged Memorization Mining and Localization The process begins with mining memorized content directly from each model’s pretraining corpus using teacher-forced prediction, where outputs agree at the first continuation token, which is the target of our editing For OLMo-7B, 6,831 unique sequences were mined from its pretraining corpus Localization is a two-step process that identifies contributing neurons in two stages Step 1 involves layer importance, where the final residual stream decomposes additively across all layers, and components are projected through the unembedding matrix to calculate an LI value indicating boost or dampening of the target token Step 2 scores each neuron by its L1 distance to r post, where a high score is received if the contribution ai·vi is close to r post in L1 distance Only neurons whose unembedded output vector (vi·WU) ranks the target token either in the top 50 of the vocabulary or in the bottom 50 are selected for editing Edit Modes and Performance Trade-offs Four edit modes span a spectrum from aggressive suppression to minimal redirection The modes are defined by the choice of distractor token s, which defines three promotion modes and a fourth mode that dampens the target directly The four edit modes include end-of-sequence (EOS), redaction marker (redact), next-best, and direct suppression Evaluation on OLMo-7B showed a success–locality trade-off across three operating points Conservative editing (redact) at 31.7% suppression has near-zero locality cost, while practical editing (next-best, k=5) at 81.5% has no catastrophic edits Mechanistic Findings and Limitations The research identified a mechanistic boundary at ∼14% of sequences unreachable by MLP-only editing Attention-layer analysis showed these failures are not driven by attention overall, but ablating the top contributing attention heads at the target position recovers 56–64% of EOS k=10 failures and 53–60% of Next-best k=5 failures The paper also noted that universal failures, approximately 921 of 6,831 memorized sequences (∼14%) resist every MLP-based configuration These universal failures are often driven by strong sequential dependencies where memorization is distributed redundantly beyond the top-k budget Furthermore, aggressive modes like EOS (k=10) can incur tail risk, with 47.8% of edits having a median shift in perplexity of +0.88 Cross-Model Generalization Success rates scale with model size rather than family, as SmolLM-360M (34.9%) and OLMo-1B (52.4%) sit well below OLMo-7B's 85.7% success rate Locality costs are architecture-dependent, with the catastrophic-outlier tail on OLMo-7B aggressive modes not appearing on smaller models like OLMo-1B or Llama2-7B The paper concludes that output vector editing is a weight-level intervention that modifies what neurons contribute to the residual stream rather than whether they are active How it works The paper presents output vector editing, a neuron-level method for mitigating memorization in LLMs that modifies what neurons contribute to the residual stream rather than whether they are active It was evaluated across four models from two architecture families and different sizes (SmolLM360M, OLMo-1B, OLMo-7B, Llama2-7B), with a deeper grid on OLMo-7B for which the pretraining corpus is fully released Three findings emerge: a success–locality trade-off where Next-best k=5 achieves 81.5% suppression with zero catastrophic failures on OLMo-7B (with the same edit-mode separation across the other three models); complementarity across edit modes covering 96.5% of OLMo-7B sequences in ensemble; and a hard “mechanism” limit at ∼14% of OLMo-7B sequences where MLP-based editing cannot reach Limitations and Ethics Statement The work aims to mitigate memorization risks, with positive impact via memorization mitigation for privacy/copyright/security, with negative-impact discussion of recoverability under adversarial probing, dual-use risk for factual rewriting, and stakeholder bias from corpus-access requirements The method can be misused for factual rewriting; reliance on pretraining corpus access biases legitimate deployment toward stakeholders with data provenance but misuse is not impossible Open access to data and code Code and mining scripts will be released upon publication under an opensource license All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.7, SlimPajama-6B, smollm-corpus) used are publicly available The paper specifies all hyperparameters (β, k), localization procedure, edit modes, and evaluation protocols required to reproduce the experiments All models (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B) and corpora (Dolma v1.

Title and authors: Tom: Now, let’s look at who wrote this. The paper is titled "Output Vector Editing for Memorization Mitigation in Large Language Models," and the authors include Ahmad Dawar Hakimi, Kaiwei Lei, Isabelle Augenstein, Hinrich Schütze >

Jane: It’s a pretty deep team of researchers from places like LMU Munich and the University of Copenhagen working on these kinds of model architectures >

Lu: They’re focusing on how the model stores knowledge in those MLP neurons because they think we need to understand that structure to fix it properly >

Meng: From what I can tell, this approach is focused on finding the specific weight edits needed rather than trying to steer the entire neural network, which is something we've seen before >

Lalam: They are specifically targeting sequences that the model has memorized, like a continuation of a prompt that appeared in its training set >

The paper's summary: Tom: The core finding they highlight is that this method can achieve up to eighty-seven point nine percent suppression on the OLMo-7B model when it’s targeting memorized continuations from its pretraining corpus >

Jane: That eighty-seven point nine percent number is pretty significant, Tom, and the paper explains it by locating those specific MLP neurons responsible for the continuation and minimally modifying their output vectors > <ref:2606.18767#pg3>

Lu: They describe this as a constrained-optimization weight edit that introduces a distractor in vocabulary space by using a rank-one update along a chosen distractor token’s unembedding direction >

Meng: So, they’re not just zeroing things out; they are changing the output vector v i to v'i, which then redirects the residual stream contribution while keeping the activation a i exactly as it was >

Lalam: They also mentioned a process for locating these neurons in two stages, first looking at layer importance and then scoring each neuron by its L1 distance to the post-sequence token >

The paper's improvements: Tom: One of the major practical takeaways they present is that there is a success–locality trade-off across three operating points, like conservative editing with redaction marker at thirty-one point seven percent suppression and near-zero locality cost > <ref:2606.18767#pg3>

Jane: And then they show a practical editing mode, called next-best with k equals five, which achieves eighty-one point five percent suppression while having no catastrophic edits on OLMo-7B >

Lu: They also found that the four edit modes—EOS, redact, next-best, and direct suppression—are complementary; each one succeeds on different sequences where the others might fail >

Meng: But they also identified a mechanistic boundary; approximately fourteen percent of memorized sequences just resist all MLP-based configurations > <ref:2606.18767#pg3,approximately 14% of memorized sequences>

Lalam: And interestingly, they found that if you look at the attention layers, ablating the top contributing attention heads at the target position recovers between sixty and sixty-four percent of those failures > <ref:2606.18767#pg3,ablating the top contributing attention heads at the target position recovers>

Conclusion: Tom: So to wrap up on "Output Vector Editing for Memorization Mitigation in Large Language Models," they show that this weight-level intervention can suppress memorization by modifying what neurons contribute to the residual stream, rather than just whether they are active >

Jane: The implication is that we have a targeted way to reduce privacy and copyright risks without having to overhaul the entire model architecture or training process >

Lu: It really shows how important it is to understand those MLP neurons as knowledge storage because if you know where they are, you can apply this kind of surgical edit >

Meng: For us on the deployment side, the practical editing mode with next-best k equals five seems like the most useful starting point because it gives high suppression without causing massive changes to behavior on OLMo-7B > <ref:2606.18767#pg1>

Lalam: I'm quite confident in this direction; I'd say they have covered a lot of ground by showing this success across four different model sizes and architectures, from 360M up to 7B parameters > <ref:2606.18767#pg1>