Contextual Linear Activation Steering of Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Contextual Linear Activation Steering of Language Models".
Jane: Here's the gist of the paper. Detailed Research Summary: Contextual Linear Activation Steering (CLAS) Contextual Linear Activation Steering (CLAS) is a novel method introduced to dynamically adapt linear activation steering within large language models (LLMs), addressing the critical limitation of using fixed steering strengths in prior approaches.
Tom: The core innovation of CLAS lies in its ability to modulate the amount of steering applied based on both the input context and the specific model block being processed.
Jane: This dynamic adaptation is achieved by utilizing a learned sensing vector (c) that interacts with block-specific steering vectors (d) via an equation such as: h tilde,t = h,t + (c times
h,t, 1: ) d.
Tom: The methodology involves a two-stage learning process.
Paper summary: Tom: So, Jane, this paper is about this new method called Contextual Linear Activation Steering of Language Models. The main idea is that instead of using one fixed steering strength for every part of the model, we can change how much we steer based on what's in the input context and which specific model block we are looking at.
Jane: That sounds really precise, Tom. But before we get into the technical bits, what's the big picture here? What problem is this paper trying to fix with this contextual steering idea?
Tom: Well, existing methods use a fixed steering strength for every token in the model, and that leads to inconsistent steering quality when you have different types of prompts. This paper proposes Contextual Linear Activation Steering, or CLAS, which dynamically adjusts those steering strengths based on the input context and the specific model block.
Lu: What I find interesting is how they achieve this dynamic adjustment. They use something they call a learned sensing vector c sub ell that interacts with block-specific steering vectors d sub ell through an equation like h t tilde equals h t plus c sub ell dot h t one d sub ell <ref:2604.24693#pg1>.
Jane: That sounds like it's adding a layer of context into the calculation. So, how do they actually learn this sensing vector, and what kind of data are they using for that?
Tom: The learning happens in two stages. First, they extract the block-specific steering vectors d sub ell by training an activation probe on a dataset of prompt concept label pairs using something called Recursive Feature Machines or RFM to derive those vectors.
Meng: So, they're using a probe trained on concept labels to get the steering directions for each block. That sounds like a solid starting point if you need task-specific knowledge for each part of the model. But what about learning that context vector?
Lu: For the context-dependent sensing vectors c sub ell, they learn them by minimizing the standard next-token prediction objective on a separate dataset consisting of prompt completion pairs. This means they use a small amount of data to train these vectors, and it requires only one forward backward pass per prompt completion pair to update them.
Paper summary: Jane: That sounds quite efficient for learning those context vectors, especially since it's much lighter than tuning the coefficients in standard linear activation steering methods that rely on an evaluator selecting a fixed coefficient.
Tom: Exactly. They compare their CLAS against standard linear activation steering and two popular parameter-efficient finetuning baselines, ReFT and LoRA, and they show CLAS outperforms both in settings with limited labeled data across eleven diverse steering tasks using four different instruction-tuned language models.
Meng: Eleven tasks is a lot of tests. That’s impressive if it holds up across all those different behaviors they are steering the models toward, like step-by-step reasoning or translation. But I wonder what happens when you try to deploy this at scale.
Lalam: From where I'm standing, the context vector learns how to modulate that linear activation based on the input context and specific model block using a learned sensing vector c sub ell. It enables context-dependent activation steering: h t tilde equals h t plus c sub ell dot h t one d sub ell > <ref:2604.24693#pg1>
Jane: So, for someone just listening to the show, what does that mean in practical terms? What’s the benefit of this contextual modulation over a fixed coefficient?
Tom: It means we can exert more precise and nuanced control over the language model's generation process. Instead of a one-size-fits-all steering direction, you can have fine-grained, context-aware control for things like stylistic constraints or tone modulation without needing extensive full fine tuning.
Lu: The paper also points out that the prompt contains the concept and y equals zero indicating that it does not. They pass the prompts x sub i N i equals one to an LLM and obtain a per-block dataset of LLM activations H l t =
h (one) l t, <ref:2604.24693#pg1>...,h (N) l t: in R k times N.
Meng: That data structure, the dataset of last-token residual stream activations, seems like a lot to manage for production. What does that mean for the computational cost when you're actually running this model?
Lalam: The method is computationally efficient because learning the sensing vectors only requires updating them through minimizing the standard next-token prediction objective on a small dataset of prompt completion pairs. It differs from tuning coefficients in linear activation steering, which uses an evaluator to select a fixed coefficient that produces the best generated outputs over a set of prompts.
Paper summary: Jane: So, we have this efficient learning process for the context vector, and then we have these specific steering vectors derived from RFM for each block. What’s the main point about what this does compared to what ReFT or LoRA can do?
Tom: The paper shows that CLAS consistently outperforms standard linear activation steering and matches or exceeds the performance of ReFT and LoRA in settings where labeled data is limited across eleven steering benchmarks. Plus, those probe-derived steering vectors used by CLAS achieve much higher accuracy for concept monitoring tasks than those learned by ReFT or LoRA.
Lu: The paper highlights that this contextual modulation is shown to enable more precise, scalable, and general control over LLM behavior than existing fixed steering directions. It establishes CLAS as a scalable, interpretable, and accurate technique for specializing and steering large language models by making the steering coefficient context-dependent >
Jane: So, to wrap up on what this means for us right now in the AI space, it seems like we have a method that lets us steer these massive language models with much finer control using only limited labeled data because the steering itself adapts to the context.
Tom: Yeah, it’s about making sure that when you ask an AI something specific, its internal mechanism is guided along the most relevant path for that exact moment in time. It shifts the control from a static setting to a dynamic one based on what's happening right now in the input data stream >
Meng: From my point of view, as someone shipping models to production, I'm still concerned about how robust these context vectors are when faced with completely novel inputs that weren't seen during training. Does this dynamic adaptation hold up for out-of-distribution behavior?
Lalam: The paper shows that the method is scalable because it doesn't require extensive full fine tuning to achieve high accuracy on those tasks >
Jane: That leads perfectly into what we should think about next. This paper by Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan, and Mikhail Belkin is proposing Contextual Linear Activation Steering of Language Models. It seems like a significant step in moving toward more adaptable steering for AI models >
Tom: Exactly. It’s about providing a scalable and interpretable way to specialize these models without needing massive amounts of labeled data for every single task >
Conclusion: Tom: So we've been talking about Contextual Linear Activation Steering, or CLAS, and now we're wrapping up this segment to talk about what this paper actually says and who wrote it.
Jane: The title is Contextual Linear Activation Steering of Language Models. Basically, the authors are showing how you can steer a language model’s internal math based on what the input context is right then.
Tom: Right, so we're moving away from those fixed steering strengths we talked about earlier and showing this dynamic adjustment based on context and model block.
Jane: It’s about making the steering coefficient depend on where you are in the input and which part of the model is processing that information.
Lu: The authors introduce two main things they train. First, block specific steering vectors derived from an activation probe using something called Recursive Feature Machines to get those initial directions for each layer or block.
Meng: And second, context dependent sensing vectors learned by minimizing the standard next-token prediction objective on a separate set of prompt completion pairs. That’s the learning mechanism we talked about earlier, right?
Tom: Exactly. It’s a two stage learning process to get those specific vectors d and then the context dependent ones c.
Jane: So what does this actually change for us if we're building these models? It means we can get much finer control over things like tone or style without needing massive fine tuning.
Tom: It shifts the control from a static setting to something dynamic based on what's happening in the input data stream right now.
Lu: The authors show that CLAS consistently outperforms standard linear activation steering across nearly all model and task pairs, often by substantial margins.
Jane: That’s strong, especially when compared against other methods like ReFT and LoRA, particularly when you have limited labeled data for those eleven steering benchmarks they tested.
Tom: And the paper highlights that the specific steering vectors derived from the probe method achieve much higher accuracy for concept monitoring tasks than those learned by ReFT or LoRA.
Lu: So in essence, CLAS is presented as a scalable and interpretable way to specialize these models by making those steering coefficients context dependent.
Jane: It sets up a lot of questions about how robust these context vectors are when the model sees something completely new it hasn't encountered before during its learning phase.
Tom: That’s the big caveat we need to watch for as we try to deploy this kind of dynamic system in a real production environment.
UC San Diego · MIT · Broad Institute of MIT and Harvard
cs.CL, cs.AI, cs.LG
Submitted: 2026-04-27
Updated: 2026-10-30
Code: https://github.com/bhdsc/CLAS
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Contextual Linear Activation Steering (CLAS) Contextual Linear Activation Steering (CLAS) is a novel method introduced to dynamically adapt linear activation steering within large language models…
Key concepts
- Contextual Linear Activation Steering (CLAS)
- A novel method that changes how a language model is steered by making the steering strength dependent on both the input context and which part of the model is being processed. It uses learned vectors to dynamically adjust the influence of steering directions.
- Block-specific steering vectors ($\mathbf{d}_\ell$)
- These are learned vectors specific to each layer or block within a large language model. They are extracted by training an activation probe on labeled data, helping the system understand how each part of the model responds to different concepts.
- Context-dependent sensing vectors ($\mathbf{c}_\ell$)
- These vectors are learned during training by minimizing prediction errors on prompt-completion pairs. They act as a mechanism to sense and determine how much steering should be applied at a particular point, based on the current context.
Terminology
Summary
Contextual Linear Activation Steering (CLAS)
Contextual Linear Activation Steering (CLAS) is a novel method introduced to dynamically adapt linear activation steering within large language models (LLMs), addressing the critical limitation of using fixed steering strengths in prior approaches. The core innovation of CLAS lies in its ability to modulate the amount of steering applied based on both the input context and the specific model block being processed. p. 1 To address these limitations, we introduce Contextual Linear Activation Steering (CLAS), which modulates
the amount of steering based on the input context and the specific model block using a learned sensing vector
cℓ ∈ R
k+1. This sensing vector enables context-dependent activation steering: h˜
ℓ,t = hℓ,t + (cℓ · [hℓ,t 1])dℓ,
where [hℓ,t 1] ∈ R
k+1 is the concatenation of hℓ,t ∈ R
k with a constant 1, and dℓ ∈ R
k
is an extracted
steering vector. To address these limitations, we introduce Contextual Linear Activation Steering (CLAS), which modulates
the amount of steering based on the input context and the specific model block using a learned sensing vector
cℓ ∈ R
k+1. This sensing vector enables context-dependent activation steering: h˜
ℓ,t = hℓ,t + (cℓ · [hℓ,t 1])dℓ,
where [hℓ,t 1] ∈ R
k+1 is the concatenation of hℓ,t ∈ R
k with a constant 1, and dℓ ∈ R
k
is an extracted
steering vector.Where the paper says it
Where the paper says it
The methodology involves a two-stage learning process. p. 1 This process is described in Section 2. Learning the sensing vectors
for CLAS, which are used to compute the context-dependent steering coefficients, differs from tuning the
fixed steering coefficients in standard activation steering methods. Extracting the steering vectors
with CLAS follows the same procedure as existing activation steering methods [5]. A dataset of prompts and
concept labels is used to train an activation probe at each block, and a direction is extracted from each probe
which serves as the steering vector dℓ. Learning the sensing vectors
for CLAS, which are used to compute the context-dependent steering coefficients, differs from tuning the
fixed steering coefficients in standard activation steering methods. The block-specific sensing vectors cℓ are
trained on a small dataset of prompt-completion pairs using the next-token prediction objective, avoiding
the need to generate and score model outputs. In this work, we form this dataset by pairing prompts3
that do
not contain concept or task-specific information with task-specific completions. We use this dataset and
objective as a practical and compute efficient way to learn the sensing vectors in transformer LLMs, as only
a single forward-backward pass is required to update the sensing vectors per prompt-completion pair.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The primary contribution of CLAS is its ability to provide context-dependent activation steering, which enables more precise and nuanced control over LLM behavior compared to methods relying on fixed coefficients or simple prompting. p. 8 Importantly, it does so
while preserving the interpretability of the activation probe directions, enabling accurate concept monitoring
while making the directions more effective for steering. More broadly, these findings demonstrate that
existing steering directions can become substantially more effective when modulated with context-dependent
coefficients, enabling more precise, scalable, and general control of LLM behavior. 50
Table 3: Accuracy averaged over 10 tasks (excludes JailbreakBench) when steering toward each task separately. CLAS consistently outperforms LAS across nearly all model-task pairs, often by substantial margins despite
occasional underperformance. In this work, we introduce Contextual Linear Activation Steering (CLAS), a method that
dynamically adapts linear activation steering to context-dependent steering strengths. Across eleven
steering benchmarks and four model families, it consistently outperforms standard linear activation
steering and matches or exceeds the performance of ReFT and LoRA in settings with limited labeled
data. We additionally evaluate whether the low-rank directions learned by ReFT and
LoRA encode interpretable concepts by testing them as linear probes on 11 concept monitoring tasks. We
find that they can be used for concept monitoring, but the probe-derived steering vectors used by CLAS
consistently achieve much higher accuracy.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
In essence, CLAS functions as a sophisticated mechanism that leverages contextual information to guide the model's internal computations along specific linear paths. This allows users to exert fine-grained, context-aware control over the language model's generation process—enabling precise steering for stylistic constraints, tone modulation, and adherence to specific instructions without necessitating extensive full fine-tuning. The findings collectively establish CLAS as a scalable, interpretable, and highly accurate technique for specializing and steering LLMs by making the steering coefficient context-dependent. p. 8 Importantly, it does so
while preserving the interpretability of the activation probe directions, enabling accurate concept monitoring
while making the directions more effective for steering. More broadly, these findings demonstrate that
existing steering directions can become substantially more effective when modulated with context-dependent
coefficients, enabling more precise, scalable, and general control of LLM behavior. Importantly, it does so
while preserving the interpretability of the activation probe directions, enabling accurate concept monitoring
while making the directions more effective for steering. More broadly, these findings demonstrate that
existing steering directions can become substantially more effective when modulated with context-dependent
coefficients, enabling more precise, scalable, and general control of LLM behavior.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Contextual Linear Activation Steering (CLAS) dynamically adapts linear activation steering to context-dependent steering strengths using a learned sensing vector cl ∈ Rk+1, which enables context-dependent activation steering: h˜l,t = hl,t + (cl · [hl,t 1])dl. This allows the system to steer model generations toward concept-specific behavior by modulating the extracted steering vector dl based on the input context and specific model block.
-
CLAS outperforms standard linear activation steering and matches or exceeds the performance of ReFT and LoRA in settings with limited labeled data across eleven steering benchmarks, demonstrating a
scalable, interpretable, and accurate method for specializing and steering large language models.
-
The system can achieve higher accuracy in concept monitoring tasks by using probe-derived steering vectors from CLAS compared to low-rank directions learned by ReFT or LoRA, as the paper notes that
probe-derived steering vectors used by CLAS consistently achieve much higher accuracy
on concept monitoring tasks.
Abstract
Linear activation steering is a powerful approach for controlling and specializing large language models by adding linear representations of concepts to their activations at inference. While effective, existing methods often steer with a fixed strength, independent of how much steering any given input requires. In this work, we introduce Contextual Linear Activation Steering (CLAS), which decomposes activation steering into a fixed steering direction and a context-dependent steering coefficient. This decomposition preserves the interpretability of the steering direction while enabling the steering strength to vary with context. It also enables CLAS to perform selective steering: steering multiple behaviors across different portions of a single generation, which cannot be achieved using a fixed steering strength. In addition, across a wide range of steering benchmarks and language models, CLAS consistently outperforms standard linear activation steering and matches or exceeds ReFT and LoRA in settings with limited labeled data. Notably, CLAS achieves this using directions that also serve as reliable concept monitors, a property that directions learned via fine-tuning lack. We therefore propose CLAS as an interpretable and accurate method for controlling and specializing large language models.
Sources
- Understanding intermediate layers using linear classifier probes
- Activation Steering for Chain-of-Thought Compression
- Representation Engineering for Large-Language Models: Survey and Research Challenges
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Training Verifiers to Solve Math Word Problems
- Efficient and accurate steering of Large Language Models through attention-guided feature learning
- The Llama 3 Herd of Models
- Designing and Interpreting Probes with Control Tasks
- Steering Language Models With Activation Engineering
- Extending Activation Steering to Broad Skills and Multiple Behaviours
- Qwen2.5 Technical Report
- A Survey of Large Language Models
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data