SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics".

Jane: Here's the gist of the paper. The gist The study investigates whether steering neural metrics toward an English pivot language can improve their correlation with human judgments, finding that test-time intervention methods are effective across diverse languages in summarization evaluation.

Tom: An increasing body of work has leveraged multilingual language models for Natural Language Generation tasks such as summarization, but a major empirical bottleneck is the shortage of accurate and robust evaluation metrics for many languages, which hinders progress A major empirical bottleneck in this area is the shortage of accurate and robust evaluation metrics for many languages, which hinders progress Recent studies suggest that multilingual language models often use English as an internal pivot language, and that misalignment with this pivot can lead to degraded downstream performance Motivated by the hypothesis that this mismatch could also apply to multilingual neural metrics, we ask whether steering their activations toward an English pivot can improve correlation with human judgments The researchers explicitly asked the research question: To what degree does mapping the activations of low-resource languages to English strengthen multilingual metric performance They focused on summarization evaluation and conducted a meta-analysis of model-based summarization metrics across eight diverse languages, examining several multilingual LLM-as-judge approaches and an encoder-based metric Two steering techniques were evaluated to improve model-based evaluation, which has not been explored before 1.

Jane: Vector-based intervention involves computing vectors from a lower-resourced (source) to a higher-resourced (target) language using parallel data, resulting in one vector per layer that represents the source to target language direction At test time, they add these vectors to activations in each corresponding layer, defined by hmix q,l = hs + ρh¯v where ρ represents the strength and direction of the steering 2.

Tom: Map-based intervention learns a linear mapping Wl between a lower-resourced (source) and a higher-resourced (target) language using parallel data to align representations by minimizing their distance The mapping is then used to project the representations in the source language into the direction of the target language at inference time as in hˆt q,l = W∗l h s q,l Experimental Setup and Metrics They examined two LLM-as-a-judge metrics and one encoder-based metric Direct prompting involves directly prompting a multilingual model to produce a numerical score on a 1–5 scale for each evaluation dimension, computed as a weighted sum over the token probabilities corresponding to the evaluation scale at the first generated token GPTScore evaluates a hypothesis using the model’s conditional generation probabilities by computing the average log-probability of its tokens under the model, conditioned on the prompt COMET is an encoder-based metric that they adapted for summarization by replacing the source input with an empty string and using system-generated summaries as hypotheses Key Findings on Steering Effectiveness The results provide strong evidence that steering improves the performance of model-based metrics Steering almost always improves correlation over the unsteered baseline, often substantially so, especially in settings where baseline performance is low The magnitude of the gains, however, can vary considerably across steering method, metrics, and languages Vector steering tends to yield larger gains, especially for COMET and for low-performing language–metric pairs Map-based steering often shows significant improvements in correlation with human judgments across both coherence and completeness dimensions For instance, Table 9 reports the results when using French as a target language instead of English for the ayaexpanse 8B model, showing large relative improvements for both Vector and Map steering Across metrics, languages with weaker baseline performance tend to benefit the most from steering Language-Specific Dynamics The study analyzed how language-specific are language vectors and steering factors They computed the cosine similarity between Language X → English vectors for each layer, observing that for every vector pair except Yoruba, the cosine similarity is relatively high, which is in line with prior findings where refusal direction generalizes to most languages except Yoruba For Map-based steering, larger values of σ generally lead to higher mean relative improvements For Vector-based steering, negative values of ρ generally outperform positive ones in aggregate, with ρ = −5 achieving the highest mean relative improvement The findings highlight the effectiveness of test-time intervention methods for neural metrics and contribute to the growing body of work that leverages model-internal insights for practical scenarios Future work could investigate whether language- and dimension-specific interventions can be combined for multilingual evaluation The study confirms that their interventions can improve the evaluative behavior of both encoderbased metrics and LLM-based evaluators The overall results provide strong evidence that steering improves the performance of model-based metrics Limitations One limitation is the smaller count of samples and human annotators in certain languages for their primary dataset Given the lack of a development set, they did not tune the ρ and σ parameter, reporting the best value per language instead While they performed experiments using French as a target language, specific source-target pairs might be preferable for future work The ethical statement acknowledges the usage of LLMs in the partial assistance of figure creation, code development, and in strengthening the coherence of parts of the manuscript The acknowledgments thank members of the MaiNLP lab for their valuable feedback and support through an ERC Consolidator Grant DIALECT 101043235 References Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda.

Title and authors: Jane: So, let's talk about the title and who wrote it. It’s "SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics."

Tom: The core idea is that they use inference-time interventions to make multilingual metrics better for generalization across languages.

Lu: They are specifically looking at summarization evaluation, and they're testing how much mapping the activations of low-resource languages to English strengthens their performance.

Meng: So, this isn't just about making one metric work better; it’s about improving the reliability of many different metrics when you use them on a variety of languages.

Lalam: I think what this means is that if we can get these metrics to align with how humans judge things more accurately across languages, then the models using those metrics are actually more trustworthy.

Tom: Exactly. And they're testing this across eight diverse languages using several different approaches, including both model-based and encoder-based metrics.

Jane: It sounds like the authors are saying that their initial findings show that current LLM-as-judge metrics have only modest and inconsistent correlation with human judgments on summarization tasks.

The paper's summary: Lu: The paper summarizes that they explicitly ask how much mapping low-resource language activations to English strengthens multilingual metric performance.

Tom: They set up a meta-analysis of model-based summarization metrics across eight diverse languages, looking at both LLM-as-judge approaches and an encoder-based metric.

Jane: What’s interesting is that they found that current LLM-based metrics show only modest and inconsistent correlation with human judgments, which sets the stage for their intervention experiments.

Meng: And they then investigate whether steering these multilingual model-based metrics toward a high-resource pivot language at test time can improve their reliability.

Lalam: They found that test-time intervention methods are effective across the board, increasing metric effectiveness for diverse languages, according to the abstract.

Tom: So the core finding is that simply mapping those internal activations to English during inference makes these metrics more reliable when they're applied to different languages.

The paper's improvements: Jane: They suggest two specific steering techniques: vector-based intervention and map-based intervention.

Lu: Vector-based intervention involves computing vectors from a lower-resourced source language to a higher-resourced target language using parallel data, resulting in one vector per layer that represents the direction of that translation.

Tom: And then at test time, you add these vectors to the activations in each corresponding layer by some strength and direction parameter rho.

Meng: That sounds like a specific math operation we need to figure out how much overhead it adds to our inference pipeline.

Jane: Vector steering tends to yield larger gains, especially for COMET and for low-performing language-metric pairs, which is an important detail.

Lu: Map-based intervention is different; it learns a linear mapping W between the source and target language representations using parallel data to align them by minimizing their distance.

Tom: Then at inference time, you use that learned mapping to project the source representation into the direction of the target language, which they call h hat q,l = W*l h s q,l.

Jane: Map-based steering showed significant improvements in correlation with human judgments for both coherence and completeness dimensions in some cases.

Conclusion: Tom: So to wrap up, the paper shows that steering activations at inference time can improve the correlation of model-based metrics with human judgments across different languages.

Jane: It seems they've demonstrated that mapping low-resource language representations toward an English pivot at test time is a way to boost metric performance.

Lu: This confirms that the approach of using parallel data to learn cross-lingual vectors or mappings is a viable method for steering in these contexts, and it builds on prior work like Wang et al. 2025a <ref:2601.15809#pg2>.

Meng: From a deployment standpoint, it's good because it's inference-time only and doesn't require retraining the massive language models they use.

Lalam: I just think this is really important because it gives us a concrete tool to address that misalignment issue they talked about—where the pivot language mismatch degrades performance.

Tom: Exactly. This work, "SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics," confirms that test-time interventions can actually improve how we evaluate multilingual models when we're looking at things like summarization.