Coherence boosting: When your pretrained language model is not paying enough attention

arXiv:2110.08294 · cs.CL · Submitted 2021-10-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Coherence boosting: When your pretrained language model is not paying enough attention".

Paper summary: Tom: So we're looking at this paper called "Coherence boosting: When your pretrained language model is not paying enough attention," and the main idea is that large language models don't really learn how distant words affect what they predict next.

Jane: It claims that long-range semantic coherence is still a challenge in automatic language generation and understanding, and they show that these models are insufficiently trained to pay enough attention to those distant words >

Lu: The authors present this procedure called coherence boosting as an inference procedure designed to increase a language model's focus on a long context >

Tom: So what does this actually do?

Jane: Well, it treats the pretrained model like an ensemble of experts that make token predictions based on different context lengths, and these different contexts are mixed together to form a predictor that is better than the base model >

Meng: But if this is just an inference trick, what does it cost us at scale?

Tom: The paper suggests it's applicable in both generation and ranking settings >

Lu: They hypothesize that the failures come from modeling choices and distribution shift, specifically because autoregressive language models are fitting a multi-objective problem simultaneously maximizing token likelihoods conditioned on many lengths of truncated context >

Jane: That means they think the base models are trying to do too much at once, so this boosting method conditions the likelihoods on the entire prompt or previously generated string >

Lalam: The coherence boosting procedure is defined by a weighted product of experts model, where you define a new language model f alpha as the softmax sum over k from one to M alpha k log f k (w1,..., wn; theta) >

Tom: That's a lot of moving parts. So where do we see the actual gains in this "coherence boosting" for next-token prediction?

Jane: They show benefits through distributional analyses of generated ordinary text and dialog responses >

Meng: Are these gains just theoretical, or can we actually deploy something that uses this without massive retraining?

Lalam: They found that coherence boosting with state of the art models for various zero shot NLP tasks yields performance gains with no additional training >

Tom: Okay, so it's not about retraining the whole model then.

Jane: Exactly. The paper shows that in experiments on the LAMBADA dataset, a boosted GPT three hundred seventeen five billion achieves a new state of the art >

Lu: They found that the optimal alpha k is always negative, which means the optimal mixture of models penalizes short range context relative to long range context >

Tom: So how does model size relate to this?

Jane: With increasing model size, the optimal alpha k and k get closer to zero, suggesting bigger models capture long range coherence better >

Paper summary: Meng: That makes sense if you think about what a huge model is supposed to do.

Lu: They also proposed something called coherence tuning as an alternative to modifying the weights of a pretrained language model by iterating training steps >

Tom: Coherence tuning sounds like it would be a way to get these improvements without needing new data for finetuning.

Jane: It involves generating a sequence, computing next token distributions under the coherence boosted version and the current model without boosting, and then performing a gradient step on the difference between those two distributions >

Lalam: This coherence tuning code runs in just a few minutes on modern hardware, which really amortizes the overhead cost of coherence boosting >

Tom: So, if we look at generation, what does this mean for creating generic text or dialog responses?

Jane: In generation of generic text and dialog responses, coherence boosting brings the frequency of occurrence of such words close to that seen in natural text >

Lu: The authors found that coherence boosting is applied across various tasks including generic text generation and multiple choice problems where examples are known to be highly coherent >

Meng: I see they tested it on summarization too. What about those metrics?

Lalam: In text summarization, the proposed coherence boosting method improves most of the metrics on both CNN and DM and XSum datasets >

Tom: So to wrap up this paper, what's the big picture for us as we look at how models handle long context?

Jane: The study concludes that coherence boosting is a simple inference time remedy and can be used as a simple alternative or complement to finetuning procedures in zero shot applications of pretrained language models >

Lu: They stress that the optimal alpha depends on tasks and models, which means you can't just set alpha to zero or negative one as in past work >

Tom: It sounds like we need to search for that optimal alpha rather than using some fixed rule.

Jane: The authors also highlight that coherence boosting corrects the bias introduced by top p truncation without sacrificing the gains in other measures >

Meng: I'm curious about what they didn't cover. Where does this method break down?

Lalam: They mention that the optimal alpha depends on tasks and models, suggesting a need to search for it rather than heuristic settings like zero or negative one >

Tom: So, the main implication is that we don't need massive retraining to fix this long range coherence issue in current models.

Jane: The paper suggests future work should consider other architectures and target domains where a strong causal link between prompt and output is known to exist >

Conclusion: Tom: So we’ve been talking about this paper, "Coherence boosting: When your pretrained language model is not paying enough attention." It’s about how these big language models struggle to connect distant words in a prompt.

Jane: Right, the core idea is that if a model doesn't pay attention to what's far away in the text it might totally mess up its next word prediction.

Lu: They introduce this inference procedure called coherence boosting, which basically makes the language model focus more on that long context when it's actually generating something.

Meng: So, if I’m deploying this, is it just another thing to add overhead to my system or does it actually give a noticeable lift in output quality?

Lalam: It seems like they found that you can get performance gains on zero shot tasks without needing any extra training on the model itself.

Tom: That's what really interests me—gains with no additional training. But how do we know this isn't just masking a different problem, you know?

Jane: The paper points out that the mechanism is rooted in modeling choices and how the model was trained, specifically because it maximizes likelihoods across many different context lengths at once.

Lu: They frame it as an ensemble of experts where each expert handles a certain length of context, and coherence boosting mixes those predictions together using a weighted product-of-experts model.

Meng: That’s interesting about the mixing part. Does that mean we can just plug in different models for different lengths? What are the practical costs there?

Lalam: They show that by adjusting these weights, you can tune it precisely, and they even have this coherence tuning algorithm which lets you get closer to that boosted performance without needing new training data.

Tom: So, if we look at the title again, "When your pretrained language model is not paying enough attention," it really hammers home the idea that the problem isn't a lack of intelligence but a specific failure in handling distance.

Jane: Exactly. It suggests that we don't need to rebuild these massive models from scratch; sometimes we just need to fine-tune how they look at their inputs during the prediction process.

Lu: The implication is that this simple inference-time fix could be a really practical way for us to improve zero shot applications of these models without getting bogged down in expensive training cycles.

Meng: It sounds like a low-cost intervention, which is good news for production deployment, provided the setup isn't overly complex to maintain.

Lalam: They emphasize that the optimal setting depends on the specific task and model you are using, so there isn't one magic number like zero or negative one that everyone should just use.

Tom: That variability in what works—that’s a key finding. It tells us we need to search for the right alpha value instead of just guessing a default setting.

Jane: And that leads into where they suggest looking next, which is exploring other model architectures and domains where we know there's a really strong connection between the prompt and the desired output. (Transition music starts)

Nikolay Malkin, Zhen Wang, Nebojsa Jojic

Université de Montréal · Ohio State University · Microsoft Research

cs.CL

Submitted: 2021-10-15

Updated: 2026-11-18

Comments: ACL 2022; code: https://github.com/zhenwang9102/coherence-boosting

DOI: 10.18653/v1/2022.acl-long.565

Code: https://github.com/zhenwang9102/coherence-boosting

License: http://creativecommons.org/licenses/by/4.0/

The gist: Long-range semantic coherence remains a challenge in automatic language generation and understanding When your pretrained language model is not paying enough attention…

Key concepts

Coherence Boosting
An inference procedure that increases a language model's attention to long contexts. It works by defining a new model as a weighted product-of-experts, mixing predictions from the base model evaluated at different context lengths to better capture distant word dependencies.
Product-of-Experts Model
A framework where the language model is viewed as an ensemble of experts, each producing a token distribution conditioned on a specific length of context. These expert distributions are then log-linearly mixed using weights to form a final predictor that is superior to the base model.
Coherence Tuning
An alternative method to finetuning. It involves generating sequences and comparing next-token distributions between the coherence-boosted version and the original model, then performing a gradient step on their KL divergence. This adjusts the model's parameters toward a more coherent state without needing task-specific training data.

Terminology

Summary

Long-range semantic coherence remains a challenge in automatic language generation and understanding When your pretrained language model is not paying enough attention, large language models have insufficiently learned the effect of distant words on next-token prediction We present coherence boosting, an inference procedure that increases a LM’s focus on a long context It is also found that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training

Pruning redundant reasoning tokens

The paper introduces coherence boosting as a simple inference-time procedure applicable in both generation and ranking settings This procedure increases the effect of distant words on predicted token distributions A pretrained model is viewed as an ensemble of experts that produce token distributions conditioned on varying lengths of context, which are log-linearly mixed to form a predictor superior to the base model Coherence boosting greatly improves prediction of words that depend on a long context In generation of generic text and dialog responses, coherence boosting brings the frequency of occurrence of such words close to that seen in natural text

How the training signal is built

The authors hypothesize that failures arise from modeling choices and distribution shift, specifically because autoregressive LMs are fit to a multi-objective problem simultaneously maximizing token likelihoods conditioned on many lengths of truncated context At generation or scoring time, likelihoods are conditioned on the entire prompt or previously generated string, specifically selected to be coherent or even guaranteed to influence the output The coherence boosting procedure is defined by a weighted product-of-experts model where the new language model f alpha is defined by f alpha (w1,..., wn; theta):= softmax sum over k=1 to M alpha k log f k (w1,. p. 2

Where the paper says it

, 𝛼𝑀 ) and produces a new language model 𝑓𝜶, defined by 𝑓𝜶 (𝑤1, . , 𝑤𝑛; 𝜃) := softmax ∑︁ 𝑀 𝑘=1 𝛼𝑘 log 𝑓𝑘 (𝑤1, .

p. 2
Where the paper says it

, 𝛼𝑀 ) and produces a new language model 𝑓𝜶, defined by 𝑓𝜶 (𝑤1, . , 𝑤𝑛; 𝜃) := softmax ∑︁ 𝑀 𝑘=1 𝛼𝑘 log 𝑓𝑘 (𝑤1, .

p. 2
Where the paper says it

, 𝛼𝑀 ) and produces a new language model 𝑓𝜶, defined by 𝑓𝜶 (𝑤1, . , 𝑤𝑛; 𝜃) := softmax ∑︁ 𝑀 𝑘=1 𝛼𝑘 log 𝑓𝑘 (𝑤1, .

.., wn; theta)

Where the gains come from

The performance of coherence boosting is demonstrated through distributional analyses of generated ordinary text and dialog responses In experiments on the LAMBADA dataset, the boosted GPT-3 175B achieves a new state of the art The optimal alpha k is always negative, indicating that the optimal mixture of models penalizes the influence of short-range context relative to long-range context With increasing model size, the optimal alpha k and k become closer to 0, suggesting bigger models capture long-range coherence better

The paper also proposes coherence tuning as an alternative to modifying the weights of a pretrained LM by iterating training steps Coherence tuning involves generating a sequence, computing next-token distributions under the coherence-boosted version and the current model without boosting, and performing a gradient step on KL(f alpha (w1..., wk; theta) k f (w1..., wk; theta)) This coherence tuning code runs in a few minutes on modern hardware, amortizing the overhead cost of coherence boosting while achieving comparable results on the WebText article completion task

Coherence boosting is applied across various tasks, including generic text generation and multiple-choice problems where examples are known to be highly coherent In dialog systems, coherence boosting with an appropriate alpha leads to responses that are more relevant to the prompt (higher BLEU and NIST) and more diverse than those from greedy decoding For text summarization, the proposed CB method improves most of the metrics on both CNN/DM and XSum datasets The results show that coherence boosting brings all metrics closer to those of human text

The study concludes that coherence boosting is a simple inference-time remedy and can be used as a simple alternative or complement to finetuning procedures in zero-shot applications of pretrained LMs The optimal alpha depends on tasks and models, suggesting the necessity of searching for an optimal alpha rather than heuristic settings like 0 or -1 The paper suggests future work should consider other architectures and target domains where a strong causal link between prompt and output is known to exist

The authors highlight that coherence boosting corrects the bias introduced by top-p truncation without sacrificing the gains in other measures The performance comparison across multiple models and inference methods shows that coherence boosting generally improves accuracy on all tasks In summary, coherence boosting illustrates the hyposensitivity of pretrained language models to long-range content and proposes a simple inference-time remedy

The paper provides extensive experimental results across multiple datasets and model sizes, showing dramatic performance gains for smaller models on certain tasks The optimal alpha is usually negative, and its absolute value tends to decrease with the model size This finding suggests that bigger models capture long-range coherence better than small models The results confirm that coherence boosting brings all metrics closer to those of human text

The authors suggest three promising research directions including coherence tuning, new domains and architectures, and efficient search proposals for non-autoregressive text generation algorithms The work contributes a simple inference-time remedy to address long-range semantic coherence in automatic language generation The paper is 2110.08294.

--- Page 5 ---

The abstract states that long-range semantic coherence remains a challenge in automatic language generation and understanding We demonstrate that large language models have insufficiently learned the effect of distant words on next-token prediction We present coherence boosting, an inference procedure that increases a LM’s focus on a long context We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses It is also found that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training

--- Page 6 ---

The paper states that the authors hypothesize that these failures arise from modeling choices and distribution shift, specifically because autoregressive LMs are fit to a multi-objective problem simultaneously maximizing token likelihoods conditioned on many lengths of truncated context Yet, at generation or scoring time, likelihoods are conditioned on the entire prompt or previously generated string, specifically selected to be coherent or even guaranteed to influence the output The two common solutions – finetuning models on one or multiple tasks (Khashabi et al., 2020; Sanh et al. p. 0

Where the paper says it

The two common solutions – finetuning models on one or multiple tasks (Khashabi et al. , 2020; Sanh et al.

, 2022) and improving models or prompts to facilitate in-context learning (Brown et al. p. 0
Where the paper says it

, 2020; Sanh et al. , 2022) and improving models or prompts to facilitate in-context learning (Brown et al.

, 2020; Schick and Schütze, 2019) – do not directly target the problem of longrange coherence

--- Page 7 ---

The paper discusses early neural language models used ‘backing-off’ schemes that interpolate between predictors with different context lengths Neural language modeling brought a need for recurrent units with better numerical properties for propagating information over long distances Attention is at the core of Transformer LMs, including GPT Language models are being trained on and adapted to ever-longer input sequences, but they remain undersensitive to distant content or syntax

--- Page 8 ---

The paper discusses the training objective for the model f as minimizing the average negative log-likelihood (NLL) of all words following the parts of a sequence that precede them This situation is different from that seen at generation time, where only the ensemble member using full context is used The authors expect sensitivity to long context, and precision in modeling its effect, to be especially diminished if the model is undertrained

--- Page 9 ---

The paper details the coherence boosting for next-token prediction by defining a new language model f alpha (w1,..., wn; theta):= softmax sum over k=1 to M alpha k log f k (w1,..., wn; theta) This is a weighted product-of-experts model where the ‘experts’ are copies of the base model f evaluated on different context lengths In multiple-choice problems, coherence boosting chooses a parameter alpha and ranks the choices by log f (ai full context; theta) + alpha log f (ai premise-free context; theta)

--- Page 10 ---

The paper discusses the mathematical formulation of coherence boosting for answer selection by defining a log-linear combination of two models: f evaluated with full context and with a partial context When alpha = -1, it is equivalent to dividing the base model’s score by the score of each answer conditioned on the prompt (short context), and thus to maximizing pointwise mutual information between the premise and the answer conditional on the premise-free context

--- Page 11 ---

The paper discusses why boosting models are better than full-length predictors because they fit all of multi-objective training simultaneously The authors show that the short-context predictors are easier to fit, while sequences in which long context affects the prediction are rare, suggesting that sensitivity to long context is diminished if the model is undertrained

--- Page 12 ---

The paper presents experimental results on LAMBADA where coherence boosting vastly reduces prediction error for all models The optimal alpha k is always negative, indicating that the optimal mixture of models penalizes the influence of short-range context relative to long-range context With increasing model size, the optimal alpha k and k become closer to 0 The results show that coherence boosting brings all metrics closer to those of human text

--- Page 13 ---

The paper shows that the optimal alpha depends on tasks and models, which means that alpha cannot be heuristically set to 0 or -1 as in past work The results also indicate that the optimal alpha is usually negative, and its absolute value tends to decrease with the model size

--- Page 14 ---

The paper presents a coherence tuning algorithm which brings the LM closer to its coherence-boosted version without training on any specialized data This algorithm involves generating a sequence, computing next-token distributions under the coherence-boosted version and the current model without boosting, and performing a gradient step on KL(f alpha (w1..., wk; theta) k f (w1..., wk; theta)) Coherence tuning code runs in a few minutes on modern hardware, amortizing the overhead cost of coherence boosting while achieving comparable results on the WebText article completion task

--- Page 15 ---

The paper shows that a sparse log-linear mixtures can be a better predictor than f max alone if the difference between prediction losses L M and L k is greater than the average KL divergence between the predictions f M and f k The coherence tuning algorithm achieves this by showing that the boosted model proportional to f(1+t) M f-t k is a better predictor than f max alone under certain conditions

--- Page 16 ---

The paper presents preliminary experiments for zero-shot abstractive summarization by applying coherence boosting to GPT-2 models The CB method improves most of the metrics on both CNN/DM and XSum datasets In the CNN/DM dataset, CB yields improvements of up to ∼3 ROUGE points

--- Page 17 ---

The paper shows that long-range information in the first few sentences is hard to capture for GPT models, and this difficulty can be alleviated by coherence boosting The output with coherence boosting is largely free of these issues, maintaining focus on limb length and diet

--- Page 18 ---

The paper shows that the optimal alpha depends on tasks and models, which means that alpha cannot be heuristically set to 0 or -1 as in past work The results confirm that coherence boosting brings all metrics closer to those of human text

--- Page 19 ---

The paper notes that the performance with GPT models presented here is far lower than that of state-of-the-art summarization-specific models, such as finetuned BART models The authors hope to see coherence boosting used as a simple alternative or complement to finetuning procedures in zero-shot applications of pretrained LMs

--- Page 20 ---

The paper shows that the optimal alpha depends on tasks and models (fourth column within each model), which means that alpha cannot be heuristically set to 0 or-1 as in past work The results confirm that coherence boosting brings all metrics closer to those of human text

--- Page 23 ---

The paper summarizes the findings by illustrating the hyposensitivity of pretrained language models to long-range content and proposes a simple inference-time remedy We hope to see coherence boosting used as a simple alternative or complement to finetuning procedures in zero-shot applications of pretrained LMs

--- Page 23 ---

The paper notes that the performance with GPT models presented here is far lower than that of state-of-the-art summarization-specific models, such as finetuned BART models We hope to see coherence boosting used as a simple alternative or complement to finetuning procedures in zero-shot applications of pretrained LMs

--- Page 9 ---

The paper details the coherence boosting for next-token prediction by defining a new language model f alpha (w1,..., wn; theta):= softmax sum over k=1 to M alpha k log f k (w1,...

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Coherence boosting for next-token prediction increases focus on long context by defining a new language model f alpha defined by f alpha (w1... w n; theta):= softmax sum M k=1 alpha k log f k (w1... w n; theta). This weighted product-of-experts model, using log-linear boosting, improves prediction of words that depend on a long context.

  2. Coherence boosting for answer selection uses a log-linear combination of two models: f evaluated with full context and with a partial context (short context), defined by log f (a i full context; theta) + alpha log f (a i premise-free context; theta). This allows ranking choices by maximizing the conditional likelihood of the answer following the prompt.

  3. Coherence tuning is proposed as an inference-time remedy that modifies weights of a pretrained LM to increase effect of distant words without training on specialized data. This algorithm iterates by computing next-token distributions under both the coherence-boosted version and current model without boosting, applying a gradient step on KL(f alpha (w1... w k; theta) k f (w1... w k; theta)).

  4. The improved AI system can generate text with higher long-range semantic coherence, as evidenced by the ability of the boosted GPT-2 Small to perform better than the original GPT-3 2.7B on LAMBADA and achieve a new state of the art on various zero-shot NLP tasks.

  5. The system can generate text for generic articles with metrics like LRn, LTF, and delta that are closer to human-written text, as shown by coherence boosting bringing all metrics closer to those of human text in Table 2.

Abstract

Long-range semantic coherence remains a challenge in automatic language generation and understanding. We demonstrate that large language models have insufficiently learned the effect of distant words on next-token prediction. We present coherence boosting, an inference procedure that increases a LM's focus on a long context. We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses. It is also found that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training.

Sources

Related papers