Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning
cs.CL
Submitted: 2026-02-06
Updated: 2026-10-31
Comments: ICLR 2026 (poster). Public on OpenReview since 2025-10-08: https://openreview.net/forum?id=vndn1Wrult. Code: https://github.com/hhh2210/echoes-as-anchors
Journal ref: The Fourteenth International Conference on Learning Representations (ICLR 2026)
Code: https://github.com/hhh2210/echoes-as-anchors
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent work improves reasoning in large reasoning models (LRMs) by test-time scaling through self-consistency and parallel thinking, adding generic "thinking tokens"…
Terminology
Abstract
Recent work improves reasoning in large reasoning models (LRMs) by test-time scaling through self-consistency and parallel thinking, adding generic "thinking tokens", or presenting the question twice, as in re-reading (RE2). However, these approaches either inject task-agnostic tokens or mandate heuristics that do not explain, and often ignore, the spontaneous repetition that many LRMs exhibit at the head of their long chain-of-thought (CoT). In this work, we investigate the model's tendency to restate the question, which we dub Echo of Prompt (EOP), as a mechanism that shapes computation from the start and can be viewed as prompt repetition inside the reasoning trace rather than in the input. We formalize its probabilistic cost by casting echo removal as rejection sampling, and define the Echo Likelihood Gap as a computable proxy linking early repetition to likelihood gains and downstream accuracy. We find that (1) in a causal intervention, reinserting an echo into failed GSM8K traces raises exact match by 7.9-10.4 points for two reasoning models but not for a non-reasoning base model; and (2) likelihood analyses controlling for length and suffix indicate that EOP's benefits go beyond verbosity, while layer-wise attention studies show that EOP increases attention from answer tokens to the answer prefix in middle layers, consistent with an attention refocusing mechanism. Building on these findings, we develop Echo-Distilled SFT (ED-SFT) to instill echoing before reasoning via supervised fine-tuning, and training-free Echoic Prompting (EP) to ground the model on the prompt midway through the trace. We evaluate on GSM8K, MathQA, Hendrycks-MATH, AIME24, and MATH-500 under identical decoding settings and budgets, and observe gains over baselines, including the insertion of thinking tokens, in most settings. Our code is publicly released at https://github.com/hhh2210/echoes-as-anchors.
Sources
- Focusing Attention: Towards Accurate Text Recognition in Natural Images
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
- Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
- Qwen3 Technical Report
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
- Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data