Thinking Inertia: LLMs Keep Thinking When Told Not To
cs.CL
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: Accepted by NeurIPS 2026. Website: https://thinking-inertia.github.io GitHub: https://github.com/thinking-inertia/code
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: Large Language Models (LLMs) increasingly ship with explicit "thinking modes", yet their counterpart, "no-thinking", has received far less attention.
Terminology
Abstract
Large Language Models (LLMs) increasingly ship with explicit "thinking modes", yet their counterpart, "no-thinking", has received far less attention. We study LLMs' no-thinking behavior along two axes. a. How to measure no-thinking? Prior work typically defines no-thinking through proxies such as a disabled thinking mode or the absence of long traces. These proxies are unreliable: disabled thinking modes may still emit reasoning, while long traces may contain filler rather than genuine inference. We instead normalize each response into a pre-answer trace and final answer, and evaluate it at three levels: (i) Empty-Thinking Rate for strict answer-only compliance; (ii) instruction-aware Question-Pre-answer Relevance for similarity between the question and pre-answer trace; and (iii) LLM-as-judge Explicit Inference Rate for visible explicit inference. Together, these metrics distinguish answer-only output, relevant but non-inferential text, and explicit inference. b. How does no-thinking vary across tasks and models? We evaluate six prompting interventions on six LLMs across Boolean, multiple-choice, and open-ended questions. We find that explicit no-think controls cannot reliably eliminate visible inference. Models instead exhibit "Thinking Inertia": explicit inference persists even under strict controls and becomes more prevalent as the answer space opens. Accuracy remains stable on Boolean and multiple-choice tasks, whereas open-ended tasks reveal a trade-off between answer-only compliance and task accuracy. Rewriting the same questions across answer spaces shows that supplying candidate answers makes answer-only responses easier to produce. These findings establish no-thinking as a non-trivial capability: stopping explicit reasoning cannot be assumed from model settings or instructions alone and deserves systematic evaluation alongside reasoning ability.
Sources
- L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
- Constitutional AI: Harmlessness from AI Feedback
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
- Training Verifiers to Solve Math Word Problems
- DeepSeek-V3 Technical Report
- Thinkless: LLM Learns When to Think
- Token-Budget-Aware LLM Reasoning
- Measuring Mathematical Problem Solving With the MATH Dataset
- Think Only When You Need with Large Hybrid-Reasoning Models
- C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness
- Large Language Models are Zero-Shot Reasoners
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Prompt Repetition Improves Non-Reasoning LLMs
- s1: Simple test-time scaling
- Self-Training Elicits Concise Reasoning in Large Language Models
- Stealing Reasoning Traces from Proprietary LLM APIs
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
- Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering