Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models
cs.CL, cs.AI
Submitted: 2026-02-02
Updated: 2026-02-02
Journal ref: 2026.emnlp-main.2168
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) often struggle with deductive judgment in syllogistic reasoning, systematically conflating semantic plausibility with formal validity a phenomenon known as content effect.
Terminology
Abstract
Large Language Models (LLMs) often struggle with deductive judgment in syllogistic reasoning, systematically conflating semantic plausibility with formal validity a phenomenon known as content effect. This bias persists even when models generate step-wise explanations, indicating that intermediate rationales may inherit the same semantic shortcuts that affect answers. Recent approaches propose mitigating this issue by increasing inference-time structural constraints, either by encouraging abstract intermediate representations or by intervening directly in the model's internal computations; however, reliably suppressing semantic interference remains an open challenge. To make formal deduction less sensitive to semantic content, we introduce a framework for abstraction-guided reasoning that explicitly separates structural inference from lexical semantics. We construct paired content-laden and abstract syllogisms and use the model's activations on abstract inputs to define an abstract reasoning space. We then learn lightweight Abstractors that, from content-conditioned residual-stream states, predict representations aligned with this space and integrate these predictions via multi-layer interventions during the forward pass. Using cross-lingual transfer as a test bed, we show that abstraction-aligned steering reduces content-driven errors and improves validity-sensitive performance. Our results position activation-level abstraction as a scalable mechanism for enhancing the robustness of formal reasoning in LLMs against semantic interference.
Sources
- Transformers as Soft Reasoners over Language
- Language models show human-like content effects on reasoning tasks
- Could Thinking Multilingually Empower LLM Reasoning?
- Gemma 3 Technical Report
- Gemma 2: Improving Open Language Models at a Practical Size
- Transformer Feed-Forward Layers Are Key-Value Memories
- Measuring Massive Multitask Language Understanding
- Understanding How CodeLLMs (Mis)Predict Types with Activation Steering
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
- Mistral 7B
- The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks
- Qwen3 Technical Report
- Qwen2.5 Technical Report
- Language Models are Multilingual Chain-of-Thought Reasoners
- Interpretable Steering of Large Language Models with Feature Guided Activation Additions
- Steering Language Models With Activation Engineering
- Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering
- SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Tasks
- Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
- BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
- Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking