When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models
cs.AI
Submitted: 2026-01-22
Updated: 2026-09-25
Comments: 34 pages, 11 figures
Code: https://github.com/HaoranZhengRaul/AMOR
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recurrent-attention hybrids aim to combine the efficiency of recurrence with the contextual recall of attention, but existing approaches typically apply attention uniformly across all positions, even
Terminology
Abstract
Recurrent-attention hybrids aim to combine the efficiency of recurrence with the contextual recall of attention, but existing approaches typically apply attention uniformly across all positions, even when the recurrent state alone is sufficient for accurate prediction. We introduce AMOR (Adaptive Metacognitive Output Router), a post-hoc hybrid architecture that selectively invokes attention based on predictive uncertainty. A recurrent backbone is augmented with entropy-gated attention blocks that activate only when the model's output entropy exceeds a dynamic threshold derived from a running batch median and scaled standard deviation. The resulting binary gate requires no learned routing parameters. Pretrained from scratch on FineWeb-Edu and with attention invoked on only 40% of positions, one of the AMOR variants (Mamba2 or Gated DeltaNet backbones) achieves the highest eight-task common-sense reasoning average at each scale among pure recurrent, pure attention, and fixed-schedule hybrid models. AMOR also improves retrieval performance over pure recurrent models while remaining competitive against fixed-schedule hybrids. Additionally, AMOR retains the long-context robustness of its recurrent backbones, where the Transformer and other hybrid architectures degrade under distribution shift. These results suggest that when attention is applied matters as much as how much: selectively allocating attention based on predictive uncertainty improves accuracy, robustness, and efficiency, offering a simple alternative to uniform or fixed routing strategies.
Sources
- Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes
- Simple linear attention language models balance the recall-throughput tradeoff
- Just read twice: closing the recall gap for recurrent language models
- Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Language Models are Few-Shot Learners
- Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Hymba: A Hybrid-head Architecture for Small Language Models
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model
- SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
- The Zamba2 Suite: Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems