Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
Researchers tested how Large Language Models (LLMs) retrieve information based on time and position rather than just meaning. They found models consistently prefer recalling tokens near the beginning or end of a prompt, showing human-like primacy and recency effects. This behavior is driven by internal mechanisms like 'induction heads' and suggests that fundamental sequential processing limitations affect both Transformer and State-Space Models equally.