Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
This work investigates if large language models (LLMs) can perform meaningful information retrieval when presented with massive, million-token document corpora. Researchers introduced BLOCKSEARCH and developed modifications like SSMax and document-level routing to combat 'attention dilution,' a phenomenon where the model's ability to focus on relevant documents collapses as corpus size increases. The findings show LLMs can retrieve effectively far beyond their training limits.