Contrastive Retrieval Heads for Improved Attention-Based Reranking
cs.IR
Submitted: 2025-10-02
Updated: 2026-10-29
Code: https://github.com/linhhtran/CoRe-Reranking
License: http://creativecommons.org/licenses/by/4.0/
The gist: The strong zero-shot and long-context capabilities of recent Large Language Models (LLMs) have paved the way for highly effective re-ranking systems.
Terminology
Abstract
The strong zero-shot and long-context capabilities of recent Large Language Models (LLMs) have paved the way for highly effective re-ranking systems. Attention-based re-rankers leverage attention weights from transformer heads to produce relevance scores, but not all heads are created equally: many contribute noise and redundancy, thus limiting performance. To address this, we introduce CoRe heads, a small set of retrieval heads identified via a contrastive scoring metric that explicitly rewards high attention heads that correlate with relevant documents, while downplaying nodes with higher attention that correlate with irrelevant documents. This relative ranking criterion isolates the most discriminative heads for re-ranking and yields a state-of-the-art list-wise re-ranker. Extensive experiments with three LLMs show that aggregated signals from CoRe heads, constituting less than 1% of all heads, substantially improve re-ranking accuracy over strong baselines. We further find that CoRe heads are concentrated in middle layers, and pruning the computation of final 50% of model layers preserves accuracy while significantly reducing inference time and memory usage.
Sources
- Phi-4 Technical Report
- Granite Embedding Models
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Extending Context Window of Large Language Models via Positional Interpolation
- NV-Retriever: Improving text embedding models with effective hard-negative mining
- Data Engineering for Scaling Language Models to 128K Context
- Retrieval-Augmented Generation for Large Language Models: A Survey
- LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
- Holistic Evaluation of Language Models
- Zero-Shot Listwise Document Reranking with a Large Language Model
- In-context Learning and Induction Heads
- Representation Learning with Contrastive Predictive Coding
- CLIPPER: Compression enables long-context synthetic data generation
- Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
- Focus Directions Make Your Language Models Pay More Attention to Relevant Contexts
Related papers
- UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA