Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers
summary
The gist
Vote-in-Context (ViC) introduces a generalized, training-free framework that rethinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM).
In short
Vote-in-Context (ViC) transforms Vision-Language Models (VLMs) into zero-shot rank fusers by treating list reranking as a reasoning task. It serializes content evidence and retriever metadata directly into the VLM prompt, allowing the model to adaptively weigh consensus across multiple lists per query. This enables powerful, training-free fusion for video retrieval.
Key concepts
- Vote-in-Context (ViC)
- ViC is a framework that rethinks list reranking and fusion as a zero-shot reasoning task for VLMs. It forces the VLM to adaptively weigh visual content evidence against retriever metadata, such as rank and multiplicity, on a per-query basis without any prior training.
- S-Grid
- The S-Grid is a compact serialization map that represents each video as an image grid, optionally paired with subtitles. This format allows the VLM to perform list-wise reasoning over video candidates by providing a structured visual and textual representation of the retrieved items.
- List-wise Reranking (M=1)
- This mode evaluates ViC as a single-list reranker where it re-evaluates the top-K items from one retriever. It uses S-Grids to reorder these candidates, achieving significant gains in precision over standard methods like CombSUM.
- Ensemble Fuser (M>1)
- In this mode, ViC constructs a candidate list by interleaving multiple retrievers. It encodes rank and consensus metadata into the list order and item multiplicity, allowing the VLM to fuse results from several sources for a single query.
Terminology used across episodes
This episode discusses
- Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers · Paper Radio
- Passage Re-ranking with BERT
- Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
- Reading Wikipedia to Answer Open-Domain Questions
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Qwen Technical Report
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages
- Gramian Multimodal Representation Learning and Alignment
The paper
Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers · Read on arXiv
Mohamed Eltahir, Ali Habibullah, Lama Ayash, Tanveer Hussain, *Naeemullah Khan
King Abdullah University of Science and Technology · Department of Computer Science, King Khalid University · Department of Computer Science, Edge Hill University
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers".
Jane: Here's the gist of the paper. Vote-in-Context (ViC) introduces a generalized, training-free framework that rethinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM).
Tom: The core concept of the framework is to serialize both content evidence and retriever metadata directly within the VLM’s prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content.
Jane: This approach allows the model to adaptively weigh all signals on a per-query basis in a zero-shot setting The core insight is to serialize both content evidence and retriever metadata directly within the VLM’s prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content The framework is evaluated in two primary modes: as a single-list reranker and as an ensemble fuser.
Tom: First, as a powerful single-list reranker (M = 1), where ViC uses S-Grids to re-evaluate the top-K items from one retriever Second, as a novel ensemble fuser (M > 1), where ViC constructs a candidate list by interleaving multiple retrievers The S-Grid serves as the compact serialization map that represents each video as an image grid, optionally paired with subtitles to enable list-wise reasoning over video candidates.
Title and authors: Tom: So we're talking about Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers. This paper is basically proposing a way for a Vision Language Model to stop just looking at one thing and start judging which video candidates are best by looking at everything together.
Jane: That sounds really complex, Tom, but in simple terms, it’s about taking the idea of fusing different search results—like getting candidates from three different video search engines—and turning that into a reasoning task for a Vision Language Model without needing to train the model itself for that specific fusion.
Lu: Exactly. The core concept is serializing both the actual content evidence and all the extra information about where those candidates came from, like their rank or how many times they appeared in a search, directly into the prompt given to this Vision Language Model so it can adaptively weigh all those signals per query.
Meng: That sounds powerful in theory, but what does that actually mean for us on a production system? We're talking about replacing some simple scoring method with a full Vision Language Model forward pass every time, which I don't think is cheap when we have thousands of queries coming in.
Lalam: I read the summary. They frame it as re-thinking list-wise reranking and fusion as a zero-shot reasoning task for a Vision Language Model because typical training-free methods usually only use rank or score signals and ignore the actual content representations of the candidates <ref:2511.01617#pg3>.
Tom: Right, so it’s moving beyond just using those rank or score signals and making the VLM actually reason about which candidate is most relevant based on both what's in it and where it came from.
Jane: And they introduce a specific way to do that called the S-Grid, which they describe as a compact serialization map that represents each video as an image grid, optionally paired with subtitles <ref:2511.01617#pg3>.
Lu: That S-Grid representation is key because it allows for list-wise reasoning over video candidates, and it maintains a computational cost proportional to the number of candidates K rather than the raw video length <ref:2511.01617#pg3>.
Meng: Proportional to K is better than scaling with the video length itself, but still, if we have a huge candidate list, we're still doing a lot of work for every single query. What’s the actual cost per query like?
Tom: The paper says it’s O(K times the Vision Language Model forward pass cost), where K is the number of candidates and CVLM is that one forward pass cost <ref:2511.01617#pg3>. It's expensive, but we know what the scaling is.
Title and authors: Jane: It’s definitely more expensive than a simple arithmetic score because you are running a full VLM process for every candidate to get that final ranking <ref:2511.01617#pg3>.
Lu: The framework operates in two stages for assembling candidates and re-ranking, starting with constructing a single candidate sequence C of length K by round-robin interleaving those truncated lists <ref:2511.01617#pg3>. Then the Vision Language Model computes a permutation that reorders that sequence to produce the final fused and reranked output Rb of length K <ref:2511.01617#pg3>.
Meng: Round-robin interleaving sounds like a lot of list management on top of the VLM process itself. How much complexity does that add compared to just taking the best item from each list and combining them?
Tom: It adds complexity by preserving duplicates, which is important because it lets the VLM see those cross-list signals, and it allows the model to adaptively weigh retriever consensus against visual-linguistic content <ref:2511.01617#pg2>.
Jane: So they are specifically looking at how much better this becomes when we look at the performance on things like ActivityNet and VATEX, where they show new state-of-the-art zero-shot retrieval performance <ref:2511.01617#pg1>.
Lu: They achieve Recall@one scores of eighty-seven point one percent for T2V and eighty-nine point zero percent for V2T on MSR-VTT, and even a ninety-nine point six percent on VATEX for V2T <ref:2511.01617#pg3>. That’s a significant jump over previous methods.
Meng: Eighty seven point one percent is good, but I need to know how this compares to something like CombSUM, which they use as a baseline <ref:2511.01617#pg3>. Because if ViC is just slightly better than what we already have, the added cost might not be worth it for real deployment.
Tom: Well, the paper shows that when you look at it as an ensemble fuser with multiple retrievers, ViC consistently outperforms strong baselines like CombSUM <ref:2511.01617#pg3>. The fusion result for ViC is eighty seven point one on MSR-VTT and seventy five point nine on MSR-VTT as the best single backbone re-ranking result <ref:2511.01617#pg3>.
Jane: It seems like the additive advantage of fusion, where you use multiple retrievers, is pretty clear in these results <ref:2511.01617#pg3>. But there’s also a point they make about how the VLM actively uses cross-list consensus as a strong relevance signal <ref:2511.01617#pg3>.
Lu: That’s because they deduplicate the candidate sequence C of length K before passing it to the Vision Language Model, which removes that multiplicity metadata <ref:2511.01617#pg3>. And when you do that, their performance drops a bit—from eighty seven point one on MSR-VTT t2v down to eighty four point two <ref:2511.01617#pg3>.
Title and authors: Meng: So the VLM isn't just blindly trusting the highest score from a single retriever; it actually learns to use that consensus signal, but I have to worry about that drop in performance when we try to optimize for maximum recall <ref:2511.01617#pg3>.
Tom: It shows the model is learning a useful pattern, even with the loss of some raw metadata, which is good because it means the fusion mechanism is actually working at a deeper level than just simple aggregation <ref:2511.01617#pg2>.
Jane: Okay, let’s talk about what they suggest as improvements for this framework <ref:2511.01617#pg4>. They point out three main tradeoffs first, the inference cost is high because you are doing a full list-wise VLM forward pass instead of something cheap like RRF or CombSUM <ref:2511.01617#pg4>.
Lu: Second, the framework is strictly bounded by the Vision Language Model’s context window, which means if your video evidence doesn't fit in there, the whole thing breaks <ref:2511.01617#pg4>. And third, VLM reliability is a factor because the fidelity of this method depends entirely on how well the Vision Language Model follows instructions <ref:2511.01617#pg4>.
Meng: The context window limit is a big deal for us. If we have long videos, or if we need to feed a lot of visual context into the model, we hit that wall quickly <ref:2511.01617#pg4>.
Tom: And the authors suggest looking at prompt engineering or even lightweight VLM fine-tuning to enable smaller models to perform robustly within these constraints <ref:2511.01617#pg4>. That’s a practical direction for us right now.
Jane: I also noticed they discuss query-aware or adaptive keyframe selection mechanisms to generate more representative S-Grids when you are working with a fixed token budget <ref:2511.01617#pg4>. That might help manage the context window issue on our end.
Lu: And scaling properties, they show that the S-Grid representation maintains computational cost proportional to the number of candidates K and not the raw video length itself <ref:2511.01617#pg3>. That’s a solid finding for scalability <ref:2511.01617#pg4>.
Meng: Proportional to K is good, but we still need that K to be manageable, and the actual VLM forward pass cost is still the bottleneck when we think about deployment speed <ref:2511.01617#pg3>.
Tom: One final scaling observation they make is that eighty point B model achieves strong zero-shot re-ranking performance, and they identify eight B as the minimum effective scale for zero-shot list-wise re-ranking in this pipeline <ref:2511.01617#pg4>. That suggests that even smaller models might be viable if we fine tune them properly <ref:2511.01617#pg4>.
Title and authors: Jane: And they confirmed that the correct item is almost always retrieved within the top thirty candidates, which means R at thirty is effectively saturated near one hundred percent across benchmarks <ref:2511.01617#pg4>. But they also noted diminishing returns beyond a moderate context size <ref:2511.01617#pg4>.
Lu: For V2T retrieval specifically, the study suggests that an input size of K equals twenty captions emerges as the most effective operating point <ref:2511.01617#pg4>. Performance plateaus or declines beyond this threshold <ref:2511.01617#pg4>.
Meng: Twenty is a specific number, which is useful for setting our operational parameters, but it still puts a hard limit on the input size we can handle before the performance curve flattens out <ref:2511.01617#pg4>.
Tom: So to wrap up this Vote-in-Context paper, it essentially reconceptualizes fusion as a zero-shot, list-wise reasoning task performed by a Vision Language Model instead of just collapsing ranked lists into a single aggregated score <ref:2511.01617#pg4>.
Jane: It’s providing us with this simple, reproducible recipe for turning modern VLMs into powerful zero-shot rerankers and fusers, which is pretty useful for anyone looking to use these models in a practical way <ref:2511.01617#pg4>.
Lu: The main contributions are releasing the framework and evaluation protocols, including an analysis of ViC’s scaling properties and its sensitivity to context size <ref:2511.01617#pg4>.
Meng: And the code is publicly available, which is important for us because we need to actually test these claims on our specific data, not just read about them <ref:2511.01617#pg4>.
Tom: So we’re leaving this paper with the idea that a VLM can be a training-free multimodal fusion model if you serialize both the content evidence and retriever metadata directly into its prompt <ref:2511.01617#pg2>.
Jane: It’s been interesting seeing how they handle those trade-offs, like the inference cost being high, but it’s a necessary trade-off for this level of list-wise reasoning <ref:2511.01617#pg4>.
Lu: Yeah, it really shows that we need to keep pushing on prompt engineering and maybe lightweight VLM fine tuning if we want smaller models to be robust enough for this kind of task <ref:2511.01617#pg4>.
Meng: For deployment, I’m still focused on how we can manage the context window constraint, because that’s where real-world systems usually fail first <ref:2511.01617#pg4>.
Tom: Well, for now, this Vote-in-Context paper gives us a clear direction for how to use these VLMs to do more than just basic retrieval by turning them into reasoning engines <ref:2511.01617#pg2>.
Jane: That’s what we’ve got for today on Vote-in-Context, turning VLMs into zero-shot rank fusers. We’ll be back next time to look at something new from arXiv.
The paper's summary: Tom: So we're looking at the summary of Vote-in-Context, and what it actually does is basically this idea that you can stop just taking one score from a search engine and start using a Vision Language Model to reason about all those different scores together.
Jane: Right, so instead of just adding up the ranks or whatever old systems did, this paper proposes you serialize both the actual visual information and all that rank metadata directly into the prompt for the Vision Language Model.
Lu: That’s because it turns list-wise reranking and fusion into a zero-shot reasoning task for that AI model. It allows it to adaptively weigh how much it should trust what different retrievers are suggesting against what the visual content actually shows.
Meng: But this sounds like a lot of work for the model, right? We’re talking about running a full Vision Language Model forward pass every time we want to rank a list of candidates.
Tom: That's where they get specific about the mechanics. They introduce something called the S-Grid, which is like a compact map that represents each video as an image grid, sometimes with subtitles included.
Jane: The S-Grid representation helps because it lets you do list-wise reasoning over video candidates without needing those super costly frame-level cross attention mechanisms <ref:2511.01617#pg3>. It’s a uniform prompt for the model to work with.
Lu: And they showed that by interleaving candidate lists and encoding the rank and multiplicity metadata in that list order, this framework, when used as an ensemble fuser with multiple retrievers, consistently beats strong baselines like CombSum.
Tom: They even show a big performance jump here—a plus of eleven point two points over just doing single-backbone reranking. It means combining the fusion approach with the individual reranking approach gives you a real additive advantage.
Jane: What’s interesting is that they found something specific about how this works. When they remove that multiplicity metadata by deduplicating the candidate sequence before feeding it to the Vision Language Model, their performance actually drops a little bit, from eighty seven point one on MSR-VTT t2v down to eighty four point two <ref:2511.01617#pg3>.
Lu: That drop shows that the Vision Language Model is actively using that cross-list consensus as a strong signal, even when you take away the raw numbers of how many times something appeared in a search.
Meng: I see they’re making trade-offs right there. The cost is high because it's running a full VLM process, and it’s strictly limited by the context window size of that AI model <ref:2511.01617#pg4>.
Tom: Yeah, and Jane mentioned they suggest looking at prompt engineering or even lightweight VLM fine-tuning if we want to make these models smaller so they can handle this without breaking the bank <ref:2511.01617#pg4>.
Jane: And for practical application, they point out that for video to text retrieval specifically, an input size of twenty captions seems to be the most effective operating point before performance just starts flattening out <ref:2511.01617#pg4>.
Lu: The paper concludes by saying this is a simple recipe for turning modern AI into zero-shot rerankers and fusers, and they released all their protocols so other people can test it on their own data <ref:2511.01617#pg4>.
Tom: So the big picture here is that we’re moving beyond just collapsing ranked lists into one score and instead using the Vision Language Model to perform a full reasoning task over all those signals simultaneously.
Jane: And it gives us a clear idea of how to use these models as powerful, training-free fusion tools for video retrieval, but we still have that context window and inference cost challenge <ref:2511.01617#pg4>.
The paper's improvements: Tom: So we're looking at how they suggest improving this Vote-in-Context framework, and what that means for us on the engineering side.
Jane: Basically, they’re talking about three main trade-offs first, which is a good way to frame it because this method isn't perfect yet.
Lu: They point out the inference cost is high because you are doing a full list-wise Vision Language Model forward pass instead of something cheap like RRF or CombSum <ref:2511.01617#pg4>. It’s a big operational hit.
Meng: That cost is definitely something I need to worry about when we think about deployment, because running a full VLM process for every query isn't cheap at scale <ref:2511.01617#pg3>.
Tom: Second, the framework is strictly bounded by the Vision Language Model’s context window, so if your video evidence doesn't fit in there, the whole ranking thing breaks <ref:2511.01617#pg4>.
Jane: And third, they stress that the reliability of this entire method depends entirely on how well that Vision Language Model follows instructions when it’s being used for fusion <ref:2511.01617#pg4>.
Lu: To address the cost and context window issues, they suggest looking at prompt engineering or even lightweight fine-tuning of the Vision Language Model to make smaller models perform robustly <ref:2511.01617#pg4>.
Meng: That would be a huge help for us because it points toward making these models more efficient without needing a massive training budget <ref:2511.01617#pg4>.
Tom: And they also suggest some application-level ideas, like using query-aware or adaptive keyframe selection mechanisms to generate better S-Grids when you have a fixed token budget <ref:2511.01617#pg4>.
Jane: That’s a practical idea for managing the context window on our end because it means we could pick the best parts of the video instead of trying to cram everything in <ref:2511.01617#pg4>.
Lu: They also analyzed scaling properties, showing that the S-Grid representation keeps the cost proportional to the number of candidates K and not the raw video length itself <ref:2511.01617#pg3>. That's good for scalability in terms of storage, even if it’s expensive for computation <ref:2511.01617#pg4>.
Tom: And they also found that eight billion parameter models achieved strong zero-shot re-ranking performance, which they identified as the minimum effective scale for this pipeline <ref:2511.01617#pg4>.
Jane: So, even though it’s expensive and limited by context, the performance gains are significant enough to make it worthwhile for specific high-quality tasks <ref:2511.01617#pg4>.
Lu: This paper really gives us a clear path forward for how to use these large models not just as simple text processors but as reasoning engines that can fuse multimodal signals in zero-shot settings <ref:2511.01617#pg2>.
Conclusion: Tom: So to wrap up, this paper on Vote-in-Context is essentially saying that we can turn Vision Language Models into zero-shot rank fusers by just serializing all that content evidence and retriever metadata directly into the prompt.
Jane: That means instead of just taking a single score from one search engine, you can use the AI to reason about all those different scores together for a better result on video retrieval.
Lu: It establishes this new way of thinking about fusion, treating it as a reasoning task performed by the Vision Language Model instead of collapsing ranked lists into one aggregated score.
Meng: For me, the main thing is that it shows how to leverage cross-list consensus as a strong relevance signal, even though you lose some raw metadata when you deduplicate those candidate sequences <ref:2511.01617#pg3>.
Tom: Yeah, I'm excited about the fact that they show the performance jump when you use multiple retrievers—that plus of eleven point two points over single-backbone reranking is a solid number <ref:2511.01617#pg3>.
Jane: It really shows that combining fusion with individual reranking gives you a complementary strength, which is something we should keep looking at <ref:2511.01617#pg3>.
Lalam: I'm pretty sure this result holds up; the AI is very sure that by using the S-Grid and encoding the rank information in that specific way, it adapts its weights per query very effectively <ref:2511.01617#pg2>.
Lu: And they confirmed that even an eight billion parameter model can achieve strong zero-shot re-ranking performance, which they call the minimum effective scale for this pipeline <ref:2511.01617#pg4>.
Meng: I still have my eye on the deployment side, though; running a full VLM forward pass every time is going to be a bottleneck unless we get better at prompt engineering or fine-tuning these models efficiently <ref:2511.01617#pg4>.
Tom: Yeah, and they point toward those prompt engineering ideas as a way to make smaller models more robust for this kind of task <ref:2511.01617#pg4>.
Jane: And for video-to-text retrieval specifically, the input size of twenty captions seems to be the sweet spot before performance starts dropping off <ref:2511.01617#pg4>.
Lu: I think this paper really gives us a clear direction for how to use these large models not just as simple text processors but as reasoning engines that can fuse multimodal signals in zero-shot settings <ref:2511.01617#pg2>.
Tom: So, Vote-in-Context is providing a simple recipe for turning modern AI into zero-shot rerankers and fusers, and they've released their code so we can test it on our own data <ref:2511.01617#pg4>.
Jane: It’s been interesting seeing how they handle the trade-offs, like the high inference cost versus the reasoning power gained <ref:2511.01617#pg4>.
Lalam: I'm pretty sure that by serializing both the content and metadata, we're solving a fundamental problem in how these models use evidence during retrieval <ref:2511.01617#pg2>.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI