Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers

arXiv:2511.01617 · cs.CV, cs.IR · Submitted 2025-11-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers".

Jane: Here's the gist of the paper. Vote-in-Context (ViC) introduces a generalized, training-free framework that rethinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM).

Tom: The core concept of the framework is to serialize both content evidence and retriever metadata directly within the VLM’s prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content.

Jane: This approach allows the model to adaptively weigh all signals on a per-query basis in a zero-shot setting The core insight is to serialize both content evidence and retriever metadata directly within the VLM’s prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content The framework is evaluated in two primary modes: as a single-list reranker and as an ensemble fuser.

Tom: First, as a powerful single-list reranker (M = 1), where ViC uses S-Grids to re-evaluate the top-K items from one retriever Second, as a novel ensemble fuser (M > 1), where ViC constructs a candidate list by interleaving multiple retrievers The S-Grid serves as the compact serialization map that represents each video as an image grid, optionally paired with subtitles to enable list-wise reasoning over video candidates.

Title and authors: Tom: So we're talking about Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers. This paper is basically proposing a way for a Vision Language Model to stop just looking at one thing and start judging which video candidates are best by looking at everything together.

Jane: That sounds really complex, Tom, but in simple terms, it’s about taking the idea of fusing different search results—like getting candidates from three different video search engines—and turning that into a reasoning task for a Vision Language Model without needing to train the model itself for that specific fusion.

Lu: Exactly. The core concept is serializing both the actual content evidence and all the extra information about where those candidates came from, like their rank or how many times they appeared in a search, directly into the prompt given to this Vision Language Model so it can adaptively weigh all those signals per query.

Meng: That sounds powerful in theory, but what does that actually mean for us on a production system? We're talking about replacing some simple scoring method with a full Vision Language Model forward pass every time, which I don't think is cheap when we have thousands of queries coming in.

Lalam: I read the summary. They frame it as re-thinking list-wise reranking and fusion as a zero-shot reasoning task for a Vision Language Model because typical training-free methods usually only use rank or score signals and ignore the actual content representations of the candidates <ref:2511.01617#pg3>.

Tom: Right, so it’s moving beyond just using those rank or score signals and making the VLM actually reason about which candidate is most relevant based on both what's in it and where it came from.

Jane: And they introduce a specific way to do that called the S-Grid, which they describe as a compact serialization map that represents each video as an image grid, optionally paired with subtitles <ref:2511.01617#pg3>.

Lu: That S-Grid representation is key because it allows for list-wise reasoning over video candidates, and it maintains a computational cost proportional to the number of candidates K rather than the raw video length <ref:2511.01617#pg3>.

Meng: Proportional to K is better than scaling with the video length itself, but still, if we have a huge candidate list, we're still doing a lot of work for every single query. What’s the actual cost per query like?

Tom: The paper says it’s O(K times the Vision Language Model forward pass cost), where K is the number of candidates and CVLM is that one forward pass cost <ref:2511.01617#pg3>. It's expensive, but we know what the scaling is.

Title and authors: Jane: It’s definitely more expensive than a simple arithmetic score because you are running a full VLM process for every candidate to get that final ranking <ref:2511.01617#pg3>.

Lu: The framework operates in two stages for assembling candidates and re-ranking, starting with constructing a single candidate sequence C of length K by round-robin interleaving those truncated lists <ref:2511.01617#pg3>. Then the Vision Language Model computes a permutation that reorders that sequence to produce the final fused and reranked output Rb of length K <ref:2511.01617#pg3>.

Meng: Round-robin interleaving sounds like a lot of list management on top of the VLM process itself. How much complexity does that add compared to just taking the best item from each list and combining them?

Tom: It adds complexity by preserving duplicates, which is important because it lets the VLM see those cross-list signals, and it allows the model to adaptively weigh retriever consensus against visual-linguistic content <ref:2511.01617#pg2>.

Jane: So they are specifically looking at how much better this becomes when we look at the performance on things like ActivityNet and VATEX, where they show new state-of-the-art zero-shot retrieval performance <ref:2511.01617#pg1>.

Lu: They achieve Recall@one scores of eighty-seven point one percent for T2V and eighty-nine point zero percent for V2T on MSR-VTT, and even a ninety-nine point six percent on VATEX for V2T <ref:2511.01617#pg3>. That’s a significant jump over previous methods.

Meng: Eighty seven point one percent is good, but I need to know how this compares to something like CombSUM, which they use as a baseline <ref:2511.01617#pg3>. Because if ViC is just slightly better than what we already have, the added cost might not be worth it for real deployment.

Tom: Well, the paper shows that when you look at it as an ensemble fuser with multiple retrievers, ViC consistently outperforms strong baselines like CombSUM <ref:2511.01617#pg3>. The fusion result for ViC is eighty seven point one on MSR-VTT and seventy five point nine on MSR-VTT as the best single backbone re-ranking result <ref:2511.01617#pg3>.

Jane: It seems like the additive advantage of fusion, where you use multiple retrievers, is pretty clear in these results <ref:2511.01617#pg3>. But there’s also a point they make about how the VLM actively uses cross-list consensus as a strong relevance signal <ref:2511.01617#pg3>.

Lu: That’s because they deduplicate the candidate sequence C of length K before passing it to the Vision Language Model, which removes that multiplicity metadata <ref:2511.01617#pg3>. And when you do that, their performance drops a bit—from eighty seven point one on MSR-VTT t2v down to eighty four point two <ref:2511.01617#pg3>.

Title and authors: Meng: So the VLM isn't just blindly trusting the highest score from a single retriever; it actually learns to use that consensus signal, but I have to worry about that drop in performance when we try to optimize for maximum recall <ref:2511.01617#pg3>.

Tom: It shows the model is learning a useful pattern, even with the loss of some raw metadata, which is good because it means the fusion mechanism is actually working at a deeper level than just simple aggregation <ref:2511.01617#pg2>.

Jane: Okay, let’s talk about what they suggest as improvements for this framework <ref:2511.01617#pg4>. They point out three main tradeoffs first, the inference cost is high because you are doing a full list-wise VLM forward pass instead of something cheap like RRF or CombSUM <ref:2511.01617#pg4>.

Lu: Second, the framework is strictly bounded by the Vision Language Model’s context window, which means if your video evidence doesn't fit in there, the whole thing breaks <ref:2511.01617#pg4>. And third, VLM reliability is a factor because the fidelity of this method depends entirely on how well the Vision Language Model follows instructions <ref:2511.01617#pg4>.

Meng: The context window limit is a big deal for us. If we have long videos, or if we need to feed a lot of visual context into the model, we hit that wall quickly <ref:2511.01617#pg4>.

Tom: And the authors suggest looking at prompt engineering or even lightweight VLM fine-tuning to enable smaller models to perform robustly within these constraints <ref:2511.01617#pg4>. That’s a practical direction for us right now.

Jane: I also noticed they discuss query-aware or adaptive keyframe selection mechanisms to generate more representative S-Grids when you are working with a fixed token budget <ref:2511.01617#pg4>. That might help manage the context window issue on our end.

Lu: And scaling properties, they show that the S-Grid representation maintains computational cost proportional to the number of candidates K and not the raw video length itself <ref:2511.01617#pg3>. That’s a solid finding for scalability <ref:2511.01617#pg4>.

Meng: Proportional to K is good, but we still need that K to be manageable, and the actual VLM forward pass cost is still the bottleneck when we think about deployment speed <ref:2511.01617#pg3>.

Tom: One final scaling observation they make is that eighty point B model achieves strong zero-shot re-ranking performance, and they identify eight B as the minimum effective scale for zero-shot list-wise re-ranking in this pipeline <ref:2511.01617#pg4>. That suggests that even smaller models might be viable if we fine tune them properly <ref:2511.01617#pg4>.

Title and authors: Jane: And they confirmed that the correct item is almost always retrieved within the top thirty candidates, which means R at thirty is effectively saturated near one hundred percent across benchmarks <ref:2511.01617#pg4>. But they also noted diminishing returns beyond a moderate context size <ref:2511.01617#pg4>.

Lu: For V2T retrieval specifically, the study suggests that an input size of K equals twenty captions emerges as the most effective operating point <ref:2511.01617#pg4>. Performance plateaus or declines beyond this threshold <ref:2511.01617#pg4>.

Meng: Twenty is a specific number, which is useful for setting our operational parameters, but it still puts a hard limit on the input size we can handle before the performance curve flattens out <ref:2511.01617#pg4>.

Tom: So to wrap up this Vote-in-Context paper, it essentially reconceptualizes fusion as a zero-shot, list-wise reasoning task performed by a Vision Language Model instead of just collapsing ranked lists into a single aggregated score <ref:2511.01617#pg4>.

Jane: It’s providing us with this simple, reproducible recipe for turning modern VLMs into powerful zero-shot rerankers and fusers, which is pretty useful for anyone looking to use these models in a practical way <ref:2511.01617#pg4>.

Lu: The main contributions are releasing the framework and evaluation protocols, including an analysis of ViC’s scaling properties and its sensitivity to context size <ref:2511.01617#pg4>.

Meng: And the code is publicly available, which is important for us because we need to actually test these claims on our specific data, not just read about them <ref:2511.01617#pg4>.

Tom: So we’re leaving this paper with the idea that a VLM can be a training-free multimodal fusion model if you serialize both the content evidence and retriever metadata directly into its prompt <ref:2511.01617#pg2>.

Jane: It’s been interesting seeing how they handle those trade-offs, like the inference cost being high, but it’s a necessary trade-off for this level of list-wise reasoning <ref:2511.01617#pg4>.

Lu: Yeah, it really shows that we need to keep pushing on prompt engineering and maybe lightweight VLM fine tuning if we want smaller models to be robust enough for this kind of task <ref:2511.01617#pg4>.

Meng: For deployment, I’m still focused on how we can manage the context window constraint, because that’s where real-world systems usually fail first <ref:2511.01617#pg4>.

Tom: Well, for now, this Vote-in-Context paper gives us a clear direction for how to use these VLMs to do more than just basic retrieval by turning them into reasoning engines <ref:2511.01617#pg2>.

Jane: That’s what we’ve got for today on Vote-in-Context, turning VLMs into zero-shot rank fusers. We’ll be back next time to look at something new from arXiv.

The paper's summary: Tom: So we're looking at the summary of Vote-in-Context, and what it actually does is basically this idea that you can stop just taking one score from a search engine and start using a Vision Language Model to reason about all those different scores together.

Jane: Right, so instead of just adding up the ranks or whatever old systems did, this paper proposes you serialize both the actual visual information and all that rank metadata directly into the prompt for the Vision Language Model.

Lu: That’s because it turns list-wise reranking and fusion into a zero-shot reasoning task for that AI model. It allows it to adaptively weigh how much it should trust what different retrievers are suggesting against what the visual content actually shows.

Meng: But this sounds like a lot of work for the model, right? We’re talking about running a full Vision Language Model forward pass every time we want to rank a list of candidates.

Tom: That's where they get specific about the mechanics. They introduce something called the S-Grid, which is like a compact map that represents each video as an image grid, sometimes with subtitles included.

Jane: The S-Grid representation helps because it lets you do list-wise reasoning over video candidates without needing those super costly frame-level cross attention mechanisms <ref:2511.01617#pg3>. It’s a uniform prompt for the model to work with.

Lu: And they showed that by interleaving candidate lists and encoding the rank and multiplicity metadata in that list order, this framework, when used as an ensemble fuser with multiple retrievers, consistently beats strong baselines like CombSum.

Tom: They even show a big performance jump here—a plus of eleven point two points over just doing single-backbone reranking. It means combining the fusion approach with the individual reranking approach gives you a real additive advantage.

Jane: What’s interesting is that they found something specific about how this works. When they remove that multiplicity metadata by deduplicating the candidate sequence before feeding it to the Vision Language Model, their performance actually drops a little bit, from eighty seven point one on MSR-VTT t2v down to eighty four point two <ref:2511.01617#pg3>.

Lu: That drop shows that the Vision Language Model is actively using that cross-list consensus as a strong signal, even when you take away the raw numbers of how many times something appeared in a search.

Meng: I see they’re making trade-offs right there. The cost is high because it's running a full VLM process, and it’s strictly limited by the context window size of that AI model <ref:2511.01617#pg4>.

Tom: Yeah, and Jane mentioned they suggest looking at prompt engineering or even lightweight VLM fine-tuning if we want to make these models smaller so they can handle this without breaking the bank <ref:2511.01617#pg4>.

Jane: And for practical application, they point out that for video to text retrieval specifically, an input size of twenty captions seems to be the most effective operating point before performance just starts flattening out <ref:2511.01617#pg4>.

Lu: The paper concludes by saying this is a simple recipe for turning modern AI into zero-shot rerankers and fusers, and they released all their protocols so other people can test it on their own data <ref:2511.01617#pg4>.

Tom: So the big picture here is that we’re moving beyond just collapsing ranked lists into one score and instead using the Vision Language Model to perform a full reasoning task over all those signals simultaneously.

Jane: And it gives us a clear idea of how to use these models as powerful, training-free fusion tools for video retrieval, but we still have that context window and inference cost challenge <ref:2511.01617#pg4>.

The paper's improvements: Tom: So we're looking at how they suggest improving this Vote-in-Context framework, and what that means for us on the engineering side.

Jane: Basically, they’re talking about three main trade-offs first, which is a good way to frame it because this method isn't perfect yet.

Lu: They point out the inference cost is high because you are doing a full list-wise Vision Language Model forward pass instead of something cheap like RRF or CombSum <ref:2511.01617#pg4>. It’s a big operational hit.

Meng: That cost is definitely something I need to worry about when we think about deployment, because running a full VLM process for every query isn't cheap at scale <ref:2511.01617#pg3>.

Tom: Second, the framework is strictly bounded by the Vision Language Model’s context window, so if your video evidence doesn't fit in there, the whole ranking thing breaks <ref:2511.01617#pg4>.

Jane: And third, they stress that the reliability of this entire method depends entirely on how well that Vision Language Model follows instructions when it’s being used for fusion <ref:2511.01617#pg4>.

Lu: To address the cost and context window issues, they suggest looking at prompt engineering or even lightweight fine-tuning of the Vision Language Model to make smaller models perform robustly <ref:2511.01617#pg4>.

Meng: That would be a huge help for us because it points toward making these models more efficient without needing a massive training budget <ref:2511.01617#pg4>.

Tom: And they also suggest some application-level ideas, like using query-aware or adaptive keyframe selection mechanisms to generate better S-Grids when you have a fixed token budget <ref:2511.01617#pg4>.

Jane: That’s a practical idea for managing the context window on our end because it means we could pick the best parts of the video instead of trying to cram everything in <ref:2511.01617#pg4>.

Lu: They also analyzed scaling properties, showing that the S-Grid representation keeps the cost proportional to the number of candidates K and not the raw video length itself <ref:2511.01617#pg3>. That's good for scalability in terms of storage, even if it’s expensive for computation <ref:2511.01617#pg4>.

Tom: And they also found that eight billion parameter models achieved strong zero-shot re-ranking performance, which they identified as the minimum effective scale for this pipeline <ref:2511.01617#pg4>.

Jane: So, even though it’s expensive and limited by context, the performance gains are significant enough to make it worthwhile for specific high-quality tasks <ref:2511.01617#pg4>.

Lu: This paper really gives us a clear path forward for how to use these large models not just as simple text processors but as reasoning engines that can fuse multimodal signals in zero-shot settings <ref:2511.01617#pg2>.

Conclusion: Tom: So to wrap up, this paper on Vote-in-Context is essentially saying that we can turn Vision Language Models into zero-shot rank fusers by just serializing all that content evidence and retriever metadata directly into the prompt.

Jane: That means instead of just taking a single score from one search engine, you can use the AI to reason about all those different scores together for a better result on video retrieval.

Lu: It establishes this new way of thinking about fusion, treating it as a reasoning task performed by the Vision Language Model instead of collapsing ranked lists into one aggregated score.

Meng: For me, the main thing is that it shows how to leverage cross-list consensus as a strong relevance signal, even though you lose some raw metadata when you deduplicate those candidate sequences <ref:2511.01617#pg3>.

Tom: Yeah, I'm excited about the fact that they show the performance jump when you use multiple retrievers—that plus of eleven point two points over single-backbone reranking is a solid number <ref:2511.01617#pg3>.

Jane: It really shows that combining fusion with individual reranking gives you a complementary strength, which is something we should keep looking at <ref:2511.01617#pg3>.

Lalam: I'm pretty sure this result holds up; the AI is very sure that by using the S-Grid and encoding the rank information in that specific way, it adapts its weights per query very effectively <ref:2511.01617#pg2>.

Lu: And they confirmed that even an eight billion parameter model can achieve strong zero-shot re-ranking performance, which they call the minimum effective scale for this pipeline <ref:2511.01617#pg4>.

Meng: I still have my eye on the deployment side, though; running a full VLM forward pass every time is going to be a bottleneck unless we get better at prompt engineering or fine-tuning these models efficiently <ref:2511.01617#pg4>.

Tom: Yeah, and they point toward those prompt engineering ideas as a way to make smaller models more robust for this kind of task <ref:2511.01617#pg4>.

Jane: And for video-to-text retrieval specifically, the input size of twenty captions seems to be the sweet spot before performance starts dropping off <ref:2511.01617#pg4>.

Lu: I think this paper really gives us a clear direction for how to use these large models not just as simple text processors but as reasoning engines that can fuse multimodal signals in zero-shot settings <ref:2511.01617#pg2>.

Tom: So, Vote-in-Context is providing a simple recipe for turning modern AI into zero-shot rerankers and fusers, and they've released their code so we can test it on our own data <ref:2511.01617#pg4>.

Jane: It’s been interesting seeing how they handle the trade-offs, like the high inference cost versus the reasoning power gained <ref:2511.01617#pg4>.

Lalam: I'm pretty sure that by serializing both the content and metadata, we're solving a fundamental problem in how these models use evidence during retrieval <ref:2511.01617#pg2>.

Mohamed Eltahir, Ali Habibullah, Lama Ayash, Tanveer Hussain, *Naeemullah Khan

King Abdullah University of Science and Technology · Department of Computer Science, King Khalid University · Department of Computer Science, Edge Hill University

cs.CV, cs.IR

Submitted: 2025-11-03

Updated: 2026-10-31

Comments: Accepted to CVPR Findings 2026

Code: https://github.com/mohammad2012191/ViC

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: Vote-in-Context (ViC) introduces a generalized, training-free framework that rethinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM).

Key concepts

Vote-in-Context (ViC)
ViC is a framework that rethinks list reranking and fusion as a zero-shot reasoning task for VLMs. It forces the VLM to adaptively weigh visual content evidence against retriever metadata, such as rank and multiplicity, on a per-query basis without any prior training.
S-Grid
The S-Grid is a compact serialization map that represents each video as an image grid, optionally paired with subtitles. This format allows the VLM to perform list-wise reasoning over video candidates by providing a structured visual and textual representation of the retrieved items.
List-wise Reranking (M=1)
This mode evaluates ViC as a single-list reranker where it re-evaluates the top-K items from one retriever. It uses S-Grids to reorder these candidates, achieving significant gains in precision over standard methods like CombSUM.
Ensemble Fuser (M>1)
In this mode, ViC constructs a candidate list by interleaving multiple retrievers. It encodes rank and consensus metadata into the list order and item multiplicity, allowing the VLM to fuse results from several sources for a single query.

Terminology

Summary

Vote-in-Context (ViC) introduces a generalized, training-free framework that rethinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM). p. 0

The core concept of the framework is to serialize both content evidence and retriever metadata directly within the VLM’s prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content. p. 0

This approach allows the model to adaptively weigh all signals on a per-query basis in a zero-shot setting The core insight is to serialize both content evidence and retriever metadata directly within the VLM’s prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content

The framework is evaluated in two primary modes: as a single-list reranker and as an ensemble fuser.

First, as a powerful single-list reranker (M = 1), where ViC uses S-Grids to re-evaluate the top-K items from one retriever Second, as a novel ensemble fuser (M > 1), where ViC constructs a candidate list by interleaving multiple retrievers

The S-Grid serves as the compact serialization map that represents each video as an image grid, optionally paired with subtitles to enable list-wise reasoning over video candidates.

The authors introduce the S-Grid, a compact serialization map that represents each video as an image grid, optionally paired with subtitles to enable list-wise reasoning over video candidates This representation is denoted as S-Grid(v) = Grid(v; s), av when applied to video retrieval

The ViC framework operates in two formal stages for candidate assembly and VLM re-ranking.

The process begins by constructing a single candidate sequence C(q) of length K The candidate sequence C(q) is formed by a round-robin (RR) interleaving of these truncated lists, preserving duplicates The VLM computes a permutation πˆ ∈ SK where the final fused and reranked output Rb(q) is the sequence C reordered by this permutation

ViC demonstrates significant performance gains across video retrieval benchmarks, establishing new state-of-the-art zero-shot retrieval performance.

Across video retrieval benchmarks including ActivityNet and VATEX, the framework establishes new state-of-the-art zero-shot retrieval performance In zero-shot settings, ViC achieves Recall@1 scores of 87. p. 01% (t2v) / 89. p. 00% (v2t) on MSR-VTT and 99. p. 06% (v2t) on VATEX

The framework's effectiveness is shown by comparing it against strong baselines like CombSUM.

ViC as a fuser (M > 1) consistently outperforms strong baselines like CombSUM The ViC fusion result (87. p. 11 on MSRVTT, Table 2) with the best single-backbone re-ranking result (75. p. 89 on MSR-VTT, Table 1) highlights the additive advantage of fusion

A key finding is that the VLM actively uses cross-list consensus as a strong relevance signal.

The ViC framework deduplicates the candidate sequence C(q) before passing it to the VLM, thus removing the multiplicity metadata The resulting performance drop (such as 87. p. 81 to 84.2 on MSR-VTT t2v) confirms that the VLM actively uses cross-list consensus as a strong relevance signal

The study highlights important trade-offs inherent in the ViC framework.

The ViC framework itself introduces three main tradeoffs. p. 9 First, its inference cost is computationally expensive, replacing the near-zero arithmetic cost of RRF or CombSUM with a full, list-wise VLM forward pass Second, the framework is strictly bounded by the VLM’s context window Finally, VLM reliability is a factor, as the framework’s fidelity depends entirely on the VLM’s instruction-following capabilities

The study suggests clear directions for future work based on identified limitations.

At the framework level, promising approaches include prompt engineering and lightweight VLM fine-tuning to enable smaller, more efficient models to perform robustly At the application level, future research could investigate query-aware or adaptive keyframe selection mechanisms to generate more representative S-Grids within a fixed token budget

The study concludes that ViC reconceptualizes fusion as a zero-shot, list-wise reasoning task performed by a VLM.

Rather than collapsing ranked lists into a single aggregated score, as in Soft Voting or RRF, ViC serializes both the content evidence E(·) and the retriever metadata (rank, multiplicity) directly into the VLM prompt This design allows the frozen VLM to adaptively weigh all available signals on a per-query basis, thereby functioning as a trainingfree, multimodal fusion model

The paper demonstrates that the S-Grid representation maintains computational cost proportional to the number of candidates (K) rather than the raw video length.

This representation maintains computational cost proportional to the number of candidates (K) rather than the raw video length, while still preserving temporal coverage and exploiting multimodal information from both visual and textual sources The cost per query is O(K ·CVLM) where K is the number of candidates, and CVLM is one forward pass cost

The study identifies 8B as the minimum effective scale for zero-shot list-wise re-ranking in the proposed ViC pipeline.

This result identifies 8B as the minimum effective scale for zero-shot list-wise re-ranking in the proposed ViC pipeline The strong performance of the 8B model, even without training, suggests that lightweight finetuning could be a very promising direction for developing highly efficient, much smaller rerankers

The study confirms that the correct item is almost always retrieved within the top 30 candidates.

Preliminary analyses confirmed that R@30 is effectively saturated near 100% across benchmarks, indicating that the correct item is almost always retrieved within the top 30 candidates However, the results indicate diminishing returns beyond a moderate context size The results indicate diminishing returns beyond a moderate context size

The study suggests that for v2t retrieval, an input size of K = 20 captions emerges as the most effective operating point.

For v2t retrieval, an input size of K = 20 captions emerges as the most effective operating point, as performance plateaus or declines beyond this threshold These observations collectively highlight the practical limitations of current VLMs’ effective context windows in list-wise relevance judgment tasks

The paper presents a simple, reproducible, and highly effective recipe for turning modern VLMs into powerful zero-shot rerankers and fusers.

We present ViC as a simple, reproducible, and highly effective recipe for turning modern VLMs into powerful zero-shot rerankers and fusers The main contributions of this work are summarized as follows

The authors release their framework and evaluation protocols, including an extensive analysis of ViC’s scaling properties.

We release our framework and evaluation protocols, including an extensive analysis of ViC’s scaling properties, its sensitivity to context size, and the performance of different assembly strategies The code and resources are publicly available at: //github. p. 1com/mohammad2012191/ViC

The paper is organized as follows.

This paper is organized as follows Section 2 reviews related work Section 3 details the proposed ViC framework and its application for video retrieval Section 4 presents the experimental results and ablation studies, followed by a discussion of the framework-s limitations and future directions

**The paper is organized as follows.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive List Fusion via Vote-in-Context (ViC) allows models to adaptively weigh retriever consensus against visual-linguistic content by serializing both content evidence and retriever metadata directly into the VLM prompt, enabling per-query decision making that is modalityagnostic.

  2. Enabling zero-shot listwise reranking for video retrieval, ViC can be used as a single-list reranker (M=1) to dramatically improve individual retriever precision, achieving up to +40 Recall@1 over previous baselines on benchmarks like MSR-VTT and VATEX.

  3. The S-Grid representation provides a Uniform Video Prompt by composing frames into an H x W image grid optionally paired with subtitles, allowing a single VLM to process candidates retrieved from any upstream model without costly frame-level cross-attention.

  4. ViC can function as an ensemble fuser (M > 1) by interleaving candidate lists and encoding rank and consensus metadata in the list order and item multiplicity, consistently outperforming strong baselines like CombSUM when fusing multiple retrievers.

  5. The framework establishes a new performance frontier, showing that incorporating fusion (ViC, M > 1) provides an additional +11.2 points improvement over single-backbone reranking (ViC, M = 1), demonstrating the complementary strengths of the two components within the ViC framework.

  6. The system can be scaled effectively: results show that even an 8B model achieves strong zero-shot re-ranking performance, identifying it as the minimum effective scale for zero-shot list-wise re-ranking in the proposed ViC pipeline.

Sources

Related papers