VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

The gist Finding compact evidence encoded once and reused as new queries arrive allows accurate video search to operate on compact evidence, encoded once and reused as new queries arrive

VEDJE Overview

VEDJE addresses the problem of needing a reusable video cache that must retain evidence for different objects, actions, and moments while remaining small enough to store at scale It achieves this by compressing features within sampled frames while keeping their representations separate across sampled times The system involves an offline frozen visual encoder processing each video once, which then writes a compact representation that a small joint encoder later reads at query time alongside the query text and the scalar first-stage score

VEDJE Pipeline

The VEDJE pipeline consists of three main components: (A) Offline indexing where compression of frame-indexed features yields the ordered cache Z(v) = [Z1;...;ZT] (B) Training-only feature-change supervision which predicts differences between the frozen features of sampled frames and (C) Online reranking where a 33M joint encoder processes the query, cache, and first-stage score

Key Contributions

The paper makes three main contributions to this work

1. Compact frame-indexed video caches: Building on cached joint encoding, we compress information within sampled frames while retaining separate frame groups, giving the joint scorer access to evidence from across the sequence under a fixed storage budget

2. Feature-change supervision for compression: We train the compressed representation with future-delta feature prediction, improving retrieval under tight storage budgets without an inference-time predictor

3. Accuracy and efficiency evaluation: Across four benchmarks, VEDJE improves R@1 in both directions over matched first stages

Training and Loss Function

VEDJE jointly trains the compressor, joint encoder, prior embedding, score head, and auxiliary training heads with four objectives defined by the loss function L = Lvtm + Lvtc + Lmlm + Ldelta The matching loss Lvtm supervises the pairwise score sθ(q, v) with cross-entropy over matched and negative pairs The contrastive loss Lvtc aligns projected text CLS representations with frozen video embeddings from the corresponding backbone through symmetric in-batch InfoNCE The masked-language loss Lmlm predicts masked caption tokens from the surrounding text with cached video tokens in context Finally, Ldelta supplies the frame-level feature-change targets by predicting differences between the frozen features of sampled frames, defined by ∆b t = Fω(Zt) ∈ R

Retrieval Performance

Across MSR-VTT (Xu et al., 2016), MSVD (Chen & Dolan, 2011), DiDeMo (Hendricks et al., 2017), and ActivityNet (Krishna et al., 2017), VEDJE consistently improves R@1 in both retrieval directions over matched first-stage retrievers On MSR-VTT, it reaches 59.8 text-to-video R@1 with a fine-tuned VideoCLIP-XL first stage (Figure 1) In the VideoPrism configuration, shrinking the per-video cache fourfold to 12 KiB preserves text-to-video recall within 0.2 points

Controlled Comparisons and Efficiency

The paper evaluates VEDJE across various cache structures, feature-change training, and joint scoring methods Feature-change supervision improves retrieval from the compressed representation without adding computation to the query path Controlled comparisons show that retaining separate frame groups over pooling improves R@1 by 2.0 points in both directions when storage remains fixed The complete VideoPrism query pipeline on an NVIDIA L40S takes 5.79 ms with a 64-token configuration, compared to approximately 4.3 ms for first-stage retrieval alone This configuration has approximately 143M total online parameters, including the 33M reranker

Conclusion

The findings establish a practical route to joint video search in which compact evidence is computed once and reused for later queries The cache is query-blind, serving every later query about v, so selectively reading or expanding the cache at query time is a natural extension The first and second stages are not currently trained jointly; closing that loop is another direction Finally, these evaluations span different video lengths but do not establish performance on multilingual video retrieval

AI Use Statement

Generative AI was used to assist in writing the manuscript and in writing code, as well as in organizing material and checking consistency The empirical results were supplied by the authors, who take full responsibility for the entire content of this submission

Ethics Statement

This work collects no new data and involves no crowdsourcing, human-subject study, biometric processing, facial recognition, or identity annotation Deployments should therefore require lawful and authorized data use, role-based access, purpose limitation, query logging and auditing

Reproducibility Statement

Algorithm 1 specifies indexing and reranking; Appendices A.2–A.5 give architecture, optimization, cache quantization, compute, code release The official VideoPrism-Base and VideoPrism-LvT-B model cards declare Apache License 2.0 The MiniLM (Wang et al., 2020) checkpoint is distributed under the MIT license The MSR-VTT, MSVD, DiDeMo, and ActivityNet are used under their original release terms

References

Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Bangalath, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong (2011). Collecting highly parallel data for paraphrase evaluation Zuozhuo Dai, Kaihui Cheng (2025). Text–video retrieval re-ranking via multi-grained cross attention and frozen image encoders Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2019). BERT: Pre-training of deep bidirectional transformers for language understanding Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell (2017). Localizing moments in video with natural language Omar Khattab and Matei Zaharia (2020). ColBERT: Efficient and effective passage search via contextualized late interaction over BERT Dohwan Ko, Ji Soo Lee, Minhyuk Choi, Zihang Meng, and Hyunwoo J. Kim (2025). Bidirectional likelihood estimation with multi-modal large language models for text-video retrieval Sanjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles (2017). Dense-captioning events in videos Ji Soo Lee, Byungoh Ko, Jaewon Cho, Howoong Lee, Jaewoon Byun, and Hyunwoo J. Kim (2025). Captioning for text-video retrieval via dual-group direct preference optimization Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi (2022). BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi (2023a). BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao (2024a). Unmasked teacher: Towards training-efficient video foundation models Huaishao Luo, Lei Ji (2022). CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji (2023). X-CLIP: End-toend multi-grained contrastive learning for video-text retrieval Alec Radford et al. (2021). Learning transferable visual models from natural language supervision Arun Reddy et al. (2025). Video-ColBERT: Contextualized late interaction for text-to-video retrieval Mitchell Keren Taraday, Shahaf Wagner, and Chaim Baskin (2026).

1. No query-time visual encoding is required, which indexed videos require no query-time visual encoding (Page 2). This allows for a system that processes compact evidence, encoded once and reused as new queries arrive, significantly reducing latency and computational overhead at runtime compared to methods that re-encode candidates online.

2. The VEDJE architecture supports joint matching by combining cached video tokens with the query text, achieving a retrieval score of c(q, v) = c(q, v) + eρρ(q, v) (Equation 5). This enables the system to refine the coarse relevance estimate from a first-stage retriever using joint content between text and compressed visual evidence.

3. Feature-change supervision allows for improved retrieval without adding computation to the query path by training a predictor Fω that learns feature changes from Zt and predicts differences between the frozen features of sampled frames (Equation 3). This auxiliary signal improves performance on tight storage budgets without increasing the latency of the actual search.

4. The system can achieve significant compression while maintaining high recall, as shown by the VideoPrism configuration which achieves shrinking the per-video cache fourfold to 12 KiB while preserving text-to-video recall within 0.2 points (Page 1). This demonstrates that accurate video search can operate on compact evidence, encoded once and reused as new queries.

5. The system provides a flexible query pipeline by integrating a residual prior, where the score is calculated as sθ(q, v) = hψ(˜c(q, v)) (Equation 5), which combines the joint representation with an embedded first-stage score to produce a final ranking. This ensures that the first-stage score and joint features also contribute complementary information.

6. The system offers controlled performance tuning by allowing users to select different cache structures, such as separate frame groups versus pooled multi-frame caches, which are shown to affect retrieval accuracy even when storage remains fixed (Table 4). This allows researchers to optimize the organization of cached evidence for specific retrieval tasks.

7. The system can be deployed efficiently by utilizing hardware-aware precision controls, such as FP8 or FP4 quantization, which reinforce the R@1 reading and show that FP8 leaves T2V and V2T values effectively unchanged from the BF16 reference (Table 16). This enables deployment on resource-constrained devices while minimizing accuracy degradation.

Finding the right video often requires distinguishing similar scenes in which different events occur. Joint matching improves retrieval, but processing rich video representations for each query is costly. VEDJE compresses features within sampled frames while keeping their representations separate in a reusable cache. Feature-change prediction supplies an auxiliary training signal that improves retrieval from the compressed cache without adding work at query time. On MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions. On MSR-VTT, it reaches 59.8 text-to-video R@1 with a fine-tuned VideoCLIP-XL first stage. In the VideoPrism configuration, shrinking the per-video cache fourfold to 12 KiB preserves text-to-video recall within 0.2 points. These results show that accurate video search can operate on compact evidence, encoded once and reused as new queries arrive.