From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction

The gist The proposed framework bridges high-performing LLM-based relevance judgment systems into efficient multimodal student models to achieve comparable performance with significantly improved latency, cost, and scalability for search relevance prediction.

How it works

1. The process begins by curating a small, high-quality human-annotated dataset to serve as ground truth 3.1 Ground Truth Curation>. Relevance prediction is defined as the task of assigning a graded relevance label to a query-item pair, reflecting how well the item satisfies the user’s intent 3.2 Prompting Strategy>. In this setting, relevance is commonly modeled using graded or ordinal labels 3.3 LLM-as-a-Judge Pipeline>. In our setting, we adopt a four-level relevance scale: Bad, Fair, Good, and Excellent 3.3 Ground Truth Curation>.

2. A reliable LLM-as-a-judge pipeline is developed through careful prompt engineering and iterative refinement using the curated dataset 3.2 Prompting Strategy>. This pipeline, denoted as E ← PLLM (x ⊕ C), evaluates an input x and context C to produce an evaluation output E 3.3 LLM-as-a-Judge Pipeline>. The model M serves as a teacher model within this pipeline, producing both a relevance label and an explanation for each query-item pair 3.3 LLM-as-a-Judge Pipeline>.

3. This teacher output, forming training examples of the form (q, m, r, e), is then used to generate large-scale labeled data for training compact multimodal student models 3.4 Distillation to Student Models>. The teacher model M is instantiated as a prompted LLM and produces [r, e] ← M(p, q, m), where p is a structured prompt 3.3 LLM-as-a-Judge Pipeline>.

4. A compact multimodal autoregressive model, denoted as Msft, is trained to predict relevance labels and optionally generate explanations [rsft, esft] ← Msft(q, m) 3.4 Distillation to Student Models>. This training is performed via supervised finetuning starting from an instruction tuned base model Mit selected from the Gemma-3 [Team and DeepMind 2025] family of multimodal large language models 3.4 Distillation to Student Models>.

Key Findings and Results

** The distilled student models achieve performance comparable to the teacher on human-annotated benchmarks, while significantly reducing inference cost and latency 3.4 Distillation to Student Models>. Specifically, GPT-4.1 achieves the highest performance, reaching a multi-class relevance accuracy of approximately 83.7% on the human-annotated benchmark 3.3 LLM-as-a-Judge Pipeline>. The distilled Gemma-3 models show strong generalization and stability across runs 5.2 Stability>. The 12B model achieves even stronger results in terms of both accuracy and F1 score on the Test 2 dataset compared to the teacher model 5.3 Explanation>. The best Pareto point is identified as Gemma-3-12B-IT-RLV-FP8 because it offers the best tradeoff between metric performance and throughput 5.4 Throughput and Performance>. The student models are substantially more cost-effective in both input and output token costs relative to GPT-4.1 8 H Cost comparison>. The final best model is Gemma-3-12B-IT-RLV-FP8 7 Conclusion>. The student models also exhibit strong generalization, stability across runs, and high-quality explanation generation, confirming that they successfully capture the behavior of the teacher system 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. The final best model is Gemma-3-12B-IT-RLV-FP8 because it is near-teacher quality, stronger than the encoder baseline, competitive with or better than the 27B model variant on the independent human Test 2 dataset, and much faster and cheaper in FP8 7 Conclusion>. The student models also exhibit strong generalization, stability across runs, and high-quality explanation generation, confirming that they successfully capture the behavior of the teacher system 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>. This work highlights the potential of combining LLM-as-a-judge systems with knowledge distillation to build scalable and practical relevance modeling pipelines for large-scale search applications 7 Conclusion>. The results show that it is possible to retain the accuracy of large, prompt-based LLM systems while enabling efficient, highthroughput relevance prediction suitable for real-world deployment 7 Conclusion>.

1. textbfKnowledge Distillation from Teacher to Student Models for Relevance Prediction Instantly Applied to Production Systems: The improved system can perform high-throughput, low-latency relevance prediction by distilling a high-performing teacher model (like GPT-4.1) into compact multimodal student models (e.g., Gemma-3 variants). This allows for practical deployment in production environments while retaining much of the teacher system’s performance while significantly improving efficiency.

2. textbfEnhanced Reasoning through Iterative Prompt Engineering: The improved system will utilize an iterative refinement process for the LLM-as-a-judge pipeline, where failure cases are analyzed and incorporated as additional rules or examples. This systematic improvement ensures that the resulting prompt is robust, enabling the model to act as a reliable evaluator that reduces ambiguity in complex query-item reasoning tasks.

3. textbf Scalable High-Quality Label Generation: The system will leverage the LLM-as-a-judge pipeline to generate large-scale, high-quality supervision data for training downstream student models. This capability allows for the creation of massive datasets from a small set of human annotations, overcoming the bottleneck of expensive human labeling and enabling scalable supervision without human annotation.

4. textbf Multimodal Robustness via Image Resolution Scaling: The improved system can be fine-tuned on higher-quality visual data (maximum edge length of 900 pixels instead of 300 pixels), as results show that higher resolution seems especially helpful for more difficult predictions, leading to greater improvement is achieved for accuracy on both Validation 2 and Test 1 datasets.

5. textbf Explanation-Augmented Reasoning for Edge Cases: The improved system will be trained with explanations, as the paper demonstrates that training with explanations improves performance for most of the models and dataset variants, particularly on edge cases from Test 1. This allows the student model to capture subtle visual patterns that require detailed analysis.

Relevance prediction between user queries and items is a fundamental component of e-commerce search systems. Traditionally, high-quality relevance labels are obtained through human judgment, which is reliable but expensive, slow, and difficult to scale. Recent advances in large language models (LLMs) enable automated relevance judgment at scale, achieving performance comparable to human annotators. However, such systems often rely on large commercial LLMs with complex prompts, resulting in high inference costs and limited throughput. In this work, we propose a framework that bridges this gap by distilling a high-performing LLM-based relevance judgment system into efficient multimodal student models. We first curate a small, high-quality human-annotated dataset to serve as ground truth. Using this dataset, we develop a reliable LLM-as-a-judge pipeline through careful prompt engineering and iterative refinement. This pipeline is then used to generate large-scale relevance labels, which serve as training data for compact multimodal models. Our distilled models achieve performance close to the teacher system while significantly improving latency, cost, and scalability. This approach enables practical deployment of high-quality relevance prediction systems in production environments.