OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation".
Jane: Here's the gist of the paper. Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods The paper presents OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records Based on this data, we introduce a three-tier hierarchical evaluation framework to assess recommendations from recall to precise expert identification Dataset Construction and Data Integrity OmniReview is defined as D = (P, A, R), where P represents the paper collection, A denotes the scholar set, and R defines the peer review records The construction methodology comprises three sequential phases: (1) entity alignment, (2) discipline taxonomy construction, and (3) task formulation The entity disambiguation phase resolves ambiguities in academic information across sources by leveraging metadata, such as titles and co-author networks This process successfully links reviews to researcher profiles, yielding 202, 756 verified records and 150, 287 identified reviewers Methodological Innovation To resolve the limitations of existing embedding-based approaches, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals This resolves the information bottleneck of semantic compression by instructing an LLM to extract and summarize fine-grained research interests from publication records Evaluation Framework Design We define three tasks to evaluate the performance of the recommendation system: (1) Task 1 (Recall): Retrieving historical ground-truth reviewers; (2) Task 2 (Discrimination): Filtering out hard-negative candidates who appear relevant superficially but lack specific domain expertise; and (3) Task 3 (Ranking): Fine-grained ranking of the best experts among the qualified candidates For Task 1, we assess retrieval effectiveness by measuring the systems capacity to assign higher confidence scores to ground-truth reviewers Performance and Findings Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation Pro-MMoE outperforms state-of-the-art baselines by margins of 1.02%, 5.39%, and 17.
Title and authors: Jane: Okay, so we know the scale of the data set now. But what does OmniReview actually *do*? How do we translate that into something useful for us in terms of research?
Tom: It’s about creating a comprehensive dataset by combining information from many different academic platforms to build these scholar profiles. They use a process called entity alignment to connect the reviews to the right researcher profiles, which is pretty smart work.
Lu: They go through three steps for construction: first entity alignment, then building a discipline taxonomy, and finally formulating the specific tasks they want the system to do with this data.
Meng: Building a taxonomy sounds like it’s imposing structure on all that messy academic information. What kind of structure are they imposing?
Jane: It’s structuring how researchers are categorized so the AI can understand their areas of expertise better, which feeds into their next big idea, which is the Pro-MMoE framework.
Tom: Right, the Pro-MMoE stands for Profiling Scholars with Multi-gate Mixture of Experts. That’s where they try to fix the problem of semantic compression by using large language models to extract those fine-grained research interests from publication records.
The paper's summary: Jane: Let's break down what that means for us, Tom. They are proposing this whole system—OmniReview and Pro-MMoE—to evaluate recommendation systems by breaking the task down into three tiers.
Tom: Right, they define these three tasks: Task one is recall, which is just about retrieving the historical ground truth reviewers. Task two is discrimination, where the system has to filter out those hard negative candidates who look relevant but don't actually have deep domain expertise.
Lu: And task three is ranking, which is the hardest part—it’s fine-grained ranking of the absolute best experts among all those qualified candidates. They are trying to assess performance across all three of these specific goals.
Meng: So they aren't just building one big system; they’re testing if their method works for finding *any* good reviewer, then filtering out the bad ones, and finally sorting the top few. That seems like a thorough way to test a recommendation engine.
Tom: They measure recall by seeing how confident the system is in assigning higher scores to those true reviewers. They’re also measuring discrimination by checking how well it separates experts from non-experts who just happen to use similar keywords.
The paper's improvements: Jane: Now they talk about what they improved compared to previous work, and it seems like they have a few key ideas. They suggest using a more sophisticated entity alignment pipeline for the initial data construction.
Tom: They want to use machine learning on top of word-level matching to resolve ambiguities in scholar names and publication titles across different sources before doing the actual linking. That should give them higher fidelity in their connections.
Lu: And they are enhancing the Pro-MMoE framework by fine-tuning the large language model part so it explicitly extracts and preserves those fine-grained expertise nuances, which is meant to help with that discrimination task.
Meng: I’m worried about that LLM extraction—if we can’t get it right, we just have noisy profiles. What about the architecture itself? How do they handle the trade-off between finding many candidates and finding absolute experts?
Tom: They introduce a Task-Adaptive Mixture of Experts architecture. This means they assign specific expert networks to different parts of the model, separating them for recall, discrimination, and ranking so the system can balance those conflicting goals dynamically.
Conclusion: Jane: So to wrap up, OmniReview gives us this massive dataset and a three-tiered evaluation framework—recall, discrimination, and ranking—to test recommendation systems against. It really shows how much context is missing from existing benchmarks.
Tom: They achieved state of the art performance across six of seven metrics, outperforming current baselines by margins like one point zero two percent for recall and seventeen point one five percent for ranking. That’s a concrete number showing their approach works better in practice on this specific set of tests.
Lu: The implication is that if you want to build a system that truly understands the nuance of academic expertise, you need this kind of structured data and a model architecture that can handle multiple objectives at once.
Meng: From a deployment view, it’s good they showed it's data efficient too. They found the approach achieves considerable data efficiency even when they only used zero point zero five as the ratio of training to evaluation data, which is important when we don't have massive amounts of perfectly annotated data available for real traffic.
Lalam: I’m confident in what the model is saying because it has access to all that literature and it’s focused on the one direction that matters for accurate identification. It’s quite sure of its answer on how to structure those expert profiles.
Yehua Huang, Penglei Sun, Zebin Chen, Zhenheng Tang, Xiaowen Chu
The Hong Kong University of Science and Technology (Guangzhou)
cs.IR, cs.AI
Submitted: 2026-02-09
Updated: 2026-11-10
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods The paper presents OmniReview…
Key concepts
- OmniReview
- A comprehensive dataset integrating reviews from multiple academic platforms. It contains 202,756 verified records linking papers to scholar profiles and reviewer information across various sources.
- Pro-MMoE
- A novel framework combining Large Language Models (LLMs) with a Multi-gate Mixture-of-Experts architecture. This system uses LLMs to create detailed semantic profiles of scholars and an MMoE structure to dynamically balance different evaluation goals for recommendation.
- Three-Tier Evaluation Framework
- A structured approach to assess recommendation performance across three distinct tasks: Recall (finding ground-truth reviewers), Discrimination (filtering out superficial candidates), and Ranking (ordering the best experts). This framework ensures a holistic assessment of the system's effectiveness.
- Multi-gate Mixture-of-Experts
- An architecture that uses multiple specialized 'expert' models to handle different evaluation tasks. A 'multi-gate' mechanism dynamically directs inputs to the most appropriate expert based on the specific goal, improving performance in complex recommendation scenarios.
Terminology
Summary
Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods The paper presents OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records Based on this data, we introduce a three-tier hierarchical evaluation framework to assess recommendations from recall to precise expert identification
Dataset Construction and Data Integrity
OmniReview is defined as D = (P, A, R), where P represents the paper collection, A denotes the scholar set, and R defines the peer review records The construction methodology comprises three sequential phases: (1) entity alignment, (2) discipline taxonomy construction, and (3) task formulation The entity disambiguation phase resolves ambiguities in academic information across sources by leveraging metadata, such as titles and co-author networks This process successfully links reviews to researcher profiles, yielding 202, 756 verified records and 150, 287 identified reviewers
Methodological Innovation
To resolve the limitations of existing embedding-based approaches, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals This resolves the information bottleneck of semantic compression by instructing an LLM to extract and summarize fine-grained research interests from publication records
Evaluation Framework Design
We define three tasks to evaluate the performance of the recommendation system: (1) Task 1 (Recall): Retrieving historical ground-truth reviewers; (2) Task 2 (Discrimination): Filtering out hard-negative candidates who appear relevant superficially but lack specific domain expertise; and (3) Task 3 (Ranking): Fine-grained ranking of the best experts among the qualified candidates For Task 1, we assess retrieval effectiveness by measuring the systems capacity to assign higher confidence scores to ground-truth reviewers
Performance and Findings
Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation Pro-MMoE outperforms state-of-the-art baselines by margins of 1.02%, 5.39%, and 17.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Data construction for OmniReview can be improved by implementing a more sophisticated entity alignment pipeline that utilizes machine learning to resolve ambiguities in scholar names and publication titles across disparate sources before applying word-level matching. This ensures higher fidelity in linking reviews to specific researcher profiles, as described in Algorithm 2.
-
The Pro-MMoE framework can be enhanced by fine-tuning the LLM profiling module to explicitly extract and preserve
fine-grained expertise nuances
to better address the information bottleneck of semantic compression. This allows the system to generate more interpretable semantic profiles that are crucial for Task 2 (Discrimination). -
The Multi-gate Mixture-of-Experts Module can be optimized by developing task-specific gating networks that dynamically balance conflicting evaluation goals with greater precision, as proposed in Section 4.4. This dynamic weighting ensures the model can better distinguish between
true experts from hard negative candidates who share superficial keywords but lack deep domain expertise.
-
The Task-specific prediction towers can be improved by utilizing the hybrid AUC-margin loss function to simultaneously optimize pairwise ranking margins and global AUC maximization, ensuring both sufficient score separation and overall ranking quality are achieved.
-
The system's performance on Task 3 (Ranking) can be further enhanced by increasing the number of experts in the MMoE architecture up to an optimal point, as shown in Figure 4. This investigation confirms that
the NDCG metric achieves its peak performance at exactly 3 experts,
suggesting that a configuration with more than three experts may not improve ranking quality. -
The model can be made more data efficient by training under varying data volume ratios, as shown in Figure 5. This demonstrates that the approach
achieves considerable data efficiency
even at the lowest ratio (0.05), allowing for practical deployment where large-scale annotated data is unavailable or costly to obtain.
Abstract
Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods. From the data perspective, existing research is hindered by the scarcity of large-scale, verified benchmarks and oversimplified evaluation metrics that fail to reflect real-world editorial workflows. To bridge this gap, we present OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records. Based on this data, we introduce a three-tier hierarchical evaluaion framework to assess recommendations from recall to precise expert identification. From the method perspective, existing embedding-based approaches suffer from the information bottleneck of semantic compression and limited interpretability. To resolve these method limitations, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning. Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals. Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation.
Sources
- SciBERT: A Pretrained Language Model for Scientific Text
- InPars: Data Augmentation for Information Retrieval using Large Language Models
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Enhancing Talent Search Ranking with Role-Aware Expert Mixtures and LLM-based Fine-Grained Job Descriptions
- FRONTIER-RevRec: A Large-scale Dataset for Reviewer Recommendation
- Query2doc: Query Expansion with Large Language Models
- On the Theoretical Limitations of Embedding-Based Retrieval
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA
- One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations