OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation
summary
The gist
Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods The paper presents OmniReview…
In short
The paper introduces OmniReview, a large dataset of 202,756 verified academic review records. It proposes Pro-MMoE, a novel framework using Large Language Models and Mixture-of-Experts to improve reviewer recommendation systems. This system uses a three-tier evaluation framework—Recall, Discrimination, and Ranking—to accurately identify expert reviewers.
Key concepts
- OmniReview
- A comprehensive dataset integrating reviews from multiple academic platforms. It contains 202,756 verified records linking papers to scholar profiles and reviewer information across various sources.
- Pro-MMoE
- A novel framework combining Large Language Models (LLMs) with a Multi-gate Mixture-of-Experts architecture. This system uses LLMs to create detailed semantic profiles of scholars and an MMoE structure to dynamically balance different evaluation goals for recommendation.
- Three-Tier Evaluation Framework
- A structured approach to assess recommendation performance across three distinct tasks: Recall (finding ground-truth reviewers), Discrimination (filtering out superficial candidates), and Ranking (ordering the best experts). This framework ensures a holistic assessment of the system's effectiveness.
- Multi-gate Mixture-of-Experts
- An architecture that uses multiple specialized 'expert' models to handle different evaluation tasks. A 'multi-gate' mechanism dynamically directs inputs to the most appropriate expert based on the specific goal, improving performance in complex recommendation scenarios.
Terminology used across episodes
This episode discusses
- OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation · Paper Radio
- SciBERT: A Pretrained Language Model for Scientific Text
- InPars: Data Augmentation for Information Retrieval using Large Language Models
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Enhancing Talent Search Ranking with Role-Aware Expert Mixtures and LLM-based Fine-Grained Job Descriptions
- FRONTIER-RevRec: A Large-scale Dataset for Reviewer Recommendation
- Query2doc: Query Expansion with Large Language Models
- On the Theoretical Limitations of Embedding-Based Retrieval
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
The paper
OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation · Read on arXiv
Yehua Huang, Penglei Sun, Zebin Chen, Zhenheng Tang, Xiaowen Chu
The Hong Kong University of Science and Technology (Guangzhou)
Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods. From the data perspective, existing research is hindered by the scarcity of large-scale, verified benchmarks and oversimplified evaluation metrics that fail to reflect real-world editorial workflows. To bridge this gap, we present OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records. Based on this data, we introduce a three-tier hierarchical evaluaion framework to assess recommendations from recall to precise expert identification. From the method perspective, existing embedding-based approaches suffer from the information bottleneck of semantic compression and limited interpretability. To resolve these method limitations, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning. Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals. Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation".
Jane: Here's the gist of the paper. Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods The paper presents OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records Based on this data, we introduce a three-tier hierarchical evaluation framework to assess recommendations from recall to precise expert identification Dataset Construction and Data Integrity OmniReview is defined as D = (P, A, R), where P represents the paper collection, A denotes the scholar set, and R defines the peer review records The construction methodology comprises three sequential phases: (1) entity alignment, (2) discipline taxonomy construction, and (3) task formulation The entity disambiguation phase resolves ambiguities in academic information across sources by leveraging metadata, such as titles and co-author networks This process successfully links reviews to researcher profiles, yielding 202, 756 verified records and 150, 287 identified reviewers Methodological Innovation To resolve the limitations of existing embedding-based approaches, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals This resolves the information bottleneck of semantic compression by instructing an LLM to extract and summarize fine-grained research interests from publication records Evaluation Framework Design We define three tasks to evaluate the performance of the recommendation system: (1) Task 1 (Recall): Retrieving historical ground-truth reviewers; (2) Task 2 (Discrimination): Filtering out hard-negative candidates who appear relevant superficially but lack specific domain expertise; and (3) Task 3 (Ranking): Fine-grained ranking of the best experts among the qualified candidates For Task 1, we assess retrieval effectiveness by measuring the systems capacity to assign higher confidence scores to ground-truth reviewers Performance and Findings Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation Pro-MMoE outperforms state-of-the-art baselines by margins of 1.02%, 5.39%, and 17.
Title and authors: Jane: Okay, so we know the scale of the data set now. But what does OmniReview actually *do*? How do we translate that into something useful for us in terms of research?
Tom: It’s about creating a comprehensive dataset by combining information from many different academic platforms to build these scholar profiles. They use a process called entity alignment to connect the reviews to the right researcher profiles, which is pretty smart work.
Lu: They go through three steps for construction: first entity alignment, then building a discipline taxonomy, and finally formulating the specific tasks they want the system to do with this data.
Meng: Building a taxonomy sounds like it’s imposing structure on all that messy academic information. What kind of structure are they imposing?
Jane: It’s structuring how researchers are categorized so the AI can understand their areas of expertise better, which feeds into their next big idea, which is the Pro-MMoE framework.
Tom: Right, the Pro-MMoE stands for Profiling Scholars with Multi-gate Mixture of Experts. That’s where they try to fix the problem of semantic compression by using large language models to extract those fine-grained research interests from publication records.
The paper's summary: Jane: Let's break down what that means for us, Tom. They are proposing this whole system—OmniReview and Pro-MMoE—to evaluate recommendation systems by breaking the task down into three tiers.
Tom: Right, they define these three tasks: Task one is recall, which is just about retrieving the historical ground truth reviewers. Task two is discrimination, where the system has to filter out those hard negative candidates who look relevant but don't actually have deep domain expertise.
Lu: And task three is ranking, which is the hardest part—it’s fine-grained ranking of the absolute best experts among all those qualified candidates. They are trying to assess performance across all three of these specific goals.
Meng: So they aren't just building one big system; they’re testing if their method works for finding *any* good reviewer, then filtering out the bad ones, and finally sorting the top few. That seems like a thorough way to test a recommendation engine.
Tom: They measure recall by seeing how confident the system is in assigning higher scores to those true reviewers. They’re also measuring discrimination by checking how well it separates experts from non-experts who just happen to use similar keywords.
The paper's improvements: Jane: Now they talk about what they improved compared to previous work, and it seems like they have a few key ideas. They suggest using a more sophisticated entity alignment pipeline for the initial data construction.
Tom: They want to use machine learning on top of word-level matching to resolve ambiguities in scholar names and publication titles across different sources before doing the actual linking. That should give them higher fidelity in their connections.
Lu: And they are enhancing the Pro-MMoE framework by fine-tuning the large language model part so it explicitly extracts and preserves those fine-grained expertise nuances, which is meant to help with that discrimination task.
Meng: I’m worried about that LLM extraction—if we can’t get it right, we just have noisy profiles. What about the architecture itself? How do they handle the trade-off between finding many candidates and finding absolute experts?
Tom: They introduce a Task-Adaptive Mixture of Experts architecture. This means they assign specific expert networks to different parts of the model, separating them for recall, discrimination, and ranking so the system can balance those conflicting goals dynamically.
Conclusion: Jane: So to wrap up, OmniReview gives us this massive dataset and a three-tiered evaluation framework—recall, discrimination, and ranking—to test recommendation systems against. It really shows how much context is missing from existing benchmarks.
Tom: They achieved state of the art performance across six of seven metrics, outperforming current baselines by margins like one point zero two percent for recall and seventeen point one five percent for ranking. That’s a concrete number showing their approach works better in practice on this specific set of tests.
Lu: The implication is that if you want to build a system that truly understands the nuance of academic expertise, you need this kind of structured data and a model architecture that can handle multiple objectives at once.
Meng: From a deployment view, it’s good they showed it's data efficient too. They found the approach achieves considerable data efficiency even when they only used zero point zero five as the ratio of training to evaluation data, which is important when we don't have massive amounts of perfectly annotated data available for real traffic.
Lalam: I’m confident in what the model is saying because it has access to all that literature and it’s focused on the one direction that matters for accurate identification. It’s quite sure of its answer on how to structure those expert profiles.
More episodes
- 2411.16152-Optimizing Winograd Convolution on ARMv8 processors
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines