UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction".
Paper summary: Jane: So we’ve got the idea that this paper, UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction, aims to address the challenge of comparing models that combine sequential modeling and feature interaction. The thesis is that current advances are often locked behind proprietary data and infrastructure, which limits academic study of scaling laws or long-sequence behavior >
Tom: They propose UniRank as an open benchmark to tackle this, offering standardized evaluation protocols and an optimized PyTorch toolkit for systematic comparison >
Lu: What’s interesting is how they categorize the ranking model approaches into three families: shallow learning methods, deep feature interaction methods, and sequence modeling methods like DIN or DIEN >
Meng: I’m interested in the training signals. They contrast two main ways to handle sequential data: a traditional single-task new impression only paradigm and a chronological pointwise autoregressive paradigm >
Jane: The traditional one uses a fixed positive-feedback sequence to predict a new impression, aiming for the loss LNIO which is the sum of binary cross entropy terms across all new impressions >
Tom: And the second one, the chronological pointwise autoregressive paradigm, constructs a full feedback sequence where the model predicts current target labels from strictly preceding behavior >
Lu: This second paradigm creates a chronological full-feedback sequence where the target loss LAR sums binary cross entropy terms for every time step t and every item k in that sequence >
Meng: So they’re essentially comparing how you structure the feedback—whether you fix it as context or build it step by step chronologically >
Jane: The evaluation shows that both unified interaction paradigms are competitive, and no single model is universally superior across all data distributions, though performance varies by dataset >
Tom: For instance, EST and HeMix are competitive on Taobao and MerRec, but the paper stresses that performance is highly dependent on the specific data distribution you're using >
Conclusion: Tom: So wrapping up UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction. The authors have given us a way to compare these complex ranking architectures systematically, even if we don't have all the private data anymore >
Jane: It really gives researchers a common language and a standardized setup to study how these different modeling families actually perform under realistic conditions across short video, advertising, and e-commerce >
Lu: The implication is that by providing this open benchmark, we can move past comparing closed implementations and start studying the underlying scaling laws more openly >
Meng: From my side, it means we have a standardized way to test what actually runs at scale without needing access to the massive proprietary infrastructure >
Lalam: UniRank gives us a standardized evaluation protocol and an optimized toolkit, which is what makes this work valuable for moving ranking research forward in an open way >
Honghao Li, Xianquan Wang, Zibin Zhang, Yi Zhang, Kangyi Lin, Yiwen Zhang
Anhui University · University of Science and Technology of China · Tencent Inc.
cs.IR
Submitted: 2026-07-22
Updated: 2026-11-18
Comments: 13 pages, 6 figures, and 9 tables. Code and data: https://github.com/salmon1802/UniRank
Code: https://github.com/shenweichen/deepctr
Project page: https://fuxictr.github.io
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
The gist: Ranking is a core stage in online advertising and recommender systems This paper proposes UniRank, an open benchmark for ranking models that unify sequential modeling and feature interaction UniRank…
Key concepts
- Unified Sequential Modeling and Feature Interaction
- This is a new approach for ranking models that tries to combine two distinct methods: sequential modeling, which understands user behavior over time, and feature interaction, which learns how different user or item characteristics influence the prediction. UniRank benchmarks models that successfully integrate both of these capabilities.
- Traditional Single-Task Paradigm
- This training method uses a fixed positive feedback sequence shared by many samples to predict a new impression. The goal is to minimize the error when predicting a new item based on this shared positive history, focusing on single prediction tasks.
- Chronological Pointwise Autoregressive Paradigm
- This paradigm builds a long sequence where the model predicts each current target label based only on the strictly preceding behaviors. It creates a full chronological feedback loop, allowing the model to learn temporal dependencies by predicting one step at a time.
Terminology
Summary
Ranking is a core stage in online advertising and recommender systems This paper proposes UniRank, an open benchmark for ranking models that unify sequential modeling and feature interaction UniRank provides standardized evaluation protocols, an optimized PyTorch toolkit, and five large-scale public datasets from diverse platforms. p. 7
The Core Contribution
UniRank is the first open benchmark dedicated to ranking architectures that unify sequential modeling and feature interaction It enables systematic comparison under standardized data, optimization, and efficiency settings. The work integrates Distributed Data Parallel (DDP), operator, mixedprecision, and attention optimizations to reduce hardware costs. p. 1 The authors release source code, training configurations, processed datasets, and complete results for 15 models on five public datasets spanning up to 700 million instances and behavior sequences exceeding 105 events. p. 1
Modeling Paradigms
The paper categorizes ranking model approaches into three broad methodological families. These paradigms are:
-
The Shallow Learning Method, which uses Logistic Regression and factorization-based models. p. 0
-
The Deep Feature Interaction Method, which embeds sparse categorical features and replaces manual crosses with neural interaction modules like DeepFM [17]. p. 0
-
The Sequence Modeling Method, which captures temporal changes in user interest using models such as DIN [89] and DIEN [13]. p. 0
Training Signal Construction
The paper contrasts two primary training paradigms for sequential data.
** Traditional Single-Task New-Impression-Only Paradigm:**
This paradigm uses a fixed positive-feedback sequence shared by multiple samples to predict a new impression. The objective is LNIO = ∑i=1 BCEy i, f theta (S+ i, x i). p. 2
** Chronological Pointwise Autoregressive Paradigm:**
This paradigm constructs a chronological full-feedback sequence where the model predicts current target labels from strictly preceding behavior S. The target loss sums binary cross-entropy terms LAR = ∑t ∈ T ∑k=1 BCEy tk, f theta (S<t, x t). p. 2
Evaluation and Performance
UniRank evaluates 15 representative models on five large-scale public datasets spanning short-video, advertising, e-commerce, and C2C marketplaces. The benchmark fixes sequence length at 100 for controlled comparison across all 15 models. p. 3 Performance analysis shows that both unified interaction paradigms are competitive, and no universally superior model exists across data distributions. p. 4 Model performance is highly dataset-dependent, with EST and HeMix being competitive on Taobao and MerRec. p. 4
Efficiency Analysis
The efficiency analysis compares total parameters and one-epoch training time under various settings. Zenith [84] has a substantially larger parameter footprint because it expands user and item ID embeddings and retains both IDs as independent tokens. p. 5 TokenFormer [90] and UltraHSTU [11] retain sequence tokens across interaction layers and apply structured sparse attention, which increases active computation. p. 5 UniRank prioritizes fair model comparison and tunes each model by validation AUC rather than peak system throughput. p. 5
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified Architecture Development: Implement token-based unified architectures that combine sequential modeling with feature interaction to improve Model FLOPs Utilization (MFU) and scale more effectively with sequence length and model capacity.
-
Training Paradigm Shift: Adopt the Chronological Pointwise Autoregressive Paradigm to construct a
full-feedback sequence
rather than apositive-only history,
which allows the model to predict current target labels from the strictly preceding sequence, resulting indense gradients
that improve the use of long interaction sequences and scale with chronological data. -
Reproducibility Standardization: Establish a common protocol for evaluation by defining six requirements—Unified data pipeline, Unified task and metric definitions, Unified model source code, Unified hyperparameters, Unified training pipeline, and Unified efficiency optimization—to ensure
reproducible and fair ranking model research.
-
Efficient Training Toolkit: Integrate Distributed Data Parallel (DDP), operator compilation using torch.compile with the Inductor backend for dense modules, mixed precision training using bf16 autocast, and attention optimization like Flash Attention to reduce hardware requirements.
-
Controlled Experimentation: Utilize handbook-style controlled experiments on tokenization strategies (Chunk, Auto, Field, Random) and activation functions (GeLU, SiLU) to standardize architectural studies and reduce the cost of rebuilding datasets and implementations.
-
Scaling Law Analysis: Conduct controlled scaling experiments on KuaiRand to determine effective scaling laws by varying token dimension, sequence length, and model depth while comparing average AUC with per-batch computation, ensuring that
effective scaling should prioritize architecture-specific dimensions according to how sequential information is preserved.
-
Optimized Training Schedules: Employ a fixed training schedule of one epoch per model with separate sparse and dense optimizers (Adagrad for sparse embeddings and AdamW for dense parameters) consistent with production practice, optimizing sparse embeddings at a default learning rate of 0.05 while AdamW optimizes dense parameters at 0.0001.
-
Hardware Efficiency Tuning: Select model configurations based on throughput, memory, storage, and sequence length scaling together rather than peak system throughput alone to make practical selection decisions for deployment and research.
Sources
- LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders
- RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
- Beyond Positive Signals: Unlocking Implicit Negative Behaviors for Enhanced Sequential User Modeling
- Bending the Scaling Law Curve in Large-Scale Recommendation Systems
- On Ranking Consistency of Pre-ranking Stage
- Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems
- UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems
- Gaussian Error Linear Units (GELUs)
- MixFormer: Co-Scaling Up Dense and Sequence in Industrial Recommenders
- HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction
- TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders
- Aggregate and Broadcast: Scalable and Efficient Feature Interaction for Recommender Systems
- MerRec: A Large-scale Multipurpose Mercari Dataset for Consumer-to-Consumer Recommendation Systems
- GraphFM: Graph Factorization Machines for Feature Interaction Modeling
- Muon is Scalable for LLM Training
- EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling
- LaProp: Separating Momentum and Adaptivity in Adam
- Decoupled Weight Decay Regularization
- Mish: A Self Regularized Non-Monotonic Activation Function
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
Related papers
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA
- One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations