UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction

arXiv:2607.19987 · cs.IR · Submitted 2026-07-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction".

Paper summary: Jane: So we’ve got the idea that this paper, UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction, aims to address the challenge of comparing models that combine sequential modeling and feature interaction. The thesis is that current advances are often locked behind proprietary data and infrastructure, which limits academic study of scaling laws or long-sequence behavior >

Tom: They propose UniRank as an open benchmark to tackle this, offering standardized evaluation protocols and an optimized PyTorch toolkit for systematic comparison >

Lu: What’s interesting is how they categorize the ranking model approaches into three families: shallow learning methods, deep feature interaction methods, and sequence modeling methods like DIN or DIEN >

Meng: I’m interested in the training signals. They contrast two main ways to handle sequential data: a traditional single-task new impression only paradigm and a chronological pointwise autoregressive paradigm >

Jane: The traditional one uses a fixed positive-feedback sequence to predict a new impression, aiming for the loss LNIO which is the sum of binary cross entropy terms across all new impressions >

Tom: And the second one, the chronological pointwise autoregressive paradigm, constructs a full feedback sequence where the model predicts current target labels from strictly preceding behavior >

Lu: This second paradigm creates a chronological full-feedback sequence where the target loss LAR sums binary cross entropy terms for every time step t and every item k in that sequence >

Meng: So they’re essentially comparing how you structure the feedback—whether you fix it as context or build it step by step chronologically >

Jane: The evaluation shows that both unified interaction paradigms are competitive, and no single model is universally superior across all data distributions, though performance varies by dataset >

Tom: For instance, EST and HeMix are competitive on Taobao and MerRec, but the paper stresses that performance is highly dependent on the specific data distribution you're using >

Conclusion: Tom: So wrapping up UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction. The authors have given us a way to compare these complex ranking architectures systematically, even if we don't have all the private data anymore >

Jane: It really gives researchers a common language and a standardized setup to study how these different modeling families actually perform under realistic conditions across short video, advertising, and e-commerce >

Lu: The implication is that by providing this open benchmark, we can move past comparing closed implementations and start studying the underlying scaling laws more openly >

Meng: From my side, it means we have a standardized way to test what actually runs at scale without needing access to the massive proprietary infrastructure >

Lalam: UniRank gives us a standardized evaluation protocol and an optimized toolkit, which is what makes this work valuable for moving ranking research forward in an open way >

Honghao Li, Xianquan Wang, Zibin Zhang, Yi Zhang, Kangyi Lin, Yiwen Zhang

Anhui University · University of Science and Technology of China · Tencent Inc.

cs.IR

Submitted: 2026-07-22

Updated: 2026-11-18

Comments: 13 pages, 6 figures, and 9 tables. Code and data: https://github.com/salmon1802/UniRank

Code: https://github.com/shenweichen/deepctr

Project page: https://fuxictr.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: Ranking is a core stage in online advertising and recommender systems This paper proposes UniRank, an open benchmark for ranking models that unify sequential modeling and feature interaction UniRank…

Key concepts

Unified Sequential Modeling and Feature Interaction
This is a new approach for ranking models that tries to combine two distinct methods: sequential modeling, which understands user behavior over time, and feature interaction, which learns how different user or item characteristics influence the prediction. UniRank benchmarks models that successfully integrate both of these capabilities.
Traditional Single-Task Paradigm
This training method uses a fixed positive feedback sequence shared by many samples to predict a new impression. The goal is to minimize the error when predicting a new item based on this shared positive history, focusing on single prediction tasks.
Chronological Pointwise Autoregressive Paradigm
This paradigm builds a long sequence where the model predicts each current target label based only on the strictly preceding behaviors. It creates a full chronological feedback loop, allowing the model to learn temporal dependencies by predicting one step at a time.

Terminology

Summary

Ranking is a core stage in online advertising and recommender systems This paper proposes UniRank, an open benchmark for ranking models that unify sequential modeling and feature interaction UniRank provides standardized evaluation protocols, an optimized PyTorch toolkit, and five large-scale public datasets from diverse platforms. p. 7

The Core Contribution

UniRank is the first open benchmark dedicated to ranking architectures that unify sequential modeling and feature interaction It enables systematic comparison under standardized data, optimization, and efficiency settings. The work integrates Distributed Data Parallel (DDP), operator, mixedprecision, and attention optimizations to reduce hardware costs. p. 1 The authors release source code, training configurations, processed datasets, and complete results for 15 models on five public datasets spanning up to 700 million instances and behavior sequences exceeding 105 events. p. 1

Modeling Paradigms

The paper categorizes ranking model approaches into three broad methodological families. These paradigms are:

  1. The Shallow Learning Method, which uses Logistic Regression and factorization-based models. p. 0

  2. The Deep Feature Interaction Method, which embeds sparse categorical features and replaces manual crosses with neural interaction modules like DeepFM [17]. p. 0

  3. The Sequence Modeling Method, which captures temporal changes in user interest using models such as DIN [89] and DIEN [13]. p. 0

Training Signal Construction

The paper contrasts two primary training paradigms for sequential data.

** Traditional Single-Task New-Impression-Only Paradigm:**

This paradigm uses a fixed positive-feedback sequence shared by multiple samples to predict a new impression. The objective is LNIO = ∑i=1 BCEy i, f theta (S+ i, x i). p. 2

** Chronological Pointwise Autoregressive Paradigm:**

This paradigm constructs a chronological full-feedback sequence where the model predicts current target labels from strictly preceding behavior S. The target loss sums binary cross-entropy terms LAR = ∑t ∈ T ∑k=1 BCEy tk, f theta (S<t, x t). p. 2

Evaluation and Performance

UniRank evaluates 15 representative models on five large-scale public datasets spanning short-video, advertising, e-commerce, and C2C marketplaces. The benchmark fixes sequence length at 100 for controlled comparison across all 15 models. p. 3 Performance analysis shows that both unified interaction paradigms are competitive, and no universally superior model exists across data distributions. p. 4 Model performance is highly dataset-dependent, with EST and HeMix being competitive on Taobao and MerRec. p. 4

Efficiency Analysis

The efficiency analysis compares total parameters and one-epoch training time under various settings. Zenith [84] has a substantially larger parameter footprint because it expands user and item ID embeddings and retains both IDs as independent tokens. p. 5 TokenFormer [90] and UltraHSTU [11] retain sequence tokens across interaction layers and apply structured sparse attention, which increases active computation. p. 5 UniRank prioritizes fair model comparison and tunes each model by validation AUC rather than peak system throughput. p. 5

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified Architecture Development: Implement token-based unified architectures that combine sequential modeling with feature interaction to improve Model FLOPs Utilization (MFU) and scale more effectively with sequence length and model capacity.

  2. Training Paradigm Shift: Adopt the Chronological Pointwise Autoregressive Paradigm to construct a full-feedback sequence rather than a positive-only history, which allows the model to predict current target labels from the strictly preceding sequence, resulting in dense gradients that improve the use of long interaction sequences and scale with chronological data.

  3. Reproducibility Standardization: Establish a common protocol for evaluation by defining six requirements—Unified data pipeline, Unified task and metric definitions, Unified model source code, Unified hyperparameters, Unified training pipeline, and Unified efficiency optimization—to ensure reproducible and fair ranking model research.

  4. Efficient Training Toolkit: Integrate Distributed Data Parallel (DDP), operator compilation using torch.compile with the Inductor backend for dense modules, mixed precision training using bf16 autocast, and attention optimization like Flash Attention to reduce hardware requirements.

  5. Controlled Experimentation: Utilize handbook-style controlled experiments on tokenization strategies (Chunk, Auto, Field, Random) and activation functions (GeLU, SiLU) to standardize architectural studies and reduce the cost of rebuilding datasets and implementations.

  6. Scaling Law Analysis: Conduct controlled scaling experiments on KuaiRand to determine effective scaling laws by varying token dimension, sequence length, and model depth while comparing average AUC with per-batch computation, ensuring that effective scaling should prioritize architecture-specific dimensions according to how sequential information is preserved.

  7. Optimized Training Schedules: Employ a fixed training schedule of one epoch per model with separate sparse and dense optimizers (Adagrad for sparse embeddings and AdamW for dense parameters) consistent with production practice, optimizing sparse embeddings at a default learning rate of 0.05 while AdamW optimizes dense parameters at 0.0001.

  8. Hardware Efficiency Tuning: Select model configurations based on throughput, memory, storage, and sequence length scaling together rather than peak system throughput alone to make practical selection decisions for deployment and research.

Sources

Related papers