EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling

arXiv:2602.10811 · cs.IR · Submitted 2026-02-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling".

Jane: Here's the gist of the paper. Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention <ref:2602.10811#pg4>.

Tom: In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modality-specific priors of content-rich signals <ref:2602.10811#pg3>.

Jane: Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation <ref:2602.10811#pg4>.

Tom: Extensive experiments show that EST exhibits a stable and efficient power-law scaling relationship, enabling predictable performance gains with model scale <ref:2602.10811#pg4>.

Paper summary: Tom: So, to wrap up this discussion on "EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling," the main point is their proposal for a fully unified model using LCA and CSA. They argue that by finding these specific information asymmetries and modality priors, we can design an architecture that scales predictably without massive computational overhead.

Jane: And the implication for us listeners is that this isn't just some theoretical AI exercise. It’s about how to build systems—like ad platforms—that can handle massive amounts of user behavior while still staying efficient enough to be practical in a real-world setting, showing gains like one point two two percent CTR lift.

Lu: They set the stage by identifying that the asymmetry between concise non-behavioral queries and massive behavioral contexts is key, which guides us toward using cross-attention specifically for those interactions.

Meng: From an engineer's view, it means we should be looking beyond standard self-attention when dealing with long sequence data because we can use content similarity to prune the search space dynamically instead of processing everything blindly.

Lalam: I am confident in their finding that this approach provides a clear roadmap for practitioners seeking to explore efficient scaling in industrial CTR prediction models by demonstrating stable power-law scaling.

Conclusion: Tom: So, we're wrapping up this look at EST, which stands for Efficiently Scalable Transformer—it’s about getting Click-Through Rate prediction to scale better without just throwing more compute at it.

Jane: Right, so the title itself says it’s about finding those efficient scaling laws using a unified modeling approach. It sounds pretty academic, but basically, they're trying to solve the problem of making these massive CTR models practical for real business use.

Lu: The authors are showing how they revisit the fundamental differences between CTR prediction and Large Language Models, pointing out some key structural properties in how information is spread across those two kinds of tasks.

Meng: I just want to know what this means for deployment, because right now, most of us are just running models on limited budgets without knowing if we’re hitting a wall soon.

Lalam: Essentially, the paper proposes a way to process all the inputs—the user's history and the ad content—in one single sequence instead of doing separate things for each part.

Tom: So, what is this unified modeling actually achieving here? Is it just making things look cleaner on paper?

Jane: No, it’s about eliminating that information bottleneck they talked about earlier. They found that when you try to scale these systems up by just aggregating user behaviors in a simple way, you lose the fine details of what actually matters.

Lu: The core idea is that there's an asymmetry between non-behavioral features and behavioral sequences, and then they use content signals to guide which parts of the long behavior sequence are actually important to look at.

Meng: So if I’m running a model, this suggests we should stop trying to feed everything into one giant matrix and instead focus on pruning what’s redundant before we start training.

Lalam: Exactly, the authors show that by using two specific modules, Lightweight Cross-Attention and Content Sparse Attention, they can do this without losing any information in the process.

Tom: It sounds like the authors are showing us a path toward predictable performance gains as we increase model size or compute budget instead of just hoping for better results.

Jane: That’s what they demonstrate through their power-law scaling relationship, which means you can actually predict how much better things will get if you double your resources.

Lu: The practical test on Taobao's advertising platform showed concrete numbers, like a point two two percent lift in Click-Through Rate and a three point two seven percent increase in Revenue Per Mile.

Meng: Those real-world results are what matter to me—seeing those actual gains on production traffic is way more valuable than just theoretical scaling laws.

Lalam: And that’s the hook, because they’re suggesting this architecture is a practical pathway for building industrial models that actually deliver measurable value.

Taobao & Tmall group of Alibaba

cs.IR

Submitted: 2026-02-11

Updated: 2026-11-12

Comments: Accepted by KDD2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 97/100

The gist: Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention <ref:2602.10811#pg4>.

Key concepts

Asymmetry in Information Density
This concept highlights the structural difference between user data types. Non-behavioral features (like a query) are concise but high-signal, while behavioral sequences (like clicks) are massive but low-density. The paper suggests that standard attention is inefficient because it treats these vastly different data scales equally.
Modality-specific Priors
This insight states that content signals, such as images or text, provide the greatest predictive power when used for computing similarity between items rather than just being treated as simple input tokens. This suggests a better way to integrate rich media into the prediction process.
Lightweight Cross-Attention (LCA)
LCA is a module designed to reduce computational cost by focusing only on high-impact interactions between non-behavioral tokens and behavioral sequences. It treats the query as the non-behavioral token and the sequence as keys/values, reducing complexity from quadratic to bilinear.
Content Sparse Attention (CSA)
CSA uses content similarity to intelligently select which parts of a long behavior sequence are most relevant for a specific prediction. By computing content similarity first and then applying top-K sparsification, it drastically reduces the computational load while retaining high predictive signal.

Terminology

Summary

Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention <ref:2602. p. 0

Where the paper says it

xj}@alibaba-inc. com 𝒂Taobao & Tmall Group of Alibaba, Beijing, China Abstract Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention.

10811#pg4>. In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modality-specific priors of content-rich signals <ref:2602. p. 0
Where the paper says it

However, such non-unified or partially unified modeling creates an information bottleneck by discarding finegrained, token-level signals essential for unlocking scaling gains. In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modalityspecific priors of content-rich signals.

10811#pg3>. Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation <ref:2602. p. 0
Where the paper says it

In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modalityspecific priors of content-rich signals. Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation.

10811#pg4>. Extensive experiments show that EST exhibits a stable and efficient power-law scaling relationship, enabling predictable performance gains with model scale <ref:2602. p. 0
Where the paper says it

EST integrates two modules: Lightweight CrossAttention (LCA), which prunes redundant self-interactions to focus on high-impact cross-feature dependencies, and Content Sparse Attention (CSA), which utilizes content similarity to dynamically select high-signal behaviors. Extensive experiments show that EST exhibits a stable and efficient power-law scaling relationship, enabling predictable performance gains with model scale.

10811#pg4>.

Foundational Distinctions

The authors identify two critical insights derived from analyzing CTR prediction models versus LLMs <ref:2602.10811#pg5>. These insights are (1) Asymmetry in Information Density, where concise, highsignal non-behavioral features N (the query) must interact with massive, low-density behavioral sequences B (the context) <ref:2602.10811#pg5>. This structural prior suggests that dense all-to-all interactions, such as standard self-attention, are computationally redundant under strict compute budget <ref:2602.10811#pg5>. The second insight is Modality-specific Priors, stating that Content signals (e.g., images and text) deliver the largest gains when utilized to compute similarity relations rather than being directly injected as token embeddings <ref:2602.10811#pg5>.

Efficient Architectural Components

The EST architecture integrates two modules to achieve fully unified modeling <ref:2602.10811#pg6>. These modules are the Lightweight CrossAttention (LCA) and Content Sparse Attention (CSA) <ref:2602. p. 0

Where the paper says it

Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation. EST integrates two modules: Lightweight CrossAttention (LCA), which prunes redundant self-interactions to focus on high-impact cross-feature dependencies, and Content Sparse Attention (CSA), which utilizes content similarity to dynamically select high-signal behaviors.

10811#pg6>.

** Lightweight CrossAttention (LCA)**

The LCA module prunes redundant self-interactions to focus on high-impact cross-feature dependencies <ref:2602.10811#pg6>. It is designed so that non-behavioral tokens N serve as queries, while the concatenated behavioral tokens B = [Bu, Bc] serve as the keys and values <ref:2602.10811#pg6>. This design results in a computational complexity reduction from quadratic to bilinear complexity when L B >> L N <ref:2602. p. 4

Where the paper says it

Formally, the complexity is reduced as follows: Projection: 𝑂 (𝐿𝑁 + 𝐿𝐵)𝑑 2 → 𝑂(𝐿𝑁 𝑑 2 + 𝐿𝐵𝑑 2 ), Attention: 𝑂 (𝐿𝑁 + 𝐿𝐵) 2 𝑑 → 𝑂(𝐿𝑁 𝐿𝐵𝑑). In a typical industrial setting where 𝐿𝐵 ≫ 𝐿𝑁 , this reduction from quadratic to bilinear complexity enables EST to handle much longer behavioral sequences without exceeding latency budget

10811#pg6>.

** Content Sparse Attention (CSA)**

The CSA module leverages content similarity to dynamically select the most relevant behaviors <ref:2602.10811#pg6>. It computes the intra-sequence similarity matrix G γ = M γM γ⊤, where each element G γ [i, j] represents the content similarity between the i-th item and j-th item within the behavior sequence B γ <ref:2602.10811#pg7>. By applying row-wise top-K sparsification, CSA reduces complexity to scale linearly with L B γ <ref:2602. p. 5

Where the paper says it

Although simple, Equation (1) is computationally expensive due to its quadratic complexity of 𝑂(𝐿 2 B𝛾 𝑑). To reduce this cost, we apply row-wise top-𝐾 sparsification to the similarity matrix G𝛾 .

10811#pg7>.

Empirical Validation and Scaling Laws

The experiments confirm the effectiveness of the proposed design across several dimensions <ref:2602.10811#pg8>.

** Ablation Study on LCA and CSA**

Compared to full self-attention, EST's LCA reduces computation by 72.23% with only a 0.03% drop in performance <ref:2602.10811#pg8>. Furthermore, building upon LCA, CSA further boosts performance by 0.14% <ref:2602.10811#pg8>.

** Power-Law Scaling**

The model exhibits a highly efficient power-law scaling relationship: predictive performance improves consistently as model parameters and compute budget increase <ref:2602.10811#pg4>. The authors fit the gain ΔGAUC to a power-law function ΔGAUC(X) = E × X α, observing that the growth trend is steeper when increasing model depth than width <ref:2602.10811#pg9>.

Industrial Impact and Deployment

The practical deployment of EST in Taobao’s display advertising platform yielded significant results <ref:2602.10811#pg10>. In the Guess What You Like scenario, EST improves Click-Through Rate (CTR) by 1.22% and Revenue Per Mile (RPM) by 3. p. 0

Where the paper says it

Deployed on Taobao’s display advertising platform, EST significantly outperforms production baselines, delivering a 3. 27% RPM (Revenue Per Mile) increase and a 1.

27% <ref:2602.10811#pg10>. This real-world testing validates the paper's claim that EST delivers significant gains of 3.27% in RPM and 1. p. 0
Where the paper says it

Deployed on Taobao’s display advertising platform, EST significantly outperforms production baselines, delivering a 3. 27% RPM (Revenue Per Mile) increase and a 1.

p. 0
Where the paper says it

Deployed on Taobao’s display advertising platform, EST significantly outperforms production baselines, delivering a 3. 27% RPM (Revenue Per Mile) increase and a 1.

22% in CTR
<ref:2602.10811#pg10>. The overall conclusion is that the architecture provides a practical pathway for scalable industrial CTR prediction models <ref:2602.10811#pg4>.

Conclusion

The paper proposes EST, an efficiently scalable transformer-based architecture that unifies heterogeneous inputs within a single token sequence <ref:2602.10811#pg8>. By introducing LCA to focus on the most informative interactions and CSA to utilize content similarity for sparse modeling of long behaviors with little overhead, EST achieves state-of-the-art performance with a clear power-law scaling trend relative to model capacity and compute cost <ref:2602. p. 8

Where the paper says it

It also adopts CSA to utilize content similarity for sparse modeling of long behaviors with little overhead. Offline evaluations confirm EST achieves state-of-the-art performance with a clear powerlaw scaling trend relative to model capacity and compute cost.

10811#pg8>. Online A/B tests further validate its real-world impact, delivering significant gains of 3. p. 8
Where the paper says it

Offline evaluations confirm EST achieves state-of-the-art performance with a clear powerlaw scaling trend relative to model capacity and compute cost. Online A/B tests further validate its real-world impact, delivering significant gains of 3.

27% in RPM and 1.22% in CTR <ref:2602. p. 0
Where the paper says it

, sharing user-side calculations across multiple candidates in the same request. Nevertheless, existing unified approaches remain partially unified: candidate-specific arXiv:2602.

10811#pg10>.

Acknowledgement

This work was sponsored by CCF-ALIMAMA TECH Kangaroo Fund (CCF-ALIMAMA OF 2025001) <ref:2602.10811#pg10>.

REFERENCE

[3] Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. 2022. Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction. p. 8

Where the paper says it

[3] Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction.

In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM). p. 8
Where the paper says it

Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM).

p. 8
Where the paper says it

Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM).

[4] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al. p. 8

Where the paper says it

In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM). [4] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al.

2025. LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders. p. 8
Where the paper says it

[4] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al. LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders.

In Proceedings of the 19th ACM Conference on Recommender Systems (RecSys). p. 8
Where the paper says it

LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders. In Proceedings of the 19th ACM Conference on Recommender Systems (RecSys).

[5] Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al. p. 8

Where the paper says it

In Proceedings of the 19th ACM Conference on Recommender Systems (RecSys). [5] Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al.

2023. TWIN: Two-Stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. p. 8
Where the paper says it

[5] Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al. TWIN: Two-Stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou.

In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). p. 8
Where the paper says it

TWIN: Two-Stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).

p. 8
Where the paper says it

TWIN: Two-Stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).

[6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang, Shuang Li, and Kun Gai. p. 8

Where the paper says it

In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). [6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang, Shuang Li, and Kun Gai.

2024. Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction Models. p. 8
Where the paper says it

[6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang, Shuang Li, and Kun Gai. Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction.

arXiv:2407. p. 8
Where the paper says it

Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction. arXiv:2407.

01607 (2024). p. 8
Where the paper says it

arXiv:2407. 01607 (2024).

[7] Huan Gui, Ruoxi Wang, Ke Yin, Long Jin, Maciej Kula, Taibai Xu, Lichan Hong, and Ed H Chi. p. 8

Where the paper says it

01607 (2024). [7] Huan Gui, Ruoxi Wang, Ke Yin, Long Jin, Maciej Kula, Taibai Xu, Lichan Hong, and Ed H Chi.

2023. Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems. p. 8
Where the paper says it

[7] Huan Gui, Ruoxi Wang, Ke Yin, Long Jin, Maciej Kula, Taibai Xu, Lichan Hong, and Ed H Chi. Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems.

arXiv:2311. p. 8
Where the paper says it

Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems. arXiv:2311.

05884 (2023). p. 8
Where the paper says it

arXiv:2311. 05884 (2023).

[8] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. p. 8

Where the paper says it

05884 (2023). [8] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He.

2017. DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction. p. 8
Where the paper says it

[8] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction.

In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI). p. 8
Where the paper says it

DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI).

[9] Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma, Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. p. 8

Where the paper says it

In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI). [9] Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma, Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al.

2025. MTGR: IndustrialScale Generative Recommendation Framework in Meituan. p. 8
Where the paper says it

[9] Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma, Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. MTGR: IndustrialScale Generative Recommendation Framework in Meituan.

In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM). p. 8
Where the paper says it

MTGR: IndustrialScale Generative Recommendation Framework in Meituan. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM).

p. 8
Where the paper says it

MTGR: IndustrialScale Generative Recommendation Framework in Meituan. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM).

[10] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. p. 8

Where the paper says it

In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM). [10] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al.

2020. Scaling Laws for Autoregressive Generative Modeling. p. 8
Where the paper says it

[10] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling Laws for Autoregressive Generative Modeling.

arXiv:2010. p. 8
Where the paper says it

Scaling Laws for Autoregressive Generative Modeling. arXiv:2010.

14701 (2020). p. 8
Where the paper says it

arXiv:2010. 14701 (2020).

[11] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. p. 8

Where the paper says it

14701 (2020). [11] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk.

2016. Session-Based Recommendations with Recurrent Neural Networks. p. 8
Where the paper says it

[11] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-Based Recommendations with Recurrent Neural Networks.

In Proceedings of the 4th International Conference on Learning Representations (ICLR). p. 8
Where the paper says it

Session-Based Recommendations with Recurrent Neural Networks. In Proceedings of the 4th International Conference on Learning Representations (ICLR).

[12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark et al. p. 8

Where the paper says it

In Proceedings of the 4th International Conference on Learning Representations (ICLR). [12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.

2022. Training Compute-Optimal Large Language Models. p. 8
Where the paper says it

[12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training Compute-Optimal Large Language Models.

In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS). p. 8
Where the paper says it

Training Compute-Optimal Large Language Models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS).

[13] Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin Liu, Han Zhu et al. p. 8

Where the paper says it

In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS). [13] Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin Liu, Han Zhu, Yuning Jiang, Jian Xu, and Bo Zheng.

2023. PS-SA: An Efficient Self-Attention via Progressive Sampling for User Behavior Sequence Modeling. p. 8
Where the paper says it

[13] Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin Liu, Han Zhu, Yuning Jiang, Jian Xu, and Bo Zheng. PS-SA: An Efficient Self-Attention via Progressive Sampling for User Behavior Sequence Modeling.

In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM). p. 8
Where the paper says it

PS-SA: An Efficient Self-Attention via Progressive Sampling for User Behavior Sequence Modeling. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM).

[14] Jie Hu, Li Shen, and Gang Sun. p. 8

Where the paper says it

In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM). [14] Jie Hu, Li Shen, and Gang Sun.

2018. Squeeze-and-Excitation Networks. p. 8
Where the paper says it

[14] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-Excitation Networks.

In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 8
Where the paper says it

Squeeze-and-Excitation Networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. p. 8

Where the paper says it

In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [15] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang.

2019. FiBiNET: Combining Feature Importance and Bilinear Feature Interaction for Click-Through Rate Prediction. p. 8
Where the paper says it

[15] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. FiBiNET: Combining Feature Importance and Bilinear Feature Interaction for Click-Through Rate Prediction.

In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys). p. 8
Where the paper says it

FiBiNET: Combining Feature Importance and Bilinear Feature Interaction for Click-Through Rate Prediction. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys).

[16] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. p. 8

Where the paper says it

In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys). [16] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin.

2016. FieldAware Factorization Machines for CTR Prediction. p. 8
Where the paper says it

[16] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. FieldAware Factorization Machines for CTR Prediction.

In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys). p. 8
Where the paper says it

FieldAware Factorization Machines for CTR Prediction. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys).

[17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown et al. p. 8

Where the paper says it

In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys). [17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

2020. Scaling Laws for Neural Language Models. p. 8
Where the paper says it

[17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models.

arXiv:2001. p. 8
Where the paper says it

Scaling Laws for Neural Language Models. arXiv:2001.

08361 (2020). p. 8
Where the paper says it

arXiv:2001. 08361 (2020).

[18] Kaiyuan Li, Dongdong Mao, Yongxiang Tang et al. p. 8

Where the paper says it

08361 (2020). [18] Kaiyuan Li, Dongdong Mao, Yongxiang Tang, Yanhua Cheng, Yanxiang Zeng, Chao Wang, Xialong Liu, and Peng Jiang.

2025. INFNet: A Taskaware Information Flow Network for Large-Scale Recommendation Systems. p. 8
Where the paper says it

[18] Kaiyuan Li, Dongdong Mao, Yongxiang Tang, Yanhua Cheng, Yanxiang Zeng, Chao Wang, Xialong Liu, and Peng Jiang. INFNet: A Taskaware Information Flow Network for Large-Scale Recommendation Systems.

arXiv:2508. p. 8
Where the paper says it

INFNet: A Taskaware Information Flow Network for Large-Scale Recommendation Systems. arXiv:2508.

11565 (2025). p. 8
Where the paper says it

arXiv:2508. 11565 (2025).

[19] Ningyuan Li, Zhilin Zhang, Tianyan Long, Yuyao Liu, Rongquan Bai et al. p. 8

Where the paper says it

11565 (2025). [19] Ningyuan Li, Zhilin Zhang, Tianyan Long, Yuyao Liu, Rongquan Bai, Yurong Chen, Xiaotie Deng, Pengjie Wang, Chuan Yu, Jian Xu, and Bo Zheng.

2025. Beyond Advertising: Mechanism Design for Platform-Wide Marketing Service QuanZhanTui. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). p. 8
Where the paper says it

Beyond Advertising: Mechanism Design for Platform-Wide Marketing Service "QuanZhanTui". In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).

p. 8
Where the paper says it

Beyond Advertising: Mechanism Design for Platform-Wide Marketing Service "QuanZhanTui". In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).

[20] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang et al. p. 8

Where the paper says it

In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). [20] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.

2024. Deepseek-v3 Technical Report. p. 8
Where the paper says it

[20] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 Technical Report.

arXiv:2412. p. 8
Where the paper says it

Deepseek-v3 Technical Report. arXiv:2412.

19437 (2024). p. 8
Where the paper says it

arXiv:2412. 19437 (2024).

[21] Zhaocheng Liu, Zhongxiang Fan, Jian Liang et al. p. 8

Where the paper says it

In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). [6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang, Shuang Li, and Kun Gai.

2023. Multi-Epoch Learning for Deep Click-Through Rate Prediction Models. p. 8
Where the paper says it

[21] Zhaocheng Liu, Zhongxiang Fan, Jian Liang, Dongying Kong, and Han Li. Multi-Epoch Learning for Deep Click-Through Rate Prediction Models.

arXiv:2305. p. 8
Where the paper says it

Multi-Epoch Learning for Deep Click-Through Rate Prediction Models. arXiv:2305.

19531 (2023). p. 8
Where the paper says it

arXiv:2305. 19531 (2023).

[24] Matthew Richardson, Ewa Dominowska, and Robert Ragno. p. 8

Where the paper says it

In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM). [24] Matthew Richardson, Ewa Dominowska, and Robert Ragno.

2007. Predicting Clicks: Estimating the Click-Through Rate for New Ads. p. 8
Where the paper says it

[24] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting Clicks: Estimating the Click-Through Rate for New Ads.

In Proceedings of the 16th International Conference on World Wide Web (WWW). p. 8
Where the paper says it

Predicting Clicks: Estimating the Click-Through Rate for New Ads. In Proceedings of the 16th International Conference on World Wide Web (WWW).

[25] Olivier Roy and Martin Vetterli. p. 8

Where the paper says it

In Proceedings of the 16th International Conference on World Wide Web (WWW). [25] Olivier Roy and Martin Vetterli.

2007. The Effective Rank: A Measure of Effective Dimensionality. p. 8
Where the paper says it

[25] Olivier Roy and Martin Vetterli. The Effective Rank: A Measure of Effective Dimensionality.

In The 15th European Signal Processing Conference (EUSIPCO). p. 8
Where the paper says it

The Effective Rank: A Measure of Effective Dimensionality. In The 15th European Signal Processing Conference (EUSIPCO).

[26] Mark Rudelson and Roman Vershynin. p. 8

Where the paper says it

In The 15th European Signal Processing Conference (EUSIPCO). [26] Mark Rudelson and Roman Vershynin.

2007. Sampling from Large Matrices: An Approach through Geometric Functional Analysis. p. 8
Where the paper says it

[26] Mark Rudelson and Roman Vershynin. Sampling from Large Matrices: An Approach through Geometric Functional Analysis.

Journal of the ACM (JACM) (2007). p. 8
Where the paper says it

Sampling from Large Matrices: An Approach through Geometric Functional Analysis. Journal of the ACM (JACM) (2007).

[27] Ying Shan, T Ryan Hoens, Jian Jiao et al. p. 8

Where the paper says it

Journal of the ACM (JACM) (2007). [27] Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao.

2016. Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features. p. 8
Where the paper says it

[27] Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features.

In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). p. 8
Where the paper says it

Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).

[28] Noam Shazeer. p. 8

Where the paper says it

In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). [28] Noam Shazeer.

2020. GLU Variants Improve Transformer. p. 8
Where the paper says it

[28] Noam Shazeer. GLU Variants Improve Transformer.

arXiv:2002. p. 8
Where the paper says it

GLU Variants Improve Transformer. arXiv:2002.

05202 (2019). p. 8
Where the paper says it

arXiv:2002. 05202 (2020).

[31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou et al. p. 8

Where the paper says it

In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM). [31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al.

2021. One Model to Serve All: STAR Topology Adaptive Recommender for Multi-Domain CTR Prediction. p. 8
Where the paper says it

[31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al. One Model to Serve All: STAR Topology Adaptive Recommender for Multi-Domain CTR Prediction.

In Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM). p. 8
Where the paper says it

One Model to Serve All: STAR Topology Adaptive Recommender for Multi-Domain CTR Prediction. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM).

[32] Weiping Song, Chence Shi, Zhiping Xiao et al. p. 8

Where the paper says it

In Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM). [32] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang.

2019. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. p. 8
Where the paper says it

[32] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. AutoInt: Automatic Feature Interaction Learning via

In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). p. 9
Where the paper says it

Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM).

[33] Ruoxi Wang, Rakesh Shivanna, Derek Cheng et al. p. 9

Where the paper says it

In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). [33] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi.

2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems. p. 9
Where the paper says it

[33] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems.

In Proceedings of the Web Conference 2021 (WWW). p. 9
Where the paper says it

DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems. In Proceedings of the Web Conference 2021 (WWW).

[34] Zhiqiang Wang, Qingyun She, and Junlin Zhang. p. 9

Where the paper says it

In Proceedings of the Web Conference 2021 (WWW). [34] Zhiqiang Wang, Qingyun She, and Junlin Zhang.

2021. MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask. p. 9
Where the paper says it

[34] Zhiqiang Wang, Qingyun She, and Junlin Zhang. MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask.

arXiv:2102. p. 9
Where the paper says it

MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask. arXiv:2102.

07619 (2021). p. 9
Where the paper says it

arXiv:2102. 07619 (2021).

[35] Bin Wu, Feifan Yang, Zhangming Chan et al. p. 9

Where the paper says it

07619 (2021). [35] Bin Wu, Feifan Yang, Zhangming Chan, Yu-Ran Gu, Jiawei Feng, Chao Yi, Xiang-Rong Sheng, Han Zhu, Jian Xu, Mang Ye, et al.

2025. MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling in CTR Prediction at Kuaishou. p. 9
Where the paper says it

[35] Bin Wu, Feifan Yang, Zhangming Chan, Yu-Ran Gu, Jiawei Feng, Chao Yi, Xiang-Rong Sheng, Han Zhu, Jian Xu, Mang Ye, et al. MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling.

arXiv:2512. p. 9
Where the paper says it

MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling. arXiv:2512.

07216 (2025). p. 9
Where the paper says it

arXiv:2512. 07216 (2025).

[36] Kailun Wu, Weijie Bian, Zhangming Chan et al. p. 8

Where the paper says it

08774 (2023). [2] Weijie Bian, Kailun Wu, Lejian Ren, Qi Pi, Yujing Zhang, Can Xiao, Xiang-Rong Sheng, Yong-Nan Zhu, Zhangming Chan, Na Mou, et al.

2022. Adversarial Gradient Driven Exploration for Deep Click-through Rate Prediction. p. 9
Where the paper says it

[36] Kailun Wu, Weijie Bian, Zhangming Chan, Lejian Ren, Shiming Xiang, ShuGuang Han, Hongbo Deng, and Bo Zheng. Adversarial Gradient Driven Exploration for Deep Click-through Rate Prediction.

In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).

[37] Xue Xia, Pong Eksombatchai, Nikil Pancha et al. p. 9

Where the paper says it

In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). [37] Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai.

2023. TransAct: Transformer-Based Realtime User Action Model for Recommendation at Pinterest. p. 9
Where the paper says it

[37] Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai. TransAct: Transformer-Based Realtime User Action Model for Recommendation at Pinterest.

In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).

[38] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng et al. p. 9

Where the paper says it

In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). [38] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng, Di Wang, Kaiyi Lin, Pengjie Wang, Jian Xu, and Bo Zheng.

2025. From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction. p. 9
Where the paper says it

[38] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng, Di Wang, Kaiyi Lin, Pengjie Wang, Jian Xu, and Bo Zheng. From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction.

arXiv:2511. p. 9
Where the paper says it

From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction. arXiv:2511.

p. 9
Where the paper says it

From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction. arXiv:2511.

12081 (2025). p. 9
Where the paper says it

arXiv:2511. 12081 (2025).

[39] An Yang, Anfeng Li, Baosong Yang et al. p. 9

Where the paper says it

12081 (2025). [39] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.

2025. Qwen3 Technical Report. p. 9
Where the paper says it

[39] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report.

arXiv:2505. p. 9
Where the paper says it

Qwen3 Technical Report. arXiv:2505.

09388 (2025). p. 9
Where the paper says it

arXiv:2505. 09388 (2025).

[40] Jia-Qi Yang, Chenglei Dai, Dan Ou et al. p. 9

Where the paper says it

09388 (2025). [40] Jia-Qi Yang, Chenglei Dai, Dan Ou, Ju Huang, De-Chuan Zhan, Qingwen Liu, Xiaoyi Zeng, and Yang Yang.

2023. COURIER: Contrastive User Intention Reconstruction for Large-Scale Pre-Train of Image Features. p. 9
Where the paper says it

[40] Jia-Qi Yang, Chenglei Dai, Dan Ou, Ju Huang, De-Chuan Zhan, Qingwen Liu, Xiaoyi Zeng, and Yang Yang. COURIER: Contrastive User Intention Reconstruction for Large-Scale Pre-Train of Image Features.

arXiv:2306. p. 9
Where the paper says it

COURIER: Contrastive User Intention Reconstruction for Large-Scale Pre-Train of Image Features. arXiv:2306.

05001 (2023). p. 9
Where the paper says it

arXiv:2306. 05001 (2023).

[41] Liren Yu, Wenming Zhang, Silu Zhou et al. p. 9

Where the paper says it

05001 (2023). [41] Liren Yu, Wenming Zhang, Silu Zhou, Tao Zhang, Zhixuan Zhang, and Dan Ou.

2025. HHFT: Hierarchical Heterogeneous Feature Transformer for Recommendation Systems. p. 9
Where the paper says it

[41] Liren Yu, Wenming Zhang, Silu Zhou, Tao Zhang, Zhixuan Zhang, and Dan Ou. HHFT: Hierarchical Heterogeneous Feature Transformer for Recommendation Systems.

arXiv:2511.20235 (2025). p. 9
Where the paper says it

arXiv:2511. 20235 (2025).

[47] Zhaoqi Zhang, Haolei Pei, Jun Guo et al. p. 9

Where the paper says it

In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM). [47] Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun.

2025. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender. p. 9
Where the paper says it

[47] Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender.

arXiv:2510. p. 9
Where the paper says it

OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender. arXiv:2510.

26104 (2025). p. 9
Where the paper says it

arXiv:2510. 26104 (2025).

[48] Zhao-Yu Zhang, Xiang-Rong Sheng et al. p. 8

Where the paper says it

In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM). [31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al.

2023. Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Models. p. 9
Where the paper says it

[48] Zhao-Yu Zhang, Xiang-Rong Sheng, Yujing Zhang, Biye Jiang, Shuguang Han, Hongbo Deng, and Bo Zheng. Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Models.

In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM).

[49] Guorui Zhou, Na Mou, Ying Fan, Qi Pi et al. p. 9

Where the paper says it

In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM). [49] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai.

2019. Deep Interest Evolution Network for Click-Through Rate Prediction. p. 9
Where the paper says it

[49] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep Interest Evolution Network for Click-Through Rate Prediction.

In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI). p. 9
Where the paper says it

Deep Interest Evolution Network for Click-Through Rate Prediction. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI).

[50] Guorui Zhou, Xiaoqiang Zhu, Chenru Song et al. p. 9

Where the paper says it

In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI). [50] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai.

2018. Deep Interest Network for ClickThrough Rate Prediction. p. 9
Where the paper says it

[50] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep Interest Network for ClickThrough Rate Prediction.

In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). p. 9
Where the paper says it

Deep Interest Network for ClickThrough Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).

[51] Jie Zhu, Zhifang Fan, Xiaoxie Zhu et al. p. 9

Where the paper says it

In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). [51] Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, et al.

2025. RankMixer: Scaling Up Ranking Models in Industrial Recommenders. p. 9
Where the paper says it

[51] Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, et al. RankMixer: Scaling Up Ranking Models in Industrial Recommenders.

In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM).

[52] Hua Zong, Qingtao Zeng, Zhengxiong Zhou et al. p. 9

Where the paper says it

In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM). [52] Hua Zong, Qingtao Zeng, Zhengxiong Zhou, Zhihua Han, Zhensong Yan, Mingjie Liu, Hechen Sun, Jiawei Liu, Yiwen Hu, Qi Wang, YiHan Xian, Wenjie Guo, Houyuan Xiang, Zhiyuan Zeng, Xiangrong Sheng, Bencheng Yan, Nan Hu, Yuheng Huang, Jinqing Lian, Ziru Xu, Yan Zhang, Ju Huang, Siran Yang, Huimin Yi, Jiamang Wang, Pengjie Wang, Zhu Han, Jian Wu, Dan Ou, Jian Xu, Haihong Tang, Yuning Jiang, Bo Zheng, and Lin Qu.

2025. RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models. p. 9
Where the paper says it

[52] Hua Zong, Qingtao Zeng, Zhengxiong Zhou, Zhihua Han, Zhensong Yan, Mingjie Liu, Hechen Sun, Jiawei Liu, Yiwen Hu, Qi Wang, YiHan Xian, Wenjie Guo, Houyuan Xiang, Zhiyuan Zeng, Xiangrong Sheng, Bencheng Yan, Nan Hu, Yuheng Huang, Jinqing Lian, Ziru Xu, Yan Zhang, Ju Huang, Siran Yang, Huimin Yi, Jiamang Wang, Pengjie Wang, Zhu Han, Jian Wu, Dan Ou, Jian Xu, Haihong Tang, Yuning Jiang, Bo Zheng, and Lin Qu. RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models.

arXiv:2509. p. 9
Where the paper says it

RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models. arXiv:2509.

20883 (2025). p. 9
Where the paper says it

arXiv:2509. 20883 (2025).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Lightweight CrossAttention (LCA) module implementation will prune redundant self-interactions by focusing only on critical cross-feature dependencies, effectively decoupling model scaling from sequence length. This allows EST to handle much longer behavioral sequences without exceeding latency budgets through reduced computational redundancy, as shown in the complexity analysis where attention moves from quadratic to bilinear complexity.

  2. Content Sparse Attention (CSA) module will leverage fixed semantic structures of pre-trained content features to guide intra-sequence interactions, yielding a training-free and computationally efficient mechanism. This allows the system to compute content similarity matrices and perform sparse attention linearly with respect to sequence length, which is crucial for maintaining efficiency when processing long user behavior sequences.

  3. The model can achieve stable and efficient power-law scaling relationships with respect to both computation overhead and model capacity, leading to predictable performance gains as the model scales. This predictability is validated by experimental results showing that EST exhibits a well-behaved scaling law where predictive performance improves consistently with increased model scale.

  4. The system can be trained efficiently using an asynchronous paradigm where sparse parameters are initialized from historical production models and only dense parameters are trained from scratch in subsequent epochs. This design mitigates overfitting and brings an additional 0.44% GAUC gain during training, improving the robustness of industrial CTR prediction models over standard one-epoch training.

  5. The model will deliver substantial real-world business impact by achieving significant gains in both Click-Through Rate (CTR) and Revenue Per Mile (RPM) when deployed in production systems like Taobao's advertising platform. Specifically, it can improve CTR by 1.22% and RPM by 3.27% in the Guess scenario of online A/B testing.

Abstract

Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention. Existing approaches typically employ early aggregation of user behaviors to maintain efficiency. However, such non-unified or partially unified modeling creates an information bottleneck by discarding fine-grained, token-level signals essential for unlocking scaling gains. In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modality-specific priors of content-rich signals. Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation. EST integrates two modules: Lightweight Cross-Attention (LCA), which prunes redundant self-interactions to focus on high-impact cross-feature dependencies, and Content Sparse Attention (CSA), which utilizes content similarity to dynamically select high-signal behaviors. Extensive experiments show that EST exhibits a stable and efficient power-law scaling relationship, enabling predictable performance gains with model scale. Deployed on Taobao's display advertising platform, EST significantly outperforms production baselines, delivering a 3.27% RPM (Revenue Per Mile) increase and a 1.22% CTR lift, establishing a practical pathway for scalable industrial CTR prediction models.

Sources

Related papers