EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling".
Jane: Here's the gist of the paper. Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention <ref:2602.10811#pg4>.
Tom: In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modality-specific priors of content-rich signals <ref:2602.10811#pg3>.
Jane: Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation <ref:2602.10811#pg4>.
Tom: Extensive experiments show that EST exhibits a stable and efficient power-law scaling relationship, enabling predictable performance gains with model scale <ref:2602.10811#pg4>.
Paper summary: Tom: So, to wrap up this discussion on "EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling," the main point is their proposal for a fully unified model using LCA and CSA. They argue that by finding these specific information asymmetries and modality priors, we can design an architecture that scales predictably without massive computational overhead.
Jane: And the implication for us listeners is that this isn't just some theoretical AI exercise. It’s about how to build systems—like ad platforms—that can handle massive amounts of user behavior while still staying efficient enough to be practical in a real-world setting, showing gains like one point two two percent CTR lift.
Lu: They set the stage by identifying that the asymmetry between concise non-behavioral queries and massive behavioral contexts is key, which guides us toward using cross-attention specifically for those interactions.
Meng: From an engineer's view, it means we should be looking beyond standard self-attention when dealing with long sequence data because we can use content similarity to prune the search space dynamically instead of processing everything blindly.
Lalam: I am confident in their finding that this approach provides a clear roadmap for practitioners seeking to explore efficient scaling in industrial CTR prediction models by demonstrating stable power-law scaling.
Conclusion: Tom: So, we're wrapping up this look at EST, which stands for Efficiently Scalable Transformer—it’s about getting Click-Through Rate prediction to scale better without just throwing more compute at it.
Jane: Right, so the title itself says it’s about finding those efficient scaling laws using a unified modeling approach. It sounds pretty academic, but basically, they're trying to solve the problem of making these massive CTR models practical for real business use.
Lu: The authors are showing how they revisit the fundamental differences between CTR prediction and Large Language Models, pointing out some key structural properties in how information is spread across those two kinds of tasks.
Meng: I just want to know what this means for deployment, because right now, most of us are just running models on limited budgets without knowing if we’re hitting a wall soon.
Lalam: Essentially, the paper proposes a way to process all the inputs—the user's history and the ad content—in one single sequence instead of doing separate things for each part.
Tom: So, what is this unified modeling actually achieving here? Is it just making things look cleaner on paper?
Jane: No, it’s about eliminating that information bottleneck they talked about earlier. They found that when you try to scale these systems up by just aggregating user behaviors in a simple way, you lose the fine details of what actually matters.
Lu: The core idea is that there's an asymmetry between non-behavioral features and behavioral sequences, and then they use content signals to guide which parts of the long behavior sequence are actually important to look at.
Meng: So if I’m running a model, this suggests we should stop trying to feed everything into one giant matrix and instead focus on pruning what’s redundant before we start training.
Lalam: Exactly, the authors show that by using two specific modules, Lightweight Cross-Attention and Content Sparse Attention, they can do this without losing any information in the process.
Tom: It sounds like the authors are showing us a path toward predictable performance gains as we increase model size or compute budget instead of just hoping for better results.
Jane: That’s what they demonstrate through their power-law scaling relationship, which means you can actually predict how much better things will get if you double your resources.
Lu: The practical test on Taobao's advertising platform showed concrete numbers, like a point two two percent lift in Click-Through Rate and a three point two seven percent increase in Revenue Per Mile.
Meng: Those real-world results are what matter to me—seeing those actual gains on production traffic is way more valuable than just theoretical scaling laws.
Lalam: And that’s the hook, because they’re suggesting this architecture is a practical pathway for building industrial models that actually deliver measurable value.
Taobao & Tmall group of Alibaba
cs.IR
Submitted: 2026-02-11
Updated: 2026-11-12
Comments: Accepted by KDD2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 97/100
The gist: Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention <ref:2602.10811#pg4>.
Key concepts
- Asymmetry in Information Density
- This concept highlights the structural difference between user data types. Non-behavioral features (like a query) are concise but high-signal, while behavioral sequences (like clicks) are massive but low-density. The paper suggests that standard attention is inefficient because it treats these vastly different data scales equally.
- Modality-specific Priors
- This insight states that content signals, such as images or text, provide the greatest predictive power when used for computing similarity between items rather than just being treated as simple input tokens. This suggests a better way to integrate rich media into the prediction process.
- Lightweight Cross-Attention (LCA)
- LCA is a module designed to reduce computational cost by focusing only on high-impact interactions between non-behavioral tokens and behavioral sequences. It treats the query as the non-behavioral token and the sequence as keys/values, reducing complexity from quadratic to bilinear.
- Content Sparse Attention (CSA)
- CSA uses content similarity to intelligently select which parts of a long behavior sequence are most relevant for a specific prediction. By computing content similarity first and then applying top-K sparsification, it drastically reduces the computational load while retaining high predictive signal.
Terminology
Summary
Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention <ref:2602. p. 0 xj}@alibaba-inc. com
𝒂Taobao & Tmall Group of Alibaba, Beijing, China
Abstract
Efficiently scaling industrial Click-Through Rate (CTR) prediction
has recently attracted significant research attention. However, such non-unified or partially unified
modeling creates an information bottleneck by discarding finegrained, token-level signals essential for unlocking scaling gains. In this work, we revisit the fundamental distinctions between
CTR prediction and Large Language Models (LLMs), identifying
two critical properties: the asymmetry in information density
between behavioral and non-behavioral features, and the modalityspecific priors of content-rich signals. In this work, we revisit the fundamental distinctions between
CTR prediction and Large Language Models (LLMs), identifying
two critical properties: the asymmetry in information density
between behavioral and non-behavioral features, and the modalityspecific priors of content-rich signals. Accordingly, we propose the
Efficiently Scalable Transformer (EST), which achieves fully unified
modeling by processing all raw inputs in a single sequence without
lossy aggregation. EST integrates two modules: Lightweight CrossAttention (LCA), which prunes redundant self-interactions to
focus on high-impact cross-feature dependencies, and Content
Sparse Attention (CSA), which utilizes content similarity to
dynamically select high-signal behaviors. Extensive experiments
show that EST exhibits a stable and efficient power-law scaling
relationship, enabling predictable performance gains with model
scale.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Foundational Distinctions
The authors identify two critical insights derived from analyzing CTR prediction models versus LLMs <ref:2602.10811#pg5>. These insights are (1) Asymmetry in Information Density, where concise, highsignal non-behavioral features N (the query) must interact with massive, low-density behavioral sequences B (the context)
<ref:2602.10811#pg5>. This structural prior suggests that dense all-to-all interactions, such as standard self-attention, are computationally redundant under strict compute budget
<ref:2602.10811#pg5>. The second insight is Modality-specific Priors, stating that Content signals (e.g., images and text) deliver the largest gains when utilized to compute similarity relations rather than being directly injected as token embeddings
<ref:2602.10811#pg5>.
Efficient Architectural Components
The EST architecture integrates two modules to achieve fully unified modeling <ref:2602.10811#pg6>. These modules are the Lightweight CrossAttention (LCA) and Content Sparse Attention (CSA) <ref:2602. p. 0 Accordingly, we propose the
Efficiently Scalable Transformer (EST), which achieves fully unified
modeling by processing all raw inputs in a single sequence without
lossy aggregation. EST integrates two modules: Lightweight CrossAttention (LCA), which prunes redundant self-interactions to
focus on high-impact cross-feature dependencies, and Content
Sparse Attention (CSA), which utilizes content similarity to
dynamically select high-signal behaviors.Where the paper says it
** Lightweight CrossAttention (LCA)**
The LCA module Formally, the complexity is reduced as follows:
Projection: 𝑂
(𝐿𝑁 + 𝐿𝐵)𝑑
2
→ 𝑂(𝐿𝑁 𝑑
2 + 𝐿𝐵𝑑
2
),
Attention: 𝑂
(𝐿𝑁 + 𝐿𝐵)
2
𝑑
→ 𝑂(𝐿𝑁 𝐿𝐵𝑑). In a typical industrial setting where 𝐿𝐵 ≫ 𝐿𝑁 , this reduction from
quadratic to bilinear complexity enables EST to handle much longer
behavioral sequences without exceeding latency budgetprunes redundant self-interactions to focus on high-impact cross-feature dependencies
<ref:2602.10811#pg6>. It is designed so that non-behavioral tokens N serve as queries, while the concatenated behavioral tokens B = [Bu, Bc] serve as the keys and values
<ref:2602.10811#pg6>. This design results in a computational complexity reduction from quadratic to bilinear complexity when L B >> L N <ref:2602. p. 4Where the paper says it
** Content Sparse Attention (CSA)**
The CSA module Although simple, Equation (1) is computationally expensive due
to its quadratic complexity of 𝑂(𝐿
2
B𝛾
𝑑). To reduce this cost, we
apply row-wise top-𝐾 sparsification to the similarity matrix G𝛾 .leverages content similarity to dynamically select the most relevant behaviors
<ref:2602.10811#pg6>. It computes the intra-sequence similarity matrix G γ = M γM γ⊤, where each element G γ [i, j] represents the content similarity between the i-th item and j-th item within the behavior sequence B γ
<ref:2602.10811#pg7>. By applying row-wise top-K sparsification, CSA reduces complexity to scale linearly with L B γ <ref:2602. p. 5Where the paper says it
Empirical Validation and Scaling Laws
The experiments confirm the effectiveness of the proposed design across several dimensions <ref:2602.10811#pg8>.
** Ablation Study on LCA and CSA**
Compared to full self-attention, EST's LCA reduces computation by 72.23% with only a 0.03% drop in performance
<ref:2602.10811#pg8>. Furthermore, building upon LCA, CSA further boosts performance by 0.14%
<ref:2602.10811#pg8>.
** Power-Law Scaling**
The model exhibits a highly efficient power-law scaling relationship: predictive performance improves consistently as model parameters and compute budget increase
<ref:2602.10811#pg4>. The authors fit the gain ΔGAUC to a power-law function ΔGAUC(X) = E × X α, observing that the growth trend is steeper when increasing model depth than width
<ref:2602.10811#pg9>.
Industrial Impact and Deployment
The practical deployment of EST in Taobao’s display advertising platform yielded significant results <ref:2602.10811#pg10>. In the Deployed on Taobao’s display advertising platform, EST
significantly outperforms production baselines, delivering a 3. 27%
RPM (Revenue Per Mile) increase and a 1. Deployed on Taobao’s display advertising platform, EST
significantly outperforms production baselines, delivering a 3. 27%
RPM (Revenue Per Mile) increase and a 1. Deployed on Taobao’s display advertising platform, EST
significantly outperforms production baselines, delivering a 3. 27%
RPM (Revenue Per Mile) increase and a 1.Guess What You Like
scenario, EST improves Click-Through Rate (CTR) by 1.22% and Revenue Per Mile (RPM) by 3. p. 0
<ref:2602.10811#pg10>. This real-world testing validates the paper's claim that EST delivers Where the paper says it
significant gains of 3.27% in RPM and 1. p. 0
<ref:2602.10811#pg10>. The overall conclusion is that the architecture provides a Where the paper says it
Where the paper says it
practical pathway for scalable industrial CTR prediction models
<ref:2602.10811#pg4>.
Conclusion
The paper proposes EST, an efficiently scalable transformer-based architecture that unifies heterogeneous inputs within a single token sequence <ref:2602.10811#pg8>. By introducing LCA to focus on the most informative interactions and CSA to utilize content similarity for sparse modeling of long behaviors with little overhead, EST achieves state-of-the-art performance with a clear power-law scaling trend relative to model capacity and compute cost <ref:2602. p. 8 It
also adopts CSA to utilize content similarity for sparse modeling
of long behaviors with little overhead. Offline evaluations confirm
EST achieves state-of-the-art performance with a clear powerlaw scaling trend relative to model capacity and compute cost. Offline evaluations confirm
EST achieves state-of-the-art performance with a clear powerlaw scaling trend relative to model capacity and compute cost. Online A/B tests further validate its real-world impact, delivering
significant gains of 3. , sharing user-side calculations across
multiple candidates in the same request. Nevertheless, existing
unified approaches remain partially unified: candidate-specific
arXiv:2602.Where the paper says it
Where the paper says it
Where the paper says it
Acknowledgement
This work was sponsored by CCF-ALIMAMA TECH Kangaroo Fund (CCF-ALIMAMA OF 2025001) <ref:2602.10811#pg10>.
REFERENCE
[3] Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. 2022. Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction. p. 8 [3] Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen,
and Sheng Chen. Sampling Is All You Need on Modeling Long-Term
User Behaviors for CTR Prediction. Sampling Is All You Need on Modeling Long-Term
User Behaviors for CTR Prediction. In Proceedings of the 31st ACM International
Conference on Information and Knowledge Management (CIKM). Sampling Is All You Need on Modeling Long-Term
User Behaviors for CTR Prediction. In Proceedings of the 31st ACM International
Conference on Information and Knowledge Management (CIKM).Where the paper says it
Where the paper says it
Where the paper says it
[4] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al. p. 8 In Proceedings of the 31st ACM International
Conference on Information and Knowledge Management (CIKM). [4] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen,
Hui Lu, Wenlin Zhao, Lele Yu, et al. [4] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen,
Hui Lu, Wenlin Zhao, Lele Yu, et al. LONGER: Scaling Up Long Sequence
Modeling in Industrial Recommenders. LONGER: Scaling Up Long Sequence
Modeling in Industrial Recommenders. In Proceedings of the 19th ACM Conference
on Recommender Systems (RecSys).Where the paper says it
Where the paper says it
Where the paper says it
[5] Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al. p. 8 In Proceedings of the 19th ACM Conference
on Recommender Systems (RecSys). [5] Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun
Hui, Dewei Leng, Yanan Niu, Yang Song, et al. [5] Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun
Hui, Dewei Leng, Yanan Niu, Yang Song, et al. TWIN: Two-Stage Interest
Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. TWIN: Two-Stage Interest
Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD). TWIN: Two-Stage Interest
Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang, Shuang Li, and Kun Gai. p. 8 In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD). [6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang,
Shuang Li, and Kun Gai. [6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang,
Shuang Li, and Kun Gai. Multi-Epoch learning with Data Augmentation
for Deep Click-Through Rate Prediction. Multi-Epoch learning with Data Augmentation
for Deep Click-Through Rate Prediction. arXiv:2407. arXiv:2407. 01607 (2024).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[7] Huan Gui, Ruoxi Wang, Ke Yin, Long Jin, Maciej Kula, Taibai Xu, Lichan Hong, and Ed H Chi. p. 8 01607 (2024). [7] Huan Gui, Ruoxi Wang, Ke Yin, Long Jin, Maciej Kula, Taibai Xu, Lichan Hong,
and Ed H Chi. [7] Huan Gui, Ruoxi Wang, Ke Yin, Long Jin, Maciej Kula, Taibai Xu, Lichan Hong,
and Ed H Chi. Hiformer: Heterogeneous Feature Interactions Learning
with Transformers for Recommender Systems. Hiformer: Heterogeneous Feature Interactions Learning
with Transformers for Recommender Systems. arXiv:2311. arXiv:2311. 05884 (2023).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[8] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. p. 8 05884 (2023). [8] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. [8] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction. DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence
(IJCAI).Where the paper says it
Where the paper says it
Where the paper says it
[9] Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma, Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. p. 8 In Proceedings of the 26th International Joint Conference on Artificial Intelligence
(IJCAI). [9] Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma,
Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. [9] Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma,
Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. MTGR: IndustrialScale Generative Recommendation Framework in Meituan. MTGR: IndustrialScale Generative Recommendation Framework in Meituan. In Proceedings of the
34th ACM International Conference on Information and Knowledge Management
(CIKM). MTGR: IndustrialScale Generative Recommendation Framework in Meituan. In Proceedings of the
34th ACM International Conference on Information and Knowledge Management
(CIKM).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[10] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. p. 8 In Proceedings of the
34th ACM International Conference on Information and Knowledge Management
(CIKM). [10] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob
Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. [10] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob
Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling Laws for Autoregressive Generative Modeling. Scaling Laws for Autoregressive Generative Modeling. arXiv:2010. arXiv:2010. 14701 (2020).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[11] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. p. 8 14701 (2020). [11] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. [11] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-Based Recommendations with Recurrent Neural Networks. Session-Based Recommendations with Recurrent Neural Networks. In
Proceedings of the 4th International Conference on Learning Representations (ICLR).Where the paper says it
Where the paper says it
Where the paper says it
[12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark et al. p. 8 In
Proceedings of the 4th International Conference on Learning Representations (ICLR). [12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya,
Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, et al. [12] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya,
Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, et al. Training Compute-Optimal Large Language
Models. Training Compute-Optimal Large Language
Models. In Proceedings of the 36th International Conference on Neural Information
Processing Systems (NeurIPS).Where the paper says it
Where the paper says it
Where the paper says it
[13] Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin Liu, Han Zhu et al. p. 8 In Proceedings of the 36th International Conference on Neural Information
Processing Systems (NeurIPS). [13] Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin
Liu, Han Zhu, Yuning Jiang, Jian Xu, and Bo Zheng. [13] Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin
Liu, Han Zhu, Yuning Jiang, Jian Xu, and Bo Zheng. PS-SA: An Efficient
Self-Attention via Progressive Sampling for User Behavior Sequence Modeling. PS-SA: An Efficient
Self-Attention via Progressive Sampling for User Behavior Sequence Modeling. In Proceedings of the 32nd ACM International Conference on Information and
Knowledge Management (CIKM).Where the paper says it
Where the paper says it
Where the paper says it
[14] Jie Hu, Li Shen, and Gang Sun. p. 8 In Proceedings of the 32nd ACM International Conference on Information and
Knowledge Management (CIKM). [14] Jie Hu, Li Shen, and Gang Sun. [14] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-Excitation Networks. Squeeze-and-Excitation Networks. In 2018
IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Where the paper says it
Where the paper says it
Where the paper says it
[15] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. p. 8 In 2018
IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [15] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. [15] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. FiBiNET: Combining
Feature Importance and Bilinear Feature Interaction for Click-Through Rate
Prediction. FiBiNET: Combining
Feature Importance and Bilinear Feature Interaction for Click-Through Rate
Prediction. In Proceedings of the 13th ACM Conference on Recommender Systems
(RecSys).Where the paper says it
Where the paper says it
Where the paper says it
[16] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. p. 8 In Proceedings of the 13th ACM Conference on Recommender Systems
(RecSys). [16] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. [16] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. FieldAware Factorization Machines for CTR Prediction. FieldAware Factorization Machines for CTR Prediction. In Proceedings of the 10th
ACM Conference on Recommender Systems (RecSys).Where the paper says it
Where the paper says it
Where the paper says it
[17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown et al. p. 8 In Proceedings of the 10th
ACM Conference on Recommender Systems (RecSys). [17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess,
Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess,
Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. Scaling Laws for Neural Language Models. arXiv:2001. arXiv:2001. 08361 (2020).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[18] Kaiyuan Li, Dongdong Mao, Yongxiang Tang et al. p. 8 08361 (2020). [18] Kaiyuan Li, Dongdong Mao, Yongxiang Tang, Yanhua Cheng, Yanxiang
Zeng, Chao Wang, Xialong Liu, and Peng Jiang. [18] Kaiyuan Li, Dongdong Mao, Yongxiang Tang, Yanhua Cheng, Yanxiang
Zeng, Chao Wang, Xialong Liu, and Peng Jiang. INFNet: A Taskaware Information Flow Network for Large-Scale Recommendation Systems. INFNet: A Taskaware Information Flow Network for Large-Scale Recommendation Systems. arXiv:2508. arXiv:2508. 11565 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[19] Ningyuan Li, Zhilin Zhang, Tianyan Long, Yuyao Liu, Rongquan Bai et al. p. 8 11565 (2025). [19] Ningyuan Li, Zhilin Zhang, Tianyan Long, Yuyao Liu, Rongquan Bai, Yurong
Chen, Xiaotie Deng, Pengjie Wang, Chuan Yu, Jian Xu, and Bo Zheng. Beyond Advertising: Mechanism Design for Platform-Wide Marketing Service
"QuanZhanTui". In Proceedings of the 31st ACM SIGKDD Conference on Knowledge
Discovery and Data Mining (KDD). Beyond Advertising: Mechanism Design for Platform-Wide Marketing Service
"QuanZhanTui". In Proceedings of the 31st ACM SIGKDD Conference on Knowledge
Discovery and Data Mining (KDD).Where the paper says it
QuanZhanTui
. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). p. 8Where the paper says it
Where the paper says it
[20] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang et al. p. 8 In Proceedings of the 31st ACM SIGKDD Conference on Knowledge
Discovery and Data Mining (KDD). [20] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu,
Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. [20] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu,
Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 Technical Report. Deepseek-v3 Technical Report. arXiv:2412. arXiv:2412. 19437 (2024).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[21] Zhaocheng Liu, Zhongxiang Fan, Jian Liang et al. p. 8 In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD). [6] Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang,
Shuang Li, and Kun Gai. [21] Zhaocheng Liu, Zhongxiang Fan, Jian Liang, Dongying Kong, and Han Li. Multi-Epoch Learning for Deep Click-Through Rate Prediction Models. Multi-Epoch Learning for Deep Click-Through Rate Prediction Models. arXiv:2305. arXiv:2305. 19531 (2023).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[24] Matthew Richardson, Ewa Dominowska, and Robert Ragno. p. 8 In Proceedings of the
29th ACM International Conference on Information and Knowledge Management
(CIKM). [24] Matthew Richardson, Ewa Dominowska, and Robert Ragno. [24] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting
Clicks: Estimating the Click-Through Rate for New Ads. Predicting
Clicks: Estimating the Click-Through Rate for New Ads. In Proceedings of the
16th International Conference on World Wide Web (WWW).Where the paper says it
Where the paper says it
Where the paper says it
[25] Olivier Roy and Martin Vetterli. p. 8 In Proceedings of the
16th International Conference on World Wide Web (WWW). [25] Olivier Roy and Martin Vetterli. [25] Olivier Roy and Martin Vetterli. The Effective Rank: A Measure of Effective
Dimensionality. The Effective Rank: A Measure of Effective
Dimensionality. In The 15th European Signal Processing Conference (EUSIPCO).Where the paper says it
Where the paper says it
Where the paper says it
[26] Mark Rudelson and Roman Vershynin. p. 8 In The 15th European Signal Processing Conference (EUSIPCO). [26] Mark Rudelson and Roman Vershynin. [26] Mark Rudelson and Roman Vershynin. Sampling from Large Matrices: An
Approach through Geometric Functional Analysis. Sampling from Large Matrices: An
Approach through Geometric Functional Analysis. Journal of the ACM (JACM)
(2007).Where the paper says it
Where the paper says it
Where the paper says it
[27] Ying Shan, T Ryan Hoens, Jian Jiao et al. p. 8 Journal of the ACM (JACM)
(2007). [27] Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. [27] Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial
Features. Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial
Features. In Proceedings of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (KDD).Where the paper says it
Where the paper says it
Where the paper says it
[28] Noam Shazeer. p. 8 In Proceedings of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (KDD). [28] Noam Shazeer. [28] Noam Shazeer. GLU Variants Improve Transformer. GLU Variants Improve Transformer. arXiv:2002. arXiv:2002. 05202
(2020).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou et al. p. 8 In Proceedings of the 33rd ACM International
Conference on Information and Knowledge Management (CIKM). [31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang
Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al. [31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang
Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al. One Model
to Serve All: STAR Topology Adaptive Recommender for Multi-Domain CTR
Prediction. One Model
to Serve All: STAR Topology Adaptive Recommender for Multi-Domain CTR
Prediction. In Proceedings of the 30th ACM International Conference on Information
and Knowledge Management (CIKM).Where the paper says it
Where the paper says it
Where the paper says it
[32] Weiping Song, Chence Shi, Zhiping Xiao et al. p. 8 In Proceedings of the 30th ACM International Conference on Information
and Knowledge Management (CIKM). [32] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang,
and Jian Tang. [32] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang,
and Jian Tang. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International
Conference on Information and Knowledge Management (CIKM).Where the paper says it
Where the paper says it
Where the paper says it
[33] Ruoxi Wang, Rakesh Shivanna, Derek Cheng et al. p. 9 In Proceedings of the 28th ACM International
Conference on Information and Knowledge Management (CIKM). [33] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong,
and Ed Chi. [33] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong,
and Ed Chi. DCN V2: Improved Deep & Cross Network and Practical
Lessons for Web-Scale Learning to Rank Systems. DCN V2: Improved Deep & Cross Network and Practical
Lessons for Web-Scale Learning to Rank Systems. In Proceedings of the Web
Conference 2021 (WWW).Where the paper says it
Where the paper says it
Where the paper says it
[34] Zhiqiang Wang, Qingyun She, and Junlin Zhang. p. 9 In Proceedings of the Web
Conference 2021 (WWW). [34] Zhiqiang Wang, Qingyun She, and Junlin Zhang. [34] Zhiqiang Wang, Qingyun She, and Junlin Zhang. MaskNet: Introducing
Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask. MaskNet: Introducing
Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask. arXiv:2102. arXiv:2102. 07619 (2021).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[35] Bin Wu, Feifan Yang, Zhangming Chan et al. p. 9 07619 (2021). [35] Bin Wu, Feifan Yang, Zhangming Chan, Yu-Ran Gu, Jiawei Feng, Chao Yi,
Xiang-Rong Sheng, Han Zhu, Jian Xu, Mang Ye, et al. [35] Bin Wu, Feifan Yang, Zhangming Chan, Yu-Ran Gu, Jiawei Feng, Chao Yi,
Xiang-Rong Sheng, Han Zhu, Jian Xu, Mang Ye, et al. MUSE: A Simple
Yet Effective Multimodal Search-Based Framework for Lifelong User Interest
Modeling. MUSE: A Simple
Yet Effective Multimodal Search-Based Framework for Lifelong User Interest
Modeling. arXiv:2512. arXiv:2512. 07216 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[36] Kailun Wu, Weijie Bian, Zhangming Chan et al. p. 8 08774 (2023). [2] Weijie Bian, Kailun Wu, Lejian Ren, Qi Pi, Yujing Zhang, Can Xiao, Xiang-Rong
Sheng, Yong-Nan Zhu, Zhangming Chan, Na Mou, et al. [36] Kailun Wu, Weijie Bian, Zhangming Chan, Lejian Ren, Shiming Xiang, ShuGuang Han, Hongbo Deng, and Bo Zheng. Adversarial Gradient Driven
Exploration for Deep Click-through Rate Prediction.Where the paper says it
Where the paper says it
[37] Xue Xia, Pong Eksombatchai, Nikil Pancha et al. p. 9 In Proceedings of the 28th
ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). [37] Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei
Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang,
and Andrew Zhai. [37] Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei
Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang,
and Andrew Zhai. TransAct: Transformer-Based Realtime User Action
Model for Recommendation at Pinterest.Where the paper says it
Where the paper says it
[38] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng et al. p. 9 In Proceedings of the 29th ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining (KDD). [38] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng, Di Wang, Kaiyi Lin, Pengjie Wang, Jian
Xu, and Bo Zheng. [38] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng, Di Wang, Kaiyi Lin, Pengjie Wang, Jian
Xu, and Bo Zheng. From Scaling to Structured Expressivity: Rethinking
Transformers for CTR Prediction. From Scaling to Structured Expressivity: Rethinking
Transformers for CTR Prediction. arXiv:2511. From Scaling to Structured Expressivity: Rethinking
Transformers for CTR Prediction. arXiv:2511. arXiv:2511. 12081 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[39] An Yang, Anfeng Li, Baosong Yang et al. p. 9 12081 (2025). [39] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng,
Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. [39] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng,
Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical
Report. Qwen3 Technical
Report. arXiv:2505. arXiv:2505. 09388 (2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[40] Jia-Qi Yang, Chenglei Dai, Dan Ou et al. p. 9 09388 (2025). [40] Jia-Qi Yang, Chenglei Dai, Dan Ou, Ju Huang, De-Chuan Zhan, Qingwen Liu,
Xiaoyi Zeng, and Yang Yang. [40] Jia-Qi Yang, Chenglei Dai, Dan Ou, Ju Huang, De-Chuan Zhan, Qingwen Liu,
Xiaoyi Zeng, and Yang Yang. COURIER: Contrastive User Intention
Reconstruction for Large-Scale Pre-Train of Image Features. COURIER: Contrastive User Intention
Reconstruction for Large-Scale Pre-Train of Image Features. arXiv:2306. arXiv:2306. 05001
(2023).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[41] Liren Yu, Wenming Zhang, Silu Zhou et al. p. 9 05001
(2023). [41] Liren Yu, Wenming Zhang, Silu Zhou, Tao Zhang, Zhixuan Zhang, and
Dan Ou. [41] Liren Yu, Wenming Zhang, Silu Zhou, Tao Zhang, Zhixuan Zhang, and
Dan Ou. HHFT: Hierarchical Heterogeneous Feature Transformer for
Recommendation Systems. arXiv:2511. 20235 (2025).Where the paper says it
Where the paper says it
Where the paper says it
[47] Zhaoqi Zhang, Haolei Pei, Jun Guo et al. p. 9 In
Proceedings of the 31st ACM International Conference on Information & Knowledge
Management (CIKM). [47] Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei
Liu, and Aixin Sun. [47] Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei
Liu, and Aixin Sun. OneTrans: Unified Feature Interaction and Sequence
Modeling with One Transformer in Industrial Recommender. OneTrans: Unified Feature Interaction and Sequence
Modeling with One Transformer in Industrial Recommender. arXiv:2510. arXiv:2510. 26104
(2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[48] Zhao-Yu Zhang, Xiang-Rong Sheng et al. p. 8 In Proceedings of the 33rd ACM International
Conference on Information and Knowledge Management (CIKM). [31] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang
Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al. [48] Zhao-Yu Zhang, Xiang-Rong Sheng, Yujing Zhang, Biye Jiang, Shuguang Han,
Hongbo Deng, and Bo Zheng. Towards Understanding the Overfitting
Phenomenon of Deep Click-Through Rate Models.Where the paper says it
Where the paper says it
[49] Guorui Zhou, Na Mou, Ying Fan, Qi Pi et al. p. 9 In Proceedings of the 31st ACM
International Conference on Information and Knowledge Management (CIKM). [49] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu,
and Kun Gai. [49] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu,
and Kun Gai. Deep Interest Evolution Network for Click-Through Rate
Prediction. Deep Interest Evolution Network for Click-Through Rate
Prediction. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence
(AAAI).Where the paper says it
Where the paper says it
Where the paper says it
[50] Guorui Zhou, Xiaoqiang Zhu, Chenru Song et al. p. 9 In Proceedings of the 33rd AAAI Conference on Artificial Intelligence
(AAAI). [50] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui
Yan, Junqi Jin, Han Li, and Kun Gai. [50] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui
Yan, Junqi Jin, Han Li, and Kun Gai. Deep Interest Network for ClickThrough Rate Prediction. Deep Interest Network for ClickThrough Rate Prediction. In Proceedings of the 24th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining (KDD).Where the paper says it
Where the paper says it
Where the paper says it
[51] Jie Zhu, Zhifang Fan, Xiaoxie Zhu et al. p. 9 In Proceedings of the 24th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining (KDD). [51] Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han,
Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, et al. [51] Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han,
Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, et al. RankMixer:
Scaling Up Ranking Models in Industrial Recommenders.Where the paper says it
Where the paper says it
[52] Hua Zong, Qingtao Zeng, Zhengxiong Zhou et al. p. 9 In Proceedings of the
34th ACM International Conference on Information and Knowledge Management
(CIKM). [52] Hua Zong, Qingtao Zeng, Zhengxiong Zhou, Zhihua Han, Zhensong Yan, Mingjie
Liu, Hechen Sun, Jiawei Liu, Yiwen Hu, Qi Wang, YiHan Xian, Wenjie Guo,
Houyuan Xiang, Zhiyuan Zeng, Xiangrong Sheng, Bencheng Yan, Nan Hu,
Yuheng Huang, Jinqing Lian, Ziru Xu, Yan Zhang, Ju Huang, Siran Yang, Huimin
Yi, Jiamang Wang, Pengjie Wang, Zhu Han, Jian Wu, Dan Ou, Jian Xu, Haihong
Tang, Yuning Jiang, Bo Zheng, and Lin Qu. [52] Hua Zong, Qingtao Zeng, Zhengxiong Zhou, Zhihua Han, Zhensong Yan, Mingjie
Liu, Hechen Sun, Jiawei Liu, Yiwen Hu, Qi Wang, YiHan Xian, Wenjie Guo,
Houyuan Xiang, Zhiyuan Zeng, Xiangrong Sheng, Bencheng Yan, Nan Hu,
Yuheng Huang, Jinqing Lian, Ziru Xu, Yan Zhang, Ju Huang, Siran Yang, Huimin
Yi, Jiamang Wang, Pengjie Wang, Zhu Han, Jian Wu, Dan Ou, Jian Xu, Haihong
Tang, Yuning Jiang, Bo Zheng, and Lin Qu. RecIS: Sparse to Dense, A
Unified Training Framework for Recommendation Models. RecIS: Sparse to Dense, A
Unified Training Framework for Recommendation Models. arXiv:2509. arXiv:2509. 20883
(2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Lightweight CrossAttention (LCA) module implementation will prune redundant self-interactions by focusing only on critical cross-feature dependencies, effectively decoupling model scaling from sequence length. This allows EST to handle much longer behavioral sequences without exceeding latency budgets through reduced computational redundancy, as shown in the complexity analysis where attention moves from quadratic to bilinear complexity.
-
Content Sparse Attention (CSA) module will leverage fixed semantic structures of pre-trained content features to guide intra-sequence interactions, yielding a training-free and computationally efficient mechanism. This allows the system to compute content similarity matrices and perform sparse attention linearly with respect to sequence length, which is crucial for maintaining efficiency when processing long user behavior sequences.
-
The model can achieve stable and efficient power-law scaling relationships with respect to both computation overhead and model capacity, leading to predictable performance gains as the model scales. This predictability is validated by experimental results showing that EST exhibits a well-behaved scaling law where predictive performance improves consistently with increased model scale.
-
The system can be trained efficiently using an asynchronous paradigm where sparse parameters are initialized from historical production models and only dense parameters are trained from scratch in subsequent epochs. This design mitigates overfitting and brings an additional 0.44% GAUC gain during training, improving the robustness of industrial CTR prediction models over standard one-epoch training.
-
The model will deliver substantial real-world business impact by achieving significant gains in both Click-Through Rate (CTR) and Revenue Per Mile (RPM) when deployed in production systems like Taobao's advertising platform. Specifically, it can improve CTR by 1.22% and RPM by 3.27% in the Guess scenario of online A/B testing.
Abstract
Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention. Existing approaches typically employ early aggregation of user behaviors to maintain efficiency. However, such non-unified or partially unified modeling creates an information bottleneck by discarding fine-grained, token-level signals essential for unlocking scaling gains. In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modality-specific priors of content-rich signals. Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation. EST integrates two modules: Lightweight Cross-Attention (LCA), which prunes redundant self-interactions to focus on high-impact cross-feature dependencies, and Content Sparse Attention (CSA), which utilizes content similarity to dynamically select high-signal behaviors. Extensive experiments show that EST exhibits a stable and efficient power-law scaling relationship, enabling predictable performance gains with model scale. Deployed on Taobao's display advertising platform, EST significantly outperforms production baselines, delivering a 3.27% RPM (Revenue Per Mile) increase and a 1.22% CTR lift, establishing a practical pathway for scalable industrial CTR prediction models.
Sources
- GPT-4 Technical Report
- Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction
- Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems
- Scaling Laws for Autoregressive Generative Modeling
- Scaling Laws for Neural Language Models
- Aggregate and Broadcast: Scalable and Efficient Feature Interaction for Recommender Systems
- DeepSeek-V3 Technical Report
- Multi-Epoch Learning for Deep Click-Through Rate Prediction Models
- GLU Variants Improve Transformer
- MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask
- MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling
- From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction
- Qwen3 Technical Report
- COURIER: Contrastive User Intention Reconstruction for Large-Scale Visual Recommendation
- HHFT: Hierarchical Heterogeneous Feature Transformer for Recommendation Systems
- OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender
- RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models
Related papers
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA
- One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations