CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent".
Jane: Here's the gist of the paper. CCFormer proposes an efficient Transformer backbone that unifies cross-field feature interaction and compressed long-sequence modeling for industrial recommendation.
Tom: Specifically, CCFormer combines feature-field separated cross attention with long-sequence subspace token mixing to exploit long-term preference signals across heterogeneous feature domains.
Jane: A hierarchical sequence compression strategy with progressively expanded receptive fields enables efficient long-sequence modeling with reduced information loss.
Tom: Extensive experiments on two public benchmarks and a large-scale industrial dataset demonstrate that CCFormer consistently outperforms stateof-the-art baselines.
Title and authors: Tom: So we're looking at the paper CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent. It sounds like they’re tackling the problem of making these big Transformer models practical for real recommendation systems where you have massive amounts of user history and item features.
Jane: Right, so the core idea here is unifying two major things: how different types of information, like user profiles and past behavior sequences, actually talk to each other, and then finding a way to handle those long sequences without making the computer crash or take forever to run.
Lu: Exactly. The authors propose a decoupled architecture that separates the input features into distinct fields—user profiles, historical behavior sequences, and target items—and they split the interaction into two parts: cross-field interaction and intra-sequence token interaction to avoid that quadratic cost of standard self-attention over the whole set.
Meng: I always worry about those quadratic costs when we talk about long sequences. If you have a million behavior entries, standard attention just explodes in complexity; what’s the actual mechanism they use to keep things manageable?
Jane: Well, they introduce this cross-field separated cross-attention module that models interactions through three directed attention flows, which lets them handle sequential and cross-domain feature interaction. Specifically for the target field, it uses two different query projections to look at the behavior and user fields separately to capture both target history relevance and compatibility.
Tom: That sounds like they’re trying to get all those necessary connections without having every single token interact with every other token in a dense way, which is smart engineering for efficiency. But how do you actually handle the sequence length itself?
Lu: For the long-sequence modeling efficiency, they use a hierarchical sequence compression strategy that applies a one-dimensional convolutional downsampling operator along the sequence dimension across deeper layers. This allows for multi-granularity modeling where shallow layers pick up short-term patterns and deeper layers extract more abstract signals.
Meng: So they’re compressing the raw history into something smaller as it goes down the network, reducing information loss while keeping the computation low, right? What about how they handle the sequence structure itself?
Title and authors: Lu: They also introduce relative temporal-positional encoding within local behavior subspaces to reduce complexity from quadratic to O(Ls m), which captures recency-aware order information. And on top of that, they have a Long-Sequence Subspace Token Mixing module that partitions the behavior sequence into compact token subspaces and processes them using a Per-channel Feed-Forward Network instead of dense pairwise attention.
Jane: That subspace mixing sounds like a clever way to get rid of the heavy self-attention cost while still modeling those important intra-sequence dependencies among historical behaviors. Does this compression strategy help with training speed too?
Tom: Absolutely. The ablation studies showed that removing the long-sequence token mixing leads to the largest degradation in AUC and GAUC, which tells you that it’s a key component for performance, but more importantly for deployment, removing the sequence compression significantly decreases training speedup from two point two one times down to one point two nine times.
Meng: one point two nine times is a big difference when we're talking about training throughput at scale. And I want to ask, how does this whole thing actually run in production? What’s the real cost versus what they’re claiming?
Jane: They show that CCFormer exhibits predictable scaling behavior with sequence length and it demonstrates a favorable effectiveness-efficiency trade-off by requiring fewer GFLOPs per sample than HSTU. And for industrial practicality, they ran online A/B tests in a video recommendation scenario and an advertising ranking scenario at Tencent where they saw gains of three point five seven percent in CTR and one point seven one percent lift in advertising revenue.
Tom: A three point five seven percent CTR gain is substantial if that’s achieved with a lower computational cost than the previous state-of-the-art, which is what they were compared against. It proves the method actually delivers on the promise of efficiency for large industrial datasets.
Lu: The paper confirms that CCFormer consistently outperforms strong sequential baselines on both Taobao and KuaiRec benchmarks, showing that this unification of cross-field interaction and compression works well across different real-world recommendation contexts.
Jane: So, to summarize, CCFormer takes the challenge of scaling long sequences in industrial recommendation by combining feature-field separated cross attention with subspace token mixing for efficient intra-sequence modeling and a hierarchical compression strategy to keep the computational cost down. What this really changes is that you can get much better cross-domain interaction while still maintaining acceptable latency for large systems.
Title and authors: Tom: So it’s about getting high performance on those big datasets without needing to scale up the model capacity to an unsustainable degree, which is exactly what we need when we ship models to production environments.
Meng: From a deployment standpoint, the fact that they showed favorable scaling behavior and demonstrated lower GFLOPs per sample than HSTU means there’s a real path for us to deploy this without immediately needing massive GPU clusters for every single user interaction.
Lu: And it's important to note, the authors also flagged that sequence compression is key for training efficiency, as removing it significantly slows down the speedup factor we saw in the training phase.
Jane: Before we wrap up this look at CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent, I want to make sure everyone has weighed in on what this means practically. Tom, what’s your final thought on the efficiency gains they highlighted?
Tom: It’s about moving beyond just sequence length scaling and showing how you can get better feature interaction across fields while keeping the computational demands manageable for real-world industrial use cases.
Jane: And from my perspective, it shows that we don't have to choose between deep, complex cross-domain modeling and being fast enough for a production system.
Lu: I think the combination of relative temporal encoding and the subspace token mixing is what really lets it capture those nuanced long-term preference signals effectively within those constraints.
Meng: For me, it’s about knowing that we can deploy something that actually performs well on real traffic without immediately having to re-engineer the entire inference pipeline for massive parallel attention.
Lalam: I'm confident in this direction; based on my analysis of the literature, this unified approach of cross-field interaction and hierarchical compression is a sound strategy for tackling industrial recommendation scaling issues.
Tom: So that’s CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent, showing how you can get high performance without necessarily needing to scale up the model capacity dramatically.
The paper's summary: Tom: So, we’re looking at CCFormer, and it basically says they unified feature interaction—cross attention—with long sequence modeling compression for industrial recommendation systems at Tencent.
Jane: That’s right. The authors are showing how they can get better cross-field interaction between things like user history and item features while also handling those massive behavior sequences efficiently with compression.
Lu: They use a hierarchical token compression strategy where the representations get condensed as they go deeper into the layers, which helps them model multi-granularity patterns.
Meng: So it’s not just one trick; it’s a whole pipeline from feature separation all the way down to sequence length management. What does that actually mean for deployment?
Tom: It means they showed this method outperforms state-of-the-art baselines on public benchmarks, but more importantly, they have online A/B tests running in Tencent’s actual video recommendation and advertising ranking systems.
Jane: Which is interesting because those tests showed a three point five seven percent gain in click-through rate and a one point seven one percent lift in advertising revenue.
Lu: Those online results suggest the trade-off they found between effectiveness and efficiency works out well for real-world traffic, showing favorable scaling behavior compared to other methods like HSTU.
Meng: Favorable scaling is good, but I need to know how much GFLOPs it actually saves per sample when we’re running this on live data.
Tom: The ablation study showed that CCFormer requires fewer GFLOPs per sample than HSTU, which means it’s a better trade-off for computational resources.
Jane: So, the core takeaway is that they managed to tackle the quadratic cost of long sequences by using subspace token mixing and hierarchical compression, while simultaneously getting richer interaction between different feature domains.
Lu: Exactly. The relative temporal-positional encoding helps capture those recency signals in the behavior sequence while the subspace mixing handles the intra-sequence dependencies without dense attention.
Meng: I’m still cautious about deployment costs, though. What happens if we try to push this to a much longer sequence than what they tested? Does it break?
Tom: The paper shows predictable scaling behavior with sequence length, which is promising for handling longer sequences in the future.
Jane: It seems like the authors are setting up CCFormer as a strong foundation for industrial recommendation systems because it balances high performance gains with manageable computational demands and real-world validation.
Lu: The fact that they’ve fully deployed this in Tencent’s production system is significant evidence that this isn't just theoretical work.
Meng: So, the immediate implication is that we can probably look at their architecture for building more robust systems where sequence length is a major bottleneck.
Tom: And before we move on to how they handled those cross-field interactions in detail, let’s just remember that this whole system is built around separating user profiles, behavior sequences, and target items into distinct semantic fields.
The paper's improvements: Tom: So, we’re looking at how CCFormer suggests improving the system further, focusing on those cross-field interactions and sequence modeling parts we just talked about.
Jane: The authors propose adding relative temporal-positional encoding to the behavior sequences, which is essentially a way to give importance based on how far apart two behaviors are in time.
Lu: That’s designed to help the model distinguish recent behaviors from older ones, making it more sensitive to things happening right now.
Meng: So we’re adding another layer of temporal awareness without just throwing more raw data at it? What's the practical impact on training time?
Tom: The ablation study confirmed that this relative encoding is important for capturing time-sensitive user interests, and it helps the model learn those patterns better.
Jane: It’s about making sure the AI doesn't just look at a long list of past actions, but understands the sequence in terms of time gaps between them.
Lu: And they also show that their subspace token mixing module is a good alternative for handling intra-sequence dependencies, which keeps the computational cost much lower than dense attention.
Meng: Lower computational cost is key for deployment because we’re running this on millions of requests; if you can cut down the complexity, it means less hardware needed per user interaction.
Tom: Exactly. And they also mention that their hierarchical compression strategy allows for multi-granularity modeling, which means shallow layers get short-term stuff and deeper layers get abstract signals.
Jane: So it’s a two-pronged approach: one part handles the sequence structure itself, and the other part manages how features from different domains talk to each other across those sequences.
Lu: The implication here is that we can build recommendation models that are both deep in their feature interaction capabilities and computationally lean enough for industrial scale.
Meng: I’m still concerned about what happens when you push sequence lengths even further than the tests showed; does the compression strategy actually hold up under heavy load?
Tom: The scaling analysis showed CCFormer exhibits predictable scaling behavior with sequence length, which suggests it can handle more data than we might initially expect.
Jane: It means the method seems robust enough to handle longer history, provided you keep those compression and mixing modules in place.
Lu: The authors are hinting at future work involving lifelong-scale sequences, which is a big ambition for this research direction.
Meng: Lifelong scale is huge; that suggests we could model users over their entire lifetime rather than just the last few months of data.
Tom: It shows the potential to model much longer user preference signals, provided the efficiency gains hold up when we extend the sequence length significantly.
Conclusion: Tom: So we’re wrapping up on CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent, which basically shows how they combined cross attention with compression to make long sequence modeling practical for recommendation.
Jane: That’s right. The paper proves you can get high performance while keeping the computational cost down by splitting feature interaction and compressing the behavior history.
Lu: It really shows that unification of those two ideas, cross-field interaction and hierarchical compression, is a solid strategy for getting better results on long sequences in industrial recommendation tasks.
Meng: I’m just thinking about the deployment implications—if this works reliably on benchmark data, does it hold up when we introduce real-world noise and massive sequence variations?
Tom: The online A/B tests at Tencent in video and advertising scenarios gave us concrete numbers, showing a three point five seven percent CTR gain in one case.
Jane: That’s substantial because it proves the efficiency gains translate directly into better business metrics like revenue lift.
Lu: It means the architecture is scalable and has good predictive scaling behavior with sequence length, which is a strong indicator for real-world use.
Meng: I’m still focused on that GFLOP count per sample; if we can get the GFLOPs down significantly compared to the HSTU baseline, that’s what makes it deployable without needing huge GPU clusters.
Tom: The ablation study showed fewer GFLOPs per sample, and they confirmed that sequence compression is key for training speedup, which is a big win for development cycles.
Jane: So, in short, CCFormer successfully balances deep feature interaction with efficient sequence management to solve the long-sequence modeling problem.
Lu: It's a solid piece of representation learning because it addresses the quadratic complexity issue head-on without losing much accuracy on those key metrics like AUC.
Meng: It’s interesting how they managed to keep the GFLOPs low while still achieving results that beat strong sequential baselines on Taobao and KuaiRec.
Tom: Yeah, so for anyone listening, CCFormer shows a concrete path forward for building industrial recommendation AI that is both powerful and computationally feasible.
Jane: It gives us a lot to think about when we look at how other large models can be made efficient enough to run on the actual traffic they see every day.
Lu: I’m looking forward to seeing how they extend this idea to lifelong-scale sequences next, because that opens up a whole new area of application.
Meng: Lifelong scale is ambitious; if that works out, it means we could build systems that truly remember users over years instead of just the last few months.
Tom: Alright, so we’ve covered CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent.
Jane: It was a really interesting look at how to make these complex Transformer models work in the real world.
Lu: Definitely a paper that shows how representation learning can be both powerful and resource-conscious.
Meng: I'm hopeful this approach gives us something tangible to start building on for our next production pipeline.
Lalam: I am confident in the findings; CCFormer provides a well-justified method for industrial ranking that is efficient and effective.
Tom: Alright, we’ll be back after the break with another paper on arXiv.
Yunlong Wang, Huizhe Zhang, Haonan Hu, Yudong Li, Bing Wen, Jianchao Tu, Chengxiang Zhuo, Zang Li
Tencent
cs.IR
Submitted: 2026-07-30
Updated: 2026-10-29
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 100/100
The gist: CCFormer proposes an efficient Transformer backbone that unifies cross-field feature interaction and compressed long-sequence modeling for industrial recommendation.
Key concepts
- Feature-Field Separated Cross Attention
- This mechanism allows the model to explicitly interact between different data domains, such as user profiles, historical behavior sequences, and target items. It uses directed attention flows to capture how features from one domain relate to another, enabling richer cross-domain insights that standard models miss.
- Long-Sequence Subspace Token Mixing
- Instead of applying dense attention over the entire long sequence, this module partitions the behavior history into smaller, compact token subspaces. It processes these subspaces using a Per-channel Feed-Forward Network (PFFN), which significantly reduces computational complexity while still effectively modeling dependencies within the sequence.
- Hierarchical Token Compression
- This strategy uses 1D convolutional downsampling across layers to progressively condense the representation of long sequences. Shallow layers capture short-term patterns, while deeper layers extract more abstract, long-term signals. This allows for efficient modeling of very long histories without losing critical information.
- Relative Temporal-Positional Encoding
- This encoding method adds positional information based on the time gap between sequential behaviors. It helps the model distinguish whether a behavior happened recently or long ago, allowing it to weight temporal relevance accurately and capture recency-aware preference patterns.
Terminology
Summary
CCFormer proposes an efficient Transformer backbone that unifies cross-field feature interaction and compressed long-sequence modeling for industrial recommendation. p. 0 Specifically, CCFormer combines feature-field separated cross attention with long-sequence subspace token mixing to exploit long-term preference signals across heterogeneous feature domains. p. 0 A hierarchical sequence compression strategy with progressively expanded receptive fields enables efficient long-sequence modeling with reduced information loss. p. 0 Extensive experiments on two public benchmarks and a large-scale industrial dataset demonstrate that CCFormer consistently outperforms stateof-the-art baselines. p. 0 Online A/B tests in a video recommendation scenario and an advertising ranking scenario at Tencent further validate its industrial practicality, yielding a 3. p. 057% CTR gain and a 1. p. 071% advertising revenue lift, respectively. p. 0
Architectural Design
CCFormer adopts a decoupled architectural paradigm that explicitly partitions the input feature space into three distinct semantic fields: user profiles, historical behavior sequences, and target items. This design disentangles the feature interaction process into two orthogonal components: crossfield interaction and intra-sequence token interaction to avoid quadratic cost of exhaustive self-attention over the full token set. p. 2 The core consists of L stacked interaction blocks where each block iteratively refines representations through decoupled interactions. p. 2
Feature Interaction Mechanisms
The model introduces a feature-field separated cross-attention module that models interactions through three directed attention flows to enable sequential and cross-domain feature interaction. This involves applying lightweight tokenwise feed-forward networks to user and target fields before computing directed cross-attention between user, behavior sequence, and target fields. p. 2 The target field uses two query projections to attend to the behavior and user fields separately, enabling it to capture both target-history relevance and compatibility. p. 2
Sequence Modeling Efficiency
To address the quadratic complexity of conventional self-attention over long sequences, CCFormer employs two key strategies for efficiency. First, it introduces relative temporal-positional encoding within local behavior subspaces to reduce complexity to O(Ls m) and capture recency-aware order information. p. 2 Second, it utilizes a Long-Sequence Subspace Token Mixing module that partitions the behavior sequence into compact token subspaces and processes them via a Per-channel Feed-Forward Network (PFFN) instead of dense pairwise attention. p. 3
Compression and Scaling Analysis
The final component is the Hierarchical Token Compression module, which applies a one-dimensional convolutional downsampling operator along the sequence dimension to progressively condense representations across deeper layers. This compression enables multi-granularity modeling where shallow layers capture short-term patterns and deeper layers extract abstract signals. p. 4 Scaling analysis shows that CCFormer exhibits predictable scaling behavior with sequence length, consistently outperforming HSTU under different settings. p. 6 Furthermore, the model demonstrates favorable effectiveness-efficiency trade-off by requiring fewer GFLOPs per sample than HSTU. p. 6
Practical Validation
The paper validates CCFormer through comprehensive experiments on public benchmarks and an internal industrial dataset, where it consistently achieves the best performance on both datasets. The results show that CCFormer improves over the strongest sequential baseline on both Taobao and KuaiRec by significant margins. p. 5 Online A/B testing in two Tencent scenarios further confirms its practicality, yielding substantial gains in metrics like CTR and advertising revenue. p. 0 The model has been fully deployed in Tencent’s production recommendation system, serving the main traffic of both scenarios. p. 0
Ablation Study Insights
Ablation studies confirm the contribution of each component, showing that removing long-sequence token mixing leads to the largest degradation in AUC and GAUC. Removing sequence compression significantly decreases training speedup from 2. p. 721× to 1.29×, indicating that sequence compression is key for training efficiency. p. 7 The relative temporal-positional encoding is also shown to be important for capturing time-sensitive user interests. p. 7 The subspace token mixing provides a lightweight alternative that incurs only a slight accuracy loss while substantially improving training efficiency. p. 7
Conclusion
CCFormer unifies feature-field separated cross attention with subspace token mixing and hierarchical compression for efficient long-sequence ranking. CCFormer consistently outperforms strong baselines offline and has been fully deployed in Tencent’s production recommendation system, delivering significant online gains in both a video recommendation and an advertising ranking scenario at comparable or lower computational cost. p. 7 Future work plans include extending CCFormer to lifelong-scale sequences and exploring compression-aware modeling for the retrieval stage. p. 7
References
[1] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameter Optimization. p. 7 In Advances in Neural Information Processing Systems, J. p. 7 Shawe-Taylor, R. p. 7 Zemel, P. Bartlett, F. p. 7 Pereira, and K. p. 7 Weinberger
[2] Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao
[3] Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang
[4] Jin Chen, Shangyu Zhang, Bin Hu, Chao Zhou
[5] Qiwei Chen, Huan Zhao, Wei Li
[6] Xiangyi Chen, Kousik Rajesh, Matthew Lawhon
[7] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen
[8] Yunlong Wang et al. 2027. CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent
[9] Qin Ding, Kevin Course, Linjian Ma
[10] Yufei Feng, Fuyu Lv, Weichen Shen
[11] Yulong Gu, Lixin Zou, and Chenliang Li
[12] Lin Guan, Jia-Qi Yang, Zhishan Zhao
[13] Huifeng Guo, Ruiming Tang, Yunming Ye
[14] Mingming Ha, Guanchen Wang, Linxun Chen
[15] Ruidong Han, Bin Yin, Shangyu Chen
[16] Xu Huang, Hao Zhang, Zhifang Fan
[17] Yuchen Jiang, Jie Zhu, Xintian Han
[18] Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation
[19] Weijiang Lai, Beihong Jin, Di Zhang, Siru Chen
[20] Yang Li, Tong Chen, Peng-Fei Zhang
[21] Guanyu Lin, Jinwei Luo, Yinfeng Li
[22] Mingyang Liu, Yong Bai, Zhangming Chan
[23] Mengyang Ma, Xiaopeng Li, Wanyu Wang
[24] Qi Pi, Weijie Bian, Guorui Zhou
[25] Qi Pi, Guorui Zhou, Yujing Zhang
[26] Fei Sun, Jun Liu, Jian Wu
[27] Dan Svenstrup, Jonas Meinertz Hansen
[28] Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao
[29] Jiakai Tang, Sunhao Dai, Teng Shi
[30] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov
[31] Yifei Xia, Suhan Ling, Fangcheng Fu
[32] Weinan Xu, Hengxu He, Minshi Tan
[33] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng
[34] Yuhao Yang, Zhi Ji, Zhaopeng Li
[35] Yanwu Yang and Panyu Zhai. 2022. Click-through rate prediction in online advertising: A literature review
[36] Dezhi Yi, Wei Guo, Wenyang Cui, Wenxuan He
[37] Jingyang Yuan, Huazuo Gao, Damai Dai
[38] Jiaqi Zhai, Lucy Liao, Xing Liu
[39] Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization
[40] Zhaoqi Zhang, Haolei Pei, Jun Guo
[41] Guorui Zhou, Xiaoqiang Zhu, Chenru Song
[42] Jie Zhu, Zhifang Fan, Xiaoxie Zhu
Note: The arXiv ID is XXXX for the paper CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent
. I have used XXXX as a placeholder for the actual ID since it was not provided in the text, but I will use it consistently as per your instructions. p. 0 The page numbers are derived from the provided text structure. p. 0
(Self-Correction: Since no arXiv ID was explicitly given, and the paper title is CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent
, I will use XXXX as a placeholder for the actual ID in all references as instructed.)
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Cross-field feature interaction for CTR prediction: CCFormer introduces a
feature-field separated cross attention
which enablessequential and cross-domain feature interaction,
allowing tokens from different domains (e.g., User Profile, Item Feature) to explicitly interact with the entire behavior sequence to capture long-term preference signals. -
Efficient long-sequence modeling: The model employs a
hierarchical sequence compression strategy with progressively expanded receptive fields
which allows forefficient long-sequence modeling with reduced information loss,
enabling the system to process ultra-long user behavior sequences while substantially reducing computational cost and achieving up to a 2.21x training speedup. -
Capturing recency-aware temporal patterns: CCFormer incorporates
relative temporal-positional encoding within local behavior subspaces,
which defines weights based on the time gap between behaviors, enabling the model todistinguish recent behaviors from outdated ones and better capture recency-aware preference patterns.
-
Efficient intra-sequence dependency modeling: The system utilizes a
Long-Sequence Subspace Token Mixing
module that partitions sequences into compact token subspaces and processes them via a Per-channel Feed-Forward Network (PFFN), effectively capturingintrasequence dependencies among historical behaviors
without the quadratic complexity of dense self-attention. -
Scalable training and inference optimization: The architecture is optimized using techniques such as
mixed precision training (BF16 or FP16)
andsparse parameters compression
using INT8 symmetric linear quantization, which reduces memory overhead by more than 35% and further decreases the size of ID-feature embedding tables by around 50%. -
Parallel prediction for high throughput: The design avoids cross-target information leakage by ensuring
target tokens are only used as queries and never attend to each other,
allowing multiple candidate items from the same user request to be scored in parallel, which can increase online peak QPS by 30% under the same inference resources.
Sources
- RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
- OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System
- Bending the Scaling Law Curve in Large-Scale Recommendation Systems
- Deep Session Interest Network for Click-Through Rate Prediction
- DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
- UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems
- MTGR: Industrial-Scale Generative Recommendation Framework in Meituan
- MixFormer: Co-Scaling Up Dense and Sequence in Industrial Recommenders
- TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders
- EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling
- From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction
- Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations
Related papers
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA
- One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations