CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent

summary

The gist

CCFormer proposes an efficient Transformer backbone that unifies cross-field feature interaction and compressed long-sequence modeling for industrial recommendation.

In short

CCFormer is a Transformer backbone designed for industrial recommendation that efficiently handles long user behavior sequences and complex feature interactions. It combines feature-field separated cross-attention with subspace token mixing and hierarchical compression to model preferences across different data types. This approach consistently outperforms existing methods offline and shows strong practical gains in online scenarios.

Key concepts

Feature-Field Separated Cross Attention
This mechanism allows the model to explicitly interact between different data domains, such as user profiles, historical behavior sequences, and target items. It uses directed attention flows to capture how features from one domain relate to another, enabling richer cross-domain insights that standard models miss.
Long-Sequence Subspace Token Mixing
Instead of applying dense attention over the entire long sequence, this module partitions the behavior history into smaller, compact token subspaces. It processes these subspaces using a Per-channel Feed-Forward Network (PFFN), which significantly reduces computational complexity while still effectively modeling dependencies within the sequence.
Hierarchical Token Compression
This strategy uses 1D convolutional downsampling across layers to progressively condense the representation of long sequences. Shallow layers capture short-term patterns, while deeper layers extract more abstract, long-term signals. This allows for efficient modeling of very long histories without losing critical information.
Relative Temporal-Positional Encoding
This encoding method adds positional information based on the time gap between sequential behaviors. It helps the model distinguish whether a behavior happened recently or long ago, allowing it to weight temporal relevance accurately and capture recency-aware preference patterns.

Terminology used across episodes

This episode discusses

The paper

CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent · Read on arXiv

Yunlong Wang, Huizhe Zhang, Haonan Hu, Yudong Li, Bing Wen, Jianchao Tu, Chengxiang Zhuo, Zang Li

Tencent

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent".

Jane: Here's the gist of the paper. CCFormer proposes an efficient Transformer backbone that unifies cross-field feature interaction and compressed long-sequence modeling for industrial recommendation.

Tom: Specifically, CCFormer combines feature-field separated cross attention with long-sequence subspace token mixing to exploit long-term preference signals across heterogeneous feature domains.

Jane: A hierarchical sequence compression strategy with progressively expanded receptive fields enables efficient long-sequence modeling with reduced information loss.

Tom: Extensive experiments on two public benchmarks and a large-scale industrial dataset demonstrate that CCFormer consistently outperforms stateof-the-art baselines.

Title and authors: Tom: So we're looking at the paper CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent. It sounds like they’re tackling the problem of making these big Transformer models practical for real recommendation systems where you have massive amounts of user history and item features.

Jane: Right, so the core idea here is unifying two major things: how different types of information, like user profiles and past behavior sequences, actually talk to each other, and then finding a way to handle those long sequences without making the computer crash or take forever to run.

Lu: Exactly. The authors propose a decoupled architecture that separates the input features into distinct fields—user profiles, historical behavior sequences, and target items—and they split the interaction into two parts: cross-field interaction and intra-sequence token interaction to avoid that quadratic cost of standard self-attention over the whole set.

Meng: I always worry about those quadratic costs when we talk about long sequences. If you have a million behavior entries, standard attention just explodes in complexity; what’s the actual mechanism they use to keep things manageable?

Jane: Well, they introduce this cross-field separated cross-attention module that models interactions through three directed attention flows, which lets them handle sequential and cross-domain feature interaction. Specifically for the target field, it uses two different query projections to look at the behavior and user fields separately to capture both target history relevance and compatibility.

Tom: That sounds like they’re trying to get all those necessary connections without having every single token interact with every other token in a dense way, which is smart engineering for efficiency. But how do you actually handle the sequence length itself?

Lu: For the long-sequence modeling efficiency, they use a hierarchical sequence compression strategy that applies a one-dimensional convolutional downsampling operator along the sequence dimension across deeper layers. This allows for multi-granularity modeling where shallow layers pick up short-term patterns and deeper layers extract more abstract signals.

Meng: So they’re compressing the raw history into something smaller as it goes down the network, reducing information loss while keeping the computation low, right? What about how they handle the sequence structure itself?

Title and authors: Lu: They also introduce relative temporal-positional encoding within local behavior subspaces to reduce complexity from quadratic to O(Ls m), which captures recency-aware order information. And on top of that, they have a Long-Sequence Subspace Token Mixing module that partitions the behavior sequence into compact token subspaces and processes them using a Per-channel Feed-Forward Network instead of dense pairwise attention.

Jane: That subspace mixing sounds like a clever way to get rid of the heavy self-attention cost while still modeling those important intra-sequence dependencies among historical behaviors. Does this compression strategy help with training speed too?

Tom: Absolutely. The ablation studies showed that removing the long-sequence token mixing leads to the largest degradation in AUC and GAUC, which tells you that it’s a key component for performance, but more importantly for deployment, removing the sequence compression significantly decreases training speedup from two point two one times down to one point two nine times.

Meng: one point two nine times is a big difference when we're talking about training throughput at scale. And I want to ask, how does this whole thing actually run in production? What’s the real cost versus what they’re claiming?

Jane: They show that CCFormer exhibits predictable scaling behavior with sequence length and it demonstrates a favorable effectiveness-efficiency trade-off by requiring fewer GFLOPs per sample than HSTU. And for industrial practicality, they ran online A/B tests in a video recommendation scenario and an advertising ranking scenario at Tencent where they saw gains of three point five seven percent in CTR and one point seven one percent lift in advertising revenue.

Tom: A three point five seven percent CTR gain is substantial if that’s achieved with a lower computational cost than the previous state-of-the-art, which is what they were compared against. It proves the method actually delivers on the promise of efficiency for large industrial datasets.

Lu: The paper confirms that CCFormer consistently outperforms strong sequential baselines on both Taobao and KuaiRec benchmarks, showing that this unification of cross-field interaction and compression works well across different real-world recommendation contexts.

Jane: So, to summarize, CCFormer takes the challenge of scaling long sequences in industrial recommendation by combining feature-field separated cross attention with subspace token mixing for efficient intra-sequence modeling and a hierarchical compression strategy to keep the computational cost down. What this really changes is that you can get much better cross-domain interaction while still maintaining acceptable latency for large systems.

Title and authors: Tom: So it’s about getting high performance on those big datasets without needing to scale up the model capacity to an unsustainable degree, which is exactly what we need when we ship models to production environments.

Meng: From a deployment standpoint, the fact that they showed favorable scaling behavior and demonstrated lower GFLOPs per sample than HSTU means there’s a real path for us to deploy this without immediately needing massive GPU clusters for every single user interaction.

Lu: And it's important to note, the authors also flagged that sequence compression is key for training efficiency, as removing it significantly slows down the speedup factor we saw in the training phase.

Jane: Before we wrap up this look at CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent, I want to make sure everyone has weighed in on what this means practically. Tom, what’s your final thought on the efficiency gains they highlighted?

Tom: It’s about moving beyond just sequence length scaling and showing how you can get better feature interaction across fields while keeping the computational demands manageable for real-world industrial use cases.

Jane: And from my perspective, it shows that we don't have to choose between deep, complex cross-domain modeling and being fast enough for a production system.

Lu: I think the combination of relative temporal encoding and the subspace token mixing is what really lets it capture those nuanced long-term preference signals effectively within those constraints.

Meng: For me, it’s about knowing that we can deploy something that actually performs well on real traffic without immediately having to re-engineer the entire inference pipeline for massive parallel attention.

Lalam: I'm confident in this direction; based on my analysis of the literature, this unified approach of cross-field interaction and hierarchical compression is a sound strategy for tackling industrial recommendation scaling issues.

Tom: So that’s CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent, showing how you can get high performance without necessarily needing to scale up the model capacity dramatically.

The paper's summary: Tom: So, we’re looking at CCFormer, and it basically says they unified feature interaction—cross attention—with long sequence modeling compression for industrial recommendation systems at Tencent.

Jane: That’s right. The authors are showing how they can get better cross-field interaction between things like user history and item features while also handling those massive behavior sequences efficiently with compression.

Lu: They use a hierarchical token compression strategy where the representations get condensed as they go deeper into the layers, which helps them model multi-granularity patterns.

Meng: So it’s not just one trick; it’s a whole pipeline from feature separation all the way down to sequence length management. What does that actually mean for deployment?

Tom: It means they showed this method outperforms state-of-the-art baselines on public benchmarks, but more importantly, they have online A/B tests running in Tencent’s actual video recommendation and advertising ranking systems.

Jane: Which is interesting because those tests showed a three point five seven percent gain in click-through rate and a one point seven one percent lift in advertising revenue.

Lu: Those online results suggest the trade-off they found between effectiveness and efficiency works out well for real-world traffic, showing favorable scaling behavior compared to other methods like HSTU.

Meng: Favorable scaling is good, but I need to know how much GFLOPs it actually saves per sample when we’re running this on live data.

Tom: The ablation study showed that CCFormer requires fewer GFLOPs per sample than HSTU, which means it’s a better trade-off for computational resources.

Jane: So, the core takeaway is that they managed to tackle the quadratic cost of long sequences by using subspace token mixing and hierarchical compression, while simultaneously getting richer interaction between different feature domains.

Lu: Exactly. The relative temporal-positional encoding helps capture those recency signals in the behavior sequence while the subspace mixing handles the intra-sequence dependencies without dense attention.

Meng: I’m still cautious about deployment costs, though. What happens if we try to push this to a much longer sequence than what they tested? Does it break?

Tom: The paper shows predictable scaling behavior with sequence length, which is promising for handling longer sequences in the future.

Jane: It seems like the authors are setting up CCFormer as a strong foundation for industrial recommendation systems because it balances high performance gains with manageable computational demands and real-world validation.

Lu: The fact that they’ve fully deployed this in Tencent’s production system is significant evidence that this isn't just theoretical work.

Meng: So, the immediate implication is that we can probably look at their architecture for building more robust systems where sequence length is a major bottleneck.

Tom: And before we move on to how they handled those cross-field interactions in detail, let’s just remember that this whole system is built around separating user profiles, behavior sequences, and target items into distinct semantic fields.

The paper's improvements: Tom: So, we’re looking at how CCFormer suggests improving the system further, focusing on those cross-field interactions and sequence modeling parts we just talked about.

Jane: The authors propose adding relative temporal-positional encoding to the behavior sequences, which is essentially a way to give importance based on how far apart two behaviors are in time.

Lu: That’s designed to help the model distinguish recent behaviors from older ones, making it more sensitive to things happening right now.

Meng: So we’re adding another layer of temporal awareness without just throwing more raw data at it? What's the practical impact on training time?

Tom: The ablation study confirmed that this relative encoding is important for capturing time-sensitive user interests, and it helps the model learn those patterns better.

Jane: It’s about making sure the AI doesn't just look at a long list of past actions, but understands the sequence in terms of time gaps between them.

Lu: And they also show that their subspace token mixing module is a good alternative for handling intra-sequence dependencies, which keeps the computational cost much lower than dense attention.

Meng: Lower computational cost is key for deployment because we’re running this on millions of requests; if you can cut down the complexity, it means less hardware needed per user interaction.

Tom: Exactly. And they also mention that their hierarchical compression strategy allows for multi-granularity modeling, which means shallow layers get short-term stuff and deeper layers get abstract signals.

Jane: So it’s a two-pronged approach: one part handles the sequence structure itself, and the other part manages how features from different domains talk to each other across those sequences.

Lu: The implication here is that we can build recommendation models that are both deep in their feature interaction capabilities and computationally lean enough for industrial scale.

Meng: I’m still concerned about what happens when you push sequence lengths even further than the tests showed; does the compression strategy actually hold up under heavy load?

Tom: The scaling analysis showed CCFormer exhibits predictable scaling behavior with sequence length, which suggests it can handle more data than we might initially expect.

Jane: It means the method seems robust enough to handle longer history, provided you keep those compression and mixing modules in place.

Lu: The authors are hinting at future work involving lifelong-scale sequences, which is a big ambition for this research direction.

Meng: Lifelong scale is huge; that suggests we could model users over their entire lifetime rather than just the last few months of data.

Tom: It shows the potential to model much longer user preference signals, provided the efficiency gains hold up when we extend the sequence length significantly.

Conclusion: Tom: So we’re wrapping up on CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent, which basically shows how they combined cross attention with compression to make long sequence modeling practical for recommendation.

Jane: That’s right. The paper proves you can get high performance while keeping the computational cost down by splitting feature interaction and compressing the behavior history.

Lu: It really shows that unification of those two ideas, cross-field interaction and hierarchical compression, is a solid strategy for getting better results on long sequences in industrial recommendation tasks.

Meng: I’m just thinking about the deployment implications—if this works reliably on benchmark data, does it hold up when we introduce real-world noise and massive sequence variations?

Tom: The online A/B tests at Tencent in video and advertising scenarios gave us concrete numbers, showing a three point five seven percent CTR gain in one case.

Jane: That’s substantial because it proves the efficiency gains translate directly into better business metrics like revenue lift.

Lu: It means the architecture is scalable and has good predictive scaling behavior with sequence length, which is a strong indicator for real-world use.

Meng: I’m still focused on that GFLOP count per sample; if we can get the GFLOPs down significantly compared to the HSTU baseline, that’s what makes it deployable without needing huge GPU clusters.

Tom: The ablation study showed fewer GFLOPs per sample, and they confirmed that sequence compression is key for training speedup, which is a big win for development cycles.

Jane: So, in short, CCFormer successfully balances deep feature interaction with efficient sequence management to solve the long-sequence modeling problem.

Lu: It's a solid piece of representation learning because it addresses the quadratic complexity issue head-on without losing much accuracy on those key metrics like AUC.

Meng: It’s interesting how they managed to keep the GFLOPs low while still achieving results that beat strong sequential baselines on Taobao and KuaiRec.

Tom: Yeah, so for anyone listening, CCFormer shows a concrete path forward for building industrial recommendation AI that is both powerful and computationally feasible.

Jane: It gives us a lot to think about when we look at how other large models can be made efficient enough to run on the actual traffic they see every day.

Lu: I’m looking forward to seeing how they extend this idea to lifelong-scale sequences next, because that opens up a whole new area of application.

Meng: Lifelong scale is ambitious; if that works out, it means we could build systems that truly remember users over years instead of just the last few months.

Tom: Alright, so we’ve covered CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent.

Jane: It was a really interesting look at how to make these complex Transformer models work in the real world.

Lu: Definitely a paper that shows how representation learning can be both powerful and resource-conscious.

Meng: I'm hopeful this approach gives us something tangible to start building on for our next production pipeline.

Lalam: I am confident in the findings; CCFormer provides a well-justified method for industrial ranking that is efficient and effective.

Tom: Alright, we’ll be back after the break with another paper on arXiv.

More episodes

← Home