Product Interaction: An Algebraic Formalism for Deep Learning Architectures

arXiv:2602.02573 · cs.LG, cs.AI · Submitted 2026-01-31 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Product Interaction: An Algebraic Formalism for Deep Learning Architectures".

Jane: The gist: Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We’re moving on to segment two where we talk about who wrote this. The paper is called "Product Interaction: An Algebraic Formalism for Deep Learning Architectures" and it was written by Haonan Dong, Chun-Wun Cheng, and Angelica I. Aviles-Rivero.

Jane: So, before we get into the details of what they did with product interactions, can you tell us what this paper is actually trying to say in plain English? What's the main point they are making about these algebraic expressions?

Tom: They’re saying that deep learning architectures have an underlying algebraic description. They build on the idea that layers are just compositions of an operator defined over a suitable algebra, and this gives us a systematic way to generate and organize those expressions by increasing their interaction order.

Lu: The authors use feature algebra and an auxiliary structural algebra whose tensor product provides a unified representation of signals, which is pretty powerful for encoding different kinds of information together.

Meng: So they are essentially providing a universal language—a set of rules—for how these complex network structures can be built, regardless of the specific task. That sounds like it could simplify model design significantly if we can leverage that formalism.

Lalam: It suggests that the complexity we see in different architectures isn't just arbitrary design choice, but rather a result of systematically increasing the order of interaction they are allowed to have.

The paper's summary: Tom: Now let’s look at the actual summary of "Product Interaction: An Algebraic Formalism for Deep Learning Architectures." They detail how they formalize equivariance by requiring compatibility between a transformation and the algebraic product, which recovers convolutional networks (translation), harmonic networks (SO(two)), tensor field networks (SO(three)), and SE(three)-equivariant attention as symmetry-constrained product interactions <ref:2602.02573#pg1>.

Jane: So, this is where the paper connects the abstract algebra to real things we know, like translation equivariance in convolutions or rotation equivariance for three dee data <ref:2602.02573#pg1>. What does that connection actually unlock for us?

Lu: It means we can use representation theory to determine the structure constants of these algebras, and those constants encode all those pairwise interactions between the elements of our algebra.

Meng: If we can fix those structure constants according to specific objectives, like enforcing a certain symmetry, it’s a huge win for controlling how the model learns. It moves constraints from being just penalties to being built into the very structure of the network's operation.

Lalam: The paper also gives examples of algebras that facilitate auxiliary structural information, like positional information or higher-dimensional hidden state information, which means we can bake in things like sequence indices or spatial samples directly into the algebra.

The paper's improvements: Tom: Moving to the improvements section of "Product Interaction: An Algebraic Formalism for Deep Learning Architectures," they empirically demonstrate that interaction order, symmetry constraint, and algebraic structure significantly affect model performance. They suggest these three things are design principles for constructing product interactions.

Jane: So, what are these design principles in action? If we want a better model than the ones we have now, what does the paper suggest we should change about how we build it?

Lu: The paper suggests using a replacement principle: position Xone equals X makes the largest contribution to self-interaction. When accuracy drops significantly when you replace X by K, that tells you which parts of the interaction are truly important for learning.

Meng: That gives us a systematic way to analyze the contribution of each self-interaction term, which is really helpful when we’re trying to prune or simplify a model structure without losing too much performance.

Lalam: Also, incorporating symmetry constraints substantially facilitates learning unconstrained structure constants, which means we don't have to guess those parameters randomly; the symmetry guides us toward good settings.

Conclusion: Tom: We’re at the conclusion of "Product Interaction: An Algebraic Formalism for Deep Learning Architectures." To wrap up, they summarize that product interactions offer a principled approach to improving network effectiveness by leveraging the design of algebraic structures. It reduces various architectures to a single underlying operation: composing multiplication operator and then taking the product element as input.

Jane: So, what does this mean for the future of designing models? What's the big picture here?

Lu: It provides a principled way to construct interactions with any self-interaction order. This means we can systematically investigate how to effectively increase that self-interaction order in product interactions, which opens up new avenues for architectural design.

Meng: From an engineering standpoint, this framework gives us a systematic perspective for investigating alternative algebraic constructions. It’s not just another layer; it's a way to understand the underlying physics of how the computation flows.

Lalam: I think this formalism gives us a principled perspective for looking at different algebraic constructions, which is exactly what we needed to move past ad-hoc architectural design and toward something more systematic.

Tsinghua University · University of Cambridge

cs.LG, cs.AI

Submitted: 2026-01-31

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist: Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order, revealing that convolutional and equivariant networks arise as

Key concepts

Product Interactions: An algebraic formalism where neural network layers are built by composing multiplication operators defined over various algebras. This allows researchers to systematically generate and organize complex network structures by increasing the order of interaction between data elements.
Interaction Order: The polynomial degree of the input variable in an expression, representing how many data elements are multiplied together in a layer's operation. Higher orders correspond to more complex architectures like attention or Mamba, while lower orders yield simpler structures like convolutions.
Symmetry Constraints: Principles that require compatibility between a transformation (like translation) and the algebraic product. These constraints restrict the possible forms of interactions, dictating how data transformations must align with the underlying mathematical structure of the network layers.

Summary

The gist: Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order, revealing that convolutional and equivariant networks arise as symmetry-constrained linear product interactions, while attention and Mamba correspond to higher-order product interactions.

Product Interactions

Product interactions are introduced as an algebraic formalism where neural network layers are constructed from compositions of a multiplication operator defined over suitable algebras. This framework allows for the systematic generation and organization of algebraic expressions by increasing interaction order. The central observation is that algebraic expressions in modern neural networks admit a unified construction in terms of linear, quadratic, and higher-order product interactions.

Architectural Realizations and Interaction Order

The paper demonstrates how different architectures realize specific product interaction orders. Specifically, convolutional and equivariant networks arise as symmetry-constrained linear product interactions. Attention mechanisms correspond to cubic and multi-level product interactions, while Mamba corresponds to a quadratic–cubic product interaction with input-dependent gating. The self-interaction order is defined as the polynomial order of X in expression O(X, Y, Z, …).

Increasing Interaction Order

The transition from State-Space Models (SSMs) to Mamba corresponds precisely to increasing the self-interaction order from 1 to 2 by using the input as the filter in the multiplication operator. Attention increases the self-interaction order of CNN from 1 to 3, achieved through composition (2) by taking filter elements K = Oquad(X) in OK(X), resulting in cubic product interaction Ocubic(X) = OOX(X)(X). Similarly, the discrete rule of Mamba increases self-interaction order by composing order 1 gating multiplication operator (8) with the quadratic product interaction operator in (10), resulting in order 3 product interaction.

Symmetry and Algebraic Structure

The symmetry principle requires compatibility between a transformation and the algebraic product, defined as O ○ Tg = Tg ○ O. This principle imposes constraints on the structure constants, restricting the form of interactions. For instance, CNNs are realized as translation equivariant multiplication operators, and Harmonic networks are SO(2) equivariant multiplication operators. The structure constants for these algebras are determined by representation theory, such as the Clebsch–Gordan coefficients for SO(3) interactions.

Empirical Findings and Design Principles

Empirical results show that interaction order, symmetry constraint, and algebraic structure significantly affect model performance. The replacement principle suggests that position X1 = X makes the largest contribution to self-interaction as the accuracy drops significantly when replacing X by K. Furthermore, incorporating symmetry constraints substantially facilitates learning of unconstrained structure constants. The rank R projection P R provides a way to obtain a structure similar to multi-head, where each rank represents a distinct feature of the data.

Conclusion

Product interactions offer a principled approach to improving network effectiveness by leveraging the design of algebraic structures. The framework reduces the construction of various architectures to a single underlying operation, namely, composition of multiplication operator and then taking product element to be input. This provides a principled way to construct interactions with arbitrary self-interaction order. The design of algebraic structures substantially affects the effectiveness of the resulting network architectures. The paper concludes by outlining future work to systematically explore how to effectively increase the self-interaction order in product interactions. The framework provides a principled perspective for investigating alternative algebraic constructions. The paper is supported by funding from the Swiss National Science Foundation (SNSF) under grant number 20HW-1 1. The work acknowledges support from the Yau Mathematical Sciences Center, Tsinghua University and the Tsinghua University Dushi Program. The paper is available as a preprint on arXiv with the identifier 2602.02573v1. The correspondence is with Angelica I. Aviles-Rivero. The preprint date is February 4, 2026. The paper contains references to Bahdanau et al., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate in International Conference on Learning Representations, 2015. It also cites Gu & Dao, 2023 for Mamba in Linear-time sequence modeling with selective state spaces arXiv preprint arXiv:2312.00752. The paper cites Vaswani et al., 1 Attention is all you need in Advances in Neural Information Processing Systems, pp. 5998–6008, 2017. It also references Cohen & Welling, 2016 for Group equivariant convolutional networks in Proceedings of the 33rd International Conference on Machine Learning, pp. 2990–2999, 2016. It also references Fuchs et al., Se(3)-transformers: 3d roto-translation equivariant attention networks in Advances in Neural Information Processing Systems, pp. 1970–1981, 2020a. It also references Thomas et al., Tensor field networks: Rotation- and translation-equivariant neural networks for 3d point clouds in Advances in Neural Information Processing Systems, pp. 10281–10292, 2018. It also references Gu & Dao, Efficiently modeling long sequences with structured state spaces in International Conference on Learning Representations, 2022. It also references Gu & Dao, Mamba: Linear-time sequence modeling with selective state spaces arXiv preprint arXiv:2312.00752. It also references Kalmman, R. E. A new approach to linear filtering and prediction problems Journal of Basic Engineering, 82:35–45, 1960. It also references Mallat, S. Group invariant scattering Communications on Pure and Applied Mathematics, 65(10):1331–1398, 2012. It also references Marchetti et al., Position: Algebra unveils deep learning—an invitation to neuroalgebraic geometry arXiv preprint arXiv:2405.05923. It also references Wang et al., Metric convolutions: A unifying theory to adaptive convolutions arXiv preprint arXiv:2406.05400. It also references Zhang et al., Tensor product attention is all you need arXiv preprint arXiv:2501.06425. It also references Yu & Erichson, 3d steerable cnns: Learning rotationally equivariant features in volumetric data in Advances in Neural Information Processing Systems, 2018.

Improvements for AI systems

Bold header: Systematic increase in architectural expressivity via interaction order

The system can transition from linear product interactions (like CNNs) to quadratic-cubic product interactions (like Mamba and Attention) by increasing self-interaction order, providing a principled way to generate and organize algebraic expressions by increasing interaction order.

Bold header: Symmetry-constrained architecture design

Equivariance constraints allow for the systematic encoding of data symmetries, such that Convolutional networks and equivariant networks arise as symmetry-constrained linear product interactions, guiding the construction of architectures like SE(3)-equivariant attention.

Bold header: Input-dependent dynamics in sequence modeling

The system can implement input-dependent dynamics by using a gating mechanism derived from composing multiplication operators, specifically showing that Mamba increases the order of SSM from 1 to 2 and that this transition is captured by taking the filter element of multiplication operator to be input.

Bold header: Multi-level product construction for complex interactions

The system can construct highly expressive architectures like SE(3)-attention by employing a multi-level product construction, where the first level is an attention type composition and subsequent levels are obtained from quadratic product interactions, increasing the self-interaction order from 1 to 3.

Bold header: Data type fusion via cross-interaction

The system can naturally capture interactions between different data modalities (e.g., text and image) by embedding them into a tensor product algebra B1⊗B1⊗A and defining the interaction as O(X, Y) = (XY t)Y t gives cross attention.

Bold header: Feature representation via Clebsch–Gordan coefficients

For equivariant networks on Lie groups like SO(3), the system can enforce compatibility between the algebraic product and group action by requiring that the product commutes with representation, leading to structure constants determined by Clebsch–Gordan coefficient (Equation 24).

Bold header: Structure-guided learnability

The system's performance can be optimized by leveraging structural operators and symmetry constraints, as experiments show that for CNNs, incorporating symmetry constraints substantially facilitates learning, and for order 3 interactions, the replacement principle provides a systematic way to analyze the contribution of each self-interaction term.

Abstract

Modern neural architectures are largely developed as distinct empirical constructions, despite recurring principles across convolutional, attention, state-space, and equivariant models. We introduce Product Interactions, an algebraic formalism that unifies these architectures under a common construction principle and turns their design into a systematic procedure. The core shift is representing data as algebraic elements and treating multiplication, rather than linear transformation alone, as a primitive architectural operation. Architectures are generated by composing multiplication operators, with input dependence organized by self-interaction order. Within this formalism, we derive convolutional and equivariant networks as symmetry-constrained first-order product interactions, recover Mamba from state-space models as an increase in self-interaction order from one to three, and establish attention as a third-order interaction from composing the same operator. Equivariance is incorporated by constraining the algebraic product via symmetry: translation equivariance yields convolution, while SO(3)-compatibility produces the Clebsch--Gordan product and tensor-field interactions. This perspective recasts architecture design as an algebraic construction problem: the choice of algebra defines admissible interactions, symmetry constrains their structure, and composition generates higher-order architectures. The framework is thus generative. Following these principles, we construct a simplified-algebra dynamical layer that improves both accuracy and throughput over Mamba on CIFAR-10. Experiments show that interaction order affects accuracy non-monotonically and symmetry constraints boost learnability. Product Interactions provides a mathematical methodology for analyzing existing architectures and systematically designing new ones.