Beyond Text: Text-Image Fusion for Scalable Two-Tower Retrieval

arXiv:2603.04836 · cs.IR · Submitted 2026-03-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Beyond Text: Text-Image Fusion for Scalable Two-Tower Retrieval".

Jane: Here's the gist of the paper. E-commerce customers make purchase decisions by jointly considering product text and visual information, yet most industrial retrieval systems rely almost exclusively on textual signals <ref:2603.04836#pg2> In this work, we study unified text–image fusion for two-tower retrieval models at a large-scale e-commerce platform, training on 20M query–item pairs <ref:2603.04836#pg2> We demonstrate that domain-specific fine-tuning and modality-specific query alignment between user queries and both product text and image modalities are crucial for effective multimodal retrieval <ref:2603.04836#pg2> Building on these insights, we propose a mixtureof-modality-experts fusion architecture with a bilinear interaction network to capture cross-modal complementary information <ref:2603.04836#pg2> Our model achieves up to 4.86% improvement in nDCG@1 for desirability and 2.36% for relevance over a strong text-only baseline, while preserving the pre-computable item embedding design required for production deployment <ref:2603.04836#pg2> Pruning redundant reasoning tokens Large-scale e-commerce retrieval systems are typically optimized around textual relevance, encoding user queries and product descriptions into a shared embedding space to enable efficient nearestneighbor search at scale <ref:2603.04836#pg2> This design fails to reflect the inherently multimodal nature of user decision-making in online shopping <ref:2603.04836#pg2> Users often rely on visual information—such as appearance, style, color, and fine-grained details conveyed by product images—to assess relevance and make interaction decisions <ref:2603.04836#pg2> This mismatch between the multimodal evidence users consider and the unimodal signals retrieval models are trained on leads to suboptimal relevance modeling <ref:2603.04836#pg2> Prior work has shown that incorporating visual signals benefits product retrieval and that multimodal pre-training improves search quality at scale <ref:2603.04836#pg2> How the training signal is built The model is trained with two complementary supervision signals: a desirability (engagement) label yeng derived from aggregated user interactions and a semantic relevance label yrel derived from humanaligned cross-encoder scores <ref:2603.04836#pg2> Both are graded categorical signals with three levels: high interaction (y = 2), low interaction (y = 1), and no interaction (y = 0) <ref:2603.04836#pg2> The multi-objective formulation combines both label sources in the final objective Ltotal = λeng E L3−hinge (y, y ˆ eng) + λrel E L3−hinge (y, y ˆ rel), with λeng = 0.7 and λrel = 0.3 <ref:2603.04836#pg2> Where the gains come from The proposed method introduces a mixture-of-modality-experts fusion module where a gating network predicts an adaptive weight α ∈

0, 1: , computing the fused item representation as hf = αht + (1 − α)hi <ref:2603.04836#pg2> To capture fine-grained feature interactions beyond linear fusion, a multi-head bilinear interaction network is introduced where the final item embedding is hx = LayerNorm hf + MLP K k=1 W(k) t ht ⊙ W(k) i hi <ref:2603.04836#pg2> The results show that MoE+Bilinear consistently achieves the strongest performance on both desirability and relevance <ref:2603.04836#pg2> Pruning redundant reasoning tokens The curriculum training strategy decomposes the learning process into three stages with increasing supervision complexity, allowing the model to gradually acquire modality-aware and cross-modal relevance patterns <ref:2603.04836#pg2> Stage I involves Domain Adaptation of CLIP Encoders, fine-tuning them on a large-scale title–image dataset using contrastive learning <ref:2603.04836#pg2> Stage II focuses on Modality-Specific Query Alignment, where user queries are explicitly aligned with individual item modalities sequentially <ref:2603.04836#pg2> Stage III is Multimodal Fusion Alignment, aligning queries with the fused product representations produced by the mixture-ofmodality-experts architecture <ref:2603.04836#pg2> Pruning redundant reasoning tokens The deployment considerations ensure that by fusing modalities entirely within the item tower, our architecture guarantees that the image encoder and bilinear interaction network are invoked only during offline embedding computation—never at serving time <ref:2603.04836#pg2> This design choice directly enables CPU-only production deployment with zero additional serving overhead compared to the text-only baseline <ref:2603.04836#pg2> The system indexes a corpus of approximately 8 million products, each represented by a 512-dimensional fused embedding stored in a ScaNN approximate nearest-neighbor (ANN) index <ref:2603.04836#pg2> Pruning redundant reasoning tokens The key enablers for the observed gains are domain fine-tuning and modality-specific query alignment, as shown by the ablation studies <ref:2603.04836#pg2> Query alignment yields substantially larger gains than domain fine-tuning, highlighting the importance of reducing the mismatch between user intent and item representations <ref:2603.04836#pg2> The bilinear interaction activations are stronger for products where relevance is jointly determined by text and image cues, confirming that both adaptive modality weighting and explicit cross-modal interaction are essential <ref:2603.04836#pg2> The paper concludes that bridging the multimodal gap yields consistent and substantial improvements in both user engagement and semantic relevance metrics <ref:2603.04836#pg2> Pruning redundant reasoning tokens The paper proposes a mixture-of-modality-experts fusion architecture with a bilinear interaction network that enables effective integration of textual and visual representations for scalable two-tower retrieval <ref:2603.04836#pg2> This approach preserves the pre-computation property and keeps serving latency identical to the text-only baseline <ref:2603.04836#pg2> Future work will explore incorporating visual encoders into the query tower to support image-based queries <ref:2603.04836#pg2> The authors thank Tianbao Yang and Xiyuan Wei for their insightful discussions and valuable guidance during this work <ref:2603.04836#pg2> The work is presented at Conference’17, July 2017, Washington, DC, USA Qujiaheng Zhang, Guangyue Xu, and Alex Fengjie Li <ref:2603.04836#pg2> Springer, 289–303 <ref:2603.04836#pg2> Pruning redundant reasoning tokens The authors acknowledge the open-source FastCLIP codebase which provided helpful references and implementation insights for their research <ref:2603.04836#pg2> The paper is available as arXiv:2603.04836v2

cs.IR: 6 Jul 2026 <ref:2603.04836#pg2> Pruning redundant reasoning tokens The paper is published in ACM, New York, NY, USA, 6 pages <ref:2603.04836#pg2> The ACM Reference Format indicates this work was presented at Conference’17, July 2017, Washington, DC, USA Qujiaheng Zhang, Guangyue Xu, and Alex Fengjie Li <ref:2603.04836#pg2> This paper is cited as ACM Reference Format: Qujiaheng Zhang, Guangyue Xu, and Alex Fengjie Li.

Jane: Beyond Text: Text–Image Fusion for Scalable Two-Tower Retrieval.

Tom: In.

Title and authors: Tom: So we're diving into the title and authors of this work, "Beyond Text: Text-Image Fusion for Scalable Two-Tower Retrieval". It’s clear they are focusing on moving past just text retrieval to something that uses both text and images together.

Jane: The authors are Qujiaheng Zhang, Guangyue Xu, and Alex Fengjie Li from Target in Minneapolis, USA. They're tackling a problem where industrial retrieval systems have been stuck almost exclusively on textual signals for years.

Lu: What they are really doing here is studying unified text image fusion for two-tower retrieval models at this large scale e-commerce platform, training on twenty million query item pairs to see how multimodal information helps.

Meng: That sounds ambitious given the scale of e-commerce data. What's the immediate implication for a company that has already built a successful text-only system? Are we just adding complexity?

Lalam: The paper makes several contributions, one being demonstrating that visual signals provide significant and consistent retrieval improvements, especially for visually driven product categories, and they point to domain fine tuning and modality specific query alignment as the key enablers.

The paper's summary: Tom: To summarize what this paper is actually doing, they show that visual signals do provide significant and consistent retrieval improvements across the board, particularly when dealing with products where appearance matters a lot. It identifies domain fine tuning and modality specific query alignment as the essential steps for making this work effectively.

Jane: The core of their proposal is building a mixture of modality experts fusion architecture combined with a bilinear interaction network to effectively integrate those textual and visual representations into one scalable two-tower retrieval model.

Lu: They introduced a multi-stage curriculum training strategy along with a multi objective loss that jointly models user engagement and semantic relevance, which is what helps them demonstrate consistent improvements across both engagement based and relevance based offline evaluation benchmarks.

Meng: The loss function sounds complex. How are they weighting the importance of desirability versus semantic relevance in that training objective? That seems like a tricky balance to get right.

Lalam: They use a multi objective formulation where the total loss is a weighted combination of two separate losses, one for desirability, which they call engagement label yeng derived from aggregated user interactions, and another for semantic relevance label yrel derived from human aligned cross encoder scores.

The paper's improvements: Tom: Their main technical improvement seems to be that they introduce that mixture of modality experts fusion module with a bilinear interaction network. This network is what lets the model capture those cross modal complementary information between the text and image representations in a way that linear fusion can't manage.

Jane: They also have this multi stage curriculum training strategy which breaks the learning process into three stages: domain adaptation, modality specific query alignment, and multimodal fusion alignment. This gradual approach helps stabilize optimization and improves the final retrieval quality.

Lu: The results they show are pretty solid; they report up to a four point eighty-six percent improvement in nDCG at one for desirability and a two point thirty-six percent for relevance over a strong text only baseline. That’s substantial when you think about how much visual cues matter in shopping.

Meng: Those numbers sound good, but what about the training process itself? They mention they are preserving the pre computable item embedding design that is needed for production deployment, right? That’s important for us if we want to ship this.

Lalam: The design choice there is smart because it means by fusing modalities entirely within the item tower, they guarantee that the image encoder and bilinear interaction network are only invoked during offline embedding computation, never at serving time.

Conclusion: Tom: So to wrap up this paper on "Beyond Text: Text-Image Fusion for Scalable Two-Tower Retrieval", it shows that bridging the multimodal gap yields consistent and substantial improvements in both user engagement and semantic relevance metrics over just text based models.

Jane: They emphasize that domain specific fine tuning and modality specific query alignment are the key enablers, meaning we need to adapt our models specifically to e-commerce semantics and align our queries with what users rely on, both text and image.

Lu: The core of it is that they propose this mixture of modality experts fusion architecture with a bilinear interaction network which enables effective integration for scalable two tower retrieval while preserving the pre computation property needed for production deployment.

Meng: For production, the fact that the image encoder and bilinear interaction network are only used during offline embedding computation means we can deploy this on CPU-only infrastructure with zero additional serving overhead compared to a text only baseline. That’s a big win for cost control at scale.

Lalam: I just want to say that the key enablers they found were domain fine tuning and modality specific query alignment, as shown by their ablation studies, which highlights the importance of reducing the mismatch between user intent and item representations.

Tom: So, it’s a complex architecture with a multi stage training process that results in solid gains, especially when you consider how much visual information users rely on during relevance judgment. What paper are we looking at next?

Qujiaheng Zhang, Guangyue Xu, Alex Fengjie Li

Target

cs.IR

Submitted: 2026-03-05

Updated: 2026-11-16

Comments: Accepted as a regular paper to the Industry & Government Program of the 2026 IEEE International Conference on Big Data (IEEE BigData 2026), Phoenix, AZ, December 14-17, 2026. 9 pages, 6 figures, 5 tables. v2: accepted version (revised title and text). (c) 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses

Code: https://github.com/OptimizationAI/FastCLIP

Project page: https://sigirecom.github.io/ecom22Papers/paper_8392.pdf

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: E-commerce customers make purchase decisions by jointly considering product text and visual information…

Key concepts

Mixture-of-Modality-Experts Fusion Architecture
This is a system that uses a gating network to decide how much weight to give between the text representation (h_t) and the image representation (h_i) when creating the final item embedding. It adaptively chooses which modality is more important for a specific product, leading to better fusion of complementary information.
Bilinear Interaction Network
This network goes beyond simple linear combination by calculating interactions between text and image features in a bilinear way. This allows the model to capture complex, non-linear relationships where the relevance of an item depends on how text and image features work together.
Curriculum Training Strategy
The training process was structured in three stages. It started with adapting encoders to the e-commerce domain, then aligned user queries with individual modalities separately, and finally fused everything together. This step-by-step approach helped the model learn complex multimodal patterns effectively.

Terminology

Summary

E-commerce customers make purchase decisions by jointly considering product text and visual information, yet most industrial retrieval systems rely almost exclusively on textual signals <ref:2603. p. 004836#pg2> In this work, we study unified text–image fusion for two-tower retrieval models at a large-scale e-commerce platform, training on 20M query–item pairs <ref:2603. p. 004836#pg2> We demonstrate that domain-specific fine-tuning and modality-specific query alignment between user queries and both product text and image modalities are crucial for effective multimodal retrieval <ref:2603. p. 004836#pg2> Building on these insights, we propose a mixtureof-modality-experts fusion architecture with a bilinear interaction network to capture cross-modal complementary information <ref:2603. p. 004836#pg2> Our model achieves up to 4. p. 086% improvement in nDCG@1 for desirability and 2. p. 036% for relevance over a strong text-only baseline, while preserving the pre-computable item embedding design required for production deployment <ref:2603. p. 004836#pg2>

Pruning redundant reasoning tokens

Large-scale e-commerce retrieval systems are typically optimized around textual relevance, encoding user queries and product descriptions into a shared embedding space to enable efficient nearestneighbor search at scale <ref:2603.04836#pg2> This design fails to reflect the inherently multimodal nature of user decision-making in online shopping <ref:2603. p. 004836#pg2> Users often rely on visual information—such as appearance, style, color, and fine-grained details conveyed by product images—to assess relevance and make interaction decisions <ref:2603. p. 004836#pg2> This mismatch between the multimodal evidence users consider and the unimodal signals retrieval models are trained on leads to suboptimal relevance modeling <ref:2603. p. 004836#pg2> Prior work has shown that incorporating visual signals benefits product retrieval and that multimodal pre-training improves search quality at scale <ref:2603. p. 004836#pg2>

How the training signal is built

The model is trained with two complementary supervision signals: a desirability (engagement) label yeng derived from aggregated user interactions and a semantic relevance label yrel derived from humanaligned cross-encoder scores <ref:2603.04836#pg2> Both are graded categorical signals with three levels: high interaction (y = 2), low interaction (y = 1), and no interaction (y = 0) <ref:2603. p. 204836#pg2> The multi-objective formulation combines both label sources in the final objective Ltotal = λeng E L3−hinge (y, y ˆ eng) + λrel E L3−hinge (y, y ˆ rel), with λeng = 0. p. 27 and λrel = 0.3 <ref:2603. p. 004836#pg2>

Where the gains come from

The proposed method introduces a mixture-of-modality-experts fusion module where a gating network predicts an adaptive weight α ∈ [0, 1], computing the fused item representation as hf = αht + (1 − α)hi <ref:2603.04836#pg2> To capture fine-grained feature interactions beyond linear fusion, a multi-head bilinear interaction network is introduced where the final item embedding is hx = LayerNorm hf + MLP K k=1 W(k) t ht ⊙ W(k) i hi <ref:2603. p. 104836#pg2> The results show that MoE+Bilinear consistently achieves the strongest performance on both desirability and relevance <ref:2603. p. 304836#pg2>

The curriculum training strategy decomposes the learning process into three stages with increasing supervision complexity, allowing the model to gradually acquire modality-aware and cross-modal relevance patterns <ref:2603.04836#pg2> Stage I involves Domain Adaptation of CLIP Encoders, fine-tuning them on a large-scale title–image dataset using contrastive learning <ref:2603. p. 204836#pg2> Stage II focuses on Modality-Specific Query Alignment, where user queries are explicitly aligned with individual item modalities sequentially <ref:2603. p. 204836#pg2> Stage III is Multimodal Fusion Alignment, aligning queries with the fused product representations produced by the mixture-ofmodality-experts architecture <ref:2603. p. 204836#pg2>

The deployment considerations ensure that by fusing modalities entirely within the item tower, our architecture guarantees that the image encoder and bilinear interaction network are invoked only during offline embedding computation—never at serving time <ref:2603.04836#pg2> This design choice directly enables CPU-only production deployment with zero additional serving overhead compared to the text-only baseline <ref:2603. p. 404836#pg2> The system indexes a corpus of approximately 8 million products, each represented by a 512-dimensional fused embedding stored in a ScaNN approximate nearest-neighbor (ANN) index <ref:2603. p. 404836#pg2>

The key enablers for the observed gains are domain fine-tuning and modality-specific query alignment, as shown by the ablation studies <ref:2603.04836#pg2> Query alignment yields substantially larger gains than domain fine-tuning, highlighting the importance of reducing the mismatch between user intent and item representations <ref:2603. p. 304836#pg2> The bilinear interaction activations are stronger for products where relevance is jointly determined by text and image cues, confirming that both adaptive modality weighting and explicit cross-modal interaction are essential <ref:2603. p. 404836#pg2> The paper concludes that bridging the multimodal gap yields consistent and substantial improvements in both user engagement and semantic relevance metrics <ref:2603. p. 404836#pg2>

The paper proposes a mixture-of-modality-experts fusion architecture with a bilinear interaction network that enables effective integration of textual and visual representations for scalable two-tower retrieval <ref:2603.04836#pg2> This approach preserves the pre-computation property and keeps serving latency identical to the text-only baseline <ref:2603. p. 104836#pg2> Future work will explore incorporating visual encoders into the query tower to support image-based queries <ref:2603. p. 404836#pg2> The authors thank Tianbao Yang and Xiyuan Wei for their insightful discussions and valuable guidance during this work <ref:2603. p. 404836#pg2> The work is presented at Conference’17, July 2017, Washington, DC, USA Qujiaheng Zhang, Guangyue Xu, and Alex Fengjie Li <ref:2603. p. 104836#pg2> Springer, 289–303 <ref:2603. p. 504836#pg2>

The authors acknowledge the open-source FastCLIP codebase which provided helpful references and implementation insights for their research <ref:2603.04836#pg2> The paper is available as arXiv:2603. p. 004836v2 [cs. p. 0 p. 0 p. 0IR] 6 Jul 2026 <ref:2603. p. 0 p. 0 p. 004836#pg2>

The paper is published in ACM, New York, NY, USA, 6 pages <ref:2603. p. 004836#pg2> The ACM Reference Format indicates this work was presented at Conference’17, July 2017, Washington, DC, USA Qujiaheng Zhang, Guangyue Xu, and Alex Fengjie Li <ref:2603. p. 104836#pg2> This paper is cited as ACM Reference Format: Qujiaheng Zhang, Guangyue Xu, and Alex Fengjie Li. p. 0 2026. Beyond Text: Text–Image Fusion for Scalable Two-Tower Retrieval. p. 0 In. ACM, New York, NY, USA, 6 pages <ref:2603.04836#pg2> The paper is also referenced in the ACM Reference Format section of page 1 <ref:2603. p. 0 p. 0 p. 004836#pg2>

The paper is cited as arXiv:2603.04836v2 [cs.IR] 6 Jul 2026 <ref:2603.04836#pg2> The paper is also referenced in the ACM Reference Format section of page 1 <ref:2603.04836#pg2>

The paper is cited as arXiv:2603.04836v2 [cs.IR] 6 Jul 2026 <ref:2603.04836#pg2> The paper is also referenced in the ACM Reference Format section of page 1 <ref:2603.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Domain-specific fine-tuning is essential for aligning multimodal representations with commercial relevance, which helps ensure that visual signals are adapted to e-commerce semantics rather than just general visual features. This leads to better performance in visually driven product categories.

  2. Explicitly aligning user queries with both product text and image modalities enables the model to learn from the same visual cues that users rely on during relevance judgment, improving query-item matching accuracy.

  3. A mixture-of-modality-experts fusion architecture with a bilinear interaction network captures cross-modal complementary information, which is shown to achieve up to 4.86% improvement in nDCG@1 for desirability and 2.36% for relevance over a strong text-only baseline.

  4. A multi-stage curriculum training strategy progressively aligns model representations: first domain adaptation, then modality-specific query alignment, and finally multimodal fusion alignment, stabilizing optimization and improving final retrieval quality.

  5. The fusion-at-item-tower design ensures that the image encoder and bilinear interaction network are invoked only during offline embedding computation, which directly enables CPU-only production deployment with zero additional serving overhead compared to the text-only baseline.

Sources

Related papers