SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce

summary

The gist

E-commerce search often suffers from vocabulary mismatch between user queries and merchant-authored product titles…

In short

SAM-D2Q expands e-commerce search by generating better queries from product titles and images. It uses a three-stage training process involving fine-tuning, data augmentation, and reinforcement learning to create business-aligned pseudoqueries. This approach improves retrieval performance and boosts key metrics like GMV in real online tests.

Key concepts

Multimodal Document Expansion
This is the core idea of generating new search queries based on product information, which includes both text (like titles) and images. The goal is to create richer, more descriptive search terms that users might use, moving beyond simple keyword matching to capture diverse user intent.
Composite Utility Function
This is a mathematical formula used during training that balances three different goals: semantic consistency (making sure the query matches the product), commercial value (predicting how valuable new terms are for sales), and visual attribute bonus (rewarding queries that highlight important visual features).
Reinforcement Learning-based Preference Alignment
This technique uses a reward system to train the model to generate queries that maximize overall business utility. The model learns which generated expansions lead to better search results, ultimately focusing on terms that are both relevant and commercially beneficial for the e-commerce platform.
Counterfactual Visual Augmentation
This involves creating synthetic training data by masking visual details from product descriptions while keeping the original images. This helps the model learn to focus on key visual attributes, making it more robust when generating queries that rely on visual information.

Terminology used across episodes

This episode discusses

The paper

SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce · Read on arXiv

Alibaba International Digital Commerce Group

E-commerce search often suffers from vocabulary mismatch between user queries and merchant-authored product titles, since short titles cannot fully cover diverse user expressions or visual product attributes. Although Doc2Query alleviates this issue by generating pseudo-queries for document expansion, traditional methods are text-only and not optimized for e-commerce business objectives. As a result, they may produce semantically plausible but commercially ineffective expansions and miss key attributes present in product images. To this end, we propose E-commerce Search-Aligned Multimodal Doc2Query (SAM-D2Q), a business-aligned multimodal document expansion framework for e-commerce search under Boolean retrieval constraints. SAM-D2Q consists of three stages: (1) task-adapted multimodal supervised fine-tuning to enhance vision-language understanding of product titles, images, and user queries; (2) multimodal data augmentation to improve perception of key visual attributes and expansion coverage; and (3) reinforcement-learning-based preference alignment toward search business objectives, encouraging the model to generate pseudo-queries that better match user intent and commercial value. Offline experiments show that SAM-D2Q substantially improves retrieval performance over traditional Doc2Query methods. Deployed in the AliExpress production search system, SAM-D2Q improves online business metrics, increasing GMV by +3.38% and Pay Count by +2.27%.

DOI: 10.1145/3799682.3840121

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce".

Jane: Here's the gist of the paper. E-commerce search often suffers from vocabulary mismatch between user queries and merchant-authored product titles, since short titles cannot fully cover diverse user expressions or visual product attributes.

Tom: SAM-D2Q proposes a business-aligned multimodal document expansion framework for e-commerce search under Boolean retrieval constraints.

Jane: Framework Overview SAM-D2Q is a business-aligned multimodal document expansion framework designed to generate pseudoqueries that are both grounded in product content and valuable for downstream commercial performance.

Tom: It consists of three stages: (1) task-adapted multimodal supervised fine-tuning to enhance vision-language understanding of product titles, images, and user queries, (2) multimodal data augmentation to improve perception of key visual attributes and expansion coverage, and (3) reinforcement-learning-based preference alignment toward search business objectives.

Paper summary: Tom: So we're looking at this paper, SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce. The main problem they're tackling is the vocabulary mismatch between what a user types and what the product title actually says. It claims that traditional text-only methods just add keywords but those additions aren't useful for actual sales, because they miss visual stuff in pictures >

Jane: Right, so if you type something general, a short title like "blue shirt" doesn't capture all the visual details—like the exact shade of blue or the sleeve length—which are often in the image > But this paper proposes SAM-D2Q as a multimodal framework that generates these expanded queries to be both grounded in what's on the product and useful for business outcomes >

Lu: What's interesting about their approach is that they aren't just focusing on semantic enrichment, they are aligning query generation with search business objectives while making sure the queries actually work within Boolean retrieval constraints > That means the generated expansion terms have to be high-precision terms that fit into an inverted index system, which is a big technical hurdle >

Meng: I'm curious about how they handle that constraint. If you generate a query term and it doesn't match the Boolean logic of the search engine, it’s useless because it won't retrieve anything > I need to know if this framework actually produces terms that are compliant with those strict Boolean rules, or if they just make pretty-sounding phrases >

Lalam: The paper outlines three stages for SAM-D2Q. First up is task-adapted multimodal supervised fine-tuning, which focuses on getting the model to better understand product titles and images together > Then they do some multimodal data augmentation to get a better sense of those key visual attributes like color or material > And finally, they use reinforcement learning based preference alignment to push the queries toward business goals >

Tom: So the training signal itself is built in a curriculum learning way. They start with just lexical expansion, focusing only on terms not covered by the text alone > Then they move to hybrid supervision where they mask visual attributes from the text while keeping the images to build some more data > And then they finish with reinforcement learning optimization based on a composite utility function >

Jane: That utility function is what really gets interesting for me. They use a composite reward including three parts: semantic consistency, commercial value, and visual attribute bonus > So it's not just about making the query sound right; it’s about scoring how consistent the query is with the product context, how valuable those new terms are based on past search views and conversion rates, and whether the new term matches a known visual attribute in that product category >

Paper summary: Lu: The commercial value reward specifically uses historical data like page views and conversion rate to estimate the potential of new terms using Bayesian smoothing for that category level estimate > That’s an attempt to tie the query generation directly into actual downstream performance metrics, which is ambitious > I wonder how robust that Bayesian smoothing holds up when you're dealing with sparse data on specific items >

Meng: From a deployment standpoint, I worry about the cost of running this. If you're doing this expansion at scale for hundreds of millions of items, how much compute does it add compared to just using a standard text expansion model > And if those historical search view numbers are biased from certain user groups, will the commercial reward reward that bias too >

Lalam: The paper shows strong results offline. For example, they say that SAM-D2Q Full increases the Total Qual from five point three six six million to seven point one three nine million at a Top-three hundred recall level > That's a significant jump in retrieval numbers just by using this framework > And when they deployed it online on AliExpress Search through A/B testing, they saw GMV increase by three point three eight percent and Pay Count increase by two point two seven percent >

Tom: Those online metrics are what really matter for business. A thirty-three percent lift in gross merchandise value and a twenty-two percent lift in pay count sounds substantial when you're talking about real revenue growth, not just recall numbers > But I have to ask, is that performance sustainable if the underlying search patterns keep changing rapidly >

Jane: It seems they also improved sparse retrieval coverage. They reported reducing the zero result page view ratio from two point zero seven percent down to one point seven two percent > That suggests that by generating these visual terms, they're finding products that wouldn't have been found otherwise > So it sounds like the main implication is that for e-commerce, we need systems that can look at images and generate queries intelligently, not just text matching >

Lu: The core idea they are pushing is integrating visual signals into document expansion in a way that respects the operational constraints of a Boolean retrieval system > They argue this moves beyond just adding keywords to creating high-precision, business-aligned terms that adhere to how the search engine actually works > This is a shift from purely linguistic enrichment to what they call SearchAligned Multimodal Doc2Query >

Meng: So, putting it simply for someone who doesn't read the paper, this AI system takes a vague user search and uses the product image alongside the text title to create a much richer, more specific search query that is designed not just to be descriptive but also to drive actual sales metrics like revenue > That’s what I’m trying to figure out how it translates from theory into something stable enough for production systems >

Paper summary: Lalam: The authors are confident in this because they used reinforcement learning based preference alignment, which is a technique they've seen work well in aligning models to specific goals > They claim that by optimizing against that composite utility function involving semantic, commercial, and visual rewards, the system produces expansion queries that meet all those criteria simultaneously >

Tom: So the authors are really betting on the idea that if you reward the model for hitting those three different objectives—being semantically consistent with the product context, having commercial value based on historical data, and being visually grounded by category attributes—the resulting search terms will be much more effective than anything we've seen before >

Jane: And that’s what makes this paper relevant to anyone in e-commerce or any field where product description is short but the visual information is crucial > The takeaway for us is that simply expanding text isn't enough anymore; you have to incorporate the multimodal nature of the product and tie your generation process directly to what actually makes a business successful >

Lu: The limitation they point out, and this I find important, is that they are proposing this under Boolean retrieval constraints, which suggests it’s specifically tailored for systems that use those structures rather than probabilistic models like BM25 > If you were using a system based purely on probabilistic matching, the implementation details of making these terms strictly compliant with the Boolean logic would be a different kind of challenge >

Meng: So it's focused on a specific type of search engine architecture. For us, if we're running massive systems that rely on learned embeddings or probabilistic ranking, we might need to adapt this idea because the direct alignment with Boolean matching is what gives them that precision advantage over other methods >

Lalam: The paper’s main contribution is showing a feasible way to deploy multimodal doc2query under those strict Boolean rules while simultaneously optimizing for business outcomes like conversion rate and GMV, which they show yields concrete gains in real-world online testing > This work validates the approach of using these multi-faceted rewards to guide query expansion in a commercial context >

Tom: So, to wrap up SAM-D2Q, it’s an industrial formulation that uses multimodal SFT, counterfactual data synthesis, and reinforcement learning with rewards for semantics, commerce, and visuals to create better search queries for e-commerce > It shows that the engineering side of aligning language models with business goals is achievable when you define your reward function this way >

Jane: And what it means for the broader world is that as products get more visual in nature, like clothing or electronics, we can stop treating product titles as just a list of words and start treating them as incomplete descriptions that need to be filled in by looking at the whole product context > This kind of system could fundamentally change how users discover things online because it goes beyond simple keyword matching into something more grounded in actual visual understanding >

Conclusion: Tom: So, we’re wrapping up on SAM-D2Q, which is this framework for generating better search queries by using both the product image and text to align with actual sales goals on e-commerce sites.

Jane: Right, it boils down to taking those vague user searches and turning them into specific queries that have a chance of actually leading to a purchase by making sure they match what the product is visually and what people actually buy.

Lu: The authors are using this three-stage process—fine-tuning for visual understanding, augmenting the data with counterfactual samples, and then using reinforcement learning to align the model with business rewards.

Meng: What’s really neat is how they build that reward function; it doesn't just reward being accurate in terms of words, but it adds a bonus for semantic consistency, commercial value based on past sales data, and matching visual attributes.

Tom: That commercial value part is what I find compelling because it ties the query generation directly to real business metrics like conversion rate and gross merchandise value.

Jane: Exactly. They are using historical page views and conversion rates to estimate how valuable a new search term might be, which is a smart way to make sure the AI isn't just generating pretty-sounding text but text that actually works for the business.

Lu: The main novelty here is applying this multimodal document expansion specifically under Boolean retrieval constraints, meaning the generated terms have to fit neatly into how search engines operate.

Meng: That’s important because if you’re deploying this at scale, those terms absolutely need to respect those structural rules of the search system.

Tom: So, the authors are showing that by optimizing against these three distinct rewards—semantics, commerce, and visuals—they can create expansion queries that perform better than just traditional text-based methods.

Jane: They did show some solid offline results where they improved retrieval quality significantly and then they tested it online on AliExpress Search to see if those performance gains translated into real money for the business.

Lu: The implication is that for e-commerce, product titles are often incomplete descriptions, and this framework provides a way to fill in those visual gaps intelligently to improve search results.

Meng: It suggests that the next evolution of search isn't just about matching keywords; it’s about understanding the whole product context visually and aligning that understanding with what drives sales.

Tom: It’s a practical approach, combining model training, data synthesis, and business logic into one system for improving how we find products online.

Jane: And that’s what we have so far on SAM-D2Q; it shows how integrating visual understanding and business goals can create much more effective search tools for e-commerce.

More episodes

← Home