Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning

arXiv:2502.09854 · cs.CL, cs.AI, cs.LG · Submitted 2025-02-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning".

Jane: The gist The SLM framework investigates how small language models can be scaled to support multimodal search and recommendation use cases while remaining efficient enough for real-time, resource-constrained deployments.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've been looking at how they scaled these small models for multimodal search and recommendation, and now we're wrapping up what this paper is all about, titled "Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning."

Jane: That framework basically takes a tiny model—a one hundred million parameter GPT-two—and trains it to generate the right prompts for both images and text search tasks <ref:2502.09854#pg1>.

Lu: Right, so the core idea is using upside-down reinforcement learning combined with synthetic data from Llama-three to make this small AI do multimodal stuff effectively <ref:2502.09854#pg1>.

Meng: From my side, I’m still curious about the actual running costs. How much memory does this small AI actually use when it's doing these multimodal tasks in a real deployment scenario?

Jane: That’s a big question for the listener here—how efficient is this thing really when you put it into a system that needs to be fast and light <ref:2502.09854#pg1>.

Tom: The paper shows that even though it’s eighty times smaller than some of those big models, its relevance and diversity scores are within six percent of models like Llama-three 8B <ref:2502.09854#pg1>. That’s a pretty solid performance number.

Lu: The authors found that this small model can handle both text-to-image and text-to-template prompts by adding specific tokens during training, which lets it manage different types of input effectively <ref:2502.09854#pg3>.

Meng: I still have to ask about the limitations mentioned in the paper, though. What is it that they say this method doesn't do? Where does it stop working?

Tom: They explicitly state that their study highlights the potential of lightweight models as practical engines for scalable multimodal discovery bridging the gap between cutting-edge research and real-world multimodal applications such as media recommendations and creative content generation <ref:2502.09854#pg1>.

Conclusion: Tom: So we're wrapping up this look at how they scaled these small models for multimodal search and recommendation, focusing on that paper, "Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning."

Jane: That framework uses a one hundred million parameter GPT-two model trained with synthetic data from Llama-three to generate prompts for both images and text search tasks <ref:2502.09854#pg1>.

Tom: The key result is that even though this SLM is up to eighty times smaller than models like Llama-three 8B, it still hits relevance and diversity scores within six percent of those larger baselines <ref:2502.09854#pg1>.

Lu: What's interesting here is that they used distillation from a much bigger model like Llama-three to teach the smaller one how to generate those prompts, which is a smart way to get good starting material without needing huge amounts of new data.

Meng: From my side, I’m still curious about the actual running costs. How much memory does this small AI actually use when it's doing these multimodal tasks in a real deployment scenario?

Jane: That’s a big question for the listener here—how efficient is this thing really when you put it into a system that needs to be fast and light?

Meng: For production, that reduced overhead is what matters most—it means we could deploy these capabilities on more resource-constrained devices or in real-time applications where speed is critical.

Jane: It shifts the focus from just building bigger models to finding smarter ways to train and use smaller ones for specific, complex tasks like creative content generation.

Tom: Yeah, so the implication here is that scalable multimodal discovery doesn't always require massive compute; it can be achieved with more targeted training methods and smaller model architectures.

Lu: And this opens up a lot of avenues for making AI tools practical instead of just theoretical.

Jane: This work shows how we can get contextually accurate prompt generation while being substantially more efficient than existing large language models, setting a new bar for what small language models can do in the multimodal space.

Adobe · Meta

cs.CL, cs.AI, cs.LG

Submitted: 2025-02-14

Updated: 2025-12-03

Comments: Accepted by ICDM 2025 MMSR

DOI: 10.1109/ICDMW69685.2025.00179

Code: https://github.com/karpathy/nanoGPT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The SLM framework investigates how small language models can be scaled to support multimodal search and recommendation use cases while remaining efficient enough for real-time,

Key concepts

Upside-Down Reinforcement Learning (UDRL): UDRL is a technique used to optimize the output of a model by framing it as an optimization problem. In this context, it trains the SLM to generate prompts that meet specific target criteria, such as desired length or relevance scores. The model learns through feedback from these reward signals to improve its generation quality iteratively.
Synthetic Data Distillation: This process uses a large language model like Llama-3 as a 'teacher' to create high-quality training data for the smaller SLM. The teacher generates millions of intent-prompt pairs, which are then curated and structured. This allows the smaller SLM to learn complex patterns and representations from the knowledge embedded in the much larger LLM without needing vast amounts of human-labeled data.
Multimodal Prompting: This refers to training a model to generate prompts that can handle different types of inputs, specifically text-to-image (T2I) and text-to-template (T2T). The SLM is augmented with specific tokens during training to understand how its generated text prompts will be used in various multimodal scenarios, enabling it to produce contextually relevant instructions for both image and template generation.
Small Language Models (SLMs): SLMs are compact language models, such as the 104M-parameter nanoGPT1 variant described. They are designed to be highly efficient for real-time applications, requiring minimal computational resources. The study demonstrates that these resource-constrained models can still achieve high performance in complex tasks like multimodal search and recommendation when properly trained.

Terminology

Reinforcement Learning — 2× in this paper · explained in 4 episodes
Distillation — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The SLM framework investigates how small language models can be scaled to support multimodal search and recommendation use cases while remaining efficient enough for real-time, resource-constrained deployments.

How it works

The proposed framework combines upside-down reinforcement learning with synthetic data distillation from a large language model (Llama-3 [1]) to train a 100M-parameter GPT-2 model [2] for multitask prompt generation

The system accepts both image and text inputs which are processed by an in-house detector for intent understanding before passing to a prompt generator

The training process involves two main stages: Synthetic Dataset Distillation and Upside-Down Reinforcement Learning

Synthetic Dataset Distillation

This stage creates a synthetic dataset using Llama-3 to enable the SLM to learn from the representations of LLM effectively

The curation process involves several steps:

Intent and Prompt Pair Generation: A dataset of 52 million intent-prompt pairs was curated by prompting the LLM with various intent descriptions and collecting its responses
Structured Prompting for Targeted Tasks: Structured prompting with few-shot examples and task specifications was used to ensure diversity and relevance in generated prompts
Labeling and Intent Selection: Intent descriptions were sourced from an in-house creative knowledge graph and augmented with diverse asset metadata from internal images and templates

Upside-Down Reinforcement Learning for SLM Optimization

Upside-down reinforcement learning (UDRL) is utilized to optimize the SLM’s generation quality and control specific attributes like length and relevance

Key aspects of the UDRL training include:

Reward-Based Prompt Generation: Upside-down reinforcement learning frames prompt generation as an optimization problem where the SLM aims to achieve target specifications for each generated output
Controlled-Length Generation: The SLM is trained to produce prompts within a desired length range (e.g., 10 to 35 words) by incorporating tokens indicating target lengths into the input
Modality-Agnostic Prompting: The SLM was trained to handle both text-to-image (T2I) and text-to-template (T2T) prompts by adding modality tokens to each training instance
Contextual Relevance and Specificity: Relevance scores are assigned to generated prompts based on a predefined metric, and prompts meeting these objectives are positively reinforced

Model Architecture and Key Capabilities

The SLM is based on nanoGPT1, a compact variant of GPT2, with 104 million parameters

Model specifications highlight its efficiency and multitask performance:

** Parameter Efficiency:**

The SLM contains approximately 1/80th the parameters of Llama-3 8b and similar state-of-the-art LLMs, making it computationally efficient

Inference Speed:

The SLM achieves a processing speed of up to 338 tokens per second on a single A10G GPU (non-batched, non-quantized, or accelerated)

Multitask Learning Capabilities:

The SLM is trained to handle both T2I and T2T prompts through the integration of task-specific tokens

Experiments and Evaluations

Quantitative evaluation assessed relevance evaluation and task adherence evaluation

Relevance Evaluation showed that:

Effective Distillation: In the few-shot setting, our SLM achieves relevance scores nearly on par with the teacher model, Llama-3 8B
Competitive Performance with Minimal Parameters: Despite being over an order of magnitude smaller, our SLM competes closely with models between 3B–8B parameters
Efficiency and Deployment Readiness: The SLM is highly efficient in both speed and memory, requiring only ∼500MB of memory while generating at 353 tokens/second

Task Adherence Evaluation (Length) showed that:

The SLM achieves precise control over prompt length, with Mean Square Error consistently around 1

Qualitative evaluations confirmed the model's quality:

** Prompt Quality:**

Human reviewers scored the prompts on Relevance and Correctness, resulting in an average relevance rating of 87%

** Multimodal Contextual Recommendation:**

For multimodal contextual recommendation, 86% of evaluators were willing to incorporate the generated images into the specified canvas

Overall, SLMs deliver contextually accurate prompt generation while being substantially more efficient and lightweight than existing LLMs

This work demonstrates the potential of small language models (SLMs) as efficient and versatile multitask learners<ref:2502.

Improvements for AI systems

Bold header: SLM as an efficient multitask learner for multimodal prompt generation

Leveraging Synthetic data is first generated from a large language model (Llama-3) using prompt engineering and few-shot examples, and then distilled into the SLM via UDRL [3], [4], this system can generate relevant text or image outputs aligned with user intent across multiple downstream use cases.

Bold header: Scalable training pipeline with Upside-Down Reinforcement Learning

The novel pipeline that distills supervision from LLMs and applies UDRL to optimize SLMs for prompt generation across multiple tasks allows the model to learn specific objectives based on desired outcomes rather than traditional reward structures.

Bold header: Competitive performance with minimal parameters

The SLM achieves relevance scores nearly on par with the teacher model, Llama-3 8B, with only a minor performance gap of approximately 3% in the few-shot setting and matches Llama-3 8B in the zero-shot setting despite being over an order of magnitude smaller.

Bold header: Precise control over prompt length

The model achieves precise control over prompt length, with a mean square error (MSE) consistently around 1, ensuring it can reliably generate prompts of desired lengths, which is essential for applications requiring specific or concise outputs.

Bold header: Real-time inference and resource efficiency

The SLM achieves a processing speed of up to 338 tokens per second on a single A10G GPU (non-batched, non-quantized, or accelerated), offering a significant advantage over baselines which require ∼80GB of GPU memory and are significantly slower.