Multimodal Graph Retrieval-Augmented Sequential Recommendation via Collaborative Filtering Paths

The gist The proposed framework, MGRASRec, is a multimodal graph retrieval-augmented framework for sequential recommendation that injects candidate-conditioned collaborative signals directly into an MLLM prompt via structured graph path retrieval and extends coverage through multimodal similarity to increase coverage beyond exact co-interaction overlap.

Problem Formulation

The problem is formally defined as predicting the probability that a candidate item is the user’s next interaction, P(Iu n+1 = Ic Su) Given a user u, we define their historical behavior sequence as Su = [Iu 1,..., Iu n], where Iu i represents the i-th item with which the user has interacted, and n denotes the length of the sequence. Each item i in the item catalog I is associated with a textual title ti and an image vi (e.g., product image or video cover). Given the historical behavior sequence Su and a candidate item Ic, the objective of multimodal sequential recommendation is to predict the probability that Ic is the user’s next interaction, i.e., P(Iu n+1 = Ic Su).

Graph-based Collaborative Filtering Retrieval

To inject focused collaborative signals conditioned on the candidate item directly into the recommendation prompt, the user-item interaction history is represented as a bipartite graph G = (U ∪ I, E), where an undirected edge (u, i) ∈ E indicates that user u has interacted with item i. When verbalizing paths in the prompt, each edge is expressed directionally as (user, watched, item) or (item, watched by, user), with the verb instantiated per domain (‘watched’ for Microlens, ‘bought’ for Amazon). For each (target user u, target item i∗) pair, a breadth-first search (BFS) is performed over G to retrieve 3-hop co-interaction paths of the form: u watched−−−→ b watched←−−−u' watched−−−→ i∗ (1) where b ∈ I is a bridge item co-interacted by both u and a bridge user u', and u' has also interacted with the target item i∗. To extend coverage, similarity paths are retrieved by computing multimodal embeddings for all items using CLIP (ViT-L/14) [15], yielding a fused item representation ei = normalize CLIPimg(vi) + CLIPtxt(ti) 2. These similarity paths are used to identify bridge items that are visually or semantically similar to items already in the target user’s history, allowing for a 4-hop similarity path (3).

Path Priority Ranking and History Selection

Since the number of retrieved paths grows unboundedly with interaction density, only top-K paths are selected, fixing K = 5 as a balance between prompt length and coverage. Paths are ranked by the number of unique bridge users passing through each bridge item, reflecting consensus behavioral evidence. Co-interaction paths are prioritized with at least 60% of top-K slots reserved as they encode more direct collaborative evidence. Relevant history selection is achieved by extracting the first-hop history item from each of the top-K ranked paths, which emerges as a byproduct of retrieval already performed. For sparse or no-path cases, remaining slots are padded with the most frequent items from the interaction history not already selected.

MLLM LoRA Fine-Tuning and Efficiency

The retrieved collaborative signals and selected history items are assembled into an augmented prompt, where if no paths exist, the CF signals section is omitted. Supervised fine-tuning (SFT) of LLaVA [10] is applied to the augmented prompt, using LoRA [7] under the PEFT framework to reduce training cost. At inference time, MGRASRec requires only a single forward pass per candidate item, which is quantified as requiring exactly one regardless of history length. This constant cost is achieved by shifting retrieval offline, where path construction requires only a BFS traversal over the interaction graph and lookups into a precomputed item-similarity table.

Overall Performance

MGRASRec achieves the best performance on all metrics across three datasets, with MGRASRec achieving AUC 89.55, HR@5 83.07, and MRR@5 62.65 on Microlens. This demonstrates that augmenting MLLMs with structured graph-based collaborative paths is the primary driver of MGRASRec’s gains, indicating that this is a promising direction for multimodal sequential recommendation. Future work could inject graph-based collaborative signals into open-set generation, where the MLLM directly generates the next item from the full item vocabulary under constrained decoding for item grounding.

1. Bold header: Structured Collaborative Signal Injection

This improvement injects candidate-conditioned collaborative signals directly into the MLLM prompt by retrieving structured paths over the user-item interaction graph, allowing MLLMs to reason over collaborative filtering signals with multimodal content through post-alignment contrastive learning.

2. Bold header: Path-Guided History Selection

The system implements path-guided history selection, which surfaces the k history items most structurally relevant to the target item at zero additional inference cost, solving the issue of impractical history truncation by extracting the first-hop history item from each of the top-K ranked paths.

3. Bold header: Constant Inference Cost for Long Histories

MGRASRec achieves efficiency by ensuring inference cost grows linearly with user history length for MLLM-MSR is avoided, as MGRASRec requires exactly one [forward pass] regardless of history length, despite handling long sequences.