Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation

Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation

This paper, accepted at CIKM '26, addresses the problem of missing modalities in multi-modal sequential recommendation systems. The authors note that real product catalogs routinely violate this assumption that every item carries every modality, citing that "34.3% of Toys & Games and 48.3% of Beauty & Personal Care items have no text description, and 26% to 41% of items have no image on the four MISSRec Amazon domains. When a modality is unavailable at serving time, a model trained on complete data loses much of its recommendation accuracy."

Proposed Method: Sequential Modality Dropout (SMD)

The authors propose SMD, described as a four-line, architecture-agnostic change that makes multi-modal sequential recommenders robust to the missing modalities they actually encounter in deployment. The mechanism works as follows: "during training, each modality stream (image and text) is independently erased with probability p for an entire user interaction history, so the model learns to predict the next item without relying on any single modality."

The mask is per-sample, not per-item: all items in user b's sequence share the same modality mask, which matches how modalities go missing in real catalogs, where an entire product category or data source tends to lack a modality at once. The implementation is a four-line PyTorch modification: if self.training and self.modality drop > 0: p, B = self.modality drop, img.shape[0]; img = img * (torch.rand(B,1,1) > p).float; txt = txt * (torch.rand(B,1,1) > p).float.

The paper also introduces an optional cross-modal reconstruction loss: Lrec = ½∥gt→v(ft(ti)) − fv(vi)∥2 + ∥gv→t(fv(vi)) − ft(ti)∥2, where gt→v and gv→t are two-layer projections trained jointly with the recommender. The total objective is L = LBCE + lambdaLrec with lambda = 0.01.

Evaluation and Results

The authors evaluate SMD on four backbones (MM-SASRec, IISAN, MISSRec, and fMRLRec) on four Amazon domains using the MISSRec benchmark. They measure robustness by retention, the fraction of a model's full-modality accuracy (HR@10) that survives when a modality is removed at test time.

Key results include:

• RQ1 (universal plug-in): On Amazon Scientific across the four backbones, SMD lifts HR@10 text retention from 18–94% to 56–99%; on MM-SASRec across four Amazon domains, it lifts text retention from 40–73% to 79–97%. Across 11 of the 16 possible (backbone, dataset) combinations, the robustness gain is 1.0 to 3.2× at essentially no cost to full-modality accuracy.

• RQ2 (extreme missingness): On a 0 to 95% per-item missing-rate sweep, MM-SASRec with SMD retains 61% of HR@10 at pmiss=0.95 versus 22% for the unmodified model (a 2.8× improvement). Per-user paired tests on 121k users confirm gains are statistically significant, with 22 of the 24 (condition, dataset) combinations significant at alpha = 0.05.

• RQ3 (reconstruction loss): An optional cross-modal reconstruction loss further lifts retention from 90% to 98% on a simple additive backbone under severe text missingness (MM-SASRec on Beauty & Personal Care with 48% text-missing items). However, on a strong dynamic-fusion backbone (MISSRec on Scientific) it adds only 3 points of text retention (70% to 73%) while costing 2.3% of full-modality HR@10.

The authors conclude: SMD is a four-line, per-sample training-time mask at the fusion point of any multi-modal sequential recommender, addressing the accuracy loss that missing modalities cause at serving time. They note that because SMD is a self-contained change at the fusion step, it can be combined with complementary techniques such as feature propagation and cross-modal reconstruction. Two open directions remain: extending beyond image and text to audio, video, or structured attributes, and handling modality corruption rather than absence.

Improvements to AI Systems:

1. Robustness to Incomplete Data at Inference: Integrate Sequential Modality Dropout (SMD) as a standard training-time augmentation in any multi-modal sequential recommender (e.g., SASRec, BERT4Rec variants). This ensures the model maintains high accuracy even when entire modality streams (e.g., all images or all text) are missing for a user’s history—a common real-world catalog condition. The improved system can serve recommendations with graceful degradation, retaining 56–99% of full-modality accuracy (vs. 18–94% without SMD) when text or images are absent.

2. Per-Sample Modality Masking for Catalog-Level Missingness: Implement SMD’s per-sample masking (masking all items in a sequence together) to align training with real-world data gaps (e.g., a product category lacking images). This allows the AI to learn modality-independent representations, so it can still predict the next item even if an entire category’s modality is unavailable at serving time. The system can handle extreme missingness (up to 95% of items missing a modality) with 2.8× better retention than baseline models.

3. Architecture-Agnostic Plug-In for Existing Models: Add SMD as a four-line change at the fusion layer of any existing multi-modal recommender (MM-SASRec, IISAN, MISSRec, fMRLRec) without retraining from scratch or altering the backbone. This enables rapid deployment across diverse production systems, improving robustness with negligible cost to full-modality accuracy (≤2.3% drop in HR@10 in worst case).

4. Optional Cross-Modal Reconstruction for Severe Missingness: Incorporate the auxiliary reconstruction loss (Lrec) to teach the model to infer missing modalities from available ones (e.g., predicting text embeddings from images). This is particularly effective for backbones with simple fusion (e.g., additive), boosting retention from 90% to 98% under severe text missingness. The improved system can proactively fill gaps in user histories, leading to more consistent recommendations even when data is sparse.

5. Adaptive Robustness Tuning: Use SMD’s dropout probability (p) as a tunable hyperparameter to balance between full-modality accuracy and robustness, based on the expected deployment environment. The system can be configured for high robustness (higher p) in catalogs with frequent missing modalities, or high accuracy (lower p) in complete-data settings, making it adaptable to diverse product domains (e.g., Toys & Games with 34.3% text-missing items vs. Scientific with near-complete data).

6. Statistical Reliability in User-Level Performance: Leverage SMD’s per-user masking to ensure that gains are consistent across individual users, not just aggregate metrics. The improved system can be validated with paired significance tests (e.g., 22/24 conditions significant at α=0.05), ensuring that robustness improvements are reliable for real user populations, not artifacts of averaging.

7. Extensibility to Other Modalities and Corruption Types: Build on SMD’s framework to extend beyond image/text to audio, video, or structured attributes, and to handle modality corruption (e.g., noisy or partially corrupted data) in addition to absence. The improved system can be generalized to multi-modal recommendation in richer domains (e.g., video platforms, e-commerce with reviews and specs), maintaining robustness across a wider range of data quality issues.