Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation
Donald Loveland, Liam Collins, Bhuvesh Kumar, Danai Koutra, Neil Shah
cs.IR, cs.LG
Submitted: 2026-08-07
Updated: 2026-08-11
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction
Terminology
Abstract
Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction histories. In these systems, items are often represented through semantic IDs (SIDs) added to the LLM vocabulary as special tokens. Ideally, SIDs imbue item token representations with semantic priors, thereby improving model generalization. However, standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs' underlying continuous geometry and forcing the LLM to relearn token relationships from interaction data. To demonstrate the consequences of this design, we first show that training from this initialization tends to organize SID embeddings around item popularity rather than semantics. We further show that, despite partially reducing the reliance on popularity and improving cold item performance, the computationally expensive process of continual pretraining (CPT) fails to reliably recover the original semantic geometry. To address these findings, we propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space. Requiring only a few lines of code and no additional training or inference overhead, this drop-in approach improves pure-SFT Recall@5 by up to 16%, reaches peak performance with up to 40% fewer SFT steps, and improves cold-item Recall@5 by up to 60%. Moreover, on datasets that benefit from additional CPT, centroid initialization reaches comparable performance while requiring half as many CPT epochs. Together, our findings show that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.
Sources
- Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation
- LLMs Need Encoders for Semantic IDs Too
- OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment
- Implicit Reasoning for Large Language Model-based Generative Recommendation
- Reasoning over Semantic IDs Enhances Generative Recommendation
- A Survey on Generative Recommendation: Data, Model, and Tasks
- MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- OneRec-Think: In-Text Reasoning for Generative Recommendation
- Qwen3 Technical Report
- OpenOneRec Technical Report
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- QProtFuse: Query-Guided Layer Fusion for Retrieval-Augmented Protein Function Prediction
- DualView: Adaptive Local-Global Fusion for Multi-Hop Document Reranking
- How Do LLM-Generated Texts Impact Term-Based Retrieval Models?