TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendation

TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendation

Abstract

Large Language Models (LLMs) have emerged as powerful tools for improving recommendation systems. The effectiveness of LLMs arises from their ability to harness rich textual information and their capacity to model heterogeneous user preferences based on users’ interaction history. However, due to the large-scale and deep architectures, LLM-based sequential recommendation approaches generally incur high inference costs, resulting in a low return on investment. To mitigate this cost, many existing approaches resort to using only the first few tokens of item descriptions, which inadvertently discards valuable information contained in the full text, thereby leading to suboptimal recommendation performance. To address this limitation, we propose a novel Token Selection approach for Preference Optimization in LLM-based sequential Recommendation, i.e., TSPORec, which accurately pinpoints informative tokens throughout the entire textual content to improve recommendation performance. Specifically, we design a three-stage pipeline to select informative tokens and introduce a novel proxy reward to facilitate the implementation. TSPORec not only enhances recommendation performance but also improves computational efficiency. Extensive experiments across two models and datasets demonstrate the superb performance (up to 31.25%) and efficiency (up to 63.4%) of our approach compared with six baseline approaches.

Introduction

Recommendation systems have become indispensable for mitigating information overload across a wide range of commercial platforms. A key factor in delivering personalized item recommendations lies in accurately modeling the heterogeneous preferences of users. Recently, Large Language Models (LLMs) have demonstrated remarkable effectiveness in capturing such preferences, owing to their vast parameter scales and strong contextual understanding. In particular, Sequential Recommendation (SR), which models diverse interests of users based on their recent interaction histories, has proven highly effective in enhancing recommendation performance when integrated with LLMs.

The goal of SR is to predict the next items a user is likely to prefer by modeling sequential patterns in their historical interactions. A key challenge in this field lies in simultaneously capturing item-level sequential dependencies and modeling the complex and heterogeneous preferences of users derived from these interaction sequences.

Existing approaches predominantly leveraged item IDentification (ID) features, overlooking the abundant textual content typically present in real-world settings. Furthermore, their reliance on shallow network architectures limit their capacity to model the complex and heterogeneous nature of user preferences. Recently, LLMs have emerged as effective solutions for overcoming the limitations of previous approaches, demonstrating strong performance in SR tasks. The core idea behind these approaches is to leverage the rich textual features and powerful representational capabilities of LLMs to extract robust representations of items and users.

However, existing LLM-based SR approaches are highly resource-intensive. For instance, given an interaction history of length S with each item represented by n tokens, a single inference step requires processing S × n tokens, which consists of both computation-intensive and memory-intensive operations. To mitigate this cost, existing approaches typically exploit only the first few tokens of each item, failing to fully exploit their rich textual features. This limitation raises an important research question: Can we identify and select the most informative tokens from textual features to maximize information utilization? Addressing this question could benefit SR systems in two key ways. First, it enables effective usage of critical information within the text. Second, it reduces computational overhead through compressing the overall token sequence length.

In this paper, we propose a novel token selection approach for preference optimization in LLM-based Sequential Recommendation, i.e., TSPORec. TSPORec is composed of three stages to select the most informative tokens from textual features. First, we pre-train an SR model based on an LLM. Next, we freeze the LLM backbone and attach a policy head, which is then trained using a newly designed proxy reward function. Using the trained policy, we extract informative tokens from item texts and subsequently retrain the LLM on the refined input sequences.

The key contributions of this paper are summarized as follows:

• We propose a new token selection pipeline that automatically identifies informative tokens from textual features. This pipeline shows strong generalization across LLMs and datasets.

• We design a novel proxy reward method to guide policy learning in an effective and targeted manner. This method can identify meaningful token importance, leading to an effective policy.

• Extensive experimental results demonstrate the superb performance (up to 29.29% in terms of recall and 31.25% in terms of NDCG) and efficiency (up to 63.4% inference overhead reduction) of TSPORec compared with six State-of-the-Art baseline approaches.

Method

The problem studied is token selection in LLM-based SR. Given an item represented by a token sequence T = t1, t2,..., tn, conventional LLM-based approaches typically derive an item embedding using only the first k tokens: ei = LLM(t1, t2,..., tk), where k ≤ n. Instead, we aim to identify an informative subset of k tokens, denoted as Tk ⊆ T, and feed them to the LLM to compute a refined item embedding: e′i = LLM(Tk). The selection of Tk is guided by a learnable policy, which prioritizes tokens that contribute most to user preference modeling. The problem is formulated as maximizing the expected reward: max R(θ).

To maximize flexibility, informative token identification is performed at the chunk level. A chunk is a subset of tokens consisting of several consecutive tokens in a token sequence. Specifically, we select ⌊k/c⌋ chunks from the token sequence T, where each chunk consists of c consecutive tokens. Token-level selection is a special case of this formulation, corresponding to c = 1. This chunk-wise strategy allows for controllable granularity in capturing local semantic structures for downstream recommendation tasks.

The overall pipeline of TSPORec consists of three stages: Pretraining, Token Selection Policy Learning, and Token Selection and Retraining.

Pretraining: In this stage, both the user LLM (denoted as g) and the item LLM (denoted as f) are pretrained employing the InfoNCE loss to obtain a foundation model that can be leveraged for reward computation. For item LLM pretraining, a special [ITEM] token is appended to the end of the textual token sequence of each item. The item embedding is then extracted as the final-layer hidden state corresponding to this [ITEM] token.

Token Selection Policy Learning: In this stage, a novel token selection policy is introduced. Given the full set of textual features for all items, exhaustively evaluating every possible token subset is computationally infeasible due to its exponential time complexity. To mitigate this burden, a differentiable proxy reward that approximates the utility of a token subset is proposed, enabling efficient optimization of the selection policy.

The informativeness of each token ti is quantified through its relevance to the item embedding using a query-key attention mechanism: info(ti) = ⟨WQ hi, WK hk+1⟩/√d, where θ = WQ, WK denotes a set of learnable parameter matrices in Rd×d, and d is the dimensionality of the hidden embeddings. The resulting score reflects the alignment between the token representation hi and the item-level representation hk+1. The probability to select Token ti is then defined as pi = exp(info(ti)) / Σj exp(info(tj)).

Given a sequence of M items that a user has interacted with, denoted as IM = T1, T2,..., TM, the joint probability of the interaction sequence is modeled as P(IM) = Πj Πl pjl, where pjl is the probability of the l-th token in Tj.

For each token sequence Tj in the interaction history IM of a user, two independent sampling passes are performed to extract distinct subsets of token chunks, yielding two derived sequences denoted Tj′ and Tj′′. Using these sampled sequences, two corresponding user embeddings are constructed via the pretrained, frozen LLM: e′u = g frozen(f frozen(IM′)) and e′′u = g frozen(f frozen(IM′′)). The full item embedding is constructed based on the full token sequence as ei = f frozen(IM).

Two cross-entropy values are computed by contrasting the user embeddings e′u and e′′u with the full item embedding ei, exploiting a set of N negative item embeddings. The reward signal r is then defined as: r = 1 if L1ce ≤ L2ce, and r = -1 otherwise. The objective is to optimize the policy parameters θ by maximizing the expected reward: R(θ) = E[r · (log P(IM′ θ) − log P(IM′′ θ))].

Token Selection and Retraining: After optimizing the policy parameters, informative token chunks are selected as follows. The probability of a chunk c is defined as P(c) = Πi pi, where pi denotes the generation probability of the i-th token in the chunk, and c is the number of tokens in c. Chunks with higher probability scores according to P(c) are selected. Using this set of high-probability chunks, a refined dataset is constructed and the model is retrained on this updated representation to improve both its performance and efficiency.

Experiments

A comprehensive evaluation of TSPORec is conducted against four state-of-the-art baselines: (1) traditional ID-based sequential recommendation models, such as SASRec and HSTU; (2) semantic initialization methods that leverage LLM-derived embeddings to initialize item ID representations, exemplified by LLMinit; and (3) hierarchical LLM-based frameworks, specifically HLLM. In addition, to study the impact of token selection, these baselines are adapted with different token selection policies, resulting in six baseline variants for comparison. The approach is evaluated on two publicly available datasets: the Amazon Book Reviews dataset and the Pixel dataset. For backbone LLMs, Qwen3-Embedding-0.6B and TinyLlama-1.1B are employed. Performance is evaluated using Recall@K (R@K) and NDCG@K (N@K).

Overall Performance: TSPORec significantly outperforms all baseline approaches by a substantial margin. In the setting where input sequences are truncated to 64 tokens and Qwen3-Embedding-0.6B serves as the frozen LLM backbone, with chunk size set to 8, TSPORec improves upon SASRec by up to 29.29% in Recall@K and 31.25% in NDCG@K on the Amazon Books dataset, yielding an average gain of 29.43% across metrics. On the Pixel dataset, it achieves improvements of up to 19.63% in Recall@K and 18.24% in NDCG@K, with an average increase of 16.79%.

The Importance of Tokens in LLM-based Recommendation: The impact of different types of input tokens is investigated by comparing random tokens, top-k logit tokens, the first k tokens, and TSPORec. TSPORec achieves the most significant performance gain by leveraging the learned token selection policy, substantially outperforming existing token selection strategies. On the Amazon Books dataset, randomly selecting tokens reduces the performance improvement of HLLM from 24.40% to just 2.14%, highlighting the critical role of input token selection in LLM-based recommendation. Surprisingly, the top-k logit tokens do not outperform the simple first-k tokens baseline, suggesting that more principled token selection mechanisms are needed. TSPORec leverages a bidirectional scoring mechanism where the importance score of each token is learned from both the standard attention mechanism, which encodes semantic meaning, and the pretrained item embedding, which provides a collaborative filtering signal.

The Impact of Token Sequence Length: Experiments with varying input token lengths show that on the Amazon Books dataset, increasing the token length leads to performance improvements for both HLLM and TSPORec. On the Pixel dataset, while moderate increases in token length initially enhance HLLM’s performance, further lengthening the input sequence eventually degrades performance, indicating that not all tokens contribute positively to recommendations. TSPORec with only 64 selected tokens achieves performance on par with or superior to the corresponding HLLM variants using 256 tokens, demonstrating that TSPORec not only enhances recommendation accuracy but also improves computational efficiency.

The Efficiency of TSPORec: By selecting informative tokens, TSPORec effectively reduces the required input length without sacrificing performance, thereby lowering overall computational overhead. Inference time measurements on an NVIDIA H100 GPU using the Amazon Books dataset show that since TSPORec matches the performance of HLLM using only 64 tokens with HLLM using 256 tokens, it reduces inference cost by 63.4% in scenarios where item embeddings are precomputed offline, and by 61.3% when both the item and user LLMs are served online.

The Generalization capability of TSPORec: Experiments using a different LLM backbone (TinyLlama-1.1B) and a different downstream SR model (LLMinit) show that TSPORec consistently outperforms HLLM across all Recall and NDCG metrics, achieving an average improvement of 4.12%. When TSPORec-selected tokens are used to initialize LLMinit, the model performance improves by 3.76%, demonstrating that the selected tokens carry informative signals for recommendation.

Case Study: A case study on the Pixel dataset reveals that TSPORec removes text segments containing function words (e.g., prepositions, pronouns, and articles) that carry limited discriminative power, and instead selects content words—nouns, adjectives, or named entities—that convey specific and semantically rich information about the item. TSPORec also tends to filter out frequent (“hot”) tokens, which are shared across many items and thus exhibit low semantic specificity, making it difficult for the user LLM to distinguish between items and learn effective representations.

Conclusion

In this paper, we propose a novel Token Selection approach tailored for LLM-driven Sequential Recommendation systems, i.e., TSPORec. We develop a three-stage workflow. First, we pre-train the foundational LLM-based sequential recommendation model to establish a robust baseline. Second, we train a dedicated policy to identify informative tokens; additionally, a proxy reward function is designed to facilitate chunk-oriented token selection, addressing the challenges of granularity in token identification. Finally, leveraging the identified tokens, we retrain the pre-trained baseline model to optimize its recommendation performance. Comprehensive experimental results demonstrate the effectiveness (up to 31.25%) and efficiency (up to 63.4%) of the proposed method.

Limitations

The proposed TSPORec employs a three-stage pipeline to select the most informative tokens from the item’s textual features. While TSPORec significantly improves both recommendation performance and inference efficiency compared to standard training methods, it incurs additional training time due to Stages 2 and 3. However, this overhead is practically acceptable, as models are typically trained once and deployed over an extended period—rendering inference efficiency a more critical concern than the one-time training cost.

Improvements to AI Systems Based on TSPORec:

1. Token-Level Input Compression for LLMs: Implement a learnable, chunk-based token selection policy that identifies and retains only the most informative tokens from long textual inputs before feeding them to an LLM. This reduces computational overhead (up to 63.4% inference cost reduction) while maintaining or improving task performance, enabling deployment of LLMs on resource-constrained devices or for high-throughput applications.

2. Preference-Optimized Token Scoring via Proxy Rewards: Integrate a differentiable proxy reward mechanism that scores token importance based on alignment with downstream task embeddings (e.g., item-level representations). This allows the AI system to automatically learn which tokens matter for a specific objective (e.g., recommendation accuracy) without exhaustive search, improving generalization across different LLM backbones and datasets.

3. Three-Stage Training Pipeline for Efficient Fine-Tuning: Adopt a pipeline of (a) pre-training a base LLM on the task, (b) freezing the backbone and training a lightweight policy head with a proxy reward, and (c) retraining on the selected token subset. This reduces the cost of adapting LLMs to new tasks, as the policy can be reused across similar tasks or models, and the final model is both faster and more accurate.

4. Semantic-Specificity Filtering for Text Representation: Use the learned token importance scores to automatically filter out low-information tokens (e.g., function words, frequent “hot” tokens) and retain content-rich tokens (nouns, adjectives, named entities). This improves the quality of text representations for tasks like sequential recommendation, search, or document classification, leading to better discrimination between similar items.

5. Adaptive Input Length Management: Enable the AI system to dynamically select the optimal number of tokens per input based on the learned policy, rather than using a fixed truncation. This allows the system to trade off between accuracy and efficiency per instance, achieving performance parity with longer inputs while using significantly fewer tokens (e.g., 64 selected tokens matching 256-token performance).

6. Cross-Model and Cross-Task Transferability: The token selection policy trained on one LLM (e.g., Qwen3-Embedding-0.6B) can be transferred to another LLM (e.g., TinyLlama-1.1B) or to a different downstream model (e.g., LLMinit), improving performance by 4% on average. This enables building reusable token-selection modules that enhance multiple AI systems without retraining from scratch.

7. Improved Sequential Recommendation Accuracy: By feeding only the most informative tokens into the LLM, the system achieves up to 31.25% improvement in NDCG and 29.29% in Recall@K over state-of-the-art baselines. This directly enhances user experience in e-commerce, content streaming, and personalized search by better predicting user preferences from interaction histories.

8. Reduced Training and Inference Overhead: The system reduces the number of tokens processed per interaction sequence, lowering memory usage and latency during both training and inference. This makes it feasible to deploy LLM-based recommenders in real-time settings, such as online shopping or news feeds, where response time is critical.

Large Language Models (LLMs) have emerged as powerful tools for improving recommendation systems. The effectiveness of LLMs arises from their ability to harness rich textual information and their capacity to model heterogeneous user preferences based on users' interaction history. However, due to the large-scale and deep architectures, LLM-based sequential recommendation approaches generally incur high inference costs, resulting in a low return on investment. To mitigate this cost, many existing approaches resort to using only the first few tokens of item descriptions, which inadvertently discards valuable information contained in the full text, thereby leading to suboptimal recommendation performance. To address this limitation, we propose a novel Token Selection approach for Preference Optimization in LLM-based sequential Recommendation, i.e., TSPORec, which accurately pinpoints informative tokens throughout the entire textual content to improve recommendation performance. Specifically, we design a three-stage pipeline to select informative tokens and introduce a novel proxy reward to facilitate the implementation. TSPORec not only enhances recommendation performance but also improves computational efficiency. Extensive experiments across two models and datasets demonstrate the superb performance (up to 31.25%) and efficiency (up to 63.4%) of our approach compared with six baseline approaches. Code is available at https://github.com/WNQzhu/TSPORec.git.