ReAlign: Reward-Guided Preference Alignment for Generative Re-Ranking
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "ReAlign: Reward-Guided Preference Alignment for Generative Re-Ranking".
Jane: Here's the gist of the paper. ReAlign proposes a reward-guided preference alignment framework that couples a listwise reward model with a candidate-aware autoregressive generator to refine generative re-ranking The objective of the re-ranking problem is to learn a conditional distribution P (L u, X) that produces ordered lists L maximizing the reward r(u, L).
Tom: Framework Overview ReAlign consists of two main components: a reward model and a generator.
Jane: The reward model acts as an evaluator that estimates the expected utility of an ordered list by jointly modeling exposure and item value under user browsing behavior.
Tom: The generator is a candidate-aware encoder–decoder model that autoregressively constructs ordered lists from the upstream candidate set.
Title and authors: Tom: So, we're talking about this paper called ReAlign: Reward-Guided Preference Alignment for Generative Re-Ranking. It sounds like it's trying to fix a gap where the list generation part and the list evaluation part don't really talk to each other very well.
Jane: Exactly. The core idea here is that instead of having a separate system that just scores lists after they're made, they couple a reward model directly with an autoregressive generator so the generator learns what users actually want by looking at those rewards.
Lu: What's really interesting about the setup is how the reward model breaks down the listwise reward into position-wise contributions. It models user exposure using something like a survival process and item value using multiple heads for different engagement signals, which gives it a way to be more detailed than just one single score.
Meng: From an engineering standpoint, I'm curious about how they handle that complexity during training. They mention the generator goes through two stages: first learning from logged exposure lists to get a baseline relevance prior, and then getting this reward-guided post-training to actually align with the reward model. That sounds like a lot of tuning.
Lalam: I see what you're saying about those two stages. The goal is to get that initial stable policy, and then use the alignment loss over standardized rewards to fine-tune it so it matches what the reward model predicts is good for users across a broad pool of lists.
Tom: Right, so they're trying to make sure the generator doesn't just make *any* list, but one that actually scores high on this detailed reward function. But how do they actually get that initial relevance prior from just the logged exposure data? That seems like a key piece of the puzzle.
Jane: Well, they use supervised pre-training on those logged exposure lists to establish that starting point before diving into the more complex reward alignment phase. It sets up a baseline where the generator already knows some basic things about what users look at.
Lu: And then during that post-training, they focus on aligning the distributions and the rankings over within-pool standardized rewards. That standardization part is important because it makes sure that even if one reward signal is way bigger than another, it doesn't completely dominate the learning process.
Meng: I worry about those biases you mentioned. If a reward model has inherent biases, aligning against that can just bake those biases into the generator's final ranking. How do they ensure the rewards they use are actually representative of what users value?
Lalam: They address that by decomposing the listwise reward itself into position-wise contributions, modeling exposure as a discrete-time survival process and item value with per-objective heads that use uncertainty weighting. That structure helps keep things calibrated.
Tom: Okay, so they're not just using one big score; they are breaking down the utility calculation into different parts—exposure and item value—and weighting them based on some uncertainty measure. That sounds like a smart way to get a more nuanced view of user engagement.
Title and authors: Jane: It does give it that interpretability you need when you're trying to understand *why* the model prefers one list over another. But let's talk about how they actually build the generator itself, since that's where the generation happens.
Lu: The generator is a candidate-aware encoder-decoder model, and it uses a matching-based selection mechanism. This means at every step, as it builds the list, the decoder creates some kind of query representation specific to that position to pick the next item from the remaining candidates.
Meng: A matching-based selection mechanism sounds complex for real-time use. When you're running this on a live feed, you need something fast. What does that look like in terms of computational cost?
Tom: They are using a shared user and item encoder to turn everything into tokens first, which is efficient. But the matching part at each position needs to be quick enough for inference on a real list of candidates.
Jane: And that leads us right into how they handle the reward model's structure. The reward model uses a unified transformer architecture over the ordered list, taking those ranked item tokens and user tokens as input to produce context-aware ranked item tokens for every single position.
Lu: That unified transformer is key because it lets the model see how all those elements interact across the entire ordered sequence of items simultaneously, creating really rich context-aware representations for each spot in the list.
Lalam: It’s about seeing the whole context at once, not just looking at one item in isolation. And when you look at their user encoder design, it takes a few different signals: static features, long-term interaction sequences via a multi-interest extraction layer with softmax attention, and short-term interactions from the most recent N interactions.
Meng: That’s a lot of input for one user token. How do you manage that representation size when you're running inference? It sounds like it could get heavy quickly.
Tom: The paper says they use RMS normalization to stabilize those user tokens after concatenating them from the static features, the long-term sequence, and the short-term sequence. That's their trick for keeping those embeddings manageable.
Jane: So, we have this powerful reward model that understands context deeply, and a generator that uses that understanding to build a list step by step based on those learned rewards. What is actually changing here for someone using this system?
Lu: The implication is that the re-ranking process moves from being just a heuristic refinement to being directly guided by what the reward model has learned about user utility, which should lead to significantly better ordered lists.
Meng: I’m still thinking about deployment. If you have a massive candidate pool, running this generator autoregressively at every step might be slow. What do they say about inference speed when you push it out?
Title and authors: Tom: They mention that for low-latency serving, they propose greedy decoding. This means at each step, the generator just takes the best item based on the selection distribution and masks it so it doesn't pick that same item again. That should make it much faster than trying to explore every possible next item.
Jane: It sounds like a practical trade-off between perfect exploration and speed for real-world applications. So, to wrap up this part of the paper, ReAlign proposes this two-stage pipeline where pre-training sets the stage and then reward alignment refines it over standardized rewards. That’s what we have here before we look at the final numbers.
Lu: Before we conclude, I think what stands out is how they structured the post-training objective to explicitly combine distribution alignment and ranking alignment using those within-pool standardized rewards. That's a solid way to make sure both aspects of the learning are being optimized simultaneously.
Meng: And I’m still wondering about the limitations they mentioned regarding that reward decomposition—how robust it is if the underlying engagement signals don't perfectly map to user value. That uncertainty weighting helps, but it’s still a point of fragility.
Lalam: They did flag that the entire framework relies on being able to model user exposure as a discrete-time survival process and item value with those specific heads for heterogeneity. If you have signals outside those defined categories, the reward estimate might break down.
Tom: So, we've seen how they build it—the two stages, the decomposition of the reward into position and value parts, and the matching mechanism in the generator. It sounds like a very structured approach to solving that preference alignment problem.
Jane: And for anyone who just listens to this show, ReAlign means we’re moving away from systems where list generation is completely divorced from knowing what actually makes a list good for the user. It’s about closing that loop.
Lu: The implication is that when you use this, you should expect better ordered lists because the generator isn't just following a prior; it's actively being steered by the learned utility signal from the reward model.
Meng: For me, I’d be watching how they handle scaling those candidate sets. If they can keep that complexity under control while maintaining those high numbers, that’s where it gets deployed for real traffic.
Lalam: I'm reasonably sure of the results here because they showed consistent outperformance against existing baselines on both KuaiRand and Avito, hitting scores like zero point eight five five one NDCG uAUC on KuaiRand and zero point seven two five eight on Avito >
Tom: It sounds like a solid piece of work that tackles a very practical problem in recommender systems by coupling the evaluation signal directly into the generation process with careful training steps to manage complexity.
Jane: Yeah, ReAlign is definitely worth checking out if you’re interested in how we can make generative systems more sensitive to what users actually prefer. That brings us to the end of this deep dive into this specific paper.
The paper's summary: Tom: So, we just talked about how ReAlign couples a reward model directly to the generator so they learn together what users actually want for ordered lists. Now, let's look at what the authors say in their summary about this whole system and why it matters for re-ranking.
Jane: Basically, they’re saying that instead of just throwing a list generation model at a set of candidates and hoping it gets the order right, you use this reward model to tell the generator exactly *why* certain lists are good. It takes that abstract idea of "preference" and turns it into a concrete mathematical signal that the generator can follow during its creation process.
Lu: What's interesting is how they structure that signal. They don't just give one score to a list, you see, they break the total reward down into pieces—position-wise contributions and item value components. It models user exposure like a survival process and the item value using different heads for various engagement signals with some uncertainty weighting on top.
Meng: That decomposition sounds useful for calibration. If you only had one big score, it’s hard to tell if the model is being rewarded because of good positioning or just because of a really popular item in the list. Having those separate components helps isolate what’s driving the utility there.
Lalam: Exactly. And then they train this generator in two stages, which is a big part of it. First, supervised pre-training on logged exposure lists to get a stable starting point—a baseline relevance prior—and then reward-guided post-training where they align the generator’s output distribution with what the reward model expects through ranking alignment and distribution alignment over standardized rewards.
Tom: That two-stage approach is where I see a lot of engineering rigor. Pre-training sets a stable policy, and the post-training refines it to match those specific reward preferences across a diverse pool of lists, using that standardization to make sure the learning isn't just chasing one extreme signal.
Jane: So for someone listening who doesn't read papers, this means we’re moving past systems where list creation is totally separate from knowing what actually pleases the user. It closes that loop between making a list and knowing if it’s good or bad.
Lu: The core idea is that the generator isn't just following some static prior; it's being actively steered by the learned utility signal from the reward model, which should result in much better ordered lists overall compared to previous methods.
Meng: I still have to think about how they manage that complexity on real traffic. If you’ve got a massive candidate pool, running this generator autoregressively at every step sounds computationally heavy for production serving.
Tom: That's a fair concern, Meng. But the authors address that with an inference strategy; they suggest greedy decoding for low-latency serving by taking the best next item and masking it so you don't pick it twice, which should make it much faster than exploring every single option at every step.
Jane: So, to wrap up this part, ReAlign proposes this two-stage training pipeline where pre-training sets the stage and then reward alignment refines it over standardized rewards. It’s about making sure the generator is genuinely guided by what users value.
Lu: And I think what really stands out is how they structured the post-training objective to explicitly combine distribution alignment and ranking alignment using those within-pool standardized rewards. That's a solid way to make sure both aspects of the learning are being optimized at once.
Meng: And I’m still thinking about those limitations they mentioned regarding that reward decomposition—how robust it is if the underlying engagement signals don't perfectly map to user value. That uncertainty weighting helps, but it’s still a point of fragility for real-world deployment.
Tom: They did flag that the entire framework relies on being able to model user exposure as a discrete-time survival process and item value with those specific heads for heterogeneity. If you have signals outside those defined categories, the reward estimate might break down.
Jane: So we’ve seen how they build it—the two stages, the decomposition of the reward into position and value parts, and the matching mechanism in the generator. It sounds like a very structured approach to solving that preference alignment problem.
Lu: And for anyone who just listens to this show, ReAlign means we’re moving away from systems where list generation is completely divorced from knowing what actually makes a list good for the user. It’s about closing that loop.
Tom: The implication is that when you use this, you should expect better ordered lists because the generator isn't just following a prior; it's actively being steered by the learned utility signal from the reward model.
Jane: For anyone who wants to know how this helps them directly, ReAlign means using an AI system where the generated list is actually optimized for what users prefer, not just what looks good on paper. That’s what we’re talking about today.
The paper's improvements: Tom: So, we've seen how ReAlign works right now, but the authors also lay out some improvements they suggest for making it even better than this current version. They aren't just stopping there; they have a plan to tighten up the mechanics further.
Jane: What are these suggestions? Are they just tweaking the math, or are they changing how we think about what we’re trying to optimize with the reward model and the generator?
Lu: The first big thing is refining that reward model architecture. They suggest it should jointly model reaching probability and item value using a unified transformer-based architecture over the ordered list. It needs to take both those ranked item tokens and user tokens as input to produce context-aware ranked item tokens for every position, not just one piece of the puzzle.
Meng: A unified architecture sounds like more parameters, which means more computation time during training and inference. What is the trade-off there?
Tom: Well, from an engineering view, that unification should help capture cross-dependencies between what a user sees and what they actually value in a single pass. It’s about getting richer context into those tokens before they get ranked.
Jane: Then there's the user encoder design. They want it to transform each user into a fixed set of user tokens that captures static features, long-term interaction sequences through some multi-interest extraction layer, and short-term interactions from the most recent N items.
Lu: That’s trying to give the AI a much richer picture of who the user is before it even starts generating a list. It's moving beyond just looking at what they clicked yesterday to understanding their long-term interests too.
Meng: A fixed set of tokens sounds like you have to pre-define those features, which takes upfront work. And how does that affect scalability? If we have millions of users, managing that input size is key for deployment.
Tom: They mention using RMS normalization to stabilize those user tokens after you concatenate the static features and the different sequence inputs. That’s their trick to keep the embedding sizes manageable on hardware when dealing with all those different data streams.
Jane: Then they suggest a new way to build the generator selection mechanism. Instead of just a matching-based selection, they propose having the decoder produce a step specific query representation at each position that matches against candidate item tokens to compute a selection probability for the next item from what’s left.
Lu: That’s making the choice at each step more explicit and context-aware, ensuring the generator is selecting based on what makes sense for *that specific spot* in the list, rather than just picking a random good item.
Meng: That sounds more computationally intensive because you're running a matching process every single time to pick the next thing from the candidate set. Does that slow down inference too much?
Tom: They are aiming for efficiency there by focusing on making that query representation very targeted, so it should still be fast enough for real-time use, but it’s a trade-off for that extra layer of precision in selection.
Jane: Finally, they propose sharpening the post-training objective. Instead of just distribution alignment and ranking alignment over standardized rewards, they suggest combining distribution alignment—minimizing cross entropy between the reward model's softmax distribution and the generator's induced distribution—with a ranking alignment loss that enforces consistency between the reward model's ordering and the generator's ordering across a diverse list pool.
Lu: That combination is clever because it forces both aspects of preference modeling to work together simultaneously during training, making sure you get both good order and good probability at once.
Tom: It’s a comprehensive training signal, which should lead to a more robust model that handles shifts in user preference better than one focused on just one alignment loss.
Jane: So we’ve seen the initial setup and now we see how they plan to upgrade the core components—the reward model's structure, the user representation, and the generator's selection process—to make it even more precise.
Meng: I guess it’s clear they are pushing for a system that is highly accurate but also structurally sound across those different types of data inputs.
Conclusion: Tom: So, to wrap up this talk on ReAlign: Reward-Guided Preference Alignment for Generative Re-Ranking, we're looking at how this whole system changes how we train these generative re-ranking models.
Jane: Basically, the authors showed that coupling a reward model directly with the generator helps it learn exactly what users prefer, rather than just guessing what looks good. It’s about closing that gap between making a list and knowing if it actually satisfies the user's underlying taste.
Lu: The main implication is that this framework allows the generator to be steered by a specific utility signal derived from user behavior, which should lead to better ordered lists in practice.
Meng: From my side, I’m still looking at deployment costs. If we want to ship this at scale on real traffic, we need those greedy decoding strategies they mentioned to keep the inference time down and manageable for production use.
Lalam: I'm pretty sure of the answer here because the results showed consistent outperformance against existing baselines on both KuaiRand and Avito, hitting scores like zero point eight five five one NDCG uAUC on KuaiRand and zero point seven two five eight on Avito.
Tom: That number is solid, but I always want to know what the actual cost of running that reward model versus the generator architecture looks like when you scale up to millions of users across different platforms.
Jane: It’s definitely a more expensive setup than a simple re-ranking step, but if the utility gain is large enough, it might be worth it for high-value applications.
Lu: The work proves that this two-stage pipeline—the pre-training and then the reward alignment—is what gives you that stability and performance boost compared to single-stage methods.
Meng: It’s a solid finding, but we still need to see how much data is required for that reward model to get those accurate signals across different user pools.
Lalam: I'm confident the signal quality is high because they used standardized rewards in the post-training, which makes the learning process much more robust against any weird scale differences in the raw engagement data.
Tom: So, ReAlign shows us a structured way to inject real user preference directly into the generation process without just relying on whatever simple prior we start with.
Jane: It’s a big step because it moves generative systems away from being purely pattern matchers and toward systems that are genuinely aligned with what users want.
Lu: The paper is a strong contribution to this area because it provides a concrete, measurable way to guide list generation using reward signals in this specific manner.
Meng: It’s definitely interesting to see how much complexity we can bake into the training process just to get that final ordering right for production. So, ReAlign: Reward-Guided Preference Alignment for Generative Re-Ranking shows us a structured way to inject real user preference directly into the generation process without just relying on whatever simple prior we start with.
Intuit Credit Karma
cs.IR
Submitted: 2026-11-11
Updated: 2026-11-11
Comments: 9 pages, 3 figures, ACM conference format
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 86/100
The gist: ReAlign proposes a reward-guided preference alignment framework that couples a listwise reward model with a candidate-aware autoregressive generator to refine generative re-ranking The objective of…
Key concepts
- Reward Model
- This model acts as an evaluator that estimates the expected utility of an ordered list. It decomposes the listwise reward into position-wise contributions by modeling user exposure as a survival process and item value using multiple heads with uncertainty weighting, providing a calibrated measure of preference.
- Generator
- A candidate-aware autoregressive model that constructs ordered lists from a set of candidates. It uses an encoder-decoder structure where the decoder generates step-specific queries to select the next best item from the remaining pool based on context and user preferences.
- Two-Stage Training Pipeline
- The generator is trained in two phases: first, supervised pre-training on logged exposure lists to establish a relevance prior. Second, reward-guided post-training aligns the generator with the reward model by optimizing complementary distribution and ranking alignment over standardized rewards.
- Listwise Reward Objective
- The objective for the reward model minimizes L = Lexp + Lvalue, which factorizes the joint distribution over user exposure and engagement signals. This allows the model to jointly learn how user exposure influences item value across an entire ordered list.
Terminology
Summary
ReAlign proposes a reward-guided preference alignment framework that couples a listwise reward model with a candidate-aware autoregressive generator to refine generative re-ranking The objective of the re-ranking problem is to learn a conditional distribution P (L u, X) that produces ordered lists L maximizing the reward r(u, L). p. 7 7131
5 Conclusion
We proposed ReAlign, a reward-guided preference alignment framework for generative re-ranking. ReAlign couples a decomposable
listwise reward model, based on survival exposure and multi-objective
value estimation, with a candidate-aware autoregressive generator
trained through supervised pre-training and reward-guided posttraining.Where the paper says it
Framework Overview
ReAlign consists of two main components: a reward model and a generator. The reward model acts as an evaluator that estimates the expected utility of an ordered list by jointly modeling exposure and item value under user browsing behavior. p. 1 1 Framework Overview
Our framework, ReAlign (Re-ranking via reward-guided alignment), consists of two main components: a reward model and a
generator. The reward model acts as an evaluator that estimates
the expected utility of an ordered list by jointly modeling exposure and item value under user browsing behavior. The reward model acts as an evaluator that estimates
the expected utility of an ordered list by jointly modeling exposure and item value under user browsing behavior. The generator is a candidate-aware encoder–decoder model that autoregressively constructs ordered lists from the upstream candidate set.Where the paper says it
Where the paper says it
How the training signal is built
The reward model decomposes the listwise reward into position-wise contributions, modeling user exposure as a discrete-time survival process and item value through multiple per-objective heads with uncertainty-based weighting. The generator is trained in two stages: supervised pre-training on logged exposure lists to establish a relevance prior, followed by reward-guided post-training that aligns the generator with the reward model through complementary distribution and ranking alignment over within-pool standardized rewards. p. 0 The reward model decomposes
the listwise reward into position-wise contributions, modeling user
exposure through a survival-based exposure module and item value
through per-objective heads for heterogeneous engagement signals. The generator is trained in two stages: supervised pre-training
on logged exposure lists to establish a relevance prior, followed
by reward-guided post-training that aligns the generator with the
reward model through complementary distribution and ranking
alignment over within-pool standardized rewards.Where the paper says it
Where the gains come from
The training objective for the reward model minimizes L = Lexp + Lvalue, which factorizes a joint distribution over exposure and engagement signals. The post-training objective combines distribution alignment and ranking alignment over a diverse list pool, with within-pool reward standardization to make both terms robust to reward scale and bounded biases. p. 0 The generator is trained in two stages: a supervised pre-training stage that
imitates logged exposure lists to establish a stable relevance prior,
followed by a reward-guided post-training stage that aligns the
generator with reward-induced listwise preferences over a diverse
list pool. The post-training objective combines distribution alignment with a ranking alignment loss, both operating on within-pool
standardized rewards to remain robust to the absolute scale and
bounded biases of the reward.Where the paper says it
Key Design Choices
The framework employs a shared user and item encoder that maps users and items into embedding tokens for downstream modeling. The reward model uses a unified transformer-based architecture over the ordered list, which takes ranked item tokens and user tokens as input to produce context-aware ranked item tokens. p. 2 To jointly model reaching probability and item value, we employ a unified transformer-based
architecture over the ordered list, as illustrated in Figure 2. This
architecture takes ranked item tokens and user tokens as input and
produces context-aware ranked item tokens for all positions. A key component of the generator is the matching-based selection mechanism. Instead of predicting over the full item vocabulary,
the decoder produces a step-specific query representation 𝑞𝑡 at
each position, which serves as a query for selecting the next item
from the candidate set.Where the paper says it
Where the paper says it
Experimental Validation
Extensive offline experiments show that ReAlign consistently outperforms existing state-of-the-art baselines. Ablation studies confirm the contribution of each design choice, showing that ablating the survival module yields the larger drop in performance. p. 7 Ablating the
survival module yields the larger drop, confirming that positiondependent exposure discounting captures genuine user browsing
behavior and is critical for accurate listwise reward estimation. Ablating the multi-objective design causes a smaller but consistent drop, showing that disentangling heterogeneous engagement
signals further improves estimation beyond a single-objective formulation. Table 4 shows that both single-stage variants underperform the
full pipeline. Post-training alone reaches competitive performance
and slightly outperforms pre-training only, but the full pipeline
outperforms both, indicating that pre-training establishes a stable
starting policy and post-training refines it to align with the rewardinduced listwise preferences. 2 Overall Performance
Table 1 summarizes the offline performance on KuaiRand and Avito. For fair comparison with scoring-based re-rankers, we report two
variants: Evaluator Only uses our reward model as a list scorer
over candidate permutations, matching the inference protocol of
the baselines; ReAlign is the full framework where the generator
produces the final list after reward-guided preference alignment. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY YaChen and LiuBo
Table 1: Overall performance comparison on KuaiRand and
Avito datasets. KuaiRand Avito
NDCG uAUC NDCG uAUC
DNN 0. 7183
CAVE 0. 7258
Evaluator Only 0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Reward Model Architecture: The reward model should jointly model reaching probability and item value using a unified transformer-based architecture over the ordered list, which takes ranked item tokens and user tokens as input to produce context-aware ranked item tokens for all positions.
-
User Encoder Design: The user encoder should transform each user into a fixed set of user tokens capturing static features, long-term interaction sequences via a multi-interest extraction layer, and short-term interaction sequences via the most recent N interactions.
-
Item Encoder Design: The item encoder should combine an item ID embedding with a linear projection of side information (like category or price) through element-wise addition to form a unified item token reflecting both identity and feature semantics.
-
Training Pipeline: The generator should undergo two stages of training: supervised pre-training on logged exposure lists to establish a stable relevance prior, followed by reward-guided post-training that aligns the generator with the reward model through complementary distribution and ranking alignment over within-pool standardized rewards.
-
Post-training Objective: The post-training objective should combine distribution alignment, defined by minimizing cross entropy between the reward model's softmax distribution and the generator's induced distribution, with a ranking alignment loss that enforces consistency between the reward model's ordering and the generator's ordering over a diverse list pool.
-
Reward Model Decomposition: The listwise reward should be decomposed into position-wise contributions by modeling user exposure as a discrete-time survival process and item value through multiple per-objective heads with uncertainty-based weighting, yielding a calibrated and interpretable listwise reward.
-
Generator Selection Mechanism: The generator should utilize a matching-based selection mechanism where the decoder produces a step specific query representation matched against the contextualized candidate item tokens to compute a selection probability for the next item from the remaining candidate set.
-
Inference Strategy: For low-latency serving, the generator should employ greedy decoding, selecting the next item by taking argmax of the selection distribution and masking it to prevent reselection, enabling efficient re-ranking without operating over the full item space.
Abstract
Re-ranking refines the upstream ranked list to maximize listwise user utility. Existing two-stage generator-evaluator methods decouple list generation from list evaluation, leaving the generator without direct access to the listwise utility signal, while scoring-based one-stage methods score items under a fixed upstream permutation context without explicitly searching the permutation space. We propose ReAlign, a reward-guided preference alignment framework that couples a listwise reward model with a candidate-aware autoregressive generator. The reward model decomposes the listwise reward into position-wise contributions, modeling user exposure through a survival-based exposure module and item value through per-objective heads for heterogeneous engagement signals. The generator is trained in two stages: supervised pre-training on logged exposure lists to establish a relevance prior, followed by reward-guided post-training that aligns the generator with the reward model through complementary distribution and ranking alignment over within-pool standardized rewards. Extensive offline experiments show that ReAlign consistently outperforms existing state-of-the-art baselines.
Sources
- Seq2Slate: Re-ranking and Slate Optimization with RNNs
- OneSearch: A Preliminary Exploration of the Unified End-to-End Generative Framework for E-commerce Search
- OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment
- Revisit Recommender System in the Permutation Prospective
- GRN: Generative Rerank Network for Context-wise Recommendation
- Generative Recommendation for Large-Scale Advertising
- Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations
- Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations
- GPR: Towards a Generative Pre-trained One-Model Paradigm for Large-Scale Advertising Recommendation
- GoalRank: Group-Relative Optimization for a Large Ranking Model
- From Generation to Consumption: Personalized List Value Estimation for Re-ranking
- OneRec-V2 Technical Report
- Globally Optimized Mutual Influence Aware Ranking in E-Commerce Search
Related papers
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA
- One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations