Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization

Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization

Published as a conference paper at COLM 2026

Abstract

Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token’s KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token’s implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model’s own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token’s contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval 2 and 12.2 points on Arena-Hard.

1. Introduction

Direct Preference Optimization (DPO) has emerged as a widely adopted method for aligning large language models with human preferences. By reparameterizing the reward function in the RLHF objective, DPO enables direct policy optimization from preference data without training a separate reward model. In the standard formulation, DPO aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as equally important to the preference signal.

However, not all tokens contribute equally to the quality of a response. A factual error in a single token can flip a preference label, while filler words rarely affect human judgment. Recent methods have sought to address this mismatch by introducing token-level importance signals into the preference optimization objective, drawing on external sources such as pre-trained teacher models, contrastive language model pairs, learned sparse masks, optimal transport matching, per-token KL constraints, prompted self-evaluation, or oracle-based token selection. While these approaches differ in their signal sources, they share a common pattern: token importance is computed prior to or outside the training loop and treated as approximately fixed throughout optimization. A natural alternative is to derive token credit from DPO’s own implicit reward, but recent work observed that this signal can assign disproportionate importance to non-critical tokens such as punctuation and line breaks, leaving its direct use largely unexplored.

In this work, we study the implicit reward not at a single snapshot but across the full trajectory of DPO training. Our analysis reveals that the token-level implicit reward undergoes substantial evolution during training: the set of tokens deemed important shifts considerably between early and late stages, and the importance ranking stabilizes only gradually over the course of optimization. Crucially, this gradual stabilization suggests that the noisy importance estimates observed by prior work reflect the early training state rather than an inherent limitation of the implicit reward. Since token importance continues to shift until convergence, any static credit assignment becomes increasingly stale as optimization progresses. This temporal perspective distinguishes our approach from methods that adapt KL regularization at the sample level or bootstrap implicit rewards at the sequence level—neither of which captures the within-training evolution of token-level importance.

Based on these observations, we show that relaxing DPO’s uniform KL coefficient to be token-specific yields an objective where each token’s contribution is scaled by a credit term. We derive that, under a variance-minimization criterion on the preference logit, the credit at each position should be proportional to the magnitude of the token’s implicit reward. Intuitively, tokens with larger implicit rewards carry more preference-relevant information and should be allowed to deviate further from the reference policy, while tokens with near-zero implicit rewards contribute little to the preference signal and benefit from tighter regularization. Since the implicit reward evolves during training, the credit should also evolve—any static approach uses an increasingly stale signal by design.

We propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that periodically extracts implicit reward magnitudes as token credit during DPO training. Se-DPO adds no external models and only a lightweight calibration network with minimal computational overhead.

Contributions:

• We observe that DPO’s token-level implicit reward shifts considerably over the course of training, with limited overlap between top-r̂t tokens at early and late stages.

• We introduce token credit to modulate per-token KL regularization strength, and analyze how it can be informed by implicit reward magnitude and estimation confidence.

• We propose Se-DPO, which derives token credit online from the model’s own evolving internal signals via a lightweight calibration network, requiring no external models and achieving up to 50.6% win rate on AlpacaEval 2 and 43.3% on Arena-Hard.

2. Preliminaries

2.1 DPO and Implicit Reward

The standard RLHF objective maximizes expected reward subject to a KL divergence constraint against a reference policy πref:

max Ex∼D,y∼πθ [r(x, y)] − β DKL[πθ(·x) ∥ πref(·x)]

DPO reparameterizes the reward in terms of the optimal policy, yielding a supervised loss over preference pairs (x, yc, yr):

LDPO = −E(x,yc,yr) log σ(β log πθ(ycx)/πref(ycx) − β log πθ(yrx)/πref(yrx))

Since language models generate tokens autoregressively, the sequence-level log-ratio decomposes as a sum of token-level terms. Rafailov et al. (2024) showed that DPO implicitly learns a token-level reward:

r̂t = β(log πθ(ytx, y<t) − log πref(ytx, y<t))

2.2 Token-Specific KL Regularization

Standard DPO applies a uniform KL coefficient β to all token positions. In this work, we consider a generalization where the KL strength varies across positions:

max Ey∼π [∑t rt − βt log(π(ytx, y<t)/πref(ytx, y<t))]

where βt controls the KL penalty at position t. At each position, the optimization problem is a KL-regularized optimization whose closed-form solution is:

π*(ytx, y<t) = (1/Zt) πref(ytx, y<t) exp(rt(yt)/βt)

Rearranging yields the implicit reward:

rt(yt) = βt(log π*(ytx, y<t) − log πref(ytx, y<t)) + βt log Zt

Define ct = β/βt and recall the token-level implicit reward r̂t = β(log πθ(ytx, y<t) − log πref(ytx, y<t)). The resulting preference logit is:

Δ̂ = ∑t ct(c) r̂t(c) − ∑t ct(r) r̂t(r)

where standard DPO is recovered when βt = β (i.e., ct = 1) for all t. A smaller βt at position t yields a larger credit ct, reducing the KL penalty and allowing the policy to deviate further from the reference at that position. The central question is how to set βt—equivalently, how to allocate per-token KL budgets.

3. Method

3.1 Token Credit Dynamics During Training

Existing token-level methods compute importance prior to or outside the training loop, treating it as approximately fixed. However, it remains unclear how token importance behaves as training progresses. To investigate, we track the implicit reward across the trajectory of standard DPO training.

Token importance evolves continuously during training. Figure 1 provides empirical motivation for our self-evolving credit design. We first revisit the implicit reward magnitude r̂t, which reflects token-level preference signals. As shown in Figure 1(a,b), top-r̂t tokens change continuously during the original training run: their rank correlation with the final checkpoint increases over time, while the early-to-final top-20% overlap remains limited.

To verify that this phenomenon also appears in the actual credit used by Se-DPO, we run a separate Se-DPO LoRA credit-dynamics probe on fixed evaluation samples. As shown in Figure 1(c–f), the learned credit ct used for per-token KL-budget allocation is also dynamic. Its correlation with the final checkpoint starts from 0.493 and gradually reaches 1.0, while the early-to-final top-20% overlap is only 33.6%. In contrast, the reference entropy Href remains static. These observations suggest that token credit is shaped by the evolving model state, motivating our online credit update rather than a static per-token KL-budget allocation.

3.2 Two Complementary Signals for Token Credit

The analysis above establishes that token credit should be extracted online. We now ask: what signals should inform the credit assignment?

The implicit reward r̂t provides a natural candidate. However, a high r̂t can arise from multiple causes: the model may have learned a meaningful preference at that position, or the reference model may simply be uncertain there, making the log-ratio noisy. Using r̂t alone conflates signal and noise.

We observe that the reference model’s token-level entropy provides a complementary signal:

Href,t = −∑v∈V πref(vx, y<t) log πref(vx, y<t)

At positions where Href,t is high, the reference model is uncertain, and the implicit reward is inherently noisier; at positions where Href,t is low, the reference model is confident, and deviations from it carry stronger signal.

This decomposition connects to the following result on optimal credit:

Proposition 1. Consider the preference logit Δ̂ = ∑t ct r̂t, where each r̂t = rt* + ϵt is a noisy estimate of the true token reward with Var[ϵt] = σt2. Since ct = β/βt > 0 by construction, we require ct ≥ 0. Assume the noise is m-dependent: Cov[ϵt, ϵt′] = 0 for t − t′ > m.

(a) Under independent noise (m = 0), the non-negative credits that minimize Var[Δ̂] subject to ∑t ctrt* = C satisfy:

ct* ∝ rt*/σt2

(b) For m-dependent noise with bounded credits ct ∈ [cmin, cmax], the cross-covariance contribution to Var[Δ̂] is at most O(m) times the diagonal term ∑t ct2σt2. The solution from (a) thus remains a valid first-order approximation when m ≪ T.

Proof sketch. For Part (a), we minimize Var[Δ̂] = ∑t ct2σt2 subject to the signal-preservation constraint ∑t ctrt* = C with ct ≥ 0. The Lagrangian first-order condition ∂L/∂ct = 2ctσt2 − λrt* = 0 directly yields ct* = λrt*/(2σt2) ∝ rt*/σt2, which is automatically non-negative. For Part (b), with bounded credits and m-dependent noise, the cross-covariance contribution to Var[Δ̂] scales as O(m · cmax2ρmax) relative to the diagonal term, remaining a bounded perturbation when m ≪ T.

Part (b) ensures that the conclusion is robust to the local correlations inherent in autoregressive generation, provided the credit values remain bounded—a property guaranteed by the mean normalization. We verify empirically that the noise autocorrelation drops below 0.05 within lag 1, yielding m/T < 0.004 (Figure 2a), confirming that the independent-noise solution from Part (a) is itself a close approximation.

We use r̂t as a proxy for rt*. For σt2, we observe that reference entropy Href,t provides an informative proxy: at high-entropy positions the reference model distributes probability mass across many tokens, so the log-ratio log πθ/πref is inherently more sensitive to small changes in πθ, increasing the estimation noise of r̂t. Empirically, we estimate σt2 as the variance of r̂t across the latter half of training checkpoints and find Spearman ρ = 0.86 between Href,t and σ̂t2 (Figure 2b–c), confirming this relationship. However, a binned analysis controlling for r̂t reveals that the correlation is strongest at low-reward positions and diminishes at high-reward positions, indicating a nonlinear interaction between the two signals. This motivates a calibration network that learns the mapping jointly (Section 3.3), rather than fixing the functional form ct ∝ r̂t/Href,t.

On potential-based shaping. The token-level implicit reward is not unique under arbitrary potential-based shaping. Se-DPO does not claim invariance to all such decompositions. Instead, it operates under the canonical token decomposition induced by the DPO log-ratio in Eq. 3, which is reproducible for a fixed policy–reference pair and provides a practical credit signal. Our ablations in Section 4.3 support this convention empirically: using r̂t alone already improves over uniform DPO, while the large gap between Static-Credit and Se-DPO shows that tracking the evolution of this canonical signal is more important than merely using a fixed token-level decomposition.

3.3 Credit Calibration Network

Proposition 1 suggests credit of the form ct ∝ r̂t/Href,t, but the proxy relationships rt* ≈ r̂t and σt2 ≈ Href,t are approximate and interact nonlinearly. Moreover, the closed-form ratio is numerically unstable at low-entropy positions where σt2 → 0, producing arbitrarily large credits for tokens with negligible reward. To account for these issues, we parameterize the credit function as a lightweight MLP fϕ that maps the two signals to a scalar credit:

c̃t = fϕ(r̂t, Href,t), s ∈ c, r

where ϕ is shared across all token positions and both response sides. Without constraint, the credit function could degenerately shrink all credits toward zero, which would trivially reduce the DPO loss. To prevent this, we normalize credits to have unit mean within each response, matching the total credit of standard DPO while allowing redistribution across tokens.

3.4 Se-DPO: Complete Algorithm

During the first W steps (the warmup phase), Se-DPO uses standard DPO with uniform credit (ct = 1), avoiding amplification of early noise when r̂t ≈ 0 for all tokens. From step W+1 onward (the guided phase), at each training step Se-DPO computes r̂t and Href,t from the current forward pass through the policy and reference models. Since DPO already evaluates both models on each batch, these quantities can be derived from the existing forward pass with minimal overhead. The calibration network fϕ then maps r̂t and Href,t to token credit.

Loss. The token credit modulates each token’s contribution to the preference loss:

LSe-DPO = −log σ(∑t ct(c) r̂t(c) − ∑t ct(r) r̂t(r))

Se-DPO adds only the forward and backward pass of the calibration network per training step. Both input signals—r̂t and Href,t—are derived from the standard DPO forward pass, requiring no additional model evaluations.

4. Experiments

4.1 Setup

Models and data. We conduct experiments on three models: Llama-3.2-3B-Instruct, Llama-3-8B-Instruct, and Gemma-2-2B-it. Following Zhu et al. (2025) and Meng et al. (2024), we use prompts from the UltraFeedback dataset and let each model generate 5 responses with a temperature of 0.8. These responses are then ranked using ArmoRM, with the highest and lowest-ranked responses selected as the chosen and rejected samples, respectively. For Llama-3-8B-Instruct, we additionally use PairRM for preference annotation to evaluate robustness across different annotators. Each dataset contains approximately 60K preference pairs. We compare against DPO, SimPO, TGDPO, and Static-Credit, using the original method-specific settings whenever applicable. The SFT row refers to the original instruction-tuned model evaluated directly, without additional supervised fine-tuning. All methods are trained for 1 epoch with the AdamW optimizer and a cosine learning rate schedule with 10% warmup.

Evaluation benchmarks. We use three prominent instruction-following benchmarks: AlpacaEval 2, Arena-Hard, and MT-Bench. For AlpacaEval 2, we report the win rate against GPT-4 Turbo. For Arena-Hard, we report the win rate against GPT-4-0314. For MT-Bench, we report the score and win rate against GPT…

Based on this paper, here are the specific improvements you can make to AI systems:

• Improvement: Replace uniform token weighting in preference optimization with a live, self-evolving credit mechanism that adjusts per-token KL regularization strength based on the model's own evolving internal signals.

• What the improved system can do: Automatically identify and prioritize tokens that carry preference-relevant information (e.g., factual claims, key decisions) while tightly regularizing filler or low-information tokens, without requiring external reward models or pre-computed importance scores.

• Improvement: Integrate reference model entropy as a complementary signal to implicit reward magnitude, using a lightweight calibration network to learn the nonlinear mapping between these signals and optimal token credit.

• What the improved system can do: Distinguish between meaningful preference signals and noise caused by reference model uncertainty, preventing the system from over-amplifying unreliable tokens (e.g., high-reward tokens at high-entropy positions where the reference model is uncertain).

• Improvement: Implement a warmup phase followed by continuous, online updating of token credit throughout training, rather than computing importance once and freezing it.

• What the improved system can do: Avoid amplifying early training noise when implicit rewards are near-zero, then adapt credit assignment as the model learns—capturing the shift in which tokens become important at different training stages (e.g., early focus on syntax, later focus on factual correctness).

• Improvement: Derive all credit signals from the model's own forward pass (implicit reward magnitudes and reference entropy), requiring only a small MLP calibration network with minimal overhead.

• What the improved system can do: Operate fully self-contained during alignment, eliminating dependencies on pre-trained teachers, contrastive LLM pairs, or external reward models—making the approach more scalable and cost-effective for deployment.

• Improvement: Apply the theoretical result that optimal token credit is proportional to |implicit reward| / noise variance, ensuring the preference logit has minimal variance while preserving signal strength.

• What the improved system can do: Produce more stable and reliable preference optimization, reducing the risk of overfitting to noisy tokens and improving generalization across different preference annotators (e.g., ArmoRM vs. PairRM) and base models.

• Improvement: Leverage the dynamic credit mechanism to improve performance on pairwise instruction-following benchmarks.

• What the improved system can do: Achieve up to 50.6% win rate on AlpacaEval 2 and 43.3% on Arena-Hard (vs. GPT-4), with gains of +8.1 to +12.2 points over strong baselines—demonstrating better adherence to complex instructions and more robust response quality, particularly in tasks where specific tokens determine correctness.

• Improvement: Ensure that performance gains are not merely artifacts of longer responses, by maintaining length-controlled win-rate advantages.

• What the improved system can do: Generate responses that are preferred by human judges even when controlling for length, indicating genuine quality improvements rather than verbosity bias—critical for real-world deployment where concise, accurate responses are valued.

Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token's KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token's implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model's own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token's contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval 2 and 12.2 points on Arena-Hard.