StaRPO: Stability-Augmented Reinforcement Policy Optimization

The gist: StaRPO introduces a stability-augmented reinforcement learning framework that explicitly incorporates reasoning stability into the optimization objective by decomposing it into two computable lightweight metrics, Autocorrelation Function (ACF) and Path Efficiency (PE), to enhance both final-answer accuracy and logical stability in large language model reasoning.

Motivation

Reinforcement learning is effective in enhancing the accuracy of large language models in complex reasoning tasks However, reasoning is not merely about producing a correct final answer; from the perspective of human users, ideal reasoning should consist of an accurate final result in terms of completing the task; a logically sound reasoning process to support the explainability and reliability of the result; and proper knowledge grounding for factual correction to improve the reliability and truthworthiness of the model’s decisions and outputs. Logical soundness is a higher-level property describing the correctness of inferential relations between premises and conclusions. In complex real-world applications, such as multi-round decision-making, mathematical theorem proving, and medical consultations, LLMs often produce reasoning chains that remain logically inconsistent or structurally erratic. These flawed reasoning must be mitigated because they risk providing users with spurious justifications or misleading conclusions.

The Problem with Existing Frameworks

Existing RL policy optimization frameworks rely on final-answer correctness as feedback signals and rarely capture the internal logical structure of the reasoning process. Consequently, the models would generate fluent and semantically relevant responses but logically inconsistent, structurally erratic, or redundant. The logical soundness is a higher-level property describing the correctness of inferential relations between premises and conclusions. In this study, we focus on augmenting the logical correctness of the LLM reasoning process, i.e., the logical stability of reasoning. Logical stability contains three aspects: (1) semantic coherence: local step-to-step consistency in meaning; (2) logical coherence: the absence of discontinuities, leaps, or contradictions; and (3) goal alignment: maintaining a consistent trajectory toward the target answer. Since reasoning stability is hard to quantify and is not a parametric property, we adopt RL as an alignment framework rather than using explicit supervision.

StaRPO Methodology

To achieve this, StaRPO decomposes stability into two computable lightweight metrics and converts them into rewards. Specifically, we introduce the Autocorrelation Function (ACF) to evaluate local step-to-step coherence, and Path Efficiency (PE) to evaluate global goal-directedness of the reasoning trajectory. These stability rewards are combined with task rewards to provide complementary and process-aware feedback.

The metrics are defined as follows:

1. Local Coherence (ACF): ACF evaluates local stability by measuring directional changes at the embedding level, where ACFk is defined as cos(∆hk, ∆hk−1) = ⟨∆hk, ∆hk−1⟩∥∆hk∥ · ∥∆hk−1∥. The ACF reward is the average over all steps as racf = 1/(K − 1) Σ X K k=2 cos ∆hk, ∆hk−1.

2. Global Progression (PE): PE evaluates global stability by calculating the overall efficiency of the reasoning trajectory, defined as η = D/L + ϵ, where D = hK − h1 and L = Σ X K k=2 hk − hk−1. The path efficiency reward is rpe = hK − h1 / (Σ K k=2 hk − hk−1 + ϵ).

StaRPO Objective Function

We extend the Group Relative Policy Optimization (GRPO) framework by incorporating stability-augmented rewards into the optimization objective. The reward for each reasoning trajectory is defined as rStaRPO(x, yi) = r(x, yi) + λacf · racf + λpe · rpe, where λacf and λpe ≥ 0 control the trade-off between task success and stability. The advantage Ai for a response yi is computed using this stability-augmented reward: Ai = rStaRPO(x, yi) − mean[rStaRPO(x, yj)] G j=1 std[rStaRPO(x, yj)] G j=1. The overall objective function of StaRPO is formulated as JStaRPO(θ) = E x∼D min w i,t(θ)Ai clip w i,t(θ), 1 − ϵ, 1 + ϵ.

Experimental Validation

Experiments on four reasoning benchmarks show that StaRPO consistently outperforms compared baselines and can enhance both final-answer accuracy and logical stability. The correlation between abnormal values of these metrics and three types of reasoning errors is significant, as shown in Table 2. Specifically, for semantic drift, both low and high ACF values are significantly associated with error occurrence. For the logical leap, all metrics show consistent significance across tests. Overall, the abnormal ACF and PE values are significantly associated with reasoning instability errors under a significance level of α = 0.05. Ablation studies confirm that both ACF and PE rewards contribute positively to reasoning performance on GSM8K and GPQA, suggesting that each component captures a distinct aspect of reasoning quality for constraining reasoning stability during RL training.

Conclusion

In this paper, we introduce StaRPO, a stability-augmented reinforcement policy optimization framework to enhance the logic stability of the LLM reasoning process. To capture the internal structure of reasoning trajectories, we focus on the notion of logical stability and decompose it into complementary local and global components. Two lightweight stability metrics, ACF and PE, are designed to capture step-wise consistency and goal-directed progression. Our experiments across four reasoning benchmarks demonstrate the effectiveness of StaRPO by comparing with existing strong baselines. The abnormal ACF and PE values are significantly associated with reasoning instability errors.

--- Page 1 ---

StaRPO: Stability-Augmented Reinforcement Policy Optimization Jinghan Zhang, Fengran Mo, Tharindu Cyril Weerasooriya, Ruimin Dai, Xiaoyan Han, Yanjie Fu, Dakuo Wang and Kunpeng Liu. 1Clemson University, 2Université de Montréal, 3Accenture (United States), 4Arizona State University, 5Northeastern University jinghaz@clemson.edu, kunpenl@clemson.edu Abstract Reinforcement learning (RL) is effective in enhancing the accuracy of large language models in complex reasoning tasks. Existing RL policy optimization frameworks rely on final-answer correctness as feedback signals and rarely capture the internal logical structure of the reasoning process. Consequently, the models would generate fluent and semantically relevant responses but logically inconsistent, structurally erratic, or redundant. To this end, we propose StaRPO, a stability-augmented reinforcement learning framework that explicitly incorporates reasoning stability into the optimization objective. Our StaRPO decomposes stability into two computable lightweight metrics: the Autocorrelation Function (ACF) to evaluate local step-to-step coherence, and Path Efficiency (PE) to evaluate global goal-directedness of the reasoning trajectory. These stability rewards are combined with task rewards to provide complementary and process-aware feedback. We validate the effectiveness of using ACF and PE rewards by showing their correlation with logic errors on two backbone models. Experiments on four reasoning benchmarks show that StaRPO consistently outperforms compared baselines and can enhance both final-answer accuracy and logical stability. 1 Introduction Recent advances in large language models (LLMs) for reasoning-intensive tasks demonstrate remarkable effectiveness. In particular, reinforcement learning (RL) has significantly raised the performance ceiling of LLMs on complex problem-solving benchmarks [Guo et al., 2025; Team, 2024; Liu et al., 2024; Pennino et al., 2025; Bandyopadhyay et al., 2025] with appropriate optimization objectives and rewards. However, reasoning is not merely about producing a correct final answer. From the perspective of human users, ideal reasoning should consist of an accurate final result in terms of completing the task; a logically sound reasoning process to support the explainability and reliability of the result; and proper knowledge grounding for factual correction to improve the reliability and truthworthiness of the model’s decisions and outputs. Although final-answer accuracy and factual correctness can usually be clearly quantified and used as feedback signals in previous studies [Zhang et al., 2024; Mo et al., 2026], the logical soundness is a higher-level property describing the correctness of inferential relations between premises and conclusions [Govier and others, 2010; Zhang et al., 2026]. This is because logic does not have a quantitative formulation, but it is useful and widely recognized by human evaluators. In complex real-world applications, such as multi-round decision-making, mathematical theorem proving, and medical consultations, LLMs often produce reasoning chains that remain logically inconsistent or structurally erratic.

1. We can implement reasoning models that maintain high logical coherence by incorporating stability-augmented rewards into Group Relative Policy Optimization (GRPO). This allows models to learn from stability-augmented rewards such as the Stepwise Autocorrelation Reward (ACF) and Path Efficiency Reward (PE), which are designed to capture local step-to-step consistency and goal-directed progression.

2. The improved system can generate reasoning chains that exhibit superior logical stability across various benchmarks. Specifically, by using the combination of ACF and PE rewards, the model can be constrained to produce outputs that avoid semantic drift (low or high ACF) and logical leap (low or high metrics), leading to more reliable inferences.

3. The system will demonstrate enhanced performance in complex reasoning tasks like mathematical problem-solving by optimizing for process accuracy. The paper shows that StaRPO provides a substantial improvement in process accuracy, such as +6.14% on Qwen-1.5B, suggesting that the stability signals enhance both final-answer accuracy and logical stability.