Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning

International Conference on Learning Representations (ICLR) 2026

Zheng Zhang1,2, Ziwei Shan1, Kaitao Song3, Yexin Li2*, Kan Ren1*
1ShanghaiTech University 2State Key Laboratory of General AI, BIGAI 3Independent Researcher
*Corresponding author
Paper Code
Figure 1a
Figure 1b

Figure 1. Reward modeling paradigms comparison and RL behavior. CRM explicitly conditions each step reward on previous steps and links it to the final outcome; empirically, CRM is more robust to reward hacking.

Abstract

Process Reward Models (PRMs) have emerged as a promising approach to enhance the reasoning capabilities of large language models (LLMs) by guiding their step-by-step reasoning toward a final answer. However, existing PRMs either treat each reasoning step in isolation, failing to capture inter-step dependencies, or struggle to align process rewards with the final outcome. Consequently, the reward signal fails to respect temporal causality in sequential reasoning and faces ambiguous credit assignment. These limitations make downstream models vulnerable to reward hacking and lead to suboptimal performance. In this work, we propose Conditional Reward Modeling (CRM) that frames LLM reasoning as a temporal process leading to a correct answer. The reward of each reasoning step is not only conditioned on the preceding steps but also explicitly linked to the final outcome of the reasoning trajectory. By enforcing conditional probability rules, our design captures the causal relationships among reasoning steps, with the link to the outcome allowing precise attribution of each intermediate step, thereby resolving credit assignment ambiguity. Further, through this consistent probabilistic modeling, the rewards produced by CRM enable more reliable cross-sample comparison. Experiments across Best-of-N sampling, beam search and reinforcement learning demonstrate that CRM consistently outperforms existing reward models, offering a principled framework for enhancing LLM reasoning. In particular, CRM is more robust to reward hacking and delivers stable downstream improvements without relying on verifiable rewards derived from ground truth.

Motivation

Process Reward Models (PRMs) provide step-wise supervision for multi-step reasoning, yet existing approaches suffer from two key limitations.

  • Isolated step modeling. Many PRMs score each step independently, ignoring the causal dependency that later steps rely on the correctness of earlier ones.
  • Weak linkage to the final outcome. Step rewards are not explicitly tied to the final correctness, leading to ambiguous credit assignment and vulnerability to reward hacking (rewards increase while task accuracy declines).

To address this, we propose CRM, which models each step’s reward as a conditional probability given all previous steps, and explicitly connects process rewards to the final outcome through the conditional probability chain rule, enabling principled and consistent credit assignment.

Methodology

CRM models LLM reasoning as a temporal process where the probability of reaching a correct answer evolves with the step index t. Since it is difficult to directly quantify “how close” a trajectory is to the correct answer, we instead model the complementary event: the reasoning process entering a wrong state such that the trajectory can no longer yield the correct final answer. We define $z$ as the index of the first step where the trajectory enters this wrong state. If no wrong state occurs throughout a trajectory of length $T$, then $z > T$ and the final answer is correct; otherwise $z \le T$ and the final answer is incorrect.

$$W(t)=\Pr(z\le t)=\sum_{i=1}^{t}p(i), \qquad S(t)=\Pr(z>t)=1-W(t)$$

A reasoning trajectory is inherently causal: the correctness of step $t$ depends on all previous steps. To capture this dependency, CRM introduces the conditional wrong-state probability:

$$h(t)=\Pr(z=t\mid z\ge t)=\frac{p(t)}{S(t-1)}$$

Naturally, $1-h(t)$ is the probability that the current step is correct given all previous steps were correct. To explicitly link intermediate steps to the final outcome, CRM applies the chain rule of probability:

$$S(t)=\Pr(z>t)=\prod_{k=1}^{t}(1-h(k))$$

With $S(T)$ representing the probability that the reasoning process reaches the correct final answer, CRM then seeks a dense step-wise reward aligned with this outcome probability. We apply Potential-Based Reward Shaping (PBRS) and choose the potential function as:

$$\Phi(s_t)\equiv \log S(t)=\sum_{k=1}^{t}\log(1-h(k))$$

Substituting this potential into PBRS yields an explicit process reward for each step transition:

$$r_t=\log(1-h(t))$$

Training objective. At step $t$, CRM takes the question $x$ and the first $t$ reasoning steps as input, and predicts $h(t)=f_\phi(x,a_{\le t})$. For correct trajectories ($l=1$), we maximize $S(T)$:

$$\mathcal{L}_S(x,y)=-\log S(T)=-\log\prod_{t=1}^{T}(1-h(t))$$

For incorrect trajectories ($l=0$), we minimize $S(T)$ and encourage the model to localize the first wrong step $z$ by maximizing $p(z)$:

$$\mathcal{L}_W(x,y)=-\log(1-S(T)),$$ $$\mathcal{L}_z(x,y,z)=-\log p(z)=-\log\left(h(z)\prod_{t=1}^{z-1}(1-h(t))\right)$$

The overall loss is:

$$\mathcal{L}=\frac{1}{|D|}\sum_{i=1}^{|D|}\Big[l_i\mathcal{L}_S(x_i,y_i)+(1-l_i)\big(\mathcal{L}_W(x_i,y_i)+\mathcal{L}_z(x_i,y_i,z_i)\big)\Big]$$

This consistent probabilistic modeling ensures that $S(t)$ has the same semantics across different samples, enabling more reliable cross-sample comparison than prior approaches.

Figure 2

Figure 2. Training objective components and their roles (e.g., correctness likelihood vs. wrong-state localization).

Figure 3

Figure 3. Cross-sample comparability: CRM yields more consistent ranking signals across questions.

Experimental Results

CRM is evaluated in three downstream settings: Best-of-N sampling (trajectory selection), beam search (step-level guidance), and RL optimization (policy improvement under dense process rewards). Across settings, CRM improves accuracy and exhibits strong robustness to reward hacking.

Best-of-N Accuracy

Model Method GSM-Plus MATH500
@8@16@32@64@128 @8@16@32@64@128
Qwen2.5-3B-Instruct ORM 66.867.266.465.765.7 51.651.451.849.049.2
PRM 67.667.967.766.966.7 54.255.255.254.254.6
PQM 68.569.268.568.268.0 53.254.454.854.855.8
IPRM 65.566.266.866.566.2 52.452.052.052.253.0
CRM (ours) 67.868.667.968.468.7 53.056.456.655.856.6

Beam Search Accuracy

Model Method MATH500 GAOKAO2023
N=4N=8N=20N=100 N=4N=8N=20N=100
Qwen2.5-Math-1.5B ORM 50.7354.8056.8058.07 35.5838.1838.4440.17
PRM 51.8055.7356.8758.00 34.7237.8438.7038.96
PQM 52.6756.6058.8758.80 36.8838.6140.6139.83
IPRM 44.2747.2748.3347.47 32.5534.4635.3234.55
CRM (ours) 54.0758.4061.0063.00 38.7039.7441.0443.55

RL Optimization (Pass@1)

VR Setting Method MATH500 MinervaMath OlympiadBench AIME25 AIME24 AMC23
VR Disabled PURE 76.030.836.713.326.670.0
PRM 71.636.332.513.310.057.5
PQM 72.034.134.313.313.352.5
CRM (ours) 77.840.039.323.343.367.5
VR Enabled Prime 81.229.440.816.626.672.5
PURE 82.440.041.323.323.370.0
VR 76.238.638.016.630.062.5
CRM + VR 80.443.042.126.633.372.5

Note: “VR” denotes verifiable reward from outcome ground-truth. “VR Disabled” means training without any verifier-based reward.

Figure 4

Figure 4. RL dynamics and reward hacking analysis (reward, length, repetition, and downstream accuracy over training).

Key Takeaways

  • Principled credit assignment: step rewards are causally aligned with the final outcome through conditional probability rules.
  • Better selection & guidance: improves Best-of-N and beam search by producing more comparable reward signals across trajectories.
  • Robust RL: mitigates reward hacking and achieves strong pass@1 even without verifiable rewards.

BibTeX

@inproceedings{zhang2026crm,
  title     = {Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning},
  author    = {Zheng Zhang and Ziwei Shan and Kaitao Song and Yexin Li and Kan Ren},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026}
}