From PPO to GRPO
We talked about PPO in the previous post, now let's first think about how PPO can be applied to finetune an LM. In particular, it optimizes the LM by maximizing the following objective: $$J_{\text{PPO}} (\theta) = E_{[q \sim P(Q), o \sim \pi_{\theta_{\text{old}}}]} \frac{1}{|o|} \sum_{t=1}^{|o|}\min[ \frac{\pi_\theta(o_t | q, o_{< t})}{\pi_{\theta_{\text{old}}} (o_t | q, o_{< t})}A_t, \text{clip}(\frac{\pi_\theta(o_t | q, o_{< t})}{\pi_{\theta_{\text{old}}}(o_t | q, o_{< t})}, 1 - \epsilon, 1 + \epsilon)A_t]$$ where $\pi_\theta$ and $\pi_{\theta_{\text{old}}}$ are the current and old policy models, and $q, o$ are questions and responses sampled from the question dataset and the old policy $\pi_{\theta_{\text{old}}}$. $A_t$ is advantage for the $t$-th token, which is computed via GAE based on the rewards $\{r_{\geq t}\}$ and a learned value function $V_\phi$ (reminder: $\hat{A}^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty} \gamma^{l} \delta_{t+l}^V$, where $\delta_t^V = - V(s_t) + r_t + \gamma V(s_{t+1})$). The value function $V$ is trained alongside the policy model. Typically people add a per-token KL penalty from a reference model in the reward at each token: $$r_t = r(q, o_{\leq t}) - \beta \log \frac{\pi_\theta(o_t | q, o_{< t})}{\pi_{ref} (o_t | q, o_{< t})}$$ where $\pi_{ref}$ is the reference model, which is the SFT or base model checkpoint before RL. The raw reward of each token $r(q, o_{\leq t})$ is a design choice, sometimes people only assign an outcome reward to the final token, sometimes people train a reward model to score the intermediate steps. The main complaint that people have about the above PPO objective is that computing the advantage through GAE involves training a value function model, which takes substantial memory and compute. To avoid this, DeepSeek came up with an easier way to compute the advantage, which leads to GRPO.

As a reminder, when we introduced value function back in the previous blog post, the motivation is to use the value function $V$ as a baseline for variance reduction. GRPO achieves this without training a value function. Specially, for each question $q$, we sample a group of outputs $\{o_1, o_2, \dots, o_G\}$ from $\pi_{\theta_{\text{old}}}$, and we get their rewards $\mathbf{r} = \{r_1, r_2, ..., r_G\}$ through either a reward model or rule-based rewards. And then we compute the advantage $\hat{A}_{i,t}$ of each token in each response as: $$\hat{A}_{i,t} = \tilde{r}_i = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$$ which means we use the group mean as a baseline, and assign the same advantage for every token in the same response. Plug this new advantage formula back into the overall training objective: $$J_{\text{GRPO}} = E_{[q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}]} \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \{ \min [\frac{\pi_\theta(o_{i,t} | q, o_{i, < t})}{\pi_{\theta_{\text{old}}}(o_{i,t}| q, o_{i,< t})} \hat{A}_{i,t}, \text{clip} (\frac{\pi_\theta(o_{i,t} | q, o_{i, < t})}{\pi_{\theta_{\text{old}}}(o_{i,t}| q, o_{i,< t})}, 1 - \epsilon, 1 + \epsilon) \hat{A}_{i,t}] - \beta \mathbb{D}_{KL} [\pi_\theta || \pi_{ref}] \}$$ At the start of optimization on a freshly collected batch, $\pi_\theta = \pi_{\theta_{\text{old}}}$, so every importance ratio equals 1 and clipping is inactive. This does not make the ratio's gradient zero: the old policy in the denominator is held fixed, while the numerator depends on the parameters being updated. Writing $o$ for a token and $h$ for its context: \[ \begin{align*} \nabla_\theta \frac{\pi_\theta(o|h)}{\pi_{\theta_{\text{old}}}(o|h)} &= \frac{\nabla_\theta \pi_\theta(o|h)}{\pi_{\theta_{\text{old}}}(o|h)} \\ &= \frac{\pi_\theta(o|h)}{\pi_{\theta_{\text{old}}}(o|h)} \nabla_\theta \log \pi_\theta(o|h), \\ \left.\nabla_\theta \frac{\pi_\theta(o|h)}{\pi_{\theta_{\text{old}}}(o|h)}\right|_{\theta=\theta_{\text{old}}} &= \left.\nabla_\theta \log \pi_\theta(o|h)\right|_{\theta=\theta_{\text{old}}}. \end{align*} \] It is the gradient at the starting policy, rather than the entire objective, that matches the ordinary policy-gradient form. After a parameter update, the current policy can differ from the old policy, so the ratios need not remain 1 when we reuse the same batch.

Note that here we are using the reverse KL, and if we use the $k_3$ estimator, this will become: $$\mathbb{D}_{KL} [\pi_\theta || \pi_{ref}] = \frac{\pi_{ref}(o_{i,t} | q, o_{i, < t})}{\pi_{\theta}(o_{i,t}| q, o_{i,< t})} - \log \frac{\pi_{ref}(o_{i,t} | q, o_{i, < t})}{\pi_{\theta}(o_{i,t}| q, o_{i,< t})} - 1$$ Now you might notice a slight mismatch here: when computing $\mathbb{D}_{KL} [\pi_\theta || \pi_{ref}]$, we should be sampling from $\pi_\theta$ while in reality we are sampling from $\pi_{\theta_{\text{old}}}$. This is fine if we are operating in the on-policy setting. If in the off-policy setting, we should multiply the $k_3$ estimator with an importance sampling weight $w_{i,t} = \frac{\pi_\theta(o_{i,t} | q, o_{i,< t})}{\pi_{\text{old}}(o_{i,t} | q, o_{i,< t})}$ (see the RPG paper for more discussion on this).

RLHF and DPO
So far, we have assumed that a reward is available. For tasks like writing a helpful response, it is often easier for humans to compare two outputs than to assign an absolute score. Both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) use these pairwise preferences to train a policy.

Learning a reward from pairwise preferences. The Bradley–Terry model assumes that comparisons are noisy: an item with a higher reward is more likely to be preferred. For two items $b_i$ and $b_j$: \[ P_\phi(b_i \succ b_j) = \frac{\exp(r_\phi(b_i))}{\exp(r_\phi(b_i)) + \exp(r_\phi(b_j))} = \sigma\!\left(r_\phi(b_i) - r_\phi(b_j)\right), \] where $\sigma(z) = 1/(1 + e^{-z})$. Given comparisons $\mathcal{D}_{\mathrm{pref}} = \{(b_i, b_j, \mu)\}$, let $\mu = 1$ when $b_i$ is preferred, $\mu = 0$ when $b_j$ is preferred, and $\mu = 0.5$ for a tie. We fit the reward model by minimizing binary cross-entropy, treating ties as equal weight on both outcomes: \[ \begin{aligned} &\mathcal{L}_{\mathrm{pref}}(\phi) = -\mathbb{E}_{(b_i,b_j,\mu)\sim\mathcal{D}_{\mathrm{pref}}}\Bigl[ \mu\log P_\phi(b_i \succ b_j) + (1-\mu)\log P_\phi(b_j \succ b_i)\Bigr]. \end{aligned} \] The same idea applies to trajectories: if $R_\phi(\tau) = \sum_t r_\phi(s_t,a_t)$, then $P_\phi(\tau^1 \succ \tau^2) = \sigma(R_\phi(\tau^1) - R_\phi(\tau^2))$. For LMs, we usually score an entire response with a scalar reward $r_\phi(x,y)$. Writing $x$ for a prompt and $y_w,y_l$ for its preferred and dispreferred responses, a dataset $\mathcal{D} = \{(x,y_w,y_l)\}$ gives the reward-model loss: \[ \mathcal{L}_{\mathrm{RM}}(\phi) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[\log\sigma\!\left(r_\phi(x,y_w)-r_\phi(x,y_l)\right)\right]. \] RLHF: optimize the learned reward. Starting from a pretrained, typically instruction-tuned LM, we keep a frozen reference $\pi_{\mathrm{ref}}$ and initialize a trainable policy $\pi_\theta$ from it. With the learned reward model held fixed, we maximize the following objective, for example using PPO: \[ \begin{aligned} J_{\mathrm{RLHF}}(\theta) &= \mathbb{E}_{x\sim P(X),\,y\sim\pi_\theta(\cdot\mid x)}\Bigl[ r_\phi(x,y) - \beta\mathbb{D}_{\mathrm{KL}}\!\left[ \pi_\theta(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\right]\Bigr]. \end{aligned} \] Here $\beta > 0$ controls how strongly we penalize moving away from the reference. Equivalently, a sampled response receives the regularized reward: \[ R_\theta(x,y) = r_\phi(x,y) - \beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}. \] The log-ratio is a sample estimate of reverse KL; its expectation over $y\sim\pi_\theta(\cdot\mid x)$ is the KL penalty above.

DPO: express the reward through the policy. For a fixed reward $r(x,y)$, the KL-regularized objective has a closed-form optimal policy when optimizing over all response distributions on the reference's support: \[ \begin{aligned} \pi^*(y\mid x) &= \frac{1}{Z(x)}\pi_{\mathrm{ref}}(y\mid x) \exp\!\left(\frac{r(x,y)}{\beta}\right), \\ Z(x) &= \sum_y \pi_{\mathrm{ref}}(y\mid x) \exp\!\left(\frac{r(x,y)}{\beta}\right). \end{aligned} \] The partition function $Z(x)$ normalizes the distribution, but summing over all possible responses is intractable. Rearranging the optimal-policy formula lets us write the reward in terms of the policy: \[ r(x,y) = \beta\log\frac{\pi^*(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} + \beta\log Z(x). \] In the Bradley–Terry model, only the reward difference between two responses to the same prompt matters, so the $\beta\log Z(x)$ terms cancel. Replacing $\pi^*$ with our trainable policy $\pi_\theta$ and substituting into the preference loss gives DPO: \[ \begin{aligned} &\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\Biggl[ \log\sigma\!\Biggl( \beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)} \Biggr)\Biggr]. \end{aligned} \] This directly trains the policy on preference pairs without fitting a separate reward model or sampling new rollouts during training. DPO conventionally defines the implicit reward as: \[ \hat{r}_\theta(x,y) = \beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}. \] More generally, the full reward-policy relation allows a prompt-dependent additive constant: \[ r_\theta(x,y) = \hat{r}_\theta(x,y) + c(x), \qquad c(x) = \beta\log Z_\theta(x). \] This constant cancels in pairwise reward differences and does not change the optimal policy under the KL-regularized objective. The policy $\pi_\theta$ is the optimal policy for its own implicit reward. Here $\pi_\theta(y\mid x)$ is the probability of the whole response, so $\log\pi_\theta(y\mid x) = \sum_t\log\pi_\theta(y_t\mid x,y_{< t})$.

On Policy Distillation
Off-policy distillation trains on teacher-generated trajectories, so the student may struggle with its own mistakes, while outcome-based RL provides only sparse feedback. On-policy distillation combines student-generated rollouts with dense teacher feedback by minimizing the per-token reverse KL between the student $\pi_\theta$ and teacher $\pi_{\text{teacher}}$ at each token in the student-generated trajectory: \[ \begin{aligned} &\mathbb{D}_{\mathrm{KL}}\!\left[\pi_\theta(\cdot \mid q, o_{< t}) \,\|\, \pi_{\text{teacher}}(\cdot \mid q, o_{< t})\right] \\ &\qquad = \mathbb{E}_{o_t \sim \pi_\theta(\cdot \mid q, o_{< t})} \left[\log \frac{\pi_\theta(o_t \mid q, o_{< t})}{\pi_{\text{teacher}}(o_t \mid q, o_{< t})}\right] \\ &\qquad = \mathbb{E}_{o_t \sim \pi_\theta(\cdot \mid q, o_{< t})} \left[\log \pi_\theta(o_t \mid q, o_{< t}) - \log \pi_{\text{teacher}}(o_t \mid q, o_{< t})\right]. \end{aligned} \]

References: