GRPO: Group Relative Policy Optimization

How GRPO turns a handful of sampled answers into a training signal without a value network: a primer, four interactive studies and the decisions they settle

draft · updated 2026-08-225 studies18 min12 sources

In short

GRPO drops PPO's value model; with beta = 0 and a rule-based reward it trains and holds the policy alone
The group is the baseline: an answer's advantage is how far it sits from its siblings
Only mixed groups carry signal, and it is largest when the model is right about half the time
On a fully on-policy step the ratio is exactly 1 and clipping never binds; it matters only when rollouts are reused or generation is off-policy
Averaging each completion's loss over its own tokens favours short correct answers and long wrong ones; a token-level or constant denominator removes that

primerwhat one GRPO step does

GRPO trains a model by sampling several answers to the same question, scoring them, and pushing probability toward the answers that scored above their own group's average

GRPO is a reinforcement-learning step, usually run after pretraining and supervised fine-tuning (DeepSeek-R1-Zero skips the fine-tuning stage). For one prompt q, the current policy samples a group of G completions o₁..o_G. (The sampling policy is written π_old; on a single-update step it is the same model as π_θ, the one being updated. Study D covers when they differ.) Each completion gets a reward r_i. Here it is 1 if the final answer is correct and 0 otherwise: a rule-based accuracy reward of the kind DeepSeek-R1 uses ("evaluates whether the response is correct", with answers "in a specified format (e.g., within a box), enabling reliable rule-based verification"). Each completion then gets an advantage Â_i, how much better or worse it did than expected, and the update raises the probability of completions with positive advantage and lowers it for negative ones.

The "expected" part is the baseline. Subtracting a baseline from the reward leaves the expected direction of the update unchanged and changes only its noise (derivation under Go deeper); DeepSeekMath describes PPO's value baseline as being there "for variance reduction". PPO learns the baseline with a separate value model (its critic), "typically another model of comparable size as the policy model". GRPO "foregoes the critic model, instead estimating the baseline from group scores". Study B (Figure 3) shows what that group baseline does to each answer; Study C (Figure 4) follows it over a training run.

Figure 1 · Read it: stage 4 is what GRPO changes. The two correct answers sit at +1.29 and the four wrong at -0.65 because each is measured against its own group's mean (1/3) and divided by the group std (0.52); Figure 3 lets you move these. Every token of a completion gets that one scalar. Held fixed: G = 6, k = 2, binary reward; clipping and the KL term (a penalty for drifting from the reference model) are left out of the strip.

Go deeper: why any baseline is allowed

For a policy π_θ and any constant b, the baseline term contributes nothing to the expected gradient:

E_o~π[ b · ∇_θ log π_θ(o) ] = b · ∇_θ Σ_o π_θ(o) = b · ∇_θ 1 = 0

so E[(r - b) ∇ log π] = E[r ∇ log π] for every constant b, while the variance of the estimate depends on b. GRPO's baseline is not a constant: the group mean contains the sample's own reward r_i, so the expected gradient is scaled by (G - 1)/G (same direction, smaller magnitude; RLOO's leave-one-out mean of the other G - 1 samples avoids this), and the division by the group std is a further sample-dependent rescaling (Study C and Dr. GRPO). In DeepSeekMath's words, the value function in PPO "is treated as a baseline in the calculation of the advantage", and GRPO "uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline". With outcome supervision DeepSeekMath "sets the advantages ... of all tokens in the output as the normalized reward" (section 4.1.2).

Awhat GRPO removes

GRPO drops PPO's value model; with a KL weight of β = 0 and a rule-based reward it trains and holds only the policy

PPO trains a value model alongside the policy to estimate each token's expected return, and holds two frozen models: a reference model for the KL penalty and a reward model for the score. DeepSeekMath's Figure 4 shows that inventory, policy and value trained, reference and reward frozen, and says why GRPO drops the value model: it "is typically another model of comparable size as the policy model" and "brings a substantial memory and computational burden". GRPO's baseline comes from the group instead (Figure 3), so the value model goes. What remains depends on two choices that are not GRPO's: TRL loads no reference model when β = 0, its default ("If 0.0 (default), the reference model is not loaded"), and a rule-based reward needs no reward model at all (DeepSeek-R1-Zero: "We do not apply the outcome or process neural reward model").

To see what each method holds, switch the reward type and β and count the boxes. No checked source gives a memory percentage; the papers say "less memory consumption" and "significantly reducing training resources", and the inventory is the honest form of the claim.

Figure 2 · Read it: boxes are equal because the value model is of comparable size to the policy; solid is trained, dashed is frozen. PPO always holds the value model; with a rule-based reward it drops the reward model too (Tulu 3 ran PPO with verifiable rewards and a value model). GRPO at TRL's defaults with a rule reward is one box. Flip point: none; this is a count, not a curve. Held fixed: the inventory of DeepSeekMath Figure 4; optimizer state, activations, KV caches and any separate generation engine are not drawn.

Count models before choosing: GRPO removes the one model besides the policy that PPO has to train; β and the reward type decide the rest for either method.

PPO: policy and value trained, reference and reward held; critic-free methods "require loading one less model copy compared to Vanilla PG and PPO".
GRPO with β = 0 (TRL default) and a rule-based reward: the policy only.
No checked source gives a percentage; quote the words, not a number.

Simplified: a model count is not a memory budget: optimizer state for the trained models, activations, KV caches and a vLLM generation copy all take memory and are not drawn.

Go deeper

DeepSeekMath eq. 2 puts PPO's KL penalty in the per-token reward, which needs π_ref; GRPO's eq. 3 puts the KL in the loss; with β = 0 TRL skips it and "the reference model is not loaded". The value model's job in PPO, the baseline "for variance reduction", is taken by the group mean (Figure 3).

Bthe advantage

The group is the baseline: an answer's advantage is how far it sits from its siblings

Figure 3 is stage 4 of Figure 1, with controls. To see how an answer's advantage depends on the group, drag the group-size (G) and correct-count (k) sliders. GRPO samples G completions for one prompt and scores each with a reward; here the reward is 1 for a correct answer and 0 for a wrong one. PPO would compare each reward to a learned value estimate; GRPO drops that value model and compares it to the group mean instead. It then divides by the group's standard deviation, so the advantage says how unusual an answer is within its own group. In TRL the group std is Bessel-corrected (it divides by G - 1, not G) and a 1e-4 floor keeps the division finite.

The sign comes from the mean, the magnitude from the std. To see the extreme case, set G to 8 and k to 1: the lone correct answer earns +2.47 while each wrong one gets -0.35. Then set k to 0 or to G: every reward equals the mean and every advantage collapses to 0.

group size G8

correct answers k1

Figure 3 · Read it: with G = 8 and k = 1 the correct answer sits 7 times further from the mean than each wrong one, so its advantage is 7 times larger in magnitude. Raise k to 4 and the two sides balance at about ±0.94. Flip point: at k = 0 or k = G every reward equals the mean, so every numerator is 0 and all G advantages are 0 (the 1e-4 floor only keeps 0/0 from becoming NaN): the group contributes no policy gradient (a KL term, if one is used, still does). Held fixed: binary rewards, scale_rewards="group" (the TRL default), no KL term.

Sample groups where the model sometimes succeeds.

When a prompt's pass rate is near 0 or 1, drop or re-weight it: almost every group it produces is a zero-signal group.
Larger G makes zero-signal groups rarer: at a pass rate of 0.9 the chance that a group comes back all-correct or all-wrong is 43% at G = 8 and 0.1% at G = 64 (0.9^G + 0.1^G); generation cost grows linearly with G. DeepSeekMath sampled 64 outputs per question; TRL's num_generations defaults to 8.
Changes if rewards are continuous (magnitudes spread out but the sign rule holds) or if scale_rewards is "batch" or "none" (std taken over the batch, or not divided at all).

Simplified: real reward functions can be continuous, and TRL sums several weighted reward functions; TRL also supports NaN rewards (ignored by nanmean/nanstd) and batch-level scaling. The sign and zero-signal behaviour shown here do not change under those options.

Go deeper

A_i = (r_i - mean(r_1..r_G)) / (std(r_1..r_G) + 1e-4)
std uses Bessel's correction (divides by G - 1), as in nanstd. With binary rewards and k correct of G: mean = k/G, var = k(G - k) / (G(G - 1)).

DeepSeekMath introduces the group-relative baseline as the replacement for PPO's critic; TRL is the reference implementation this figure follows.

Ctraining dynamics

Only mixed groups carry signal, and it is largest when the model is right about half the time

Figure 3 is one group at one step. Over many steps each prompt's pass rate moves, and with it the size of the update that prompt can contribute. For a binary reward the group std is sqrt(k(G - k) / (G(G - 1))): largest at k = G/2, zero at k = 0 or k = G. So a prompt contributes most while the model gets it right about half the time, and nothing once it is always right or always wrong. Three prompts are simulated, each with a toy one-parameter policy: a single logit (the unbounded number a sigmoid turns into a probability p), updated with lr = 0.03. They start at different difficulties: easy at 60%, medium at 20%, hard at 2%. Each step draws G samples; the update is lr times the summed advantage-weighted score function (the gradient of the log-probability of the sampled outcome).

To watch the three prompts train, drag the step slider or press play. At G = 8 the easy prompt passes 90% by step 23 and then drifts into the zero-signal corner: at a pass rate of 0.96, 72% of its groups come back all-correct. The medium prompt crosses 50% at step 15 and climbs fastest. The hard prompt barely moves for about a hundred steps: at p = 0.02 only 15% of groups contain a correct answer, the other 85% add nothing. A mixed group's step is not small (a k = 1 group moves the logit 66% as far as a k = 4 group), but because mixed groups are rare the expected update is only about a tenth of the peak. It crosses 50% only at step 127.

training step t60

group size G8

Figure 4 · Read it: left, each prompt's pass rate by step, solid up to the cursor and faint beyond it; p is the policy's true probability of a correct answer, not the sampled k/G. Right, the expected update size as a fraction of its peak, against pass rate. The solid curve is TRL's default (divide by the group std, scale_rewards="group"); the dashed curve is the same quantity without the std division (scale_rewards="none"). Both peak at p = 0.5 and are 0 at the ends; at G = 8 the solid curve is still at 25% of its peak at p = 0.05 or 0.95 while the dashed one is at 19%. Flip point: p = 0.5. Dividing by the std flattens the curve, so too-easy and too-hard prompts get relatively more weight; Dr. GRPO calls this a question-level difficulty bias. Watch the easy prompt's dot slide down the right side into the zero-signal corner while the hard prompt waits in the left one. Held fixed: lr = 0.03 on the logit, binary reward, seeds 7 to 9 (one per prompt), one independent policy per prompt; the trajectories use the std-normalised update.

Keep the batch in the middle band: drop or resample prompts whose group comes back all-correct or all-wrong.

TRL logs the share of zero-std groups per step as frac_reward_zero_std (with group scaling; under batch scaling it flags the batch std); if it climbs, the batch is saturating.
DAPO's dynamic sampling over-samples and filters out prompts with accuracy 0 or 1 before the update; over a run "the number of samples with accuracy equal to 1 continues to increase", the filter is the last and largest step of its ablation (42 to 50 on AIME 2024), and "convergence time is even reduced". A curriculum that keeps pass rates near 0.5 does the same job up front.
Dr. GRPO shows the std division up-weights too-easy and too-hard questions: "Questions with lower standard deviations (e.g., those that are too easy or too hard ...) are given higher weights during policy updates". TRL's scale_rewards="none" removes it; the zero-signal ends do not change.
Changes if rewards are continuous: the std is rarely exactly 0, but low-variance groups still give small updates, so the band rule holds in softer form.

Simplified: a Bernoulli policy per prompt is not a language model. In a real run prompts share parameters, so progress on easy prompts moves hard ones too, and clipping, KL, token-level credit and response length all shape the curve. The zero-signal ends are exact in every GRPO variant: identical rewards give zero advantages and so zero policy-gradient signal; a KL term, if present, still contributes a gradient. Where the peak sits depends on the reward distribution and the normalisation (the solid and dashed curves differ for that reason).

Go deeper

For a Bernoulli policy with logit θ, the score function is ∂ log π(r) / ∂θ = r - p. With group-relative advantages A_i = (r_i - m) / (s + ε):

Σ_i A_i (r_i - p) = Σ_i (r_i - m)(r_i - p) / (s + ε) = (G - 1) s² / (s + ε) = (k(G - k) / G) / (s + ε)

which is never negative (the update always pushes toward the correct answer) and is 0 exactly when k = 0 or k = G. The solid right-hand curve is its expectation over k ~ Binomial(G, p). Without the std division the sum is k(G - k)/G, whose expectation is (G - 1) p (1 - p): the dashed curve. Both are drawn as a fraction of their own peak at p = 0.5 so the shapes can be compared. TRL's per-token loss normalisation (loss_type="dapo", the default) divides by the number of tokens in the batch, a constant 1/G in this one-token toy, and is absorbed into lr. For G from 2 to 16 both expectations peak at exactly p = 0.5 (checked numerically; experiments/toy_training_dynamics.js). Simulation and curves: experiments/toy_training_dynamics.js in the explainer folder.

Dclipping and the ratio

On a fully on-policy step the ratio is exactly 1, so clipping never binds; it starts to matter only when rollouts are reused or generation is off-policy (sampled by weights that lag the ones being updated)

GRPO keeps PPO's clipped surrogate (the objective PPO maximises in place of the true return). For each token the objective is min(r·Â, clip(r, 1 - ε_low, 1 + ε_high)·Â), where the ratio r = π_θ / π_old compares the probability of the sampled token under the policy being updated with its probability under the policy that generated it. The clip removes the incentive to push r outside the band: once r passes 1 + ε_high with Â > 0, or drops below 1 - ε_low with Â < 0, the objective goes flat and that token contributes no gradient. Only one side binds per token, decided by the sign of Â; TRL counts a token as low-clipped only when Â < 0 and high-clipped only when Â > 0.

When the policy takes a single update per batch of rollouts, π_old is π_θ, r = 1 for every token, and nothing is clipped. DeepSeekMath trained that way ("The policy model only has a single update following each exploration stage", section 4.2) and, in its appendix analysis under that stated assumption, notes that "we can remove the min and clip operation" (A.1.5); TRL's default num_iterations = 1 does the same, the old log-probabilities being the current ones with the gradient detached. Clipping matters when rollouts are reused for several updates (DAPO: 16 gradient updates per rollout) or generation runs off-policy. Drag r away from 1 and watch which line goes flat; switch to clip-higher and the Â > 0 line keeps rising until 1.28.

ratio r = π_θ / π_old1.00

Figure 5 · Read it: the horizontal axis is the ratio r; the vertical axis is the per-token objective for |Â| = 1. At r = 1 both lines pass through ±1 with unit slope (+1 for Â > 0, -1 for Â < 0): the plain policy gradient, which is all GRPO ever computes on a one-update step. Flip point: r = 1 on a one-update step (in TRL: num_iterations = 1 with aligned accumulation and no vLLM importance-sampling correction). Moving right, the Â > 0 line goes flat at 1 + ε_high (shaded: slope 0, no gradient); moving left, the Â < 0 line goes flat at 1 - ε_low. With symmetric ε = 0.2 a token whose probability must rise is capped at r = 1.2. DAPO notes that this lets a 0.9-probability token reach 1.08 but a 0.01 token only 0.012; clip-higher lifts the cap to 1.28 and leaves the Â < 0 side at 0.8. Held fixed: |Â| = 1, one token, token-level ratio; GSPO draws the same picture for a sequence-level ratio with a band about 700 times tighter (3e-4 against 0.2).

One update per rollout: ε does not matter. Reusing rollouts or generating off-policy: decouple ε_high and watch the clip ratio.

With num_iterations = 1 (TRL default), accumulation aligned to steps_per_generation, and the vLLM importance-sampling path off, TRL reuses the detached current log-probs and the ratio is exactly 1; otherwise it recomputes the old log-probs, and the ratio is 1 only up to numerics or, with misaligned accumulation, genuinely off-policy. RLOO measured PPO's clip active "<5% of the time per batch".
If you reuse rollouts, DAPO's ε_low = 0.2, ε_high = 0.28 keeps low-probability tokens explorable and counters entropy collapse (the policy's output distribution narrowing quickly; DAPO: "the entropy of the policy decreases quickly as training progresses"); TRL: epsilon_high=0.28, and read clip_ratio/low_mean, high_mean, region_mean.
If token-level ratios are noisy (long sequences; mixture-of-experts models whose expert routing shifts between old and new policy), GSPO clips a sequence-level, length-normalised ratio (π_θ(y|x)/π_old(y|x))^1/|y| at 3e-4 / 4e-4; TRL: importance_sampling_level="sequence" with loss_type="grpo".

Simplified: |Â| is fixed at 1 and one token is shown. In a run every token of a completion shares that completion's Â (Figure 3), the loss is aggregated across tokens and completions (loss_type), an optional KL term is added, and TRL can apply an importance-sampling correction for the vLLM/trainer mismatch. None of that changes where the band sits or that r = 1 on a one-update step.

Go deeper

For Â > 0: d/dr min(r, clip(r)) is 1 for r < 1 + ε_high and 0 beyond. For Â < 0: d/dr of -max(r, clip(r)) is -1 for r > 1 - ε_low and 0 below. TRL computes coef_1 = exp(log π_θ - log π_old), coef_2 = clamp(coef_1, 1 - ε_low, 1 + ε_high), loss = -min(coef_1·A, coef_2·A) (lines 3184, 3201-3209). When π_old = π_θ the value of coef_1 is 1 and its gradient is ∇log π_θ, so the update is Â ∇log π_θ: the REINFORCE direction with the group-relative advantage.

Elength and the loss

Averaging each completion's loss over its own tokens favours short correct answers and long wrong ones; a token-level or constant denominator removes that

After the advantage (Figure 3) and the ratio (Figure 5), the per-token terms have to be added up. DeepSeekMath's objective averages each completion over its own tokens and then over the group: (1/G) Σ_i (1/|o_i|) Σ_t. That gives every completion the same total weight, 1/G, whatever its length, so each token of a short completion carries more weight than each token of a long one. Dr. GRPO names the consequence: "For positive advantages ... this bias results in greater gradient updates for shorter responses, leading the policy to favor brevity in correct answers. Conversely, for negative advantages ... longer responses are penalized less due to their larger |o_i|, causing the policy to prefer lengthier responses among incorrect ones".

The fix is a denominator that does not depend on the completion's own length. DAPO divides by the total number of tokens in the batch, 1/Σ_i|o_i| Σ_i Σ_t; Dr. GRPO divides by a constant, the generation budget. TRL ships all three: loss_type="grpo" (per-sequence mean, "Not recommended due to length bias"), "dapo" (the default, sum over active tokens in the batch) and "dr_grpo" (sum over batch size × max_completion_length).

The figure shows two completions in one group: one correct, one wrong. To see the length bias, set the wrong one long and the correct one short, then switch the denominator between the three loss types and watch the per-token heights. To make the bias vanish, drag the two lengths equal: under grpo the per-token weights then match.

correct answer, tokens300

wrong answer, tokens700

Figure 6 · Read it: width is length in tokens, height is the weight a token gets in the loss before it is multiplied by the advantage (±0.71 for both completions at G = 2), area is the completion's total weight. Under grpo the two areas are always equal, so the shorter completion's tokens are taller: at 300 vs 700 tokens each correct token weighs 2.33 times each wrong token. Under dapo every token in the batch has the same height, 1/Σ|o_i|, so the long wrong answer's total weight grows with its length. Under dr_grpo the height is a constant 1/(G·L_max), here with L_max = 1024, so totals grow with length and no longer depend on what else is in the batch. Flip point: equal lengths, where the per-token ratio under grpo is 1.00. Held fixed: G = 2, equal |Â|, one group in the batch, L_max = 1024.

Do not train long chains of thought with loss_type="grpo". Keep TRL's default dapo, or use dr_grpo with scale_rewards="none" to match the Dr. GRPO paper.

TRL marks grpo "Not recommended due to length bias" and defaults to dapo.
DAPO's token-level loss: sample-level averaging "leads to an unhealthy increase in entropy" and under-penalises long bad samples; in the ablation it moves AIME 2024 from 41 to 42 before dynamic sampling takes it to 50.
Dr. GRPO reports that its fix "prevents the response length from growing wildly" and that "the length of incorrect responses is substantially reduced".
bnpo divides by the active tokens in the local micro-batch, so "results may slightly vary depending on the local batch size"; in this one-group figure it coincides with dapo.

Simplified: two completions and one group. In a run the dapo denominator is the token count of the whole generation batch and the advantages differ per completion; the weight shown here is the aggregation factor only. Dr. GRPO's claim about response length is about a full training run, which this figure does not simulate.

Go deeper

TRL's aggregation (trainer lines 3241-3273): grpo: ((per_token_loss * mask).sum(-1) / mask.sum(-1).clamp(min=1.0)).mean(); bnpo: (per_token_loss * mask).sum() / mask.sum().clamp(min=1.0); dr_grpo: (per_token_loss * mask).sum() / (per_token_loss.size(0) * self.max_completion_length); dapo: the sum over the generation batch's active tokens, rescaled per accumulation window (num_items_in_batch). With two completions of lengths L₁, L₂: per-token weights 1/(2L_i), 1/(L₁ + L₂), 1/(2·1024); totals 1/2, L_i/(L₁ + L₂), L_i/2048.

Decision table

Study	Situation	Call	What would change it
A	Budgeting models in memory for RL post-training	Count models: PPO trains policy and value and holds reference and reward; GRPO trains the policy and, at TRL defaults with rule rewards, holds nothing else	A learned reward model or beta > 0 adds frozen models to either method; no checked source gives a percentage
B	Choosing prompts and group size for GRPO	Sample groups where the model sometimes succeeds; a group that is all right or all wrong teaches nothing	If rewards are continuous or scale_rewards is 'batch' or 'none', magnitudes change but the sign rule holds
C	Choosing which prompts stay in the batch as training progresses	Keep the batch in the middle band: drop or resample prompts whose group comes back all-correct or all-wrong	With continuous rewards the std is rarely exactly 0 and the rule holds in softer form; scale_rewards='none' removes the std flattening but not the zero-signal ends
D	Choosing epsilon, epsilon_high and num_iterations	One update per rollout: epsilon does not matter. Reusing rollouts or generating off-policy: decouple epsilon_high (DAPO 0.28) and watch clip_ratio	Token-level ratios on long sequences or MoE routing push toward the sequence-level (GSPO) variant
E	Choosing loss_type for long chain-of-thought training	Do not use loss_type='grpo' for long chain-of-thought training; keep TRL's default 'dapo', or 'dr_grpo' with scale_rewards='none' to match the paper	bnpo coincides with dapo at one group but depends on the local batch size

Decisions

Four decisions an engineer makes around GRPO: the call first, the evidence table under it. Every cell carries its source; where no checked source supports a number, the cell says so. Quotes are verbatim from the papers or the TRL source at commit 6d484ba (2026-08-22).

1 · SFT or RL (GRPO with verifiable rewards)

Call: demonstrations and no verifier: SFT, or distillation from a stronger model (R1's distilled 32B scores 72.6 on AIME 2024 where RL on the same base reached 47.0). A verifier and a model that already sometimes gets it right: RL sharpens it ("RL enhances Maj@K's performance but not Pass@K"). Run a small cold-start SFT first; R1 reports "better performance against DeepSeek-R1-Zero" with it.

Evidence table

Question	SFT	RL with GRPO	Source
Training signal	Imitation; the gradient coefficient is "always set to 1"	A per-sample coefficient set by the advantage	DeepSeekMath App. A.1.1 and eq. 21
Data needed	Demonstrations (q, o)	Questions plus a reward function; outputs sampled from the live policy	DeepSeekMath Table 10
Penalises wrong answers	No	Yes, through negative advantages; GRPO "uniquely adjusts its gradient coefficient based on the reward value"	DeepSeekMath 5.2.1
What RL improves	n/a	"RL enhances Maj@K's performance but not Pass@K" (majority vote over K samples vs any-correct over K samples)	DeepSeekMath 5.2.2
RL with no SFT at all	n/a	Works (R1-Zero, "without any supervised data") but "struggles with challenges like poor readability, and language mixing"	R1 2.2, 2.2.4
Why a cold start	"thousands of cold-start data" of long CoT before RL	R1 reports "better performance against DeepSeek-R1-Zero" with it	R1 2.3.1
Small models	Distillation from a stronger model: Distill-Qwen-32B 72.6 AIME 2024	RL on Qwen-32B-Base for 10K+ steps: 47.0; RL on small models "may not even achieve the performance of distillation"	R1 Table 6, 4.1
Verifier	None	A reliable checker; neural reward models "may suffer from reward hacking"	R1 2.2.2
Compute per example	One forward and backward	G rollouts per prompt (DeepSeekMath 64, TRL default 8) plus the policy pass; a reference model only when beta is not 0	DeepSeekMath 4.2; TRL config L477, L674-680
Online vs offline	Offline (SFT, RFT = rejection-sampling fine-tuning, DPO)	Online; "Online RFT significantly outperforms RFT on two benchmarks"	DeepSeekMath 5.2.1, Fig. 5

2 · GRPO, PPO, DPO, RLOO, REINFORCE++

Call: offline preference pairs and no budget for online sampling: DPO. A verifier or reward model and online RL without a critic: GRPO, or RLOO / REINFORCE++, which differ in the baseline (leave-one-out mean, batch mean) and in where the std is taken. PPO stays the reference when a critic is affordable. The one checked head-to-head number, REINFORCE++'s Table 4 (GRPO 22.58 vs PPO 21.85 on one 7B setup), is too narrow to settle the choice.

Evidence table

Dimension	PPO	GRPO	DPO	RLOO	REINFORCE++	Source
Value model	Trained alongside the policy	None: "obviates the need for additional value function approximation"	None	None	None	DeepSeekMath 4.1.1; RLOO
Models held	Policy, reference, reward, value	Policy, reference, reward (no reference when beta = 0)	Policy, reference	Policy, reference, reward	Policy, reference, reward	DeepSeekMath Fig. 4; TRL config L677
Memory claim, in the source's words	baseline	"less memory consumption"; "significantly reducing training resources"	"computationally lightweight"	"one less model copy"	not stated	DeepSeekMath 6, Fig. 4; DPO abstract; RLOO. No checked source gives a percentage.
Baseline	Learned V(s) with GAE (generalised advantage estimation)	Mean of the G samples	None (pairwise logit)	Leave-one-out mean of the other k - 1	Batch mean and std	PPO eq. 11-12; DeepSeekMath 4.1.1; RLOO 2.3; REINFORCE++ eq. 5
Preference pairs	No	No	Yes (o+, o-)	No	No	DeepSeekMath Table 10; DPO abstract
On-policy sampling	Yes	Yes	No ("eliminating the need for sampling from the LM during fine-tuning")	Yes	Yes	DPO abstract; DeepSeekMath Table 10
Clipping	Yes, "say, ε = 0.2" (the paper writes ϵ)	Yes	No	No; PPO's clip active "<5% of the time per batch"	PPO-style clip	PPO eq. 7; DeepSeekMath eq. 3; RLOO 3.2; REINFORCE++
KL placement	In the per-token reward	In the loss (the k3 estimator, "guaranteed to be positive")	Implicit through beta in the logit	In the reward	In the per-token reward (k1); its "w/ Baseline" variant uses a separate k2 KL loss	DeepSeekMath eq. 2 vs 3-4; REINFORCE++ eq. 4, 3.2
Reported comparisons	DeepSeekMath Fig. 5: GRPO above Online RFT above RFT. RLOO (k = 4) beats PPO by "10.3, 14.5, and 32.1" win-rate points on its three setups. REINFORCE++ Table 4 (Qwen2.5-7B-Base): REINFORCE++ with baseline 24.10 average accuracy, GRPO 22.58, PPO 21.85; its Figure 4 shows GRPO overfitting a 30-prompt set. That Table 4 is the only checked head-to-head GRPO vs PPO number.

3 · GRPO and its variants (TRL <code>loss_type</code> and friends)

Call: start from TRL's defaults (loss_type="dapo", scale_rewards="group", beta=0). Add epsilon_high=0.28 and mask_truncated_completions=True if you reuse rollouts or train long outputs (DAPO). Use dr_grpo + scale_rewards="none" to reproduce Dr. GRPO; importance_sampling_level="sequence" for GSPO on long sequences or MoE. Do not pick loss_type="grpo" for long CoT (Figure 6).

Evidence table

Variant	What changes	Failure it fixes	Reported result	TRL	Source
GRPO (paper)	A = (r - mean)/std; loss (1/G) Σ_i (1/\|o_i\|) Σ_t; symmetric ε; KL in the loss	baseline	DeepSeekMath-RL 7B: 88.2 GSM8K, 51.7 MATH	`loss_type="grpo"`, `scale_rewards="group"`; docstring: "Not recommended due to length bias"	DeepSeekMath eq. 3, Table 5; TRL config L796-801
Dr. GRPO	Drop 1/\|o_i\| (divide by a constant) and drop the std	Length bias (Figure 6) and difficulty bias (Figure 4)	43.3% AIME 2024 from a 7B base, "27 hours compute on 8×A100 GPUs"	`loss_type="dr_grpo"` + `scale_rewards="none"`	Dr. GRPO 3; TRL L3252-3256, config L789-792
DAPO	ε_low 0.2 / ε_high 0.28; dynamic sampling; token-level loss; overlong shaping; no KL	Entropy collapse; zero-gradient prompts; long bad samples under-penalised; truncation noise	50 AIME 2024 on Qwen2.5-32B at 50% of R1-Zero-Qwen-32B's steps; ablation 30, 36, 38, 41, 42, 50	`loss_type="dapo"` (default), `epsilon_high=0.28`, `mask_truncated_completions=True`; dynamic sampling is not a flag (watch `frac_reward_zero_std`)	DAPO 3.1-3.4, Table 1; TRL L3257-3263, config L698-706, L829-836
BNPO	Token-level loss over the local micro-batch	Same target as DAPO's loss, local scope	none reported in the checked sources	`loss_type="bnpo"`; "results may slightly vary depending on the local batch size"	TRL config L807-810, trainer L3247-3251
GSPO	Sequence-level ratio (π/π_old)^1/\|y\|, clipped at 3e-4 / 4e-4	Token-ratio noise over long sequences; MoE routing instability	"higher training efficiency than GRPO" on Qwen3-30B-A3B while clipping more tokens	`importance_sampling_level="sequence"` with `loss_type="grpo"`	GSPO 2-4; TRL L3173-3177, L916-922
CISPO, SAPO, LUSPO, VESPO	Present in TRL as further `loss_type` values; their papers were not among the sources checked for this page, so nothing is claimed about them here			`loss_type=` cispo / sapo / luspo / vespo	TRL config L811-826

4 · Hyperparameters: what the sources used, what TRL defaults to

Call: the sources agree on lr 1e-6 (DeepSeekMath, DAPO, TRL); they disagree on everything else, none states a training temperature, and none ablates G. G: 8 (TRL) to 64 (DeepSeekMath). beta: 0.04 with a learned reward model, 0 with rule rewards (DAPO, TRL default). Clipping and μ only matter together (Figure 5). R1 reports none of these, and a beta of 0.001 sometimes attributed to R1 is not in the paper.

Evidence table

Knob	DeepSeekMath	DAPO	GSPO	TRL default	Note
Group size G	64	16	not stated	8 (config L477)	No ablation on G in the checked sources
beta (KL)	0.04	removed	not stated	0.0; no reference model loaded (L674-680)	DeepSeekMath's 0.04 went with a learned reward model
ε / ε_high	ε, value not stated	0.2 / 0.28	3e-4 / 4e-4 (its GRPO baseline 0.2 / 0.27)	0.2 / None (L686-706)	Only matters off-policy (Figure 5)
Temperature	not stated for RL	not stated for training (evaluation 1.0, top-p 0.7)	not stated	1.0 (L525-526)	R1 evaluates at 0.6, top-p 0.95; no checked source states or ablates a training temperature
Max completion length	1024	16,384 + 4,096 soft cache	not stated	512 (L491-492)	Caps what RL can learn; R1's lengths grow toward ~10k tokens
num_iterations (μ)	1	16 updates per rollout	4 mini-batch updates per rollout	1 (L682-685)	μ = 1 means r = 1 and nothing clipped (Figure 5)
scale_rewards	std within group	std within group	std within group	"group" (L782-793)	Dr. GRPO recommends none (Figure 4)
loss_type	sequence mean, then group mean	token-level	sequence-level ratio, grpo aggregation	"dapo" (L794-801)	Figure 6
Learning rate	1e-6	1e-6	not stated	1e-6 (L421)

References

Hugging Face (2026). trl/trainer/grpo_trainer.py: lines 2787-2810 (advantages), 2852 (frac_reward_zero_std), 3149-3155 (old log-probs = detached current ones when num_iterations = 1), 3184 and 3201-3209 (ratio and clipped loss), 3361-3372 (clip metrics); grpo_config.py: 477-478 (num_generations 8), 682-706 (num_iterations 1, epsilon 0.2, epsilon_high None, 'Paper DAPO recommends 0.28'), 753-762 (importance_sampling_level), 782-793 (scale_rewards); main at commit 6d484ba, accessed 2026-08-22 · code
Hugging Face (2026). trl/trainer/utils.py, nanstd (Bessel-corrected), main branch, accessed 2026-08-22 · code
Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (section 4.1: GRPO objective, 'GRPO foregoes the critic model'; section 4.2: 'For each question, we sample 64 outputs') · paper
Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale (section 3.2, Dynamic Sampling: 'over-sample and filter out prompts with the accuracy equal to 1 and 0') · paper
Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective (section 3.1: the std term gives too-easy and too-hard questions higher weight; Dr. GRPO removes it) · paper
DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (section 2.2.2: rule-based accuracy and format rewards) · paper
Schulman et al. (2017). Proximal Policy Optimization Algorithms (section 3, eq. 7: clipped surrogate; 'say, eps = 0.2') · paper
Zheng et al. (2025). Group Sequence Policy Optimization (sequence-level importance ratio; clip ranges 3e-4 / 4e-4 vs 0.2 / 0.27 for the GRPO baseline) · paper
Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (section 2.3: leave-one-out baseline; section 3.2: PPO clipping active '<5% of the time per batch'; section 5: one less model copy, win-rate gains) · paper
Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model (abstract: 'computationally lightweight', 'eliminating the need for sampling from the LM during fine-tuning') · paper
Lambert et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training (section 7: RLVR with PPO and a value model; a constant reward when a completion is verifiably correct) · paper
Hu (2025). REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models (eq. 4-5: KL in the per-token reward, batch normalisation; Figure 3: GRPO overfitting a 30-prompt set; Table 4: 24.10 / 22.58 / 21.85) · paper

Revisions and earlier versions

Earlier versions are kept in the repository history. Revisions are cut after substantial changes only.

r1 (draft, unreleased) · 2026-08-22

all · added First release (in preparation): start-here primer, Study B (group-relative advantage), Study C (training dynamics), Study D (ratio and clipping), Study E (length and the loss), and five decision tables. Working iterations before release are kept in the record folder (drafts/), not here.