How GRPO turns a handful of sampled answers into a training signal without a value network: a primer, four interactive studies and the decisions they settle
GRPO trains a model by sampling several answers to the same question, scoring them, and pushing probability toward the answers that scored above their own group's average
GRPO is a reinforcement-learning step, usually run after pretraining and supervised fine-tuning (DeepSeek-R1-Zero skips the fine-tuning stage). For one prompt q, the current policy samples a group of G completions o1..oG. (The sampling policy is written πold; on a single-update step it is the same model as πθ, the one being updated. Study D covers when they differ.) Each completion gets a reward ri. Here it is 1 if the final answer is correct and 0 otherwise: a rule-based accuracy reward of the kind DeepSeek-R1 uses ("evaluates whether the response is correct", with answers "in a specified format (e.g., within a box), enabling reliable rule-based verification"). Each completion then gets an advantage Âi, how much better or worse it did than expected, and the update raises the probability of completions with positive advantage and lowers it for negative ones.
The "expected" part is the baseline. Subtracting a baseline from the reward leaves the expected direction of the update unchanged and changes only its noise (derivation under Go deeper); DeepSeekMath describes PPO's value baseline as being there "for variance reduction". PPO learns the baseline with a separate value model (its critic), "typically another model of comparable size as the policy model". GRPO "foregoes the critic model, instead estimating the baseline from group scores". Study B (Figure 3) shows what that group baseline does to each answer; Study C (Figure 4) follows it over a training run.
Figure 1 · Read it: stage 4 is what GRPO changes. The two correct answers sit at +1.29 and the four wrong at -0.65 because each is measured against its own group's mean (1/3) and divided by the group std (0.52); Figure 3 lets you move these. Every token of a completion gets that one scalar. Held fixed: G = 6, k = 2, binary reward; clipping and the KL term (a penalty for drifting from the reference model) are left out of the strip.Go deeper: why any baseline is allowed
For a policy πθ and any constant b, the baseline term contributes nothing to the expected gradient:
Eo~π[ b · ∇θ log πθ(o) ] = b · ∇θ Σo πθ(o) = b · ∇θ 1 = 0
so E[(r - b) ∇ log π] = E[r ∇ log π] for every constant b, while the variance of the estimate depends on b. GRPO's baseline is not a constant: the group mean contains the sample's own reward ri, so the expected gradient is scaled by (G - 1)/G (same direction, smaller magnitude; RLOO's leave-one-out mean of the other G - 1 samples avoids this), and the division by the group std is a further sample-dependent rescaling (Study C and Dr. GRPO). In DeepSeekMath's words, the value function in PPO "is treated as a baseline in the calculation of the advantage", and GRPO "uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline". With outcome supervision DeepSeekMath "sets the advantages ... of all tokens in the output as the normalized reward" (section 4.1.2).
GRPO drops PPO's value model; with a KL weight of β = 0 and a rule-based reward it trains and holds only the policy
PPO trains a value model alongside the policy to estimate each token's expected return, and holds two frozen models: a reference model for the KL penalty and a reward model for the score. DeepSeekMath's Figure 4 shows that inventory, policy and value trained, reference and reward frozen, and says why GRPO drops the value model: it "is typically another model of comparable size as the policy model" and "brings a substantial memory and computational burden". GRPO's baseline comes from the group instead (Figure 3), so the value model goes. What remains depends on two choices that are not GRPO's: TRL loads no reference model when β = 0, its default ("If 0.0 (default), the reference model is not loaded"), and a rule-based reward needs no reward model at all (DeepSeek-R1-Zero: "We do not apply the outcome or process neural reward model").
To see what each method holds, switch the reward type and β and count the boxes. No checked source gives a memory percentage; the papers say "less memory consumption" and "significantly reducing training resources", and the inventory is the honest form of the claim.
Figure 2 · Read it: boxes are equal because the value model is of comparable size to the policy; solid is trained, dashed is frozen. PPO always holds the value model; with a rule-based reward it drops the reward model too (Tulu 3 ran PPO with verifiable rewards and a value model). GRPO at TRL's defaults with a rule reward is one box. Flip point: none; this is a count, not a curve. Held fixed: the inventory of DeepSeekMath Figure 4; optimizer state, activations, KV caches and any separate generation engine are not drawn.
Count models before choosing: GRPO removes the one model besides the policy that PPO has to train; β and the reward type decide the rest for either method.
PPO: policy and value trained, reference and reward held; critic-free methods "require loading one less model copy compared to Vanilla PG and PPO".
GRPO with β = 0 (TRL default) and a rule-based reward: the policy only.
No checked source gives a percentage; quote the words, not a number.
Simplified: a model count is not a memory budget: optimizer state for the trained models, activations, KV caches and a vLLM generation copy all take memory and are not drawn.
Go deeper
DeepSeekMath eq. 2 puts PPO's KL penalty in the per-token reward, which needs πref; GRPO's eq. 3 puts the KL in the loss; with β = 0 TRL skips it and "the reference model is not loaded". The value model's job in PPO, the baseline "for variance reduction", is taken by the group mean (Figure 3).
The group is the baseline: an answer's advantage is how far it sits from its siblings
Figure 3 is stage 4 of Figure 1, with controls. To see how an answer's advantage depends on the group, drag the group-size (G) and correct-count (k) sliders. GRPO samples G completions for one prompt and scores each with a reward; here the reward is 1 for a correct answer and 0 for a wrong one. PPO would compare each reward to a learned value estimate; GRPO drops that value model and compares it to the group mean instead. It then divides by the group's standard deviation, so the advantage says how unusual an answer is within its own group. In TRL the group std is Bessel-corrected (it divides by G - 1, not G) and a 1e-4 floor keeps the division finite.
The sign comes from the mean, the magnitude from the std. To see the extreme case, set G to 8 and k to 1: the lone correct answer earns +2.47 while each wrong one gets -0.35. Then set k to 0 or to G: every reward equals the mean and every advantage collapses to 0.
Figure 3 · Read it: with G = 8 and k = 1 the correct answer sits 7 times further from the mean than each wrong one, so its advantage is 7 times larger in magnitude. Raise k to 4 and the two sides balance at about ±0.94. Flip point: at k = 0 or k = G every reward equals the mean, so every numerator is 0 and all G advantages are 0 (the 1e-4 floor only keeps 0/0 from becoming NaN): the group contributes no policy gradient (a KL term, if one is used, still does). Held fixed: binary rewards, scale_rewards="group" (the TRL default), no KL term.
Sample groups where the model sometimes succeeds.
When a prompt's pass rate is near 0 or 1, drop or re-weight it: almost every group it produces is a zero-signal group.
Larger G makes zero-signal groups rarer: at a pass rate of 0.9 the chance that a group comes back all-correct or all-wrong is 43% at G = 8 and 0.1% at G = 64 (0.9G + 0.1G); generation cost grows linearly with G. DeepSeekMath sampled 64 outputs per question; TRL's num_generations defaults to 8.
Changes if rewards are continuous (magnitudes spread out but the sign rule holds) or if scale_rewards is "batch" or "none" (std taken over the batch, or not divided at all).
Simplified: real reward functions can be continuous, and TRL sums several weighted reward functions; TRL also supports NaN rewards (ignored by nanmean/nanstd) and batch-level scaling. The sign and zero-signal behaviour shown here do not change under those options.
Go deeper
A_i = (r_i - mean(r_1..r_G)) / (std(r_1..r_G) + 1e-4) std uses Bessel's correction (divides by G - 1), as in nanstd. With binary rewards and k correct of G: mean = k/G, var = k(G - k) / (G(G - 1)).
DeepSeekMath introduces the group-relative baseline as the replacement for PPO's critic; TRL is the reference implementation this figure follows.
Only mixed groups carry signal, and it is largest when the model is right about half the time
Figure 3 is one group at one step. Over many steps each prompt's pass rate moves, and with it the size of the update that prompt can contribute. For a binary reward the group std is sqrt(k(G - k) / (G(G - 1))): largest at k = G/2, zero at k = 0 or k = G. So a prompt contributes most while the model gets it right about half the time, and nothing once it is always right or always wrong. Three prompts are simulated, each with a toy one-parameter policy: a single logit (the unbounded number a sigmoid turns into a probability p), updated with lr = 0.03. They start at different difficulties: easy at 60%, medium at 20%, hard at 2%. Each step draws G samples; the update is lr times the summed advantage-weighted score function (the gradient of the log-probability of the sampled outcome).
To watch the three prompts train, drag the step slider or press play. At G = 8 the easy prompt passes 90% by step 23 and then drifts into the zero-signal corner: at a pass rate of 0.96, 72% of its groups come back all-correct. The medium prompt crosses 50% at step 15 and climbs fastest. The hard prompt barely moves for about a hundred steps: at p = 0.02 only 15% of groups contain a correct answer, the other 85% add nothing. A mixed group's step is not small (a k = 1 group moves the logit 66% as far as a k = 4 group), but because mixed groups are rare the expected update is only about a tenth of the peak. It crosses 50% only at step 127.
Figure 4 · Read it: left, each prompt's pass rate by step, solid up to the cursor and faint beyond it; p is the policy's true probability of a correct answer, not the sampled k/G. Right, the expected update size as a fraction of its peak, against pass rate. The solid curve is TRL's default (divide by the group std, scale_rewards="group"); the dashed curve is the same quantity without the std division (scale_rewards="none"). Both peak at p = 0.5 and are 0 at the ends; at G = 8 the solid curve is still at 25% of its peak at p = 0.05 or 0.95 while the dashed one is at 19%. Flip point: p = 0.5. Dividing by the std flattens the curve, so too-easy and too-hard prompts get relatively more weight; Dr. GRPO calls this a question-level difficulty bias. Watch the easy prompt's dot slide down the right side into the zero-signal corner while the hard prompt waits in the left one. Held fixed: lr = 0.03 on the logit, binary reward, seeds 7 to 9 (one per prompt), one independent policy per prompt; the trajectories use the std-normalised update.
Keep the batch in the middle band: drop or resample prompts whose group comes back all-correct or all-wrong.
TRL logs the share of zero-std groups per step as frac_reward_zero_std (with group scaling; under batch scaling it flags the batch std); if it climbs, the batch is saturating.
DAPO's dynamic sampling over-samples and filters out prompts with accuracy 0 or 1 before the update; over a run "the number of samples with accuracy equal to 1 continues to increase", the filter is the last and largest step of its ablation (42 to 50 on AIME 2024), and "convergence time is even reduced". A curriculum that keeps pass rates near 0.5 does the same job up front.
Dr. GRPO shows the std division up-weights too-easy and too-hard questions: "Questions with lower standard deviations (e.g., those that are too easy or too hard ...) are given higher weights during policy updates". TRL's scale_rewards="none" removes it; the zero-signal ends do not change.
Changes if rewards are continuous: the std is rarely exactly 0, but low-variance groups still give small updates, so the band rule holds in softer form.
Simplified: a Bernoulli policy per prompt is not a language model. In a real run prompts share parameters, so progress on easy prompts moves hard ones too, and clipping, KL, token-level credit and response length all shape the curve. The zero-signal ends are exact in every GRPO variant: identical rewards give zero advantages and so zero policy-gradient signal; a KL term, if present, still contributes a gradient. Where the peak sits depends on the reward distribution and the normalisation (the solid and dashed curves differ for that reason).
Go deeper
For a Bernoulli policy with logit θ, the score function is ∂ log π(r) / ∂θ = r - p. With group-relative advantages A_i = (r_i - m) / (s + ε):
which is never negative (the update always pushes toward the correct answer) and is 0 exactly when k = 0 or k = G. The solid right-hand curve is its expectation over k ~ Binomial(G, p). Without the std division the sum is k(G - k)/G, whose expectation is (G - 1) p (1 - p): the dashed curve. Both are drawn as a fraction of their own peak at p = 0.5 so the shapes can be compared. TRL's per-token loss normalisation (loss_type="dapo", the default) divides by the number of tokens in the batch, a constant 1/G in this one-token toy, and is absorbed into lr. For G from 2 to 16 both expectations peak at exactly p = 0.5 (checked numerically; experiments/toy_training_dynamics.js). Simulation and curves: experiments/toy_training_dynamics.js in the explainer folder.
On a fully on-policy step the ratio is exactly 1, so clipping never binds; it starts to matter only when rollouts are reused or generation is off-policy (sampled by weights that lag the ones being updated)
GRPO keeps PPO's clipped surrogate (the objective PPO maximises in place of the true return). For each token the objective is min(r·Â, clip(r, 1 - εlow, 1 + εhigh)·Â), where the ratio r = πθ / πold compares the probability of the sampled token under the policy being updated with its probability under the policy that generated it. The clip removes the incentive to push r outside the band: once r passes 1 + εhigh with  > 0, or drops below 1 - εlow with  < 0, the objective goes flat and that token contributes no gradient. Only one side binds per token, decided by the sign of Â; TRL counts a token as low-clipped only when  < 0 and high-clipped only when  > 0.
When the policy takes a single update per batch of rollouts, πold is πθ, r = 1 for every token, and nothing is clipped. DeepSeekMath trained that way ("The policy model only has a single update following each exploration stage", section 4.2) and, in its appendix analysis under that stated assumption, notes that "we can remove the min and clip operation" (A.1.5); TRL's default num_iterations = 1 does the same, the old log-probabilities being the current ones with the gradient detached. Clipping matters when rollouts are reused for several updates (DAPO: 16 gradient updates per rollout) or generation runs off-policy. Drag r away from 1 and watch which line goes flat; switch to clip-higher and the  > 0 line keeps rising until 1.28.
Figure 5 · Read it: the horizontal axis is the ratio r; the vertical axis is the per-token objective for |Â| = 1. At r = 1 both lines pass through ±1 with unit slope (+1 for  > 0, -1 for  < 0): the plain policy gradient, which is all GRPO ever computes on a one-update step. Flip point: r = 1 on a one-update step (in TRL: num_iterations = 1 with aligned accumulation and no vLLM importance-sampling correction). Moving right, the  > 0 line goes flat at 1 + εhigh (shaded: slope 0, no gradient); moving left, the  < 0 line goes flat at 1 - εlow. With symmetric ε = 0.2 a token whose probability must rise is capped at r = 1.2. DAPO notes that this lets a 0.9-probability token reach 1.08 but a 0.01 token only 0.012; clip-higher lifts the cap to 1.28 and leaves the  < 0 side at 0.8. Held fixed: |Â| = 1, one token, token-level ratio; GSPO draws the same picture for a sequence-level ratio with a band about 700 times tighter (3e-4 against 0.2).
One update per rollout: ε does not matter. Reusing rollouts or generating off-policy: decouple εhigh and watch the clip ratio.
With num_iterations = 1 (TRL default), accumulation aligned to steps_per_generation, and the vLLM importance-sampling path off, TRL reuses the detached current log-probs and the ratio is exactly 1; otherwise it recomputes the old log-probs, and the ratio is 1 only up to numerics or, with misaligned accumulation, genuinely off-policy. RLOO measured PPO's clip active "<5% of the time per batch".
If you reuse rollouts, DAPO's εlow = 0.2, εhigh = 0.28 keeps low-probability tokens explorable and counters entropy collapse (the policy's output distribution narrowing quickly; DAPO: "the entropy of the policy decreases quickly as training progresses"); TRL: epsilon_high=0.28, and read clip_ratio/low_mean, high_mean, region_mean.
If token-level ratios are noisy (long sequences; mixture-of-experts models whose expert routing shifts between old and new policy), GSPO clips a sequence-level, length-normalised ratio (πθ(y|x)/πold(y|x))1/|y| at 3e-4 / 4e-4; TRL: importance_sampling_level="sequence" with loss_type="grpo".
Simplified: |Â| is fixed at 1 and one token is shown. In a run every token of a completion shares that completion's  (Figure 3), the loss is aggregated across tokens and completions (loss_type), an optional KL term is added, and TRL can apply an importance-sampling correction for the vLLM/trainer mismatch. None of that changes where the band sits or that r = 1 on a one-update step.
Go deeper
For  > 0: d/dr min(r, clip(r)) is 1 for r < 1 + εhigh and 0 beyond. For  < 0: d/dr of -max(r, clip(r)) is -1 for r > 1 - εlow and 0 below. TRL computes coef_1 = exp(log πθ - log πold), coef_2 = clamp(coef_1, 1 - εlow, 1 + εhigh), loss = -min(coef_1·A, coef_2·A) (lines 3184, 3201-3209). When πold = πθ the value of coef_1 is 1 and its gradient is ∇log πθ, so the update is  ∇log πθ: the REINFORCE direction with the group-relative advantage.
Averaging each completion's loss over its own tokens favours short correct answers and long wrong ones; a token-level or constant denominator removes that
After the advantage (Figure 3) and the ratio (Figure 5), the per-token terms have to be added up. DeepSeekMath's objective averages each completion over its own tokens and then over the group: (1/G) Σi (1/|oi|) Σt. That gives every completion the same total weight, 1/G, whatever its length, so each token of a short completion carries more weight than each token of a long one. Dr. GRPO names the consequence: "For positive advantages ... this bias results in greater gradient updates for shorter responses, leading the policy to favor brevity in correct answers. Conversely, for negative advantages ... longer responses are penalized less due to their larger |oi|, causing the policy to prefer lengthier responses among incorrect ones".
The fix is a denominator that does not depend on the completion's own length. DAPO divides by the total number of tokens in the batch, 1/Σi|oi| Σi Σt; Dr. GRPO divides by a constant, the generation budget. TRL ships all three: loss_type="grpo" (per-sequence mean, "Not recommended due to length bias"), "dapo" (the default, sum over active tokens in the batch) and "dr_grpo" (sum over batch size × max_completion_length).
The figure shows two completions in one group: one correct, one wrong. To see the length bias, set the wrong one long and the correct one short, then switch the denominator between the three loss types and watch the per-token heights. To make the bias vanish, drag the two lengths equal: under grpo the per-token weights then match.
Figure 6 · Read it: width is length in tokens, height is the weight a token gets in the loss before it is multiplied by the advantage (±0.71 for both completions at G = 2), area is the completion's total weight. Under grpo the two areas are always equal, so the shorter completion's tokens are taller: at 300 vs 700 tokens each correct token weighs 2.33 times each wrong token. Under dapo every token in the batch has the same height, 1/Σ|oi|, so the long wrong answer's total weight grows with its length. Under dr_grpo the height is a constant 1/(G·Lmax), here with Lmax = 1024, so totals grow with length and no longer depend on what else is in the batch. Flip point: equal lengths, where the per-token ratio under grpo is 1.00. Held fixed: G = 2, equal |Â|, one group in the batch, Lmax = 1024.
Do not train long chains of thought with loss_type="grpo". Keep TRL's default dapo, or use dr_grpo with scale_rewards="none" to match the Dr. GRPO paper.
TRL marks grpo "Not recommended due to length bias" and defaults to dapo.
DAPO's token-level loss: sample-level averaging "leads to an unhealthy increase in entropy" and under-penalises long bad samples; in the ablation it moves AIME 2024 from 41 to 42 before dynamic sampling takes it to 50.
Dr. GRPO reports that its fix "prevents the response length from growing wildly" and that "the length of incorrect responses is substantially reduced".
bnpo divides by the active tokens in the local micro-batch, so "results may slightly vary depending on the local batch size"; in this one-group figure it coincides with dapo.
Simplified: two completions and one group. In a run the dapo denominator is the token count of the whole generation batch and the advantages differ per completion; the weight shown here is the aggregation factor only. Dr. GRPO's claim about response length is about a full training run, which this figure does not simulate.
Go deeper
TRL's aggregation (trainer lines 3241-3273): grpo: ((per_token_loss * mask).sum(-1) / mask.sum(-1).clamp(min=1.0)).mean(); bnpo: (per_token_loss * mask).sum() / mask.sum().clamp(min=1.0); dr_grpo: (per_token_loss * mask).sum() / (per_token_loss.size(0) * self.max_completion_length); dapo: the sum over the generation batch's active tokens, rescaled per accumulation window (num_items_in_batch). With two completions of lengths L1, L2: per-token weights 1/(2Li), 1/(L1 + L2), 1/(2·1024); totals 1/2, Li/(L1 + L2), Li/2048.
Decision table
Study
Situation
Call
What would change it
A
Budgeting models in memory for RL post-training
Count models: PPO trains policy and value and holds reference and reward; GRPO trains the policy and, at TRL defaults with rule rewards, holds nothing else
A learned reward model or beta > 0 adds frozen models to either method; no checked source gives a percentage
B
Choosing prompts and group size for GRPO
Sample groups where the model sometimes succeeds; a group that is all right or all wrong teaches nothing
If rewards are continuous or scale_rewards is 'batch' or 'none', magnitudes change but the sign rule holds
C
Choosing which prompts stay in the batch as training progresses
Keep the batch in the middle band: drop or resample prompts whose group comes back all-correct or all-wrong
With continuous rewards the std is rarely exactly 0 and the rule holds in softer form; scale_rewards='none' removes the std flattening but not the zero-signal ends
D
Choosing epsilon, epsilon_high and num_iterations
One update per rollout: epsilon does not matter. Reusing rollouts or generating off-policy: decouple epsilon_high (DAPO 0.28) and watch clip_ratio
Token-level ratios on long sequences or MoE routing push toward the sequence-level (GSPO) variant
E
Choosing loss_type for long chain-of-thought training
Do not use loss_type='grpo' for long chain-of-thought training; keep TRL's default 'dapo', or 'dr_grpo' with scale_rewards='none' to match the paper
bnpo coincides with dapo at one group but depends on the local batch size
Decisions
Four decisions an engineer makes around GRPO: the call first, the evidence table under it. Every cell carries its source; where no checked source supports a number, the cell says so. Quotes are verbatim from the papers or the TRL source at commit 6d484ba (2026-08-22).
1 · SFT or RL (GRPO with verifiable rewards)
Call: demonstrations and no verifier: SFT, or distillation from a stronger model (R1's distilled 32B scores 72.6 on AIME 2024 where RL on the same base reached 47.0). A verifier and a model that already sometimes gets it right: RL sharpens it ("RL enhances Maj@K's performance but not Pass@K"). Run a small cold-start SFT first; R1 reports "better performance against DeepSeek-R1-Zero" with it.
Evidence table
Question
SFT
RL with GRPO
Source
Training signal
Imitation; the gradient coefficient is "always set to 1"
A per-sample coefficient set by the advantage
DeepSeekMath App. A.1.1 and eq. 21
Data needed
Demonstrations (q, o)
Questions plus a reward function; outputs sampled from the live policy
DeepSeekMath Table 10
Penalises wrong answers
No
Yes, through negative advantages; GRPO "uniquely adjusts its gradient coefficient based on the reward value"
DeepSeekMath 5.2.1
What RL improves
n/a
"RL enhances Maj@K's performance but not Pass@K" (majority vote over K samples vs any-correct over K samples)
DeepSeekMath 5.2.2
RL with no SFT at all
n/a
Works (R1-Zero, "without any supervised data") but "struggles with challenges like poor readability, and language mixing"
R1 2.2, 2.2.4
Why a cold start
"thousands of cold-start data" of long CoT before RL
R1 reports "better performance against DeepSeek-R1-Zero" with it
R1 2.3.1
Small models
Distillation from a stronger model: Distill-Qwen-32B 72.6 AIME 2024
RL on Qwen-32B-Base for 10K+ steps: 47.0; RL on small models "may not even achieve the performance of distillation"
R1 Table 6, 4.1
Verifier
None
A reliable checker; neural reward models "may suffer from reward hacking"
R1 2.2.2
Compute per example
One forward and backward
G rollouts per prompt (DeepSeekMath 64, TRL default 8) plus the policy pass; a reference model only when beta is not 0
Online; "Online RFT significantly outperforms RFT on two benchmarks"
DeepSeekMath 5.2.1, Fig. 5
2 · GRPO, PPO, DPO, RLOO, REINFORCE++
Call: offline preference pairs and no budget for online sampling: DPO. A verifier or reward model and online RL without a critic: GRPO, or RLOO / REINFORCE++, which differ in the baseline (leave-one-out mean, batch mean) and in where the std is taken. PPO stays the reference when a critic is affordable. The one checked head-to-head number, REINFORCE++'s Table 4 (GRPO 22.58 vs PPO 21.85 on one 7B setup), is too narrow to settle the choice.
Evidence table
Dimension
PPO
GRPO
DPO
RLOO
REINFORCE++
Source
Value model
Trained alongside the policy
None: "obviates the need for additional value function approximation"
None
None
None
DeepSeekMath 4.1.1; RLOO
Models held
Policy, reference, reward, value
Policy, reference, reward (no reference when beta = 0)
Policy, reference
Policy, reference, reward
Policy, reference, reward
DeepSeekMath Fig. 4; TRL config L677
Memory claim, in the source's words
baseline
"less memory consumption"; "significantly reducing training resources"
"computationally lightweight"
"one less model copy"
not stated
DeepSeekMath 6, Fig. 4; DPO abstract; RLOO. No checked source gives a percentage.
Baseline
Learned V(s) with GAE (generalised advantage estimation)
In the loss (the k3 estimator, "guaranteed to be positive")
Implicit through beta in the logit
In the reward
In the per-token reward (k1); its "w/ Baseline" variant uses a separate k2 KL loss
DeepSeekMath eq. 2 vs 3-4; REINFORCE++ eq. 4, 3.2
Reported comparisons
DeepSeekMath Fig. 5: GRPO above Online RFT above RFT. RLOO (k = 4) beats PPO by "10.3, 14.5, and 32.1" win-rate points on its three setups. REINFORCE++ Table 4 (Qwen2.5-7B-Base): REINFORCE++ with baseline 24.10 average accuracy, GRPO 22.58, PPO 21.85; its Figure 4 shows GRPO overfitting a 30-prompt set. That Table 4 is the only checked head-to-head GRPO vs PPO number.
3 · GRPO and its variants (TRL <code>loss_type</code> and friends)
Call: start from TRL's defaults (loss_type="dapo", scale_rewards="group", beta=0). Add epsilon_high=0.28 and mask_truncated_completions=True if you reuse rollouts or train long outputs (DAPO). Use dr_grpo + scale_rewards="none" to reproduce Dr. GRPO; importance_sampling_level="sequence" for GSPO on long sequences or MoE. Do not pick loss_type="grpo" for long CoT (Figure 6).
Evidence table
Variant
What changes
Failure it fixes
Reported result
TRL
Source
GRPO (paper)
A = (r - mean)/std; loss (1/G) Σi (1/|oi|) Σt; symmetric ε; KL in the loss
baseline
DeepSeekMath-RL 7B: 88.2 GSM8K, 51.7 MATH
loss_type="grpo", scale_rewards="group"; docstring: "Not recommended due to length bias"
DeepSeekMath eq. 3, Table 5; TRL config L796-801
Dr. GRPO
Drop 1/|oi| (divide by a constant) and drop the std
Length bias (Figure 6) and difficulty bias (Figure 4)
43.3% AIME 2024 from a 7B base, "27 hours compute on 8×A100 GPUs"
loss_type="bnpo"; "results may slightly vary depending on the local batch size"
TRL config L807-810, trainer L3247-3251
GSPO
Sequence-level ratio (π/πold)1/|y|, clipped at 3e-4 / 4e-4
Token-ratio noise over long sequences; MoE routing instability
"higher training efficiency than GRPO" on Qwen3-30B-A3B while clipping more tokens
importance_sampling_level="sequence" with loss_type="grpo"
GSPO 2-4; TRL L3173-3177, L916-922
CISPO, SAPO, LUSPO, VESPO
Present in TRL as further loss_type values; their papers were not among the sources checked for this page, so nothing is claimed about them here
loss_type= cispo / sapo / luspo / vespo
TRL config L811-826
4 · Hyperparameters: what the sources used, what TRL defaults to
Call: the sources agree on lr 1e-6 (DeepSeekMath, DAPO, TRL); they disagree on everything else, none states a training temperature, and none ablates G. G: 8 (TRL) to 64 (DeepSeekMath). beta: 0.04 with a learned reward model, 0 with rule rewards (DAPO, TRL default). Clipping and μ only matter together (Figure 5). R1 reports none of these, and a beta of 0.001 sometimes attributed to R1 is not in the paper.
Evidence table
Knob
DeepSeekMath
DAPO
GSPO
TRL default
Note
Group size G
64
16
not stated
8 (config L477)
No ablation on G in the checked sources
beta (KL)
0.04
removed
not stated
0.0; no reference model loaded (L674-680)
DeepSeekMath's 0.04 went with a learned reward model
ε / ε_high
ε, value not stated
0.2 / 0.28
3e-4 / 4e-4 (its GRPO baseline 0.2 / 0.27)
0.2 / None (L686-706)
Only matters off-policy (Figure 5)
Temperature
not stated for RL
not stated for training (evaluation 1.0, top-p 0.7)
not stated
1.0 (L525-526)
R1 evaluates at 0.6, top-p 0.95; no checked source states or ablates a training temperature
Max completion length
1024
16,384 + 4,096 soft cache
not stated
512 (L491-492)
Caps what RL can learn; R1's lengths grow toward ~10k tokens
Earlier versions are kept in the repository history. Revisions are cut after substantial changes only.
r1 (current) · 2026-08-22
all · added First release (in preparation): start-here primer, Study B (group-relative advantage), Study C (training dynamics), Study D (ratio and clipping), Study E (length and the loss), and five decision tables. Working iterations before release are kept in the record folder (drafts/), not here.