Senior LevelAI, LLMs, Embeddings & MCP
How does Group Relative Policy Optimization (GRPO) optimize LLM reasoning without a critic model?
❓ Interview Prompt: Explain Group Relative Policy Optimization (GRPO - DeepSeek R1), baseline elimination, and reward scaling.
Standard PPO reinforcement learning requires training 4 large models simultaneously (Policy, Reference, Reward, and Value/Critic models), consuming enormous GPU memory. Group Relative Policy Optimization (GRPO) eliminates the Critic model.
How GRPO Works (Key Innovation in DeepSeek-R1 / DeepSeek-Math):
1
For each input prompt, the Policy model generates a group of $G$ candidate outputs (e.g. $G = 8$ answers).
2
Evaluates the reward score for each output in the group.
3
Calculates the relative advantage of each output compared to the group mean and standard deviation:
$$A_i = \frac{r_i - \text{mean}(\{r_1..r_G\})}{\text{std}(\{r_1..r_G\})}$$
4
Updates policy weights directly using relative group advantage, cutting memory usage in half!
Code Implementation Example
typescript
function calculateGrpoAdvantages(rewards: number[]): number[] {
const mean = rewards.reduce((a, b) => a + b, 0) / rewards.length;
const variance = rewards.reduce((a, b) => a + Math.pow(b - mean, 2), 0) / rewards.length;
const std = Math.sqrt(variance) || 1e-8; // Prevent div by zero
// Normalize group rewards to compute relative advantages
return rewards.map(r => (r - mean) / std);
}💡GRPO relative advantage score calculation across candidate response group.
🎯 Interview Key Takeaway
GRPO samples a group of outputs per prompt and calculates relative group advantages, eliminating the heavy Critic model in RL training.
#GRPO#DeepSeek#RL#PPO#Reasoning#Senior
Back to all AI, LLMs, Embeddings & MCP Questions→