Senior LevelAI, LLMs, Embeddings & MCP
Compare Direct Preference Optimization (DPO) vs Proximal Policy Optimization (PPO) for LLM alignment.
❓ Interview Prompt: Deep dive into mathematical formulations: PPO Actor-Critic reward loss vs DPO analytical implicit reward formulation.
1. PPO (Proximal Policy Optimization - Traditional RLHF):
•
Requires running 4 separate models simultaneously: Policy Model, Reference Model, Reward Model, and Value (Critic) Model.
•
Uses reinforcement learning policy gradients to maximize reward model outputs.
•
Highly unstable training dynamics; susceptible to reward hacking.
2. DPO (Direct Preference Optimization - Modern Standard):
•
Mathematical re-parametrization of the Bradley-Terry preference model.
•
Proves that the optimal policy can be derived directly from preference data without a separate reward model!
•
Objective loss directly increases probability of chosen completion while decreasing probability of rejected completion:
$$\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right]$$
Code Implementation Example
python
# PyTorch DPO Loss Function (Python)
import torch
import torch.nn.functional as F
def dpo_loss(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rejected_logps, beta=0.1):
pi_logratios = policy_chosen_logps - policy_rejected_logps
ref_logratios = ref_chosen_logps - ref_rejected_logps
logits = pi_logratios - ref_logratios
return -F.logsigmoid(beta * logits).mean()💡Direct Preference Optimization (DPO) loss calculation function in PyTorch.
🎯 Interview Key Takeaway
DPO eliminates the need for separate Reward and Critic models, optimizing preference alignment directly on policy loss.
#DPO#PPO#RLHF#Alignment#Senior
Back to all AI, LLMs, Embeddings & MCP Questions→