Senior LevelAI, LLMs, Embeddings & MCP
How does DeepSpeed ZeRO-3 (Zero Redundancy Optimizer) partition model states across thousands of GPUs?
❓ Interview Prompt: Explain ZeRO-Stage 1, Stage 2, and Stage 3 partitioning of optimizer states, gradients, and model parameters.
When training massive models (100B+ parameters), standard data parallelism duplicates all model states across every GPU, causing Out-Of-Memory (OOM) crashes. DeepSpeed ZeRO (Zero Redundancy Optimizer) eliminates state duplication.
ZeRO Partitioning Stages:
1
ZeRO-Stage 1: Partitions Optimizer States (Adam FP32 momentum/variance) across GPUs (saves 4x memory).
2
ZeRO-Stage 2: Partitions Gradients across GPUs (saves additional memory).
3
ZeRO-Stage 3: Partitions Model Parameters across GPUs! Each GPU holds only $1/N$ of the model weights, gathering parameters on-demand during forward/backward passes.
Impact:
Enables training models with hundreds of billions of parameters across clusters without specialized tensor-parallel code modifications.
Code Implementation Example
json
{
"zero_optimization": {
"stage": 3,
"offload_optimizer": { "device": "cpu", "pin_memory": true },
"offload_param": { "device": "cpu", "pin_memory": true },
"overlap_comm": true,
"allreduce_bucket_size": 5e8
}
}💡DeepSpeed ZeRO-Stage 3 JSON configuration with CPU memory offloading.
🎯 Interview Key Takeaway
DeepSpeed ZeRO-3 partitions optimizer states, gradients, and model parameters across GPUs to eliminate memory duplication.
#DeepSpeed#ZeRO-3#Distributed#GPU Training#Senior
Back to all AI, LLMs, Embeddings & MCP Questions→