Intermediate LevelAI, LLMs, Embeddings & MCP
Compare Tensor Parallelism vs Pipeline Parallelism for multi-GPU LLM training/inference.
❓ Interview Prompt: Explain multi-GPU scaling strategies: Tensor Parallelism (intra-layer matrix slicing) vs Pipeline Parallelism (inter-layer stage partitioning).
When a model exceeds the memory of a single GPU, distributed parallelism splits the model across multiple GPUs:
1. Tensor Parallelism (TP - Megatron-LM):
Splits individual weight matrices within a layer across GPUs.
•
Requires high-speed GPU-to-GPU interconnects (NVLink) due to frequent communication per layer.
•
Best for: Single-node multi-GPU clusters (e.g. 8x H100s).
2. Pipeline Parallelism (PP):
Splits sequential layers across GPUs (e.g. Layers 1-20 on GPU 1, Layers 21-40 on GPU 2).
•
Uses micro-batches to minimize GPU pipeline bubble idle time.
•
Best for: Multi-node clusters connected over Ethernet/InfiniBand.
Code Implementation Example
typescript
// Parallelism allocation decision
function selectParallelism(numNodes: number, gpusPerNode: number) {
return {
tensorParallelSize: gpusPerNode, // e.g. 8 (intra-node NVLink)
pipelineParallelSize: numNodes // e.g. 4 (inter-node network)
};
}💡Distributed multi-GPU parallelism configuration strategy.
🎯 Interview Key Takeaway
Tensor Parallelism splits weight matrices across GPUs via NVLink; Pipeline Parallelism partitions sequential model layers across nodes.
#Tensor Parallelism#Pipeline Parallelism#Distributed#GPUs#Intermediate
Back to all AI, LLMs, Embeddings & MCP Questions→