How does Volcano Batch Scheduler optimize AI/ML training and High-Performance Computing (HPC) jobs on K8s?
❓ Interview Prompt: Explain Volcano gang scheduling, fair-share queuing, GPU topology-aware placement, and PodGroup all-or-nothing allocation for distributed ML training.
The default Kubernetes scheduler is designed for microservices — it schedules pods independently, one at a time. This model breaks for distributed AI/ML training jobs that require all workers to start simultaneously.
The Problem: Distributed Training Deadlock:
A PyTorch DDP job needs 8 GPU workers. The default scheduler may place 7 of 8 pods (holding 7 GPUs) while the 8th waits for a free GPU — deadlock. The 7 running workers hold resources without being able to train. Neither can complete.
Volcano Gang Scheduling:
Gang scheduling (all-or-nothing): A PodGroup (all workers + parameter server) is only scheduled when all pods can start simultaneously. If 8/8 GPUs aren't available, wait — don't hold partial resources.
Volcano Queue System:
GPU Topology-Aware Scheduling:
Modern GPU servers have multiple PCIe root complexes or NVLink bridges — GPU pairs on the same NVLink switch have 600 GB/s bandwidth; across PCIe is 16 GB/s. Volcano places workers on topologically optimal GPU pairs to maximize training throughput.
Code Implementation Example
# Volcano Queue — GPU compute budget for ML team
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
name: ml-team
spec:
weight: 1
capability:
nvidia.com/gpu: "32" # ML team gets 32 GPUs max
reclaimable: true # other teams can reclaim idle GPUs
---
# Volcano Job — PyTorch DDP training with gang scheduling
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
name: pytorch-training
spec:
minAvailable: 8 # GANG: all 8 pods must start together!
queue: ml-team
schedulerName: volcano
tasks:
- replicas: 8 # 8 GPU workers
name: worker
template:
spec:
containers:
- name: pytorch
image: pytorch/pytorch:2.1-cuda12.1
command:
- torchrun
- --nproc_per_node=1
- --nnodes=8
- train.py
resources:
limits:
nvidia.com/gpu: 1 # 1 GPU per worker
restartPolicy: OnFailureVolcano gang scheduling (minAvailable) prevents distributed training deadlock — all pods start together or none start. GPU topology awareness maximizes NVLink bandwidth for inter-GPU communication.