How do you execute zero-downtime Kubernetes control plane and worker node upgrades?
❓ Interview Prompt: Explain the Kubernetes version skew policy, control plane upgrade sequence, and safe rolling node drain/upgrade procedures with PodDisruptionBudgets.
Kubernetes cluster upgrades require careful sequencing to maintain availability. Kubernetes version skew policy: components can be at most 2 minor versions apart.
Upgrade Order (Mandatory Sequence):
Control Plane Upgrade:
With kubeadm: kubeadm upgrade apply v1.30.0 handles components in correct order. With managed services (EKS, GKE, AKS): use the provider's upgrade API — they handle control plane automatically.
Worker Node Rolling Upgrade:
For each node:
apt upgrade kubeadm kubectl kubelet, then kubeadm upgrade node.PDB Interaction During Drain:
If draining would violate any PDB (e.g., API deployment has minAvailable: 2 but only 2 pods running), drain blocks until pods reschedule elsewhere. This is by design — PDBs prevent zero-downtime guarantees from being violated.
Code Implementation Example
#!/bin/bash
# Zero-downtime rolling node upgrade script
NODES=$(kubectl get nodes -l node-role.kubernetes.io/worker -o name)
for NODE in $NODES; do
echo "=== Upgrading $NODE ==="
# Step 1: Cordon — mark unschedulable
kubectl cordon $NODE
echo " Cordoned: $NODE"
# Step 2: Drain — evict pods (respects PDBs, waits up to 5min)
kubectl drain $NODE \
--ignore-daemonsets \ # don't try to evict DaemonSet pods
--delete-emptydir-data \ # allow deleting emptyDir volumes
--grace-period=60 \ # 60s for graceful shutdown
--timeout=5m # timeout waiting for evictions
echo " Drained: $NODE"
# Step 3: SSH + upgrade kubelet (cloud-init, Ansible, or SSM)
NODE_IP=$(kubectl get $NODE -o jsonpath='{.status.addresses[0].address}')
ssh ubuntu@$NODE_IP \
"sudo apt-get update && sudo apt-get install -y kubelet=1.30.0-1.1 \
&& sudo systemctl daemon-reload && sudo systemctl restart kubelet"
# Step 4: Wait for node to become Ready
kubectl wait --for=condition=Ready $NODE --timeout=5m
# Step 5: Uncordon — allow scheduling again
kubectl uncordon $NODE
echo " Uncordoned: $NODE"
# Wait 30s before upgrading next node (let pods reschedule)
echo " Waiting 30s before next node..."
sleep 30
doneUpgrade control plane first (kube-apiserver → controllers → schedulers), then rolling drain/upgrade worker nodes one-by-one. PDBs block drain if eviction would violate availability guarantees.