How does Node Problem Detector (NPD) report kernel deadlocks and hardware faults to the K8s control plane?
❓ Interview Prompt: Explain Node Problem Detector DaemonSet, system log monitors, custom problem plugins, and NodeCondition reporting for proactive node replacement.
Node Problem Detector (NPD) is a Kubernetes DaemonSet that monitors node health at the system level and reports problems as NodeConditions or Events to the Kubernetes API server — enabling automated remediation.
What NPD Monitors:
INFO: task ... blocked for more than 120 seconds)./var/log/kern.log.Architecture:
NPD runs as a DaemonSet with elevated privileges to read /dev/kmsg, /var/log/, and systemd journal.
It has two types of monitors:
NodeCondition Reporting:
NPD writes custom NodeConditions (e.g., KernelDeadlock=True) to the Node object. Cluster Autoscaler and tools like Node Auto Provisioner can use these conditions to automatically terminate and replace unhealthy nodes.
Code Implementation Example
# NPD DaemonSet configuration (simplified)
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-problem-detector
namespace: kube-system
spec:
template:
spec:
hostPID: true # access host process IDs
hostNetwork: true # access host network
containers:
- name: node-problem-detector
image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14
command:
- /node-problem-detector
- --logtostderr
- --system-log-monitors=/config/kernel-monitor.json
- --custom-plugin-monitors=/config/custom-plugin.json
securityContext:
privileged: true # required to read kernel logs
volumeMounts:
- mountPath: /var/log
name: log
- mountPath: /dev/kmsg
name: kmsg
volumes:
- name: log
hostPath:
path: /var/log
- name: kmsg
hostPath:
path: /dev/kmsg
# Custom plugin: check disk I/O latency
# /config/custom-disk-check.sh:
# #!/bin/bash
# LATENCY=$(iostat -x 1 1 | awk '/sda/ {print $14}')
# if (( $(echo "$LATENCY > 100" | bc -l) )); then exit 1; fi # >100ms = failNode Problem Detector converts kernel-level failures (deadlocks, OOM, hardware errors) into Kubernetes NodeConditions — enabling automated node termination and replacement via cluster autoscalers.