How does KEDA (Kubernetes Event-driven Autoscaling) scale workloads based on Kafka queue depth?
❓ Interview Prompt: Explain KEDA ScaledObject, external scalers (Kafka, SQS, Redis), scale-to-zero from event queues, and integration with HPA for event-driven workloads.
KEDA (Kubernetes Event-Driven Autoscaling) extends the Kubernetes HPA with the ability to scale based on external event sources — Kafka topics, SQS queues, Redis lists, cron schedules — scaling from 0 to N and back to 0.
How KEDA Works:
Scale-to-Zero Advantage:
Standard HPA has a minimum of 1 replica. KEDA can scale to 0 when there are no events — critical for cost optimization in batch processing, nightly jobs, and event-driven workers.
50+ Built-in Scalers:
Kafka, AWS SQS/SNS, GCP Pub/Sub, Azure Service Bus, Redis Lists, RabbitMQ, Prometheus metrics, cron schedule, and more.
Code Implementation Example
# KEDA ScaledObject: scale worker based on Kafka consumer lag
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-processor-scaler
spec:
scaleTargetRef:
name: order-processor # Deployment to scale
# Scale to zero when no lag (no messages to process)
minReplicaCount: 0 # scale completely to zero!
maxReplicaCount: 100 # max 100 worker replicas
# Polling interval (check Kafka lag every 15s)
pollingInterval: 15
# Keep scaled-down for 5min before scaling to 0 (reduces thrashing)
cooldownPeriod: 300
triggers:
# Kafka trigger: scale based on consumer group lag
- type: kafka
metadata:
bootstrapServers: kafka-svc:9092
consumerGroup: order-processors
topic: orders
lagThreshold: "50" # 1 replica per 50 messages of lag
offsetResetPolicy: latest
---
# Optional: ScaledJob for batch jobs (run-to-completion, not deployments)
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: report-generator
spec:
jobTargetRef:
template:
spec:
containers:
- name: report-gen
image: report-gen:latest
triggers:
- type: aws-sqs-queue
metadata:
queueURL: https://sqs.us-east-1.amazonaws.com/123/reports
queueLength: "5" # 1 job per 5 SQS messagesKEDA scales Deployments from 0→N based on external queue depth (Kafka, SQS, Redis). Scale-to-zero is the key differentiator from HPA — critical for batch workloads and cost optimization.