Senior LevelSystem Design & Architecture
How do you implement Dynamic Load Shedding based on CPU utilization and queue delay?
❓ Interview Prompt: Explain Load Shedding, adaptive concurrency limits (Netflix Concurrency-Limits), CPU/memory saturation thresholds, and rejecting non-critical traffic.
When a backend server approaches CPU or memory saturation, queuing incoming requests makes latency skyrocket for ALL users (Saturated Queue Collapse):
Dynamic Load Shedding Solution:
1
Server monitors current CPU utilization and HTTP queue delay.
2
When CPU exceeds $85\%$ or queue delay exceeds $100\text{ms}$, the server sheds load by rejecting non-critical background requests immediately with '503 Service Unavailable'.
3
Preserves CPU capacity strictly for critical user workflows (e.g. checkout payments).
Code Implementation Example
typescript
import { LittleLawLimiter } from './concurrency';
const limiter = new LittleLawLimiter({ maxQueueDelayMs: 100, targetCpuRatio: 0.85 });
app.use((req, res, next) => {
if (!limiter.shouldAcceptRequest(req.priority)) {
// Dynamic Load Shedding: Drop non-critical traffic immediately!
return res.status(503).json({ error: 'Server saturated; request shed' });
}
next();
});💡Dynamic Load Shedding middleware shedding traffic under CPU saturation.
🎯 Interview Key Takeaway
Implement Dynamic Load Shedding to drop non-critical traffic when CPU or queue delay exceeds saturation bounds.
#System Design#Load Shedding#Resilience#Concurrency#Senior
Back to all System Design & Architecture Questions→