What is Tail Latency Amplification and how do Hedged Requests mitigate high p99 latency?
❓ Interview Prompt: Explain Tail Latency Amplification, fan-out microservice queries, $p99$ latency accumulation, and Hedged Requests / Tied Requests.
Tail Latency Amplification occurs in fan-out microservice architectures: if a service makes 100 parallel sub-requests to render a page, and each sub-request has a $99\%$ success rate within $50\text{ms}$, the probability of the main request completing within $50\text{ms}$ drops to $0.99^{100} \approx 36.6\%$!
Hedged Requests Solution:
If a primary sub-request has not returned within $p95$ time ($20\text{ms}$), the client automatically sends an identical Hedged Request to a secondary server node and accepts whichever response arrives first, canceling the slower request!
Code Implementation Example
export async function fetchWithHedgedRequest(url: string, hedgeDelayMs = 20) {
const controller = new AbortController();
const primary = fetch(url, { signal: controller.signal });
const hedgeTimer = new Promise((resolve) => setTimeout(resolve, hedgeDelayMs));
const winner = await Promise.race([primary, hedgeTimer]);
if (winner !== undefined) return winner; // Primary finished fast!
// Primary timed out at p95; fire Hedged Request to secondary server!
const secondary = fetch('${url}?hedge=1', { signal: controller.signal });
return await Promise.race([primary, secondary]);
}Use Hedged Requests to fire duplicate requests to secondary servers after p95 timeouts, eliminating tail latency bottlenecks.