Intermediate LevelAI, LLMs, Embeddings & MCP
What is the Key-Value (KV) Cache in Transformer inference and how does it save GPU memory bandwidth?
❓ Interview Prompt: Explain autoregressive generation redundancy, Key and Value vector retention, and GPU memory bandwidth bottlenecks.
During LLM text generation, the model predicts text token-by-token autoregressively. Without optimization, generating token 100 requires re-computing Attention Key ($K$) and Value ($V$) matrices for all previous 99 tokens.
The KV Cache Solution:
The KV Cache stores pre-computed Key and Value tensor representations for all past tokens in GPU VRAM memory.
•
When generating token 100, the GPU calculates $Q, K, V$ for only token 100, retrieving past $K, V$ matrices directly from cache memory.
•
Reduces compute from $O(N^2)$ to $O(N)$ per token, dramatically reducing inference latency!
Code Implementation Example
typescript
// KV Cache VRAM calculation formula (Bytes)
function calculateKvCacheBytes(layers: number, heads: number, headDim: number, seqLen: number, batchSize: number): number {
// 2 for (Key + Value), 2 for FP16 (2 bytes per float)
return 2 * 2 * layers * heads * headDim * seqLen * batchSize;
}
// Llama-3-70B (80 layers, 8 KV heads, 128 head dim, 4096 seq, batch 1)
const cacheSizeMb = calculateKvCacheBytes(80, 8, 128, 4096, 1) / (1024 * 1024);
console.log('KV Cache size:', cacheSizeMb.toFixed(1), 'MB per request'); // ~1280 MB💡Calculating KV Cache GPU VRAM memory consumption per active request.
🎯 Interview Key Takeaway
The KV Cache stores pre-computed attention Key and Value matrices for past tokens, converting autoregressive generation from O(N^2) to O(N).
#KV Cache#Transformer#Inference#VRAM#Intermediate
Back to all AI, LLMs, Embeddings & MCP Questions→