Intermediate LevelAI, LLMs, Embeddings & MCP
How does PagedAttention (vLLM) eliminate GPU memory fragmentation during LLM serving?
❓ Interview Prompt: Explain physical vs virtual GPU memory allocation, PagedAttention block tables, and vLLM throughput improvements.
Traditional LLM serving engines pre-allocate contiguous GPU VRAM blocks for a request's maximum potential sequence length (e.g. 4096 tokens). Because most requests finish early, 60% to 80% of GPU VRAM is wasted on internal memory fragmentation!
PagedAttention (vLLM Engine):
Inspired by Virtual Memory Paging in Operating Systems:
1
Partitions the KV Cache into small non-contiguous physical memory blocks (e.g. 16 tokens per block).
2
Maintains a Virtual Block Table mapping logical token sequences to physical GPU memory pages.
3
Allocates new memory pages dynamically on-demand as tokens are generated.
Impact:
Eliminates memory fragmentation, boosting LLM serving request batch sizes and token throughput by 2x to 4x.
Code Implementation Example
typescript
// Conceptual vLLM PagedAttention Block Table Mapping
interface BlockTable {
logicalBlockId: number;
physicalBlockAddress: number; // Non-contiguous GPU VRAM block pointer
}
const requestBlockTable: BlockTable[] = [
{ logicalBlockId: 0, physicalBlockAddress: 0x7f01 }, // Tokens 0-15
{ logicalBlockId: 1, physicalBlockAddress: 0x3a09 } // Tokens 16-31
];💡Virtual block table mapping logical KV cache blocks to non-contiguous GPU VRAM pages.
🎯 Interview Key Takeaway
PagedAttention eliminates GPU memory fragmentation by allocating non-contiguous KV cache pages dynamically like OS virtual memory.
#PagedAttention#vLLM#GPU Memory#Serving#Intermediate
Back to all AI, LLMs, Embeddings & MCP Questions→