Intermediate LevelAI, LLMs, Embeddings & MCP
How does Continuous Batching (Iteration-level batching) optimize LLM serving throughput?
❓ Interview Prompt: Compare traditional sequence-level batching vs iteration-level continuous batching in vLLM/TGI.
Traditional batching waits for all requests in a batch to finish generation before starting the next batch. Because prompt lengths vary, GPUs sit idle waiting for the longest request (padding waste).
Continuous Batching (Iteration-Level Batching):
•
Operates at the single-token iteration step level.
•
As soon as a request finishes generating its final token, it is immediately evicted from the batch.
•
A new incoming request is inserted into the active batch on the very next token iteration!
•
Keeps GPU compute utilization at nearly 100%.
Code Implementation Example
typescript
// Iteration-level scheduling concept
interface BatchSlot { requestId: string; currentStep: number; isDone: boolean; }
function scheduleIteration(activeSlots: BatchSlot[], waitingQueue: BatchSlot[]) {
// Evict completed slots
const running = activeSlots.filter(s => !s.isDone);
// Immediately insert waiting requests into freed slots!
while (running.length < MAX_BATCH_SIZE && waitingQueue.length > 0) {
running.push(waitingQueue.shift()!);
}
return running;
}💡Continuous batching scheduling concept at token iteration step.
🎯 Interview Key Takeaway
Continuous batching schedules requests at the single-token iteration level, filling freed GPU batch slots instantly.
#Continuous Batching#vLLM#TGI#Serving#Intermediate
Back to all AI, LLMs, Embeddings & MCP Questions→