Beginner LevelAI, LLMs, Embeddings & MCP
What is the LLM Context Window and what happens when input exceeds token limits?
❓ Interview Prompt: Explain context windows, maximum sequence lengths, attention memory overhead, and context truncation strategies.
The Context Window is the maximum sequence length of combined input (prompt) and output (completion) tokens that an LLM can process in a single inference call.
Key Concepts:
•
Modern context sizes range from 8,000 tokens (8k) up to 1,000,000+ tokens (1M+ in Gemini 1.5 Pro).
•
Quadratic Attention Bottleneck ($O(N^2)$ memory growth) makes long contexts computationally expensive.
What Happens When Exceeded:
1
API Error: Most provider APIs return an error (
400 Context Length Exceeded).2
Needle-in-a-Haystack Degradation: Information in the middle of extremely long contexts may be forgotten ("Lost in the Middle" phenomenon).
Mitigation Strategies:
Sliding window summarization, document chunking, and selective RAG retrieval.
Code Implementation Example
typescript
function truncateContext(messages: { role: string; content: string }[], maxTokens: number): typeof messages {
let currentTokens = 0;
const pruned = [];
// Iterate backwards from most recent messages
for (let i = messages.length - 1; i >= 0; i--) {
const msgTokens = estimateTokens(messages[i].content);
if (currentTokens + msgTokens > maxTokens) break;
pruned.unshift(messages[i]);
currentTokens += msgTokens;
}
return pruned;
}💡Pruned conversation history backwards to fit inside a maximum context token limit.
🎯 Interview Key Takeaway
The context window limits total input + output tokens per request. Exceeding limits requires message pruning or summarization.
#Context Window#Tokens#Pruning#Beginner
Back to all AI, LLMs, Embeddings & MCP Questions→