Intermediate LevelAI, LLMs, Embeddings & MCP
How does Semantic Caching (GPTCache) prevent duplicate LLM API requests?
❓ Interview Prompt: Explain exact match caching vs semantic vector caching, similarity thresholds, and cost/latency reduction.
Standard key-value caches (like Redis) only match identical query strings. Semantic Caching embeds user queries to match semantically equivalent questions.
How It Works:
1
When a user asks "How do I reset my password?", the semantic cache checks vector similarity against past cached questions.
2
If a cached question (e.g. "What is the step to reset password?") has a cosine similarity $> 0.96$, the cache returns the stored LLM answer instantly!
3
Benefits: Sub-10ms response latency and 100% savings on model API tokens for cached questions.
Code Implementation Example
typescript
class SemanticCache {
private cacheStore: { vector: number[]; answer: string }[] = [];
async get(query: string, threshold: number = 0.95): Promise<string | null> {
const queryVec = await embedText(query);
for (const entry of this.cacheStore) {
if (cosineSimilarity(queryVec, entry.vector) >= threshold) {
return entry.answer; // Semantic cache hit!
}
}
return null; // Cache miss
}
}💡Semantic cache lookup using embedding vector cosine similarity.
🎯 Interview Key Takeaway
Semantic caching uses vector similarity search to return stored answers for semantically equivalent user questions instantly.
#Semantic Cache#GPTCache#Optimization#Vector Search#Intermediate
Back to all AI, LLMs, Embeddings & MCP Questions→