Beginner LevelAI, LLMs, Embeddings & MCP
What is the difference between Dense Embeddings and Sparse Embeddings (BM25)?
❓ Interview Prompt: Compare dense neural vector representations with sparse keyword term frequency representations.
1. Dense Embeddings (e.g. OpenAI text-embedding-3, BGE):
•
High-dimensional vectors where almost every index contains a non-zero floating-point number (e.g. 1536 dimensions).
•
Generated by neural transformer models.
•
Strength: Captures deep semantic context, synonyms, and intent (e.g. connects "physician" to "doctor").
2. Sparse Embeddings (e.g. BM25, TF-IDF, SPLADE):
•
High-dimensional vectors where most values are zero, with non-zero values representing token term frequency weights.
•
Strength: Captures exact keyword matches, serial numbers, and specific proper nouns (e.g. matching SKU code "ERR-9921").
Code Implementation Example
typescript
// Dense Vector: Continuous float array (768 dimensions)
const denseVector = [0.012, -0.421, 0.109, 0.884];
// Sparse Vector: Map of specific term indices to weights
const sparseVector = {
indices: [1042, 8891, 14002], // Vocabulary token indices
values: [0.85, 1.20, 0.45] // BM25 term frequency weights
};💡Dense float vector structure vs sparse index-weight representation.
🎯 Interview Key Takeaway
Dense embeddings capture semantic meaning; sparse embeddings (BM25) capture exact keyword matches and serial numbers.
#Dense Embeddings#Sparse Embeddings#BM25#Beginner
Back to all AI, LLMs, Embeddings & MCP Questions→