Beginner LevelAI, LLMs, Embeddings & MCP
What are Text Chunking Strategies (Fixed-size vs Sentence vs Semantic Chunking) for RAG?
❓ Interview Prompt: Explain text chunking, token boundary selection, overlap parameters, and semantic document splitting.
Before embedding documents into a vector store, large files must be split into smaller text segments called chunks.
Chunking Strategies:
1
Fixed-Size Chunking: Splits text into uniform character or token counts (e.g. 500 characters). Simple, but can slice sentences mid-thought.
2
Sentence / Paragraph Chunking: Respects natural punctuation boundaries (
\n\n or .).3
Recursive Character Chunking: Tries splitting by paragraphs, then sentences, then words recursively to maintain target chunk size.
4
Semantic Chunking: Computes embedding distance between consecutive sentences and splits where semantic distance spikes.
Code Implementation Example
typescript
function recursiveChunkText(text: string, chunkSize: number = 500, overlap: number = 50): string[] {
const paragraphs = text.split('\n\n');
const chunks: string[] = [];
let currentChunk = '';
for (const para of paragraphs) {
if ((currentChunk + para).length <= chunkSize) {
currentChunk += (currentChunk ? '\n\n' : '') + para;
} else {
if (currentChunk) chunks.push(currentChunk);
currentChunk = para;
}
}
if (currentChunk) chunks.push(currentChunk);
return chunks;
}💡Paragraph-aware text chunking implementation with overlap.
🎯 Interview Key Takeaway
Recursive paragraph chunking preserves sentence boundary context better than arbitrary fixed-length splitting.
#Chunking#RAG#Text Processing#Beginner
Back to all AI, LLMs, Embeddings & MCP Questions→