Senior LevelAI, LLMs, Embeddings & MCP
What is Mixture-of-Depths (MoD) and how does it dynamically allocate compute per token?
❓ Interview Prompt: Explain Mixture-of-Depths (MoD), adaptive compute allocation, router capacity bounds, and skipping transformer layers.
While Mixture-of-Experts (MoE) dynamically routes tokens across horizontal expert sub-networks, Mixture-of-Depths (MoD) dynamically routes tokens across vertical layer depth.
Core Concept (Google DeepMind):
•
Simple tokens (e.g. punctuation, common words "the") require very little computation.
•
Complex tokens (e.g. key code logic, hard mathematical steps) require deep computation.
•
MoD uses a router at specific transformer layers to skip layer processing entirely for easy tokens, passing them through residual connections.
•
Reduces total FLOPs per token by 50% while maintaining dense model accuracy!
Code Implementation Example
typescript
// Conceptual Mixture-of-Depths Layer Router
function processModLayer(tokenEmbeddings: number[][], capacityK: number) {
// 1. Router scores tokens for current layer relevance
const scores = tokenEmbeddings.map(emb => calculateLayerRelevance(emb));
// 2. Select top-K tokens that get to execute layer compute
const topKIndices = new Set(getTopKIndices(scores, capacityK));
// 3. Simple tokens skip layer compute via residual connection!
return tokenEmbeddings.map((emb, i) =>
topKIndices.has(i) ? executeLayerCompute(emb) : emb
);
}💡Mixture-of-Depths router selecting top-K tokens for vertical layer execution.
🎯 Interview Key Takeaway
Mixture-of-Depths dynamically skips specific transformer layers for simple tokens, capping compute FLOPs per token.
#Mixture-of-Depths#MoD#Adaptive Compute#Architecture#Senior
Back to all AI, LLMs, Embeddings & MCP Questions→