Senior LevelSystem Design & Architecture
How do you implement an Idempotent Consumer in Event-Driven Architecture to handle duplicate delivery?
❓ Interview Prompt: Explain Idempotent Consumer Pattern, handling duplicate event delivery in Kafka/RabbitMQ, message deduplication tables, and atomic execution.
Message brokers (Kafka, RabbitMQ) guarantee At-Least-Once Delivery, meaning duplicate events WILL arrive at consumers during network retries!
Idempotent Consumer Pattern:
1
Every event message includes a unique 'eventId' or 'messageId' in its header.
2
Before processing an event, the consumer checks a Deduplication Store (PostgreSQL/Redis).
3
Process the event payload and write 'eventId' to the deduplication store inside the same local database transaction!
4
If duplicate 'eventId' arrives later $\rightarrow$ Ignore message immediately.
Code Implementation Example
typescript
export async function processIdempotentEvent(event: { id: string; payload: any }) {
return await db.$transaction(async (tx) => {
// Check if message ID was already processed
const processed = await tx.processedEvents.findUnique({ where: { id: event.id } });
if (processed) return; // Duplicate message ignored safely!
await handleBusinessLogic(event.payload);
await tx.processedEvents.create({ data: { id: event.id, processedAt: new Date() } });
});
}💡Implementing an Idempotent Consumer with database event deduplication.
🎯 Interview Key Takeaway
Build Idempotent Consumers with deduplication tables to handle duplicate message delivery safely in event-driven systems.
#System Design#Idempotent Consumer#Kafka#EDA#Senior
Back to all System Design & Architecture Questions→