Senior LevelPython
How do you architect resilient distributed task queues in Python using Celery with RabbitMQ brokers, Redis result backends, and acknowledgment worker patterns?
❓ Interview Prompt: Explain Celery distributed task queue, AMQP message brokers (RabbitMQ), Redis result backends, late acknowledgments (`task_acks_late`), and task prefetch tuning.
Celery is Python's distributed task queue framework:
Production Architecture:
1
Broker (RabbitMQ): AMQP broker for reliable task message persistence and routing.
2
Result Backend (Redis): High-speed key-value store for task execution status.
3
Resilience Configuration: Set 'task_acks_late = True' and 'worker_prefetch_multiplier = 1' so workers acknowledge task messages ONLY AFTER execution succeeds. If a worker pod crashes mid-task, RabbitMQ automatically redelivers the message to a surviving worker!
Code Implementation Example
python
from celery import Celery
app = Celery(
'tasks',
broker='amqp://guest@localhost//',
backend='redis://localhost:6379/0'
)
app.conf.update(
task_acks_late=True,
worker_prefetch_multiplier=1
)
@app.task(bind=True, max_retries=3)
def send_welcome_email(self, user_id):
print(f'Sending welcome email to user {user_id}')💡Configuring resilient Celery task worker with late acknowledgments.
🎯 Interview Key Takeaway
Configure Celery with task_acks_late=True and worker_prefetch_multiplier=1 to guarantee zero task loss during worker container crashes.
#Celery#RabbitMQ#Redis#Task Queue#Senior
Back to all Python Questions→