Celery doesn't promise exactly-once execution. In many setups, it's at-least-once, so your tasks need to survive running twice. Here's a practical breakdown.

Why a task runs twice

  • acks_late=True + worker crash: the broker redelivers the task, possibly after partial work.
  • Redis visibility timeout: unacknowledged messages become visible again (1 hour by default). Long tasks or long countdown/eta values can be redelivered while still running.
  • Retries: a task that fails after a side effect repeats that side effect.
  • Double submissions: double-clicks, duplicate webhooks, client retries.
  • Celery Beat: overlapping runs, or more than one Beat process.
  • Queuing inside a transaction: the worker can start before the data is committed.

1. Make tasks idempotent

# Not idempotent @shared_task def give_bonus(user_id): Wallet.objects.filter(user_id=user_id).update(balance=F("balance") + 10) # Idempotent @shared_task def recalculate_total(order_id): order = Order.objects.get(pk=order_id) order.total = calculate_total(order) order.save(update_fields=["total"]) 

Compute from source data, prefer update_or_create(), and check state before acting.

2. Use a unique constraint as a safety net

Check-then-act fails when two copies run at the same moment. Let the database decide:

from django.db import IntegrityError, transaction @shared_task def create_payout(order_id): order = Order.objects.get(pk=order_id) try: with transaction.atomic(): Payout.objects.create(order=order, amount=order.total) except IntegrityError: return # duplicate run 

For concurrent updates to one row, use select_for_update() inside transaction.atomic().

3. Use a lock for non-overlapping work

@shared_task(bind=True) def generate_invoice(self, order_id): lock = redis_client.lock(f"lock:invoice:{order_id}", timeout=300) if not lock.acquire(blocking=False): return "skipped: already running" try: build_invoice(order_id) finally: try: lock.release() except redis.exceptions.LockError: pass 

Rules of thumb:

  • Always set a timeout, or a crashed worker blocks the task forever
  • Make the timeout longer than the task's normal runtime
  • A lock is not idempotency: a duplicate arriving after the lock is released still runs
  • cache.add() works as a simple lock only on shared backends like Redis, not local-memory cache

4. Idempotency keys for webhooks

class ProcessedEvent(models.Model): event_id = models.CharField(max_length=200, unique=True) processed_at = models.DateTimeField(auto_now_add=True) @shared_task def handle_payment_webhook(event_id, payload): try: with transaction.atomic(): ProcessedEvent.objects.create(event_id=event_id) apply_payment(payload) except IntegrityError: return # already handled 

The key and the business change share one transaction, so a failed run leaves no key behind and a retry can still succeed.

5. Queue after commit

def place_order(request): with transaction.atomic(): order = Order.objects.create(...) transaction.on_commit(lambda: process_order.delay(order.id)) 

Pass IDs, not model objects.

6. Check your config

CELERY_TASK_ACKS_LATE = True CELERY_TASK_REJECT_ON_WORKER_LOST = True CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 43200} # 12 hours 

Set visibility_timeout higher than your longest task and longest countdown. For Beat, run exactly one instance and lock long periodic tasks.

Which fix when?

Situation Approach
Any retryable task Idempotent design
Create a record once Unique constraint + IntegrityError
Concurrent updates select_for_update()
Expensive, must not overlap Redis lock with timeout
Webhooks, API retries Idempotency key in the database
Periodic tasks Lock + single Beat instance

Full guide

The complete post has a pre-ship checklist and an FAQ:

👉 https://blog.reiteachaigean.com/blog/celery-prevent-duplicate-task-execution

More on Celery and large datasets:

How do you handle duplicate tasks in production?