Celery vs RQ vs Dramatiq for FOIA Batch Processing #

Within Async Batch Processing, the task framework you choose is not a matter of taste — it is the mechanism that decides whether a deadline-bound record survives a worker crash or vanishes without a trace. Celery, RQ, and Dramatiq all dispatch background jobs, and any of them will run a demo. The differences that matter for public-records work show up only under failure: when a broker restarts mid-batch, when a worker is killed while holding a task, when the same message is delivered twice, and when a job has exhausted its retries and needs somewhere safe to land. This guide compares the three on exactly those axes and gives a clear recommendation by scenario, framing every trade-off around one non-negotiable requirement: never silently lose a task that carries a statutory deadline.

The Decision That Actually Matters: Durability #

A FOIA batch is deadline-bound work under 5 U.S.C. § 552(a)(6)(A)(i), which gives an agency 20 business days to respond. If a task disappears — because a worker died after pulling it off the broker but before finishing, and the framework had already acknowledged the message — nobody gets an error. The record simply never processes, the production ships incomplete, and the failure surfaces weeks later as a complaint or a lawsuit. That is the difference between at-most-once and at-least-once delivery, and it is the first question to ask of any framework.

At-most-once delivery acknowledges the message when it is handed to a worker. If the worker dies, the message is gone. At-least-once delivery holds the message on the broker until the worker confirms completion; a crash re-delivers it, so the task runs again rather than disappearing. For deadline-bound records you want at-least-once, paired with idempotent tasks so a re-delivery is harmless. Everything else — priority, retries, dead-letter routing, operational weight — is secondary to getting this right, because every other capability assumes the task still exists.

The three frameworks land differently. Celery defaults to at-most-once but offers strong at-least-once semantics through acks_late=True plus task_reject_on_worker_lost, at the cost of the most complex operational surface. RQ is Redis-only and at-most-once by default; durability requires care, and it has no native priority. Dramatiq was designed after both and ships at-least-once behavior, automatic retries, and dead-letter handling as defaults rather than options, with a middleware model that is simpler to operate than Celery.

Durability is not a property of the framework alone, though — it is a property of the framework plus the acknowledgment mode plus the broker. A framework can promise at-least-once delivery and still lose records if the broker acknowledges eagerly, if the result backend drops state on restart, or if the task is not idempotent and a re-delivery double-counts a disclosure. That is why the recommendation below never stops at a framework name: it pairs each choice with the acknowledgment setting, the persistence requirement, and the idempotency discipline that together make a batch actually deadline-safe. Treat “which framework” as the first of three coupled decisions rather than the whole answer.

Capability Comparison #

The matrix summarizes where each framework stands on the axes that decide a batch’s fate. Read the durability column first; the rest only matters once a task is guaranteed to still exist after a crash.

The same comparison in table form, with the acknowledgment detail spelled out:

How a Task Looks in Each #

The task-definition ergonomics differ, and the differences map directly onto the durability story. In Celery, the durability guarantees are opt-in flags you must remember to set:

from celery import Celery

app = Celery("foia", broker="amqp://localhost", backend="redis://localhost")


@app.task(bind=True, acks_late=True, max_retries=5,
          autoretry_for=(ConnectionError,), retry_backoff=True)
def retrieve_document(self, doc_id: str, request_id: str) -> str:
    # acks_late keeps the message on the broker until the task finishes, so a
    # worker crash re-delivers this deadline-bound record instead of dropping
    # it. Without acks_late, Celery acknowledges on receipt and the record is
    # gone if the worker dies mid-task.
    return fetch(doc_id)

RQ keeps the surface minimal, but you inherit at-most-once delivery unless you add retries and lean on separate queues to fake priority:

from redis import Redis
from rq import Queue, Retry

# No native priority: a dedicated high-urgency queue, worked first, is the
# standard approximation. Retry() adds the bounded retries that RQ's default
# at-most-once delivery otherwise lacks.
high = Queue("foia-high", connection=Redis())
high.enqueue(retrieve_document, "DOC-1", "REQ-1",
             retry=Retry(max=3, interval=[10, 30, 60]))

Dramatiq puts the durable behavior in the defaults — retries and dead-letter routing happen without extra flags:

import dramatiq
from dramatiq.brokers.rabbitmq import RabbitmqBroker

dramatiq.set_broker(RabbitmqBroker(url="amqp://localhost"))


@dramatiq.actor(max_retries=5, min_backoff=10_000, time_limit=600_000)
def retrieve_document(doc_id: str, request_id: str) -> None:
    # Messages that exhaust max_retries move to the dead-letter queue
    # automatically, so no deadline-bound record vanishes without a trace and
    # a human can reconcile the DLQ before the production is certified.
    process(fetch(doc_id))

Notice the pattern: Celery gives you the most control but demands you configure durability correctly; RQ gives you the least surface area but the weakest defaults; Dramatiq gives you safe defaults with less to tune. For an agency, “safe by default” is worth a great deal, because the failure mode of a forgotten flag is a missed statutory deadline nobody sees until it is too late.

Recommendation by Scenario #

• High-volume intake with true priority tiers. When expedited requests, litigation-hold productions, and routine backlog must be worked in a strict order, Celery’s native broker priorities are the only first-class fit, and its maturity justifies the operational weight. This is the path detailed in Managing High-Volume Intake with Celery Task Queues, and the ordering logic it consumes is set by Priority Scoring Algorithms.
• A new build that must be durable without a dedicated ops team. Choose Dramatiq. At-least-once delivery, automatic retries, and a dead-letter queue are defaults, so the batch is deadline-safe before anyone tunes it, and the middleware model is far lighter to run than a multi-node Celery deployment with its separate result backend, beat scheduler, and monitoring surface.
• A small, self-contained job on infrastructure you already run on Redis. RQ is the pragmatic pick when the batch is modest, priority is not required, and simplicity wins — provided you consciously add Retry(...) and treat Redis persistence as part of your durability story rather than assuming it.
• Any framework, one hard rule. Whatever you pick, make the task idempotent and enable at-least-once delivery, so a re-delivered message is safe and a lost message is impossible. The streaming execution body that runs inside these tasks is covered in Streaming Large PDF Batches with Asyncio and Backpressure, and the routing that feeds them is governed by Async Queue Management.

Operational Trade-offs Beyond the Framework #

The framework decides delivery semantics; the deployment decides whether those semantics hold. Three operational facts dominate the choice regardless of which name you pick. First, the broker is the real durability boundary: an at-least-once framework in front of a Redis instance with no persistence configured is still at-most-once in practice, because a broker restart drops the in-flight messages. Second, visibility timeouts and acknowledgment windows must exceed your longest task, or the broker re-delivers a task that is still running and you double-process a record — a particular risk for the large scanned binders that take minutes, not seconds, to parse. Size the window to the tail of your task-duration distribution, not the median, and make the task idempotent so an occasional overlap is harmless rather than a duplicated disclosure. Third, the dead-letter queue is not optional plumbing — it is the evidence trail. A record that exhausts its retries must land somewhere a human reviews before the production is certified, and every terminal outcome should emit a structured line in line with Audit Logging Architecture so the agency can prove what processed, what failed, and what was reconciled.

Compliance Verification Checklist #

• The chosen framework runs with at-least-once delivery enabled (acks_late on Celery, durable defaults on Dramatiq, deliberate Retry(...) plus persistence on RQ), so a worker crash re-delivers a task rather than dropping it.
• Every task is idempotent, keyed on a stable identifier, so a re-delivered deadline-bound record is processed safely rather than duplicated.
• The broker is configured for persistence, and its durability guarantees are verified — not assumed — because the broker, not the framework, is the real boundary.
• Visibility timeouts and acknowledgment windows exceed the longest expected task, preventing re-delivery of still-running work.
• Retries use bounded, backed-off attempts scoped to transient failures, so a persistent fault does not retry into the statutory deadline.
• Records that exhaust their retries land on a dead-letter queue that a human reconciles before the production is certified complete.
• Every terminal task outcome emits a structured audit line, so the agency can prove which records processed, failed, or were re-queued within the response window.

FAQ #

Related #

• Async Batch Processing — the parent system these frameworks execute
• Streaming Large PDF Batches with Asyncio and Backpressure
• Managing High-Volume Intake with Celery Task Queues
• Async Queue Management
• Audit Logging Architecture

← Back to Document Retrieval & Parsing