Background Jobs & Queues
Run work asynchronously outside the request path — emails, exports, integrations, scheduled tasks, and anything slow enough to break a web request.
Options
Queue Backend*
Required Capabilities*
Failure & Durability*
Decision Points
Does any job charge money, send external messages, or call a paid API?
If yes
Use transactional outbox and make handlers idempotent. Store an idempotency key on the job.
If no
Standard retry + DLQ is sufficient.
Do you already run Redis or a broker?
If yes
Use it for jobs — a second persistence dependency is rarely justified.
If no
Start with a database-backed queue; migrate only when volume demands it.
Do you need scheduled/cron jobs in addition to on-demand enqueues?
If yes
Enqueue from a single scheduler process (not per-worker cron) to avoid duplicates in a horizontally-scaled deployment.
If no
Pure on-demand enqueue is simpler — add scheduled capability only when you actually have recurring jobs.
Is at-least-once delivery sufficient, or do you need exactly-once?
If yes
Exactly-once requires transactional outbox plus idempotent handlers — no library gives it to you for free.
If no
At-least-once with idempotent handlers is the pragmatic production default.
Do you have mixed-priority workloads (user-visible vs batch)?
If yes
Use at least two queues (default, bulk) with separate worker pools so a long batch job never starves user-triggered work.
If no
A single queue is simpler and fine for homogeneous workloads.
Are your handlers idempotent by contract?
If yes
Aggressive retries are safe — store an idempotency key per job and dedupe on handler entry.
If no
Lean on transactional outbox and accept retries will sometimes double-invoke side effects unless you add keys.
Do you need DAG pipelines (jobs that spawn dependent jobs)?
If yes
Use a durable workflow engine (Temporal, Inngest, BullMQ Flows) — rolling your own DAG orchestration is a year-long tarpit.
If no
Flat enqueue is simpler and covers the majority of use cases.
Do you need a dead-letter queue for failures?
If yes
Any production queue needs a DLQ with alerting on depth growth — silent job failure is a common outage source.
If no
Skip only for best-effort one-off jobs where losing the job is acceptable.
Do you need per-tenant queue isolation for noisy neighbors?
If yes
Shard queues by tenant or add per-tenant concurrency caps so one customer bursting to 10k jobs does not stall everyone else.
If no
A shared queue is fine in single-tenant or low-variance workloads.
Do long-running jobs need to be cancellable mid-run?
If yes
Pass a cancellation token through the handler and checkpoint progress so cancellation is responsive without data loss.
If no
If jobs complete quickly, retry-on-failure is simpler than implementing graceful cancellation.
Do you need per-job-type retry and backoff configuration?
If yes
Different failure modes need different backoff — network errors retry fast, rate-limit errors retry slow. Configure per job class.
If no
A single global retry policy (5 attempts, exponential backoff) is the pragmatic default.
Are jobs CPU-bound (heavy compute) or IO-bound (external calls)?
If yes
CPU-bound: use a worker pool sized to core count. Avoid async in the same process — it will not help and may hurt.
If no
IO-bound: use async/concurrent workers to maximize throughput on waiting time.
Do you need queue-depth and worker-lag observability?
If yes
Emit per-queue depth, processing latency, retry count, and DLQ size to Prometheus/Datadog — and alert on them.
If no
The built-in queue dashboard (Sidekiq Web, BullMQ Board) is enough for small teams.
Do you need to persist job results for later retrieval?
If yes
Store results in a separate results table keyed by job ID — clients poll or receive webhook/SSE when done.
If no
Fire-and-forget jobs are simpler; only persist results when a user UI depends on them.
Do job payloads contain sensitive data?
If yes
Encrypt payloads at rest (envelope encryption with KMS) — queue storage is usually less hardened than your primary DB.
If no
Plaintext payloads are fine for internal, non-PII work.
Should similar jobs be batched for efficiency?
If yes
Coalesce jobs (e.g., "send digest for user X") within a short window — a single batch handler beats N individual invocations for I/O.
If no
Per-job execution is simpler to reason about and debug.
Do you need an admin UI to list, retry, and cancel jobs?
If yes
Mount the queue library dashboard (Sidekiq Web, Oban Web, BullMQ Board) behind admin auth — zero-effort ops leverage.
If no
CLI tools and logs are enough for a small team; add UI when non-engineers need to investigate job failures.
Tradeoffs
Primary DB absorbs queue write load; row-level locks contend with application queries
Enqueue happens outside DB transaction — jobs can run for state that was rolled back
Additional table, polling worker, and idempotency discipline — the payoff is no duplicated side effects
Go Deeper
Implementation Examples
Mature ecosystem job libraries — study their docs for retry, scheduling, and priority patterns.
Redis-backed job queue for Node — flows, rate limiting, and a solid web dashboard.
Managed durable execution platform — useful when you don't want to operate a queue at all.