Queues, Not Cron: Designing Background Jobs That Don't Wake You Up at 3AM

A cron job that runs a script every five minutes is the fastest way to ship background work, and one of the most common sources of on-call pages I've inherited when taking over a system.

The failure mode is always the same shape: the job runs long, the next scheduled run starts before the first one finished, now you have two processes fighting over the same rows, and someone gets paged at 3am to figure out why customers are getting double-charged.

Why queues beat cron for anything that touches state

A proper queue (Redis-backed, SQS, or your database as a queue table if you're small) gives you three things cron doesn't:

• Idempotency by design. Each job processes one unit of work with a unique identifier. Retries don't double-process, because the job checks its own completion state first.
• Backpressure. If your worker pool is saturated, jobs wait in the queue instead of piling up as competing cron invocations.
• Visibility. A failed job goes to a dead-letter queue where you can inspect and retry it. A failed cron script just... didn't run, and you find out when a customer complains.

The three rules I don't compromise on

1. Every job is idempotent, verified by a unique key — not by hoping it doesn't run twice. Before doing the work, the job checks: has this exact operation already been completed? A payment job keyed on an idempotency key you generated at request time, not on the job's own retry count.

2. Jobs are small and single-purpose. "Process the daily report" is not a job — it's a workflow. Break it into "fetch the data," "generate the report," "send the report," each independently retryable. If step three fails, you don't want to re-run steps one and two.

3. Failure is loud, and success is quiet. Alerting on job failures needs to happen automatically, not via a dashboard someone has to remember to check. And once you're alerting on failure, resist the urge to also alert on every success — you'll train the team to ignore the channel.

The retry strategy that actually matters

Exponential backoff with jitter, and a maximum attempt count that lands the job in a dead-letter state instead of retrying forever. I've seen systems where a broken job retried every few seconds for four days, quietly consuming a large chunk of the worker pool's capacity while nobody noticed the queue depth climbing on every other job type.

When cron is still fine

Cron is fine for genuinely idempotent, stateless triggers — "check if there's new work to enqueue," not "do the work." Use cron to kick off a queued job, not to be the job. That one distinction removes most of the failure modes above without giving up the simplicity of a scheduled trigger.

The actual payoff

None of this is exotic. It's a few hours of deliberate design against a checklist. What it buys you is the difference between "the queue depth alert fired and we fixed it during business hours" and "customers noticed before we did."