Starter
30-day free trial$12.00/ month
512 MB RAM0.5 vCPUs
Products · Background workers
Run background jobs, queues, and cron schedules.
Nothing to operate.
Push a worker and Darwa gives it a queue, a scheduler, retry policy, a dead-letter queue, concurrency limits, and per-job logs. When the backlog grows it adds workers — and tells you when the queue will be clear.
— No Redis to run— No cron box— No hidden failed jobs
Waiting3,204
job_8f21a0
thumbnail · order-4821.jpg
priority 1 · queued 2s
job_8f21a1
invoice.pdf → email
priority 1 · queued 3s
job_8f2200
nightly-export
delayed · runs 02:00
job_8f2201
embed 240 documents
priority 5 · queued 41s
+ 3,200 more
Processing8 workers
job_8f21f4RUNNING
resize · hero-banner.png
worker 3 · 1.9s
job_8f21f6RUNNING
stripe.invoice.send
worker 5 · 0.6s
job_8f21f9RUNNING
summarise support thread
worker 7 · 4.2s
Failed · dead letter3
job_8f1d02FAILED
stripe.charge.create
attempt 3 of 5 · HTTP 429
job_8f1c88FAILED
webhook → crm.acme.com
attempt 5 of 5 · timeout
job_8f1b41FAILED
pdf → text extraction
attempt 2 of 5 · OOM at 512 MB
Depth 3,204 · 8 workers · 41 jobs/s · avg 1.9s
ETAQueue clears in 6m 20s at the current rate
PLANScaling 8 → 26 workers brings that to 1m 05s
COST+$0.38 for the burst, workers released after
→Applied automatically — your rule allows up to 40
JOB 01deploy
Bring the worker framework you already use.
Your existing BullMQ, Celery, or Sidekiq worker deploys as-is. Darwa supplies the broker, the scheduler, and the runtime around it.
BullMQ
Celery
Sidekiq
Asynq
Oban
RQ
Laravel Queue
Custom Docker
Git and Docker deploymentPush to a branch or supply an image — both build and release the same way.
Always-on workersThe process stays resident between jobs. No cold start per message, no invocation ceiling.
Automatic redeploysEvery commit ships a new worker version without dropping a job.
Worker versioningIn-flight jobs finish on the version that started them; new jobs pick up the latest.
Managed brokerA Redis-compatible queue is provisioned with the worker. Bring your own if you prefer.
darwa.yaml
worker: image-processing runtime: node22 start: node worker.js queue: name: images type: priority concurrency: 20 scale: min: 1 max: 40 on: queue_depth retries: attempts: 5 backoff: exponential dead_letter: true rate_limit: stripe: 100/minute
JOB 02queues
Every queue pattern, without assembling one.
FIFO for ordering, priority for urgency, delayed for later, scheduled for recurring — declared in config rather than built out of primitives.
FIFO queues
Strict ordering per queue or per key, so jobs for one customer never overtake each other.
Priority queues
Ten priority levels. A password reset jumps ahead of a nightly export without a second queue.
Delayed jobs
Run in ten minutes, at 02:00, or on a timestamp you compute — held by the broker, not a sleeping process.
Redis-compatible
Existing BullMQ or Sidekiq code connects unchanged; the connection string is injected for you.
Per-key deduplication
Enqueue the same job id twice and only one runs — useful for webhook storms.
Batch enqueue
Push thousands of jobs in one call and track them as a group with a single progress figure.
| Trigger | What starts the job |
|---|---|
| HTTP request | POST a payload to a worker endpoint and get a job id back |
| Queue message | Anything your app pushes to a Redis-compatible queue |
| Schedule | Cron expressions, or plain-language intervals |
| Database event | Row inserted, updated, or deleted in your managed Postgres |
| Storage upload | A file landing in a bucket starts the job that processes it |
| Webhook | Signed third-party callbacks, verified before your code runs |
JOB 03cron
Scheduling that does not need its own container.
Cron expressions or plain intervals, attached to the worker that runs them. Missed windows are recorded rather than silently skipped.
Cron or interval“*/5 * * * *” or “every 5 minutes” — both accepted, both shown as the next three run times.
Overlap policyQueue the next run, skip it, or cancel the previous one. Chosen per schedule, not globally.
Timezone awareDaily at 07:00 means 07:00 where your business is, including across daylight saving.
Recorded missesIf a schedule could not run, it appears as a missed window with the reason.
Manual runTrigger any schedule immediately from the dashboard or CLI to test it.
every minute health-sweep next 09:43
hourly refresh-search-index next 10:00
weekdays 07:00 overnight-digest next Mon
0 2 * * * nightly-export next 02:00
monthly 1st invoice-run next Aug 1
MISShealth-sweep skipped 04:12 — deploy in progress
JOB 04retries · dlq
Failures retry themselves, then land somewhere you can see.
Exponential backoff by default, a dead-letter queue with a real interface, and a diagnosis instead of a stack trace.
Attempt schedule · exponential backoff
try 10stry 2+2stry 3+8stry 4+32stry 5+2m
Attempts, backoff curve, and ceiling are per queue. A rate-limited third party gets patient retries; a malformed payload fails fast and goes straight to the dead-letter queue.
Why it failed — not just that it failed
FAILstripe.charge.create — HTTP 429
Attempt 3 of 5 · 41 similar failures in 60s
CAUSEYour Stripe key allows 100 req/min; workers sent 340
FIXRate-limit the stripe tool to 100/min at the pool level
RETRYRetry after 30s — the window resets then
→darwa queue limit stripe 100/minute
Dead-letter queue
InspectFull payload, attempt history, and the error from each try
RetrySend one job or a filtered selection back to the front of the queue
ReplayRe-run against the current worker version with the original payload
Edit and retryFix a malformed field and requeue without touching your code
ExportDownload as JSON for analysis or to reproduce locally
DeleteDiscard permanently, with the action recorded in activity history
JOB 05concurrency
Different work needs different limits.
Image processing wants twenty workers. A rate-limited API wants five. An expensive model call wants two. Set it per queue and let the platform hold the line.
Worker pool · image-processing
6 busy · 2 idle · 18 starting — filled is working, dashed is being added now
1→5→8→26→40 max
| Queue | Concurrency | Scale range | Rate limit | Why |
|---|---|---|---|---|
| image-processing | 20 | 1 → 40 | — | CPU bound, safe to widen |
| email-sending | 5 | 1 → 5 | 300/min | Provider throttles above this |
| stripe-billing | 4 | 1 → 4 | 100/min | API key quota |
| ai-summaries | 2 | 1 → 3 | 20/min | Model cost per call |
| nightly-export | 1 | 1 → 1 | — | Must not run twice |
Rate limits are enforced across the whole pool, not per worker — so scaling to forty workers cannot accidentally quadruple the requests you send a third party.
JOB 06metrics · logs
Per-job logs, not one giant stream to grep.
Open a job and see only its lines, its payload, its attempts, and its timings. Metrics answer the question you actually have: is the backlog growing or shrinking?
Waiting
3,204
peaked at 8,900 · 09:12
Running
8
26 starting, 40 allowed
Throughput
41 /s
+18% after the last deploy
Average processing
1.9 s
p95 4.4 s · slowest: pdf extraction
Failed · 24h
3
of 892,410 · 0.0003%
Retry rate
1.2%
mostly Stripe 429s at 09:41
Ask the queue a question
1Postgres is the bottleneck — 780 ms per job in query time
2Only 8 of 40 allowed workers are running
3Stripe calls waited 2.1 s average — you are at the limit
4Memory 480 MB of 512 MB — GC pauses adding 120 ms
FIXAdd an index, raise workers to 26, bump memory to 1 GB
→Projected: queue clears in 1m 05s instead of 6m 20s
One job, its own log
09:41:02 picked up by worker 7 (attempt 1)
09:41:02 payload: { thread_id: 4821, msgs: 14 }
09:41:03 postgres: loaded 14 messages (41 ms)
09:41:06 model call: 1,204 tokens (2.9 s)
09:41:06 wrote summary to threads.summary
DONE4.2 s total · $0.004 · no retries
Cost is attributed per job, so an expensive queue is obvious before the invoice explains it.
JOB 07cost
Workers idle at 3 a.m. and you still pay for them.
Darwa watches the shape of your backlog over the week and proposes floors and ceilings that match it — with the saving stated before you agree.
image-processing min 10 workers held 24/7
IDLEBetween 22:00 and 07:00 depth stays under 20 jobs
PLANOvernight floor 10 → 2, daytime ceiling unchanged
SAVE$42/month · queue still clears within 90s overnight
RISKA 3 a.m. burst would take 40s longer to drain
→darwa cost apply 1
JOB 08queued
What is queued next.
Not shipped yet. Everything above works without it — this is where the product is heading.
Visual workflow builder · preview
Upload fileResize imageGenerate thumbnailStore in bucketSend email
Orchestration without orchestration code — each node keeps its own retries, concurrency, and logs.
01Visual workflow builderSoonChain steps without writing orchestration code — upload, resize, thumbnail, store, notify.
02Durable executionSoonA worker that dies two hours in resumes from its last completed step instead of starting over.
03AI workflow generatorSoonDescribe the pipeline in a sentence and get a working workflow you can edit.
04Human approval stepsSoonPause a job mid-flight until a person signs off, then continue from that point.
05Workflow replay & time travelSoonRe-run a failed workflow from any step with the original payload.
06Distributed map/reduceSoonFan one job into thousands of shards and collect the results.
07Multi-region queue replicationSoonQueues mirrored across regions with automatic failover.
08Event busSoonPublish once, let many workers subscribe, without wiring queues by hand.
JOB 09pricing
Basic and Premium worker pricing, in one place.
Choose a Darwa zone to see exact worker plans and usage rates. Queueing, scheduling, retries, and dead-letter handling are included.
Worker plansUSD · per worker · month
Standard
$25.30/ month
2 GB RAM1 vCPU
Pro
$50.62/ month
4 GB RAM2 vCPUs
Pro Plus
$101.23/ month
8 GB RAM2 vCPUs
Pro Max
$202.45/ month
16 GB RAM4 vCPUs
Pro Ultra
$404.90/ month
32 GB RAM8 vCPUs
Usage pricingExact rates for virginia-1
StorageManaged
$0.18/ GB-month
- Included IOPS
- 3,000
- Extra IOPS
- $0.01 / IOPS-mo
- Included throughput
- 125 MB/s
- Extra throughput
- $0.09 / MB/s-mo
Internet transferPer GB
$0.00inbound
- First 10 TB / month
- $0.15
- Next 40 TB / month
- $0.14
- Next 100 TB / month
- $0.12
- Over 150 TB / month
- $0.08
NetworkOptional
Includedshared ingress + TLS
- Dedicated public IPv4
- $6.08 / month
- Same availability zone
- Free
- Cross-zone transfer
- $0.02 / GB each way
- Included outbound
- 0 GB
Get started
Move one queue over and watch the board.
Your worker code does not change. The broker, scheduler, retries, and dead-letter queue stop being your problem.
Background worker FAQ
The practical questions, answered.
What a worker is, how it differs from a cron job, what happens when a job fails, and when a web service is the better fit.
What is a background worker?+
A background worker is a process that runs without receiving public web traffic. It consumes jobs from a queue, reacts to events, or runs on a schedule, which suits work that takes too long for a web request such as sending email, processing uploads, generating reports, or calling a slow third-party API.
How is a background worker different from a cron job?+
A cron job runs on a fixed schedule and exits. A worker stays running and processes jobs as they arrive. Darwa runs both, so a nightly report can use a schedule while an image pipeline consumes a queue continuously.
Do I need to run my own message broker?+
No. Darwa provides managed queues and schedulers, so there is no separate broker to provision, patch, or monitor. Your worker code stays the same; the queue infrastructure is operated for you.
What happens when a job fails?+
Failed jobs are retried automatically. A job that keeps failing after its retries moves to a dead-letter queue, where it is held for inspection instead of being lost or blocking the rest of the queue.
Can workers scale automatically?+
Yes. Queue-depth autoscaling adds worker capacity as a backlog grows and reduces it when the queue drains, so a traffic spike does not require manual intervention. Concurrency controls cap how many jobs run at once when a downstream API has its own rate limit.
Can I see why a specific job failed?+
Yes. Per-job logs and monitoring show each run and its output, so a failure can be traced to the job that caused it rather than to a single combined log stream.
Should I use a background worker or a web service?+
Use a web service for anything that answers an HTTP request, and a background worker for work that should not receive public traffic. Most applications run both: the web service accepts the request and enqueues the job, and the worker performs it.
Where do workers store their state?+
Workers are for processing rather than durable storage, so pair them with a Darwa managed database or cloud storage bucket for state that has to outlive a job.
What does background worker hosting include?+
Background worker hosting runs non-HTTP processes with managed scheduling, queueing, retries, concurrency limits, logs, and queue-depth autoscaling. It is a job queue platform for teams that want to run background jobs in the cloud without operating brokers and worker servers themselves.
Can I use Darwa for Celery worker hosting or BullMQ queue hosting?+
Yes. Celery worker hosting connects Python workers to the managed queue, while BullMQ queue hosting connects Node.js workers through a Redis-compatible endpoint. Existing Sidekiq, RQ, Asynq, Oban, and Laravel Queue workers can use the same model.
Is cron job hosting different from scheduled task hosting?+
They describe the same core need: run a command on a defined schedule. Darwa cron job hosting accepts cron expressions or readable intervals, while scheduled task hosting adds the same logs, retry behavior, alerts, and durable execution history used by other worker jobs.