How to Run Background Workers on a VPS

You can run background workers on the same VPS as your web app, reliably and for free, without a managed worker platform. A worker is just a long-running process that pulls jobs off a queue and does the slow work: sending email, generating reports, resizing images, calling a slow API. Run it as a systemd service or a Compose service next to your web app, give it restart-on-failure, and scale it by running more copies. I run every product's workers this way. It is simpler than the managed pitch and it costs nothing extra.

Do I need a managed worker service?

No, not for a small or mid-size product. Managed worker platforms sell you autoscaling and a dashboard, but a background worker is fundamentally a process in a loop. It connects to your queue, blocks until a job arrives, does the work, acknowledges the job, and repeats. That loop runs perfectly well on a box you already own, sitting right next to the database and the queue it talks to.

The colocation is a feature. Your worker reaches the queue and the database over the private network in a fraction of a millisecond, instead of a round trip to a managed endpoint. And you are not paying a second vendor for compute you already have idle on your VPS. This is the same argument I make for running the job queue itself yourself instead of SQS: the whole pipeline lives on one box you control.

Run the worker as a supervised process

The one rule for a worker: it must restart when it dies, because it will die. A bad job, a memory leak, a dropped connection. You do not want a crashed worker to just stop processing jobs silently.

Two clean ways to supervise it:

• As a Compose service, use the same image as your web app with a different command, and set restart: unless-stopped. Docker brings it back after a crash or a reboot.
• As a systemd service, write a unit with Restart=on-failure and RestartSec=5. systemd captures its logs to the journal and restarts it on crash. This pairs naturally with the timer approach in reliable cron jobs on a VPS.

Either way the worker self-heals. Add a memory limit so a leaking worker gets killed and restarted before it starves the box, rather than taking the whole machine down with it.

How do I scale background workers?

Horizontally, by running more copies. This is the quiet advantage of a queue-based design: workers are stateless consumers, so ten workers pulling from the same queue process ten times the throughput with no coordination. With Compose you set the replica count. With systemd you use a templated unit and start worker@1 through worker@4.

Scale the count to your workload, not to a fantasy. Watch the queue depth. If jobs are piling up faster than they clear, add workers or a bigger box. If the queue is usually empty, you have too many. Concurrency inside each worker is a second lever: for I/O-bound jobs like API calls, one worker can handle many jobs at once; for CPU-bound jobs, match worker count to cores. Get the queue depth on a dashboard so this is a decision, not a guess, using the same self-hosted monitoring in self-host monitoring instead of Datadog.

Make jobs safe to retry

Workers fail mid-job, and the queue will redeliver. So every job has to be safe to run twice. Design jobs to be idempotent: check whether the work is already done before doing it, use a unique key so a retried "send invoice" job does not send two. This single discipline turns "worker crashed halfway" from a data-corruption incident into a non-event, because the redelivered job simply finishes what the first one started.

Handle the poison job too. A job that fails every time will loop forever and block the queue. Cap retries and move persistent failures to a dead-letter queue you can inspect, instead of letting one bad job jam the pipeline. Log what each job did, not just that it ran, so when something looks wrong weeks later you have a trail.

That is the whole system: supervised processes, more copies to scale, idempotent jobs, a dead-letter queue for the stubborn ones. It runs on the HostSSH box you already have, with no worker vendor and no extra bill. Fold the worker units into your standard box setup the way I describe in one runbook for twenty apps, and every product gets reliable background processing by default.