Test Your Backups With Restore Drills, Not Faith

A backup you have never restored is not a backup. It is a hope. I run a portfolio of about twenty companies on my own servers, and the single cheapest insurance I buy is a restore drill: I take a real backup, spin up a clean box, and rebuild the app from nothing. If that fails, I would rather learn it on a Tuesday than during an outage. Most people who lose data had backups. They just found out too late that the backups were empty, corrupt, or missing the one thing they needed.

Why do backups fail when you finally need them?

They fail quietly. A cron job breaks and nobody notices because success is silent and only failure should be loud, except nobody wired the alert. The dump runs but excludes a schema you added last quarter. The file lands in a bucket you lost access to. The database was mid-write and the snapshot is inconsistent. None of these show up until you type the restore command with your heart pounding.

The pattern is always the same. Backups get treated as a job that runs, not a result you verify. Running is easy. Restoring is the only test that matches reality. If you have never done the restore, you have tested nothing.

How often should you run a restore drill?

Match the drill to the cost of losing the data. For my revenue-generating apps, I restore monthly. For the one Postgres box that sits under most of the portfolio, I drill more often because the blast radius is larger. A hobby project can go quarterly. The rule is simple: if the thought of restoring makes you nervous, you are overdue.

A drill has three parts.

• Pull the latest backup to a fresh machine, not the production one.
• Rebuild the database and the app from that backup alone, no shortcuts.
• Confirm the app boots, queries return real rows, and a known record is present.

Time it. The number you get is your real recovery time, and it is almost always longer than you assumed. Write it down. That number is what you actually promise your users, whether you meant to or not.

What a good restore drill checks

Restoring the database file is the easy half. The drill has to prove you can rebuild the whole environment, because a database with no app around it is useless.

Check that you still have the schema migrations, the environment variables, the TLS certs, and the exact service versions. I keep secrets managed without a cloud vendor precisely so a rebuild does not depend on some dashboard I might get locked out of. Check that your restore does not silently depend on a machine that no longer exists. The worst backups are the ones that only restore onto the box that died.

Also check the age of the data. A backup that finished twelve hours ago means you lose twelve hours on a bad day. If that is unacceptable, you need more frequent snapshots or point-in-time recovery, not a bigger nightly dump. Deciding that number up front is the whole point of disaster recovery planning when you self-host.

Make the drill boring and automatic

The goal is to remove drama. A drill that requires heroics will not happen twice. Script it. The script provisions a throwaway VPS, pulls the newest backup, runs the restore, boots the app, hits one endpoint, checks one row, and reports pass or fail. Then it destroys the box so the drill costs you an hour of compute and nothing else.

Once it is scripted, schedule it. Now the machine tests the machine, and you only get pinged when something breaks. This is the same reason I own my backups when I self-host: control means nothing if you never exercise it. A backup you own but cannot restore is worse than a managed backup you trust, because you carry the confidence without the proof.

I built and run this on plain VPS boxes through HostSSH, which keeps the drill cheap: a fresh server for an hour costs pennies, and I would rather spend those pennies every month than discover on the worst day of the quarter that my recovery plan was fiction.

Run the drill. Time it. Fix what breaks. Then you own a recovery process instead of a folder full of files you pray still work.