The self-hosted backup stack that actually restores

The short version: dump the databases, back up with restic, verify the bytes you wrote, and drill a restore every month. Every number below comes from a run on my own machine.

Five thousand rows in a Vaultwarden-shaped SQLite database. The file that was supposed to contain them: 4,096 bytes.

Not a corruption bug. SQLite in WAL mode, doing exactly what it is designed to do. The table lived in the write-ahead log, the main db.sqlite3 held almost nothing, and a backup that copies only the main file gets a database with no schema in it. I reproduced the setup on purpose: a SQLite database in WAL mode with 5,000 rows inserted and the connection held open. Then I copied the main database file, the way a naive backup script would, opened the copy, and got this:

sqlite3.OperationalError: no such table: t

Five thousand rows, absent from a copy that looked perfectly healthy. The file existed. The file had a plausible size for an SQLite header page. Every “back up your appdata folder” guide on the internet would have told me I was fine.

I built this stack because I got tired of that class of failure. Not disk failures. Those are rare and loud. The failures that actually eat homelab data are quiet: a copy taken at the wrong moment, a dump that silently wrote 0 bytes, a repository with one damaged pack file that nobody read, a restore script that reports success while skipping a file.

This is the whole thing, end to end, with the numbers from my own run: about 25 minutes of setup and an hour a month of babysitting.

The bench

Everything below was run in a Debian 12 container on an Unraid host: 20 cores, 31 GiB of RAM, restic 0.18.0, SQLite 3.40.1, PostgreSQL 15.

The test data was 162 MiB of pretend appdata — 300 photos, one 80 MiB video, 150 PDFs, a 5,000-row SQLite vault database, a 20,000-row Postgres table, four compose files, and 200 thumbnail files that should never be in a backup. That is roughly the shape of a small homelab: mostly media, a little configuration, and two databases that matter more than all the rest combined.

I am not going to pretend this is a 2 TB Immich library. It is not, and read-verification timings scale with bytes, not with my optimism. I will flag where the numbers will move.

Three ways a backup lies to you

It was never restorable. The backup job runs, the log says “completed”, the folder has files in it. Nobody has ever restored it, so nobody knows that the archive is missing the one directory that matters, or that the database inside it is an empty stub. Most common failure, easiest to fix: restore something on a schedule and check the result.

It captured a moving target. Copying a live database file is like photocopying a book while someone rips pages out. Sometimes you get lucky. With WAL mode you usually do not. The fix is a dump through the database’s own consistent-snapshot mechanism, before the file-level backup runs.

The bytes rotted and nothing read them. Silent bit rot, a bad USB cable, a half-finished upload. restic stores checksums for everything, but the default restic check only validates structure and metadata. It does not read your data. In my test, a repository with a deliberately flipped byte in a pack file passed restic check with exit code 0 and the message “no errors were found”. Only --read-data-subset caught it. More on that in a moment, because it is the most important paragraph in this article.

The stack

Four pieces, in this order:

apps writing to /mnt/user/appdata
          |
          v
[1] pre-backup dump      sqlite3 .backup / pg_dump -Fc  ->  /mnt/user/dumps
          |
          v
[2] restic backup        appdata (minus caches) + dumps  ->  local repo
          |
          v
[3] restic check         structure nightly, 5% of data read weekly
          |
          v
[4] restore-verify.sh    restore to a temp dir, compare checksums, open the DBs

Piece 1 is the part everyone skips. Pieces 2 and 3 are commodity. Piece 4 is the only part that tells you the truth.

Why restic rather than Borg, Kopia, Duplicati or Veeam? Restic is a single static binary, it deduplicates across snapshots, it compresses by default since 0.15, it writes to almost any backend without a helper daemon, and — the part I care about — it fails loudly and can verify what it wrote. Borg is excellent and faster on a local repo, but it wants a Borg binary on the remote if you use one. Kopia has a decent GUI and a policy system; if you want a UI, use Kopia and keep everything else in this guide. Duplicati’s database-backed model has a long history of needing that database to restore, which is a dependency I do not want in a disaster.

If you already run Borg or Kopia, the structure below still applies. The dump step and the drill step are not restic-specific, and they are the two steps that matter.

Where the bytes go

The 3-2-1 rule is three copies, two media, one off-site. My mapping for a small homelab:

The third copy is the one people skip and the only one that survives your house. B2 list pricing when I last looked was about $6 per TB per month with free egress up to three times your stored volume; check the current page before you budget, because object storage pricing moves and I am not going to quote a number I cannot re-verify here.

One rule for the off-site copy: use an application key scoped to a single bucket with write and list permission only, no delete. Restic can then add snapshots but cannot remove them, and neither can anything that gets root on your server. Pruning that repo becomes a manual, deliberate, occasional job run from somewhere else. That is the correct amount of friction for the copy that saves you from ransomware.

Build the repository

export RESTIC_REPOSITORY=/mnt/user/backups/restic
export RESTIC_PASSWORD_FILE=/root/.restic-pass

openssl rand -base64 24 > /root/.restic-pass
chmod 600 /root/.restic-pass
restic init

On my bench, restic init took 2.9 seconds. The password file is the single point of failure for everything else: if you lose it, the repository is noise. Put a copy in your password manager, and a copy on paper somewhere that is not the same building as the server. Do not put it in the same restic repository. I have seen that suggestion in a forum thread and it is exactly as clever as locking the spare key inside the car.

The repository is a directory of content-addressed packs. It is not a folder you can browse with ls and copy files out of. That is the trade you make for deduplication and verifiable integrity.

Dump the databases first

This is the step that separates a backup from a copy.

SQLite apps: Vaultwarden, Mealie, Paperless on older installs

sqlite3 .backup uses SQLite’s online backup API. It reads a consistent snapshot while the app keeps writing. Run it against the live file — do not stop the container first, and do not cp the file.

My first attempt at this failed in a way worth showing:

==> sqlite dump: /appdata/vaultwarden/data/db.sqlite3 -> /dumps/vaultwarden/db.sqlite3
Error: database is locked

The app was mid-transaction. sqlite3’s shell command has no busy timeout by default, so .backup gave up instantly. The fix is one flag and a retry:

sqlite3 -cmd ".timeout 8000" "$db" ".backup '$out'"

With the timeout set and three attempts, the same dump succeeded while a writer was inserting a row every 20 milliseconds. The dump passed PRAGMA integrity_check and contained 5,278 rows — the 5,000 originals plus 278 that the fake app wrote during the window.

That number is the whole argument for this approach. A file copy would have captured a torn view of the database. A dump captured a transactionally consistent point in time, including writes that happened during the dump, and the row count proves the movement was real.

PostgreSQL apps: Immich, Paperless-ngx, Nextcloud with pgsql

pg_dump -Fc --no-owner --no-privileges -d "$PG_DSN" -f "$DUMPDIR/immich/immich.dump"

-Fc is the custom format: compressed, single file, restorable with pg_restore in parallel and with --clean --if-exists when you need to overwrite. Plain SQL dumps are fine for tiny databases and painful for big ones.

Against a 20,000-row table with an index and a jsonb column, pg_dump took 66 ms and produced 627,714 bytes. Restoring it into a fresh database returned 20,000 rows and an identical maximum timestamp.

Two things bit me, both ordinary:

• The postgres user could not write to my dump directory. In production this is a bind mount with the right ownership, or a --user on the dump command. If you run pg_dump from a systemd unit as root against a containerised Postgres, you will hit a permissions variant of this. Dump to a path the database user owns, then move it.
• pg_restore needs read access to the file too. My drill first failed on this, which is a good argument for the drill: the failure was in the recovery path, not the backup path, and it would have surfaced at 3 a.m. on the worst day.

One caveat I cannot test on this bench: a real Immich database carries the pgvecto.rs (or pgvector) extension and its types. pg_restore into a database without that extension installed will fail on the first vector column. Install the extension in the target database before restoring — in Immich’s case, start the stack once with an empty database so its init scripts create the extension, then restore over it.

The dump hook is short enough to keep in one file. I have put the tested version up as a download: pre-backup-dump.sh. It loops over known SQLite paths, skips the ones that do not exist, runs the integrity check on every dump, and exits non-zero if anything is wrong.

The backup command

restic backup /mnt/user/appdata /mnt/user/dumps \
  --exclude '**/thumbs' \
  --exclude '**/cache' \
  --exclude '*.tmp' \
  --tag nightly \
  --host bacchus

Two paths, one snapshot. That matters later: on restore you get appdata and dumps back together, and you know they came from the same moment.

The excludes are doing more work than they look like. My test set had 200 thumbnail files totalling 12 MiB, and every one of them is regenerable. On a real Immich install the thumbs directory is not small, and it does not matter exactly how big it is, because every byte in it can be regenerated. Frigate’s cache is worse. Excluding it cuts both backup time and storage, and costs you nothing at restore time except a re-render.

Real numbers from the first run:

Files:         458 new,     0 changed,     0 unmodified
Dirs:           19 new,     0 changed,     0 unmodified
Data Blobs:    531 new
Tree Blobs:     20 new
Added to the repository: 151.732 MiB (146.101 MiB stored)
processed 458 files, 153.068 MiB in 0:01

1.4 seconds wall clock, 146 MiB stored for 153 MiB of data. Compression saved 4.83%, and that is with a dataset deliberately hard to compress: random bytes standing in for JPEGs and video, which are already compressed. Real photo libraries compress even worse. Text-heavy appdata (configs, XML, JSON, SQL dumps) does much better.

The second run, after adding one 200 KiB photo and editing one compose file:

Files:           1 new,     2 changed,   456 unmodified
Added to the repository: 1.990 MiB (246.222 KiB stored)
processed 459 files, 153.396 MiB in 0:00

Two-thirds of a second, and the repository grew by a quarter of a megabyte instead of 153 MiB. That is deduplication doing its job: unchanged files are not re-uploaded, they are referenced. This is why a restic repository stays small even with 30 daily snapshots of the same data.

If you are backing up to B2 over a home connection, the first run is the painful one. 153 MiB took 1.4 seconds locally; 1 TB of photos on a 20 Mbit upload is about five days. Start the off-site copy before you need it, and stagger it so it does not fight your nightly local backup for disk I/O.

Retention that does not eat the disk

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

That policy keeps a week of dailies, a month of weeklies and half a year of monthlies: 17 snapshots at most, and because of deduplication each one costs roughly the size of the change since the last, not the size of the dataset.

In my run, forget applied the policy to two snapshots and kept both — one as the oldest daily, oldest weekly and oldest monthly, the other under all three categories — then --prune rewrote the repository packs. The repo stayed at 147 MiB because both snapshots referenced the same data. On a repository where the data has actually changed, prune takes real time and real disk: it reads every pack that contains data no snapshot references any more and writes new packs. On a 1 TB repo, I would budget a few hours the first time (an estimate, not a measurement — my repo was 147 MiB).

Schedule it weekly, not nightly. Pruning on every backup is wasted I/O and it raises the window where an interrupted prune leaves you with temporary packs to clean up.

Retention is a business decision disguised as a technical one. Saving six months of monthlies is cheap in restic terms and valuable when you discover in March that a file went missing in January. Saving 30 dailies of a photo library is mostly pointless: the files do not change. My split for a homelab: 7 dailies, 4 weeklies, 6 monthlies for appdata and dumps; a separate policy for media that keeps 14 dailies and then monthlies forever, because photos are not regenerable and forget policies are where people lose them. Read the forget output before you add --prune. Every deleted snapshot prints what it is deleting and why.

Verification that means something

restic has two levels of checking and the difference is the whole point.

restic check reads the index, walks the trees and confirms that every snapshot points at blobs that exist. It does not read the blob contents. It is fast and it catches structural damage.

restic check --read-data-subset=5% reads 5% of your actual data and verifies the cryptographic hash of every blob it touches. This is the one that catches bit rot, a failing USB controller, a truncated upload.

I proved the difference on purpose. I copied the repository, flipped one byte at offset 500 of a 17 MB pack file, and ran both:

$ restic check
check all packs
check snapshots, trees and blobs
no errors were found
(exit code 0)

$ restic check --read-data-subset=100%
read 100.0% of data packs
pack 105a9b83fdaab5c16b970fb4b487c086731c5a98859006e25c1de69c64a156be contains 2 errors:
  [blob 67996c6e...: decrypting blob <data/67996c6e> from 105a9b83 failed: ciphertext verification failed
   unexpected pack id 94b66855...]
Fatal: repository contains errors
(exit code 1)

Two things to take from that. First, a nightly restic check that passes tells you almost nothing about whether your data is intact. Second, and worse, look at what a restore from that damaged repository did:

Summary: Restored 477 / 478 files/dirs (152.551 MiB / 153.396 MiB) in 0:00
Fatal: There were 1 errors

It restored 477 files, buried the shortfall in the middle of a long output, and exited 1. If your restore script pipes output to a log and checks nothing, you have a restore that is missing a file and looks finished. The missing file in this case was the 80 MiB video — the largest thing in the dataset, and the thing most likely to be missing in a real failure, because big files get chunked into many blobs and any one of them can rot.

My schedule:

• Nightly: restic check after the backup. Cheap, catches index and tree damage.
• Weekly: restic check --read-data-subset=5% rotating through the repository. Five percent a week covers everything in about five months, and each run costs a fifth of a full read.
• Monthly: full --read-data-subset=100% on the local repository, because it is a local disk read and 153 MiB took 841 ms. On 2 TB, that is a different calculation — run it monthly on the repository you care most about and 10% on the rest.

Both checks take an exclusive lock. Do not run them while a backup is writing; restic will queue behind the lock and your backup will look hung.

The drill

This is the part that turns a pile of scripts into a backup system. Once a month, restore the newest snapshot to a scratch directory, compare it against the source, and open the databases.

The script is restore-verify.sh. The important parts are not the restic calls — those are two lines — but the comparison:

restic restore latest --target "$TARGET" --verify
manifest "$SRC" > /tmp/src.manifest
manifest "$TARGET$SRC" > /tmp/restored.manifest
diff -u /tmp/src.manifest /tmp/restored.manifest

A manifest of path, size and SHA-256 for every file, sorted, compared line by line. Not a file count. Counts hide swaps, truncations and same-size corruption.

The first version of this script failed, and the failure taught me the most useful rule in the whole guide. It compared the live db.sqlite3 against the restored copy and reported a checksum mismatch. Of course it did: the app had written to the live database after the backup finished. A live database file is a moving target by definition, and comparing it to a snapshot is a test that can never pass reliably.

The fix is a deny list. The manifest skips live databases, WAL and journal sidecars, cache directories and temp files, and instead verifies:

• appdata files (everything that should be byte-identical),
• the dump directory (byte-identical, because dumps are written once and never touched again),
• the restored dump opened with SQLite: PRAGMA integrity_check and a row count.

That last step is the one that would have caught the 4,096-byte disaster at the top of this article. Checksums prove the bytes travelled. Opening the database proves the bytes are a database.

The full run on my bench:

==> newest snapshot
783f4566  2026-11-06 22:06:17  bacchus  nightly  153.396 MiB
==> restoring into /root/lab/restore-check
Summary: Restored 478 files/dirs (153.396 MiB) in 0:00
verifying files in /root/lab/restore-check
finished verifying 459 files in /root/lab/restore-check (took 55ms)
==> appdata
    source files:   457
    restored files: 457
    checksums: identical
==> database dumps
    source files:   1
    restored files: 1
    checksums: identical
==> sqlite integrity of restored dump
    integrity_check: ok
    rows: 5278
PASS

2.4 seconds end to end, including restic’s own verification of the restored files. On 153 MiB that is trivial. On 2 TB the restore is the slow part — expect hours over USB, and test it once so you know the number instead of guessing it during an incident.

Then the PostgreSQL half, which is where the drill earns its keep:

restic restore latest --target /restore --verify
createdb immich_drill
pg_restore -d immich_drill --no-owner --no-privileges /restore/dumps/immich/immich.dump
psql -d immich_drill -tAc 'select count(*) from assets'

20,000 rows before, 20,000 rows after, and an identical maximum timestamp on the newest asset. No amount of backup logging gives you that sentence. Only a restore does.

Restore into a throwaway database, never over the live one. The drill is supposed to be boring; restoring over production is how a test becomes an incident.

Recovering one app, end to end

The general shape for a container app:

1. Stop the container. Half-restored databases plus a running app is how you get a second incident.
2. restic restore latest --target /tmp/restore --include /mnt/user/appdata/immich
3. Move the old directory aside (mv immich immich.broken), do not delete it. Disk is cheap, and you may want the logs.
4. Restore the database: pg_restore or copy the verified dump over the SQLite file.
5. Fix ownership: chown -R 99:100 /mnt/user/appdata/immich on Unraid, or the UID/GID from the compose file elsewhere.
6. Start the container, then check the app’s own health page before you tell anyone it worked.

The ownership step is the one people forget, and it produces a container that starts, logs permission errors, and looks like a failed restore when it is a failed chown.

For Immich specifically: the upload directory and the database must come from the same snapshot. If you restore photos from Tuesday and the database from Monday, Immich will show assets whose files are missing and files whose assets do not exist. One snapshot, two paths, restored together — that is why the backup command takes both paths in a single run.

If Immich is the library you are worried about, I wrote the standalone version of this with an external USB disk, the exact restore order, and a test where I pulled the cable mid-write: Immich backup to an external drive.

And a backup drill is worth nothing if Docker will not start on the morning you need it, which is its own failure with its own error strings: Unraid Docker service failed to start. And a password manager is a smaller database with a sharper version of the same trap: Vaultwarden backup and restore is a restore drill where the file everyone copies came back empty.

Vaultwarden is easier and less forgiving. The SQLite dump is the whole state apart from rsa_key.pem and config.json, which live beside it in appdata. Restore all three, keep the same ADMIN_TOKEN, and check that existing clients can still sync before you rotate anything.

Put it on a timer

Three files: the wrapper, a service, a timer. I have put all three in the downloads folder — backup-nightly.sh, restic-backup.service, restic-backup.timer.

The service is deliberately boring:

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
ExecStart=/root/bin/backup-nightly.sh
Nice=10
IOSchedulingClass=idle

EnvironmentFile keeps the repository path and password file out of the unit, so the unit can live in a git repo. Nice and IOSchedulingClass matter more than they look: a backup that saturates the array at 02:30 will make every other container on the box stutter, and on a media server you will notice it in Plex before you notice it in the logs.

The timer:

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true

Persistent=true is the one line I would fight for. If the machine is off at 02:30, systemd runs the job at the next boot instead of skipping the night. Without it, a reboot at the wrong time silently costs you a day of snapshots.

The wrapper exists to do four things in order, and to stop at the first failure:

1. Dump the databases.
2. Run the backup.
3. Run restic check.
4. Ping the dead-man switch — only on a clean run.

Every step is || die, and die writes to the log, pings the /fail endpoint and exits non-zero. That sounds obvious. It is also the thing most cron jobs get wrong: restic backup ... ; restic check in a crontab means the second command’s exit code is the only one that reaches the mailer, and the first failure disappears.

I tested both paths, because a wrapper that has never failed is a wrapper you do not know. First run, with a stub restic on PATH that printed Fatal: unable to open repository and exited 1:

=== FAILURE RUN ===
/root/lab/bin/backup-nightly.sh: line 23: /root/bin/pre-backup-dump.sh: No such file or directory
FAILED: database dump
wrapper exit=1
--- pings received ---
/ping/abc/fail

That output is from the version before I fixed the script’s own path handling, which is a small humiliation I am leaving in because it proves the wrapper stops at the first problem and tells the monitor. Then the success run with the real binary:

--- 2026-11-06T22:10:01+00:00 starting
==> sqlite dump: .../vaultwarden/data/db.sqlite3 -> .../dumps/vaultwarden/db.sqlite3
    integrity_check: ok
Files:         460 new,     0 changed,     0 unmodified
Added to the repository: 515.061 KiB (11.749 KiB stored)
processed 460 files, 156.519 MiB in 0:00
snapshot 9ee564cc saved
check all packs
no errors were found
--- 2026-11-06T22:10:03+00:00 OK
wrapper exit=0
--- pings received ---
/ping/abc

Note the line I did not expect: no parent snapshot found, will read all files. My first backup used --host bacchus; the wrapper passes --host "$(hostname -s)", and the container’s hostname is not bacchus, so restic could not find the previous snapshot to diff against. It still stored only 11.7 KiB of new data — deduplication is content-based, not snapshot-based — but it re-read all 156 MiB to work that out. On a local disk that is a second. On 2 TB over USB it is an hour of pointless I/O every night.

Pick one host string and use it everywhere. --host is part of how restic groups snapshots, and inconsistency there is a slow, silent performance bug.

On Unraid, the same wrapper runs from the User Scripts plugin on a cron schedule, or from a small container with /mnt/user/backups and the appdata share bind-mounted read-only. If you use the container route, remember that the container’s hostname changes when it is recreated — set --host explicitly to the Unraid box’s name and do not rely on the default.

One more failure mode worth knowing: restic documents exit code 3 for “some source files could not be read”, distinct from 1 for a fatal error. My wrapper treats anything non-zero as a failure, which is the correct amount of nuance for a homelab. If you have a directory full of files that are permanently unreadable, you will get a nightly alert until you exclude it — which is the behaviour you want from a backup system, even if it is annoying.

Monitoring the thing that matters

A backup that fails silently is worse than no backup, because it buys false confidence. Two signals are worth alerting on:

The backup had to finish. Not “the job started”. restic exits non-zero on any failure, including a partial read. Run it from a systemd timer or cron, and have the wrapper curl a dead-man’s-switch endpoint only on exit 0. Healthchecks.io, Uptime Kuma with a push monitor, or a five-line webhook to anything you already run. If the ping does not arrive in 26 hours, you get an email. That catches the failure mode cron cannot: the timer that stopped firing.

The drill passed. A monthly run of restore-verify.sh, with the same dead-man switch. Twenty-six hours of silence from the drill is also an alert. This is the check that would have caught the 4,096-byte file, and it is the only check that tests the recovery path rather than the backup path.

I would add a third: disk space on the repository, at 80% and 90%. A full repository disk stops backups in a way that is easy to miss for a week.

What this stack does not do

• Bare-metal recovery. It restores files and databases, not a bootable system. Reinstalling the OS and Docker, then restoring appdata, is a two-hour job you should rehearse once. Unraid’s flash drive configuration is a separate backup — grab the flash backup, or zip the config folder monthly.
• Immich’s vector extension, tested. I restored a 20,000-row Postgres table, not a real Immich schema with pgvecto.rs. The mechanism is the same; the extension install order is the part that will bite you first.
• Anything on a remote backend. Every number here is a local filesystem repository. B2 and S3 add latency, retries and a class of error I have not exercised. The restic commands are identical; the timings are not.
• Unraid 7.1 specifics. This ran in a Debian container on Unraid, not against Unraid’s own tooling. The paths I use (/mnt/user/appdata, /mnt/user/backups) match Unraid’s layout, but I have not tested the Unraid appdata backup plugin alongside this, and you should not run both against the same directories without thinking about locks.
• Ransomware-proofing by default. That requires the append-only off-site key described above. A repository mounted read-write from the same server that gets encrypted is not a backup.

What it costs

Ignoring the disk you already own: one external HDD (£60-120 for 2-4 TB when I last looked, one-off) and object storage for the third copy. At 200 GB of appdata and dumps, B2 is a couple of dollars a month at list price. At 2 TB of photos it is about $12 a month, plus egress if you ever restore the lot (free up to three times your stored volume, which a full restore stays inside).

The expensive part is not storage. It is the hour a month you spend reading the drill output. That hour is the product.

Start here

If you do nothing else this week:

1. Install restic, create a repository on a disk that is not your array, and back up appdata once.
2. Add the dump step for your two most important apps. SQLite first, because it is where the silent data loss lives.
3. Run restic check --read-data-subset=5% and read the output.
4. Restore the snapshot to /tmp and diff it against the source. Fix whatever the diff tells you.
5. Put both jobs on a timer with a dead-man switch.

The scripts are downloadable and unglamorous: pre-backup-dump.sh, restore-verify.sh. Change the paths, run them, and keep the output of the first successful drill somewhere you will find it — because the day you need it, you will not want to be reading this article again.

If you want the compose files and Unraid templates from the bench, they are coming in the articles index as the rest of this series lands. The backup pack — dump hooks, drill script, systemd units, and the retention policies I actually use — is the first paid download, and it will be the same code as the free scripts plus the units and the checklist.