Backup architecture

Four independent copies, four different ways to lose them.

sonda.rogerle.com · restic 0.19.1 · reviewed 2026-09-02

This box holds four separate backup legs. They are not redundancy for its own sake: each one survives a failure the others do not. A single shared repo survives a dead server but not a bad passphrase; a local mirror survives the remote host vanishing but not the building it sits in.

Read this first. Until 2026-06-18 there was only leg 1, and an older version of this page documented only that. If you onboard a box from the Repo A steps alone you will build one leg of four and reasonably believe you are finished. That is the failure this page exists to prevent.

The four legs

1

Offsite push — Repo A

/root/scripts/backup.sh · daily 03:30 · survives: this box dying

restic pushes /etc, /root, /home over SFTP to the shared fleet repo, deduplicated across hosts and scoped by hostname.

sftp:backups@backups.rogerle.com:/home/backups/restic-rl
restic backup --host "$HOST" --tag "$HOST" --tag nightly

Retention runs Sundays 05:00: --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --prune, always --host "$(hostname -f)" so one box can never prune another's snapshots. Integrity check 06:30 on the 1st.

2

Local mirror — Repo B

/root/scripts/restic-mirror-pull.sh · daily 04:30 · survives: Repo A vanishing

restic copy pulls Repo A into a second repository on this box's own disk — currently 101 GB under /backup/restic-mirror. Different hardware, different continent, different failure mode.

RESTIC_FROM_REPOSITORY=sftp:backups@backups.rogerle.com:/home/backups/restic-rl
RESTIC_REPOSITORY=/backup/restic-mirror
A mirror-pull is not a backup of this box. It proves someone else's data arrived here. If leg 1 fails while leg 2 succeeds, a naive "last restic activity" check reads green while nothing was backed up. That is why the health feed keys on backup done and deliberately not on mirror-pull done.
3

Porão replica

pushed from Helsinki · daily 04:17 · survives: Helsinki being unreachable

The crew's shared memory. Helsinki pushes the Porão database here nightly; the local MariaDB schema porao (files, file_versions, tokens) backs the dormant failover at mcp.rogerle.net, deployed by Calafate 2026-06-13. Engine lives in /opt/porao.

Lag is up to 24 hours, by design. This is a nightly mirror, not replication. At the time of writing the newest replicated row is 02:18 today — correct, but anything written to Porão since the 04:17 push is not here yet. Failing over does not lose the day's work silently; it loses it loudly, and someone must re-enter it.
4

Encrypted Google Drive

/root/scripts/weekly-gdrive-backup.sh · Sundays 05:30 · survives: losing every RL server at once

tar + zstd of /etc /home /opt /root /srv /var/spool/cron, uploaded through an rclone crypt remote so Google stores ciphertext only. Keeps 6 weekly copies (RETENTION_COUNT=6); the oldest is deleted after a successful upload, never before.

The passphrase is the single point of failure. It is per-server and lives in 1Password. Lose it and the box and the ciphertext is unrecoverable — there is no second copy of the key anywhere. Runbook: Porão projects/rclone.md.

Schedule — do not collide with these

WhenWhatLeg
03:30 dailyrestic backup → Repo A1
04:17 dailyHelsinki → Sonda Porão push (inbound)3
04:30 dailyrestic mirror-pull → Repo B2
05:00 Sunrestic forget + prune1
05:30 Sunencrypted Drive upload4
06:30 monthlyrestic check (integrity)1

Ordering is deliberate: the mirror pulls after the push, and prune runs before the Drive upload so the weekly tarball reflects a pruned repo. Anything new goes outside these windows.

Reading the log

All four legs append to /var/log/rl-backup.log. The success markers are what monitoring keys on, so they matter more than they look:

=== 2026-09-02T03:30:55+01:00 backup done   host=sonda.rogerle.com ===
=== 2026-09-02T04:54:49+01:00 mirror-pull done ===
[2026-08-30T05:37:36+01:00] gdrive-backup: END — success (3.5GB uploaded, ...)

site-health-check --json exposes backups.restic_last and backups.gdrive_last_success from these lines. Drive is weekly — six days old is normal; alert past eight, not past one.

Onboarding a new box (leg 1)

Roughly five minutes. There is no restic init — the repo already exists and initialising over it would be a very bad afternoon.

# EPEL provides restic on AlmaLinux 8 / 9 / 10
dnf install -y epel-release && dnf install -y restic

ssh-copy-id backups@backups.rogerle.com
ssh backups@backups.rogerle.com 'echo OK'

scp sonda.rogerle.com:/etc/cron.d/restic-backup /etc/cron.d/restic-backup
# then MOVE YOUR SLOT so boxes do not all hit the repo at once:
sed -i 's|^30 3 |15 4 |' /etc/cron.d/restic-backup    # backup   -> 04:15
sed -i 's|^0 5  |45 5 |' /etc/cron.d/restic-backup    # prune    -> 05:45 Sun

Then add legs 2–4 as the box warrants. A box holding nothing unique may legitimately stop at leg 1 — but that is a decision to make out loud, not by forgetting.

Verify, do not assume. A backup you have never restored is a hypothesis. Restore one file from each leg after onboarding — restic restore --target /tmp/x --include /etc/hostname latest is enough to prove the repo, the password file and the SFTP path all work together.