onboard any rl server in six steps
How to add a new RogerLe.com server to the shared encrypted backup repo. Every onboarded host pushes snapshots to one repo on backups.rogerle.com, deduplicated across hosts and tagged by hostname. The pilot lives on Sonda.
~5 minutes per server. No restic init — the repo already exists.
rest-server append-only mode to bound that.
/root/.restic/password available somewhere you can copy from — pull from any already-onboarded host, or paste from the 1Password vault.restic on AlmaLinux 8 / 9 / 10).backups.rogerle.com./root/.restic/password on every onboarded host AND Roger's 1Password vault. Don't reduce that to one place.
The install path depends on whether you're on AlmaLinux 9/10 (EPEL has it) or AlmaLinux 8 (EPEL does not have it — use the official static binary from GitHub instead). Both paths land the same working tool, and both interop on the same repo because the on-disk format is stable across versions.
AlmaLinux 9 / 10 — from EPEL
# EPEL may already be enabled — this is idempotent
dnf install -y epel-release
dnf install -y restic
restic version
AlmaLinux 8 — official static binary from GitHub
# restic is a single statically-linked Go binary — no dependencies. # Same binary works on AL8/9/10. Check latest version at github.com/restic/restic/releases. VER=0.18.1 curl -L "https://github.com/restic/restic/releases/download/v${VER}/restic_${VER}_linux_amd64.bz2" \ -o /tmp/restic.bz2 bunzip2 /tmp/restic.bz2 install -m 0755 /tmp/restic /usr/local/bin/restic /bin/rm /tmp/restic restic version
dnf search restic on AlmaLinux 8.10 with EPEL enabled returns "No matches found." The GitHub binary is the official distribution path for systems without a package — that's what the restic project documents.
Restic talks to the backups VPS over SFTP. The new host needs passwordless SSH into the backups account. Create a root key if one doesn't exist, then push the public half.
# Generate a root key if /root/.ssh/id_ed25519 doesn't exist: [ -f /root/.ssh/id_ed25519 ] || ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519 # Push the pubkey to the backups@ account (one-time password prompt): ssh-copy-id backups@backups.rogerle.com # Verify passwordless SSH works: ssh backups@backups.rogerle.com 'echo OK'
ssh-copy-id isn't allowed by the destination's sshd_config (some hardened boxes block password auth entirely), fall back to: cat /root/.ssh/id_ed25519.pub | ssh -o PreferredAuthentications=publickey backups@backups.rogerle.com 'cat >> ~/.ssh/authorized_keys' from a host that already has access.
The 65-byte password file is identical on every onboarded host. Pull it from Sonda (or any other onboarded host), or paste from the 1Password vault.
mkdir -p /root/.restic chmod 700 /root/.restic scp sonda.rogerle.com:/root/.restic/password /root/.restic/password chmod 600 /root/.restic/password
Or — if scp from Sonda isn't reachable from this host — vi /root/.restic/password, paste the 64 base64 characters + newline from 1Password, then chmod 600.
One script, host-agnostic — it reads hostname -f at runtime for the --host and --tag. Same file works everywhere.
mkdir -p /root/scripts scp sonda.rogerle.com:/root/scripts/backup.sh /root/scripts/backup.sh chmod 750 /root/scripts/backup.sh
#!/bin/bash
# /root/scripts/backup.sh — restic backup driver for RogerLe.com servers
set -euo pipefail
export RESTIC_REPOSITORY="sftp:backups@backups.rogerle.com:/home/backups/restic-rl"
export RESTIC_PASSWORD_FILE="/root/.restic/password"
HOST="$(hostname -f)"
LOG=/var/log/rl-backup.log
exec >>"$LOG" 2>&1
echo "=== $(date -Iseconds) backup start host=$HOST ==="
restic backup \
--host "$HOST" \
--tag "$HOST" \
--tag nightly \
--exclude='/root/.cache' \
--exclude='/home/*/.cache' \
--exclude='/var/log/journal' \
--exclude='/var/log/btmp*' \
--exclude='/var/log/wtmp*' \
--exclude='/var/lib/mysql' \
/etc \
/root \
/home \
/var/log
echo "=== $(date -Iseconds) backup done host=$HOST ==="
Three jobs: nightly backup, weekly forget+prune, monthly integrity check. The cron file scoping (--host "$(hostname -f)") makes prune and forget operate only on this server's snapshots, but the file ships with Sonda's time slots — every other host must shift its three minute fields to a unique slot from the fleet schedule below.
# Step 5a — drop the cron file (ships with Sonda's slot baked in) scp sonda.rogerle.com:/etc/cron.d/restic-backup /etc/cron.d/restic-backup # Step 5b — shift the three minute fields to this host's assigned slot, # picking the row for this host from the table below. Example for forum: sed -i 's|^30 3 |15 4 |' /etc/cron.d/restic-backup # backup → 04:15 sed -i 's|^0 5 |45 5 |' /etc/cron.d/restic-backup # prune → 05:45 Sun sed -i 's|^0 6 1 |15 7 1 |' /etc/cron.d/restic-backup # check → 07:15 on the 1st systemctl reload crond
Fleet schedule (15-min stagger). All three windows are staggered because the backups VPS is HDD-backed (concurrent writes serialize on seek), and forget --prune takes an exclusive repo lock that collides if two hosts run simultaneously:
| Host | Backup (daily) |
Prune (Sundays) |
Check (1st of month) |
|---|---|---|---|
| sonda | 03:30 | 05:00 | 06:30 |
| admin (ns1+rldns+status, DediRock) | 03:45 | 05:15 | 06:45 |
| ns2 (serverroom.net) | 04:00 | 05:30 | 07:00 |
| forum.slbenfica.org (RackNerd) | 04:15 | 05:45 | 07:15 |
| directadmin (RackNerd) | 04:30 | 06:00 | 07:30 |
| corp / camperbudget (Hetzner — pending) | 04:45 | 06:15 | 07:45 |
Backup window ends ~04:50 (last host done). Prune window ends ~06:20. Check window ends ~07:50. No collisions even on the rare Sunday-falls-on-the-1st months. If you add a new host later, take the next 15-min slot in each column.
mysql-backup cron.daily job (which typically fires around 03:11 via anacron)Run the script directly to seed the first snapshot and verify everything works before cron does it for you.
/root/scripts/backup.sh tail -20 /var/log/rl-backup.log
You should see lines like Added to the repository: X.XX MiB (Y.YY MiB stored) — the stored number is the deduplicated delta. On a fresh AL9 host where Sonda has already been onboarded, expect the first snapshot to be +3–5GB of net storage even if the host is 30GB total; that's the cross-host dedup magic. First backup runtime is typically 30 seconds to a few minutes depending on data volume.
Confirm the new host appears in the repo and that snapshots filter cleanly by hostname.
# This host's snapshots only: restic snapshots --host "$(hostname -f)" # All hosts in the repo (you should see Sonda + this new host): restic snapshots # Repo size and dedup stats: restic stats --mode raw-data # Optional: a deep integrity check (takes a few minutes on a large repo): restic check
If restic snapshots needs the env-vars set (it does, when run outside the script):
export RESTIC_REPOSITORY="sftp:backups@backups.rogerle.com:/home/backups/restic-rl" export RESTIC_PASSWORD_FILE="/root/.restic/password"
The restore mechanic is the same on every host. List snapshots, pick one by ID, restore to a target directory. Round-trip is byte-identical — verified on the Sonda pilot.
# List this host's snapshots, latest first: restic snapshots --host "$(hostname -f)" # Restore a single file or directory to /tmp/recover: restic restore latest --host "$(hostname -f)" \ --target /tmp/recover \ --include /home/sonda.rogerle.com/www/index.php # Restore an entire site: restic restore latest --host "$(hostname -f)" \ --target /tmp/recover \ --include /home/photos.rogerle.com # Restore from a specific older snapshot by ID: restic restore a1b2c3d4 --target /tmp/recover --include /etc/nginx
/tmp/recover first, then move files into place after inspection. Direct restore --target / overwrites live files and is almost never what you want.
Backups VPS
192.227.170.94 · DediRock · sshd-only · 2GB / 1TB HDDMode
600 · root:root · 65 bytes (64 base64 + newline)Log file
/var/log/rl-backup.logMail target
root (cron MAILTO)Notably excluded
/var/lib/mysql (live), /var/log/journal, *.cachePer-host scope
forget runs with--host $(hostname -f)
/var/lib/mysql live produces corrupt restores. The existing /etc/cron.daily/mysql-backup writes mysqldump output under /root/mysql/sql-dumps/ (or similar — check the host) which gets picked up via /root. If a host has dumps somewhere else, audit before assuming.restic version first. Don't mix repo writes from a pre-0.10 client (you won't on EPEL 8/9/10, but worth knowing)./root/.forward or postfix relay. Silent integrity-check failures are how repos rot.hostname -f. If a server's FQDN changes, old snapshots stay tagged with the old name. restic tag --set new.name --host old.name can rewrite them; do this if you ever rename a box.restic init on a new host. The shared repo already exists. init would create a second repo at the same path — restic stops you, but if it didn't, you'd encrypt over the existing one with a new key.restic backup at the same instant against the same repo is safe (restic uses repo-side locks), but stagger the cron times if you have many hosts to keep one slow run from blocking the others.Five of six active RL servers are onboarded. One pending — the consolidated corp/camperbudget Hetzner box, after Mudança's migration lands on June 12. rl.rogerle.com is being cancelled (content already rebuilt on Sonda) and is not onboarded.
04:45 / 06:15 / 07:45.Today, every onboarded host shares one password and write access to the entire repo. If any host gets root-compromised, the attacker can restic forget --keep-last 0 --prune and erase every host's history. Defense against that is append-only mode: replace the sftp backend on backups.rogerle.com with the rest-server binary in append-only mode. New snapshots write fine; forget and prune become impossible from a compromised host. A separate maintenance key on a non-prod machine handles pruning out-of-band.
Worth doing before the fleet is fully onboarded. Estimated ~1 hour: install rest-server on the backups VPS, switch the systemd unit, change every host's RESTIC_REPOSITORY URL from sftp:... to rest:https://....