feat(admin): add Backup and Example-Backup-With-Borg

Administrator-Guide.md

@@ -8,6 +8,7 @@ recommended.
 - [[Release Process]]
 - [[Installation]]
 - [[Configuration]]
+- [[Backup]]
 - [[Upgrading]]
 - [[Troubleshooting]]
 - [[Reverse Proxy]]

Backup.md

@@ -0,0 +1,39 @@
+# Backup
+
+All TfStated data is stored in its SQLite database. Backing up this database can
+be achieved through different means.
+
+## VACUUM INTO
+
+SQLite has a `VACUUM` command that is used to optimize the structure of the
+database. It also has the side effect of shrinking its size by reclaiming disk
+space that is no longer used by table data.
+
+`VACUUM INTO` is a variant that runs this process without interfering with other
+writes and also outputs a perfect working copy of your database at the point in
+time you run this command. Everything is kept as is, including the `PRAGMA` and
+indexes. You can then backup this output file. Example:
+
+``` sql
+umask 077
+echo -n "VACUUM INTO '/tmp/tfstated.db';" | sqlite3 /var/lib/tfstated/tfstated.db
+```
+
+This is the recommended way to backup your TfStated database as you can pair it
+up with the backup software of your choice (for example
+[borg](./Example-Backup-With-Borg)).
+
+Another option is to output this backup file
+directly to a remote storage mounted on your server.
+
+## Litestream
+
+[Litestream](https://litestream.io/) is a background service that continuously
+monitors and replicates an SQLite database to an S3 compatible object storage.
+
+Litestream is a good backup solution that has the advantage of providing point
+in time recovery. The downsides are that it is another service to manage and
+that it requires a write lock on the database when its checkpointing occurs.
+
+Since Litestream is more complex to operate, the TfStated project recommends to
+pair this solution with a `VACUUM INTO` style backup.

Example-Backup-With-Borg.md

@@ -0,0 +1,86 @@
+# Example Backup with Borg
+
+Here is a complete example of how to backup a `/var/lib/tfstated/tfstated.db`
+SQLite database file using [borg](https://www.borgbackup.org/) on a Debian 12
+bookworm server using a bash script, a systemd service and a systemd timer.
+
+### Script
+
+The `/etc/borg/tfstated.sh` script should belong to `root:root` with 0500
+permissions (`r-x------`):
+
+``` shell
+#!/usr/bin/env bash
+set -euo pipefail
+
+archiveSuffix=".failed"
+
+# Run borg init if the repo doesn't exist yet
+if ! borg list > /dev/null; then
+    borg init --encryption none
+fi
+
+archiveName="tfstated-sqlite3-$(date +%Y-%m-%dT%H:%M:%S)"
+rm -f /tmp/tfstated.db; umask 077; printf '%s' "VACUUM INTO '/tmp/tfstated.db'" \
+   | sqlite3 /srv/tfstated/sqlite.db
+borg create \
+     --compression auto,zstd \
+     "::${archiveName}${archiveSuffix}" \
+     /tmp/tfstated.db
+rm -f /tmp/tfstated.db
+borg rename "::${archiveName}${archiveSuffix}" "${archiveName}"
+borg prune \
+     --keep-daily=14 --keep-monthly=3 --keep-weekly=4 \
+     --glob-archives '*-tfstated-sqlite3-*'
+
+borg compact
+```
+
+Please change the destination hostname and retention options to your liking. You
+can also encrypt your borg backups for additional security, but remember that
+your OpenTofu/terraform states are already encrypted at rest in the SQLite
+database.
+
+### Systemd service
+The `/etc/systemd/system/borg-job-tfstated.service` systemd service file should
+belong to `root:root` with 0444 permissions (`r--r--r--`):
+
+``` ini
+[Unit]
+Description=BorgBackup job tfstated
+
+[Service]
+Environment="BORG_REPO=ssh://borg@myth.adyxax.org/srv/borg/tfstated"
+Environment="BORG_RSH=ssh -i /etc/borg/tfstated.key -o StrictHostKeyChecking=accept-new"
+CPUSchedulingPolicy=idle
+ExecStart=/etc/borg/tfstated.sh
+Group=root
+IOSchedulingClass=idle
+PrivateTmp=true
+ProtectSystem=strict
+ReadWritePaths=/root/.cache/borg
+ReadWritePaths=/root/.config/borg
+User=root
+```
+
+This service file uses environment variables to pass information about the
+`BORG_REPO` and the `BORG_RSH` command to use. Change them to your liking.
+
+### Systemd timer
+
+The `/etc/systemd/system/borg-job-tfstated.timer` systemd timer file should
+belong to `root:root` with 0444 permissions (`r--r--r--`):
+
+``` ini
+[Unit]
+Description=BorgBackup job tfstated timer
+
+[Timer]
+FixedRandomDelay=true
+OnCalendar=daily
+Persistent=true
+RandomizedDelaySec=3600
+
+[Install]
+WantedBy=timers.target
+```