diff --git a/Administrator-Guide.md b/Administrator-Guide.md index d203d01..e9e7060 100644 --- a/Administrator-Guide.md +++ b/Administrator-Guide.md @@ -8,6 +8,7 @@ recommended. - [[Release Process]] - [[Installation]] - [[Configuration]] +- [[Backup]] - [[Upgrading]] - [[Troubleshooting]] - [[Reverse Proxy]] diff --git a/Backup.md b/Backup.md new file mode 100644 index 0000000..82932b7 --- /dev/null +++ b/Backup.md @@ -0,0 +1,39 @@ +# Backup + +All TfStated data is stored in its SQLite database. Backing up this database can +be achieved through different means. + +## VACUUM INTO + +SQLite has a `VACUUM` command that is used to optimize the structure of the +database. It also has the side effect of shrinking its size by reclaiming disk +space that is no longer used by table data. + +`VACUUM INTO` is a variant that runs this process without interfering with other +writes and also outputs a perfect working copy of your database at the point in +time you run this command. Everything is kept as is, including the `PRAGMA` and +indexes. You can then backup this output file. Example: + +``` sql +umask 077 +echo -n "VACUUM INTO '/tmp/tfstated.db';" | sqlite3 /var/lib/tfstated/tfstated.db +``` + +This is the recommended way to backup your TfStated database as you can pair it +up with the backup software of your choice (for example +[borg](./Example-Backup-With-Borg)). + +Another option is to output this backup file +directly to a remote storage mounted on your server. + +## Litestream + +[Litestream](https://litestream.io/) is a background service that continuously +monitors and replicates an SQLite database to an S3 compatible object storage. + +Litestream is a good backup solution that has the advantage of providing point +in time recovery. The downsides are that it is another service to manage and +that it requires a write lock on the database when its checkpointing occurs. + +Since Litestream is more complex to operate, the TfStated project recommends to +pair this solution with a `VACUUM INTO` style backup. diff --git a/Example-Backup-With-Borg.md b/Example-Backup-With-Borg.md new file mode 100644 index 0000000..2ab6bf7 --- /dev/null +++ b/Example-Backup-With-Borg.md @@ -0,0 +1,86 @@ +# Example Backup with Borg + +Here is a complete example of how to backup a `/var/lib/tfstated/tfstated.db` +SQLite database file using [borg](https://www.borgbackup.org/) on a Debian 12 +bookworm server using a bash script, a systemd service and a systemd timer. + +### Script + +The `/etc/borg/tfstated.sh` script should belong to `root:root` with 0500 +permissions (`r-x------`): + +``` shell +#!/usr/bin/env bash +set -euo pipefail + +archiveSuffix=".failed" + +# Run borg init if the repo doesn't exist yet +if ! borg list > /dev/null; then + borg init --encryption none +fi + +archiveName="tfstated-sqlite3-$(date +%Y-%m-%dT%H:%M:%S)" +rm -f /tmp/tfstated.db; umask 077; printf '%s' "VACUUM INTO '/tmp/tfstated.db'" \ + | sqlite3 /srv/tfstated/sqlite.db +borg create \ + --compression auto,zstd \ + "::${archiveName}${archiveSuffix}" \ + /tmp/tfstated.db +rm -f /tmp/tfstated.db +borg rename "::${archiveName}${archiveSuffix}" "${archiveName}" +borg prune \ + --keep-daily=14 --keep-monthly=3 --keep-weekly=4 \ + --glob-archives '*-tfstated-sqlite3-*' + +borg compact +``` + +Please change the destination hostname and retention options to your liking. You +can also encrypt your borg backups for additional security, but remember that +your OpenTofu/terraform states are already encrypted at rest in the SQLite +database. + +### Systemd service +The `/etc/systemd/system/borg-job-tfstated.service` systemd service file should +belong to `root:root` with 0444 permissions (`r--r--r--`): + +``` ini +[Unit] +Description=BorgBackup job tfstated + +[Service] +Environment="BORG_REPO=ssh://borg@myth.adyxax.org/srv/borg/tfstated" +Environment="BORG_RSH=ssh -i /etc/borg/tfstated.key -o StrictHostKeyChecking=accept-new" +CPUSchedulingPolicy=idle +ExecStart=/etc/borg/tfstated.sh +Group=root +IOSchedulingClass=idle +PrivateTmp=true +ProtectSystem=strict +ReadWritePaths=/root/.cache/borg +ReadWritePaths=/root/.config/borg +User=root +``` + +This service file uses environment variables to pass information about the +`BORG_REPO` and the `BORG_RSH` command to use. Change them to your liking. + +### Systemd timer + +The `/etc/systemd/system/borg-job-tfstated.timer` systemd timer file should +belong to `root:root` with 0444 permissions (`r--r--r--`): + +``` ini +[Unit] +Description=BorgBackup job tfstated timer + +[Timer] +FixedRandomDelay=true +OnCalendar=daily +Persistent=true +RandomizedDelaySec=3600 + +[Install] +WantedBy=timers.target +```