DEPLOYMENT: a pre-deploy backup sequence for the D13 auth change

Asked whether there is an easy way to take a full backup before deploying,
since this touches auth. There is - the backup sidecar is already running and
scripts/db-backup.sh takes a one-shot dump - but three things about THIS deploy
were not written down anywhere.

Timing. The container starts with `alembic upgrade head && exec gunicorn`, so
the migration runs seconds after redeploy and there is no window afterwards. The
dump has to be taken before, not after.

Retention. The scheduled job prunes to the newest BACKUP_KEEP (14) files
matching wpsuite-*.sql.gz*, so on a daily cadence a pre-deploy dump is deleted
in a fortnight - exactly when a slow-burning problem would surface. Copying it
to a name outside the glob protects it.

Rollback is not just the database. The new code has no password_hash in its
model and the old code requires it, so restoring without also rolling the code
back leaves schema and application disagreeing. Recorded as both steps in
order, with a note to capture the current commit FIRST, since that is easy to
forget and impossible to reconstruct afterwards.

Also flagged what this particular dump is: the last copy of every password hash
that will ever exist. BACKUP_ENC_PASSPHRASE must be set before taking it - the
script warns and writes plaintext otherwise - and its retention deserves a
deliberate decision rather than the default fortnight, because bcrypt is not
plaintext but is crackable offline given a copy and time.

And a step to prove the dump is readable before deploying, because an untested
dump is not a backup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-24 11:55:33 -05:00
parent 60b5b0c1f2
commit c99666796a

View File

@@ -324,6 +324,56 @@ docker compose up -d --build webserver # front-end change (html/) — rebuild
docker compose up -d --build api # backend change (server/) docker compose up -d --build api # backend change (server/)
``` ```
## Before deploying the D13 auth change — take a backup first
**Timing is the whole point.** The container's start command is
`alembic upgrade head && exec gunicorn`, so migrations run *seconds after you
redeploy*. Migration `b7e4f1a20c93` drops `users.password_hash`. There is no window
afterwards: back up **before** you redeploy, not after.
```bash
# 1. Record what you are rolling back TO. Do this first; it is easy to forget
# and impossible to reconstruct under pressure.
git -C /path/to/repo rev-parse --short HEAD
# 2. One-shot dump, using the sidecar that is already running.
docker compose exec backup sh /scripts/db-backup.sh
# -> ./backups/wpsuite-<UTC timestamp>.sql.gz[.enc] on the host
# 3. Take it OUT of the rotation. The scheduled job prunes to the newest
# BACKUP_KEEP (default 14) files matching wpsuite-*.sql.gz*, so on a daily
# cadence this dump is deleted in a fortnight. A prefix that does not match
# the glob is enough to protect it.
docker compose exec backup sh -c 'cd /backups && cp "$(ls -1t wpsuite-*.sql.gz* | head -1)" "pre-d13-$(ls -1t wpsuite-*.sql.gz* | head -1)"'
# 4. Prove it is readable BEFORE you deploy. An untested dump is not a backup.
docker compose exec backup sh -c 'openssl enc -d -aes-256-cbc -pbkdf2 -pass env:BACKUP_ENC_PASSPHRASE -in /backups/pre-d13-*.sql.gz.enc | gunzip -c | grep -c "INSERT INTO public.users"'
# (drop the openssl stage for an unencrypted .sql.gz)
```
### Two things about this particular dump
**It is the last copy of every password hash that will ever exist.** After the
migration the column is gone; this file is where those bcrypt hashes live from then
on. Make sure `BACKUP_ENC_PASSPHRASE` is set before step 2 — the script warns loudly
if it is not, and writes plaintext — and decide deliberately how long to keep the
file. bcrypt is not plaintext, but it is crackable offline given time and a copy.
**Restoring the database is not, by itself, a rollback.** The new code has no
`password_hash` in its model and the old code requires it, so a restore without a
matching code rollback leaves you with a schema and an application that disagree. A
real rollback is both, in this order:
```bash
# redeploy the commit from step 1 (Portainer: point the stack back and rebuild)
docker compose exec backup sh /scripts/db-restore.sh /backups/pre-d13-wpsuite-<ts>.sql.gz.enc
```
`db-restore.sh` dumps are taken with `--clean --if-exists`, so restoring **drops and
recreates** objects before loading. It overwrites whatever is currently there.
---
## Backups & retention ## Backups & retention
A **`backup` sidecar** (in `docker-compose.yml`) runs `pg_dump` on a schedule and A **`backup` sidecar** (in `docker-compose.yml`) runs `pg_dump` on a schedule and