From c99666796a77c6e893b888f05648755f61b3a995 Mon Sep 17 00:00:00 2001 From: Cody Schaefer Date: Mon, 24 Aug 2026 11:55:33 -0500 Subject: [PATCH] DEPLOYMENT: a pre-deploy backup sequence for the D13 auth change Asked whether there is an easy way to take a full backup before deploying, since this touches auth. There is - the backup sidecar is already running and scripts/db-backup.sh takes a one-shot dump - but three things about THIS deploy were not written down anywhere. Timing. The container starts with `alembic upgrade head && exec gunicorn`, so the migration runs seconds after redeploy and there is no window afterwards. The dump has to be taken before, not after. Retention. The scheduled job prunes to the newest BACKUP_KEEP (14) files matching wpsuite-*.sql.gz*, so on a daily cadence a pre-deploy dump is deleted in a fortnight - exactly when a slow-burning problem would surface. Copying it to a name outside the glob protects it. Rollback is not just the database. The new code has no password_hash in its model and the old code requires it, so restoring without also rolling the code back leaves schema and application disagreeing. Recorded as both steps in order, with a note to capture the current commit FIRST, since that is easy to forget and impossible to reconstruct afterwards. Also flagged what this particular dump is: the last copy of every password hash that will ever exist. BACKUP_ENC_PASSPHRASE must be set before taking it - the script warns and writes plaintext otherwise - and its retention deserves a deliberate decision rather than the default fortnight, because bcrypt is not plaintext but is crackable offline given a copy and time. And a step to prove the dump is readable before deploying, because an untested dump is not a backup. Co-Authored-By: Claude Opus 5 (1M context) --- DEPLOYMENT.md | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index b286409..a769462 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -324,6 +324,56 @@ docker compose up -d --build webserver # front-end change (html/) — rebuild docker compose up -d --build api # backend change (server/) ``` +## Before deploying the D13 auth change — take a backup first + +**Timing is the whole point.** The container's start command is +`alembic upgrade head && exec gunicorn`, so migrations run *seconds after you +redeploy*. Migration `b7e4f1a20c93` drops `users.password_hash`. There is no window +afterwards: back up **before** you redeploy, not after. + +```bash +# 1. Record what you are rolling back TO. Do this first; it is easy to forget +# and impossible to reconstruct under pressure. +git -C /path/to/repo rev-parse --short HEAD + +# 2. One-shot dump, using the sidecar that is already running. +docker compose exec backup sh /scripts/db-backup.sh +# -> ./backups/wpsuite-.sql.gz[.enc] on the host + +# 3. Take it OUT of the rotation. The scheduled job prunes to the newest +# BACKUP_KEEP (default 14) files matching wpsuite-*.sql.gz*, so on a daily +# cadence this dump is deleted in a fortnight. A prefix that does not match +# the glob is enough to protect it. +docker compose exec backup sh -c 'cd /backups && cp "$(ls -1t wpsuite-*.sql.gz* | head -1)" "pre-d13-$(ls -1t wpsuite-*.sql.gz* | head -1)"' + +# 4. Prove it is readable BEFORE you deploy. An untested dump is not a backup. +docker compose exec backup sh -c 'openssl enc -d -aes-256-cbc -pbkdf2 -pass env:BACKUP_ENC_PASSPHRASE -in /backups/pre-d13-*.sql.gz.enc | gunzip -c | grep -c "INSERT INTO public.users"' +# (drop the openssl stage for an unencrypted .sql.gz) +``` + +### Two things about this particular dump + +**It is the last copy of every password hash that will ever exist.** After the +migration the column is gone; this file is where those bcrypt hashes live from then +on. Make sure `BACKUP_ENC_PASSPHRASE` is set before step 2 — the script warns loudly +if it is not, and writes plaintext — and decide deliberately how long to keep the +file. bcrypt is not plaintext, but it is crackable offline given time and a copy. + +**Restoring the database is not, by itself, a rollback.** The new code has no +`password_hash` in its model and the old code requires it, so a restore without a +matching code rollback leaves you with a schema and an application that disagree. A +real rollback is both, in this order: + +```bash +# redeploy the commit from step 1 (Portainer: point the stack back and rebuild) +docker compose exec backup sh /scripts/db-restore.sh /backups/pre-d13-wpsuite-.sql.gz.enc +``` + +`db-restore.sh` dumps are taken with `--clean --if-exists`, so restoring **drops and +recreates** objects before loading. It overwrites whatever is currently there. + +--- + ## Backups & retention A **`backup` sidecar** (in `docker-compose.yml`) runs `pg_dump` on a schedule and