DEPLOYMENT: a pre-deploy backup sequence for the D13 auth change
Asked whether there is an easy way to take a full backup before deploying, since this touches auth. There is - the backup sidecar is already running and scripts/db-backup.sh takes a one-shot dump - but three things about THIS deploy were not written down anywhere. Timing. The container starts with `alembic upgrade head && exec gunicorn`, so the migration runs seconds after redeploy and there is no window afterwards. The dump has to be taken before, not after. Retention. The scheduled job prunes to the newest BACKUP_KEEP (14) files matching wpsuite-*.sql.gz*, so on a daily cadence a pre-deploy dump is deleted in a fortnight - exactly when a slow-burning problem would surface. Copying it to a name outside the glob protects it. Rollback is not just the database. The new code has no password_hash in its model and the old code requires it, so restoring without also rolling the code back leaves schema and application disagreeing. Recorded as both steps in order, with a note to capture the current commit FIRST, since that is easy to forget and impossible to reconstruct afterwards. Also flagged what this particular dump is: the last copy of every password hash that will ever exist. BACKUP_ENC_PASSPHRASE must be set before taking it - the script warns and writes plaintext otherwise - and its retention deserves a deliberate decision rather than the default fortnight, because bcrypt is not plaintext but is crackable offline given a copy and time. And a step to prove the dump is readable before deploying, because an untested dump is not a backup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -324,6 +324,56 @@ docker compose up -d --build webserver # front-end change (html/) — rebuild
|
||||
docker compose up -d --build api # backend change (server/)
|
||||
```
|
||||
|
||||
## Before deploying the D13 auth change — take a backup first
|
||||
|
||||
**Timing is the whole point.** The container's start command is
|
||||
`alembic upgrade head && exec gunicorn`, so migrations run *seconds after you
|
||||
redeploy*. Migration `b7e4f1a20c93` drops `users.password_hash`. There is no window
|
||||
afterwards: back up **before** you redeploy, not after.
|
||||
|
||||
```bash
|
||||
# 1. Record what you are rolling back TO. Do this first; it is easy to forget
|
||||
# and impossible to reconstruct under pressure.
|
||||
git -C /path/to/repo rev-parse --short HEAD
|
||||
|
||||
# 2. One-shot dump, using the sidecar that is already running.
|
||||
docker compose exec backup sh /scripts/db-backup.sh
|
||||
# -> ./backups/wpsuite-<UTC timestamp>.sql.gz[.enc] on the host
|
||||
|
||||
# 3. Take it OUT of the rotation. The scheduled job prunes to the newest
|
||||
# BACKUP_KEEP (default 14) files matching wpsuite-*.sql.gz*, so on a daily
|
||||
# cadence this dump is deleted in a fortnight. A prefix that does not match
|
||||
# the glob is enough to protect it.
|
||||
docker compose exec backup sh -c 'cd /backups && cp "$(ls -1t wpsuite-*.sql.gz* | head -1)" "pre-d13-$(ls -1t wpsuite-*.sql.gz* | head -1)"'
|
||||
|
||||
# 4. Prove it is readable BEFORE you deploy. An untested dump is not a backup.
|
||||
docker compose exec backup sh -c 'openssl enc -d -aes-256-cbc -pbkdf2 -pass env:BACKUP_ENC_PASSPHRASE -in /backups/pre-d13-*.sql.gz.enc | gunzip -c | grep -c "INSERT INTO public.users"'
|
||||
# (drop the openssl stage for an unencrypted .sql.gz)
|
||||
```
|
||||
|
||||
### Two things about this particular dump
|
||||
|
||||
**It is the last copy of every password hash that will ever exist.** After the
|
||||
migration the column is gone; this file is where those bcrypt hashes live from then
|
||||
on. Make sure `BACKUP_ENC_PASSPHRASE` is set before step 2 — the script warns loudly
|
||||
if it is not, and writes plaintext — and decide deliberately how long to keep the
|
||||
file. bcrypt is not plaintext, but it is crackable offline given time and a copy.
|
||||
|
||||
**Restoring the database is not, by itself, a rollback.** The new code has no
|
||||
`password_hash` in its model and the old code requires it, so a restore without a
|
||||
matching code rollback leaves you with a schema and an application that disagree. A
|
||||
real rollback is both, in this order:
|
||||
|
||||
```bash
|
||||
# redeploy the commit from step 1 (Portainer: point the stack back and rebuild)
|
||||
docker compose exec backup sh /scripts/db-restore.sh /backups/pre-d13-wpsuite-<ts>.sql.gz.enc
|
||||
```
|
||||
|
||||
`db-restore.sh` dumps are taken with `--clean --if-exists`, so restoring **drops and
|
||||
recreates** objects before loading. It overwrites whatever is currently there.
|
||||
|
||||
---
|
||||
|
||||
## Backups & retention
|
||||
|
||||
A **`backup` sidecar** (in `docker-compose.yml`) runs `pg_dump` on a schedule and
|
||||
|
||||
Reference in New Issue
Block a user