diff --git a/DEPLOY-runbook-2026-08-04.md b/DEPLOY-runbook-2026-08-04.md new file mode 100644 index 0000000..fa4a9b9 --- /dev/null +++ b/DEPLOY-runbook-2026-08-04.md @@ -0,0 +1,233 @@ +# Deploy runbook — WP Suite, 2026-08-04 + +**For:** IT / whoever administers the Docker host and Portainer +**From:** n.siegfried@prime-controls.com +**Expected duration:** 10–15 minutes, including the backup +**Expected downtime:** under a minute, while containers are recreated + +--- + +## Fill these in before handing this over + +| Thing | Value | +|---|---| +| Docker host (SSH target) | `________________` | +| Stack name in Portainer | `________________` | +| Site URL | `https://________________` | +| Stack directory on the host (holds `docker-compose.yml` / `backups/`) | `________________` | + +Container names are fixed by the compose file and are the same on every host: +`nginx_webserver`, `wp_api`, `wp_db`, `wp_db_backup`. + +--- + +## What this deploy changes + +Front-end and nginx changes, plus pending database migrations that run +automatically. Two things make it more than a routine restart: + +1. **The nginx config and the entire `html/` directory are baked into the + container image at build time.** A plain restart deploys nothing — the stack + must be re-pulled and re-built. +2. **A pending migration rewrites existing rows** in the `users.role` column + (values `user` → `project_user`). That is why step 1 is a backup and not + optional. + +Migrations run themselves when the `wp_api` container starts. There is nothing +to type and **no new environment variables** — do not change the stack's +environment variables. + +--- + +## Step 0 — Record the current state (needed for rollback) + +SSH to the Docker host and run: + +```bash +docker exec wp_api alembic -c server/alembic.ini current +docker inspect nginx_webserver --format 'nginx image: {{.Image}}' +docker inspect wp_api --format 'api image: {{.Image}}' +``` + +**Copy the output into your ticket.** Also note the Git commit the Portainer +stack is currently on (Portainer → the stack → the Git reference / last-updated +commit). Without these, rollback is guesswork. + +--- + +## Step 1 — Back up the database + +On the Docker host: + +```bash +docker exec wp_db_backup /scripts/db-backup.sh +``` + +This triggers the stack's existing backup sidecar once, on demand. Expected +output ends with a line like: + +``` +[db-backup] wrote 1.4M /backups/wpsuite-20260804-141233Z.sql.gz.enc +``` + +Confirm the file is on the host (substitute the stack directory): + +```bash +ls -lt /backups | head -3 +``` + +**Record that filename.** Do not continue until you have seen the `wrote …` +line and the file in that listing. + +- A `.sql.gz.enc` extension means backups are encrypted — expected and correct. +- A `.sql.gz` extension plus a `WARNING: BACKUP_ENC_PASSPHRASE not set` line + means backups are unencrypted. Not a blocker for this deploy; report it back. +- **No SSH access?** Portainer → **Containers** → `wp_db_backup` → **Console** → + connect with `/bin/sh`, then run `/scripts/db-backup.sh`. Same result: the + dump lands on the host, because `/backups` is a bind mount. + +--- + +## Step 2 — Redeploy the stack in Portainer + +1. Portainer → **Stacks** → select the stack. +2. **Pull and redeploy** — with re-pull / re-build **enabled**. +3. Wait for it to report success. + +A plain "restart" or "stop/start" will **not** deploy this change. See "What +this deploy changes" above. + +--- + +## Step 3 — Confirm the containers came up + +```bash +docker ps --filter name=nginx_webserver --filter name=wp_api --filter name=wp_db +``` + +All three must be `Up`, and `wp_db` should show `(healthy)`. Then check the API +applied its migrations cleanly: + +```bash +docker logs wp_api --tail 40 +``` + +You are looking for Alembic `Running upgrade …` lines followed by gunicorn +starting up, and **no** traceback. The API deliberately refuses to start if a +migration fails, so a restarting `wp_api` container means the migration failed — +go to Rollback. + +Then verify nginx's own view of its config: + +```bash +docker exec nginx_webserver nginx -t +``` + +Expected: `syntax is ok` / `test is successful`. + +--- + +## Step 4 — Confirm the response headers + +```bash +curl -sI https:///work-package-suite.html | grep -Ei 'cache-control|content-security-policy' +``` + +Add `-k` if the site uses an internal or self-signed certificate. + +**Both lines must come back.** Expected, approximately: + +``` +cache-control: no-cache, must-revalidate +content-security-policy: default-src 'self'; script-src 'self' 'unsafe-inline'; ... +``` + +If the `content-security-policy` line is **missing** while `cache-control` is +present, the deploy is bad — go to Rollback and send me the nginx log. (This is +the specific regression this deploy fixes; the two headers must coexist.) + +Also confirm the API is reachable through the proxy: + +```bash +curl -s https:///api/health # → {"ok": true} +``` + +--- + +## Step 5 — Hard-reload once in a browser + +Open the site and press **Ctrl+Shift+R** (Cmd+Shift+R on macOS) once. The app +uses a service worker; a normal reload can serve the previous version and make a +good deploy look broken. + +Sanity check: log in, and confirm the home page offers to select or create a +project. + +**Deploy complete.** Please report back: the step 0 output, the backup filename, +and the two header lines from step 4. + +--- + +## Rollback + +Pick the case that matches. + +### Case A — nginx won't start, or the CSP header is missing + +The database is untouched by this, so this is a code-only rollback. In Portainer, +redeploy the stack pinned to the **previous Git commit** recorded in step 0 +(Portainer → the stack → change the Git reference to that commit → Pull and +redeploy). Then re-run step 3 and step 4. + +**Before you do:** grab the log, because it is what I need to fix this. + +```bash +docker logs nginx_webserver --tail 100 +``` + +Send me that output. If the container is in a restart loop the log still works. + +### Case B — `wp_api` is restarting / a migration failed + +```bash +docker logs wp_api --tail 100 +``` + +Send me that output. **Do not restore the database and do not roll the API back +without contacting me first** — if the `users.role` migration already committed, +rolling the API back to the previous image without converting those values back +will break logins. That conversion is a one-line command, but it has to match +what actually ran. + +Reach me at n.siegfried@prime-controls.com. + +### Case C — restoring the backup (only if I ask for it) + +Destructive: this drops and recreates the current schema and data. For an +encrypted dump, on the Docker host, in the `backups` directory: + +```bash +export BACKUP_ENC_PASSPHRASE='' +openssl enc -d -aes-256-cbc -pbkdf2 -pass env:BACKUP_ENC_PASSPHRASE \ + -in wpsuite-.sql.gz.enc \ + | gunzip \ + | docker exec -i wp_db psql -U wpsuite -d wpsuite +unset BACKUP_ENC_PASSPHRASE +``` + +For an unencrypted dump, drop the `openssl` stage and pipe `gunzip` straight +into `psql`. Substitute the real values if `POSTGRES_USER` / `POSTGRES_DB` are +not `wpsuite`. + +--- + +## Notes + +- Do not add or change environment variables for this deploy. +- Do not run `docker compose down -v` — the `-v` flag deletes the `pgdata` + volume and with it the entire database. +- `docker compose …` commands are avoided throughout this runbook on purpose: + for a Portainer-managed Git stack the compose project lives under Portainer's + own data directory, so `docker compose` from an SSH session usually can't find + it. The `docker exec ` form used here works from any directory. +- Full background documentation: `DEPLOYMENT.md` in the repository.