# Deploy runbook: WP Suite Okta cutover (wave 10) **For:** IT / whoever administers the Docker host and Portainer **From:** m.mabrey@prime-controls.com **Revised:** 2026-09-03. First version of this runbook. **Expected duration:** 20-30 minutes, including the backup and the live sign-in check **Expected downtime:** under a minute, while containers are recreated This is a separate runbook from `DEPLOY-runbook-2026-08-04.md`, not a revision of it. That one still applies to its own deploy. Read this one in full before starting. It has a step that the earlier one does not: this deploy removes the local password system entirely, and one part of that removal cannot be undone by the usual means. See "What this deploy changes" below. --- ## Fill these in before handing this over | Thing | Value | |---|---| | Docker host (SSH target) | `________________` | | Stack name in Portainer | `________________` | | Site URL | `https://________________` | | Stack directory on the host (holds `docker-compose.yml` / `backups/`) | `________________` | | A real Okta account assigned to the app integration, for the live check in Step 4 | `________________` | Container names are fixed by the compose file and are the same on every host: `nginx_webserver`, `wp_api`, `wp_db`, `wp_db_backup`. --- ## What this deploy changes Sign-in changes from a local username/password to Okta OIDC, completely. Not a toggle, not a fallback. Three things make this more than a routine deploy: 1. **Five new environment variables are required**, and without them nobody can sign in: `OKTA_ISSUER`, `OKTA_CLIENT_ID`, `OKTA_CLIENT_SECRET`, `OKTA_REDIRECT_URI`, and optionally `OKTA_IDENTITY_CLAIM`. This is a genuine change from how deploys usually go here. Do not skip Step 1.5. 2. **A migration drops the `password_hash` column** (`1d60a608bb51`), and it is **one-way in practice.** Its `downgrade()` re-adds the column, but empty. The real password hashes are gone the moment this commits, and no `alembic downgrade` brings them back. If this deploy needs to be undone after that point, going back to the old local-password code does not work on its own. See the Rollback section, Case C. This is why Step 1's backup is not optional the way it sometimes reads in other runbooks. 3. **There is no break-glass path**, by design (recorded decision D16). If Okta is unreachable or misconfigured after this deploy, the app is unreachable for everyone, admins included, until Okta is fixed. That is expected behavior, not a bug to roll back from. See Rollback, Case A/B, before assuming something is broken. The visible change for people using the app: the login page becomes "Sign in with Okta" instead of a username/password form. Nothing else in the app's day-to-day behavior changes. --- ## Step 0: Record the current state (needed for rollback) SSH to the Docker host and run: ```bash docker exec wp_api alembic -c server/alembic.ini current docker inspect nginx_webserver --format 'nginx image: {{.Image}}' docker inspect wp_api --format 'api image: {{.Image}}' ``` **Copy the output into your ticket.** Also note the Git commit the Portainer stack is currently on (Portainer, the stack, the Git reference / last-updated commit). Without these, rollback is guesswork. Expected output of the first command before this deploy: `a1b8c6d4e2f9 (head)`. If it shows anything else, stop and check with me before continuing. This runbook assumes that starting point. --- ## Step 1: Back up the database On the Docker host: ```bash docker exec wp_db_backup /scripts/db-backup.sh ``` Expected output ends with a line like: ``` [db-backup] wrote 1.4M /backups/wpsuite-20260903-141233Z.sql.gz.enc ``` Confirm the file is on the host (substitute the stack directory): ```bash ls -lt /backups | head -3 ``` **Record that filename.** Do not continue until you have seen the `wrote ...` line and the file in that listing. This backup matters more than usual for this deploy. Once the migration in Step 3 commits, this file becomes the *only* way to get local password hashes back, for any reason. Treat it as the point you would restore to, not routine housekeeping. - A `.sql.gz.enc` extension means backups are encrypted. Expected and correct. - A `.sql.gz` extension plus a `WARNING: BACKUP_ENC_PASSPHRASE not set` line means backups are unencrypted. Not a blocker for this deploy; report it back. - **No SSH access?** Portainer, **Containers**, `wp_db_backup`, **Console**, connect with `/bin/sh`, then run `/scripts/db-backup.sh`. Same result: the dump lands on the host, because `/backups` is a bind mount. --- ## Step 1.5: Confirm the Okta app integration is actually ready Do this before redeploying, not after. Everything here is checked in Okta's own admin console and in the values that will go into the stack's environment variables. Nothing touches the WP Suite host yet. 1. The Okta app integration exists (Sign-in method: OIDC, Authorization Code, Application type: Web Application), and the account listed in the fill-in table above is assigned to it. 2. `OKTA_REDIRECT_URI` matches a "Sign-in redirect URI" registered on that app integration **exactly**: scheme, host, and path, including whether it ends in `/api/auth/okta/callback`. 3. `OKTA_ISSUER`, `OKTA_CLIENT_ID`, and `OKTA_CLIENT_SECRET` are the values from that same app integration, not a different one. 4. If your Okta configuration puts the directory identity somewhere other than the `preferred_username` claim, `OKTA_IDENTITY_CLAIM` is set to the right claim name. If unsure, leave it unset; `preferred_username` is the default. Add all five to the stack's **Environment variables** in Portainer now, before Step 2. `OKTA_CLIENT_SECRET` should be handled the same way `AUTH_SECRET_KEY` already is: not typed anywhere it will be logged. If any of items 1-3 above are not yet confirmed, stop here and get them confirmed first. A wrong redirect URI or an unassigned account will not corrupt anything, but it does mean nobody signs in after this deploy until it is fixed. See Rollback, Case B, which is the ordinary way that gets fixed and does not involve the database at all. --- ## Step 2: Redeploy the stack in Portainer 1. Portainer, **Stacks**, select the stack. 2. **Pull and redeploy**, with re-pull / re-build **enabled**. 3. Wait for it to report success. A plain "restart" or "stop/start" will not pick up new code, and will not pick up the environment variables added in Step 1.5 either. --- ## Step 3: Confirm the containers came up ```bash docker ps --filter name=nginx_webserver --filter name=wp_api --filter name=wp_db ``` All three must be `Up`, and `wp_db` should show `(healthy)`. Then check the API applied its migration cleanly: ```bash docker logs wp_api --tail 40 ``` You are looking for an Alembic `Running upgrade a1b8c6d4e2f9 -> 1d60a608bb51` line followed by gunicorn starting up, and no traceback. The API refuses to start if a migration fails, so a restarting `wp_api` container means it failed. Go to Rollback, Case B, and read the "did the migration commit" note there before doing anything to the database. Confirm the database landed on the new revision: ```bash docker exec wp_api alembic -c server/alembic.ini current ``` Expected: `1d60a608bb51 (head)`. **Once you see this, you have passed the point of no return described above.** The backup from Step 1 is now the only way back to a working local-password system, if that is ever needed. Then verify nginx's own view of its config: ```bash docker exec nginx_webserver nginx -t ``` Expected: `syntax is ok` / `test is successful`. --- ## Step 4: Confirm Okta sign-in actually works, live This is the step that matters most for this deploy. A clean container start does not by itself prove sign-in works, and there is currently no startup log line that confirms Okta config is good (logged separately as a follow-up, not fixed as part of this runbook). The only real proof is a live sign-in. 1. Open the site's normal URL in a private/incognito window. It should land on `login.html` with a "Sign in with Okta" button, not a username/password form. 2. Click it. You should be redirected to your organization's actual Okta sign-in page (the real Okta domain from `OKTA_ISSUER`, not this app's own domain). 3. Sign in with the account from the fill-in table. You should land back on the WP Suite site, signed in. 4. If this is the account's first-ever sign-in, it is now JIT-provisioned as a regular user (`project_user`). To make it an admin, on the Docker host: ```bash docker exec -it wp_api python -m server.manage_users promote --role admin ``` This only works on an account that has already signed in once through Okta. It promotes an existing row; it does not create one. That is deliberate (recorded decision D16): there is no other admin-bootstrap path. If step 2 or 3 fails (redirected to an Okta error page, redirected back to `login.html` with an error, or nothing happens), this is almost always a configuration problem from Step 1.5, not a code or database problem. Go to Rollback, Case B, before considering anything more drastic. Also confirm the API is reachable through the proxy and the redirect itself is wired up: ```bash curl -s https:///api/health # -> {"ok": true} curl -sI https:///api/auth/okta/login | grep -i ^location # -> your Okta authorize URL ``` --- ## Step 5: Hard-reload once in a browser, then sanity-check Press **Ctrl+Shift+R** (Cmd+Shift+R on macOS) once. The app uses a service worker; a normal reload can serve the previous version. 1. Signed in as the account from Step 4, the home page offers to select or create a project, same as before. 2. Open **Admin Console** as the promoted admin account. The user table shows the account you just signed in with. There is no password column, no "reset password" action anywhere in the UI. 3. Open a project and confirm a work package can be opened and edited normally. Sign-in is the only thing this deploy changes, so the rest of the app should look untouched. **Deploy complete.** Please report back: the Step 0 output, the backup filename from Step 1, and confirmation that Step 4's live sign-in worked. --- ## Rollback Read this before assuming a rollback is needed. Cases A and B below do **not** touch the database and are the far more likely outcome of something going wrong here. Okta configuration is fiddly and easy to get slightly wrong. Case C is the severe, destructive one, and should be a last resort, not a first reaction. ### Case A: nginx won't start, or containers won't come up at all Same as any other deploy: the database is untouched by container start-up failures. In Portainer, redeploy the stack pinned to the **previous Git commit** recorded in Step 0, then re-run Step 3. ```bash docker logs nginx_webserver --tail 100 docker logs wp_api --tail 100 ``` Send me whichever of those is relevant. ### Case B: containers are up, but Okta sign-in doesn't work This is a configuration problem, not a data problem, and does **not** need a code rollback or a database restore. Check, in order: 1. Is `wp_api`'s log showing anything at all when a sign-in is attempted? `docker logs wp_api --tail 100`. 2. Do the five `OKTA_*` values in the stack's environment variables actually match the Okta app integration (Step 1.5)? A copy-paste error in `OKTA_CLIENT_SECRET` or a redirect URI that's off by a trailing slash are the two most common causes. 3. Is the account assigned to the Okta app integration? An unassigned account gets denied by Okta itself, before it ever reaches this app. 4. If a specific person can't sign in but others can, check `OKTA_IDENTITY_CLAIM`. The claim it reads may not carry that person's directory identity in the format expected. Confirm with security which claim Okta is actually issuing. Fix the environment variable(s) in Portainer, then redeploy (Pull and redeploy is fine; the migration already applied and does not run again). No backup restore, no code rollback. If it's still not working after checking all four, send me the `wp_api` log from item 1 along with which of items 2-4 you already ruled out. ### Case C: the decision is made to abandon Okta and restore local-password sign-in This is the case the point-of-no-return warning in "What this deploy changes" is about. Only reach for this if Case A and B do not apply. That is, Okta itself is working correctly but a decision has been made to go back to the old system entirely. **This cannot be done with a code rollback alone.** The old code expects a real `password_hash` on every user row. After Step 3 commits, that column is either gone or (if `alembic downgrade` is run) present but empty. Either way, nobody's stored password survived, including admins'. The only way to get a working local-password system back is to restore the full database from the Step 1 backup, which also rolls back every other change made since that backup: new work packages, comments, uploaded files, everything. **Do not do this without confirming with me first.** If it is confirmed, for an encrypted dump, on the Docker host, in the `backups` directory: ```bash export BACKUP_ENC_PASSPHRASE='' openssl enc -d -aes-256-cbc -pbkdf2 -pass env:BACKUP_ENC_PASSPHRASE \ -in wpsuite-.sql.gz.enc \ | gunzip \ | docker exec -i wp_db psql -U wpsuite -d wpsuite unset BACKUP_ENC_PASSPHRASE ``` For an unencrypted dump, drop the `openssl` stage and pipe `gunzip` straight into `psql`. Substitute the real values if `POSTGRES_USER` / `POSTGRES_DB` are not `wpsuite`. After restoring, redeploy pinned to the Git commit recorded in Step 0, since the restored database matches the old schema, not this one. Reach me at m.mabrey@prime-controls.com. --- ## Notes - Do not run `docker compose down -v`. The `-v` flag deletes the `pgdata` volume and with it the entire database. - `docker exec ` is used throughout rather than `docker compose ...`, because a Portainer-managed Git stack's compose project lives under Portainer's own data directory and usually isn't reachable from an ad hoc SSH session the same way. - There is currently no startup log line confirming Okta config is valid (`okta_auth.describe()` exists but nothing calls it yet, logged as a follow-up). Step 4's live sign-in is the real verification until that's wired in. - Full background documentation: `DEPLOYMENT.md` and `server/README.md` in the repository. `docs/waves/decisions-2026-09-03.md` (D16, D17) records why there is no break-glass path and why this runbook exists as its own document.