Project-SDE-WP-Suite/DEPLOY-runbook-2026-09-03.md

Deploy runbook: WP Suite Okta cutover (wave 10)

For: IT / whoever administers the Docker host and Portainer From: m.mabrey@prime-controls.com Revised: 2026-09-03. First version of this runbook. Expected duration: 20-30 minutes, including the backup and the live sign-in check Expected downtime: under a minute, while containers are recreated

This is a separate runbook from DEPLOY-runbook-2026-08-04.md, not a revision of it. That one still applies to its own deploy. Read this one in full before starting. It has a step that the earlier one does not: this deploy removes the local password system entirely, and one part of that removal cannot be undone by the usual means. See "What this deploy changes" below.

---

Fill these in before handing this over

Thing	Value
Docker host (SSH target)	________________
Stack name in Portainer	________________
Site URL	https://________________
Stack directory on the host (holds docker-compose.yml / backups/)	________________
A real Okta account assigned to the app integration, for the live check in Step 4	________________

Container names are fixed by the compose file and are the same on every host: nginx_webserver, wp_api, wp_db, wp_db_backup.

---

What this deploy changes

Sign-in changes from a local username/password to Okta OIDC, completely. Not a toggle, not a fallback. Three things make this more than a routine deploy:

1. Five new environment variables are required, and without them nobody can sign in: OKTA_ISSUER, OKTA_CLIENT_ID, OKTA_CLIENT_SECRET, OKTA_REDIRECT_URI, and optionally OKTA_IDENTITY_CLAIM. This is a genuine change from how deploys usually go here. Do not skip Step 1.5.

2. A migration drops the password_hash column (1d60a608bb51), and it is one-way in practice. Its downgrade() re-adds the column, but empty. The real password hashes are gone the moment this commits, and no alembic downgrade brings them back. If this deploy needs to be undone after that point, going back to the old local-password code does not work on its own. See the Rollback section, Case C. This is why Step 1's backup is not optional the way it sometimes reads in other runbooks.

3. There is no break-glass path, by design (recorded decision D16). If Okta is unreachable or misconfigured after this deploy, the app is unreachable for everyone, admins included, until Okta is fixed. That is expected behavior, not a bug to roll back from. See Rollback, Case A/B, before assuming something is broken.

The visible change for people using the app: the login page becomes "Sign in with Okta" instead of a username/password form. Nothing else in the app's day-to-day behavior changes.

---

Step 0: Record the current state (needed for rollback)

SSH to the Docker host and run:

docker exec wp_api alembic -c server/alembic.ini current
docker inspect nginx_webserver --format 'nginx image: {{.Image}}'
docker inspect wp_api          --format 'api image:   {{.Image}}'

Copy the output into your ticket. Also note the Git commit the Portainer stack is currently on (Portainer, the stack, the Git reference / last-updated commit). Without these, rollback is guesswork.

Expected output of the first command before this deploy: a1b8c6d4e2f9 (head). If it shows anything else, stop and check with me before continuing. This runbook assumes that starting point.

---

Step 1: Back up the database

On the Docker host:

docker exec wp_db_backup /scripts/db-backup.sh

Expected output ends with a line like:

[db-backup] wrote 1.4M /backups/wpsuite-20260903-141233Z.sql.gz.enc

Confirm the file is on the host (substitute the stack directory):

ls -lt <stack-dir>/backups | head -3

Record that filename. Do not continue until you have seen the wrote ... line and the file in that listing.

This backup matters more than usual for this deploy. Once the migration in Step 3 commits, this file becomes the only way to get local password hashes back, for any reason. Treat it as the point you would restore to, not routine housekeeping.

• A .sql.gz.enc extension means backups are encrypted. Expected and correct.
• A .sql.gz extension plus a WARNING: BACKUP_ENC_PASSPHRASE not set line means backups are unencrypted. Not a blocker for this deploy; report it back.
• No SSH access? Portainer, Containers, wp_db_backup, Console, connect with /bin/sh, then run /scripts/db-backup.sh. Same result: the dump lands on the host, because /backups is a bind mount.

---

Step 1.5: Confirm the Okta app integration is actually ready

Do this before redeploying, not after. Everything here is checked in Okta's own admin console and in the values that will go into the stack's environment variables. Nothing touches the WP Suite host yet.

1. The Okta app integration exists (Sign-in method: OIDC, Authorization Code, Application type: Web Application), and the account listed in the fill-in table above is assigned to it.
2. OKTA_REDIRECT_URI matches a "Sign-in redirect URI" registered on that app integration exactly: scheme, host, and path, including whether it ends in /api/auth/okta/callback.
3. OKTA_ISSUER, OKTA_CLIENT_ID, and OKTA_CLIENT_SECRET are the values from that same app integration, not a different one.
4. If your Okta configuration puts the directory identity somewhere other than the preferred_username claim, OKTA_IDENTITY_CLAIM is set to the right claim name. If unsure, leave it unset; preferred_username is the default.

Add all five to the stack's Environment variables in Portainer now, before Step 2. OKTA_CLIENT_SECRET should be handled the same way AUTH_SECRET_KEY already is: not typed anywhere it will be logged.

If any of items 1-3 above are not yet confirmed, stop here and get them confirmed first. A wrong redirect URI or an unassigned account will not corrupt anything, but it does mean nobody signs in after this deploy until it is fixed. See Rollback, Case B, which is the ordinary way that gets fixed and does not involve the database at all.

---

Step 2: Redeploy the stack in Portainer

1. Portainer, Stacks, select the stack.
2. Pull and redeploy, with re-pull / re-build enabled.
3. Wait for it to report success.

A plain "restart" or "stop/start" will not pick up new code, and will not pick up the environment variables added in Step 1.5 either.

---

Step 3: Confirm the containers came up

docker ps --filter name=nginx_webserver --filter name=wp_api --filter name=wp_db

All three must be Up, and wp_db should show (healthy). Then check the API applied its migration cleanly:

docker logs wp_api --tail 40

You are looking for an Alembic Running upgrade a1b8c6d4e2f9 -> 1d60a608bb51 line followed by gunicorn starting up, and no traceback. The API refuses to start if a migration fails, so a restarting wp_api container means it failed. Go to Rollback, Case B, and read the "did the migration commit" note there before doing anything to the database.

Confirm the database landed on the new revision:

docker exec wp_api alembic -c server/alembic.ini current

Expected: 1d60a608bb51 (head). Once you see this, you have passed the point of no return described above. The backup from Step 1 is now the only way back to a working local-password system, if that is ever needed.

Then verify nginx's own view of its config:

docker exec nginx_webserver nginx -t

Expected: syntax is ok / test is successful.

---

Step 4: Confirm Okta sign-in actually works, live

This is the step that matters most for this deploy. A clean container start does not by itself prove sign-in works, and there is currently no startup log line that confirms Okta config is good (logged separately as a follow-up, not fixed as part of this runbook). The only real proof is a live sign-in.

1. Open the site's normal URL in a private/incognito window. It should land on login.html with a "Sign in with Okta" button, not a username/password form.

2. Click it. You should be redirected to your organization's actual Okta sign-in page (the real Okta domain from OKTA_ISSUER, not this app's own domain).

3. Sign in with the account from the fill-in table. You should land back on the WP Suite site, signed in.

4. If this is the account's first-ever sign-in, it is now JIT-provisioned as a regular user (project_user). To make it an admin, on the Docker host:

   docker exec -it wp_api python -m server.manage_users promote <username> --role admin
   

   This only works on an account that has already signed in once through Okta. It promotes an existing row; it does not create one. That is deliberate (recorded decision D16): there is no other admin-bootstrap path.

If step 2 or 3 fails (redirected to an Okta error page, redirected back to login.html with an error, or nothing happens), this is almost always a configuration problem from Step 1.5, not a code or database problem. Go to Rollback, Case B, before considering anything more drastic.

Also confirm the API is reachable through the proxy and the redirect itself is wired up:

curl -s https://<site-url>/api/health                                  # -> {"ok": true}
curl -sI https://<site-url>/api/auth/okta/login | grep -i ^location    # -> your Okta authorize URL

---

Step 5: Hard-reload once in a browser, then sanity-check

Press Ctrl+Shift+R (Cmd+Shift+R on macOS) once. The app uses a service worker; a normal reload can serve the previous version.

1. Signed in as the account from Step 4, the home page offers to select or create a project, same as before.
2. Open Admin Console as the promoted admin account. The user table shows the account you just signed in with. There is no password column, no "reset password" action anywhere in the UI.
3. Open a project and confirm a work package can be opened and edited normally. Sign-in is the only thing this deploy changes, so the rest of the app should look untouched.

Deploy complete. Please report back: the Step 0 output, the backup filename from Step 1, and confirmation that Step 4's live sign-in worked.

---

Rollback

Read this before assuming a rollback is needed. Cases A and B below do not touch the database and are the far more likely outcome of something going wrong here. Okta configuration is fiddly and easy to get slightly wrong. Case C is the severe, destructive one, and should be a last resort, not a first reaction.

Case A: nginx won't start, or containers won't come up at all

Same as any other deploy: the database is untouched by container start-up failures. In Portainer, redeploy the stack pinned to the previous Git commit recorded in Step 0, then re-run Step 3.

docker logs nginx_webserver --tail 100
docker logs wp_api --tail 100

Send me whichever of those is relevant.

Case B: containers are up, but Okta sign-in doesn't work

This is a configuration problem, not a data problem, and does not need a code rollback or a database restore. Check, in order:

1. Is wp_api's log showing anything at all when a sign-in is attempted? docker logs wp_api --tail 100.
2. Do the five OKTA_* values in the stack's environment variables actually match the Okta app integration (Step 1.5)? A copy-paste error in OKTA_CLIENT_SECRET or a redirect URI that's off by a trailing slash are the two most common causes.
3. Is the account assigned to the Okta app integration? An unassigned account gets denied by Okta itself, before it ever reaches this app.
4. If a specific person can't sign in but others can, check OKTA_IDENTITY_CLAIM. The claim it reads may not carry that person's directory identity in the format expected. Confirm with security which claim Okta is actually issuing.

Fix the environment variable(s) in Portainer, then redeploy (Pull and redeploy is fine; the migration already applied and does not run again). No backup restore, no code rollback.

If it's still not working after checking all four, send me the wp_api log from item 1 along with which of items 2-4 you already ruled out.

Case C: the decision is made to abandon Okta and restore local-password sign-in

This is the case the point-of-no-return warning in "What this deploy changes" is about. Only reach for this if Case A and B do not apply. That is, Okta itself is working correctly but a decision has been made to go back to the old system entirely.

This cannot be done with a code rollback alone. The old code expects a real password_hash on every user row. After Step 3 commits, that column is either gone or (if alembic downgrade is run) present but empty. Either way, nobody's stored password survived, including admins'. The only way to get a working local-password system back is to restore the full database from the Step 1 backup, which also rolls back every other change made since that backup: new work packages, comments, uploaded files, everything.

Do not do this without confirming with me first. If it is confirmed, for an encrypted dump, on the Docker host, in the backups directory:

export BACKUP_ENC_PASSPHRASE='<the passphrase, from the stack env vars>'
openssl enc -d -aes-256-cbc -pbkdf2 -pass env:BACKUP_ENC_PASSPHRASE \
  -in wpsuite-<timestamp>.sql.gz.enc \
  | gunzip \
  | docker exec -i wp_db psql -U wpsuite -d wpsuite
unset BACKUP_ENC_PASSPHRASE

For an unencrypted dump, drop the openssl stage and pipe gunzip straight into psql. Substitute the real values if POSTGRES_USER / POSTGRES_DB are not wpsuite. After restoring, redeploy pinned to the Git commit recorded in Step 0, since the restored database matches the old schema, not this one.

Reach me at m.mabrey@prime-controls.com.

---

Notes

• Do not run docker compose down -v. The -v flag deletes the pgdata volume and with it the entire database.
• docker exec <container-name> is used throughout rather than docker compose ..., because a Portainer-managed Git stack's compose project lives under Portainer's own data directory and usually isn't reachable from an ad hoc SSH session the same way.
• There is currently no startup log line confirming Okta config is valid (okta_auth.describe() exists but nothing calls it yet, logged as a follow-up). Step 4's live sign-in is the real verification until that's wired in.
• Full background documentation: DEPLOYMENT.md and server/README.md in the repository. docs/waves/decisions-2026-09-03.md (D16, D17) records why there is no break-glass path and why this runbook exists as its own document.