n.siegfried/Project-SDE-WP-Suite
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

T10.9: rollback-aware deploy runbook for the Okta cutover

Browse Source 
New DEPLOY-runbook-2026-09-03.md, separate from the 2026-08-04 runbook.
Names the five new OKTA_* env vars, treats the pre-deploy backup as the
only way back once 1d60a608bb51 (drop_local_password) commits since its
downgrade() restores the column but not the data, and splits Rollback
into the fixable case (Okta app integration misconfigured, fix and
redeploy api, no data at risk) versus the severe case (abandoning Okta
for local-password code, which only a destructive backup restore can
reach). States D16's no-break-glass posture plainly.

D17 records the decision and why: staged-deploy-sequence docs and
backlog-only were both considered and declined in favor of the runbook.

Logged to backlog.md rather than fixed here: okta_auth.describe() has
no caller (BL-028), users.failed_attempts/locked_until are vestigial
(BL-029), DEPLOY-login-portal.md is fully stale (BL-030).
This commit is contained in:
Matt Mabrey
2026-09-03 14:25:18 -07:00
parent d7d1e93dd8
commit d6eae0d846
4 changed files with 471 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

355
DEPLOY-runbook-2026-09-03.md Normal file
View File

@@ -0,0 +1,355 @@
# Deploy runbook: WP Suite Okta cutover (wave 10)
**For:** IT / whoever administers the Docker host and Portainer
**From:** m.mabrey@prime-controls.com
**Revised:** 2026-09-03. First version of this runbook.
**Expected duration:** 20-30 minutes, including the backup and the live sign-in check
**Expected downtime:** under a minute, while containers are recreated
This is a separate runbook from `DEPLOY-runbook-2026-08-04.md`, not a revision of it.
That one still applies to its own deploy. Read this one in full before starting.
It has a step that the earlier one does not: this deploy removes the local password
system entirely, and one part of that removal cannot be undone by the usual means.
See "What this deploy changes" below.
---
## Fill these in before handing this over
| Thing | Value |
|---|---|
| Docker host (SSH target) | `________________` |
| Stack name in Portainer | `________________` |
| Site URL | `https://________________` |
| Stack directory on the host (holds `docker-compose.yml` / `backups/`) | `________________` |
| A real Okta account assigned to the app integration, for the live check in Step 4 | `________________` |
Container names are fixed by the compose file and are the same on every host:
`nginx_webserver`, `wp_api`, `wp_db`, `wp_db_backup`.
---
## What this deploy changes
Sign-in changes from a local username/password to Okta OIDC, completely. Not a
toggle, not a fallback. Three things make this more than a routine deploy:
1. **Five new environment variables are required**, and without them nobody can
sign in: `OKTA_ISSUER`, `OKTA_CLIENT_ID`, `OKTA_CLIENT_SECRET`,
`OKTA_REDIRECT_URI`, and optionally `OKTA_IDENTITY_CLAIM`. This is a genuine
change from how deploys usually go here. Do not skip Step 1.5.
2. **A migration drops the `password_hash` column** (`1d60a608bb51`), and it is
**one-way in practice.** Its `downgrade()` re-adds the column, but empty.
The real password hashes are gone the moment this commits, and no `alembic
downgrade` brings them back. If this deploy needs to be undone after that
point, going back to the old local-password code does not work on its own.
See the Rollback section, Case C. This is why Step 1's backup is not
optional the way it sometimes reads in other runbooks.
3. **There is no break-glass path**, by design (recorded decision D16). If Okta
is unreachable or misconfigured after this deploy, the app is unreachable for
everyone, admins included, until Okta is fixed. That is expected behavior,
not a bug to roll back from. See Rollback, Case A/B, before assuming
something is broken.
The visible change for people using the app: the login page becomes "Sign in
with Okta" instead of a username/password form. Nothing else in the app's
day-to-day behavior changes.
---
## Step 0: Record the current state (needed for rollback)
SSH to the Docker host and run:
```bash
docker exec wp_api alembic -c server/alembic.ini current
docker inspect nginx_webserver --format 'nginx image: {{.Image}}'
docker inspect wp_api --format 'api image: {{.Image}}'
```
**Copy the output into your ticket.** Also note the Git commit the Portainer
stack is currently on (Portainer, the stack, the Git reference / last-updated
commit). Without these, rollback is guesswork.
Expected output of the first command before this deploy: `a1b8c6d4e2f9 (head)`.
If it shows anything else, stop and check with me before continuing. This
runbook assumes that starting point.
---
## Step 1: Back up the database
On the Docker host:
```bash
docker exec wp_db_backup /scripts/db-backup.sh
```
Expected output ends with a line like:
```
[db-backup] wrote 1.4M /backups/wpsuite-20260903-141233Z.sql.gz.enc
```
Confirm the file is on the host (substitute the stack directory):
```bash
ls -lt <stack-dir>/backups | head -3
```
**Record that filename.** Do not continue until you have seen the `wrote ...`
line and the file in that listing.
This backup matters more than usual for this deploy. Once the migration in
Step 3 commits, this file becomes the *only* way to get local password hashes
back, for any reason. Treat it as the point you would restore to, not routine
housekeeping.
- A `.sql.gz.enc` extension means backups are encrypted. Expected and correct.
- A `.sql.gz` extension plus a `WARNING: BACKUP_ENC_PASSPHRASE not set` line
means backups are unencrypted. Not a blocker for this deploy; report it back.
- **No SSH access?** Portainer, **Containers**, `wp_db_backup`, **Console**,
connect with `/bin/sh`, then run `/scripts/db-backup.sh`. Same result: the
dump lands on the host, because `/backups` is a bind mount.
---
## Step 1.5: Confirm the Okta app integration is actually ready
Do this before redeploying, not after. Everything here is checked in Okta's own
admin console and in the values that will go into the stack's environment
variables. Nothing touches the WP Suite host yet.
1. The Okta app integration exists (Sign-in method: OIDC, Authorization Code,
Application type: Web Application), and the account listed in the fill-in
table above is assigned to it.
2. `OKTA_REDIRECT_URI` matches a "Sign-in redirect URI" registered on that app
integration **exactly**: scheme, host, and path, including whether it ends
in `/api/auth/okta/callback`.
3. `OKTA_ISSUER`, `OKTA_CLIENT_ID`, and `OKTA_CLIENT_SECRET` are the values from
that same app integration, not a different one.
4. If your Okta configuration puts the directory identity somewhere other than
the `preferred_username` claim, `OKTA_IDENTITY_CLAIM` is set to the right
claim name. If unsure, leave it unset; `preferred_username` is the default.
Add all five to the stack's **Environment variables** in Portainer now, before
Step 2. `OKTA_CLIENT_SECRET` should be handled the same way `AUTH_SECRET_KEY`
already is: not typed anywhere it will be logged.
If any of items 1-3 above are not yet confirmed, stop here and get them
confirmed first. A wrong redirect URI or an unassigned account will not corrupt
anything, but it does mean nobody signs in after this deploy until it is fixed.
See Rollback, Case B, which is the ordinary way that gets fixed and does not
involve the database at all.
---
## Step 2: Redeploy the stack in Portainer
1. Portainer, **Stacks**, select the stack.
2. **Pull and redeploy**, with re-pull / re-build **enabled**.
3. Wait for it to report success.
A plain "restart" or "stop/start" will not pick up new code, and will not pick
up the environment variables added in Step 1.5 either.
---
## Step 3: Confirm the containers came up
```bash
docker ps --filter name=nginx_webserver --filter name=wp_api --filter name=wp_db
```
All three must be `Up`, and `wp_db` should show `(healthy)`. Then check the API
applied its migration cleanly:
```bash
docker logs wp_api --tail 40
```
You are looking for an Alembic `Running upgrade a1b8c6d4e2f9 -> 1d60a608bb51`
line followed by gunicorn starting up, and no traceback. The API refuses to
start if a migration fails, so a restarting `wp_api` container means it failed.
Go to Rollback, Case B, and read the "did the migration commit" note there
before doing anything to the database.
Confirm the database landed on the new revision:
```bash
docker exec wp_api alembic -c server/alembic.ini current
```
Expected: `1d60a608bb51 (head)`. **Once you see this, you have passed the
point of no return described above.** The backup from Step 1 is now the only
way back to a working local-password system, if that is ever needed.
Then verify nginx's own view of its config:
```bash
docker exec nginx_webserver nginx -t
```
Expected: `syntax is ok` / `test is successful`.
---
## Step 4: Confirm Okta sign-in actually works, live
This is the step that matters most for this deploy. A clean container start
does not by itself prove sign-in works, and there is currently no startup log
line that confirms Okta config is good (logged separately as a follow-up, not
fixed as part of this runbook). The only real proof is a live sign-in.
1. Open the site's normal URL in a private/incognito window. It should land on
`login.html` with a "Sign in with Okta" button, not a username/password
form.
2. Click it. You should be redirected to your organization's actual Okta
sign-in page (the real Okta domain from `OKTA_ISSUER`, not this app's own
domain).
3. Sign in with the account from the fill-in table. You should land back on
the WP Suite site, signed in.
4. If this is the account's first-ever sign-in, it is now JIT-provisioned as a
regular user (`project_user`). To make it an admin, on the Docker host:
```bash
docker exec -it wp_api python -m server.manage_users promote <username> --role admin
```
This only works on an account that has already signed in once through Okta.
It promotes an existing row; it does not create one. That is deliberate
(recorded decision D16): there is no other admin-bootstrap path.
If step 2 or 3 fails (redirected to an Okta error page, redirected back to
`login.html` with an error, or nothing happens), this is almost always a
configuration problem from Step 1.5, not a code or database problem. Go to
Rollback, Case B, before considering anything more drastic.
Also confirm the API is reachable through the proxy and the redirect itself is
wired up:
```bash
curl -s https://<site-url>/api/health # -> {"ok": true}
curl -sI https://<site-url>/api/auth/okta/login | grep -i ^location # -> your Okta authorize URL
```
---
## Step 5: Hard-reload once in a browser, then sanity-check
Press **Ctrl+Shift+R** (Cmd+Shift+R on macOS) once. The app uses a service
worker; a normal reload can serve the previous version.
1. Signed in as the account from Step 4, the home page offers to select or
create a project, same as before.
2. Open **Admin Console** as the promoted admin account. The user table shows
the account you just signed in with. There is no password column, no
"reset password" action anywhere in the UI.
3. Open a project and confirm a work package can be opened and edited
normally. Sign-in is the only thing this deploy changes, so the rest of
the app should look untouched.
**Deploy complete.** Please report back: the Step 0 output, the backup
filename from Step 1, and confirmation that Step 4's live sign-in worked.
---
## Rollback
Read this before assuming a rollback is needed. Cases A and B below do **not**
touch the database and are the far more likely outcome of something going
wrong here. Okta configuration is fiddly and easy to get slightly wrong. Case
C is the severe, destructive one, and should be a last resort, not a first
reaction.
### Case A: nginx won't start, or containers won't come up at all
Same as any other deploy: the database is untouched by container start-up
failures. In Portainer, redeploy the stack pinned to the **previous Git
commit** recorded in Step 0, then re-run Step 3.
```bash
docker logs nginx_webserver --tail 100
docker logs wp_api --tail 100
```
Send me whichever of those is relevant.
### Case B: containers are up, but Okta sign-in doesn't work
This is a configuration problem, not a data problem, and does **not** need a
code rollback or a database restore. Check, in order:
1. Is `wp_api`'s log showing anything at all when a sign-in is attempted?
`docker logs wp_api --tail 100`.
2. Do the five `OKTA_*` values in the stack's environment variables actually
match the Okta app integration (Step 1.5)? A copy-paste error in
`OKTA_CLIENT_SECRET` or a redirect URI that's off by a trailing slash are
the two most common causes.
3. Is the account assigned to the Okta app integration? An unassigned account
gets denied by Okta itself, before it ever reaches this app.
4. If a specific person can't sign in but others can, check
`OKTA_IDENTITY_CLAIM`. The claim it reads may not carry that person's
directory identity in the format expected. Confirm with security which
claim Okta is actually issuing.
Fix the environment variable(s) in Portainer, then redeploy (Pull and redeploy
is fine; the migration already applied and does not run again). No backup
restore, no code rollback.
If it's still not working after checking all four, send me the `wp_api` log
from item 1 along with which of items 2-4 you already ruled out.
### Case C: the decision is made to abandon Okta and restore local-password sign-in
This is the case the point-of-no-return warning in "What this deploy changes"
is about. Only reach for this if Case A and B do not apply. That is, Okta
itself is working correctly but a decision has been made to go back to the old
system entirely.
**This cannot be done with a code rollback alone.** The old code expects a
real `password_hash` on every user row. After Step 3 commits, that column is
either gone or (if `alembic downgrade` is run) present but empty. Either way,
nobody's stored password survived, including admins'. The only way to get a
working local-password system back is to restore the full database from the
Step 1 backup, which also rolls back every other change made since that
backup: new work packages, comments, uploaded files, everything.
**Do not do this without confirming with me first.** If it is confirmed, for
an encrypted dump, on the Docker host, in the `backups` directory:
```bash
export BACKUP_ENC_PASSPHRASE='<the passphrase, from the stack env vars>'
openssl enc -d -aes-256-cbc -pbkdf2 -pass env:BACKUP_ENC_PASSPHRASE \
-in wpsuite-<timestamp>.sql.gz.enc \
| gunzip \
| docker exec -i wp_db psql -U wpsuite -d wpsuite
unset BACKUP_ENC_PASSPHRASE
```
For an unencrypted dump, drop the `openssl` stage and pipe `gunzip` straight
into `psql`. Substitute the real values if `POSTGRES_USER` / `POSTGRES_DB` are
not `wpsuite`. After restoring, redeploy pinned to the Git commit recorded in
Step 0, since the restored database matches the old schema, not this one.
Reach me at m.mabrey@prime-controls.com.
---
## Notes
- Do not run `docker compose down -v`. The `-v` flag deletes the `pgdata`
volume and with it the entire database.
- `docker exec <container-name>` is used throughout rather than
`docker compose ...`, because a Portainer-managed Git stack's compose project
lives under Portainer's own data directory and usually isn't reachable from
an ad hoc SSH session the same way.
- There is currently no startup log line confirming Okta config is valid
(`okta_auth.describe()` exists but nothing calls it yet, logged as a
follow-up). Step 4's live sign-in is the real verification until that's
wired in.
- Full background documentation: `DEPLOYMENT.md` and `server/README.md` in the
repository. `docs/waves/decisions-2026-09-03.md` (D16, D17) records why
there is no break-glass path and why this runbook exists as its own document.

46
docs/waves/backlog.md
View File

@@ -594,3 +594,49 @@ deliberately deferred.
and pinned; the systematic guard is its own small task, not a hotfix rider.
- **Suggested wave or follow-up:** next housekeeping pass, paired with BL-026
(both are "deploys should be boring" work).
### BL-028 — `okta_auth.describe()` is never called
- **Found during:** T10.9 (writing the deploy runbook's live-verification step)
- **Where:** `server/okta_auth.py` (`describe()`), `server/app.py` (no caller anywhere)
- **What:** `describe()` exists specifically to shout `"*** FAKE OKTA PROVIDER
ACTIVE..."` or report missing config at a glance, and both
`server/.env.example` and `server/README.md` tell an operator to "check the
startup log line" for it. Nothing prints it. `app.py` never imports or calls
`describe()` at process start, so that log line does not exist and an operator
following the docs will not find it.
- **Why not now:** a runbook is documentation, not server code; wiring a
startup log call is a real (if small) change to `app.py` and wants its own
diff and its own verification, not a rider on T10.9.
- **Suggested wave or follow-up:** next housekeeping pass. One call
(`logger.info(okta_auth.describe())` near startup) plus updating
`DEPLOY-runbook-2026-09-03.md` Step 4 to check the log line once it exists.
### BL-029 — `users.failed_attempts` / `users.locked_until` are vestigial
- **Found during:** T10.6
- **Where:** `server/models.py` (`User.failed_attempts`, `User.locked_until`)
- **What:** both columns exist to support local-password lockout, which T10.4
removed. They are still reset to `0`/`None` on every Okta sign-in but nothing
increments them anymore — dead columns, not a bug, but schema drift from the
D15 cutover.
- **Why not now:** T10.6 is documentation scope; dropping columns is a migration
and belongs with the rest of the local-password cleanup, not folded into a
docs task.
- **Suggested wave or follow-up:** next housekeeping pass, alongside any other
post-cutover schema tidy-up.
### BL-030 — `DEPLOY-login-portal.md` is fully stale
- **Found during:** T10.9
- **Where:** `DEPLOY-login-portal.md` (repo root)
- **What:** the original username/password login rollout doc. References
`bcrypt`, `create-admin --password`, `AUTH_SECRET_KEY` as the only secret,
and a login form — none of which describe the app since T10.4/T10.5. Someone
handed this to IT today would be told to do things that no longer work.
- **Why not now:** no task currently owns deploy-doc cleanup as a category;
deleting or archiving a doc is a product/records call (`CLAUDE.md`'s "removed
fields are hidden, not deleted" spirit likely applies to docs too, but that is
worth confirming rather than assuming).
- **Suggested wave or follow-up:** next housekeeping pass — needs Nick on
whether to delete, archive, or rewrite it as historical record.

45
docs/waves/decisions-2026-09-03.md
View File

@@ -75,3 +75,48 @@ of scripting a login form. No live Okta tenant needed, and T10.4 no longer depen
T10.7's timing.
`wave-10.md`'s T10.4 bullet is updated to reflect this full scope.
## D17: A dedicated rollback-aware runbook for the Okta cutover deploy
Raised while preparing to close out wave 10: T10.4's migration
(`server/alembic/versions/1d60a608bb51_drop_local_password.py`) drops the
`password_hash` column, and its `downgrade()` re-adds the column with
`server_default=''`. That restores the schema, not the data — the real bcrypt hashes
are destroyed the moment `upgrade()`'s `op.drop_column` commits, and no amount of
`alembic downgrade` brings them back. `DEPLOY-runbook-2026-08-04.md`, the existing
precedent for how a deploy of this repo is handed to IT, has no equivalent case in its
own Rollback section — its migrations are additive or reversible, so nothing there
warns an operator that this one is different.
Three options were on the table: write this runbook now; also document a staged
deploy sequence (ship T10.5's Okta-live-alongside-password state first, verify real
sign-ins, then ship T10.4's column drop as a separate follow-up deploy); or log the
hazard to `backlog.md` and stop there. Decided: write the runbook only. Staged
sequencing is real risk reduction but is a second deploy plan on top of a wave that is
otherwise a single cutover (D15's "full replacement," not a toggle) — worth
proposing on its own if IT wants it, not worth building unasked. Backlog-only was
rejected because the hazard is concrete and dated (this wave, this migration), not a
someday item.
### What the runbook has to do differently from the 2026-08-04 precedent
- Name the five `OKTA_*` environment variables as newly required for this deploy —
the precedent's own "no new environment variables" note does not apply here and
restating it unchanged would be actively wrong.
- Treat the pre-deploy backup (`docker exec wp_db_backup /scripts/db-backup.sh`) as
the only way back once `1d60a608bb51` commits, not as routine due diligence.
- Separate two failure modes that look similar but are not: an Okta app integration
that is misconfigured after a clean deploy (redirect URI, client secret, an
unassigned test account) is fixable in place — fix the config, redeploy the `api`
service, no data at risk, migration already applied and stays applied. Deciding to
abandon Okta and restore local-password code is the severe case — the empty
`password_hash` column means the old code has nothing to check a password against,
so the only way back is the destructive backup restore (`DEPLOY-runbook-2026-08-04.md`
Case C's own procedure, reused here). Conflating these two would send an operator
straight to a destructive restore for a problem that a config fix would have solved.
- Fold in D16's no-break-glass posture: if Okta is down or misconfigured, the app is
down for everyone, including admins, by design — not a defect to roll back from.
Filed as **T10.9** in `wave-10.md` rather than folded into T10.6 (already merged and
verified) or T10.8 (UI/test verification, a different kind of check). New scope found
after the task that raised it closed gets a new ID, per `CLAUDE.md`.

25
docs/waves/wave-10.md
View File

@@ -120,6 +120,31 @@ Depends only on `main` as it stands after `D15`. Not sequenced behind any other
- **T10.8 — Verification.** 390px and 1440px, full suite, done-when checks per task,
matching the rigor D13 was held to.
- **T10.9 — Rollback-aware deploy runbook.** Raised after hazard review found
`DEPLOY-runbook-2026-08-04.md`'s Rollback section has no case for a migration whose
`downgrade()` cannot restore the data it drops — see `D17`. `T10.4`'s
`1d60a608bb51_drop_local_password` is exactly that: the schema comes back, the
bcrypt hashes do not.
Built: `DEPLOY-runbook-2026-09-03.md`, following the 2026-08-04 runbook's structure
(fill-in table, numbered deploy steps, case-by-case Rollback section, Notes). Names
the five `OKTA_*` vars as newly required (the precedent's "no new environment
variables" note does not carry over), treats the pre-deploy backup as the only way
back once the migration commits, and splits Rollback into the fixable case (Okta app
integration misconfigured — fix and redeploy `api`, no data at risk, migration stays
applied) versus the severe case (abandoning Okta for local-password code — only the
destructive backup restore gets there, reusing the 2026-08-04 runbook's own Case C
procedure). D16's no-break-glass posture is stated plainly rather than left implicit.
Logged, not fixed here (out of scope): `okta_auth.describe()`'s startup log line
(referenced by `DEPLOYMENT.md`/`server/README.md`) has no caller anywhere in
`server/app.py` — nothing actually prints it at process start. The runbook's Step 4
therefore verifies via a live Okta sign-in rather than a log line, and this gap is
flagged in `docs/waves/backlog.md` as a candidate fix (wiring `describe()` into
startup) since it directly bears on deploy verifiability. `DEPLOY-login-portal.md`
is now fully stale (bcrypt, `create-admin --password`, none of which still exist) —
not touched, no task claims it.
## Still open
- The OIDC claim mapping (`T10.3`).