T10.9: rollback-aware deploy runbook for the Okta cutover

New DEPLOY-runbook-2026-09-03.md, separate from the 2026-08-04 runbook.
Names the five new OKTA_* env vars, treats the pre-deploy backup as the
only way back once 1d60a608bb51 (drop_local_password) commits since its
downgrade() restores the column but not the data, and splits Rollback
into the fixable case (Okta app integration misconfigured, fix and
redeploy api, no data at risk) versus the severe case (abandoning Okta
for local-password code, which only a destructive backup restore can
reach). States D16's no-break-glass posture plainly.

D17 records the decision and why: staged-deploy-sequence docs and
backlog-only were both considered and declined in favor of the runbook.

Logged to backlog.md rather than fixed here: okta_auth.describe() has
no caller (BL-028), users.failed_attempts/locked_until are vestigial
(BL-029), DEPLOY-login-portal.md is fully stale (BL-030).
This commit is contained in:
2026-09-03 14:25:18 -07:00
parent d7d1e93dd8
commit d6eae0d846
4 changed files with 471 additions and 0 deletions

View File

@@ -75,3 +75,48 @@ of scripting a login form. No live Okta tenant needed, and T10.4 no longer depen
T10.7's timing.
`wave-10.md`'s T10.4 bullet is updated to reflect this full scope.
## D17: A dedicated rollback-aware runbook for the Okta cutover deploy
Raised while preparing to close out wave 10: T10.4's migration
(`server/alembic/versions/1d60a608bb51_drop_local_password.py`) drops the
`password_hash` column, and its `downgrade()` re-adds the column with
`server_default=''`. That restores the schema, not the data — the real bcrypt hashes
are destroyed the moment `upgrade()`'s `op.drop_column` commits, and no amount of
`alembic downgrade` brings them back. `DEPLOY-runbook-2026-08-04.md`, the existing
precedent for how a deploy of this repo is handed to IT, has no equivalent case in its
own Rollback section — its migrations are additive or reversible, so nothing there
warns an operator that this one is different.
Three options were on the table: write this runbook now; also document a staged
deploy sequence (ship T10.5's Okta-live-alongside-password state first, verify real
sign-ins, then ship T10.4's column drop as a separate follow-up deploy); or log the
hazard to `backlog.md` and stop there. Decided: write the runbook only. Staged
sequencing is real risk reduction but is a second deploy plan on top of a wave that is
otherwise a single cutover (D15's "full replacement," not a toggle) — worth
proposing on its own if IT wants it, not worth building unasked. Backlog-only was
rejected because the hazard is concrete and dated (this wave, this migration), not a
someday item.
### What the runbook has to do differently from the 2026-08-04 precedent
- Name the five `OKTA_*` environment variables as newly required for this deploy —
the precedent's own "no new environment variables" note does not apply here and
restating it unchanged would be actively wrong.
- Treat the pre-deploy backup (`docker exec wp_db_backup /scripts/db-backup.sh`) as
the only way back once `1d60a608bb51` commits, not as routine due diligence.
- Separate two failure modes that look similar but are not: an Okta app integration
that is misconfigured after a clean deploy (redirect URI, client secret, an
unassigned test account) is fixable in place — fix the config, redeploy the `api`
service, no data at risk, migration already applied and stays applied. Deciding to
abandon Okta and restore local-password code is the severe case — the empty
`password_hash` column means the old code has nothing to check a password against,
so the only way back is the destructive backup restore (`DEPLOY-runbook-2026-08-04.md`
Case C's own procedure, reused here). Conflating these two would send an operator
straight to a destructive restore for a problem that a config fix would have solved.
- Fold in D16's no-break-glass posture: if Okta is down or misconfigured, the app is
down for everyone, including admins, by design — not a defect to roll back from.
Filed as **T10.9** in `wave-10.md` rather than folded into T10.6 (already merged and
verified) or T10.8 (UI/test verification, a different kind of check). New scope found
after the task that raised it closed gets a new ID, per `CLAUDE.md`.