Matt Mabrey
d6eae0d846
New DEPLOY-runbook-2026-09-03.md, separate from the 2026-08-04 runbook. Names the five new OKTA_* env vars, treats the pre-deploy backup as the only way back once 1d60a608bb51 (drop_local_password) commits since its downgrade() restores the column but not the data, and splits Rollback into the fixable case (Okta app integration misconfigured, fix and redeploy api, no data at risk) versus the severe case (abandoning Okta for local-password code, which only a destructive backup restore can reach). States D16's no-break-glass posture plainly. D17 records the decision and why: staged-deploy-sequence docs and backlog-only were both considered and declined in favor of the runbook. Logged to backlog.md rather than fixed here: okta_auth.describe() has no caller (BL-028), users.failed_attempts/locked_until are vestigial (BL-029), DEPLOY-login-portal.md is fully stale (BL-030).
123 lines
7.1 KiB
Markdown
123 lines
7.1 KiB
Markdown
# Decisions — 2026-09-03
|
|
|
|
## D16: Okta admin bootstrap, break-glass posture, and the real scope of T10.4
|
|
|
|
Raised during hazard review for T10.4 (remove the local password path). D15 settled
|
|
*that* local passwords go away and Okta OIDC is the sole replacement; it did not settle
|
|
how an admin account gets named once there is no password to set, or what happens if
|
|
Okta itself is unreachable. Both are decided here.
|
|
|
|
### Admin bootstrap
|
|
|
|
`server/manage_users.py` stays the bootstrap tool — its own docstring already says so
|
|
("the `/api/auth/users` endpoint needs an existing admin, so you have to bootstrap one
|
|
here") — but it changes from *creating* an account to *promoting* one:
|
|
|
|
- An operator with shell/DB access on the server runs it against an account that
|
|
already signed in through Okta once and was JIT-provisioned by T10.3 (landing at
|
|
`project_user`, per that task). The command sets `role = admin` on that existing row
|
|
by username.
|
|
- It does **not** create a `User` row from scratch and does not touch `password_hash`
|
|
(the column is gone after T10.4's migration).
|
|
|
|
Rejected: minting a brand-new admin row by hand-typed username. `OKTA_IDENTITY_CLAIM`'s
|
|
exact format is still unconfirmed by security (open item carried from D15/wave-10.md).
|
|
A hand-typed username that doesn't exactly match what Okta actually sends produces a
|
|
second, orphaned account instead of promoting the real one. Promoting an
|
|
already-JIT-provisioned row sidesteps that entirely — it never has to guess the future
|
|
claim value.
|
|
|
|
Ongoing (non-bootstrap) admin naming needs no new work: `html/users.js` already has a
|
|
live role dropdown (`roleSelect`, gated by server-supplied `grantable_roles`) that lets
|
|
an existing admin promote any other account, including one JIT-provisioned via Okta.
|
|
That path is unrelated to the password removal and keeps working unchanged.
|
|
|
|
### Break glass
|
|
|
|
No break-glass path, by design. If Okta is unreachable or misconfigured, the app is
|
|
unreachable for everyone, including admins, until Okta is restored.
|
|
|
|
This matches the precedent already on record for the abandoned LDAPS design (D13/D14):
|
|
"LDAPS is the *only* path, no local fallback, no break-glass." Carried forward
|
|
deliberately rather than assumed to still apply, given Okta's failure modes differ from
|
|
an internal LDAP bind — considered and confirmed, not defaulted into.
|
|
|
|
Rejected: a toggleable emergency local login gated behind an env flag. It would
|
|
reintroduce a stored local credential, exactly what D15 exists to eliminate, for a
|
|
scenario (Okta down) judged less likely and less costly than the standing risk of a
|
|
forgotten emergency backdoor.
|
|
|
|
The server-shell CLI (`manage_users.py`, promoting an existing row) is not a formal
|
|
break-glass mechanism — it cannot help if no account has ever signed in through Okta —
|
|
but it is the same trust tier as "someone with SSH/container access to prod could
|
|
already edit the database directly," and it costs no new engineering.
|
|
|
|
### T10.4 scope correction
|
|
|
|
Hazard review found real call sites of `hash_password` / `verify_password` /
|
|
`password_problem` that the original T10.4 bullet in `wave-10.md` didn't name and that
|
|
break the moment those functions are deleted:
|
|
|
|
- `create_user()` and `admin_reset_password()` in `server/app.py` (the admin console's
|
|
"add user" and "reset password" routes).
|
|
- `html/users.js`'s "add user" form (`nu-password` field) and "Reset password" button.
|
|
- `server/manage_users.py`'s `create`, `create-admin`, and `reset-password` subcommands
|
|
(see bootstrap section above for its replacement).
|
|
- `tests/browser_check.py` and `tests/launcher_check.py`, which call
|
|
`auth.hash_password()` to seed fixture rows.
|
|
|
|
Also found: `server/smoketest.py` and `server/seed_demo.py` authenticate via
|
|
`POST /api/auth/login`, which T10.4 removes, and CLAUDE.md's own verification section
|
|
names both scripts as required checks. Fix folded into T10.4 rather than deferred to
|
|
T10.7: both scripts switch to minting a session with `auth.create_token()` and setting
|
|
the cookie directly, the same technique `tests/browser_check.py` already uses instead
|
|
of scripting a login form. No live Okta tenant needed, and T10.4 no longer depends on
|
|
T10.7's timing.
|
|
|
|
`wave-10.md`'s T10.4 bullet is updated to reflect this full scope.
|
|
|
|
## D17: A dedicated rollback-aware runbook for the Okta cutover deploy
|
|
|
|
Raised while preparing to close out wave 10: T10.4's migration
|
|
(`server/alembic/versions/1d60a608bb51_drop_local_password.py`) drops the
|
|
`password_hash` column, and its `downgrade()` re-adds the column with
|
|
`server_default=''`. That restores the schema, not the data — the real bcrypt hashes
|
|
are destroyed the moment `upgrade()`'s `op.drop_column` commits, and no amount of
|
|
`alembic downgrade` brings them back. `DEPLOY-runbook-2026-08-04.md`, the existing
|
|
precedent for how a deploy of this repo is handed to IT, has no equivalent case in its
|
|
own Rollback section — its migrations are additive or reversible, so nothing there
|
|
warns an operator that this one is different.
|
|
|
|
Three options were on the table: write this runbook now; also document a staged
|
|
deploy sequence (ship T10.5's Okta-live-alongside-password state first, verify real
|
|
sign-ins, then ship T10.4's column drop as a separate follow-up deploy); or log the
|
|
hazard to `backlog.md` and stop there. Decided: write the runbook only. Staged
|
|
sequencing is real risk reduction but is a second deploy plan on top of a wave that is
|
|
otherwise a single cutover (D15's "full replacement," not a toggle) — worth
|
|
proposing on its own if IT wants it, not worth building unasked. Backlog-only was
|
|
rejected because the hazard is concrete and dated (this wave, this migration), not a
|
|
someday item.
|
|
|
|
### What the runbook has to do differently from the 2026-08-04 precedent
|
|
|
|
- Name the five `OKTA_*` environment variables as newly required for this deploy —
|
|
the precedent's own "no new environment variables" note does not apply here and
|
|
restating it unchanged would be actively wrong.
|
|
- Treat the pre-deploy backup (`docker exec wp_db_backup /scripts/db-backup.sh`) as
|
|
the only way back once `1d60a608bb51` commits, not as routine due diligence.
|
|
- Separate two failure modes that look similar but are not: an Okta app integration
|
|
that is misconfigured after a clean deploy (redirect URI, client secret, an
|
|
unassigned test account) is fixable in place — fix the config, redeploy the `api`
|
|
service, no data at risk, migration already applied and stays applied. Deciding to
|
|
abandon Okta and restore local-password code is the severe case — the empty
|
|
`password_hash` column means the old code has nothing to check a password against,
|
|
so the only way back is the destructive backup restore (`DEPLOY-runbook-2026-08-04.md`
|
|
Case C's own procedure, reused here). Conflating these two would send an operator
|
|
straight to a destructive restore for a problem that a config fix would have solved.
|
|
- Fold in D16's no-break-glass posture: if Okta is down or misconfigured, the app is
|
|
down for everyone, including admins, by design — not a defect to roll back from.
|
|
|
|
Filed as **T10.9** in `wave-10.md` rather than folded into T10.6 (already merged and
|
|
verified) or T10.8 (UI/test verification, a different kind of check). New scope found
|
|
after the task that raised it closed gets a new ID, per `CLAUDE.md`.
|