Project-SDE-WP-Suite/docs/waves/decisions-2026-09-03.md

Decisions — 2026-09-03

D16: Okta admin bootstrap, break-glass posture, and the real scope of T10.4

Raised during hazard review for T10.4 (remove the local password path). D15 settled that local passwords go away and Okta OIDC is the sole replacement; it did not settle how an admin account gets named once there is no password to set, or what happens if Okta itself is unreachable. Both are decided here.

Admin bootstrap

server/manage_users.py stays the bootstrap tool — its own docstring already says so ("the /api/auth/users endpoint needs an existing admin, so you have to bootstrap one here") — but it changes from creating an account to promoting one:

• An operator with shell/DB access on the server runs it against an account that already signed in through Okta once and was JIT-provisioned by T10.3 (landing at project_user, per that task). The command sets role = admin on that existing row by username.
• It does not create a User row from scratch and does not touch password_hash (the column is gone after T10.4's migration).

Rejected: minting a brand-new admin row by hand-typed username. OKTA_IDENTITY_CLAIM's exact format is still unconfirmed by security (open item carried from D15/wave-10.md). A hand-typed username that doesn't exactly match what Okta actually sends produces a second, orphaned account instead of promoting the real one. Promoting an already-JIT-provisioned row sidesteps that entirely — it never has to guess the future claim value.

Ongoing (non-bootstrap) admin naming needs no new work: html/users.js already has a live role dropdown (roleSelect, gated by server-supplied grantable_roles) that lets an existing admin promote any other account, including one JIT-provisioned via Okta. That path is unrelated to the password removal and keeps working unchanged.

Break glass

No break-glass path, by design. If Okta is unreachable or misconfigured, the app is unreachable for everyone, including admins, until Okta is restored.

This matches the precedent already on record for the abandoned LDAPS design (D13/D14): "LDAPS is the only path, no local fallback, no break-glass." Carried forward deliberately rather than assumed to still apply, given Okta's failure modes differ from an internal LDAP bind — considered and confirmed, not defaulted into.

Rejected: a toggleable emergency local login gated behind an env flag. It would reintroduce a stored local credential, exactly what D15 exists to eliminate, for a scenario (Okta down) judged less likely and less costly than the standing risk of a forgotten emergency backdoor.

The server-shell CLI (manage_users.py, promoting an existing row) is not a formal break-glass mechanism — it cannot help if no account has ever signed in through Okta — but it is the same trust tier as "someone with SSH/container access to prod could already edit the database directly," and it costs no new engineering.

T10.4 scope correction

Hazard review found real call sites of hash_password / verify_password / password_problem that the original T10.4 bullet in wave-10.md didn't name and that break the moment those functions are deleted:

• create_user() and admin_reset_password() in server/app.py (the admin console's "add user" and "reset password" routes).
• html/users.js's "add user" form (nu-password field) and "Reset password" button.
• server/manage_users.py's create, create-admin, and reset-password subcommands (see bootstrap section above for its replacement).
• tests/browser_check.py and tests/launcher_check.py, which call auth.hash_password() to seed fixture rows.

Also found: server/smoketest.py and server/seed_demo.py authenticate via POST /api/auth/login, which T10.4 removes, and CLAUDE.md's own verification section names both scripts as required checks. Fix folded into T10.4 rather than deferred to T10.7: both scripts switch to minting a session with auth.create_token() and setting the cookie directly, the same technique tests/browser_check.py already uses instead of scripting a login form. No live Okta tenant needed, and T10.4 no longer depends on T10.7's timing.

wave-10.md's T10.4 bullet is updated to reflect this full scope.

D17: A dedicated rollback-aware runbook for the Okta cutover deploy

Raised while preparing to close out wave 10: T10.4's migration (server/alembic/versions/1d60a608bb51_drop_local_password.py) drops the password_hash column, and its downgrade() re-adds the column with server_default=''. That restores the schema, not the data — the real bcrypt hashes are destroyed the moment upgrade()'s op.drop_column commits, and no amount of alembic downgrade brings them back. DEPLOY-runbook-2026-08-04.md, the existing precedent for how a deploy of this repo is handed to IT, has no equivalent case in its own Rollback section — its migrations are additive or reversible, so nothing there warns an operator that this one is different.

Three options were on the table: write this runbook now; also document a staged deploy sequence (ship T10.5's Okta-live-alongside-password state first, verify real sign-ins, then ship T10.4's column drop as a separate follow-up deploy); or log the hazard to backlog.md and stop there. Decided: write the runbook only. Staged sequencing is real risk reduction but is a second deploy plan on top of a wave that is otherwise a single cutover (D15's "full replacement," not a toggle) — worth proposing on its own if IT wants it, not worth building unasked. Backlog-only was rejected because the hazard is concrete and dated (this wave, this migration), not a someday item.

What the runbook has to do differently from the 2026-08-04 precedent

• Name the five OKTA_* environment variables as newly required for this deploy — the precedent's own "no new environment variables" note does not apply here and restating it unchanged would be actively wrong.
• Treat the pre-deploy backup (docker exec wp_db_backup /scripts/db-backup.sh) as the only way back once 1d60a608bb51 commits, not as routine due diligence.
• Separate two failure modes that look similar but are not: an Okta app integration that is misconfigured after a clean deploy (redirect URI, client secret, an unassigned test account) is fixable in place — fix the config, redeploy the api service, no data at risk, migration already applied and stays applied. Deciding to abandon Okta and restore local-password code is the severe case — the empty password_hash column means the old code has nothing to check a password against, so the only way back is the destructive backup restore (DEPLOY-runbook-2026-08-04.md Case C's own procedure, reused here). Conflating these two would send an operator straight to a destructive restore for a problem that a config fix would have solved.
• Fold in D16's no-break-glass posture: if Okta is down or misconfigured, the app is down for everyone, including admins, by design — not a defect to roll back from.

Filed as T10.9 in wave-10.md rather than folded into T10.6 (already merged and verified) or T10.8 (UI/test verification, a different kind of check). New scope found after the task that raised it closed gets a new ID, per CLAUDE.md.