Files
Project-SDE-WP-Suite/docs/waves/wave-13.md
Matt Mabrey f30989d4b8 T13.4: run the Okta sync on a schedule (D18)
A daemon timer thread starts from a new lifespan hook in app.py and calls the
T13.3 sync every OKTA_SYNC_INTERVAL_SECONDS (default 900, minimum 300). With no
OKTA_API_TOKEN it starts nothing and the startup log says so.

The api runs 2 gunicorn workers, so each starts its own timer. A single atomic
conditional UPDATE on one app_settings row (key okta_sync) is the gate: exactly
one worker wins each cycle, on SQLite and Postgres, with no lock held. The row
survives a restart, so a restart inside the interval starts no second run. A
fetch is now capped at 120 s in total, so a run is always shorter than the
minimum interval and cannot overlap the next. No migration needed.

docker-compose.yml passes OKTA_API_TOKEN, OKTA_ORG_URL,
OKTA_SYNC_PROFILE_ATTRIBUTE and OKTA_SYNC_INTERVAL_SECONDS to the api service.

tests/okta_sync_schedule_check.py: 38 checks including an 8-worker race on a
file database (exactly one winner, 16 rounds) and fail-closed through the
scheduled path. okta_auth_check.py still 22/22.
2026-10-16 11:34:13 -07:00

16 KiB

Wave 13 — Okta/AD deprovisioning sync

Items: D18 Depends on: wave 10 merged (it is). Not blocked by wave 11 or wave 12, but shares server/app.py account-state surface with wave 12 (CR-020) — sequence commits to avoid an avoidable conflict. Decision record: docs/waves/decisions-2026-09-17.md

Six tasks, in build order. T13.3's fail-closed behavior is the most important done-when list in this wave — do not relax it to ship faster.


T13.1 — D18: idle timeout with an absolute ceiling

  • Items: D18
  • Depends on: nothing (first task, independent of the rest)
  • Blocks: nothing
  • Surface: server/
  • Files: server/auth.py, server/app.py (auth_gate middleware), server/.env.example, DEPLOYMENT.md

Revised 2026-09-23 — originally just "shrink AUTH_SESSION_HOURS". Matt asked whether an idle timeout would be a better fit than a flat session length. It is, but not by itself — see the decision record's reasoning on why idle-with-no-ceiling is actually a worse fit for this item's own threat model than a flat expiry would have been. Build both.

Do:

  • Add login_at to the JWT payload in create_token() — the original sign-in time, distinct from iat, which becomes "when THIS token was issued" once tokens start getting reissued. login_at never changes across reissues; it's what the absolute ceiling is measured from.
  • Add AUTH_IDLE_MINUTES (default 30). AUTH_SESSION_HOURS stays the name for the absolute ceiling, default changing from 12 to a proposed 8 — update its docstring/comment in auth.py and .env.example, since its MEANING is changing (session length -> hard ceiling on top of a sliding idle window), not just its value.
  • In auth_gate (server/app.py), after confirming a request is authenticated: if now < login_at + AUTH_SESSION_HOURS (the ceiling hasn't passed), compute new_exp = min(now + AUTH_IDLE_MINUTES, login_at + AUTH_SESSION_HOURS). If new_exp is meaningfully later than the current token's exp (throttle this — do not reissue on every single request, only when enough time has passed to be worth a new cookie; a few minutes of slack is fine), mint a refreshed token carrying forward sub/username/ role/ver/login_at unchanged, and set it on the response.
  • If the ceiling HAS passed, do not refresh — let the existing token expire on its own terms (it may already be invalid, or may tick over within the idle window; either way, no new one is issued past the ceiling).
  • The middleware should not need a DB round trip to do this — everything needed (sub, username, role, ver, login_at) is already in the validated claims. get_current_user's existing per-request is_active/ token_version check is unaffected and still runs separately.

Do not: reissue the cookie on every request unconditionally — that's a Set-Cookie header on every API call for no benefit over a coarser refresh. Do not let the ceiling check silently vanish for tokens issued before this change ships — a token with no login_at claim should fall back to treating its own iat as login_at, not bypass the ceiling entirely.

Done when:

  • a session with continuous activity stays alive past 30 minutes but is cut off at the AUTH_SESSION_HOURS ceiling regardless
  • a session with no activity for 30+ minutes is rejected on its next request
  • the cookie is not rewritten on every single request — verify the refresh is throttled, not unconditional
  • a pre-existing token with no login_at claim (simulating a session issued before this shipped) still gets a hard ceiling, via the iat fallback
  • .env.example and DEPLOYMENT.md explain both variables and flag both defaults as proposed, not confirmed against Okta's own session policy
  • existing session/auth tests updated for the new mechanism, not just a new number

T13.2 — D18: Okta Management API client

  • Items: D18
  • Depends on: nothing (independent of T13.1)
  • Blocks: T13.3
  • Surface: server/
  • Files: new server/okta_sync.py (or extend server/okta_auth.py — task's call), server/.env.example

Do: A small client for Okta's user-list endpoint, authenticated with a new, separate credential (e.g. OKTA_API_TOKEN) — not the OIDC client secret used for sign-in. Fetch the full user list (paginated per Okta's API) rather than one-by-one lookups per local user; this app has ~20-odd accounts today, and a list-and-diff is simpler and cheaper than N calls. Return each Okta user's identity-claim value (matching OKTA_IDENTITY_CLAIM, already confirmed live per wave 10) and status.

Follow the existing pattern for external credentials in this repo (MICRON_DB_URL, SMTP_PASSWORD): env-only, never logged, never returned to the browser in an error message.

Do not: reuse OKTA_CLIENT_ID/OKTA_CLIENT_SECRET for this. Sign-in and the management API are different trust boundaries with different scopes; conflating them means a compromise or rotation of one affects the other unnecessarily.

Done when:

  • the client authenticates with its own credential, distinct from the OIDC client (2026-10-16: OKTA_API_TOKEN, sent as Authorization: SSWS. tests/okta_sync_check.py sets a different OIDC client secret and asserts it is never sent.)
  • it fetches the complete Okta user list, handling pagination (2026-10-16: follows Link: rel="next", 200 per page, 2-page case checked. A next link to another host is refused, a chain longer than 50 pages raises instead of cutting the list off, redirects are never followed.)
  • an auth failure or malformed response raises a clear, specific error rather than returning an empty list indistinguishable from "everyone was deprovisioned" — this distinction is what T13.3 depends on (2026-10-16: OktaSyncError for HTTP 401/403/404/429/5xx, redirects, network errors, timeouts, non-JSON, non-list, a bad item (never skipped), and an empty list. 31 failure checks. Mutation test: making the client return [] on an error fails 10 of them.)
  • the credential is never logged or surfaced in any API response (2026-10-16: error messages carry the class name and HTTP status only, never the URL or a header. describe() has no token. Nothing is wired to an API response yet; T13.4 wires the startup log.)

T13.3 — D18: the sync job

  • Items: D18
  • Depends on: T13.2
  • Blocks: T13.4
  • Surface: server/
  • Files: server/okta_sync.py, server/app.py (or wherever log_event lives)

Do: Compare the Okta user list (T13.2) against local users rows. For any local is_active=True user whose Okta identity is missing from the list, or present with a non-active status, set is_active=False and write an AuditLog row (actor="system:okta_sync", action e.g. user_deprovisioned_by_sync, detail naming the Okta status found). Never touch a user already is_active=False. Never re-enable anyone.

The fail-closed rule, non-negotiable: if T13.2's client raises an error of any kind (network, auth, malformed response, timeout), this task takes no action on any account for that run and logs the failure clearly (server log, at minimum). An error is never treated as "Okta returned zero active users." Write a test that asserts this directly: feed the sync a failing client and assert zero rows changed and zero AuditLog entries written.

Done when:

  • a user missing from Okta's list, or present but not active, is disabled with a correctly-detailed audit row (2026-10-16: see "Decided 2026-10-16" below for which statuses count as "not active". Audit row: actor system:okta_sync, action user_deprovisioned_by_sync, entity_type user, detail {okta_status, via}; a missing user is recorded as MISSING.)
  • a user already disabled is left alone (no duplicate audit row each run) (2026-10-16: a second run over the same data writes no new rows.)
  • an active Okta user already active locally produces no audit row (only changes are logged, not a clean bill of health every cycle) (2026-10-16: verified in tests/okta_sync_job_check.py.)
  • a simulated Okta API failure results in zero account changes and zero audit rows — this is the one check that must never be skipped or weakened (2026-10-16: 10 stand-in failures (OktaSyncError, RuntimeError, ConnectionError, TimeoutError, ValueError, KeyError, empty list, None, two kinds of garbage), 4 through the real client (HTTP 401/403/429/500), the default path with no token, a failed commit, and a wrong identity attribute. Each: ok is False, zero accounts changed, zero audit rows. Mutation test: removing the fail-closed branch and the zero-match guard fails 28 of these.)
  • the job never re-enables an account under any input (2026-10-16: a disabled account that is ACTIVE in Okta stays disabled, and okta_sync.py contains no assignment of is_active to True.)

Decided 2026-10-16 (Matt, in chat): which Okta statuses disable an account. The spec says "a non-active status". Read literally, that also catches PASSWORD_EXPIRED, LOCKED_OUT and RECOVERY, which people leave on their own, and this job never re-enables, so one AD password expiry would lock someone out of the app until an admin stepped in. Decided: an account disables when it is missing from Okta's list, or every Okta entry for it is SUSPENDED or DEPROVISIONED (DISABLING_STATUSES in server/okta_sync.py). Okta's default user list already leaves out DEPROVISIONED users, so those arrive as "missing".

Added 2026-10-16, inside decision 4's "anything the sync can't confidently parse": if there are active local accounts and none of them matches any Okta user, the run is refused and nothing changes. That pattern means the identity attribute is wrong (OKTA_SYNC_PROFILE_ATTRIBUTE), not that the whole company left. It is a guard, not a new feature. Raise it in the PR.

Open, for the PR / Nick: which Okta profile attribute equals the preferred_username claim in this tenant. The client assumes profile.login, Okta's default. Confirm before the first live run. A wrong value is caught by the zero-match guard above only when it matches nobody at all; a value that matches some accounts and not others would disable the rest. See BL-069.

T13.4 — D18: run it on a schedule

  • Items: D18
  • Depends on: T13.3
  • Blocks: T13.6
  • Surface: server/, docker-compose.yml
  • Files: server/app.py (startup hook) or a new sidecar per the backup container's pattern — task's call, per the decision record's noted alternative

Do: Wire T13.3 to run on an interval (proposed 15 minutes, env-overridable — e.g. OKTA_SYNC_INTERVAL_SECONDS, matching BACKUP_INTERVAL_SECONDS's naming). Default choice is an in-process background task in the api container; if a separate container is chosen instead, follow the backup service's shape (own Dockerfile or script, internal network plus whatever egress reaching Okta requires — check whether outbound as currently defined is sufficient or Okta needs a distinct allowance).

Done when:

  • the job runs automatically on the configured interval without manual invocation (2026-10-16: a daemon timer thread starts from the app's lifespan hook (server/app.py), checks once a minute, and ran the sync on its own in tests/okta_sync_schedule_check.py. With OKTA_API_TOKEN unset it starts nothing and the startup log says Okta sync DISABLED. Booted the real app both ways; the thread starts and ends with the app.)
  • interval is env-configurable with a sane default (2026-10-16: OKTA_SYNC_INTERVAL_SECONDS, default 900. A non-number uses the default. A value under 300 is raised to 300. Passed through docker-compose.yml.)
  • a container restart does not produce a duplicate/overlapping run, and a slow cycle does not stack with the next one (2026-10-16: the api runs 2 gunicorn workers, so each starts its own timer. One row in app_settings (key okta_sync) is the gate: a worker claims a cycle with one atomic conditional UPDATE, so exactly one wins. Checked with 8 racing workers on a file database, 15 rounds plus the first-insert race: exactly 1 winner every time. The row outlives a restart, so a restart inside the interval starts nothing. A slow run cannot overlap the next: a fetch is capped at 120 s in total and the interval cannot go below 300 s. Inside one process the thread runs one cycle at a time. This needed no migration: app_settings already exists.)

Decided 2026-10-16 (task's call, per this task's "Do"): in-process timer in the api container, as D18 proposed, not a sidecar. The backup pattern was the alternative; it would need its own image and would not remove the 2-worker duplicate question, because the sync reads the application database anyway. Network check, as the task asked: the outbound network the api service already joins is an ordinary bridge with a gateway, so Okta is reachable with no network change. The host firewall must allow https to the Okta org. One change to T13.2's code: fetch_users() now also stops after 120 seconds in total (TOTAL_DEADLINE_SECONDS), which is what lets the scheduler promise no overlap. Tested in tests/okta_sync_check.py section 8.

T13.5 — D18: admin visibility

  • Items: D18
  • Depends on: T13.3
  • Blocks: T13.6
  • Surface: html/
  • Files: html/users.js / html/admin.js (wherever audit history is already surfaced)

Do: Confirm an auto-disable reads clearly wherever admins already look at account history — the actor string (system:okta_sync) should be self-explanatory in context, not require reading server logs to understand. Do not build new UI beyond making sure the existing audit surface renders this actor sensibly. Email/notification to admins on auto-disable is a noted fast-follow (the decision record flags it as open, not required here) — do not build it in this task; log it instead if it's tempting to add.

Done when:

  • an auto-disabled account's audit entry is visible and legible in the existing admin UI without special-casing
  • nothing here silently assumes CR-019's activity view exists yet — this must work standalone

T13.6 — D18: verification

  • Items: D18
  • Depends on: T13.4, T13.5
  • Blocks: nothing
  • Surface: server/
  • Files: as touched above

Do: Full verification per CLAUDE.md. Beyond the usual run: specifically re-run T13.3's fail-closed test in isolation and confirm it still passes after T13.4's scheduling wrapper is in place — the scheduling layer must not introduce a path that swallows the client's error and proceeds anyway.

Done when:

  • all D18 acceptance criteria in decisions-2026-09-17.md are met or a failure is stated with a reason
  • the fail-closed behavior is verified end to end through the scheduled wrapper, not just the bare sync function
  • smoke test and seed_demo.py both still pass
  • full test suite passes

Wave 13 exit criteria

  • AUTH_SESSION_HOURS defaults to 2, documented
  • the sync job runs on a schedule and correctly disables accounts Okta no longer shows as active
  • every auto-disable is individually audited with a clearly non-human actor
  • an Okta API failure of any kind changes zero accounts — verified, not assumed
  • the sync never re-enables an account
  • D18 fully accounted for, no open acceptance criteria