From 103e4d40ee59965fc1fee071a5f38e0b8c6abc36 Mon Sep 17 00:00:00 2001 From: Matt Mabrey Date: Fri, 16 Oct 2026 11:31:39 -0700 Subject: [PATCH] T13.3: Okta deprovisioning sync job, fail closed (D18) run_sync() compares the Okta user list (T13.2) to active local accounts and disables the ones Okta no longer vouches for. One AuditLog row per change, actor system:okta_sync, action user_deprovisioned_by_sync. Fail closed: any exception from the fetch, an empty or untrusted answer, or a failed save changes nothing and writes nothing. The fetch runs before the database is read, so a failed fetch has no path to a change. It only ever disables; an already-disabled account is untouched and gets no new row. Decided with Matt 2026-10-16: disable on missing, SUSPENDED or DEPROVISIONED. PASSWORD_EXPIRED, LOCKED_OUT and RECOVERY do not disable (the job never re-enables, so a literal "not ACTIVE" would lock people out for good). Added a guard: no match between any active local account and any Okta user is refused as a wrong identity attribute, not read as a mass offboarding. tests/okta_sync_job_check.py: 71 checks; removing the fail-closed branch and the guard fails 28 of them. BL-069 logs the missing per-run cap. --- docs/waves/backlog.md | 17 +++ docs/waves/wave-13.md | 36 +++++- server/okta_sync.py | 139 +++++++++++++++++++++- tests/okta_sync_job_check.py | 218 +++++++++++++++++++++++++++++++++++ 4 files changed, 402 insertions(+), 8 deletions(-) create mode 100644 tests/okta_sync_job_check.py diff --git a/docs/waves/backlog.md b/docs/waves/backlog.md index e28f292..1060d14 100644 --- a/docs/waves/backlog.md +++ b/docs/waves/backlog.md @@ -662,3 +662,20 @@ deliberately deferred. worth confirming rather than assuming). - **Suggested wave or follow-up:** next housekeeping pass — needs Nick on whether to delete, archive, or rewrite it as historical record. + +### BL-069 - The sync has no cap on one run and no last-admin guard + +- **Found during:** T13.3 (D18) +- **Where:** `server/okta_sync.py` `run_sync()` +- **What:** a run disables every account the rules select, however many. If the + identity attribute is partly wrong (it matches some accounts and not others), + one run can disable most of the company, including every admin. The + zero-match guard only stops the case where nothing matches. Disabled accounts + cannot sign in, so re-enabling needs an admin who is still active, or the + `manage_users.py` command line on the host. +- **Why not now:** D18 does not ask for it, and the right limit is a product + call: a count, a share of accounts, a rule that the last active admin never + disables, or all three. +- **Suggested wave or follow-up:** before the first live run, or in T13.6 if + Matt decides. Needs Matt. + diff --git a/docs/waves/wave-13.md b/docs/waves/wave-13.md index 479939d..821dbf6 100644 --- a/docs/waves/wave-13.md +++ b/docs/waves/wave-13.md @@ -144,18 +144,44 @@ assert zero rows changed and zero `AuditLog` entries written. **Done when:** -- [ ] a user missing from Okta's list, or present but not active, is disabled +- [x] a user missing from Okta's list, or present but not active, is disabled with a correctly-detailed audit row -- [ ] a user already disabled is left alone (no duplicate audit row each run) -- [ ] an active Okta user already active locally produces no audit row (only + *(2026-10-16: see "Decided 2026-10-16" below for which statuses count as "not active". Audit row: actor `system:okta_sync`, action `user_deprovisioned_by_sync`, `entity_type` user, `detail` `{okta_status, via}`; a missing user is recorded as `MISSING`.)* +- [x] a user already disabled is left alone (no duplicate audit row each run) + *(2026-10-16: a second run over the same data writes no new rows.)* +- [x] an active Okta user already active locally produces no audit row (only changes are logged, not a clean bill of health every cycle) -- [ ] **a simulated Okta API failure results in zero account changes and zero + *(2026-10-16: verified in `tests/okta_sync_job_check.py`.)* +- [x] **a simulated Okta API failure results in zero account changes and zero audit rows** — this is the one check that must never be skipped or weakened -- [ ] the job never re-enables an account under any input + *(2026-10-16: 10 stand-in failures (OktaSyncError, RuntimeError, ConnectionError, TimeoutError, ValueError, KeyError, empty list, None, two kinds of garbage), 4 through the real client (HTTP 401/403/429/500), the default path with no token, a failed commit, and a wrong identity attribute. Each: `ok` is False, zero accounts changed, zero audit rows. Mutation test: removing the fail-closed branch and the zero-match guard fails 28 of these.)* +- [x] the job never re-enables an account under any input + *(2026-10-16: a disabled account that is ACTIVE in Okta stays disabled, and `okta_sync.py` contains no assignment of `is_active` to True.)* --- +**Decided 2026-10-16 (Matt, in chat):** which Okta statuses disable an account. +The spec says "a non-active status". Read literally, that also catches +`PASSWORD_EXPIRED`, `LOCKED_OUT` and `RECOVERY`, which people leave on their +own, and this job never re-enables, so one AD password expiry would lock someone +out of the app until an admin stepped in. Decided: an account disables when it is +**missing** from Okta's list, or every Okta entry for it is **SUSPENDED** or +**DEPROVISIONED** (`DISABLING_STATUSES` in `server/okta_sync.py`). Okta's default +user list already leaves out DEPROVISIONED users, so those arrive as "missing". + +**Added 2026-10-16, inside decision 4's "anything the sync can't confidently +parse":** if there are active local accounts and **none** of them matches any +Okta user, the run is refused and nothing changes. That pattern means the +identity attribute is wrong (`OKTA_SYNC_PROFILE_ATTRIBUTE`), not that the whole +company left. It is a guard, not a new feature. Raise it in the PR. + +**Open, for the PR / Nick:** which Okta profile attribute equals the +`preferred_username` claim in this tenant. The client assumes `profile.login`, +Okta's default. Confirm before the first live run. A wrong value is caught by the +zero-match guard above only when it matches nobody at all; a value that matches +some accounts and not others would disable the rest. See `BL-069`. + ### T13.4 — D18: run it on a schedule - **Items:** `D18` diff --git a/server/okta_sync.py b/server/okta_sync.py index ee48abc..16b92a8 100644 --- a/server/okta_sync.py +++ b/server/okta_sync.py @@ -26,6 +26,10 @@ would disable every account on an Okta outage. The cases that raise are: - a page limit reached (a cut-off list would make real users look missing) - a "next page" link that points to a different host +T13.3 (the sync job, `run_sync` below) uses that list to disable local accounts. +Its rules are in its own docstring. In short: it only ever disables, it never +re-enables, it writes one audit row per change, and any failure means NO change. + The token is read from the environment each time it is used. It is never logged, never put in an error message, and never returned to a browser. Errors carry the exception class name and the HTTP status only. httpx exceptions can include the @@ -33,13 +37,16 @@ request URL, so the URL is not copied into the message either. """ import logging import os -from dataclasses import dataclass -from typing import Optional +import uuid +from dataclasses import dataclass, field +from typing import Callable, Optional from urllib.parse import urlparse import httpx +from sqlalchemy import select +from sqlalchemy.orm import Session -from . import okta_auth +from . import models, okta_auth log = logging.getLogger("wpsuite.okta_sync") @@ -203,3 +210,129 @@ def fetch_users(http: Optional[httpx.Client] = None) -> "list[OktaUser]": raise _fail("Okta returned no users, which is not credible for a real org") log.info("Okta user list fetched: %d user(s)", len(users)) return users + + +# ── T13.3: the sync job ─────────────────────────────────────────────────────── +# Which Okta statuses mean "this person is gone or was blocked on purpose". +# Decided by Matt, 2026-10-16: a literal "anything that is not ACTIVE" would also +# catch PASSWORD_EXPIRED, LOCKED_OUT and RECOVERY, which people leave on their +# own, and because this job never re-enables, one AD password expiry would lock a +# person out of this app until an admin stepped in. A user who is MISSING from the +# list disables too (Okta's default list leaves out DEPROVISIONED users). +DISABLING_STATUSES = frozenset({"SUSPENDED", "DEPROVISIONED"}) + +SYNC_ACTOR = "system:okta_sync" +SYNC_ACTION = "user_deprovisioned_by_sync" + + +@dataclass +class SyncResult: + ok: bool # False = the run took NO action at all + error: str = "" # why, when ok is False + okta_users: int = 0 # size of the Okta list + checked: int = 0 # active local accounts looked at + disabled: "list[str]" = field(default_factory=list) # usernames disabled this run + + +def _gen_id(prefix: str) -> str: + # Same shape as server/app.py gen_id. Not imported: app.py will import this + # module (T13.4), and the reverse import would be circular. + return f"{prefix}_{uuid.uuid4().hex[:12]}" + + +def run_sync(db: Session, fetch: Optional[Callable[[], "list[OktaUser]"]] = None) -> SyncResult: + """Compare the Okta user list to local accounts and disable the ones Okta no + longer vouches for. + + The rules, none of which may be relaxed (D18): + + 1. FAIL CLOSED. If fetching raises ANY exception, or the answer cannot be + trusted, this function changes nothing and writes nothing. A failure is + never read as "nobody is active". The fetch happens before the database + is read, so there is no code path from a failed fetch to a change. + 2. ONLY DISABLE. It sets is_active to False. No input makes it set True. + An account that is already disabled is not touched and gets no new + audit row, so a run that finds nothing new writes nothing. + 3. One AuditLog row per disabled account, actor "system:okta_sync", so it + is never confused with an admin's action. Rows and changes are saved in + one transaction. If saving fails, none of it is kept. + 4. Only changes are logged. A person who is still active in Okta and still + active here produces no row. + + One guard beyond the rules above, inside decision 4's "anything the sync can't + confidently parse": if there are active local accounts and NONE of them + matches any Okta user, the identity attribute is almost certainly wrong (see + OKTA_SYNC_PROFILE_ATTRIBUTE). That is a setup mistake, not an offboarding of + the whole company, so the run is refused and nothing changes. + + A local account matches an Okta user by username or by email, ignoring case, + the same two fields auth.find_user() uses. If the Okta list holds the same + identity twice with different statuses, the account is disabled only when + every one of them is a disabling status. + + `fetch` is for tests. In production it is None and fetch_users() is used. + """ + fetch = fetch or fetch_users + + # 1. The fetch. Nothing below runs unless it returned a list. + try: + okta_users = fetch() + except Exception as exc: # an error of ANY kind, not only OktaSyncError + msg = str(exc) if isinstance(exc, OktaSyncError) else f"unexpected {type(exc).__name__}" + log.error("Okta sync SKIPPED, no account was changed: %s", msg) + return SyncResult(ok=False, error=msg) + if not okta_users: + # fetch_users() already raises on an empty list. This covers a stand-in + # fetch (and any future change) that returns one: same rule. + msg = "the fetch returned no users" + log.error("Okta sync SKIPPED, no account was changed: %s", msg) + return SyncResult(ok=False, error=msg) + + try: + by_identity: "dict[str, list[str]]" = {} + for u in okta_users: + by_identity.setdefault(u.identity.strip().lower(), []).append(u.status.strip().upper()) + + locals_ = list(db.scalars(select(models.User).where(models.User.is_active.is_(True)))) + plan: "list[tuple[models.User, str]]" = [] + matched = 0 + for local in locals_: + statuses: "list[str]" = [] + for key in {(local.username or "").strip().lower(), (local.email or "").strip().lower()}: + if key: + statuses.extend(by_identity.get(key, [])) + if statuses: + matched += 1 + if all(st in DISABLING_STATUSES for st in statuses): + plan.append((local, sorted(set(statuses))[0])) + else: + plan.append((local, "MISSING")) + + if locals_ and matched == 0: + msg = (f"none of {len(locals_)} active local account(s) matches any of " + f"{len(okta_users)} Okta user(s); the identity attribute is probably wrong") + log.error("Okta sync SKIPPED, no account was changed: %s", msg) + return SyncResult(ok=False, error=msg, okta_users=len(okta_users), checked=len(locals_)) + + planned = [(local.username, status) for local, status in plan] + checked = len(locals_) + for local, status in plan: + local.is_active = False + db.add(models.AuditLog( + id=_gen_id("ev"), actor=SYNC_ACTOR, action=SYNC_ACTION, + entity_type="user", entity_id=local.id, project_id=None, + summary=local.username[:400], + detail={"okta_status": status, "via": "okta_sync"}, + )) + db.commit() + except Exception as exc: + db.rollback() + log.error("Okta sync FAILED, nothing was kept: unexpected %s", type(exc).__name__) + return SyncResult(ok=False, error=f"unexpected {type(exc).__name__}") + + for name, status in planned: + log.warning("Okta sync disabled %r (Okta status: %s)", name, status) + log.info("Okta sync done: %d Okta user(s), %d active local account(s) checked, %d disabled", + len(okta_users), checked, len(planned)) + return SyncResult(ok=True, okta_users=len(okta_users), checked=checked, + disabled=[name for name, _ in planned]) diff --git a/tests/okta_sync_job_check.py b/tests/okta_sync_job_check.py new file mode 100644 index 0000000..43cc56a --- /dev/null +++ b/tests/okta_sync_job_check.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +"""Does the Okta deprovisioning sync fail closed? - D18 / T13.3. + +The rule that must never be skipped or weakened: when Okta cannot be asked, or +its answer cannot be trusted, the sync changes NO account and writes NO audit +row. Treating "could not reach Okta" as "nobody is active" would lock the whole +company out on one bad afternoon. + +Also checked: the sync only ever disables (it never re-enables, under any +input), writes one audit row per change with a non-human actor, leaves an +already-disabled account alone, logs only changes, and keeps temporary Okta +states (PASSWORD_EXPIRED, LOCKED_OUT, RECOVERY) from disabling anyone. + +Self-contained: its own in-memory SQLite, no server, no network, no browser. +Exit 0 all passed, 1 a failure. +""" +import os +import sys + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, ROOT) + +import httpx # noqa: E402 +from sqlalchemy import create_engine, select # noqa: E402 +from sqlalchemy.orm import sessionmaker # noqa: E402 +from sqlalchemy.pool import StaticPool # noqa: E402 + +from server import models, okta_auth, okta_sync # noqa: E402 +from server.okta_sync import OktaSyncError, OktaUser, run_sync # noqa: E402 + +fails = 0 + + +def chk(label, ok, extra=""): + global fails + print(("PASS " if ok else "FAIL ") + label + (f" [{extra}]" if (extra and not ok) else "")) + if not ok: + fails += 1 + + +engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool) +models.Base.metadata.create_all(engine) +Session = sessionmaker(bind=engine, autoflush=False, future=True) + + +def seed(rows): + """rows: list of (username, is_active[, email]). Wipes users and audit rows first.""" + with Session() as db: + db.query(models.AuditLog).delete() + db.query(models.User).delete() + for r in rows: + name, active = r[0], r[1] + db.add(models.User(id="u_" + name, username=name, is_active=active, + email=(r[2] if len(r) > 2 else ""))) + db.commit() + + +def state(): + with Session() as db: + users = {u.username: u.is_active for u in db.scalars(select(models.User))} + audits = [a.to_dict() for a in db.scalars(select(models.AuditLog))] + return users, audits + + +def fixed(*pairs): + return lambda: [OktaUser(identity=i, status=s) for i, s in pairs] + + +# ── 1. the failure rule ─────────────────────────────────────────────────────── +print("\n1. FAIL CLOSED: a failed or untrusted fetch changes nothing") +BASE = [("alice", True), ("bob", True), ("carol", True)] + + +def boom(exc): + def f(): + raise exc + return f + + +failing = [ + ("OktaSyncError (auth failure)", boom(OktaSyncError("HTTP 401"))), + ("RuntimeError", boom(RuntimeError("boom"))), + ("ConnectionError", boom(ConnectionError("down"))), + ("TimeoutError", boom(TimeoutError("slow"))), + ("ValueError", boom(ValueError("bad"))), + ("KeyError", boom(KeyError("x"))), + ("fetch returns an empty list", lambda: []), + ("fetch returns None", lambda: None), + ("fetch returns garbage (strings)", lambda: ["alice", "bob"]), + ("fetch returns garbage (dicts)", lambda: [{"identity": "alice"}]), +] +for label, fetch in failing: + seed(BASE) + before = state() + with Session() as db: + res = run_sync(db, fetch=fetch) + after = state() + chk(f"{label}: ok is False", res.ok is False) + chk(f"{label}: zero accounts changed", after[0] == before[0] == {"alice": True, "bob": True, "carol": True}) + chk(f"{label}: zero audit rows", after[1] == [] and res.disabled == []) + +print("\n ...and through the real client, not a stand-in") +okta_auth.ISSUER = "https://example.okta.com/oauth2/default" +os.environ["OKTA_API_TOKEN"] = "tok" +for code in (401, 403, 429, 500): + seed(BASE) + c = httpx.Client(transport=httpx.MockTransport(lambda r, k=code: httpx.Response(k, json={}))) + with Session() as db: + res = run_sync(db, fetch=lambda c=c: okta_sync.fetch_users(http=c)) + u, a = state() + chk(f"real client, HTTP {code}: nothing changed, no audit rows", + (not res.ok) and all(u.values()) and a == []) +seed(BASE) +os.environ["OKTA_API_TOKEN"] = "" +with Session() as db: + res = run_sync(db) # fetch=None: the default path, no token set +u, a = state() +chk("default fetch with no token: nothing changed, no audit rows", (not res.ok) and all(u.values()) and a == []) + +print("\n ...and when saving fails") +seed(BASE) +with Session() as db: + real_commit = db.commit + + def bad_commit(): + raise RuntimeError("disk full") + + db.commit = bad_commit + res = run_sync(db, fetch=fixed(("alice", "ACTIVE"))) # bob and carol would be disabled +u, a = state() +chk("a failed commit keeps no change and no audit row", (not res.ok) and all(u.values()) and a == []) + +print("\n ...and when no local account matches any Okta user (wrong identity attribute)") +seed(BASE) +with Session() as db: + res = run_sync(db, fetch=fixed(("zed@x.com", "ACTIVE"), ("yan@x.com", "ACTIVE"))) +u, a = state() +chk("zero matches is refused, not read as 'everyone left'", (not res.ok) and all(u.values()) and a == []) +chk("...and the error says why", "identity attribute" in res.error) + +# ── 2. what it does disable ─────────────────────────────────────────────────── +print("\n2. a good run disables the right accounts") +seed([("alice", True), ("bob", True), ("carol", True), ("dave", True), + ("erin", True), ("frank", True), ("gina", True), ("hank", True)]) +okta = fixed(("alice", "ACTIVE"), ("bob", "SUSPENDED"), + # carol is MISSING from the list + ("dave", "DEPROVISIONED"), ("erin", "PASSWORD_EXPIRED"), + ("frank", "LOCKED_OUT"), ("gina", "RECOVERY"), ("hank", "ACTIVE")) +with Session() as db: + res = run_sync(db, fetch=okta) +users, audits = state() +chk("run is ok", res.ok, res.error) +chk("missing, SUSPENDED and DEPROVISIONED are disabled", + sorted(res.disabled) == ["bob", "carol", "dave"] and + [users[n] for n in ("bob", "carol", "dave")] == [False, False, False]) +chk("ACTIVE stays active", users["alice"] is True and users["hank"] is True) +chk("PASSWORD_EXPIRED, LOCKED_OUT and RECOVERY do NOT disable (temporary states)", + users["erin"] is True and users["frank"] is True and users["gina"] is True) +chk("one audit row per disabled account", len(audits) == 3, str(len(audits))) +row = next(a for a in audits if a["summary"] == "bob") +chk("audit actor is the sync job, not a person", {a["actor"] for a in audits} == {"system:okta_sync"}) +chk("audit action is user_deprovisioned_by_sync", {a["action"] for a in audits} == {"user_deprovisioned_by_sync"}) +chk("audit row names the user and the Okta status found", + row["entity_type"] == "user" and row["entity_id"] == "u_bob" and row["detail"].get("okta_status") == "SUSPENDED") +chk("a missing user is recorded as MISSING", + next(a for a in audits if a["summary"] == "carol")["detail"].get("okta_status") == "MISSING") + +print("\n ...run it again: only changes are logged") +with Session() as db: + res2 = run_sync(db, fetch=okta) +users2, audits2 = state() +chk("second run is ok, disables nobody", res2.ok and res2.disabled == []) +chk("second run writes no new audit rows (no duplicates)", len(audits2) == 3) +chk("accounts unchanged by the second run", users2 == users) + +# ── 3. never re-enables, never touches a disabled account ───────────────────── +print("\n3. never re-enables, never touches an already-disabled account") +seed([("alice", True), ("hank", False), ("gina", False)]) +with Session() as db: + res = run_sync(db, fetch=fixed(("alice", "ACTIVE"), ("hank", "ACTIVE"))) # gina missing +users, audits = state() +chk("a disabled account that is ACTIVE in Okta stays disabled", users["hank"] is False) +chk("a disabled account that is missing from Okta stays as it is", users["gina"] is False) +chk("no audit row for either (nothing changed)", audits == [] and res.disabled == []) +src = open(os.path.join(ROOT, "server", "okta_sync.py")).read() +chk("no assignment of is_active to True anywhere in okta_sync.py", + "is_active = True" not in src and "is_active=True" not in src.replace("is_active.is_(True)", "")) + +# ── 4. matching ─────────────────────────────────────────────────────────────── +print("\n4. matching") +seed([("ALICE", True), ("bob", True, "Bob@X.com"), ("carol", True)]) +with Session() as db: + res = run_sync(db, fetch=fixed(("alice", "ACTIVE"), ("bob@x.com", "ACTIVE"), ("carol", "SUSPENDED"))) +users, _ = state() +chk("username match ignores case", users["ALICE"] is True) +chk("email match works, ignoring case", users["bob"] is True) +chk("a matching SUSPENDED user is disabled", users["carol"] is False) + +seed([("alice", True), ("bob", True), ("carol", True)]) +with Session() as db: + run_sync(db, fetch=fixed(("alice", "ACTIVE"), ("bob", "ACTIVE"), ("bob", "SUSPENDED"), + ("carol", "SUSPENDED"), ("carol", "DEPROVISIONED"))) +users, _ = state() +chk("an identity listed twice, one entry active: stays active", users["bob"] is True) +chk("an identity listed twice, every entry disabling: disabled", users["carol"] is False) + +# ── 5. the secret never reaches the audit row or the result ─────────────────── +print("\n5. no secret in what is stored or returned") +seed(BASE) +os.environ["OKTA_API_TOKEN"] = "SECRET-TOKEN-123" +c = httpx.Client(transport=httpx.MockTransport(lambda r: httpx.Response(401, json={}))) +with Session() as db: + res = run_sync(db, fetch=lambda: okta_sync.fetch_users(http=c)) +chk("the failure message has no token", "SECRET-TOKEN-123" not in res.error and "example.okta.com" not in res.error) + +print() +print("ALL PASSED" if not fails else f"{fails} FAILED") +sys.exit(0 if not fails else 1)