T13.3: Okta deprovisioning sync job, fail closed (D18)

run_sync() compares the Okta user list (T13.2) to active local accounts and
disables the ones Okta no longer vouches for. One AuditLog row per change,
actor system:okta_sync, action user_deprovisioned_by_sync.

Fail closed: any exception from the fetch, an empty or untrusted answer, or a
failed save changes nothing and writes nothing. The fetch runs before the
database is read, so a failed fetch has no path to a change. It only ever
disables; an already-disabled account is untouched and gets no new row.

Decided with Matt 2026-10-16: disable on missing, SUSPENDED or DEPROVISIONED.
PASSWORD_EXPIRED, LOCKED_OUT and RECOVERY do not disable (the job never
re-enables, so a literal "not ACTIVE" would lock people out for good).
Added a guard: no match between any active local account and any Okta user is
refused as a wrong identity attribute, not read as a mass offboarding.

tests/okta_sync_job_check.py: 71 checks; removing the fail-closed branch and the
guard fails 28 of them. BL-069 logs the missing per-run cap.
This commit is contained in:
2026-10-16 11:31:39 -07:00
parent 2f7de6e33b
commit 103e4d40ee
4 changed files with 402 additions and 8 deletions

View File

@@ -662,3 +662,20 @@ deliberately deferred.
worth confirming rather than assuming).
- **Suggested wave or follow-up:** next housekeeping pass — needs Nick on
whether to delete, archive, or rewrite it as historical record.
### BL-069 - The sync has no cap on one run and no last-admin guard
- **Found during:** T13.3 (D18)
- **Where:** `server/okta_sync.py` `run_sync()`
- **What:** a run disables every account the rules select, however many. If the
identity attribute is partly wrong (it matches some accounts and not others),
one run can disable most of the company, including every admin. The
zero-match guard only stops the case where nothing matches. Disabled accounts
cannot sign in, so re-enabling needs an admin who is still active, or the
`manage_users.py` command line on the host.
- **Why not now:** D18 does not ask for it, and the right limit is a product
call: a count, a share of accounts, a rule that the last active admin never
disables, or all three.
- **Suggested wave or follow-up:** before the first live run, or in T13.6 if
Matt decides. Needs Matt.

View File

@@ -144,18 +144,44 @@ assert zero rows changed and zero `AuditLog` entries written.
**Done when:**
- [ ] a user missing from Okta's list, or present but not active, is disabled
- [x] a user missing from Okta's list, or present but not active, is disabled
with a correctly-detailed audit row
- [ ] a user already disabled is left alone (no duplicate audit row each run)
- [ ] an active Okta user already active locally produces no audit row (only
*(2026-10-16: see "Decided 2026-10-16" below for which statuses count as "not active". Audit row: actor `system:okta_sync`, action `user_deprovisioned_by_sync`, `entity_type` user, `detail` `{okta_status, via}`; a missing user is recorded as `MISSING`.)*
- [x] a user already disabled is left alone (no duplicate audit row each run)
*(2026-10-16: a second run over the same data writes no new rows.)*
- [x] an active Okta user already active locally produces no audit row (only
changes are logged, not a clean bill of health every cycle)
- [ ] **a simulated Okta API failure results in zero account changes and zero
*(2026-10-16: verified in `tests/okta_sync_job_check.py`.)*
- [x] **a simulated Okta API failure results in zero account changes and zero
audit rows** — this is the one check that must never be skipped or
weakened
- [ ] the job never re-enables an account under any input
*(2026-10-16: 10 stand-in failures (OktaSyncError, RuntimeError, ConnectionError, TimeoutError, ValueError, KeyError, empty list, None, two kinds of garbage), 4 through the real client (HTTP 401/403/429/500), the default path with no token, a failed commit, and a wrong identity attribute. Each: `ok` is False, zero accounts changed, zero audit rows. Mutation test: removing the fail-closed branch and the zero-match guard fails 28 of these.)*
- [x] the job never re-enables an account under any input
*(2026-10-16: a disabled account that is ACTIVE in Okta stays disabled, and `okta_sync.py` contains no assignment of `is_active` to True.)*
---
**Decided 2026-10-16 (Matt, in chat):** which Okta statuses disable an account.
The spec says "a non-active status". Read literally, that also catches
`PASSWORD_EXPIRED`, `LOCKED_OUT` and `RECOVERY`, which people leave on their
own, and this job never re-enables, so one AD password expiry would lock someone
out of the app until an admin stepped in. Decided: an account disables when it is
**missing** from Okta's list, or every Okta entry for it is **SUSPENDED** or
**DEPROVISIONED** (`DISABLING_STATUSES` in `server/okta_sync.py`). Okta's default
user list already leaves out DEPROVISIONED users, so those arrive as "missing".
**Added 2026-10-16, inside decision 4's "anything the sync can't confidently
parse":** if there are active local accounts and **none** of them matches any
Okta user, the run is refused and nothing changes. That pattern means the
identity attribute is wrong (`OKTA_SYNC_PROFILE_ATTRIBUTE`), not that the whole
company left. It is a guard, not a new feature. Raise it in the PR.
**Open, for the PR / Nick:** which Okta profile attribute equals the
`preferred_username` claim in this tenant. The client assumes `profile.login`,
Okta's default. Confirm before the first live run. A wrong value is caught by the
zero-match guard above only when it matches nobody at all; a value that matches
some accounts and not others would disable the rest. See `BL-069`.
### T13.4 — D18: run it on a schedule
- **Items:** `D18`

View File

@@ -26,6 +26,10 @@ would disable every account on an Okta outage. The cases that raise are:
- a page limit reached (a cut-off list would make real users look missing)
- a "next page" link that points to a different host
T13.3 (the sync job, `run_sync` below) uses that list to disable local accounts.
Its rules are in its own docstring. In short: it only ever disables, it never
re-enables, it writes one audit row per change, and any failure means NO change.
The token is read from the environment each time it is used. It is never logged,
never put in an error message, and never returned to a browser. Errors carry the
exception class name and the HTTP status only. httpx exceptions can include the
@@ -33,13 +37,16 @@ request URL, so the URL is not copied into the message either.
"""
import logging
import os
from dataclasses import dataclass
from typing import Optional
import uuid
from dataclasses import dataclass, field
from typing import Callable, Optional
from urllib.parse import urlparse
import httpx
from sqlalchemy import select
from sqlalchemy.orm import Session
from . import okta_auth
from . import models, okta_auth
log = logging.getLogger("wpsuite.okta_sync")
@@ -203,3 +210,129 @@ def fetch_users(http: Optional[httpx.Client] = None) -> "list[OktaUser]":
raise _fail("Okta returned no users, which is not credible for a real org")
log.info("Okta user list fetched: %d user(s)", len(users))
return users
# ── T13.3: the sync job ───────────────────────────────────────────────────────
# Which Okta statuses mean "this person is gone or was blocked on purpose".
# Decided by Matt, 2026-10-16: a literal "anything that is not ACTIVE" would also
# catch PASSWORD_EXPIRED, LOCKED_OUT and RECOVERY, which people leave on their
# own, and because this job never re-enables, one AD password expiry would lock a
# person out of this app until an admin stepped in. A user who is MISSING from the
# list disables too (Okta's default list leaves out DEPROVISIONED users).
DISABLING_STATUSES = frozenset({"SUSPENDED", "DEPROVISIONED"})
SYNC_ACTOR = "system:okta_sync"
SYNC_ACTION = "user_deprovisioned_by_sync"
@dataclass
class SyncResult:
ok: bool # False = the run took NO action at all
error: str = "" # why, when ok is False
okta_users: int = 0 # size of the Okta list
checked: int = 0 # active local accounts looked at
disabled: "list[str]" = field(default_factory=list) # usernames disabled this run
def _gen_id(prefix: str) -> str:
# Same shape as server/app.py gen_id. Not imported: app.py will import this
# module (T13.4), and the reverse import would be circular.
return f"{prefix}_{uuid.uuid4().hex[:12]}"
def run_sync(db: Session, fetch: Optional[Callable[[], "list[OktaUser]"]] = None) -> SyncResult:
"""Compare the Okta user list to local accounts and disable the ones Okta no
longer vouches for.
The rules, none of which may be relaxed (D18):
1. FAIL CLOSED. If fetching raises ANY exception, or the answer cannot be
trusted, this function changes nothing and writes nothing. A failure is
never read as "nobody is active". The fetch happens before the database
is read, so there is no code path from a failed fetch to a change.
2. ONLY DISABLE. It sets is_active to False. No input makes it set True.
An account that is already disabled is not touched and gets no new
audit row, so a run that finds nothing new writes nothing.
3. One AuditLog row per disabled account, actor "system:okta_sync", so it
is never confused with an admin's action. Rows and changes are saved in
one transaction. If saving fails, none of it is kept.
4. Only changes are logged. A person who is still active in Okta and still
active here produces no row.
One guard beyond the rules above, inside decision 4's "anything the sync can't
confidently parse": if there are active local accounts and NONE of them
matches any Okta user, the identity attribute is almost certainly wrong (see
OKTA_SYNC_PROFILE_ATTRIBUTE). That is a setup mistake, not an offboarding of
the whole company, so the run is refused and nothing changes.
A local account matches an Okta user by username or by email, ignoring case,
the same two fields auth.find_user() uses. If the Okta list holds the same
identity twice with different statuses, the account is disabled only when
every one of them is a disabling status.
`fetch` is for tests. In production it is None and fetch_users() is used.
"""
fetch = fetch or fetch_users
# 1. The fetch. Nothing below runs unless it returned a list.
try:
okta_users = fetch()
except Exception as exc: # an error of ANY kind, not only OktaSyncError
msg = str(exc) if isinstance(exc, OktaSyncError) else f"unexpected {type(exc).__name__}"
log.error("Okta sync SKIPPED, no account was changed: %s", msg)
return SyncResult(ok=False, error=msg)
if not okta_users:
# fetch_users() already raises on an empty list. This covers a stand-in
# fetch (and any future change) that returns one: same rule.
msg = "the fetch returned no users"
log.error("Okta sync SKIPPED, no account was changed: %s", msg)
return SyncResult(ok=False, error=msg)
try:
by_identity: "dict[str, list[str]]" = {}
for u in okta_users:
by_identity.setdefault(u.identity.strip().lower(), []).append(u.status.strip().upper())
locals_ = list(db.scalars(select(models.User).where(models.User.is_active.is_(True))))
plan: "list[tuple[models.User, str]]" = []
matched = 0
for local in locals_:
statuses: "list[str]" = []
for key in {(local.username or "").strip().lower(), (local.email or "").strip().lower()}:
if key:
statuses.extend(by_identity.get(key, []))
if statuses:
matched += 1
if all(st in DISABLING_STATUSES for st in statuses):
plan.append((local, sorted(set(statuses))[0]))
else:
plan.append((local, "MISSING"))
if locals_ and matched == 0:
msg = (f"none of {len(locals_)} active local account(s) matches any of "
f"{len(okta_users)} Okta user(s); the identity attribute is probably wrong")
log.error("Okta sync SKIPPED, no account was changed: %s", msg)
return SyncResult(ok=False, error=msg, okta_users=len(okta_users), checked=len(locals_))
planned = [(local.username, status) for local, status in plan]
checked = len(locals_)
for local, status in plan:
local.is_active = False
db.add(models.AuditLog(
id=_gen_id("ev"), actor=SYNC_ACTOR, action=SYNC_ACTION,
entity_type="user", entity_id=local.id, project_id=None,
summary=local.username[:400],
detail={"okta_status": status, "via": "okta_sync"},
))
db.commit()
except Exception as exc:
db.rollback()
log.error("Okta sync FAILED, nothing was kept: unexpected %s", type(exc).__name__)
return SyncResult(ok=False, error=f"unexpected {type(exc).__name__}")
for name, status in planned:
log.warning("Okta sync disabled %r (Okta status: %s)", name, status)
log.info("Okta sync done: %d Okta user(s), %d active local account(s) checked, %d disabled",
len(okta_users), checked, len(planned))
return SyncResult(ok=True, okta_users=len(okta_users), checked=checked,
disabled=[name for name, _ in planned])

View File

@@ -0,0 +1,218 @@
#!/usr/bin/env python3
"""Does the Okta deprovisioning sync fail closed? - D18 / T13.3.
The rule that must never be skipped or weakened: when Okta cannot be asked, or
its answer cannot be trusted, the sync changes NO account and writes NO audit
row. Treating "could not reach Okta" as "nobody is active" would lock the whole
company out on one bad afternoon.
Also checked: the sync only ever disables (it never re-enables, under any
input), writes one audit row per change with a non-human actor, leaves an
already-disabled account alone, logs only changes, and keeps temporary Okta
states (PASSWORD_EXPIRED, LOCKED_OUT, RECOVERY) from disabling anyone.
Self-contained: its own in-memory SQLite, no server, no network, no browser.
Exit 0 all passed, 1 a failure.
"""
import os
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
import httpx # noqa: E402
from sqlalchemy import create_engine, select # noqa: E402
from sqlalchemy.orm import sessionmaker # noqa: E402
from sqlalchemy.pool import StaticPool # noqa: E402
from server import models, okta_auth, okta_sync # noqa: E402
from server.okta_sync import OktaSyncError, OktaUser, run_sync # noqa: E402
fails = 0
def chk(label, ok, extra=""):
global fails
print(("PASS " if ok else "FAIL ") + label + (f" [{extra}]" if (extra and not ok) else ""))
if not ok:
fails += 1
engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
models.Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine, autoflush=False, future=True)
def seed(rows):
"""rows: list of (username, is_active[, email]). Wipes users and audit rows first."""
with Session() as db:
db.query(models.AuditLog).delete()
db.query(models.User).delete()
for r in rows:
name, active = r[0], r[1]
db.add(models.User(id="u_" + name, username=name, is_active=active,
email=(r[2] if len(r) > 2 else "")))
db.commit()
def state():
with Session() as db:
users = {u.username: u.is_active for u in db.scalars(select(models.User))}
audits = [a.to_dict() for a in db.scalars(select(models.AuditLog))]
return users, audits
def fixed(*pairs):
return lambda: [OktaUser(identity=i, status=s) for i, s in pairs]
# ── 1. the failure rule ───────────────────────────────────────────────────────
print("\n1. FAIL CLOSED: a failed or untrusted fetch changes nothing")
BASE = [("alice", True), ("bob", True), ("carol", True)]
def boom(exc):
def f():
raise exc
return f
failing = [
("OktaSyncError (auth failure)", boom(OktaSyncError("HTTP 401"))),
("RuntimeError", boom(RuntimeError("boom"))),
("ConnectionError", boom(ConnectionError("down"))),
("TimeoutError", boom(TimeoutError("slow"))),
("ValueError", boom(ValueError("bad"))),
("KeyError", boom(KeyError("x"))),
("fetch returns an empty list", lambda: []),
("fetch returns None", lambda: None),
("fetch returns garbage (strings)", lambda: ["alice", "bob"]),
("fetch returns garbage (dicts)", lambda: [{"identity": "alice"}]),
]
for label, fetch in failing:
seed(BASE)
before = state()
with Session() as db:
res = run_sync(db, fetch=fetch)
after = state()
chk(f"{label}: ok is False", res.ok is False)
chk(f"{label}: zero accounts changed", after[0] == before[0] == {"alice": True, "bob": True, "carol": True})
chk(f"{label}: zero audit rows", after[1] == [] and res.disabled == [])
print("\n ...and through the real client, not a stand-in")
okta_auth.ISSUER = "https://example.okta.com/oauth2/default"
os.environ["OKTA_API_TOKEN"] = "tok"
for code in (401, 403, 429, 500):
seed(BASE)
c = httpx.Client(transport=httpx.MockTransport(lambda r, k=code: httpx.Response(k, json={})))
with Session() as db:
res = run_sync(db, fetch=lambda c=c: okta_sync.fetch_users(http=c))
u, a = state()
chk(f"real client, HTTP {code}: nothing changed, no audit rows",
(not res.ok) and all(u.values()) and a == [])
seed(BASE)
os.environ["OKTA_API_TOKEN"] = ""
with Session() as db:
res = run_sync(db) # fetch=None: the default path, no token set
u, a = state()
chk("default fetch with no token: nothing changed, no audit rows", (not res.ok) and all(u.values()) and a == [])
print("\n ...and when saving fails")
seed(BASE)
with Session() as db:
real_commit = db.commit
def bad_commit():
raise RuntimeError("disk full")
db.commit = bad_commit
res = run_sync(db, fetch=fixed(("alice", "ACTIVE"))) # bob and carol would be disabled
u, a = state()
chk("a failed commit keeps no change and no audit row", (not res.ok) and all(u.values()) and a == [])
print("\n ...and when no local account matches any Okta user (wrong identity attribute)")
seed(BASE)
with Session() as db:
res = run_sync(db, fetch=fixed(("zed@x.com", "ACTIVE"), ("yan@x.com", "ACTIVE")))
u, a = state()
chk("zero matches is refused, not read as 'everyone left'", (not res.ok) and all(u.values()) and a == [])
chk("...and the error says why", "identity attribute" in res.error)
# ── 2. what it does disable ───────────────────────────────────────────────────
print("\n2. a good run disables the right accounts")
seed([("alice", True), ("bob", True), ("carol", True), ("dave", True),
("erin", True), ("frank", True), ("gina", True), ("hank", True)])
okta = fixed(("alice", "ACTIVE"), ("bob", "SUSPENDED"),
# carol is MISSING from the list
("dave", "DEPROVISIONED"), ("erin", "PASSWORD_EXPIRED"),
("frank", "LOCKED_OUT"), ("gina", "RECOVERY"), ("hank", "ACTIVE"))
with Session() as db:
res = run_sync(db, fetch=okta)
users, audits = state()
chk("run is ok", res.ok, res.error)
chk("missing, SUSPENDED and DEPROVISIONED are disabled",
sorted(res.disabled) == ["bob", "carol", "dave"] and
[users[n] for n in ("bob", "carol", "dave")] == [False, False, False])
chk("ACTIVE stays active", users["alice"] is True and users["hank"] is True)
chk("PASSWORD_EXPIRED, LOCKED_OUT and RECOVERY do NOT disable (temporary states)",
users["erin"] is True and users["frank"] is True and users["gina"] is True)
chk("one audit row per disabled account", len(audits) == 3, str(len(audits)))
row = next(a for a in audits if a["summary"] == "bob")
chk("audit actor is the sync job, not a person", {a["actor"] for a in audits} == {"system:okta_sync"})
chk("audit action is user_deprovisioned_by_sync", {a["action"] for a in audits} == {"user_deprovisioned_by_sync"})
chk("audit row names the user and the Okta status found",
row["entity_type"] == "user" and row["entity_id"] == "u_bob" and row["detail"].get("okta_status") == "SUSPENDED")
chk("a missing user is recorded as MISSING",
next(a for a in audits if a["summary"] == "carol")["detail"].get("okta_status") == "MISSING")
print("\n ...run it again: only changes are logged")
with Session() as db:
res2 = run_sync(db, fetch=okta)
users2, audits2 = state()
chk("second run is ok, disables nobody", res2.ok and res2.disabled == [])
chk("second run writes no new audit rows (no duplicates)", len(audits2) == 3)
chk("accounts unchanged by the second run", users2 == users)
# ── 3. never re-enables, never touches a disabled account ─────────────────────
print("\n3. never re-enables, never touches an already-disabled account")
seed([("alice", True), ("hank", False), ("gina", False)])
with Session() as db:
res = run_sync(db, fetch=fixed(("alice", "ACTIVE"), ("hank", "ACTIVE"))) # gina missing
users, audits = state()
chk("a disabled account that is ACTIVE in Okta stays disabled", users["hank"] is False)
chk("a disabled account that is missing from Okta stays as it is", users["gina"] is False)
chk("no audit row for either (nothing changed)", audits == [] and res.disabled == [])
src = open(os.path.join(ROOT, "server", "okta_sync.py")).read()
chk("no assignment of is_active to True anywhere in okta_sync.py",
"is_active = True" not in src and "is_active=True" not in src.replace("is_active.is_(True)", ""))
# ── 4. matching ───────────────────────────────────────────────────────────────
print("\n4. matching")
seed([("ALICE", True), ("bob", True, "Bob@X.com"), ("carol", True)])
with Session() as db:
res = run_sync(db, fetch=fixed(("alice", "ACTIVE"), ("bob@x.com", "ACTIVE"), ("carol", "SUSPENDED")))
users, _ = state()
chk("username match ignores case", users["ALICE"] is True)
chk("email match works, ignoring case", users["bob"] is True)
chk("a matching SUSPENDED user is disabled", users["carol"] is False)
seed([("alice", True), ("bob", True), ("carol", True)])
with Session() as db:
run_sync(db, fetch=fixed(("alice", "ACTIVE"), ("bob", "ACTIVE"), ("bob", "SUSPENDED"),
("carol", "SUSPENDED"), ("carol", "DEPROVISIONED")))
users, _ = state()
chk("an identity listed twice, one entry active: stays active", users["bob"] is True)
chk("an identity listed twice, every entry disabling: disabled", users["carol"] is False)
# ── 5. the secret never reaches the audit row or the result ───────────────────
print("\n5. no secret in what is stored or returned")
seed(BASE)
os.environ["OKTA_API_TOKEN"] = "SECRET-TOKEN-123"
c = httpx.Client(transport=httpx.MockTransport(lambda r: httpx.Response(401, json={})))
with Session() as db:
res = run_sync(db, fetch=lambda: okta_sync.fetch_users(http=c))
chk("the failure message has no token", "SECRET-TOKEN-123" not in res.error and "example.okta.com" not in res.error)
print()
print("ALL PASSED" if not fails else f"{fails} FAILED")
sys.exit(0 if not fails else 1)