T13.4: run the Okta sync on a schedule (D18)

A daemon timer thread starts from a new lifespan hook in app.py and calls the
T13.3 sync every OKTA_SYNC_INTERVAL_SECONDS (default 900, minimum 300). With no
OKTA_API_TOKEN it starts nothing and the startup log says so.

The api runs 2 gunicorn workers, so each starts its own timer. A single atomic
conditional UPDATE on one app_settings row (key okta_sync) is the gate: exactly
one worker wins each cycle, on SQLite and Postgres, with no lock held. The row
survives a restart, so a restart inside the interval starts no second run. A
fetch is now capped at 120 s in total, so a run is always shorter than the
minimum interval and cannot overlap the next. No migration needed.

docker-compose.yml passes OKTA_API_TOKEN, OKTA_ORG_URL,
OKTA_SYNC_PROFILE_ATTRIBUTE and OKTA_SYNC_INTERVAL_SECONDS to the api service.

tests/okta_sync_schedule_check.py: 38 checks including an 8-worker race on a
file database (exactly one winner, 16 rounds) and fail-closed through the
scheduled path. okta_auth_check.py still 22/22.
This commit is contained in:
2026-10-16 11:34:13 -07:00
parent 103e4d40ee
commit f30989d4b8
7 changed files with 427 additions and 6 deletions

View File

@@ -49,6 +49,18 @@ services:
# here instead, so an operator who leaves .env's copy commented out gets
# the real default, not a broken one.
OKTA_IDENTITY_CLAIM: ${OKTA_IDENTITY_CLAIM:-preferred_username}
# Okta deprovisioning sync (D18, wave 13). OFF until OKTA_API_TOKEN is set; the
# startup log says which. The token is a read-only Okta API token, NOT the OIDC
# client secret above. Empty values here mean "unset" to the code, so the
# ${VAR:-} form is safe for all four (unlike OKTA_IDENTITY_CLAIM above, whose
# code default only applies when the variable is absent).
# Reaching Okta needs outbound https. The `outbound` network the api service
# already joins is an ordinary bridge with a gateway, so no new network is
# needed; the host firewall must allow https to the Okta org.
OKTA_API_TOKEN: ${OKTA_API_TOKEN:-}
OKTA_ORG_URL: ${OKTA_ORG_URL:-}
OKTA_SYNC_PROFILE_ATTRIBUTE: ${OKTA_SYNC_PROFILE_ATTRIBUTE:-}
OKTA_SYNC_INTERVAL_SECONDS: ${OKTA_SYNC_INTERVAL_SECONDS:-}
# Optional — SMTP password for WP-assignment emails. Email is off by
# default and enabled from the Admin console; this is the only email
# secret and it is never stored in the DB. Leave unset until configured.

View File

@@ -202,14 +202,28 @@ is sufficient or Okta needs a distinct allowance).
**Done when:**
- [ ] the job runs automatically on the configured interval without manual
- [x] the job runs automatically on the configured interval without manual
invocation
- [ ] interval is env-configurable with a sane default
- [ ] a container restart does not produce a duplicate/overlapping run, and a
*(2026-10-16: a daemon timer thread starts from the app's lifespan hook (`server/app.py`), checks once a minute, and ran the sync on its own in `tests/okta_sync_schedule_check.py`. With `OKTA_API_TOKEN` unset it starts nothing and the startup log says `Okta sync DISABLED`. Booted the real app both ways; the thread starts and ends with the app.)*
- [x] interval is env-configurable with a sane default
*(2026-10-16: `OKTA_SYNC_INTERVAL_SECONDS`, default 900. A non-number uses the default. A value under 300 is raised to 300. Passed through `docker-compose.yml`.)*
- [x] a container restart does not produce a duplicate/overlapping run, and a
slow cycle does not stack with the next one
*(2026-10-16: the api runs 2 gunicorn workers, so each starts its own timer. One row in `app_settings` (key `okta_sync`) is the gate: a worker claims a cycle with one atomic conditional UPDATE, so exactly one wins. Checked with 8 racing workers on a file database, 15 rounds plus the first-insert race: exactly 1 winner every time. The row outlives a restart, so a restart inside the interval starts nothing. A slow run cannot overlap the next: a fetch is capped at 120 s in total and the interval cannot go below 300 s. Inside one process the thread runs one cycle at a time. This needed no migration: `app_settings` already exists.)*
---
**Decided 2026-10-16 (task's call, per this task's "Do"):** in-process timer in the
`api` container, as D18 proposed, not a sidecar. The `backup` pattern was the
alternative; it would need its own image and would not remove the 2-worker
duplicate question, because the sync reads the application database anyway.
**Network check, as the task asked:** the `outbound` network the `api` service
already joins is an ordinary bridge with a gateway, so Okta is reachable with no
network change. The host firewall must allow https to the Okta org.
**One change to T13.2's code:** `fetch_users()` now also stops after 120 seconds in
total (`TOTAL_DEADLINE_SECONDS`), which is what lets the scheduler promise no
overlap. Tested in `tests/okta_sync_check.py` section 8.
### T13.5 — D18: admin visibility
- **Items:** `D18`

View File

@@ -88,6 +88,12 @@ OKTA_REDIRECT_URI=CHANGE_ME
# profile attribute of the same name. Set this only if your tenant builds the
# claim from a different attribute.
# OKTA_SYNC_PROFILE_ATTRIBUTE=login
#
# How often the sync runs, in seconds. Default 900 (15 minutes), proposed and not
# confirmed with IT. A value under 300 is raised to 300. Every api worker checks
# once a minute and one of them wins each cycle, so raising the worker count does
# not raise the call rate to Okta.
# OKTA_SYNC_INTERVAL_SECONDS=900
# ── Email notifications (optional) ─────────────────────────────────────────────
# WP-assignment emails are OFF by default and are turned on from the Admin

View File

@@ -10,6 +10,7 @@ Interactive docs: http://<host>/api/docs
"""
import base64
import csv
import contextlib
import hashlib
import hmac
import io
@@ -31,8 +32,8 @@ from sqlalchemy import select, delete, func, or_
from sqlalchemy.orm import Session
from starlette.middleware.sessions import SessionMiddleware
from .db import Base, engine, get_db
from . import models, auth, notify, assets_db, okta_auth, okta_fake
from .db import Base, SessionLocal, engine, get_db
from . import models, auth, notify, assets_db, okta_auth, okta_fake, okta_sync
log = logging.getLogger("wpsuite.app")
@@ -47,7 +48,21 @@ if engine.dialect.name == "sqlite":
# Interactive docs are handy in dev but hand an attacker the full API map in prod,
# so enable them only on the SQLite dev fallback (production runs on Postgres).
_docs_enabled = engine.dialect.name == "sqlite"
@contextlib.asynccontextmanager
async def _lifespan(_app):
"""Start the Okta deprovisioning sync (D18 / T13.4) when the process starts and
stop it when the process ends. With no OKTA_API_TOKEN it starts nothing and
says so in the startup log."""
stop = okta_sync.start_scheduler(SessionLocal)
try:
yield
finally:
if stop is not None:
stop.set()
app = FastAPI(
lifespan=_lifespan,
title="Work Package Suite API",
docs_url="/api/docs" if _docs_enabled else None,
redoc_url=None,

View File

@@ -37,13 +37,17 @@ request URL, so the URL is not copied into the message either.
"""
import logging
import os
import threading
import time
import uuid
from datetime import datetime, timedelta, timezone
from dataclasses import dataclass, field
from typing import Callable, Optional
from urllib.parse import urlparse
import httpx
from sqlalchemy import select
from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from . import models, okta_auth
@@ -57,6 +61,10 @@ PAGE_LIMIT = 200
# raises when the bound is reached. 50 pages of 200 is 10,000 people.
MAX_PAGES = 50
TIMEOUT_SECONDS = 15.0
# The whole fetch, all pages together, must finish inside this. T13.4 relies on it:
# a run is always shorter than the shortest allowed interval, so one cycle can
# never still be running when the next is due.
TOTAL_DEADLINE_SECONDS = 120.0
# Which Okta *profile attribute* matches each OIDC claim that OKTA_IDENTITY_CLAIM
# may name. The sign-in flow reads a claim from the ID token. This API returns
@@ -171,10 +179,13 @@ def fetch_users(http: Optional[httpx.Client] = None) -> "list[OktaUser]":
url: Optional[str] = f"{base}/api/v1/users"
params: Optional[dict] = {"limit": PAGE_LIMIT}
users: "list[OktaUser]" = []
started = time.monotonic()
try:
for _ in range(MAX_PAGES):
if not url:
break
if time.monotonic() - started > TOTAL_DEADLINE_SECONDS:
raise _fail(f"took longer than {int(TOTAL_DEADLINE_SECONDS)} seconds in total")
try:
resp = client.get(url, params=params, headers=headers,
follow_redirects=False)
@@ -336,3 +347,107 @@ def run_sync(db: Session, fetch: Optional[Callable[[], "list[OktaUser]"]] = None
len(okta_users), checked, len(planned))
return SyncResult(ok=True, okta_users=len(okta_users), checked=checked,
disabled=[name for name, _ in planned])
# ── T13.4: running it on a schedule ───────────────────────────────────────────
# The api container runs gunicorn with 2 workers, and a later deploy may run more
# than one container. Each worker starts its own timer thread below. Without a
# gate, all of them would call Okta every cycle and write overlapping changes.
# The gate is one row in app_settings (key "okta_sync"). Its updated_at is "when
# the last run was claimed". A worker claims a run with ONE atomic conditional
# UPDATE: it succeeds only if the last claim is older than the interval. Exactly
# one worker wins each cycle, on SQLite and on Postgres, with no lock held.
#
# The row survives a restart, so a restart inside the interval does not start a
# second run. A slow run cannot overlap the next: a fetch is bounded to
# TOTAL_DEADLINE_SECONDS (120) and the interval can never be set below
# INTERVAL_MIN_SECONDS (300). Inside one process the timer thread runs one cycle at
# a time, so cycles cannot stack there either.
CLAIM_KEY = "okta_sync"
INTERVAL_DEFAULT_SECONDS = 900 # 15 minutes (D18, proposed, not confirmed)
INTERVAL_MIN_SECONDS = 300
TICK_SECONDS = 60 # how often each worker checks if a run is due
START_DELAY_SECONDS = 10 # let the app finish starting first
def interval_seconds() -> int:
"""OKTA_SYNC_INTERVAL_SECONDS, default 900. A value that is not a number uses
the default. A value under 300 is raised to 300 (see above for why)."""
raw = os.getenv("OKTA_SYNC_INTERVAL_SECONDS", "").strip()
if not raw:
return INTERVAL_DEFAULT_SECONDS
try:
value = int(raw)
except ValueError:
log.warning("OKTA_SYNC_INTERVAL_SECONDS=%r is not a whole number; using %d",
raw, INTERVAL_DEFAULT_SECONDS)
return INTERVAL_DEFAULT_SECONDS
if value < INTERVAL_MIN_SECONDS:
log.warning("OKTA_SYNC_INTERVAL_SECONDS=%d is under the minimum; using %d",
value, INTERVAL_MIN_SECONDS)
return INTERVAL_MIN_SECONDS
return value
def claim(db: Session, interval: int, now: Optional[datetime] = None) -> bool:
"""Try to claim this cycle. True for exactly one caller per interval."""
now = now or datetime.now(timezone.utc)
cutoff = now - timedelta(seconds=interval)
try:
row = db.get(models.AppSetting, CLAIM_KEY)
if row is None:
db.add(models.AppSetting(key=CLAIM_KEY, value={}, updated_at=now))
db.commit() # the insert is the claim
return True
except IntegrityError:
db.rollback() # another worker inserted first: it won
return False
db.rollback() # end the read; the UPDATE starts fresh
result = db.execute(
update(models.AppSetting)
.where(models.AppSetting.key == CLAIM_KEY, models.AppSetting.updated_at <= cutoff)
.values(updated_at=now)
)
db.commit()
return result.rowcount == 1
def run_if_due(session_factory: Callable[[], Session],
fetch: Optional[Callable[[], "list[OktaUser]"]] = None,
now: Optional[datetime] = None) -> Optional[SyncResult]:
"""One timer tick: claim the cycle, and if it is ours, run the sync. Returns
the SyncResult, or None when the cycle was not ours (or could not be claimed).
Nothing here can turn a failure into a change: run_sync() owns that rule, and
anything that goes wrong around it is logged and ends the tick."""
try:
with session_factory() as db:
if not claim(db, interval_seconds(), now=now):
return None
with session_factory() as db:
return run_sync(db, fetch=fetch)
except Exception as exc:
log.error("Okta sync tick failed, no account was changed: unexpected %s", type(exc).__name__)
return None
def start_scheduler(session_factory: Callable[[], Session]) -> "Optional[threading.Event]":
"""Start the timer thread. Returns a stop Event (set it to end the thread), or
None when the sync is off because OKTA_API_TOKEN or the org URL is missing.
Logs one line either way, so an operator can see it in `docker compose logs api`."""
log.info(describe())
if not is_configured():
return None
stop = threading.Event()
def loop():
if stop.wait(START_DELAY_SECONDS):
return
while not stop.is_set():
run_if_due(session_factory)
if stop.wait(TICK_SECONDS):
return
threading.Thread(target=loop, name="okta-sync", daemon=True).start()
log.info("Okta sync scheduled: every %d seconds (checked each %d s by every worker; "
"one wins each cycle)", interval_seconds(), TICK_SECONDS)
return stop

View File

@@ -230,6 +230,23 @@ every_log = " ".join(cap.lines)
chk("no log line holds the API token", TOKEN not in every_log)
chk("no log line holds the client secret", CLIENT_SECRET not in every_log)
# ── 8. the whole fetch is time-bounded (T13.4 relies on it) ──────────────────
print("\n8. the whole fetch is time-bounded")
reset_env()
saved = okta_sync.TOTAL_DEADLINE_SECONDS
okta_sync.TOTAL_DEADLINE_SECONDS = 0.0
def slow_pages(req):
return httpx.Response(200, json=[user("alice")],
headers={"Link": f'<{ORG}/api/v1/users?after=x>; rel="next"'})
raises(slow_pages, "a fetch that runs past the total deadline")
okta_sync.TOTAL_DEADLINE_SECONDS = saved
chk("the real deadline is under the shortest allowed interval",
okta_sync.TOTAL_DEADLINE_SECONDS < okta_sync.INTERVAL_MIN_SECONDS)
print()
print("ALL PASSED" if not fails else f"{fails} FAILED")
sys.exit(0 if not fails else 1)

View File

@@ -0,0 +1,242 @@
#!/usr/bin/env python3
"""Does the Okta sync run on a schedule without doubling up? - D18 / T13.4.
The api container runs 2 gunicorn workers, and each starts its own timer. Checked
here: exactly one of them wins each cycle (including a race on a real file
database), a restart inside the interval does not start a second run, the
interval is configurable and cannot go below the minimum, the timer thread starts
and stops, nothing starts when the sync is not configured, and a failure inside
the scheduled path still changes nothing.
Self-contained: a throwaway SQLite file, threads, no server, no network, no browser.
Exit 0 all passed, 1 a failure.
"""
import os
import sys
import tempfile
import threading
import time
from datetime import datetime, timedelta, timezone
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
from sqlalchemy import create_engine, select # noqa: E402
from sqlalchemy.orm import sessionmaker # noqa: E402
from server import models, okta_auth, okta_sync # noqa: E402
from server.okta_sync import OktaSyncError, OktaUser # noqa: E402
fails = 0
def chk(label, ok, extra=""):
global fails
print(("PASS " if ok else "FAIL ") + label + (f" [{extra}]" if (extra and not ok) else ""))
if not ok:
fails += 1
tmp = tempfile.mkdtemp(prefix="oktasync_")
engine = create_engine(f"sqlite:///{os.path.join(tmp, 't.db')}",
connect_args={"check_same_thread": False, "timeout": 20}, future=True)
models.Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine, autoflush=False, future=True)
INTERVAL = 900
os.environ["OKTA_SYNC_INTERVAL_SECONDS"] = str(INTERVAL)
T0 = datetime(2026, 10, 16, 12, 0, 0, tzinfo=timezone.utc)
def wipe(users=()):
with Session() as db:
db.query(models.AuditLog).delete()
db.query(models.User).delete()
db.query(models.AppSetting).filter(models.AppSetting.key == okta_sync.CLAIM_KEY).delete()
for name, active in users:
db.add(models.User(id="u_" + name, username=name, is_active=active))
db.commit()
def users_state():
with Session() as db:
return ({u.username: u.is_active for u in db.scalars(select(models.User))},
db.query(models.AuditLog).count())
# ── 1. interval ───────────────────────────────────────────────────────────────
print("\n1. interval")
def iv(v):
if v is None:
os.environ.pop("OKTA_SYNC_INTERVAL_SECONDS", None)
else:
os.environ["OKTA_SYNC_INTERVAL_SECONDS"] = v
return okta_sync.interval_seconds()
chk("default is 900 seconds (15 minutes)", iv(None) == 900 and iv("") == 900)
chk("an env value is used", iv("600") == 600 and iv("3600") == 3600)
chk("not a number: the default", iv("soon") == 900 and iv("1.5") == 900)
chk("under the minimum is raised to 300 (a slow cycle can never overlap the next)",
iv("60") == 300 and iv("0") == 300 and iv("-5") == 300)
chk("a fetch is bounded below the minimum interval",
okta_sync.TOTAL_DEADLINE_SECONDS < okta_sync.INTERVAL_MIN_SECONDS)
iv(str(INTERVAL))
# ── 2. the claim ──────────────────────────────────────────────────────────────
print("\n2. the claim gate")
wipe()
with Session() as db:
chk("first ever claim wins (the insert is the claim)", okta_sync.claim(db, INTERVAL, now=T0) is True)
with Session() as db:
chk("a second claim in the same instant loses", okta_sync.claim(db, INTERVAL, now=T0) is False)
with Session() as db:
chk("one second before the interval is up: loses",
okta_sync.claim(db, INTERVAL, now=T0 + timedelta(seconds=INTERVAL - 1)) is False)
with Session() as db:
chk("when the interval is up: wins",
okta_sync.claim(db, INTERVAL, now=T0 + timedelta(seconds=INTERVAL)) is True)
with Session() as db:
chk("and then the next caller loses again",
okta_sync.claim(db, INTERVAL, now=T0 + timedelta(seconds=INTERVAL + 1)) is False)
# ── 3. the race ───────────────────────────────────────────────────────────────
print("\n3. exactly one of many workers wins (a real race on a file database)")
def race(n=8):
barrier = threading.Barrier(n)
wins, errors = [], []
def worker():
try:
with Session() as db:
barrier.wait()
wins.append(okta_sync.claim(db, INTERVAL, now=datetime.now(timezone.utc)))
except Exception as exc: # a lost race may surface as an error
errors.append(type(exc).__name__)
wins.append(False)
ts = [threading.Thread(target=worker) for _ in range(n)]
[t.start() for t in ts]
[t.join() for t in ts]
return sum(1 for w in wins if w), errors
wipe()
won, errs = race()
chk("no row yet, 8 workers at once: exactly 1 wins (the insert race)", won == 1, f"{won} {errs}")
bad = []
for i in range(15):
with Session() as db: # age the claim so the next race is due again
db.query(models.AppSetting).filter(models.AppSetting.key == okta_sync.CLAIM_KEY).update(
{"updated_at": datetime.now(timezone.utc) - timedelta(seconds=INTERVAL + 5)})
db.commit()
won, errs = race()
if won != 1:
bad.append((i, won, errs))
chk("row due, 8 workers at once, 15 rounds: exactly 1 wins every round", not bad, str(bad))
# ── 4. one tick ───────────────────────────────────────────────────────────────
print("\n4. a tick runs the sync once per cycle")
wipe([("alice", True), ("bob", True)])
calls = []
def fetch_ok():
calls.append(1)
return [OktaUser("alice", "ACTIVE"), OktaUser("bob", "SUSPENDED")]
r1 = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0)
chk("first tick claims and runs", r1 is not None and r1.ok and r1.disabled == ["bob"])
r2 = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0 + timedelta(seconds=30))
chk("a tick 30 seconds later does nothing (not due)", r2 is None and len(calls) == 1)
r3 = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0 + timedelta(seconds=INTERVAL))
chk("a tick when due runs again", r3 is not None and r3.ok and len(calls) == 2)
print("\n ...a restart inside the interval does not start a second run")
wipe([("alice", True), ("bob", True)])
calls.clear()
okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0)
# "restart": a brand-new caller with nothing in memory, 2 minutes later
r = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0 + timedelta(seconds=120))
chk("the claim row outlives the process: no duplicate run", r is None and len(calls) == 1)
print("\n ...two workers tick in the same instant")
wipe([("alice", True), ("bob", True)])
calls.clear()
results = []
barrier = threading.Barrier(2)
def tick():
barrier.wait()
results.append(okta_sync.run_if_due(Session, fetch=fetch_ok))
ts = [threading.Thread(target=tick) for _ in range(2)]
[t.start() for t in ts]
[t.join() for t in ts]
chk("Okta is asked once, not twice", len(calls) == 1, str(len(calls)))
chk("exactly one worker got a result", sum(1 for r in results if r is not None) == 1)
_, audits = users_state()
chk("one audit row, not two", audits == 1, str(audits))
# ── 5. fail closed through the scheduled path ─────────────────────────────────
print("\n5. FAIL CLOSED through the scheduled wrapper (not just run_sync)")
for label, exc in (("OktaSyncError", OktaSyncError("HTTP 500")), ("RuntimeError", RuntimeError("x")),
("TimeoutError", TimeoutError("t"))):
wipe([("alice", True), ("bob", True)])
def failing(e=exc):
raise e
r = okta_sync.run_if_due(Session, fetch=failing, now=T0)
u, a = users_state()
chk(f"{label} in the scheduled path: ok is False, zero changes, zero audit rows",
r is not None and r.ok is False and all(u.values()) and a == 0)
wipe([("alice", True), ("bob", True)])
def broken_factory():
raise RuntimeError("database is down")
chk("the database failing at claim time: the tick ends quietly, returns None",
okta_sync.run_if_due(broken_factory, fetch=fetch_ok, now=T0) is None)
u, a = users_state()
chk("...and nothing changed", all(u.values()) and a == 0)
# ── 6. the thread ─────────────────────────────────────────────────────────────
print("\n6. the timer thread")
os.environ["OKTA_API_TOKEN"] = ""
okta_auth.ISSUER = "https://example.okta.com/oauth2/default"
chk("no token: nothing starts", okta_sync.start_scheduler(Session) is None)
chk("...no thread named okta-sync", not any(t.name == "okta-sync" for t in threading.enumerate()))
os.environ["OKTA_API_TOKEN"] = "tok"
wipe([("alice", True), ("bob", True)])
okta_sync.START_DELAY_SECONDS = 0.05
okta_sync.TICK_SECONDS = 0.05
ran = []
real_fetch = okta_sync.fetch_users
okta_sync.fetch_users = lambda http=None: (ran.append(1) or
[OktaUser("alice", "ACTIVE"), OktaUser("bob", "DEPROVISIONED")])
stop = okta_sync.start_scheduler(Session)
chk("configured: a stop Event comes back", stop is not None)
time.sleep(1.2) # about 20 ticks
u, a = users_state()
chk("the thread ran the sync on its own (bob was disabled without a manual call)", u["bob"] is False)
chk("many ticks, one run (the claim gate holds)", len(ran) == 1 and a == 1, f"ran={len(ran)} audits={a}")
th = [t for t in threading.enumerate() if t.name == "okta-sync"]
chk("one timer thread is alive", len(th) == 1 and th[0].is_alive())
chk("the thread is a daemon (never blocks shutdown)", th[0].daemon)
stop.set()
th[0].join(timeout=3)
chk("setting the stop Event ends the thread", not th[0].is_alive())
okta_sync.fetch_users = real_fetch
os.environ["OKTA_API_TOKEN"] = ""
print()
print("ALL PASSED" if not fails else f"{fails} FAILED")
sys.exit(0 if not fails else 1)