T13.4: run the Okta sync on a schedule (D18)
A daemon timer thread starts from a new lifespan hook in app.py and calls the T13.3 sync every OKTA_SYNC_INTERVAL_SECONDS (default 900, minimum 300). With no OKTA_API_TOKEN it starts nothing and the startup log says so. The api runs 2 gunicorn workers, so each starts its own timer. A single atomic conditional UPDATE on one app_settings row (key okta_sync) is the gate: exactly one worker wins each cycle, on SQLite and Postgres, with no lock held. The row survives a restart, so a restart inside the interval starts no second run. A fetch is now capped at 120 s in total, so a run is always shorter than the minimum interval and cannot overlap the next. No migration needed. docker-compose.yml passes OKTA_API_TOKEN, OKTA_ORG_URL, OKTA_SYNC_PROFILE_ATTRIBUTE and OKTA_SYNC_INTERVAL_SECONDS to the api service. tests/okta_sync_schedule_check.py: 38 checks including an 8-worker race on a file database (exactly one winner, 16 rounds) and fail-closed through the scheduled path. okta_auth_check.py still 22/22.
This commit is contained in:
@@ -49,6 +49,18 @@ services:
|
|||||||
# here instead, so an operator who leaves .env's copy commented out gets
|
# here instead, so an operator who leaves .env's copy commented out gets
|
||||||
# the real default, not a broken one.
|
# the real default, not a broken one.
|
||||||
OKTA_IDENTITY_CLAIM: ${OKTA_IDENTITY_CLAIM:-preferred_username}
|
OKTA_IDENTITY_CLAIM: ${OKTA_IDENTITY_CLAIM:-preferred_username}
|
||||||
|
# Okta deprovisioning sync (D18, wave 13). OFF until OKTA_API_TOKEN is set; the
|
||||||
|
# startup log says which. The token is a read-only Okta API token, NOT the OIDC
|
||||||
|
# client secret above. Empty values here mean "unset" to the code, so the
|
||||||
|
# ${VAR:-} form is safe for all four (unlike OKTA_IDENTITY_CLAIM above, whose
|
||||||
|
# code default only applies when the variable is absent).
|
||||||
|
# Reaching Okta needs outbound https. The `outbound` network the api service
|
||||||
|
# already joins is an ordinary bridge with a gateway, so no new network is
|
||||||
|
# needed; the host firewall must allow https to the Okta org.
|
||||||
|
OKTA_API_TOKEN: ${OKTA_API_TOKEN:-}
|
||||||
|
OKTA_ORG_URL: ${OKTA_ORG_URL:-}
|
||||||
|
OKTA_SYNC_PROFILE_ATTRIBUTE: ${OKTA_SYNC_PROFILE_ATTRIBUTE:-}
|
||||||
|
OKTA_SYNC_INTERVAL_SECONDS: ${OKTA_SYNC_INTERVAL_SECONDS:-}
|
||||||
# Optional — SMTP password for WP-assignment emails. Email is off by
|
# Optional — SMTP password for WP-assignment emails. Email is off by
|
||||||
# default and enabled from the Admin console; this is the only email
|
# default and enabled from the Admin console; this is the only email
|
||||||
# secret and it is never stored in the DB. Leave unset until configured.
|
# secret and it is never stored in the DB. Leave unset until configured.
|
||||||
|
|||||||
@@ -202,14 +202,28 @@ is sufficient or Okta needs a distinct allowance).
|
|||||||
|
|
||||||
**Done when:**
|
**Done when:**
|
||||||
|
|
||||||
- [ ] the job runs automatically on the configured interval without manual
|
- [x] the job runs automatically on the configured interval without manual
|
||||||
invocation
|
invocation
|
||||||
- [ ] interval is env-configurable with a sane default
|
*(2026-10-16: a daemon timer thread starts from the app's lifespan hook (`server/app.py`), checks once a minute, and ran the sync on its own in `tests/okta_sync_schedule_check.py`. With `OKTA_API_TOKEN` unset it starts nothing and the startup log says `Okta sync DISABLED`. Booted the real app both ways; the thread starts and ends with the app.)*
|
||||||
- [ ] a container restart does not produce a duplicate/overlapping run, and a
|
- [x] interval is env-configurable with a sane default
|
||||||
|
*(2026-10-16: `OKTA_SYNC_INTERVAL_SECONDS`, default 900. A non-number uses the default. A value under 300 is raised to 300. Passed through `docker-compose.yml`.)*
|
||||||
|
- [x] a container restart does not produce a duplicate/overlapping run, and a
|
||||||
slow cycle does not stack with the next one
|
slow cycle does not stack with the next one
|
||||||
|
*(2026-10-16: the api runs 2 gunicorn workers, so each starts its own timer. One row in `app_settings` (key `okta_sync`) is the gate: a worker claims a cycle with one atomic conditional UPDATE, so exactly one wins. Checked with 8 racing workers on a file database, 15 rounds plus the first-insert race: exactly 1 winner every time. The row outlives a restart, so a restart inside the interval starts nothing. A slow run cannot overlap the next: a fetch is capped at 120 s in total and the interval cannot go below 300 s. Inside one process the thread runs one cycle at a time. This needed no migration: `app_settings` already exists.)*
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Decided 2026-10-16 (task's call, per this task's "Do"):** in-process timer in the
|
||||||
|
`api` container, as D18 proposed, not a sidecar. The `backup` pattern was the
|
||||||
|
alternative; it would need its own image and would not remove the 2-worker
|
||||||
|
duplicate question, because the sync reads the application database anyway.
|
||||||
|
**Network check, as the task asked:** the `outbound` network the `api` service
|
||||||
|
already joins is an ordinary bridge with a gateway, so Okta is reachable with no
|
||||||
|
network change. The host firewall must allow https to the Okta org.
|
||||||
|
**One change to T13.2's code:** `fetch_users()` now also stops after 120 seconds in
|
||||||
|
total (`TOTAL_DEADLINE_SECONDS`), which is what lets the scheduler promise no
|
||||||
|
overlap. Tested in `tests/okta_sync_check.py` section 8.
|
||||||
|
|
||||||
### T13.5 — D18: admin visibility
|
### T13.5 — D18: admin visibility
|
||||||
|
|
||||||
- **Items:** `D18`
|
- **Items:** `D18`
|
||||||
|
|||||||
@@ -88,6 +88,12 @@ OKTA_REDIRECT_URI=CHANGE_ME
|
|||||||
# profile attribute of the same name. Set this only if your tenant builds the
|
# profile attribute of the same name. Set this only if your tenant builds the
|
||||||
# claim from a different attribute.
|
# claim from a different attribute.
|
||||||
# OKTA_SYNC_PROFILE_ATTRIBUTE=login
|
# OKTA_SYNC_PROFILE_ATTRIBUTE=login
|
||||||
|
#
|
||||||
|
# How often the sync runs, in seconds. Default 900 (15 minutes), proposed and not
|
||||||
|
# confirmed with IT. A value under 300 is raised to 300. Every api worker checks
|
||||||
|
# once a minute and one of them wins each cycle, so raising the worker count does
|
||||||
|
# not raise the call rate to Okta.
|
||||||
|
# OKTA_SYNC_INTERVAL_SECONDS=900
|
||||||
|
|
||||||
# ── Email notifications (optional) ─────────────────────────────────────────────
|
# ── Email notifications (optional) ─────────────────────────────────────────────
|
||||||
# WP-assignment emails are OFF by default and are turned on from the Admin
|
# WP-assignment emails are OFF by default and are turned on from the Admin
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ Interactive docs: http://<host>/api/docs
|
|||||||
"""
|
"""
|
||||||
import base64
|
import base64
|
||||||
import csv
|
import csv
|
||||||
|
import contextlib
|
||||||
import hashlib
|
import hashlib
|
||||||
import hmac
|
import hmac
|
||||||
import io
|
import io
|
||||||
@@ -31,8 +32,8 @@ from sqlalchemy import select, delete, func, or_
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
from starlette.middleware.sessions import SessionMiddleware
|
from starlette.middleware.sessions import SessionMiddleware
|
||||||
|
|
||||||
from .db import Base, engine, get_db
|
from .db import Base, SessionLocal, engine, get_db
|
||||||
from . import models, auth, notify, assets_db, okta_auth, okta_fake
|
from . import models, auth, notify, assets_db, okta_auth, okta_fake, okta_sync
|
||||||
|
|
||||||
log = logging.getLogger("wpsuite.app")
|
log = logging.getLogger("wpsuite.app")
|
||||||
|
|
||||||
@@ -47,7 +48,21 @@ if engine.dialect.name == "sqlite":
|
|||||||
# Interactive docs are handy in dev but hand an attacker the full API map in prod,
|
# Interactive docs are handy in dev but hand an attacker the full API map in prod,
|
||||||
# so enable them only on the SQLite dev fallback (production runs on Postgres).
|
# so enable them only on the SQLite dev fallback (production runs on Postgres).
|
||||||
_docs_enabled = engine.dialect.name == "sqlite"
|
_docs_enabled = engine.dialect.name == "sqlite"
|
||||||
|
@contextlib.asynccontextmanager
|
||||||
|
async def _lifespan(_app):
|
||||||
|
"""Start the Okta deprovisioning sync (D18 / T13.4) when the process starts and
|
||||||
|
stop it when the process ends. With no OKTA_API_TOKEN it starts nothing and
|
||||||
|
says so in the startup log."""
|
||||||
|
stop = okta_sync.start_scheduler(SessionLocal)
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
if stop is not None:
|
||||||
|
stop.set()
|
||||||
|
|
||||||
|
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
|
lifespan=_lifespan,
|
||||||
title="Work Package Suite API",
|
title="Work Package Suite API",
|
||||||
docs_url="/api/docs" if _docs_enabled else None,
|
docs_url="/api/docs" if _docs_enabled else None,
|
||||||
redoc_url=None,
|
redoc_url=None,
|
||||||
|
|||||||
@@ -37,13 +37,17 @@ request URL, so the URL is not copied into the message either.
|
|||||||
"""
|
"""
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
import uuid
|
import uuid
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from typing import Callable, Optional
|
from typing import Callable, Optional
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select, update
|
||||||
|
from sqlalchemy.exc import IntegrityError
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from . import models, okta_auth
|
from . import models, okta_auth
|
||||||
@@ -57,6 +61,10 @@ PAGE_LIMIT = 200
|
|||||||
# raises when the bound is reached. 50 pages of 200 is 10,000 people.
|
# raises when the bound is reached. 50 pages of 200 is 10,000 people.
|
||||||
MAX_PAGES = 50
|
MAX_PAGES = 50
|
||||||
TIMEOUT_SECONDS = 15.0
|
TIMEOUT_SECONDS = 15.0
|
||||||
|
# The whole fetch, all pages together, must finish inside this. T13.4 relies on it:
|
||||||
|
# a run is always shorter than the shortest allowed interval, so one cycle can
|
||||||
|
# never still be running when the next is due.
|
||||||
|
TOTAL_DEADLINE_SECONDS = 120.0
|
||||||
|
|
||||||
# Which Okta *profile attribute* matches each OIDC claim that OKTA_IDENTITY_CLAIM
|
# Which Okta *profile attribute* matches each OIDC claim that OKTA_IDENTITY_CLAIM
|
||||||
# may name. The sign-in flow reads a claim from the ID token. This API returns
|
# may name. The sign-in flow reads a claim from the ID token. This API returns
|
||||||
@@ -171,10 +179,13 @@ def fetch_users(http: Optional[httpx.Client] = None) -> "list[OktaUser]":
|
|||||||
url: Optional[str] = f"{base}/api/v1/users"
|
url: Optional[str] = f"{base}/api/v1/users"
|
||||||
params: Optional[dict] = {"limit": PAGE_LIMIT}
|
params: Optional[dict] = {"limit": PAGE_LIMIT}
|
||||||
users: "list[OktaUser]" = []
|
users: "list[OktaUser]" = []
|
||||||
|
started = time.monotonic()
|
||||||
try:
|
try:
|
||||||
for _ in range(MAX_PAGES):
|
for _ in range(MAX_PAGES):
|
||||||
if not url:
|
if not url:
|
||||||
break
|
break
|
||||||
|
if time.monotonic() - started > TOTAL_DEADLINE_SECONDS:
|
||||||
|
raise _fail(f"took longer than {int(TOTAL_DEADLINE_SECONDS)} seconds in total")
|
||||||
try:
|
try:
|
||||||
resp = client.get(url, params=params, headers=headers,
|
resp = client.get(url, params=params, headers=headers,
|
||||||
follow_redirects=False)
|
follow_redirects=False)
|
||||||
@@ -336,3 +347,107 @@ def run_sync(db: Session, fetch: Optional[Callable[[], "list[OktaUser]"]] = None
|
|||||||
len(okta_users), checked, len(planned))
|
len(okta_users), checked, len(planned))
|
||||||
return SyncResult(ok=True, okta_users=len(okta_users), checked=checked,
|
return SyncResult(ok=True, okta_users=len(okta_users), checked=checked,
|
||||||
disabled=[name for name, _ in planned])
|
disabled=[name for name, _ in planned])
|
||||||
|
|
||||||
|
|
||||||
|
# ── T13.4: running it on a schedule ───────────────────────────────────────────
|
||||||
|
# The api container runs gunicorn with 2 workers, and a later deploy may run more
|
||||||
|
# than one container. Each worker starts its own timer thread below. Without a
|
||||||
|
# gate, all of them would call Okta every cycle and write overlapping changes.
|
||||||
|
# The gate is one row in app_settings (key "okta_sync"). Its updated_at is "when
|
||||||
|
# the last run was claimed". A worker claims a run with ONE atomic conditional
|
||||||
|
# UPDATE: it succeeds only if the last claim is older than the interval. Exactly
|
||||||
|
# one worker wins each cycle, on SQLite and on Postgres, with no lock held.
|
||||||
|
#
|
||||||
|
# The row survives a restart, so a restart inside the interval does not start a
|
||||||
|
# second run. A slow run cannot overlap the next: a fetch is bounded to
|
||||||
|
# TOTAL_DEADLINE_SECONDS (120) and the interval can never be set below
|
||||||
|
# INTERVAL_MIN_SECONDS (300). Inside one process the timer thread runs one cycle at
|
||||||
|
# a time, so cycles cannot stack there either.
|
||||||
|
CLAIM_KEY = "okta_sync"
|
||||||
|
INTERVAL_DEFAULT_SECONDS = 900 # 15 minutes (D18, proposed, not confirmed)
|
||||||
|
INTERVAL_MIN_SECONDS = 300
|
||||||
|
TICK_SECONDS = 60 # how often each worker checks if a run is due
|
||||||
|
START_DELAY_SECONDS = 10 # let the app finish starting first
|
||||||
|
|
||||||
|
|
||||||
|
def interval_seconds() -> int:
|
||||||
|
"""OKTA_SYNC_INTERVAL_SECONDS, default 900. A value that is not a number uses
|
||||||
|
the default. A value under 300 is raised to 300 (see above for why)."""
|
||||||
|
raw = os.getenv("OKTA_SYNC_INTERVAL_SECONDS", "").strip()
|
||||||
|
if not raw:
|
||||||
|
return INTERVAL_DEFAULT_SECONDS
|
||||||
|
try:
|
||||||
|
value = int(raw)
|
||||||
|
except ValueError:
|
||||||
|
log.warning("OKTA_SYNC_INTERVAL_SECONDS=%r is not a whole number; using %d",
|
||||||
|
raw, INTERVAL_DEFAULT_SECONDS)
|
||||||
|
return INTERVAL_DEFAULT_SECONDS
|
||||||
|
if value < INTERVAL_MIN_SECONDS:
|
||||||
|
log.warning("OKTA_SYNC_INTERVAL_SECONDS=%d is under the minimum; using %d",
|
||||||
|
value, INTERVAL_MIN_SECONDS)
|
||||||
|
return INTERVAL_MIN_SECONDS
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def claim(db: Session, interval: int, now: Optional[datetime] = None) -> bool:
|
||||||
|
"""Try to claim this cycle. True for exactly one caller per interval."""
|
||||||
|
now = now or datetime.now(timezone.utc)
|
||||||
|
cutoff = now - timedelta(seconds=interval)
|
||||||
|
try:
|
||||||
|
row = db.get(models.AppSetting, CLAIM_KEY)
|
||||||
|
if row is None:
|
||||||
|
db.add(models.AppSetting(key=CLAIM_KEY, value={}, updated_at=now))
|
||||||
|
db.commit() # the insert is the claim
|
||||||
|
return True
|
||||||
|
except IntegrityError:
|
||||||
|
db.rollback() # another worker inserted first: it won
|
||||||
|
return False
|
||||||
|
db.rollback() # end the read; the UPDATE starts fresh
|
||||||
|
result = db.execute(
|
||||||
|
update(models.AppSetting)
|
||||||
|
.where(models.AppSetting.key == CLAIM_KEY, models.AppSetting.updated_at <= cutoff)
|
||||||
|
.values(updated_at=now)
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
return result.rowcount == 1
|
||||||
|
|
||||||
|
|
||||||
|
def run_if_due(session_factory: Callable[[], Session],
|
||||||
|
fetch: Optional[Callable[[], "list[OktaUser]"]] = None,
|
||||||
|
now: Optional[datetime] = None) -> Optional[SyncResult]:
|
||||||
|
"""One timer tick: claim the cycle, and if it is ours, run the sync. Returns
|
||||||
|
the SyncResult, or None when the cycle was not ours (or could not be claimed).
|
||||||
|
Nothing here can turn a failure into a change: run_sync() owns that rule, and
|
||||||
|
anything that goes wrong around it is logged and ends the tick."""
|
||||||
|
try:
|
||||||
|
with session_factory() as db:
|
||||||
|
if not claim(db, interval_seconds(), now=now):
|
||||||
|
return None
|
||||||
|
with session_factory() as db:
|
||||||
|
return run_sync(db, fetch=fetch)
|
||||||
|
except Exception as exc:
|
||||||
|
log.error("Okta sync tick failed, no account was changed: unexpected %s", type(exc).__name__)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def start_scheduler(session_factory: Callable[[], Session]) -> "Optional[threading.Event]":
|
||||||
|
"""Start the timer thread. Returns a stop Event (set it to end the thread), or
|
||||||
|
None when the sync is off because OKTA_API_TOKEN or the org URL is missing.
|
||||||
|
Logs one line either way, so an operator can see it in `docker compose logs api`."""
|
||||||
|
log.info(describe())
|
||||||
|
if not is_configured():
|
||||||
|
return None
|
||||||
|
stop = threading.Event()
|
||||||
|
|
||||||
|
def loop():
|
||||||
|
if stop.wait(START_DELAY_SECONDS):
|
||||||
|
return
|
||||||
|
while not stop.is_set():
|
||||||
|
run_if_due(session_factory)
|
||||||
|
if stop.wait(TICK_SECONDS):
|
||||||
|
return
|
||||||
|
|
||||||
|
threading.Thread(target=loop, name="okta-sync", daemon=True).start()
|
||||||
|
log.info("Okta sync scheduled: every %d seconds (checked each %d s by every worker; "
|
||||||
|
"one wins each cycle)", interval_seconds(), TICK_SECONDS)
|
||||||
|
return stop
|
||||||
|
|||||||
@@ -230,6 +230,23 @@ every_log = " ".join(cap.lines)
|
|||||||
chk("no log line holds the API token", TOKEN not in every_log)
|
chk("no log line holds the API token", TOKEN not in every_log)
|
||||||
chk("no log line holds the client secret", CLIENT_SECRET not in every_log)
|
chk("no log line holds the client secret", CLIENT_SECRET not in every_log)
|
||||||
|
|
||||||
|
# ── 8. the whole fetch is time-bounded (T13.4 relies on it) ──────────────────
|
||||||
|
print("\n8. the whole fetch is time-bounded")
|
||||||
|
reset_env()
|
||||||
|
saved = okta_sync.TOTAL_DEADLINE_SECONDS
|
||||||
|
okta_sync.TOTAL_DEADLINE_SECONDS = 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def slow_pages(req):
|
||||||
|
return httpx.Response(200, json=[user("alice")],
|
||||||
|
headers={"Link": f'<{ORG}/api/v1/users?after=x>; rel="next"'})
|
||||||
|
|
||||||
|
|
||||||
|
raises(slow_pages, "a fetch that runs past the total deadline")
|
||||||
|
okta_sync.TOTAL_DEADLINE_SECONDS = saved
|
||||||
|
chk("the real deadline is under the shortest allowed interval",
|
||||||
|
okta_sync.TOTAL_DEADLINE_SECONDS < okta_sync.INTERVAL_MIN_SECONDS)
|
||||||
|
|
||||||
print()
|
print()
|
||||||
print("ALL PASSED" if not fails else f"{fails} FAILED")
|
print("ALL PASSED" if not fails else f"{fails} FAILED")
|
||||||
sys.exit(0 if not fails else 1)
|
sys.exit(0 if not fails else 1)
|
||||||
|
|||||||
242
tests/okta_sync_schedule_check.py
Normal file
242
tests/okta_sync_schedule_check.py
Normal file
@@ -0,0 +1,242 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Does the Okta sync run on a schedule without doubling up? - D18 / T13.4.
|
||||||
|
|
||||||
|
The api container runs 2 gunicorn workers, and each starts its own timer. Checked
|
||||||
|
here: exactly one of them wins each cycle (including a race on a real file
|
||||||
|
database), a restart inside the interval does not start a second run, the
|
||||||
|
interval is configurable and cannot go below the minimum, the timer thread starts
|
||||||
|
and stops, nothing starts when the sync is not configured, and a failure inside
|
||||||
|
the scheduled path still changes nothing.
|
||||||
|
|
||||||
|
Self-contained: a throwaway SQLite file, threads, no server, no network, no browser.
|
||||||
|
Exit 0 all passed, 1 a failure.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
sys.path.insert(0, ROOT)
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine, select # noqa: E402
|
||||||
|
from sqlalchemy.orm import sessionmaker # noqa: E402
|
||||||
|
|
||||||
|
from server import models, okta_auth, okta_sync # noqa: E402
|
||||||
|
from server.okta_sync import OktaSyncError, OktaUser # noqa: E402
|
||||||
|
|
||||||
|
fails = 0
|
||||||
|
|
||||||
|
|
||||||
|
def chk(label, ok, extra=""):
|
||||||
|
global fails
|
||||||
|
print(("PASS " if ok else "FAIL ") + label + (f" [{extra}]" if (extra and not ok) else ""))
|
||||||
|
if not ok:
|
||||||
|
fails += 1
|
||||||
|
|
||||||
|
|
||||||
|
tmp = tempfile.mkdtemp(prefix="oktasync_")
|
||||||
|
engine = create_engine(f"sqlite:///{os.path.join(tmp, 't.db')}",
|
||||||
|
connect_args={"check_same_thread": False, "timeout": 20}, future=True)
|
||||||
|
models.Base.metadata.create_all(engine)
|
||||||
|
Session = sessionmaker(bind=engine, autoflush=False, future=True)
|
||||||
|
INTERVAL = 900
|
||||||
|
os.environ["OKTA_SYNC_INTERVAL_SECONDS"] = str(INTERVAL)
|
||||||
|
T0 = datetime(2026, 10, 16, 12, 0, 0, tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def wipe(users=()):
|
||||||
|
with Session() as db:
|
||||||
|
db.query(models.AuditLog).delete()
|
||||||
|
db.query(models.User).delete()
|
||||||
|
db.query(models.AppSetting).filter(models.AppSetting.key == okta_sync.CLAIM_KEY).delete()
|
||||||
|
for name, active in users:
|
||||||
|
db.add(models.User(id="u_" + name, username=name, is_active=active))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def users_state():
|
||||||
|
with Session() as db:
|
||||||
|
return ({u.username: u.is_active for u in db.scalars(select(models.User))},
|
||||||
|
db.query(models.AuditLog).count())
|
||||||
|
|
||||||
|
|
||||||
|
# ── 1. interval ───────────────────────────────────────────────────────────────
|
||||||
|
print("\n1. interval")
|
||||||
|
def iv(v):
|
||||||
|
if v is None:
|
||||||
|
os.environ.pop("OKTA_SYNC_INTERVAL_SECONDS", None)
|
||||||
|
else:
|
||||||
|
os.environ["OKTA_SYNC_INTERVAL_SECONDS"] = v
|
||||||
|
return okta_sync.interval_seconds()
|
||||||
|
|
||||||
|
chk("default is 900 seconds (15 minutes)", iv(None) == 900 and iv("") == 900)
|
||||||
|
chk("an env value is used", iv("600") == 600 and iv("3600") == 3600)
|
||||||
|
chk("not a number: the default", iv("soon") == 900 and iv("1.5") == 900)
|
||||||
|
chk("under the minimum is raised to 300 (a slow cycle can never overlap the next)",
|
||||||
|
iv("60") == 300 and iv("0") == 300 and iv("-5") == 300)
|
||||||
|
chk("a fetch is bounded below the minimum interval",
|
||||||
|
okta_sync.TOTAL_DEADLINE_SECONDS < okta_sync.INTERVAL_MIN_SECONDS)
|
||||||
|
iv(str(INTERVAL))
|
||||||
|
|
||||||
|
# ── 2. the claim ──────────────────────────────────────────────────────────────
|
||||||
|
print("\n2. the claim gate")
|
||||||
|
wipe()
|
||||||
|
with Session() as db:
|
||||||
|
chk("first ever claim wins (the insert is the claim)", okta_sync.claim(db, INTERVAL, now=T0) is True)
|
||||||
|
with Session() as db:
|
||||||
|
chk("a second claim in the same instant loses", okta_sync.claim(db, INTERVAL, now=T0) is False)
|
||||||
|
with Session() as db:
|
||||||
|
chk("one second before the interval is up: loses",
|
||||||
|
okta_sync.claim(db, INTERVAL, now=T0 + timedelta(seconds=INTERVAL - 1)) is False)
|
||||||
|
with Session() as db:
|
||||||
|
chk("when the interval is up: wins",
|
||||||
|
okta_sync.claim(db, INTERVAL, now=T0 + timedelta(seconds=INTERVAL)) is True)
|
||||||
|
with Session() as db:
|
||||||
|
chk("and then the next caller loses again",
|
||||||
|
okta_sync.claim(db, INTERVAL, now=T0 + timedelta(seconds=INTERVAL + 1)) is False)
|
||||||
|
|
||||||
|
# ── 3. the race ───────────────────────────────────────────────────────────────
|
||||||
|
print("\n3. exactly one of many workers wins (a real race on a file database)")
|
||||||
|
|
||||||
|
|
||||||
|
def race(n=8):
|
||||||
|
barrier = threading.Barrier(n)
|
||||||
|
wins, errors = [], []
|
||||||
|
|
||||||
|
def worker():
|
||||||
|
try:
|
||||||
|
with Session() as db:
|
||||||
|
barrier.wait()
|
||||||
|
wins.append(okta_sync.claim(db, INTERVAL, now=datetime.now(timezone.utc)))
|
||||||
|
except Exception as exc: # a lost race may surface as an error
|
||||||
|
errors.append(type(exc).__name__)
|
||||||
|
wins.append(False)
|
||||||
|
|
||||||
|
ts = [threading.Thread(target=worker) for _ in range(n)]
|
||||||
|
[t.start() for t in ts]
|
||||||
|
[t.join() for t in ts]
|
||||||
|
return sum(1 for w in wins if w), errors
|
||||||
|
|
||||||
|
|
||||||
|
wipe()
|
||||||
|
won, errs = race()
|
||||||
|
chk("no row yet, 8 workers at once: exactly 1 wins (the insert race)", won == 1, f"{won} {errs}")
|
||||||
|
bad = []
|
||||||
|
for i in range(15):
|
||||||
|
with Session() as db: # age the claim so the next race is due again
|
||||||
|
db.query(models.AppSetting).filter(models.AppSetting.key == okta_sync.CLAIM_KEY).update(
|
||||||
|
{"updated_at": datetime.now(timezone.utc) - timedelta(seconds=INTERVAL + 5)})
|
||||||
|
db.commit()
|
||||||
|
won, errs = race()
|
||||||
|
if won != 1:
|
||||||
|
bad.append((i, won, errs))
|
||||||
|
chk("row due, 8 workers at once, 15 rounds: exactly 1 wins every round", not bad, str(bad))
|
||||||
|
|
||||||
|
# ── 4. one tick ───────────────────────────────────────────────────────────────
|
||||||
|
print("\n4. a tick runs the sync once per cycle")
|
||||||
|
wipe([("alice", True), ("bob", True)])
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_ok():
|
||||||
|
calls.append(1)
|
||||||
|
return [OktaUser("alice", "ACTIVE"), OktaUser("bob", "SUSPENDED")]
|
||||||
|
|
||||||
|
|
||||||
|
r1 = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0)
|
||||||
|
chk("first tick claims and runs", r1 is not None and r1.ok and r1.disabled == ["bob"])
|
||||||
|
r2 = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0 + timedelta(seconds=30))
|
||||||
|
chk("a tick 30 seconds later does nothing (not due)", r2 is None and len(calls) == 1)
|
||||||
|
r3 = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0 + timedelta(seconds=INTERVAL))
|
||||||
|
chk("a tick when due runs again", r3 is not None and r3.ok and len(calls) == 2)
|
||||||
|
|
||||||
|
print("\n ...a restart inside the interval does not start a second run")
|
||||||
|
wipe([("alice", True), ("bob", True)])
|
||||||
|
calls.clear()
|
||||||
|
okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0)
|
||||||
|
# "restart": a brand-new caller with nothing in memory, 2 minutes later
|
||||||
|
r = okta_sync.run_if_due(Session, fetch=fetch_ok, now=T0 + timedelta(seconds=120))
|
||||||
|
chk("the claim row outlives the process: no duplicate run", r is None and len(calls) == 1)
|
||||||
|
|
||||||
|
print("\n ...two workers tick in the same instant")
|
||||||
|
wipe([("alice", True), ("bob", True)])
|
||||||
|
calls.clear()
|
||||||
|
results = []
|
||||||
|
barrier = threading.Barrier(2)
|
||||||
|
|
||||||
|
|
||||||
|
def tick():
|
||||||
|
barrier.wait()
|
||||||
|
results.append(okta_sync.run_if_due(Session, fetch=fetch_ok))
|
||||||
|
|
||||||
|
|
||||||
|
ts = [threading.Thread(target=tick) for _ in range(2)]
|
||||||
|
[t.start() for t in ts]
|
||||||
|
[t.join() for t in ts]
|
||||||
|
chk("Okta is asked once, not twice", len(calls) == 1, str(len(calls)))
|
||||||
|
chk("exactly one worker got a result", sum(1 for r in results if r is not None) == 1)
|
||||||
|
_, audits = users_state()
|
||||||
|
chk("one audit row, not two", audits == 1, str(audits))
|
||||||
|
|
||||||
|
# ── 5. fail closed through the scheduled path ─────────────────────────────────
|
||||||
|
print("\n5. FAIL CLOSED through the scheduled wrapper (not just run_sync)")
|
||||||
|
for label, exc in (("OktaSyncError", OktaSyncError("HTTP 500")), ("RuntimeError", RuntimeError("x")),
|
||||||
|
("TimeoutError", TimeoutError("t"))):
|
||||||
|
wipe([("alice", True), ("bob", True)])
|
||||||
|
|
||||||
|
def failing(e=exc):
|
||||||
|
raise e
|
||||||
|
|
||||||
|
r = okta_sync.run_if_due(Session, fetch=failing, now=T0)
|
||||||
|
u, a = users_state()
|
||||||
|
chk(f"{label} in the scheduled path: ok is False, zero changes, zero audit rows",
|
||||||
|
r is not None and r.ok is False and all(u.values()) and a == 0)
|
||||||
|
|
||||||
|
wipe([("alice", True), ("bob", True)])
|
||||||
|
|
||||||
|
|
||||||
|
def broken_factory():
|
||||||
|
raise RuntimeError("database is down")
|
||||||
|
|
||||||
|
|
||||||
|
chk("the database failing at claim time: the tick ends quietly, returns None",
|
||||||
|
okta_sync.run_if_due(broken_factory, fetch=fetch_ok, now=T0) is None)
|
||||||
|
u, a = users_state()
|
||||||
|
chk("...and nothing changed", all(u.values()) and a == 0)
|
||||||
|
|
||||||
|
# ── 6. the thread ─────────────────────────────────────────────────────────────
|
||||||
|
print("\n6. the timer thread")
|
||||||
|
os.environ["OKTA_API_TOKEN"] = ""
|
||||||
|
okta_auth.ISSUER = "https://example.okta.com/oauth2/default"
|
||||||
|
chk("no token: nothing starts", okta_sync.start_scheduler(Session) is None)
|
||||||
|
chk("...no thread named okta-sync", not any(t.name == "okta-sync" for t in threading.enumerate()))
|
||||||
|
|
||||||
|
os.environ["OKTA_API_TOKEN"] = "tok"
|
||||||
|
wipe([("alice", True), ("bob", True)])
|
||||||
|
okta_sync.START_DELAY_SECONDS = 0.05
|
||||||
|
okta_sync.TICK_SECONDS = 0.05
|
||||||
|
ran = []
|
||||||
|
real_fetch = okta_sync.fetch_users
|
||||||
|
okta_sync.fetch_users = lambda http=None: (ran.append(1) or
|
||||||
|
[OktaUser("alice", "ACTIVE"), OktaUser("bob", "DEPROVISIONED")])
|
||||||
|
stop = okta_sync.start_scheduler(Session)
|
||||||
|
chk("configured: a stop Event comes back", stop is not None)
|
||||||
|
time.sleep(1.2) # about 20 ticks
|
||||||
|
u, a = users_state()
|
||||||
|
chk("the thread ran the sync on its own (bob was disabled without a manual call)", u["bob"] is False)
|
||||||
|
chk("many ticks, one run (the claim gate holds)", len(ran) == 1 and a == 1, f"ran={len(ran)} audits={a}")
|
||||||
|
th = [t for t in threading.enumerate() if t.name == "okta-sync"]
|
||||||
|
chk("one timer thread is alive", len(th) == 1 and th[0].is_alive())
|
||||||
|
chk("the thread is a daemon (never blocks shutdown)", th[0].daemon)
|
||||||
|
stop.set()
|
||||||
|
th[0].join(timeout=3)
|
||||||
|
chk("setting the stop Event ends the thread", not th[0].is_alive())
|
||||||
|
okta_sync.fetch_users = real_fetch
|
||||||
|
os.environ["OKTA_API_TOKEN"] = ""
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("ALL PASSED" if not fails else f"{fails} FAILED")
|
||||||
|
sys.exit(0 if not fails else 1)
|
||||||
Reference in New Issue
Block a user