Project-SDE-WP-Suite/DEPLOYMENT.md

Deployment

Audience: the IT admin standing this up inside the firewall. This covers the SQL-backed deployment — NGINX serving the static front end and a Python API backed by PostgreSQL.

The repo already contains everything needed to run it as a Docker stack: Dockerfile, docker-compose.yml, the nginx/ config, the front end in html/, and the API in server/. The detailed container reference (endpoints, password rotation, day-to-day commands) lives in server/README.md — this doc is the start-to-finish guide.

                         [ your TLS reverse proxy / traefik ]   ← HTTPS terminates here
                                        │  (external "proxy" network)
                                   ┌────▼────┐   internal network   ┌──────────┐   ┌────────────┐
   browser ───────────────────────│  nginx  │ ───── /api/ ───────> │   api    │ → │  postgres  │
                                   │ (html/) │                      │ FastAPI  │   │   (db)     │
                                   └─────────┘                      └──────────┘   └────────────┘

Everything runs inside your firewall; the app makes no outbound internet calls (logo and scripts are local).

Architecture note: all static files live under html/ and are baked into the nginx image at build time (not bind-mounted). So after any front-end change you rebuild the webserver image (see Updating below). The API image is built from the root Dockerfile.

---

1. Prerequisites

• A Linux host with Docker and Docker Compose v2 (docker compose …).
• An external Docker network named proxy that your TLS-terminating reverse proxy also sits on (the compose file marks it external: true):

  docker network create proxy
  

  If you don't run a separate reverse proxy, you can instead publish the nginx container's port 80 directly (see the note in step 4) and terminate TLS there.
• The repository checked out on the host.

2. Create the database credentials (.env)

Create a file named .env in the project root (same folder as docker-compose.yml). It is git-ignored and must never be committed.

# .env  — project root
POSTGRES_DB=wpsuite
POSTGRES_USER=wpsuite
POSTGRES_PASSWORD=<strong-random-password>

# REQUIRED — signs login session cookies. If unset, `docker compose up` errors
# out and the API refuses to start. Generate once and keep it stable:
#   openssl rand -base64 48
AUTH_SECRET_KEY=<strong-random-secret>

# Encrypts database backups at rest (AES-256). Set this BEFORE the DB holds
# customer IP. Keep the passphrase OFF this host — losing it makes dumps
# unrecoverable:  openssl rand -base64 32
BACKUP_ENC_PASSPHRASE=<strong-random-passphrase>

# OPTIONAL — SMTP password for WP-assignment email + password-reset links. Email
# is OFF by default and enabled from the Admin console; the host/port/from-address
# are configured there, but the password is only ever read from this variable
# (never stored in the DB or shown in the UI). Leave unset until you have SMTP
# details.
# SMTP_PASSWORD=<smtp-app-password>

# OPTIONAL — password-reset link lifetime (minutes) and the per-account send
# cooldown (seconds). Defaults shown; both only matter once email is enabled.
# AUTH_RESET_MINUTES=60
# AUTH_RESET_COOLDOWN_SECONDS=120

The API builds its own DB connection string from the POSTGRES_* values and encodes the password automatically, so a password with special characters (@ ! # : / …) works without any manual escaping. DATABASE_URL is optional and only needed if you want to point the API at some other database; if you do set it, you must URL-encode the password yourself, and it's ignored whenever the three POSTGRES_* values are present.

Generate a strong password with openssl rand -base64 32.

Portainer note: for a Git-based stack these go in the stack's Environment variables section (Portainer doesn't read a local .env). Set POSTGRES_DB / POSTGRES_USER / POSTGRES_PASSWORD / AUTH_SECRET_KEY / BACKUP_ENC_PASSPHRASE (and SMTP_PASSWORD, if you enable email) there.

These are the only credentials in the system, and they never appear in the compose file or in git.

3. Point your reverse proxy at the nginx container

The nginx container listens on port 80 on the proxy network and expects TLS to be terminated upstream (by your reverse proxy / traefik). Route your chosen hostname (e.g. wp-suite.company.local) to the nginx_webserver container on that network. The container already proxies /api/ to the api service internally — no extra app config needed.

Serve it over HTTPS, and forward the scheme. The bundled nginx sets the security response headers (CSP, HSTS, X-Frame-Options, nosniff) and passes X-Forwarded-Proto: https to the API, which is what makes the session cookie Secure. If you front the stack with your own proxy instead, make sure it terminates TLS and forwards X-Forwarded-Proto: https — otherwise the login cookie won't get the Secure flag. HSTS also assumes the site is only ever reached over HTTPS.

4. Bring it up

From the project root:

docker compose up -d --build      # builds the api + nginx images, starts all three containers
docker compose ps                 # confirm nginx_webserver, wp_api, wp_db are running/healthy
docker compose logs -f api        # watch the API start (Ctrl-C to stop following)

The database schema is created automatically on first API start — no manual CREATE TABLE. The Postgres data lives in the named volume pgdata and survives docker compose down (only down -v deletes it).

No separate reverse proxy? Publish nginx directly by adding a ports: mapping to the webserver service (e.g. "8080:80") and terminate TLS at whatever sits in front of it. The internal api/db containers should never be published.

5. Verify

# API liveness (from the host, through the proxy hostname)
curl https://wp-suite.company.local/api/health        # → {"ok": true}

# Interactive API docs
#   https://wp-suite.company.local/api/docs

Then load the site in a browser: the home page should prompt to select or create a project. Create one, complete an SOP, and confirm a row appears:

docker compose exec db psql -U wpsuite -d wpsuite -c "select id, name from projects;"

Automated smoke test

server/smoketest.py exercises the whole stack end-to-end (health → project → SOP → Work Package → the AWP issue gate → status → metrics → comments → cascade cleanup). Stdlib only — no pip/jq.

# Through the proxy (use --insecure for a self-signed internal cert):
python3 server/smoketest.py https://wp-suite.company.local --insecure

# Or from inside the api container (hits FastAPI directly):
docker compose exec api python /app/server/smoketest.py http://localhost:8000

# Add --keep to leave a demo project in the DB so you can open it in the UI.

Exit code 0 and "ALL PASS" means the API, the Python logic, and SQL are all working. It cleans up after itself (the test project and its SOP/WPs are deleted via cascade); a single tagged test comment remains (there's no comment delete endpoint).

Loadable demo project

server/seed_demo.py populates a realistic DEMO project (a complete SOP plus a spread of Work Packages: issued, gated, a multi-discipline master with split instances, an overdue one, an over-threshold draft) so there's data to look at.

python3 server/seed_demo.py https://wp-suite.company.local --insecure
python3 server/seed_demo.py https://wp-suite.company.local --clean   # remove it later

What shows where: the DEMO project, its SOP, and its Work Packages are all API/SQL-backed, so they appear in the home-page project picker and render in the Creator/Dashboard as soon as any user opens the project. Inspect them at the SQL layer with smoketest.py or:

docker compose exec db psql -U wpsuite -d wpsuite \
  -c "select number, subject, status from work_packages order by number;"

---

What is stored in SQL today

The API + Postgres are the system of record. Everything below is server-stored and shared across every user who opens the project:

Data	Stored in PostgreSQL today?
Projects	Yes — the front end is API-first (/api/projects), falling back to the browser only if the API is unreachable.
Comments / feedback	Yes — every feedback surface posts to /api/feedback.
SOPs	Yes — pulled from /api/sops on load and written through on every save.
Work Packages	Yes — same write-through to /api/wps (+ issue / status / archive / metrics), including the owner assignment (assignee_id).

Saves go through a durable client-side sync outbox: edits are written to the API immediately, and if the device is offline they queue and retry when it reconnects (4xx rejections are dropped rather than retried forever). The browser cache is only an offline fallback that reconciles through that outbox — so two users on the same project see the same server-stored SOP and Work Packages.

Data model (PostgreSQL)

Table	Holds	Key columns
projects	top-level construction projects	name, number, client, division, site, sample, archived_at, data
sops	project SOP baselines	project_id → projects, name, number, complete, data (full SOP JSON)
work_packages	individual IWPs	project_id → projects, sop_id → sops, parent_id (split instances), number, subject, type, status, assignee_id (owner), issued_at, archived_at, data (full WP JSON)
comments	feedback from any page	source, sop_id, wp_id, step, author, text, extra
users	login accounts	username, password_hash (bcrypt), role, full_name, email, is_active, auto_add_projects + auto_add_role (default membership on new projects), login-lockout + token_version fields
project_members	per-project access control	user_id → users, project_id → projects
audit_log	append-only activity trail	actor, action, entity_type, entity_id, project_id, summary, detail
notifications	in-app record + email outbox	user_id, kind, wp_id, subject, status (pending / sent / failed / skipped)
app_settings	admin-configured settings (e.g. email)	key, value (JSON)

The complete client document is stored verbatim in each row's data JSON column; frequently-listed fields are promoted to real columns for filtering.

Endpoints (summary)

Projects GET/POST /api/projects, GET/DELETE /api/projects/{id}, POST /api/projects/{id}/archive · SOPs GET/POST /api/sops, GET /api/sops/latest, GET/DELETE /api/sops/{id} · Work Packages GET/POST /api/wps, GET/DELETE /api/wps/{id}, POST /api/wps/{id}/issue, POST /api/wps/{id}/status, POST /api/wps/{id}/archive, GET /api/wps/metrics · Comments POST /api/comments (and /api/feedback), GET /api/comments · Auth POST /api/auth/login / logout, GET /api/auth/me, admin user management under /api/auth/users (including POST /api/auth/users/{id}/auto-add) · Admin-only GET/PUT /api/settings, POST /api/settings/test-email, GET /api/notifications, GET /api/projects/{id}/members. List/latest/metrics accept a project_id (and sop_id) filter. GET /api/projects and GET /api/wps both take archived=exclude|only|all and default to exclude — anything that needs to see archived rows (the admin console, the demo cleanup) must ask for them. Full reference and request shapes: /api/docs and server/README.md.

---

Updating after a change

git pull
docker compose up -d --build webserver   # front-end change (html/) — rebuild the baked image
docker compose up -d --build api         # backend change (server/)

Backups & retention

A backup sidecar (in docker-compose.yml) runs pg_dump on a schedule and writes gzipped, timestamped dumps to ./backups/ on the host. It starts with the stack — no cron to set up.

• Cadence / retention: daily, keeping the newest 14 dumps. Override in .env with BACKUP_INTERVAL_SECONDS (seconds between dumps) and BACKUP_KEEP (how many to keep).
• Encryption at rest: set BACKUP_ENC_PASSPHRASE in .env and dumps are written AES-256-encrypted as *.sql.gz.enc. Do this before any customer IP goes in — without it the dumps (and every offsite copy) are plaintext. Store the passphrase somewhere other than this host; if you lose it the backups can't be restored.
• Ad-hoc backup now: docker compose exec backup sh /scripts/db-backup.sh
• Restore (destructive — overwrites current data): docker compose exec backup sh /scripts/db-restore.sh /backups/wpsuite-YYYYMMDD-HHMMSSZ.sql.gz.enc
• Offsite — do this: the dumps live in ./backups/ on the host; if the host/volume dies, so do they. Sync that folder offsite from the host (e.g. a cron running rclone/aws s3 sync). The db/backup containers are on an egress-less internal network on purpose, so offsite must be pushed from the host.
• Test restores quarterly: load the latest dump into a throwaway database and confirm it applies. An untested backup is not a backup.

Field devices & data at rest

The field view (PWA) caches a project's Work Packages/SOP in the browser's localStorage so it works offline — i.e. customer IP sits on the device. localStorage is not encrypted and is not a security boundary. Signing out clears the cached project data, but for any tablet/phone that opens customer-IP projects:

• Require full-disk encryption (BitLocker / FileVault / Android FBE / iOS is encrypted by default) and a device passcode.
• Enrol field devices in MDM so a lost device can be remotely wiped, and keep the browser profile per-user on shared devices.
• Users should sign out when handing off a shared device (clears the cache).

Email notifications (optional)

Work-package owner assignment works out of the box (in-app only). Optional email on assignment is OFF by default and is turned on from the Admin console → Notifications & email card, where an admin sets the SMTP host / port / TLS / From address and flips the master toggle.

• The SMTP password is never stored in the database. It is read only from the SMTP_PASSWORD environment variable (see the .env block in step 2 and the api service in docker-compose.yml). The UI shows only whether it is set.
• Email stays effectively off until all of: the toggle is on, SMTP host + From are configured, and SMTP_PASSWORD is present. Until then, assignments are still recorded in-app (status skipped); nothing is sent.
• Notification emails carry only a WP number and a deep link — never the work package contents — so customer IP stays behind the login.
• Use the card's Send test email button to confirm SMTP before enabling.

Self-service password reset

Turning email on also enables Forgot password on the login page. Until then the link explains that an admin must reset it (server/manage_users.py, or the Admin console's Reset password button).

• The emailed link carries a short-lived signed token — AUTH_RESET_MINUTES (default 60). It is single-use: completing a reset bumps the account's token_version, which both burns the link and signs out that user's other sessions. A completed reset also clears any login lockout.
• /api/auth/forgot-password answers identically for unknown accounts, so it can't be used to discover usernames. Misses are recorded in the audit log (password_reset_miss) instead.
• One reset mail per account+client per AUTH_RESET_COOLDOWN_SECONDS (default 120) so the form can't be used to flood someone's inbox. The throttle is per worker and in-memory; the token expiry is the real control.
• Reset mails are sent immediately, not through the notifications outbox — a reset link must never be persisted where an admin could read it and take over an account.
• Set app_base_url in the admin card, or the emailed link will be relative and therefore useless.

Permissions roles

User.role is the permissions role; User.project_role is the person's job function on the project (Project Manager, Superintendent, …) and grants nothing. Both are set in the Admin console's user table.

Role	May do
admin	User administration, app settings, and every project
project_admin	On assigned projects: delete work packages, change a completed SOP, delete the project
project_user	Create/edit work packages, author a SOP up to completion; may archive a WP but not delete one

Enforced server-side by require_project_admin in server/app.py; the front end only hides controls to avoid dead-end clicks. Accounts created before this change carried the role user, which the migration rewrites to project_user.

Feature flags

Admin console → Features. bim_enabled is OFF by default: the SOP creator hides the BIM/VDC section and every project is install-only (IWP). A SOP that already has BIM enabled keeps its data — it just stops being offered — so turning the flag off never deletes BIM types, gates, or sequence steps.

Release gates (constraints + predecessors)

A work package reaches Issued only when both gates are met:

1. every constraint is Cleared or N/A — a hard gate, no override;
2. every predecessor work package (data.predecessors, a list of WP ids) is Closed.

Enforced by enforce_release_gates() on every path that can set a status — /api/wps (the browser and the offline outbox both save through it), /api/wps/{id}/issue, and /api/wps/{id}/status. Also:

• Overridable, deliberately. Planners legitimately release ahead of upstream close-out, so the predecessor gate accepts data.gateOverride = {reason, by, at}. A blank reason is not an override. The server writes a gate_overridden audit event naming the reason and what was skipped, and the reason prints on the package. Changing the predecessor set clears the override.
• Cycles are refused (check_predecessor_cycle) — direct and through a chain, with a 400 explaining which package already waits on this one.
• A deleted predecessor does not block. It would otherwise freeze everything downstream of a package someone removed.
• The Creator's picker hides itself and any package that already waits on it, so a cycle is hard to build in the first place; the dashboard refuses to issue a blocked package and points at the form for the logged override.

data.seq (the SOP sequence phase) is still stored and shown, but it is descriptive — it gates nothing.

Critical constraints reopened after release

A constraint marked Critical on the SOP that reopens after the package was released emails the owner, PM, CM and everyone on the package's distribution list (minus whoever reopened it), and writes a constraint_reopened audit event.

Detected by comparing incoming constraints against the stored ones inside the normal upsert — not a separate endpoint, because the browser saves through the sync outbox, which only replays POST /api/wps; anything hung off another route would be lost offline. It fires only on a real transition (cleared/N-A → open), so re-saving an already-open constraint doesn't re-announce, and never for a package that was never released or a non-critical constraint. Bodies carry the constraint name, WP number and a link — never the package contents.

Localization (dates, times, numbers)

Three levels, most specific first — resolved in html/wp-format.js:

1. the user's own preference — Language & time in the top-right menu (users.locale / users.timezone, via POST /api/auth/preferences)
2. the app default — Admin console → Features → Localization defaults (default_locale / default_timezone)
3. the browser, as before

Timezone names are validated against the server's own zoneinfo database, and the picker is fed from GET /api/timezones so it can only offer what will be accepted. Calendar dates (a due date, a kitting date) are formatted from their parts and are never shifted by a timezone — only real instants (MIMO windows, history, notifications) are converted. Use the shared helpers (wpFormatDate, wpFormatDateTime, wpFormatTime, wpFormatNumber) rather than toLocaleString(), or a page will quietly ignore the preference.

Top-bar chrome (project switcher + search)

html/wp-chrome.js + wp-chrome.css inject a project switcher and a centered global search into whichever top bar a page has — the dark .wp-appbar or the older .header. It is skipped inside an iframe, so the embedded WP creator does not get a second bar.

• Switching project reloads the current page with ?project=<id>; every page already resolves its project from that parameter.
• Search calls GET /api/search?q=, which is scoped to the caller's projects (scope_to_access) and hides archived work packages, archived projects, and anything belonging to an archived project. LIKE wildcards in the query are escaped, so searching 100% matches a literal 100%. Two-character minimum.
• Ctrl/Cmd-K focuses the field from anywhere.

Schema migrations (Alembic)

Schema is managed by Alembic (server/alembic/). The API container runs alembic upgrade head on startup (see the Dockerfile CMD), so deploys apply pending migrations automatically.

• The baseline migration is idempotent: on a fresh database it creates every table; on a database whose tables already exist (made by the old create_all) it adopts the schema as-is — no manual alembic stamp needed.
• Local dev on SQLite still auto-creates tables for a zero-config run; Postgres is migrations-only.
• To change the schema: edit server/models.py, then generate and review a migration before committing:

  # from the project root (against your dev SQLite or a staging DB)
  python -m alembic -c server/alembic.ini revision --autogenerate -m "describe the change"
  python -m alembic -c server/alembic.ini upgrade head     # apply locally to test
  

  The next docker compose up -d --build api applies it in production on startup.

Local trial without Postgres

For a quick local look, the API falls back to a SQLite file when DATABASE_URL is unset (sqlite:///./wpsuite.db) — see server/README.md § Local dev. The front end alone can also be served statically from html/ (it falls back to browser storage when the API isn't reachable).

Per-project permissions

users.role is the account's default permissions role. A membership row can override it per project (project_members.role), so someone can be Project Admin on one job and a plain Project User on another. Empty means "inherit the account's role", which is how every pre-existing membership behaves.

Resolved by effective_role() in server/app.py; require_project_admin() uses it, so deleting a work package, changing a completed SOP and deleting a project are all judged on that project. An app admin is admin everywhere and bypasses membership entirely.

Set it in Admin console → User administration → Project access (its own column, showing how many projects each account can reach). The dialog ticks project access and picks the role on each; /api/auth/users/{id}/projects takes {project_ids: [...], roles: {project_id: role}} and only accepts the two project-scoped roles. Changes are audit-logged as project_access_changed.

Who appears in the SOP's people pickers is GET /api/projects/{id}/members — the project's members plus app admins, each with their effective role on that project. A project with nobody assigned shows only the admins, which is why assigning people is the first step on a new job.

Default members on new projects

Memberships are also created automatically. Admin console → Default members on new projects flags accounts (users.auto_add_projects) that belong on every job — the PM who runs them all, the QC lead — with the role they should hold there (users.auto_add_role, sharing project_members.role's value space, '' = inherit the account's own).

• It applies only to projects created after the flag is set. Nothing is back-filled onto existing jobs; use Project access for those.
• App admins are skipped (they already reach every project) and the flag is cleared if an account is promoted to admin. Inactive accounts are skipped.
• Runs in add_default_members() on the is_new branch of upsert_project, so it covers every route into project creation — the home page, the sample project, the demo seeder. An update never re-runs it.
• If the creator is themselves a flagged member, the membership created for them as creator carries their auto_add_role, so they aren't silently downgraded on the one job they started.
• Audit-logged once per project as project_access_granted with detail.reason = "auto_add_projects".

Archiving a project

A finished job is archived rather than deleted: projects.archived_at, set from Admin console → Projects (or POST /api/projects/{id}/archive, which needs Project Admin on that project, same bar as deleting it).

An archived project is hidden and frozen:

• It leaves the home picker, the app-bar switcher and global search, because GET /api/projects defaults to archived=exclude.
• It is still readable by id, so a deep link renders it — with a read-only banner from wp-chrome.js — and the admin console still lists it under ?archived=all.
• Every write that lands on it is refused with 409 by require_project_writable(): saving a project, SOP or work package, deleting either, issuing, status changes, WP archiving, and comments on its WPs/SOPs. Moving a work package into or out of an archived project is refused too. 409 rather than 403 is deliberate — nobody lacks a permission, the project's state is the objection, and the browser outbox (html/project-data.js) retires 4xx ops instead of retrying them forever.
• Unarchiving and deleting stay allowed: unarchive is the one write an archived project must accept, and archive-then-delete is a normal sequence.

Nothing is removed, and unarchiving restores all of it. server/smoketest.py asserts the whole round trip.

Asset freshness (why the app can't run half-updated)

A page must never run against a stylesheet or script from a previous deploy. Three things enforce that, and all three are needed:

1. Cache-Control: no-cache on HTML/CSS/JS — set by NGINX (nginx/conf.d/wp-suite.conf) and by the dev server (_NoCacheCode in server/app.py). With no header at all the browser applies heuristic freshness, roughly 10% of each file's age, so the least recently changed file gets the longest lifetime — which is exactly how HTML and CSS drift apart. ETag/Last-Modified still make each revalidation a cheap 304.
2. The service worker fetches code with cache: 'no-cache' (html/sw.js) and precaches with cache: 'reload'. A plain fetch(req) inherits the request's default cache mode and consults the browser HTTP cache, so "network-first" alone was not enough. Non-ok responses fall back to the cache rather than replacing a page the cache could still serve, and cache keys drop the query string so in-app links (?project=…&tab=…) still resolve offline.
3. Components whose CSS-missing state is broken carry their own critical layout. The embedded creator's iframe keeps its sizing inline (and sizeWPFrame() re-applies it), and the work-package panel injects a floor of positioning rules from wp-creation-app.js. Both had failure modes — a 300×150 iframe, and panel controls dumped loose into the form — that a missing rule turned into a broken page rather than a plain one.

If you change the shell file list in sw.js, bump CACHE.

NGINX note: the Cache-Control value comes from a map $uri $wp_cache_control at http level, applied with a single server-level add_header. Do not move it into a location block: nginx does not inherit add_header into a block that declares its own, so a location ~* \.(html|css|js)$ setting only Cache-Control silently drops the CSP / HSTS / X-Frame-Options / nosniff headers for exactly those files. After deploying, confirm both are present on one response:

curl -sI https://wp-suite.company.local/work-package-suite.html >   | grep -Ei 'cache-control|content-security-policy'