Project-SDE-WP-Suite/DEPLOYMENT.md

# Deployment

Audience: the IT admin standing this up inside the firewall. This covers the
**SQL-backed deployment** — NGINX serving the static front end and a Python API
backed by **PostgreSQL**.

The repo already contains everything needed to run it as a Docker stack:
`Dockerfile`, `docker-compose.yml`, the `nginx/` config, the front end in
`html/`, and the API in `server/`. The detailed container reference (endpoints,
password rotation, day-to-day commands) lives in
[`server/README.md`](server/README.md) — this doc is the start-to-finish guide.

```
                         [ your TLS reverse proxy / traefik ]   ← HTTPS terminates here
                                        │  (external "proxy" network)
                                   ┌────▼────┐   internal network   ┌──────────┐   ┌────────────┐
   browser ───────────────────────│  nginx  │ ───── /api/ ───────> │   api    │ → │  postgres  │
                                   │ (html/) │                      │ FastAPI  │   │   (db)     │
                                   └─────────┘                      └──────────┘   └────────────┘
```

Everything runs inside your firewall; the app makes **no outbound internet
calls** (logo and scripts are local).

> **Architecture note:** all static files live under **`html/`** and are *baked
> into the nginx image* at build time (not bind-mounted). So after any front-end
> change you rebuild the `webserver` image (see *Updating* below). The API image
> is built from the root `Dockerfile`.

---

## 1. Prerequisites

- A Linux host with **Docker** and **Docker Compose v2** (`docker compose …`).
- An external Docker network named `proxy` that your TLS-terminating reverse
  proxy also sits on (the compose file marks it `external: true`):
  ```bash
  docker network create proxy
  ```
  If you don't run a separate reverse proxy, you can instead publish the nginx
  container's port 80 directly (see the note in step 4) and terminate TLS there.
- The repository checked out on the host.

## 2. Create the database credentials (`.env`)

Create a file named `.env` in the **project root** (same folder as
`docker-compose.yml`). It is git-ignored and must never be committed.

```bash
# .env  — project root
POSTGRES_DB=wpsuite
POSTGRES_USER=wpsuite
POSTGRES_PASSWORD=<strong-random-password>

# REQUIRED — signs login session cookies. If unset, `docker compose up` errors
# out and the API refuses to start. Generate once and keep it stable:
#   openssl rand -base64 48
AUTH_SECRET_KEY=<strong-random-secret>

# Encrypts database backups at rest (AES-256). Set this BEFORE the DB holds
# customer IP. Keep the passphrase OFF this host — losing it makes dumps
# unrecoverable:  openssl rand -base64 32
BACKUP_ENC_PASSPHRASE=<strong-random-passphrase>

# OPTIONAL — SMTP password for WP-assignment email notifications. Email is OFF
# by default and enabled from the Admin console; the host/port/from-address are
# configured there, but the password is only ever read from this variable (never
# stored in the DB or shown in the UI). Leave unset until you have SMTP details.
# SMTP_PASSWORD=<smtp-app-password>
```

The API builds its own DB connection string from the `POSTGRES_*`
values and **encodes the password automatically**, so a password with special
characters (`@ ! # : /` …) works without any manual escaping. `DATABASE_URL`
is **optional** and only needed if you want to point the API at some other
database; if you do set it, you must URL-encode the password yourself, and it's
ignored whenever the three `POSTGRES_*` values are present.

Generate a strong password with `openssl rand -base64 32`.

> **Portainer note:** for a Git-based stack these go in the stack's
> **Environment variables** section (Portainer doesn't read a local `.env`).
> Set `POSTGRES_DB` / `POSTGRES_USER` / `POSTGRES_PASSWORD` / `AUTH_SECRET_KEY` /
> `BACKUP_ENC_PASSPHRASE` (and `SMTP_PASSWORD`, if you enable email) there.

These are the only credentials in the system, and they never appear in the
compose file or in git.

## 3. Point your reverse proxy at the nginx container

The nginx container listens on port **80** on the `proxy` network and expects
TLS to be terminated upstream (by your reverse proxy / traefik). Route your
chosen hostname (e.g. `wp-suite.company.local`) to the `nginx_webserver`
container on that network. The container already proxies `/api/` to the `api`
service internally — no extra app config needed.

> **Serve it over HTTPS, and forward the scheme.** The bundled nginx sets the
> security response headers (CSP, HSTS, `X-Frame-Options`, `nosniff`) and passes
> `X-Forwarded-Proto: https` to the API, which is what makes the session cookie
> `Secure`. If you front the stack with your **own** proxy instead, make sure it
> terminates TLS and forwards `X-Forwarded-Proto: https` — otherwise the login
> cookie won't get the `Secure` flag. HSTS also assumes the site is only ever
> reached over HTTPS.

## 4. Bring it up

From the project root:

```bash
docker compose up -d --build      # builds the api + nginx images, starts all three containers
docker compose ps                 # confirm nginx_webserver, wp_api, wp_db are running/healthy
docker compose logs -f api        # watch the API start (Ctrl-C to stop following)
```

The database schema is **created automatically** on first API start — no manual
`CREATE TABLE`. The Postgres data lives in the named volume `pgdata` and
survives `docker compose down` (only `down -v` deletes it).

> No separate reverse proxy? Publish nginx directly by adding a `ports:` mapping
> to the `webserver` service (e.g. `"8080:80"`) and terminate TLS at whatever
> sits in front of it. The internal `api`/`db` containers should **never** be
> published.

## 5. Verify

```bash
# API liveness (from the host, through the proxy hostname)
curl https://wp-suite.company.local/api/health        # → {"ok": true}

# Interactive API docs
#   https://wp-suite.company.local/api/docs
```

Then load the site in a browser: the home page should prompt to **select or
create a project**. Create one, complete an SOP, and confirm a row appears:

```bash
docker compose exec db psql -U wpsuite -d wpsuite -c "select id, name from projects;"
```

### Automated smoke test

`server/smoketest.py` exercises the whole stack end-to-end (health → project →
SOP → Work Package → the AWP issue gate → status → metrics → comments → cascade
cleanup). Stdlib only — no pip/jq.

```bash
# Through the proxy (use --insecure for a self-signed internal cert):
python3 server/smoketest.py https://wp-suite.company.local --insecure

# Or from inside the api container (hits FastAPI directly):
docker compose exec api python /app/server/smoketest.py http://localhost:8000

# Add --keep to leave a demo project in the DB so you can open it in the UI.
```

Exit code 0 and "ALL PASS" means the API, the Python logic, and SQL are all
working. It cleans up after itself (the test project and its SOP/WPs are
deleted via cascade); a single tagged test comment remains (there's no comment
delete endpoint).

### Loadable demo project

`server/seed_demo.py` populates a realistic **DEMO** project (a complete SOP plus
a spread of Work Packages: issued, gated, a multi-discipline master with split
instances, an overdue one, an over-threshold draft) so there's data to look at.

```bash
python3 server/seed_demo.py https://wp-suite.company.local --insecure
python3 server/seed_demo.py https://wp-suite.company.local --clean   # remove it later
```

> **What shows where:** the DEMO **project**, its **SOP**, and its **Work
> Packages** are all API/SQL-backed, so they appear in the home-page project
> picker and render in the Creator/Dashboard as soon as any user opens the
> project. Inspect them at the SQL layer with `smoketest.py` or:
> ```bash
> docker compose exec db psql -U wpsuite -d wpsuite \
>   -c "select number, subject, status from work_packages order by number;"
> ```

---

## What is stored in SQL today

The API + Postgres are the system of record. Everything below is server-stored
and shared across every user who opens the project:

| Data | Stored in PostgreSQL today? |
|------|------------------------------|
| **Projects** | **Yes** — the front end is API-first (`/api/projects`), falling back to the browser only if the API is unreachable. |
| **Comments / feedback** | **Yes** — every feedback surface posts to `/api/feedback`. |
| **SOPs** | **Yes** — pulled from `/api/sops` on load and written through on every save. |
| **Work Packages** | **Yes** — same write-through to `/api/wps` (+ issue / status / archive / metrics), including the owner assignment (`assignee_id`). |

Saves go through a **durable client-side sync outbox**: edits are written to the
API immediately, and if the device is offline they queue and retry when it
reconnects (4xx rejections are dropped rather than retried forever). The browser
cache is only an offline fallback that reconciles through that outbox — so two
users on the same project see the same server-stored SOP and Work Packages.

## Data model (PostgreSQL)

| Table | Holds | Key columns |
|-------|-------|-------------|
| `projects` | top-level construction projects | `name`, `number`, `client`, `division`, `site`, `sample`, `data` |
| `sops` | project SOP baselines | `project_id` → projects, `name`, `number`, `complete`, `data` (full SOP JSON) |
| `work_packages` | individual IWPs | `project_id` → projects, `sop_id` → sops, `parent_id` (split instances), `number`, `subject`, `type`, `status`, `assignee_id` (owner), `issued_at`, `archived_at`, `data` (full WP JSON) |
| `comments` | feedback from any page | `source`, `sop_id`, `wp_id`, `step`, `author`, `text`, `extra` |
| `users` | login accounts | `username`, `password_hash` (bcrypt), `role`, `full_name`, `email`, `is_active`, login-lockout + `token_version` fields |
| `project_members` | per-project access control | `user_id` → users, `project_id` → projects |
| `audit_log` | append-only activity trail | `actor`, `action`, `entity_type`, `entity_id`, `project_id`, `summary`, `detail` |
| `notifications` | in-app record + email outbox | `user_id`, `kind`, `wp_id`, `subject`, `status` (pending / sent / failed / skipped) |
| `app_settings` | admin-configured settings (e.g. email) | `key`, `value` (JSON) |

The complete client document is stored verbatim in each row's `data` JSON
column; frequently-listed fields are promoted to real columns for filtering.

### Endpoints (summary)

Projects `GET/POST /api/projects`, `GET/DELETE /api/projects/{id}` ·
SOPs `GET/POST /api/sops`, `GET /api/sops/latest`, `GET/DELETE /api/sops/{id}` ·
Work Packages `GET/POST /api/wps`, `GET/DELETE /api/wps/{id}`,
`POST /api/wps/{id}/issue`, `POST /api/wps/{id}/status`, `POST /api/wps/{id}/archive`,
`GET /api/wps/metrics` ·
Comments `POST /api/comments` (and `/api/feedback`), `GET /api/comments` ·
Auth `POST /api/auth/login` / `logout`, `GET /api/auth/me`, admin user management
under `/api/auth/users` · Admin-only `GET/PUT /api/settings`,
`POST /api/settings/test-email`, `GET /api/notifications`,
`GET /api/projects/{id}/members`.
List/latest/metrics accept a `project_id` (and `sop_id`) filter. Full reference
and request shapes: `/api/docs` and [`server/README.md`](server/README.md).

---

## Updating after a change

```bash
git pull
docker compose up -d --build webserver   # front-end change (html/) — rebuild the baked image
docker compose up -d --build api         # backend change (server/)
```

## Backups & retention

A **`backup` sidecar** (in `docker-compose.yml`) runs `pg_dump` on a schedule and
writes gzipped, timestamped dumps to `./backups/` on the host. It starts with the
stack — no cron to set up.

- **Cadence / retention:** daily, keeping the newest 14 dumps. Override in `.env`
  with `BACKUP_INTERVAL_SECONDS` (seconds between dumps) and `BACKUP_KEEP` (how many
  to keep).
- **Encryption at rest:** set `BACKUP_ENC_PASSPHRASE` in `.env` and dumps are
  written AES-256-encrypted as `*.sql.gz.enc`. **Do this before any customer IP
  goes in** — without it the dumps (and every offsite copy) are plaintext. Store
  the passphrase somewhere other than this host; if you lose it the backups can't
  be restored.
- **Ad-hoc backup now:** `docker compose exec backup sh /scripts/db-backup.sh`
- **Restore (destructive — overwrites current data):**
  `docker compose exec backup sh /scripts/db-restore.sh /backups/wpsuite-YYYYMMDD-HHMMSSZ.sql.gz.enc`
- **Offsite — do this:** the dumps live in `./backups/` on the host; if the host/volume
  dies, so do they. Sync that folder offsite from the **host** (e.g. a cron running
  `rclone`/`aws s3 sync`). The `db`/`backup` containers are on an egress-less
  `internal` network on purpose, so offsite must be pushed from the host.
- **Test restores quarterly:** load the latest dump into a throwaway database and
  confirm it applies. An untested backup is not a backup.

## Field devices & data at rest

The field view (PWA) caches a project's Work Packages/SOP in the browser's
localStorage so it works offline — i.e. **customer IP sits on the device**.
localStorage is not encrypted and is not a security boundary. Signing out clears
the cached project data, but for any tablet/phone that opens customer-IP projects:

- **Require full-disk encryption** (BitLocker / FileVault / Android FBE / iOS is
  encrypted by default) and a device passcode.
- **Enrol field devices in MDM** so a lost device can be remotely wiped, and keep
  the browser profile per-user on shared devices.
- Users should **sign out** when handing off a shared device (clears the cache).

## Email notifications (optional)

Work-package **owner assignment** works out of the box (in-app only). Optional
**email** on assignment is **OFF by default** and is turned on from the **Admin
console → Notifications & email** card, where an admin sets the SMTP host / port /
TLS / From address and flips the master toggle.

- The **SMTP password is never stored in the database.** It is read only from the
  `SMTP_PASSWORD` environment variable (see the `.env` block in step 2 and the
  `api` service in `docker-compose.yml`). The UI shows only whether it is set.
- Email stays effectively off until **all** of: the toggle is on, SMTP host + From
  are configured, and `SMTP_PASSWORD` is present. Until then, assignments are
  still recorded in-app (status `skipped`); nothing is sent.
- Notification emails carry only a **WP number and a deep link** — never the work
  package contents — so customer IP stays behind the login.
- Use the card's **Send test email** button to confirm SMTP before enabling.

## Schema migrations (Alembic)

Schema is managed by **Alembic** (`server/alembic/`). The API container runs
`alembic upgrade head` on startup (see the `Dockerfile` CMD), so **deploys apply
pending migrations automatically**.

- The **baseline** migration is idempotent: on a fresh database it creates every
  table; on a database whose tables already exist (made by the old `create_all`)
  it adopts the schema as-is — no manual `alembic stamp` needed.
- Local dev on SQLite still auto-creates tables for a zero-config run; Postgres is
  migrations-only.
- **To change the schema:** edit `server/models.py`, then generate and review a
  migration before committing:
  ```bash
  # from the project root (against your dev SQLite or a staging DB)
  python -m alembic -c server/alembic.ini revision --autogenerate -m "describe the change"
  python -m alembic -c server/alembic.ini upgrade head     # apply locally to test
  ```
  The next `docker compose up -d --build api` applies it in production on startup.

## Local trial without Postgres

For a quick local look, the API falls back to a SQLite file when `DATABASE_URL`
is unset (`sqlite:///./wpsuite.db`) — see [`server/README.md`](server/README.md)
§ *Local dev*. The front end alone can also be served statically from `html/`
(it falls back to browser storage when the API isn't reachable).