diff --git a/.gitignore b/.gitignore index 7ddb774..073ff7e 100644 --- a/.gitignore +++ b/.gitignore @@ -15,5 +15,16 @@ wpsuite.db # Runtime directories (created by containers) logs/ +# Database backup dumps (large + sensitive) — keep the folder, ignore contents +/backups/* +!/backups/.gitkeep + # Local server logs *.log + +# Local scratch / test artifacts (curl cookie jars hold live session tokens) +_*.txt +cookies.txt + +# Claude Code local workspace (agent memory, session data) +.claude/ diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index d071308..fe7f079 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -51,9 +51,25 @@ Create a file named `.env` in the **project root** (same folder as POSTGRES_DB=wpsuite POSTGRES_USER=wpsuite POSTGRES_PASSWORD= + +# REQUIRED — signs login session cookies. If unset, `docker compose up` errors +# out and the API refuses to start. Generate once and keep it stable: +# openssl rand -base64 48 +AUTH_SECRET_KEY= + +# Encrypts database backups at rest (AES-256). Set this BEFORE the DB holds +# customer IP. Keep the passphrase OFF this host — losing it makes dumps +# unrecoverable: openssl rand -base64 32 +BACKUP_ENC_PASSPHRASE= + +# OPTIONAL — SMTP password for WP-assignment email notifications. Email is OFF +# by default and enabled from the Admin console; the host/port/from-address are +# configured there, but the password is only ever read from this variable (never +# stored in the DB or shown in the UI). Leave unset until you have SMTP details. +# SMTP_PASSWORD= ``` -That's it — the API now builds its own connection string from these three +The API builds its own DB connection string from the `POSTGRES_*` values and **encodes the password automatically**, so a password with special characters (`@ ! # : /` …) works without any manual escaping. `DATABASE_URL` is **optional** and only needed if you want to point the API at some other @@ -64,7 +80,8 @@ Generate a strong password with `openssl rand -base64 32`. > **Portainer note:** for a Git-based stack these go in the stack's > **Environment variables** section (Portainer doesn't read a local `.env`). -> Set `POSTGRES_DB` / `POSTGRES_USER` / `POSTGRES_PASSWORD` there. +> Set `POSTGRES_DB` / `POSTGRES_USER` / `POSTGRES_PASSWORD` / `AUTH_SECRET_KEY` / +> `BACKUP_ENC_PASSPHRASE` (and `SMTP_PASSWORD`, if you enable email) there. These are the only credentials in the system, and they never appear in the compose file or in git. @@ -77,6 +94,14 @@ chosen hostname (e.g. `wp-suite.company.local`) to the `nginx_webserver` container on that network. The container already proxies `/api/` to the `api` service internally — no extra app config needed. +> **Serve it over HTTPS, and forward the scheme.** The bundled nginx sets the +> security response headers (CSP, HSTS, `X-Frame-Options`, `nosniff`) and passes +> `X-Forwarded-Proto: https` to the API, which is what makes the session cookie +> `Secure`. If you front the stack with your **own** proxy instead, make sure it +> terminates TLS and forwards `X-Forwarded-Proto: https` — otherwise the login +> cookie won't get the `Secure` flag. HSTS also assumes the site is only ever +> reached over HTTPS. + ## 4. Bring it up From the project root: @@ -145,12 +170,10 @@ python3 server/seed_demo.py https://wp-suite.company.local --insecure python3 server/seed_demo.py https://wp-suite.company.local --clean # remove it later ``` -> **What shows where:** the DEMO **project** is API/SQL-backed, so it appears in -> the home-page project picker right away (this is the visible proof that the -> projects → SQL path works end-to-end). The DEMO **SOP and Work Packages** are -> written to SQL too, but the current front end still reads SOPs/WPs from the -> browser, so they won't render in the Creator/Dashboard until the Phase 2 -> wiring. Inspect them at the SQL layer with `smoketest.py` or: +> **What shows where:** the DEMO **project**, its **SOP**, and its **Work +> Packages** are all API/SQL-backed, so they appear in the home-page project +> picker and render in the Creator/Dashboard as soon as any user opens the +> project. Inspect them at the SQL layer with `smoketest.py` or: > ```bash > docker compose exec db psql -U wpsuite -d wpsuite \ > -c "select number, subject, status from work_packages order by number;" @@ -160,20 +183,21 @@ python3 server/seed_demo.py https://wp-suite.company.local --clean # remove it ## What is stored in SQL today -Be aware of the current persistence split — the API + Postgres are fully -deployed, and: +The API + Postgres are the system of record. Everything below is server-stored +and shared across every user who opens the project: | Data | Stored in PostgreSQL today? | |------|------------------------------| | **Projects** | **Yes** — the front end is API-first (`/api/projects`), falling back to the browser only if the API is unreachable. | | **Comments / feedback** | **Yes** — every feedback surface posts to `/api/feedback`. | -| **SOPs** | Endpoints exist (`/api/sops`); the front end still keeps the SOP in the browser (namespaced per project). Wiring it to the API is the remaining **Phase 2** step. | -| **Work Packages** | Same — `/api/wps` (+ issue/status/metrics) exist and are ready; the creator still saves to the browser per project. | +| **SOPs** | **Yes** — pulled from `/api/sops` on load and written through on every save. | +| **Work Packages** | **Yes** — same write-through to `/api/wps` (+ issue / status / archive / metrics), including the owner assignment (`assignee_id`). | -So a fresh deployment gives you **shared, server-stored projects and comments -immediately**. Moving SOPs and Work Packages off the browser and onto the API -(so they're shared across users too) is a front-end change only — the database -and endpoints are already in place. +Saves go through a **durable client-side sync outbox**: edits are written to the +API immediately, and if the device is offline they queue and retry when it +reconnects (4xx rejections are dropped rather than retried forever). The browser +cache is only an offline fallback that reconciles through that outbox — so two +users on the same project see the same server-stored SOP and Work Packages. ## Data model (PostgreSQL) @@ -181,8 +205,13 @@ and endpoints are already in place. |-------|-------|-------------| | `projects` | top-level construction projects | `name`, `number`, `client`, `division`, `site`, `sample`, `data` | | `sops` | project SOP baselines | `project_id` → projects, `name`, `number`, `complete`, `data` (full SOP JSON) | -| `work_packages` | individual IWPs | `project_id` → projects, `sop_id` → sops, `parent_id` (split instances), `number`, `subject`, `type`, `status`, `issued_at`, `data` (full WP JSON) | +| `work_packages` | individual IWPs | `project_id` → projects, `sop_id` → sops, `parent_id` (split instances), `number`, `subject`, `type`, `status`, `assignee_id` (owner), `issued_at`, `archived_at`, `data` (full WP JSON) | | `comments` | feedback from any page | `source`, `sop_id`, `wp_id`, `step`, `author`, `text`, `extra` | +| `users` | login accounts | `username`, `password_hash` (bcrypt), `role`, `full_name`, `email`, `is_active`, login-lockout + `token_version` fields | +| `project_members` | per-project access control | `user_id` → users, `project_id` → projects | +| `audit_log` | append-only activity trail | `actor`, `action`, `entity_type`, `entity_id`, `project_id`, `summary`, `detail` | +| `notifications` | in-app record + email outbox | `user_id`, `kind`, `wp_id`, `subject`, `status` (pending / sent / failed / skipped) | +| `app_settings` | admin-configured settings (e.g. email) | `key`, `value` (JSON) | The complete client document is stored verbatim in each row's `data` JSON column; frequently-listed fields are promoted to real columns for filtering. @@ -192,8 +221,13 @@ column; frequently-listed fields are promoted to real columns for filtering. Projects `GET/POST /api/projects`, `GET/DELETE /api/projects/{id}` · SOPs `GET/POST /api/sops`, `GET /api/sops/latest`, `GET/DELETE /api/sops/{id}` · Work Packages `GET/POST /api/wps`, `GET/DELETE /api/wps/{id}`, -`POST /api/wps/{id}/issue`, `POST /api/wps/{id}/status`, `GET /api/wps/metrics` · -Comments `POST /api/comments` (and `/api/feedback`), `GET /api/comments`. +`POST /api/wps/{id}/issue`, `POST /api/wps/{id}/status`, `POST /api/wps/{id}/archive`, +`GET /api/wps/metrics` · +Comments `POST /api/comments` (and `/api/feedback`), `GET /api/comments` · +Auth `POST /api/auth/login` / `logout`, `GET /api/auth/me`, admin user management +under `/api/auth/users` · Admin-only `GET/PUT /api/settings`, +`POST /api/settings/test-email`, `GET /api/notifications`, +`GET /api/projects/{id}/members`. List/latest/metrics accept a `project_id` (and `sop_id`) filter. Full reference and request shapes: `/api/docs` and [`server/README.md`](server/README.md). @@ -209,27 +243,77 @@ docker compose up -d --build api # backend change (server/) ## Backups & retention -The whole dataset is in the `pgdata` volume — back it up on a schedule: +A **`backup` sidecar** (in `docker-compose.yml`) runs `pg_dump` on a schedule and +writes gzipped, timestamped dumps to `./backups/` on the host. It starts with the +stack — no cron to set up. -```bash -# Backup (run from project root) -docker compose exec -T db pg_dump -U wpsuite wpsuite > backup-$(date +%F).sql +- **Cadence / retention:** daily, keeping the newest 14 dumps. Override in `.env` + with `BACKUP_INTERVAL_SECONDS` (seconds between dumps) and `BACKUP_KEEP` (how many + to keep). +- **Encryption at rest:** set `BACKUP_ENC_PASSPHRASE` in `.env` and dumps are + written AES-256-encrypted as `*.sql.gz.enc`. **Do this before any customer IP + goes in** — without it the dumps (and every offsite copy) are plaintext. Store + the passphrase somewhere other than this host; if you lose it the backups can't + be restored. +- **Ad-hoc backup now:** `docker compose exec backup sh /scripts/db-backup.sh` +- **Restore (destructive — overwrites current data):** + `docker compose exec backup sh /scripts/db-restore.sh /backups/wpsuite-YYYYMMDD-HHMMSSZ.sql.gz.enc` +- **Offsite — do this:** the dumps live in `./backups/` on the host; if the host/volume + dies, so do they. Sync that folder offsite from the **host** (e.g. a cron running + `rclone`/`aws s3 sync`). The `db`/`backup` containers are on an egress-less + `internal` network on purpose, so offsite must be pushed from the host. +- **Test restores quarterly:** load the latest dump into a throwaway database and + confirm it applies. An untested backup is not a backup. -# Restore -docker compose exec -T db psql -U wpsuite -d wpsuite < backup-YYYY-MM-DD.sql -``` +## Field devices & data at rest -## Schema migrations (important) +The field view (PWA) caches a project's Work Packages/SOP in the browser's +localStorage so it works offline — i.e. **customer IP sits on the device**. +localStorage is not encrypted and is not a security boundary. Signing out clears +the cached project data, but for any tablet/phone that opens customer-IP projects: -Tables are auto-created on API startup (`Base.metadata.create_all`). This -creates **missing tables**, but it does **not** alter existing ones. The -multi-project work added the `projects` table and new columns -(`sops.project_id`, `work_packages.project_id` / `parent_id` / `issued_at`): +- **Require full-disk encryption** (BitLocker / FileVault / Android FBE / iOS is + encrypted by default) and a device passcode. +- **Enrol field devices in MDM** so a lost device can be remotely wiped, and keep + the browser profile per-user on shared devices. +- Users should **sign out** when handing off a shared device (clears the cache). -- On a **fresh** database these appear automatically — nothing to do. -- On a database that **already has data** from an older schema, add the new - columns with a migration (introduce **Alembic**) or apply them manually with - `ALTER TABLE` before deploying — don't rely on `create_all` for column changes. +## Email notifications (optional) + +Work-package **owner assignment** works out of the box (in-app only). Optional +**email** on assignment is **OFF by default** and is turned on from the **Admin +console → Notifications & email** card, where an admin sets the SMTP host / port / +TLS / From address and flips the master toggle. + +- The **SMTP password is never stored in the database.** It is read only from the + `SMTP_PASSWORD` environment variable (see the `.env` block in step 2 and the + `api` service in `docker-compose.yml`). The UI shows only whether it is set. +- Email stays effectively off until **all** of: the toggle is on, SMTP host + From + are configured, and `SMTP_PASSWORD` is present. Until then, assignments are + still recorded in-app (status `skipped`); nothing is sent. +- Notification emails carry only a **WP number and a deep link** — never the work + package contents — so customer IP stays behind the login. +- Use the card's **Send test email** button to confirm SMTP before enabling. + +## Schema migrations (Alembic) + +Schema is managed by **Alembic** (`server/alembic/`). The API container runs +`alembic upgrade head` on startup (see the `Dockerfile` CMD), so **deploys apply +pending migrations automatically**. + +- The **baseline** migration is idempotent: on a fresh database it creates every + table; on a database whose tables already exist (made by the old `create_all`) + it adopts the schema as-is — no manual `alembic stamp` needed. +- Local dev on SQLite still auto-creates tables for a zero-config run; Postgres is + migrations-only. +- **To change the schema:** edit `server/models.py`, then generate and review a + migration before committing: + ```bash + # from the project root (against your dev SQLite or a staging DB) + python -m alembic -c server/alembic.ini revision --autogenerate -m "describe the change" + python -m alembic -c server/alembic.ini upgrade head # apply locally to test + ``` + The next `docker compose up -d --build api` applies it in production on startup. ## Local trial without Postgres diff --git a/Dockerfile b/Dockerfile index 7335448..8088b88 100644 --- a/Dockerfile +++ b/Dockerfile @@ -4,7 +4,8 @@ COPY server/requirements.txt ./server/ RUN pip install --no-cache-dir -r server/requirements.txt COPY server/ ./server/ EXPOSE 8000 -# --preload imports the app once in the master (so create_all runs a single time) -# before forking workers, preventing a table-creation race on first startup. -CMD ["gunicorn", "-k", "uvicorn.workers.UvicornWorker", "--preload", \ - "-b", "0.0.0.0:8000", "--workers", "2", "server.app:app"] \ No newline at end of file +# Apply any pending DB migrations, THEN start the app. `alembic upgrade head` is +# safe on both fresh and existing databases (the baseline migration adopts an +# existing schema, so no manual stamp is needed). `exec` hands PID 1 to gunicorn +# for correct signal handling; --preload imports the app once before forking. +CMD ["sh", "-c", "alembic -c server/alembic.ini upgrade head && exec gunicorn -k uvicorn.workers.UvicornWorker --preload -b 0.0.0.0:8000 --workers 2 server.app:app"] \ No newline at end of file diff --git a/backups/.gitkeep b/backups/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docker-compose.yml b/docker-compose.yml index 0d436ea..5f1e9f2 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -27,9 +27,14 @@ services: POSTGRES_HOST: db # Optional full-URL override (must be URL-encoded if used). DATABASE_URL: ${DATABASE_URL:-} - # Signs login session cookies. MUST be set (see server/.env.example). - AUTH_SECRET_KEY: ${AUTH_SECRET_KEY} + # Signs login session cookies. REQUIRED — compose fails fast if it's unset, + # and the API refuses to start in production without it (see server/auth.py). + AUTH_SECRET_KEY: ${AUTH_SECRET_KEY:?set AUTH_SECRET_KEY in .env (see server/.env.example)} AUTH_SESSION_HOURS: ${AUTH_SESSION_HOURS:-12} + # Optional — SMTP password for WP-assignment emails. Email is off by + # default and enabled from the Admin console; this is the only email + # secret and it is never stored in the DB. Leave unset until configured. + SMTP_PASSWORD: ${SMTP_PASSWORD:-} restart: unless-stopped depends_on: db: @@ -55,6 +60,36 @@ services: networks: - internal + # Scheduled pg_dump backups. Writes gzipped, timestamped dumps to ./backups on + # the host (sync that folder offsite from the host — this container has no + # internet egress). See scripts/db-backup.sh and DEPLOYMENT.md § Backups. + backup: + build: + context: . + dockerfile: scripts/backup.Dockerfile # postgres client + openssl + container_name: wp_db_backup + environment: + POSTGRES_USER: ${POSTGRES_USER} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + POSTGRES_DB: ${POSTGRES_DB} + PGHOST: db + BACKUP_DIR: /backups + BACKUP_KEEP: ${BACKUP_KEEP:-14} # keep the newest N dumps + BACKUP_INTERVAL_SECONDS: ${BACKUP_INTERVAL_SECONDS:-86400} # 86400 = daily + # Set BACKUP_ENC_PASSPHRASE in .env to encrypt dumps at rest (AES-256). + # Required once the DB holds customer IP. Keep the passphrase off this host. + BACKUP_ENC_PASSPHRASE: ${BACKUP_ENC_PASSPHRASE:-} + volumes: + - ./scripts:/scripts:ro + - ./backups:/backups + entrypoint: ["/bin/sh", "/scripts/backup-cron.sh"] + restart: unless-stopped + depends_on: + db: + condition: service_healthy + networks: + - internal + volumes: pgdata: nginx_logs: diff --git a/html/admin.html b/html/admin.html index 153a3c9..fdb71e9 100644 --- a/html/admin.html +++ b/html/admin.html @@ -6,18 +6,21 @@ Admin Console — Work Package Suite + + + + +
+ + Prime Controls + Work Package Suite | Admin Console + +
+