building out frontend and docker

This commit is contained in:
C-West8
2026-10-23 06:52:20 -05:00
parent 42a1ce7d30
commit c9ce46d7bc
58 changed files with 2665 additions and 50 deletions

View File

@@ -7,7 +7,7 @@
| Direction | One-way, SharePoint → ACC |
| Latency | Poll every ~3 min (configurable); "Run now" for immediate sync |
| Operations | Create, update (new ACC version), rename, move, delete |
| Scope | Many projects per server; each project is a 1:1 pair of SharePoint root ↔ ACC root |
| Scope | Many projects per server; each project is a 1:1 pair of SharePoint root and ACC root |
| ACC ownership | Only touch ACC items this tool created; hand-added ACC content is left alone |
| Ignored files | Temp/lock/system files are skipped silently (not issues) |
| Retries | 3 attempts across runs, then surface as an issue |
@@ -164,14 +164,42 @@ MAX_ATTEMPTS=3
HISTORY_RETENTION_DAYS=7
```
Env vars are declared and validated with `zod` in `src/env.ts` (SvelteKit's `defineEnvVars`); the server refuses to start on bad config. Only the variables in use so far are declared; the Graph, APS and admin ones get added along with the code that reads them. Run exactly **one** replica, because the locks live in-process.
Env vars are declared and validated with `zod` in `src/env.ts` (SvelteKit's `defineEnvVars`); the server refuses to start on bad config. Only the variables in use so far are declared; the Graph and APS ones get added along with the adapters that read them. Run exactly **one** replica, because the locks live in-process.
`SYNC_BACKEND` picks the SharePoint/ACC connection:
- **`live`**: the real Graph/APS adapters. Until those exist, a `NotConnectedBackend` stands in. Runs fail as an outage (no attempts charged) and the add-project form explains why.
- **`simulated`**: `FakeDrive`/`FakeAcc` behind a `SimulatedBackend` that resolves any pasted path, seeds sample files and makes random edits. The database is in-memory in this mode, so the fake services and the sync state never disagree.
`ADMIN_USERNAME`/`ADMIN_PASSWORD` turn the admin pages on; `SESSION_SECRET` keeps logins valid across restarts.
Two settings live outside `src/env.ts`:
- **`ORIGIN`** is read at build time (`paths.origin` in `vite.config.ts`). SvelteKit 3 fixes the app's address at build time, and adapter-node otherwise assumes HTTPS. So over plain HTTP, form posts fail the CSRF check, and the session cookie is marked `Secure`, unless `ORIGIN` is set. Behind an HTTPS reverse proxy, leave it unset.
- **`TZ`** is the container's time zone. Times are formatted on the server first, so it should match the users' zone.
### Deployment
- `Dockerfile`: a multi-stage build on `node:24-bookworm-slim`. The runtime stage holds `build/`, production `node_modules` (`better-sqlite3`, `zod`; everything else is bundled) and `drizzle/`, runs as `node`, and keeps the database at `/data/sync.db`.
- `docker-compose.yml` is the Portainer stack: one service, the `sync-data` volume, required settings enforced with `${VAR:?}`, and `stop_grace_period: 2m` so a run in progress can finish.
- `GET /healthz` backs the image's `HEALTHCHECK`.
- On startup, runs left without an outcome (the process was killed mid-run) are closed as failed. The next run carries on from the saved item state.
- Operations are covered in [runbook.md](runbook.md).
### Code conventions
- **Imports:** never `../`. Anything outside the current folder goes through the `#lib/...` alias (`package.json` `imports`). Every local import carries its `.ts` extension, which SvelteKit 3's tsconfig requires.
- **Internal links:** always built with `resolve()` from `$app/paths` (enforced by lint).
## Source layout
```
src/env.ts # env var declarations + validation
src/hooks.server.ts # init → startApp(config); admin route guard
src/lib/server/
container.ts # (todo) composition root: the only place adapters are chosen
container.ts # composition root: the only place adapters are chosen
app.ts # holds the running services (survives dev reloads)
forms.ts logger.ts
sync/
core/ # pure TS, no SvelteKit imports
model.ts ports.ts errors.ts
@@ -182,16 +210,22 @@ src/lib/server/
tree.ts ignore.ts
coordinator.ts # per-project lock + manual cooldown
scheduler.ts # non-overlapping interval tasks
testing/ # FakeDrive, FakeAcc, harness (real engine + in-memory SQLite)
project-service.ts # admin operations: create/edit/reset/delete, retry
testing/ # harness: real engine + in-memory SQLite + fakes
status/ # read models for the UI: summaries, issues, file tree
auth/ # admin credentials, signed session tokens, cookie helpers
adapters/
sqlite/ # Drizzle schema, migrations runner, repositories
graph/ # (todo) GraphChangeSource, auth (msal-node)
aps/ # (todo) ApsDocumentTarget, auth, upload flow
simulated/ # FakeDrive, FakeAcc, SimulatedBackend (demo + tests)
not-connected.ts # placeholder for live mode until Graph/APS exist
graph/ # (todo) GraphChangeSource + SourceRootResolver, auth (msal-node)
aps/ # (todo) ApsDocumentTarget + TargetRootResolver, upload flow
http/ # (todo) shared fetch wrapper: retry, backoff, Retry-After
auth/ # (todo) admin session cookie
src/lib/components/ # Badge, FileTree, RunNowButton, ProjectFields, TimeAgo, AutoRefresh
src/lib/styles/app.css # design tokens + shared styles
src/routes/ # projects, project status + history, admin, healthz
drizzle/ # generated SQL migrations (npm run db:generate)
src/hooks.server.ts # (todo) init → container → scheduler; admin route guard
src/routes/ # (todo) UI
Dockerfile docker-compose.yml
```
## Open questions (resolved by spikes)
@@ -205,11 +239,11 @@ src/routes/ # (todo) UI
0. **Access requests:** start now; they're the slowest part. See [access-setup.md](access-setup.md).
1. **Spikes** against a sandbox ACC project and test SharePoint site. Answer the open questions above and record the results here.
2. ✅ **Scaffold:** SvelteKit 3 + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier. (Dockerfile moves to step 9.)
3. ✅ **Core:** ports, IgnorePolicy, ingest, planner, reconciler and SyncEngine, tested against FakeDrive/FakeAcc.
4. ✅ **SQLite repositories** and migrations.
5. **GraphChangeSource.**
6. **ApsDocumentTarget**, including the fallbacks chosen in step 1.
7. **Wiring:** ✅ coordinator, scheduler. Still to do: composition root, `init` hook, retention job schedule.
8. **UI:** status pages, Run now, admin login, project add/edit/reset.
9. **Ship:** Docker image, Portainer stack file, runbook.
2. **Scaffold (done):** SvelteKit 3 + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier. (Dockerfile moves to step 9.)
3. **Core (done):** ports, IgnorePolicy, ingest, planner, reconciler and SyncEngine, tested against FakeDrive/FakeAcc.
4. **SQLite repositories and migrations (done).**
5. **GraphChangeSource** + SharePoint URL resolver.
6. **ApsDocumentTarget** + ACC folder URL resolver, including the fallbacks chosen in step 1.
7. **Wiring (done):** coordinator, scheduler, composition root, `init` hook, history pruning, simulated mode.
8. **UI (done):** projects list, project status (title block, issues with retry, file tree), 7-day history with per-run detail, Run now with cooldown, admin login, add/edit/resume/reset/remove.
9. **Ship (done):** Docker image, Portainer stack file, runbook. The image was tested locally. A first Portainer deploy is still to do.

84
docs/runbook.md Normal file
View File

@@ -0,0 +1,84 @@
# Runbook
How to deploy, update, back up and troubleshoot the sync server.
## What runs
- **One container**, built from the `Dockerfile`. Never run more than one copy: the locks that stop two syncs of the same project overlapping live inside the process.
- **One volume**, `sync-data`, mounted at `/data`. It holds `sync.db`, the SQLite database with the projects, the record of which ACC item belongs to which SharePoint item, and the last 7 days of history.
- **Port 3000** inside the container. `HOST_PORT` sets the port on the server (default 3000).
- **Health check:** `GET /healthz` answers `{"status":"ok"}` once the server has started. Docker checks it every 30 seconds, and Portainer shows the result as healthy or unhealthy.
- **Logs:** one JSON object per line on stdout. View them in Portainer under the container's logs.
## Choose how people will reach it
Decide this before the first deploy, because it changes how the image is built.
- **Behind an HTTPS reverse proxy** (recommended). The proxy must pass the original `Host` header on, which most do by default. Leave `ORIGIN` empty.
- **Plain HTTP** on the VPN, for example `http://sync-server:3000`. Set `ORIGIN` to exactly the address people will type, with no trailing slash. `ORIGIN` is fixed when the image is built, so the app only works at that one address: not by IP if `ORIGIN` uses a name, and not on a different port.
If `ORIGIN` is wrong or missing over plain HTTP, pages still load, but every button that submits something fails with "Cross-site POST form submissions are forbidden".
## Settings
Set these in the Portainer stack's **Environment variables**, or in a `.env` file next to `docker-compose.yml` when using the command line. Never put them in the repository.
| Variable | Required | Default | What it is |
| ----------------------------- | --------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ADMIN_USERNAME` | yes | | Login for the admin pages |
| `ADMIN_PASSWORD` | yes | | Password for the admin pages |
| `SESSION_SECRET` | yes | | Signs admin logins. Any long random string, e.g. the output of `node -e "console.log(crypto.randomBytes(32).toString('hex'))"`. Changing it logs admins out |
| `ORIGIN` | see above | | The app's address when served over plain HTTP. Build time only |
| `HOST_PORT` | no | `3000` | Port on the server |
| `SYNC_BACKEND` | no | `live` | `simulated` runs the in-memory demo instead (nothing is saved) |
| `TZ` | no | `America/Chicago` | Time zone the status pages show times in |
| `POLL_INTERVAL_SECONDS` | no | `180` | How often every project is checked for changes |
| `MANUAL_RUN_COOLDOWN_SECONDS` | no | `30` | Minimum gap between "Run now" presses for one project |
| `MAX_ATTEMPTS` | no | `3` | Failed tries before a file is reported as an issue |
| `HISTORY_RETENTION_DAYS` | no | `7` | How long sync history is kept |
The Microsoft Graph and APS credentials will be added to this table when the live connections are built.
## First deploy (Portainer)
1. **Stacks → Add stack**, give it a name such as `sharepoint-acc-sync`, and choose **Repository**.
2. Enter the repository URL and the branch (`refs/heads/main`). The compose path is `docker-compose.yml`. If the repository is private, turn on authentication and use a read-only access token.
3. Under **Environment variables**, add the settings above.
4. **Deploy the stack.** Portainer builds the image from the `Dockerfile`, which takes a few minutes the first time.
5. Check it's working:
- the container shows as **healthy** in Portainer;
- the logs show `Sync server started` with the expected `backend`;
- the projects page opens at the chosen address, and the admin login works.
From the command line on the Docker host, the equivalent is `docker compose up -d --build` in a clone of the repository, with the settings in `.env`.
## Updating
1. In Portainer, open the stack and use **Pull and redeploy**. Make sure the image is rebuilt, not reused: after the first update, check that the change is actually live.
2. On the command line: `git pull`, then `docker compose up -d --build`.
When the container stops, a sync in progress gets up to 2 minutes to finish. If it doesn't finish in time, it's marked in the history as stopped by a restart, and the next sync picks up whatever was left. Nothing needs cleaning up by hand.
## Backups
Back up `sync.db`. Losing it means the tool forgets which ACC files it created. It would then treat every SharePoint file as new, and could not tidy up the ACC copies it made before.
To take a copy while the server is running:
```sh
docker exec <container> node -e "require('better-sqlite3')('/data/sync.db').backup('/data/backup.db').then(() => console.log('backed up'))"
docker cp <container>:/data/backup.db ./sync-backup.db
```
To restore one: stop the stack, put the backup in the volume as `sync.db`, delete any `sync.db-wal` and `sync.db-shm` beside it, then start the stack.
## Troubleshooting
| Symptom | Likely cause and fix |
| ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Deploy fails with "required variable … is missing a value" | A required setting isn't set. Add it to the stack's environment variables |
| Buttons fail with "Cross-site POST form submissions are forbidden" | Plain HTTP without the right `ORIGIN`, or the app was opened at a different address. Set `ORIGIN` and rebuild |
| Admin login appears to work but doesn't stick | Plain HTTP without `ORIGIN`: the login cookie is marked HTTPS-only, so the browser drops it. Set `ORIGIN` and rebuild |
| Container is unhealthy or keeps restarting | Check the logs. Bad settings are reported by name at startup |
| Times on the pages are off by some hours | `TZ` doesn't match the office's time zone |
| A project shows **Paused** | Its SharePoint folder was deleted or couldn't be found. Nothing was deleted in ACC. Fix the folder, then **Resume** it on the admin page |