From 09a5f8fe0d22af6900e87c24af9705125ade5b7c Mon Sep 17 00:00:00 2001 From: C-West8 <125926137+C-West8@users.noreply.github.com> Date: Thu, 22 Oct 2026 22:05:33 -0500 Subject: [PATCH] planning docs --- README.md | 24 +++++ docs/access-setup.md | 32 ++++++ docs/design.md | 234 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 290 insertions(+) create mode 100644 README.md create mode 100644 docs/access-setup.md create mode 100644 docs/design.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..0414ad6 --- /dev/null +++ b/README.md @@ -0,0 +1,24 @@ +# SharePoint → ACC Sync + +Keeps project folders in SharePoint mirrored into Autodesk Construction Cloud (ACC / Autodesk Forma Docs), with a few minutes' delay at most. One server can sync many projects. Each project pairs one SharePoint folder with one ACC folder. + +There is also a small web page that shows each project's file tree, sync status, open issues and the last 7 days of sync history. It has a "Run now" button as well. + +## What it does + +- **One-way sync.** SharePoint is the source of truth. Nothing is ever written back to SharePoint. +- **Every file operation carries over.** Creates, edits (as new ACC versions), renames, moves and deletes are all mirrored. Removing something from the project's SharePoint root counts as deleting it. +- **Files added to ACC by hand are left alone.** The sync only ever changes ACC items it created. +- **Temp and lock files are ignored.** That covers `~$*`, `*.tmp`, `Thumbs.db` and similar. +- **Failures are retried 3 times**, then reported as an issue on the status page. + +## Status + +Planning. See: + +- [docs/design.md](docs/design.md): architecture, interfaces, data model and build order +- [docs/access-setup.md](docs/access-setup.md): what IT and the ACC admin need to set up + +## Stack + +SvelteKit (TypeScript) on `adapter-node`, running as a single long-lived Docker container. It stores its state in SQLite on a mounted volume. It talks to Microsoft Graph for SharePoint and to the Autodesk Platform Services (APS) Data Management API for ACC. diff --git a/docs/access-setup.md b/docs/access-setup.md new file mode 100644 index 0000000..1230951 --- /dev/null +++ b/docs/access-setup.md @@ -0,0 +1,32 @@ +# Access setup + +The sync server needs two app identities: one for Microsoft (to read SharePoint) and one for Autodesk (to write to ACC). Both are shared by every project. Adding a new project only means granting these identities access to the new folders. + +Nothing here needs a user to stay logged in. Both use app-only (client credentials) auth. + +## Microsoft / SharePoint (ask IT) + +1. **Create an Entra ID app registration**, e.g. "SharePoint ACC Sync". + - Single tenant. + - Create a **client secret** (or a certificate, if IT prefers). Note its expiry date, because the sync stops when it expires. + - Send us the **Tenant ID**, **Client ID**, and the secret, securely. They go into Portainer as secrets. +2. **Grant Microsoft Graph application permissions (read-only)**, with admin consent. Pick one: + - **`Sites.Selected` (recommended, least privilege).** The app can read only the sites IT explicitly grants it. Each new project's site needs a one-time grant by IT (`POST /sites/{site-id}/permissions` with role `read`). + - **`Sites.Read.All`.** The app can read every SharePoint site in the tenant. No per-project step, but much broader access. +3. **For each project:** confirm the site is granted (if using `Sites.Selected`), then give us the URL of the project's root folder. + +The tool only ever reads from SharePoint, so no write permission is needed. + +## Autodesk / ACC (ask the ACC account admin) + +1. **Create an APS app** at → *My Apps*. + - App type: **Server-to-Server** (app-only). + - API access: **Data Management API**. + - Send us the **Client ID** and **Client Secret**. +2. **In ACC Account Admin → Custom Integrations, add the APS app's Client ID** so it can access the account's projects. +3. **Possibly: a service user.** Some ACC Docs operations may need to run "as" a user, i.e. an account that is a project member with edit/upload permission on the target folder. We'll confirm this during the build spikes before asking for it. +4. **For each project:** tell us which ACC project, and which folder under *Project Files*, is the sync root. + +## For the build spikes + +A **sandbox ACC project** and a **test SharePoint site/folder** we can freely create, move and delete files in. The early tests check how ACC handles moves and restores, and they shouldn't run against real project data. diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..ea6f434 --- /dev/null +++ b/docs/design.md @@ -0,0 +1,234 @@ +# Design + +## Goals and constraints + +| Decision | Choice | +| --- | --- | +| Direction | One-way, SharePoint → ACC | +| Latency | Poll every ~3 min (configurable); "Run now" for immediate sync | +| Operations | Create, update (new ACC version), rename, move, delete | +| Scope | Many projects per server; each project is a 1:1 pair of SharePoint root ↔ ACC root | +| ACC ownership | Only touch ACC items this tool created; hand-added ACC content is left alone | +| Ignored files | Temp/lock/system files are skipped silently (not issues) | +| Retries | 3 attempts across runs, then surface as an issue | +| Files | Small/medium documents; no large Revit models | +| Hosting | One Docker container (Portainer), SQLite on a volume | +| Credentials | One Microsoft app registration + one APS app, shared by all projects | +| UI access | Viewing and "Run now" are open (behind VPN). Adding, editing or resetting projects requires the single admin login from env | + +## Architecture + +The sync logic is plain TypeScript with no SvelteKit imports. SvelteKit hosts it: it starts the scheduler from the `init` hook and serves the status UI. The core depends only on interfaces (ports); SharePoint, ACC and SQLite are adapters plugged in at one composition root. + +``` + ┌──────────────── SvelteKit (adapter-node) ────────────────┐ + │ routes/ (status UI, Run now, admin) hooks.server.ts │ + │ │ (init → start) │ + │ ▼ │ │ + │ SyncCoordinator ◄── scheduler tick ────┘ │ + │ (per-project lock + cooldown) │ + │ │ │ + │ ▼ │ + │ SyncEngine ── IgnorePolicy │ + │ ┌──────┼───────────────┐ │ + │ ▼ ▼ ▼ │ + │ ChangeSource Repositories DocumentTarget │ + │ (Graph) (SQLite) (APS / ACC) │ + └──────────────────────────────────────────────────────────┘ +``` + +### Each run has two phases + +1. **Ingest.** Read changes from SharePoint since the project's saved cursor and write them to `items` as the *desired* state (name, parent, content tag, deleted flag). Save the new cursor in the same transaction. Nothing touches ACC in this phase. +2. **Reconcile.** For every item whose desired state differs from its *synced* state, work out the operation and apply it to ACC. On success, copy desired → synced. On failure, increment `attempt_count` and record the error. + +This split gives us several things without extra code: + +- **Retries.** Failed items still differ, so the next run picks them up again. No separate retry queue is needed. +- **Crash safety.** If the container dies mid-run, the cursor is already saved and the pending work is still in the database. +- **Inferred renames and moves.** Graph delta only reports "this item now looks like X". The operation comes from comparing that to what we last synced: + +| Desired vs synced | Operation | +| --- | --- | +| no target id yet | create (folder) / upload (file) | +| `deleted` | delete | +| name differs | rename | +| parent differs | move | +| content tag differs | upload new version | + + A single change can need more than one of these, for example a move plus a rename. + +**Order within reconcile:** folder creates and moves from shallowest to deepest, then files, then deletes. That way a destination folder always exists before anything moves into it. + +### Scope and delta queries + +Graph delta for SharePoint only works at the **drive root**, not on subfolders. So each project runs delta on its whole drive and keeps only items under the project's root folder. It checks this with the parent chain stored in `items`. + +An item whose new parent is outside the root counts as deleted. That covers "removed from the root = deleted in ACC". + +**Safety rule:** if the project's root folder itself disappears or moves, the project is paused with a project-level issue and **nothing is deleted**. That is almost always a mistake, not an intent to wipe ACC. + +## Interfaces (ports) + +These are sketches; final signatures get settled during the build. + +```ts +// sync/core/ports.ts + +export interface SourceItem { + sourceId: string; + parentSourceId: string | null; + name: string; + isFolder: boolean; + contentTag: string | null; // Graph cTag; changes only when content changes + modifiedAt: Date; + size: number | null; +} + +export type SourceChange = + | { kind: 'upsert'; item: SourceItem } + | { kind: 'delete'; sourceId: string }; + +export interface ChangeSource { + /** Changes since `cursor` (null = full enumeration), with the cursor to save. */ + getChanges(root: SourceRoot, cursor: string | null): Promise<{ changes: SourceChange[]; cursor: string }>; + download(root: SourceRoot, sourceId: string): Promise<{ stream: ReadableStream; size: number }>; +} + +export interface TargetRef { id: string; isFolder: boolean } + +export interface DocumentTarget { + createFolder(parentId: string, name: string): Promise; + uploadFile(parentId: string, name: string, content: FileContent): Promise; + uploadVersion(ref: TargetRef, name: string, content: FileContent): Promise; + rename(ref: TargetRef, name: string): Promise; + /** May return a new ref if the target can only move by copy + delete. */ + move(ref: TargetRef, newParentId: string): Promise; + delete(ref: TargetRef): Promise; + /** Bring back a previously deleted item (keeps history where possible). */ + restore(ref: TargetRef, parentId: string, name: string, content?: FileContent): Promise; +} + +export interface IgnorePolicy { + isIgnored(name: string, isFolder: boolean): boolean; +} +``` + +Repositories are split by consumer (interface segregation): + +- `ProjectRepository`: CRUD, cursor, pause and reset +- `ItemRepository`: desired/synced state, pending diff query, tree query for the UI +- `RunRepository`: start/finish runs, record events, prune old history + +Both fallback strategies, move-by-copy and restore-by-reupload, live **inside the ACC adapter**. They are ACC-specific details, so the engine never branches on them. + +### Coordinator + +`SyncCoordinator.requestRun(projectId, trigger)` is the only way a run starts, whether from the scheduler or from "Run now". It holds one in-process lock per project and returns one of: + +- `started` +- `already-running` (the caller attaches to the current run) +- `{ cooldown: secondsLeft }` (the last manual run was less than 30 s ago) + +The scheduler goes through enabled projects one at a time, and one failing project never blocks the others. Graph and APS clients share one HTTP layer that handles `429`/`Retry-After` and backoff, because rate limits apply per app, not per project. + +## Data model (SQLite via Drizzle) + +``` +projects + id, name, enabled, paused_reason + sp_site_id, sp_drive_id, sp_root_item_id, sp_display_path + acc_hub_id, acc_project_id, acc_root_folder_id, acc_display_path + delta_cursor, first_synced_at, last_run_at, created_at + +items PK (project_id, source_id) + -- desired (from SharePoint) + parent_source_id, name, is_folder, content_tag, source_modified_at, size, deleted + -- synced (what ACC has) + target_id, synced_parent_target_id, synced_name, synced_content_tag, synced_at + -- bookkeeping + status ('synced' | 'pending' | 'failed' | 'deleted') + attempt_count, last_error, last_attempt_at + +runs + id, project_id, trigger ('schedule' | 'manual'), started_at, finished_at + outcome ('success' | 'partial' | 'failed'), error + created, updated, renamed, moved, deleted, failed -- counts + +run_events + id, run_id, project_id, source_id, path, action, outcome, message, at +``` + +- IDs are stored, not paths, so renaming a root folder doesn't break a project. Display paths are shown in the UI only. +- **Issues** = items with `status = 'failed'`, plus a project-level issue when it is paused or its latest run failed outright (bad auth, root missing). +- A nightly job deletes `runs` and `run_events` older than `HISTORY_RETENTION_DAYS`. `items` is current state, so it is never pruned. +- After a failed item has used up its attempts, it is only retried when its content or location changes again, or when someone clicks retry. +- Deleted items keep their `target_id` (a tombstone). That lets a SharePoint restore revive the same ACC item instead of creating a new one. +- A project's roots are locked once `first_synced_at` is set. Changing them requires an explicit **reset**, which clears the project's items and cursor. + +## UI + +| Route | Who | What | +| --- | --- | --- | +| `/` | anyone | Projects list: status, last run, open issue count | +| `/projects/[id]` | anyone | File tree with status badges and last update time, an issues panel and a "Run now" button | +| `/projects/[id]/history` | anyone | Runs from the last 7 days, expandable to per-file events | +| `/admin/login` | — | Single admin login | +| `/admin/projects/new`, `/admin/projects/[id]` | admin | Add/edit project (paste SharePoint URL and pick ACC folder; server resolves and checks the IDs), enable/disable, reset | + +- The status pages read only from SQLite. They never call SharePoint or ACC. +- While a run is active, the project page refreshes its data every few seconds. + +## Configuration (env) + +``` +GRAPH_TENANT_ID, GRAPH_CLIENT_ID, GRAPH_CLIENT_SECRET +APS_CLIENT_ID, APS_CLIENT_SECRET +ADMIN_USERNAME, ADMIN_PASSWORD, SESSION_SECRET +DATABASE_PATH=/data/sync.db +POLL_INTERVAL_SECONDS=180 +MANUAL_RUN_COOLDOWN_SECONDS=30 +MAX_ATTEMPTS=3 +HISTORY_RETENTION_DAYS=7 +``` + +Env is validated with `zod` at startup; the server refuses to start on bad config. Run exactly **one** replica, because the locks live in-process. + +## Source layout + +``` +src/lib/server/ + config.ts # zod-validated env + container.ts # composition root: the only place adapters are chosen + sync/ + core/ # ports, SyncEngine, reconcile planner, IgnorePolicy (pure TS) + coordinator.ts # locks, cooldown, scheduler loop, retention job + adapters/ + graph/ # GraphChangeSource, auth (msal-node) + aps/ # ApsDocumentTarget, auth, upload flow + sqlite/ # Drizzle schema + repositories + http/ # shared fetch wrapper: retry, backoff, Retry-After + auth/ # admin session cookie +src/hooks.server.ts # init → container → coordinator.start(); admin route guard +src/routes/ # UI +``` + +## Open questions (resolved by spikes) + +1. **Move a file in ACC natively?** The APS docs show folder moves via `PATCH folders` (changing the parent). Single-file moves are not clearly documented. If not supported, the fallback is copy then delete, which may lose ACC version history. +2. **Revive a deleted ACC item?** Can we add a version to, or restore, a deleted item through the API so that history and links are kept? If not, the fallback is to upload a new item. +3. **App-only auth for ACC writes:** whether some ACC Docs operations need an `x-user-id` (a service user that is a member of each project). +4. **Graph delta with a subfolder root:** confirm the drive-wide filtering approach and how folder deletes report their children. + +## Build order + +0. **Access requests:** start now; they're the slowest part. See [access-setup.md](access-setup.md). +1. **Spikes** against a sandbox ACC project and test SharePoint site. Answer the open questions above and record the results here. +2. **Scaffold:** SvelteKit + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier, Dockerfile. +3. **Core:** ports, IgnorePolicy, reconcile planner and SyncEngine, unit-tested against in-memory fakes. Most of the logic lives here. +4. **SQLite repositories** and migrations. +5. **GraphChangeSource.** +6. **ApsDocumentTarget**, including the fallbacks chosen in step 1. +7. **Coordinator:** scheduler, locks, cooldown, retention job, hooked into `init`. +8. **UI:** status pages, Run now, admin login, project add/edit/reset. +9. **Ship:** Docker image, Portainer stack file, runbook.