first draft
This commit is contained in:
@@ -19,13 +19,13 @@ The tool only ever reads from SharePoint, so no write permission is needed.
|
||||
|
||||
## Autodesk / ACC (ask the ACC account admin)
|
||||
|
||||
1. **Create an APS app** at <https://aps.autodesk.com> → *My Apps*.
|
||||
1. **Create an APS app** at <https://aps.autodesk.com> → _My Apps_.
|
||||
- App type: **Server-to-Server** (app-only).
|
||||
- API access: **Data Management API**.
|
||||
- Send us the **Client ID** and **Client Secret**.
|
||||
2. **In ACC Account Admin → Custom Integrations, add the APS app's Client ID** so it can access the account's projects.
|
||||
3. **Possibly: a service user.** Some ACC Docs operations may need to run "as" a user, i.e. an account that is a project member with edit/upload permission on the target folder. We'll confirm this during the build spikes before asking for it.
|
||||
4. **For each project:** tell us which ACC project, and which folder under *Project Files*, is the sync root.
|
||||
4. **For each project:** tell us which ACC project, and which folder under _Project Files_, is the sync root.
|
||||
|
||||
## For the build spikes
|
||||
|
||||
|
||||
177
docs/design.md
177
docs/design.md
@@ -2,19 +2,19 @@
|
||||
|
||||
## Goals and constraints
|
||||
|
||||
| Decision | Choice |
|
||||
| --- | --- |
|
||||
| Direction | One-way, SharePoint → ACC |
|
||||
| Latency | Poll every ~3 min (configurable); "Run now" for immediate sync |
|
||||
| Operations | Create, update (new ACC version), rename, move, delete |
|
||||
| Scope | Many projects per server; each project is a 1:1 pair of SharePoint root ↔ ACC root |
|
||||
| ACC ownership | Only touch ACC items this tool created; hand-added ACC content is left alone |
|
||||
| Ignored files | Temp/lock/system files are skipped silently (not issues) |
|
||||
| Retries | 3 attempts across runs, then surface as an issue |
|
||||
| Files | Small/medium documents; no large Revit models |
|
||||
| Hosting | One Docker container (Portainer), SQLite on a volume |
|
||||
| Credentials | One Microsoft app registration + one APS app, shared by all projects |
|
||||
| UI access | Viewing and "Run now" are open (behind VPN). Adding, editing or resetting projects requires the single admin login from env |
|
||||
| Decision | Choice |
|
||||
| ------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Direction | One-way, SharePoint → ACC |
|
||||
| Latency | Poll every ~3 min (configurable); "Run now" for immediate sync |
|
||||
| Operations | Create, update (new ACC version), rename, move, delete |
|
||||
| Scope | Many projects per server; each project is a 1:1 pair of SharePoint root ↔ ACC root |
|
||||
| ACC ownership | Only touch ACC items this tool created; hand-added ACC content is left alone |
|
||||
| Ignored files | Temp/lock/system files are skipped silently (not issues) |
|
||||
| Retries | 3 attempts across runs, then surface as an issue |
|
||||
| Files | Small/medium documents; no large Revit models |
|
||||
| Hosting | One Docker container (Portainer), SQLite on a volume |
|
||||
| Credentials | One Microsoft app registration + one APS app, shared by all projects |
|
||||
| UI access | Viewing and "Run now" are open (behind VPN). Adding, editing or resetting projects requires the single admin login from env |
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -39,8 +39,8 @@ The sync logic is plain TypeScript with no SvelteKit imports. SvelteKit hosts it
|
||||
|
||||
### Each run has two phases
|
||||
|
||||
1. **Ingest.** Read changes from SharePoint since the project's saved cursor and write them to `items` as the *desired* state (name, parent, content tag, deleted flag). Save the new cursor in the same transaction. Nothing touches ACC in this phase.
|
||||
2. **Reconcile.** For every item whose desired state differs from its *synced* state, work out the operation and apply it to ACC. On success, copy desired → synced. On failure, increment `attempt_count` and record the error.
|
||||
1. **Ingest.** Read changes from SharePoint since the project's saved cursor and write them to `items` as the _desired_ state (name, parent, content tag, deleted flag). Save the new cursor in the same transaction. Nothing touches ACC in this phase.
|
||||
2. **Reconcile.** For every item whose desired state differs from its _synced_ state, work out the operation and apply it to ACC. On success, copy desired → synced. On failure, increment `attempt_count` and record the error.
|
||||
|
||||
This split gives us several things without extra code:
|
||||
|
||||
@@ -48,15 +48,15 @@ This split gives us several things without extra code:
|
||||
- **Crash safety.** If the container dies mid-run, the cursor is already saved and the pending work is still in the database.
|
||||
- **Inferred renames and moves.** Graph delta only reports "this item now looks like X". The operation comes from comparing that to what we last synced:
|
||||
|
||||
| Desired vs synced | Operation |
|
||||
| --- | --- |
|
||||
| no target id yet | create (folder) / upload (file) |
|
||||
| `deleted` | delete |
|
||||
| name differs | rename |
|
||||
| parent differs | move |
|
||||
| content tag differs | upload new version |
|
||||
| Desired vs synced | Operation |
|
||||
| ------------------- | ------------------------------- |
|
||||
| no target id yet | create (folder) / upload (file) |
|
||||
| `deleted` | delete |
|
||||
| name differs | rename |
|
||||
| parent differs | move |
|
||||
| content tag differs | upload new version |
|
||||
|
||||
A single change can need more than one of these, for example a move plus a rename.
|
||||
A single change can need more than one of these, for example a move plus a rename.
|
||||
|
||||
**Order within reconcile:** folder creates and moves from shallowest to deepest, then files, then deletes. That way a destination folder always exists before anything moves into it.
|
||||
|
||||
@@ -66,69 +66,38 @@ Graph delta for SharePoint only works at the **drive root**, not on subfolders.
|
||||
|
||||
An item whose new parent is outside the root counts as deleted. That covers "removed from the root = deleted in ACC".
|
||||
|
||||
**Safety rule:** if the project's root folder itself disappears or moves, the project is paused with a project-level issue and **nothing is deleted**. That is almost always a mistake, not an intent to wipe ACC.
|
||||
**Safety rule:** if the project's root folder is deleted, or is missing from a full listing, the project is paused with a project-level issue and **nothing is deleted**. That is almost always a mistake, not an intent to wipe ACC. Moving or renaming the root within the same library is harmless, because projects track it by ID.
|
||||
|
||||
**Restored folders:** a folder delete cascades "deleted" to everything stored under it. If a deleted folder later reappears, its contents may not be reported one by one, so ingest asks for one follow-up **full listing** in the same run, which brings them back.
|
||||
|
||||
## Interfaces (ports)
|
||||
|
||||
These are sketches; final signatures get settled during the build.
|
||||
The source of truth is [`src/lib/server/sync/core/ports.ts`](../src/lib/server/sync/core/ports.ts):
|
||||
|
||||
```ts
|
||||
// sync/core/ports.ts
|
||||
|
||||
export interface SourceItem {
|
||||
sourceId: string;
|
||||
parentSourceId: string | null;
|
||||
name: string;
|
||||
isFolder: boolean;
|
||||
contentTag: string | null; // Graph cTag; changes only when content changes
|
||||
modifiedAt: Date;
|
||||
size: number | null;
|
||||
}
|
||||
|
||||
export type SourceChange =
|
||||
| { kind: 'upsert'; item: SourceItem }
|
||||
| { kind: 'delete'; sourceId: string };
|
||||
|
||||
export interface ChangeSource {
|
||||
/** Changes since `cursor` (null = full enumeration), with the cursor to save. */
|
||||
getChanges(root: SourceRoot, cursor: string | null): Promise<{ changes: SourceChange[]; cursor: string }>;
|
||||
download(root: SourceRoot, sourceId: string): Promise<{ stream: ReadableStream<Uint8Array>; size: number }>;
|
||||
}
|
||||
|
||||
export interface TargetRef { id: string; isFolder: boolean }
|
||||
|
||||
export interface DocumentTarget {
|
||||
createFolder(parentId: string, name: string): Promise<TargetRef>;
|
||||
uploadFile(parentId: string, name: string, content: FileContent): Promise<TargetRef>;
|
||||
uploadVersion(ref: TargetRef, name: string, content: FileContent): Promise<void>;
|
||||
rename(ref: TargetRef, name: string): Promise<void>;
|
||||
/** May return a new ref if the target can only move by copy + delete. */
|
||||
move(ref: TargetRef, newParentId: string): Promise<TargetRef>;
|
||||
delete(ref: TargetRef): Promise<void>;
|
||||
/** Bring back a previously deleted item (keeps history where possible). */
|
||||
restore(ref: TargetRef, parentId: string, name: string, content?: FileContent): Promise<TargetRef>;
|
||||
}
|
||||
|
||||
export interface IgnorePolicy {
|
||||
isIgnored(name: string, isFolder: boolean): boolean;
|
||||
}
|
||||
```
|
||||
- `ChangeSource` (SharePoint): `getChanges(root, cursor)` returns changes, the next cursor and whether it was a full listing. `download(root, id)` returns the file content.
|
||||
- `DocumentTarget` (ACC): `createFolder`, `uploadFile`, `uploadVersion`, `rename`, `move`, `delete`, `restore`.
|
||||
- `IgnorePolicy`: decides which names are skipped.
|
||||
|
||||
Repositories are split by consumer (interface segregation):
|
||||
|
||||
- `ProjectRepository`: CRUD, cursor, pause and reset
|
||||
- `ItemRepository`: desired/synced state, pending diff query, tree query for the UI
|
||||
- `RunRepository`: start/finish runs, record events, prune old history
|
||||
- `ProjectRepository`: create, pause/resume, first-synced and last-run stamps, reset
|
||||
- `ItemRepository`: load a project's items, commit an ingest with its cursor (atomically), save reconcile results
|
||||
- `RunRepository`: start/finish runs, record events, list history, prune old history
|
||||
|
||||
Adapters throw `ServiceUnavailableError` for outages (network, auth, throttling that outlasted retries). The run stops early and no item is charged an attempt; any other error counts against the item that hit it.
|
||||
|
||||
Both fallback strategies, move-by-copy and restore-by-reupload, live **inside the ACC adapter**. They are ACC-specific details, so the engine never branches on them.
|
||||
|
||||
### Coordinator
|
||||
|
||||
`SyncCoordinator.requestRun(projectId, trigger)` is the only way a run starts, whether from the scheduler or from "Run now". It holds one in-process lock per project and returns one of:
|
||||
`SyncCoordinator` is the only way a run starts. It holds one in-process lock per project. `requestManualRun(projectId)` ("Run now") returns one of:
|
||||
|
||||
- `started`
|
||||
- `already-running` (the caller attaches to the current run)
|
||||
- `{ cooldown: secondsLeft }` (the last manual run was less than 30 s ago)
|
||||
- `already-running` (the page just shows the run in progress)
|
||||
- `cooldown` with `retryAfterSeconds` (the last manual request for this project was under 30 s ago)
|
||||
- `unavailable` with a reason (unknown, disabled or paused project)
|
||||
|
||||
`runScheduledPass()` is what the timer calls. An `IntervalScheduler` drives it, and the history-pruning job, without ever overlapping a task with itself.
|
||||
|
||||
The scheduler goes through enabled projects one at a time, and one failing project never blocks the others. Graph and APS clients share one HTTP layer that handles `429`/`Retry-After` and backoff, because rate limits apply per app, not per project.
|
||||
|
||||
@@ -145,7 +114,7 @@ items PK (project_id, source_id)
|
||||
-- desired (from SharePoint)
|
||||
parent_source_id, name, is_folder, content_tag, source_modified_at, size, deleted
|
||||
-- synced (what ACC has)
|
||||
target_id, synced_parent_target_id, synced_name, synced_content_tag, synced_at
|
||||
target_id, synced_parent_source_id, synced_name, synced_content_tag, synced_deleted, synced_at
|
||||
-- bookkeeping
|
||||
status ('synced' | 'pending' | 'failed' | 'deleted')
|
||||
attempt_count, last_error, last_attempt_at
|
||||
@@ -153,12 +122,15 @@ items PK (project_id, source_id)
|
||||
runs
|
||||
id, project_id, trigger ('schedule' | 'manual'), started_at, finished_at
|
||||
outcome ('success' | 'partial' | 'failed'), error
|
||||
created, updated, renamed, moved, deleted, failed -- counts
|
||||
created, updated, renamed, moved, deleted, restored, failed -- counts
|
||||
|
||||
run_events
|
||||
id, run_id, project_id, source_id, path, action, outcome, message, at
|
||||
id, run_id, project_id, source_id, path, action, message, at
|
||||
-- action: created | updated | renamed | moved | deleted | restored | failed
|
||||
```
|
||||
|
||||
The synced parent is stored as a SharePoint ID, so "was it moved?" is a plain comparison with the desired parent. The ACC folder to move into is looked up from the parent item when needed.
|
||||
|
||||
- IDs are stored, not paths, so renaming a root folder doesn't break a project. Display paths are shown in the UI only.
|
||||
- **Issues** = items with `status = 'failed'`, plus a project-level issue when it is paused or its latest run failed outright (bad auth, root missing).
|
||||
- A nightly job deletes `runs` and `run_events` older than `HISTORY_RETENTION_DAYS`. `items` is current state, so it is never pruned.
|
||||
@@ -168,13 +140,13 @@ run_events
|
||||
|
||||
## UI
|
||||
|
||||
| Route | Who | What |
|
||||
| --- | --- | --- |
|
||||
| `/` | anyone | Projects list: status, last run, open issue count |
|
||||
| `/projects/[id]` | anyone | File tree with status badges and last update time, an issues panel and a "Run now" button |
|
||||
| `/projects/[id]/history` | anyone | Runs from the last 7 days, expandable to per-file events |
|
||||
| `/admin/login` | — | Single admin login |
|
||||
| `/admin/projects/new`, `/admin/projects/[id]` | admin | Add/edit project (paste SharePoint URL and pick ACC folder; server resolves and checks the IDs), enable/disable, reset |
|
||||
| Route | Who | What |
|
||||
| --------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------- |
|
||||
| `/` | anyone | Projects list: status, last run, open issue count |
|
||||
| `/projects/[id]` | anyone | File tree with status badges and last update time, an issues panel and a "Run now" button |
|
||||
| `/projects/[id]/history` | anyone | Runs from the last 7 days, expandable to per-file events |
|
||||
| `/admin/login` | — | Single admin login |
|
||||
| `/admin/projects/new`, `/admin/projects/[id]` | admin | Add/edit project (paste SharePoint URL and pick ACC folder; server resolves and checks the IDs), enable/disable, reset |
|
||||
|
||||
- The status pages read only from SQLite. They never call SharePoint or ACC.
|
||||
- While a run is active, the project page refreshes its data every few seconds.
|
||||
@@ -192,25 +164,34 @@ MAX_ATTEMPTS=3
|
||||
HISTORY_RETENTION_DAYS=7
|
||||
```
|
||||
|
||||
Env is validated with `zod` at startup; the server refuses to start on bad config. Run exactly **one** replica, because the locks live in-process.
|
||||
Env vars are declared and validated with `zod` in `src/env.ts` (SvelteKit's `defineEnvVars`); the server refuses to start on bad config. Only the variables in use so far are declared; the Graph, APS and admin ones get added along with the code that reads them. Run exactly **one** replica, because the locks live in-process.
|
||||
|
||||
## Source layout
|
||||
|
||||
```
|
||||
src/env.ts # env var declarations + validation
|
||||
src/lib/server/
|
||||
config.ts # zod-validated env
|
||||
container.ts # composition root: the only place adapters are chosen
|
||||
container.ts # (todo) composition root: the only place adapters are chosen
|
||||
sync/
|
||||
core/ # ports, SyncEngine, reconcile planner, IgnorePolicy (pure TS)
|
||||
coordinator.ts # locks, cooldown, scheduler loop, retention job
|
||||
core/ # pure TS, no SvelteKit imports
|
||||
model.ts ports.ts errors.ts
|
||||
ingest.ts # SharePoint changes → desired state
|
||||
plan.ts # desired vs synced → action, status, work order
|
||||
reconciler.ts # applies actions to ACC
|
||||
engine.ts # one run of one project
|
||||
tree.ts ignore.ts
|
||||
coordinator.ts # per-project lock + manual cooldown
|
||||
scheduler.ts # non-overlapping interval tasks
|
||||
testing/ # FakeDrive, FakeAcc, harness (real engine + in-memory SQLite)
|
||||
adapters/
|
||||
graph/ # GraphChangeSource, auth (msal-node)
|
||||
aps/ # ApsDocumentTarget, auth, upload flow
|
||||
sqlite/ # Drizzle schema + repositories
|
||||
http/ # shared fetch wrapper: retry, backoff, Retry-After
|
||||
auth/ # admin session cookie
|
||||
src/hooks.server.ts # init → container → coordinator.start(); admin route guard
|
||||
src/routes/ # UI
|
||||
sqlite/ # Drizzle schema, migrations runner, repositories
|
||||
graph/ # (todo) GraphChangeSource, auth (msal-node)
|
||||
aps/ # (todo) ApsDocumentTarget, auth, upload flow
|
||||
http/ # (todo) shared fetch wrapper: retry, backoff, Retry-After
|
||||
auth/ # (todo) admin session cookie
|
||||
drizzle/ # generated SQL migrations (npm run db:generate)
|
||||
src/hooks.server.ts # (todo) init → container → scheduler; admin route guard
|
||||
src/routes/ # (todo) UI
|
||||
```
|
||||
|
||||
## Open questions (resolved by spikes)
|
||||
@@ -224,11 +205,11 @@ src/routes/ # UI
|
||||
|
||||
0. **Access requests:** start now; they're the slowest part. See [access-setup.md](access-setup.md).
|
||||
1. **Spikes** against a sandbox ACC project and test SharePoint site. Answer the open questions above and record the results here.
|
||||
2. **Scaffold:** SvelteKit + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier, Dockerfile.
|
||||
3. **Core:** ports, IgnorePolicy, reconcile planner and SyncEngine, unit-tested against in-memory fakes. Most of the logic lives here.
|
||||
4. **SQLite repositories** and migrations.
|
||||
2. ✅ **Scaffold:** SvelteKit 3 + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier. (Dockerfile moves to step 9.)
|
||||
3. ✅ **Core:** ports, IgnorePolicy, ingest, planner, reconciler and SyncEngine, tested against FakeDrive/FakeAcc.
|
||||
4. ✅ **SQLite repositories** and migrations.
|
||||
5. **GraphChangeSource.**
|
||||
6. **ApsDocumentTarget**, including the fallbacks chosen in step 1.
|
||||
7. **Coordinator:** scheduler, locks, cooldown, retention job, hooked into `init`.
|
||||
7. **Wiring:** ✅ coordinator, scheduler. Still to do: composition root, `init` hook, retention job schedule.
|
||||
8. **UI:** status pages, Run now, admin login, project add/edit/reset.
|
||||
9. **Ship:** Docker image, Portainer stack file, runbook.
|
||||
|
||||
Reference in New Issue
Block a user