Files
sharepoint-acc-sync/docs/design.md
2026-10-22 22:05:33 -05:00

12 KiB

Design

Goals and constraints

Decision Choice
Direction One-way, SharePoint → ACC
Latency Poll every ~3 min (configurable); "Run now" for immediate sync
Operations Create, update (new ACC version), rename, move, delete
Scope Many projects per server; each project is a 1:1 pair of SharePoint root ↔ ACC root
ACC ownership Only touch ACC items this tool created; hand-added ACC content is left alone
Ignored files Temp/lock/system files are skipped silently (not issues)
Retries 3 attempts across runs, then surface as an issue
Files Small/medium documents; no large Revit models
Hosting One Docker container (Portainer), SQLite on a volume
Credentials One Microsoft app registration + one APS app, shared by all projects
UI access Viewing and "Run now" are open (behind VPN). Adding, editing or resetting projects requires the single admin login from env

Architecture

The sync logic is plain TypeScript with no SvelteKit imports. SvelteKit hosts it: it starts the scheduler from the init hook and serves the status UI. The core depends only on interfaces (ports); SharePoint, ACC and SQLite are adapters plugged in at one composition root.

           ┌──────────────── SvelteKit (adapter-node) ────────────────┐
           │  routes/ (status UI, Run now, admin)   hooks.server.ts   │
           │              │                         (init → start)    │
           │              ▼                               │           │
           │      SyncCoordinator  ◄── scheduler tick ────┘           │
           │   (per-project lock + cooldown)                          │
           │              │                                           │
           │              ▼                                           │
           │         SyncEngine ── IgnorePolicy                       │
           │       ┌──────┼───────────────┐                           │
           │       ▼      ▼               ▼                           │
           │ ChangeSource  Repositories   DocumentTarget              │
           │   (Graph)      (SQLite)        (APS / ACC)               │
           └──────────────────────────────────────────────────────────┘

Each run has two phases

  1. Ingest. Read changes from SharePoint since the project's saved cursor and write them to items as the desired state (name, parent, content tag, deleted flag). Save the new cursor in the same transaction. Nothing touches ACC in this phase.
  2. Reconcile. For every item whose desired state differs from its synced state, work out the operation and apply it to ACC. On success, copy desired → synced. On failure, increment attempt_count and record the error.

This split gives us several things without extra code:

  • Retries. Failed items still differ, so the next run picks them up again. No separate retry queue is needed.
  • Crash safety. If the container dies mid-run, the cursor is already saved and the pending work is still in the database.
  • Inferred renames and moves. Graph delta only reports "this item now looks like X". The operation comes from comparing that to what we last synced:
Desired vs synced Operation
no target id yet create (folder) / upload (file)
deleted delete
name differs rename
parent differs move
content tag differs upload new version

A single change can need more than one of these, for example a move plus a rename.

Order within reconcile: folder creates and moves from shallowest to deepest, then files, then deletes. That way a destination folder always exists before anything moves into it.

Scope and delta queries

Graph delta for SharePoint only works at the drive root, not on subfolders. So each project runs delta on its whole drive and keeps only items under the project's root folder. It checks this with the parent chain stored in items.

An item whose new parent is outside the root counts as deleted. That covers "removed from the root = deleted in ACC".

Safety rule: if the project's root folder itself disappears or moves, the project is paused with a project-level issue and nothing is deleted. That is almost always a mistake, not an intent to wipe ACC.

Interfaces (ports)

These are sketches; final signatures get settled during the build.

// sync/core/ports.ts

export interface SourceItem {
  sourceId: string;
  parentSourceId: string | null;
  name: string;
  isFolder: boolean;
  contentTag: string | null;   // Graph cTag; changes only when content changes
  modifiedAt: Date;
  size: number | null;
}

export type SourceChange =
  | { kind: 'upsert'; item: SourceItem }
  | { kind: 'delete'; sourceId: string };

export interface ChangeSource {
  /** Changes since `cursor` (null = full enumeration), with the cursor to save. */
  getChanges(root: SourceRoot, cursor: string | null): Promise<{ changes: SourceChange[]; cursor: string }>;
  download(root: SourceRoot, sourceId: string): Promise<{ stream: ReadableStream<Uint8Array>; size: number }>;
}

export interface TargetRef { id: string; isFolder: boolean }

export interface DocumentTarget {
  createFolder(parentId: string, name: string): Promise<TargetRef>;
  uploadFile(parentId: string, name: string, content: FileContent): Promise<TargetRef>;
  uploadVersion(ref: TargetRef, name: string, content: FileContent): Promise<void>;
  rename(ref: TargetRef, name: string): Promise<void>;
  /** May return a new ref if the target can only move by copy + delete. */
  move(ref: TargetRef, newParentId: string): Promise<TargetRef>;
  delete(ref: TargetRef): Promise<void>;
  /** Bring back a previously deleted item (keeps history where possible). */
  restore(ref: TargetRef, parentId: string, name: string, content?: FileContent): Promise<TargetRef>;
}

export interface IgnorePolicy {
  isIgnored(name: string, isFolder: boolean): boolean;
}

Repositories are split by consumer (interface segregation):

  • ProjectRepository: CRUD, cursor, pause and reset
  • ItemRepository: desired/synced state, pending diff query, tree query for the UI
  • RunRepository: start/finish runs, record events, prune old history

Both fallback strategies, move-by-copy and restore-by-reupload, live inside the ACC adapter. They are ACC-specific details, so the engine never branches on them.

Coordinator

SyncCoordinator.requestRun(projectId, trigger) is the only way a run starts, whether from the scheduler or from "Run now". It holds one in-process lock per project and returns one of:

  • started
  • already-running (the caller attaches to the current run)
  • { cooldown: secondsLeft } (the last manual run was less than 30 s ago)

The scheduler goes through enabled projects one at a time, and one failing project never blocks the others. Graph and APS clients share one HTTP layer that handles 429/Retry-After and backoff, because rate limits apply per app, not per project.

Data model (SQLite via Drizzle)

projects
  id, name, enabled, paused_reason
  sp_site_id, sp_drive_id, sp_root_item_id, sp_display_path
  acc_hub_id, acc_project_id, acc_root_folder_id, acc_display_path
  delta_cursor, first_synced_at, last_run_at, created_at

items                                   PK (project_id, source_id)
  -- desired (from SharePoint)
  parent_source_id, name, is_folder, content_tag, source_modified_at, size, deleted
  -- synced (what ACC has)
  target_id, synced_parent_target_id, synced_name, synced_content_tag, synced_at
  -- bookkeeping
  status  ('synced' | 'pending' | 'failed' | 'deleted')
  attempt_count, last_error, last_attempt_at

runs
  id, project_id, trigger ('schedule' | 'manual'), started_at, finished_at
  outcome ('success' | 'partial' | 'failed'), error
  created, updated, renamed, moved, deleted, failed   -- counts

run_events
  id, run_id, project_id, source_id, path, action, outcome, message, at
  • IDs are stored, not paths, so renaming a root folder doesn't break a project. Display paths are shown in the UI only.
  • Issues = items with status = 'failed', plus a project-level issue when it is paused or its latest run failed outright (bad auth, root missing).
  • A nightly job deletes runs and run_events older than HISTORY_RETENTION_DAYS. items is current state, so it is never pruned.
  • After a failed item has used up its attempts, it is only retried when its content or location changes again, or when someone clicks retry.
  • Deleted items keep their target_id (a tombstone). That lets a SharePoint restore revive the same ACC item instead of creating a new one.
  • A project's roots are locked once first_synced_at is set. Changing them requires an explicit reset, which clears the project's items and cursor.

UI

Route Who What
/ anyone Projects list: status, last run, open issue count
/projects/[id] anyone File tree with status badges and last update time, an issues panel and a "Run now" button
/projects/[id]/history anyone Runs from the last 7 days, expandable to per-file events
/admin/login — Single admin login
/admin/projects/new, /admin/projects/[id] admin Add/edit project (paste SharePoint URL and pick ACC folder; server resolves and checks the IDs), enable/disable, reset
  • The status pages read only from SQLite. They never call SharePoint or ACC.
  • While a run is active, the project page refreshes its data every few seconds.

Configuration (env)

GRAPH_TENANT_ID, GRAPH_CLIENT_ID, GRAPH_CLIENT_SECRET
APS_CLIENT_ID, APS_CLIENT_SECRET
ADMIN_USERNAME, ADMIN_PASSWORD, SESSION_SECRET
DATABASE_PATH=/data/sync.db
POLL_INTERVAL_SECONDS=180
MANUAL_RUN_COOLDOWN_SECONDS=30
MAX_ATTEMPTS=3
HISTORY_RETENTION_DAYS=7

Env is validated with zod at startup; the server refuses to start on bad config. Run exactly one replica, because the locks live in-process.

Source layout

src/lib/server/
  config.ts                 # zod-validated env
  container.ts              # composition root: the only place adapters are chosen
  sync/
    core/                   # ports, SyncEngine, reconcile planner, IgnorePolicy (pure TS)
    coordinator.ts          # locks, cooldown, scheduler loop, retention job
  adapters/
    graph/                  # GraphChangeSource, auth (msal-node)
    aps/                    # ApsDocumentTarget, auth, upload flow
    sqlite/                 # Drizzle schema + repositories
    http/                   # shared fetch wrapper: retry, backoff, Retry-After
  auth/                     # admin session cookie
src/hooks.server.ts         # init → container → coordinator.start(); admin route guard
src/routes/                 # UI

Open questions (resolved by spikes)

  1. Move a file in ACC natively? The APS docs show folder moves via PATCH folders (changing the parent). Single-file moves are not clearly documented. If not supported, the fallback is copy then delete, which may lose ACC version history.
  2. Revive a deleted ACC item? Can we add a version to, or restore, a deleted item through the API so that history and links are kept? If not, the fallback is to upload a new item.
  3. App-only auth for ACC writes: whether some ACC Docs operations need an x-user-id (a service user that is a member of each project).
  4. Graph delta with a subfolder root: confirm the drive-wide filtering approach and how folder deletes report their children.

Build order

  1. Access requests: start now; they're the slowest part. See access-setup.md.
  2. Spikes against a sandbox ACC project and test SharePoint site. Answer the open questions above and record the results here.
  3. Scaffold: SvelteKit + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier, Dockerfile.
  4. Core: ports, IgnorePolicy, reconcile planner and SyncEngine, unit-tested against in-memory fakes. Most of the logic lives here.
  5. SQLite repositories and migrations.
  6. GraphChangeSource.
  7. ApsDocumentTarget, including the fallbacks chosen in step 1.
  8. Coordinator: scheduler, locks, cooldown, retention job, hooked into init.
  9. UI: status pages, Run now, admin login, project add/edit/reset.
  10. Ship: Docker image, Portainer stack file, runbook.