first draft
This commit is contained in:
5
.env.example
Normal file
5
.env.example
Normal file
@@ -0,0 +1,5 @@
|
||||
DATABASE_PATH=local.db
|
||||
POLL_INTERVAL_SECONDS=180
|
||||
MANUAL_RUN_COOLDOWN_SECONDS=30
|
||||
MAX_ATTEMPTS=3
|
||||
HISTORY_RETENTION_DAYS=7
|
||||
25
.gitignore
vendored
Normal file
25
.gitignore
vendored
Normal file
@@ -0,0 +1,25 @@
|
||||
node_modules
|
||||
|
||||
# Output
|
||||
.output
|
||||
.vercel
|
||||
.netlify
|
||||
.wrangler
|
||||
/.svelte-kit
|
||||
/build
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Env
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
!.env.test
|
||||
|
||||
# Vite
|
||||
vite.config.js.timestamp-*
|
||||
vite.config.ts.timestamp-*
|
||||
# SQLite
|
||||
*.db
|
||||
10
.prettierignore
Normal file
10
.prettierignore
Normal file
@@ -0,0 +1,10 @@
|
||||
# Package Managers
|
||||
package-lock.json
|
||||
pnpm-lock.yaml
|
||||
yarn.lock
|
||||
bun.lock
|
||||
bun.lockb
|
||||
|
||||
# Miscellaneous
|
||||
/static/
|
||||
/drizzle/
|
||||
3
.vscode/extensions.json
vendored
Normal file
3
.vscode/extensions.json
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"recommendations": ["svelte.svelte-vscode", "esbenp.prettier-vscode", "dbaeumer.vscode-eslint"]
|
||||
}
|
||||
@@ -14,7 +14,7 @@ There is also a small web page that shows each project's file tree, sync status,
|
||||
|
||||
## Status
|
||||
|
||||
Planning. See:
|
||||
In development. The sync core and database layer are built and tested against simulated SharePoint and ACC. The real SharePoint/ACC connections and the web UI are next. See:
|
||||
|
||||
- [docs/design.md](docs/design.md): architecture, interfaces, data model and build order
|
||||
- [docs/access-setup.md](docs/access-setup.md): what IT and the ACC admin need to set up
|
||||
|
||||
@@ -19,13 +19,13 @@ The tool only ever reads from SharePoint, so no write permission is needed.
|
||||
|
||||
## Autodesk / ACC (ask the ACC account admin)
|
||||
|
||||
1. **Create an APS app** at <https://aps.autodesk.com> → *My Apps*.
|
||||
1. **Create an APS app** at <https://aps.autodesk.com> → _My Apps_.
|
||||
- App type: **Server-to-Server** (app-only).
|
||||
- API access: **Data Management API**.
|
||||
- Send us the **Client ID** and **Client Secret**.
|
||||
2. **In ACC Account Admin → Custom Integrations, add the APS app's Client ID** so it can access the account's projects.
|
||||
3. **Possibly: a service user.** Some ACC Docs operations may need to run "as" a user, i.e. an account that is a project member with edit/upload permission on the target folder. We'll confirm this during the build spikes before asking for it.
|
||||
4. **For each project:** tell us which ACC project, and which folder under *Project Files*, is the sync root.
|
||||
4. **For each project:** tell us which ACC project, and which folder under _Project Files_, is the sync root.
|
||||
|
||||
## For the build spikes
|
||||
|
||||
|
||||
129
docs/design.md
129
docs/design.md
@@ -3,7 +3,7 @@
|
||||
## Goals and constraints
|
||||
|
||||
| Decision | Choice |
|
||||
| --- | --- |
|
||||
| ------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Direction | One-way, SharePoint → ACC |
|
||||
| Latency | Poll every ~3 min (configurable); "Run now" for immediate sync |
|
||||
| Operations | Create, update (new ACC version), rename, move, delete |
|
||||
@@ -39,8 +39,8 @@ The sync logic is plain TypeScript with no SvelteKit imports. SvelteKit hosts it
|
||||
|
||||
### Each run has two phases
|
||||
|
||||
1. **Ingest.** Read changes from SharePoint since the project's saved cursor and write them to `items` as the *desired* state (name, parent, content tag, deleted flag). Save the new cursor in the same transaction. Nothing touches ACC in this phase.
|
||||
2. **Reconcile.** For every item whose desired state differs from its *synced* state, work out the operation and apply it to ACC. On success, copy desired → synced. On failure, increment `attempt_count` and record the error.
|
||||
1. **Ingest.** Read changes from SharePoint since the project's saved cursor and write them to `items` as the _desired_ state (name, parent, content tag, deleted flag). Save the new cursor in the same transaction. Nothing touches ACC in this phase.
|
||||
2. **Reconcile.** For every item whose desired state differs from its _synced_ state, work out the operation and apply it to ACC. On success, copy desired → synced. On failure, increment `attempt_count` and record the error.
|
||||
|
||||
This split gives us several things without extra code:
|
||||
|
||||
@@ -49,14 +49,14 @@ This split gives us several things without extra code:
|
||||
- **Inferred renames and moves.** Graph delta only reports "this item now looks like X". The operation comes from comparing that to what we last synced:
|
||||
|
||||
| Desired vs synced | Operation |
|
||||
| --- | --- |
|
||||
| ------------------- | ------------------------------- |
|
||||
| no target id yet | create (folder) / upload (file) |
|
||||
| `deleted` | delete |
|
||||
| name differs | rename |
|
||||
| parent differs | move |
|
||||
| content tag differs | upload new version |
|
||||
|
||||
A single change can need more than one of these, for example a move plus a rename.
|
||||
A single change can need more than one of these, for example a move plus a rename.
|
||||
|
||||
**Order within reconcile:** folder creates and moves from shallowest to deepest, then files, then deletes. That way a destination folder always exists before anything moves into it.
|
||||
|
||||
@@ -66,69 +66,38 @@ Graph delta for SharePoint only works at the **drive root**, not on subfolders.
|
||||
|
||||
An item whose new parent is outside the root counts as deleted. That covers "removed from the root = deleted in ACC".
|
||||
|
||||
**Safety rule:** if the project's root folder itself disappears or moves, the project is paused with a project-level issue and **nothing is deleted**. That is almost always a mistake, not an intent to wipe ACC.
|
||||
**Safety rule:** if the project's root folder is deleted, or is missing from a full listing, the project is paused with a project-level issue and **nothing is deleted**. That is almost always a mistake, not an intent to wipe ACC. Moving or renaming the root within the same library is harmless, because projects track it by ID.
|
||||
|
||||
**Restored folders:** a folder delete cascades "deleted" to everything stored under it. If a deleted folder later reappears, its contents may not be reported one by one, so ingest asks for one follow-up **full listing** in the same run, which brings them back.
|
||||
|
||||
## Interfaces (ports)
|
||||
|
||||
These are sketches; final signatures get settled during the build.
|
||||
The source of truth is [`src/lib/server/sync/core/ports.ts`](../src/lib/server/sync/core/ports.ts):
|
||||
|
||||
```ts
|
||||
// sync/core/ports.ts
|
||||
|
||||
export interface SourceItem {
|
||||
sourceId: string;
|
||||
parentSourceId: string | null;
|
||||
name: string;
|
||||
isFolder: boolean;
|
||||
contentTag: string | null; // Graph cTag; changes only when content changes
|
||||
modifiedAt: Date;
|
||||
size: number | null;
|
||||
}
|
||||
|
||||
export type SourceChange =
|
||||
| { kind: 'upsert'; item: SourceItem }
|
||||
| { kind: 'delete'; sourceId: string };
|
||||
|
||||
export interface ChangeSource {
|
||||
/** Changes since `cursor` (null = full enumeration), with the cursor to save. */
|
||||
getChanges(root: SourceRoot, cursor: string | null): Promise<{ changes: SourceChange[]; cursor: string }>;
|
||||
download(root: SourceRoot, sourceId: string): Promise<{ stream: ReadableStream<Uint8Array>; size: number }>;
|
||||
}
|
||||
|
||||
export interface TargetRef { id: string; isFolder: boolean }
|
||||
|
||||
export interface DocumentTarget {
|
||||
createFolder(parentId: string, name: string): Promise<TargetRef>;
|
||||
uploadFile(parentId: string, name: string, content: FileContent): Promise<TargetRef>;
|
||||
uploadVersion(ref: TargetRef, name: string, content: FileContent): Promise<void>;
|
||||
rename(ref: TargetRef, name: string): Promise<void>;
|
||||
/** May return a new ref if the target can only move by copy + delete. */
|
||||
move(ref: TargetRef, newParentId: string): Promise<TargetRef>;
|
||||
delete(ref: TargetRef): Promise<void>;
|
||||
/** Bring back a previously deleted item (keeps history where possible). */
|
||||
restore(ref: TargetRef, parentId: string, name: string, content?: FileContent): Promise<TargetRef>;
|
||||
}
|
||||
|
||||
export interface IgnorePolicy {
|
||||
isIgnored(name: string, isFolder: boolean): boolean;
|
||||
}
|
||||
```
|
||||
- `ChangeSource` (SharePoint): `getChanges(root, cursor)` returns changes, the next cursor and whether it was a full listing. `download(root, id)` returns the file content.
|
||||
- `DocumentTarget` (ACC): `createFolder`, `uploadFile`, `uploadVersion`, `rename`, `move`, `delete`, `restore`.
|
||||
- `IgnorePolicy`: decides which names are skipped.
|
||||
|
||||
Repositories are split by consumer (interface segregation):
|
||||
|
||||
- `ProjectRepository`: CRUD, cursor, pause and reset
|
||||
- `ItemRepository`: desired/synced state, pending diff query, tree query for the UI
|
||||
- `RunRepository`: start/finish runs, record events, prune old history
|
||||
- `ProjectRepository`: create, pause/resume, first-synced and last-run stamps, reset
|
||||
- `ItemRepository`: load a project's items, commit an ingest with its cursor (atomically), save reconcile results
|
||||
- `RunRepository`: start/finish runs, record events, list history, prune old history
|
||||
|
||||
Adapters throw `ServiceUnavailableError` for outages (network, auth, throttling that outlasted retries). The run stops early and no item is charged an attempt; any other error counts against the item that hit it.
|
||||
|
||||
Both fallback strategies, move-by-copy and restore-by-reupload, live **inside the ACC adapter**. They are ACC-specific details, so the engine never branches on them.
|
||||
|
||||
### Coordinator
|
||||
|
||||
`SyncCoordinator.requestRun(projectId, trigger)` is the only way a run starts, whether from the scheduler or from "Run now". It holds one in-process lock per project and returns one of:
|
||||
`SyncCoordinator` is the only way a run starts. It holds one in-process lock per project. `requestManualRun(projectId)` ("Run now") returns one of:
|
||||
|
||||
- `started`
|
||||
- `already-running` (the caller attaches to the current run)
|
||||
- `{ cooldown: secondsLeft }` (the last manual run was less than 30 s ago)
|
||||
- `already-running` (the page just shows the run in progress)
|
||||
- `cooldown` with `retryAfterSeconds` (the last manual request for this project was under 30 s ago)
|
||||
- `unavailable` with a reason (unknown, disabled or paused project)
|
||||
|
||||
`runScheduledPass()` is what the timer calls. An `IntervalScheduler` drives it, and the history-pruning job, without ever overlapping a task with itself.
|
||||
|
||||
The scheduler goes through enabled projects one at a time, and one failing project never blocks the others. Graph and APS clients share one HTTP layer that handles `429`/`Retry-After` and backoff, because rate limits apply per app, not per project.
|
||||
|
||||
@@ -145,7 +114,7 @@ items PK (project_id, source_id)
|
||||
-- desired (from SharePoint)
|
||||
parent_source_id, name, is_folder, content_tag, source_modified_at, size, deleted
|
||||
-- synced (what ACC has)
|
||||
target_id, synced_parent_target_id, synced_name, synced_content_tag, synced_at
|
||||
target_id, synced_parent_source_id, synced_name, synced_content_tag, synced_deleted, synced_at
|
||||
-- bookkeeping
|
||||
status ('synced' | 'pending' | 'failed' | 'deleted')
|
||||
attempt_count, last_error, last_attempt_at
|
||||
@@ -153,12 +122,15 @@ items PK (project_id, source_id)
|
||||
runs
|
||||
id, project_id, trigger ('schedule' | 'manual'), started_at, finished_at
|
||||
outcome ('success' | 'partial' | 'failed'), error
|
||||
created, updated, renamed, moved, deleted, failed -- counts
|
||||
created, updated, renamed, moved, deleted, restored, failed -- counts
|
||||
|
||||
run_events
|
||||
id, run_id, project_id, source_id, path, action, outcome, message, at
|
||||
id, run_id, project_id, source_id, path, action, message, at
|
||||
-- action: created | updated | renamed | moved | deleted | restored | failed
|
||||
```
|
||||
|
||||
The synced parent is stored as a SharePoint ID, so "was it moved?" is a plain comparison with the desired parent. The ACC folder to move into is looked up from the parent item when needed.
|
||||
|
||||
- IDs are stored, not paths, so renaming a root folder doesn't break a project. Display paths are shown in the UI only.
|
||||
- **Issues** = items with `status = 'failed'`, plus a project-level issue when it is paused or its latest run failed outright (bad auth, root missing).
|
||||
- A nightly job deletes `runs` and `run_events` older than `HISTORY_RETENTION_DAYS`. `items` is current state, so it is never pruned.
|
||||
@@ -169,7 +141,7 @@ run_events
|
||||
## UI
|
||||
|
||||
| Route | Who | What |
|
||||
| --- | --- | --- |
|
||||
| --------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------- |
|
||||
| `/` | anyone | Projects list: status, last run, open issue count |
|
||||
| `/projects/[id]` | anyone | File tree with status badges and last update time, an issues panel and a "Run now" button |
|
||||
| `/projects/[id]/history` | anyone | Runs from the last 7 days, expandable to per-file events |
|
||||
@@ -192,25 +164,34 @@ MAX_ATTEMPTS=3
|
||||
HISTORY_RETENTION_DAYS=7
|
||||
```
|
||||
|
||||
Env is validated with `zod` at startup; the server refuses to start on bad config. Run exactly **one** replica, because the locks live in-process.
|
||||
Env vars are declared and validated with `zod` in `src/env.ts` (SvelteKit's `defineEnvVars`); the server refuses to start on bad config. Only the variables in use so far are declared; the Graph, APS and admin ones get added along with the code that reads them. Run exactly **one** replica, because the locks live in-process.
|
||||
|
||||
## Source layout
|
||||
|
||||
```
|
||||
src/env.ts # env var declarations + validation
|
||||
src/lib/server/
|
||||
config.ts # zod-validated env
|
||||
container.ts # composition root: the only place adapters are chosen
|
||||
container.ts # (todo) composition root: the only place adapters are chosen
|
||||
sync/
|
||||
core/ # ports, SyncEngine, reconcile planner, IgnorePolicy (pure TS)
|
||||
coordinator.ts # locks, cooldown, scheduler loop, retention job
|
||||
core/ # pure TS, no SvelteKit imports
|
||||
model.ts ports.ts errors.ts
|
||||
ingest.ts # SharePoint changes → desired state
|
||||
plan.ts # desired vs synced → action, status, work order
|
||||
reconciler.ts # applies actions to ACC
|
||||
engine.ts # one run of one project
|
||||
tree.ts ignore.ts
|
||||
coordinator.ts # per-project lock + manual cooldown
|
||||
scheduler.ts # non-overlapping interval tasks
|
||||
testing/ # FakeDrive, FakeAcc, harness (real engine + in-memory SQLite)
|
||||
adapters/
|
||||
graph/ # GraphChangeSource, auth (msal-node)
|
||||
aps/ # ApsDocumentTarget, auth, upload flow
|
||||
sqlite/ # Drizzle schema + repositories
|
||||
http/ # shared fetch wrapper: retry, backoff, Retry-After
|
||||
auth/ # admin session cookie
|
||||
src/hooks.server.ts # init → container → coordinator.start(); admin route guard
|
||||
src/routes/ # UI
|
||||
sqlite/ # Drizzle schema, migrations runner, repositories
|
||||
graph/ # (todo) GraphChangeSource, auth (msal-node)
|
||||
aps/ # (todo) ApsDocumentTarget, auth, upload flow
|
||||
http/ # (todo) shared fetch wrapper: retry, backoff, Retry-After
|
||||
auth/ # (todo) admin session cookie
|
||||
drizzle/ # generated SQL migrations (npm run db:generate)
|
||||
src/hooks.server.ts # (todo) init → container → scheduler; admin route guard
|
||||
src/routes/ # (todo) UI
|
||||
```
|
||||
|
||||
## Open questions (resolved by spikes)
|
||||
@@ -224,11 +205,11 @@ src/routes/ # UI
|
||||
|
||||
0. **Access requests:** start now; they're the slowest part. See [access-setup.md](access-setup.md).
|
||||
1. **Spikes** against a sandbox ACC project and test SharePoint site. Answer the open questions above and record the results here.
|
||||
2. **Scaffold:** SvelteKit + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier, Dockerfile.
|
||||
3. **Core:** ports, IgnorePolicy, reconcile planner and SyncEngine, unit-tested against in-memory fakes. Most of the logic lives here.
|
||||
4. **SQLite repositories** and migrations.
|
||||
2. ✅ **Scaffold:** SvelteKit 3 + adapter-node, TypeScript strict, Drizzle + better-sqlite3, zod, Vitest, ESLint/Prettier. (Dockerfile moves to step 9.)
|
||||
3. ✅ **Core:** ports, IgnorePolicy, ingest, planner, reconciler and SyncEngine, tested against FakeDrive/FakeAcc.
|
||||
4. ✅ **SQLite repositories** and migrations.
|
||||
5. **GraphChangeSource.**
|
||||
6. **ApsDocumentTarget**, including the fallbacks chosen in step 1.
|
||||
7. **Coordinator:** scheduler, locks, cooldown, retention job, hooked into `init`.
|
||||
7. **Wiring:** ✅ coordinator, scheduler. Still to do: composition root, `init` hook, retention job schedule.
|
||||
8. **UI:** status pages, Run now, admin login, project add/edit/reset.
|
||||
9. **Ship:** Docker image, Portainer stack file, runbook.
|
||||
|
||||
10
drizzle.config.ts
Normal file
10
drizzle.config.ts
Normal file
@@ -0,0 +1,10 @@
|
||||
import { defineConfig } from 'drizzle-kit';
|
||||
|
||||
export default defineConfig({
|
||||
schema: './src/lib/server/adapters/sqlite/schema.ts',
|
||||
out: './drizzle',
|
||||
dialect: 'sqlite',
|
||||
dbCredentials: { url: process.env.DATABASE_PATH ?? 'local.db' },
|
||||
verbose: true,
|
||||
strict: true
|
||||
});
|
||||
76
drizzle/0000_init.sql
Normal file
76
drizzle/0000_init.sql
Normal file
@@ -0,0 +1,76 @@
|
||||
CREATE TABLE `items` (
|
||||
`project_id` text NOT NULL,
|
||||
`source_id` text NOT NULL,
|
||||
`parent_source_id` text NOT NULL,
|
||||
`name` text NOT NULL,
|
||||
`is_folder` integer NOT NULL,
|
||||
`content_tag` text,
|
||||
`source_modified_at` integer NOT NULL,
|
||||
`size` integer,
|
||||
`deleted` integer NOT NULL,
|
||||
`target_id` text,
|
||||
`synced_parent_source_id` text,
|
||||
`synced_name` text,
|
||||
`synced_content_tag` text,
|
||||
`synced_deleted` integer NOT NULL,
|
||||
`synced_at` integer,
|
||||
`status` text NOT NULL,
|
||||
`attempt_count` integer DEFAULT 0 NOT NULL,
|
||||
`last_error` text,
|
||||
`last_attempt_at` integer,
|
||||
PRIMARY KEY(`project_id`, `source_id`),
|
||||
FOREIGN KEY (`project_id`) REFERENCES `projects`(`id`) ON UPDATE no action ON DELETE cascade
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE INDEX `items_project_status` ON `items` (`project_id`,`status`);--> statement-breakpoint
|
||||
CREATE TABLE `projects` (
|
||||
`id` text PRIMARY KEY NOT NULL,
|
||||
`name` text NOT NULL,
|
||||
`enabled` integer DEFAULT true NOT NULL,
|
||||
`paused_reason` text,
|
||||
`sp_site_id` text NOT NULL,
|
||||
`sp_drive_id` text NOT NULL,
|
||||
`sp_root_item_id` text NOT NULL,
|
||||
`sp_display_path` text NOT NULL,
|
||||
`acc_hub_id` text NOT NULL,
|
||||
`acc_project_id` text NOT NULL,
|
||||
`acc_root_folder_id` text NOT NULL,
|
||||
`acc_display_path` text NOT NULL,
|
||||
`delta_cursor` text,
|
||||
`first_synced_at` integer,
|
||||
`last_run_at` integer,
|
||||
`created_at` integer NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE `run_events` (
|
||||
`id` integer PRIMARY KEY AUTOINCREMENT NOT NULL,
|
||||
`run_id` text NOT NULL,
|
||||
`project_id` text NOT NULL,
|
||||
`source_id` text NOT NULL,
|
||||
`path` text NOT NULL,
|
||||
`action` text NOT NULL,
|
||||
`message` text,
|
||||
`at` integer NOT NULL,
|
||||
FOREIGN KEY (`run_id`) REFERENCES `runs`(`id`) ON UPDATE no action ON DELETE cascade
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE INDEX `run_events_run` ON `run_events` (`run_id`);--> statement-breakpoint
|
||||
CREATE TABLE `runs` (
|
||||
`id` text PRIMARY KEY NOT NULL,
|
||||
`project_id` text NOT NULL,
|
||||
`trigger` text NOT NULL,
|
||||
`started_at` integer NOT NULL,
|
||||
`finished_at` integer,
|
||||
`outcome` text,
|
||||
`error` text,
|
||||
`created` integer DEFAULT 0 NOT NULL,
|
||||
`updated` integer DEFAULT 0 NOT NULL,
|
||||
`renamed` integer DEFAULT 0 NOT NULL,
|
||||
`moved` integer DEFAULT 0 NOT NULL,
|
||||
`deleted` integer DEFAULT 0 NOT NULL,
|
||||
`restored` integer DEFAULT 0 NOT NULL,
|
||||
`failed` integer DEFAULT 0 NOT NULL,
|
||||
FOREIGN KEY (`project_id`) REFERENCES `projects`(`id`) ON UPDATE no action ON DELETE cascade
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE INDEX `runs_project_started` ON `runs` (`project_id`,`started_at`);
|
||||
542
drizzle/meta/0000_snapshot.json
Normal file
542
drizzle/meta/0000_snapshot.json
Normal file
@@ -0,0 +1,542 @@
|
||||
{
|
||||
"version": "6",
|
||||
"dialect": "sqlite",
|
||||
"id": "a7309db7-6811-48f5-bb28-822a92adf5d5",
|
||||
"prevId": "00000000-0000-0000-0000-000000000000",
|
||||
"tables": {
|
||||
"items": {
|
||||
"name": "items",
|
||||
"columns": {
|
||||
"project_id": {
|
||||
"name": "project_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"source_id": {
|
||||
"name": "source_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"parent_source_id": {
|
||||
"name": "parent_source_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"name": {
|
||||
"name": "name",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"is_folder": {
|
||||
"name": "is_folder",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"content_tag": {
|
||||
"name": "content_tag",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"source_modified_at": {
|
||||
"name": "source_modified_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"size": {
|
||||
"name": "size",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"deleted": {
|
||||
"name": "deleted",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"target_id": {
|
||||
"name": "target_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"synced_parent_source_id": {
|
||||
"name": "synced_parent_source_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"synced_name": {
|
||||
"name": "synced_name",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"synced_content_tag": {
|
||||
"name": "synced_content_tag",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"synced_deleted": {
|
||||
"name": "synced_deleted",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"synced_at": {
|
||||
"name": "synced_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"status": {
|
||||
"name": "status",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"attempt_count": {
|
||||
"name": "attempt_count",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
},
|
||||
"last_error": {
|
||||
"name": "last_error",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"last_attempt_at": {
|
||||
"name": "last_attempt_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"items_project_status": {
|
||||
"name": "items_project_status",
|
||||
"columns": [
|
||||
"project_id",
|
||||
"status"
|
||||
],
|
||||
"isUnique": false
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"items_project_id_projects_id_fk": {
|
||||
"name": "items_project_id_projects_id_fk",
|
||||
"tableFrom": "items",
|
||||
"tableTo": "projects",
|
||||
"columnsFrom": [
|
||||
"project_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {
|
||||
"items_project_id_source_id_pk": {
|
||||
"columns": [
|
||||
"project_id",
|
||||
"source_id"
|
||||
],
|
||||
"name": "items_project_id_source_id_pk"
|
||||
}
|
||||
},
|
||||
"uniqueConstraints": {},
|
||||
"checkConstraints": {}
|
||||
},
|
||||
"projects": {
|
||||
"name": "projects",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "text",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"name": {
|
||||
"name": "name",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"enabled": {
|
||||
"name": "enabled",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": true
|
||||
},
|
||||
"paused_reason": {
|
||||
"name": "paused_reason",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"sp_site_id": {
|
||||
"name": "sp_site_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"sp_drive_id": {
|
||||
"name": "sp_drive_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"sp_root_item_id": {
|
||||
"name": "sp_root_item_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"sp_display_path": {
|
||||
"name": "sp_display_path",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"acc_hub_id": {
|
||||
"name": "acc_hub_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"acc_project_id": {
|
||||
"name": "acc_project_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"acc_root_folder_id": {
|
||||
"name": "acc_root_folder_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"acc_display_path": {
|
||||
"name": "acc_display_path",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"delta_cursor": {
|
||||
"name": "delta_cursor",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"first_synced_at": {
|
||||
"name": "first_synced_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"last_run_at": {
|
||||
"name": "last_run_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
}
|
||||
},
|
||||
"indexes": {},
|
||||
"foreignKeys": {},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"checkConstraints": {}
|
||||
},
|
||||
"run_events": {
|
||||
"name": "run_events",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "integer",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"autoincrement": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"project_id": {
|
||||
"name": "project_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"source_id": {
|
||||
"name": "source_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"path": {
|
||||
"name": "path",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"action": {
|
||||
"name": "action",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"message": {
|
||||
"name": "message",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"at": {
|
||||
"name": "at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"run_events_run": {
|
||||
"name": "run_events_run",
|
||||
"columns": [
|
||||
"run_id"
|
||||
],
|
||||
"isUnique": false
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"run_events_run_id_runs_id_fk": {
|
||||
"name": "run_events_run_id_runs_id_fk",
|
||||
"tableFrom": "run_events",
|
||||
"tableTo": "runs",
|
||||
"columnsFrom": [
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"checkConstraints": {}
|
||||
},
|
||||
"runs": {
|
||||
"name": "runs",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "text",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"project_id": {
|
||||
"name": "project_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"trigger": {
|
||||
"name": "trigger",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"started_at": {
|
||||
"name": "started_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false
|
||||
},
|
||||
"finished_at": {
|
||||
"name": "finished_at",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"outcome": {
|
||||
"name": "outcome",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"error": {
|
||||
"name": "error",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false,
|
||||
"autoincrement": false
|
||||
},
|
||||
"created": {
|
||||
"name": "created",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
},
|
||||
"updated": {
|
||||
"name": "updated",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
},
|
||||
"renamed": {
|
||||
"name": "renamed",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
},
|
||||
"moved": {
|
||||
"name": "moved",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
},
|
||||
"deleted": {
|
||||
"name": "deleted",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
},
|
||||
"restored": {
|
||||
"name": "restored",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
},
|
||||
"failed": {
|
||||
"name": "failed",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"autoincrement": false,
|
||||
"default": 0
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"runs_project_started": {
|
||||
"name": "runs_project_started",
|
||||
"columns": [
|
||||
"project_id",
|
||||
"started_at"
|
||||
],
|
||||
"isUnique": false
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"runs_project_id_projects_id_fk": {
|
||||
"name": "runs_project_id_projects_id_fk",
|
||||
"tableFrom": "runs",
|
||||
"tableTo": "projects",
|
||||
"columnsFrom": [
|
||||
"project_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"checkConstraints": {}
|
||||
}
|
||||
},
|
||||
"views": {},
|
||||
"enums": {},
|
||||
"_meta": {
|
||||
"schemas": {},
|
||||
"tables": {},
|
||||
"columns": {}
|
||||
},
|
||||
"internal": {
|
||||
"indexes": {}
|
||||
}
|
||||
}
|
||||
13
drizzle/meta/_journal.json
Normal file
13
drizzle/meta/_journal.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"version": "7",
|
||||
"dialect": "sqlite",
|
||||
"entries": [
|
||||
{
|
||||
"idx": 0,
|
||||
"version": "6",
|
||||
"when": 1792725790862,
|
||||
"tag": "0000_init",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
42
eslint.config.js
Normal file
42
eslint.config.js
Normal file
@@ -0,0 +1,42 @@
|
||||
import prettier from 'eslint-config-prettier';
|
||||
import path from 'node:path';
|
||||
import js from '@eslint/js';
|
||||
import svelte from 'eslint-plugin-svelte';
|
||||
import { defineConfig, includeIgnoreFile } from 'eslint/config';
|
||||
import globals from 'globals';
|
||||
import ts from 'typescript-eslint';
|
||||
|
||||
const gitignorePath = path.resolve(import.meta.dirname, '.gitignore');
|
||||
|
||||
export default defineConfig(
|
||||
includeIgnoreFile(gitignorePath),
|
||||
js.configs.recommended,
|
||||
ts.configs.recommended,
|
||||
svelte.configs.recommended,
|
||||
prettier,
|
||||
svelte.configs.prettier,
|
||||
{
|
||||
languageOptions: { globals: { ...globals.browser, ...globals.node } },
|
||||
rules: {
|
||||
// typescript-eslint strongly recommend that you do not use the no-undef lint rule on TypeScript projects.
|
||||
// see: https://typescript-eslint.io/troubleshooting/faqs/eslint/#i-get-errors-from-the-no-undef-rule-about-global-variables-not-being-defined-even-though-there-are-no-typescript-errors
|
||||
'no-undef': 'off'
|
||||
}
|
||||
},
|
||||
{
|
||||
files: ['**/*.svelte', '**/*.svelte.ts', '**/*.svelte.js'],
|
||||
languageOptions: {
|
||||
parserOptions: {
|
||||
projectService: true,
|
||||
extraFileExtensions: ['.svelte'],
|
||||
parser: ts.parser
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
rules: {
|
||||
// Allow `const { omitted, ...rest } = obj` to drop fields.
|
||||
'@typescript-eslint/no-unused-vars': ['error', { ignoreRestSiblings: true }]
|
||||
}
|
||||
}
|
||||
);
|
||||
4969
package-lock.json
generated
Normal file
4969
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
52
package.json
Normal file
52
package.json
Normal file
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"name": "sharepoint-acc-sync",
|
||||
"private": true,
|
||||
"version": "0.0.1",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite dev",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview",
|
||||
"prepare": "svelte-kit sync || echo ''",
|
||||
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json",
|
||||
"check:watch": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json --watch",
|
||||
"lint": "prettier --check . && eslint .",
|
||||
"format": "prettier --write .",
|
||||
"test:unit": "vitest",
|
||||
"test": "npm run test:unit -- --run",
|
||||
"db:push": "drizzle-kit push",
|
||||
"db:generate": "drizzle-kit generate",
|
||||
"db:migrate": "drizzle-kit migrate",
|
||||
"db:studio": "drizzle-kit studio"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^10.0.1",
|
||||
"@sveltejs/adapter-node": "^6.0.0",
|
||||
"@sveltejs/kit": "^3.0.0",
|
||||
"@sveltejs/vite-plugin-svelte": "^7.2.0",
|
||||
"@types/better-sqlite3": "^9.6.0",
|
||||
"@types/node": "^24",
|
||||
"drizzle-kit": "^0.31.10",
|
||||
"drizzle-orm": "^0.45.2",
|
||||
"eslint": "^10.4.1",
|
||||
"eslint-config-prettier": "^10.1.8",
|
||||
"eslint-plugin-svelte": "^3.19.0",
|
||||
"globals": "^17.6.0",
|
||||
"prettier": "^3.8.3",
|
||||
"prettier-plugin-svelte": "^4.1.0",
|
||||
"svelte": "^5.57.1",
|
||||
"svelte-check": "^4.6.0",
|
||||
"typescript": "^6.0.3",
|
||||
"typescript-eslint": "^8.60.1",
|
||||
"vite": "^8.3.0",
|
||||
"vitest": "^4.1.8"
|
||||
},
|
||||
"imports": {
|
||||
"#lib": "./src/lib/index.js",
|
||||
"#lib/*": "./src/lib/*"
|
||||
},
|
||||
"dependencies": {
|
||||
"better-sqlite3": "^13.0.2",
|
||||
"zod": "^4.6.7"
|
||||
}
|
||||
}
|
||||
11
prettier.config.js
Normal file
11
prettier.config.js
Normal file
@@ -0,0 +1,11 @@
|
||||
/** @type {import("prettier").Config} */
|
||||
const config = {
|
||||
useTabs: true,
|
||||
singleQuote: true,
|
||||
trailingComma: 'none',
|
||||
printWidth: 100,
|
||||
plugins: ['prettier-plugin-svelte'],
|
||||
overrides: [{ files: '*.svelte', options: { parser: 'svelte' } }]
|
||||
};
|
||||
|
||||
export default config;
|
||||
13
src/app.d.ts
vendored
Normal file
13
src/app.d.ts
vendored
Normal file
@@ -0,0 +1,13 @@
|
||||
// See https://svelte.dev/docs/kit/types#app.d.ts
|
||||
// for information about these interfaces
|
||||
declare global {
|
||||
namespace App {
|
||||
// interface Error {}
|
||||
// interface Locals {}
|
||||
// interface PageData {}
|
||||
// interface PageState {}
|
||||
// interface Platform {}
|
||||
}
|
||||
}
|
||||
|
||||
export {};
|
||||
12
src/app.html
Normal file
12
src/app.html
Normal file
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta name="text-scale" content="scale" />
|
||||
%sveltekit.head%
|
||||
</head>
|
||||
<body data-sveltekit-preload-data="hover">
|
||||
<div style="display: contents">%sveltekit.body%</div>
|
||||
</body>
|
||||
</html>
|
||||
28
src/env.ts
Normal file
28
src/env.ts
Normal file
@@ -0,0 +1,28 @@
|
||||
import { defineEnvVars } from '@sveltejs/kit/env';
|
||||
import { z } from 'zod';
|
||||
|
||||
const positiveInt = (fallback: number) =>
|
||||
z.coerce.number<string | undefined>().int().positive().default(fallback);
|
||||
|
||||
export const variables = defineEnvVars({
|
||||
DATABASE_PATH: {
|
||||
description: 'Path to the SQLite database file (mount a volume here in Docker).',
|
||||
schema: z.string().min(1).default('local.db')
|
||||
},
|
||||
POLL_INTERVAL_SECONDS: {
|
||||
description: 'How often every project is checked for SharePoint changes.',
|
||||
schema: positiveInt(180)
|
||||
},
|
||||
MANUAL_RUN_COOLDOWN_SECONDS: {
|
||||
description: 'Minimum gap between "Run now" requests for the same project.',
|
||||
schema: positiveInt(30)
|
||||
},
|
||||
MAX_ATTEMPTS: {
|
||||
description: 'Failed attempts before an item is reported as an issue.',
|
||||
schema: positiveInt(3)
|
||||
},
|
||||
HISTORY_RETENTION_DAYS: {
|
||||
description: 'How long run history is kept.',
|
||||
schema: positiveInt(7)
|
||||
}
|
||||
});
|
||||
1
src/lib/assets/favicon.svg
Normal file
1
src/lib/assets/favicon.svg
Normal file
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="107" height="128" viewBox="0 0 107 128"><title>svelte-logo</title><path d="M94.157 22.819c-10.4-14.885-30.94-19.297-45.792-9.835L22.282 29.608A29.92 29.92 0 0 0 8.764 49.65a31.5 31.5 0 0 0 3.108 20.231 30 30 0 0 0-4.477 11.183 31.9 31.9 0 0 0 5.448 24.116c10.402 14.887 30.942 19.297 45.791 9.835l26.083-16.624A29.92 29.92 0 0 0 98.235 78.35a31.53 31.53 0 0 0-3.105-20.232 30 30 0 0 0 4.474-11.182 31.88 31.88 0 0 0-5.447-24.116" style="fill:#ff3e00"/><path d="M45.817 106.582a20.72 20.72 0 0 1-22.237-8.243 19.17 19.17 0 0 1-3.277-14.503 18 18 0 0 1 .624-2.435l.49-1.498 1.337.981a33.6 33.6 0 0 0 10.203 5.098l.97.294-.09.968a5.85 5.85 0 0 0 1.052 3.878 6.24 6.24 0 0 0 6.695 2.485 5.8 5.8 0 0 0 1.603-.704L69.27 76.28a5.43 5.43 0 0 0 2.45-3.631 5.8 5.8 0 0 0-.987-4.371 6.24 6.24 0 0 0-6.698-2.487 5.7 5.7 0 0 0-1.6.704l-9.953 6.345a19 19 0 0 1-5.296 2.326 20.72 20.72 0 0 1-22.237-8.243 19.17 19.17 0 0 1-3.277-14.502 17.99 17.99 0 0 1 8.13-12.052l26.081-16.623a19 19 0 0 1 5.3-2.329 20.72 20.72 0 0 1 22.237 8.243 19.17 19.17 0 0 1 3.277 14.503 18 18 0 0 1-.624 2.435l-.49 1.498-1.337-.98a33.6 33.6 0 0 0-10.203-5.1l-.97-.294.09-.968a5.86 5.86 0 0 0-1.052-3.878 6.24 6.24 0 0 0-6.696-2.485 5.8 5.8 0 0 0-1.602.704L37.73 51.72a5.42 5.42 0 0 0-2.449 3.63 5.79 5.79 0 0 0 .986 4.372 6.24 6.24 0 0 0 6.698 2.486 5.8 5.8 0 0 0 1.602-.704l9.952-6.342a19 19 0 0 1 5.295-2.328 20.72 20.72 0 0 1 22.237 8.242 19.17 19.17 0 0 1 3.277 14.503 18 18 0 0 1-8.13 12.053l-26.081 16.622a19 19 0 0 1-5.3 2.328" style="fill:#fff"/></svg>
|
||||
|
After Width: | Height: | Size: 1.5 KiB |
1
src/lib/index.ts
Normal file
1
src/lib/index.ts
Normal file
@@ -0,0 +1 @@
|
||||
// place files you want to import through the `#lib` alias in this folder.
|
||||
16
src/lib/server/adapters/sqlite/database.ts
Normal file
16
src/lib/server/adapters/sqlite/database.ts
Normal file
@@ -0,0 +1,16 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { drizzle, type BetterSQLite3Database } from 'drizzle-orm/better-sqlite3';
|
||||
import { migrate } from 'drizzle-orm/better-sqlite3/migrator';
|
||||
import * as schema from './schema';
|
||||
|
||||
export type Db = BetterSQLite3Database<typeof schema>;
|
||||
|
||||
/** Opens (or creates) the database and brings its schema up to date. */
|
||||
export function openDatabase(path: string, migrationsFolder = 'drizzle'): Db {
|
||||
const client = new Database(path);
|
||||
client.pragma('journal_mode = WAL');
|
||||
client.pragma('foreign_keys = ON');
|
||||
const db = drizzle(client, { schema });
|
||||
migrate(db, { migrationsFolder });
|
||||
return db;
|
||||
}
|
||||
88
src/lib/server/adapters/sqlite/item-repository.ts
Normal file
88
src/lib/server/adapters/sqlite/item-repository.ts
Normal file
@@ -0,0 +1,88 @@
|
||||
import { eq } from 'drizzle-orm';
|
||||
import type { ItemState } from '../../sync/core/model';
|
||||
import type { ItemRepository } from '../../sync/core/ports';
|
||||
import type { Db } from './database';
|
||||
import { items, projects } from './schema';
|
||||
|
||||
type ItemRow = typeof items.$inferSelect;
|
||||
type Tx = Parameters<Parameters<Db['transaction']>[0]>[0];
|
||||
|
||||
export class SqliteItemRepository implements ItemRepository {
|
||||
constructor(private readonly db: Db) {}
|
||||
|
||||
async listByProject(projectId: string): Promise<ItemState[]> {
|
||||
return this.db.select().from(items).where(eq(items.projectId, projectId)).all().map(toItem);
|
||||
}
|
||||
|
||||
async commitIngest(projectId: string, states: ItemState[], cursor: string): Promise<void> {
|
||||
this.db.transaction((tx) => {
|
||||
upsert(tx, projectId, states);
|
||||
tx.update(projects).set({ deltaCursor: cursor }).where(eq(projects.id, projectId)).run();
|
||||
});
|
||||
}
|
||||
|
||||
async save(projectId: string, states: ItemState[]): Promise<void> {
|
||||
this.db.transaction((tx) => upsert(tx, projectId, states));
|
||||
}
|
||||
}
|
||||
|
||||
function upsert(tx: Tx, projectId: string, states: ItemState[]): void {
|
||||
for (const state of states) {
|
||||
const row = toRow(projectId, state);
|
||||
tx.insert(items)
|
||||
.values(row)
|
||||
.onConflictDoUpdate({ target: [items.projectId, items.sourceId], set: row })
|
||||
.run();
|
||||
}
|
||||
}
|
||||
|
||||
function toRow(projectId: string, { sourceId, desired, synced, ...rest }: ItemState): ItemRow {
|
||||
return {
|
||||
projectId,
|
||||
sourceId,
|
||||
parentSourceId: desired.parentSourceId,
|
||||
name: desired.name,
|
||||
isFolder: desired.isFolder,
|
||||
contentTag: desired.contentTag,
|
||||
sourceModifiedAt: desired.modifiedAt,
|
||||
size: desired.size,
|
||||
deleted: desired.deleted,
|
||||
targetId: synced.targetId,
|
||||
syncedParentSourceId: synced.parentSourceId,
|
||||
syncedName: synced.name,
|
||||
syncedContentTag: synced.contentTag,
|
||||
syncedDeleted: synced.deleted,
|
||||
syncedAt: synced.at,
|
||||
status: rest.status,
|
||||
attemptCount: rest.attemptCount,
|
||||
lastError: rest.lastError,
|
||||
lastAttemptAt: rest.lastAttemptAt
|
||||
};
|
||||
}
|
||||
|
||||
function toItem(row: ItemRow): ItemState {
|
||||
return {
|
||||
sourceId: row.sourceId,
|
||||
desired: {
|
||||
parentSourceId: row.parentSourceId,
|
||||
name: row.name,
|
||||
isFolder: row.isFolder,
|
||||
contentTag: row.contentTag,
|
||||
modifiedAt: row.sourceModifiedAt,
|
||||
size: row.size,
|
||||
deleted: row.deleted
|
||||
},
|
||||
synced: {
|
||||
targetId: row.targetId,
|
||||
parentSourceId: row.syncedParentSourceId,
|
||||
name: row.syncedName,
|
||||
contentTag: row.syncedContentTag,
|
||||
deleted: row.syncedDeleted,
|
||||
at: row.syncedAt
|
||||
},
|
||||
status: row.status,
|
||||
attemptCount: row.attemptCount,
|
||||
lastError: row.lastError,
|
||||
lastAttemptAt: row.lastAttemptAt
|
||||
};
|
||||
}
|
||||
95
src/lib/server/adapters/sqlite/project-repository.ts
Normal file
95
src/lib/server/adapters/sqlite/project-repository.ts
Normal file
@@ -0,0 +1,95 @@
|
||||
import { eq } from 'drizzle-orm';
|
||||
import type { NewProject, Project } from '../../sync/core/model';
|
||||
import type { Clock, ProjectRepository } from '../../sync/core/ports';
|
||||
import type { Db } from './database';
|
||||
import { items, projects } from './schema';
|
||||
|
||||
type ProjectRow = typeof projects.$inferSelect;
|
||||
|
||||
export class SqliteProjectRepository implements ProjectRepository {
|
||||
constructor(
|
||||
private readonly db: Db,
|
||||
private readonly clock: Clock
|
||||
) {}
|
||||
|
||||
async get(id: string): Promise<Project | null> {
|
||||
const row = this.db.select().from(projects).where(eq(projects.id, id)).get();
|
||||
return row ? toProject(row) : null;
|
||||
}
|
||||
|
||||
async list(): Promise<Project[]> {
|
||||
return this.db.select().from(projects).orderBy(projects.name).all().map(toProject);
|
||||
}
|
||||
|
||||
async create(project: NewProject): Promise<Project> {
|
||||
const row = this.db
|
||||
.insert(projects)
|
||||
.values({
|
||||
id: crypto.randomUUID(),
|
||||
name: project.name,
|
||||
spSiteId: project.source.siteId,
|
||||
spDriveId: project.source.driveId,
|
||||
spRootItemId: project.source.rootItemId,
|
||||
spDisplayPath: project.sourceDisplayPath,
|
||||
accHubId: project.target.hubId,
|
||||
accProjectId: project.target.projectId,
|
||||
accRootFolderId: project.target.rootFolderId,
|
||||
accDisplayPath: project.targetDisplayPath,
|
||||
createdAt: this.clock()
|
||||
})
|
||||
.returning()
|
||||
.get();
|
||||
return toProject(row);
|
||||
}
|
||||
|
||||
async pause(id: string, reason: string): Promise<void> {
|
||||
this.update(id, { pausedReason: reason });
|
||||
}
|
||||
|
||||
async resume(id: string): Promise<void> {
|
||||
this.update(id, { pausedReason: null });
|
||||
}
|
||||
|
||||
async markFirstSynced(id: string, at: Date): Promise<void> {
|
||||
this.update(id, { firstSyncedAt: at });
|
||||
}
|
||||
|
||||
async setLastRunAt(id: string, at: Date): Promise<void> {
|
||||
this.update(id, { lastRunAt: at });
|
||||
}
|
||||
|
||||
async reset(id: string): Promise<void> {
|
||||
this.db.transaction((tx) => {
|
||||
tx.delete(items).where(eq(items.projectId, id)).run();
|
||||
tx.update(projects)
|
||||
.set({ deltaCursor: null, firstSyncedAt: null, pausedReason: null })
|
||||
.where(eq(projects.id, id))
|
||||
.run();
|
||||
});
|
||||
}
|
||||
|
||||
private update(id: string, values: Partial<ProjectRow>): void {
|
||||
this.db.update(projects).set(values).where(eq(projects.id, id)).run();
|
||||
}
|
||||
}
|
||||
|
||||
function toProject(row: ProjectRow): Project {
|
||||
return {
|
||||
id: row.id,
|
||||
name: row.name,
|
||||
enabled: row.enabled,
|
||||
pausedReason: row.pausedReason,
|
||||
source: { siteId: row.spSiteId, driveId: row.spDriveId, rootItemId: row.spRootItemId },
|
||||
sourceDisplayPath: row.spDisplayPath,
|
||||
target: {
|
||||
hubId: row.accHubId,
|
||||
projectId: row.accProjectId,
|
||||
rootFolderId: row.accRootFolderId
|
||||
},
|
||||
targetDisplayPath: row.accDisplayPath,
|
||||
deltaCursor: row.deltaCursor,
|
||||
firstSyncedAt: row.firstSyncedAt,
|
||||
lastRunAt: row.lastRunAt,
|
||||
createdAt: row.createdAt
|
||||
};
|
||||
}
|
||||
28
src/lib/server/adapters/sqlite/run-repository.spec.ts
Normal file
28
src/lib/server/adapters/sqlite/run-repository.spec.ts
Normal file
@@ -0,0 +1,28 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { createHarness } from '../../sync/testing/harness';
|
||||
|
||||
describe('SqliteRunRepository', () => {
|
||||
it('prunes old runs together with their events', async () => {
|
||||
const h = await createHarness();
|
||||
h.drive.addFile(h.rootId, 'a.txt');
|
||||
const old = await h.sync();
|
||||
const cutoff = new Date(Date.UTC(2026, 9, 2));
|
||||
const recent = await h.runs.start(h.project.id, 'manual', new Date(Date.UTC(2026, 9, 3)));
|
||||
|
||||
expect(await h.runs.pruneBefore(cutoff)).toBe(1);
|
||||
|
||||
expect((await h.runs.listSince(h.project.id, new Date(0))).map((r) => r.runId)).toEqual([
|
||||
recent
|
||||
]);
|
||||
expect(await h.runs.listEvents(old.runId)).toEqual([]);
|
||||
});
|
||||
|
||||
it('reports an unfinished run with no outcome', async () => {
|
||||
const h = await createHarness();
|
||||
const runId = await h.runs.start(h.project.id, 'manual', new Date());
|
||||
|
||||
const [run] = await h.runs.listSince(h.project.id, new Date(0));
|
||||
|
||||
expect(run).toMatchObject({ runId, outcome: null, finishedAt: null });
|
||||
});
|
||||
});
|
||||
71
src/lib/server/adapters/sqlite/run-repository.ts
Normal file
71
src/lib/server/adapters/sqlite/run-repository.ts
Normal file
@@ -0,0 +1,71 @@
|
||||
import { and, asc, desc, eq, gte, lt } from 'drizzle-orm';
|
||||
import type { RunCounts, RunEvent, RunOutcome, RunRecord, RunTrigger } from '../../sync/core/model';
|
||||
import type { RunRepository } from '../../sync/core/ports';
|
||||
import type { Db } from './database';
|
||||
import { runEvents, runs } from './schema';
|
||||
|
||||
type RunRow = typeof runs.$inferSelect;
|
||||
|
||||
export class SqliteRunRepository implements RunRepository {
|
||||
constructor(private readonly db: Db) {}
|
||||
|
||||
async start(projectId: string, trigger: RunTrigger, at: Date): Promise<string> {
|
||||
const id = crypto.randomUUID();
|
||||
this.db.insert(runs).values({ id, projectId, trigger, startedAt: at }).run();
|
||||
return id;
|
||||
}
|
||||
|
||||
async finish(
|
||||
runId: string,
|
||||
result: { finishedAt: Date; outcome: RunOutcome; error: string | null; counts: RunCounts }
|
||||
): Promise<void> {
|
||||
const { finishedAt, outcome, error, counts } = result;
|
||||
this.db
|
||||
.update(runs)
|
||||
.set({ finishedAt, outcome, error, ...counts })
|
||||
.where(eq(runs.id, runId))
|
||||
.run();
|
||||
}
|
||||
|
||||
async addEvent(event: RunEvent): Promise<void> {
|
||||
this.db.insert(runEvents).values(event).run();
|
||||
}
|
||||
|
||||
async listSince(projectId: string, since: Date): Promise<RunRecord[]> {
|
||||
return this.db
|
||||
.select()
|
||||
.from(runs)
|
||||
.where(and(eq(runs.projectId, projectId), gte(runs.startedAt, since)))
|
||||
.orderBy(desc(runs.startedAt))
|
||||
.all()
|
||||
.map(toRunRecord);
|
||||
}
|
||||
|
||||
async listEvents(runId: string): Promise<RunEvent[]> {
|
||||
return this.db
|
||||
.select()
|
||||
.from(runEvents)
|
||||
.where(eq(runEvents.runId, runId))
|
||||
.orderBy(asc(runEvents.id))
|
||||
.all()
|
||||
.map(({ id, ...event }) => event);
|
||||
}
|
||||
|
||||
async pruneBefore(cutoff: Date): Promise<number> {
|
||||
return this.db.delete(runs).where(lt(runs.startedAt, cutoff)).run().changes;
|
||||
}
|
||||
}
|
||||
|
||||
function toRunRecord(row: RunRow): RunRecord {
|
||||
const { id, projectId, trigger, startedAt, finishedAt, outcome, error, ...counts } = row;
|
||||
return {
|
||||
runId: id,
|
||||
projectId,
|
||||
trigger,
|
||||
startedAt,
|
||||
finishedAt,
|
||||
outcome,
|
||||
error,
|
||||
counts
|
||||
};
|
||||
}
|
||||
99
src/lib/server/adapters/sqlite/schema.ts
Normal file
99
src/lib/server/adapters/sqlite/schema.ts
Normal file
@@ -0,0 +1,99 @@
|
||||
import { index, integer, primaryKey, sqliteTable, text } from 'drizzle-orm/sqlite-core';
|
||||
import { EVENT_ACTIONS } from '../../sync/core/model';
|
||||
|
||||
const timestamp = (name: string) => integer(name, { mode: 'timestamp_ms' });
|
||||
const flag = (name: string) => integer(name, { mode: 'boolean' });
|
||||
const count = (name: string) => integer(name).notNull().default(0);
|
||||
|
||||
export const projects = sqliteTable('projects', {
|
||||
id: text('id').primaryKey(),
|
||||
name: text('name').notNull(),
|
||||
enabled: flag('enabled').notNull().default(true),
|
||||
pausedReason: text('paused_reason'),
|
||||
spSiteId: text('sp_site_id').notNull(),
|
||||
spDriveId: text('sp_drive_id').notNull(),
|
||||
spRootItemId: text('sp_root_item_id').notNull(),
|
||||
spDisplayPath: text('sp_display_path').notNull(),
|
||||
accHubId: text('acc_hub_id').notNull(),
|
||||
accProjectId: text('acc_project_id').notNull(),
|
||||
accRootFolderId: text('acc_root_folder_id').notNull(),
|
||||
accDisplayPath: text('acc_display_path').notNull(),
|
||||
deltaCursor: text('delta_cursor'),
|
||||
firstSyncedAt: timestamp('first_synced_at'),
|
||||
lastRunAt: timestamp('last_run_at'),
|
||||
createdAt: timestamp('created_at').notNull()
|
||||
});
|
||||
|
||||
export const items = sqliteTable(
|
||||
'items',
|
||||
{
|
||||
projectId: text('project_id')
|
||||
.notNull()
|
||||
.references(() => projects.id, { onDelete: 'cascade' }),
|
||||
sourceId: text('source_id').notNull(),
|
||||
// desired (from SharePoint)
|
||||
parentSourceId: text('parent_source_id').notNull(),
|
||||
name: text('name').notNull(),
|
||||
isFolder: flag('is_folder').notNull(),
|
||||
contentTag: text('content_tag'),
|
||||
sourceModifiedAt: timestamp('source_modified_at').notNull(),
|
||||
size: integer('size'),
|
||||
deleted: flag('deleted').notNull(),
|
||||
// synced (what ACC has)
|
||||
targetId: text('target_id'),
|
||||
syncedParentSourceId: text('synced_parent_source_id'),
|
||||
syncedName: text('synced_name'),
|
||||
syncedContentTag: text('synced_content_tag'),
|
||||
syncedDeleted: flag('synced_deleted').notNull(),
|
||||
syncedAt: timestamp('synced_at'),
|
||||
// bookkeeping
|
||||
status: text('status', { enum: ['pending', 'synced', 'failed', 'deleted'] }).notNull(),
|
||||
attemptCount: count('attempt_count'),
|
||||
lastError: text('last_error'),
|
||||
lastAttemptAt: timestamp('last_attempt_at')
|
||||
},
|
||||
(t) => [
|
||||
primaryKey({ columns: [t.projectId, t.sourceId] }),
|
||||
index('items_project_status').on(t.projectId, t.status)
|
||||
]
|
||||
);
|
||||
|
||||
export const runs = sqliteTable(
|
||||
'runs',
|
||||
{
|
||||
id: text('id').primaryKey(),
|
||||
projectId: text('project_id')
|
||||
.notNull()
|
||||
.references(() => projects.id, { onDelete: 'cascade' }),
|
||||
trigger: text('trigger', { enum: ['schedule', 'manual'] }).notNull(),
|
||||
startedAt: timestamp('started_at').notNull(),
|
||||
finishedAt: timestamp('finished_at'),
|
||||
outcome: text('outcome', { enum: ['success', 'partial', 'failed'] }),
|
||||
error: text('error'),
|
||||
created: count('created'),
|
||||
updated: count('updated'),
|
||||
renamed: count('renamed'),
|
||||
moved: count('moved'),
|
||||
deleted: count('deleted'),
|
||||
restored: count('restored'),
|
||||
failed: count('failed')
|
||||
},
|
||||
(t) => [index('runs_project_started').on(t.projectId, t.startedAt)]
|
||||
);
|
||||
|
||||
export const runEvents = sqliteTable(
|
||||
'run_events',
|
||||
{
|
||||
id: integer('id').primaryKey({ autoIncrement: true }),
|
||||
runId: text('run_id')
|
||||
.notNull()
|
||||
.references(() => runs.id, { onDelete: 'cascade' }),
|
||||
projectId: text('project_id').notNull(),
|
||||
sourceId: text('source_id').notNull(),
|
||||
path: text('path').notNull(),
|
||||
action: text('action', { enum: EVENT_ACTIONS }).notNull(),
|
||||
message: text('message'),
|
||||
at: timestamp('at').notNull()
|
||||
},
|
||||
(t) => [index('run_events_run').on(t.runId)]
|
||||
);
|
||||
115
src/lib/server/sync/coordinator.spec.ts
Normal file
115
src/lib/server/sync/coordinator.spec.ts
Normal file
@@ -0,0 +1,115 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import type { ProjectSyncRunner } from './core/engine';
|
||||
import { emptyCounts, type Project, type RunTrigger } from './core/model';
|
||||
import type { ProjectRepository } from './core/ports';
|
||||
import { SyncCoordinator } from './coordinator';
|
||||
import { silentLogger } from './testing/harness';
|
||||
|
||||
function project(id: string, overrides: Partial<Project> = {}): Project {
|
||||
return { id, enabled: true, pausedReason: null, ...overrides } as Project;
|
||||
}
|
||||
|
||||
/** A runner whose runs stay in flight until `finish` is called. */
|
||||
class ControlledRunner implements ProjectSyncRunner {
|
||||
readonly started: Array<[string, RunTrigger]> = [];
|
||||
private readonly pending: Array<() => void> = [];
|
||||
|
||||
runProject(projectId: string, trigger: RunTrigger) {
|
||||
this.started.push([projectId, trigger]);
|
||||
return new Promise<never>((resolve) => {
|
||||
this.pending.push(() =>
|
||||
resolve({ runId: 'r', outcome: 'success', error: null, counts: emptyCounts() } as never)
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
finishAll() {
|
||||
this.pending.splice(0).forEach((finish) => finish());
|
||||
}
|
||||
}
|
||||
|
||||
function setup(projects: Project[]) {
|
||||
let now = 0;
|
||||
const runner = new ControlledRunner();
|
||||
const repo = {
|
||||
get: async (id: string) => projects.find((p) => p.id === id) ?? null,
|
||||
list: async () => projects
|
||||
} as ProjectRepository;
|
||||
const coordinator = new SyncCoordinator({
|
||||
runner,
|
||||
projects: repo,
|
||||
clock: () => new Date(now),
|
||||
logger: silentLogger,
|
||||
manualCooldownMs: 30_000
|
||||
});
|
||||
return { runner, coordinator, advance: (ms: number) => (now += ms) };
|
||||
}
|
||||
|
||||
describe('SyncCoordinator', () => {
|
||||
it('starts a manual run and reports it as already running on a second click', async () => {
|
||||
const { runner, coordinator } = setup([project('p1')]);
|
||||
|
||||
expect(await coordinator.requestManualRun('p1')).toEqual({ status: 'started' });
|
||||
expect(await coordinator.requestManualRun('p1')).toEqual({ status: 'already-running' });
|
||||
expect(runner.started).toEqual([['p1', 'manual']]);
|
||||
});
|
||||
|
||||
it('enforces the manual cooldown after a run finishes', async () => {
|
||||
const { runner, coordinator, advance } = setup([project('p1')]);
|
||||
await coordinator.requestManualRun('p1');
|
||||
runner.finishAll();
|
||||
await coordinator.drain();
|
||||
|
||||
advance(10_000);
|
||||
expect(await coordinator.requestManualRun('p1')).toEqual({
|
||||
status: 'cooldown',
|
||||
retryAfterSeconds: 20
|
||||
});
|
||||
|
||||
advance(20_000);
|
||||
expect(await coordinator.requestManualRun('p1')).toEqual({ status: 'started' });
|
||||
});
|
||||
|
||||
it('keeps cooldowns per project', async () => {
|
||||
const { coordinator } = setup([project('p1'), project('p2')]);
|
||||
await coordinator.requestManualRun('p1');
|
||||
expect(await coordinator.requestManualRun('p2')).toEqual({ status: 'started' });
|
||||
});
|
||||
|
||||
it('refuses manual runs for disabled, paused or unknown projects', async () => {
|
||||
const { coordinator } = setup([
|
||||
project('off', { enabled: false }),
|
||||
project('paused', { pausedReason: 'Root deleted' })
|
||||
]);
|
||||
|
||||
expect(await coordinator.requestManualRun('off')).toMatchObject({ status: 'unavailable' });
|
||||
expect(await coordinator.requestManualRun('paused')).toEqual({
|
||||
status: 'unavailable',
|
||||
reason: 'Root deleted'
|
||||
});
|
||||
expect(await coordinator.requestManualRun('nope')).toMatchObject({ status: 'unavailable' });
|
||||
});
|
||||
|
||||
it('runs scheduled passes sequentially, skipping busy, disabled and paused projects', async () => {
|
||||
const { runner, coordinator } = setup([
|
||||
project('busy'),
|
||||
project('off', { enabled: false }),
|
||||
project('paused', { pausedReason: 'x' }),
|
||||
project('a'),
|
||||
project('b')
|
||||
]);
|
||||
await coordinator.requestManualRun('busy');
|
||||
|
||||
const pass = coordinator.runScheduledPass();
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
expect(runner.started.map(([id]) => id)).toEqual(['busy', 'a']);
|
||||
|
||||
runner.finishAll();
|
||||
await new Promise((r) => setTimeout(r));
|
||||
expect(runner.started.map(([id]) => id)).toEqual(['busy', 'a', 'b']);
|
||||
|
||||
runner.finishAll();
|
||||
await pass;
|
||||
});
|
||||
});
|
||||
79
src/lib/server/sync/coordinator.ts
Normal file
79
src/lib/server/sync/coordinator.ts
Normal file
@@ -0,0 +1,79 @@
|
||||
import type { ProjectSyncRunner } from './core/engine';
|
||||
import { errorMessage } from './core/errors';
|
||||
import type { RunTrigger } from './core/model';
|
||||
import type { Clock, Logger, ProjectRepository } from './core/ports';
|
||||
|
||||
export type RunRequestResult =
|
||||
| { status: 'started' }
|
||||
| { status: 'already-running' }
|
||||
| { status: 'cooldown'; retryAfterSeconds: number }
|
||||
| { status: 'unavailable'; reason: string };
|
||||
|
||||
export interface SyncCoordinatorDeps {
|
||||
runner: ProjectSyncRunner;
|
||||
projects: ProjectRepository;
|
||||
clock: Clock;
|
||||
logger: Logger;
|
||||
manualCooldownMs: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* The only way a sync run starts. Guarantees at most one run per project at a
|
||||
* time and rate-limits manual "Run now" requests per project.
|
||||
*/
|
||||
export class SyncCoordinator {
|
||||
private readonly active = new Map<string, Promise<void>>();
|
||||
private readonly lastManualRequest = new Map<string, number>();
|
||||
|
||||
constructor(private readonly deps: SyncCoordinatorDeps) {}
|
||||
|
||||
isRunning(projectId: string): boolean {
|
||||
return this.active.has(projectId);
|
||||
}
|
||||
|
||||
async requestManualRun(projectId: string): Promise<RunRequestResult> {
|
||||
const project = await this.deps.projects.get(projectId);
|
||||
if (!project) return { status: 'unavailable', reason: 'Project not found.' };
|
||||
if (!project.enabled) return { status: 'unavailable', reason: 'Project is disabled.' };
|
||||
if (project.pausedReason) return { status: 'unavailable', reason: project.pausedReason };
|
||||
if (this.isRunning(projectId)) return { status: 'already-running' };
|
||||
|
||||
const now = this.deps.clock().getTime();
|
||||
const elapsed = now - (this.lastManualRequest.get(projectId) ?? -Infinity);
|
||||
if (elapsed < this.deps.manualCooldownMs) {
|
||||
return {
|
||||
status: 'cooldown',
|
||||
retryAfterSeconds: Math.ceil((this.deps.manualCooldownMs - elapsed) / 1000)
|
||||
};
|
||||
}
|
||||
|
||||
this.lastManualRequest.set(projectId, now);
|
||||
void this.launch(projectId, 'manual');
|
||||
return { status: 'started' };
|
||||
}
|
||||
|
||||
/** Syncs every enabled, unpaused, idle project, one after another. */
|
||||
async runScheduledPass(): Promise<void> {
|
||||
for (const project of await this.deps.projects.list()) {
|
||||
if (!project.enabled || project.pausedReason || this.isRunning(project.id)) continue;
|
||||
await this.launch(project.id, 'schedule');
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolves once every in-flight run has finished. */
|
||||
async drain(): Promise<void> {
|
||||
await Promise.all(this.active.values());
|
||||
}
|
||||
|
||||
private launch(projectId: string, trigger: RunTrigger): Promise<void> {
|
||||
const run = this.deps.runner
|
||||
.runProject(projectId, trigger)
|
||||
.then(() => undefined)
|
||||
.catch((error) => {
|
||||
this.deps.logger.error('Sync run crashed', { projectId, error: errorMessage(error) });
|
||||
})
|
||||
.finally(() => this.active.delete(projectId));
|
||||
this.active.set(projectId, run);
|
||||
return run;
|
||||
}
|
||||
}
|
||||
243
src/lib/server/sync/core/engine.spec.ts
Normal file
243
src/lib/server/sync/core/engine.spec.ts
Normal file
@@ -0,0 +1,243 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { createHarness } from '../testing/harness';
|
||||
import { ServiceUnavailableError } from './errors';
|
||||
import { ROOT_DELETED } from './ingest';
|
||||
import { WAITING_FOR_PARENT } from './reconciler';
|
||||
|
||||
describe('SyncEngine', () => {
|
||||
it('mirrors the initial tree and ignores temp files', async () => {
|
||||
const h = await createHarness();
|
||||
const drawings = h.drive.addFolder(h.rootId, 'Drawings');
|
||||
h.drive.addFile(drawings, 'plan.pdf');
|
||||
h.drive.addFile(drawings, '~$plan.docx');
|
||||
h.drive.addFile(h.rootId, 'notes.txt');
|
||||
h.drive.addFile(h.drive.driveRootId, 'outside-root.txt');
|
||||
|
||||
const result = await h.sync();
|
||||
|
||||
expect(result.outcome).toBe('success');
|
||||
expect(result.counts.created).toBe(3);
|
||||
expect(h.acc.tree()).toEqual(['Drawings/', 'Drawings/plan.pdf@1', 'notes.txt@1']);
|
||||
});
|
||||
|
||||
it('uploads edits as new versions', async () => {
|
||||
const h = await createHarness();
|
||||
const file = h.drive.addFile(h.rootId, 'spec.docx');
|
||||
await h.sync();
|
||||
|
||||
h.drive.edit(file, 'spec v2');
|
||||
const result = await h.sync();
|
||||
|
||||
expect(result.counts.updated).toBe(1);
|
||||
expect(h.acc.tree()).toEqual(['spec.docx@2']);
|
||||
expect(h.acc.latestContent('spec.docx')).toBe('spec v2');
|
||||
});
|
||||
|
||||
it('applies a move and rename of the same file', async () => {
|
||||
const h = await createHarness();
|
||||
const a = h.drive.addFolder(h.rootId, 'A');
|
||||
const b = h.drive.addFolder(h.rootId, 'B');
|
||||
const file = h.drive.addFile(a, 'old.txt');
|
||||
await h.sync();
|
||||
|
||||
h.drive.move(file, b);
|
||||
h.drive.rename(file, 'new.txt');
|
||||
const result = await h.sync();
|
||||
|
||||
expect(result.counts).toMatchObject({ moved: 1, renamed: 1, created: 0 });
|
||||
expect(h.acc.tree()).toEqual(['A/', 'B/', 'B/new.txt@1']);
|
||||
});
|
||||
|
||||
it('creates a folder before files moved into it', async () => {
|
||||
const h = await createHarness();
|
||||
const file = h.drive.addFile(h.rootId, 'report.pdf');
|
||||
await h.sync();
|
||||
|
||||
const archive = h.drive.addFolder(h.rootId, 'Archive');
|
||||
h.drive.move(file, archive);
|
||||
await h.sync();
|
||||
|
||||
expect(h.acc.tree()).toEqual(['Archive/', 'Archive/report.pdf@1']);
|
||||
});
|
||||
|
||||
it('deletes a folder with a single ACC call that covers its contents', async () => {
|
||||
const h = await createHarness();
|
||||
const folder = h.drive.addFolder(h.rootId, 'Old');
|
||||
h.drive.addFile(folder, 'a.txt');
|
||||
h.drive.addFile(folder, 'b.txt');
|
||||
await h.sync();
|
||||
|
||||
h.drive.remove(folder);
|
||||
const result = await h.sync();
|
||||
|
||||
expect(result.counts.deleted).toBe(1);
|
||||
expect(h.acc.calls.filter((c) => c.startsWith('delete'))).toEqual(['delete Old']);
|
||||
expect(h.acc.tree()).toEqual([]);
|
||||
expect((await h.itemNamed('a.txt')).status).toBe('deleted');
|
||||
});
|
||||
|
||||
it('treats items moved out of the root as deleted', async () => {
|
||||
const h = await createHarness();
|
||||
const file = h.drive.addFile(h.rootId, 'leaving.txt');
|
||||
await h.sync();
|
||||
|
||||
h.drive.move(file, h.drive.driveRootId);
|
||||
await h.sync();
|
||||
|
||||
expect(h.acc.tree()).toEqual([]);
|
||||
});
|
||||
|
||||
it('treats a rename to an ignored name as a delete', async () => {
|
||||
const h = await createHarness();
|
||||
const file = h.drive.addFile(h.rootId, 'draft.docx');
|
||||
await h.sync();
|
||||
|
||||
h.drive.rename(file, 'draft.tmp');
|
||||
await h.sync();
|
||||
|
||||
expect(h.acc.tree()).toEqual([]);
|
||||
});
|
||||
|
||||
it('revives the same ACC items when a deleted folder is restored', async () => {
|
||||
const h = await createHarness();
|
||||
const folder = h.drive.addFolder(h.rootId, 'Specs');
|
||||
h.drive.addFile(folder, 'spec.docx');
|
||||
await h.sync();
|
||||
const before = await h.itemNamed('spec.docx');
|
||||
|
||||
h.drive.remove(folder);
|
||||
await h.sync();
|
||||
h.drive.restore(folder);
|
||||
const result = await h.sync();
|
||||
|
||||
expect(result.counts.restored).toBe(2);
|
||||
expect(result.counts.created).toBe(0);
|
||||
expect(h.acc.tree()).toEqual(['Specs/', 'Specs/spec.docx@1']);
|
||||
expect((await h.itemNamed('spec.docx')).synced.targetId).toBe(before.synced.targetId);
|
||||
});
|
||||
|
||||
it('never touches files added to ACC by hand', async () => {
|
||||
const h = await createHarness();
|
||||
const folder = h.drive.addFolder(h.rootId, 'Shared');
|
||||
await h.sync();
|
||||
const accFolder = (await h.itemNamed('Shared')).synced.targetId!;
|
||||
h.acc.addByHand(h.acc.rootFolderId, 'manual.pdf');
|
||||
h.acc.addByHand(accFolder, 'inside.pdf');
|
||||
|
||||
h.drive.addFile(h.rootId, 'synced.txt');
|
||||
h.drive.rename(folder, 'Shared Renamed');
|
||||
await h.sync();
|
||||
|
||||
expect(h.acc.tree()).toEqual([
|
||||
'Shared Renamed/',
|
||||
'Shared Renamed/inside.pdf@1',
|
||||
'manual.pdf@1',
|
||||
'synced.txt@1'
|
||||
]);
|
||||
});
|
||||
|
||||
describe('failures', () => {
|
||||
it('retries a failing item, then reports it after the last attempt', async () => {
|
||||
const h = await createHarness({ maxAttempts: 3 });
|
||||
h.drive.addFile(h.rootId, 'bad.dwg');
|
||||
h.drive.addFile(h.rootId, 'good.txt');
|
||||
h.acc.failOn('bad.dwg', new Error('Upload rejected'));
|
||||
|
||||
const first = await h.sync();
|
||||
expect(first.outcome).toBe('partial');
|
||||
expect((await h.itemNamed('bad.dwg')).status).toBe('pending');
|
||||
|
||||
await h.sync();
|
||||
await h.sync();
|
||||
const failed = await h.itemNamed('bad.dwg');
|
||||
expect(failed).toMatchObject({
|
||||
status: 'failed',
|
||||
attemptCount: 3,
|
||||
lastError: 'Upload rejected'
|
||||
});
|
||||
|
||||
const uploadsBefore = h.acc.calls.length;
|
||||
await h.sync();
|
||||
expect(h.acc.calls.length).toBe(uploadsBefore);
|
||||
expect(h.acc.tree()).toEqual(['good.txt@1']);
|
||||
});
|
||||
|
||||
it('gives a failed item fresh attempts when it changes in SharePoint', async () => {
|
||||
const h = await createHarness({ maxAttempts: 1 });
|
||||
const file = h.drive.addFile(h.rootId, 'bad.dwg');
|
||||
h.acc.failOn('bad.dwg', new Error('Upload rejected'), 1);
|
||||
await h.sync();
|
||||
expect((await h.itemNamed('bad.dwg')).status).toBe('failed');
|
||||
|
||||
h.drive.edit(file, 'fixed');
|
||||
await h.sync();
|
||||
|
||||
expect((await h.itemNamed('bad.dwg')).status).toBe('synced');
|
||||
expect(h.acc.latestContent('bad.dwg')).toBe('fixed');
|
||||
});
|
||||
|
||||
it('holds children without charging attempts while their folder fails', async () => {
|
||||
const h = await createHarness();
|
||||
const folder = h.drive.addFolder(h.rootId, 'Broken');
|
||||
h.drive.addFile(folder, 'child.txt');
|
||||
h.acc.failOn('Broken', new Error('Folder create failed'), 1);
|
||||
|
||||
await h.sync();
|
||||
expect(await h.itemNamed('child.txt')).toMatchObject({
|
||||
status: 'pending',
|
||||
attemptCount: 0,
|
||||
lastError: WAITING_FOR_PARENT
|
||||
});
|
||||
|
||||
await h.sync();
|
||||
expect(h.acc.tree()).toEqual(['Broken/', 'Broken/child.txt@1']);
|
||||
});
|
||||
|
||||
it('stops the run on an outage without charging attempts', async () => {
|
||||
const h = await createHarness();
|
||||
h.drive.addFile(h.rootId, 'a.txt');
|
||||
h.drive.addFile(h.rootId, 'b.txt');
|
||||
h.acc.failOn('a.txt', new ServiceUnavailableError('ACC is down'), 1);
|
||||
|
||||
const result = await h.sync();
|
||||
|
||||
expect(result).toMatchObject({ outcome: 'failed', error: 'ACC is down' });
|
||||
expect((await h.itemNamed('a.txt')).attemptCount).toBe(0);
|
||||
expect(h.acc.calls).toEqual(['uploadFile a.txt']);
|
||||
|
||||
await h.sync();
|
||||
expect(h.acc.tree()).toEqual(['a.txt@1', 'b.txt@1']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('root safety', () => {
|
||||
it('pauses instead of deleting when the root folder is deleted', async () => {
|
||||
const h = await createHarness();
|
||||
h.drive.addFile(h.rootId, 'keep.txt');
|
||||
await h.sync();
|
||||
|
||||
h.drive.remove(h.rootId);
|
||||
const result = await h.sync();
|
||||
|
||||
expect(result).toMatchObject({ outcome: 'failed', error: ROOT_DELETED });
|
||||
expect((await h.projects.get(h.project.id))!.pausedReason).toBe(ROOT_DELETED);
|
||||
expect(h.acc.tree()).toEqual(['keep.txt@1']);
|
||||
});
|
||||
});
|
||||
|
||||
it('records run history and per-file events', async () => {
|
||||
const h = await createHarness();
|
||||
const folder = h.drive.addFolder(h.rootId, 'Docs');
|
||||
h.drive.addFile(folder, 'a.txt');
|
||||
|
||||
const result = await h.sync();
|
||||
|
||||
const [run] = await h.runs.listSince(h.project.id, new Date(0));
|
||||
expect(run).toMatchObject({ runId: result.runId, trigger: 'schedule', outcome: 'success' });
|
||||
const events = await h.runs.listEvents(result.runId);
|
||||
expect(events.map((e) => `${e.action} ${e.path}`)).toEqual([
|
||||
'created Docs',
|
||||
'created Docs/a.txt'
|
||||
]);
|
||||
});
|
||||
});
|
||||
104
src/lib/server/sync/core/engine.ts
Normal file
104
src/lib/server/sync/core/engine.ts
Normal file
@@ -0,0 +1,104 @@
|
||||
import { errorMessage } from './errors';
|
||||
import { ingestChanges } from './ingest';
|
||||
import {
|
||||
emptyCounts,
|
||||
type ItemState,
|
||||
type Project,
|
||||
type RunOutcome,
|
||||
type RunResult,
|
||||
type RunTrigger
|
||||
} from './model';
|
||||
import type {
|
||||
ChangeSource,
|
||||
Clock,
|
||||
IgnorePolicy,
|
||||
ItemRepository,
|
||||
Logger,
|
||||
ProjectRepository,
|
||||
RunRepository
|
||||
} from './ports';
|
||||
import type { Reconciler } from './reconciler';
|
||||
|
||||
export interface ProjectSyncRunner {
|
||||
runProject(projectId: string, trigger: RunTrigger): Promise<RunResult>;
|
||||
}
|
||||
|
||||
export interface SyncEngineDeps {
|
||||
source: ChangeSource;
|
||||
projects: ProjectRepository;
|
||||
items: ItemRepository;
|
||||
runs: RunRepository;
|
||||
reconciler: Reconciler;
|
||||
ignore: IgnorePolicy;
|
||||
clock: Clock;
|
||||
logger: Logger;
|
||||
maxAttempts: number;
|
||||
}
|
||||
|
||||
/** Runs one sync of one project: ingest SharePoint changes, then reconcile ACC. */
|
||||
export class SyncEngine implements ProjectSyncRunner {
|
||||
constructor(private readonly deps: SyncEngineDeps) {}
|
||||
|
||||
async runProject(projectId: string, trigger: RunTrigger): Promise<RunResult> {
|
||||
const { projects, items, runs, clock, logger } = this.deps;
|
||||
const project = await projects.get(projectId);
|
||||
if (!project) throw new Error(`Project ${projectId} not found.`);
|
||||
|
||||
const runId = await runs.start(projectId, trigger, clock());
|
||||
let counts = emptyCounts();
|
||||
let outcome: RunOutcome = 'failed';
|
||||
let error: string | null;
|
||||
|
||||
try {
|
||||
const stored = await items.listByProject(projectId);
|
||||
const current = new Map(stored.map((item) => [item.sourceId, item]));
|
||||
const pausedReason = await this.ingest(project, current);
|
||||
|
||||
if (pausedReason) {
|
||||
await projects.pause(projectId, pausedReason);
|
||||
error = pausedReason;
|
||||
logger.warn('Project paused', { projectId, reason: pausedReason });
|
||||
} else {
|
||||
if (!project.firstSyncedAt) await projects.markFirstSynced(projectId, clock());
|
||||
const result = await this.deps.reconciler.reconcile(project, runId, current);
|
||||
counts = result.counts;
|
||||
error = result.abortedWith;
|
||||
outcome = result.abortedWith ? 'failed' : counts.failed > 0 ? 'partial' : 'success';
|
||||
}
|
||||
} catch (caught) {
|
||||
error = errorMessage(caught);
|
||||
logger.error('Sync run failed', { projectId, runId, error });
|
||||
}
|
||||
|
||||
const finishedAt = clock();
|
||||
await runs.finish(runId, { finishedAt, outcome, error, counts });
|
||||
await projects.setLastRunAt(projectId, finishedAt);
|
||||
return { runId, outcome, error, counts };
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads SharePoint changes into `current` and commits them with their cursor.
|
||||
* Follows up with one full listing when ingest asks for it. Returns a pause
|
||||
* reason if the batch was rejected.
|
||||
*/
|
||||
private async ingest(project: Project, current: Map<string, ItemState>): Promise<string | null> {
|
||||
let cursor = project.deltaCursor;
|
||||
for (;;) {
|
||||
const batch = await this.deps.source.getChanges(project.source, cursor);
|
||||
const result = ingestChanges({
|
||||
rootId: project.source.rootItemId,
|
||||
items: current,
|
||||
changes: batch.changes,
|
||||
fullSync: batch.fullSync,
|
||||
ignore: this.deps.ignore,
|
||||
maxAttempts: this.deps.maxAttempts
|
||||
});
|
||||
if (result.kind === 'paused') return result.reason;
|
||||
|
||||
await this.deps.items.commitIngest(project.id, result.changed, batch.cursor);
|
||||
for (const item of result.changed) current.set(item.sourceId, item);
|
||||
if (!result.needsFullListing) return null;
|
||||
cursor = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
15
src/lib/server/sync/core/errors.ts
Normal file
15
src/lib/server/sync/core/errors.ts
Normal file
@@ -0,0 +1,15 @@
|
||||
/**
|
||||
* A remote service can't be used right now (outage, network, auth, throttling
|
||||
* that outlasted retries). Adapters throw this so the run stops early instead of
|
||||
* charging a failed attempt to every item it would otherwise touch.
|
||||
*/
|
||||
export class ServiceUnavailableError extends Error {
|
||||
constructor(message: string, options?: ErrorOptions) {
|
||||
super(message, options);
|
||||
this.name = 'ServiceUnavailableError';
|
||||
}
|
||||
}
|
||||
|
||||
export function errorMessage(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
26
src/lib/server/sync/core/ignore.spec.ts
Normal file
26
src/lib/server/sync/core/ignore.spec.ts
Normal file
@@ -0,0 +1,26 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { PatternIgnorePolicy } from './ignore';
|
||||
|
||||
describe('PatternIgnorePolicy', () => {
|
||||
const policy = new PatternIgnorePolicy();
|
||||
|
||||
it.each([
|
||||
'~$Report.docx',
|
||||
'~WRL0003.tmp',
|
||||
'upload.TMP',
|
||||
'.~lock.plan.odt#',
|
||||
'thumbs.db',
|
||||
'.DS_Store'
|
||||
])('ignores %s', (name) => expect(policy.isIgnored(name)).toBe(true));
|
||||
|
||||
it.each(['Report.docx', 'tmp-notes.txt', 'plan.dwg', 'desktop.ini.bak'])('keeps %s', (name) =>
|
||||
expect(policy.isIgnored(name)).toBe(false)
|
||||
);
|
||||
|
||||
it('accepts custom patterns', () => {
|
||||
const custom = new PatternIgnorePolicy(['*.bak', 'draft?.txt']);
|
||||
expect(custom.isIgnored('a.bak')).toBe(true);
|
||||
expect(custom.isIgnored('draft1.txt')).toBe(true);
|
||||
expect(custom.isIgnored('draft10.txt')).toBe(false);
|
||||
});
|
||||
});
|
||||
35
src/lib/server/sync/core/ignore.ts
Normal file
35
src/lib/server/sync/core/ignore.ts
Normal file
@@ -0,0 +1,35 @@
|
||||
import type { IgnorePolicy } from './ports';
|
||||
|
||||
/** Office lock files, temp/partial downloads and OS metadata. */
|
||||
export const DEFAULT_IGNORE_PATTERNS = [
|
||||
'~$*',
|
||||
'~*.tmp',
|
||||
'*.tmp',
|
||||
'.~lock.*#',
|
||||
'*.crdownload',
|
||||
'*.partial',
|
||||
'Thumbs.db',
|
||||
'desktop.ini',
|
||||
'.DS_Store'
|
||||
];
|
||||
|
||||
/** Matches names against case-insensitive `*` / `?` wildcard patterns. */
|
||||
export class PatternIgnorePolicy implements IgnorePolicy {
|
||||
private readonly patterns: RegExp[];
|
||||
|
||||
constructor(patterns: readonly string[] = DEFAULT_IGNORE_PATTERNS) {
|
||||
this.patterns = patterns.map(toRegExp);
|
||||
}
|
||||
|
||||
isIgnored(name: string): boolean {
|
||||
return this.patterns.some((pattern) => pattern.test(name));
|
||||
}
|
||||
}
|
||||
|
||||
function toRegExp(pattern: string): RegExp {
|
||||
const body = pattern
|
||||
.split('')
|
||||
.map((ch) => (ch === '*' ? '.*' : ch === '?' ? '.' : ch.replace(/[.+^${}()|[\]\\#]/g, '\\$&')))
|
||||
.join('');
|
||||
return new RegExp(`^${body}$`, 'i');
|
||||
}
|
||||
123
src/lib/server/sync/core/ingest.spec.ts
Normal file
123
src/lib/server/sync/core/ingest.spec.ts
Normal file
@@ -0,0 +1,123 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { PatternIgnorePolicy } from './ignore';
|
||||
import { ingestChanges, ROOT_MISSING, type IngestInput } from './ingest';
|
||||
import type { ItemState, SourceChange, SourceItem } from './model';
|
||||
|
||||
const ROOT = 'root';
|
||||
|
||||
function source(sourceId: string, parentSourceId: string, overrides: Partial<SourceItem> = {}) {
|
||||
return {
|
||||
sourceId,
|
||||
parentSourceId,
|
||||
name: sourceId,
|
||||
isFolder: false,
|
||||
contentTag: 'c1',
|
||||
modifiedAt: new Date(0),
|
||||
size: 1,
|
||||
...overrides
|
||||
} satisfies SourceItem;
|
||||
}
|
||||
|
||||
const upsert = (item: SourceItem): SourceChange => ({ kind: 'upsert', item });
|
||||
const folder = (id: string, parent: string) =>
|
||||
source(id, parent, { isFolder: true, contentTag: null });
|
||||
|
||||
function run(
|
||||
changes: SourceChange[],
|
||||
items: ItemState[] = [],
|
||||
overrides: Partial<IngestInput> = {}
|
||||
) {
|
||||
const result = ingestChanges({
|
||||
rootId: ROOT,
|
||||
items: new Map(items.map((i) => [i.sourceId, i])),
|
||||
changes,
|
||||
fullSync: false,
|
||||
ignore: new PatternIgnorePolicy(),
|
||||
maxAttempts: 3,
|
||||
...overrides
|
||||
});
|
||||
if (result.kind !== 'ok') throw new Error(result.reason);
|
||||
return result;
|
||||
}
|
||||
|
||||
const byId = (items: ItemState[]) => Object.fromEntries(items.map((i) => [i.sourceId, i]));
|
||||
|
||||
describe('ingestChanges', () => {
|
||||
it('accepts children reported before their parent', () => {
|
||||
const { changed } = run([upsert(source('file', 'dir')), upsert(folder('dir', ROOT))]);
|
||||
expect(changed.map((i) => i.sourceId).sort()).toEqual(['dir', 'file']);
|
||||
});
|
||||
|
||||
it('skips untracked items outside the root', () => {
|
||||
const { changed } = run([upsert(source('elsewhere', 'other-folder'))]);
|
||||
expect(changed).toEqual([]);
|
||||
});
|
||||
|
||||
it('keeps the last change when an item appears twice', () => {
|
||||
const { changed } = run([
|
||||
upsert(source('f', ROOT, { name: 'first' })),
|
||||
upsert(source('f', ROOT, { name: 'second' }))
|
||||
]);
|
||||
expect(changed[0].desired.name).toBe('second');
|
||||
});
|
||||
|
||||
it('skips everything under an ignored folder', () => {
|
||||
const ignore = { isIgnored: (name: string) => name === 'skip' };
|
||||
const { changed } = run([upsert(folder('skip', ROOT)), upsert(source('f', 'skip'))], [], {
|
||||
ignore
|
||||
});
|
||||
expect(changed).toEqual([]);
|
||||
});
|
||||
|
||||
it('cascades a folder delete to everything tracked beneath it', () => {
|
||||
const first = run([
|
||||
upsert(folder('a', ROOT)),
|
||||
upsert(folder('b', 'a')),
|
||||
upsert(source('f', 'b'))
|
||||
]).changed;
|
||||
const { changed } = run([{ kind: 'delete', sourceId: 'a' }], first);
|
||||
expect(changed.map((i) => [i.sourceId, i.desired.deleted]).sort()).toEqual([
|
||||
['a', true],
|
||||
['b', true],
|
||||
['f', true]
|
||||
]);
|
||||
});
|
||||
|
||||
it('deletes tracked items missing from a full listing', () => {
|
||||
const first = run([upsert(source('gone', ROOT)), upsert(source('kept', ROOT))]).changed;
|
||||
const { changed } = run([upsert(folder(ROOT, 'drive')), upsert(source('kept', ROOT))], first, {
|
||||
fullSync: true
|
||||
});
|
||||
expect(byId(changed).gone.desired.deleted).toBe(true);
|
||||
expect(byId(changed).kept).toBeUndefined();
|
||||
});
|
||||
|
||||
it('pauses when a full listing does not include the root', () => {
|
||||
const result = ingestChanges({
|
||||
rootId: ROOT,
|
||||
items: new Map(),
|
||||
changes: [],
|
||||
fullSync: true,
|
||||
ignore: new PatternIgnorePolicy(),
|
||||
maxAttempts: 3
|
||||
});
|
||||
expect(result).toEqual({ kind: 'paused', reason: ROOT_MISSING });
|
||||
});
|
||||
|
||||
it('keeps attempts for metadata-only changes but resets them for real ones', () => {
|
||||
const [tracked] = run([upsert(source('f', ROOT))]).changed;
|
||||
const failing = { ...tracked, attemptCount: 3, lastError: 'boom' };
|
||||
|
||||
const touched = run([upsert(source('f', ROOT, { modifiedAt: new Date(5) }))], [failing]);
|
||||
expect(touched.changed[0].attemptCount).toBe(3);
|
||||
|
||||
const edited = run([upsert(source('f', ROOT, { contentTag: 'c2' }))], [failing]);
|
||||
expect(edited.changed[0]).toMatchObject({ attemptCount: 0, lastError: null });
|
||||
});
|
||||
|
||||
it('asks for a full listing when a deleted folder comes back', () => {
|
||||
const first = run([upsert(folder('a', ROOT))]).changed;
|
||||
const deleted = run([{ kind: 'delete', sourceId: 'a' }], first).changed;
|
||||
expect(run([upsert(folder('a', ROOT))], deleted).needsFullListing).toBe(true);
|
||||
});
|
||||
});
|
||||
200
src/lib/server/sync/core/ingest.ts
Normal file
200
src/lib/server/sync/core/ingest.ts
Normal file
@@ -0,0 +1,200 @@
|
||||
import type { DesiredState, ItemState, SourceChange, SourceItem } from './model';
|
||||
import { withStatus } from './plan';
|
||||
import type { IgnorePolicy } from './ports';
|
||||
import { ItemTree } from './tree';
|
||||
|
||||
export const ROOT_DELETED = 'The SharePoint root folder was deleted.';
|
||||
export const ROOT_MISSING = 'The SharePoint root folder was not found in a full listing.';
|
||||
|
||||
export interface IngestInput {
|
||||
rootId: string;
|
||||
items: ReadonlyMap<string, ItemState>;
|
||||
changes: readonly SourceChange[];
|
||||
fullSync: boolean;
|
||||
ignore: IgnorePolicy;
|
||||
maxAttempts: number;
|
||||
}
|
||||
|
||||
export type IngestResult =
|
||||
| {
|
||||
kind: 'ok';
|
||||
changed: ItemState[];
|
||||
/**
|
||||
* A deleted folder came back. Its contents may not be reported individually,
|
||||
* so the caller should follow up with a full listing to revive them.
|
||||
*/
|
||||
needsFullListing: boolean;
|
||||
}
|
||||
| { kind: 'paused'; reason: string };
|
||||
|
||||
/**
|
||||
* Folds a batch of SharePoint changes into the desired state of the project's
|
||||
* items. Pure: returns only the items that changed and never touches ACC.
|
||||
*
|
||||
* Anything that leaves the root's scope (moved out, under an ignored or deleted
|
||||
* folder, renamed to an ignored name, or absent from a full listing) is treated
|
||||
* as deleted. If the root itself disappears the batch is rejected instead, so a
|
||||
* misconfiguration can never cascade into deleting everything in ACC.
|
||||
*/
|
||||
export function ingestChanges(input: IngestInput): IngestResult {
|
||||
const { rootId, changes, fullSync, ignore, maxAttempts } = input;
|
||||
|
||||
const latest = new Map<string, SourceChange>();
|
||||
for (const change of changes) latest.set(changeId(change), change);
|
||||
|
||||
const rootChange = latest.get(rootId);
|
||||
if (rootChange?.kind === 'delete') return { kind: 'paused', reason: ROOT_DELETED };
|
||||
if (fullSync && !rootChange) return { kind: 'paused', reason: ROOT_MISSING };
|
||||
latest.delete(rootId);
|
||||
|
||||
const working = new Map(input.items);
|
||||
const touched = new Set<string>();
|
||||
let revivedFolder = false;
|
||||
|
||||
const setDesired = (sourceId: string, desired: DesiredState) => {
|
||||
const current = working.get(sourceId);
|
||||
if (current && sameDesired(current.desired, desired)) return;
|
||||
const base = current ?? newItem(sourceId, desired);
|
||||
const resetAttempts = !current || actionableChange(current.desired, desired);
|
||||
if (current?.desired.deleted && !desired.deleted && desired.isFolder) revivedFolder = true;
|
||||
working.set(sourceId, {
|
||||
...base,
|
||||
desired,
|
||||
...(resetAttempts && { attemptCount: 0, lastError: null })
|
||||
});
|
||||
touched.add(sourceId);
|
||||
};
|
||||
|
||||
const markDeleted = (sourceId: string) => {
|
||||
const current = working.get(sourceId);
|
||||
if (current && !current.desired.deleted) {
|
||||
setDesired(sourceId, { ...current.desired, deleted: true });
|
||||
}
|
||||
};
|
||||
|
||||
const upserts = new Map<string, SourceItem>();
|
||||
for (const change of latest.values()) {
|
||||
if (change.kind === 'upsert') upserts.set(change.item.sourceId, change.item);
|
||||
else markDeleted(change.sourceId);
|
||||
}
|
||||
|
||||
if (fullSync) {
|
||||
for (const sourceId of input.items.keys()) if (!latest.has(sourceId)) markDeleted(sourceId);
|
||||
}
|
||||
|
||||
const inScope = scopeResolver(rootId, upserts, working, ignore);
|
||||
for (const item of upserts.values()) {
|
||||
if (inScope(item.sourceId)) setDesired(item.sourceId, toDesired(item));
|
||||
else markDeleted(item.sourceId);
|
||||
}
|
||||
|
||||
const tree = new ItemTree(rootId, working);
|
||||
for (const sourceId of [...touched]) {
|
||||
const item = working.get(sourceId)!;
|
||||
if (item.desired.deleted && item.desired.isFolder) {
|
||||
for (const descendant of tree.desiredDescendants(sourceId)) markDeleted(descendant.sourceId);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
kind: 'ok',
|
||||
changed: [...touched].map((id) => withStatus(working.get(id)!, maxAttempts)),
|
||||
needsFullListing: revivedFolder && !fullSync
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* An item is in scope when every ancestor up to the root is a live, non-ignored
|
||||
* folder. Newer information from this batch wins over stored state.
|
||||
*/
|
||||
function scopeResolver(
|
||||
rootId: string,
|
||||
upserts: ReadonlyMap<string, SourceItem>,
|
||||
known: ReadonlyMap<string, ItemState>,
|
||||
ignore: IgnorePolicy
|
||||
): (sourceId: string) => boolean {
|
||||
const memo = new Map<string, boolean>();
|
||||
|
||||
const resolve = (sourceId: string): boolean => {
|
||||
if (sourceId === rootId) return true;
|
||||
const cached = memo.get(sourceId);
|
||||
if (cached !== undefined) return cached;
|
||||
memo.set(sourceId, false); // guards against parent cycles
|
||||
|
||||
const fresh = upserts.get(sourceId);
|
||||
const stored = known.get(sourceId);
|
||||
let result: boolean;
|
||||
if (fresh) {
|
||||
result =
|
||||
!ignore.isIgnored(fresh.name, fresh.isFolder) &&
|
||||
fresh.parentSourceId !== null &&
|
||||
resolveParent(fresh.parentSourceId);
|
||||
} else {
|
||||
result = !!stored && !stored.desired.deleted && resolveParent(stored.desired.parentSourceId);
|
||||
}
|
||||
memo.set(sourceId, result);
|
||||
return result;
|
||||
};
|
||||
|
||||
const resolveParent = (parentId: string): boolean => {
|
||||
if (parentId === rootId) return true;
|
||||
const isFolder = upserts.get(parentId)?.isFolder ?? known.get(parentId)?.desired.isFolder;
|
||||
return isFolder === true && resolve(parentId);
|
||||
};
|
||||
|
||||
return resolve;
|
||||
}
|
||||
|
||||
function changeId(change: SourceChange): string {
|
||||
return change.kind === 'upsert' ? change.item.sourceId : change.sourceId;
|
||||
}
|
||||
|
||||
function toDesired(item: SourceItem): DesiredState {
|
||||
return {
|
||||
parentSourceId: item.parentSourceId!,
|
||||
name: item.name,
|
||||
isFolder: item.isFolder,
|
||||
contentTag: item.contentTag,
|
||||
modifiedAt: item.modifiedAt,
|
||||
size: item.size,
|
||||
deleted: false
|
||||
};
|
||||
}
|
||||
|
||||
function newItem(sourceId: string, desired: DesiredState): ItemState {
|
||||
return {
|
||||
sourceId,
|
||||
desired,
|
||||
synced: {
|
||||
targetId: null,
|
||||
parentSourceId: null,
|
||||
name: null,
|
||||
contentTag: null,
|
||||
deleted: false,
|
||||
at: null
|
||||
},
|
||||
status: 'pending',
|
||||
attemptCount: 0,
|
||||
lastError: null,
|
||||
lastAttemptAt: null
|
||||
};
|
||||
}
|
||||
|
||||
/** Changes that alter what needs doing in ACC (as opposed to display metadata). */
|
||||
function actionableChange(a: DesiredState, b: DesiredState): boolean {
|
||||
return (
|
||||
a.parentSourceId !== b.parentSourceId ||
|
||||
a.name !== b.name ||
|
||||
a.contentTag !== b.contentTag ||
|
||||
a.deleted !== b.deleted
|
||||
);
|
||||
}
|
||||
|
||||
function sameDesired(a: DesiredState, b: DesiredState): boolean {
|
||||
return (
|
||||
!actionableChange(a, b) &&
|
||||
a.isFolder === b.isFolder &&
|
||||
a.size === b.size &&
|
||||
a.modifiedAt.getTime() === b.modifiedAt.getTime()
|
||||
);
|
||||
}
|
||||
145
src/lib/server/sync/core/model.ts
Normal file
145
src/lib/server/sync/core/model.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
/** Where a project's files come from in SharePoint (all Graph IDs). */
|
||||
export interface SourceRoot {
|
||||
siteId: string;
|
||||
driveId: string;
|
||||
/** The folder whose contents are mirrored. Never stored as an item itself. */
|
||||
rootItemId: string;
|
||||
}
|
||||
|
||||
/** Where a project's files go in ACC (all APS IDs). */
|
||||
export interface TargetRoot {
|
||||
hubId: string;
|
||||
projectId: string;
|
||||
rootFolderId: string;
|
||||
}
|
||||
|
||||
export interface Project {
|
||||
id: string;
|
||||
name: string;
|
||||
enabled: boolean;
|
||||
/** Set when the sync refuses to continue (e.g. the SharePoint root vanished). */
|
||||
pausedReason: string | null;
|
||||
source: SourceRoot;
|
||||
sourceDisplayPath: string;
|
||||
target: TargetRoot;
|
||||
targetDisplayPath: string;
|
||||
deltaCursor: string | null;
|
||||
firstSyncedAt: Date | null;
|
||||
lastRunAt: Date | null;
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
export type NewProject = Pick<
|
||||
Project,
|
||||
'name' | 'source' | 'sourceDisplayPath' | 'target' | 'targetDisplayPath'
|
||||
>;
|
||||
|
||||
/** An item as SharePoint currently describes it. */
|
||||
export interface SourceItem {
|
||||
sourceId: string;
|
||||
parentSourceId: string | null;
|
||||
name: string;
|
||||
isFolder: boolean;
|
||||
/** Changes only when file content changes (Graph `cTag`). Null for folders. */
|
||||
contentTag: string | null;
|
||||
modifiedAt: Date;
|
||||
size: number | null;
|
||||
}
|
||||
|
||||
export type SourceChange =
|
||||
{ kind: 'upsert'; item: SourceItem } | { kind: 'delete'; sourceId: string };
|
||||
|
||||
export interface FileContent {
|
||||
size: number;
|
||||
stream: ReadableStream<Uint8Array>;
|
||||
}
|
||||
|
||||
export interface TargetRef {
|
||||
id: string;
|
||||
isFolder: boolean;
|
||||
}
|
||||
|
||||
/** What SharePoint says the item should look like. */
|
||||
export interface DesiredState {
|
||||
parentSourceId: string;
|
||||
name: string;
|
||||
isFolder: boolean;
|
||||
contentTag: string | null;
|
||||
modifiedAt: Date;
|
||||
size: number | null;
|
||||
deleted: boolean;
|
||||
}
|
||||
|
||||
/** What we last successfully wrote to ACC. */
|
||||
export interface SyncedState {
|
||||
targetId: string | null;
|
||||
parentSourceId: string | null;
|
||||
name: string | null;
|
||||
contentTag: string | null;
|
||||
deleted: boolean;
|
||||
at: Date | null;
|
||||
}
|
||||
|
||||
export type ItemStatus = 'pending' | 'synced' | 'failed' | 'deleted';
|
||||
|
||||
export interface ItemState {
|
||||
sourceId: string;
|
||||
desired: DesiredState;
|
||||
synced: SyncedState;
|
||||
status: ItemStatus;
|
||||
attemptCount: number;
|
||||
lastError: string | null;
|
||||
lastAttemptAt: Date | null;
|
||||
}
|
||||
|
||||
export type SyncAction =
|
||||
| { kind: 'create' }
|
||||
| { kind: 'restore' }
|
||||
| { kind: 'delete' }
|
||||
| { kind: 'update'; move: boolean; rename: boolean; content: boolean };
|
||||
|
||||
export type RunTrigger = 'schedule' | 'manual';
|
||||
export type RunOutcome = 'success' | 'partial' | 'failed';
|
||||
|
||||
export const EVENT_ACTIONS = [
|
||||
'created',
|
||||
'updated',
|
||||
'renamed',
|
||||
'moved',
|
||||
'deleted',
|
||||
'restored',
|
||||
'failed'
|
||||
] as const;
|
||||
export type EventAction = (typeof EVENT_ACTIONS)[number];
|
||||
|
||||
export type RunCounts = Record<EventAction, number>;
|
||||
|
||||
export function emptyCounts(): RunCounts {
|
||||
return Object.fromEntries(EVENT_ACTIONS.map((a) => [a, 0])) as RunCounts;
|
||||
}
|
||||
|
||||
export interface RunResult {
|
||||
runId: string;
|
||||
outcome: RunOutcome;
|
||||
error: string | null;
|
||||
counts: RunCounts;
|
||||
}
|
||||
|
||||
export interface RunRecord extends Omit<RunResult, 'outcome'> {
|
||||
/** Null while the run is still in progress. */
|
||||
outcome: RunOutcome | null;
|
||||
projectId: string;
|
||||
trigger: RunTrigger;
|
||||
startedAt: Date;
|
||||
finishedAt: Date | null;
|
||||
}
|
||||
|
||||
export interface RunEvent {
|
||||
runId: string;
|
||||
projectId: string;
|
||||
sourceId: string;
|
||||
path: string;
|
||||
action: EventAction;
|
||||
message: string | null;
|
||||
at: Date;
|
||||
}
|
||||
67
src/lib/server/sync/core/plan.spec.ts
Normal file
67
src/lib/server/sync/core/plan.spec.ts
Normal file
@@ -0,0 +1,67 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import type { DesiredState, ItemState, SyncedState } from './model';
|
||||
import { computeStatus, planAction } from './plan';
|
||||
|
||||
const desired: DesiredState = {
|
||||
parentSourceId: 'p',
|
||||
name: 'a.txt',
|
||||
isFolder: false,
|
||||
contentTag: 'c1',
|
||||
modifiedAt: new Date(0),
|
||||
size: 1,
|
||||
deleted: false
|
||||
};
|
||||
const synced: SyncedState = {
|
||||
targetId: 't',
|
||||
parentSourceId: 'p',
|
||||
name: 'a.txt',
|
||||
contentTag: 'c1',
|
||||
deleted: false,
|
||||
at: new Date(0)
|
||||
};
|
||||
|
||||
function item(d: Partial<DesiredState> = {}, s: Partial<SyncedState> = {}, attemptCount = 0) {
|
||||
return {
|
||||
sourceId: 'x',
|
||||
desired: { ...desired, ...d },
|
||||
synced: { ...synced, ...s },
|
||||
status: 'pending',
|
||||
attemptCount,
|
||||
lastError: null,
|
||||
lastAttemptAt: null
|
||||
} satisfies ItemState;
|
||||
}
|
||||
|
||||
describe('planAction', () => {
|
||||
it.each([
|
||||
['in sync', item(), null],
|
||||
['never synced', item({}, { targetId: null }), { kind: 'create' }],
|
||||
['deleted in ACC, alive in SharePoint', item({}, { deleted: true }), { kind: 'restore' }],
|
||||
['deleted in SharePoint', item({ deleted: true }), { kind: 'delete' }],
|
||||
['deleted before it was ever synced', item({ deleted: true }, { targetId: null }), null],
|
||||
['already deleted in both', item({ deleted: true }, { deleted: true }), null],
|
||||
[
|
||||
'moved, renamed and edited',
|
||||
item({ parentSourceId: 'q', name: 'b.txt', contentTag: 'c2' }),
|
||||
{ kind: 'update', move: true, rename: true, content: true }
|
||||
],
|
||||
[
|
||||
'folder (content tags ignored)',
|
||||
item({ isFolder: true, contentTag: null }, { contentTag: null }),
|
||||
null
|
||||
]
|
||||
])('%s', (_label, state, expected) => {
|
||||
expect(planAction(state)).toEqual(expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeStatus', () => {
|
||||
it.each([
|
||||
[item(), 'synced'],
|
||||
[item({ deleted: true }, { deleted: true }), 'deleted'],
|
||||
[item({ name: 'new' }, {}, 2), 'pending'],
|
||||
[item({ name: 'new' }, {}, 3), 'failed']
|
||||
] as const)('%#: %s', (state, expected) => {
|
||||
expect(computeStatus(state, 3)).toBe(expected);
|
||||
});
|
||||
});
|
||||
39
src/lib/server/sync/core/plan.ts
Normal file
39
src/lib/server/sync/core/plan.ts
Normal file
@@ -0,0 +1,39 @@
|
||||
import type { ItemState, ItemStatus, SyncAction } from './model';
|
||||
import type { ItemTree } from './tree';
|
||||
|
||||
/** The ACC operation needed to make the synced state match the desired state, if any. */
|
||||
export function planAction({ desired, synced }: ItemState): SyncAction | null {
|
||||
if (desired.deleted) return synced.targetId && !synced.deleted ? { kind: 'delete' } : null;
|
||||
if (!synced.targetId) return { kind: 'create' };
|
||||
if (synced.deleted) return { kind: 'restore' };
|
||||
|
||||
const move = desired.parentSourceId !== synced.parentSourceId;
|
||||
const rename = desired.name !== synced.name;
|
||||
const content = !desired.isFolder && desired.contentTag !== synced.contentTag;
|
||||
return move || rename || content ? { kind: 'update', move, rename, content } : null;
|
||||
}
|
||||
|
||||
export function computeStatus(item: ItemState, maxAttempts: number): ItemStatus {
|
||||
if (!planAction(item)) return item.desired.deleted ? 'deleted' : 'synced';
|
||||
return item.attemptCount >= maxAttempts ? 'failed' : 'pending';
|
||||
}
|
||||
|
||||
export function withStatus(item: ItemState, maxAttempts: number): ItemState {
|
||||
return { ...item, status: computeStatus(item, maxAttempts) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Items with outstanding work and attempts left, ordered so parents exist before
|
||||
* children: folders (shallow first), then files, then deletes (shallow first, so
|
||||
* a folder delete covers its contents).
|
||||
*/
|
||||
export function workQueue(
|
||||
items: Iterable<ItemState>,
|
||||
tree: ItemTree,
|
||||
maxAttempts: number
|
||||
): ItemState[] {
|
||||
const rank = (item: ItemState) => (item.desired.deleted ? 2 : item.desired.isFolder ? 0 : 1);
|
||||
return [...items]
|
||||
.filter((item) => computeStatus(item, maxAttempts) === 'pending')
|
||||
.sort((a, b) => rank(a) - rank(b) || tree.depth(a.sourceId) - tree.depth(b.sourceId));
|
||||
}
|
||||
93
src/lib/server/sync/core/ports.ts
Normal file
93
src/lib/server/sync/core/ports.ts
Normal file
@@ -0,0 +1,93 @@
|
||||
import type {
|
||||
FileContent,
|
||||
ItemState,
|
||||
NewProject,
|
||||
Project,
|
||||
RunEvent,
|
||||
RunOutcome,
|
||||
RunCounts,
|
||||
RunRecord,
|
||||
RunTrigger,
|
||||
SourceChange,
|
||||
SourceRoot,
|
||||
TargetRef
|
||||
} from './model';
|
||||
|
||||
export interface ChangeBatch {
|
||||
changes: SourceChange[];
|
||||
/** Cursor to pass next time. */
|
||||
cursor: string;
|
||||
/** True when this is a complete listing: anything tracked but absent has been removed. */
|
||||
fullSync: boolean;
|
||||
}
|
||||
|
||||
export interface ChangeSource {
|
||||
/** Changes since `cursor`; a null cursor (or an expired one) yields a full listing. */
|
||||
getChanges(root: SourceRoot, cursor: string | null): Promise<ChangeBatch>;
|
||||
download(root: SourceRoot, sourceId: string): Promise<FileContent>;
|
||||
}
|
||||
|
||||
/**
|
||||
* The destination document store. Implementations own any fallbacks they need
|
||||
* (e.g. move by copy + delete), so callers never branch on target quirks.
|
||||
*/
|
||||
export interface DocumentTarget {
|
||||
createFolder(parentId: string, name: string): Promise<TargetRef>;
|
||||
uploadFile(parentId: string, name: string, content: FileContent): Promise<TargetRef>;
|
||||
uploadVersion(ref: TargetRef, name: string, content: FileContent): Promise<void>;
|
||||
rename(ref: TargetRef, name: string): Promise<void>;
|
||||
/** May return a different ref if the move had to be done by copy + delete. */
|
||||
move(ref: TargetRef, newParentId: string): Promise<TargetRef>;
|
||||
delete(ref: TargetRef): Promise<void>;
|
||||
/** Bring back a deleted item, keeping its history where possible. */
|
||||
restore(
|
||||
ref: TargetRef,
|
||||
parentId: string,
|
||||
name: string,
|
||||
content?: FileContent
|
||||
): Promise<TargetRef>;
|
||||
}
|
||||
|
||||
export interface IgnorePolicy {
|
||||
isIgnored(name: string, isFolder: boolean): boolean;
|
||||
}
|
||||
|
||||
export interface ProjectRepository {
|
||||
get(id: string): Promise<Project | null>;
|
||||
list(): Promise<Project[]>;
|
||||
create(project: NewProject): Promise<Project>;
|
||||
pause(id: string, reason: string): Promise<void>;
|
||||
resume(id: string): Promise<void>;
|
||||
markFirstSynced(id: string, at: Date): Promise<void>;
|
||||
setLastRunAt(id: string, at: Date): Promise<void>;
|
||||
/** Forget all sync state so the project starts over from a full listing. */
|
||||
reset(id: string): Promise<void>;
|
||||
}
|
||||
|
||||
export interface ItemRepository {
|
||||
listByProject(projectId: string): Promise<ItemState[]>;
|
||||
/** Atomically store ingested changes and the cursor they were read up to. */
|
||||
commitIngest(projectId: string, items: ItemState[], cursor: string): Promise<void>;
|
||||
save(projectId: string, items: ItemState[]): Promise<void>;
|
||||
}
|
||||
|
||||
export interface RunRepository {
|
||||
start(projectId: string, trigger: RunTrigger, at: Date): Promise<string>;
|
||||
finish(
|
||||
runId: string,
|
||||
result: { finishedAt: Date; outcome: RunOutcome; error: string | null; counts: RunCounts }
|
||||
): Promise<void>;
|
||||
addEvent(event: RunEvent): Promise<void>;
|
||||
listSince(projectId: string, since: Date): Promise<RunRecord[]>;
|
||||
listEvents(runId: string): Promise<RunEvent[]>;
|
||||
/** Delete runs (and their events) started before `cutoff`; returns how many runs were removed. */
|
||||
pruneBefore(cutoff: Date): Promise<number>;
|
||||
}
|
||||
|
||||
export type Clock = () => Date;
|
||||
|
||||
export interface Logger {
|
||||
info(message: string, meta?: Record<string, unknown>): void;
|
||||
warn(message: string, meta?: Record<string, unknown>): void;
|
||||
error(message: string, meta?: Record<string, unknown>): void;
|
||||
}
|
||||
237
src/lib/server/sync/core/reconciler.ts
Normal file
237
src/lib/server/sync/core/reconciler.ts
Normal file
@@ -0,0 +1,237 @@
|
||||
import { errorMessage, ServiceUnavailableError } from './errors';
|
||||
import {
|
||||
emptyCounts,
|
||||
type EventAction,
|
||||
type ItemState,
|
||||
type Project,
|
||||
type RunCounts,
|
||||
type SyncAction,
|
||||
type SyncedState,
|
||||
type TargetRef
|
||||
} from './model';
|
||||
import { planAction, withStatus, workQueue } from './plan';
|
||||
import type { ChangeSource, Clock, DocumentTarget, ItemRepository, RunRepository } from './ports';
|
||||
import { ItemTree } from './tree';
|
||||
|
||||
export const WAITING_FOR_PARENT = 'Waiting for its parent folder to sync.';
|
||||
|
||||
export interface ReconcilerDeps {
|
||||
source: ChangeSource;
|
||||
target: DocumentTarget;
|
||||
items: ItemRepository;
|
||||
runs: RunRepository;
|
||||
clock: Clock;
|
||||
maxAttempts: number;
|
||||
}
|
||||
|
||||
export interface ReconcileResult {
|
||||
counts: RunCounts;
|
||||
/** Set when a service outage stopped the run early. */
|
||||
abortedWith: string | null;
|
||||
}
|
||||
|
||||
type RecordEvent = (action: EventAction, message?: string) => Promise<void>;
|
||||
|
||||
/** Pushes every pending item's desired state to ACC, one item at a time. */
|
||||
export class Reconciler {
|
||||
constructor(private readonly deps: ReconcilerDeps) {}
|
||||
|
||||
async reconcile(
|
||||
project: Project,
|
||||
runId: string,
|
||||
items: Map<string, ItemState>
|
||||
): Promise<ReconcileResult> {
|
||||
const { maxAttempts } = this.deps;
|
||||
const tree = new ItemTree(project.source.rootItemId, items);
|
||||
const counts = emptyCounts();
|
||||
|
||||
for (const { sourceId } of workQueue(items.values(), tree, maxAttempts)) {
|
||||
// Re-read: an earlier folder delete may already have covered this item.
|
||||
const item = items.get(sourceId)!;
|
||||
const action = planAction(item);
|
||||
if (!action) continue;
|
||||
|
||||
const record: RecordEvent = async (eventAction, message) => {
|
||||
counts[eventAction]++;
|
||||
await this.deps.runs.addEvent({
|
||||
runId,
|
||||
projectId: project.id,
|
||||
sourceId,
|
||||
path: tree.path(sourceId),
|
||||
action: eventAction,
|
||||
message: message ?? null,
|
||||
at: this.deps.clock()
|
||||
});
|
||||
};
|
||||
|
||||
try {
|
||||
await this.apply(project, items, tree, item, action, record);
|
||||
} catch (error) {
|
||||
if (error instanceof ServiceUnavailableError) {
|
||||
return { counts, abortedWith: error.message };
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
return { counts, abortedWith: null };
|
||||
}
|
||||
|
||||
private async apply(
|
||||
project: Project,
|
||||
items: Map<string, ItemState>,
|
||||
tree: ItemTree,
|
||||
item: ItemState,
|
||||
action: SyncAction,
|
||||
record: RecordEvent
|
||||
): Promise<void> {
|
||||
const parentTargetId = this.targetParentId(project, items, item);
|
||||
const needsParent = action.kind !== 'delete' && (action.kind !== 'update' || action.move);
|
||||
if (needsParent && !parentTargetId) {
|
||||
await this.store(project, items, [{ ...item, lastError: WAITING_FOR_PARENT }]);
|
||||
return;
|
||||
}
|
||||
|
||||
const synced: SyncedState = { ...item.synced };
|
||||
const now = this.deps.clock();
|
||||
try {
|
||||
await this.execute(project, item, action, parentTargetId!, synced, record);
|
||||
const done: ItemState = {
|
||||
...item,
|
||||
synced: { ...synced, at: now },
|
||||
attemptCount: 0,
|
||||
lastError: null,
|
||||
lastAttemptAt: now
|
||||
};
|
||||
await this.store(project, items, [done]);
|
||||
if (action.kind === 'delete' && item.desired.isFolder) {
|
||||
await this.coverDeletedContents(project, items, tree, item.sourceId);
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof ServiceUnavailableError) {
|
||||
// Keep partial progress but don't charge an attempt for an outage.
|
||||
await this.store(project, items, [{ ...item, synced }]);
|
||||
throw error;
|
||||
}
|
||||
const message = errorMessage(error);
|
||||
await this.store(project, items, [
|
||||
{
|
||||
...item,
|
||||
synced,
|
||||
attemptCount: item.attemptCount + 1,
|
||||
lastError: message,
|
||||
lastAttemptAt: now
|
||||
}
|
||||
]);
|
||||
await record('failed', message);
|
||||
}
|
||||
}
|
||||
|
||||
/** Runs the target calls for one action, recording progress into `synced` as each succeeds. */
|
||||
private async execute(
|
||||
project: Project,
|
||||
item: ItemState,
|
||||
action: SyncAction,
|
||||
parentTargetId: string,
|
||||
synced: SyncedState,
|
||||
record: RecordEvent
|
||||
): Promise<void> {
|
||||
const { target } = this.deps;
|
||||
const { desired } = item;
|
||||
const adopt = (ref: TargetRef) =>
|
||||
Object.assign(synced, {
|
||||
targetId: ref.id,
|
||||
parentSourceId: desired.parentSourceId,
|
||||
name: desired.name,
|
||||
contentTag: desired.contentTag,
|
||||
deleted: false
|
||||
});
|
||||
|
||||
switch (action.kind) {
|
||||
case 'create': {
|
||||
const ref = desired.isFolder
|
||||
? await target.createFolder(parentTargetId, desired.name)
|
||||
: await target.uploadFile(
|
||||
parentTargetId,
|
||||
desired.name,
|
||||
await this.download(project, item)
|
||||
);
|
||||
adopt(ref);
|
||||
return record('created');
|
||||
}
|
||||
case 'restore': {
|
||||
const stale = !desired.isFolder && desired.contentTag !== synced.contentTag;
|
||||
const content = stale ? await this.download(project, item) : undefined;
|
||||
adopt(await target.restore(refOf(item), parentTargetId, desired.name, content));
|
||||
return record('restored');
|
||||
}
|
||||
case 'delete': {
|
||||
await target.delete(refOf(item));
|
||||
synced.deleted = true;
|
||||
return record('deleted');
|
||||
}
|
||||
case 'update': {
|
||||
let ref = refOf(item);
|
||||
if (action.move) {
|
||||
ref = await target.move(ref, parentTargetId);
|
||||
Object.assign(synced, { targetId: ref.id, parentSourceId: desired.parentSourceId });
|
||||
await record('moved');
|
||||
}
|
||||
if (action.rename) {
|
||||
await target.rename(ref, desired.name);
|
||||
synced.name = desired.name;
|
||||
await record('renamed');
|
||||
}
|
||||
if (action.content) {
|
||||
await target.uploadVersion(ref, desired.name, await this.download(project, item));
|
||||
synced.contentTag = desired.contentTag;
|
||||
await record('updated');
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Deleting a folder in ACC takes its contents with it; reflect that in their synced state. */
|
||||
private async coverDeletedContents(
|
||||
project: Project,
|
||||
items: Map<string, ItemState>,
|
||||
tree: ItemTree,
|
||||
folderId: string
|
||||
): Promise<void> {
|
||||
const covered = tree
|
||||
.syncedDescendants(folderId)
|
||||
.filter((d) => d.synced.targetId && !d.synced.deleted)
|
||||
.map((d) => ({ ...d, synced: { ...d.synced, deleted: true } }));
|
||||
await this.store(project, items, covered);
|
||||
}
|
||||
|
||||
private targetParentId(
|
||||
project: Project,
|
||||
items: ReadonlyMap<string, ItemState>,
|
||||
item: ItemState
|
||||
): string | null {
|
||||
const parentId = item.desired.parentSourceId;
|
||||
if (parentId === project.source.rootItemId) return project.target.rootFolderId;
|
||||
const parent = items.get(parentId);
|
||||
return parent?.synced.targetId && !parent.synced.deleted ? parent.synced.targetId : null;
|
||||
}
|
||||
|
||||
private download(project: Project, item: ItemState) {
|
||||
return this.deps.source.download(project.source, item.sourceId);
|
||||
}
|
||||
|
||||
/** Persists items with fresh statuses and mirrors them into the in-memory map. */
|
||||
private async store(
|
||||
project: Project,
|
||||
items: Map<string, ItemState>,
|
||||
updated: ItemState[]
|
||||
): Promise<void> {
|
||||
if (updated.length === 0) return;
|
||||
const withStatuses = updated.map((item) => withStatus(item, this.deps.maxAttempts));
|
||||
await this.deps.items.save(project.id, withStatuses);
|
||||
for (const item of withStatuses) items.set(item.sourceId, item);
|
||||
}
|
||||
}
|
||||
|
||||
function refOf(item: ItemState): TargetRef {
|
||||
return { id: item.synced.targetId!, isFolder: item.desired.isFolder };
|
||||
}
|
||||
73
src/lib/server/sync/core/tree.ts
Normal file
73
src/lib/server/sync/core/tree.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
import type { ItemState } from './model';
|
||||
|
||||
type ParentOf = (item: ItemState) => string | null;
|
||||
|
||||
const desiredParent: ParentOf = (item) => item.desired.parentSourceId;
|
||||
const syncedParent: ParentOf = (item) => item.synced.parentSourceId;
|
||||
|
||||
/**
|
||||
* Read-only view over a project's items. Reads the map live, so it reflects
|
||||
* entries replaced after construction.
|
||||
*/
|
||||
export class ItemTree {
|
||||
constructor(
|
||||
private readonly rootId: string,
|
||||
private readonly items: ReadonlyMap<string, ItemState>
|
||||
) {}
|
||||
|
||||
/** Desired ancestors, nearest first, stopping before the root. Cycle-safe. */
|
||||
ancestors(sourceId: string): ItemState[] {
|
||||
const result: ItemState[] = [];
|
||||
const seen = new Set([sourceId]);
|
||||
let parentId = this.items.get(sourceId)?.desired.parentSourceId;
|
||||
while (parentId && parentId !== this.rootId && !seen.has(parentId)) {
|
||||
const parent = this.items.get(parentId);
|
||||
if (!parent) break;
|
||||
result.push(parent);
|
||||
seen.add(parentId);
|
||||
parentId = parent.desired.parentSourceId;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
depth(sourceId: string): number {
|
||||
return this.ancestors(sourceId).length;
|
||||
}
|
||||
|
||||
/** Display path relative to the project root, e.g. `Drawings/Level 1/plan.pdf`. */
|
||||
path(sourceId: string): string {
|
||||
const item = this.items.get(sourceId);
|
||||
if (!item) return sourceId;
|
||||
return [...this.ancestors(sourceId).reverse(), item].map((i) => i.desired.name).join('/');
|
||||
}
|
||||
|
||||
/** Items under `folderId` according to SharePoint. */
|
||||
desiredDescendants(folderId: string): ItemState[] {
|
||||
return this.descendants(folderId, desiredParent);
|
||||
}
|
||||
|
||||
/** Items under `folderId` as they currently sit in ACC. */
|
||||
syncedDescendants(folderId: string): ItemState[] {
|
||||
return this.descendants(folderId, syncedParent);
|
||||
}
|
||||
|
||||
private descendants(folderId: string, parentOf: ParentOf): ItemState[] {
|
||||
const children = new Map<string, ItemState[]>();
|
||||
for (const item of this.items.values()) {
|
||||
const parentId = parentOf(item);
|
||||
if (parentId) children.set(parentId, [...(children.get(parentId) ?? []), item]);
|
||||
}
|
||||
const result: ItemState[] = [];
|
||||
const seen = new Set([folderId]);
|
||||
const queue = [folderId];
|
||||
while (queue.length > 0) {
|
||||
for (const child of children.get(queue.shift()!) ?? []) {
|
||||
if (seen.has(child.sourceId)) continue;
|
||||
seen.add(child.sourceId);
|
||||
result.push(child);
|
||||
queue.push(child.sourceId);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
}
|
||||
38
src/lib/server/sync/scheduler.spec.ts
Normal file
38
src/lib/server/sync/scheduler.spec.ts
Normal file
@@ -0,0 +1,38 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { IntervalScheduler } from './scheduler';
|
||||
import { silentLogger } from './testing/harness';
|
||||
|
||||
describe('IntervalScheduler', () => {
|
||||
beforeEach(() => vi.useFakeTimers());
|
||||
afterEach(() => vi.useRealTimers());
|
||||
|
||||
it('waits for a slow task to finish before scheduling the next run', async () => {
|
||||
const scheduler = new IntervalScheduler(silentLogger);
|
||||
let runs = 0;
|
||||
scheduler.every('slow', 1000, async () => {
|
||||
runs++;
|
||||
await new Promise((resolve) => setTimeout(resolve, 5000));
|
||||
});
|
||||
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(runs).toBe(1);
|
||||
await vi.advanceTimersByTimeAsync(4000);
|
||||
expect(runs).toBe(1);
|
||||
await vi.advanceTimersByTimeAsync(2000);
|
||||
expect(runs).toBe(2);
|
||||
scheduler.stop();
|
||||
});
|
||||
|
||||
it('keeps going after a task throws, and stops cleanly', async () => {
|
||||
const scheduler = new IntervalScheduler(silentLogger);
|
||||
const task = vi.fn().mockRejectedValueOnce(new Error('boom')).mockResolvedValue(undefined);
|
||||
scheduler.every('flaky', 1000, task);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(2000);
|
||||
expect(task).toHaveBeenCalledTimes(2);
|
||||
|
||||
scheduler.stop();
|
||||
await vi.advanceTimersByTimeAsync(5000);
|
||||
expect(task).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
39
src/lib/server/sync/scheduler.ts
Normal file
39
src/lib/server/sync/scheduler.ts
Normal file
@@ -0,0 +1,39 @@
|
||||
import { errorMessage } from './core/errors';
|
||||
import type { Logger } from './core/ports';
|
||||
|
||||
/**
|
||||
* Runs async tasks on fixed intervals. The next run of a task is scheduled only
|
||||
* after the previous one settles, so a slow task never overlaps itself.
|
||||
*/
|
||||
export class IntervalScheduler {
|
||||
private readonly timers = new Set<ReturnType<typeof setTimeout>>();
|
||||
private stopped = false;
|
||||
|
||||
constructor(private readonly logger: Logger) {}
|
||||
|
||||
every(name: string, intervalMs: number, task: () => Promise<unknown>): void {
|
||||
const tick = async () => {
|
||||
try {
|
||||
await task();
|
||||
} catch (error) {
|
||||
this.logger.error('Scheduled task failed', { task: name, error: errorMessage(error) });
|
||||
}
|
||||
schedule();
|
||||
};
|
||||
const schedule = () => {
|
||||
if (this.stopped) return;
|
||||
const timer = setTimeout(() => {
|
||||
this.timers.delete(timer);
|
||||
void tick();
|
||||
}, intervalMs);
|
||||
this.timers.add(timer);
|
||||
};
|
||||
schedule();
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.stopped = true;
|
||||
for (const timer of this.timers) clearTimeout(timer);
|
||||
this.timers.clear();
|
||||
}
|
||||
}
|
||||
145
src/lib/server/sync/testing/fake-acc.ts
Normal file
145
src/lib/server/sync/testing/fake-acc.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
import type { FileContent, TargetRef } from '../core/model';
|
||||
import type { DocumentTarget } from '../core/ports';
|
||||
|
||||
interface AccNode {
|
||||
id: string;
|
||||
parentId: string | null;
|
||||
name: string;
|
||||
isFolder: boolean;
|
||||
deleted: boolean;
|
||||
versions: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* In-memory stand-in for ACC Docs. Deleting a folder hides its contents (as in
|
||||
* ACC); restoring a folder does not bring its contents back.
|
||||
*/
|
||||
export class FakeAcc implements DocumentTarget {
|
||||
readonly rootFolderId = 'acc-root';
|
||||
readonly calls: string[] = [];
|
||||
private readonly nodes = new Map<string, AccNode>();
|
||||
private readonly failures = new Map<string, { error: Error; remaining: number }>();
|
||||
private nextId = 1;
|
||||
|
||||
constructor() {
|
||||
this.nodes.set(this.rootFolderId, {
|
||||
id: this.rootFolderId,
|
||||
parentId: null,
|
||||
name: 'Project Files',
|
||||
isFolder: true,
|
||||
deleted: false,
|
||||
versions: []
|
||||
});
|
||||
}
|
||||
|
||||
/** Make the next `times` operations on an item with this name throw `error`. */
|
||||
failOn(name: string, error: Error, times = Infinity): void {
|
||||
this.failures.set(name, { error, remaining: times });
|
||||
}
|
||||
|
||||
/** Adds a file directly, as if someone uploaded it by hand in ACC. */
|
||||
addByHand(parentId: string, name: string): string {
|
||||
return this.insert(parentId, name, false, ['by hand']);
|
||||
}
|
||||
|
||||
/** Live paths, e.g. `Drawings/`, `Drawings/plan.pdf@2` (file@version count). */
|
||||
tree(): string[] {
|
||||
return [...this.nodes.values()]
|
||||
.filter((node) => node.id !== this.rootFolderId && this.isAlive(node))
|
||||
.map((node) => this.path(node) + (node.isFolder ? '/' : `@${node.versions.length}`))
|
||||
.sort();
|
||||
}
|
||||
|
||||
latestContent(path: string): string | undefined {
|
||||
const node = [...this.nodes.values()].find((n) => this.isAlive(n) && this.path(n) === path);
|
||||
return node?.versions.at(-1);
|
||||
}
|
||||
|
||||
async createFolder(parentId: string, name: string): Promise<TargetRef> {
|
||||
this.enter('createFolder', name);
|
||||
return { id: this.insert(parentId, name, true, []), isFolder: true };
|
||||
}
|
||||
|
||||
async uploadFile(parentId: string, name: string, content: FileContent): Promise<TargetRef> {
|
||||
this.enter('uploadFile', name);
|
||||
return { id: this.insert(parentId, name, false, [await read(content)]), isFolder: false };
|
||||
}
|
||||
|
||||
async uploadVersion(ref: TargetRef, name: string, content: FileContent): Promise<void> {
|
||||
const node = this.node(ref, 'uploadVersion');
|
||||
node.versions.push(await read(content));
|
||||
node.name = name;
|
||||
}
|
||||
|
||||
async rename(ref: TargetRef, name: string): Promise<void> {
|
||||
this.node(ref, 'rename').name = name;
|
||||
}
|
||||
|
||||
async move(ref: TargetRef, newParentId: string): Promise<TargetRef> {
|
||||
this.node(ref, 'move').parentId = newParentId;
|
||||
return ref;
|
||||
}
|
||||
|
||||
async delete(ref: TargetRef): Promise<void> {
|
||||
this.node(ref, 'delete').deleted = true;
|
||||
}
|
||||
|
||||
async restore(
|
||||
ref: TargetRef,
|
||||
parentId: string,
|
||||
name: string,
|
||||
content?: FileContent
|
||||
): Promise<TargetRef> {
|
||||
const node = this.node(ref, 'restore');
|
||||
Object.assign(node, { deleted: false, parentId, name });
|
||||
if (content) node.versions.push(await read(content));
|
||||
return ref;
|
||||
}
|
||||
|
||||
private insert(parentId: string, name: string, isFolder: boolean, versions: string[]): string {
|
||||
if (!this.nodes.has(parentId)) throw new Error(`No ACC folder ${parentId}`);
|
||||
const id = `acc-${this.nextId++}`;
|
||||
this.nodes.set(id, { id, parentId, name, isFolder, deleted: false, versions });
|
||||
return id;
|
||||
}
|
||||
|
||||
private node(ref: TargetRef, operation: string): AccNode {
|
||||
const node = this.nodes.get(ref.id);
|
||||
if (!node) throw new Error(`No ACC item ${ref.id}`);
|
||||
this.enter(operation, node.name);
|
||||
return node;
|
||||
}
|
||||
|
||||
private enter(operation: string, name: string): void {
|
||||
this.calls.push(`${operation} ${name}`);
|
||||
const failure = this.failures.get(name);
|
||||
if (failure && failure.remaining > 0) {
|
||||
failure.remaining--;
|
||||
throw failure.error;
|
||||
}
|
||||
}
|
||||
|
||||
private isAlive(node: AccNode): boolean {
|
||||
for (
|
||||
let n: AccNode | undefined = node;
|
||||
n;
|
||||
n = n.parentId ? this.nodes.get(n.parentId) : undefined
|
||||
) {
|
||||
if (n.deleted) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
private path(node: AccNode): string {
|
||||
const names: string[] = [];
|
||||
for (let n: AccNode | undefined = node; n && n.id !== this.rootFolderId;) {
|
||||
names.unshift(n.name);
|
||||
n = n.parentId ? this.nodes.get(n.parentId) : undefined;
|
||||
}
|
||||
return names.join('/');
|
||||
}
|
||||
}
|
||||
|
||||
async function read(content: FileContent): Promise<string> {
|
||||
return new Response(content.stream).text();
|
||||
}
|
||||
127
src/lib/server/sync/testing/fake-drive.ts
Normal file
127
src/lib/server/sync/testing/fake-drive.ts
Normal file
@@ -0,0 +1,127 @@
|
||||
import type { FileContent, SourceChange, SourceItem, SourceRoot } from '../core/model';
|
||||
import type { ChangeBatch, ChangeSource } from '../core/ports';
|
||||
|
||||
interface DriveNode {
|
||||
id: string;
|
||||
parentId: string | null;
|
||||
name: string;
|
||||
isFolder: boolean;
|
||||
content: string;
|
||||
version: number;
|
||||
deleted: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* In-memory stand-in for a SharePoint document library that behaves like Graph
|
||||
* delta: a null cursor lists everything; otherwise each item changed since the
|
||||
* cursor is reported once, in its latest state.
|
||||
*/
|
||||
export class FakeDrive implements ChangeSource {
|
||||
readonly driveRootId = 'drive-root';
|
||||
private readonly nodes = new Map<string, DriveNode>();
|
||||
private readonly log: string[] = [];
|
||||
private nextId = 1;
|
||||
|
||||
constructor() {
|
||||
this.nodes.set(this.driveRootId, {
|
||||
id: this.driveRootId,
|
||||
parentId: null,
|
||||
name: 'root',
|
||||
isFolder: true,
|
||||
content: '',
|
||||
version: 0,
|
||||
deleted: false
|
||||
});
|
||||
}
|
||||
|
||||
addFolder(parentId: string, name: string): string {
|
||||
return this.add(parentId, name, true, '');
|
||||
}
|
||||
|
||||
addFile(parentId: string, name: string, content = `${name} v1`): string {
|
||||
return this.add(parentId, name, false, content);
|
||||
}
|
||||
|
||||
edit(id: string, content: string): void {
|
||||
this.change(id, (node) => {
|
||||
node.content = content;
|
||||
node.version++;
|
||||
});
|
||||
}
|
||||
|
||||
rename(id: string, name: string): void {
|
||||
this.change(id, (node) => (node.name = name));
|
||||
}
|
||||
|
||||
move(id: string, parentId: string): void {
|
||||
this.change(id, (node) => (node.parentId = parentId));
|
||||
}
|
||||
|
||||
/** Reports only the removed item, not its contents: the harder case for the engine. */
|
||||
remove(id: string): void {
|
||||
this.change(id, (node) => (node.deleted = true));
|
||||
}
|
||||
|
||||
/** Reports only the restored item, not its contents. */
|
||||
restore(id: string): void {
|
||||
this.change(id, (node) => (node.deleted = false));
|
||||
}
|
||||
|
||||
async getChanges(_root: SourceRoot, cursor: string | null): Promise<ChangeBatch> {
|
||||
const end = String(this.log.length);
|
||||
if (cursor === null) {
|
||||
const changes = [...this.nodes.values()]
|
||||
.filter((node) => this.isAlive(node))
|
||||
.map((node) => this.toChange(node));
|
||||
return { changes, cursor: end, fullSync: true };
|
||||
}
|
||||
const ids = [...new Set(this.log.slice(Number(cursor)).reverse())].reverse();
|
||||
return {
|
||||
changes: ids.map((id) => this.toChange(this.nodes.get(id)!)),
|
||||
cursor: end,
|
||||
fullSync: false
|
||||
};
|
||||
}
|
||||
|
||||
async download(_root: SourceRoot, sourceId: string): Promise<FileContent> {
|
||||
const bytes = new TextEncoder().encode(this.nodes.get(sourceId)!.content);
|
||||
return { size: bytes.length, stream: new Response(bytes).body! };
|
||||
}
|
||||
|
||||
private add(parentId: string, name: string, isFolder: boolean, content: string): string {
|
||||
const id = `sp-${this.nextId++}`;
|
||||
this.nodes.set(id, { id, parentId, name, isFolder, content, version: 1, deleted: false });
|
||||
this.log.push(id);
|
||||
return id;
|
||||
}
|
||||
|
||||
private change(id: string, mutate: (node: DriveNode) => void): void {
|
||||
mutate(this.nodes.get(id)!);
|
||||
this.log.push(id);
|
||||
}
|
||||
|
||||
private isAlive(node: DriveNode): boolean {
|
||||
for (
|
||||
let n: DriveNode | undefined = node;
|
||||
n;
|
||||
n = n.parentId ? this.nodes.get(n.parentId) : undefined
|
||||
) {
|
||||
if (n.deleted) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
private toChange(node: DriveNode): SourceChange {
|
||||
if (node.deleted) return { kind: 'delete', sourceId: node.id };
|
||||
const item: SourceItem = {
|
||||
sourceId: node.id,
|
||||
parentSourceId: node.parentId,
|
||||
name: node.name,
|
||||
isFolder: node.isFolder,
|
||||
contentTag: node.isFolder ? null : `c${node.version}`,
|
||||
modifiedAt: new Date(Date.UTC(2026, 0, 1, 0, 0, node.version)),
|
||||
size: node.isFolder ? null : node.content.length
|
||||
};
|
||||
return { kind: 'upsert', item };
|
||||
}
|
||||
}
|
||||
70
src/lib/server/sync/testing/harness.ts
Normal file
70
src/lib/server/sync/testing/harness.ts
Normal file
@@ -0,0 +1,70 @@
|
||||
import { openDatabase } from '../../adapters/sqlite/database';
|
||||
import { SqliteItemRepository } from '../../adapters/sqlite/item-repository';
|
||||
import { SqliteProjectRepository } from '../../adapters/sqlite/project-repository';
|
||||
import { SqliteRunRepository } from '../../adapters/sqlite/run-repository';
|
||||
import { SyncEngine } from '../core/engine';
|
||||
import { PatternIgnorePolicy } from '../core/ignore';
|
||||
import type { Logger } from '../core/ports';
|
||||
import { Reconciler } from '../core/reconciler';
|
||||
import { FakeAcc } from './fake-acc';
|
||||
import { FakeDrive } from './fake-drive';
|
||||
|
||||
export const silentLogger: Logger = { info() {}, warn() {}, error() {} };
|
||||
|
||||
/** A real engine over in-memory SQLite, a fake SharePoint drive and a fake ACC. */
|
||||
export async function createHarness({ maxAttempts = 3 } = {}) {
|
||||
let now = Date.UTC(2026, 9, 1);
|
||||
const clock = () => new Date((now += 1000));
|
||||
|
||||
const db = openDatabase(':memory:');
|
||||
const projects = new SqliteProjectRepository(db, clock);
|
||||
const items = new SqliteItemRepository(db);
|
||||
const runs = new SqliteRunRepository(db);
|
||||
|
||||
const drive = new FakeDrive();
|
||||
const acc = new FakeAcc();
|
||||
const rootId = drive.addFolder(drive.driveRootId, 'Project X');
|
||||
|
||||
const reconciler = new Reconciler({
|
||||
source: drive,
|
||||
target: acc,
|
||||
items,
|
||||
runs,
|
||||
clock,
|
||||
maxAttempts
|
||||
});
|
||||
const engine = new SyncEngine({
|
||||
source: drive,
|
||||
projects,
|
||||
items,
|
||||
runs,
|
||||
reconciler,
|
||||
ignore: new PatternIgnorePolicy(),
|
||||
clock,
|
||||
logger: silentLogger,
|
||||
maxAttempts
|
||||
});
|
||||
|
||||
const project = await projects.create({
|
||||
name: 'Project X',
|
||||
source: { siteId: 'site', driveId: 'drive', rootItemId: rootId },
|
||||
sourceDisplayPath: '/sites/ProjectX/Documents/Project X',
|
||||
target: { hubId: 'hub', projectId: 'acc-project', rootFolderId: acc.rootFolderId },
|
||||
targetDisplayPath: 'Project X/Project Files'
|
||||
});
|
||||
|
||||
return {
|
||||
db,
|
||||
drive,
|
||||
acc,
|
||||
rootId,
|
||||
projects,
|
||||
items,
|
||||
runs,
|
||||
engine,
|
||||
project,
|
||||
sync: () => engine.runProject(project.id, 'schedule'),
|
||||
itemNamed: async (name: string) =>
|
||||
(await items.listByProject(project.id)).find((i) => i.desired.name === name)!
|
||||
};
|
||||
}
|
||||
12
src/routes/+layout.svelte
Normal file
12
src/routes/+layout.svelte
Normal file
@@ -0,0 +1,12 @@
|
||||
<script lang="ts">
|
||||
import favicon from '#lib/assets/favicon.svg';
|
||||
import type { LayoutProps } from './$types';
|
||||
|
||||
let { children }: LayoutProps = $props();
|
||||
</script>
|
||||
|
||||
<svelte:head>
|
||||
<link rel="icon" href={favicon} />
|
||||
</svelte:head>
|
||||
|
||||
{@render children()}
|
||||
2
src/routes/+page.svelte
Normal file
2
src/routes/+page.svelte
Normal file
@@ -0,0 +1,2 @@
|
||||
<h1>Welcome to SvelteKit</h1>
|
||||
<p>Visit <a href="https://svelte.dev/docs/kit">svelte.dev/docs/kit</a> to read the documentation</p>
|
||||
3
static/robots.txt
Normal file
3
static/robots.txt
Normal file
@@ -0,0 +1,3 @@
|
||||
# allow crawling everything by default
|
||||
User-agent: *
|
||||
Disallow:
|
||||
8
tsconfig.json
Normal file
8
tsconfig.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"extends": "$app/tsconfig",
|
||||
"compilerOptions": {
|
||||
"strict": true,
|
||||
"types": ["$app/types", "node"]
|
||||
},
|
||||
"include": ["src", "vite.config.ts", "drizzle.config.ts"]
|
||||
}
|
||||
30
vite.config.ts
Normal file
30
vite.config.ts
Normal file
@@ -0,0 +1,30 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
import adapter from '@sveltejs/adapter-node';
|
||||
import { sveltekit } from '@sveltejs/kit/vite';
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [
|
||||
sveltekit({
|
||||
compilerOptions: {
|
||||
// Force runes mode for the project, except for libraries. Can be removed in svelte 6.
|
||||
runes: ({ filename }) =>
|
||||
filename.split(/[/\\]/).includes('node_modules') ? undefined : true
|
||||
},
|
||||
adapter: adapter()
|
||||
})
|
||||
],
|
||||
test: {
|
||||
expect: { requireAssertions: true },
|
||||
projects: [
|
||||
{
|
||||
extends: './vite.config.ts',
|
||||
test: {
|
||||
name: 'server',
|
||||
environment: 'node',
|
||||
include: ['src/**/*.{test,spec}.{js,ts}'],
|
||||
exclude: ['src/**/*.svelte.{test,spec}.{js,ts}']
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user