Cordy/Cairn

Fork 0

Change-feed endpoint: GET /api/v1/changes (unblocks near-instant desktop sync) #478

New issue

Closed

opened 2026-09-10 17:56:06 +00:00 by Cordy · 2 comments

Cordy commented

2026-09-10 17:56:06 +00:00

Owner

Why

The desktop client (Cordy/cairn-desktop) syncs over WebDAV, and plain WebDAV has no change feed. The client's only option is to PROPFIND the whole tree on a timer. On a large tree that is expensive for the server and slow to react — a change made on machine A appears on machine B "sometime in the next 30 seconds", and the cost scales with tree size rather than with change volume.

A cheap cursor-based delta endpoint fixes both. This is a small addition to an existing API surface, not a new subsystem, and it is what makes sync feel instant rather than eventually-consistent.

Referenced as a known gap in cairn-desktop/docs/design-spec.md §3 and §10. The client ships poll-only in v1 and adopts this when it exists — so this is not a blocker for the client, it is an upgrade.

Proposed shape

GET /api/v1/changes?since=<cursor>&limit=500

{
  "changes": [
    { "path": "home/nikola/notes.md", "type": "write",  "at": "2026-09-10T17:00:00Z" },
    { "path": "home/nikola/old.txt",  "type": "delete", "at": "2026-09-10T17:00:04Z" },
    { "path": "home/nikola/new.txt",  "type": "move", "from": "home/nikola/tmp.txt", "at": "…" }
  ],
  "cursor": "opaque-string",
  "more": false
}

since omitted → return only the current cursor with an empty change list, so a fresh client can establish a starting point without replaying history.
more: true → the client should immediately request again with the new cursor.
Cursors are opaque to the client. Do not let them become a documented integer that someone starts arithmetic on.

Three design points that matter

1. Scope it to the caller. Changes must be filtered to what that user can actually see — their home, their group spaces, honouring the same read-only and app-owned rules internal/storage/scope already enforces. A global feed would leak the existence and names of other users' files. This is the security-relevant part of the issue.

2. Cursor expiry is mandatory, not optional. The journal will be pruned. A client that has been offline longer than the retention window must be told so explicitly rather than silently receiving an incomplete delta:

{ "expired": true }

On expired, the client falls back to a full reconciliation. Getting this wrong is a silent-divergence bug — the worst class for a sync product, because nothing appears to fail.

3. The feed is advisory, never authoritative. Cairn's whole thesis is that the directory tree on disk is the data model, so files legitimately change underneath us — rsync, a ZFS rollback, an S3 tool, an admin with mv. Those changes never pass through Cairn and cannot appear in a journal. The endpoint must therefore be documented as an optimisation over periodic full reconciliation, not a replacement for it. The client is already built this way: watchers are a hint, the full rescan is the backstop.

Where the data comes from

Cairn already writes an append-only per-object access log in internal/audit covering writes, deletes and renames (never listings) — the same events this feed needs. Two options worth weighing in the design pass:

Derive from the audit log. No new write path. But audit is admin-facing, carries actor identity, and has its own retention policy; reusing it directly couples two features with different privacy scopes and different lifetimes.
A separate lightweight journal at the storage chokepoint — same place quotas are already enforced. Cleaner separation, one more append per write.

I lean to the second, but it is worth a look at the audit driver before deciding.

Interaction with v0.6 (#138)

A change journal is runtime state, so it falls squarely under the state-in-backend work. Build it consistent with whatever #138 lands: if runtime state lives in .cairn-state/, the journal lives there too. Sequencing is already right — this is v0.7, after v0.6 closes — but do not design the journal in isolation from #138's storage decisions.

Also worth confirming against #138's two-writer analysis: if two pods can ever serve the same bucket, the cursor must remain monotonic across both, or clients will see changes go backwards.

Acceptance criteria

A client can establish a cursor, make changes over WebDAV, and receive exactly those changes.
Changes outside the caller's scope never appear.
An expired cursor returns expired: true rather than a partial list.
A file modified directly on disk (not through Cairn) is documented as not appearing — with the full-reconciliation fallback stated in the handbook.
Endpoint is covered by tests in internal/api, following the existing patterns there.

Out of scope

Push notification (WebSocket/SSE). Polling this endpoint is already dramatically cheaper than polling PROPFIND, and push adds a connection-management problem for a fraction of the remaining benefit. Revisit only if the beta shows polling latency is still the complaint.

## Why The desktop client (`Cordy/cairn-desktop`) syncs over WebDAV, and **plain WebDAV has no change feed**. The client's only option is to `PROPFIND` the whole tree on a timer. On a large tree that is expensive for the server and slow to react — a change made on machine A appears on machine B "sometime in the next 30 seconds", and the cost scales with tree size rather than with change volume. A cheap cursor-based delta endpoint fixes both. This is a small addition to an existing API surface, not a new subsystem, and it is what makes sync feel instant rather than eventually-consistent. Referenced as a known gap in `cairn-desktop/docs/design-spec.md` §3 and §10. The client ships poll-only in v1 and adopts this when it exists — so this is not a blocker for the client, it is an upgrade. ## Proposed shape ``` GET /api/v1/changes?since=<cursor>&limit=500 ``` ```json { "changes": [ { "path": "home/nikola/notes.md", "type": "write", "at": "2026-09-10T17:00:00Z" }, { "path": "home/nikola/old.txt", "type": "delete", "at": "2026-09-10T17:00:04Z" }, { "path": "home/nikola/new.txt", "type": "move", "from": "home/nikola/tmp.txt", "at": "…" } ], "cursor": "opaque-string", "more": false } ``` - `since` omitted → return only the current cursor with an empty change list, so a fresh client can establish a starting point without replaying history. - `more: true` → the client should immediately request again with the new cursor. - Cursors are **opaque** to the client. Do not let them become a documented integer that someone starts arithmetic on. ## Three design points that matter **1. Scope it to the caller.** Changes must be filtered to what that user can actually see — their home, their group spaces, honouring the same read-only and app-owned rules `internal/storage/scope` already enforces. A global feed would leak the existence and names of other users' files. This is the security-relevant part of the issue. **2. Cursor expiry is mandatory, not optional.** The journal will be pruned. A client that has been offline longer than the retention window **must** be told so explicitly rather than silently receiving an incomplete delta: ```json { "expired": true } ``` On `expired`, the client falls back to a full reconciliation. Getting this wrong is a silent-divergence bug — the worst class for a sync product, because nothing appears to fail. **3. The feed is advisory, never authoritative.** Cairn's whole thesis is that the directory tree on disk *is* the data model, so files legitimately change underneath us — `rsync`, a ZFS rollback, an S3 tool, an admin with `mv`. Those changes never pass through Cairn and cannot appear in a journal. The endpoint must therefore be documented as an **optimisation over** periodic full reconciliation, not a replacement for it. The client is already built this way: watchers are a hint, the full rescan is the backstop. ## Where the data comes from Cairn already writes an append-only per-object access log in `internal/audit` covering writes, deletes and renames (never listings) — the same events this feed needs. Two options worth weighing in the design pass: - **Derive from the audit log.** No new write path. But audit is admin-facing, carries actor identity, and has its own retention policy; reusing it directly couples two features with different privacy scopes and different lifetimes. - **A separate lightweight journal at the storage chokepoint** — same place quotas are already enforced. Cleaner separation, one more append per write. I lean to the second, but it is worth a look at the audit driver before deciding. ## Interaction with v0.6 (#138) **A change journal is runtime state**, so it falls squarely under the state-in-backend work. Build it consistent with whatever #138 lands: if runtime state lives in `.cairn-state/`, the journal lives there too. Sequencing is already right — this is v0.7, after v0.6 closes — but do not design the journal in isolation from #138's storage decisions. Also worth confirming against #138's two-writer analysis: if two pods can ever serve the same bucket, the cursor must remain monotonic across both, or clients will see changes go backwards. ## Acceptance criteria - A client can establish a cursor, make changes over WebDAV, and receive exactly those changes. - Changes outside the caller's scope never appear. - An expired cursor returns `expired: true` rather than a partial list. - A file modified directly on disk (not through Cairn) is documented as **not** appearing — with the full-reconciliation fallback stated in the handbook. - Endpoint is covered by tests in `internal/api`, following the existing patterns there. ## Out of scope Push notification (WebSocket/SSE). Polling this endpoint is already dramatically cheaper than polling `PROPFIND`, and push adds a connection-management problem for a fraction of the remaining benefit. Revisit only if the beta shows polling latency is still the complaint.

Cordy added this to the v0.7 — Closed beta (dogfood + hardening) milestone

2026-09-10 17:56:06 +00:00

Cordy commented

2026-09-10 22:25:22 +00:00

Author

Owner

Design assessment (2026-09-11, pre-build — recorded now, build deferred; the client ships poll-only v1 regardless):

Feasible cleanly, and the sequencing concern is already resolved. #138 closed 2026-08-10 and internal/statestore exists, so the journal has its settled home in encrypted .cairn-state/ from day one — no need to design around an unlanded dependency.

1. Journal source: separate lightweight journal, as leaned. The decorator precedent already sits at exactly the right layer — quota and the #423 marks decorator both wrap the shared storage.Driver, the single chokepoint for API uploads, WebDAV PUT/DELETE/MOVE, tus finalisation and peering inbox delivery. A journal decorator recording write/delete/move (Rename supplies from/to) is a small, well-trodden pattern. The journal needs no actor identity — filtering happens at read time — which cleanly sidesteps the audit-log privacy/retention coupling. Deriving from audit is rejected.

2. Persistence tension + resolution: epoch cursors. The statestore persists whole JSON objects; a journal persisted append-per-write would rewrite the whole object on every file mutation (an extra S3 PUT per write). Resolution follows from this issue's own point 3 (advisory, never authoritative): in-memory ring buffer, debounced flush (every few seconds / N entries), and the cursor carries an epoch — journal loss or unclean shutdown bumps the epoch, and any old-epoch cursor answers {"expired": true}, triggering the client's full reconciliation. Incompleteness is always declared, never partial, so the silent-divergence class is structurally closed rather than merely tested against.

3. The bulk of the work is the visibility filter, as predicted here. Mapping backend-absolute journal paths to the caller's view (own home, member spaces, app-owned exclusions, .cairn-state/ never) and translating into the caller's namespace is the security-critical part and where most of the TDD effort belongs. Ingredients exist (internal/storage/scope rules, Spaces.Members). One decision to record when building: a newly joined space's history is not replayed — the client's reconciliation covers it; replaying pre-membership history would be both complex and a mild disclosure.

4. Bonus since this issue was written: the #476 SSE hub now exists. Push stays out of scope as stated, but when the beta wants it, it is one Publish(user, "changes") from the same decorator plus a topic case in the already-deployed client wiring — the cost of "revisit later" has dropped to near zero.

Cursor shape: opaque base64 of (epoch, seq), monotonic within an epoch; single-replica Recreate keeps monotonicity trivially today, and the #138 two-writer caveat carries over unchanged (multi-replica would need a shared sequencer — recorded, not built).

No conflicts with the current tree. Effort estimate: one normal house TDD pass — decorator + ring/flush + epoch store (statestore), the /api/v1/changes handler with scope filtering, handbook section stating the advisory contract and reconciliation backstop, tests in internal/api per the acceptance list above.

Design assessment (2026-09-11, pre-build — recorded now, build deferred; the client ships poll-only v1 regardless): **Feasible cleanly, and the sequencing concern is already resolved.** #138 closed 2026-08-10 and `internal/statestore` exists, so the journal has its settled home in encrypted `.cairn-state/` from day one — no need to design around an unlanded dependency. **1. Journal source: separate lightweight journal, as leaned.** The decorator precedent already sits at exactly the right layer — quota and the #423 marks decorator both wrap the shared `storage.Driver`, the single chokepoint for API uploads, WebDAV PUT/DELETE/MOVE, tus finalisation and peering inbox delivery. A journal decorator recording `write`/`delete`/`move` (Rename supplies from/to) is a small, well-trodden pattern. The journal needs **no actor identity** — filtering happens at read time — which cleanly sidesteps the audit-log privacy/retention coupling. Deriving from audit is rejected. **2. Persistence tension + resolution: epoch cursors.** The statestore persists whole JSON objects; a journal persisted append-per-write would rewrite the whole object on every file mutation (an extra S3 PUT per write). Resolution follows from this issue's own point 3 (advisory, never authoritative): in-memory ring buffer, debounced flush (every few seconds / N entries), and the cursor carries an **epoch** — journal loss or unclean shutdown bumps the epoch, and any old-epoch cursor answers `{"expired": true}`, triggering the client's full reconciliation. Incompleteness is always declared, never partial, so the silent-divergence class is structurally closed rather than merely tested against. **3. The bulk of the work is the visibility filter, as predicted here.** Mapping backend-absolute journal paths to the caller's view (own home, member spaces, app-owned exclusions, `.cairn-state/` never) and translating into the caller's namespace is the security-critical part and where most of the TDD effort belongs. Ingredients exist (`internal/storage/scope` rules, `Spaces.Members`). One decision to record when building: **a newly joined space's history is not replayed** — the client's reconciliation covers it; replaying pre-membership history would be both complex and a mild disclosure. **4. Bonus since this issue was written:** the #476 SSE hub now exists. Push stays out of scope as stated, but when the beta wants it, it is one `Publish(user, "changes")` from the same decorator plus a topic case in the already-deployed client wiring — the cost of "revisit later" has dropped to near zero. **Cursor shape:** opaque base64 of `(epoch, seq)`, monotonic within an epoch; single-replica `Recreate` keeps monotonicity trivially today, and the #138 two-writer caveat carries over unchanged (multi-replica would need a shared sequencer — recorded, not built). No conflicts with the current tree. Effort estimate: one normal house TDD pass — decorator + ring/flush + epoch store (statestore), the `/api/v1/changes` handler with scope filtering, handbook section stating the advisory contract and reconciliation backstop, tests in `internal/api` per the acceptance list above.

Cordy referenced this issue from a commit

2026-09-11 20:35:36 +00:00

one-shot: #478 change feed - journal decorator + /api/v1/changes (red-then-green)

Cordy referenced this issue from a commit

2026-09-11 20:39:02 +00:00

one-shot: #478 change feed - journal decorator + /api/v1/changes (red-then-green)

forgejo-actions referenced this issue from a commit

2026-09-11 20:51:58 +00:00

#478: advisory change feed - journal decorator + GET /api/v1/changes

Cordy referenced this issue

2026-09-11 20:54:58 +00:00

#478: advisory change feed for sync clients #489

Cordy referenced this issue from a commit

2026-09-11 20:57:43 +00:00

Merge pull request '#478: advisory change feed for sync clients' (#489) from fix-478 into main

Cordy commented

2026-09-11 21:06:22 +00:00

Author

Owner

Shipped in v0.6.163 (PR #489, merged; live on the dogfood — cairn_build_info{version="v0.6.163"}).

What was built (per the design assessment above, all points held):

internal/changes — journal + decorator:
- In-memory ring (4096 entries), strictly ordered Seq, entries {seq, at, type write|delete|move, path, from?} with backend-absolute paths and no actor identity.
- Opaque base64 (epoch, seq) cursors. Any cursor from another epoch, from a pruned region of the ring, or from the future answers expired — never a partial list.
- Persistence via statestore (debounced 2 s) into changes.json, added to the s3 statePaths list so it lives in .cairn-state/. A clean shutdown resumes the epoch (changeJournal.Close() wired into the graceful-shutdown path), so cursors survive ordinary rollouts; a crash or lost journal starts a fresh epoch — the state is re-marked dirty immediately on resume, so only a properly closed journal is ever resumed.
- Decorator (explicit methods, no embedding) wraps directly above marks/audit and inside unscopedStore: REST, WebDAV, TUS, peering deliveries and public-share uploads are all recorded; deletes are recorded logically (above trash); only successful operations land — a hold/lock/read-only refusal never appears. .cairn-state/ writes are structurally excluded (statestore binds the raw driver) plus a defensive path skip.
GET /api/v1/changes?since=&limit=500 (internal/api/changes.go):
- Session/app-password auth; no since → {changes:[], cursor, more:false} (client does its own first scan).
- Per-caller visibility filter mirroring scope.resolve: own home translated /home/<user>/x → /home/x; member spaces (IdP groups, -ro read-only groups, app-owned via spacestore RoleFor) unchanged; other homes, /.trash/, state never. A move with one end out of view degrades honestly: moved away → delete at the old path, moved in → write at the new one. Invisible spans are drained server-side so pages are never empty while more is true.
docs/handbook/change-feed.md — protocol, what the feed contains, the advisory contract (backend-direct changes never appear; no space-history replay; reconciliation backstop required), operational notes.

TDD: 13 tests (8 journal/decorator, 5 endpoint incl. scope filter, move degradation, epoch expiry, dirty-crash epoch bump, clean-close resume), witnessed red on the runner before implementation, full suite green.

Known gaps, documented as the advisory contract: admin trash restores call the trash driver below the journal wrap and are not recorded (they are backend-direct from the feed's point of view); SSE changes topic push and the multi-replica sequencer stay deferred as agreed.

Closing.

Shipped in **v0.6.163** (PR #489, merged; live on the dogfood — `cairn_build_info{version="v0.6.163"}`). **What was built** (per the design assessment above, all points held): - `internal/changes` — journal + decorator: - In-memory ring (4096 entries), strictly ordered `Seq`, entries `{seq, at, type write|delete|move, path, from?}` with **backend-absolute paths and no actor identity**. - Opaque base64 `(epoch, seq)` cursors. Any cursor from another epoch, from a pruned region of the ring, or from the future answers **expired — never a partial list**. - Persistence via statestore (debounced 2 s) into `changes.json`, added to the s3 statePaths list so it lives in `.cairn-state/`. A **clean shutdown resumes the epoch** (`changeJournal.Close()` wired into the graceful-shutdown path), so cursors survive ordinary rollouts; a crash or lost journal starts a fresh epoch — the state is re-marked dirty immediately on resume, so only a properly closed journal is ever resumed. - Decorator (explicit methods, no embedding) wraps **directly above marks/audit and inside `unscopedStore`**: REST, WebDAV, TUS, peering deliveries and public-share uploads are all recorded; deletes are recorded logically (above trash); **only successful operations** land — a hold/lock/read-only refusal never appears. `.cairn-state/` writes are structurally excluded (statestore binds the raw driver) plus a defensive path skip. - `GET /api/v1/changes?since=&limit=500` (`internal/api/changes.go`): - Session/app-password auth; no `since` → `{changes:[], cursor, more:false}` (client does its own first scan). - **Per-caller visibility filter mirroring scope.resolve**: own home translated `/home/<user>/x → /home/x`; member spaces (IdP groups, `-ro` read-only groups, app-owned via spacestore RoleFor) unchanged; other homes, `/.trash/`, state never. A move with one end out of view degrades honestly: moved away → `delete` at the old path, moved in → `write` at the new one. Invisible spans are drained server-side so pages are never empty while `more` is true. - `docs/handbook/change-feed.md` — protocol, what the feed contains, the advisory contract (backend-direct changes never appear; no space-history replay; reconciliation backstop required), operational notes. **TDD**: 13 tests (8 journal/decorator, 5 endpoint incl. scope filter, move degradation, epoch expiry, dirty-crash epoch bump, clean-close resume), witnessed red on the runner before implementation, full suite green. **Known gaps, documented as the advisory contract**: admin trash *restores* call the trash driver below the journal wrap and are not recorded (they are backend-direct from the feed's point of view); SSE `changes` topic push and the multi-replica sequencer stay deferred as agreed. Closing.

Cordy closed this issue

2026-09-11 21:06:25 +00:00

Cordy referenced this issue

2026-09-18 18:05:08 +00:00

File versioning: design ADR, per-driver version storage, restore/download UI (#530, 4/4) #541