Horizontal scalability #742

Closed
opened 2026-09-25 09:52:25 +00:00 by hyrsh · 45 comments
Collaborator

Since AlpineFiles is a single instance installation we need to address the possibility to scale it horizontally.

Things to consider if scaling occurs:

  • installation environment (docker, raw binary, kubernetes)
  • attached storage (S3, NAS/SAN, NFS, POSIX, etc.)
  • local user handling (local file has to be distributed)
  • storage write locks or collisions
  • the local config must not share a directory with user data (separation provides more flexibility and easier handling)
  • the local config can be at a fixed file path (e.g. /alpinefiles/config/local.json)
  • the local data directory can also be at a fixed path (e.g. /alpinefiles/data)

There is one thing that is necessary if using "local" storage. This kind of storage applies not only to a local installation but also to an installation in Kubernetes when using PVCs since they work like local directory mounts from an applications point of view. So if no external storage is directly addressed like S3 or NFS we have to introduce a human understandable directory where the local user data always gets written if nothing gets configured and default values are used. The same applies for Docker volumes.

Proposal for the default directory (user data) --> /alpinefiles/data
Proposal for the default config file --> /alpinefiles/config/local.json

The most critical part of scaling comes down to the active state the application has. In our case we need to look for possible pitfalls regarding storage and local user handling.

Scenario: Raw Binary (on e.g. 3x machines)

The binary is executed on multiple linux hosts and the initial admin gets written to the local file on machine #1.
The documentation has to push the user to distribute the created file to machines #2 and #3 (e.g. with some sync job) or use a common mount (e.g. NFS) to keep local users in sync. The same applies to the data directory.
There also has to be a load balancer in front of all machines since we cannot provide reasonable balancing mechanisms on a network level.

Scenario: Docker

As in the raw binary example we need to document that the docker volumes have to be synced across all hosts. The other option would be to directly use external volumes with docker (e.g. NFS, S3).

Scenario: Kubernetes

The mounted PVC must be in RWX access mode or a pre-filled local user config must be created as Secret to be mounted to the respective path. More elegant would be if the initial wizard would offer a checkbox "Create Kubernetes Secret" that would create a Secret with the entered credentials for the first admin (if the checkbox is checked the first thing is to try to communicate with the Kubernetes API and log + notify if there are missing RBAC permissions to create a secret or if the API is at all reachable). If this is done, we can use the built-in scaling of Kubernetes.

If I can think of more. I will add another issue.

Since AlpineFiles is a single instance installation we need to address the possibility to scale it horizontally. Things to consider if scaling occurs: - installation environment (docker, raw binary, kubernetes) - attached storage (S3, NAS/SAN, NFS, POSIX, etc.) - local user handling (local file has to be distributed) - storage write locks or collisions - the local config must not share a directory with user data (separation provides more flexibility and easier handling) - the local config can be at a fixed file path (e.g. /alpinefiles/config/local.json) - the local data directory can also be at a fixed path (e.g. /alpinefiles/data) There is one thing that is necessary if using "local" storage. This kind of storage applies not only to a local installation but also to an installation in Kubernetes when using PVCs since they work like local directory mounts from an applications point of view. So if no external storage is directly addressed like S3 or NFS we have to introduce a human understandable directory where the local user data always gets written if nothing gets configured and default values are used. The same applies for Docker volumes. Proposal for the default directory (user data) --> /alpinefiles/data Proposal for the default config file --> /alpinefiles/config/local.json The most critical part of scaling comes down to the active state the application has. In our case we need to look for possible pitfalls regarding storage and local user handling. **Scenario: Raw Binary (on e.g. 3x machines)** The binary is executed on multiple linux hosts and the initial admin gets written to the local file on machine #1. The documentation has to push the user to distribute the created file to machines #2 and #3 (e.g. with some sync job) or use a common mount (e.g. NFS) to keep local users in sync. The same applies to the data directory. There also has to be a load balancer in front of all machines since we cannot provide reasonable balancing mechanisms on a network level. **Scenario: Docker** As in the raw binary example we need to document that the docker volumes have to be synced across all hosts. The other option would be to directly use external volumes with docker (e.g. NFS, S3). **Scenario: Kubernetes** The mounted PVC must be in RWX access mode or a pre-filled local user config must be created as Secret to be mounted to the respective path. More elegant would be if the initial wizard would offer a checkbox "Create Kubernetes Secret" that would create a Secret with the entered credentials for the first admin (if the checkbox is checked the first thing is to try to communicate with the Kubernetes API and log + notify if there are missing RBAC permissions to create a secret or if the API is at all reachable). If this is done, we can use the built-in scaling of Kubernetes. If I can think of more. I will add another issue.
Owner

Notes from the #922 research (2026-10-16) for the horizontal-scaling design. Source: scratch/b7e/batch907/922-research-2.md.

Version data safety: for file versions, #922 is designed so data safety needs no cross-instance lock (record-then-publish). A shared lock would only be needed for ordering, behind a PathSerializer interface.

What #742 must still solve elsewhere:

  • statestore boot lock: it refuses to start a second instance.
  • #315 user file locks: they live in one process's memory plus a JSON state file, so they must become a shared store.
  • ownermeta: its manifests are written without coordination.
  • Other in-memory state: sessions, rate limits and caches.

Approaches that do not work on Garage or NFS:

  • No fencing: neither can check a fencing token, so locks implemented as lease objects on the backend are not safe.
  • No compare-and-swap: Garage offers none. Its docs call conditional writes "structurally impossible", and K2V has no CAS either.

A coordination service (Redis, etcd, NATS KV or Postgres) is likely needed once more than one instance runs.

**Notes from the #922 research (2026-10-16)** for the horizontal-scaling design. Source: `scratch/b7e/batch907/922-research-2.md`. **Version data safety:** for file versions, #922 is designed so data safety needs no cross-instance lock (record-then-publish). A shared lock would only be needed for ordering, behind a `PathSerializer` interface. **What #742 must still solve elsewhere:** - **statestore boot lock:** it refuses to start a second instance. - **#315 user file locks:** they live in one process's memory plus a JSON state file, so they must become a shared store. - **ownermeta:** its manifests are written without coordination. - **Other in-memory state:** sessions, rate limits and caches. **Approaches that do not work on Garage or NFS:** - **No fencing:** neither can check a fencing token, so locks implemented as lease objects on the backend are not safe. - **No compare-and-swap:** Garage offers none. Its docs call conditional writes "structurally impossible", and K2V has no CAS either. A coordination service (Redis, etcd, NATS KV or Postgres) is likely needed once more than one instance runs.
Owner

From #922 (v0.6.347), what this means for scale-out:

  • Versions are safe with several instances on one backend. Every save records its content before it publishes, and each save's commit check is made against the backend. The local check ran two instances on one data root: 480 concurrent writes, 0 lost.
  • The cross-instance lock has a home. The versions engine takes its commit lock through versions.PathSerializer. Today that is an in-process implementation. A cross-instance one plugs in there without touching the engine; its Held.Lost() reports a lost lease, and the engine then skips pruning. Decision D5: never lease objects on the backend.
  • Clocks must be synchronised. The one-hour prune grace compares key times written by different instances, so instances need NTP (skew well under 30 minutes).
  • The statestore boot lock is keyed by hostname. Two processes with the same hostname are both admitted; a second instance on another hostname is refused.
  • Gaps that remain: sessions are per instance, owner metadata races, and a WebDAV If-Match check can be overtaken by a save in between (#951).
From #922 (v0.6.347), what this means for scale-out: - **Versions are safe with several instances on one backend.** Every save records its content before it publishes, and each save's commit check is made against the backend. The local check ran two instances on one data root: 480 concurrent writes, 0 lost. - **The cross-instance lock has a home.** The versions engine takes its commit lock through `versions.PathSerializer`. Today that is an in-process implementation. A cross-instance one plugs in there without touching the engine; its `Held.Lost()` reports a lost lease, and the engine then skips pruning. Decision D5: never lease objects on the backend. - **Clocks must be synchronised.** The one-hour prune grace compares key times written by different instances, so instances need NTP (skew well under 30 minutes). - **The statestore boot lock is keyed by hostname.** Two processes with the same hostname are both admitted; a second instance on another hostname is refused. - **Gaps that remain:** sessions are per instance, owner metadata races, and a WebDAV `If-Match` check can be overtaken by a save in between (#951).
Owner

Multi-instance notes from batch 3 (v0.6.350, 2026-10-19). Each is safe today on one instance and needs the cross-instance work tracked here.

  1. statestore writes every state file through a fixed temp name, so two instances writing the same file at once can corrupt it. One case: two instances writing the space registry during a convergence. Use a unique temp name plus an atomic rename, and compare-and-set where the backend allows.
  2. spaceids writers outside the convergence (Ensure, Rename, RetireAlias, Forget) still persist the whole in-memory cache, so a stale instance can overwrite another's fields. The convergence writers re-read the record on disk and change only their own fields since v0.6.350. The other writers need the same.
  3. Space restore and purge serialize through the versions engine's serializer seam, which is meant for short locks but is held here for a whole S3 copy, and Lost() is ignored. Correctness no longer depends on the lock: the restore and purge each write their own flag object and back off if they see the other's. Throughput and lease-loss handling do.
  4. Stale spaces cache: an instance started before a space was deleted elsewhere can judge a real, interrupted restore entry as stale and drop it. Nothing is destroyed (the folder and tombstone stay), but only an admin can then bring the space back.
  5. Narrow window in delete: an instance with a stale registry can delete a space just after another instance records MovingTo and before it copies anything. If an S3 delete and a convergence copy then overlap, a duplicate partial copy can be left at the new name. No file is lost.
  6. Stale MovingTo clear: in one narrow race, a stale instance can clear another instance's MovingTo. If that instance then crashes mid-copy, the admin row reads folder-exists; no file is lost.
Multi-instance notes from batch 3 (v0.6.350, 2026-10-19). Each is safe today on one instance and needs the cross-instance work tracked here. 1. **statestore** writes every state file through a fixed temp name, so two instances writing the same file at once can corrupt it. One case: two instances writing the space registry during a convergence. Use a unique temp name plus an atomic rename, and compare-and-set where the backend allows. 2. **spaceids writers outside the convergence** (`Ensure`, `Rename`, `RetireAlias`, `Forget`) still persist the whole in-memory cache, so a stale instance can overwrite another's fields. The convergence writers re-read the record on disk and change only their own fields since v0.6.350. The other writers need the same. 3. **Space restore and purge** serialize through the versions engine's serializer seam, which is meant for short locks but is held here for a whole S3 copy, and `Lost()` is ignored. Correctness no longer depends on the lock: the restore and purge each write their own flag object and back off if they see the other's. Throughput and lease-loss handling do. 4. **Stale spaces cache:** an instance started before a space was deleted elsewhere can judge a real, interrupted restore entry as stale and drop it. Nothing is destroyed (the folder and tombstone stay), but only an admin can then bring the space back. 5. **Narrow window in delete:** an instance with a stale registry can delete a space just after another instance records `MovingTo` and before it copies anything. If an S3 delete and a convergence copy then overlap, a duplicate partial copy can be left at the new name. No file is lost. 6. **Stale `MovingTo` clear:** in one narrow race, a stale instance can clear another instance's `MovingTo`. If that instance then crashes mid-copy, the admin row reads `folder-exists`; no file is lost.
Owner

Another gap from the batch 4 research (research-b4.md §7.4, from reading the code).

The change journal is per instance. changes.Open loads one statestore key at start. Each instance then keeps its own ring and flushes it to that same key (internal/changes/changes.go:95-193).

Two instances that start from the same clean state resume the same epoch and sequence, and then diverge. A cursor issued before the divergence is accepted by both, because its hash is in both histories. A client that keeps polling one instance then silently misses changes made through the other. That breaks the change feed's promise to never bridge a gap silently.

  • Minimum fix: each instance gets its own epoch, so cross-instance cursors expire instead of skipping changes.
  • Real fix: a shared journal.

This is not yet in ARCHITECTURE's list of scale-out gaps.

Another gap from the batch 4 research (`research-b4.md` §7.4, from reading the code). **The change journal is per instance.** `changes.Open` loads one statestore key at start. Each instance then keeps its own ring and flushes it to that same key (`internal/changes/changes.go:95-193`). Two instances that start from the same clean state resume the same epoch and sequence, and then diverge. A cursor issued before the divergence is accepted by both, because its hash is in both histories. A client that keeps polling one instance then silently misses changes made through the other. That breaks the change feed's promise to never bridge a gap silently. - **Minimum fix:** each instance gets its own epoch, so cross-instance cursors expire instead of skipping changes. - **Real fix:** a shared journal. This is not yet in ARCHITECTURE's list of scale-out gaps.
Owner

More from batch 4 (2026-10-20), after a CI failure in TestConvergeTwoInstancesRandom931. The race was real and also exists on v0.6.350; it is fixed in v0.6.351.

Fixed: a second instance decided "folder-missing" from its out-of-date copy of the space registry after the other instance had already moved the folder. The refusal is now re-checked against the registry on disk. A guarded DropMove write also stops the "copies-left" reason from landing on a space another instance has already finished.

Still open, needs the cross-instance work tracked here:

  1. No compare-and-swap on the space registry. Two instances writing it at the same moment can lose one write. In about 1 in 1,000 to 3,000 random interleavings in the test rig, a finished move is overwritten. The files are all in the new folder, but the space reads as "folder missing". Retry now does not help; Finish rename by an admin does, and no files are lost.
    • Fix options: compare-and-swap writes for the registry, or have the S3 rename recognise a second mover that started after the source markers were deleted.
  2. Finish rename on a stale instance. Where the instance's registry copy already shows the rename as finished, it answers rename-finished without re-reading the registry.
  3. One stale "moving" answer. On the instance that recorded the move, the first member read after the other instance finished it answers "moving" once.
More from batch 4 (2026-10-20), after a CI failure in `TestConvergeTwoInstancesRandom931`. The race was real and also exists on v0.6.350; it is fixed in v0.6.351. **Fixed:** a second instance decided "folder-missing" from its out-of-date copy of the space registry after the other instance had already moved the folder. The refusal is now re-checked against the registry on disk. A guarded `DropMove` write also stops the "copies-left" reason from landing on a space another instance has already finished. **Still open, needs the cross-instance work tracked here:** 1. **No compare-and-swap on the space registry.** Two instances writing it at the same moment can lose one write. In about 1 in 1,000 to 3,000 random interleavings in the test rig, a finished move is overwritten. The files are all in the new folder, but the space reads as "folder missing". Retry now does not help; Finish rename by an admin does, and no files are lost. - Fix options: compare-and-swap writes for the registry, or have the S3 rename recognise a second mover that started after the source markers were deleted. 2. **Finish rename on a stale instance.** Where the instance's registry copy already shows the rename as finished, it answers `rename-finished` without re-reading the registry. 3. **One stale "moving" answer.** On the instance that recorded the move, the first member read after the other instance finished it answers "moving" once.
Owner

Notes from the #985 research (research-985-1 §8, -2 §6, -3 §7, -4 §6), all to be held by the multi-instance design:

  1. OIDC login state is per process (oidclib.go:119, 204, 274). A callback that reaches another instance fails as "unknown or expired state", today as the bare "login failed" page. Needs sticky sessions or a shared state store. VERIFIED (code).
  2. Throttle is per instance (memory, throttle.go:13-16): N instances give N times the attempt budget. AD lockout stays the only global cap; this strengthens #992 (refused-credential cache) and #991 (timing).
  3. Certificates card: each instance computes its own TLS state (trust.go:10-11). Proposed: one statestore document with one entry per instance id, each instance rewriting only its own entry every 5 min (the spacealerts merge), "not reporting since" after 15 min, pruned after 24 h; plus low-cardinality per-instance metrics. Depends on item 1 of comment 9729 (unique temp names) on the local store.
  4. Observed LDAP/OIDC facts in Users & access are per instance (own connections, resolver map, credential cache): labelled "as seen by this instance" until a shared store exists; never merged across instances.
  5. Shared files device-pauses.json (mass-miss) and the audit JSONL need this issue's writer rules; each instance may raise its own mass-miss bell item.
  6. Config that must be identical on every instance: auth.passwordChangeUrl, help texts, TLS trust.
Notes from the #985 research (research-985-1 §8, -2 §6, -3 §7, -4 §6), all to be held by the multi-instance design: 1. **OIDC login state is per process** (`oidclib.go:119, 204, 274`). A callback that reaches another instance fails as "unknown or expired state", today as the bare "login failed" page. Needs sticky sessions or a shared state store. VERIFIED (code). 2. **Throttle is per instance** (memory, `throttle.go:13-16`): N instances give N times the attempt budget. AD lockout stays the only global cap; this strengthens #992 (refused-credential cache) and #991 (timing). 3. **Certificates card:** each instance computes its own TLS state (`trust.go:10-11`). Proposed: one statestore document with one entry per instance id, each instance rewriting only its own entry every 5 min (the `spacealerts` merge), "not reporting since" after 15 min, pruned after 24 h; plus low-cardinality per-instance metrics. Depends on item 1 of comment 9729 (unique temp names) on the local store. 4. **Observed LDAP/OIDC facts** in Users & access are per instance (own connections, resolver map, credential cache): labelled "as seen by this instance" until a shared store exists; never merged across instances. 5. **Shared files** `device-pauses.json` (mass-miss) and the audit JSONL need this issue's writer rules; each instance may raise its own mass-miss bell item. 6. Config that must be identical on every instance: `auth.passwordChangeUrl`, help texts, TLS trust.
Owner

Decision (Nikola, 2026-10-27): multi-instance is a v0.7 goal.

Next: research the coordination store following the playbook: Redis, etcd, NATS KV, Postgres, and the option of no extra store (backend-only, per comment 9551). Then a decision for Nikola, and a plan. The plan covers the issues that depend on this one: #1011 (whole-file rewrites of state), #945, #944, #942, #950, #958 (versions across instances), #1020 (lease handover at upgrade) and #1019 (split deployments).

From now on, new multi-instance findings are added here as comments rather than as new issues.

**Decision (Nikola, 2026-10-27):** multi-instance is a v0.7 goal. **Next:** research the coordination store following the playbook: Redis, etcd, NATS KV, Postgres, and the option of no extra store (backend-only, per comment 9551). Then a decision for Nikola, and a plan. The plan covers the issues that depend on this one: #1011 (whole-file rewrites of state), #945, #944, #942, #950, #958 (versions across instances), #1020 (lease handover at upgrade) and #1019 (split deployments). From now on, new multi-instance findings are added here as comments rather than as new issues.
Owner

Research: coordination for multi-instance, 2026-10-27. Full file: scratch/b7e/backlog/research-742-coordination.md, with claims marked VERIFIED, INFERRED or UNVERIFIED.

What breaks today with 2 replicas:

  • On S3, the second instance refuses to start, because the startup lock is keyed by hostname (statestore/lock.go:306).
  • On a shared POSIX volume nothing refuses, so the roughly 45 state files become last-writer-wins (#1011).
  • Per process today:
    • sessions and their revocation;
    • back-channel logout, OIDC login state and device-flow codes;
    • share password gates and throttles;
    • the change journal and the event hub;
    • user and WebDAV locks, tus uploads, background loops;
    • the caches that decide access.
  • New, recorded here:
    • The WOPI token key is random per boot (office/token.go:37), so a Collabora callback that reaches another instance fails.
    • The S3 audit seal is one hash chain whose next number comes from a listing; two instances would fork it.
    • From R1 (v0.6.359): on the S3 state backend the download-mark secret has no true create-only write (a read-back after 2 s), and the marks ledger is per instance, so Trace finds a mark only on the instance that served it.

The backend can't coordinate: Garage accepts If-None-Match and silently ignores it (VERIFIED: its docs, and a Lance report with Garage v2.4.1). NFS locks are unreliable (ownCloud's cluster docs). Correctness has to come from compare-and-swap with fencing tokens on the store that holds the state (Kleppmann; Jepsen on etcd).

Recommendation, tiered:

  • Tier 0 (one instance, the default): no new dependency; add the missing startup lock on POSIX.
  • Tier 1 (2+ instances): one required store, PostgreSQL. It holds the state documents (still age-encrypted), the short-lived shared state, leases with fencing, and change notifications.
    • Valkey only as a cache, ever: its replication is asynchronous, and Redis 8 is AGPL.
    • NATS JetStream KV is the second choice (oCIS and OpenCloud use it), but Jepsen found data loss and split-brain with its default fsync.
    • etcd needs 3 nodes, too heavy for small operators.

Order of work:

  • Step 0, no new dependency:
    • Version-checked, record-level writes for every state file (#1011).
    • A stable WOPI key; OIDC state in an encrypted cookie.
    • A journal epoch per instance.
    • A coordination interface with an in-process version, plus the #1020 lease hand-over.
    • #945, #944, #950.
  • Step 1: the Postgres driver, with an adopt-or-refuse migration.
  • Step 2: sessions, device flow, share gates and throttles move into the store.
  • Step 3:
    • Cross-instance commit ordering (#958, #942).
    • Singleton background jobs; cache invalidation.
    • A shared change journal, locks, the event hub, and per-instance audit chains.
  • Step 4: tus, Helm and compose, then #1019.

Correction: comment 9551 ("never lease objects on the backend") was posted by Cordy, summarising the #922 research. It isn't Manuel's.

Open questions for Nikola (being asked now) and for Manuel:

  • Does Manuel agree with the tiers?
  • Do his customers usually have a managed Postgres?
  • Do they need the store itself highly available?
  • Do they need several instances on NFS/SAN, or S3 only?
**Research: coordination for multi-instance, 2026-10-27.** Full file: `scratch/b7e/backlog/research-742-coordination.md`, with claims marked VERIFIED, INFERRED or UNVERIFIED. **What breaks today with 2 replicas:** - On S3, the second instance refuses to start, because the startup lock is keyed by hostname (`statestore/lock.go:306`). - On a shared POSIX volume nothing refuses, so the roughly 45 state files become last-writer-wins (#1011). - Per process today: - sessions and their revocation; - back-channel logout, OIDC login state and device-flow codes; - share password gates and throttles; - the change journal and the event hub; - user and WebDAV locks, tus uploads, background loops; - the caches that decide access. - **New, recorded here:** - The WOPI token key is random per boot (`office/token.go:37`), so a Collabora callback that reaches another instance fails. - The S3 audit seal is one hash chain whose next number comes from a listing; two instances would fork it. - From R1 (v0.6.359): on the S3 state backend the download-mark secret has no true create-only write (a read-back after 2 s), and the marks ledger is per instance, so Trace finds a mark only on the instance that served it. **The backend can't coordinate:** Garage accepts `If-None-Match` and silently ignores it (VERIFIED: its docs, and a Lance report with Garage v2.4.1). NFS locks are unreliable (ownCloud's cluster docs). Correctness has to come from compare-and-swap with fencing tokens on the store that holds the state (Kleppmann; Jepsen on etcd). **Recommendation, tiered:** - **Tier 0 (one instance, the default):** no new dependency; add the missing startup lock on POSIX. - **Tier 1 (2+ instances):** one required store, PostgreSQL. It holds the state documents (still age-encrypted), the short-lived shared state, leases with fencing, and change notifications. - Valkey only as a cache, ever: its replication is asynchronous, and Redis 8 is AGPL. - NATS JetStream KV is the second choice (oCIS and OpenCloud use it), but Jepsen found data loss and split-brain with its default fsync. - etcd needs 3 nodes, too heavy for small operators. **Order of work:** - **Step 0, no new dependency:** - Version-checked, record-level writes for every state file (#1011). - A stable WOPI key; OIDC state in an encrypted cookie. - A journal epoch per instance. - A coordination interface with an in-process version, plus the #1020 lease hand-over. - #945, #944, #950. - **Step 1:** the Postgres driver, with an adopt-or-refuse migration. - **Step 2:** sessions, device flow, share gates and throttles move into the store. - **Step 3:** - Cross-instance commit ordering (#958, #942). - Singleton background jobs; cache invalidation. - A shared change journal, locks, the event hub, and per-instance audit chains. - **Step 4:** tus, Helm and compose, then #1019. Correction: comment 9551 ("never lease objects on the backend") was posted by Cordy, summarising the #922 research. It isn't Manuel's. **Open questions** for Nikola (being asked now) and for Manuel: - Does Manuel agree with the tiers? - Do his customers usually have a managed Postgres? - Do they need the store itself highly available? - Do they need several instances on NFS/SAN, or S3 only?
Owner

Decisions (Nikola, 2026-10-27):

  • Coordination store: waits for Manuel's answers below.
  • Disaster recovery: yes. In a multi-instance setup, Cairn also writes a periodic age-encrypted state snapshot into .cairn-state/ in the bucket, so the bucket plus the recovery key still restores everything.
  • Scope for v0.7: 2-3 replicas at one site, on S3 and on POSIX/NFS. No multi-region.
  • Resumable uploads (tus) across instances: deep research first (running now).

@hyrsh, questions for you before the store is chosen:

  1. Do your public-sector customers usually have a managed PostgreSQL? Would it be acceptable as a required dependency for running 2+ instances (one instance stays dependency-free)? Or would they prefer Redis/Valkey or NATS?
  2. Do they need the coordination store itself to be highly available, or is one managed instance with backups enough?
  3. Do they run several instances on NFS/SAN, or do they expect S3 for that?
  4. Is the Kubernetes Secret for the first admin still wanted once local users live in the store?
  5. Are fixed, separate config and data paths still a requirement?
**Decisions (Nikola, 2026-10-27):** - **Coordination store:** waits for Manuel's answers below. - **Disaster recovery:** yes. In a multi-instance setup, Cairn also writes a periodic age-encrypted state snapshot into `.cairn-state/` in the bucket, so the bucket plus the recovery key still restores everything. - **Scope for v0.7:** 2-3 replicas at one site, on S3 **and on POSIX/NFS**. No multi-region. - **Resumable uploads (tus) across instances:** deep research first (running now). @hyrsh, questions for you before the store is chosen: 1. Do your public-sector customers usually have a managed PostgreSQL? Would it be acceptable as a required dependency for running 2+ instances (one instance stays dependency-free)? Or would they prefer Redis/Valkey or NATS? 2. Do they need the coordination store itself to be highly available, or is one managed instance with backups enough? 3. Do they run several instances on NFS/SAN, or do they expect S3 for that? 4. Is the Kubernetes Secret for the first admin still wanted once local users live in the store? 5. Are fixed, separate config and data paths still a requirement?
Owner

Research 2: resumable uploads, NFS, and locks across instances (2026-10-27). Full file: scratch/b7e/backlog/research-742-tus-nfs.md.

Uploads (tus). Each upload becomes a list of write-once, uniquely named pieces, sealed with the .cairn-state age keys. One progress record holds the confirmed offset and the piece list, and it changes only when the expected version still matches. No shared volume, no sticky sessions, no locks.

  • One instance: the record and the pieces stay in the local spool.
  • 2+ instances: the record moves to the coordination store, and the pieces go to the backend (S3 objects, or files on the NFS root).
  • A resume on another instance takes over as the current writer; the old request fails its next progress update and stops. This is what IETF resumable-upload draft-14 recommends.
  • Finishing streams through the normal Store.Write chain, so quota, locks, holds, versions, encryption and the key check all still apply.
  • Cost: with 2+ instances, each byte crosses the backend about 3 times.
  • Rejected: a shared spool volume, required sticky sessions, adopting tusd (no cross-instance lock, no compare-and-swap in its s3store), and assembling one encrypted stream from S3 multipart parts.
  • Garage supports multipart with no minimum part size, plus abort-incomplete lifecycle rules (VERIFIED).

POSIX/NFS with several instances.

  • What holds across clients: what the posix driver does for file bytes (unique temp names, close-before-rename, hard-link records with the retransmit check, open+fstat).
  • What does not:
    • the shared append-only files (audit, marks ledger, today's spool);
    • the write-time If-Match check, which reads cached attributes;
    • folder listings on other instances, which can lag up to about 60 s.
  • Requirements, checked at boot: NFSv4.1+, hard, no local_lock/nolock, lookupcache=positive, NTP. NFS locks are never used for correctness.
  • The #922 extended NFS probe has never been run.

Locks (WebDAV and user locks). One lock record per file in the coordination store, taken by compare-and-swap. Expiry follows the store's clock, and the write path reads the store directly.

Single-instance findings from this research (in the backlog, planned for an uploads batch):

  • High: one global lock in the tus handler serialises all uploads on an instance.
  • High: finishing runs on the request's connection, so a disconnect or proxy timeout stops it, and a lost final reply says "expired".
  • Unfinished uploads are plaintext in the spool.
  • S3 uploads are a single PUT (5 GB cap on AWS).
  • Quota is not reserved.
  • Lock taking is check-then-write.

Questions for Manuel (@hyrsh):

  • Which NFS version and mount options do your customers use, and do their exports allow hard links?
  • Do their load balancers impose a response timeout on uploads?
**Research 2: resumable uploads, NFS, and locks across instances (2026-10-27).** Full file: `scratch/b7e/backlog/research-742-tus-nfs.md`. **Uploads (tus).** Each upload becomes a list of write-once, uniquely named pieces, sealed with the `.cairn-state` age keys. One progress record holds the confirmed offset and the piece list, and it changes only when the expected version still matches. No shared volume, no sticky sessions, no locks. - **One instance:** the record and the pieces stay in the local spool. - **2+ instances:** the record moves to the coordination store, and the pieces go to the backend (S3 objects, or files on the NFS root). - **A resume on another instance** takes over as the current writer; the old request fails its next progress update and stops. This is what IETF resumable-upload draft-14 recommends. - **Finishing** streams through the normal `Store.Write` chain, so quota, locks, holds, versions, encryption and the key check all still apply. - **Cost:** with 2+ instances, each byte crosses the backend about 3 times. - **Rejected:** a shared spool volume, required sticky sessions, adopting tusd (no cross-instance lock, no compare-and-swap in its s3store), and assembling one encrypted stream from S3 multipart parts. - Garage supports multipart with no minimum part size, plus abort-incomplete lifecycle rules (VERIFIED). **POSIX/NFS with several instances.** - **What holds across clients:** what the posix driver does for file bytes (unique temp names, close-before-rename, hard-link records with the retransmit check, open+fstat). - **What does not:** - the shared append-only files (audit, marks ledger, today's spool); - the write-time `If-Match` check, which reads cached attributes; - folder listings on other instances, which can lag up to about 60 s. - **Requirements, checked at boot:** NFSv4.1+, `hard`, no `local_lock`/`nolock`, `lookupcache=positive`, NTP. NFS locks are never used for correctness. - The #922 extended NFS probe has never been run. **Locks (WebDAV and user locks).** One lock record per file in the coordination store, taken by compare-and-swap. Expiry follows the store's clock, and the write path reads the store directly. **Single-instance findings from this research** (in the backlog, planned for an uploads batch): - **High:** one global lock in the tus handler serialises all uploads on an instance. - **High:** finishing runs on the request's connection, so a disconnect or proxy timeout stops it, and a lost final reply says "expired". - Unfinished uploads are plaintext in the spool. - S3 uploads are a single PUT (5 GB cap on AWS). - Quota is not reserved. - Lock taking is check-then-write. **Questions for Manuel (@hyrsh):** - Which NFS version and mount options do your customers use, and do their exports allow hard links? - Do their load balancers impose a response timeout on uploads?
Owner

Decisions (Nikola, 2026-10-27), research 2:

  • Uploads with 2+ instances: pieces on the backend plus a progress record in the store, accepting about 3x backend traffic. One instance keeps the local spool.
  • Single-instance upload fixes, in the next uploads batch:
    • parallel uploads (no global lock);
    • finishing continues after a disconnect and reports "finishing" or "finished";
    • unfinished uploads are encrypted whenever encryption is on;
    • the declared size is reserved against quota at the start.
  • NFS with 2+ instances: refuse to start on mounts below NFSv4.1 or with soft, local_lock or nolock. One instance only warns.
  • Extended NFS probe: approved. Two short-lived pods on two nodes on a scratch volume, through GitOps. Cairn's data volumes stay untouched, and everything is removed afterwards.
**Decisions (Nikola, 2026-10-27), research 2:** - **Uploads with 2+ instances:** pieces on the backend plus a progress record in the store, accepting about 3x backend traffic. One instance keeps the local spool. - **Single-instance upload fixes, in the next uploads batch:** - parallel uploads (no global lock); - finishing continues after a disconnect and reports "finishing" or "finished"; - unfinished uploads are encrypted whenever encryption is on; - the declared size is reserved against quota at the start. - **NFS with 2+ instances:** refuse to start on mounts below NFSv4.1 or with `soft`, `local_lock` or `nolock`. One instance only warns. - **Extended NFS probe:** approved. Two short-lived pods on two nodes on a scratch volume, through GitOps. Cairn's data volumes stay untouched, and everything is removed afterwards.
Owner

Multi-instance finding (CI, 2026-10-27, v0.6.361 branch; pre-existing, also on v0.6.360): two instances converging one space rename can lose the "folder is now X" registry update.

  • Instance B finishes the move and saves SetBackendDir.
  • Instance A was refused with folder-exists. It then runs ForgetMove, which read the whole registry file before B's save and writes its stale copy after it.
  • Result: the files are all under the new folder, but the record points at the old one, so a fresh instance sees folder-missing.
  • Impact: no data is lost, and Finish rename recovers it. The CI test now accepts this outcome and a deterministic test reproduces it (convergeforget931_test.go).
  • Fix belongs to the coordination step: compare-and-swap on the registry record. Garage has no conditional write, so it can't be done on the S3 state backend today. Not clearing MovingTo after folder-exists would reverse a safety rule (a foreign folder could get merged into).
**Multi-instance finding (CI, 2026-10-27, v0.6.361 branch; pre-existing, also on v0.6.360):** two instances converging one space rename can lose the "folder is now X" registry update. - Instance B finishes the move and saves `SetBackendDir`. - Instance A was refused with folder-exists. It then runs `ForgetMove`, which read the whole registry file before B's save and writes its stale copy after it. - **Result:** the files are all under the new folder, but the record points at the old one, so a fresh instance sees `folder-missing`. - **Impact:** no data is lost, and Finish rename recovers it. The CI test now accepts this outcome and a deterministic test reproduces it (`convergeforget931_test.go`). - **Fix belongs to the coordination step:** compare-and-swap on the registry record. Garage has no conditional write, so it can't be done on the S3 state backend today. Not clearing MovingTo after folder-exists would reverse a safety rule (a foreign folder could get merged into).
Owner

Multi-instance finding (BAP review, v0.6.363 branch; pre-existing class #1011):

  • LocalStore loads local-users.json once per process and rewrites it whole.
  • After an admin reset on instance A, instance B still has the old hash. The old password keeps working on B, and the one-time password is refused there.
  • Any write on B rewrites the file from its stale map and drops A's reset: NoteSignIn every 15 minutes, an admin change, or another account's new-password step.

This is not fail-safe. The fix belongs with #1011: record-level, version-checked writes, then the coordination store.

**Multi-instance finding (BAP review, v0.6.363 branch; pre-existing class #1011):** - `LocalStore` loads `local-users.json` once per process and rewrites it whole. - After an admin reset on instance A, instance B still has the old hash. The old password keeps working on B, and the one-time password is refused there. - Any write on B rewrites the file from its stale map and drops A's reset: `NoteSignIn` every 15 minutes, an admin change, or another account's new-password step. This is not fail-safe. The fix belongs with #1011: record-level, version-checked writes, then the coordination store.
Owner

Multi-instance finding (CI, v0.6.363; present since v0.6.356 via #1007): when two instances restored one deleted space at once, one restore could answer 200 while the other moved the folder back into the trash, leaving an empty live space.

Fixed in v0.6.363 for the normal race:

  • Revive treats a space already live on its folder as success.
  • mayRollBack reads the intents before the entry.

Left open: if the losing restore's revive fails with a real error (a state write failure) while the other's resumed restore finishes after its checks, it can still roll the folder back. This needs a shared lock across instances (coordination store). The restore hook can also run on both instances when both answer 200.

**Multi-instance finding (CI, v0.6.363; present since v0.6.356 via #1007):** when two instances restored one deleted space at once, one restore could answer 200 while the other moved the folder back into the trash, leaving an empty live space. **Fixed in v0.6.363 for the normal race:** - `Revive` treats a space already live on its folder as success. - `mayRollBack` reads the intents before the entry. **Left open:** if the losing restore's revive fails with a real error (a state write failure) while the other's resumed restore finishes after its checks, it can still roll the folder back. This needs a shared lock across instances (coordination store). The restore hook can also run on both instances when both answer 200.
Owner

Multi-instance notes from #904 (v0.6.364):

  • On an instance that did not declare the restore, the WebDAV X-Cairn-Data-Epoch (read from the restores store) and the change feed's dataEpoch can disagree for up to about 60 s.
  • The restores store re-reads storage-restores.json under its lock on the device and Basic hot path every 30 s; on S3 that is one bucket GET per 30 s per instance. This is the same pattern as the #895 pause store.
  • Acks, ticks, closes and dismissals are record-level read-merge-write without compare-and-swap.
  • The witness counter assumes one instance per bucket; it tolerates a racing write with a 60 s time condition.
  • NoteApp's per-process "already noted" set is only a write throttle.
  • The API's folder-scope check reads a copy up to 30 s old, but the store decides again on the disk record.
**Multi-instance notes from #904 (v0.6.364):** - On an instance that did not declare the restore, the WebDAV `X-Cairn-Data-Epoch` (read from the restores store) and the change feed's `dataEpoch` can disagree for up to about 60 s. - The restores store re-reads `storage-restores.json` under its lock on the device and Basic hot path every 30 s; on S3 that is one bucket GET per 30 s per instance. This is the same pattern as the #895 pause store. - Acks, ticks, closes and dismissals are record-level read-merge-write without compare-and-swap. - The witness counter assumes one instance per bucket; it tolerates a racing write with a 60 s time condition. - `NoteApp`'s per-process "already noted" set is only a write throttle. - The API's folder-scope check reads a copy up to 30 s old, but the store decides again on the disk record.
Owner

Multi-instance findings from batch M1 (v0.6.368, #1026 Remove partial copy):

  • Finish rename vs a removal starting on another instance: Finish rename checks for a running removal lease before adopting the folder; a removal that starts on another instance just after that check can delete one batch (≤1000 keys proven to be the copy's) before its next per-batch registry check sees the folder is now the space's. Needs a compare-and-swap or an ordered marker (Finish rename writes "adopting" first; the removal checks it after StartRun).
  • hold.Store is a per-instance cache never re-read (hold.go); Set/Lift write the whole map from it. Only the partial-copy removal reads a fresh Snapshot().
  • Two removals started at the same moment on two instances can both run (no CAS on the lease); they delete the same proven keys.
  • Run counts and the preview cache are per instance; other instances show "Removing…" without counts.
  • runUnreadableAfter (5 min) is tied to the API's 3-minute lease only by a comment.
Multi-instance findings from batch M1 (v0.6.368, #1026 Remove partial copy): - **Finish rename vs a removal starting on another instance:** Finish rename checks for a running removal lease before adopting the folder; a removal that starts on another instance just after that check can delete one batch (≤1000 keys proven to be the copy's) before its next per-batch registry check sees the folder is now the space's. Needs a compare-and-swap or an ordered marker (Finish rename writes "adopting" first; the removal checks it after StartRun). - `hold.Store` is a per-instance cache never re-read (`hold.go`); Set/Lift write the whole map from it. Only the partial-copy removal reads a fresh `Snapshot()`. - Two removals started at the same moment on two instances can both run (no CAS on the lease); they delete the same proven keys. - Run counts and the preview cache are per instance; other instances show "Removing…" without counts. - `runUnreadableAfter` (5 min) is tied to the API's 3-minute lease only by a comment.
Owner

It is very common for software architects to suggest PostgreSQL for this scenario. PostgreSQL has a built-in feature called LISTEN / NOTIFY, which allows it to act as a lightweight pub/sub message broker alongside its primary job as a database.
While using Postgres can simplify your tech stack (since you likely already need a database for file metadata), it has strict architectural trade-offs compared to NATS.
Here is the direct comparison for a horizontally scaled file server setup:

Architectural Breakdown

Feature NATS PostgreSQL (LISTEN / NOTIFY)
Primary Design Built from the ground up for ultra-fast, ephemeral messaging and streaming. Built for data persistence, ACID transactions, and complex queries.
Message Delivery Speed Sub-millisecond latency. Memory-first architecture. Higher latency. Messages go through the database engine and transaction logs.
Throughput Capacity Millions of messages per second without breaking a sweat. Bound by disk I/O and connection limits; usually tops out at a few thousand notifications/sec.
Payload Size Limit 1 MB by default (configurable up to 64 MB if needed). Strict 8,000-byte limit per notification payload.
Queue Groups / Load Balancing Native support. Easily splits background jobs among instances. Not natively supported for NOTIFY. Every listening server gets every message.
Infrastructure Overhead Adds a new piece of infrastructure to maintain (NATS clusters). Zero new infrastructure if you are already using Postgres for metadata.

Why the Architect Suggested Postgres (The Pros)

  1. Tech Stack Simplicity: You don't have to deploy, secure, and monitor a NATS cluster. You get messaging "for free" inside your existing database.
  2. Transactional Messaging: You can update a file record in the database and trigger the cache invalidation event in the exact same database transaction. If the database write fails, the message never sends. This prevents "phantom" cache updates.

Where Postgres Fails for File Servers (The Cons)

  1. The 8KB Payload Limit

The NOTIFY payload in Postgres is strictly capped at 8,000 bytes. While this is enough to send a simple "file_id: 1234 updated" string, it becomes a major bottleneck if you want to pass larger event metadata or small binary payloads directly through the event stream.

  1. The "Thundering Herd" Problem (No Queue Groups)

Postgres LISTEN / NOTIFY is strictly a broadcast mechanism. If you have 10 file server instances listening for events, all 10 instances will receive every single notification.
• For Cache Invalidation, this is perfect (everyone needs to clear their cache).
• For Background Tasks (like generating a thumbnail), this is a disaster. All 10 servers will try to generate the same thumbnail simultaneously unless you write a complex distributed locking mechanism on top of it. NATS handles this natively out of the box with Queue Groups.

  1. Connection Exhaustion

Postgres requires a persistent, open database connection for every active LISTEN session. In highly scaled environments, tying up valuable database connection pools just to wait for real-time events can starve your actual API of data connections. NATS, by contrast, handles hundreds of thousands of concurrent connections effortlessly.

The Verdict: Which one should you pick?

• Choose PostgreSQL if your file server handles a modest workload, you only need it for simple cache invalidation (not background job routing), and your top priority is keeping your infrastructure as simple as possible.
• Choose NATS if your file server needs to handle heavy traffic, routes asynchronous background jobs (thumbnails, virus scans), requires real-time WebSocket broadcasting, or needs room to scale linearly without hitting database bottlenecks.
Are your file servers mostly doing simple cache clearing, or are you also trying to coordinate background processing (like video transcoding, compression, or sync jobs)?

It is very common for software architects to suggest PostgreSQL for this scenario. PostgreSQL has a built-in feature called LISTEN / NOTIFY, which allows it to act as a lightweight pub/sub message broker alongside its primary job as a database. While using Postgres can simplify your tech stack (since you likely already need a database for file metadata), it has strict architectural trade-offs compared to NATS. Here is the direct comparison for a horizontally scaled file server setup: Architectural Breakdown Feature NATS PostgreSQL (LISTEN / NOTIFY) Primary Design Built from the ground up for ultra-fast, ephemeral messaging and streaming. Built for data persistence, ACID transactions, and complex queries. Message Delivery Speed Sub-millisecond latency. Memory-first architecture. Higher latency. Messages go through the database engine and transaction logs. Throughput Capacity Millions of messages per second without breaking a sweat. Bound by disk I/O and connection limits; usually tops out at a few thousand notifications/sec. Payload Size Limit 1 MB by default (configurable up to 64 MB if needed). Strict 8,000-byte limit per notification payload. Queue Groups / Load Balancing Native support. Easily splits background jobs among instances. Not natively supported for NOTIFY. Every listening server gets every message. Infrastructure Overhead Adds a new piece of infrastructure to maintain (NATS clusters). Zero new infrastructure if you are already using Postgres for metadata. Why the Architect Suggested Postgres (The Pros) 1. Tech Stack Simplicity: You don't have to deploy, secure, and monitor a NATS cluster. You get messaging "for free" inside your existing database. 2. Transactional Messaging: You can update a file record in the database and trigger the cache invalidation event in the exact same database transaction. If the database write fails, the message never sends. This prevents "phantom" cache updates. Where Postgres Fails for File Servers (The Cons) 1. The 8KB Payload Limit The NOTIFY payload in Postgres is strictly capped at 8,000 bytes. While this is enough to send a simple "file_id: 1234 updated" string, it becomes a major bottleneck if you want to pass larger event metadata or small binary payloads directly through the event stream. 2. The "Thundering Herd" Problem (No Queue Groups) Postgres LISTEN / NOTIFY is strictly a broadcast mechanism. If you have 10 file server instances listening for events, all 10 instances will receive every single notification. • For Cache Invalidation, this is perfect (everyone needs to clear their cache). • For Background Tasks (like generating a thumbnail), this is a disaster. All 10 servers will try to generate the same thumbnail simultaneously unless you write a complex distributed locking mechanism on top of it. NATS handles this natively out of the box with Queue Groups. 3. Connection Exhaustion Postgres requires a persistent, open database connection for every active LISTEN session. In highly scaled environments, tying up valuable database connection pools just to wait for real-time events can starve your actual API of data connections. NATS, by contrast, handles hundreds of thousands of concurrent connections effortlessly. The Verdict: Which one should you pick? • Choose PostgreSQL if your file server handles a modest workload, you only need it for simple cache invalidation (not background job routing), and your top priority is keeping your infrastructure as simple as possible. • Choose NATS if your file server needs to handle heavy traffic, routes asynchronous background jobs (thumbnails, virus scans), requires real-time WebSocket broadcasting, or needs room to scale linearly without hitting database bottlenecks. Are your file servers mostly doing simple cache clearing, or are you also trying to coordinate background processing (like video transcoding, compression, or sync jobs)?
Owner

Decision (Nikola, 2026-11-05): running several instances stays a requirement for launch (v0.7 Feature complete), after the 2026-11-04 reprioritisation.

On the store (reply to the NATS vs Postgres comparison above): that comparison is about LISTEN/NOTIFY messaging. What #742 mostly needs is durable state with compare-and-set: the state files (#1011), leases with fencing, sessions, locks and upload progress records. At 2-3 instances this is a few hundred operations a second, so throughput does not decide it.

  • Postgres: a commit is on disk by default, and several records can change in one transaction. It is an extra server to run (or a managed one).
  • NATS: it can run inside the Cairn binary, so the instances can form their own cluster with nothing extra to install; OpenCloud and oCIS use it. But Jepsen (NATS 2.12.1, Dec 2025, https://jepsen.io/analyses/nats-2.12.1) found acknowledged writes lost because the default flushes to disk every 2 min, plus split-brain after power loss and committed writes lost when a few files on a minority of nodes are corrupted. It needs sync_interval: always.
  • Compute: not measured yet; a benchmark on the homelab is proposed before the choice.

Step 0 (no new dependency: #1011 record-level writes, stable WOPI key, a separate journal epoch per instance, the POSIX startup lock) does not depend on the choice and can start now.

**Decision (Nikola, 2026-11-05):** running several instances stays a requirement for launch (v0.7 Feature complete), after the 2026-11-04 reprioritisation. **On the store (reply to the NATS vs Postgres comparison above):** that comparison is about LISTEN/NOTIFY messaging. What #742 mostly needs is durable state with compare-and-set: the state files (#1011), leases with fencing, sessions, locks and upload progress records. At 2-3 instances this is a few hundred operations a second, so throughput does not decide it. - **Postgres:** a commit is on disk by default, and several records can change in one transaction. It is an extra server to run (or a managed one). - **NATS:** it can run inside the Cairn binary, so the instances can form their own cluster with nothing extra to install; OpenCloud and oCIS use it. But Jepsen (NATS 2.12.1, Dec 2025, https://jepsen.io/analyses/nats-2.12.1) found acknowledged writes lost because the default flushes to disk every 2 min, plus split-brain after power loss and committed writes lost when a few files on a minority of nodes are corrupted. It needs `sync_interval: always`. - **Compute:** not measured yet; a benchmark on the homelab is proposed before the choice. Step 0 (no new dependency: #1011 record-level writes, stable WOPI key, a separate journal epoch per instance, the POSIX startup lock) does not depend on the choice and can start now.
Owner

Research 3: Step 0, the work that needs no store (2026-11-05). Full file: scratch/b7e/backlog/research-742-step0.md (code at 666c3d3b, claims marked VERIFIED, INFERRED or UNVERIFIED).

Inventory: 64 bound state documents now, not ~45, plus 8 record folders.

  • 19 are still rewritten whole from one process's memory: app passwords, local users, groups, admins, holds, locks, notify, OCM, license, settings and 9 others.
  • 25 already read, merge and write, but nothing checks a version.
  • 12 places run their own lease or claim as "save, then read back". None is fenced.

Recommendations:

  1. #1011 (L, 3 waves): statestore gets a versioned API (LoadV, SaveIf, Update(fn)) with an opaque version. Files and .cairn-state objects check it by content hash under the process lock. Every writer applies only its own change to the document as read. Caches that decide access re-read every 10 s. The chosen store later implements the same API, so no store code changes again.
    • Without a store this gives full correctness within one process, and only detection across processes: Garage ignores conditional writes, and NFS locks cannot fence a temp-then-rename write.
  2. Stable WOPI key (S): a keyring document beside item-id-key, with a key id in each token so keys can rotate.
    • Finding: this alone changes nothing users see. The editor session table (#1056) is per process, so another instance still answers 404 editor-session-unknown. The table has to move to the store (Step 2).
  3. OIDC state (M): the per-flow cookie from #999 carries the sealed state (AES-GCM, bound to the state hash), SameSite=Lax, 10 min, next capped at 1 KiB. Sign-ins then survive restarts, and the uncapped in-memory state map (a memory DoS) goes away.
  4. Journal epoch (S): finding: a per-instance epoch doesn't fix comment 9902.
    • Since #904 a cursor already expires on the other instance once the two histories differ.
    • A client that only polls A still never sees B's changes; only a shared journal fixes that.
    • Keying epochs by hostname would force a full rescan after every Kubernetes rollout.
    • Proposal: a writer guard now, plus a Log interface for the shared journal.
  5. POSIX startup lock (S-M): flock on the state directory plus the existing heartbeat lock, and the decided NFS mount rules from /proc/mounts. cairnd restored then refuses on POSIX too.
  6. coord interface (M): records with CAS and TTL, leases with fencing tokens and the #1020 take-over rule, best-effort notifications, a log and an instance registry. It comes with an in-process implementation and a conformance suite that the NATS and Postgres drivers must pass later.
  7. #945 (M), #944 (M), #950 (S-M): all still open; none needs the store. Proposal: move them to v0.7.

Waves:

  • W1: versioned statestore, coord, POSIX lock, keyring.
  • W2: High-severity writers, WOPI codec, OIDC cookie, #945, #944.
  • W3: Medium writers, read-merge-write stores, leases, journal guard, #950.
  • W4: docs.

Eight questions for Nikola, each with a recommendation, are in section 11 of the file.

**Research 3: Step 0, the work that needs no store (2026-11-05).** Full file: `scratch/b7e/backlog/research-742-step0.md` (code at 666c3d3b, claims marked VERIFIED, INFERRED or UNVERIFIED). **Inventory:** 64 bound state documents now, not ~45, plus 8 record folders. - 19 are still rewritten whole from one process's memory: app passwords, local users, groups, admins, holds, locks, notify, OCM, license, settings and 9 others. - 25 already read, merge and write, but nothing checks a version. - 12 places run their own lease or claim as "save, then read back". None is fenced. **Recommendations:** 1. **#1011 (L, 3 waves):** `statestore` gets a versioned API (`LoadV`, `SaveIf`, `Update(fn)`) with an opaque version. Files and `.cairn-state` objects check it by content hash under the process lock. Every writer applies only its own change to the document as read. Caches that decide access re-read every 10 s. The chosen store later implements the same API, so no store code changes again. - Without a store this gives full correctness within one process, and only detection across processes: Garage ignores conditional writes, and NFS locks cannot fence a temp-then-rename write. 2. **Stable WOPI key (S):** a keyring document beside `item-id-key`, with a key id in each token so keys can rotate. - **Finding:** this alone changes nothing users see. The editor session table (#1056) is per process, so another instance still answers 404 `editor-session-unknown`. The table has to move to the store (Step 2). 3. **OIDC state (M):** the per-flow cookie from #999 carries the sealed state (AES-GCM, bound to the state hash), `SameSite=Lax`, 10 min, `next` capped at 1 KiB. Sign-ins then survive restarts, and the uncapped in-memory state map (a memory DoS) goes away. 4. **Journal epoch (S):** **finding:** a per-instance epoch doesn't fix comment 9902. - Since #904 a cursor already expires on the other instance once the two histories differ. - A client that only polls A still never sees B's changes; only a shared journal fixes that. - Keying epochs by hostname would force a full rescan after every Kubernetes rollout. - Proposal: a writer guard now, plus a `Log` interface for the shared journal. 5. **POSIX startup lock (S-M):** `flock` on the state directory plus the existing heartbeat lock, and the decided NFS mount rules from `/proc/mounts`. `cairnd restored` then refuses on POSIX too. 6. **`coord` interface (M):** records with CAS and TTL, leases with fencing tokens and the #1020 take-over rule, best-effort notifications, a log and an instance registry. It comes with an in-process implementation and a conformance suite that the NATS and Postgres drivers must pass later. 7. **#945 (M), #944 (M), #950 (S-M):** all still open; none needs the store. Proposal: move them to v0.7. **Waves:** - W1: versioned statestore, `coord`, POSIX lock, keyring. - W2: High-severity writers, WOPI codec, OIDC cookie, #945, #944. - W3: Medium writers, read-merge-write stores, leases, journal guard, #950. - W4: docs. Eight questions for Nikola, each with a recommendation, are in section 11 of the file.
Owner

Decisions (Nikola, 2026-11-05):

Step 0 (research-742-step0.md): all eight recommendations approved.

  1. The settings API answers 409 when two admins save at once.
  2. WOPI and OIDC keys in one key file, separate from the never-rotating item-id key, rotated by an admin "end all editor sessions" action.
  3. WOPI across instances waits for the editor session table to move into the store (Step 2).
  4. No per-instance journal epoch; a writer guard plus a Log interface for the shared journal (Step 3).
  5. A second process with the same hostname is refused while the first is alive.
  6. No home-made compare-and-swap on POSIX now.
  7. #945, #944 and #950 move into v0.7 (#950 records copied files on both backends).
  8. OIDC state single-use per instance plus the IdP's one-time code and nonce until Step 2.
    Built in 4 waves alongside batch 2.

Benchmark (research-742-benchmark.md): approved as designed. Published numbers from x86 VMs on pve2 (every store node 4 vCPU / 4 GB), plus a limited Pi run. Compared: embedded NATS 2.15 (3 nodes, sync_interval: always and default), Postgres single, Postgres with a synchronous standby, Postgres with 2 Cairn instances. Correctness first (lost acknowledged writes, stale compare-and-swap wins, reused fencing tokens under 8 fault types), then cost, then latency; the method and decision criteria go on the wiki before any run. Nikola creates a power-only Proxmox token for the benchmark VMs when the harness is ready. Results are published on the repo wiki.

**Decisions (Nikola, 2026-11-05):** **Step 0** (research-742-step0.md): all eight recommendations approved. 1. The settings API answers 409 when two admins save at once. 2. WOPI and OIDC keys in one key file, separate from the never-rotating item-id key, rotated by an admin "end all editor sessions" action. 3. WOPI across instances waits for the editor session table to move into the store (Step 2). 4. No per-instance journal epoch; a writer guard plus a `Log` interface for the shared journal (Step 3). 5. A second process with the same hostname is refused while the first is alive. 6. No home-made compare-and-swap on POSIX now. 7. #945, #944 and #950 move into v0.7 (#950 records copied files on both backends). 8. OIDC state single-use per instance plus the IdP's one-time code and nonce until Step 2. Built in 4 waves alongside batch 2. **Benchmark** (research-742-benchmark.md): approved as designed. Published numbers from x86 VMs on pve2 (every store node 4 vCPU / 4 GB), plus a limited Pi run. Compared: embedded NATS 2.15 (3 nodes, `sync_interval: always` and default), Postgres single, Postgres with a synchronous standby, Postgres with 2 Cairn instances. Correctness first (lost acknowledged writes, stale compare-and-swap wins, reused fencing tokens under 8 fault types), then cost, then latency; the method and decision criteria go on the wiki before any run. Nikola creates a power-only Proxmox token for the benchmark VMs when the harness is ready. Results are published on the repo wiki.
Owner

Decisions (Nikola, 2026-11-06), Step 0 spec points (research-742-step0-unsettled.md), all approved:

  • A1: "End all editor sessions" rotates only the WOPI key.
  • A2: a settings save without a version is refused (428).
  • A3: on 409 the page keeps the edits, no auto refresh; each settings form sends the version it was loaded with (fixes a silent revert found in the research); the scan panel no longer reloads over unsaved edits.
  • A4: copy-record, scan-outage and data-export leases stay inside their own records.
  • A5: a second process on the same host waits up to about 130 s.
  • A6: locks re-read every 10 s.
  • A7: the extra read per state save on S3 is accepted and measured (requests and bytes).
  • A8: peering transfers interrupted at start become a hard gate for Step 1; new transfer records get Instance and Run stamped now.
  • A9-A10: statestore.Write kept one release; deterministic two-process tests in CI, random soak manual.
  • The 13 new strings of spec §16 are approved.

Benchmark criteria (approved): correctness, then availability (writes resume on the surviving instances after one node is lost, no operator action; Postgres with automatic failover not tested), then cost (operator burden first; differences within the run-to-run spread don't decide), then latency (p99 at 2,000 users above 50 ms per operation, only with clearly separated confidence intervals). Tie-break: Postgres. Postgres with 2 Cairn instances gets the client-side faults only. The method page goes on the wiki now; no external review posts.

**Decisions (Nikola, 2026-11-06), Step 0 spec points** (research-742-step0-unsettled.md), all approved: - A1: "End all editor sessions" rotates only the WOPI key. - A2: a settings save without a version is refused (428). - A3: on 409 the page keeps the edits, no auto refresh; each settings form sends the version it was loaded with (fixes a silent revert found in the research); the scan panel no longer reloads over unsaved edits. - A4: copy-record, scan-outage and data-export leases stay inside their own records. - A5: a second process on the same host waits up to about 130 s. - A6: locks re-read every 10 s. - A7: the extra read per state save on S3 is accepted and measured (requests and bytes). - A8: peering transfers interrupted at start become a hard gate for Step 1; new transfer records get `Instance` and `Run` stamped now. - A9-A10: `statestore.Write` kept one release; deterministic two-process tests in CI, random soak manual. - The 13 new strings of spec §16 are approved. **Benchmark criteria** (approved): correctness, then availability (writes resume on the surviving instances after one node is lost, no operator action; Postgres with automatic failover not tested), then cost (operator burden first; differences within the run-to-run spread don't decide), then latency (p99 at 2,000 users above 50 ms per operation, only with clearly separated confidence intervals). Tie-break: Postgres. Postgres with 2 Cairn instances gets the client-side faults only. The method page goes on the wiki now; no external review posts.
Owner

Step 0 shipped in v0.6.401 (PR #1138), live on both dogfoods (pods on v0.6.401, ready, no restarts, no new WARN or ERROR lines). Running more than one Cairn process on one storage root is still not supported; v0.6.401 refuses a second one.

What it does: start lock (flock + heartbeat on the state directory and the POSIX data root; a sealed lock object on S3; a failed read never takes over a live lock; a lock sealed to other keys is taken over only after 130 s without change; a process whose lock was taken over exits), instance key file for WOPI and OIDC state, versioned state saves for every read-merge-write store (guard test, two-process soak), settings 409/428, change-journal writer guard, local delete that cannot leave a usable device behind. Reviews: three wave reviews, a final review and two focused re-reviews of the lock fixes. Residuals: ARCHITECTURE.md, "What Step 0 does not yet do".

S3 cost at idle (A7), Garage request counters over 5 minutes each, whole Garage (other buckets are near idle), no users active:

v0.6.400 v0.6.401
GetObject 12.4/min 30.9/min
HeadObject 35.0/min 54.8/min
ListObjectsV2 8.2/min 9.6/min
PutObject 3.0/min 4.4/min
Total 58.6/min 99.7/min
Bytes read / written 61 / 5 KB/min 47 / 2 KB/min

So about 41 more requests a minute at idle (the version reads before each save, the 10 s re-reads of spaces and quotas, the lock heartbeat), about 1.8 million a month; at typical cloud GET prices that is well under a franc a month. Bytes did not grow. The v0.6.401 sample began about 2.5 minutes after start, so it may include some start-up reads.

Benchmark: harness phase 0 is built (branch bench742, own module bench/coordbench): null, embedded NATS (single node and 3-node R3) and PostgreSQL drivers, a shared conformance suite that the deliberately broken drivers fail, open-loop load generator, history checkers. First findings: an ADR-50 atomic batch over two KV subjects of one stream works on nats-server 2.15.1 (so fenced writes in NATS are one atomic write, as designed); NATS TTLs are whole seconds; only the stream leader answers a direct get when asked, so fresh reads are possible. Next: VMs on pve2 per the plan (Terraform via homelab-config), then the measured runs, results on the wiki page.

**Step 0 shipped in v0.6.401** (PR #1138), live on both dogfoods (pods on v0.6.401, ready, no restarts, no new WARN or ERROR lines). Running more than one Cairn process on one storage root is still not supported; v0.6.401 refuses a second one. What it does: start lock (flock + heartbeat on the state directory and the POSIX data root; a sealed lock object on S3; a failed read never takes over a live lock; a lock sealed to other keys is taken over only after 130 s without change; a process whose lock was taken over exits), instance key file for WOPI and OIDC state, versioned state saves for every read-merge-write store (guard test, two-process soak), settings 409/428, change-journal writer guard, local delete that cannot leave a usable device behind. Reviews: three wave reviews, a final review and two focused re-reviews of the lock fixes. Residuals: ARCHITECTURE.md, "What Step 0 does not yet do". **S3 cost at idle (A7)**, Garage request counters over 5 minutes each, whole Garage (other buckets are near idle), no users active: | | v0.6.400 | v0.6.401 | |---|---|---| | GetObject | 12.4/min | 30.9/min | | HeadObject | 35.0/min | 54.8/min | | ListObjectsV2 | 8.2/min | 9.6/min | | PutObject | 3.0/min | 4.4/min | | **Total** | **58.6/min** | **99.7/min** | | Bytes read / written | 61 / 5 KB/min | 47 / 2 KB/min | So about 41 more requests a minute at idle (the version reads before each save, the 10 s re-reads of spaces and quotas, the lock heartbeat), about 1.8 million a month; at typical cloud GET prices that is well under a franc a month. Bytes did not grow. The v0.6.401 sample began about 2.5 minutes after start, so it may include some start-up reads. **Benchmark:** harness phase 0 is built (branch bench742, own module `bench/coordbench`): null, embedded NATS (single node and 3-node R3) and PostgreSQL drivers, a shared conformance suite that the deliberately broken drivers fail, open-loop load generator, history checkers. First findings: an ADR-50 atomic batch over two KV subjects of one stream works on nats-server 2.15.1 (so fenced writes in NATS are one atomic write, as designed); NATS TTLs are whole seconds; only the stream leader answers a direct get when asked, so fresh reads are possible. Next: VMs on pve2 per the plan (Terraform via homelab-config), then the measured runs, results on the wiki page.
Owner

Benchmark: test bed up, measured runs started (2026-11-06 23:30 UTC).

  • Six VMs on pve2 (homelab-config #48/#50, #49 and fixes #51-#56): 3 store nodes, PostgreSQL 18 primary + standby, controller. Calibration: fdatasync p50 about 0.65 ms (fio, 4 KiB), 0.29 ms per 8 KiB WAL sync (pg_test_fsync); harness timer lateness p99 1.09 ms without spin, 27 µs with -spin 2ms, so the runs use spin.
  • Self-tests on the real servers: PostgreSQL conformance passes over TLS (and caught one SQL type bug the sandbox could not, fixed); the broken plain-sequence journal skips 159 acknowledged entries on a real server, as the design predicted.
  • One harness bug found on the VMs and fixed (0.3.0): in a 3-node NATS preload the two non-preloading nodes stopped their servers and the R3 preload lost quorum.
  • Preloads: NATS 1 M journal entries in about 1 min, PostgreSQL in about 8.5 min.
  • The campaign runs unattended on the controller VM (bench-only SSH key, Proxmox token limited to VM.PowerMgmt on the six VMs, 21 days). First half of the performance matrix (N-def, N-sync, P1) now, then P2, second half, then the fault trials: about 80 h in total. Results go on the wiki page when done.
**Benchmark: test bed up, measured runs started** (2026-11-06 23:30 UTC). - Six VMs on pve2 (homelab-config #48/#50, #49 and fixes #51-#56): 3 store nodes, PostgreSQL 18 primary + standby, controller. Calibration: fdatasync p50 about 0.65 ms (fio, 4 KiB), 0.29 ms per 8 KiB WAL sync (pg_test_fsync); harness timer lateness p99 1.09 ms without spin, 27 µs with `-spin 2ms`, so the runs use spin. - Self-tests on the real servers: PostgreSQL conformance passes over TLS (and caught one SQL type bug the sandbox could not, fixed); the broken plain-sequence journal skips 159 acknowledged entries on a real server, as the design predicted. - One harness bug found on the VMs and fixed (0.3.0): in a 3-node NATS preload the two non-preloading nodes stopped their servers and the R3 preload lost quorum. - Preloads: NATS 1 M journal entries in about 1 min, PostgreSQL in about 8.5 min. - The campaign runs unattended on the controller VM (bench-only SSH key, Proxmox token limited to VM.PowerMgmt on the six VMs, 21 days). First half of the performance matrix (N-def, N-sync, P1) now, then P2, second half, then the fault trials: about 80 h in total. Results go on the wiki page when done.
Owner

v0.6.403 residuals (#1105, #1140, #1106, #1107), also in ARCHITECTURE.md:

  • R1 download limit: with two racing instances one extra download per instance can pass a limit (until a Step 1 coord driver passes coordtest). R2: the limit's audit event and bell can come twice.
  • R3 a link edit or revoke reaches other instances within 10 s.
  • R4 link password sessions and the password limiter stay per process.
  • R5 link-key joins instance-keys.json, which Step 1 still has to bind.
  • R6 gate before a second instance: an instance older than v0.6.403 serves view-only and upload-only links as download links. Close by reusing oldInstances (spacekeys.go) with v0.6.403 as the minimum.
  • Office discovery failure backoff (30 s) and the view-only listing are per process.
  • #1106: usage figures and TUS reservations per process; 4 background measures per process; unflushed group observations lost on a crash; held changes visible elsewhere after the minute + 30 s flush + 10 s re-read; newest-at merge depends on clocks; personal-quota-seen.json is rewritten whole on each flush.
  • #1107: manager rights on another instance can lag up to 10 s (hand-over and delete decide on a fresh read); ownsAppSpace and member-end items read List() without the 10 s re-read (F4); power mode canBreakLocks costs one uncached directory lookup per listing.
**v0.6.403 residuals** (#1105, #1140, #1106, #1107), also in ARCHITECTURE.md: - R1 download limit: with two racing instances one extra download per instance can pass a limit (until a Step 1 coord driver passes coordtest). R2: the limit's audit event and bell can come twice. - R3 a link edit or revoke reaches other instances within 10 s. - R4 link password sessions and the password limiter stay per process. - R5 `link-key` joins `instance-keys.json`, which Step 1 still has to bind. - R6 **gate before a second instance**: an instance older than v0.6.403 serves view-only and upload-only links as download links. Close by reusing `oldInstances` (spacekeys.go) with v0.6.403 as the minimum. - Office discovery failure backoff (30 s) and the view-only listing are per process. - #1106: usage figures and TUS reservations per process; 4 background measures per process; unflushed group observations lost on a crash; held changes visible elsewhere after the minute + 30 s flush + 10 s re-read; newest-at merge depends on clocks; `personal-quota-seen.json` is rewritten whole on each flush. - #1107: manager rights on another instance can lag up to 10 s (hand-over and delete decide on a fresh read); `ownsAppSpace` and member-end items read `List()` without the 10 s re-read (F4); power mode `canBreakLocks` costs one uncached directory lookup per listing.
Owner

v0.6.404 multi-instance residuals (full texts in ARCHITECTURE.md).

  • #1109 branding: settings via Fresh (10 s); images content-addressed in the shared state, index only through statestore.UpdatePath; hourly sweep under coord lease branding-sweep; per-process caches bounded. Residuals: posix window where a Claim and a sweep removal can both land (WARN, re-upload fixes it); a re-upload racing an eviction's delete; a refused save leaves claimed images until the sweep; mixed versions: v0.6.403 drops branding at its first settings save.
  • #1103 sharing e-mails: R1-R15. Notably at-most-once optional mail; per-process Stop page rate limit and rate bucket; LDAP group members from the 30-day members-seen index; a new instance-keys.json voids mailed Stop links; end reminders on their own lease link-ending (up to ~20 min late after a hard kill); limits as two coord updates (a send near a limit can be refused for a moment, never exceeded); shareMailOptional reaches other instances within 10 s and the pin is per instance (set it the same everywhere).
  • #1101 thumbnails: no new statestore documents or leases. M1 cache drops only on the instance whose change journal saw the change (others keep unreachable entries until eviction, the 30-day sweep or a restart; sealed on encrypted instances) until Step 3's shared log; M2 refusal notes and converter backoff per instance; M3 switch changes with the settings refresh; M4 the Previews card shows the answering instance. Compile-cache checks and the stuck-render count are per pod.
**v0.6.404 multi-instance residuals** (full texts in ARCHITECTURE.md). - #1109 branding: settings via Fresh (10 s); images content-addressed in the shared state, index only through `statestore.UpdatePath`; hourly sweep under coord lease `branding-sweep`; per-process caches bounded. Residuals: posix window where a Claim and a sweep removal can both land (WARN, re-upload fixes it); a re-upload racing an eviction's delete; a refused save leaves claimed images until the sweep; mixed versions: v0.6.403 drops `branding` at its first settings save. - #1103 sharing e-mails: R1-R15. Notably at-most-once optional mail; per-process Stop page rate limit and rate bucket; LDAP group members from the 30-day members-seen index; a new `instance-keys.json` voids mailed Stop links; end reminders on their own lease `link-ending` (up to ~20 min late after a hard kill); limits as two coord updates (a send near a limit can be refused for a moment, never exceeded); `shareMailOptional` reaches other instances within 10 s and the pin is per instance (set it the same everywhere). - #1101 thumbnails: no new statestore documents or leases. M1 cache drops only on the instance whose change journal saw the change (others keep unreachable entries until eviction, the 30-day sweep or a restart; sealed on encrypted instances) until Step 3's shared log; M2 refusal notes and converter backoff per instance; M3 switch changes with the settings refresh; M4 the Previews card shows the answering instance. Compile-cache checks and the stuck-render count are per pod.
Owner

v0.6.405 multi-instance residuals (full texts in ARCHITECTURE.md).

  • #1110/#924 versions: R1 v0.6.404 running alongside (or a downgrade then upgrade) prunes versions inside the minimum period without a marker, so such chains can read complete; R2 chains from before v0.6.405 carry no marker until first removed/moved/copied; R3 two keeps at the limit on two instances can exceed it by one; R4 prune at save uses the cached minimum period (a person's delete reads it fresh); R5 an s3 keep dying between copy and delete leaves both keys; R6 the sweep cursor lives in versions-state.json; R8 a stopped sweep can skip one chain for one night; R9 two instances may each post one ceiling notice in a race. Sweep under lease versions-sweep.
  • #1104 restore: M1-M21, notably the change feed stays one-writer until Step 3; move-record entries buffered in the last 2 s before a crash are lost; added-item and rename steps check then act (content is If-Match); names from before the move record come from the 4096-entry change ring; G4 holds only once every instance runs v0.6.405; Garage key.json race on first start; PATCH marker of a dead instance blocks a start up to 2 min; bell dedup is detection-only on Garage; the owner-busy lease stays with the starting instance on resume. Leases rewind.<rootkey> (60 s) and moverec-compact.
**v0.6.405 multi-instance residuals** (full texts in ARCHITECTURE.md). - #1110/#924 versions: R1 v0.6.404 running alongside (or a downgrade then upgrade) prunes versions inside the minimum period without a marker, so such chains can read complete; R2 chains from before v0.6.405 carry no marker until first removed/moved/copied; R3 two keeps at the limit on two instances can exceed it by one; R4 prune at save uses the cached minimum period (a person's delete reads it fresh); R5 an s3 keep dying between copy and delete leaves both keys; R6 the sweep cursor lives in versions-state.json; R8 a stopped sweep can skip one chain for one night; R9 two instances may each post one ceiling notice in a race. Sweep under lease `versions-sweep`. - #1104 restore: M1-M21, notably the change feed stays one-writer until Step 3; move-record entries buffered in the last 2 s before a crash are lost; added-item and rename steps check then act (content is If-Match); names from before the move record come from the 4096-entry change ring; G4 holds only once every instance runs v0.6.405; Garage key.json race on first start; PATCH marker of a dead instance blocks a start up to 2 min; bell dedup is detection-only on Garage; the owner-busy lease stays with the starting instance on resume. Leases `rewind.<rootkey>` (60 s) and `moverec-compact`.
Owner

The fault campaign fault742 is finished: 230 cells, chain4b ended 2026-11-11 03:38 UTC.

Raw result: 175 cells pass and 51 are marked elle INVALID. Only 44 of those 51 fail on elle alone. 005-P1-F5-r4 has no history (a harness error).

Investigation (full report: scratch/b7e/backlog/elle-invalid-742.md, and on cb-ctl ~/coordbench/investigate-elle/). The 44 elle-only failures come from the harness, not the stores:

  1. Wrong consistency model for these clocks. The harness checks elle's default, strict-serializable, using each VM's own wall clock. The VMs sync to different NTP servers. Every inversion found is between ops on different hosts, with gaps of 155 µs to 2.6 ms.
  2. Clocks after a power cut. A VM restarted after a power cut runs on a whole-second hardware clock for its first seconds, because chrony only slews. This produces gaps of 66 to 771 ms in the F3 cells.
  3. History lost on crash. A dead node loses the unflushed tail of its own history, while the other replicas kept its last appends. Elle then aborts (exit 255, reported as INVALID).

All 44 cells pass -c serializable. With a corrected history, they also pass strict-serializable within a stated clock tolerance.

What this means

  • P1 (Postgres) is clean on all 89 cells that produced a history.
  • N-sync (NATS, sync) is clean on 70 of 70.
  • N-def (NATS, default) is clean on 60 of 70. Its only real failures are the 10 F4 cells: a power cut on all three VMs without fsync loses acknowledged writes and breaks fencing.

Next steps (harness only; no re-run needed)

  • Close open invokes as info, and add indeterminate appends for values that were seen but whose write was lost.
  • Report elle exit 255 as ERROR, not INVALID.
  • Gate on -c serializable, and report strict-serializable as a second check with an explicit clock tolerance.
  • Re-check all 230 histories on a copy.
  • For future campaigns: one local chrony server, and wait for clock sync after a reboot.

After that, the P2 fault campaign, the wiki page and the teardown, once you agree.

The **fault campaign `fault742` is finished**: 230 cells, chain4b ended 2026-11-11 03:38 UTC. **Raw result:** 175 cells pass and 51 are marked elle INVALID. Only 44 of those 51 fail on elle alone. 005-P1-F5-r4 has no history (a harness error). **Investigation (full report: `scratch/b7e/backlog/elle-invalid-742.md`, and on cb-ctl `~/coordbench/investigate-elle/`).** The 44 elle-only failures come from the harness, not the stores: 1. **Wrong consistency model for these clocks.** The harness checks elle's default, strict-serializable, using each VM's own wall clock. The VMs sync to different NTP servers. Every inversion found is between ops on different hosts, with gaps of 155 µs to 2.6 ms. 2. **Clocks after a power cut.** A VM restarted after a power cut runs on a whole-second hardware clock for its first seconds, because chrony only slews. This produces gaps of 66 to 771 ms in the F3 cells. 3. **History lost on crash.** A dead node loses the unflushed tail of its own history, while the other replicas kept its last appends. Elle then aborts (exit 255, reported as INVALID). All 44 cells pass `-c serializable`. With a corrected history, they also pass strict-serializable within a stated clock tolerance. **What this means** - P1 (Postgres) is clean on all 89 cells that produced a history. - N-sync (NATS, sync) is clean on 70 of 70. - N-def (NATS, default) is clean on 60 of 70. Its only real failures are the 10 F4 cells: a power cut on all three VMs without fsync loses acknowledged writes and breaks fencing. **Next steps (harness only; no re-run needed)** - Close open invokes as `info`, and add indeterminate appends for values that were seen but whose write was lost. - Report elle exit 255 as ERROR, not INVALID. - Gate on `-c serializable`, and report strict-serializable as a second check with an explicit clock tolerance. - Re-check all 230 histories on a copy. - For future campaigns: one local chrony server, and wait for clock sync after a reboot. After that, the P2 fault campaign, the wiki page and the teardown, once you agree.
Owner

Harness fixed, fault742 re-checked, P2 fault campaign started (2026-11-11)

Harness (coordbench 0.3.2). bench742 commit 0a24708b, homelab-config fe70285:

  • Elle now gates on -c serializable. Strict-serializable is still reported, but separately and not as a gate.
  • Crash holes are closed as info.
  • Appends lost with a crashed node's history are recovered (152).
  • An elle-cli crash is reported as ERROR.

Re-check of all 230 fault742 cells (copies only; the results are untouched). Report: scratch/b7e/backlog/recheck-fault742.md.

  • 219 PASS.
  • 10 FAIL: all N-def F4, a power cut on all three VMs without fsync. These are real.
  • 1 ERROR: 005-P1-F5-r4 has no history.
  • Overall: P1 and N-sync are clean everywhere.

P2 fault campaign.

  • cb-ctl's disk was grown from 32 to 96 GB (homelab-config PR #57; the plan was one in-place change).
  • The playbook ran with pg_mode p2.
  • chain5 (unit coordbench-chain5) runs check, a fresh P2 preload, then faults --seed 7420 --configs P2 --campaign fault742p2, about 8 h.

Then the wiki page, and the teardown once Nikola agrees.

**Harness fixed, `fault742` re-checked, P2 fault campaign started** (2026-11-11) **Harness (coordbench 0.3.2).** bench742 commit 0a24708b, homelab-config fe70285: - Elle now gates on `-c serializable`. Strict-serializable is still reported, but separately and not as a gate. - Crash holes are closed as `info`. - Appends lost with a crashed node's history are recovered (152). - An elle-cli crash is reported as ERROR. **Re-check of all 230 `fault742` cells** (copies only; the results are untouched). Report: `scratch/b7e/backlog/recheck-fault742.md`. - **219 PASS.** - **10 FAIL:** all N-def F4, a power cut on all three VMs without fsync. These are real. - **1 ERROR:** 005-P1-F5-r4 has no history. - **Overall:** P1 and N-sync are clean everywhere. **P2 fault campaign.** - cb-ctl's disk was grown from 32 to 96 GB (homelab-config PR #57; the plan was one in-place change). - The playbook ran with `pg_mode p2`. - `chain5` (unit `coordbench-chain5`) runs check, a fresh P2 preload, then `faults --seed 7420 --configs P2 --campaign fault742p2`, about 8 h. Then the wiki page, and the teardown once Nikola agrees.
Owner

Benchmark: fault-campaign results are on the wiki (2026-11-12): Coordination store benchmark. The results sit above the published method, which is unchanged. The changelog lists every deviation.

Correctness (step 1), 340 trials, 10 per configuration and fault, after rechecks and reruns:

trials PASS FAIL ERROR
N-def (NATS, default fsync) 70 60 10 0
N-sync (NATS, sync_interval: always) 70 70 0 0
P1 (one PostgreSQL) 90 90 0 0
P2 (PostgreSQL + synchronous standby) 110 110 0 0
  • N-def is disqualified. Power-cutting all three NATS VMs (F4) loses 493 to 1,123 acknowledged appends per trial and breaks fencing, in 10 of 10 trials. This matches Jepsen.
  • N-sync, P1 and P2: none observed. No lost acknowledged write, no duplicate or regressing fencing token, and elle serializable is valid in every trial.
  • Every other stored FAIL or ERROR was the harness, not a store. Causes:
    • cross-host wall-clock comparisons in elle strict-serializable and in the fencing check;
    • the RTC clock of a VM revived after a power cut;
    • crash holes in the histories;
    • LazyFS silently dropping fsyncs once its daemon ran out of file descriptors;
    • one 409 start refusal.
      The page lists each one with the reason its verdict changed.
  • The P1 F5 reruns (005-P1-F5-r4 and 009-P1-F5-r3, same plan lines and seeds, harness 0.3.3) both PASS, with 0 dropped LazyFS writes.

Still open before a store is chosen:

  • Availability (step 2): the N-sync write-unavailability window still has to be extracted from the histories. P1 fails step 2 by rule. P2 fails it as tested (writes stall while the standby is down; promotion is manual).
  • Cost and NATS latency (steps 3 and 4): need the analysis of the perf742 matrix.
  • The 20 NATS F5 cells ran LazyFS with the same 1024 open-file limit, and they have no LazyFS log. They still need a check.

Raw data: 6 archives with their sha256 sums, listed on the page. Nothing has been torn down.

**Benchmark: fault-campaign results are on the wiki** (2026-11-12): [Coordination store benchmark](http://192.168.10.245/Cordy/Cairn/wiki/Coordination-store-benchmark). The results sit above the published method, which is unchanged. The changelog lists every deviation. **Correctness (step 1), 340 trials, 10 per configuration and fault, after rechecks and reruns:** | | trials | PASS | FAIL | ERROR | |---|---|---|---|---| | N-def (NATS, default fsync) | 70 | 60 | **10** | 0 | | N-sync (NATS, `sync_interval: always`) | 70 | 70 | 0 | 0 | | P1 (one PostgreSQL) | 90 | 90 | 0 | 0 | | P2 (PostgreSQL + synchronous standby) | 110 | 110 | 0 | 0 | - **N-def is disqualified.** Power-cutting all three NATS VMs (F4) loses 493 to 1,123 acknowledged appends per trial and breaks fencing, in 10 of 10 trials. This matches Jepsen. - **N-sync, P1 and P2:** none observed. No lost acknowledged write, no duplicate or regressing fencing token, and elle serializable is valid in every trial. - **Every other stored FAIL or ERROR was the harness,** not a store. Causes: - cross-host wall-clock comparisons in elle strict-serializable and in the fencing check; - the RTC clock of a VM revived after a power cut; - crash holes in the histories; - LazyFS silently dropping fsyncs once its daemon ran out of file descriptors; - one 409 start refusal. The page lists each one with the reason its verdict changed. - The P1 F5 reruns (005-P1-F5-r4 and 009-P1-F5-r3, same plan lines and seeds, harness 0.3.3) both PASS, with 0 dropped LazyFS writes. **Still open before a store is chosen:** - Availability (step 2): the N-sync write-unavailability window still has to be extracted from the histories. P1 fails step 2 by rule. P2 fails it as tested (writes stall while the standby is down; promotion is manual). - Cost and NATS latency (steps 3 and 4): need the analysis of the perf742 matrix. - The 20 NATS F5 cells ran LazyFS with the same 1024 open-file limit, and they have no LazyFS log. They still need a check. Raw data: 6 archives with their sha256 sums, listed on the page. Nothing has been torn down.
Owner

Wiki updated with the results for all four decision steps (harness 0.3.4 reruns, chains 8 and 9): Coordination store benchmark

  • Step 1: N-sync, P1 and P2 lost no acknowledged write in any of their 290 trials; N-def lost writes in 10/10 F4 trials and is out.
  • Step 2: only NATS keeps writing after one store VM is lost; since 0.3.4 leases also come back (about 20 s, max 25 s). P1 and P2 fail; P-HA passes by design, untested.
  • Step 3: operator burden favours N-sync (no DB servers); cost is about +80 m CPU and +235 MiB per instance, so the 256 MiB pod default would have to rise to about 384 to 512 MiB.
  • Step 4: does not decide. With distinct seeds N-sync writes are about 1.4x P2 at p50 (perf742's 2x was a seed artefact); every L2000 p99 is under 10 ms.
  • Finding: the steps favour N-sync, with caveats (memory, 3-node minimum, 20 to 30 s disturbed periods, open Jepsen reports, stale reads unmeasured). Decision pending (Nikola). A 5 x 20 min interleaved latency rerun is advised before quoting latency publicly.
Wiki updated with the results for all four decision steps (harness 0.3.4 reruns, chains 8 and 9): [Coordination store benchmark](/Cordy/Cairn/wiki/Coordination-store-benchmark) - Step 1: N-sync, P1 and P2 lost no acknowledged write in any of their 290 trials; N-def lost writes in 10/10 F4 trials and is out. - Step 2: only NATS keeps writing after one store VM is lost; since 0.3.4 leases also come back (about 20 s, max 25 s). P1 and P2 fail; P-HA passes by design, untested. - Step 3: operator burden favours N-sync (no DB servers); cost is about +80 m CPU and +235 MiB per instance, so the 256 MiB pod default would have to rise to about 384 to 512 MiB. - Step 4: does not decide. With distinct seeds N-sync writes are about 1.4x P2 at p50 (perf742's 2x was a seed artefact); every L2000 p99 is under 10 ms. - Finding: the steps favour N-sync, with caveats (memory, 3-node minimum, 20 to 30 s disturbed periods, open Jepsen reports, stale reads unmeasured). Decision pending (Nikola). A 5 x 20 min interleaved latency rerun is advised before quoting latency publicly.
Owner

Decision (Nikola, 2026-11-13): the coordination store is NATS, configuration N-sync (NATS JetStream embedded in the Cairn pods, 3-way replication, fsync on every write). Basis: the benchmark results on the wiki page Coordination store benchmark (harness 0.3.4, chains 1 to 9).

Why, by the agreed four steps:

  1. Correctness: N-sync, P1 and P2 pass every campaign (no lost acknowledged write, no stale compare-and-set win, no reused fencing token). N-def is out: it loses acknowledged writes when all three NATS nodes lose power at once.
  2. Availability: N-sync keeps accepting writes and gets every lease back about 20 s (max 25 s) after a store node is powered off, before the node returns. P2 (PostgreSQL with synchronous standby) stops writes for 54 to 64 s and needs manual promotion if the primary is lost.
  3. Operator cost: N-sync needs no database servers for 3 instances; the cost is about +235 MiB memory and +79 m CPU per Cairn instance.
  4. Latency: N-sync is about 1.4 to 1.7x P2 at p50 with fair seeds; every L2000 p99 is under 9 ms, far below the 50 ms line, so latency does not decide.

Scope added to #742 for v0.7 (Nikola):

  • External NATS option: large deployments can point Cairn at an external NATS cluster instead of the embedded one (connection settings, TLS, credentials, the same streams and KV buckets), at launch and in the v0.7 milestone.
  • Deployment docs are a first-class deliverable: multi-instance deployment is where operators struggle, so the repo handbook gets a complete, tested section (single instance, 3 embedded instances, external NATS; Kubernetes and plain hosts; sizing incl. the pod memory limit; TLS and secrets; backups and restore; adding and replacing a node; failure behaviour and what operators see; upgrades and rollback; troubleshooting). The website docs will be built from it, so it must be exact.

Caveats the wiring batch must cover: raise the default pod memory limit (about 384 to 512 MiB, measured with Cairn itself); per-call timeouts and retries for writes and leases (20 to 30 s of timed-out calls and up to 13 s lease gaps after a leader change); a third NATS-only node for 2-instance setups; node recovery under a memory cap (untested); open NATS correctness reports (#7549, #7556, #7567) tracked; stale replica reads (unmeasured; read from the leader where it decides). Also Wave B of #1011 (state on the store, early invalidation, minimum-version gate).

The test VMs are kept for integration tests of the wiring batch.

**Decision (Nikola, 2026-11-13): the coordination store is NATS, configuration N-sync** (NATS JetStream embedded in the Cairn pods, 3-way replication, fsync on every write). Basis: the benchmark results on the wiki page [Coordination store benchmark](wiki/Coordination-store-benchmark) (harness 0.3.4, chains 1 to 9). **Why, by the agreed four steps:** 1. Correctness: N-sync, P1 and P2 pass every campaign (no lost acknowledged write, no stale compare-and-set win, no reused fencing token). N-def is out: it loses acknowledged writes when all three NATS nodes lose power at once. 2. Availability: N-sync keeps accepting writes and gets every lease back about 20 s (max 25 s) after a store node is powered off, before the node returns. P2 (PostgreSQL with synchronous standby) stops writes for 54 to 64 s and needs manual promotion if the primary is lost. 3. Operator cost: N-sync needs no database servers for 3 instances; the cost is about +235 MiB memory and +79 m CPU per Cairn instance. 4. Latency: N-sync is about 1.4 to 1.7x P2 at p50 with fair seeds; every L2000 p99 is under 9 ms, far below the 50 ms line, so latency does not decide. **Scope added to #742 for v0.7 (Nikola):** - **External NATS option:** large deployments can point Cairn at an external NATS cluster instead of the embedded one (connection settings, TLS, credentials, the same streams and KV buckets), at launch and in the v0.7 milestone. - **Deployment docs are a first-class deliverable:** multi-instance deployment is where operators struggle, so the repo handbook gets a complete, tested section (single instance, 3 embedded instances, external NATS; Kubernetes and plain hosts; sizing incl. the pod memory limit; TLS and secrets; backups and restore; adding and replacing a node; failure behaviour and what operators see; upgrades and rollback; troubleshooting). The website docs will be built from it, so it must be exact. **Caveats the wiring batch must cover:** raise the default pod memory limit (about 384 to 512 MiB, measured with Cairn itself); per-call timeouts and retries for writes and leases (20 to 30 s of timed-out calls and up to 13 s lease gaps after a leader change); a third NATS-only node for 2-instance setups; node recovery under a memory cap (untested); open NATS correctness reports (#7549, #7556, #7567) tracked; stale replica reads (unmeasured; read from the leader where it decides). Also Wave B of #1011 (state on the store, early invalidation, minimum-version gate). The test VMs are kept for integration tests of the wiring batch.
Owner

Step 1 research (2026-11-14): research-742-step1-cairn.md and research-742-step1-market.md.

Key findings:

  • Cairn already has the seams: all 74 state documents go through statestore.Versioned, all leases/claims through coord (memcoord, statecoord) with a conformance suite. Step 1 = a NATS coord driver + a NATS-backed state store + new start path + chart + docs.
  • Not enough for official multi-instance on its own: sessions, the WOPI session table, two-step sign-in, device-flow codes, share password gates, the sign-in throttle, the live-update hub and the change feed epoch are per process (Collabora calls server-to-server, so sticky sessions don't rescue editing).
  • NATS facts that shape the design: a standalone server cannot be turned into a cluster in place (data must be migrated); a 1-node cluster never starts JetStream; the server name is the node identity; KV reads may come from a stale follower (decisions must read from the leader; CAS is still checked at the leader); a plain KV update drops a key's TTL; values are 1 MiB by default; the bundled server version sets upgrade-from rules.
  • Jepsen-reported issues #7549, #7556, #7567, #7545, #7564 are open; sync_interval: always mitigates the crash cases; membership changes are the least-tested operation.
  • Memory: the chart is already at 512Mi since v0.6.404 (PDF thumbnails); a revived NATS node peaked at 830 to 900 MB while catching up.
  • Others that embed NATS embed a single node and send high availability to an external cluster; a clustered server inside each app pod has little precedent, so our runbooks and tests carry more weight.

Open points go to Nikola next.

**Step 1 research (2026-11-14)**: research-742-step1-cairn.md and research-742-step1-market.md. Key findings: - Cairn already has the seams: all 74 state documents go through `statestore.Versioned`, all leases/claims through `coord` (memcoord, statecoord) with a conformance suite. Step 1 = a NATS coord driver + a NATS-backed state store + new start path + chart + docs. - Not enough for official multi-instance on its own: sessions, the WOPI session table, two-step sign-in, device-flow codes, share password gates, the sign-in throttle, the live-update hub and the change feed epoch are per process (Collabora calls server-to-server, so sticky sessions don't rescue editing). - NATS facts that shape the design: a standalone server cannot be turned into a cluster in place (data must be migrated); a 1-node cluster never starts JetStream; the server name is the node identity; KV reads may come from a stale follower (decisions must read from the leader; CAS is still checked at the leader); a plain KV update drops a key's TTL; values are 1 MiB by default; the bundled server version sets upgrade-from rules. - Jepsen-reported issues #7549, #7556, #7567, #7545, #7564 are open; `sync_interval: always` mitigates the crash cases; membership changes are the least-tested operation. - Memory: the chart is already at 512Mi since v0.6.404 (PDF thumbnails); a revived NATS node peaked at 830 to 900 MB while catching up. - Others that embed NATS embed a single node and send high availability to an external cluster; a clustered server inside each app pod has little precedent, so our runbooks and tests carry more weight. Open points go to Nikola next.
Owner

Step 1 decisions (Nikola, 2026-11-14):

  • Two batches in v0.7, multi-instance announced only after both. Batch A (v0.6.409): NATS coord driver and state store (sealed values), embedded and external modes, membership replaces the start lock, #1011 Wave B, live-update hub over NATS, chart (StatefulSet for embedded), cairnd cluster/backup/restore/migrate, full deployment docs. Batch B: sessions, WOPI sessions, two-step, device codes, OIDC state, logout replay, share password gates and the throttle as expiring NATS records; shared change journal.
  • Topology: 1 instance keeps the single-process mode (no NATS); embedded NATS with exactly 3 or 5 instances; 2-server sites use an external NATS cluster.
  • 1 to 3 and back: offline cairnd migrate with a short maintenance window, runbook, tested both ways.
  • Accepted: keep Cairn's lease code on NATS (leader reads, clock check, one power-cut validation run); leader reads for every access decision; Cairn seals every value, NATS at-rest encryption on by default; sync_interval: always forced; mTLS between nodes, client port private; 8 MiB values; sealed backup every 15 min kept 7 days plus cairnd backup/restore; pinned NATS version with upgrade-from rules and n/n-1 skew; external NATS checked at start (R3, version, peers); 1 GiB pod memory until measured; readiness green and degraded during a NATS outage; NFS refused for the store; a "Known limitations" docs section linking the open Jepsen issues.
    Next: mockups and spec for batch A.
**Step 1 decisions (Nikola, 2026-11-14):** - **Two batches in v0.7**, multi-instance announced only after both. Batch A (v0.6.409): NATS coord driver and state store (sealed values), embedded and external modes, membership replaces the start lock, #1011 Wave B, live-update hub over NATS, chart (StatefulSet for embedded), `cairnd cluster/backup/restore/migrate`, full deployment docs. Batch B: sessions, WOPI sessions, two-step, device codes, OIDC state, logout replay, share password gates and the throttle as expiring NATS records; shared change journal. - **Topology:** 1 instance keeps the single-process mode (no NATS); embedded NATS with exactly 3 or 5 instances; 2-server sites use an external NATS cluster. - **1 to 3 and back:** offline `cairnd migrate` with a short maintenance window, runbook, tested both ways. - **Accepted:** keep Cairn's lease code on NATS (leader reads, clock check, one power-cut validation run); leader reads for every access decision; Cairn seals every value, NATS at-rest encryption on by default; `sync_interval: always` forced; mTLS between nodes, client port private; 8 MiB values; sealed backup every 15 min kept 7 days plus `cairnd backup/restore`; pinned NATS version with upgrade-from rules and n/n-1 skew; external NATS checked at start (R3, version, peers); 1 GiB pod memory until measured; readiness green and degraded during a NATS outage; NFS refused for the store; a "Known limitations" docs section linking the open Jepsen issues. Next: mockups and spec for batch A.
Owner

Batch A mockup and spec decisions (Nikola, 2026-11-14): mockup 742a-instances-mockup.html approved (Instances section on Overview, member-down and majority-lost cards, rolling-update skew, bell items, 503 coord-unavailable, CLI outputs) with the research alignment; all 37 researched points accepted (NATS 2.15.3 if the modules are downloaded first, R5 for 5 members, a fail-fast gate so requests don't stall during majority loss, heartbeat-age liveness, at-rest key rotation by rolling restart and re-seal, per-instance sealed audit chains on POSIX, a hardened local admin socket for cairnd cluster); the owner and item-id manifests get version-checked writes in batch A. Next: spec/plan fold, then the build.

**Batch A mockup and spec decisions (Nikola, 2026-11-14):** mockup `742a-instances-mockup.html` approved (Instances section on Overview, member-down and majority-lost cards, rolling-update skew, bell items, 503 `coord-unavailable`, CLI outputs) with the research alignment; all 37 researched points accepted (NATS 2.15.3 if the modules are downloaded first, R5 for 5 members, a fail-fast gate so requests don't stall during majority loss, heartbeat-age liveness, at-rest key rotation by rolling restart and re-seal, per-instance sealed audit chains on POSIX, a hardened local admin socket for `cairnd cluster`); the owner and item-id manifests get version-checked writes in batch A. Next: spec/plan fold, then the build.
Owner

Batch A (v0.6.409) progress, 2026-11-14:

  • W2 (start path, membership, outage answers, cross-instance invalidation, version gates, observability) merged and reviewed by two reviewers; one fix round (fx2) closed every finding. Notable fixes: an unreadable single-instance document no longer counts as "nothing there" (the start path fails closed instead of creating a fresh cluster), settings changes now reach every member (background re-read), restore/export share one lease and check the record inside the swap, backups pause while the cluster is frozen, owner-manifest record names are HMACs, and /metrics never calls NATS.
  • W3 merged: Overview Instances section and cards, rail dot, bell items, localised coord-unavailable / cluster-maintenance, sign-in/guest texts, the four-language maintenance page, Helm chart 0.8.0 (StatefulSet, headless Service, PDB, anti-affinity, grace 120 s).

Decisions (researched in research-742a-w3-unsettled.md):

  1. Every per-deploy coordination key gets a CAIRN_COORD_* variable; the chart sets them all (each member advertises its pod DNS name, per the NATS guidance for TLS routes). No config.json snippet needed on Kubernetes.
  2. External NATS mode on Kubernetes also runs as a StatefulSet (stable member names, no false "unreachable" alarms or new audit chains on rollouts).
  3. The chart refuses an in-place switch while the single-instance data claim exists, and the data claim carries helm.sh/resource-policy: keep.
  4. Guest page: "If it keeps happening, tell the person who invited you." (strings round).
  5. Rotation progress stays out of the Overview in batch A (CLI and metrics have it).
  6. External mode shows the amber "not reachable" card with its own texts (strings round).

Next: W3 review, docs (multi-instance pages), then the VM and Kubernetes runs.

Batch A (v0.6.409) progress, 2026-11-14: - W2 (start path, membership, outage answers, cross-instance invalidation, version gates, observability) merged and reviewed by two reviewers; one fix round (fx2) closed every finding. Notable fixes: an unreadable single-instance document no longer counts as "nothing there" (the start path fails closed instead of creating a fresh cluster), settings changes now reach every member (background re-read), restore/export share one lease and check the record inside the swap, backups pause while the cluster is frozen, owner-manifest record names are HMACs, and `/metrics` never calls NATS. - W3 merged: Overview Instances section and cards, rail dot, bell items, localised `coord-unavailable` / `cluster-maintenance`, sign-in/guest texts, the four-language maintenance page, Helm chart 0.8.0 (StatefulSet, headless Service, PDB, anti-affinity, grace 120 s). Decisions (researched in `research-742a-w3-unsettled.md`): 1. Every per-deploy coordination key gets a `CAIRN_COORD_*` variable; the chart sets them all (each member advertises its pod DNS name, per the NATS guidance for TLS routes). No config.json snippet needed on Kubernetes. 2. External NATS mode on Kubernetes also runs as a StatefulSet (stable member names, no false "unreachable" alarms or new audit chains on rollouts). 3. The chart refuses an in-place switch while the single-instance data claim exists, and the data claim carries `helm.sh/resource-policy: keep`. 4. Guest page: "If it keeps happening, tell the person who invited you." (strings round). 5. Rotation progress stays out of the Overview in batch A (CLI and metrics have it). 6. External mode shows the amber "not reachable" card with its own texts (strings round). Next: W3 review, docs (multi-instance pages), then the VM and Kubernetes runs.
Owner

Batch A shipped: v0.6.409 (PR #1181, tag on 0d954d45). Several instances is in preview.

Coordination: embedded NATS JetStream (3 or 5 members, R3/R5, fsync on every write, per-member at-rest keys) or an external NATS cluster. Every per-deploy key has a CAIRN_COORD_* variable.

Outage behaviour: 503 coord-unavailable / cluster-maintenance, a fail-fast gate, reads keep working, and /readyz stays green.

Cluster lifecycle:

  • cluster record, membership, version gates and the legacy fence;
  • offline cairnd migrate --to nats / --to local;
  • backups and restore, including into new instances;
  • cluster rescue / certs [--ca] / status.

Key rotation: the cluster decides which key it seals with (option D), with a start guard. The re-seal covers values and stored objects, cairnd cluster reseal runs it on demand, and the ledger follows option (b+).

Kubernetes: Overview Instances section, bell kinds, 21 cluster audit verbs, metrics and endpoint. Chart 0.8.0 (StatefulSet in both modes, PDB, per-pod volumes, data claim kept on uninstall).

Docs: full handbook: modes, Kubernetes, plain hosts, runbooks R1-R12, Known limitations.

Verification:

  • Three hosts, two rounds plus a recheck: every runbook. 13 bugs found, all fixed and confirmed on the hosts.
  • Power-cut lease validation, strict, 30 trials: no overlap, tokens strictly increasing, survivors take the lease back within TTL + 30 s.
  • Kubernetes in a homelab scratch namespace: bootstrap, rolling upgrade, pod deletion, node drain, grow 1→3 / shrink 3→1, external NATS. Six findings, all fixed or documented.
  • Dogfoods on v0.6.409 (local mode): both pods ready, no restarts, no new WARN/ERROR lines.

Residuals for batch B and later:

  • Per instance until batch B: sessions, two-step steps, device codes, OIDC state, jti cache, link password gates, throttles, WOPI sessions, change journal, resumable uploads, commit ordering, lock maps.
  • Legacy fence: never start a release older than v0.6.409 on migrated storage.
  • Clocks: leases need synchronised clocks (refused above 30 s).
  • Outages:
    • Up to 1 s of refusals after an outage.
    • The first request of an outage waits 5 s.
    • When a cut-off member rejoins, changes pause for 6 to 12 s.
  • Two-step sign-in in an outage (accepted): a setup made on another instance in the 10-15 s before the outage is not seen. WebDAV Basic with the account password and no cached answer is refused. A member with no good start-record read answers home-held. Strict switch: #1179.
  • Dead members and data layout:
    • Dead member records linger up to 7 days.
    • Hot documents are not split (Wave C).
    • Record directories are not watched (10 s bound).
    • Start-time sweeps run on every instance.
    • A create rewrites the folder's whole owner manifest.
  • External mode:
    • Values are encrypted, not signed (#1176).
    • Key and member records are unauthenticated; they can only add refusals.
    • A move to another NATS server logs nothing (backlog).
  • Rotation:
    • Back-out on plain hosts needs the new identity as legacy first.
    • File content follows each instance's configuration (#1178).
    • Deployment custody (#1177).
  • Kubernetes: members restart after migrate --to local until the StatefulSet is scaled to 0 (documented).

Next: batch B (b10v07), the remaining per-instance state above, then the multi-instance announcement. This issue stays open for batch B.

**Batch A shipped: v0.6.409** (PR #1181, tag on 0d954d45). Several instances is in **preview**. **Coordination:** embedded NATS JetStream (3 or 5 members, R3/R5, fsync on every write, per-member at-rest keys) or an external NATS cluster. Every per-deploy key has a `CAIRN_COORD_*` variable. **Outage behaviour:** 503 `coord-unavailable` / `cluster-maintenance`, a fail-fast gate, reads keep working, and `/readyz` stays green. **Cluster lifecycle:** - cluster record, membership, version gates and the legacy fence; - offline `cairnd migrate --to nats` / `--to local`; - backups and restore, including into new instances; - `cluster rescue` / `certs [--ca]` / `status`. **Key rotation:** the cluster decides which key it seals with (option D), with a start guard. The re-seal covers values and stored objects, `cairnd cluster reseal` runs it on demand, and the ledger follows option (b+). **Kubernetes:** Overview Instances section, bell kinds, 21 cluster audit verbs, metrics and endpoint. Chart 0.8.0 (StatefulSet in both modes, PDB, per-pod volumes, data claim kept on uninstall). **Docs:** full handbook: modes, Kubernetes, plain hosts, runbooks R1-R12, Known limitations. **Verification:** - **Three hosts, two rounds plus a recheck:** every runbook. 13 bugs found, all fixed and confirmed on the hosts. - **Power-cut lease validation, strict, 30 trials:** no overlap, tokens strictly increasing, survivors take the lease back within TTL + 30 s. - **Kubernetes in a homelab scratch namespace:** bootstrap, rolling upgrade, pod deletion, node drain, grow 1→3 / shrink 3→1, external NATS. Six findings, all fixed or documented. - **Dogfoods on v0.6.409 (local mode):** both pods ready, no restarts, no new WARN/ERROR lines. **Residuals for batch B and later:** - **Per instance until batch B:** sessions, two-step steps, device codes, OIDC state, jti cache, link password gates, throttles, WOPI sessions, change journal, resumable uploads, commit ordering, lock maps. - **Legacy fence:** never start a release older than v0.6.409 on migrated storage. - **Clocks:** leases need synchronised clocks (refused above 30 s). - **Outages:** - Up to 1 s of refusals after an outage. - The first request of an outage waits 5 s. - When a cut-off member rejoins, changes pause for 6 to 12 s. - **Two-step sign-in in an outage (accepted):** a setup made on another instance in the 10-15 s before the outage is not seen. WebDAV Basic with the account password and no cached answer is refused. A member with no good start-record read answers `home-held`. Strict switch: #1179. - **Dead members and data layout:** - Dead member records linger up to 7 days. - Hot documents are not split (Wave C). - Record directories are not watched (10 s bound). - Start-time sweeps run on every instance. - A create rewrites the folder's whole owner manifest. - **External mode:** - Values are encrypted, not signed (#1176). - Key and member records are unauthenticated; they can only add refusals. - A move to another NATS server logs nothing (backlog). - **Rotation:** - Back-out on plain hosts needs the new identity as legacy first. - File content follows each instance's configuration (#1178). - Deployment custody (#1177). - **Kubernetes:** members restart after `migrate --to local` until the StatefulSet is scaled to 0 (documented). **Next:** batch B (b10v07), the remaining per-instance state above, then the multi-instance announcement. This issue stays open for batch B.
Owner

v0.6.410 shipped (patch on batch A, PR #1182, tag on 2e3719f7). Live on both dogfoods.

Fixed

  • fx15: cairnd migrate --to nats --dry-run without -config on an S3 instance read the built-in defaults and reported "state documents 0", exit 0. It now names where it read (Single-instance state: ...). When it finds no state document, it refuses with exit 2 and asks for -config.
  • fx15b: holds-confirm-empty and restored had the same blind spot and wrote to the local default directory instead of the bucket. They now name where they write and refuse without -config when the defaults hold no state. audit-verify without -config says so.
  • fx16 (found by CI): cairnd cluster ... run by another user could report "connection ended (broken pipe)" instead of the refusal. The member refuses before it reads the request, and the client stopped at its failed send. The client now reads the answer after a failed send.
  • CI races fixed on the test side:
    • the adminsock raw-write EPIPE;
    • the pdfium recycle log line written after FirstPage returns;
    • the spaceids two-writer starvation under load or -race (backlog 2026-11-13).

New finding: #1183, filed in this milestone for batch 10. A second process using a member's name can join: its own embedded NATS server (same server name) disturbs JetStream, the real member's heartbeat fails, and the duplicate check sees no live record. Already present in v0.6.409; timing-dependent.

Backlog: pdfium can run one render past RecycleEvery, because the slot is freed before finish() retires the module.

v0.6.410 shipped (patch on batch A, PR #1182, tag on 2e3719f7). Live on both dogfoods. **Fixed** - fx15: `cairnd migrate --to nats --dry-run` without `-config` on an S3 instance read the built-in defaults and reported "state documents 0", exit 0. It now names where it read (`Single-instance state: ...`). When it finds no state document, it refuses with exit 2 and asks for `-config`. - fx15b: `holds-confirm-empty` and `restored` had the same blind spot and wrote to the local default directory instead of the bucket. They now name where they write and refuse without `-config` when the defaults hold no state. `audit-verify` without `-config` says so. - fx16 (found by CI): `cairnd cluster ...` run by another user could report "connection ended (broken pipe)" instead of the refusal. The member refuses before it reads the request, and the client stopped at its failed send. The client now reads the answer after a failed send. - CI races fixed on the test side: - the adminsock raw-write EPIPE; - the pdfium recycle log line written after FirstPage returns; - the spaceids two-writer starvation under load or `-race` (backlog 2026-11-13). **New finding:** #1183, filed in this milestone for batch 10. A second process using a member's name can join: its own embedded NATS server (same server name) disturbs JetStream, the real member's heartbeat fails, and the duplicate check sees no live record. Already present in v0.6.409; timing-dependent. **Backlog:** pdfium can run one render past `RecycleEvery`, because the slot is freed before `finish()` retires the module.
Owner

Dogfood migration dry runs (migrate --to nats --dry-run -config ..., read-only), as data points for sizing:

enc openbao (v0.6.410)
state documents 37, 43 KiB 59, 93 KiB
record directories 5 records, 259 B
coordination records 4 (idp fingerprints, mfa)
largest value 9 KiB activity-v2.json 19 KiB (warns at 4 MiB)
result every item can be copied every item can be copied, about 1 s

The openbao run shows the v0.6.410 line Single-instance state: S3 bucket cairn-openbao, state prefix /.cairn-state, so the fx15 source naming works live.

Observation for batch 10: the copy carries the older document generations next to their successors (tags.json/tags-v2.json, favorites, recents, activity/activity-v2, first-signins.prev.json). To check whether anything still reads the v1 files; if nothing does, the migration could skip them or a cleanup could remove them.

Dogfood migration dry runs (`migrate --to nats --dry-run -config ...`, read-only), as data points for sizing: | | enc | openbao (v0.6.410) | |---|---|---| | state documents | 37, 43 KiB | 59, 93 KiB | | record directories | | 5 records, 259 B | | coordination records | | 4 (idp fingerprints, mfa) | | largest value | 9 KiB | activity-v2.json 19 KiB (warns at 4 MiB) | | result | every item can be copied | every item can be copied, about 1 s | The openbao run shows the v0.6.410 line `Single-instance state: S3 bucket cairn-openbao, state prefix /.cairn-state`, so the fx15 source naming works live. Observation for batch 10: the copy carries the older document generations next to their successors (`tags.json`/`tags-v2.json`, `favorites`, `recents`, `activity`/`activity-v2`, `first-signins.prev.json`). To check whether anything still reads the v1 files; if nothing does, the migration could skip them or a cleanup could remove them.
Owner

Batch B research and decisions (2026-11-18). Research: 7 files in .superpowers/sdd/2026-11-18-b10v07/ (sign-in state, editor sessions and link passwords, change feed, cluster lifecycle, keys, residual census, uploads).

Main findings

  • Two processes with one member name can corrupt the cluster (#1183). NATS routes accept a second server with a peer's name, and Raft counts votes by a peer id derived from the name, so lost acknowledged writes are possible (inferred from the nats-server 2.15.3 source).
  • A wrong key rotation back-out can make files unopenable with no warning (the ledger only learns keys in a legacy slot) (#1178).
  • Ending sessions (admin two-step reset, refusing a directory account, identity-provider logout) only takes effect on one instance.
  • Document editing fails across instances.
  • Sync clients can miss changes silently.
  • Uploads of 16 MiB or more, inbound peering and the bell break without affinity.
  • Trash purge and the review reports run on every member.

Decisions (Nikola)

  • Batch B is split. Batch 10 (v0.6.411), safety first:
    • #1183 duplicate guard, in three layers;
    • #1185 plus the 5-member export acknowledgement;
    • #1178 holder-aware ledger, and file content follows the cluster decision;
    • in-record lease Lost();
    • bell re-read, leases for the trash purge, the review reports and device maintenance, Activity re-read;
    • small runtime fixes, two new audit verbs, the back-out warning, and the Known limitations rewrite.
  • Batch 11 (v0.6.412), then the announcement:
    • shared sign-in state: sessions with cluster-wide ending, two-step, device codes, OIDC state, logout replay, throttle;
    • #1179 strict mode;
    • shared editor session table;
    • link password cookie plus a per-link failure cap;
    • change feed on a stream;
    • rotation progress on the Overview;
    • the ship-ready upload fix: any member accepts any piece and finishes any upload, no affinity rule, nothing deferred.
  • Every other researched point is accepted as recommended (decision-sheet-b10.md, decisions-b10.md).
Batch B research and decisions (2026-11-18). Research: 7 files in `.superpowers/sdd/2026-11-18-b10v07/` (sign-in state, editor sessions and link passwords, change feed, cluster lifecycle, keys, residual census, uploads). **Main findings** - Two processes with one member name can corrupt the cluster (#1183). NATS routes accept a second server with a peer's name, and Raft counts votes by a peer id derived from the name, so lost acknowledged writes are possible (inferred from the nats-server 2.15.3 source). - A wrong key rotation back-out can make files unopenable with no warning (the ledger only learns keys in a legacy slot) (#1178). - Ending sessions (admin two-step reset, refusing a directory account, identity-provider logout) only takes effect on one instance. - Document editing fails across instances. - Sync clients can miss changes silently. - Uploads of 16 MiB or more, inbound peering and the bell break without affinity. - Trash purge and the review reports run on every member. **Decisions (Nikola)** - Batch B is split. **Batch 10 (v0.6.411), safety first:** - #1183 duplicate guard, in three layers; - #1185 plus the 5-member export acknowledgement; - #1178 holder-aware ledger, and file content follows the cluster decision; - in-record lease `Lost()`; - bell re-read, leases for the trash purge, the review reports and device maintenance, Activity re-read; - small runtime fixes, two new audit verbs, the back-out warning, and the Known limitations rewrite. - **Batch 11 (v0.6.412), then the announcement:** - shared sign-in state: sessions with cluster-wide ending, two-step, device codes, OIDC state, logout replay, throttle; - #1179 strict mode; - shared editor session table; - link password cookie plus a per-link failure cap; - change feed on a stream; - rotation progress on the Overview; - the ship-ready upload fix: any member accepts any piece and finishes any upload, no affinity rule, nothing deferred. - Every other researched point is accepted as recommended (`decision-sheet-b10.md`, `decisions-b10.md`).
Owner

Batch 10 (v0.6.411) status, 2026-11-19.

Mockup 742b1-safety-mockup.html rev 1 approved by Nikola as drawn: every frame, the 10 new strings in EN/DE/FR/IT, and all 7 open points as recommended.

Build, branch b10v07: all four waves are merged.

  • W1, 4 tasks, plus fix round fx1 for 1 High and 3 Medium: duplicate detection must not depend on clocks, a paused lease holder must never act after Until, the ledger merge, and a CI flake whose root cause is proven.
  • W2, 5 tasks, plus fix round fx2 for 5 Medium and 6 Low: the instanceSeed check in both orders, a forced start never drops an entry with unknown holders, reports run once per period, real-main ports, the bell test wait.
  • W3 UI: web suite (323 files) and a11y suite (93 files) green.
  • W4 docs: Known limitations rewritten; R13 "Forget a lost instance".

Found on the way, fixed in this batch (single instance too)

  • Every personal or administrator data export with more than one collector has failed at the ZIP step since #1023: each collector's staged files overwrote the previous one's.
  • A stale copy-record run could overwrite a newer run's result.
  • Files sealed before a deployment-identity rotation with instanceSeed opened only through the recovery identity.

Next: the final review, then the run on the kept VMs (duplicate member, export with 3 and 5 members, lease under a stopped holder, key back-out, cluster forget), then the release.

Batch 10 (v0.6.411) status, 2026-11-19. **Mockup** `742b1-safety-mockup.html` rev 1 approved by Nikola as drawn: every frame, the 10 new strings in EN/DE/FR/IT, and all 7 open points as recommended. **Build, branch `b10v07`:** all four waves are merged. - **W1, 4 tasks, plus fix round fx1** for 1 High and 3 Medium: duplicate detection must not depend on clocks, a paused lease holder must never act after Until, the ledger merge, and a CI flake whose root cause is proven. - **W2, 5 tasks, plus fix round fx2** for 5 Medium and 6 Low: the `instanceSeed` check in both orders, a forced start never drops an entry with unknown holders, reports run once per period, real-main ports, the bell test wait. - **W3 UI:** web suite (323 files) and a11y suite (93 files) green. - **W4 docs:** Known limitations rewritten; R13 "Forget a lost instance". **Found on the way, fixed in this batch (single instance too)** - Every personal or administrator data export with more than one collector has failed at the ZIP step since #1023: each collector's staged files overwrote the previous one's. - A stale copy-record run could overwrite a newer run's result. - Files sealed before a deployment-identity rotation with `instanceSeed` opened only through the recovery identity. **Next:** the final review, then the run on the kept VMs (duplicate member, export with 3 and 5 members, lease under a stopped holder, key back-out, cluster forget), then the release.
Owner

v0.6.411 shipped (batch B part 1, "safety first"): PR #1186, tag on cf31bffa. Live on both dogfoods (homelab-config 9b16293).

What shipped

  • Duplicate member guard (#1183): route probe, NATS's own duplicate refusal, the record check; after join the newer process stops.
  • Export end for 3 and 5 members (#1185): members acknowledge and keep NATS up until all have; the export is refused while a member is down.
  • In-record leases and job leases: stop at Until by their own clock, also while the store hangs or the holder is paused.
  • Keys follow the cluster's decision (#1178): holder-aware ledger, cairnd cluster forget, key-drain-forced and key-drained, back-out WARN, remote crypto role refused with several instances, instanceSeed checked between members.
  • Census: the bell, Activity, the readiness list, last sign-ins and space cosmetics follow other members; the trash purge, scheduled reports and device maintenance run once per cluster.
  • External mode: logs disconnect and reconnect, re-checks with retries, and records the new server at once.
  • Migration: converted v1 documents are skipped at migrate --to nats.
  • Known limitations rewritten.

Fixed, single instance too

  • Data exports with more than one collector failed since #1023.
  • The outage watch dropped refusals made right after the last leader contact.
  • instanceSeed files sealed before a rotation opened only through the recovery identity.
  • End share race.
  • cluster remove now retries a member change still in flight (10202).

Verification

  • VM run on three hosts (run-s13.md): duplicate refused in 3 to 157 ms; export with 3 members about 6 s, with 5 about 5 s; forget works.
  • The rc2 recheck of the reconnect fix had no degraded spell in 381 polls.
  • One more finding there: a member hung at the very start of an export makes the export refuse. That is the safe outcome, and it is documented.

Follow-ups

  • #1187: the remote crypto role follows the cluster decision (v1.x).
  • Batch 11 (v0.6.412): the shared sign-in state, editor sessions, change feed, uploads, strict mode (#1179) and rotation progress, then the announcement.
**v0.6.411 shipped** (batch B part 1, "safety first"): PR #1186, tag on cf31bffa. Live on both dogfoods (homelab-config 9b16293). **What shipped** - **Duplicate member guard (#1183):** route probe, NATS's own duplicate refusal, the record check; after join the newer process stops. - **Export end for 3 and 5 members (#1185):** members acknowledge and keep NATS up until all have; the export is refused while a member is down. - **In-record leases and job leases:** stop at Until by their own clock, also while the store hangs or the holder is paused. - **Keys follow the cluster's decision (#1178):** holder-aware ledger, `cairnd cluster forget`, `key-drain-forced` and `key-drained`, back-out WARN, remote crypto role refused with several instances, `instanceSeed` checked between members. - **Census:** the bell, Activity, the readiness list, last sign-ins and space cosmetics follow other members; the trash purge, scheduled reports and device maintenance run once per cluster. - **External mode:** logs disconnect and reconnect, re-checks with retries, and records the new server at once. - **Migration:** converted v1 documents are skipped at `migrate --to nats`. - **Known limitations rewritten.** **Fixed, single instance too** - Data exports with more than one collector failed since #1023. - The outage watch dropped refusals made right after the last leader contact. - `instanceSeed` files sealed before a rotation opened only through the recovery identity. - End share race. - `cluster remove` now retries a member change still in flight (10202). **Verification** - VM run on three hosts (`run-s13.md`): duplicate refused in 3 to 157 ms; export with 3 members about 6 s, with 5 about 5 s; `forget` works. - The rc2 recheck of the reconnect fix had no degraded spell in 381 polls. - One more finding there: a member hung at the very start of an export makes the export refuse. That is the safe outcome, and it is documented. **Follow-ups** - #1187: the remote crypto role follows the cluster decision (v1.x). - Batch 11 (v0.6.412): the shared sign-in state, editor sessions, change feed, uploads, strict mode (#1179) and rotation progress, then the announcement.
Owner

R2-Q8 live check (batch 11 input), 2026-11-19: Collabora does not take a WOPI lock on Cairn.

Test: enc dogfood, v0.6.411. A new test document (wopi-lock-check.docx in Nikola's Personal folder) was opened in Collabora, two words were typed, and it was saved.

The pod log shows exactly these WOPI calls:

  • GET /wopi/files/<id> (CheckFileInfo, 283 bytes);
  • GET /wopi/files/<id>/contents (GetFile);
  • POST /wopi/files/<id>/contents (PutFile, 200).

No POST /wopi/files/<id> came, which is where LOCK, REFRESH_LOCK and UNLOCK would arrive. The file never showed a lock.

Cause, VERIFIED at cf31bffa:

  • Cairn implements the WOPI lock state machine (internal/api/wopi.go:721).
  • But CheckFileInfo (wopi.go:142-152) never sets SupportsLocks, SupportsGetLock or SupportsUpdate, so Collabora never uses it.

Consequences:

  • The handbook's statement that the editor holds a lock while a document is open is false today.
  • The "join an open session" logic and lock-based conflict protection against WebDAV and desktop clients are not in effect during editing.

This goes into the batch 11 decision round, researched first: advertise lock support (and what that changes for Collabora, OnlyOffice, the WOPI validator and multi-instance), or keep locks off and correct the docs.

**R2-Q8 live check (batch 11 input), 2026-11-19:** Collabora does **not** take a WOPI lock on Cairn. Test: enc dogfood, v0.6.411. A new test document (`wopi-lock-check.docx` in Nikola's Personal folder) was opened in Collabora, two words were typed, and it was saved. The pod log shows exactly these WOPI calls: - `GET /wopi/files/<id>` (CheckFileInfo, 283 bytes); - `GET /wopi/files/<id>/contents` (GetFile); - `POST /wopi/files/<id>/contents` (PutFile, 200). No `POST /wopi/files/<id>` came, which is where LOCK, REFRESH_LOCK and UNLOCK would arrive. The file never showed a lock. Cause, VERIFIED at cf31bffa: - Cairn implements the WOPI lock state machine (`internal/api/wopi.go:721`). - But CheckFileInfo (`wopi.go:142-152`) never sets `SupportsLocks`, `SupportsGetLock` or `SupportsUpdate`, so Collabora never uses it. Consequences: - The handbook's statement that the editor holds a lock while a document is open is false today. - The "join an open session" logic and lock-based conflict protection against WebDAV and desktop clients are not in effect during editing. This goes into the batch 11 decision round, researched first: advertise lock support (and what that changes for Collabora, OnlyOffice, the WOPI validator and multi-instance), or keep locks off and correct the docs.
Owner

Batch 11 research and decisions (2026-11-20). Research: .superpowers/sdd/2026-11-20-b11v07/research-b11-r8-wopi-locks.md (editor locks) and research-b11-r9-refresh.md (the batch 11 designs re-checked against v0.6.411).

Editor locks
Users expect a document to lock by itself while it is open in the browser editor:

  • the lock is shared by everyone co-editing;
  • writes from outside the editor are refused;
  • it is released when the last person closes, or about 30 minutes after a crash.

Nextcloud (files_lock), oCIS, Seafile Pro and Box all work this way. Nobody locks by hand to edit; manual locks are a separate feature.

Decided: editor locks on. Cairn advertises lock support for writers. The file list shows who is in the editor, with Join. Outside writes are refused while the file is open. Required with the switch:

  • your own manual lock never blocks your own editor;
  • after an editor crash, the reopened session takes over its own lock;
  • no manual lock while the file is open in the editor;
  • Break lock on an editor lock ends that editing session.

WebDAV stops showing lock tokens to readers.

Batch 11 is split in two

  • 11a, v0.6.412: shared building blocks (auth bucket, sealed records, coord-key classification for migrate and backup), nats.go 1.56.0 (a security release), the ship-ready upload design, and the change journal on a stream.
  • 11b, v0.6.413: shared sign-in records, strict mode #1179, the link password gate and per-link cap, the shared editor session table with editor locks, and rotation progress. Then the multi-instance announcement.

All other points in decision-sheet-b11.md are accepted as recommended.

**Batch 11 research and decisions (2026-11-20).** Research: `.superpowers/sdd/2026-11-20-b11v07/research-b11-r8-wopi-locks.md` (editor locks) and `research-b11-r9-refresh.md` (the batch 11 designs re-checked against v0.6.411). **Editor locks** Users expect a document to lock by itself while it is open in the browser editor: - the lock is shared by everyone co-editing; - writes from outside the editor are refused; - it is released when the last person closes, or about 30 minutes after a crash. Nextcloud (files_lock), oCIS, Seafile Pro and Box all work this way. Nobody locks by hand to edit; manual locks are a separate feature. Decided: **editor locks on**. Cairn advertises lock support for writers. The file list shows who is in the editor, with Join. Outside writes are refused while the file is open. Required with the switch: - your own manual lock never blocks your own editor; - after an editor crash, the reopened session takes over its own lock; - no manual lock while the file is open in the editor; - Break lock on an editor lock ends that editing session. WebDAV stops showing lock tokens to readers. **Batch 11 is split in two** - **11a, v0.6.412:** shared building blocks (auth bucket, sealed records, coord-key classification for migrate and backup), nats.go 1.56.0 (a security release), the ship-ready upload design, and the change journal on a stream. - **11b, v0.6.413:** shared sign-in records, strict mode #1179, the link password gate and per-link cap, the shared editor session table with editor locks, and rotation progress. Then the multi-instance announcement. All other points in `decision-sheet-b11.md` are accepted as recommended.
Owner

v0.6.412 shipped (batch B part 2a, PR #1188, merge e147b92b, tag v0.6.412). Live on both dogfoods (homelab-config d201778): ready, 0 restarts, only the standing WARN lines; manual checks on #1005.

What it brings:

  • Uploads on any instance: any member accepts any piece of a resumable upload and finishes any upload. Upload records with an epoch and piece runs are shared; pieces are sealed write-once objects under .cairn-state/tus/; a newer request takes an upload over (the older answers 409 with Upload-Offset); a crash mid-finish rolls back to the last stored piece and the client's resend finishes on any member, never storing a file twice; shared quota reservations; one leased sweep; 503, never 404, during outages. The web client pauses and resumes by itself for up to 10 minutes. The "v0.6.411 only" upload affinity rule is gone from the docs.
  • The shared change feed: the change list for sync clients is a sealed NATS stream shared by every member, with epochs that rotate at the switch, after a gap, a crash, a data-epoch change or a stream loss.
  • Building blocks for 11b: the <prefix>_auth bucket, sealed coord records, the coord-key classification table (also fixes dup.* in backups), switch-on with the cluster's minimum version.
  • nats.go 1.56.0 (security release), tested in embedded and external modes.
  • Operator facts from the VM run (run-u14): TMPDIR on disk for S3 (start WARN on tmpfs), the JetStream account size formula (refused at once with the formula when too small), S3 retry WARN.

CI note: the first PR run failed on a race in TestUploadFinishCrashRealMain742c (a HEAD fences a running PATCH by design, and the test polled HEAD while the PATCH streamed); the test now waits on a deterministic seam. Lesson added to the 11b brief.

Next: batch 11b (v0.6.413): spec and plan written (2026-11-21-b12v07/), spec points settled as consequences of the decisions (decisions-b12.md), whole-page mockup drawn and waiting for approval: shared sign-in records, #1179 strict mode, link gate cookie and per-link cap, the shared editor session table with editor locks, P5 rotation progress; then the multi-instance announcement.

**v0.6.412 shipped** (batch B part 2a, PR #1188, merge e147b92b, tag v0.6.412). Live on both dogfoods (homelab-config d201778): ready, 0 restarts, only the standing WARN lines; manual checks on #1005. What it brings: - **Uploads on any instance:** any member accepts any piece of a resumable upload and finishes any upload. Upload records with an epoch and piece runs are shared; pieces are sealed write-once objects under `.cairn-state/tus/`; a newer request takes an upload over (the older answers 409 with `Upload-Offset`); a crash mid-finish rolls back to the last stored piece and the client's resend finishes on any member, never storing a file twice; shared quota reservations; one leased sweep; 503, never 404, during outages. The web client pauses and resumes by itself for up to 10 minutes. The "v0.6.411 only" upload affinity rule is gone from the docs. - **The shared change feed:** the change list for sync clients is a sealed NATS stream shared by every member, with epochs that rotate at the switch, after a gap, a crash, a data-epoch change or a stream loss. - **Building blocks for 11b:** the `<prefix>_auth` bucket, sealed coord records, the coord-key classification table (also fixes `dup.*` in backups), switch-on with the cluster's minimum version. - **nats.go 1.56.0** (security release), tested in embedded and external modes. - Operator facts from the VM run (run-u14): `TMPDIR` on disk for S3 (start WARN on tmpfs), the JetStream account size formula (refused at once with the formula when too small), S3 retry WARN. CI note: the first PR run failed on a race in `TestUploadFinishCrashRealMain742c` (a HEAD fences a running PATCH by design, and the test polled HEAD while the PATCH streamed); the test now waits on a deterministic seam. Lesson added to the 11b brief. **Next: batch 11b (v0.6.413)**: spec and plan written (`2026-11-21-b12v07/`), spec points settled as consequences of the decisions (decisions-b12.md), whole-page mockup drawn and waiting for approval: shared sign-in records, #1179 strict mode, link gate cookie and per-link cap, the shared editor session table with editor locks, P5 rotation progress; then the multi-instance announcement.
Owner

v0.6.413 shipped: several instances are supported (batch B part 2b, PR #1195, merge b2b65727, tag v0.6.413). Live on both dogfoods (homelab-config 4100739): ready, 0 restarts, only the standing WARN lines; manual checks on #1005.

What it brings (all switched on when every member runs v0.6.413):

  • Shared sign-in in <prefix>_auth (sealed, never backed up or migrated): sessions end on every member at once (sign-out, disable, password or two-step reset, back-channel logout by sub); outage sessions are kept on the member and promoted on recovery; two-step tickets, device codes, OIDC state and logout jti are decided once across members; the sign-in throttle is shared.
  • #1179 strict mode: an administrator setting that pauses all new sign-ins during a coordination outage (form, OIDC start, Basic with an account password); app passwords, API keys and emergency accounts keep working; audited.
  • Link passwords: a sealed cookie (every mode, survives restarts), a per-link cap of 100 failures per hour with a cluster-wide soft budget of one attempt per minute (NIST SP 800-63B 5.2.2), 1,024-character limit.
  • Document editing: a shared editor session table and editor locks: the file is locked by the editor while open, everyone co-edits in one session, outside writes (incl. your own sync and WebDAV clients) answer 423 until everyone closes, "In the editor: X" with Join, End editing for everyone, guests read-only while members edit, lock tokens hidden in lockdiscovery, a crash take-over by the same session.
  • Key rotation progress on the Overview and in cairnd cluster status (retiring keys, backups, next step, re-seal pass total and phase).
  • Operations: the "preview" wording is gone; external NATS refusals name a server without room for a stream (JetStream 10005) instead of retrying silently; every start step that lasts over 15 s logs which one.

Process: 16 tasks in 5 waves, wave reviews W1-W3 with 6 fix rounds, final review (High: a write guest could take over the members' editor lock, fixed), VM run on three members with v0.6.413-rc1 (all scripted steps passed), three CI fixes (a race in an outage test that hid a real bug in how the auth watch judged "behind" during a majority loss; a probe-bucket cleanup; external streams that could not be placed).

What stays, with issues: #1189 inbound peering affinity (peering state in NATS), #1190 If-Match and conflict mode fail across instances, #1191 shared move marks, #1192 guests in the members' editor session, #1193 "Your sessions" and End all, #1194 optional inactivity timeout; #1176, #1177, #1187, #957 as before. Known limitations in multi-instance.md list the residuals.

Closing #742: the multi-instance announcement has shipped; follow-ups are tracked in the issues above.

**v0.6.413 shipped: several instances are supported** (batch B part 2b, PR #1195, merge b2b65727, tag v0.6.413). Live on both dogfoods (homelab-config 4100739): ready, 0 restarts, only the standing WARN lines; manual checks on #1005. What it brings (all switched on when every member runs v0.6.413): - **Shared sign-in** in `<prefix>_auth` (sealed, never backed up or migrated): sessions end on every member at once (sign-out, disable, password or two-step reset, back-channel logout by sub); outage sessions are kept on the member and promoted on recovery; two-step tickets, device codes, OIDC state and logout jti are decided once across members; the sign-in throttle is shared. - **#1179 strict mode**: an administrator setting that pauses all new sign-ins during a coordination outage (form, OIDC start, Basic with an account password); app passwords, API keys and emergency accounts keep working; audited. - **Link passwords**: a sealed cookie (every mode, survives restarts), a per-link cap of 100 failures per hour with a cluster-wide soft budget of one attempt per minute (NIST SP 800-63B 5.2.2), 1,024-character limit. - **Document editing**: a shared editor session table and **editor locks**: the file is locked by the editor while open, everyone co-edits in one session, outside writes (incl. your own sync and WebDAV clients) answer 423 until everyone closes, "In the editor: X" with Join, End editing for everyone, guests read-only while members edit, lock tokens hidden in lockdiscovery, a crash take-over by the same session. - **Key rotation progress** on the Overview and in `cairnd cluster status` (retiring keys, backups, next step, re-seal pass total and phase). - **Operations**: the "preview" wording is gone; external NATS refusals name a server without room for a stream (JetStream 10005) instead of retrying silently; every start step that lasts over 15 s logs which one. Process: 16 tasks in 5 waves, wave reviews W1-W3 with 6 fix rounds, final review (High: a write guest could take over the members' editor lock, fixed), VM run on three members with v0.6.413-rc1 (all scripted steps passed), three CI fixes (a race in an outage test that hid a real bug in how the auth watch judged "behind" during a majority loss; a probe-bucket cleanup; external streams that could not be placed). **What stays, with issues:** #1189 inbound peering affinity (peering state in NATS), #1190 If-Match and conflict mode fail across instances, #1191 shared move marks, #1192 guests in the members' editor session, #1193 "Your sessions" and End all, #1194 optional inactivity timeout; #1176, #1177, #1187, #957 as before. Known limitations in multi-instance.md list the residuals. Closing #742: the multi-instance announcement has shipped; follow-ups are tracked in the issues above.
Cordy closed this issue 2026-11-22 01:15:28 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#742
No description provided.