Horizontal scalability #742
Closed
opened 2026-09-25 09:52:25 +00:00 by hyrsh
·
45 comments
No Branch/Tag specified
main
b12v07
b11v07
b10v07
ci-rc-publish
fx15-s3dry
b9v07
b8v07
bench742
b7v07
legal-texts-v0.2
b6v07
b5v07
b4v07
b3v07
sx1139
st742
b2v07
ix1102
fu1098
zp1097
wd1099
tr1100
qf1077
nc1070
ci1
rp1072
lk1073
ab1071
fi1074
si1070
ra1054
wo1056
oc1057
tf1055
sv1063
fg1059
st1058
fs929
mu1039
ls1033
cf975
lf835
dx1023
ml896
a11y2
av1022
r913
r7
r6
r5
r45
r23
r1
b10
b9
fix-b5
fix-b4
fix-b3
fix-b2
fix-b1
fix-922
fix-923
fix-911
fix-343b
batch-a
v0.6.413
v0.6.412
v0.6.411
v0.6.410
v0.6.409
v0.6.408
v0.6.407
v0.6.406
v0.6.405
v0.6.404
v0.6.403
v0.6.402
v0.6.401
v0.6.400
v0.6.399
v0.6.398
v0.6.397
v0.6.396
v0.6.395
v0.6.394
v0.6.393
v0.6.392
v0.6.391
v0.6.390
v0.6.389
v0.6.388
v0.6.387
v0.6.386
v0.6.385
v0.6.384
v0.6.383
v0.6.382
v0.6.381
v0.6.380
v0.6.379
v0.6.378
v0.6.377
v0.6.376
v0.6.375
v0.6.374
v0.6.373
v0.6.372
v0.6.371
v0.6.370
v0.6.369
v0.6.368
v0.6.367
v0.6.366
v0.6.365
v0.6.364
v0.6.363
v0.6.362
v0.6.361
v0.6.360
v0.6.359
v0.6.358
v0.6.357
v0.6.356
v0.6.355
v0.6.354
v0.6.353
v0.6.352
v0.6.351
v0.6.350
v0.6.349
v0.6.348
v0.6.347
v0.6.346
v0.6.345
v0.6.344
v0.6.343
v0.6.342
v0.6.341
v0.6.340
v0.6.339
v0.6.338
v0.6.337
v0.6.336
v0.6.335
v0.6.334
v0.6.333
v0.6.332
v0.6.331
v0.6.330
v0.6.329
v0.6.328
v0.6.327
v0.6.326
v0.6.325
v0.6.324
v0.6.323
v0.6.322
v0.6.321
v0.6.320
v0.6.319
v0.6.318
v0.6.317
v0.6.316
v0.6.315
v0.6.314
v0.6.313
v0.6.312
v0.6.311
v0.6.310
v0.6.309
v0.6.308
v0.6.307
v0.6.306
v0.6.305
v0.6.304
v0.6.303
v0.6.302
v0.6.301
v0.6.300
v0.6.299
v0.6.298
v0.6.297
v0.6.296
v0.6.295
v0.6.294
v0.6.293
v0.6.292
v0.6.291
v0.6.290
v0.6.289
v0.6.288
v0.6.287
v0.6.286
v0.6.285
v0.6.284
v0.6.283
v0.6.282
v0.6.281
v0.6.280
v0.6.279
v0.6.278
v0.6.277
v0.6.276
v0.6.275
v0.6.274
v0.6.273
v0.6.272
v0.6.271
v0.6.270
v0.6.269
v0.6.268
v0.6.267
v0.6.266
v0.6.265
v0.6.264
v0.6.263
v0.6.262
v0.6.261
v0.6.260
v0.6.259
v0.6.258
v0.6.257
v0.6.256
v0.6.255
v0.6.254
v0.6.253
v0.6.252
v0.6.251
v0.6.250
v0.6.249
v0.6.248
v0.6.247
v0.6.246
v0.6.245
v0.6.244
v0.6.243
v0.6.242
v0.6.241
v0.6.240
v0.6.239
v0.6.238
v0.6.237
v0.6.236
v0.6.235
v0.6.234
v0.6.233
v0.6.232
v0.6.231
v0.6.230
v0.6.229
v0.6.228
v0.6.227
v0.6.226
v0.6.225
v0.6.224
v0.6.223
v0.6.222
v0.6.221
v0.6.220
v0.6.219
v0.6.218
v0.6.217
v0.6.216
v0.6.215
v0.6.214
v0.6.213
v0.6.212
v0.6.211
v0.6.210
v0.6.209
v0.6.208
v0.6.207
v0.6.206
v0.6.205
v0.6.204
v0.6.203
v0.6.202
v0.6.201
v0.6.200
v0.6.199
v0.6.198
v0.6.197
v0.6.196
v0.6.195
v0.6.194
v0.6.193
v0.6.192
v0.6.191
v0.6.190
v0.6.189
v0.6.188
v0.6.187
v0.6.186
v0.6.185
v0.6.184
v0.6.183
v0.6.182
v0.6.181
v0.6.180
v0.6.179
v0.6.178
v0.6.177
v0.6.176
v0.6.175
v0.6.174
v0.6.173
v0.6.172
v0.6.171
v0.6.170
v0.6.169
v0.6.168
v0.6.167
v0.6.166
v0.6.165
v0.6.164
v0.6.163
v0.6.162
v0.6.161
v0.6.160
v0.6.159
v0.6.158
v0.6.157
v0.6.156
v0.6.155
v0.6.154
v0.6.153
v0.6.152
v0.6.151
v0.6.150
v0.6.149
v0.6.148
v0.6.147
v0.6.146
v0.6.145
v0.6.144
v0.6.143
v0.6.142
v0.6.141
v0.6.140
v0.6.139
v0.6.138
v0.6.137
v0.6.136
v0.6.135
v0.6.134
v0.6.133
v0.6.132
v0.6.131
v0.6.130
v0.6.129
v0.6.128
v0.6.127
v0.6.126
v0.6.125
v0.6.124
v0.6.123
v0.6.122
v0.6.121
v0.6.120
v0.6.119
v0.6.118
v0.6.117
v0.6.116
v0.6.115
v0.6.114
v0.6.113
v0.6.112
v0.6.111
v0.6.110
v0.6.109
v0.6.108
v0.6.107
v0.6.106
v0.6.105
v0.6.104
v0.6.103
v0.6.102
v0.6.101
v0.6.100
v0.6.99
v0.6.98
v0.6.97
v0.6.96
v0.6.95
v0.6.94
v0.6.93
v0.6.92
v0.6.91
v0.6.90
v0.6.89
v0.6.88
v0.6.87
v0.6.86
v0.6.85
v0.6.84
v0.6.83
v0.6.82
v0.6.81
v0.6.80
v0.6.79
v0.6.78
v0.6.77
v0.6.76
v0.6.75
v0.6.74
v0.6.73
v0.6.72
v0.6.71
v0.6.70
v0.6.69
v0.6.68
v0.6.67
v0.6.66
v0.6.65
v0.6.64
v0.6.63
v0.6.62
v0.6.61
v0.6.60
v0.6.59
v0.6.58
v0.6.57
v0.6.56
v0.6.55
v0.6.54
v0.6.53
v0.6.52
v0.6.51
v0.6.50
v0.6.49
v0.6.48
v0.6.47
v0.6.46
v0.6.45
v0.6.44
v0.6.43
v0.6.42
v0.6.41
v0.6.40
v0.6.39
v0.6.38
v0.6.37
v0.6.36
v0.6.35
v0.6.34
v0.6.33
v0.6.32
v0.6.31
v0.6.30
v0.6.29
v0.6.28
v0.6.25
v0.6.24
v0.6.23
v0.6.22
v0.6.21
v0.6.20
v0.6.19
v0.6.18
v0.6.17
v0.6.16
v0.6.15
v0.6.14
v0.6.12
v0.6.11
v0.6.10
v0.6.9
v0.6.8
v0.6.7
v0.6.6
v0.6.5
v0.6.4
v0.6.3
v0.6.2
v0.6.1
v0.6.0
v0.5.7
v0.5.6
v0.5.5
v0.5.4
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.12
v0.4.11
v0.4.10
v0.4.9
v0.4.8
v0.4.7
v0.4.6
v0.4.5
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.30
v0.3.29
v0.3.28
v0.3.27
v0.3.26
v0.3.25
v0.3.24
v0.3.23
v0.3.22
v0.3.21
v0.3.20
v0.3.19
v0.3.18
v0.3.17
v0.3.16
v0.3.15
v0.3.14
v0.3.13
v0.3.12
v0.3.11
v0.3.10
v0.3.9
v0.3.8
v0.3.7
v0.3.6
v0.3.5
v0.3.4
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.0.3
v0.0.2
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference: Cordy/Cairn#742
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Since AlpineFiles is a single instance installation we need to address the possibility to scale it horizontally.
Things to consider if scaling occurs:
There is one thing that is necessary if using "local" storage. This kind of storage applies not only to a local installation but also to an installation in Kubernetes when using PVCs since they work like local directory mounts from an applications point of view. So if no external storage is directly addressed like S3 or NFS we have to introduce a human understandable directory where the local user data always gets written if nothing gets configured and default values are used. The same applies for Docker volumes.
Proposal for the default directory (user data) --> /alpinefiles/data
Proposal for the default config file --> /alpinefiles/config/local.json
The most critical part of scaling comes down to the active state the application has. In our case we need to look for possible pitfalls regarding storage and local user handling.
Scenario: Raw Binary (on e.g. 3x machines)
The binary is executed on multiple linux hosts and the initial admin gets written to the local file on machine #1.
The documentation has to push the user to distribute the created file to machines #2 and #3 (e.g. with some sync job) or use a common mount (e.g. NFS) to keep local users in sync. The same applies to the data directory.
There also has to be a load balancer in front of all machines since we cannot provide reasonable balancing mechanisms on a network level.
Scenario: Docker
As in the raw binary example we need to document that the docker volumes have to be synced across all hosts. The other option would be to directly use external volumes with docker (e.g. NFS, S3).
Scenario: Kubernetes
The mounted PVC must be in RWX access mode or a pre-filled local user config must be created as Secret to be mounted to the respective path. More elegant would be if the initial wizard would offer a checkbox "Create Kubernetes Secret" that would create a Secret with the entered credentials for the first admin (if the checkbox is checked the first thing is to try to communicate with the Kubernetes API and log + notify if there are missing RBAC permissions to create a secret or if the API is at all reachable). If this is done, we can use the built-in scaling of Kubernetes.
If I can think of more. I will add another issue.
Notes from the #922 research (2026-10-16) for the horizontal-scaling design. Source:
scratch/b7e/batch907/922-research-2.md.Version data safety: for file versions, #922 is designed so data safety needs no cross-instance lock (record-then-publish). A shared lock would only be needed for ordering, behind a
PathSerializerinterface.What #742 must still solve elsewhere:
Approaches that do not work on Garage or NFS:
A coordination service (Redis, etcd, NATS KV or Postgres) is likely needed once more than one instance runs.
From #922 (v0.6.347), what this means for scale-out:
versions.PathSerializer. Today that is an in-process implementation. A cross-instance one plugs in there without touching the engine; itsHeld.Lost()reports a lost lease, and the engine then skips pruning. Decision D5: never lease objects on the backend.If-Matchcheck can be overtaken by a save in between (#951).Multi-instance notes from batch 3 (v0.6.350, 2026-10-19). Each is safe today on one instance and needs the cross-instance work tracked here.
Ensure,Rename,RetireAlias,Forget) still persist the whole in-memory cache, so a stale instance can overwrite another's fields. The convergence writers re-read the record on disk and change only their own fields since v0.6.350. The other writers need the same.Lost()is ignored. Correctness no longer depends on the lock: the restore and purge each write their own flag object and back off if they see the other's. Throughput and lease-loss handling do.MovingToand before it copies anything. If an S3 delete and a convergence copy then overlap, a duplicate partial copy can be left at the new name. No file is lost.MovingToclear: in one narrow race, a stale instance can clear another instance'sMovingTo. If that instance then crashes mid-copy, the admin row readsfolder-exists; no file is lost.Another gap from the batch 4 research (
research-b4.md§7.4, from reading the code).The change journal is per instance.
changes.Openloads one statestore key at start. Each instance then keeps its own ring and flushes it to that same key (internal/changes/changes.go:95-193).Two instances that start from the same clean state resume the same epoch and sequence, and then diverge. A cursor issued before the divergence is accepted by both, because its hash is in both histories. A client that keeps polling one instance then silently misses changes made through the other. That breaks the change feed's promise to never bridge a gap silently.
This is not yet in ARCHITECTURE's list of scale-out gaps.
More from batch 4 (2026-10-20), after a CI failure in
TestConvergeTwoInstancesRandom931. The race was real and also exists on v0.6.350; it is fixed in v0.6.351.Fixed: a second instance decided "folder-missing" from its out-of-date copy of the space registry after the other instance had already moved the folder. The refusal is now re-checked against the registry on disk. A guarded
DropMovewrite also stops the "copies-left" reason from landing on a space another instance has already finished.Still open, needs the cross-instance work tracked here:
rename-finishedwithout re-reading the registry.Notes from the #985 research (research-985-1 §8, -2 §6, -3 §7, -4 §6), all to be held by the multi-instance design:
oidclib.go:119, 204, 274). A callback that reaches another instance fails as "unknown or expired state", today as the bare "login failed" page. Needs sticky sessions or a shared state store. VERIFIED (code).throttle.go:13-16): N instances give N times the attempt budget. AD lockout stays the only global cap; this strengthens #992 (refused-credential cache) and #991 (timing).trust.go:10-11). Proposed: one statestore document with one entry per instance id, each instance rewriting only its own entry every 5 min (thespacealertsmerge), "not reporting since" after 15 min, pruned after 24 h; plus low-cardinality per-instance metrics. Depends on item 1 of comment 9729 (unique temp names) on the local store.device-pauses.json(mass-miss) and the audit JSONL need this issue's writer rules; each instance may raise its own mass-miss bell item.auth.passwordChangeUrl, help texts, TLS trust.Decision (Nikola, 2026-10-27): multi-instance is a v0.7 goal.
Next: research the coordination store following the playbook: Redis, etcd, NATS KV, Postgres, and the option of no extra store (backend-only, per comment 9551). Then a decision for Nikola, and a plan. The plan covers the issues that depend on this one: #1011 (whole-file rewrites of state), #945, #944, #942, #950, #958 (versions across instances), #1020 (lease handover at upgrade) and #1019 (split deployments).
From now on, new multi-instance findings are added here as comments rather than as new issues.
Research: coordination for multi-instance, 2026-10-27. Full file:
scratch/b7e/backlog/research-742-coordination.md, with claims marked VERIFIED, INFERRED or UNVERIFIED.What breaks today with 2 replicas:
statestore/lock.go:306).office/token.go:37), so a Collabora callback that reaches another instance fails.The backend can't coordinate: Garage accepts
If-None-Matchand silently ignores it (VERIFIED: its docs, and a Lance report with Garage v2.4.1). NFS locks are unreliable (ownCloud's cluster docs). Correctness has to come from compare-and-swap with fencing tokens on the store that holds the state (Kleppmann; Jepsen on etcd).Recommendation, tiered:
Order of work:
Correction: comment 9551 ("never lease objects on the backend") was posted by Cordy, summarising the #922 research. It isn't Manuel's.
Open questions for Nikola (being asked now) and for Manuel:
Decisions (Nikola, 2026-10-27):
.cairn-state/in the bucket, so the bucket plus the recovery key still restores everything.@hyrsh, questions for you before the store is chosen:
Research 2: resumable uploads, NFS, and locks across instances (2026-10-27). Full file:
scratch/b7e/backlog/research-742-tus-nfs.md.Uploads (tus). Each upload becomes a list of write-once, uniquely named pieces, sealed with the
.cairn-stateage keys. One progress record holds the confirmed offset and the piece list, and it changes only when the expected version still matches. No shared volume, no sticky sessions, no locks.Store.Writechain, so quota, locks, holds, versions, encryption and the key check all still apply.POSIX/NFS with several instances.
If-Matchcheck, which reads cached attributes;hard, nolocal_lock/nolock,lookupcache=positive, NTP. NFS locks are never used for correctness.Locks (WebDAV and user locks). One lock record per file in the coordination store, taken by compare-and-swap. Expiry follows the store's clock, and the write path reads the store directly.
Single-instance findings from this research (in the backlog, planned for an uploads batch):
Questions for Manuel (@hyrsh):
Decisions (Nikola, 2026-10-27), research 2:
soft,local_lockornolock. One instance only warns.Multi-instance finding (CI, 2026-10-27, v0.6.361 branch; pre-existing, also on v0.6.360): two instances converging one space rename can lose the "folder is now X" registry update.
SetBackendDir.ForgetMove, which read the whole registry file before B's save and writes its stale copy after it.folder-missing.convergeforget931_test.go).Multi-instance finding (BAP review, v0.6.363 branch; pre-existing class #1011):
LocalStoreloadslocal-users.jsononce per process and rewrites it whole.NoteSignInevery 15 minutes, an admin change, or another account's new-password step.This is not fail-safe. The fix belongs with #1011: record-level, version-checked writes, then the coordination store.
Multi-instance finding (CI, v0.6.363; present since v0.6.356 via #1007): when two instances restored one deleted space at once, one restore could answer 200 while the other moved the folder back into the trash, leaving an empty live space.
Fixed in v0.6.363 for the normal race:
Revivetreats a space already live on its folder as success.mayRollBackreads the intents before the entry.Left open: if the losing restore's revive fails with a real error (a state write failure) while the other's resumed restore finishes after its checks, it can still roll the folder back. This needs a shared lock across instances (coordination store). The restore hook can also run on both instances when both answer 200.
Multi-instance notes from #904 (v0.6.364):
X-Cairn-Data-Epoch(read from the restores store) and the change feed'sdataEpochcan disagree for up to about 60 s.storage-restores.jsonunder its lock on the device and Basic hot path every 30 s; on S3 that is one bucket GET per 30 s per instance. This is the same pattern as the #895 pause store.NoteApp's per-process "already noted" set is only a write throttle.Multi-instance findings from batch M1 (v0.6.368, #1026 Remove partial copy):
hold.Storeis a per-instance cache never re-read (hold.go); Set/Lift write the whole map from it. Only the partial-copy removal reads a freshSnapshot().runUnreadableAfter(5 min) is tied to the API's 3-minute lease only by a comment.It is very common for software architects to suggest PostgreSQL for this scenario. PostgreSQL has a built-in feature called LISTEN / NOTIFY, which allows it to act as a lightweight pub/sub message broker alongside its primary job as a database.
While using Postgres can simplify your tech stack (since you likely already need a database for file metadata), it has strict architectural trade-offs compared to NATS.
Here is the direct comparison for a horizontally scaled file server setup:
Architectural Breakdown
Feature NATS PostgreSQL (LISTEN / NOTIFY)
Primary Design Built from the ground up for ultra-fast, ephemeral messaging and streaming. Built for data persistence, ACID transactions, and complex queries.
Message Delivery Speed Sub-millisecond latency. Memory-first architecture. Higher latency. Messages go through the database engine and transaction logs.
Throughput Capacity Millions of messages per second without breaking a sweat. Bound by disk I/O and connection limits; usually tops out at a few thousand notifications/sec.
Payload Size Limit 1 MB by default (configurable up to 64 MB if needed). Strict 8,000-byte limit per notification payload.
Queue Groups / Load Balancing Native support. Easily splits background jobs among instances. Not natively supported for NOTIFY. Every listening server gets every message.
Infrastructure Overhead Adds a new piece of infrastructure to maintain (NATS clusters). Zero new infrastructure if you are already using Postgres for metadata.
Why the Architect Suggested Postgres (The Pros)
Where Postgres Fails for File Servers (The Cons)
The NOTIFY payload in Postgres is strictly capped at 8,000 bytes. While this is enough to send a simple "file_id: 1234 updated" string, it becomes a major bottleneck if you want to pass larger event metadata or small binary payloads directly through the event stream.
Postgres LISTEN / NOTIFY is strictly a broadcast mechanism. If you have 10 file server instances listening for events, all 10 instances will receive every single notification.
• For Cache Invalidation, this is perfect (everyone needs to clear their cache).
• For Background Tasks (like generating a thumbnail), this is a disaster. All 10 servers will try to generate the same thumbnail simultaneously unless you write a complex distributed locking mechanism on top of it. NATS handles this natively out of the box with Queue Groups.
Postgres requires a persistent, open database connection for every active LISTEN session. In highly scaled environments, tying up valuable database connection pools just to wait for real-time events can starve your actual API of data connections. NATS, by contrast, handles hundreds of thousands of concurrent connections effortlessly.
The Verdict: Which one should you pick?
• Choose PostgreSQL if your file server handles a modest workload, you only need it for simple cache invalidation (not background job routing), and your top priority is keeping your infrastructure as simple as possible.
• Choose NATS if your file server needs to handle heavy traffic, routes asynchronous background jobs (thumbnails, virus scans), requires real-time WebSocket broadcasting, or needs room to scale linearly without hitting database bottlenecks.
Are your file servers mostly doing simple cache clearing, or are you also trying to coordinate background processing (like video transcoding, compression, or sync jobs)?
Decision (Nikola, 2026-11-05): running several instances stays a requirement for launch (v0.7 Feature complete), after the 2026-11-04 reprioritisation.
On the store (reply to the NATS vs Postgres comparison above): that comparison is about LISTEN/NOTIFY messaging. What #742 mostly needs is durable state with compare-and-set: the state files (#1011), leases with fencing, sessions, locks and upload progress records. At 2-3 instances this is a few hundred operations a second, so throughput does not decide it.
sync_interval: always.Step 0 (no new dependency: #1011 record-level writes, stable WOPI key, a separate journal epoch per instance, the POSIX startup lock) does not depend on the choice and can start now.
Research 3: Step 0, the work that needs no store (2026-11-05). Full file:
scratch/b7e/backlog/research-742-step0.md(code at666c3d3b, claims marked VERIFIED, INFERRED or UNVERIFIED).Inventory: 64 bound state documents now, not ~45, plus 8 record folders.
Recommendations:
statestoregets a versioned API (LoadV,SaveIf,Update(fn)) with an opaque version. Files and.cairn-stateobjects check it by content hash under the process lock. Every writer applies only its own change to the document as read. Caches that decide access re-read every 10 s. The chosen store later implements the same API, so no store code changes again.item-id-key, with a key id in each token so keys can rotate.editor-session-unknown. The table has to move to the store (Step 2).SameSite=Lax, 10 min,nextcapped at 1 KiB. Sign-ins then survive restarts, and the uncapped in-memory state map (a memory DoS) goes away.Loginterface for the shared journal.flockon the state directory plus the existing heartbeat lock, and the decided NFS mount rules from/proc/mounts.cairnd restoredthen refuses on POSIX too.coordinterface (M): records with CAS and TTL, leases with fencing tokens and the #1020 take-over rule, best-effort notifications, a log and an instance registry. It comes with an in-process implementation and a conformance suite that the NATS and Postgres drivers must pass later.Waves:
coord, POSIX lock, keyring.Eight questions for Nikola, each with a recommendation, are in section 11 of the file.
Decisions (Nikola, 2026-11-05):
Step 0 (research-742-step0.md): all eight recommendations approved.
Loginterface for the shared journal (Step 3).Built in 4 waves alongside batch 2.
Benchmark (research-742-benchmark.md): approved as designed. Published numbers from x86 VMs on pve2 (every store node 4 vCPU / 4 GB), plus a limited Pi run. Compared: embedded NATS 2.15 (3 nodes,
sync_interval: alwaysand default), Postgres single, Postgres with a synchronous standby, Postgres with 2 Cairn instances. Correctness first (lost acknowledged writes, stale compare-and-swap wins, reused fencing tokens under 8 fault types), then cost, then latency; the method and decision criteria go on the wiki before any run. Nikola creates a power-only Proxmox token for the benchmark VMs when the harness is ready. Results are published on the repo wiki.Decisions (Nikola, 2026-11-06), Step 0 spec points (research-742-step0-unsettled.md), all approved:
InstanceandRunstamped now.statestore.Writekept one release; deterministic two-process tests in CI, random soak manual.Benchmark criteria (approved): correctness, then availability (writes resume on the surviving instances after one node is lost, no operator action; Postgres with automatic failover not tested), then cost (operator burden first; differences within the run-to-run spread don't decide), then latency (p99 at 2,000 users above 50 ms per operation, only with clearly separated confidence intervals). Tie-break: Postgres. Postgres with 2 Cairn instances gets the client-side faults only. The method page goes on the wiki now; no external review posts.
Step 0 shipped in v0.6.401 (PR #1138), live on both dogfoods (pods on v0.6.401, ready, no restarts, no new WARN or ERROR lines). Running more than one Cairn process on one storage root is still not supported; v0.6.401 refuses a second one.
What it does: start lock (flock + heartbeat on the state directory and the POSIX data root; a sealed lock object on S3; a failed read never takes over a live lock; a lock sealed to other keys is taken over only after 130 s without change; a process whose lock was taken over exits), instance key file for WOPI and OIDC state, versioned state saves for every read-merge-write store (guard test, two-process soak), settings 409/428, change-journal writer guard, local delete that cannot leave a usable device behind. Reviews: three wave reviews, a final review and two focused re-reviews of the lock fixes. Residuals: ARCHITECTURE.md, "What Step 0 does not yet do".
S3 cost at idle (A7), Garage request counters over 5 minutes each, whole Garage (other buckets are near idle), no users active:
So about 41 more requests a minute at idle (the version reads before each save, the 10 s re-reads of spaces and quotas, the lock heartbeat), about 1.8 million a month; at typical cloud GET prices that is well under a franc a month. Bytes did not grow. The v0.6.401 sample began about 2.5 minutes after start, so it may include some start-up reads.
Benchmark: harness phase 0 is built (branch bench742, own module
bench/coordbench): null, embedded NATS (single node and 3-node R3) and PostgreSQL drivers, a shared conformance suite that the deliberately broken drivers fail, open-loop load generator, history checkers. First findings: an ADR-50 atomic batch over two KV subjects of one stream works on nats-server 2.15.1 (so fenced writes in NATS are one atomic write, as designed); NATS TTLs are whole seconds; only the stream leader answers a direct get when asked, so fresh reads are possible. Next: VMs on pve2 per the plan (Terraform via homelab-config), then the measured runs, results on the wiki page.Benchmark: test bed up, measured runs started (2026-11-06 23:30 UTC).
-spin 2ms, so the runs use spin.v0.6.403 residuals (#1105, #1140, #1106, #1107), also in ARCHITECTURE.md:
link-keyjoinsinstance-keys.json, which Step 1 still has to bind.oldInstances(spacekeys.go) with v0.6.403 as the minimum.personal-quota-seen.jsonis rewritten whole on each flush.ownsAppSpaceand member-end items readList()without the 10 s re-read (F4); power modecanBreakLockscosts one uncached directory lookup per listing.v0.6.404 multi-instance residuals (full texts in ARCHITECTURE.md).
statestore.UpdatePath; hourly sweep under coord leasebranding-sweep; per-process caches bounded. Residuals: posix window where a Claim and a sweep removal can both land (WARN, re-upload fixes it); a re-upload racing an eviction's delete; a refused save leaves claimed images until the sweep; mixed versions: v0.6.403 dropsbrandingat its first settings save.instance-keys.jsonvoids mailed Stop links; end reminders on their own leaselink-ending(up to ~20 min late after a hard kill); limits as two coord updates (a send near a limit can be refused for a moment, never exceeded);shareMailOptionalreaches other instances within 10 s and the pin is per instance (set it the same everywhere).v0.6.405 multi-instance residuals (full texts in ARCHITECTURE.md).
versions-sweep.rewind.<rootkey>(60 s) andmoverec-compact.The fault campaign
fault742is finished: 230 cells, chain4b ended 2026-11-11 03:38 UTC.Raw result: 175 cells pass and 51 are marked elle INVALID. Only 44 of those 51 fail on elle alone. 005-P1-F5-r4 has no history (a harness error).
Investigation (full report:
scratch/b7e/backlog/elle-invalid-742.md, and on cb-ctl~/coordbench/investigate-elle/). The 44 elle-only failures come from the harness, not the stores:All 44 cells pass
-c serializable. With a corrected history, they also pass strict-serializable within a stated clock tolerance.What this means
Next steps (harness only; no re-run needed)
info, and add indeterminate appends for values that were seen but whose write was lost.-c serializable, and report strict-serializable as a second check with an explicit clock tolerance.After that, the P2 fault campaign, the wiki page and the teardown, once you agree.
Harness fixed,
fault742re-checked, P2 fault campaign started (2026-11-11)Harness (coordbench 0.3.2). bench742 commit
0a24708b, homelab-config fe70285:-c serializable. Strict-serializable is still reported, but separately and not as a gate.info.Re-check of all 230
fault742cells (copies only; the results are untouched). Report:scratch/b7e/backlog/recheck-fault742.md.P2 fault campaign.
pg_mode p2.chain5(unitcoordbench-chain5) runs check, a fresh P2 preload, thenfaults --seed 7420 --configs P2 --campaign fault742p2, about 8 h.Then the wiki page, and the teardown once Nikola agrees.
Benchmark: fault-campaign results are on the wiki (2026-11-12): Coordination store benchmark. The results sit above the published method, which is unchanged. The changelog lists every deviation.
Correctness (step 1), 340 trials, 10 per configuration and fault, after rechecks and reruns:
sync_interval: always)The page lists each one with the reason its verdict changed.
Still open before a store is chosen:
Raw data: 6 archives with their sha256 sums, listed on the page. Nothing has been torn down.
Wiki updated with the results for all four decision steps (harness 0.3.4 reruns, chains 8 and 9): Coordination store benchmark
Decision (Nikola, 2026-11-13): the coordination store is NATS, configuration N-sync (NATS JetStream embedded in the Cairn pods, 3-way replication, fsync on every write). Basis: the benchmark results on the wiki page Coordination store benchmark (harness 0.3.4, chains 1 to 9).
Why, by the agreed four steps:
Scope added to #742 for v0.7 (Nikola):
Caveats the wiring batch must cover: raise the default pod memory limit (about 384 to 512 MiB, measured with Cairn itself); per-call timeouts and retries for writes and leases (20 to 30 s of timed-out calls and up to 13 s lease gaps after a leader change); a third NATS-only node for 2-instance setups; node recovery under a memory cap (untested); open NATS correctness reports (#7549, #7556, #7567) tracked; stale replica reads (unmeasured; read from the leader where it decides). Also Wave B of #1011 (state on the store, early invalidation, minimum-version gate).
The test VMs are kept for integration tests of the wiring batch.
Step 1 research (2026-11-14): research-742-step1-cairn.md and research-742-step1-market.md.
Key findings:
statestore.Versioned, all leases/claims throughcoord(memcoord, statecoord) with a conformance suite. Step 1 = a NATS coord driver + a NATS-backed state store + new start path + chart + docs.sync_interval: alwaysmitigates the crash cases; membership changes are the least-tested operation.Open points go to Nikola next.
Step 1 decisions (Nikola, 2026-11-14):
cairnd cluster/backup/restore/migrate, full deployment docs. Batch B: sessions, WOPI sessions, two-step, device codes, OIDC state, logout replay, share password gates and the throttle as expiring NATS records; shared change journal.cairnd migratewith a short maintenance window, runbook, tested both ways.sync_interval: alwaysforced; mTLS between nodes, client port private; 8 MiB values; sealed backup every 15 min kept 7 days pluscairnd backup/restore; pinned NATS version with upgrade-from rules and n/n-1 skew; external NATS checked at start (R3, version, peers); 1 GiB pod memory until measured; readiness green and degraded during a NATS outage; NFS refused for the store; a "Known limitations" docs section linking the open Jepsen issues.Next: mockups and spec for batch A.
Batch A mockup and spec decisions (Nikola, 2026-11-14): mockup
742a-instances-mockup.htmlapproved (Instances section on Overview, member-down and majority-lost cards, rolling-update skew, bell items, 503coord-unavailable, CLI outputs) with the research alignment; all 37 researched points accepted (NATS 2.15.3 if the modules are downloaded first, R5 for 5 members, a fail-fast gate so requests don't stall during majority loss, heartbeat-age liveness, at-rest key rotation by rolling restart and re-seal, per-instance sealed audit chains on POSIX, a hardened local admin socket forcairnd cluster); the owner and item-id manifests get version-checked writes in batch A. Next: spec/plan fold, then the build.Batch A (v0.6.409) progress, 2026-11-14:
/metricsnever calls NATS.coord-unavailable/cluster-maintenance, sign-in/guest texts, the four-language maintenance page, Helm chart 0.8.0 (StatefulSet, headless Service, PDB, anti-affinity, grace 120 s).Decisions (researched in
research-742a-w3-unsettled.md):CAIRN_COORD_*variable; the chart sets them all (each member advertises its pod DNS name, per the NATS guidance for TLS routes). No config.json snippet needed on Kubernetes.helm.sh/resource-policy: keep.Next: W3 review, docs (multi-instance pages), then the VM and Kubernetes runs.
Batch A shipped: v0.6.409 (PR #1181, tag on
0d954d45). Several instances is in preview.Coordination: embedded NATS JetStream (3 or 5 members, R3/R5, fsync on every write, per-member at-rest keys) or an external NATS cluster. Every per-deploy key has a
CAIRN_COORD_*variable.Outage behaviour: 503
coord-unavailable/cluster-maintenance, a fail-fast gate, reads keep working, and/readyzstays green.Cluster lifecycle:
cairnd migrate --to nats/--to local;cluster rescue/certs [--ca]/status.Key rotation: the cluster decides which key it seals with (option D), with a start guard. The re-seal covers values and stored objects,
cairnd cluster resealruns it on demand, and the ledger follows option (b+).Kubernetes: Overview Instances section, bell kinds, 21 cluster audit verbs, metrics and endpoint. Chart 0.8.0 (StatefulSet in both modes, PDB, per-pod volumes, data claim kept on uninstall).
Docs: full handbook: modes, Kubernetes, plain hosts, runbooks R1-R12, Known limitations.
Verification:
Residuals for batch B and later:
home-held. Strict switch: #1179.migrate --to localuntil the StatefulSet is scaled to 0 (documented).Next: batch B (b10v07), the remaining per-instance state above, then the multi-instance announcement. This issue stays open for batch B.
v0.6.410 shipped (patch on batch A, PR #1182, tag on
2e3719f7). Live on both dogfoods.Fixed
cairnd migrate --to nats --dry-runwithout-configon an S3 instance read the built-in defaults and reported "state documents 0", exit 0. It now names where it read (Single-instance state: ...). When it finds no state document, it refuses with exit 2 and asks for-config.holds-confirm-emptyandrestoredhad the same blind spot and wrote to the local default directory instead of the bucket. They now name where they write and refuse without-configwhen the defaults hold no state.audit-verifywithout-configsays so.cairnd cluster ...run by another user could report "connection ended (broken pipe)" instead of the refusal. The member refuses before it reads the request, and the client stopped at its failed send. The client now reads the answer after a failed send.-race(backlog 2026-11-13).New finding: #1183, filed in this milestone for batch 10. A second process using a member's name can join: its own embedded NATS server (same server name) disturbs JetStream, the real member's heartbeat fails, and the duplicate check sees no live record. Already present in v0.6.409; timing-dependent.
Backlog: pdfium can run one render past
RecycleEvery, because the slot is freed beforefinish()retires the module.Dogfood migration dry runs (
migrate --to nats --dry-run -config ..., read-only), as data points for sizing:The openbao run shows the v0.6.410 line
Single-instance state: S3 bucket cairn-openbao, state prefix /.cairn-state, so the fx15 source naming works live.Observation for batch 10: the copy carries the older document generations next to their successors (
tags.json/tags-v2.json,favorites,recents,activity/activity-v2,first-signins.prev.json). To check whether anything still reads the v1 files; if nothing does, the migration could skip them or a cleanup could remove them.Batch B research and decisions (2026-11-18). Research: 7 files in
.superpowers/sdd/2026-11-18-b10v07/(sign-in state, editor sessions and link passwords, change feed, cluster lifecycle, keys, residual census, uploads).Main findings
Decisions (Nikola)
Lost();decision-sheet-b10.md,decisions-b10.md).Batch 10 (v0.6.411) status, 2026-11-19.
Mockup
742b1-safety-mockup.htmlrev 1 approved by Nikola as drawn: every frame, the 10 new strings in EN/DE/FR/IT, and all 7 open points as recommended.Build, branch
b10v07: all four waves are merged.instanceSeedcheck in both orders, a forced start never drops an entry with unknown holders, reports run once per period, real-main ports, the bell test wait.Found on the way, fixed in this batch (single instance too)
instanceSeedopened only through the recovery identity.Next: the final review, then the run on the kept VMs (duplicate member, export with 3 and 5 members, lease under a stopped holder, key back-out, cluster forget), then the release.
v0.6.411 shipped (batch B part 1, "safety first"): PR #1186, tag on
cf31bffa. Live on both dogfoods (homelab-config 9b16293).What shipped
cairnd cluster forget,key-drain-forcedandkey-drained, back-out WARN, remote crypto role refused with several instances,instanceSeedchecked between members.migrate --to nats.Fixed, single instance too
instanceSeedfiles sealed before a rotation opened only through the recovery identity.cluster removenow retries a member change still in flight (10202).Verification
run-s13.md): duplicate refused in 3 to 157 ms; export with 3 members about 6 s, with 5 about 5 s;forgetworks.Follow-ups
R2-Q8 live check (batch 11 input), 2026-11-19: Collabora does not take a WOPI lock on Cairn.
Test: enc dogfood, v0.6.411. A new test document (
wopi-lock-check.docxin Nikola's Personal folder) was opened in Collabora, two words were typed, and it was saved.The pod log shows exactly these WOPI calls:
GET /wopi/files/<id>(CheckFileInfo, 283 bytes);GET /wopi/files/<id>/contents(GetFile);POST /wopi/files/<id>/contents(PutFile, 200).No
POST /wopi/files/<id>came, which is where LOCK, REFRESH_LOCK and UNLOCK would arrive. The file never showed a lock.Cause, VERIFIED at
cf31bffa:internal/api/wopi.go:721).wopi.go:142-152) never setsSupportsLocks,SupportsGetLockorSupportsUpdate, so Collabora never uses it.Consequences:
This goes into the batch 11 decision round, researched first: advertise lock support (and what that changes for Collabora, OnlyOffice, the WOPI validator and multi-instance), or keep locks off and correct the docs.
Batch 11 research and decisions (2026-11-20). Research:
.superpowers/sdd/2026-11-20-b11v07/research-b11-r8-wopi-locks.md(editor locks) andresearch-b11-r9-refresh.md(the batch 11 designs re-checked against v0.6.411).Editor locks
Users expect a document to lock by itself while it is open in the browser editor:
Nextcloud (files_lock), oCIS, Seafile Pro and Box all work this way. Nobody locks by hand to edit; manual locks are a separate feature.
Decided: editor locks on. Cairn advertises lock support for writers. The file list shows who is in the editor, with Join. Outside writes are refused while the file is open. Required with the switch:
WebDAV stops showing lock tokens to readers.
Batch 11 is split in two
All other points in
decision-sheet-b11.mdare accepted as recommended.v0.6.412 shipped (batch B part 2a, PR #1188, merge
e147b92b, tag v0.6.412). Live on both dogfoods (homelab-config d201778): ready, 0 restarts, only the standing WARN lines; manual checks on #1005.What it brings:
.cairn-state/tus/; a newer request takes an upload over (the older answers 409 withUpload-Offset); a crash mid-finish rolls back to the last stored piece and the client's resend finishes on any member, never storing a file twice; shared quota reservations; one leased sweep; 503, never 404, during outages. The web client pauses and resumes by itself for up to 10 minutes. The "v0.6.411 only" upload affinity rule is gone from the docs.<prefix>_authbucket, sealed coord records, the coord-key classification table (also fixesdup.*in backups), switch-on with the cluster's minimum version.TMPDIRon disk for S3 (start WARN on tmpfs), the JetStream account size formula (refused at once with the formula when too small), S3 retry WARN.CI note: the first PR run failed on a race in
TestUploadFinishCrashRealMain742c(a HEAD fences a running PATCH by design, and the test polled HEAD while the PATCH streamed); the test now waits on a deterministic seam. Lesson added to the 11b brief.Next: batch 11b (v0.6.413): spec and plan written (
2026-11-21-b12v07/), spec points settled as consequences of the decisions (decisions-b12.md), whole-page mockup drawn and waiting for approval: shared sign-in records, #1179 strict mode, link gate cookie and per-link cap, the shared editor session table with editor locks, P5 rotation progress; then the multi-instance announcement.v0.6.413 shipped: several instances are supported (batch B part 2b, PR #1195, merge
b2b65727, tag v0.6.413). Live on both dogfoods (homelab-config 4100739): ready, 0 restarts, only the standing WARN lines; manual checks on #1005.What it brings (all switched on when every member runs v0.6.413):
<prefix>_auth(sealed, never backed up or migrated): sessions end on every member at once (sign-out, disable, password or two-step reset, back-channel logout by sub); outage sessions are kept on the member and promoted on recovery; two-step tickets, device codes, OIDC state and logout jti are decided once across members; the sign-in throttle is shared.cairnd cluster status(retiring keys, backups, next step, re-seal pass total and phase).Process: 16 tasks in 5 waves, wave reviews W1-W3 with 6 fix rounds, final review (High: a write guest could take over the members' editor lock, fixed), VM run on three members with v0.6.413-rc1 (all scripted steps passed), three CI fixes (a race in an outage test that hid a real bug in how the auth watch judged "behind" during a majority loss; a probe-bucket cleanup; external streams that could not be placed).
What stays, with issues: #1189 inbound peering affinity (peering state in NATS), #1190 If-Match and conflict mode fail across instances, #1191 shared move marks, #1192 guests in the members' editor session, #1193 "Your sessions" and End all, #1194 optional inactivity timeout; #1176, #1177, #1187, #957 as before. Known limitations in multi-instance.md list the residuals.
Closing #742: the multi-instance announcement has shipped; follow-ups are tracked in the issues above.