Table of Contents
- Coordination store benchmark: method, decision criteria and results
- Results
- Verdict
- The four steps
- Step 1. Correctness (disqualifying)
- Step 2. Availability (pass or fail)
- Step 3. Cost
- Step 4. Latency
- Harness changes 0.3.0 to 0.3.4
- Limits of the evidence
- Open risks
- Raw data
- Setup (reference)
- Method and decision criteria (published 2026-11-06)
Coordination store benchmark: method, decision criteria and results
How Cairn will choose the store that lets several instances work together (#742). This page fixed the method and the decision criteria before any measured run, so the results cannot steer them. The design was approved on 2026-11-05 and the decision criteria were made precise on 2026-11-06, before publication and before any measured run. Any change to the method after publication is listed in the changelog at the end, with the reason.
Status (2026-11-13): all four decision steps evaluated. The benchmark favours N-sync. Decision pending (Nikola).
The results come first. The method and decision criteria as published on 2026-11-06 follow unchanged from "Method and decision criteria" on.
Tags used in the results: VERIFIED (from the stored results, logs or source), INFERRED (reasoned from verified facts), UNVERIFIED (not checked or not checkable).
Names used below: N-def = NATS JetStream embedded in Cairn, 3 nodes, default sync_interval. N-sync = the same with sync_interval: always. P1 = one PostgreSQL 18 server. P2 = PostgreSQL primary plus one synchronous standby. P-HA = highly available PostgreSQL run by the customer (not tested; see step 2 of the method).
Results
Verdict
What the four steps favour: N-sync (embedded NATS JetStream, 3 replicas,
sync_interval: always, 3 Cairn instances). This is the benchmark's finding under the published method. The store decision itself is pending (Nikola).
- Step 1, correctness: N-sync, P1 and P2 lost no acknowledged write in any trial. N-def lost acknowledged writes in 10 of 10 trials when all three NATS VMs lost power, so it is out.
- Step 2, availability: only NATS kept working after one store VM was lost, for writes and (since harness 0.3.4) for leases. P1 and P2 stopped all writes until the lost VM came back. P-HA passes by design, untested.
- Step 3, cost: operator burden comes first and favours N-sync (no database servers). Inside each Cairn instance N-sync costs more: about +80 millicores CPU and about +235 MiB memory at L2000.
- Step 4, latency: does not decide. N-sync writes take about 1.4x as long as P2 at the median (+0.7 to +1.9 ms), but every p99 is far below 50 ms.
Caveats that go with this finding:
- The default Cairn pod memory limit (256 MiB) would have to rise, to about 384 to 512 MiB.
- NATS needs 3 nodes. A 2-instance deployment needs a NATS-only third server.
- After a node is lost, NATS writes keep timing out for about 20 to 30 s, and leases can be unavailable for up to about 25 s. Cairn needs a timeout and a retry on every store call.
- Open Jepsen reports against NATS cover faults this benchmark did not test (corrupted files, power cut plus pauses). Stale reads from replicas were not measured.
- The latency figures come from 2 short runs per cell. They are good enough to show that latency does not decide. They are not good enough to quote publicly.
The four steps
Step 1. Correctness (disqualifying)
What was tested. Each trial ("cell") starts a fresh store, runs the normal load plus a correctness workload, injects one fault at a random time, heals it, and reads everything back. The checkers then look for lost acknowledged writes, duplicate or outdated lease tokens, stale fenced writes, and (with elle) whether the history is serializable. Ten trials per configuration and fault. The fault list is in "Fault classes" below.
Campaigns.
| Campaign | Harness | Configurations | Faults | Cells |
|---|---|---|---|---|
fault742 |
0.3.0 / 0.3.1, re-checked with 0.3.2 | N-def, N-sync, P1 | all that apply | 230 |
fault742p2 |
0.3.2 (cells 001 to 044) and 0.3.3 | P2 | all that apply | 110 |
fault742-p1f5-rerun |
0.3.3 | P1 | F5 (2 cells rerun with the same seeds) | 2 |
fault742-lt |
0.3.4 | N-def, N-sync | F3, F6 (rerun with lease timeouts, for step 2) | 40 |
Result per configuration (VERIFIED). "Stored" is the verdict the campaign wrote at the time. The final count is after every recheck and rerun.
| Configuration | Trials | PASS | FAIL | ERROR | Stored PASS / FAIL / ERROR |
|---|---|---|---|---|---|
| N-def | 90 | 80 | 10 | 0 | 64 / 26 / 0 |
| N-sync | 90 | 90 | 0 | 0 | 70 / 20 / 0 |
| P1 | 90 | 90 | 0 | 0 | 81 / 8 / 1 |
| P2 | 110 | 110 | 0 | 0 | 107 / 3 / 0 |
| all | 380 | 370 | 10 | 0 | 322 / 57 / 1 |
- All 10 FAILs are N-def F4 (all three NATS VMs power-cut) and they are real. Each cell lost 493 to 1,123 acknowledged appends (7 cells), or about 3,000 journal entries (3 cells), and showed stale fenced writes and token regressions. This matches Jepsen's NATS 2.12.1 finding for the default
sync_interval. N-def is out. NATS stays a candidate only through N-sync. - N-sync, P1 and P2: zero lost acknowledged writes, zero stale compare-and-set wins, zero reused or outdated tokens, in every trial. With 10 trials per cell this means "none observed": a per-trial failure rate up to about 26% is still consistent with it.
- Every other non-PASS stored verdict (47 FAIL, 1 ERROR) was a harness fault, not store behaviour. Each was found, fixed, and the cell re-checked or rerun. See "Harness changes" below.
fault742-lt(0.3.4): all 40 cells PASS (20 N-def, 20 N-sync).
Why stored verdicts changed, NATS (fault742).
| Config | Fault | Final PASS / FAIL | Stored PASS / FAIL | Why it changed |
|---|---|---|---|---|
| N-def | F1 | 10 / 0 | 10 / 0 | |
| N-def | F2 | 10 / 0 | 9 / 1 | strict-serializable inversion between hosts (191 us): clock skew |
| N-def | F3 | 10 / 0 | 0 / 10 | revived VM ran on a whole-second hardware clock until chrony synced; crash holes in the history aborted elle |
| N-def | F4 | 0 / 10 | 0 / 10 | real loss (see above) |
| N-def | F5 | 10 / 0 | 9 / 1 | invoke left open by the kill made elle abort |
| N-def | F6 | 10 / 0 | 9 / 1 | cross-host clock skew (1.2 ms) |
| N-def | F7 | 10 / 0 | 7 / 3 | cross-host clock skew (77 us to 1.26 ms) |
| N-sync | F1, F2 | 20 / 0 | 20 / 0 | |
| N-sync | F3 | 10 / 0 | 0 / 10 | revived-VM clock (8 cells), crash holes in the history (2 cells) |
| N-sync | F4 | 10 / 0 | 0 / 10 | crash holes only: 93 appends lost with the crashed nodes' own history files; the store had them |
| N-sync | F5 | 10 / 0 | 10 / 0 | clean (see "F5 and LazyFS" below) |
| N-sync | F6, F7 | 20 / 0 | 20 / 0 |
PostgreSQL.
- P1: all 8 stored FAILs were cross-host clock skew (155 us to 2.6 ms; all valid under serializable). One cell (005-P1-F5-r4) had no history after a LazyFS bug. Both 005 and 009 P1 F5 were rerun with 0.3.3, same plan lines and seeds: both PASS, with a clean LazyFS log.
- P2: 3 stored FAILs (016, 018, 019) came from the old fencing check, which compared a lease grant on one host with a write on another host by wall clock. Store order was clean. They PASS with the published 0.3.3 checker. All 9 strict-serializable INVALIDs and all 3 of these FAILs fall in one 95-minute window (cells 013 to 025), so a few ms of clock offset between VMs then (INFERRED).
- P2 F5 first attempts (009, 023, 045) were invalid: the LazyFS daemon ran out of file descriptors and silently dropped fsyncs. They were retried with 0.3.3: 0 dropped writes. Even the invalid attempts lost no acknowledged append.
F5 and LazyFS (VERIFIED). LazyFS 0.3.1 reports success for an fsync it did not do when it runs out of file descriptors. This hit PostgreSQL. A separate check showed the 20 NATS F5 cells were not affected: the NATS store has only about 140 files, the whole system had at most 640 open file handles in every NATS F5 snapshot (limit 1024 for the daemon alone), and the one surviving NATS LazyFS log (229-N-def-F5-r5, all 3 nodes) shows 0 dropped writes. No NATS F5 rerun was needed.
Not a correctness failure, but noted: stale reads were not measured (see "Open risks").
Step 2. Availability (pass or fail)
The rule. After one store VM is lost (F3, a hard power cut), do writes succeed again on every surviving Cairn instance, without anyone acting? How long the window is does not decide among configurations that pass.
How it was measured. From the correctness histories (about 30 writes per second per instance), not from the load generator. A gap of more than 1 s between successful writes counts as an outage. "Writes resume for good" is the end of the last such gap before the fault heals. Lease figures count successful acquires and fenced writes per lease name, with a 4 s gap threshold (the workload itself pauses a holder for 3 s). p50 and max are over the 10 trials. VERIFIED unless tagged.
One store VM power-cut (F3; for P2 also the standby, F8a). Times in seconds after the fault. The VM is powered on again after 30 s. Where two values are given, the first is from fault742 and the second from fault742-lt (0.3.4).
| N-def | N-sync | P1 | P2, primary cut (F3) | P2, standby cut (F8a) | |
|---|---|---|---|---|---|
| survivors write again while the VM is still down | yes, 10/10 | yes, 10/10 | no, 0/10 | no, 0/10 | no, 0/10 |
| first outage ends, p50 / max | 5.5 / 17.2; 5.9 / 21.9 | 5.4 / 12.0; 5.5 / 11.3 | 39.5 / 39.7 | 53.8 / 60.2 | 64.0 / 64.8 |
| writes resume for good, p50 / max | 10.1 / 29.4; 10.8 / 21.9 | 14.2 / 22.0; 7.8 / 16.7 | 39.5 / 39.7 | 53.8 / 60.2 | 64.0 / 64.8 |
| last timed-out write, p50 / max | 27.7 / 33.9; 26.3 / 34.2 | 26.1 / 57.7; 24.8 / 30.7 | 46.7 / 54.8 | 54.5 / 60.2 | none time out; writes hang |
| every lease back, p50 / max | not measurable before 0.3.4; 20.5 / 24.6 | not measurable before 0.3.4; 20.2 / 25.4 | 39.5 / 39.7 | 53.8 / 60.2 | 64.0 / 64.8 |
| what ends the window | NATS elects a new leader on the two survivors | same | VM back and PostgreSQL started | VM back, then the standby re-attaches | VM back, then wal_sender_timeout (60 s) on the primary |
In plain words:
- NATS: the two surviving instances lose writes for about 5 s, then write again while the cut node is still off. Single writes keep timing out (5 s or 10 s client timeouts) for about 25 to 30 s. Since 0.3.4, every lease is back after about 20 s (max 25 s), well before the cut node returns at about 41 s.
- P1: nothing is written until the server is back. In production that is an operator action or the power returning.
- P2: worse than P1 here. Losing the primary stops writes until it is back and the standby has re-attached. Losing the standby also stops every commit: commits wait for the synchronous standby, and they stay blocked until it is back and the primary drops the old connection. A permanent loss needs a manual promotion (F8b, not run) or a change of
synchronous_standby_names.
Why leases were "not measurable" before 0.3.4. In fault742, a NATS lease call that was in flight when the fault hit never returned: it had no timeout of its own and waited until the end of the run. So in 20 of 20 F3 cells the survivors made no successful lease call until the cut node came back, and in 20 of 20 F6 cells some lease never came back. That was the harness, not NATS. 0.3.4 gives every lease call a 5 s timeout, and F3 and F6 were rerun for both NATS configurations (fault742-lt). No lease call then took longer than 5.001 s.
Partition of one NATS node from the other two (F6, 30 s).
| N-def (0.3.4) | N-sync (0.3.4) | |
|---|---|---|
| majority writes resume for good, p50 / max | 14.3 / 17.4 | 17.0 / 30.5 |
| leases during the partition, back for good, p50 / max | 18.1 / 26.0 | 21.2 / 24.1 |
| lease gap when the partition heals, longest, p50 / max | 8.6 / 10.4 | 8.3 / 10.4 |
| every lease back after the heal, p50 / max | 38.3 / 40.7 | 38.0 / 41.3 |
| last timed-out write, p50 / max | 37.6 / 39.6 | 38.5 / 39.1 |
The majority side keeps working during the partition. The larger disturbance is at the heal, when the isolated node rejoins and leaders change (INFERRED). With PostgreSQL, F6 cuts one Cairn instance off the primary: the other instances see no gap, and they take over its leases within the 2 s lease TTL.
Other faults, briefly. Killing or stopping the PostgreSQL server stops all writes until it restarts (P1 about 21 s, P2 about 25 s, the extra time is the standby re-attaching). Pausing it for 6 s blocks every write for about 6.3 s. Killing one NATS node caused a write gap above 1 s in 2 of 10 (N-def) and 1 of 10 (N-sync) cells, up to 6.5 s. Pausing one NATS node for 6 s blocked the survivors' writes for 5 to 7 s. Killing, stopping or pausing one Cairn instance with PostgreSQL does not affect the others.
Lease refusals (0.3.4 detail). In the perf runs, 0 to 0.25% of lease cycles were refused, the level expected by design. In the fault cells, 456 of about 5.47 million refusals were "refused while free". Most of those were checker artefacts (history lost with a powered-off node, clock offsets of 20 to 45 ms after a power cut, a 15 ms miss at the TTL boundary). The real NATS property behind the rest: after a leader change, an expired lease can stay in place for up to about 13 s, because the expiry is only applied by a leader with quorum. This is not a correctness failure, but Cairn would inherit it. VERIFIED counts; causes INFERRED.
Step 2 result.
| Configuration | Step 2 | Write window after one store VM is lost (F3), p50 / max |
|---|---|---|
| N-def | pass (already out at step 1) | 10.8 / 21.9 s; leases 20.5 / 24.6 s |
| N-sync | pass, writes and leases | 7.8 / 16.7 s; leases 20.2 / 25.4 s |
| P1 | fail | writes resume only after the server is back (39.5 / 39.7 s in the test) |
| P1-2 | fail (takes P1's result; not run) | as P1 |
| P2 | fail as tested | primary 53.8 / 60.2 s, standby 64.0 / 64.8 s, both only after the VM is back |
| P-HA | passes by design, untested | not measured |
Step 3. Cost
Step 3 compares N-sync with P2 / P-HA (P-HA uses P2's measured results, as the method says). Operator burden comes first, then resources.
Operator burden.
| N-sync | P2 / P-HA | |
|---|---|---|
| extra servers to run and patch | 0 at 3 Cairn instances; 1 NATS-only server at 2 instances | P2: 2 PostgreSQL VMs. P-HA: the customer's PostgreSQL servers plus failover tooling (Patroni, CloudNativePG) or a managed service |
| minimum Cairn instances for HA | 3 (Raft quorum) | 2 |
Resources (perf742-lt and perf742-lt-p2, harness 0.3.4, 2 runs per cell; VERIFIED). CPU includes the load generator's 2 ms timer spin, so compare differences, not absolute values.
| N-sync L200 | N-sync L2000 | P2 L200 | P2 L2000 | |
|---|---|---|---|---|
| CPU per Cairn instance (cores) | 0.070 | 0.472 | 0.056 | 0.393 |
| max memory (RSS) per Cairn instance (MiB) | 211 to 224 | 289 to 306 | 55 to 63 | 59 to 65 |
| total busy CPU, all VMs (cores) | 0.305 | 1.416 | 0.299 (incl. 2 PG VMs) | 1.327 (incl. pg1 0.167, pg2 0.022) |
| disk written per write op, marginal (KB) / flushes | 98.9 / 4.3 | 21.0 / 2.8 | ||
| network, all VMs (MB/s) | 1.68 | 3.27 | 1.39 | 2.90 |
| store size on disk | about 558 MB per node, 3 copies | about 662 MB, plus the standby copy |
- Inside one Cairn instance at L2000, N-sync costs +79 millicores CPU and about +235 MiB memory more than P2. Both are above the method's thresholds (25 m, 13 MiB) and the run ranges do not overlap, so the difference counts. At L200 the CPU difference (+14 m) is below the threshold; the memory difference (about +155 MiB) is not. The earlier perf742 runs gave the same picture (+70 m, +214 MiB).
- Total CPU is about equal (+0.09 cores for N-sync at L2000), but P2 needs two extra VMs, each with its own operating system (about 470 to 500 MiB used on each).
- Disk: N-sync writes about 4.7x the bytes per write op (3 replicas, an fsync per write). Neither configuration comes near the disk's limits at these rates (INFERRED).
The 256 MiB pod limit. The current Cairn pod default is 0.5 CPU and 256 MiB. P2's client uses at most 65 MiB, which leaves about 190 MiB for Cairn. N-sync does not fit at L2000. It was run once more with a 256 MiB cgroup cap and GOMEMLIMIT 230 MiB (perf742-lt-cap256, 2 runs per level):
| N-sync under a 256 MiB cap | L200 | L2000 |
|---|---|---|
| max RSS per instance (MiB) | 211 to 243 | 250 to 256 |
| cgroup memory peak | 214 to 254 M | 256 M, the cap, on all 3 nodes in both runs |
| OOM kills / errors / timeouts | 0 / 0 / 0 | 0 / 0 / 0 |
| state.update p50 / p99 (ms) | 4.72 / 7.05 | 2.76 / 5.81 |
| tail | normal | one of the two runs had 60 to 140 ms spikes (signin p99 66 ms, lease.cycle max 137 ms) |
So the benchmark process alone (load generator plus embedded NATS, without any Cairn code) survives L2000 at 256 MiB, but with zero headroom and with occasional stalls. The default memory limit would have to rise, to about 384 to 512 MiB (INFERRED). Recovery of a power-cut node under a cap was not tested: uncapped, a revived node's cgroup peaked at 829 to 896 MB while it caught up.
Step 3 result (INFERRED per the method). Operator burden comes first and favours N-sync: no database servers and no failover tooling, against two PostgreSQL VMs for P2 or the customer's HA setup for P-HA. The resource cost goes the other way and is reported as a cost, not a disqualification: about +80 m CPU and +235 MiB per instance, and a higher default pod memory limit.
Step 4. Latency
The rule. Latency decides only if an operation's p99 at L2000 is above 50 ms for one configuration and below it for the other, with confidence intervals on opposite sides of 50 ms.
Fair comparison (both harness 0.3.4, different seeds per instance, same plan shape: 2 runs per cell, 1 min warm-up, 5 min measured). Latency is measured from the planned send time. Per run the three instances are merged; the table gives the median of the 2 run values, in ms. VERIFIED.
N-sync from chain 8 (perf742-lt, 2026-11-13 about 01:00 to 02:30 UTC); P2 from chain 9 (perf742-lt-p2, 17:16 to 17:41 UTC); N-def (out at step 1) for reference.
L2000 (about 496 operations per second in total)
| Operation | N-sync p50 / p99 | P2 p50 / p99 | Ratio N-sync / P2, p50 / p99 | N-def p50 / p99 |
|---|---|---|---|---|
| state.update | 2.76 / 5.42 | 1.91 / 3.73 | 1.44 / 1.45 | 0.67 / 1.81 |
| upload.progress | 2.80 / 4.69 | 1.98 / 4.16 | 1.41 / 1.13 | 0.69 / 1.43 |
| journal.append | 2.43 / 4.69 | 1.72 / 3.86 | 1.42 / 1.22 | 0.45 / 0.89 |
| lease.cycle (acquire + release) | 4.72 / 7.71 | 2.79 / 5.14 | 1.69 / 1.50 | 0.99 / 1.99 |
| lease.renew | 0.59 / 4.00 | 1.79 / 3.74 | 0.33 / 1.07 | 0.49 / 1.00 |
| lock.write | 4.57 / 7.21 | 2.96 / 3.94 | 1.54 / 1.83 | 0.94 / 1.39 |
| signin | 4.71 / 7.11 | 3.10 / 4.53 | 1.52 / 1.57 | 1.15 / 1.93 |
| session.get | 0.24 / 0.52 | 0.21 / 0.42 | 1.14 / 1.23 | 0.23 / 0.51 |
| state.get | 0.24 / 0.90 | 0.23 / 0.49 | 1.05 / 1.84 | 0.24 / 0.89 |
| journal.read | 0.47 / 1.32 | 0.34 / 0.69 | 1.37 / 1.90 | 0.46 / 1.29 |
| notify.delivery | 0.12 / 0.28 | 0.28 / 0.78 | 0.45 / 0.35 | 0.12 / 0.27 |
What this says:
- N-sync durable writes take 1.4x to 1.7x as long as P2 at p50 (+0.7 to +1.9 ms) and 1.1x to 1.8x at p99. The cause is an fsync on the leader and a follower before every acknowledgement (INFERRED).
- Reads are equal at p50. P2's read p99 is about half of N-sync's. Notifications are 2 to 3x faster on NATS.
- lease.renew is faster on NATS because two thirds of renews are refusals, which NATS answers from a read and PostgreSQL from an UPDATE (INFERRED).
- N-sync has a heavier extreme tail: single maxima of 53 ms (lease.cycle) and 29 ms (journal.append) against at most 10 ms for P2. Cause UNVERIFIED.
- At L200 (about 53 ops/s) N-sync is 1.1x to 1.8x slower at p50, but its p99 is lower than P2's for most writes (P2 L200 write p99 8.9 to 10.3 ms).
- Both configurations hold the offered rate (achieved / offered 0.99 to 1.00), with 0 errors and 0 timeouts.
Step 4 result. The highest run p99 of any operation at L2000 is 8.6 ms (N-sync signin) and 5.5 ms (P2 lease.cycle). No run p99 anywhere is near 50 ms. With 2 runs there is no confidence interval, but every value is more than 39 ms below the line. Step 4 does not decide. The only p99 above 50 ms in all the new data is N-sync's signin in one capped run under the 256 MiB limit (about 50 signins per run), which is not the step 4 cell.
Why the earlier perf742 numbers were wrong, and by how much. The first performance matrix (perf742, harness 0.3.0, 5 runs of 20 min) gave every instance the same workload seed. The three instances therefore picked the same keys at the same planned instants. They refused each other's leases about 2 times in 3 and collided on compare-and-set updates, which then retried. This inflated mostly the tails, and hit N-sync hardest (INFERRED: its slower writes widened the collision window).
| L2000 | perf742 (same seeds) | 0.3.4 (distinct seeds) | Change |
|---|---|---|---|
| N-sync state.update p50 / p99 (ms) | 4.41 / 23.12 | 2.76 / 5.42 | p50 -37%, p99 -77% |
| N-sync journal.append p50 / p99 | 2.98 / 6.42 | 2.43 / 4.69 | -18% / -27% |
| N-sync retries per state.update | 1.14 | under 0.005 | gone |
| N-sync lease.cycle refused | about 65% | 0.25% | gone |
| P2 state.update p50 / p99 | 2.11 / 10.14 | 1.91 / 3.73 | p50 -9%, p99 -63% |
| P2 journal.append p50 / p99 | 1.39 / 4.62 | 1.72 / 3.86 | p50 +24%, p99 -16% |
| P2 retries per state.update | 0.33 | 0.0004 | gone |
| N-sync / P2 ratio, state.update p50 | 2.09x | 1.44x | |
| N-sync / P2 ratio, journal.append p50 | 2.14x | 1.42x |
- About 40% of N-sync's old state.update median and about 75% of its p99 was this artefact. For P2 it was about 9% of the median and about 60% of the p99.
- N-def did not change (0.66 / 1.74 ms before, 0.67 / 1.81 ms after). That confirms the measurement itself did not change, only the workload.
- P2's journal.append median rose by 0.33 ms in the rerun. It is flat over each 5-minute run, so it is not a warm-up spike. Cause UNVERIFIED (the test bed is 6 days older, and the warm-up shorter). It makes the N-sync / P2 ratio smaller, so it does not favour P2.
- So the old statement "N-sync writes take about 2x as long as P2" overstated the gap. The fair figure is about 1.4x.
Harness changes 0.3.0 to 0.3.4
The harness is coordbench (Cairn repository, branch bench742, bench/coordbench). Each version fixed something that had produced false verdicts or distorted numbers. Which binary ran each cell is VERIFIED from the per-cell facts files.
| Version | Commits | What was wrong | Effect on earlier numbers | Used for |
|---|---|---|---|---|
| 0.3.0 | up to 9dce0452 |
In a 3-node NATS preload, the two nodes that were not preloading stopped their servers, and the preload lost quorum. | Found on the VMs before any counted run. None. | perf742; fault742 cells 001 to 004 |
| 0.3.1 | 256705c0, cfa13bf8 |
elle-cli needed a real JSON history; the LazyFS config section was wrong; LazyFS creates files as root, so PostgreSQL panicked on a root-owned file after an F5 cell. | 005-P1-F5-r4 ended with no history (ERROR); later rerun. | fault742 cells 005 to 230 |
| 0.3.2 | 0a24708b |
(a) elle's default model is strict-serializable, which compares wall clocks of different VMs; sub-ms clock offsets failed whole histories. (b) A crashed node loses the unflushed end of its own history, and a killed client leaves open calls; elle-cli crashed on both, which was reported as INVALID. | Most stored NATS FAILs (all F3, N-sync F4, several clock-skew cells) became PASS on recheck. The N-def F4 losses stayed FAIL. | recheck of all 230 fault742 cells; P2 cells 001 to 044 |
| 0.3.3 | c0dc3ee0, cbf7f124, 156b7511 |
(a) The fencing check compared a lease grant on one host with a write on another by wall clock. (b) The LazyFS daemon kept a 1024 open-file limit and, when it ran out, reported fsyncs as done without doing them. (c) One failed LazyFS unmount stopped a whole chain. | (a) P2 016, 018, 019 FAIL became PASS. (b) P2 F5 first attempts 009, 023, 045 invalid and retried; P1 F5 009 suspect after its fault, rerun with 005. NATS F5 checked separately: not affected. | P2 F5 retries and cells 046 to 110; P1 F5 reruns |
| 0.3.4 | af1d62c4 to 2216b349 |
(a) Every instance ran the same workload seed. (b) NATS decided "lease held" from a replica's possibly lagging copy. (c) NATS lease calls in the correctness workload had no timeout and could hang until the end of the run. Also new: refusal detail (who held the lease, and why), memory caps, single-cell reruns, a recorded harness version. | (a) perf742 lease and compare-and-set figures, and the N-sync / P2 ratio, were distorted (see step 4). (c) NATS lease availability after F3 and F6 could not be measured; rerun as fault742-lt. |
fault742-lt, perf742-lt, perf742-lt-cap256, perf742-lt-p2 |
Notes:
- The fault campaigns before 0.3.4 also used the same seed on every instance. That raised contention in the correctness workload, but it does not change what the checkers verify, so the step 1 verdicts stand (INFERRED).
- Published binaries: 0.3.3 sha256
7cf33435a0cc5648a3707f68e45e8c878d6d731a865bfac62ebabcba3a408f0c; 0.3.4 sha256ac2e54045e19405d5ce77ae78746372a60f41647c6b83288f3bd4509834b9846(on all six VMs in chains 8 and 9). The harness CHANGELOG (bench/coordbench/CHANGELOG.md) describes 0.3.4 in detail. - Other pitfalls that were handled by procedure: a VM revived after a power cut runs on a whole-second hardware clock until chrony corrects it; a chain started while an Ansible deploy was still running got 409 start refusals;
coordbench checkrewrites history files in place, so every recheck ran on copies; plan numbering depends on the configuration list, so single-cell reruns used the original plan filtered to keep the same seeds.
Limits of the evidence
- Latency rests on 2 short runs per cell. Chain 8 (N-sync, N-def) and chain 9 (P2) ran 2 runs of 5 minutes each, about 15 hours apart, not interleaved as the method asks. Run ranges are narrow (for example P2 state.update p50 1.90 to 1.92 ms) and N-def matched the earlier matrix, so there is no sign of drift. But 2 runs give no confidence interval, and 5 minutes is short for p99.9 and maxima.
- What a "publishable" latency rerun would add: N-sync and P2 interleaved, 5 runs of 20 minutes each with 0.3.4 (about 4 hours at L2000 only). That gives bootstrap confidence intervals as the method asks, and removes time-of-day effects. Recommended before quoting latency publicly. Not needed for the decision, because latency does not decide: every p99 is far below 50 ms.
- Per-operation disk figures from the 6-minute runs include each run's store reset; the "marginal" figures remove that share.
- Ten trials per fault: "none observed" still allows a per-trial failure rate up to about 26%.
- Not run: P1-2 (PostgreSQL with 2 Cairn instances), F8b (primary power cut plus manual promotion), the small-node profile, stress cells other than the ramp. The ramp could not find a saturation point: the load generator, not the store, was the limit at about 8,000 ops/s.
- One host, one NVMe disk, a hypervisor hard stop rather than a physical power cut (see "What this benchmark does not show").
- Clocks: chrony against public pools, error bounds of about 10 to 27 ms, so strict-serializable results stay information only.
- P2 cells 001 to 044 (except the F5 retries) were judged by 0.3.2; only 016, 018 and 019 were re-checked with 0.3.3. Those 41 cells had 0 stale-fence hits, so a recheck would not change them (INFERRED).
Open risks
These do not change steps 1 to 4, but they should be stated next to any choice of N-sync.
- Open NATS Jepsen reports (all still open on 2026-11-13, reported against 2.12.1; the 2.15.0 release notes do not list them as fixed):
- nats-server #7549: single-bit errors in
.blkdata files, even on a minority of nodes, can lose acknowledged writes, also withsync_interval: always. - nats-server #7556: corrupted snapshot files on a minority of nodes can make nodes delete a stream and take the cluster down.
- nats-server #7567: a single-node power failure combined with pauses or partitions can cause split-brain and lost acknowledged writes. It was reported with a positive
sync_interval, so it bears most on the default setting; this benchmark covered it only in part.
- nats-server #7549: single-bit errors in
- Stale reads from replicas were not measured. NATS serves direct gets from any replica, and Cairn makes access decisions from reads (a revoked session, a lock check). No perf run recorded stale reads (
oracle_reads0). This needs its own check before Cairn relies on reads for access decisions. - Node recovery under a memory cap is untested. Uncapped, a revived node's cgroup peaked at 829 to 896 MB while it caught up. Whether a node capped at 384 to 512 MiB recovers from a power cut without OOM or long stalls is UNVERIFIED.
- Disturbed periods after a leader change. Writes keep timing out for about 20 to 30 s after a node is lost. Leases are back after about 20 to 25 s, have an 8 to 10 s gap when a partition heals, and an expired lease can stay in place for up to about 13 s after a leader change. Cairn needs per-call timeouts and retries on every store call.
- Not tested at all: a full disk, clock steps, adding or removing cluster members, upgrades. With embedded NATS, Cairn owns cluster membership, upgrades and recovery.
- If a site already runs highly available PostgreSQL, its operator burden is already paid, and P2's lower write latency and memory apply there. Offering a PostgreSQL driver for such sites would be a separate product decision, not what this benchmark chose. P-HA failover was not tested by us.
Raw data
Archives on the operator workstation in ~/coordbench-archive/. The first five were checked after transfer (remote stream sha256 equals local sha256, zstd -t, file count matches the controller); chain8 and chain9 with zstd -t and a full tar listing. No secrets: env files, keys, tokens and TLS material were excluded and the contents grep-checked. Checksums from SHA256SUMS.
| Archive | Contents | Size (bytes) | sha256 |
|---|---|---|---|
results.tar.zst |
controller results: fault742, fault742p2 (with attempt dirs), the aborted 005-P1-F5-r4 chain, perf742, calibration, preload |
5,571,264,912 | 5bd95b18652d6a6da9662a05c078ab390a588cc2c2a0c75e8d91e2238343c293 |
home-coordbench.tar.zst |
controller run-book home without env files: chain scripts and logs, run book, elle investigation, rechecks (incl. recheck-p2-pub), P1 F5 rerun scripts |
418,511,436 | bdc26f9e4fca4053d16972053623758f7636675f60ae98a78fd05c67d35bf3fc |
ctl-journal.tar.zst |
controller journals of the chain units and of the Ansible / 409 window | 13,761 | 1b40256028f54f7a257228a9d2e6f1ce3ebc5070729c00ecdfff5342b9dc98b9 |
pg1-logs.tar.zst |
PostgreSQL primary logs (all EMFILE and PANIC evidence) and the LazyFS log of P2 045's first attempt | 1,432,842 | 05e164ebe068453dab702ee623d264191c71c11079d33b5eef092d82632c43b8 |
pg2-logs.tar.zst |
PostgreSQL standby logs | 92,532 | 16aa1be31693f503bb9f45fd5f7f605e8f6a846b6aec2ca975cac8ddbc60e6f6 |
fault742-p1f5-rerun.tar.zst |
the two P1 F5 reruns (005, 009) | 37,521,683 | 57ec7b67a8f563821d7decc4039e46383845e7cf1b19066748576f501b39d5ee |
chain8.tar.zst |
fault742-lt, perf742-lt, perf742-lt-cap256 (harness 0.3.4) |
769,873,850 | 90515a617efb3f30c15d63353efb71b4cc466562d4c260638058dd6c375719b0 |
chain9.tar.zst |
perf742-lt-p2 (harness 0.3.4) |
1,003,609 | 8f25ed722a078e1c1fa2d84faf963b41450782efbc64e6b633a5457b882bde3f |
- Per cell:
planline,control.json/control.log,node*/(summaries, HdrHistogram logs, histories),history-all.jsonl,history-elle.json,check.txt,cell.json,facts/,snap/,logs/. - The 0.3.3 verdicts for P2 016/018/019 are in
recheck-p2-pub/*/check-pub.txt(home archive); the storedcheck.txtthere still says FAIL (0.3.2). - To re-check a cell: unpack it to a scratch directory and run
coordbench check -dir <cell copy> -elle <elle-cli jar>. It rewrites the history files in that directory, so never run it on the only copy. - Harness source: branch
bench742,bench/coordbench(0.3.4 at2216b349). Test bed: homelab-config,ansible/playbooks/coordbench.yml. - The workstation holds the only off-VM copy so far.
- The test VMs are kept for now. The only NATS LazyFS logs (cell 229, on the three instance VMs under
/run) are not in the archive and would be lost on reboot.
Setup (reference)
All VMs run on one Proxmox host with a local NVMe SSD, Debian 13, chrony against public NTP pools, a host firewall between bench VMs, TLS on all store traffic.
| Role | Count | Size | What runs there |
|---|---|---|---|
| Cairn instance VM | 3 | 4 vCPU, 4 GB | coordbench node (driver, load generator, correctness workload); for N-def and N-sync also the embedded NATS server |
| PostgreSQL primary VM | 1 | 4 vCPU, 4 GB | PostgreSQL 18 (P1 and P2) |
| PostgreSQL standby VM | 1 | 4 vCPU, 4 GB | PostgreSQL 18 synchronous standby (P2 only) |
| Controller VM | 1 | 2 vCPU, 2 GB | coordbench control and check, elle-cli, run book, results |
- Versions: nats-server 2.15.1, nats.go 1.53.1; PostgreSQL 18.6, pgx 5.12.0; Go 1.26.10; LazyFS 0.3.1; elle-cli 0.1.11 on Java 21.
- Calibration (2026-11-06): fdatasync p50 about 0.65 ms (fio, 4 KiB); 0.29 ms per 8 KiB WAL sync (
pg_test_fsync); all runs use the 2 ms timer spin (timer lateness p99 27 us with spin). - Fault cells: 1 min warm-up and 5 min at L2000 with the correctness workload (list-append on 1,000 keys, a journal set, fenced lease writes with a 2 s TTL) and one fault at a random time, then final reads.
Fault classes
| Fault | NATS (N-def, N-sync) | PostgreSQL (P1, P2) |
|---|---|---|
| F1 | SIGTERM one Cairn instance (with its NATS server), restart after 20 s | SIGTERM the PostgreSQL primary, restart after 20 s |
| F1i | n/a | SIGTERM one Cairn instance, restart after 20 s |
| F2 | kill -9 one Cairn instance (with its NATS server) |
kill -9 the PostgreSQL primary |
| F2i | n/a | kill -9 one Cairn instance |
| F3 | power cut (hypervisor hard stop) of one instance VM, 30 s | power cut of the PostgreSQL primary VM, 30 s |
| F4 | power cut of all three instance VMs | P2 only: power cut of primary and standby VMs |
| F5 | LazyFS under one node's NATS data: clear unsynced data, then kill | LazyFS under the primary's data directory: clear unsynced data, then kill PostgreSQL |
| F6 | network partition of one NATS node from the others, 30 s | partition of one Cairn instance from the primary, 30 s |
| F7 | pause (SIGSTOP) one Cairn instance for 6 s (3x the lease TTL) | same |
| F7s | n/a | pause the PostgreSQL primary for 6 s |
| F8a | n/a | P2 only: power cut of the standby VM |
Method and decision criteria (published 2026-11-06)
Why this benchmark
One Cairn instance needs no extra service, and that stays the default. Running 2-3 instances at one site behind one load balancer is a v0.7 goal. File content is not the problem: each save records its content before it publishes, so several instances can already write to one storage backend without losing data (#922). What the instances must share is state, and that needs a coordination store.
What Cairn needs from that store:
- Durable records with compare-and-set. Accounts, app passwords, groups, admins, shares, legal holds, settings and the space registry. A write must succeed only if the record still has the version the writer read, so a revoked app password or a removed admin cannot come back because two instances wrote at the same moment (#1011).
- Leases with fencing tokens. Background jobs that must run on one instance at a time, and the per-path ordering of saves. A lease can expire while its holder is paused, so every write made under a lease carries a token that only grows, and the store refuses a write whose token is no longer current (Kleppmann, "How to do distributed locking").
- Sessions with expiry, extension and revocation that every instance sees.
- Locks for WebDAV, manual and office-editor locks: one record per file, taken only if no other holder exists.
- Upload progress records for resumable uploads that continue on another instance: the record changes only if the expected version still matches, and the losing writer stops.
- A change journal that sync clients read after a cursor, never skipping an entry that becomes visible later.
- Notifications between instances, for change nudges and cache invalidation.
The storage backend cannot do this job. Garage, a common self-hosted S3 store, accepts conditional-write headers and ignores them (Garage known issues), and NFS locks cannot fence a write.
The two candidates are PostgreSQL, which many IT departments already run or buy as a managed service, and NATS JetStream embedded in the Cairn binary, which lets the instances form their own cluster with nothing extra to install (oCIS and OpenCloud use NATS for their persistent stores). Valkey/Redis was ruled out because its replication is asynchronous and can lose acknowledged writes on failover; etcd because it needs a three-node quorum of its own. Both reasons are in the #742 research.
At 2-3 instances the expected load is a few hundred operations per second. Throughput is therefore not what decides the choice; correctness under failure is.
Candidates and exact configurations
Every configuration runs behind one Go interface that mirrors the coordination layer Cairn will use. The load generator runs inside the same process as the store driver, as Cairn's request handlers will.
The main comparison runs with 3 Cairn instances, within the 2-3 instance scope. P1-2 adds PostgreSQL with 2 Cairn instances, tested with client-side faults only (see "Fault types").
| Name | Store | Cairn instances | Copies of each write | Extra servers |
|---|---|---|---|---|
| N-def | NATS JetStream embedded in each instance, 3-node cluster, sync_interval at its default |
3 | 3 (R3) | none |
| N-sync | as N-def, with sync_interval: always |
3 | 3 (R3) | none |
| P1 | one PostgreSQL server, synchronous_commit = on |
3 | 1 | 1 |
| P2 | P1 plus one synchronous streaming standby | 3 | 2 | 2 |
| P1-2 | P1 with two Cairn instances | 2 | 1 | 1 |
| null | in-memory no-op driver, no store | 3 | none | none |
P-HA names a highly available PostgreSQL that the customer provides (a managed high-availability service, Patroni or CloudNativePG). It is not a configuration we run; it appears in the decision only as described in step 2 and is labelled "not tested by us" wherever it appears.
Versions
- nats-server v2.15.1 with nats.go v1.53.1 (the client version nats-server v2.15.1 requires).
- PostgreSQL 18, the current 18.x minor release at run time; the exact version is recorded with the results. Client: pgx v5.
- Go 1.26 for the harness and both drivers.
nats-server v2.15.0 (2026-09-17) changed what sync_interval: always syncs for replicated streams: it now syncs the Raft log entries and not the upper stream layer (release notes). Throughput figures measured on earlier versions do not apply, which is why only 2.15.1 is measured. Results for 2.12.x would differ.
NATS (N-def, N-sync)
- nats-server runs inside each Cairn instance. The instance connects to its own server in-process, without TCP, and the server opens no client port. The three servers form a cluster over routes with TLS and credentials.
- JetStream file storage, replication factor 3.
JetStreamMaxMemory64 MiB andGOMEMLIMITset explicitly, because the memory-store default is a share of system RAM, which inside a container can be the host's (NATS sizing). - Key-value buckets
state,sessions,locksandleaseswith history 1 and per-key TTL enabled; a streamjournalwith a 30-day maximum age. The stream behindstateallows atomic batch publish. - N-def leaves
sync_intervalat its default: the server acknowledges a write before it is flushed to disk, and flushes about every two minutes. N-sync setssync_interval: always(NATS docs: sync_interval). N-def is measured because it is the default; Jepsen's analysis of NATS 2.12.1 found acknowledged writes lost with it.
How each operation is built, and why:
- Read: a key-value get. NATS documents that these reads are served by any replica, including out-of-date ones ("We do not provide read-after-write consistency", ADR-8). Compare-and-set stays safe, because the leader checks the expected revision, but a read on one instance can miss a write just acknowledged on another. The benchmark therefore measures the stale-read rate (see Metrics). If the Go client can reach a read served by the leader, that read mode is measured as well; if not, the results say so.
- Create-only and compare-and-set: key-value create and update by revision. For keys with a TTL, the update is a raw publish carrying both the expected last revision and the TTL, because the client's key-value update resets the key's TTL, so a lease renewed that way would never expire.
- Lease token: the key-value revision returned when the lease is taken. Revisions only grow.
- Fenced write: one atomic batch (ADR-50) that re-asserts the lease at its current revision and writes the record at its expected version. Whether a batch may span two keys of one bucket this way is the first thing the harness's correctness suite checks.
- Journal: publish to the
journalstream, read by sequence. - Notifications: core NATS publish and subscribe.
Why three nodes. JetStream replicates each write with Raft, which accepts a write only when a majority of nodes has it. With 3 nodes the quorum is 2 of 3, so the cluster keeps accepting writes after one node fails. With 2 nodes the quorum is 2 of 2, so the loss of either node stops writes. Embedded NATS therefore needs 3 nodes. A deployment with 2 Cairn instances and embedded NATS needs a third node that runs only NATS, and that node counts as one extra server under operator burden (step 3 of the decision criteria). The main comparison runs with 3 Cairn instances, so no NATS-only node is part of it, and no two-instance NATS configuration is measured.
PostgreSQL (P1, P2, P1-2)
- Server settings:
synchronous_commit = on,fsync = on,full_page_writes = on(all defaults),shared_buffersat 25% of RAM,max_connections = 100, autovacuum on, with vacuum events logged so their effect is visible. - P2: one streaming standby with
synchronous_standby_names = 'FIRST 1 (standby1)'; all reads go to the primary. A commit is acknowledged only after the standby has flushed it (PostgreSQL 18: synchronous replication). Promotion is manual (pg_promote()); no automatic failover tool is part of this round. - Client: a pgx v5 pool of 16 connections per instance plus one dedicated connection for
LISTEN, TLS withverify-full, every statement prepared, isolationREAD COMMITTED(the default). No connection pooler, because transaction pooling breaksLISTEN. - Any server setting not listed here stays at the PostgreSQL 18 default.
Schema:
CREATE TABLE state (
ns text NOT NULL, key text NOT NULL,
version bigint NOT NULL, value bytea NOT NULL,
expires_at timestamptz, PRIMARY KEY (ns, key)
) WITH (fillfactor = 70);
CREATE INDEX state_expiry ON state (expires_at) WHERE expires_at IS NOT NULL;
-- sessions and locks are rows in state under their own namespace
CREATE TABLE leases (
name text PRIMARY KEY, holder text NOT NULL,
token bigint NOT NULL, expires_at timestamptz NOT NULL
);
CREATE TABLE journal_head (journal text PRIMARY KEY, seq bigint NOT NULL);
CREATE TABLE journal (
journal text NOT NULL, seq bigint NOT NULL, at timestamptz NOT NULL DEFAULT now(),
entry bytea NOT NULL, PRIMARY KEY (journal, seq)
);
How each operation is built, and why:
-
Compare-and-set:
UPDATE ... WHERE version = $expected RETURNING version; zero rows is a conflict. -
Lease: an insert that takes over the row only if the old lease has expired, incrementing
token. Expiry follows the database server's clock (clock_timestamp()), not the instances' clocks. -
Fenced write: one statement that share-locks the lease row at the expected token and updates the record at its expected version, so a concurrent lease takeover waits for the commit:
WITH l AS (SELECT 1 FROM leases WHERE name=$4 AND token=$5 AND expires_at > clock_timestamp() FOR SHARE) UPDATE state SET value=$6, version=version+1 WHERE ns=$1 AND key=$2 AND version=$3 AND EXISTS (SELECT 1 FROM l) RETURNING version; -
Journal: one transaction increments the
journal_headrow, inserts the entry and callspg_notify. A plain sequence is not used: sequence values are assigned before commit and never rolled back (PostgreSQL 18: sequence functions), so a reader polling for entries after its cursor could skip a lower number that commits later. The head row serialises appenders, so commit order equals sequence order. -
Notifications:
pg_notify, delivered at commit. -
Expiry: one instance, holding a lease, deletes expired rows in batches of 1,000 every 60 seconds. Its cost is part of the measurement.
Null baseline
An in-memory driver with the same interface. It measures what the load generator and the Cairn-side code cost on their own, so that a store's cost inside an instance is the difference between that store's run and the null run at the same load. It also serves as the reference in the harness self-tests.
Decision criteria, in fixed order
The criteria are applied in four fixed steps, in this order. A later step is used only between configurations that pass the earlier ones. If the configurations are still even after step 4, the tie-break below decides.
Step 1. Correctness (disqualifying). Under every fault listed below that applies to it (see "Fault types"), the configuration we would ship must show, in all 10 trials:
- zero lost acknowledged writes: every write the store acknowledged is present after recovery, including every acknowledged journal entry;
- zero stale compare-and-set wins: no two successful compare-and-set operations on one key used the same expected version;
- zero reused or outdated fencing tokens: no two lease acquisitions returned the same token, and no fenced write was accepted with a token lower than one already issued.
One violation in one trial disqualifies the configuration. Operations whose outcome is unknown (a timeout or a lost connection) may or may not take effect; they are counted and reported, not treated as failures. N-def is measured for comparison; if it fails, NATS stays a candidate only through N-sync.
Stale reads are measured and reported for every configuration. They are not one of the three disqualifying checks, because compare-and-set stays safe when a read is stale, but Cairn makes access decisions (a revoked session, a lock check) from reads, so the results will state how Cairn handles them for the chosen store.
Step 2. Availability (pass or fail). A configuration passes if, after one store node is lost, writes succeed again on every surviving Cairn instance without any operator action. "One store node is lost" means fault F3 on any one VM that holds store data: for NATS any one of the three Cairn instance VMs; for P1 the PostgreSQL server; for P2 the primary (F3) or the standby (F8). P1-2 has the same store as P1 and takes P1's result. For each trial the write-unavailability window is recorded, and the median and maximum over the 10 trials are reported.
Configurations that pass are preferred to those that do not. Among those that pass, the length of the window does not decide further, so a difference such as 3 s against 5 s does not choose the store.
If no configuration passes step 2, step 2 does not decide, and step 3 compares every configuration that passed step 1. Because P-HA counts as passing (see below), this applies when PostgreSQL has not passed step 1 and neither NATS configuration passes step 2.
A single PostgreSQL server fails this step as tested. In P1, losing the server stops all writes until it is back, and bringing it back is an operator action. P1-2 takes the same result. P2 is expected not to pass either (commits block while the synchronous standby is down, and losing the primary needs a manual promotion); it is measured and reported as measured.
Highly available PostgreSQL provided by the customer (P-HA) counts as passing this step, not tested by us. A managed high-availability service, Patroni or CloudNativePG promotes a standby without operator action, which is what this step asks for. No automatic failover tool is part of this round, so the pass rests on how these setups are designed, not on our trials. Whether a given setup also keeps every acknowledged write through a failover depends on its replication settings (an asynchronous standby can miss the last commits) and is not tested by us either. P-HA runs the same PostgreSQL server with the same settings as P1 and P2, so it passes step 1 only if they do. In steps 3 and 4, P-HA uses the measured results of P2, the tested configuration with the same shape (a primary with a synchronous standby), and its operator burden is the database servers and failover tooling the customer runs, or the managed service it buys.
In the results, step 2 is shown as follows:
| Configuration | Step 2 (availability) | Write-unavailability window |
|---|---|---|
| N-def | pass or fail, as measured | median and maximum over 10 trials |
| N-sync | pass or fail, as measured | median and maximum over 10 trials |
| P1 (single PostgreSQL server) | fail | recorded per trial; writes resume only after the server is restarted |
| P1-2 | fail (takes P1's result) | as P1 |
| P2 | pass or fail, as measured | median and maximum over 10 trials |
| P-HA (customer-provided highly available PostgreSQL) | passes by design, untested | not measured |
A PostgreSQL result here therefore describes the tested configurations, not PostgreSQL in general.
Step 3. Cost. Between configurations that pass steps 1 and 2, in this order:
- Operator burden first: how many extra servers the operator runs and keeps up to date, and the minimum number of Cairn instances the configuration needs. A deployment with 2 Cairn instances and embedded NATS counts its NATS-only third node as one extra server. P-HA counts the database servers and failover tooling the customer runs, or the managed service it buys.
- Resource cost: the store's added CPU and memory inside one Cairn instance, then total system cost (instances plus database servers), disk writes and fsyncs per logical write, and network traffic per operation. CPU and memory are reported together, because with
GOMEMLIMITset, memory pressure shows up as garbage-collection CPU rather than as resident memory.
A resource difference decides only when both of these hold:
- the ranges of the run medians do not overlap: for each configuration, the range from the lowest to the highest of its 5 run medians;
- for CPU and memory inside a Cairn instance, the difference is at least 5% of the pod limit, that is 25 millicores of CPU or about 13 MiB of memory.
Otherwise the difference is reported as "no measurable difference" and does not decide.
The pod limit is the current default for a Cairn pod: 0.5 CPU and 256 MiB. Embedded NATS runs inside the Cairn process, so for N-def and N-sync the results state, at L200 and L2000, whether one Cairn instance with the embedded server stays within those defaults and with how much headroom. Exceeding them is a cost, reported as such (the default limits would have to rise, and every operator would carry that), not a disqualification.
Step 4. Latency. Latency decides only if, at level L2000 (2,000 users), the p99 latency of an operation, measured from its planned send time, is above 50 ms for one configuration and below it for the other, and only when the confidence intervals sit on opposite sides of 50 ms: the lower bound of one configuration's 95% confidence interval for that p99 is above 50 ms and the upper bound of the other's is below 50 ms. An interval that contains 50 ms does not decide.
The operations that count are all thirteen operation types O1 to O13 (see "Operations") as measured in the L2000 cell, compared pairwise between configurations that passed the earlier steps. The L2000-storm cell is excluded from this step: it is reported for information and does not count, and neither do the stress cells (S-ramp, S-hot, S-journal).
Throughput is deliberately not a criterion. Cairn's expected rates are low; the saturation point of each configuration is reported for information.
Tie-break. If the configurations are still even after steps 1 to 4, PostgreSQL is chosen. Reason: a tie means the measured criteria cannot separate the stores, so the decision falls to the risks this benchmark does not measure. For those, embedded NATS has open public Jepsen reports on faults this benchmark cannot cover (corrupted data files on a minority of nodes, #7549 and #7556) or covers only in part (a single power cut combined with pauses, #7567), while PostgreSQL has standard recovery tooling (backups, point-in-time recovery, high-availability tools) that operators and managed services already use. With embedded NATS, Cairn itself would own cluster membership, upgrades and recovery.
Workload
Operations
The workload is derived operation by operation from what Cairn will send to the store.
| # | Operation | What in Cairn issues it | Semantics required | Value size |
|---|---|---|---|---|
| O1 | state.get |
reads of authoritative records (app passwords, users, groups, admins, shares, holds, settings, space registry); access-deciding caches revalidating | latest committed version; fresh for access decisions | lognormal, median 1 KiB, p99 16 KiB |
| O2 | state.update |
read, apply one change, compare-and-set, retry on conflict (at most 8 tries, jittered backoff) | compare-and-set on version; latency measured over the whole loop | as O1 |
| O3 | state.create |
create-only records (keys, export claims, outage records) | exactly one winner | 256 B - 1 KiB |
| O4 | session.get |
every browser request | read by token hash, with expiry | 300 B |
| O5 | session.create / extend / revoke |
sign-in, "Stay signed in" extension, revocation | create with expiry; extension by compare-and-set (two tabs extending at once add one step, not two); delete | 300 B |
| O6 | lease.acquire / renew / release |
singleton background jobs; per-path ordering of saves | create-only with expiry; renew keeping the token; token that only grows | 200 B |
| O7 | state.fencedUpdate |
a job holding a lease writes state (exports, restores, removals) | accepted only if the lease token is still current and the record version matches, atomically | 1 KiB |
| O8 | lock.acquire / check / refresh / release |
WebDAV, manual and office-editor locks | create-only with expiry; fresh read on check; compare-and-set refresh | 300 B |
| O9 | upload.progress |
resumable uploads across instances: one record per upload, one update per confirmed piece | compare-and-set; the losing writer stops | 2-4 KiB |
| O10 | journal.append |
change journal | totally ordered, no gaps as seen by readers, durable | 400 B |
| O11 | journal.read(after, limit) |
change-feed polls by sync clients and WebDAV | entries after a cursor, never skipping one that becomes visible later | n x 400 B |
| O12 | notify.publish / subscribe |
change nudges to browsers, cache invalidation | delivered to every other instance; best effort | 200 B |
| O13 | throttle.incr |
sign-in throttle shared by all instances | compare-and-set counter on a hot key, with expiry | 100 B |
Values are random bytes. Cairn stores encrypted records, so compressible test data would flatter both stores.
Per-user rates (modelled)
An "active user" is one signed in during the busy hour. Each rate is an assumption:
| Assumption | Value | |
|---|---|---|
| A1 | users with the web app open | 60% of active users |
| A2 | authenticated API calls per open browser | 10 per minute |
| A3 | sync devices | 1.5 per active user, each polling the change feed every 30 s |
| A4 | file changes | 30 per active user per busy hour |
| A5 | store operations per file change | 1 lock check, 1 lease acquire and 1 release, 1 journal append, 1 notification, 0.5 record updates |
| A6 | large uploads | 1 per user per 4 hours, 100 MiB on average, 8 MiB pieces: about 15 store writes each |
| A7 | sign-ins | 1 per user per 8 hours; storm: 50% of users within 15 minutes; about 5 store operations each |
| A8 | document editing | 5% of active users, one lock operation per 5 minutes |
| A9 | background, per cluster | 12 singleton leases renewed every 20 s; 3 instances revalidating 10 hot records every 10 s |
The 30-second poll in A3 is the Nextcloud desktop client's default (remotePollInterval).
Resulting mix per active user, in operations per second: session.get 0.100, state.get 0.050, journal.read 0.050, lease operations 0.017, lock.check 0.008, journal.append 0.008, notify 0.008, state.update 0.004, upload.progress 0.001, sign-in operations 0.0002, lock writes 0.0002. In total about 0.25 operations per active user per second, 84% reads.
Calibration before the runs. Cairn exports a request counter per route (cairn_http_requests_total). Fourteen days of that counter from our own Cairn deployments, divided by the number of signed-in users, give measured values for A2 and A3. The workload profile is adjusted to them before the first measured run, and the change is recorded in the changelog below. Our own deployments have few users, so the rates remain modelled, and the results will say so.
Levels
| Level | Users | Operations/s | Writes/s | Purpose |
|---|---|---|---|---|
| L0 | 0 | 3.6 (background only) | 0.6 | idle footprint |
| L200 | 200 | about 53 | about 8 | small office |
| L2000 | 2,000 | about 500 | about 80 | large site |
| L2000-storm | 2,000 plus sign-in storm | about 505 | about 86 | 15-minute storm on top of L2000 |
| S-ramp | steps of x1, x2, x4, x8, x16 of L2000, 3 min each | 500 to 8,000 | 80 to 1,280 | saturation point; a step ends the ramp when p99 exceeds 250 ms or errors exceed 0.1% |
| S-hot | 50 concurrent state.update on one key, then on 10 keys |
as fast as possible | all | conflict rate, retries per success, end-to-end update latency |
| S-journal | L2000 plus 2,500 journal reads/s (clients polling every 5 s) | about 3,000 | as L2000 | cost of the change feed |
Key popularity follows a Zipf distribution (s = 1.1) over the preloaded keys, plus the explicit hot keys. Preloaded before every level: 50,000 records, 1,200 live and 10,000 expired sessions, 1,000,000 journal entries, 100 locks and 50 leases.
Faults
Correctness workload
During every fault trial a separate correctness workload runs at L2000 write intensity on 1,000 dedicated keys, apart from the performance keys:
- Each key holds a list of operation ids that only grows. A write is a compare-and-set that appends its own unique id. After healing, final reads return every list.
- Every operation is logged as invoked, then as succeeded, failed or unknown, with a monotonic timestamp and the instance it ran on.
- Journal appends carry unique ids.
- For NATS, plain register keys are also checked for linearizability twice: once with the default reads (stale reads are expected, per ADR-8) and once with writes and compare-and-set only.
Fault types
10 trials per fault and configuration, at randomised times. The last column states what published sources lead us to expect; the trials test it.
| Fault | How it is applied | Applies to | Expected from sources | |
|---|---|---|---|---|
| F1 | graceful restart of one instance (rolling update) | SIGTERM, then restart | all | no loss |
| F2 | process crash of one instance, or of the PostgreSQL server | kill -9; the operating system keeps its page cache |
all; for P1-2 one Cairn instance only | no loss: Jepsen saw none with process crashes; PostgreSQL flushes WAL at commit (PostgreSQL 18: asynchronous commit) |
| F3 | power cut of one VM | hypervisor hard stop of one store VM, which discards its page cache; for P1 and P2 the primary; for P1-2 one Cairn instance VM | all; for P1-2 one Cairn instance VM only | P1: no loss, unavailable until restart. N-def: a single power cut plus pauses can lose writes and cause split-brain (nats-server #7567). N-sync: no loss expected |
| F4 | power cut of all store VMs | all within 2 seconds | all except P1-2 | N-def: loss of acknowledged writes up to the flush interval (Jepsen lost 131,418 of 930,005 in one run). N-sync, P1, P2: no loss expected |
| F5 | loss of unsynced data on one node | LazyFS under the store's data directory: drop unsynced data, kill, with pauses before and after (Jepsen's pattern) | all except P1-2 | N-def: loss, as in Jepsen. N-sync, P1, P2: no loss expected |
| F6 | network partition | firewall drop between one NATS node and the others; between one Cairn instance and PostgreSQL | all | NATS: the minority is unavailable, the majority continues. PostgreSQL: the isolated instance fails its operations. No loss either way |
| F7 | lease holder paused past expiry | pause the holder for twice the lease TTL, let another instance take the lease, resume the old holder, which then tries a fenced write | all | the old holder's write is refused |
| F8 | standby power cut; primary power cut then manual promotion | hard stop of the standby; hard stop of the primary, then pg_promote() on the standby |
P2 | standby loss blocks commits until it returns ("Such transaction commits may never be completed if any one of the synchronous standbys should crash", PostgreSQL 18); promotion loses no acknowledged write |
P1-2 (PostgreSQL with 2 Cairn instances) runs only the client-side faults:
- F1: graceful restart of one Cairn instance;
- F2 on one Cairn instance (
kill -9); - F3 on one Cairn instance VM: the lease holder dies without releasing its lease, which must expire and be taken over by the other instance;
- F6: partition between one Cairn instance and PostgreSQL; the other instance must continue and take over the leases;
- F7: lease holder paused past expiry, then a fenced write.
The store-side faults (F2 on the PostgreSQL server, F3 and F4 on the PostgreSQL VM, F5 under the PostgreSQL data directory) are covered by the single-PostgreSQL configuration P1: P1-2 uses the same server with the same settings, so those outcomes do not depend on the number of Cairn instances. The fault time goes instead to the lease behaviour with exactly two instances, where the loss or isolation of one leaves a single candidate to take over. A store-side effect that only appears with two clients would not be seen. F8 applies to P2 only.
The null baseline is not fault-tested; it holds nothing durable.
How the history is checked
Histories are written as JSON lines and checked with elle-cli:
- the grow-only lists with Elle's
list-appendmodel (single-key transactions). A missing id is a lost write; two successful writes from the same predecessor are a stale compare-and-set win; - journal appends as a Jepsen
set(every acknowledged entry present after recovery), plus a gap check: no reader ever sees sequence n+k and later sees n appear; - NATS register keys with Knossos
cas-register; - fencing with a custom checker over the lease log and the guarded record's own "last token" field.
Results are reported per cell as "violations in k of 10 trials". Zero in 10 is reported as "none observed": with 10 trials, a per-trial probability of up to about 26% is still consistent with that result (95% upper bound). As Jepsen notes, tests can show that bugs exist, not that they are absent.
Metrics and how they are measured
Latency and throughput
- Load is open loop: operations arrive as a Poisson process at the level's rate, split evenly across the instances. Each operation has a planned send time, and latency is measured from that planned time, not from the moment a worker became free. A closed-loop generator waits for slow responses and so stops measuring exactly when latency is high ("coordinated omission", wrk2).
- Per operation type: p50, p95, p99, p99.9 and maximum, recorded in HdrHistogram interval logs every 10 seconds, both corrected and uncorrected. Histograms are merged within a run, never across runs.
- Achieved against offered throughput; error and timeout rate; compare-and-set conflict rate and retries per successful
state.update. - If an instance's backlog of due operations keeps growing, the step is marked overloaded and ended instead of queueing silently.
- Notification latency is measured by echo: instance A publishes, every other instance echoes, and A records half the round trip, which avoids clock differences between VMs.
Stale reads
For every configuration: the share of reads that start after another instance's write was acknowledged and still return an older version (stale-read rate), and how long after the acknowledgement such reads still occur (staleness window).
Resources
- CPU (cores) and resident memory per process; Go runtime heap, GC pauses and goroutines for every benchmark process.
- Store cost inside a Cairn instance = the process with the store driver minus the same process with the null driver at the same offered load. For embedded NATS, a second view comes from CPU profiles: the load generator's goroutines carry profiler labels and the NATS server's do not.
- PostgreSQL: server CPU and memory on its VM, connections in use, and its statistics views (database, WAL and I/O); the client-side cost of pgx in the Cairn process by the same null-driver subtraction.
- Disk: bytes written and write operations per second per VM, bytes written per logical write, fsync calls per second per process (counted with bpftrace in a separate calibration run, so tracing never affects measured runs), and growth of the store's data on disk.
- Network: bytes per second per VM and per logical operation.
- Saturation of the shared physical disk, so a result limited by that one disk is labelled as such.
Idle footprint and availability
- Idle footprint at L0 over 30 minutes: CPU, memory, disk writes per minute and network per minute, per instance and in total (cluster heartbeats for NATS; WAL and checkpoints for PostgreSQL).
- Cold start to the first successful operation; rejoin and catch-up time of a restarted node; standby catch-up for P2.
- Write-unavailability window per fault: time from the fault to the first successful write on each surviving instance, and to the last failed write after healing.
Environment
Main profile (published numbers)
- One x86-64 server with a local NVMe SSD, running KVM virtual machines.
- Each Cairn instance and each PostgreSQL server is its own VM of the same size: 4 vCPU, 4 GB RAM, 32 GB virtual disk on the local NVMe. With NATS, the Cairn instance VMs are also the NATS nodes. A smaller VM (2 vCPU, 2 GB) runs the controller and the checkers.
- Debian 13 on every VM; clocks synchronised with chrony; a host firewall allowing only the benchmark's ports between benchmark VMs.
- TLS on all store traffic: NATS routes, PostgreSQL client connections and replication.
- Metrics scraped by Prometheus every 5 seconds.
- Recorded with the results: CPU model, disk model, measured fsync latency (fio and
pg_test_fsync), network round trip, kernel and hypervisor versions, and the VM disk cache mode.
Why this shape. A customer site that runs 2-3 Cairn instances typically has x86 servers or VMs with 2-4 vCPU and 4-8 GB each, at one site, on a LAN with well under 1 ms round trip, with the store's data on local SSD or network block storage, and PostgreSQL on its own VM or as a managed service. The main profile follows that. VMs are used rather than containers because only a VM stop discards that VM's page cache like a power cut does; a stopped container shares the host's page cache and cannot lose unsynced writes, which would hide exactly the failures F3 and F4 look for.
Fairness. Every store node gets the same vCPU, RAM and disk. Both per-instance cost and total system cost are reported, since NATS runs inside 3 instances and PostgreSQL adds 1 or 2 servers.
Where the setup differs from a customer site.
- All VMs share one physical NVMe, so three NATS nodes' fsyncs land on one device that three real servers would not share. Disk saturation is recorded and affected results are labelled.
- The virtual network between VMs on one host is faster than a real LAN. A sensitivity run at L2000 adds 0.1 ms one-way delay on every VM with
tc netem(about 0.2 ms extra round trip). - Runs are scheduled when the host has no other work; a run that overlaps another job on the host is flagged and repeated.
Small-node profile (limited, labelled)
A limited run on a Kubernetes cluster of Raspberry Pi arm64 nodes (4 cores, 8 GB) answers one narrower question: what embedded NATS costs inside a Cairn pod with the current default pod limits (0.5 CPU, 256 MiB), and with more headroom (2 CPU, 1 GiB). It runs L0 and L200 and faults F2 and F3, 3 repetitions each. These nodes keep persistent volumes on NFS, which NATS advises against for JetStream storage, and the database shares a node with a Cairn stand-in. Its results are labelled "homelab, not representative" and are not used for the decision except to answer that question.
Procedure
- Harness self-tests, before any measured run. All drivers, including the null driver, must pass one shared conformance suite: exactly one winner among 50 concurrent creators; compare-and-set conflicts; expiry, and renewal that keeps the expiry; fencing (a paused holder is refused); the journal gap check with 20 concurrent appenders; notification fan-out. Then checker validation: deliberately broken drivers (a compare-and-set that ignores the version, a journal on a plain sequence, a NATS lease renewed with the TTL-resetting update) must make the checkers fail. A checker that never fails proves nothing.
- Calibration. fio sync writes,
pg_test_fsync,nats bench kv, a pgbench custom script, null-driver baselines and fsync counts. If the harness's numbers are far fromnats bench kvor pgbench on the same VMs, the harness is investigated before any result is used. - Performance runs. Every configuration runs with 3 Cairn instance VMs, except P1-2, which runs with 2. Each cell (configuration x level) is reset to the preloaded snapshot, then runs 5 minutes of warm-up at the cell's load, 20 minutes of measurement and 1 minute of drain. Ramp steps are 3 minutes each; the idle cell measures 30 minutes. 5 repetitions per cell; 3 for idle, ramp and hot-key cells.
- Fault runs. 10 trials per applicable fault and configuration, with the correctness workload running. P1-2 runs only the client-side faults (see "Fault types").
- Order. Cells are run in a randomised order, interleaved across configurations, so slow drift (temperature, host load, SSD state) does not line up with one store. Seeds are recorded.
- Variance. Per cell: the median of the run medians with minimum and maximum; for p99, a bootstrap 95% confidence interval over runs. When a difference counts as measurable is defined in steps 3 and 4 of the decision criteria; any other difference is reported as "no measurable difference".
The full matrix takes about 52 hours of machine time (36 hours of performance runs and 16 hours of fault runs), plus about 8 hours for the small-node profile. The fault time is lower than an earlier estimate because P1-2 runs only the client-side faults.
Raw data kept per run: HdrHistogram interval logs, operation histories (fault runs), a metrics snapshot of the run window, store statistics before and after (NATS server and JetStream reports, PostgreSQL statistics views), kernel and store logs, and the exact configuration and binary version.
What this benchmark does not show
- Faults not covered: bit flips and truncation of data files (Jepsen found NATS can lose or delete a stream when files on a minority of nodes are corrupted: #7549, #7556), a full disk, clock steps between nodes, adding or removing cluster members, and automatic PostgreSQL failover (Patroni, CloudNativePG, managed high-availability services). P-HA counts as passing step 2 by design, not tested by us; see step 2 of the decision criteria. The nats-server 2.15.0 release notes do not list #7549, #7556 or #7567 as fixed.
- A hard stop is not a physical power cut. It discards the VM's page cache but does not cut power to the physical drive, so drive write caches are not exercised.
- One host, one disk, one virtual network. Results limited by the shared NVMe are labelled; the network effect is covered only by the one netem run.
- No network block storage, no NFS for store data in the main profile, and no managed PostgreSQL service.
- The rates are modelled. They are calibrated against our own small deployments, not measured at customer sites.
- Our own harness and drivers. A different driver design could do better or worse on either store. "Failure to optimise" is a known benchmarking pitfall (Raasveldt et al., "Fair Benchmarking Considered Difficult", DBTest 2018); every setting is published so it can be checked.
- Versions move. Results hold for nats-server 2.15.1 and PostgreSQL 18. NATS 2.15.0 already changed sync behaviour once.
- No observed loss is not proof of none. Ten clean trials still allow a per-trial failure probability of up to about 26%.
- Not a general store comparison. The workload is Cairn's: small records, low rates, mostly reads. It says nothing about either store for other workloads.
Reproducing it
The harness is one Go program, coordbench, in the Cairn repository under bench/coordbench, as its own Go module so its dependencies do not enter Cairn's. It has these subcommands:
node: one per Cairn instance; hosts the driver, the load generator for its share of users, metrics and a control API;control: orchestrates runs, sets rates, collects histograms and histories, applies faults;check: runs the custom checks and calls elle-cli;report: builds tables and charts from the raw data;calibrate: fio,pg_test_fsync,nats bench kv, the null-driver baseline.
The harness, the workload profile files, the seeds and the raw data (HdrHistogram logs, operation histories, metrics snapshots) will be published with the results, with the repository tag the runs used.
Sources
- Jepsen: NATS 2.12.1
- NATS docs: sync_interval, sizing and resources
- NATS ADR-8: JetStream key-value, ADR-50: atomic batch publish
- nats-server releases (v2.15.0, v2.15.1), issues #7549, #7556, #7567
- PostgreSQL 18 docs: asynchronous commit, synchronous replication, sequence functions, pgbench good practices
- wrk2 (coordinated omission), elle-cli, LazyFS
- Kleppmann: How to do distributed locking
- Raasveldt et al.: Fair Benchmarking Considered Difficult
Changelog of this page
- 2026-11-05: method and decision criteria approved.
- 2026-11-06: decision criteria made precise and the page published, before any measured run. Changes, with reasons:
- Availability is now its own step 2, a pass rule, instead of part of cost: running several instances is about surviving the loss of one node, which is a different kind of thing from CPU and memory, and a pass rule keeps a few seconds of difference from choosing the store. The page now says plainly that PostgreSQL with automatic failover was not tested.
- Cost (step 3) keeps operator burden first. A resource difference now decides only when the ranges of the run medians do not overlap and, for CPU and memory, the difference is at least 5% of the pod limit, so that run noise or a tiny difference cannot choose the store. Embedded NATS is measured against the current Cairn pod defaults (0.5 CPU, 256 MiB); exceeding them is a cost, not a disqualification.
- Latency (step 4) now lists the operations that count and decides only when the confidence intervals sit on opposite sides of 50 ms, so run noise cannot decide.
- A tie-break was added: PostgreSQL, because the faults this benchmark cannot cover have open Jepsen reports against NATS, and PostgreSQL has standard recovery tooling.
- P1-2 (PostgreSQL with 2 Cairn instances) runs only the client-side faults F1, F2 and F3 on a Cairn instance, F6 and F7; the store-side faults are covered by P1, which uses the same server and settings.
- Correctness (step 1) is unchanged in substance; the fencing check is named "reused or outdated fencing tokens" to match what it already checked.
- 2026-11-06: node count and availability rule settled, before any measured run. Changes, with reasons:
- The main comparison runs with 3 Cairn instances, within the 2-3 instance scope. Embedded NATS needs 3 nodes to keep accepting writes after one fails (Raft quorum 2 of 3), so a deployment with 2 Cairn instances and embedded NATS needs a third, NATS-only node, which now counts as an extra server under operator burden (step 3). P1-2 (PostgreSQL with 2 Cairn instances) stays as a configuration, with client-side faults only.
- Step 2 now states that a single PostgreSQL server fails it as tested, and that a highly available PostgreSQL provided by the customer (P-HA: managed service, Patroni, CloudNativePG) counts as passing, labelled "not tested by us". The results table shows P1 as failing and P-HA as "passes by design, untested". Reason: a site that needs the store to stay writable after losing one node would run PostgreSQL with automatic failover, so step 2 judges that setup, while stating plainly that we did not test it.
- The rule for the case where no configuration passes step 2 is stated on its own: step 2 does not decide, and step 3 compares every configuration that passed step 1.
- Step 4 states explicitly that the L2000-storm cell is excluded from the latency decision.
- The run-time estimate is about 52 hours instead of about 56, because P1-2 runs fewer faults.
- 2026-11-12: fault campaign results added at the top of the page. Deviations from the published method, made after runs had started, with reasons:
- The history checker was changed after the first campaign, because its verdicts were wrong, not because they were unwelcome: elle now gates on serializable and reports strict-serializable as information (strict depends on the wall clocks of different VMs); fencing gates on store order and reports the cross-host wall-clock comparison as information. Both directions were kept honest: the 10 real N-def F4 failures still fail, and every recheck ran on copies of the original histories.
- The F5 cells hit by the LazyFS file-descriptor bug were rerun with the fixed harness; the invalid first attempts are kept as evidence.
- Faults on PostgreSQL were split by target (F1/F2 on the server, F1i/F2i on a Cairn instance, F7s pause of the server, F8a standby power cut), so each target got 10 trials. F7 paused the holder for 6 s, three times the 2 s lease TTL.
- P1-2 and F8b were not run.
- Each fault cell ran 1 minute of warm-up and 5 minutes of load, not the performance runs' timing.