v0.6.409: several instances (preview), #742 Step 1 batch A (NATS embedded/external), #1011 Wave B #1181

Merged
Cordy merged 143 commits from b9v07 into main 2026-11-18 01:20:43 +00:00
Owner

Several instances, preview (#742 Step 1 batch A, #1011 Wave B)

Cairn can run as 3 or 5 instances that share one storage. A single instance (local mode) is unchanged apart from what CHANGELOG "Changed, read this before upgrading" lists.

Coordination

  • Embedded NATS JetStream: R3/R5, fsync on every write, and per-member at-rest keys.
  • An external NATS cluster as the other option.
  • Every per-deploy key has a CAIRN_COORD_* variable.

State

  • Sealed state values in NATS (natsstate), leases and records (natscoord), and a fail-fast gate.
  • Answers during an outage: 503 coord-unavailable and cluster-maintenance. Reads keep working, and /readyz stays green.

Cluster lifecycle

  • Cluster record, membership, version gates, and the legacy fence against old binaries.
  • Offline cairnd migrate --to nats / --to local, with the U4 rule.
  • Backups and restore, including into new instances.
  • cluster rescue, cluster certs [--ca], and cluster status, all over the admin socket.

Key rotation

  • The cluster decides which key it seals with: one rolling restart rotates the recovery identity.
  • A start guard, a re-seal of values and stored objects (retired keys included), and cairnd cluster reseal.
  • Ledger option (b+): records only ever add protection.
  • Audit cluster-keys-decided and cluster-resealed.

Observability

  • Overview Instances section and cards, bell kinds, the 21 cluster audit verbs, metrics and the cluster endpoint.

Kubernetes and docs

  • Helm chart 0.8.0: StatefulSet (embedded and external), PDB, anti-affinity, per-pod volumes, and grace of at least 120 s. The data claim is kept on uninstall.
  • Handbook: multi-instance, Kubernetes and hosts pages, runbooks R1-R12, configuration reference, CHANGELOG and ARCHITECTURE.md.

Verification

  • Many review rounds and fix rounds (fx2-fx14). The full suites pass: Go, web 322/322, a11y 92/92, all real-main tests.
  • Three hosts (two VM rounds): every runbook. The rounds found 13 bugs, all fixed and confirmed on the hosts.
  • Power-cut lease validation, strict: 30 trials. No overlap, tokens always increased, and survivors take a lease back within TTL + 30 s.
  • Kubernetes (homelab, scratch namespace): bootstrap, rolling upgrade, pod deletion, node drain, grow 1→3 and shrink 3→1, external NATS. Six findings, fixed or documented.

Follow-ups: #1176, #1177, #1178, #1179 (batch B and later). Known limitations are in docs/handbook/multi-instance.md.

## Several instances, preview (#742 Step 1 batch A, #1011 Wave B) Cairn can run as 3 or 5 instances that share one storage. A single instance (local mode) is unchanged apart from what CHANGELOG "Changed, read this before upgrading" lists. **Coordination** - Embedded NATS JetStream: R3/R5, fsync on every write, and per-member at-rest keys. - An external NATS cluster as the other option. - Every per-deploy key has a `CAIRN_COORD_*` variable. **State** - Sealed state values in NATS (natsstate), leases and records (natscoord), and a fail-fast gate. - Answers during an outage: 503 `coord-unavailable` and `cluster-maintenance`. Reads keep working, and `/readyz` stays green. **Cluster lifecycle** - Cluster record, membership, version gates, and the legacy fence against old binaries. - Offline `cairnd migrate --to nats` / `--to local`, with the U4 rule. - Backups and restore, including into new instances. - `cluster rescue`, `cluster certs [--ca]`, and `cluster status`, all over the admin socket. **Key rotation** - The cluster decides which key it seals with: one rolling restart rotates the recovery identity. - A start guard, a re-seal of values and stored objects (retired keys included), and `cairnd cluster reseal`. - Ledger option (b+): records only ever add protection. - Audit `cluster-keys-decided` and `cluster-resealed`. **Observability** - Overview Instances section and cards, bell kinds, the 21 cluster audit verbs, metrics and the cluster endpoint. **Kubernetes and docs** - Helm chart 0.8.0: StatefulSet (embedded and external), PDB, anti-affinity, per-pod volumes, and grace of at least 120 s. The data claim is kept on uninstall. - Handbook: multi-instance, Kubernetes and hosts pages, runbooks R1-R12, configuration reference, CHANGELOG and ARCHITECTURE.md. ## Verification - Many review rounds and fix rounds (fx2-fx14). The full suites pass: Go, web 322/322, a11y 92/92, all real-main tests. - **Three hosts (two VM rounds):** every runbook. The rounds found 13 bugs, all fixed and confirmed on the hosts. - **Power-cut lease validation, strict:** 30 trials. No overlap, tokens always increased, and survivors take a lease back within TTL + 30 s. - **Kubernetes (homelab, scratch namespace):** bootstrap, rolling upgrade, pod deletion, node drain, grow 1→3 and shrink 3→1, external NATS. Six findings, fixed or documented. Follow-ups: #1176, #1177, #1178, #1179 (batch B and later). Known limitations are in `docs/handbook/multi-instance.md`.
Cordy added 142 commits 2026-11-18 00:25:31 +00:00
go.mod and go.sum from the SDD folder's n1-go.mod.resolved and
n1-go.sum.resolved. Indirect bumps by minimum version selection:
golang.org/x/crypto v0.57.0 -> v0.58.0, golang.org/x/sys v0.48.0 ->
v0.50.0; new: klauspost/compress v1.20.4, minio/highwayhash v1.0.4,
nats-io/jwt/v2 v2.8.3, nats-io/nkeys v0.4.17, nats-io/nuid v1.0.1,
golang.org/x/time v0.16.0, antithesis-sdk-go v0.9.0-default-no-op.
nats-server, nats.go, jwt/v2 and nkeys move to the direct block
(natstest imports them); no version changes. go.sum is byte-identical
to the resolved file.

internal/config: the coordination block (local, embedded, external)
with ResolveCoordination (defaults, CAIRN_COORD_MODE/NAME/PEERS and the
*_FILE variables for the secret files, every spec S2.2 rule incl. the
cross-field rules and the 100-byte admin socket path), checked at Load.
The configuration reference gets the keys and variables (the drift
tests require them; the docs task expands the section).

internal/coord: ErrUnavailable (spec S6.6).

internal/natsserver/natstest: 1, 3 and 5-node in-process JetStream
clusters, routes through per-pair loopback proxies (Isolate cuts a node
off), Stop/Restart, StreamLeader, MetaLeader, route and client TLS,
operator mode with a .creds file.
ConnectExternal: tls:// only (an unexported test variable lifts it),
no inline URL credentials, exactly one credential form (creds, nkey
seed, user and password), reconnect for ever, first connect bounded by
ctx and callTimeout.

Preflight (spec S4.2): jetstream, servers, test-bucket (<prefix>_probe,
R3 or R1 with allowSingleReplica, always deleted), bucket-replicas (one
check per Cairn bucket), version (>= 2.15.0), max-payload (>= 8 MiB),
tls, credentials, sync-always (unchecked, WARN). Each refusal is logged
at ERROR with the mockup text and named in the error, which wraps
ErrPreflight. Recheck (spec S4.3) runs the same list read-only without
the probe, never refuses and logs nothing; a test watches the account's
stream create/update/delete API subjects (with a positive control).

The wire test seals a marker with an age stub through a recording
proxy; n5 extends it to the real store.
internal/coord/natscoord: records in <prefix>_coord (history 1, file,
Replicas 1/3/5, MaxValue + 64 KiB, markers 1 min, direct allowed); Get
and every CAS check read from the stream leader (STREAM.MSG.GET); Create
and Update are raw publishes with the expected subject sequence, Update
carrying the stored Nats-TTL; Delete purges with a 1 min marker TTL;
coord.NewLeases on the records; Notify on <prefix>.n.<topic>; member
records instance.<KeyPart> with a 7-day TTL renewed per heartbeat; Log
on stream <prefix>_log_<name>. Every call under min(caller,
CallTimeout) (listings ReadTimeout); timeouts, no responders, closed
connections, 10008 and 10005 wrap coord.ErrUnavailable.

10164 (wrong last sequence, constant) is retried within the deadline,
not mapped to ErrConflict: nats-server 2.15.3 acks a write before it
clears the subject's in-process mark (applyStreamMsgOp), so a CAS right
after an ack can meet it (Leases/Exclusive50 on 3 nodes: Release
"version changed"). The reference driver mapped it to a conflict.

Mutants (must fail):
go test -tags natscoord_mutant_kvupdate -run TTLSurvivesUpdate742a
--- FAIL: TestTTLSurvivesUpdate742a (0.51s)
    natscoord742a_test.go:88: after update 1 the record's Nats-TTL is 0s, want 2s
FAIL	github.com/nikolasomborski/cairn/internal/coord/natscoord	0.513s
go test -tags natscoord_mutant_shortttl -run InstancesLiveByHeartbeatAge742a
--- FAIL: TestInstancesLiveByHeartbeatAge742a (0.71s)
    natscoord742a_test.go:432: member record Nats-TTL 1s, want 7 days
FAIL	github.com/nikolasomborski/cairn/internal/coord/natscoord	0.714s
StartEmbedded refuses before the server opens the store: peers other
than 3 or 5, a store directory on a network or memory file system,
under the temporary directory, on storage.root's file system,
world-writable, foreign-owned or short of space, a member directory
held by another process, a member file of another name or cluster, a
member file whose at-rest key fingerprint is neither the current nor
the old key's (rotation guard), and wrapped store keys that neither key
opens (probe, so the server never deletes a block it cannot decrypt).

nats-server v2.15.3 opens a Raft log's key file with the current key
only, so a rotation restart with JetStreamOldKey failed with "Can't
start JetStream"; StartEmbedded re-wraps those key files (same seed,
nonce and cipher) before the server starts; streams and blocks convert
in the server.
internal/statestore/natsstate: <prefix>_state KV bucket (history 1, file,
R replicas, MaxValueSize 8 MiB + 64 KiB, marker TTL 1 min); values sealed
with age to the state recipients as "cairn-state/1\n<name>\n<plaintext>",
the name checked on every read (ErrMisplaced wraps ErrUndecryptable);
versions are decimal revisions; LoadV, CAS and probes are leader reads
(STREAM.MSG.GET); Load is a direct read only within FreshFor of the last
leader contact and at or above the watch's floor; SaveIf reads first unless
this store opened the expected revision, so a value it cannot open is never
overwritten; writes refused while the connection is down (no reconnect
buffer replay); 8 MiB refused before publish (ERROR line), WARN once a day
per document at 4 MiB, OnSize for the gauge; ordered metadata watch with
floors, MarkChanged for bound documents, m.cluster to onMeta, invalidate
everything on each (re)start; the re-seal pass and CountRetiredOnly.
internal/coord: Gate (fail-fast, 10 s without leader contact plus a failed
call). internal/statestore: ErrTooLarge, PathFor, MarkChanged,
MarkAllChanged, Due. internal/storage: ErrCoordUnavailable alias.

Mutants (must fail, and do):
go test -tags natsstate_mutant_alwaysdirect -run 'IsolatedReplicaReadIsNotGood742a|DirectReadBelowFloorGoesToLeader742a'
  --- FAIL: TestIsolatedReplicaReadIsNotGood742a: the cut-off member's Load succeeded with "v1" (stale read counted as good)
  --- FAIL: TestDirectReadBelowFloorGoesToLeader742a: Load below the floor returned "v1" from the held-back replica
go test -tags natsstate_mutant_nogate -run GateRefusesAtOnceWhileDown742a
  --- FAIL: TestGateRefusesAtOnceWhileDown742a: Load took 503ms while the gate is shut, want at once (and LoadV, List, ...)
- MaxPayload = 8 MiB + 64 KiB: an 8 MiB plaintext plus its sealing (~2.3 KiB)
  was refused by the server; TestExactMaxValueSavesOnEmbedded742a saves an
  exactly 8 MiB document through natsstate on three embedded members.
- natstest.DefaultMaxPayload equals MaxPayload (checked from natsserver).
- max-payload refusal: "the NATS server allows values up to <n>; Cairn needs
  at least 8 MiB + 64 KiB (set max_payload to 9MB)" (strings round).
- Embedded Conn/SysConn and ConnectExternal: nats.ReconnectBufSize(-1).
- TestExternalWireCarriesNoPlaintext742a: real natsstate store (documents,
  records, leader and direct reads, a listing) instead of the sealing stub.
- TestHealthInProcess742a (review-w1a S3): KV handle on the meta leader's
  connection (the stopped follower is another node), bucket create under the
  30 s formation deadline, waits for the store statistics; 25 runs green.
- review-w1a S1: a cut-off stream leader answers until the server steps it
  down (10 to 20 s). LeaderProbe is now STREAM.INFO from the stream leader
  with a majority heard from within 3 s (quorumSeen); a committed write is a
  proof too; a failed probe drops it. Leader reads and listings prove first
  when the last proof is 10 s (proofFor) old; leader answers count as contact
  (and open the gate) only while the proof is younger than 10 s; direct reads
  need a proof within FreshFor. TestIsolatedLeaderReadIsNotGood742a: past
  13 s + 1 s after the cut neither Load nor LoadV returns the old value and
  LeaderProbe fails (red before: Load returned "v1" 14 s after the cut).
- review-w1a S2: listKeys reads kv.WatchFiltered itself; updates that end
  before the end marker are coord.ErrUnavailable, never a short list
  (TestListFailsWhenConnectionCloses742a; red before: 54 of 3000 names, no
  error).
- 10164 (subject busy) retried in publish and purge until the call deadline,
  never ErrConflict (TestSubjectBusyIsNotConflict742a,
  TestBackToBackSaveIfR3742a). TestExactMaxValueSaves742a.
- Existing tests changed: TestDirectReadNeedsRecentLeaderContact742a (a plain
  leader read no longer reopens direct reads; a probe or committed write
  does), TestGateProbeReopens742a (the probe is counted by the proofs hook).
- Mutants still fail: natsstate_mutant_alwaysdirect
  (TestIsolatedReplicaReadIsNotGood742a: stale read counted as good),
  natsstate_mutant_nogate (TestGateRefusesAtOnceWhileDown742a).
- review-w1a S2: records.List lists through kv.WatchFiltered; updates that
  end before the end marker are coord.ErrUnavailable (errListCut), never a
  short list; keys sorted and compacted (TestListFailsWhenConnectionCloses742a;
  red before: 33 of 3000 keys, no error).
- Create, Update, Delete, Log.Append and Notify.Publish refuse at once with
  coord.ErrUnavailable while the connection is not up (nats.go would buffer
  and send later); ErrReconnectBufExceeded maps to ErrUnavailable.
  TestWritesRefusedWhileDisconnected742a (red before: each write waited the
  2 s deadline, and the Create, the Delete and the log append landed after
  the reconnect).
- Mutants still fail: natscoord_mutant_kvupdate, natscoord_mutant_shortttl.
login.html: ERR login-coord-unavailable and cluster-maintenance (drafts,
strings round) right after login-state-unreadable; neither marks a field
nor counts toward the failure hint, both show the help line, on the
sign-in form and the new-password step. The shared ERR block is the same
on setup.html and storage-setup.html (qa843 rule), so they gain both
entries too. guest.html: coord-unavailable (errCoordUnavail, the
mockup's text) and cluster-maintenance (errClusterMaint, draft) after
holds-unavailable. maintenancePage picks en/de/fr/it from
Accept-Language (q-values, primary subtag) with the drafted title,
sentence and retry hint; no script. webdist copies synced.
- local mode (the default) renders exactly chart 0.7.0's workload,
  Service, PVC and NetworkPolicy (goldens from base fc919b56)
- embedded: StatefulSet <fullname>-member, 3 or 5 (others refuse),
  Parallel, RollingUpdate, volumeClaimTemplates cairn-member at
  /var/lib/cairn, CAIRN_COORD_MODE/NAME/PEERS and the route file
  variables, route Secrets projected at /etc/cairn-route, port 6222,
  headless Service (publishNotReadyAddresses), PDB maxUnavailable 1,
  required (or preferred) anti-affinity, 1Gi/512Mi, GOMEMLIMIT 85 percent
- external: the Deployment with coordination.replicas, RollingUpdate
  maxUnavailable 1 maxSurge 0, PDB above one replica, preferred
  anti-affinity, credentials Secret at /etc/cairn-nats
- both: terminationGracePeriodSeconds 120 (refused below: drain overlaps
  the 60 s clean-stop backup, 10 s leave, flushes, 30 s NATS stop),
  liveness /healthz, readiness /readyz, no startup probe; POSIX needs an
  existing ReadWriteMany claim; an in-place switch is refused while the
  release's single-instance data volume exists (helm would delete it)
- NetworkPolicy: 6222 in and out between the release's server pods only
- helmlite: int, until, hasKey, merge; lookup answerable in tests
- ci.yml: helm lint/template of both coordination modes and the 2-replica refusal
Cordy merged commit 0d954d4535 into main 2026-11-18 01:20:43 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1181
No description provided.