Instance lock: guard against two instances writing one backend

Cordy commented

2026-08-09 03:17:22 +00:00

Owner

Depends on #152. Single replica + Recreate is the supported topology, but with state in the backend, two instances pointed at one bucket becomes a silent corruption mode instead of an obvious misconfiguration — last-writer-wins on every state object.

Design to settle:

A lock object (.cairn-state/.lock) holding an instance id + heartbeat timestamp, refreshed on an interval; considered stale after N missed beats (survivable across Recreate rollouts, where old and new pod briefly overlap — the TTL must tolerate a normal rollout window).
On boot with a live foreign lock: refuse to start with a message naming the other holder. On stale lock: take over.
Honest limits, written down: S3 has no compare-and-swap on Garage's feature set — this lock is advisory, a tripwire against misconfiguration, not a distributed-consensus primitive. Say so in the code comment and the docs; do not oversell it.
Interaction with #154's migration and with Recreate rollouts explicitly tested.

Alternative to consider and reject (or adopt) in the issue: no lock, docs-only warning. The argument for the lock is that the failure it prevents (two prod instances sharing a bucket after a copy-pasted config) is exactly the kind of quiet disaster the sovereignty audience cannot debug.

Depends on #152. Single replica + `Recreate` is the supported topology, but with state in the backend, two instances pointed at one bucket becomes a *silent* corruption mode instead of an obvious misconfiguration — last-writer-wins on every state object. **Design to settle:** - A lock object (`.cairn-state/.lock`) holding an instance id + heartbeat timestamp, refreshed on an interval; considered stale after N missed beats (survivable across `Recreate` rollouts, where old and new pod briefly overlap — the TTL must tolerate a normal rollout window). - On boot with a live foreign lock: **refuse to start** with a message naming the other holder. On stale lock: take over. - Honest limits, written down: S3 has no compare-and-swap on Garage's feature set — this lock is advisory, a tripwire against misconfiguration, not a distributed-consensus primitive. Say so in the code comment and the docs; do not oversell it. - Interaction with #154's migration and with `Recreate` rollouts explicitly tested. Alternative to consider and reject (or adopt) in the issue: no lock, docs-only warning. The argument for the lock is that the failure it prevents (two prod instances sharing a bucket after a copy-pasted config) is exactly the kind of quiet disaster the sovereignty audience cannot debug.

Cordy added this to the v0.6 — Stateless instance (state-in-backend) milestone

2026-08-09 03:17:22 +00:00

Cordy referenced this issue

2026-08-09 03:17:59 +00:00

v0.6 acceptance: the delete-the-volume dogfood #159

Cordy referenced this issue

2026-08-09 03:18:12 +00:00

State-in-backend: instance state lives encrypted in the storage backend, not on local disk #138

Cordy referenced this issue

2026-08-09 03:41:14 +00:00

v0.6 ADR: state store abstraction — envelope, atomicity, and the config/state line (gates the milestone) #152

Cordy referenced this issue

2026-08-09 03:41:24 +00:00

State-in-backend: instance state lives encrypted in the storage backend, not on local disk #138

Cordy referenced this issue from a commit

2026-08-09 06:34:41 +00:00

feat(#156): advisory instance lock (D8) - heartbeat, stale takeover, graceful release

Cordy referenced this issue from a commit

2026-08-09 06:35:07 +00:00

test(#156): lock - fresh, foreign-live refusal, stale/own/unreadable takeover, release

forgejo-actions referenced this issue from a commit

2026-08-09 06:48:09 +00:00

feat: #155 reserved-prefix guard, #156 instance lock, #160 setup mode + storage card (splices)

Cordy referenced this issue

2026-08-09 06:54:56 +00:00

v0.6 session 3: reserved-prefix guard (#155), instance lock (#156), storage connect + setup mode (#160) #165

Cordy referenced this issue from a commit

2026-08-09 06:55:16 +00:00

Merge pull request 'v0.6 session 3: reserved-prefix guard (#155), instance lock (#156), storage connect + setup mode (#160)' (#165) from feat-s3-guards into main

Cordy commented

2026-08-09 07:02:40 +00:00

Author

Owner

Shipped in v0.5.5 (PR #165), live on the dogfood — the instance now maintains .cairn-state/.lock (encrypted like all state).

Adopted, not docs-only — the alternative was considered and rejected as the issue anticipated: the failure this prevents (two prod instances sharing a bucket after a copy-pasted config) is a quiet disaster the sovereignty audience cannot debug, and a refused boot with a name in the error is a support ticket instead.

Design as ratified (D8) plus one decision made during implementation: the instance ID is the hostname, not a random ID. A crashed container restarts with the same hostname and takes over its own still-fresh lock immediately — a random ID would strand a crash-looping pod for the entire stale window, turning any crash into a 5-minute outage. A second deployment gets a different pod hostname and is refused with holder, PID, heartbeat age, and the stale horizon in the message. Parameters: 60s heartbeat, 5min stale (comfortably above a Recreate rollout; with maxSurge: 0 the old pod Releases on SIGTERM before the new one boots, so the window only matters after SIGKILL). Unreadable lock objects (e.g. after an encryption-identity rotation) are taken over with a warning — an advisory lock must never brick a boot permanently.

Ordering vs #154: acquired before the migration loop, so a second instance can't interleave its migration with a live instance's state. Released in the graceful-shutdown path before "shutdown complete".

Honest limits, in the code comment where the next reader will look: no CAS on Garage's S3, acquisition is read-then-write and racy at the seconds scale — a tripwire against standing misconfiguration, not consensus. Not oversold.

Tests: fresh acquire, live-foreign refusal (names the holder, does NOT clobber the lock), stale takeover, own-lock reacquire (crash-restart path), unreadable takeover, release deletes.

Shipped in **v0.5.5** (PR #165), live on the dogfood — the instance now maintains `.cairn-state/.lock` (encrypted like all state). **Adopted, not docs-only** — the alternative was considered and rejected as the issue anticipated: the failure this prevents (two prod instances sharing a bucket after a copy-pasted config) is a quiet disaster the sovereignty audience cannot debug, and a refused boot with a name in the error is a support ticket instead. Design as ratified (D8) plus one decision made during implementation: **the instance ID is the hostname, not a random ID**. A crashed container restarts with the same hostname and takes over its own still-fresh lock immediately — a random ID would strand a crash-looping pod for the entire stale window, turning any crash into a 5-minute outage. A second deployment gets a different pod hostname and is refused with holder, PID, heartbeat age, and the stale horizon in the message. Parameters: 60s heartbeat, 5min stale (comfortably above a Recreate rollout; with `maxSurge: 0` the old pod Releases on SIGTERM before the new one boots, so the window only matters after SIGKILL). Unreadable lock objects (e.g. after an encryption-identity rotation) are taken over with a warning — an advisory lock must never brick a boot permanently. **Ordering vs #154:** acquired *before* the migration loop, so a second instance can't interleave its migration with a live instance's state. Released in the graceful-shutdown path before "shutdown complete". **Honest limits, in the code comment where the next reader will look:** no CAS on Garage's S3, acquisition is read-then-write and racy at the seconds scale — a tripwire against standing misconfiguration, not consensus. Not oversold. Tests: fresh acquire, live-foreign refusal (names the holder, does NOT clobber the lock), stale takeover, own-lock reacquire (crash-restart path), unreadable takeover, release deletes.

Cordy closed this issue

2026-08-09 07:02:46 +00:00

Rows
Columns

Instance lock: guard against two instances writing one backend #156