Cordy/Cairn

Fork 0

v0.6 ADR: state store abstraction — envelope, atomicity, and the config/state line (gates the milestone) #152

New issue

Closed

opened 2026-08-09 03:16:26 +00:00 by Cordy · 2 comments

Cordy commented

2026-08-09 03:16:26 +00:00

Owner

The #100-equivalent for v0.6: every other issue in this milestone builds on the decisions made here. Deliverable is a decision record (issue comment or IMPLEMENTED-STATE.md, house pattern) plus the skeleton package.

The abstraction. A tiny statestore interface the existing atomic-JSON stores can be ported onto without changing their logic:

type Store interface {
    Load(name string) ([]byte, error)   // ErrNotExist when absent
    Save(name string, data []byte) error // atomic per name
}

Two implementations: localStore (today's write-temp-then-rename under a directory — keeps posix/dev behavior byte-identical) and backendStore (objects under a reserved .cairn-state/ prefix in the configured storage driver, encrypted).

Decisions this ADR must land:

Atomicity on S3. No rename exists; a single whole-object PUT is atomic per object on Garage/S3. Decide whether that suffices (likely yes for whole-file JSON at admin-action frequency) or whether an ETag/If-Match guard is needed against lost updates — and write down why.
Encryption envelope. State objects are ciphertext in the bucket, non-negotiable (peering.json holds live peer credentials, local-users.json verifiers). Encrypt to the recovery/instance identity via the existing age machinery; decide whether the deployment identity or the recovery recipient (or both, like file data) are the recipients, and confirm the sovereignty check (age -d offline) works for state exactly as for files.
The config/state line, stated once. config.json + secret = backend connection, encryption/recovery identity, IdP. Everything else is state. Consequence to record: the storage connection can never become a runtime setting (the pointer cannot live in what it points to) — settles the settings-panel question permanently.
What stays local. Genuine scratch: tus spool, peering blob staging. Memory-only stays memory-only (challenges, spent tokens, in-flight transfers).
Driver selection. posix driver → .cairn-state/ under the data root (naturally persistent, keeps true rename atomicity via localStore semantics); s3 driver → backendStore. One switch at boot, not per store.

Skeleton: package + both implementations + tests (round-trip, absent-name, concurrent saves on localStore; backendStore against the in-memory/posix driver test double). No store ported yet — that is the next issue.

The #100-equivalent for v0.6: every other issue in this milestone builds on the decisions made here. Deliverable is a decision record (issue comment or `IMPLEMENTED-STATE.md`, house pattern) plus the skeleton package. **The abstraction.** A tiny `statestore` interface the existing atomic-JSON stores can be ported onto without changing their logic: ```go type Store interface { Load(name string) ([]byte, error) // ErrNotExist when absent Save(name string, data []byte) error // atomic per name } ``` Two implementations: `localStore` (today's write-temp-then-rename under a directory — keeps posix/dev behavior byte-identical) and `backendStore` (objects under a reserved `.cairn-state/` prefix in the configured storage driver, encrypted). **Decisions this ADR must land:** 1. **Atomicity on S3.** No rename exists; a single whole-object PUT is atomic per object on Garage/S3. Decide whether that suffices (likely yes for whole-file JSON at admin-action frequency) or whether an ETag/If-Match guard is needed against lost updates — and write down why. 2. **Encryption envelope.** State objects are ciphertext in the bucket, non-negotiable (`peering.json` holds live peer credentials, `local-users.json` verifiers). Encrypt to the recovery/instance identity via the existing age machinery; decide whether the deployment identity or the recovery recipient (or both, like file data) are the recipients, and confirm the sovereignty check (`age -d` offline) works for state exactly as for files. 3. **The config/state line, stated once.** `config.json` + secret = backend connection, encryption/recovery identity, IdP. Everything else is state. Consequence to record: the storage connection can never become a runtime setting (the pointer cannot live in what it points to) — settles the settings-panel question permanently. 4. **What stays local.** Genuine scratch: tus spool, peering blob staging. Memory-only stays memory-only (challenges, spent tokens, in-flight transfers). 5. **Driver selection.** posix driver → `.cairn-state/` under the data root (naturally persistent, keeps true rename atomicity via localStore semantics); s3 driver → backendStore. One switch at boot, not per store. Skeleton: package + both implementations + tests (round-trip, absent-name, concurrent saves on localStore; backendStore against the in-memory/posix driver test double). No store ported yet — that is the next issue.

Cordy added this to the v0.6 — Stateless instance (state-in-backend) milestone

2026-08-09 03:16:26 +00:00

Cordy referenced this issue

2026-08-09 03:16:40 +00:00

Port the nine state stores onto statestore + invert boot ordering #153

Cordy referenced this issue

2026-08-09 03:17:09 +00:00

.cairn-state/ must be invisible and unreachable from every user surface #155

Cordy referenced this issue

2026-08-09 03:17:22 +00:00

Instance lock: guard against two instances writing one backend #156

Cordy referenced this issue

2026-08-09 03:18:12 +00:00

State-in-backend: instance state lives encrypted in the storage backend, not on local disk #138

Cordy referenced this issue

2026-08-09 03:40:54 +00:00

Storage connect from the WebUI: bootstrap layer, setup mode, and the wizard #160

Cordy commented

2026-08-09 03:41:14 +00:00

Author

Owner

Decision record — ratified with Nikola 2026-08-09. All design questions across the milestone settled in one pass so implementation can proceed in large sessions. This comment is the ADR; the remaining work on this issue is the skeleton package only.

D1 — Interface: Load(name) ([]byte, error) (ErrNotExist when absent) / Save(name, data []byte) error (atomic per name) / Delete(name) error (needed only for the lock's graceful release). Nothing more; migration and lock compose from these.

D2 — S3 atomicity: single whole-object PUT, no ETag/CAS guard. Writes are admin-action-rate on a single-writer topology with an advisory lock (#156); conditional-write support is not something the design should depend on across S3 implementations (Garage included).

D3 — Encryption envelope: state objects age-encrypted to the recovery recipient always, plus the deployment identity when configured — mirroring file data, so the offline sovereignty drill (age -d) is identical for state. s3 driver with no encryption identities refuses to boot with a one-line fix message: state (peering credentials, auth verifiers) is never plaintext in a bucket, and no silent local fallback that would quietly retain the update-wipes-state failure mode.

D4 — Naming: existing basenames + .age under the reserved prefix (.cairn-state/settings.json.age). Names reveal feature usage; contents reveal nothing.

D5 — Boot order: config → storage driver (+encryption) → statestore → migration (#154) → settings → auth → license → rest. Configured-but-unreachable backend → refuse to start (a file server without its storage is down, not degraded; the restart policy is the retry loop). storage.driver: "setup" → setup mode, not a crash (#160).

D6 — Write-through: synchronous Save; a failed Save fails the mutating request. Holds especially: no fire-and-forget, ever.

D7 — Migration (#154): per-name: local-only → upload + rename .migrated; backend-only → adopt; both identical (byte compare) → adopt + rename; both differing → refuse to start with checksums and the exact operator instruction. Newest-wins rejected: an automatic pick on holds.json could silently release a legal hold.

D8 — Lock (#156): adopted. .cairn-state/.lock, ~60s heartbeat, ~5min stale threshold (tolerates Recreate rollout overlap), Delete on graceful shutdown, refuse boot on a live foreign holder, take over stale locks. Framed honestly in code and docs as advisory — a misconfiguration tripwire, not consensus.

D9 — Reserved prefix (#155): one innermost reserved storage-driver decorator (mirror of scope being outermost); statestore attaches below it. Prefix additionally hidden server-side from unscoped-admin root listings.

D10 — License (#157): boot re-activation from config/env key when state absent; activation failure logs loudly and continues unlicensed — a license-server outage must never down a customer's file server. Air-gapped signed-blob path preserved (#91 owns the policy).

D11 — Storage connect (#160, new in scope): three-layer model — config (operator, read-only) / bootstrap (machine-written local pointer, /data/.cairn/bootstrap.json, plaintext 0600, precedence env > config > bootstrap) / state (with the data). Setup mode behind the explicit storage.driver: "setup" sentinel; wizard applies via probe → persist → graceful restart through the one normal boot path (hot-swap rejected). The line, final form: the pointer to your data lives locally; everything it points to lives with the data.

**Decision record — ratified with Nikola 2026-08-09.** All design questions across the milestone settled in one pass so implementation can proceed in large sessions. This comment is the ADR; the remaining work on this issue is the skeleton package only. **D1 — Interface:** `Load(name) ([]byte, error)` (ErrNotExist when absent) / `Save(name, data []byte) error` (atomic per name) / `Delete(name) error` (needed only for the lock's graceful release). Nothing more; migration and lock compose from these. **D2 — S3 atomicity:** single whole-object PUT, **no ETag/CAS guard**. Writes are admin-action-rate on a single-writer topology with an advisory lock (#156); conditional-write support is not something the design should depend on across S3 implementations (Garage included). **D3 — Encryption envelope:** state objects age-encrypted to the **recovery recipient always, plus the deployment identity when configured** — mirroring file data, so the offline sovereignty drill (`age -d`) is identical for state. **s3 driver with no encryption identities refuses to boot** with a one-line fix message: state (peering credentials, auth verifiers) is never plaintext in a bucket, and no silent local fallback that would quietly retain the update-wipes-state failure mode. **D4 — Naming:** existing basenames + `.age` under the reserved prefix (`.cairn-state/settings.json.age`). Names reveal feature usage; contents reveal nothing. **D5 — Boot order:** config → storage driver (+encryption) → statestore → migration (#154) → settings → auth → license → rest. Configured-but-unreachable backend → **refuse to start** (a file server without its storage is down, not degraded; the restart policy is the retry loop). `storage.driver: "setup"` → **setup mode**, not a crash (#160). **D6 — Write-through:** synchronous `Save`; a failed Save fails the mutating request. Holds especially: no fire-and-forget, ever. **D7 — Migration (#154):** per-name: local-only → upload + rename `.migrated`; backend-only → adopt; both identical (byte compare) → adopt + rename; **both differing → refuse to start** with checksums and the exact operator instruction. Newest-wins rejected: an automatic pick on `holds.json` could silently release a legal hold. **D8 — Lock (#156): adopted.** `.cairn-state/.lock`, ~60s heartbeat, ~5min stale threshold (tolerates Recreate rollout overlap), Delete on graceful shutdown, refuse boot on a live foreign holder, take over stale locks. Framed honestly in code and docs as advisory — a misconfiguration tripwire, not consensus. **D9 — Reserved prefix (#155):** one innermost `reserved` storage-driver decorator (mirror of scope being outermost); statestore attaches below it. Prefix additionally hidden server-side from unscoped-admin root listings. **D10 — License (#157):** boot re-activation from config/env key when state absent; activation failure logs loudly and continues unlicensed — a license-server outage must never down a customer's file server. Air-gapped signed-blob path preserved (#91 owns the policy). **D11 — Storage connect (#160, new in scope):** three-layer model — config (operator, read-only) / **bootstrap** (machine-written local pointer, `/data/.cairn/bootstrap.json`, plaintext 0600, precedence env > config > bootstrap) / state (with the data). Setup mode behind the explicit `storage.driver: "setup"` sentinel; wizard applies via probe → persist → **graceful restart** through the one normal boot path (hot-swap rejected). The line, final form: *the pointer to your data lives locally; everything it points to lives with the data.*

Cordy referenced this issue

2026-08-09 03:41:24 +00:00

State-in-backend: instance state lives encrypted in the storage backend, not on local disk #138

Cordy referenced this issue from a commit

2026-08-09 03:49:29 +00:00

ci: session 1 — EnsureKey boot re-activation (#157) + statestore skeleton (#152)

forgejo-actions referenced this issue from a commit

2026-08-09 03:50:40 +00:00

feat: v0.6 session 1 — license boot re-activation via EnsureKey (#157), statestore skeleton (#152)

Cordy referenced this issue

2026-08-09 03:51:27 +00:00

v0.6 session 1: license boot re-activation (#157) + statestore skeleton (#152) #161

Cordy referenced this issue from a commit

2026-08-09 03:51:45 +00:00

Merge pull request 'v0.6 session 1: license boot re-activation (#157) + statestore skeleton (#152)' (#161) from feat/v06-session1 into main

Cordy commented

2026-08-09 03:58:19 +00:00

Author

Owner

Skeleton shipped in v0.5.1 (PR #161); with the ADR above ratified, this issue is complete.

internal/statestore exactly per D1–D4: Store interface (Load/Save/Delete + ErrNotExist, single-segment name guard against traversal), Local (temp-then-rename under a directory, byte-identical to pre-#152 files — a Local pointed at /data/.cairn reads today's state unchanged, which is #153's zero-migration path for posix), and Backend (age-encrypted whole-object PUTs under /.cairn-state/, objects named <name>.age, refuses to operate without recipients or identities per D3, ensure-once prefix creation, the D2 no-CAS rationale in the code comment where the next reader will look for it).

Tests prove the properties that matter: round-trips on both; the raw backend object is an age file under the reserved prefix containing no plaintext; a foreign identity cannot open it; deletes idempotent; traversal names rejected.

Deliberately wired nowhere yet — #153 owns the port and boot inversion, with one recorded breadcrumb: verify the s3 driver maps its 404 to fs.ErrNotExist before attaching Backend to it.

Skeleton shipped in **v0.5.1** (PR #161); with the ADR above ratified, this issue is complete. `internal/statestore` exactly per D1–D4: `Store` interface (Load/Save/Delete + `ErrNotExist`, single-segment name guard against traversal), **`Local`** (temp-then-rename under a directory, byte-identical to pre-#152 files — a `Local` pointed at `/data/.cairn` reads today's state unchanged, which is #153's zero-migration path for posix), and **`Backend`** (age-encrypted whole-object PUTs under `/.cairn-state/`, objects named `<name>.age`, refuses to operate without recipients or identities per D3, ensure-once prefix creation, the D2 no-CAS rationale in the code comment where the next reader will look for it). Tests prove the properties that matter: round-trips on both; the raw backend object is an age file under the reserved prefix containing no plaintext; a foreign identity cannot open it; deletes idempotent; traversal names rejected. Deliberately wired nowhere yet — #153 owns the port and boot inversion, with one recorded breadcrumb: verify the s3 driver maps its 404 to `fs.ErrNotExist` before attaching `Backend` to it.

Cordy closed this issue

2026-08-09 03:58:24 +00:00

Cordy referenced this issue

2026-08-09 04:45:23 +00:00

Port the nine state stores onto statestore + invert boot ordering #153

Cordy referenced this issue

2026-08-10 00:26:00 +00:00

State-in-backend: instance state lives encrypted in the storage backend, not on local disk #138