State-in-backend: instance state lives encrypted in the storage backend, not on local disk #138

New issue

Closed

opened 2026-08-07 21:01:42 +00:00 by Cordy · 3 comments

Cordy commented

2026-08-07 21:01:42 +00:00

Owner

Decided with Nikola (2026-08-07). This is the v0.6 milestone's core issue — design first, then implementation.

Motivation

Dogfooding v0.4.x surfaced that every runtime state file dies with /data. On the dogfood, /data was an emptyDir, so each redeploy silently reset: license-state.json (the "my license disappears after updates" report), settings.json (incl. peeringEnabled), shares.json (every share link died), holds.json (legal holds silently released by a version bump — the worst item for our compliance story), peering.json (trust state incl. live peer credentials), app-passwords.json, local-users.json. The dogfood is patched with a PVC (homelab-config cairn-enc-data), but every customer who runs the container without persistence has the identical failure mode waiting.

The deep fix: stop keeping instance state on local disk. Precedent is MinIO, which stores its whole configuration and IAM in a hidden .minio.sys prefix inside the object store it serves — the server binary is stateless, and pointing a fresh server at an existing volume resurrects the instance completely.

Design

Each atomic-JSON store reads/writes .cairn-state/<name>.json inside the configured storage backend instead of /data/.cairn/<name>.json:

S3 driver: objects under a reserved .cairn-state/ prefix in the bucket. The prefix must be invisible to and unreachable from all user-facing surfaces (WebDAV, UI listing, shares, peering delivery) — same treatment as other internal paths.
posix/NFS driver: .cairn-state/ under the data root. These deployments already persist the data root, so they get durability for free and keep true rename-based atomicity.
Encrypted at rest, non-negotiable: state is encrypted to the instance's recovery/instance identity via the existing age machinery. peering.json holds live credentials (keys peers issued us) and local-users.json holds verifiers; a bucket compromise must not yield either in the clear.

The config/state line becomes exact: config.json (+ its secret) answers "where is my data, how do I unlock it, who authenticates my users" — backend connection, encryption/recovery identity, IdP. Everything else is state and lives with the data. Corollary: the backend connection itself can never move into runtime settings (the pointer to storage can't live in the storage), which settles the "connect S3 from the settings panel" question — that surface is a read-only card plus, later, a first-run wizard and a supervised migration tool, never a live toggle.

Stays local (genuine scratch): tus upload spool, peering blob staging (peer-blobs). Already memory-only and staying that way: challenges, spent tokens, in-flight transfer records.

Questions the design pass must answer

Atomicity on S3. Current stores do write-temp-then-rename; S3 has no rename. A single whole-object PUT is atomic per object, which suffices for whole-file JSON — but confirm Garage's semantics, and decide whether we need read-after-write verification or a version/etag guard against lost updates.
Boot ordering. Storage must initialize before settings/license/peering load — today's order is roughly the reverse. Map the new dependency graph in main.go, including the failure mode "backend unreachable at boot" (serve degraded? refuse to start? retry loop with health probe implications?).
Write-through caching. State reads happen on every request path today via in-memory structs; keep that. Writes are rare (admin actions) — write-through to the backend, fail the mutating request if the PUT fails (no fire-and-forget for holds or peering).
Migration. On boot: local state present + backend state absent → upload once, then prefer backend; local copy retained read-only as a fallback for one release cycle. Explicitly define the conflict case (both present, differing) — newest-wins is wrong for holds; propose refuse-and-tell-operator.
Two-writer hazard. Single replica + Recreate is the supported topology, but document what happens if two instances share a bucket (they must — peering already assumes distinct buckets per instance? verify) and consider a lightweight instance-lock object with TTL.
License re-activation fold-in. With state in the backend the license survives updates anyway; still add boot re-activation from config/env key (CAIRN_LICENSE_KEY already wins per config.go) for the fresh-instance/DR path — it becomes a small corner of this issue instead of its own feature.
Backup story. After this, backing up the bucket backs up the whole instance. Update docs/handbook/deployment.md accordingly, including the sharp-edge table of what was previously local-only.

Acceptance

Kill the pod, delete its volume entirely, restart against the same bucket + config: license, settings, shares, holds, peering, app passwords, local users all intact.
Full update cycle (tag → deploy) on an instance with no persistent /data loses nothing.
posix-driver instance behaves identically before/after (state just moved under the data root).
Raw state objects in the bucket are ciphertext; sovereignty check (pull object out of Garage, age -d with recovery key) works for state exactly as it does for file data.
User-facing surfaces cannot list, read, share, or deliver into .cairn-state/.

Decided with Nikola (2026-08-07). This is the v0.6 milestone's core issue — design first, then implementation. ## Motivation Dogfooding v0.4.x surfaced that **every runtime state file dies with `/data`**. On the dogfood, `/data` was an `emptyDir`, so each redeploy silently reset: `license-state.json` (the "my license disappears after updates" report), `settings.json` (incl. `peeringEnabled`), `shares.json` (**every share link died**), `holds.json` (**legal holds silently released by a version bump** — the worst item for our compliance story), `peering.json` (trust state incl. live peer credentials), `app-passwords.json`, `local-users.json`. The dogfood is patched with a PVC (homelab-config `cairn-enc-data`), but every customer who runs the container without persistence has the identical failure mode waiting. The deep fix: **stop keeping instance state on local disk.** Precedent is MinIO, which stores its whole configuration and IAM in a hidden `.minio.sys` prefix inside the object store it serves — the server binary is stateless, and pointing a fresh server at an existing volume resurrects the instance completely. ## Design Each atomic-JSON store reads/writes `.cairn-state/<name>.json` **inside the configured storage backend** instead of `/data/.cairn/<name>.json`: - **S3 driver:** objects under a reserved `.cairn-state/` prefix in the bucket. The prefix must be invisible to and unreachable from all user-facing surfaces (WebDAV, UI listing, shares, peering delivery) — same treatment as other internal paths. - **posix/NFS driver:** `.cairn-state/` under the data root. These deployments already persist the data root, so they get durability for free and keep true rename-based atomicity. - **Encrypted at rest, non-negotiable:** state is encrypted to the instance's recovery/instance identity via the existing age machinery. `peering.json` holds live credentials (keys peers issued us) and `local-users.json` holds verifiers; a bucket compromise must not yield either in the clear. **The config/state line becomes exact:** `config.json` (+ its secret) answers "where is my data, how do I unlock it, who authenticates my users" — backend connection, encryption/recovery identity, IdP. Everything else is state and lives with the data. Corollary: the backend connection itself can never move into runtime settings (the pointer to storage can't live in the storage), which settles the "connect S3 from the settings panel" question — that surface is a read-only card plus, later, a first-run wizard and a supervised migration tool, never a live toggle. **Stays local (genuine scratch):** tus upload spool, peering blob staging (`peer-blobs`). Already memory-only and staying that way: challenges, spent tokens, in-flight transfer records. ## Questions the design pass must answer 1. **Atomicity on S3.** Current stores do write-temp-then-rename; S3 has no rename. A single whole-object PUT is atomic per object, which suffices for whole-file JSON — but confirm Garage's semantics, and decide whether we need read-after-write verification or a version/etag guard against lost updates. 2. **Boot ordering.** Storage must initialize before settings/license/peering load — today's order is roughly the reverse. Map the new dependency graph in `main.go`, including the failure mode "backend unreachable at boot" (serve degraded? refuse to start? retry loop with health probe implications?). 3. **Write-through caching.** State reads happen on every request path today via in-memory structs; keep that. Writes are rare (admin actions) — write-through to the backend, fail the mutating request if the PUT fails (no fire-and-forget for holds or peering). 4. **Migration.** On boot: local state present + backend state absent → upload once, then prefer backend; local copy retained read-only as a fallback for one release cycle. Explicitly define the conflict case (both present, differing) — newest-wins is wrong for holds; propose refuse-and-tell-operator. 5. **Two-writer hazard.** Single replica + Recreate is the supported topology, but document what happens if two instances share a bucket (they must — peering already assumes distinct buckets per instance? verify) and consider a lightweight instance-lock object with TTL. 6. **License re-activation fold-in.** With state in the backend the license survives updates anyway; still add boot re-activation from config/env key (`CAIRN_LICENSE_KEY` already wins per config.go) for the fresh-instance/DR path — it becomes a small corner of this issue instead of its own feature. 7. **Backup story.** After this, backing up the bucket backs up the whole instance. Update `docs/handbook/deployment.md` accordingly, including the sharp-edge table of what was previously local-only. ## Acceptance - Kill the pod, delete its volume entirely, restart against the same bucket + config: license, settings, shares, holds, peering, app passwords, local users all intact. - Full update cycle (tag → deploy) on an instance with **no** persistent `/data` loses nothing. - posix-driver instance behaves identically before/after (state just moved under the data root). - Raw state objects in the bucket are ciphertext; sovereignty check (pull object out of Garage, `age -d` with recovery key) works for state exactly as it does for file data. - User-facing surfaces cannot list, read, share, or deliver into `.cairn-state/`.

Cordy added this to the v0.6 — Stateless instance (state-in-backend) milestone

#	Issue	Answers question(s)	Depends on
#152	ADR: statestore abstraction — interface, local + backend impls, encryption envelope, S3 atomicity, the config/state line, what stays local	1, 3, plus the line	— (gates all)
#153	Port the nine stores (notify-seen joined the list since this brief) + boot-order inversion + write-through	2, 3	#152
#154	Disk→backend migration, refuse-on-conflict (holds must never auto-merge)	4	#153
#155	`.cairn-state/` invisible/unreachable from every user surface incl. unscoped admins	(new — implied)	#152
#156	Instance lock — advisory two-writer tripwire, honest about its limits	5	#152
#157	License re-activation at boot from config/env key — independent, can ship first	6	—
#158	Docs: backup story, sharp-edge table, ARCHITECTURE § — write last, from fact (#124 pattern)	7	#159 passing
#159	Acceptance: the delete-the-volume dogfood — closes the milestone	all	#152–#157

Rows
Columns