State regression: newer stores live on local disk on s3 instances (spaceids, spacemeta, spacequota, tags, favorites, images...) #820

Closed
opened 2026-09-29 17:13:48 +00:00 by Cordy · 1 comment
Owner

Found during the 2026-09-29 docs audit.

The v0.6 promise (ARCHITECTURE §11, #159 acceptance): delete the volume, recreate from Git, and the instance comes back whole from the bucket. On s3, only the paths in cmd/cairnd/main.go's statePaths are bound to the backend (statestore.Bind); every other store falls back to plain local files under /data/.cairn/.

Not in statePaths (local disk on s3 instances):

Store Since Loss on a volume-less redeploy means
spaceids.json #768 Space identities re-minted. Every ID-keyed record below is orphaned.
spacemeta.json #768 Subtitles, descriptions, image refs gone
spacequota.json #802 All space limits silently lifted
spacealerts.json #801 Notification times re-stamped (minor)
tags.json #743 era Everyone's tags gone
favorites.json, recents.json, activity.json #292 etc. Per-user lists and activity gone
spaceimg/, avatars/ (binary dirs) #770, #757 Space images and profile pictures gone

preview-cache/ is legitimately local (a cache).

Why nobody noticed: both dogfoods mount a PVC at /data (cairn-enc-data, cairn-openbao-data), which masks it. ARCHITECTURE §11 even states that recents and favourites are in the backend, and deployment.md says "What still lives locally: genuine scratch only". Both are currently untrue.

Fix direction (design pass first):

  1. Add the JSON stores to statePaths. Existing migration semantics apply (adopt-or-refuse per file).
  2. Images and avatars are binary blobs in a directory: move them behind the statestore (e.g. Load/Save of <id>.jpg names) or into a reserved backend prefix, with a one-time migration of existing files.
  3. A regression guard: a test that walks every filepath.Join(filepath.Dir(cfg.SettingsPath), …) store opened in main and asserts it is either bound or on an explicit "scratch" allow-list, so the next new store can't forget.
  4. Re-run the #159 delete-the-volume acceptance on a dogfood.
  5. Correct ARCHITECTURE §11 and deployment.md in the same change.
Found during the 2026-09-29 docs audit. **The v0.6 promise (ARCHITECTURE §11, #159 acceptance):** delete the volume, recreate from Git, and the instance comes back whole from the bucket. On s3, only the paths in `cmd/cairnd/main.go`'s `statePaths` are bound to the backend (`statestore.Bind`); every other store falls back to plain local files under `/data/.cairn/`. **Not in `statePaths` (local disk on s3 instances):** | Store | Since | Loss on a volume-less redeploy means | |---|---|---| | `spaceids.json` | #768 | **Space identities re-minted.** Every ID-keyed record below is orphaned. | | `spacemeta.json` | #768 | Subtitles, descriptions, image refs gone | | `spacequota.json` | #802 | **All space limits silently lifted** | | `spacealerts.json` | #801 | Notification times re-stamped (minor) | | `tags.json` | #743 era | Everyone's tags gone | | `favorites.json`, `recents.json`, `activity.json` | #292 etc. | Per-user lists and activity gone | | `spaceimg/`, `avatars/` (binary dirs) | #770, #757 | Space images and profile pictures gone | `preview-cache/` is legitimately local (a cache). **Why nobody noticed:** both dogfoods mount a PVC at `/data` (`cairn-enc-data`, `cairn-openbao-data`), which masks it. ARCHITECTURE §11 even states that recents and favourites are in the backend, and `deployment.md` says "What still lives locally: genuine scratch only". Both are currently untrue. **Fix direction (design pass first):** 1. Add the JSON stores to `statePaths`. Existing migration semantics apply (adopt-or-refuse per file). 2. Images and avatars are binary blobs in a directory: move them behind the statestore (e.g. `Load`/`Save` of `<id>.jpg` names) or into a reserved backend prefix, with a one-time migration of existing files. 3. A regression guard: a test that walks every `filepath.Join(filepath.Dir(cfg.SettingsPath), …)` store opened in main and asserts it is either bound or on an explicit "scratch" allow-list, so the next new store can't forget. 4. Re-run the #159 delete-the-volume acceptance on a dogfood. 5. Correct ARCHITECTURE §11 and `deployment.md` in the same change.
Cordy closed this issue 2026-09-29 18:06:06 +00:00
Author
Owner

Shipped in v0.6.310 (PR #822) and proven on the Bao dogfood, 2026-09-29.

Rollout: both dogfoods booted with files=27 (19 existing + 8 new bindings); the one-time migration adopted the local files and pictures without a refusal.

Drill (no data deleted): Bao's /data was switched from the cairn-openbao-data PVC to an emptyDir via GitOps (homelab-config cd54cab, volume renamed for SSA), then reverted (8a61a0a). Baseline taken before the switch vs. the instance running on an empty volume, compared by SHA-256 over the API responses:

Checked Result
11 spaces incl. subtitle, description, image bytes identical
Space limits (two 3 GiB) identical
Profile picture bytes identical
Favourites, 11 recents, 3 tags identical
Notification preferences identical

Back on the PVC afterwards, clean boot (the old local copies are *.migrated, so nothing to reconcile). What stays local is documented in deployment.md; the audit log and marks ledger are tracked in #823. Closing.

**Shipped in v0.6.310 (PR #822) and proven on the Bao dogfood, 2026-09-29.** Rollout: both dogfoods booted with `files=27` (19 existing + 8 new bindings); the one-time migration adopted the local files and pictures without a refusal. Drill (no data deleted): Bao's `/data` was switched from the `cairn-openbao-data` PVC to an `emptyDir` via GitOps (homelab-config cd54cab, volume renamed for SSA), then reverted (8a61a0a). Baseline taken before the switch vs. the instance running on an empty volume, compared by SHA-256 over the API responses: | Checked | Result | |---|---| | 11 spaces incl. subtitle, description, image bytes | identical | | Space limits (two 3 GiB) | identical | | Profile picture bytes | identical | | Favourites, 11 recents, 3 tags | identical | | Notification preferences | identical | Back on the PVC afterwards, clean boot (the old local copies are `*.migrated`, so nothing to reconcile). What stays local is documented in deployment.md; the audit log and marks ledger are tracked in #823. Closing.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#820
No description provided.