#742 Wave A: finish record-level state (re-read OCM, peering, licence; writers outside the pattern; guard) and bounded grace when state cannot be read #1011

Closed
opened 2026-10-26 10:56:17 +00:00 by Cordy · 3 comments
Owner

Found in the #1007 audit of all 35 statestore.Write callers (.superpowers/sdd/2026-10-26-batch8/task-b-report.md, audit table and follow-ups; final-review.md R2). v0.6.356 fixed the temp-name race for every file and added read-merge-write to the space registry, spaces.json, spacemeta.json and spacequota.json.

Still whole-file writes from memory (severity High once several instances share one state):

  • app passwords: a revoked device can come back;
  • local users, groups, admins: a removed admin can come back;
  • public shares and OCM state: a revoked share can come back;
  • legal holds: a hold set on one instance can be erased;
  • notify-seen.

Medium and Low files: the full table is in the report.

Stale reads: spacestore and spacequota still answer reads from a cache that is refreshed only on this instance's own write or restore paths. A member removed on one instance keeps access on the others until then (High), and a cap set elsewhere is not enforced (High). spacemeta is Low.

Fix direction: the record-level read-merge-write pattern used in v0.6.356 (re-read under the process lock, apply one change, write), plus a short re-read cadence for caches that decide access. Part of #742 before multi-instance is supported.

Found in the #1007 audit of all 35 `statestore.Write` callers (`.superpowers/sdd/2026-10-26-batch8/task-b-report.md`, audit table and follow-ups; final-review.md R2). v0.6.356 fixed the temp-name race for every file and added read-merge-write to the space registry, `spaces.json`, `spacemeta.json` and `spacequota.json`. **Still whole-file writes from memory (severity High once several instances share one state):** - app passwords: a revoked device can come back; - local users, groups, admins: a removed admin can come back; - public shares and OCM state: a revoked share can come back; - legal holds: a hold set on one instance can be erased; - notify-seen. **Medium and Low files:** the full table is in the report. **Stale reads:** spacestore and spacequota still answer reads from a cache that is refreshed only on this instance's own write or restore paths. A member removed on one instance keeps access on the others until then (High), and a cap set elsewhere is not enforced (High). spacemeta is Low. **Fix direction:** the record-level read-merge-write pattern used in v0.6.356 (re-read under the process lock, apply one change, write), plus a short re-read cadence for caches that decide access. Part of #742 before multi-instance is supported.
Author
Owner

Research for batch 8 (2026-11-12/13; research-1011.md and research-1011-unsettled.md).

Status: v0.6.401 (#742 Step 0) already moved every writer this issue names to version-checked read-merge-write with retries, a guard test, and 10 s re-reads of the deciding caches. Real cross-instance prevention needs conditional writes, which Garage cannot do, so it belongs to the #742 Step 1 store.

Found still open (Wave A): OCM trusted servers/contacts/invites/incoming shares, peering keys and paused peers, and the licence seat count are cached and never re-read; some record writers (space requests, mail outbox, zip tickets, copy-record head, audit-forwarding secrets) and update callbacks (second-factor index, IdP fingerprint, guest codes) bypass the pattern; the guard misses both.

Decisions (Nikola):

  • This issue becomes Wave A and closes when it ships in v0.6.408; Wave B (state on the chosen store, early invalidation, minimum-version gate) goes to the #742 Step 1 wiring batch.
  • When state cannot be re-read: a fixed 5-minute grace after the last good read, then grants (app passwords, API keys, local users, groups, admins, spaces, links, OCM, peering) refuse with 503 state-unreadable; legal holds pause background deletes at once and refuse every delete after 5 minutes; licence seats alert only; break-glass always works; log, metrics, live Overview banner, audit. Sources: NIST SP 800-53 SA-8(24), IA-5(13), SC-24, SI-17; CWE-636; OWASP ASVS 5.0 V8/V16; Kubernetes authorizer cache; OCSP soft-fail lessons; Nextcloud token cache.
  • Five small gaps fixed in the same batch (last-good-read time, missing holds.json, shares cache back-off, break-glass test, spaces class).
Research for batch 8 (2026-11-12/13; research-1011.md and research-1011-unsettled.md). **Status:** v0.6.401 (#742 Step 0) already moved every writer this issue names to version-checked read-merge-write with retries, a guard test, and 10 s re-reads of the deciding caches. Real cross-instance prevention needs conditional writes, which Garage cannot do, so it belongs to the #742 Step 1 store. **Found still open (Wave A):** OCM trusted servers/contacts/invites/incoming shares, peering keys and paused peers, and the licence seat count are cached and never re-read; some record writers (space requests, mail outbox, zip tickets, copy-record head, audit-forwarding secrets) and update callbacks (second-factor index, IdP fingerprint, guest codes) bypass the pattern; the guard misses both. **Decisions (Nikola):** - This issue becomes Wave A and closes when it ships in v0.6.408; Wave B (state on the chosen store, early invalidation, minimum-version gate) goes to the #742 Step 1 wiring batch. - When state cannot be re-read: a fixed 5-minute grace after the last good read, then grants (app passwords, API keys, local users, groups, admins, spaces, links, OCM, peering) refuse with 503 `state-unreadable`; legal holds pause background deletes at once and refuse every delete after 5 minutes; licence seats alert only; break-glass always works; log, metrics, live Overview banner, audit. Sources: NIST SP 800-53 SA-8(24), IA-5(13), SC-24, SI-17; CWE-636; OWASP ASVS 5.0 V8/V16; Kubernetes authorizer cache; OCSP soft-fail lessons; Nextcloud token cache. - Five small gaps fixed in the same batch (last-good-read time, missing holds.json, shares cache back-off, break-glass test, spaces class).
Cordy changed title from #742: most state files are rewritten whole from one instance's memory (app passwords, local users, groups, admins, shares, OCM, holds, notify marks): lost updates can resurrect revoked access to #742 Wave A: finish record-level state (re-read OCM, peering, licence; writers outside the pattern; guard) and bounded grace when state cannot be read 2026-11-12 23:48:45 +00:00
Author
Owner

Mockup and spec decisions (Nikola, 2026-11-13):

  • Mockup 1011-unreadable-state-mockup.html approved: Overview card amber during the 5-minute grace, red after; rail dot; bell summary after episodes that reached the bound; refusal texts in the app and on the sign-in page (login-state-unreadable); licence alert-only; endpoint /api/v1/admin/state-health; audit group state; last-good-read timestamp metrics.
  • A missing (after it existed) or unreadable holds.json at start: Cairn starts and treats every existing path as held (create-only marker; ended by restore or cairnd holds-confirm-empty, audited).
  • Break-glass names with a wrong password also answer 503 while state is past the bound, so the status never reveals which names are emergency accounts (OWASP ASVS 6.3.8).
  • The other 18 points accepted as researched (refuse only requests that use the unreadable grant; Retry-After 30 s, no WebDAV client drops stored passwords on 503; runtime admins keep seeing the card; space-groups tracked too).
    Next: spec and plan updated, then the build (v0.6.408).
Mockup and spec decisions (Nikola, 2026-11-13): - Mockup `1011-unreadable-state-mockup.html` approved: Overview card amber during the 5-minute grace, red after; rail dot; bell summary after episodes that reached the bound; refusal texts in the app and on the sign-in page (`login-state-unreadable`); licence alert-only; endpoint `/api/v1/admin/state-health`; audit group `state`; last-good-read timestamp metrics. - A missing (after it existed) or unreadable `holds.json` at start: Cairn starts and treats every existing path as held (create-only marker; ended by restore or `cairnd holds-confirm-empty`, audited). - Break-glass names with a wrong password also answer 503 while state is past the bound, so the status never reveals which names are emergency accounts (OWASP ASVS 6.3.8). - The other 18 points accepted as researched (refuse only requests that use the unreadable grant; Retry-After 30 s, no WebDAV client drops stored passwords on 503; runtime admins keep seeing the card; space-groups tracked too). Next: spec and plan updated, then the build (v0.6.408).
Author
Owner

Shipped in v0.6.408 (PR #1175, tag on f913ca61), live on both dogfoods (v0.6.408, ready, 0 restarts, healthz 200, no new WARN/ERROR).

  • Wave A: 10 s re-reads for OCM trust, contacts, invites and received shares, peering keys and paused peers, and licence seats; the last plain record saves (space requests, mail outbox, zip tickets, copy-record head, audit-forwarding secrets, keydrain, data-export records) and blind update callbacks (second-factor index, IdP fingerprint, guest codes) decide on the record as stored; the guard test catches new ones (by name, with an allowlist).
  • Bounded grace: each instance tracks the last good read of 14 to 16 state stores (read every 10 to 15 s, traffic or not). For 5 minutes Cairn answers from its last copy, then requests that depend on an unreadable grant answer 503 state-unreadable (Retry-After 30, never 401, no device pause or session revoke, throttle unchanged). Legal holds: background deletes incl. the Deleted-forever pass pause at the first failed read; past 5 minutes every delete, rename and overwrite of an existing file answers 503 holds-unavailable on every surface; a missing or unreadable holds.json at start starts held (cairnd holds-confirm-empty). Break-glass always works and is never revealed. Overview card, rail dot, bell, metrics, audit group state, GET /api/v1/admin/state-health.
  • Wave B (state on the chosen store, early invalidation, minimum-version gate) continues with the #742 store wiring.

Docs: deployment.md "When state cannot be read", admin-policy.md, security-standards.md, ARCHITECTURE.md, CHANGELOG. Residuals and small findings are in the backlog; nothing open for this issue.

Shipped in **v0.6.408** (PR #1175, tag on f913ca61), live on both dogfoods (v0.6.408, ready, 0 restarts, healthz 200, no new WARN/ERROR). - **Wave A:** 10 s re-reads for OCM trust, contacts, invites and received shares, peering keys and paused peers, and licence seats; the last plain record saves (space requests, mail outbox, zip tickets, copy-record head, audit-forwarding secrets, keydrain, data-export records) and blind update callbacks (second-factor index, IdP fingerprint, guest codes) decide on the record as stored; the guard test catches new ones (by name, with an allowlist). - **Bounded grace:** each instance tracks the last good read of 14 to 16 state stores (read every 10 to 15 s, traffic or not). For 5 minutes Cairn answers from its last copy, then requests that depend on an unreadable grant answer 503 `state-unreadable` (Retry-After 30, never 401, no device pause or session revoke, throttle unchanged). Legal holds: background deletes incl. the Deleted-forever pass pause at the first failed read; past 5 minutes every delete, rename and overwrite of an existing file answers 503 `holds-unavailable` on every surface; a missing or unreadable holds.json at start starts held (`cairnd holds-confirm-empty`). Break-glass always works and is never revealed. Overview card, rail dot, bell, metrics, audit group `state`, `GET /api/v1/admin/state-health`. - Wave B (state on the chosen store, early invalidation, minimum-version gate) continues with the #742 store wiring. Docs: deployment.md "When state cannot be read", admin-policy.md, security-standards.md, ARCHITECTURE.md, CHANGELOG. Residuals and small findings are in the backlog; nothing open for this issue.
Cordy closed this issue 2026-11-13 16:40:44 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1011
No description provided.