#528 - openbao custody: group/content reads fail with 403 — diagnosis, discriminator, and hardening scope - Cordy/Cairn

Cordy commented

2026-09-14 00:28:54 +00:00

Owner

Observed on files-bao (v0.6.187), 2026-09-14 ~00:15Z: opening any file in the space Fall 3 fails; the raw endpoint returns 500 with {"error":"encrypt: openbao read groups/Fall 3: 403 Forbidden"}. Pod log shows every /api/v1/files/raw and /api/v1/preview/pdf in the session returning 500 (same 63-byte body). Visible symptoms on the phone: image preview opens to a broken-image "?", Edit shows the raw error JSON.

Why some previews still "work": tile thumbnails and converted previews are served from the per-version disk caches, so anything previewed before the failure began keeps rendering. No fresh decrypt demonstrably succeeded in this session — which matters for the diagnosis below. The odt that opened fine was a cached conversion.

Code path (internal/storage/encrypt/openbao.go): keysFor → fetch(ctx, "groups/Fall 3") → GET {base}/v1/{mount}/data/{prefix}/groups/Fall 3 with X-Vault-Token → non-200 → encrypt: openbao read %s: %s. The keyset cache is memory-only, so every pod restart re-reads from OpenBao — and the dogfoods restarted ~10 times on 2026-09-13 for the #389 waves, which is why this surfaced now regardless of when the credential actually went bad.

Two candidate root causes — OpenBao/Vault returns 403 for BOTH:

Token expired or revoked. Cairn never renews the token (no renewal code; token used as a static bearer). If setup minted a token with the default TTL (768h ≈ 32 days) rather than a periodic/orphan token, it would expire ~one month after the openbao dogfood was set up — which lands almost exactly now.
Policy lacks groups/* under the prefix (users/* granted, groups/* missed), and Fall 3 is simply the first group content opened since the last restart.

10-second discriminator: open any file under /home on files-bao.

Home also fails (openbao read users/nikola-test: 403 Forbidden) → token-wide → cause 1.
Home works → groups-only → cause 2.

Server-side check, whichever way it points: bao token lookup on Cairn's token (ttl/expire_time), and bao token capabilities <token> <mount>/data/<prefix>/groups/Fall 3.

Remediation (ops, likely): re-mint as an orphan periodic token (or a long/renewable TTL with renewal), policy path "<mount>/data/<prefix>/*" { capabilities = ["create","read","update"] } (+ the matching metadata path if used), update the instance config Secret, restart.

Product hardening scope (this issue):

Boot-time custody self-check: on startup with openbao custody, probe a canary read (and token lookup-self for TTL) and log loudly — today the instance boots green and fails only on first decrypt.
Surface custody failures in the admin Encryption & keys card (LastError pattern, like the office discovery card) instead of raw JSON at the content endpoint.
Token self-renewal when the token is renewable; warn at boot when it is not and carries an expiry.
Metric + alert: custody read failures as a counter (cairn_custody_errors_total) so a CairnCustodyFailing alert can fire — CairnDown cannot see this failure mode (instance healthy, content dead).
Path escaping: url(rel) concatenates the space name unescaped. "Fall 3" survives via Go's URL re-encoding, but %, ? or # in a space name would corrupt the request path — escape per segment (url.PathEscape).
UX: the file-open path should render a readable error state instead of raw JSON (phone screenshot shows the bare body).

Observed on files-bao (v0.6.187), 2026-09-14 ~00:15Z: opening any file in the space **Fall 3** fails; the raw endpoint returns 500 with `{"error":"encrypt: openbao read groups/Fall 3: 403 Forbidden"}`. Pod log shows every `/api/v1/files/raw` and `/api/v1/preview/pdf` in the session returning 500 (same 63-byte body). Visible symptoms on the phone: image preview opens to a broken-image "?", Edit shows the raw error JSON. **Why some previews still "work":** tile thumbnails and converted previews are served from the per-version disk caches, so anything previewed before the failure began keeps rendering. No **fresh** decrypt demonstrably succeeded in this session — which matters for the diagnosis below. The odt that opened fine was a cached conversion. **Code path** (`internal/storage/encrypt/openbao.go`): `keysFor` → `fetch(ctx, "groups/Fall 3")` → GET `{base}/v1/{mount}/data/{prefix}/groups/Fall 3` with `X-Vault-Token` → non-200 → `encrypt: openbao read %s: %s`. The keyset cache is memory-only, so every pod restart re-reads from OpenBao — and the dogfoods restarted ~10 times on 2026-09-13 for the #389 waves, which is why this surfaced now regardless of when the credential actually went bad. **Two candidate root causes** — OpenBao/Vault returns 403 for BOTH: 1. **Token expired or revoked.** Cairn never renews the token (no renewal code; token used as a static bearer). If setup minted a token with the default TTL (768h ≈ 32 days) rather than a periodic/orphan token, it would expire ~one month after the openbao dogfood was set up — which lands almost exactly now. 2. **Policy lacks `groups/*`** under the prefix (users/* granted, groups/* missed), and Fall 3 is simply the first group content opened since the last restart. **10-second discriminator:** open any file under `/home` on files-bao. - Home also fails (`openbao read users/nikola-test: 403 Forbidden`) → token-wide → cause 1. - Home works → groups-only → cause 2. Server-side check, whichever way it points: `bao token lookup` on Cairn's token (ttl/expire_time), and `bao token capabilities <token> <mount>/data/<prefix>/groups/Fall 3`. **Remediation (ops, likely):** re-mint as an orphan **periodic** token (or a long/renewable TTL with renewal), policy `path "<mount>/data/<prefix>/*" { capabilities = ["create","read","update"] }` (+ the matching `metadata` path if used), update the instance config Secret, restart. **Product hardening scope (this issue):** - **Boot-time custody self-check:** on startup with openbao custody, probe a canary read (and token lookup-self for TTL) and log loudly — today the instance boots green and fails only on first decrypt. - **Surface custody failures in the admin Encryption & keys card** (LastError pattern, like the office discovery card) instead of raw JSON at the content endpoint. - **Token self-renewal** when the token is renewable; warn at boot when it is not and carries an expiry. - **Metric + alert:** custody read failures as a counter (`cairn_custody_errors_total`) so a `CairnCustodyFailing` alert can fire — `CairnDown` cannot see this failure mode (instance healthy, content dead). - **Path escaping:** `url(rel)` concatenates the space name unescaped. "Fall 3" survives via Go's URL re-encoding, but `%`, `?` or `#` in a space name would corrupt the request path — escape per segment (`url.PathEscape`). - **UX:** the file-open path should render a readable error state instead of raw JSON (phone screenshot shows the bare body).

Cordy added this to the v0.7 — Closed beta (dogfood + hardening) milestone

2026-09-14 00:47:01 +00:00

Cordy commented

2026-09-14 00:48:43 +00:00

Author

Owner

Root cause confirmed: the custody token expired — default 768h TTL, dead on schedule

Evidence chain:

Discriminator run by Nikola: /home open fails identically — {"error":"encrypt: openbao read users/nikola-test: 403 Forbidden"} → token-wide, not a groups policy gap.
openbao-0 (ns openbao) is 32 days old with 0 restarts — never resealed, so not the seal (sealed would 503 anyway).
The smoking gun in openbao-0's log, ~50 minutes before the first 403:
2026-09-13T23:25:27Z [INFO] expiration: revoked lease: lease_id=auth/token/create/h5e13255c…
The expiration manager revoked a token minted via auth/token/create. 32 days = the default 768h token TTL; the token was minted during the same setup session that started the pod. Cairn never renews (static bearer), so it died exactly one TTL later.

Why it surfaced as "space Fall 3": the keyset cache is memory-only and the dogfoods restarted repeatedly on 2026-09-13 for the #389 waves; the first post-restart decrypt anyone attempted happened to be group content.

Runbook (operator — needs the root token, so this is Nikola's)

kubectl -n openbao exec -it openbao-0 -- sh
export BAO_ADDR=http://127.0.0.1:8200
bao login                      # root token from the setup notes

bao policy read cairn          # verify the policy still covers secret/{data,metadata}/cairn/*

# mint the replacement as an ORPHAN PERIODIC token:
bao token create -policy=cairn -orphan -period=768h -display-name=cairn-openbao-dogfood

Then put the new token into the openbao dogfood's (untracked) config Secret in ns cairn and kubectl -n cairn rollout restart deploy/cairn-openbao. Content in Fall 3 and /home should open immediately after (fresh decrypts; caches were never affected).

Important caveat on -period: a periodic token is renewable forever but must still be renewed within each period — it does not refresh on use. Until Cairn self-renews, this token dies again in 32 days, same as this one. That turns this issue's hardening list from nice-to-have into a fuse:

Token self-renewal (custody=openbao): background renew-self at period/2 when the token is renewable; WARN loudly at boot when it is not renewable and carries an expiry. This is the actual fix.
Boot-time custody self-check (token lookup-self + canary read) so an instance with dead custody says so at startup instead of booting green.
Surfacing in the Encryption & keys admin card + cairn_custody_errors_total metric for a CairnCustodyFailing alert — this failure mode was invisible to every existing alert for ~an hour of active use.

Happy to build the renewal + boot-check + metric as the next TDD run on request — it is self-contained (httptest OpenBao stub) and closes the fuse before day 32.

## Root cause confirmed: the custody token expired — default 768h TTL, dead on schedule Evidence chain: 1. Discriminator run by Nikola: `/home` open fails identically — `{"error":"encrypt: openbao read users/nikola-test: 403 Forbidden"}` → token-wide, not a groups policy gap. 2. `openbao-0` (ns `openbao`) is **32 days old with 0 restarts** — never resealed, so not the seal (sealed would 503 anyway). 3. The smoking gun in openbao-0's log, ~50 minutes before the first 403: `2026-09-13T23:25:27Z [INFO] expiration: revoked lease: lease_id=auth/token/create/h5e13255c…` The expiration manager revoked a token minted via `auth/token/create`. 32 days = the default **768h** token TTL; the token was minted during the same setup session that started the pod. Cairn never renews (static bearer), so it died exactly one TTL later. Why it surfaced as "space Fall 3": the keyset cache is memory-only and the dogfoods restarted repeatedly on 2026-09-13 for the #389 waves; the first post-restart decrypt anyone attempted happened to be group content. ### Runbook (operator — needs the root token, so this is Nikola's) ``` kubectl -n openbao exec -it openbao-0 -- sh export BAO_ADDR=http://127.0.0.1:8200 bao login # root token from the setup notes bao policy read cairn # verify the policy still covers secret/{data,metadata}/cairn/* # mint the replacement as an ORPHAN PERIODIC token: bao token create -policy=cairn -orphan -period=768h -display-name=cairn-openbao-dogfood ``` Then put the new token into the openbao dogfood's (untracked) config Secret in ns `cairn` and `kubectl -n cairn rollout restart deploy/cairn-openbao`. Content in Fall 3 and /home should open immediately after (fresh decrypts; caches were never affected). **Important caveat on `-period`:** a periodic token is renewable forever but must still be *renewed* within each period — it does **not** refresh on use. Until Cairn self-renews, this token dies again in 32 days, same as this one. That turns this issue's hardening list from nice-to-have into a fuse: - **Token self-renewal** (custody=openbao): background renew-self at period/2 when the token is renewable; WARN loudly at boot when it is not renewable and carries an expiry. This is the actual fix. - **Boot-time custody self-check** (token lookup-self + canary read) so an instance with dead custody says so at startup instead of booting green. - Surfacing in the Encryption & keys admin card + `cairn_custody_errors_total` metric for a `CairnCustodyFailing` alert — this failure mode was invisible to every existing alert for ~an hour of active use. Happy to build the renewal + boot-check + metric as the next TDD run on request — it is self-contained (httptest OpenBao stub) and closes the fuse before day 32.

Cordy referenced this issue from a commit

2026-09-14 01:17:17 +00:00

one-shot: #528 red - custody maintenance + metrics tests (expected to fail)

forgejo-actions referenced this issue from a commit

2026-09-14 01:17:25 +00:00

test(#528): custody token maintenance + custody metrics (red)

Cordy referenced this issue from a commit

2026-09-14 01:20:58 +00:00

one-shot: #528 green - custody maintenance, metrics, wiring

Cordy referenced this issue from a commit

2026-09-14 01:27:20 +00:00

one-shot: #528 finisher - rename colliding test helper, rerun gates, commit

Cordy referenced this issue from a commit

2026-09-14 01:30:28 +00:00

one-shot: #528 finisher 2 - rebase and land the green tree

forgejo-actions referenced this issue from a commit

2026-09-14 01:30:35 +00:00

feat(#528): custody token maintenance - boot self-check, renewal, metrics

Cordy referenced this issue

2026-09-14 01:31:45 +00:00

feat(#528): custody token maintenance — boot self-check, renewal, metrics #534

Cordy referenced this issue from a commit

2026-09-14 01:32:52 +00:00

Merge pull request 'feat(#528): custody token maintenance — boot self-check, renewal, metrics' (#534) from fix-528 into main

Cordy referenced this issue from a commit

2026-09-14 01:44:09 +00:00

one-shot: #528 - wire StartMaintenance into buildEncryption (the path dogfoods take)

forgejo-actions referenced this issue from a commit

2026-09-14 01:44:25 +00:00

fix(#528): start custody maintenance on the in-process path too

Cordy referenced this issue

2026-09-14 01:45:52 +00:00

fix(#528): start custody maintenance on the in-process path too #536

Cordy referenced this issue from a commit

2026-09-14 01:47:00 +00:00

Merge pull request 'fix(#528): start custody maintenance on the in-process path too' (#536) from fix-528c into main

Cordy commented

2026-09-14 01:54:35 +00:00

Author

Owner

Hardening shipped — v0.6.189 + v0.6.190, alerts live

Ops side (Nikola): root token recovered, replacement minted as orphan periodic 768h, config Secret updated, instance restarted — content decrypts again.

Product side:

v0.6.189 (PR #534, TDD red→green on the runner): openbao_maint.go — LookupSelf/RenewSelf/SelfCheck (canary-404-is-healthy), renewAfter clamp [1m, 12h] at half TTL, process-global custody stats with sticky CustodyLastError(), fetch/put failures counted; obs gains SetCustodyProvider emitting cairn_custody_errors_total / cairn_custody_token_ttl_seconds / cairn_custody_token_renewable (licence-gauge pattern).
v0.6.190 (PR #536): the wiring fix — v0.6.189 started maintenance only in buildKeys (standalone crypto role); in-process instances build through buildEncryption's near-duplicate switch and got nothing. Caught because the deployed boot log lacked the self-check line; now wired on both paths.
homelab-config: CairnCustodyFailing (critical, any increase(cairn_custody_errors_total[15m])) and CairnCustodyTokenExpiring (warning, TTL < 3d for 30m) in the cairn-alerts group.

Live verification (cairn-openbao, v0.6.190 boot):

custody self-check ok token_ttl=767h14m44s renewable=true

The renewal loop now keeps that sawtoothing — 2026-10-16 is a non-event. During an actual outage the lookup retries every minute at ERROR, pumping the counter, so the critical alert fires within ~20 minutes instead of never.

Remaining scope (keeping this open):

Surface CustodyLastError() + token TTL in the admin Encryption & keys card (needs the encryption payload + frontend card work).
url(rel) path escaping per segment (%, ?, # in space names).
Refactor candidate: buildKeys and buildEncryption duplicate the recovery parsing and the whole custody switch — that duplication is precisely what let v0.6.189 wire the wrong door.

## Hardening shipped — v0.6.189 + v0.6.190, alerts live **Ops side (Nikola):** root token recovered, replacement minted as orphan periodic 768h, config Secret updated, instance restarted — content decrypts again. **Product side:** - v0.6.189 (PR #534, TDD red→green on the runner): `openbao_maint.go` — `LookupSelf`/`RenewSelf`/`SelfCheck` (canary-404-is-healthy), `renewAfter` clamp [1m, 12h] at half TTL, process-global custody stats with sticky `CustodyLastError()`, fetch/put failures counted; obs gains `SetCustodyProvider` emitting `cairn_custody_errors_total` / `cairn_custody_token_ttl_seconds` / `cairn_custody_token_renewable` (licence-gauge pattern). - v0.6.190 (PR #536): the wiring fix — v0.6.189 started maintenance only in `buildKeys` (standalone crypto role); in-process instances build through `buildEncryption`'s near-duplicate switch and got nothing. Caught because the deployed boot log lacked the self-check line; now wired on both paths. - homelab-config: `CairnCustodyFailing` (critical, any `increase(cairn_custody_errors_total[15m])`) and `CairnCustodyTokenExpiring` (warning, TTL < 3d for 30m) in the cairn-alerts group. **Live verification (cairn-openbao, v0.6.190 boot):** ``` custody self-check ok token_ttl=767h14m44s renewable=true ``` The renewal loop now keeps that sawtoothing — 2026-10-16 is a non-event. During an actual outage the lookup retries every minute at ERROR, pumping the counter, so the critical alert fires within ~20 minutes instead of never. **Remaining scope (keeping this open):** - Surface `CustodyLastError()` + token TTL in the admin Encryption & keys card (needs the encryption payload + frontend card work). - `url(rel)` path escaping per segment (`%`, `?`, `#` in space names). - Refactor candidate: `buildKeys` and `buildEncryption` duplicate the recovery parsing and the whole custody switch — that duplication is precisely what let v0.6.189 wire the wrong door.

forgejo-actions referenced this issue from a commit

2026-09-14 02:06:47 +00:00

test(#528): url segment escaping + custody status in encryption payload (red)

forgejo-actions referenced this issue from a commit

2026-09-14 02:15:30 +00:00

feat(#528): custody status on the admin card, path escaping, provider dedup

Cordy referenced this issue

2026-09-14 02:20:08 +00:00

#528 finish: custody status on the admin card, KV path escaping, provider dedup (v0.6.191) #537

Cordy referenced this issue from a commit

2026-09-14 02:21:21 +00:00

Merge pull request '#528 finish: custody status on the admin card, KV path escaping, provider dedup (v0.6.191)' (#537) from fix-528d into main

Cordy commented

2026-09-14 02:29:53 +00:00

Author

Owner

v0.6.191 shipped (PR #537) — remaining scope done, closing.

Encryption & keys card now shows the custody token's live state under Custody (OpenBao custody only, while maintenance is active): whether it renews itself before expiry, TTL as a badge (hours, days past 48 h), a warning row for a non-renewable token, and a Last custody error row when one has occurred. Backed by a new custodyStatus object in GET /api/v1/admin/encryption sourced from CustodyStats()/CustodyLastError() — counters and TTL only, the token never reaches the payload. i18n ×4.
Path escaping: url(rel) now escapes each segment with url.PathEscape, so a #, % or ? in a user or space name can no longer corrupt the KV request path (a % previously made the URL unparseable outright).
Provider dedup: buildEncryption collapsed to encrypt.New(store, buildKeys(cfg, log)) after verifying the two switches were token-identical — the duplicate is what hid the v0.6.189 wiring gap.

Verified live on the dogfood after the bump: version=v0.6.191, custody self-check ok token_ttl=767h25m55s renewable=true — i.e. the dedup preserved the maintenance wiring on the in-process path.

Full incident record: root cause (default 768 h token TTL, revoked lease), operational fix (orphan periodic token), productized hardening (boot self-check + background renewal, v0.6.189/190), Prometheus alerts (CairnCustodyFailing, CairnCustodyTokenExpiring) in homelab-config, and this release's admin-card visibility + escaping + dedup.

**v0.6.191 shipped (PR #537) — remaining scope done, closing.** - **Encryption & keys card** now shows the custody token's live state under Custody (OpenBao custody only, while maintenance is active): whether it renews itself before expiry, TTL as a badge (hours, days past 48 h), a warning row for a non-renewable token, and a Last custody error row when one has occurred. Backed by a new `custodyStatus` object in `GET /api/v1/admin/encryption` sourced from `CustodyStats()`/`CustodyLastError()` — counters and TTL only, the token never reaches the payload. i18n ×4. - **Path escaping:** `url(rel)` now escapes each segment with `url.PathEscape`, so a `#`, `%` or `?` in a user or space name can no longer corrupt the KV request path (a `%` previously made the URL unparseable outright). - **Provider dedup:** `buildEncryption` collapsed to `encrypt.New(store, buildKeys(cfg, log))` after verifying the two switches were token-identical — the duplicate is what hid the v0.6.189 wiring gap. Verified live on the dogfood after the bump: `version=v0.6.191`, `custody self-check ok token_ttl=767h25m55s renewable=true` — i.e. the dedup preserved the maintenance wiring on the in-process path. Full incident record: root cause (default 768 h token TTL, revoked lease), operational fix (orphan periodic token), productized hardening (boot self-check + background renewal, v0.6.189/190), Prometheus alerts (CairnCustodyFailing, CairnCustodyTokenExpiring) in homelab-config, and this release's admin-card visibility + escaping + dedup.

Cordy closed this issue

2026-09-14 02:29:56 +00:00

Cordy referenced this issue

2026-09-14 02:48:24 +00:00

File versioning: design ADR, per-driver version storage, restore/download UI (#530, 4/4) #541

forgejo-actions referenced this issue from a commit

2026-09-14 22:43:40 +00:00

docs: align all surfaces with v0.6.180-201 (#533/#550 accounts+groups, #528 custody, #389 phone)

Cordy referenced this issue

2026-09-14 22:44:35 +00:00

docs: full alignment run for v0.6.180–201 — local accounts, groups, custody, phone #554

Cordy referenced this issue