Cordy/Cairn
2
1
Fork 0
Code Issues 20 Pull requests Projects Releases Packages Wiki Activity Actions

feat(#528): custody token maintenance — boot self-check, renewal, metrics #534

Merged
Cordy merged 9 commits from fix-528 into main 2026-09-14 01:32:50 +00:00
Conversation 0 Commits 9 Files changed 7 +437 -3
Cordy commented 2026-09-14 01:31:45 +00:00
Owner
Copy link

Closes the failure class behind last night's incident: the openbao custody token hit its default 768h TTL, OpenBao's expiration manager revoked it, every fresh decrypt 403'd, and nothing said so until a user opened a file.

  • Boot self-check (custody=openbao): token lookup-self + a canary KV read at startup. Dead custody logs ERROR at boot; a non-renewable expiring token logs WARN with its remaining lifetime; healthy logs INFO with ttl/renewability. The canary treats 404 as healthy — the token is accepted on the data tree without requiring any secret to exist.
  • Background renewal: renewable tokens are renewed at half their TTL (clamped [1m, 12h]) — the periodic token from the runbook now actually lives forever. Lookup/renewal failures log ERROR and retry each minute, which also pumps the error counter, so the alert fires during an outage.
  • Metrics (licence-gauge pattern, #437): cairn_custody_errors_total (fetch/put/lookup/renew failures), cairn_custody_token_ttl_seconds (0 = non-expiring), cairn_custody_token_renewable — emitted only while a custody provider is active. CustodyLastError() exported for the admin Encryption & keys card as a follow-up.

TDD, red witnessed on the runner first (missing symbols across both packages), then green on the full suite. The test helper is maintBao — the existing suite already owns the name fakeBao as a type, which the red build-failure masked; caught and renamed in the green run.

After this merges and the dogfood restarts, the boot log should show custody self-check ok token_ttl=… renewable=true, and the renewal loop makes the 2026-10-16 fuse a non-event. A CairnCustodyFailing PrometheusRule lands in homelab-config next.

Closes the failure class behind last night's incident: the openbao custody token hit its default 768h TTL, OpenBao's expiration manager revoked it, every fresh decrypt 403'd, and nothing said so until a user opened a file. - **Boot self-check** (custody=openbao): token `lookup-self` + a canary KV read at startup. Dead custody logs ERROR at boot; a non-renewable expiring token logs WARN with its remaining lifetime; healthy logs INFO with ttl/renewability. The canary treats 404 as healthy — the token is accepted on the data tree without requiring any secret to exist. - **Background renewal:** renewable tokens are renewed at half their TTL (clamped [1m, 12h]) — the periodic token from the runbook now actually lives forever. Lookup/renewal failures log ERROR and retry each minute, which also pumps the error counter, so the alert fires during an outage. - **Metrics** (licence-gauge pattern, #437): `cairn_custody_errors_total` (fetch/put/lookup/renew failures), `cairn_custody_token_ttl_seconds` (0 = non-expiring), `cairn_custody_token_renewable` — emitted only while a custody provider is active. `CustodyLastError()` exported for the admin Encryption & keys card as a follow-up. TDD, red witnessed on the runner first (missing symbols across both packages), then green on the full suite. The test helper is `maintBao` — the existing suite already owns the name `fakeBao` as a type, which the red build-failure masked; caught and renamed in the green run. After this merges and the dogfood restarts, the boot log should show `custody self-check ok token_ttl=… renewable=true`, and the renewal loop makes the 2026-10-16 fuse a non-event. A `CairnCustodyFailing` PrometheusRule lands in homelab-config next.
Witnessed failing on the runner: LookupSelf/RenewSelf/SelfCheck/
renewAfter/CustodyStats/CustodyLastError and obs.SetCustodyProvider
do not exist yet. Covers: lookup-self parsing, 403 counting with a
sticky last error, renew-self lease parsing and failure counting,
the canary-404-is-healthy rule, fetch-failure counting, the renewal
interval clamp [1m, 12h] at half TTL, StartMaintenance registering
the stats provider, and the Prometheus exposition of
cairn_custody_errors_total / _token_ttl_seconds / _token_renewable
gated on an active provider.
feat(#528): custody token maintenance - boot self-check, renewal, metrics
All checks were successful
ci / test-and-build (pull_request) Successful in 53s
Details
b3b790252a 
The openbao dogfood died silently when its custody token hit the
default 768h TTL: OpenBao's expiration manager revoked it, every
fresh decrypt 403d, and nothing said so until a user opened a file.
Three changes close that class:

- Boot self-check: with custody=openbao, cairnd now runs token
  lookup-self plus a canary KV read at startup and logs an ERROR
  when custody is dead, a WARN when the token is non-renewable and
  expiring (with its remaining lifetime), and an INFO with ttl and
  renewability when healthy.
- Background renewal: a renewable token is renewed at half its TTL
  (clamped to [1m, 12h]), so the periodic token from the incident
  runbook now actually lives forever. Renewal and lookup failures
  log at ERROR and retry every minute.
- Metrics: custody read/write/renewal failures increment
  cairn_custody_errors_total; cairn_custody_token_ttl_seconds and
  cairn_custody_token_renewable expose the lease state - emitted
  only when a custody provider is active, mirroring the licence
  gauge pattern (#437). CustodyLastError() is exported for the
  admin surface to pick up in a follow-up.

Tests (witnessed red first on the runner): lookup/renew parsing and
failure counting against an httptest OpenBao, canary-404-is-healthy,
fetch-failure counting, the renewal clamp, provider registration,
and the exposition format incl. absence while inactive. The test
helper is maintBao: the existing suite already owns fakeBao.
Cordy scheduled this pull request to auto merge when all checks succeed 2026-09-14 01:31:51 +00:00
Cordy merged commit 337cba6ff8 into main 2026-09-14 01:32:50 +00:00
Cordy deleted branch fix-528 2026-09-14 01:32:52 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#534
No description provided.
Delete branch "fix-528"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?