OpenBao custody: live e2e against a real OpenBao instance #112

New issue

Closed

opened 2026-08-04 17:23:30 +00:00 by Cordy · 2 comments

Cordy commented

2026-08-04 17:23:30 +00:00

Owner

The OpenBao key-custody backend (#29, v0.3.28) is unit-tested against a fake KV v2 only. Before any customer runs keyCustody: "openbao" — and AD parity is one of its selling points — it needs a pass against the real thing. Moved onto the v0.6 closed-beta milestone per the 2026-08-04 roadmap redesign.

Plan (same pattern as the Keycloak/interop e2e work):

Deploy OpenBao on k3s via GitOps (small StatefulSet or the upstream chart; dev-mode is acceptable for the test, token auth): enable a KV v2 mount, create a scoped token for Cairn.
Spin a test Cairn instance (or temporarily a second config on the runner) with keyCustody: "openbao" + recovery identity, storage on a scratch bucket.
One-shot interop-openbao.yml on the runner (LAN test host):
- user write → identity lazily provisioned under {prefix}/users/<u> — verify via OpenBao API; roundtrip read.
- space write (X-Cairn-Group path) → {prefix}/groups/<space> provisioned; member roundtrip.
- restart/second replica → same identities adopted (no re-mint, cas respected).
- recovery: pull raw object, age -d with recovery key.
- postQuantum: true variant: hybrid provision + upgrade-on-touch of a pre-seeded classical secret (legacyIdentities retained), old object still decrypts.
Record results here; note any KV v2 semantic differences from the fake (cas error shape, metadata versioning) and fix the provider if reality disagrees.

Open question: keep the OpenBao instance standing (it doubles as a custody option for other homelab experiments) or tear down after green — decide when it exists.

The OpenBao key-custody backend (#29, v0.3.28) is unit-tested against a fake KV v2 only. Before any customer runs `keyCustody: "openbao"` — and AD parity is one of its selling points — it needs a pass against the real thing. Moved onto the v0.6 closed-beta milestone per the 2026-08-04 roadmap redesign. **Plan (same pattern as the Keycloak/interop e2e work):** 1. Deploy OpenBao on k3s via GitOps (small StatefulSet or the upstream chart; dev-mode is acceptable for the test, token auth): enable a KV v2 mount, create a scoped token for Cairn. 2. Spin a test Cairn instance (or temporarily a second config on the runner) with `keyCustody: "openbao"` + recovery identity, storage on a scratch bucket. 3. One-shot `interop-openbao.yml` on the runner (LAN test host): - user write → identity lazily provisioned under `{prefix}/users/<u>` — verify via OpenBao API; roundtrip read. - space write (`X-Cairn-Group` path) → `{prefix}/groups/<space>` provisioned; member roundtrip. - restart/second replica → same identities adopted (no re-mint, cas respected). - recovery: pull raw object, `age -d` with recovery key. - **postQuantum: true** variant: hybrid provision + upgrade-on-touch of a pre-seeded classical secret (`legacyIdentities` retained), old object still decrypts. 4. Record results here; note any KV v2 semantic differences from the fake (cas error shape, metadata versioning) and fix the provider if reality disagrees. Open question: keep the OpenBao instance standing (it doubles as a custody option for other homelab experiments) or tear down after green — decide when it exists.

Cordy added this to the v0.7 — Closed beta (dogfood + hardening) milestone

2026-08-04 17:23:30 +00:00

Cordy commented

2026-08-12 21:15:01 +00:00

Author

Owner

Triage: tandem (needs a real OpenBao). The openbao key-custody backend exists and is unit-tested; this is a live e2e against an actual OpenBao KV v2 mount (provision + token). I can prep a test workflow/checklist; you stand up the instance.

**Triage: tandem (needs a real OpenBao).** The `openbao` key-custody backend exists and is unit-tested; this is a live e2e against an actual OpenBao KV v2 mount (provision + token). I can prep a test workflow/checklist; you stand up the instance.

Cordy commented

2026-08-13 01:36:04 +00:00

Author

Owner

Verified — OpenBao custody works end to end against a real OpenBao instance.

Deployed (GitOps):

OpenBao single-node, integrated (raft) storage — k3s/openbao/openbao.yaml. Plain HTTP on the cluster network, disable_mlock, health probe lenient enough to stay routable while sealed (so init/unseal reach it). Test-grade: one unseal key, no auto-unseal — a pod restart re-seals and needs a manual bao operator unseal.
KV v2 mounted at secret/; least-privilege cairn policy (create/read/update/delete on secret/data/cairn/*, +list on secret/metadata/cairn/*); a periodic token issued to Cairn.
Test Cairn instance cairn-openbao — k3s/cairn/openbao-dogfood.yaml, own Garage bucket cairn-openbao, OIDC against the same realm, HTTPS via Caddy. Encryption config: keyCustody: openbao, openbaoMount: secret, openbaoPathPrefix: cairn, postQuantum: true with a hybrid (age-keygen -pq) recovery identity.

Verification:

Boot log: storage encryption enabled custody=openbao recovery=true, auth=oidc, version=v0.6.16. No flavour-mismatch refusal, so the hybrid recovery key correctly matches postQuantum: true.
Per-user minting: after a real Keycloak login + file upload as nikola-test, bao kv list secret/cairn/users returns nikola-test; the age identity is stored at secret/cairn/users/nikola-test in OpenBao — and not in the Keycloak user profile (no cairnAgeIdentity attribute). Uniform custody without writable IdP attributes, which is the point of the mode (AD deployments).
Round-trip: the uploaded file downloads/opens cleanly → Cairn fetched the key back from OpenBao and decrypted.
Isolation: separate bucket, no cross-instance file visibility. The Spaces folders shown are virtual group roots derived from the OIDC groups claim, empty on the fresh bucket — not another instance's data.

The OpenBao deployment and the test instance stay in GitOps for future regression. Closing as done.

**Verified — OpenBao custody works end to end against a real OpenBao instance.** **Deployed (GitOps):** - **OpenBao** single-node, integrated (raft) storage — `k3s/openbao/openbao.yaml`. Plain HTTP on the cluster network, `disable_mlock`, health probe lenient enough to stay routable while sealed (so init/unseal reach it). Test-grade: one unseal key, no auto-unseal — a pod restart re-seals and needs a manual `bao operator unseal`. - KV v2 mounted at `secret/`; least-privilege `cairn` policy (`create/read/update/delete` on `secret/data/cairn/*`, `+list` on `secret/metadata/cairn/*`); a periodic token issued to Cairn. - **Test Cairn instance** `cairn-openbao` — `k3s/cairn/openbao-dogfood.yaml`, own Garage bucket `cairn-openbao`, OIDC against the same realm, HTTPS via Caddy. Encryption config: `keyCustody: openbao`, `openbaoMount: secret`, `openbaoPathPrefix: cairn`, `postQuantum: true` with a hybrid (`age-keygen -pq`) recovery identity. **Verification:** - Boot log: `storage encryption enabled custody=openbao recovery=true`, `auth=oidc`, `version=v0.6.16`. No flavour-mismatch refusal, so the hybrid recovery key correctly matches `postQuantum: true`. - Per-user minting: after a real Keycloak login + file upload as `nikola-test`, `bao kv list secret/cairn/users` returns `nikola-test`; the age identity is stored at `secret/cairn/users/nikola-test` in OpenBao — and **not** in the Keycloak user profile (no `cairnAgeIdentity` attribute). Uniform custody without writable IdP attributes, which is the point of the mode (AD deployments). - Round-trip: the uploaded file downloads/opens cleanly → Cairn fetched the key back from OpenBao and decrypted. - Isolation: separate bucket, no cross-instance file visibility. The Spaces folders shown are virtual group roots derived from the OIDC `groups` claim, empty on the fresh bucket — not another instance's data. The OpenBao deployment and the test instance stay in GitOps for future regression. Closing as done.

Cordy closed this issue

2026-08-13 01:36:10 +00:00