Cordy/Cairn

Fork 0

Deep dive: one legacy key slot is a design ceiling on key rotation #349

New issue

Closed

opened 2026-08-28 00:22:02 +00:00 by Cordy · 2 comments

Cordy commented

2026-08-28 00:22:02 +00:00

Owner

The finding

Every key provider holds exactly one retained identity:

StaticKeys{Identity, Recovery, RecoveryLegacy, LegacyIdentity} (internal/storage/encrypt/static.go)
KeycloakKeys{Recovery, RecoveryLegacy} (keycloak.go)
OpenBaoKeys{Recovery, RecoveryLegacy} (openbao.go)
cmd/cairnd/main.go seals state to RecoveryIdentity + DeploymentIdentity and opens with the two *Legacy singulars.

Because rotation is not retroactive, a stored object stays readable only by the identities it was written to. One legacy slot therefore means an instance can be rotated once. A second rotation silently evicts the still-load-bearing key and orphans every object that depended on it.

Note the asymmetry: per-user and per-space identities do not have this problem — resolveKeyset keeps a full all []age.Identity list and retains every older identity for decryption (that is what makes upgrade-on-touch lossless). The restriction exists only for the deployment-level identities, which are parsed from single config strings.

Why this is a deep dive and not a fix

Widening RecoveryLegacy to a list is a small change on its face, and it may well be the answer. But it decides several things at once, and getting them wrong is unrecoverable by construction:

Config surface. recoveryIdentityLegacy is a documented scalar. Does it become an array, does a second field appear, or do keys move out of config entirely into the state store (which #138 already relocated everything else into)?
Interaction with #138. State now lives in the backend, sealed to these very identities. A key list that lives in the state that the key list unseals is circular — the same rule that keeps the storage connection out of live settings applies here, and the boundary needs stating explicitly.
Unbounded retention. Every retained key is a live decryption path for anything it ever covered. "Keep them all forever" is safe against data loss and bad against key hygiene — the opposite failure of today's behaviour. Retirement needs a defensible policy, which is what the coverage report (#348) makes possible.
PQ flavour rules. Retained keys are decrypt-only, so flavour homogeneity does not bind them the way it binds Recipients(). Worth confirming rather than assuming, since it determines whether a mixed classical/hybrid retained set is legal.
Operator model. However many slots exist, the operator has to reason about which key covers what. The current answer is a doc paragraph. The good answer is probably a key inventory in the admin UI, of which #348's report is the first half.

Where it stands today

Documented rather than fixed, in docs/handbook/encryption.md → "Rotating the recovery identity", which states the one-slot constraint as a hard rule and tells operators to finish retiring the old key before rotating again. That is correct guidance for current behaviour and buys time, but it leaves a real ceiling in place.

Outcome wanted from this issue

A decision — recorded as an ADR under docs/adr/ — on how deployment-level key history should be represented, covering the five points above. Implementation follows from it and may well be small.

#348 — legacy-key coverage report (makes the current single slot survivable; prerequisite for any retirement policy)
#138 — state-in-backend (owns the config-vs-state boundary this has to respect)

## The finding Every key provider holds exactly **one** retained identity: - `StaticKeys{Identity, Recovery, RecoveryLegacy, LegacyIdentity}` (`internal/storage/encrypt/static.go`) - `KeycloakKeys{Recovery, RecoveryLegacy}` (`keycloak.go`) - `OpenBaoKeys{Recovery, RecoveryLegacy}` (`openbao.go`) - `cmd/cairnd/main.go` seals state to `RecoveryIdentity` + `DeploymentIdentity` and opens with the two `*Legacy` singulars. Because rotation is not retroactive, a stored object stays readable only by the identities it was written to. One legacy slot therefore means **an instance can be rotated once**. A second rotation silently evicts the still-load-bearing key and orphans every object that depended on it. Note the asymmetry: per-user and per-space identities do *not* have this problem — `resolveKeyset` keeps a full `all []age.Identity` list and retains every older identity for decryption (that is what makes upgrade-on-touch lossless). The restriction exists only for the deployment-level identities, which are parsed from single config strings. ## Why this is a deep dive and not a fix Widening `RecoveryLegacy` to a list is a small change on its face, and it may well be the answer. But it decides several things at once, and getting them wrong is unrecoverable by construction: - **Config surface.** `recoveryIdentityLegacy` is a documented scalar. Does it become an array, does a second field appear, or do keys move out of config entirely into the state store (which #138 already relocated everything else into)? - **Interaction with #138.** State now lives in the backend, sealed to these very identities. A key list that lives in the state that the key list unseals is circular — the same rule that keeps the storage connection out of live settings applies here, and the boundary needs stating explicitly. - **Unbounded retention.** Every retained key is a live decryption path for anything it ever covered. "Keep them all forever" is safe against data loss and bad against key hygiene — the opposite failure of today's behaviour. Retirement needs a defensible policy, which is what the coverage report (#348) makes possible. - **PQ flavour rules.** Retained keys are decrypt-only, so flavour homogeneity does not bind them the way it binds `Recipients()`. Worth confirming rather than assuming, since it determines whether a mixed classical/hybrid retained set is legal. - **Operator model.** However many slots exist, the operator has to reason about which key covers what. The current answer is a doc paragraph. The good answer is probably a key inventory in the admin UI, of which #348's report is the first half. ## Where it stands today Documented rather than fixed, in `docs/handbook/encryption.md` → "Rotating the recovery identity", which states the one-slot constraint as a hard rule and tells operators to finish retiring the old key before rotating again. That is correct guidance for current behaviour and buys time, but it leaves a real ceiling in place. ## Outcome wanted from this issue A decision — recorded as an ADR under `docs/adr/` — on how deployment-level key history should be represented, covering the five points above. Implementation follows from it and may well be small. ## Related - #348 — legacy-key coverage report (makes the current single slot survivable; prerequisite for any retirement policy) - #138 — state-in-backend (owns the config-vs-state boundary this has to respect)

Cordy added this to the v0.7 — Closed beta (dogfood + hardening) milestone

2026-08-28 00:22:02 +00:00

Cordy referenced this issue

2026-08-28 00:52:37 +00:00

feat(admin): build out the Encryption & keys page #350

Cordy referenced this issue from a commit

2026-09-03 16:50:07 +00:00

docs: ADR 0002 - deployment-key rotation as rotate/drain/verify/discard (#349)

Cordy referenced this issue

2026-09-03 16:50:16 +00:00

ADR 0002: deployment-key rotation — rotate, drain, verify, discard (#349) #408

Cordy referenced this issue from a commit

2026-09-03 16:50:32 +00:00

Merge pull request 'ADR 0002: deployment-key rotation — rotate, drain, verify, discard (#349)' (#408) from adr-349 into main

Cordy commented

2026-09-03 16:50:46 +00:00

Author

Owner

Decided and recorded: ADR 0002 — rotate, drain, verify, discard (PR #408, operator direction 2026-09-03).

The keychain option is rejected outright: no growing retained-key list at the deployment level. Instead the two-key shape (active + one retiring) stays, and rotation becomes a completable operation — a managed background drain re-encrypts everything the retiring key covers (through the normal write pipeline, so custody re-resolution and PQ upgrade-on-touch ride along), #348's header scan verifies coverage has reached zero, and only then is the old key discarded. Next rotation is allowed from that state.

Answers to the five questions in this issue's brief:

Config surface — unchanged; the scalar legacy slots are correct for a bounded model.
#138 boundary — secrets stay in config/env, never state (circularity). The state store gains only a public-halves drain ledger, which powers a refuse-to-boot tripwire when an undrained key vanishes from config — today's silent orphaning becomes a named refusal.
Retention — no unbounded retention by design; drain-then-discard is the policy, #348 makes it provable.
PQ flavour — homogeneity binds the encrypt-side recipient set only; an active-hybrid/retiring-classical window is legal and the drain resolves it.
Operator model — Encryption & keys shows the retiring key, drain progress, and the "covers nothing — safe to remove" verdict.

Header-surgery rewrap (MinIO-style) was evaluated and rejected with reasoning in the ADR: on S3 the object must be re-uploaded either way, and the hand-rolled header serialization loses on trust surface, flavour rules, and PQ synergy.

Precedent this follows: Vault transit's rotate→rewrap→raise-min-decryption-version cycle, MinIO's payload-untouched key rotation, NIST SP 800-57's re-encrypt-on-compromise. Sources in the ADR.

Implementation path: #348 builds the scan (now explicitly the drain's first pass), then a follow-up issue for the drain job + ledger + tripwire + admin surface + handbook rewrite. This issue can close once the ADR reads right to you.

**Decided and recorded: [ADR 0002 — rotate, drain, verify, discard](../src/branch/main/docs/adr/0002-key-rotation-drain.md)** (PR #408, operator direction 2026-09-03). The keychain option is rejected outright: no growing retained-key list at the deployment level. Instead the two-key shape (active + one retiring) stays, and rotation becomes a *completable* operation — a managed background drain re-encrypts everything the retiring key covers (through the normal write pipeline, so custody re-resolution and PQ upgrade-on-touch ride along), #348's header scan verifies coverage has reached zero, and only then is the old key discarded. Next rotation is allowed from that state. Answers to the five questions in this issue's brief: 1. **Config surface** — unchanged; the scalar legacy slots are correct for a bounded model. 2. **#138 boundary** — secrets stay in config/env, never state (circularity). The state store gains only a *public-halves* drain ledger, which powers a refuse-to-boot tripwire when an undrained key vanishes from config — today's silent orphaning becomes a named refusal. 3. **Retention** — no unbounded retention by design; drain-then-discard is the policy, #348 makes it provable. 4. **PQ flavour** — homogeneity binds the encrypt-side recipient set only; an active-hybrid/retiring-classical window is legal and the drain resolves it. 5. **Operator model** — Encryption & keys shows the retiring key, drain progress, and the "covers nothing — safe to remove" verdict. Header-surgery rewrap (MinIO-style) was evaluated and rejected with reasoning in the ADR: on S3 the object must be re-uploaded either way, and the hand-rolled header serialization loses on trust surface, flavour rules, and PQ synergy. Precedent this follows: Vault transit's rotate→rewrap→raise-min-decryption-version cycle, MinIO's payload-untouched key rotation, NIST SP 800-57's re-encrypt-on-compromise. Sources in the ADR. Implementation path: #348 builds the scan (now explicitly the drain's first pass), then a follow-up issue for the drain job + ledger + tripwire + admin surface + handbook rewrite. This issue can close once the ADR reads right to you.

Cordy commented

2026-09-03 19:17:48 +00:00

Author

Owner

Closing: the outcome this issue asked for — a recorded decision — is delivered and approved. ADR 0002 (docs/adr/0002-key-rotation-drain.md, PR #408) fixes the bounded two-key model: rotate → drain → verify → discard, at most two keys ever, no growing keychain.

Already real in v0.6.115/116: the verify step (#348 coverage scan) and the Rotation tab lifecycle, exercised on both dogfoods including a live rotation on cairn-openbao and the mixed classical/hybrid window on cairn-enc.

Implementation of the drain (re-encrypt job, ledger, boot tripwire, admin surface) is tracked in #411, including the two flags raised 2026-09-03: ownership preserved by construction + regression test, and locked-file deferral with the approved UI.

Closing: the outcome this issue asked for — a recorded decision — is delivered and approved. ADR 0002 (`docs/adr/0002-key-rotation-drain.md`, PR #408) fixes the bounded two-key model: rotate → drain → verify → discard, at most two keys ever, no growing keychain. Already real in v0.6.115/116: the verify step (#348 coverage scan) and the Rotation tab lifecycle, exercised on both dogfoods including a live rotation on cairn-openbao and the mixed classical/hybrid window on cairn-enc. Implementation of the drain (re-encrypt job, ledger, boot tripwire, admin surface) is tracked in **#411**, including the two flags raised 2026-09-03: ownership preserved by construction + regression test, and locked-file deferral with the approved UI.

Cordy closed this issue

2026-09-03 19:17:51 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#349

No description provided.

Rows
Columns