Cordy/Cairn
2
1
Fork 0
Code Issues 20 Pull requests Projects Releases Packages Wiki Activity Actions

Key rotation: managed re-encryption (drain job + ledger + boot tripwire) #411

New issue
Closed
opened 2026-09-03 19:17:41 +00:00 by Cordy · 5 comments
Cordy commented 2026-09-03 19:17:41 +00:00
Owner
Copy link

Implements the drain half of ADR 0002 (docs/adr/0002-key-rotation-drain.md, merged via PR #408). #348 shipped the verify half (coverage scan + Rotation tab); this issue makes the rotate→drain→verify→discard cycle complete without waiting on users to touch files.

Drain job

  • Admin-triggered background job from the Rotation tab (Re-encrypt button in the card slot reserved for it since v0.6.115).
  • Walks the backend at the encrypt layer (below quota/ownermeta/filelock/trash/hold/audit/scope). For each retired-only object: decrypt with the retiring key — which by definition opens the whole drain set — and rewrite through the encrypted write path. That re-resolves the owner's current custody recipient, stamps the new recovery key, and upgrades the flavour to hybrid in the same pass.
  • Checkpointed on the listing cursor (resumable after interruption/restart), throttled (objects-per-second budget, Pi-friendly), convergent (new writes already carry the active key).
  • One key-drain audit event per run, with counts.

Locked-file deferral (decided 2026-09-03)

Never rewrite beneath a live lock — a stale re-encryption racing an Office save could clobber the newer content. Instead:

  • Consult the lock store before each rewrite; skip live-locked files, count them as deferred — locked, in use.
  • UI per the approved mockup (drain-locked-files-mockup.html): amber inline notice during the run ("N files are locked and in use — skipped, not stuck") with an expandable list showing path, lock type and holder; end-of-pass summary splits re-encrypted / deferred (locked) / failed; stepper sub-label counts only the true remainder; hint copy tells the operator not to break locks — deferral converges on its own (a save re-encrypts under the new key; the next pass catches the rest).
  • Verify stays gated until the count is genuinely zero.

Ownership guarantee (flagged 2026-09-03)

Safe by construction, pinned by test:

  • The drain writes below ownermeta, so no owner re-stamping can occur — there is no code path by which the drain could attribute a file to anyone.
  • Owner records are themselves encrypted objects and get drained content-identically (decrypt → re-encrypt of byte-identical plaintext).
  • Regression test required: drain an object plus its owner sidecar, assert the listing's Owner is unchanged.

Safety pieces (per ADR 0002)

  • Drain ledger: public halves of the active and retiring keys recorded in the state store.
  • Boot tripwire: refuse to start if an undrained key vanishes from config — replaces today's silent orphaning.

UI + docs

  • Rotation tab: Re-encrypt button, progress card (stepper step 2 goes live), Recovery ↔ Rotation cross-link.
  • Handbook: drain section + the rotation guide page at the canonical URL the idle tab already links (https://www.swisscairn.ch/docs/encryption#rotation).
  • Copy nits from the #348 screenshots: "1 objects skipped" needs the singular form; root-level group renders as a cryptic /… (2) — give the root group a legible label.

Known cost

Drained objects get a new modified time → sync clients re-download the drained set once. Document it.

Testbeds

Both dogfoods are live drain candidates: cairn-enc has 3 classical retired-only objects (first-ever green discard within reach), cairn-openbao has 18 hybrid ones from the 2026-09 rotation.

Implements the drain half of ADR 0002 (`docs/adr/0002-key-rotation-drain.md`, merged via PR #408). #348 shipped the **verify** half (coverage scan + Rotation tab); this issue makes the rotate→**drain**→verify→discard cycle complete without waiting on users to touch files. ## Drain job - Admin-triggered background job from the Rotation tab (**Re-encrypt** button in the card slot reserved for it since v0.6.115). - Walks the backend at the **encrypt layer** (below quota/ownermeta/filelock/trash/hold/audit/scope). For each retired-only object: decrypt with the **retiring** key — which by definition opens the whole drain set — and rewrite through the encrypted write path. That re-resolves the owner's current custody recipient, stamps the new recovery key, and upgrades the flavour to hybrid in the same pass. - **Checkpointed** on the listing cursor (resumable after interruption/restart), **throttled** (objects-per-second budget, Pi-friendly), **convergent** (new writes already carry the active key). - One `key-drain` audit event per run, with counts. ## Locked-file deferral (decided 2026-09-03) Never rewrite beneath a live lock — a stale re-encryption racing an Office save could clobber the newer content. Instead: - Consult the lock store before each rewrite; skip live-locked files, count them as **deferred — locked, in use**. - UI per the approved mockup (`drain-locked-files-mockup.html`): amber inline notice during the run ("N files are locked and in use — skipped, not stuck") with an expandable list showing path, lock type and holder; end-of-pass summary splits **re-encrypted / deferred (locked) / failed**; stepper sub-label counts only the true remainder; hint copy tells the operator not to break locks — deferral converges on its own (a save re-encrypts under the new key; the next pass catches the rest). - Verify stays gated until the count is genuinely zero. ## Ownership guarantee (flagged 2026-09-03) Safe by construction, pinned by test: - The drain writes below `ownermeta`, so no owner re-stamping can occur — there is no code path by which the drain could attribute a file to anyone. - Owner records are themselves encrypted objects and get drained **content-identically** (decrypt → re-encrypt of byte-identical plaintext). - **Regression test required:** drain an object plus its owner sidecar, assert the listing's Owner is unchanged. ## Safety pieces (per ADR 0002) - **Drain ledger:** public halves of the active and retiring keys recorded in the state store. - **Boot tripwire:** refuse to start if an undrained key vanishes from config — replaces today's silent orphaning. ## UI + docs - Rotation tab: Re-encrypt button, progress card (stepper step 2 goes live), Recovery ↔ Rotation cross-link. - Handbook: drain section + the rotation guide page at the canonical URL the idle tab already links (`https://www.swisscairn.ch/docs/encryption#rotation`). - Copy nits from the #348 screenshots: "1 object**s** skipped" needs the singular form; root-level group renders as a cryptic `/… (2)` — give the root group a legible label. ## Known cost Drained objects get a new modified time → sync clients re-download the drained set once. Document it. ## Testbeds Both dogfoods are live drain candidates: **cairn-enc** has 3 classical retired-only objects (first-ever green discard within reach), **cairn-openbao** has 18 hybrid ones from the 2026-09 rotation.
Cordy closed this issue 2026-09-03 20:05:10 +00:00
Cordy commented 2026-09-03 20:12:40 +00:00
Author
Owner
Copy link

Shipped in v0.6.117 (PR #412, merged; both dogfoods rolled and healthy).

Design decisions recorded 2026-09-03

  • Drain placement: rewrites go through the in-process encrypt decorator captured below quota/ownermeta/filelock — ownership preservation is by construction (no code path can re-stamp an owner; owner sidecars are drained content-identically) and pinned by TestDrainMovesRetiredOnly's sidecar assertion.
  • Per-user custody: the drain synthesises auth.ContextWithUser from the /home/<user>/ segment, so keycloak-profile and OpenBao custody re-resolve the owner's current recipient — and PQ upgrade-on-touch rides along, exactly as ADR 0002 promised.
  • Locked files (Nikola's flag): never rewritten (a stale rewrite could race a client save) — deferred with path/type/holder via filelock.Covering, shown in the amber notice + expandable list from the approved drain-locked-files-mockup.html; copy says outright that nothing is stuck and locks must not be broken.
  • State objects: re-seal via the statestore (Load with retained identities, Save to current recipients) — never through the content path, so state is never double-wrapped.
  • No persisted checkpoint: the pass is idempotent (classification skips already-active objects), so pause/resume/restart is simply running it again.
  • Ledger + tripwire: key-rotation.json beside the other runtime state (statestore-bound on s3); scan-proven zero marks entries drained; an undrained key vanishing from config refuses boot; CAIRN_KEYDRAIN_FORCE=1 is the documented escape hatch. Handbook section rewritten.

Operator pass wanted: both dogfoods are live testbeds — cairn-enc has 3 classical retired-only objects (one Re-encrypt press away from the first-ever verified discard), cairn-openbao ~18 hybrid. Deploys wiped sessions; fresh login needed.

**Shipped in v0.6.117** (PR #412, merged; both dogfoods rolled and healthy). **Design decisions recorded 2026-09-03** - *Drain placement:* rewrites go through the in-process encrypt decorator captured below quota/ownermeta/filelock — ownership preservation is by construction (no code path can re-stamp an owner; owner sidecars are drained content-identically) and pinned by `TestDrainMovesRetiredOnly`'s sidecar assertion. - *Per-user custody:* the drain synthesises `auth.ContextWithUser` from the `/home/<user>/` segment, so keycloak-profile and OpenBao custody re-resolve the owner's current recipient — and PQ upgrade-on-touch rides along, exactly as ADR 0002 promised. - *Locked files (Nikola's flag):* never rewritten (a stale rewrite could race a client save) — deferred with path/type/holder via `filelock.Covering`, shown in the amber notice + expandable list from the approved `drain-locked-files-mockup.html`; copy says outright that nothing is stuck and locks must not be broken. - *State objects:* re-seal via the statestore (Load with retained identities, Save to current recipients) — never through the content path, so state is never double-wrapped. - *No persisted checkpoint:* the pass is idempotent (classification skips already-active objects), so pause/resume/restart is simply running it again. - *Ledger + tripwire:* `key-rotation.json` beside the other runtime state (statestore-bound on s3); scan-proven zero marks entries drained; an undrained key vanishing from config refuses boot; `CAIRN_KEYDRAIN_FORCE=1` is the documented escape hatch. Handbook section rewritten. **Operator pass wanted:** both dogfoods are live testbeds — cairn-enc has 3 classical retired-only objects (one Re-encrypt press away from the first-ever verified discard), cairn-openbao ~18 hybrid. Deploys wiped sessions; fresh login needed.
Cordy commented 2026-09-03 20:30:08 +00:00
Author
Owner
Copy link

v0.6.118 (PR #413, dogfoods rolled): first real drain on cairn-enc surfaced two things.

  1. Principal-less paths — /test/cairn-handoff.md, /cairn-fix-first.md, /sovereignty-check.txt are pre-per-user-homes relics at the backend root; per-user custody has no principal to resolve, so the encrypt layer refused. They now re-encrypt directly to the active deployment-level recipients (Drainer.Fallback): such objects open only via the retiring recovery key today, so this preserves exactly their current access, on the new key.
  2. Bogus-principal edge — a file directly under /home no longer resolves its own name as a username, so per-user custody can never be asked to mint IdP keys for a principal that does not exist.

Open observation: /home/Mülltrennung.docx lists but does not open (not found) — the same object the scan counts as "skipped". Likely a unicode-normalisation ghost (NFD vs NFC umlaut in the S3 key). Not a drain problem; if it is not visible/deletable in the UI, it becomes its own issue against the s3 driver.

**v0.6.118** (PR #413, dogfoods rolled): first real drain on cairn-enc surfaced two things. 1. *Principal-less paths* — `/test/cairn-handoff.md`, `/cairn-fix-first.md`, `/sovereignty-check.txt` are pre-per-user-homes relics at the backend root; per-user custody has no principal to resolve, so the encrypt layer refused. They now re-encrypt **directly to the active deployment-level recipients** (`Drainer.Fallback`): such objects open only via the retiring recovery key today, so this preserves exactly their current access, on the new key. 2. *Bogus-principal edge* — a file directly under `/home` no longer resolves its own name as a username, so per-user custody can never be asked to mint IdP keys for a principal that does not exist. **Open observation:** `/home/Mülltrennung.docx` lists but does not open (`not found`) — the same object the scan counts as "skipped". Likely a unicode-normalisation ghost (NFD vs NFC umlaut in the S3 key). Not a drain problem; if it is not visible/deletable in the UI, it becomes its own issue against the s3 driver.
Cordy commented 2026-09-03 20:52:01 +00:00
Author
Owner
Copy link

v0.6.119 (PR #415, dogfoods rolled): a finished drain pass now hands straight to verification — OnDone (on completion, not on pause) starts the coverage scan, so the Rotation card refreshes to the true remaining count instead of the stale pre-pass figure, and a zero result marks the ledger drained without any further click.

Found on cairn-openbao: the pass moved all 18 objects, but the headline (fed by the last scan) kept saying “18 objects remaining”, and a second Re-encrypt press just reported 0/0/0 until a manual Rescan. Decision recorded: auto-verify instead of a “Continue” button — the flow self-advances and the ledger benefits as a side effect.

Dogfood state: both instances reached the green “Covers nothing” verdict — cairn-enc (3 classical drained; the #414 ghost counts as skipped) and cairn-openbao (18 hybrid drained). Both are cleared for the discard step: remove recoveryIdentityLegacy from the untracked config Secret, restart, expect a clean boot with the ledger retiring the entry.

**v0.6.119** (PR #415, dogfoods rolled): a finished drain pass now hands straight to verification — `OnDone` (on completion, not on pause) starts the coverage scan, so the Rotation card refreshes to the true remaining count instead of the stale pre-pass figure, and a zero result marks the ledger drained without any further click. Found on cairn-openbao: the pass moved all 18 objects, but the headline (fed by the last scan) kept saying “18 objects remaining”, and a second Re-encrypt press just reported 0/0/0 until a manual Rescan. Decision recorded: auto-verify instead of a “Continue” button — the flow self-advances and the ledger benefits as a side effect. **Dogfood state:** both instances reached the green “Covers nothing” verdict — cairn-enc (3 classical drained; the #414 ghost counts as skipped) and cairn-openbao (18 hybrid drained). Both are cleared for the discard step: remove `recoveryIdentityLegacy` from the untracked config Secret, restart, expect a clean boot with the ledger retiring the entry.
Cordy commented 2026-09-03 21:15:49 +00:00
Author
Owner
Copy link

v0.6.120 (PR #416, dogfoods rolled): the tripwire (and the ledger-open failure path) now cancels the heartbeat and releases the state-backend instance lock before exiting. Found live during cairn-enc's first discard: every refused boot stranded the lock, so the recovery boot (key restored or CAIRN_KEYDRAIN_FORCE=1) sat in CrashLoopBackOff for the full 5-minute stale window.

And for the record — the tripwire's first live catch was a genuine one: the cairn-enc discard was attempted without a post-drain verified scan (the drain had moved all 3 objects, but the last completed scan still said 3 covered; the operator's rescan that day was on cairn-openbao). Boot refused exactly as designed; recovery via the documented CAIRN_KEYDRAIN_FORCE=1 escape hatch; ledger entry dropped with the loud warning. v0.6.119's auto-verify makes this scenario much harder to reproduce, since a finished pass now runs the scan itself.

**v0.6.120** (PR #416, dogfoods rolled): the tripwire (and the ledger-open failure path) now cancels the heartbeat and releases the state-backend instance lock before exiting. Found live during cairn-enc's first discard: every refused boot stranded the lock, so the recovery boot (key restored or `CAIRN_KEYDRAIN_FORCE=1`) sat in CrashLoopBackOff for the full 5-minute stale window. **And for the record — the tripwire's first live catch was a genuine one:** the cairn-enc discard was attempted without a post-drain verified scan (the drain had moved all 3 objects, but the last *completed* scan still said 3 covered; the operator's rescan that day was on cairn-openbao). Boot refused exactly as designed; recovery via the documented `CAIRN_KEYDRAIN_FORCE=1` escape hatch; ledger entry dropped with the loud warning. v0.6.119's auto-verify makes this scenario much harder to reproduce, since a finished pass now runs the scan itself.
Cordy commented 2026-09-03 22:55:05 +00:00
Author
Owner
Copy link

Closing on operator sign-off: rotate → drain → verify → discard completed end-to-end on both dogfoods (cairn-enc: 3 classical objects incl. the first-ever tripwire catch and escape-hatch recovery; cairn-openbao: 18 hybrid objects through the clean happy path). Shipped across v0.6.117–120 with three dogfound fixes along the way. Follow-ups live in #414 (s3 driver unicode roundtrip).

Closing on operator sign-off: rotate → drain → verify → discard completed end-to-end on both dogfoods (cairn-enc: 3 classical objects incl. the first-ever tripwire catch and escape-hatch recovery; cairn-openbao: 18 hybrid objects through the clean happy path). Shipped across v0.6.117–120 with three dogfound fixes along the way. Follow-ups live in #414 (s3 driver unicode roundtrip).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
scout-654
fix-escguard
fix-vard
fix-middot
fix-552d
fix-552c
fix-543
scout-tf
v0.6.265
v0.6.264
v0.6.263
v0.6.262
v0.6.261
v0.6.260
v0.6.259
v0.6.258
v0.6.257
v0.6.256
v0.6.255
v0.6.254
v0.6.253
v0.6.252
v0.6.251
v0.6.250
v0.6.249
v0.6.248
v0.6.247
v0.6.246
v0.6.245
v0.6.244
v0.6.243
v0.6.242
v0.6.241
v0.6.240
v0.6.239
v0.6.238
v0.6.237
v0.6.236
v0.6.235
v0.6.234
v0.6.233
v0.6.232
v0.6.231
v0.6.230
v0.6.229
v0.6.228
v0.6.227
v0.6.226
v0.6.225
v0.6.224
v0.6.223
v0.6.222
v0.6.221
v0.6.220
v0.6.219
v0.6.218
v0.6.217
v0.6.216
v0.6.215
v0.6.214
v0.6.213
v0.6.212
v0.6.211
v0.6.210
v0.6.209
v0.6.208
v0.6.207
v0.6.206
v0.6.205
v0.6.204
v0.6.203
v0.6.202
v0.6.201
v0.6.200
v0.6.199
v0.6.198
v0.6.197
v0.6.196
v0.6.195
v0.6.194
v0.6.193
v0.6.192
v0.6.191
v0.6.190
v0.6.189
v0.6.188
v0.6.187
v0.6.186
v0.6.185
v0.6.184
v0.6.183
v0.6.182
v0.6.181
v0.6.180
v0.6.179
v0.6.178
v0.6.177
v0.6.176
v0.6.175
v0.6.174
v0.6.173
v0.6.172
v0.6.171
v0.6.170
v0.6.169
v0.6.168
v0.6.167
v0.6.166
v0.6.165
v0.6.164
v0.6.163
v0.6.162
v0.6.161
v0.6.160
v0.6.159
v0.6.158
v0.6.157
v0.6.156
v0.6.155
v0.6.154
v0.6.153
v0.6.152
v0.6.151
v0.6.150
v0.6.149
v0.6.148
v0.6.147
v0.6.146
v0.6.145
v0.6.144
v0.6.143
v0.6.142
v0.6.141
v0.6.140
v0.6.139
v0.6.138
v0.6.137
v0.6.136
v0.6.135
v0.6.134
v0.6.133
v0.6.132
v0.6.131
v0.6.130
v0.6.129
v0.6.128
v0.6.127
v0.6.126
v0.6.125
v0.6.124
v0.6.123
v0.6.122
v0.6.121
v0.6.120
v0.6.119
v0.6.118
v0.6.117
v0.6.116
v0.6.115
v0.6.114
v0.6.113
v0.6.112
v0.6.111
v0.6.110
v0.6.109
v0.6.108
v0.6.107
v0.6.106
v0.6.105
v0.6.104
v0.6.103
v0.6.102
v0.6.101
v0.6.100
v0.6.99
v0.6.98
v0.6.97
v0.6.96
v0.6.95
v0.6.94
v0.6.93
v0.6.92
v0.6.91
v0.6.90
v0.6.89
v0.6.88
v0.6.87
v0.6.86
v0.6.85
v0.6.84
v0.6.83
v0.6.82
v0.6.81
v0.6.80
v0.6.79
v0.6.78
v0.6.77
v0.6.76
v0.6.75
v0.6.74
v0.6.73
v0.6.72
v0.6.71
v0.6.70
v0.6.69
v0.6.68
v0.6.67
v0.6.66
v0.6.65
v0.6.64
v0.6.63
v0.6.62
v0.6.61
v0.6.60
v0.6.59
v0.6.58
v0.6.57
v0.6.56
v0.6.55
v0.6.54
v0.6.53
v0.6.52
v0.6.51
v0.6.50
v0.6.49
v0.6.48
v0.6.47
v0.6.46
v0.6.45
v0.6.44
v0.6.43
v0.6.42
v0.6.41
v0.6.40
v0.6.39
v0.6.38
v0.6.37
v0.6.36
v0.6.35
v0.6.34
v0.6.33
v0.6.32
v0.6.31
v0.6.30
v0.6.29
v0.6.28
v0.6.25
v0.6.24
v0.6.23
v0.6.22
v0.6.21
v0.6.20
v0.6.19
v0.6.18
v0.6.17
v0.6.16
v0.6.15
v0.6.14
v0.6.12
v0.6.11
v0.6.10
v0.6.9
v0.6.8
v0.6.7
v0.6.6
v0.6.5
v0.6.4
v0.6.3
v0.6.2
v0.6.1
v0.6.0
v0.5.7
v0.5.6
v0.5.5
v0.5.4
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.12
v0.4.11
v0.4.10
v0.4.9
v0.4.8
v0.4.7
v0.4.6
v0.4.5
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.30
v0.3.29
v0.3.28
v0.3.27
v0.3.26
v0.3.25
v0.3.24
v0.3.23
v0.3.22
v0.3.21
v0.3.20
v0.3.19
v0.3.18
v0.3.17
v0.3.16
v0.3.15
v0.3.14
v0.3.13
v0.3.12
v0.3.11
v0.3.10
v0.3.9
v0.3.8
v0.3.7
v0.3.6
v0.3.5
v0.3.4
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.0.3
v0.0.2
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
v0.7 — Closed beta (dogfood + hardening)
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#411
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?