v0.6.350: spaces safety (#935 restore a deleted space, #962 delete by identity, #931/#960 rename never adopts or merges, #961 missing folder, #930, #938, space key name) #970

Merged
Cordy merged 84 commits from fix-b3 into main 2026-10-19 04:46:38 +00:00
Owner

Batch 3 of the backlog run. Decisions on #935 and #931 (2026-10-18) and the approved mockup (2026-10-19).

What changes

  • #962 / #935: deleting a space finds its folder through the space's id, so a renamed space's files go to Deleted files and a legal hold refuses the delete (403 legal-hold). The group delete with ?space=delete uses the same path.
    • "Restore space" un-deletes the space: same id, name, owner, settings, limit and folder. It comes back deactivated, with its members restored for review.
    • The name stays reserved while the Deleted files entry exists. Delete forever frees the name and its former names.
    • A restore or purge interrupted by a crash or by another instance is finished or refused, never stranded.
  • #931 / #960: rename convergence records MovingTo and only finishes a move it started itself. It never adopts or merges a folder it did not move.
    • S3 refuses a folder rename onto an existing folder, as POSIX does.
    • The admin row shows "Rename not finished" with the reason, plus "Finish rename…" (audited, shows the folder's facts) and "Retry now".
  • #961: a space whose folder is missing is no longer silently recreated empty. It answers 503 space-folder-missing; admins get a bell item, a row state and "Check again".
  • Space encryption key (found on the way): since #771, a rename convergence split a space's files across two keys. Reads used the new name and silently provisioned a second key; on the dogfoods, recovery identities hid this.
    • The key name is now recorded in the registry and in spacekeys.json.
    • Reads never create a key and also try the folder and former names.
    • A missing key answers 503 key-unavailable.
  • #930, #938: the configured read-only suffix applies to new space names, and three /meta and rename-answer fixes.
  • Peering: delivery resolves the recipient through the registry and never creates a missing or deleted space's folder.
  • UI: the approved mockup scenes 1a to 9b, in four languages.

Process: Tasks A, B, C, D, E and I, each with its own review and fix rounds. Then a final whole-branch review and three final fix rounds, the last review verdict being "ready to release". Every package passes, -race on the batch-3 tests, go vet, gofmt, all web checks, and no U+2014. The SDD trail is in .superpowers/sdd/2026-10-18-batch3/.

Follow-ups filed: #963, #964, #965, #966, #967, #968, #969, plus notes on #742.

Release gate: a visual check of every mockup scene on the live dogfoods after the bump.

Closes #930, #938, #962, #935, #931, #960, #961.

Batch 3 of the backlog run. Decisions on #935 and #931 (2026-10-18) and the approved mockup (2026-10-19). **What changes** - **#962 / #935:** deleting a space finds its folder through the space's id, so a renamed space's files go to Deleted files and a legal hold refuses the delete (403 `legal-hold`). The group delete with `?space=delete` uses the same path. - "Restore space" un-deletes the space: same id, name, owner, settings, limit and folder. It comes back deactivated, with its members restored for review. - The name stays reserved while the Deleted files entry exists. Delete forever frees the name and its former names. - A restore or purge interrupted by a crash or by another instance is finished or refused, never stranded. - **#931 / #960:** rename convergence records `MovingTo` and only finishes a move it started itself. It never adopts or merges a folder it did not move. - S3 refuses a folder rename onto an existing folder, as POSIX does. - The admin row shows "Rename not finished" with the reason, plus "Finish rename…" (audited, shows the folder's facts) and "Retry now". - **#961:** a space whose folder is missing is no longer silently recreated empty. It answers 503 `space-folder-missing`; admins get a bell item, a row state and "Check again". - **Space encryption key (found on the way):** since #771, a rename convergence split a space's files across two keys. Reads used the new name and silently provisioned a second key; on the dogfoods, recovery identities hid this. - The key name is now recorded in the registry and in `spacekeys.json`. - Reads never create a key and also try the folder and former names. - A missing key answers 503 `key-unavailable`. - **#930, #938:** the configured read-only suffix applies to new space names, and three `/meta` and rename-answer fixes. - **Peering:** delivery resolves the recipient through the registry and never creates a missing or deleted space's folder. - **UI:** the approved mockup scenes 1a to 9b, in four languages. **Process:** Tasks A, B, C, D, E and I, each with its own review and fix rounds. Then a final whole-branch review and three final fix rounds, the last review verdict being "ready to release". Every package passes, `-race` on the batch-3 tests, `go vet`, gofmt, all web checks, and no U+2014. The SDD trail is in `.superpowers/sdd/2026-10-18-batch3/`. **Follow-ups filed:** #963, #964, #965, #966, #967, #968, #969, plus notes on #742. **Release gate:** a visual check of every mockup scene on the live dogfoods after the bump. Closes #930, #938, #962, #935, #931, #960, #961.
The S3 driver's folder Rename copied into an existing destination prefix
and merged the two folders; the POSIX driver refuses (Go's rename refuses
any existing destination directory, EEXIST, storage.ErrAlreadyExists).
S3 now checks the destination before copying and again before deleting
the source; something that landed there meanwhile refuses the rename,
the copies are taken back and the source keeps its folder markers.

storage.WithMergeInto lets a caller merge on S3 (the POSIX driver never
merges): the WebDAV overwrite's restore of a parked destination uses it,
so a MOVE that fails half way on S3 still puts the original back as
before. Callers audited: REST rename (now 409 on S3 too), WebDAV MOVE
(parks the destination first; RFC 4918 overwrite unchanged), trash
delete/restore, owner transfer, peering, versions chain moves and space
convergence (next commit).
A tombstone linked to the entry holding the space's folder reserves the
name until Release, whatever KeepUntil says. The members at delete are kept
as a snapshot that grants nothing; Undelete brings the space back
deactivated with them. Release drops the tombstone once the entry is
purged.
A Delete marked with WithSpaceRoot records the space id and backend folder
in the sidecar and reports the entry. Restoring a whole-space entry goes
through the space lifecycle (SpaceLife): the folder goes back to the exact
backend folder it was deleted from and the record is revived, or the
restore is refused with space-gone, space-name-taken or space-folder-taken
and nothing moves. An entry that names no space is never restored as a
bare folder. The space's owner sees and handles someone else's deletion
under the ID d:<spaceID>. Purge, AdminPurge and the sweep release the name.

TestSpaceDeleteWithGuard and TestNumberedRestoreUnchanged change: a whole
space entry naming no space is now refused as space-gone.
deleteSpace and an app group's ?space=delete share one helper, trashSpace:
the backend folder resolved by space id moves into the deleter's Deleted
files as an entry naming the space, a legal hold anywhere inside refuses
with the legal-hold answer, a missing folder logs a WARN and still
tombstones, and the tombstone links the entry (the name stays reserved
while it exists). SpacesAPI.SpaceLife is the trash's view of the space
side: Check (space-gone, space-name-taken, space-folder-taken), Revive
(deactivated, snapshot members back, audited space-restore) and Release
(tombstone dropped and identity retired when the entry is purged). The
trash API answers a refused space restore 409 with its code. main.go wires
it.

TestNoRawBackendDirCompare accepts the one new case-sensitive NFC
comparison (a deleted space comes back to its own folder only).
The rename convergence records the target folder in the persisted
registry before it moves anything; SetBackendDir clears it. Convergence
resumes or finishes only a move whose MovingTo is the space's current
name (next commit).
Decision (Nikola, 2026-10-18): never adopt automatically. Convergence
decides from storage and the persisted registry alone (research-931
section 4(d)): it records MovingTo before moving, resumes its own
half-copied folder (S3, storage.WithMergeInto) or finishes its own
moved folder, and treats a Stat error as unknown, never as missing.
A folder at the new name it did not make, a missing folder, or both
gone: nothing moves, a rate-limited WARN 'spaces: rename not finished'
(reason=folder-exists|folder-missing) and audit space-converge-failed
with the reason. A rename refused before anything reached the new name
(a hold, the driver's ErrAlreadyExists) clears MovingTo, so a later
folder there never passes for its own. A stale MovingTo after a second
rename no longer matches and is ignored.

Administrators (session only, #890; anyone else gets the space 404):
- inventory rows carry renameUnfinished {reason, folder, spaceFolder,
  folderExists}
- POST /api/v1/spaces/{name}/finish-rename {useFolder}: the audited
  adoption (space-converge-adopted), refused 409 rename-finished,
  folder-changed, rename-running, no-folder, folder-claimed,
  old-folder-not-empty
- POST /api/v1/spaces/{name}/retry-rename: one convergence now, answers
  {finished, reason, note}
The second-check refusal deleted every key it had copied. Two movers
of one folder (two instances converging one space) write the same keys,
so the refusing one deleted the other's copies just before that one
deleted the source: every file lost. The copies now stay (the source is
still complete), the source markers go back only when the source is
still complete, and both the refusal and a failed marker restore WARN.
The S3 driver marks a folder rename that failed after part of the
folder reached the destination (storage.ErrPartialMove). The
convergence keeps MovingTo for such a failure and on any doubt; it
clears it only for a refusal (ErrAlreadyExists, a hold) or when a check
of the new name, detached from the expired context, answers not-found.
A real half copy is resumed at the next start or by Retry now instead
of turning into a permanent folder-exists.
Before moving, the convergence records with MovingTo the folder markers
of the space's bound folders (spaceids Rec.MovingMarks). The S3 move
deletes them at the source first; after a crash mid-copy the resume
writes them at the destination before recording the folder, so devices
limited to folders in the space end ok instead of different.
- spaceids Rec.ConvergeFailure: why the last attempt left the rename
  unfinished, written only when it changes, cleared on convergence and
  on a rename; the inventory's renameUnfinished shows it where the plan
  alone cannot (legal-hold, error, a refused case-only rename,
  old-folder-not-empty).
- A hold (SpacesAPI.HoldAffects) stops the attempt before MovingTo is
  recorded: one registry write the first time, none after.
- A resume whose old folder was made again empty removes it (base
  driver, EmptyDirRemover) and finishes; holding files on a driver that
  cannot merge (posix) it is refused as old-folder-not-empty.
- The rename WARN is rate-limited per space and reason.
- A restore writes "restoring" into the entry's sidecar before the
  folder moves. A retry, the boot pass (FinishSpaceRestores) and the
  sweep finish it from what storage holds: folder still in the trash
  (on s3 maybe partly copied already), or already back; a revived space
  only loses its leftover row. Purge, AdminPurge and the sweep refuse
  such an entry (409 space-restoring).
- removeEntry no longer ignores the folder delete: a failure keeps the
  entry, the name and the identity; not-found means a restore took it
  (no release) unless an earlier purge stopped after the folder.
- Restore and purge lock per space id through a versions.PathSerializer
  (main.go: the engine's, the #742 seam), not one global mutex.
- Check accepts the live, deactivated space whose tombstone was never
  written, never names the space itself as another space, and reads a
  missing backend folder as the name (as trashSpace).
Back to the pre-#935 restore (never numbered) for a whole space folder
deleted as a folder when a live space record or a registered folder
holds exactly that folder, or when no space lifecycle is wired (power
mode). Otherwise space-gone, now saying the folder no longer belongs to
a space. TestNumberedRestoreUnchanged asserts ErrAlreadyExists again.
- Only visible files make the old folder "still hold files"; the
  space's trash and version history move into the adopted folder (a
  clash refuses as hidden-data-conflict, nothing moved), and the
  emptied old folder is removed.
- "Another space uses this folder" also covers a store record of that
  name and, for an app-owned space, a directory group in an
  authoritative listing; an unreadable listing answers 503
  directory-unavailable (fails closed, as the take-over).
- A hold on the old folder refuses (legal-hold).
- trashSpace (deleteSpace and the group ?space=delete) refuses 409
  space-rename-running while MovingTo is recorded, a folder sits at both
  the old and the new name, or a convergence of the space runs here; it
  holds the convergence guard while it moves the folder
- restore (spaceLife.Check) refuses space-folder-taken while the
  identity records MovingTo
- the convergence (boot sweep, after a rename) skips a tombstoned space;
  Retry now and Finish rename answer 409 space-deleted
- finishRestore resumes a restore's own partial s3 copy with
  storage.WithMergeInto (#960 refuses an existing target); a target the
  driver refuses drops the intent and answers space-folder-taken
  (review B finding 3, B's merge checklist)
- The restore intent is its own object, <entry>.restoring (with the device
  folder markers found in the trash); a purge writes <entry>.purging. Each
  side writes its flag, then reads the other's, and withdraws on finding
  it: no lock shared across instances. A purge reads the entry again after
  its delete and stops (404, no release) when a restore took the folder.
- finishRestore rolls back only a folder this call moved. A folder at the
  target it did not move is never touched; an entry whose tombstone no
  longer links it is removed (INFO line). A resumed copy merges into the
  target only under the tombstone of this entry and a passing Check.
- A restore whose folder another restore moved finishes that restore.
- A resumed restore writes the recorded device folder markers back (WARN
  when it cannot).
- The sweep finishes only entries no tombstone links: one could-not-finish
  WARN per tick.
- Texts: rolled-back restores say so; space-restoring asks to try again in
  a moment (administrators are told what to do); new code space-purging.
A retried delete then starts clean and links its entry; no entry outlives
a tombstone that does not link it, so the name never frees early. WARN
when the folder cannot go back.
The sweeper reads it on every tick; nothing ordered that read after the
write. A wiring guard checks the order.
Group keys were looked up by the backend folder name (encrypt.GroupFromPath
-> OpenBao groups/<folder>, Keycloak group <folder>). A rename's
convergence (and Finish rename) moves the folder with its ciphertext
unchanged, so on OpenBao custody every file encrypted before the move was
read with a new, lazily provisioned groups/<new name> key afterwards
(unreadable without the recovery identity), and on Keycloak custody a
directory space renamed in Keycloak and relinked could not be read on its
old folder until the move (group <old name> not found). Pre-existing since
#771; reproduces on main 49793042.

Read side only, no custody format change and no key written or moved:
- spaceids: Rec.KeyName records the folder a renamed space leaves when its
  folder moves (SetBackendDir), persisted, never changed after.
  KeyScope maps any folder of a space (its own, or one named like its
  name or a former name) to that key name, plus a directory space's
  current name.
- encrypt: the drivers resolve /spaces/<folder> through SetGroupResolver
  (wired from the registry in main.go and the stacktest rig);
  WithGroupName carries a directory space's current name. Keycloak
  custody looks the key up under the current name first, then the key
  name. cryptosvc forwards it (X-Cairn-Group-Name).
- stacktest: Opts.Keys runs the stack on a per-space custody (fakes).
Only a new space, a restore and a convergence make a space folder. The
identity registry records when a space's folder was first made or seen
(folderMade); the first open of a space whose folder was never made
(a directory space nobody opened yet) still makes it once. A recorded
folder that is gone answers 503 space-folder-missing on REST, WebDAV
and TUS, with one WARN per space per hour. The start pass records the
folders that exist (the upgrade). Decided from storage and the registry
file (#742): a stale instance reads the file before it would make a
folder, and whole-registry writes keep every recorded folder.
qa693: the fact-row confirm count is 23 (Finish rename is a new askDanger).
Review D findings 1-4, 8-11:
- Custodies look a space's key up under every name it may sit under
  (Keycloak: the current name's group, the key name's group; both: the key
  name's other Unicode form, #934) without making anything. A read decrypts
  with every key found and never makes one (no key: a clear error, or the
  recovery identity alone when one is configured, and a rate-limited WARN).
  A write uses the first name holding a key (current name first; a WARN
  when two do) and makes one only when none does (finding 1: a key name's
  cached group no longer answers before the current name; finding 2: a
  relink to an existing group keeps the old group's key; finding 3).
- spaceids.KeyScope reads the registry again on a miss (once per folder
  per 10 s) and reports whether the folder is a known space; the encrypt
  layer marks an unknown one (X-Cairn-Group-Unregistered over cryptosvc)
  and no key is made for it (finding 4, #742). Dot names under /spaces
  (its owner manifest) are scoped as before.
- encrypt.Driver.Open reads the object before the keys: a missing file is
  not found, not a space with no key yet.
- The encrypt layer replaces a group name left in the context (finding 8).
- ByBackendDir uses a folder index rebuilt with the name index (finding 9).
- Tests: the four findings on posix and s3 (stacktest, OpenBao and
  Keycloak fakes, a two-instance test), custody unit tests, C's registry
  writers keeping KeyName (finding 10), the wire header.
- cryptosvc wire-protocol comment lists the group headers (finding 11).
Cordy merged commit c69c6025e1 into main 2026-10-19 04:46:38 +00:00
Author
Owner

Status (2026-10-19, paused). v0.6.350 is merged and tagged, the image is built, and both dogfoods were bumped (homelab-config 630ffd9). Both boot cleanly on v0.6.350:

  • spaces: recorded the folders of existing spaces: 5 on files, 13 on files-bao.
  • No errors.

Still open, when Nikola is back and signed in on both instances:

  1. Live check on a throwaway space, on both instances:
    • rename, converge, then read files (the space key name fix);
    • deactivate, delete, check the Deleted files space row, Restore space (comes back deactivated with members), then reactivate and read;
    • delete again, then Delete forever (the name becomes free);
    • Administration › Spaces and Administration › Deleted files;
    • #938 /meta of a missing name.
  2. Visual check of the mockup scenes (the release gate from the reviews).
  3. Clean up the test data, then post a result comment on each issue: #930, #938, #962, #935, #931, #960, #961.

Next in the backlog run after that: batch 4 (#916, #920, #915, #926 items 3 and 5).

**Status (2026-10-19, paused).** v0.6.350 is merged and tagged, the image is built, and both dogfoods were bumped (homelab-config 630ffd9). Both boot cleanly on v0.6.350: - `spaces: recorded the folders of existing spaces`: 5 on files, 13 on files-bao. - No errors. **Still open, when Nikola is back and signed in on both instances:** 1. **Live check** on a throwaway space, on both instances: - rename, converge, then read files (the space key name fix); - deactivate, delete, check the Deleted files space row, Restore space (comes back deactivated with members), then reactivate and read; - delete again, then Delete forever (the name becomes free); - Administration › Spaces and Administration › Deleted files; - #938 `/meta` of a missing name. 2. **Visual check** of the mockup scenes (the release gate from the reviews). 3. **Clean up** the test data, then post a result comment on each issue: #930, #938, #962, #935, #931, #960, #961. Next in the backlog run after that: batch 4 (#916, #920, #915, #926 items 3 and 5).
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#970
No description provided.