v0.6.361: storage robustness and resumable uploads #1027

Merged
Cordy merged 17 commits from r45 into main 2026-10-27 12:22:46 +00:00
Owner

Storage (R4):

  • #956: refuse S3 stores without CopyObject, at start and in the setup wizard.
  • #925 items 1 and 4: "too large for the storage" is a clear 413. S3 folder moves that would not fit are refused before the first copy.
  • #954: a WebDAV overwrite interrupted by a crash never hides the only copy. Recovery runs at start and hourly, and a name deleted after the park comes back as "(recovered)".
  • #980: S3 no longer creates a missing folder silently.
  • #969: long S3 rename copies converge, and half copies are reported. The Remove button follows in #1026.

Resumable uploads (decisions on #742, 2026-10-27):

  • Uploads run in parallel.
  • Finishing survives a disconnect and reports finishing or finished.
  • The spool is sealed with the state keys.
  • The declared size is reserved against quota, and released after 15 minutes without progress.
  • Deletes always pass the instance cap.

Review: per task (R4 had 2 blockers, fixed), the uploads task (6 should-fix, fixed), and the whole branch (one should-fix, fixed). The texts were approved by Nikola on 2026-10-27.

Closes #956, closes #925, closes #954, closes #980, closes #969.

**Storage (R4):** - #956: refuse S3 stores without CopyObject, at start and in the setup wizard. - #925 items 1 and 4: "too large for the storage" is a clear 413. S3 folder moves that would not fit are refused before the first copy. - #954: a WebDAV overwrite interrupted by a crash never hides the only copy. Recovery runs at start and hourly, and a name deleted after the park comes back as "(recovered)". - #980: S3 no longer creates a missing folder silently. - #969: long S3 rename copies converge, and half copies are reported. The Remove button follows in #1026. **Resumable uploads** (decisions on #742, 2026-10-27): - Uploads run in parallel. - Finishing survives a disconnect and reports finishing or finished. - The spool is sealed with the state keys. - The declared size is reserved against quota, and released after 15 minutes without progress. - Deletes always pass the instance cap. **Review:** per task (R4 had 2 blockers, fixed), the uploads task (6 should-fix, fixed), and the whole branch (one should-fix, fixed). The texts were approved by Nikola on 2026-10-27. Closes #956, closes #925, closes #954, closes #980, closes #969.
F1: one lock per upload, never held while a body is read or a file is
stored; a newer PATCH of the same upload ends the older request
(draft-14 take-over) and only its pieces commit.
F2: storing the finished upload runs detached from the request (bounded,
2 h); HEAD answers 503 upload-finishing, then 200 upload-finished with
the ETag; the final PATCH waits up to 20 s, else answers finishing; a
repeat of the final PATCH answers 204. The web client waits for
finished instead of restarting or calling the upload expired.
F3: the spool is a versioned record plus write-once pieces, sealed with
age to the state keys (recovery, deployment) when encryption is on, or
to a key held in memory when the process holds neither; a pre-upgrade
plaintext spool resumes as the upload's first piece.
F5: the declared size is reserved in the quota driver at create (507
quota-exceeded when it does not fit), counted by Room and by every
other write, released at finish, termination and expiry, restored at a
restart.
S1: a storing refusal that arrives after the final PATCH stopped waiting
is kept in the record (phase refused, status and code); HEAD carries its
X-Cairn-Code at the last piece's offset, the next PATCH there answers it
once and clears it. The web client stops with it after one such round.
S2: the web room check leaves out files this browser resumes.
S3: a reservation without progress for 15 minutes is released (Sweep,
every 5 minutes with reservations); the next PATCH reserves again and is
refused with quota-exceeded when it no longer fits.
S4: the finished record is saved before the pieces go; a failed save is
a WARN and keeps the pieces.
S5: the web client sends a best-effort DELETE before it drops a resume
URL (every drop except gone).
S6: records are "<id>.rec" (v0.6.360 reads only "<id>.json" and answers
404 after a rollback); a legacy sidecar is renamed when adopted.
N1: the finish bound is 2 h plus the length at 5 MB/s.
N2: state keys age cannot seal to together refuse the start.
N3: test for a piece written but never committed.
N4: Sweep expires a writer with no byte for Expiry.
B1 (#925 item 4): a folder delete (move into a trash tree) is not refused
by the version key check; the driver's raw key limit still applies.
B2 (#954): a parked entry whose name was deleted after the park goes
beside it as "(recovered)" (trash.DeletedSince); Run passes once more
after ParkGrace. Parks are aged by their newest key too (long S3 parks,
pre-#954 random tokens).
S1: S3 ParkedEntries walks folder by folder, page by page, never listing
.cairn-* or trash trees; passes run where the group pass lease says
(grouprefs.LeaseHere).
S2: trash and statestore folder caches remember a folder only once it
exists and make it again when a write finds it gone.
S3: copies-left is the space's own target, no half copy until renamed away.
S4: a drop whose shared folder vanished answers 404 with a WARN.
S5: alias retire answers 503 when a half copy cannot be checked.
S6 (#956): the setup wizard's storage check runs the CopyObject probe
(code store-no-copy).
Nits: convergence reason name-too-long; idle error names its bound.
S1 quota: a write into a trash tree skips the instance cap, so a delete
works when reservations fill the instance.
Nit 1 tus finish releases the reservation right after the write.
Nit 2 finish keeps a refusal itself when its request stopped waiting.
Nit 3 tus create refuses a target whose folder is missing (404 not-found);
the web app words it.
Nit 5/6 posix parked search skips the root .cairn folder; an unreadable
folder is skipped (silently on access denied, else one INFO per folder).
Docs: CHANGELOG, deployment.md, spaces-quota.md (nits 7, 8, 10).
Root cause (pre-existing since the CI-flake review 2 ForgetMove, same on
9ccc20a9): A records the move, B joins it and copies first; A's copy is
refused before its first key (folder-exists: B's copies are at the new
name) and A forgets the MovingTo it wrote. ForgetMove is a whole-registry
read-modify-write: read before B's SetBackendDir, written after it, it
overwrote the finished move (last writer wins, no compare-and-swap, ADR
D2/#156, spaceids.changeDiskLocked). End state: files all in Stray, the
registry Tiny with no MovingTo and no reason; a fresh instance refuses
folder-missing and Finish rename converges it, as for the copies-left
lost update the test already accepted.

- TestConvergeLostUpdateForgetMove931 reproduces the interleaving
  deterministically (a hooked registry state store orders A's stale read,
  B's SetBackendDir, A's write) and checks the end state, the fresh
  instance's folder-missing row, Finish rename and the converged space.
- lostUpdate931 accepts this end state too (one finished, the other
  folder-exists; old folder, no MovingTo, no reason), nothing else;
  TestLostUpdateSignature931 covers it.
Cordy merged commit a1f32f4817 into main 2026-10-27 12:22:46 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1027
No description provided.