v0.6.348: storage robustness (#940 S3 read loop, #939 temp sweep + reserved names, #925 reserve on S3 + quota walk) #955

Merged
Cordy merged 29 commits from fix-b1 into main 2026-10-17 15:47:17 +00:00
Owner

Backlog cleanup, batch 1. These items needed no decision and no mockup.

Changes

#940: S3 ranged reads

A ranged read could loop forever, or return wrong bytes. Now:

  • An empty 206, a 416 or a wrong Content-Range ends the read with a visible io.ErrUnexpectedEOF (WARN with reason).
  • When a store ignores Range, Cairn skips forward to the right offset itself.
  • Download marking (internal/marks/embed.go) no longer loops on a short inner object. The reviewer measured about 80,000 GETs in 3 s before this fix.

#939: temp files and reserved names

  • posix data-root sweep. Hidden temps older than 24 h are removed, timed by the storage server's clock. It runs at start and then every 6 h after the previous sweep ends. It:
    • stays on the data root's own file system and skips .snapshot and .zfs;
    • never removes a temp that this server still has open;
    • removes files relative to an open handle on their parent folder.
  • TMPDIR sweep (internal/tempfiles). One helper creates every Cairn temp file (s3 spools, cairn-dec, cairn-drain), and the same rule sweeps them at start and every 6 h.
  • Reserved names (closes a quota bypass):
    • Any name containing .cairn-replace- is refused at the reserved layer on every backend. WebDAV's own parking passes an internal marker.
    • posix refuses .cairn-upload- as a rename or copy target.
    • TUS create/finalize and public upload answer 400.
  • Quota now counts hidden entries. The quota walk lists hidden names through an internal marker, and copyDir skips hidden names.
  • Trash restore of tagged names:
    • A set-aside name (<name>.cairn-replace-<token>) restores to its own name.
    • Any other tagged name is restored with the tag removed and reported as renamed.
    • A tagged home or space root, or a name that would become reserved, is refused as damaged.
    • Flat mode is handled.
  • Other names: space, local, peer, directory account and directory group names may not hold the tag or start with .cairn-. Config users are checked at boot.

#925 item 2: reserveBytes

  • FreeKnown is added. The reserve check now applies only where free space is known (posix), so S3 no longer refuses every write or lists the bucket on each write.
  • A WARN is logged at boot when the reserve can't apply.

#925 item 3: quota usage walk

  • Concurrent lookups share one walk (singleflight).
  • The walk is detached from the requesting call: values are kept, but the caller's cancellation no longer stops it.
  • A cancelled, failed or panicked walk never caches a partial figure.
  • Stale-while-revalidate. The last figure stays in use while a background refresh runs. Credits made during a refresh are kept (per-flight delta), and the TTL counts from the end of the walk.
  • Failure handling:
    • a failed walk logs a WARN and backs off;
    • a hung walk is abandoned after 31 min, with at most 2 walkers per tree;
    • cached figures are evicted after 24 h.

Reviews

  • Each task was reviewed separately.
  • Six fix rounds, each re-reviewed.
  • A final review of the whole branch.
  • A fresh review of the docs, checked against the code.

Tests and docs

  • Tests:
    • All 63 packages pass.
    • -race passes on quota, posix and tempfiles.
    • posix, stacktest, trash, quota, scope and auth also pass as root, since CI runs as root.
  • Docs: CHANGELOG v0.6.348; deployment.md (temporary files and the temp sweep, reserved names); admin-policy.md (reserveBytes, how usage is measured, restore rules); encryption.md; local-accounts.md; spaces-quota.md; ARCHITECTURE.md; README.

Issues

  • Follow-ups filed: #954 (recovery of parked replacements, with the added cases).
  • #953 needs one more run of the Garage probe: P9, the delimiter listing.

Closes #940, closes #939.

Backlog cleanup, batch 1. These items needed no decision and no mockup. ## Changes ### #940: S3 ranged reads A ranged read could loop forever, or return wrong bytes. Now: - An empty 206, a 416 or a wrong `Content-Range` ends the read with a visible `io.ErrUnexpectedEOF` (WARN with `reason`). - When a store ignores `Range`, Cairn skips forward to the right offset itself. - Download marking (`internal/marks/embed.go`) no longer loops on a short inner object. The reviewer measured about 80,000 GETs in 3 s before this fix. ### #939: temp files and reserved names - **posix data-root sweep.** Hidden temps older than 24 h are removed, timed by the storage server's clock. It runs at start and then every 6 h after the previous sweep ends. It: - stays on the data root's own file system and skips `.snapshot` and `.zfs`; - never removes a temp that this server still has open; - removes files relative to an open handle on their parent folder. - **TMPDIR sweep** (`internal/tempfiles`). One helper creates every Cairn temp file (s3 spools, `cairn-dec`, `cairn-drain`), and the same rule sweeps them at start and every 6 h. - **Reserved names** (closes a quota bypass): - Any name containing `.cairn-replace-` is refused at the `reserved` layer on every backend. WebDAV's own parking passes an internal marker. - posix refuses `.cairn-upload-` as a rename or copy target. - TUS create/finalize and public upload answer 400. - **Quota now counts hidden entries.** The quota walk lists hidden names through an internal marker, and `copyDir` skips hidden names. - **Trash restore of tagged names:** - A set-aside name (`<name>.cairn-replace-<token>`) restores to its own name. - Any other tagged name is restored with the tag removed and reported as renamed. - A tagged home or space root, or a name that would become reserved, is refused as damaged. - Flat mode is handled. - **Other names:** space, local, peer, directory account and directory group names may not hold the tag or start with `.cairn-`. Config users are checked at boot. ### #925 item 2: reserveBytes - `FreeKnown` is added. The reserve check now applies only where free space is known (posix), so S3 no longer refuses every write or lists the bucket on each write. - A WARN is logged at boot when the reserve can't apply. ### #925 item 3: quota usage walk - Concurrent lookups share one walk (singleflight). - The walk is detached from the requesting call: values are kept, but the caller's cancellation no longer stops it. - A cancelled, failed or panicked walk never caches a partial figure. - **Stale-while-revalidate.** The last figure stays in use while a background refresh runs. Credits made during a refresh are kept (per-flight delta), and the TTL counts from the end of the walk. - **Failure handling:** - a failed walk logs a WARN and backs off; - a hung walk is abandoned after 31 min, with at most 2 walkers per tree; - cached figures are evicted after 24 h. ## Reviews - Each task was reviewed separately. - Six fix rounds, each re-reviewed. - A final review of the whole branch. - A fresh review of the docs, checked against the code. ## Tests and docs - **Tests:** - All 63 packages pass. - `-race` passes on quota, posix and tempfiles. - posix, stacktest, trash, quota, scope and auth also pass as root, since CI runs as root. - **Docs:** CHANGELOG v0.6.348; `deployment.md` (temporary files and the temp sweep, reserved names); `admin-policy.md` (reserveBytes, how usage is measured, restore rules); `encryption.md`; `local-accounts.md`; `spaces-quota.md`; `ARCHITECTURE.md`; README. ## Issues - Follow-ups filed: #954 (recovery of parked replacements, with the added cases). - #953 needs one more run of the Garage probe: P9, the delimiter listing. Closes #940, closes #939.
posix: RunTempSweeper removes upload temps (.cairn-upload-<16 hex>)
older than 24 h, by the newest of mtime and change time, anywhere under
the data root (version folders included): at once and every 6 h, in the
background after the server serves. Regular files only; no link
followed, no folder, no parked WebDAV replacement removed. One INFO line
per sweep that removed anything, one WARN per failure kind per sweep.

s3: spools are made by createSpool (cairn-s3-upload-<digits> in TMPDIR);
SweepSpools removes those older than 24 h at boot.
- posix Rename refuses an upload temp name as destination, Copy any
  reserved name (ErrInvalidPath, as Write and Mkdir); a parked WebDAV
  replacement name stays allowed for Rename (#496 parks by rename).
- Sweep walks folder by folder through os.Root handles: removes relative
  to the opened folder, skips folders swapped meanwhile, other file
  systems (device id) and .snapshot/.zfs folders; checks ctx per folder.
- Temps this driver has open are held in memory and never swept.
- Age cutoff from the storage server's clock, read from a held probe
  temp; no probe, no sweep (one WARN until a probe works again).
- Next sweep 6 h after the last one ended; one INFO "posix: temp sweep
  done" per sweep; read and remove WARNs once per path per process.
- tempfiles: ctime also on openbsd, dragonfly, solaris/illumos, aix; a
  missing TMPDIR is quiet.
- Unreadable-folder test skips as root (CI).
docs: no long dash in the changed ARCHITECTURE line
All checks were successful
ci / test-and-build (pull_request) Successful in 8m30s
555307fa0b
Cordy merged commit 6fcce1f0af into main 2026-10-17 15:47:17 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#955
No description provided.