Storage: hidden temp files left by a crash are never cleaned up (incl. inside version folders) #939

Closed
opened 2026-10-16 19:08:32 +00:00 by Cordy · 2 comments
Owner

Found in the fix-922 Tasks 3-4 review (2026-10-16).

What happens: posix writes, snapshots and records go through hidden temp names (.cairn-upload-*, and hidden link names in record mode). A crash between creating the temp file and the final rename or link leaves the temp behind. That was already true for Write. Since #922 such leftovers can also sit inside .cairn-versions folders and hold full copies or extra hard links. SizeByPrefix doesn't count them, so they use disk space that quotas and size reports never show.

To do:

  • Add a sweeper, at boot and periodically, that removes hidden temp names older than a safe age (e.g. 24 h), with one log line per sweep.
  • Check the same for s3 spools.
  • Test it with planted leftovers.
Found in the fix-922 Tasks 3-4 review (2026-10-16). **What happens:** posix writes, snapshots and records go through hidden temp names (`.cairn-upload-*`, and hidden link names in record mode). A crash between creating the temp file and the final rename or link leaves the temp behind. That was already true for `Write`. Since #922 such leftovers can also sit inside `.cairn-versions` folders and hold full copies or extra hard links. `SizeByPrefix` doesn't count them, so they use disk space that quotas and size reports never show. **To do:** - Add a sweeper, at boot and periodically, that removes hidden temp names older than a safe age (e.g. 24 h), with one log line per sweep. - Check the same for s3 spools. - Test it with planted leftovers.
Cordy closed this issue 2026-10-17 15:47:21 +00:00
Author
Owner

Shipped in v0.6.348 (PR #955), running on both dogfoods.

Sweep inside the posix data root. Hidden upload temps older than 24 h are removed:

  • When: at start, then 6 h after each sweep finishes.
  • Age: measured on the storage server's clock.
  • Safety: temps this server still has open are never touched. The sweep stays on the data root's own filesystem, skips .snapshot and .zfs, and removes only through a handle on the parent folder.
  • On files: the first sweep logged posix: temp sweep done folders=19 removed=0 bytes=0 duration=43ms.

Sweep of the temp folder (TMPDIR). All three temp kinds, S3 upload spools (cairn-s3-upload-*), decrypted plaintext (cairn-dec-*) and key-drain files (cairn-drain-*), are now made by one helper (internal/tempfiles) and swept at start and every 6 h.

Found during review and fixed in the same release:

  • Quota bypass. A user could hide a file from quota counting by moving it to a hidden name. Fixed in three parts:
    • .cairn-replace- names are now reserved on every backend.
    • The quota walk now counts hidden entries.
    • Folder copies no longer carry hidden entries.
  • Reserved names. Space, account, peer and directory group names may no longer contain the tag or start with .cairn-.
  • Restoring tagged names. Deleted-files entries with a tagged name are restored safely.

Follow-up: set-aside WebDAV entries (.cairn-replace-*) left behind by a crash are never swept, because one may be the only copy of a file. Recovering them is tracked in #954.

**Shipped in v0.6.348 (PR #955), running on both dogfoods.** **Sweep inside the posix data root.** Hidden upload temps older than 24 h are removed: - **When:** at start, then 6 h after each sweep finishes. - **Age:** measured on the storage server's clock. - **Safety:** temps this server still has open are never touched. The sweep stays on the data root's own filesystem, skips `.snapshot` and `.zfs`, and removes only through a handle on the parent folder. - **On files:** the first sweep logged `posix: temp sweep done folders=19 removed=0 bytes=0 duration=43ms`. **Sweep of the temp folder (`TMPDIR`).** All three temp kinds, S3 upload spools (`cairn-s3-upload-*`), decrypted plaintext (`cairn-dec-*`) and key-drain files (`cairn-drain-*`), are now made by one helper (`internal/tempfiles`) and swept at start and every 6 h. **Found during review and fixed in the same release:** - **Quota bypass.** A user could hide a file from quota counting by moving it to a hidden name. Fixed in three parts: - `.cairn-replace-` names are now reserved on every backend. - The quota walk now counts hidden entries. - Folder copies no longer carry hidden entries. - **Reserved names.** Space, account, peer and directory group names may no longer contain the tag or start with `.cairn-`. - **Restoring tagged names.** Deleted-files entries with a tagged name are restored safely. **Follow-up:** set-aside WebDAV entries (`.cairn-replace-*`) left behind by a crash are never swept, because one may be the only copy of a file. Recovering them is tracked in #954.
Author
Owner

Live result:

  • Temp sweep: both starts on files (posix) log posix: temp sweep done. On v0.6.350 today it logged folders=20 removed=0: no leftover temps.
  • Reserved names (v0.6.349 live check, PR #959), on both dogfoods: an upload to x.cairn-replace-abc was refused with 400 invalid path, and so was a rename to a.txt.cairn-replace-abc.

The quota count of hidden names is covered by tests only; it needs a leftover temp file to show.

**Live result:** - **Temp sweep:** both starts on files (posix) log `posix: temp sweep done`. On v0.6.350 today it logged `folders=20 removed=0`: no leftover temps. - **Reserved names (v0.6.349 live check, PR #959), on both dogfoods:** an upload to `x.cairn-replace-abc` was refused with 400 `invalid path`, and so was a rename to `a.txt.cairn-replace-abc`. The quota count of hidden names is covered by tests only; it needs a leftover temp file to show.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#939
No description provided.