WebDAV MOVE/COPY onto an existing file destroys it without a version or a trash entry #908

Closed
opened 2026-10-15 09:55:35 +00:00 by Cordy · 7 comments
Owner

Found during the #900 research (comment 9164), by reading the code; not yet confirmed by a test.

internal/dav/dav.go:493-507 with internal/storage/versions/versions.go:467 and :502: a WebDAV MOVE or COPY with Overwrite: T onto an existing file replaces it with no version of the old content and no Deleted files entry, for every client (not only devices). That is silent data loss, and it contradicts what users expect from versioning (#541).

Also from the same research (versions hardening, needed before any "never destroy" mode for #900):

  • versions are pruned by count at write time (default 10, versions.go:283), so 11 quick overwrites remove every clean version (NCSC principle 3 asks for time-based retention);
  • version saving fails open: a write still goes through when saving the version fails (versions.go:235).

To do: failing tests first through the real stack (WebDAV MOVE and COPY with overwrite, REST copy/rename with overwrite), then the fix (version or trash the replaced file), then decide on the two hardening points.

Found during the #900 research (comment 9164), by reading the code; not yet confirmed by a test. `internal/dav/dav.go:493-507` with `internal/storage/versions/versions.go:467` and `:502`: a WebDAV MOVE or COPY with `Overwrite: T` onto an existing file replaces it with no version of the old content and no Deleted files entry, for every client (not only devices). That is silent data loss, and it contradicts what users expect from versioning (#541). Also from the same research (versions hardening, needed before any "never destroy" mode for #900): - versions are pruned by count at write time (default 10, `versions.go:283`), so 11 quick overwrites remove every clean version (NCSC principle 3 asks for time-based retention); - version saving fails open: a write still goes through when saving the version fails (`versions.go:235`). To do: failing tests first through the real stack (WebDAV MOVE and COPY with overwrite, REST copy/rename with overwrite), then the fix (version or trash the replaced file), then decide on the two hardening points.
Author
Owner

Deep research on 908-3: fail closed for every overwrite, or only for restore? (2026-10-15)

Nikola asked why the earlier research recommended the narrow option. Full file: cairn-work/scratch/b7e/batch907/908-3-failclosed-research.md.

Where the narrow conclusion came from:

  • An inherited code comment. Both research files took the comment at versions.go:234-237 as settled ("a mint failure must never fail the write"). VERSIONING.md never discusses a failed mint.
  • Issue scope. #900 said "at least for limited devices" and handed the general question on.
  • One real argument: refusing a save hurts some clients. Collabora gives up after 5 failed saves when the last user leaves, and rclone does not retry a failed PUT but deletes the target. In Cairn that goes to trash with its history, so it is confusing but nothing is lost.

Hypotheses checked against the code:

  • Quota: versions don't count against quota, so a nearly full user can still save.
  • Blast radius: small; version folders sit beside the data.
  • Correlated failure: mostly true, but there are five exceptions where only the version save fails:
    • S3 errors (the S3 client never retries, and each mint adds 6-8 requests);
    • a wrong-owner .cairn-versions folder;
    • S3's 1024-byte key limit;
    • a disk-space window where the new file fits but old plus new does not;
    • the 99-per-second key limit.
  • Mass overwrite (ransomware, a bulk script): version copies fill the disk during the event. From then on, fail open overwrites clean files with no history. This is the strongest case for fail closed.

Precedent:

  • Fail closed by design: systems with built-in history (Seafile, S3 versioning, copy-on-write filesystems, Git).
  • Fail open: Nextcloud's copy-before-write ignores the copy result, even on restore, and its users complain about silent history loss.
  • Fail closed when full: SharePoint counts versions in quota and goes read-only when full.

Two bugs found (fix regardless):

  1. Autosave coalescing destroys the version from before an edit. Mint replaces the newest version by the same author when it is under 10 minutes old (versions.go:247-254), so the second autosave of an editing session deletes the pre-session version. TestCoalesceReplaces asserts this; admin-policy.md:220 promises the opposite.
  2. TUS marks the upload complete before the final write (tus.go:419-431). A refused finalize (423, 403 today) is then reported as success to the retrying web uploader.

Recommended final rule: "Cairn never replaces a file without keeping its current content as a version; if it cannot, the change is refused and the file is unchanged." Universal, but only once these preconditions ship:

  1. autosaves inside the 10-minute window skip the mint instead of replacing (this keeps the pre-edit version);
  2. no versions of empty files;
  3. server-side Copy for mints plus S3 retries;
  4. the TUS offset fix;
  5. a refused editor save writes a rescue copy beside the file and notifies the user;
  6. refuse creating paths whose version key would exceed the backend limit.

Until the preconditions ship: restore-only, as in the plan's default.

Decisions (my recommendation in brackets):

  • F1 scope [every overwrite, once the preconditions ship; until then restore-only];
  • F2 autosave [skip instead of replace, in batch A regardless];
  • F3 refused editor save [refuse plus rescue copy plus bell];
  • F4 TUS offset bug [fix in batch A];
  • F5 how versions are saved [server-side Copy plus S3 retries];
  • F6 over-long version keys [refuse creation];
  • F7 alerting [metrics plus admin bell].

New strings only, no mockup. About 2-3 extra builder days.

## Deep research on 908-3: fail closed for every overwrite, or only for restore? (2026-10-15) Nikola asked why the earlier research recommended the narrow option. Full file: `cairn-work/scratch/b7e/batch907/908-3-failclosed-research.md`. **Where the narrow conclusion came from:** - **An inherited code comment.** Both research files took the comment at `versions.go:234-237` as settled ("a mint failure must never fail the write"). VERSIONING.md never discusses a failed mint. - **Issue scope.** #900 said "at least for limited devices" and handed the general question on. - **One real argument:** refusing a save hurts some clients. Collabora gives up after 5 failed saves when the last user leaves, and rclone does not retry a failed PUT but deletes the target. In Cairn that goes to trash with its history, so it is confusing but nothing is lost. **Hypotheses checked against the code:** - **Quota:** versions don't count against quota, so a nearly full user can still save. - **Blast radius:** small; version folders sit beside the data. - **Correlated failure:** mostly true, but there are five exceptions where only the version save fails: - S3 errors (the S3 client never retries, and each mint adds 6-8 requests); - a wrong-owner `.cairn-versions` folder; - S3's 1024-byte key limit; - a disk-space window where the new file fits but old plus new does not; - the 99-per-second key limit. - **Mass overwrite** (ransomware, a bulk script): version copies fill the disk during the event. From then on, fail open overwrites clean files with no history. This is the strongest case for fail closed. **Precedent:** - **Fail closed by design:** systems with built-in history (Seafile, S3 versioning, copy-on-write filesystems, Git). - **Fail open:** Nextcloud's copy-before-write ignores the copy result, even on restore, and its users complain about silent history loss. - **Fail closed when full:** SharePoint counts versions in quota and goes read-only when full. **Two bugs found (fix regardless):** 1. **Autosave coalescing destroys the version from before an edit.** `Mint` replaces the newest version by the same author when it is under 10 minutes old (`versions.go:247-254`), so the second autosave of an editing session deletes the pre-session version. `TestCoalesceReplaces` asserts this; admin-policy.md:220 promises the opposite. 2. **TUS marks the upload complete before the final write** (`tus.go:419-431`). A refused finalize (423, 403 today) is then reported as success to the retrying web uploader. **Recommended final rule:** "Cairn never replaces a file without keeping its current content as a version; if it cannot, the change is refused and the file is unchanged." Universal, but only once these preconditions ship: 1. autosaves inside the 10-minute window skip the mint instead of replacing (this keeps the pre-edit version); 2. no versions of empty files; 3. server-side Copy for mints plus S3 retries; 4. the TUS offset fix; 5. a refused editor save writes a rescue copy beside the file and notifies the user; 6. refuse creating paths whose version key would exceed the backend limit. Until the preconditions ship: restore-only, as in the plan's default. **Decisions** (my recommendation in brackets): - F1 scope [every overwrite, once the preconditions ship; until then restore-only]; - F2 autosave [skip instead of replace, in batch A regardless]; - F3 refused editor save [refuse plus rescue copy plus bell]; - F4 TUS offset bug [fix in batch A]; - F5 how versions are saved [server-side Copy plus S3 retries]; - F6 over-long version keys [refuse creation]; - F7 alerting [metrics plus admin bell]. New strings only, no mockup. About 2-3 extra builder days.
Author
Owner

Decisions (Nikola, 2026-10-15) on comment 9177

All recommendations accepted. All of it goes into batch A.

  • F1 (c): every overwrite is refused when its version cannot be kept. This lands in the same release as the preconditions F2 to F6. There is no setting to turn it off.
  • F2 (b): autosaves by the same author inside the merge window skip the mint, so the pre-edit version survives. No versions of 0-byte files.
  • F3 (b): a refused editor (WOPI) save is written as a rescue copy, and the editing user gets a bell item.
  • F4: the TUS finalize offset bug is fixed in batch A.
  • F5: the mint uses a server-side Copy, with S3 retries.
  • F6: creating a path whose version key would be too long is refused up front with a clear message.
  • F7: metrics cairn_versions_mint_failures_total{cause} and cairn_versions_refused_writes_total{surface}, and an admin bell on the first refusal, then at most hourly.

The per-surface behaviour table and the tests from comment 9177 section 7 are the build contract. The new strings (rescue copy name, user bell, admin bell, public upload link refusal) go to Nikola for text approval before the UI task.

## Decisions (Nikola, 2026-10-15) on comment 9177 All recommendations accepted. All of it goes into batch A. - **F1 (c):** every overwrite is refused when its version cannot be kept. This lands in the same release as the preconditions F2 to F6. There is no setting to turn it off. - **F2 (b):** autosaves by the same author inside the merge window skip the mint, so the pre-edit version survives. No versions of 0-byte files. - **F3 (b):** a refused editor (WOPI) save is written as a rescue copy, and the editing user gets a bell item. - **F4:** the TUS finalize offset bug is fixed in batch A. - **F5:** the mint uses a server-side Copy, with S3 retries. - **F6:** creating a path whose version key would be too long is refused up front with a clear message. - **F7:** metrics `cairn_versions_mint_failures_total{cause}` and `cairn_versions_refused_writes_total{surface}`, and an admin bell on the first refusal, then at most hourly. The per-surface behaviour table and the tests from comment 9177 section 7 are the build contract. The new strings (rescue copy name, user bell, admin bell, public upload link refusal) go to Nikola for text approval before the UI task.
Author
Owner

Strings approved (Nikola, 2026-10-15)

Nikola approved all five new strings in EN/DE/FR/IT, as listed in the batch A plan's Strings table:

  • errNameTooLong;
  • the rescue copy name {name} (unsaved edits {when}){ext};
  • the editor bell version-rescue;
  • the admin bell version-refusals, with its four cause labels;
  • pubUploadRetry.

Also recorded, a ruling from the build review: an autosave skips making a version only when the newest version is itself an autosave mint by the same person, inside the window. Without that limit, two saves from a sync client followed by an editor autosave lost the second save's content.

## Strings approved (Nikola, 2026-10-15) Nikola approved all five new strings in EN/DE/FR/IT, as listed in the batch A plan's Strings table: - `errNameTooLong`; - the rescue copy name `{name} (unsaved edits {when}){ext}`; - the editor bell `version-rescue`; - the admin bell `version-refusals`, with its four cause labels; - `pubUploadRetry`. Also recorded, a ruling from the build review: an autosave skips making a version only when the newest version is itself an autosave mint by the same person, inside the window. Without that limit, two saves from a sync client followed by an editor autosave lost the second save's content.
Author
Owner

Shipped in v0.6.343 (PR #910), follow-ups in v0.6.344 (PR #912)

Every overwrite now keeps the old content as a version, or is refused with version-not-saved (503 + Retry-After: 30, or 507). This covers PUT, REST upload, public link, TUS, WOPI, MOVE, COPY, REST rename, restore and reports.

Also in this release:

  • Autosaves: an autosave skips the version only after an earlier autosave version by the same person in the window. Empty files get no versions.
  • How versions are made: by server-side copy, with S3 retries.
  • TUS: a refused finalize stays resumable and is never reported as complete.
  • Long names: a name too long for its version key is refused up front.
  • Editor saves: a refused save tries a rescue copy and sends a bell item to the editor. Admins get a bell item and two metrics.
  • MOVE/COPY onto a file: the replaced content is kept, and the two version chains merge.
  • Folders: a replaced folder goes to Deleted files.
  • Into itself: moving or copying an item onto itself or into its own subtree is refused. This also fixes a COPY on POSIX that never finished.

Live check (local instance, v0.6.343), all PASS:

  • MOVE/COPY onto a file keeps the replaced content as the newest version. A replaced folder restores as "dirX 2". Self-move and copy into a subfolder answer 403 at once.
  • With chmod 0500 on a .cairn-versions folder, every write is refused with 503 version-not-saved + Retry-After and the file is unchanged: WebDAV PUT, MOVE and COPY, REST upload, TUS and WOPI.
  • TUS: after the refusal, HEAD reports 10 of 20 bytes. Once the folder is fixed, the re-sent last chunk finalizes.
  • Both metrics count, the admin and user bells appear, and the WOPI rescue copy is written.

Still open: a real Collabora session plus Finder and rclone on the POSIX dogfood. Doing these with Nikola, then closing.

## Shipped in v0.6.343 (PR #910), follow-ups in v0.6.344 (PR #912) Every overwrite now keeps the old content as a version, or is refused with `version-not-saved` (503 + `Retry-After: 30`, or 507). This covers PUT, REST upload, public link, TUS, WOPI, MOVE, COPY, REST rename, restore and reports. Also in this release: - **Autosaves:** an autosave skips the version only after an earlier autosave version by the same person in the window. Empty files get no versions. - **How versions are made:** by server-side copy, with S3 retries. - **TUS:** a refused finalize stays resumable and is never reported as complete. - **Long names:** a name too long for its version key is refused up front. - **Editor saves:** a refused save tries a rescue copy and sends a bell item to the editor. Admins get a bell item and two metrics. - **MOVE/COPY onto a file:** the replaced content is kept, and the two version chains merge. - **Folders:** a replaced folder goes to Deleted files. - **Into itself:** moving or copying an item onto itself or into its own subtree is refused. This also fixes a COPY on POSIX that never finished. **Live check (local instance, v0.6.343), all PASS:** - MOVE/COPY onto a file keeps the replaced content as the newest version. A replaced folder restores as "dirX 2". Self-move and copy into a subfolder answer 403 at once. - With `chmod 0500` on a `.cairn-versions` folder, every write is refused with 503 `version-not-saved` + `Retry-After` and the file is unchanged: WebDAV PUT, MOVE and COPY, REST upload, TUS and WOPI. - TUS: after the refusal, HEAD reports 10 of 20 bytes. Once the folder is fixed, the re-sent last chunk finalizes. - Both metrics count, the admin and user bells appear, and the WOPI rescue copy is written. **Still open:** a real Collabora session plus Finder and rclone on the POSIX dogfood. Doing these with Nikola, then closing.
Author
Owner

Live check on files-bao (S3 + OpenBao, v0.6.344), in Nikola's browser session: PASS

  • REST rename onto an existing file: b.txt holds a's content, and b's replaced content (B2) is now its newest version. Version contents read back as [B2, B1].
  • Restore the oldest version: 200; the file contains B1, and the version list grows from 2 to 3.
  • Rename onto itself: REST answers 200 with no change; WebDAV MOVE onto itself answers 403. The versions and the content are unchanged.
  • Collabora: a new .odt, edited and saved twice. The first save made one autosave-marked version (_1a_) of the content before editing; the second save inside the window made no version.
  • #906: admin upload into /spaces/test/.trash/ answers 403 trash-managed. Restoring an entry whose name is taken lands as lc343-a2 2.txt and the answer gives the person's own path.
  • #903: a PROPFIND on /dav/home/ that names the quota properties returns the figures for the home bucket.

All test files were deleted and purged afterwards.

Still open: a forced refusal with a rescue copy in a real Collabora session needs the POSIX dogfood, because the S3 version prefix cannot be made unwritable. Finder and rclone also need Nikola.

## Live check on files-bao (S3 + OpenBao, v0.6.344), in Nikola's browser session: PASS - **REST rename onto an existing file:** b.txt holds a's content, and b's replaced content (B2) is now its newest version. Version contents read back as [B2, B1]. - **Restore the oldest version:** 200; the file contains B1, and the version list grows from 2 to 3. - **Rename onto itself:** REST answers 200 with no change; WebDAV MOVE onto itself answers 403. The versions and the content are unchanged. - **Collabora:** a new .odt, edited and saved twice. The first save made one autosave-marked version (`_1a_`) of the content before editing; the second save inside the window made no version. - **#906:** admin upload into `/spaces/test/.trash/` answers 403 `trash-managed`. Restoring an entry whose name is taken lands as `lc343-a2 2.txt` and the answer gives the person's own path. - **#903:** a PROPFIND on `/dav/home/` that names the quota properties returns the figures for the home bucket. All test files were deleted and purged afterwards. **Still open:** a forced refusal with a rescue copy in a real Collabora session needs the POSIX dogfood, because the S3 version prefix cannot be made unwritable. Finder and rclone also need Nikola.
Author
Owner

Live check on files (POSIX, v0.6.344) with a real Collabora session: server side PASS

Setup: in a test folder, .cairn-versions was set to 0500 through a temporary debug container, then reverted and the test folder purged.

  • REST upload over an existing file: 503, Retry-After: 30, version-not-saved. The file kept T2.
  • WebDAV PUT: 503 version-not-saved.
  • New file in the same folder: 201, so new files are never refused.
  • Collabora save: refused. Log: wopi save refused, then wopi rescue copy saved. The copy rescue (unsaved edits 2026-10-15 17-16).odt holds the edits. The next refused saves refreshed the same copy and made no new ones.
  • Bells: version-rescue for the editor (name and copy path) and version-refusals for the admin (count 1, cause io).
  • WARN line: versions: mint failed path=... surface=... cause=io, written once per folder and cause.

Finding (new, #913): after the refused save, Collabora reloaded the document. The typed text vanished from the editor (it is still in the rescue copy). Later Collabora asked "Document has been changed in storage", even though Cairn only ever answered 503. To be researched in #913.

The Finder and rclone checks are tracked in #914 (Nikola). This issue closes after those.

## Live check on files (POSIX, v0.6.344) with a real Collabora session: server side PASS Setup: in a test folder, `.cairn-versions` was set to 0500 through a temporary debug container, then reverted and the test folder purged. - **REST upload over an existing file:** 503, `Retry-After: 30`, `version-not-saved`. The file kept T2. - **WebDAV PUT:** 503 `version-not-saved`. - **New file in the same folder:** 201, so new files are never refused. - **Collabora save:** refused. Log: `wopi save refused`, then `wopi rescue copy saved`. The copy `rescue (unsaved edits 2026-10-15 17-16).odt` holds the edits. The next refused saves refreshed the **same** copy and made no new ones. - **Bells:** `version-rescue` for the editor (name and copy path) and `version-refusals` for the admin (count 1, cause io). - **WARN line:** `versions: mint failed path=... surface=... cause=io`, written once per folder and cause. **Finding (new, #913):** after the refused save, Collabora reloaded the document. The typed text vanished from the editor (it is still in the rescue copy). Later Collabora asked "Document has been changed in storage", even though Cairn only ever answered 503. To be researched in #913. The Finder and rclone checks are tracked in #914 (Nikola). This issue closes after those.
Author
Owner

Closing as fixed (v0.6.343/344): server-side live checks passed on both dogfoods (comments above; internal/dav/dav.go:426,584 on v0.6.358 main). The Finder and rclone steps stay tracked in #914 and #1005 section F.

Closing as fixed (v0.6.343/344): server-side live checks passed on both dogfoods (comments above; `internal/dav/dav.go:426,584` on v0.6.358 main). The Finder and rclone steps stay tracked in #914 and #1005 section F.
Cordy closed this issue 2026-10-26 23:43:07 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#908
No description provided.