v0.6.349: versions correctness (#951 If-Match at commit, #947 plain sizes, #941/#943 recorded moves, #946 pinned replacement times) #959

Merged
Cordy merged 16 commits from fix-b2 into main 2026-10-17 20:40:26 +00:00
Owner

This is backlog batch 2. Every decision was made by Nikola after deep research, and each one is recorded on its issue.

#951: conditional saves at commit

  • If-Match and If-None-Match (including *) are checked a second time at the commit step, under the path lock.
  • If the check fails, WebDAV answers 412 and drops the body. WOPI answers 409 with code 1010, and makes no rescue copy and no size hint.
  • New: storage.Precondition / ErrPreconditionFailed. Weak ETags are accepted on both sides.
  • Out of scope, in #957: DELETE/MOVE/COPY and the RFC 4918 If header.

#947: plain version sizes

  • The size is derived from each version's own age header, without needing a key. It is cached in memory (4096 entries) and nothing new is stored.
  • On S3 this takes a ranged GET of 8 KiB, falling back to 64 KiB. An unreadable header shows the stored size and is retried. Non-age records are cached.
  • The panel no longer shows a dangling dot for an empty version.

#941 / #943: recorded moves

  • Moves across key scopes, and S3 moves of unrecorded files, now record the file in the destination's chain first, so the moved file still names its record.
  • The old record is deleted only when the identity is exact (the new ExactRecorder). The move re-checks the source after reading it, so a twin save during the move is safe.
  • If recording fails, the move falls back to recording at the old path and then a plain rename.
  • On S3 in put/none modes, an ETag+size match is used. It is enabled only when the boot probe confirms that ETags are the content MD5 and that they distinguish contents.

#946: pinned replacement times

  • When histories merge, the replacement time is written into the key, following the corrected algorithm from the research: r<10-digit unix>, keys up to 98 bytes. v0.6.347 and v0.6.348 can still parse these keys.
  • Destination pins are written before the moved content is published, and pruning happens after the merge.
  • Pins are never changed or undone.
  • After a failed publish, the destination's current content gets a fresh record. A pinned duplicate takes no count slot, but only where identity is exact.

Reviews

  • Per-task reviews, then 6 fix rounds, each re-reviewed (round 6 is narrowing only).
  • A final whole-branch review.
  • A fresh docs review against the code, followed by fixes.

Tests

  • The full suite passes.
  • versions passes under -race (166 tests).
  • dav, s3, posix, stacktest and api 947/951 pass.
  • As root: posix, stacktest, trash, quota, scope, auth, versions, dav, api.
  • The pin model reports 0 early contents, including the failed-publish branch.

Docs

CHANGELOG v0.6.349, VERSIONING (§1, §2 pinned keys, §6, §7, §7a, §9), document-editing, deployment, file-browser.

Issues

  • Follow-ups: #956 (stores without CopyObject), #957, #958 (overlapping saves published out of key order).

Closes #951, closes #947, closes #941, closes #943, closes #946.

This is backlog batch 2. Every decision was made by Nikola after deep research, and each one is recorded on its issue. ## #951: conditional saves at commit - If-Match and If-None-Match (including `*`) are checked a second time at the commit step, under the path lock. - If the check fails, WebDAV answers 412 and drops the body. WOPI answers 409 with code 1010, and makes no rescue copy and no size hint. - New: `storage.Precondition` / `ErrPreconditionFailed`. Weak ETags are accepted on both sides. - Out of scope, in #957: DELETE/MOVE/COPY and the RFC 4918 `If` header. ## #947: plain version sizes - The size is derived from each version's own age header, without needing a key. It is cached in memory (4096 entries) and nothing new is stored. - On S3 this takes a ranged GET of 8 KiB, falling back to 64 KiB. An unreadable header shows the stored size and is retried. Non-age records are cached. - The panel no longer shows a dangling dot for an empty version. ## #941 / #943: recorded moves - Moves across key scopes, and S3 moves of unrecorded files, now record the file in the destination's chain first, so the moved file still names its record. - The old record is deleted only when the identity is exact (the new `ExactRecorder`). The move re-checks the source after reading it, so a twin save during the move is safe. - If recording fails, the move falls back to recording at the old path and then a plain rename. - On S3 in put/none modes, an ETag+size match is used. It is enabled only when the boot probe confirms that ETags are the content MD5 and that they distinguish contents. ## #946: pinned replacement times - When histories merge, the replacement time is written into the key, following the corrected algorithm from the research: `r<10-digit unix>`, keys up to 98 bytes. v0.6.347 and v0.6.348 can still parse these keys. - Destination pins are written before the moved content is published, and pruning happens after the merge. - Pins are never changed or undone. - After a failed publish, the destination's current content gets a fresh record. A pinned duplicate takes no count slot, but only where identity is exact. ## Reviews - Per-task reviews, then 6 fix rounds, each re-reviewed (round 6 is narrowing only). - A final whole-branch review. - A fresh docs review against the code, followed by fixes. ## Tests - The full suite passes. - versions passes under `-race` (166 tests). - dav, s3, posix, stacktest and api 947/951 pass. - As root: posix, stacktest, trash, quota, scope, auth, versions, dav, api. - The pin model reports 0 early contents, including the failed-publish branch. ## Docs CHANGELOG v0.6.349, VERSIONING (§1, §2 pinned keys, §6, §7, §7a, §9), document-editing, deployment, file-browser. ## Issues - Follow-ups: #956 (stores without CopyObject), #957, #958 (overlapping saves published out of key order). Closes #951, closes #947, closes #941, closes #943, closes #946.
The early check stays (it saves the upload). The precondition rides the
Write ctx only (storage.WithPrecondition, claimed once by the first
recorded write); write.commit evaluates it right after the path lock and
before any snapshot. A mismatch publishes nothing, drops the body's record
and returns storage.ErrPreconditionFailed: WebDAV 412, WOPI 409 with
COOLStatusCode 1010 (no rescue copy, no size hint). Weak ETags stay
accepted (documented). DELETE/MOVE/COPY preconditions are #957.
Each listed version's size is derived from its own age header (no key,
age's chunk rule), cached in memory under the record's path, stored
size and ETag; nothing new is stored. A header that cannot be read
shows the stored size for that row (rate-limited WARN) and is read
again next time. s3 reads the head in one bounded ranged GET (new
ReadHead), posix by one open and read. Unencrypted instances are
unchanged. The versions panel drops the separator when a version has
no size text (empty version).
A move whose plain rename would not carry the moved file's record now
moves as a recorded copy: CopyRecorded records the content in the
destination's chain and publishes it naming that record, the source is
deleted, and the chains merge without the source's old record. The move
plan's commit takes no serializer (Rename holds both paths) and claims
no precondition.

- #941: a move across key scopes (personal to space, space to space and
  back) asks the encrypt layer (new CrossScope on Driver and
  RemoteDriver); the moved file names its record, no duplicate.
- #943 copy mode: an unrecorded source on s3 moves server-side when
  MovesByCopyRecorded (mode copy, REPLACE honoured, within the copy
  limit); above the limit today's path (documented duplicate).
- #943 put and none modes: s3 RecordOf finds the record by size and ETag
  (newest match) when the object names none, gated by a boot probe that
  ETags survive PUT, CopyObject and listing (etag_match on the probe
  line). Unstable ETags keep today's duplicates; nothing is lost.
- s3fake: UnstableETag, ListETagDiffers; stacktest Fault forwards
  MovesByCopyRecorded.
- Tests: move941_test.go, move943_test.go, s3/record943_test.go;
  TestS3WriteRecorded/none and keydrain's s3-none row updated for the
  #943 decision (stated in the report).
A move onto a file with its own history writes the replacement time into
the version key where the merged chain would otherwise age an entry from
the other file's dates (pinned keys, <12 digits>r<unix>[a|u], at most 98
bytes), using the corrected algorithm of research-946 5.2: the moved
file's entries are pinned as they are carried, the destination's only
where needed and before the moved entries arrive. ageCut ages a pinned
entry from its pin and skips pinned entries as replacers. The 87-byte
creation guard stays: a pin that does not fit is not written (WARN). The
one-hour tolerance applies to the replaced content only.
- precondition.go: "W/" is stripped from the stored ETag too, so a backend
  that returns a weak ETag matches itself (WOPI sends the file's own ETag
  as If-Match). Table rows: weak stored vs strong client, weak vs weak,
  strong vs weak, plus If-None-Match on a weak stored tag.
- TestPreconditionNotClaimedByOthers951: an internal write, a copy and a
  rename on the SAME ctx precede the user's write; it still evaluates the
  precondition (ErrPreconditionFailed).
- Refusal tests (If-Match, If-None-Match *, before-snapshot, stat fails):
  no .cairn-upload-* / cairn-s3-upload-* temp left, the cached quota figure
  unchanged, and a fresh quota walk counts only the file and its owner
  manifest.
- C1: the move finds the source's record again once the source is read;
  a source saved meanwhile is moved as unrecorded content (fresh key,
  the old record follows the chain).
- I2: the old record is deleted only when the raw driver's answer is
  exact (storage.ExactRecorder: posix link mode, a record the s3 object
  names) and not a refused write's; posix copy mode and s3 ETag matches
  keep it (a possible duplicate, never a loss).
- I1, M4: the s3 ETag probe requires ETags equal to the bytes' MD5 and
  different for other bytes of one length; the match never answers a
  refused write's record and never matches an empty ETag.
- M1: the move's record is pending until the move ends. M2: a lost hold
  deletes no old record. M3: a retry after a failed delete of the source
  only deletes the source.
- A source with no record whose move cannot be recorded falls back to
  the plain rename (recorded in its own chain first), with one WARN.
C-1/I-1: move.record writes the destination's pins (K counted in) before
the driver makes K; the merge after a recorded move computes only the
moved file's pins; Rename prunes after the merge. I-2: the source's old
record travelling beside K is pinned at its replacement time. m-1: no pin
for entries within the record grace older than the moved content's
record. m-3: K bounds in-flight entries on a retry. m-4: a re-keyed
current record is not pinned at its own time. m-2: Recorder contract
documented. m-5: the model checks the state with K in the chain before
the carry.
I-A: a recorded move pins the destination before its publish; when the
publish fails (move.drop), the pins it wrote are renamed back to their
keys, best effort with one WARN, so the content that stays current does
not age from the failed move.

m-1 residual: Unmint finds a record that another instance's merge pinned
while its write was in flight, so a refused (412) or failed write leaves
no pinned version of its body.
A pin is written once and never changes (research-946 §4(a)): another
instance's move onto the same file may rely on the pins a recorded move
wrote before its publish, so undoing them after a failed publish made
old versions age early. The undo is removed. Instead, when the publish
fails and RecordOf still answers the replaced content's pinned record,
that content gets a fresh, unpinned record (best effort, one WARN):
it ages from its real replacement, the pinned record stays as a
duplicate in the list. The Unmint pinned-name fallback stays.
Engine.duplicate compared RecordOf identities, which are guesses in posix
copy mode (size and mtime) and on s3 where the ETag need not follow the
content. Two different contents could be exempted from the count cap.
Now an entry is a duplicate only on equal identities from a driver whose
identities are exact (new storage.ExactIdentifier: posix link mode; s3
with etag_match) or when both name the same record exactly
(storage.ExactRecorder: s3 object metadata).
Cordy merged commit 4979304259 into main 2026-10-17 20:40:26 +00:00
Author
Owner

Live check on both dogfoods (2026-10-18)

Run on files (posix/NFS, link mode) and files-bao (Garage, copy mode). Both are on v0.6.349 and were checked from a browser session. Every check passed on both.

  • #939 reserved names: an upload to x.cairn-replace-abc was refused with 400 invalid path, and so was a rename to a.txt.cairn-replace-abc.
  • #947 sizes: a 6-byte file saved three times lists versions of 6 bytes (before this change they showed the encrypted size of about 304).
  • #941 cross-scope move: I moved /home/b12-live/a.txt, which has 2 versions, into /spaces/test and back.
    • The destination still listed exactly the same 2 versions.
    • Its current content was not listed among them.
    • No duplicate was added.
  • #951 conditional saves (WebDAV, session):
    • PUT with the current If-Match answered 204.
    • PUT again with the now stale If-Match answered 412.
    • If-None-Match: * on the existing file answered 412.
    • The file kept the accepted content.
  • #951 WOPI:
    • PutFile with the current X-COOL-WOPI-Timestamp answered 200.
    • PutFile again with the stale timestamp answered 409 {"COOLStatusCode":1010}.
    • No rescue copy was created.
  • No content lost: the final version list held all four earlier contents (stuvwx, mnopqr, ghijkl, abcdef) under the current wopi01.

The test folders were deleted and their trash entries purged on both instances.

The race at the commit step itself (another save landing during the upload) is covered by the rig tests on all backends; I can't time it reliably by hand.

## Live check on both dogfoods (2026-10-18) Run on files (posix/NFS, link mode) and files-bao (Garage, copy mode). Both are on v0.6.349 and were checked from a browser session. Every check passed on both. - **#939 reserved names:** an upload to `x.cairn-replace-abc` was refused with 400 `invalid path`, and so was a rename to `a.txt.cairn-replace-abc`. - **#947 sizes:** a 6-byte file saved three times lists versions of `6` bytes (before this change they showed the encrypted size of about 304). - **#941 cross-scope move:** I moved `/home/b12-live/a.txt`, which has 2 versions, into `/spaces/test` and back. - The destination still listed exactly the same 2 versions. - Its current content was not listed among them. - No duplicate was added. - **#951 conditional saves (WebDAV, session):** - PUT with the current `If-Match` answered 204. - PUT again with the now stale `If-Match` answered 412. - `If-None-Match: *` on the existing file answered 412. - The file kept the accepted content. - **#951 WOPI:** - PutFile with the current `X-COOL-WOPI-Timestamp` answered 200. - PutFile again with the stale timestamp answered 409 `{"COOLStatusCode":1010}`. - No rescue copy was created. - **No content lost:** the final version list held all four earlier contents (`stuvwx`, `mnopqr`, `ghijkl`, `abcdef`) under the current `wopi01`. The test folders were deleted and their trash entries purged on both instances. The race at the commit step itself (another save landing during the upload) is covered by the rig tests on all backends; I can't time it reliably by hand.
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#959
No description provided.