Versions: two saves in the same second can lose a version (key sequence per author, not atomic) #922

Closed
opened 2026-10-15 21:22:13 +00:00 by Cordy · 9 comments
Owner

This is a data-loss class, confirmed live on a local v0.6.345 instance (live345 Finding 1, 3 out of 3 runs, script scratch/b7e/live345/t_5x.sh). Earlier notes from the batch A reviews point to the same cause.

What happens: an office autosave by emil and a write by olga to the same file, in the same second, lose olga's write. The editor's next save overwrites it and no version is kept.

  • With real Collabora this needs the person to choose "Overwrite" in the conflict dialog. Over WebDAV/REST it can happen on its own.

Cause (internal/storage/versions/versions.go):

  • Version keys look like <rfc3339-second>_<seq>[a]_<author>.v.
  • The sequence is picked per author: one author's existing keys decide the next number.
  • Keys sort as text. So within one second, emil's autosave key _1a_emil sorts after olga's later _1_olga. The autosave skip then sees "the newest version is my autosave" and skips minting.

Also from the reviews:

  • Same author, same second: two concurrent mints of one file can both find the same sequence free (Stat, then Write, is not atomic). Both write one key, so one version is lost.
  • An Unmint by either one (same content, or a failed MOVE/COPY) then deletes the key the other kept.
  • _9a_ sorts against _10_ as text.

Fix direction (research first):

  • Make the key order time-exact and author-independent, for example nanosecond time plus a random or global sequence.
  • Pick and create keys atomically (create-exclusive on POSIX, a conditional PUT on S3).
  • The skip rule should look at the true newest version.
  • Old keys must still parse.
  • Tests through the rig, posix and s3, with concurrent writers.
This is a data-loss class, confirmed live on a local v0.6.345 instance (live345 Finding 1, 3 out of 3 runs, script `scratch/b7e/live345/t_5x.sh`). Earlier notes from the batch A reviews point to the same cause. **What happens:** an office autosave by emil and a write by olga to the same file, in the same second, lose olga's write. The editor's next save overwrites it and no version is kept. - With real Collabora this needs the person to choose "Overwrite" in the conflict dialog. Over WebDAV/REST it can happen on its own. **Cause** (`internal/storage/versions/versions.go`): - Version keys look like `<rfc3339-second>_<seq>[a]_<author>.v`. - The sequence is picked per author: one author's existing keys decide the next number. - Keys sort as text. So within one second, emil's autosave key `_1a_emil` sorts after olga's later `_1_olga`. The autosave skip then sees "the newest version is my autosave" and skips minting. **Also from the reviews:** - Same author, same second: two concurrent mints of one file can both find the same sequence free (Stat, then Write, is not atomic). Both write one key, so one version is lost. - An Unmint by either one (same content, or a failed MOVE/COPY) then deletes the key the other kept. - `_9a_` sorts against `_10_` as text. **Fix direction (research first):** - Make the key order time-exact and author-independent, for example nanosecond time plus a random or global sequence. - Pick and create keys atomically (create-exclusive on POSIX, a conditional PUT on S3). - The skip rule should look at the true newest version. - Old keys must still parse. - Tests through the rig, posix and s3, with concurrent writers.
Author
Owner

Research (2026-10-16)

Full findings: cairn-work/scratch/b7e/batch907/922-research.md. Rig tests are in 922tools/race922_test.go; the prototype diff is 922tools/prototype-p1p2.diff.

Confirmed on posix, posix-encrypt, s3 and s3-encrypt:

  • (a) Autosave by one person, a save by another in the same second (the live345 case): olga's write is lost, and no version keeps it.
  • (b1) Same author, same second: both mints get the same key, so one version is lost.
  • (b2) Unmint: an Unmint deletes the shared key, and the chain ends empty.
  • (c) Pruning removes the newest: pruning frees seq 1, the next mint reuses it, and it sorts as the oldest, so it is pruned at once. With a cap of 3, twelve saves in one second keep w1-w3 and lose w9-w11.
  • (d) NEW, write race: two concurrent writers both version the same old content, so the content in between is never kept. Stress test (8 goroutines × 20 rounds): 49 to 112 of 161 contents lost. #908's "never replace without keeping" does not hold under concurrency.
  • (e) NEW, refusals: on posix, more than 99 saves of one file per second are refused.

Market:

  • Garage: conditional writes are documented as "structurally impossible". v2.4.1 accepts If-None-Match and ignores it (lance#9762).
  • AWS: supports If-None-Match on PUT (2024) and on CopyObject (2025-10).
  • Nextcloud: names versions by whole second and retries on duplicates; complaints appear on the forum.
  • oCIS: names revisions to the nanosecond and orders them by parsed time.
  • restic: names are unique by construction and each is written once. This is the model recommended here.

Recommendation:

  • Key format: YYYY-MM-DDTHH:MM:SS.nnnnnnnnnZ_<12 random digits>[a]_<author>.v. It has the same three fields, so v0.6.346, the API and the UI still read it.
  • Ordering: by parsed time, then the numeric middle field (this also fixes the legacy _9_/_10_ order), then the text. The mint time is the later of now and the newest version + 1 ns.
  • Prune and Unmint: prune never deletes the key it just minted, and an Unmint deletes only its own key.
  • One per-file lock inside the process around read, mint and write, covering Write, Rename (both paths, in sorted order), Copy and Delete. It waits on the request context instead of refusing.
  • No conditional writes. The seq search and the 99-per-second limit go away.

Prototype: keys and ordering alone fix (a) and (c). With the lock added, the stress test loses 0 contents in all variants, also under -race. Four tests hard-code the old key format.

Decisions for Nikola:

  1. Atomicity: lock + unique keys (rec.), conditional writes, or both.
  2. Random token: 12 digits (rec.; the longest key grows from 67 to 87 bytes), none, or 20.
  3. Lock scope: the whole write (rec.; a long upload makes the next save of the same file wait), or only the commit (needs a driver hook).
  4. Old keys on the dogfoods: leave them (rec.).
  5. Metrics: drop the "no free key" refusal cause (rec.).

Mockup: none (no UI change). Size: S-M, about 1 day.

## Research (2026-10-16) Full findings: `cairn-work/scratch/b7e/batch907/922-research.md`. Rig tests are in `922tools/race922_test.go`; the prototype diff is `922tools/prototype-p1p2.diff`. **Confirmed on posix, posix-encrypt, s3 and s3-encrypt:** - **(a) Autosave by one person, a save by another in the same second (the live345 case):** olga's write is lost, and no version keeps it. - **(b1) Same author, same second:** both mints get the same key, so one version is lost. - **(b2) Unmint:** an Unmint deletes the shared key, and the chain ends empty. - **(c) Pruning removes the newest:** pruning frees seq 1, the next mint reuses it, and it sorts as the oldest, so it is pruned at once. With a cap of 3, twelve saves in one second keep w1-w3 and lose w9-w11. - **(d) NEW, write race:** two concurrent writers both version the same old content, so the content in between is never kept. Stress test (8 goroutines × 20 rounds): 49 to 112 of 161 contents lost. #908's "never replace without keeping" does not hold under concurrency. - **(e) NEW, refusals:** on posix, more than 99 saves of one file per second are refused. **Market:** - **Garage:** conditional writes are documented as "structurally impossible". v2.4.1 accepts `If-None-Match` and ignores it (lance#9762). - **AWS:** supports `If-None-Match` on PUT (2024) and on CopyObject (2025-10). - **Nextcloud:** names versions by whole second and retries on duplicates; complaints appear on the forum. - **oCIS:** names revisions to the nanosecond and orders them by parsed time. - **restic:** names are unique by construction and each is written once. This is the model recommended here. **Recommendation:** - **Key format:** `YYYY-MM-DDTHH:MM:SS.nnnnnnnnnZ_<12 random digits>[a]_<author>.v`. It has the same three fields, so v0.6.346, the API and the UI still read it. - **Ordering:** by parsed time, then the numeric middle field (this also fixes the legacy `_9_`/`_10_` order), then the text. The mint time is the later of now and the newest version + 1 ns. - **Prune and Unmint:** prune never deletes the key it just minted, and an Unmint deletes only its own key. - **One per-file lock inside the process** around read, mint and write, covering Write, Rename (both paths, in sorted order), Copy and Delete. It waits on the request context instead of refusing. - **No conditional writes.** The seq search and the 99-per-second limit go away. **Prototype:** keys and ordering alone fix (a) and (c). With the lock added, the stress test loses 0 contents in all variants, also under `-race`. Four tests hard-code the old key format. **Decisions for Nikola:** 1. **Atomicity:** lock + unique keys (rec.), conditional writes, or both. 2. **Random token:** 12 digits (rec.; the longest key grows from 67 to 87 bytes), none, or 20. 3. **Lock scope:** the whole write (rec.; a long upload makes the next save of the same file wait), or only the commit (needs a driver hook). 4. **Old keys on the dogfoods:** leave them (rec.). 5. **Metrics:** drop the "no free key" refusal cause (rec.). **Mockup:** none (no UI change). **Size:** S-M, about 1 day.
Author
Owner

Follow-up research: holding across instances (#742) (2026-10-16)

Nikola: horizontal scaling (#742) will be needed, so the fix must hold across several Cairn instances that share one backend. Full findings: cairn-work/scratch/b7e/batch907/922-research-2.md.

Main finding: a lock alone cannot make this safe across instances on Garage or NFS.

  • An expired lease can end up held by two instances at once.
  • The standard cure is a fencing token that the storage checks on each write. That is impossible on Garage (no compare-and-swap, K2V included) and on NFS.
  • So data safety must not depend on a lock.

Competitors:

  • Nextcloud, ownCloud 10 and oCIS make cross-node writes safe through Redis, a database or NATS. Nextcloud users hit 423 errors and stale-lock TTLs; ownCloud's own advice is "never rely on NFS locks".
  • Seafile updates its head pointer with compare-and-swap in its database.
  • Delta Lake and Iceberg need DynamoDB on S3.
  • restic writes everything once, under unique names.
  • lance silently lost about three quarters of appends on Garage.

Garage: its docs call conditional writes "structurally impossible". A third-party report shows v2.4.1 accepts a create-only PUT and overwrites anyway. It does promise read-after-write. The deployed version is not documented; a probe is ready (922tools/garage-probe-922.sh).

New recommendation: record-then-publish.

  • Every write first stores its content under a unique version key, then makes it visible at the path.
  • Before a replace, the current content is recorded too, unless a record of it already exists.
  • The file itself stays last-writer-wins. No content ever lives only at the path, so no mix of writers on any number of instances can lose one. This needs no lock, no clock and no storage condition (proof in §6 C').
  • The same principle is behind restic, Seafile and JuiceFS.

Cost of recording:

  • posix: a hard link, so no data copy. That is cheaper than today's mint, which copies the whole file.
  • Garage: the same number of requests as today, and no extra bytes thanks to block deduplication.
  • AWS S3 / MinIO: one extra stored copy of each file's current content.

What to build now (nothing thrown away for #742):

  • Keys: the research-1 format with the 12-digit random token. Versions are ordered by parsed commit time, clamped above the newest version the writer can see.
  • One PathSerializer interface: in-process now. #742 can plug in a cross-instance implementation later, for ordering only.
  • Lock scope: only the short commit step, so a long upload never blocks an autosave of the same file. This needs a small Recorder capability in the posix, s3, encrypt and reserved layers.
  • Autosave coalescing: removes only the author's own autosave, and only when that autosave is exactly what was replaced. This fixes the live345 case without a lock.
  • Prune never deletes the newest version or the record of the current content. Unmint deletes only its own key.
  • Tests: add a two-instance rig, Stack.Twin: a second stack over the same posix directory or s3fake.

#742 still needs (noted there): the statestore boot lock (it refuses a second instance), a shared #315 lock store, and ownermeta.

Decisions for Nikola:

  1. Record-then-publish plus the in-process lock (rec.), or locks only.
  2. Lock only the commit step (rec.), or the whole write.
  3. Hide the current content's record in the Versions API (rec., no UI change), or show a "Current version" row (needs a mockup).
  4. Accept the extra storage on AWS S3 and MinIO (rec.).
  5. Decide the cross-instance lock together with #742's shared-state design; never use lease objects on the backend (rec.).
  6. Hard links on posix, with a copy fallback detected at boot (rec.).
  7. Accept a sub-millisecond window for tools that write directly into the posix tree (rec.).

Probes waiting for permission:

  • Garage: creates and deletes a temporary cairn-probe-922-* bucket on TrueNAS.
  • NFS: a scratch PVC (922tools/nfs-probe-922.yaml) to check hard links, O_EXCL and flock across two nodes.

Size: M, about 3-4 days.

## Follow-up research: holding across instances (#742) (2026-10-16) Nikola: horizontal scaling (#742) will be needed, so the fix must hold across several Cairn instances that share one backend. Full findings: `cairn-work/scratch/b7e/batch907/922-research-2.md`. **Main finding:** a lock alone cannot make this safe across instances on Garage or NFS. - An expired lease can end up held by two instances at once. - The standard cure is a fencing token that the storage checks on each write. That is impossible on Garage (no compare-and-swap, K2V included) and on NFS. - So data safety must not depend on a lock. **Competitors:** - Nextcloud, ownCloud 10 and oCIS make cross-node writes safe through Redis, a database or NATS. Nextcloud users hit 423 errors and stale-lock TTLs; ownCloud's own advice is "never rely on NFS locks". - Seafile updates its head pointer with compare-and-swap in its database. - Delta Lake and Iceberg need DynamoDB on S3. - restic writes everything once, under unique names. - lance silently lost about three quarters of appends on Garage. **Garage:** its docs call conditional writes "structurally impossible". A third-party report shows v2.4.1 accepts a create-only PUT and overwrites anyway. It does promise read-after-write. The deployed version is not documented; a probe is ready (`922tools/garage-probe-922.sh`). **New recommendation: record-then-publish.** - Every write first stores its content under a unique version key, then makes it visible at the path. - Before a replace, the current content is recorded too, unless a record of it already exists. - The file itself stays last-writer-wins. No content ever lives only at the path, so no mix of writers on any number of instances can lose one. This needs no lock, no clock and no storage condition (proof in §6 C'). - The same principle is behind restic, Seafile and JuiceFS. **Cost of recording:** - **posix:** a hard link, so no data copy. That is cheaper than today's mint, which copies the whole file. - **Garage:** the same number of requests as today, and no extra bytes thanks to block deduplication. - **AWS S3 / MinIO:** one extra stored copy of each file's current content. **What to build now (nothing thrown away for #742):** - **Keys:** the research-1 format with the 12-digit random token. Versions are ordered by parsed commit time, clamped above the newest version the writer can see. - **One `PathSerializer` interface:** in-process now. #742 can plug in a cross-instance implementation later, for ordering only. - **Lock scope:** only the short commit step, so a long upload never blocks an autosave of the same file. This needs a small `Recorder` capability in the posix, s3, encrypt and reserved layers. - **Autosave coalescing:** removes only the author's own autosave, and only when that autosave is exactly what was replaced. This fixes the live345 case without a lock. - **Prune** never deletes the newest version or the record of the current content. **Unmint** deletes only its own key. - **Tests:** add a two-instance rig, `Stack.Twin`: a second stack over the same posix directory or s3fake. **#742 still needs** (noted there): the statestore boot lock (it refuses a second instance), a shared #315 lock store, and ownermeta. **Decisions for Nikola:** 1. Record-then-publish plus the in-process lock (rec.), or locks only. 2. Lock only the commit step (rec.), or the whole write. 3. Hide the current content's record in the Versions API (rec., no UI change), or show a "Current version" row (needs a mockup). 4. Accept the extra storage on AWS S3 and MinIO (rec.). 5. Decide the cross-instance lock together with #742's shared-state design; never use lease objects on the backend (rec.). 6. Hard links on posix, with a copy fallback detected at boot (rec.). 7. Accept a sub-millisecond window for tools that write directly into the posix tree (rec.). **Probes waiting for permission:** - **Garage:** creates and deletes a temporary `cairn-probe-922-*` bucket on TrueNAS. - **NFS:** a scratch PVC (`922tools/nfs-probe-922.yaml`) to check hard links, O_EXCL and flock across two nodes. **Size:** M, about 3-4 days.
Author
Owner

Decisions (Nikola, 2026-10-16)

All seven recommendations from comment 9444 are accepted:

  1. Record-then-publish, plus a short in-process lock.
  2. The lock covers only the commit step.
  3. The record of the current content is hidden in the Versions API, so the UI does not change.
  4. The extra stored copy on AWS S3 and MinIO is accepted.
  5. The cross-instance lock is designed with #742. It is never built as lease objects on the backend.
  6. POSIX uses hard links, with a copy fallback detected at boot.
  7. A window of under a millisecond is accepted for tools that write directly into the POSIX tree.

Nikola also allowed the two probes: a temporary Garage bucket on TrueNAS, and a scratch NFS volume.

## Decisions (Nikola, 2026-10-16) All seven recommendations from comment 9444 are accepted: 1. Record-then-publish, plus a short in-process lock. 2. The lock covers only the commit step. 3. The record of the current content is hidden in the Versions API, so the UI does not change. 4. The extra stored copy on AWS S3 and MinIO is accepted. 5. The cross-instance lock is designed with #742. It is never built as lease objects on the backend. 6. POSIX uses hard links, with a copy fallback detected at boot. 7. A window of under a millisecond is accepted for tools that write directly into the POSIX tree. Nikola also allowed the two probes: a temporary Garage bucket on TrueNAS, and a scratch NFS volume.
Author
Owner

NFS probe results (2026-10-16)

The probe ran on a scratch PVC on the nfs-monitoring storage class, with two pods on two nodes: probe A on pi2, probe B on pi1. Mount: NFS 4.2, hard,proto=tcp. The PVC and both jobs were deleted afterwards.

Check Result
Hard link works: a and a.rec share an inode, links=2
ln onto an existing name refused, EEXIST
Rename over an existing file while a hard link holds the old content atomic, old content kept: a.rec still reads "one", a reads "two"
O_EXCL create exclusive: the second create is refused
flock not available: "Bad file descriptor" on both nodes. The "B refused" line is that error, not a real exclusion.

Meaning for the design:

  • POSIX on this NFS export supports record-then-publish with hard links (decision 6).
  • NFS file locks are unusable, which confirms decision 5: no backend locks.

The Garage probe is still pending: Claude has no shell on TrueNAS.

## NFS probe results (2026-10-16) The probe ran on a scratch PVC on the `nfs-monitoring` storage class, with two pods on two nodes: probe A on pi2, probe B on pi1. Mount: NFS 4.2, `hard,proto=tcp`. The PVC and both jobs were deleted afterwards. | Check | Result | |---|---| | Hard link | **works**: `a` and `a.rec` share an inode, links=2 | | `ln` onto an existing name | **refused**, EEXIST | | Rename over an existing file while a hard link holds the old content | **atomic, old content kept**: `a.rec` still reads "one", `a` reads "two" | | `O_EXCL` create | **exclusive**: the second create is refused | | `flock` | **not available**: "Bad file descriptor" on both nodes. The "B refused" line is that error, not a real exclusion. | **Meaning for the design:** - POSIX on this NFS export supports record-then-publish with hard links (decision 6). - NFS file locks are unusable, which confirms decision 5: no backend locks. The Garage probe is still pending: Claude has no shell on TrueNAS.
Author
Owner

Approved (Nikola, 2026-10-16): spec rulings with visible effects

Nikola: "i view them all as improvements".

  • R6: a version's "by" is the person who saved that content.
  • R7: an editing burst keeps one more version than today; still at most one autosave version per person per window.
  • R5: saving identical content leaves the file untouched.
  • R3 trade-off: while a version folder cannot be written, the following saves of an editing burst are also refused (rescue copy and bell as in #908), instead of passing without a version.

Spec: docs/superpowers/specs/2026-10-16-fix-922-design.md. Plan: docs/superpowers/plans/2026-10-16-fix-922.md (branch fix-922). Build starts now. The Garage probe (with P8, metadata) will be run by Nikola on TrueNAS.

## Approved (Nikola, 2026-10-16): spec rulings with visible effects Nikola: "i view them all as improvements". - **R6:** a version's "by" is the person who saved that content. - **R7:** an editing burst keeps one more version than today; still at most one autosave version per person per window. - **R5:** saving identical content leaves the file untouched. - **R3 trade-off:** while a version folder cannot be written, the following saves of an editing burst are also refused (rescue copy and bell as in #908), instead of passing without a version. Spec: `docs/superpowers/specs/2026-10-16-fix-922-design.md`. Plan: `docs/superpowers/plans/2026-10-16-fix-922.md` (branch `fix-922`). Build starts now. The Garage probe (with P8, metadata) will be run by Nikola on TrueNAS.
Cordy closed this issue 2026-10-17 02:10:42 +00:00
Author
Owner

Shipped in v0.6.347 (PR #952, merged 760d7c1d). Both dogfoods run it. I'll close this issue once the dogfood checks have passed.

What changed

  • Record, then publish. Every save now stores its own content as a version record first, and only then becomes the file's content.
  • Commit check. A short in-process commit step (versions.PathSerializer) checks that the content being replaced has a record, and records it if it has none.
  • Version keys. Keys look like …T…nnnnnnnnnZ_<12 random digits>[a|u]_<author>.v. They are unique and ordered by time, with no limit per second.
  • How a file is matched to its record:
    • posix: a hard link (copy mode where links are unavailable);
    • S3: object metadata x-amz-meta-cairn-record, with a boot probe.
  • Version list. The record of the current content is hidden.
  • Retention. Pruning has a one-hour grace. A save that has waited more than 30 minutes remakes its record. A version's age now counts from when it was replaced.
  • Key drain. It keeps a file and its record together, and leaves alone any file that someone saves while it runs.

Proof

  • Probes on main: all four fault tests fail (live345 olga-2-IMPORTANT lost; 12 saves in one second leave w4 w3 w2; the stress run loses about half). They all pass on the branch.
  • Local live check:
    • On v0.6.346 the faults reproduce.
    • On v0.6.347 all checks pass: t_5x 3/3; identical saves keep ETag and Last-Modified; WOPI refusal plus rescue copy; restore.
    • Two instances on one data root: 480 concurrent writes, 0 lost, 0 refused, both instances list the same versions.
  • Dogfood boot lines:
    • files (posix on NFS): versions: records by hard link.
    • files-bao (Garage): s3: records published by CopyObject. Garage keeps object metadata on CopyObject, so the design's S3 path runs in its best mode. The boot probe replaces the manual P8 probe.

Rulings

  • R1: random-token keys, ordered by time.
  • R3: if only a save's own record fails, the save is published unrecorded (WARN + cairn_versions_unrecorded_writes_total); the next save records it.
  • R4: a refusal after the upload keeps the body as a u version, and the WOPI rescue copy is written from it.
  • R5: an identical save leaves the file untouched.
  • R6: "by" names who saved that content.
  • R7: a burst of editor autosaves keeps its first and last save.
  • R8: S3 identity is object metadata plus a boot probe.
  • R9: posix identity is open+fstat.
  • R13: Cairn refuses to start when the storage stack cannot record.
  • R17: the drain keeps a file and its record together.
  • R19–R22: see #934 and the ADR amendment for hard links. Backups must keep hard links (rsync -aH).

Found and filed during the work

#939, #940, #941, #942, #943, #944, #945, #946, #947, #949, #950, #951.

**Shipped in v0.6.347** (PR #952, merged 760d7c1d). Both dogfoods run it. I'll close this issue once the dogfood checks have passed. ## What changed - **Record, then publish.** Every save now stores its own content as a version record first, and only then becomes the file's content. - **Commit check.** A short in-process commit step (`versions.PathSerializer`) checks that the content being replaced has a record, and records it if it has none. - **Version keys.** Keys look like `…T…nnnnnnnnnZ_<12 random digits>[a|u]_<author>.v`. They are unique and ordered by time, with no limit per second. - **How a file is matched to its record:** - posix: a hard link (copy mode where links are unavailable); - S3: object metadata `x-amz-meta-cairn-record`, with a boot probe. - **Version list.** The record of the current content is hidden. - **Retention.** Pruning has a one-hour grace. A save that has waited more than 30 minutes remakes its record. A version's age now counts from when it was replaced. - **Key drain.** It keeps a file and its record together, and leaves alone any file that someone saves while it runs. ## Proof - **Probes on main:** all four fault tests fail (live345 `olga-2-IMPORTANT` lost; 12 saves in one second leave w4 w3 w2; the stress run loses about half). They all pass on the branch. - **Local live check:** - On v0.6.346 the faults reproduce. - On v0.6.347 all checks pass: t_5x 3/3; identical saves keep ETag and Last-Modified; WOPI refusal plus rescue copy; restore. - **Two instances on one data root:** 480 concurrent writes, 0 lost, 0 refused, both instances list the same versions. - **Dogfood boot lines:** - files (posix on NFS): `versions: records by hard link`. - files-bao (Garage): `s3: records published by CopyObject`. Garage keeps object metadata on CopyObject, so the design's S3 path runs in its best mode. The boot probe replaces the manual P8 probe. ## Rulings - **R1:** random-token keys, ordered by time. - **R3:** if only a save's own record fails, the save is published unrecorded (WARN + `cairn_versions_unrecorded_writes_total`); the next save records it. - **R4:** a refusal after the upload keeps the body as a `u` version, and the WOPI rescue copy is written from it. - **R5:** an identical save leaves the file untouched. - **R6:** "by" names who saved that content. - **R7:** a burst of editor autosaves keeps its first and last save. - **R8:** S3 identity is object metadata plus a boot probe. - **R9:** posix identity is open+fstat. - **R13:** Cairn refuses to start when the storage stack cannot record. - **R17:** the drain keeps a file and its record together. - **R19–R22:** see #934 and the ADR amendment for hard links. Backups must keep hard links (`rsync -aH`). ## Found and filed during the work #939, #940, #941, #942, #943, #944, #945, #946, #947, #949, #950, #951.
Author
Owner

Manual Garage probe (2026-10-17): this run is invalid.

Every request answered 404 NoSuchBucket for cairn-probe-922-cairn-probe-922-20261017: the prefix was doubled in BUCKET, so the script addressed a bucket that does not exist. Its verdicts were read from those error answers and mean nothing (P1–P4 "IGNORED", P6 "FAIL", P7 "20 stale", P8 "MODE none"). The script now runs a preflight PUT and stops with exit 3 when the bucket isn't writable, so a wrong name can no longer produce verdicts.

What the release relies on is already measured. Cairn's own boot probe ran on the real Garage bucket at the v0.6.347 rollout of files-bao and logged s3: records published by CopyObject. That probe does a PUT with x-amz-meta-cairn-record, a HEAD, a CopyObject (COPY directive) and a HEAD of the copy, so Garage keeps object metadata on PUT and on CopyObject. That is P8's question, answered in production.

Still open (informational, the design does not depend on them):

  • P1–P4: conditional writes (expected ignored on Garage);
  • P5: ETag on copy;
  • P6/P7: list-after-write and read-after-overwrite (Garage documents both as consistent; worth confirming);
  • P8 REPLACE.

A rerun with BUCKET set to the exact name from garage bucket create answers them.

**Manual Garage probe (2026-10-17): this run is invalid.** Every request answered `404 NoSuchBucket` for `cairn-probe-922-cairn-probe-922-20261017`: the prefix was doubled in `BUCKET`, so the script addressed a bucket that does not exist. Its verdicts were read from those error answers and mean nothing (P1–P4 "IGNORED", P6 "FAIL", P7 "20 stale", P8 "MODE none"). The script now runs a preflight PUT and stops with exit 3 when the bucket isn't writable, so a wrong name can no longer produce verdicts. **What the release relies on is already measured.** Cairn's own boot probe ran on the real Garage bucket at the v0.6.347 rollout of files-bao and logged `s3: records published by CopyObject`. That probe does a PUT with `x-amz-meta-cairn-record`, a HEAD, a CopyObject (COPY directive) and a HEAD of the copy, so Garage keeps object metadata on PUT and on CopyObject. That is P8's question, answered in production. **Still open (informational, the design does not depend on them):** - P1–P4: conditional writes (expected ignored on Garage); - P5: ETag on copy; - P6/P7: list-after-write and read-after-overwrite (Garage documents both as consistent; worth confirming); - P8 REPLACE. A rerun with `BUCKET` set to the exact name from `garage bucket create` answers them.
Author
Owner

Dogfood checks for v0.6.347 (2026-10-17): all pass

Boot lines

  • files (posix on NFS): versions: records by hard link.
  • files-bao (Garage): s3: records published by CopyObject.

The live345 race

Each run did a WOPI PutFile, a REST upload of olga-2-IMPORTANT, then a second PutFile. This ran 3 times on each instance.

files files-bao
olga-2-IMPORTANT kept as a version 3/3 3/3
current content emil-save-2 3/3 3/3
current content missing from the version list 3/3 3/3

On v0.6.346 the same race lost olga-2-IMPORTANT every time.

Same-content save

Last-Modified stays the same, and no version is added. Checked on both instances.

Versions panel (files-bao)

A file saved three times shows 2 versions and no row for the current content.

All test files were removed and their trash entries purged. Closing.

## Dogfood checks for v0.6.347 (2026-10-17): all pass ### Boot lines - **files** (posix on NFS): `versions: records by hard link`. - **files-bao** (Garage): `s3: records published by CopyObject`. ### The live345 race Each run did a WOPI PutFile, a REST upload of `olga-2-IMPORTANT`, then a second PutFile. This ran 3 times on each instance. | | files | files-bao | |---|---|---| | `olga-2-IMPORTANT` kept as a version | 3/3 | 3/3 | | current content `emil-save-2` | 3/3 | 3/3 | | current content missing from the version list | 3/3 | 3/3 | On v0.6.346 the same race lost `olga-2-IMPORTANT` every time. ### Same-content save Last-Modified stays the same, and no version is added. Checked on both instances. ### Versions panel (files-bao) A file saved three times shows 2 versions and no row for the current content. All test files were removed and their trash entries purged. Closing.
Author
Owner

Valid Garage probe run (2026-10-17, second run, with the right bucket name). Every result matches what the design expects.

Probe Result What it means for Cairn
P1–P3: conditional PUT and CopyObject (If-None-Match, If-Match) ignored As documented for Garage. #922 never relies on conditional writes, so there is no impact. #949 (create-only PUT) cannot use this on Garage.
P4: 20 parallel create-only PUTs 20/20 answered 200, last writer wins Same as P1–P3.
P5: CopyObject keeps the ETag same ETag Not used: the design does not depend on ETags.
P6: list-after-write 50/50 Listings are consistent, which version chains need.
P7: read-after-overwrite 0/20 stale Reads are consistent, which the commit check needs.
P8: user metadata on PUT, and on CopyObject with COPY kept Mode copy, the same as the files-bao boot line.
P8: CopyObject with REPLACE honoured A file copy records server-side without streaming.

The manual probe agrees with Cairn's boot probe on files-bao. No action needed.

**Valid Garage probe run (2026-10-17, second run, with the right bucket name).** Every result matches what the design expects. | Probe | Result | What it means for Cairn | |---|---|---| | P1–P3: conditional PUT and CopyObject (`If-None-Match`, `If-Match`) | ignored | As documented for Garage. #922 never relies on conditional writes, so there is no impact. #949 (create-only PUT) cannot use this on Garage. | | P4: 20 parallel create-only PUTs | 20/20 answered 200, last writer wins | Same as P1–P3. | | P5: CopyObject keeps the ETag | same ETag | Not used: the design does not depend on ETags. | | P6: list-after-write | 50/50 | Listings are consistent, which version chains need. | | P7: read-after-overwrite | 0/20 stale | Reads are consistent, which the commit check needs. | | P8: user metadata on PUT, and on CopyObject with COPY | kept | Mode `copy`, the same as the files-bao boot line. | | P8: CopyObject with REPLACE | honoured | A file copy records server-side without streaming. | The manual probe agrees with Cairn's boot probe on files-bao. No action needed.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#922
No description provided.