Version history silently not written on cairn-enc: every version mint fails with Garage 403 "Invalid signature" #670

New issue

Closed

opened 2026-09-23 03:01:32 +00:00 by Cordy · 0 comments

Cordy commented

2026-09-23 03:01:32 +00:00

Owner

Found in the #656 QA pass (v0.6.247, files.c0rdyceps.ch / cairn-enc, S3 backend = Garage bucket cairn-enc).

Matrix row: §3 "Versions drill-down: Edit a file, open Versions → rows with restore/download actions"; §7 "Collabora edit: save round-trips".

Steps

_qa → New → Excel spreadsheet qa-sheet.xlsx (opens in Collabora).
Type QA656 into A1 and save (⌘S), then Back. Collabora saves again on close.
Details pane → Versions.

Expected: at least one previous version (the empty template) with Restore / Download.

Seen: "No previous versions." The save itself worked (POST /wopi/files/…/contents → 200, twice), but the server log shows the version write failing every time:

level=WARN msg="versions: mint failed" path=/home/nikola-test/_qa/qa-sheet.xlsx
  err="s3 put: status 403: <Error><Code>AccessDenied</Code>
       <Message>Forbidden: Invalid signature</Message>
       <Resource>/cairn-enc/home/nikola-test/_qa/.cairn-versions/qa-sheet.xlsx/2026-09-23T02:58:26Z_1_nikola-test.v</Resource>
       <Region>garage</Region></Error>"

The same 403 appears for .cairn-meta.json on every metadata write in _qa (02:57:44, 02:57:56, 02:58:26, 02:58:37). GET /api/v1/versions returns {"items":[]} for every file checked on this instance, including older documents in Personal.

Why this is serious

It fails silently: WARN in the log, 200 to the client, no UI signal. Users and admins believe edits are versioned, and they aren't. Anyone relying on "restore the previous version" after a bad edit or ransomware-style overwrite has nothing to restore.
Retention and holds policy presumably assume versions exist.

Likely cause (hypothesis, not verified): the version object key embeds an RFC 3339 timestamp with : characters (…/2026-09-23T02:58:26Z_1_nikola-test.v). A SigV4 canonical-URI encoding mismatch on : between the Cairn S3 client and Garage produces exactly Invalid signature, while other object PUTs, without colons, succeed. Worth checking files-bao's backend for the same pattern, and adding a CI check (e.g. to interop-s3) that mints a version against Garage and lists it back.

Also noted: .cairn-meta.json, an internal metadata file, is itself being put through version minting on every tag/favourite write. Probably unintended.

Screenshot: 670-no-versions.jpg (outputs folder)

Found in the #656 QA pass (v0.6.247, files.c0rdyceps.ch / cairn-enc, S3 backend = Garage bucket `cairn-enc`). **Matrix row:** §3 "Versions drill-down: *Edit a file, open Versions → rows with restore/download actions*"; §7 "Collabora edit: *save round-trips*". **Steps** 1. `_qa` → New → Excel spreadsheet `qa-sheet.xlsx` (opens in Collabora). 2. Type `QA656` into A1 and save (⌘S), then Back. Collabora saves again on close. 3. Details pane → **Versions**. **Expected:** at least one previous version (the empty template) with Restore / Download. **Seen:** **"No previous versions."** The save itself worked (`POST /wopi/files/…/contents` → **200**, twice), but the server log shows the version write failing every time: ``` level=WARN msg="versions: mint failed" path=/home/nikola-test/_qa/qa-sheet.xlsx err="s3 put: status 403: <Error><Code>AccessDenied</Code> <Message>Forbidden: Invalid signature</Message> <Resource>/cairn-enc/home/nikola-test/_qa/.cairn-versions/qa-sheet.xlsx/2026-09-23T02:58:26Z_1_nikola-test.v</Resource> <Region>garage</Region></Error>" ``` The same 403 appears for `.cairn-meta.json` on every metadata write in `_qa` (02:57:44, 02:57:56, 02:58:26, 02:58:37). `GET /api/v1/versions` returns `{"items":[]}` for every file checked on this instance, including older documents in Personal. **Why this is serious** - It fails **silently**: WARN in the log, 200 to the client, no UI signal. Users and admins believe edits are versioned, and they aren't. Anyone relying on "restore the previous version" after a bad edit or ransomware-style overwrite has nothing to restore. - Retention and holds policy presumably assume versions exist. **Likely cause (hypothesis, not verified):** the version object key embeds an RFC 3339 timestamp with `:` characters (`…/2026-09-23T02:58:26Z_1_nikola-test.v`). A SigV4 canonical-URI encoding mismatch on `:` between the Cairn S3 client and Garage produces exactly `Invalid signature`, while other object PUTs, without colons, succeed. Worth checking files-bao's backend for the same pattern, and adding a CI check (e.g. to interop-s3) that mints a version against Garage and lists it back. **Also noted:** `.cairn-meta.json`, an internal metadata file, is itself being put through version minting on every tag/favourite write. Probably unintended. **Screenshot:** `670-no-versions.jpg` (outputs folder)