s3 driver: ghost object with decomposed-unicode key — lists but cannot be opened, invisible on every user surface #414
Labels
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Cordy/Cairn#414
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by the first real drain on cairn-enc (#411, v0.6.117/118).
Symptom
/home/Mülltrennung.docxexists as a backend object on the cairn-enc Garage bucket:not found: file does not exist, even in-process with the exact path the listing itself returned,Operator history: the file was uploaded once and then deleted, and it sits directly under
/home(not inside any user's home) — a pre-per-user-homes upload.Hypothesis
Unicode normalisation mismatch on the S3 key. macOS clients (Finder/WebDAV) write filenames decomposed (NFD:
u+ combining diaeresis); other surfaces send the composed form (NFC). Two renderings, two byte-different keys. The delete removed the twin it could address; the orphan remained. That the in-process list → open roundtrip also fails suggests the s3 driver (or Garage's list encoding) alters the key bytes somewhere between listing and GET — e.g. URL-decoding of the ListObjects response not matching the key used on GET.To investigate
192.168.10.32:3900, bucketcairn-enc, prefixhome/) and compare to what the driver's List returns.Cleanup (operator, independent of the fix)
Delete the exact key directly against Garage once its bytes are known. Until then the object is harmless: the scan reports it as skipped, the drain as failed-with-reason; it blocks nothing.
Evidence captured (2026-09-03). Direct
s3api list-objects-v2against Garage, key bytes viaascii():NFD confirmed —
u+ combining diaeresis (U+0308), the macOS decomposed form, consistent with a Finder/WebDAV upload. Noterepr()is useless for this check (combining marks are printable, so NFD and NFC render identically);ascii()is the tool.This sharpens the driver question: the object's key is plain NFD bytes and Garage serves it fine, yet Cairn's own List → Open roundtrip fails in-process — so something between the s3 driver's listing and its GET normalises or otherwise alters the key bytes. The reproduction test stands: PUT an NFD key through the driver, List, Open with the listed path; must be byte-identical.
Operator cleanup via
delete-objectwith the exact listed key (piped, never retyped — a Mac keyboard emits NFC, which would address the wrong bytes).Operator cleanup complete (2026-09-03). The NFD object was deleted with
s3api delete-objectusing the key piped byte-faithfully from the listing; a filtered re-list confirms it is gone. The cairn-enc backend is now clean — every object on exactly one key.What remains here is the driver fix: the List → Open roundtrip must be byte-identical for non-NFC keys, pinned by a test that PUTs a decomposed-umlaut key through the driver and opens it via its own listing. Until then, any NFD-named upload (macOS Finder/WebDAV is the standard source) risks becoming a new ghost the moment it is renamed or deleted through a surface that normalises.
Root cause (found in source, not hypothesised).
storage.CleanPathhas normalised every client path to NFC since #183 — and the s3 driver routes all paths through it viakey(), including paths returned by its own List. So for a legacy NFD key: List returns true NFD bytes → Open/Delete re-clean them to NFC → GET/DELETE address a nonexistent key. DELETE is idempotent (204 on a missing key), which is exactly how the original "successful" delete produced the ghost. #183's policy was half-finished: inputs normalise, but pre-#183 NFD objects became permanently unaddressable.Design (2026-09-03) — canonical writes + equivalence-tolerant addressing:
resolve(ctx, key)tries the given (NFC) form, then the NFD alternate, where existence means exact object, dir marker, or any child under the prefix. Wired into Stat/List/Open/Delete/Copy-source.Stat/Listreport the backend-true bytes, keeping roundtrips byte-faithful.storage.AltUnicodeFormshort-circuits for ASCII/form-agnostic paths — zero extra requests on ~all traffic; a couple of HEADs only when a unicode path misses.Note for later: the POSIX driver has the same theoretical exposure for externally-placed NFD files;
storage.AltUnicodeFormis deliberately in the storage package so posix can adopt the same fallback if it ever surfaces there.Shipped in v0.6.121 (PR #417, merged; both dogfoods rolled and healthy).
The driver now addresses canonically equivalent legacy keys:
storage.AltUnicodeForm+resolve()in Stat/List/Open/Delete/Copy-source, backend-true paths in Stat/List, in-place overwrites on legacy twins, canonical NFC for all new keys, zero cost on ASCII paths. Regression suite runs against an in-process byte-opaque fake S3 with rune-constructed keys (unicode literals in test sources are exposed to the very bug class under test — three transport layers normalised them during this build alone).Had this existed two months ago, the Mülltrennung ghost could never have formed: its delete would have resolved to the stored NFD bytes and actually removed the object.