Versions on AWS S3 / MinIO: create-only PUT for new files (If-None-Match), D4 follow-up #949

Open
opened 2026-10-17 02:01:21 +00:00 by Cordy · 1 comment
Owner

From the fix-922 spec §14 (2026-10-16), follow-up to decision D4.

Background: since v0.6.347 every save records its content before publishing it (#922). On AWS S3 and MinIO the record of each file's current content is a second stored copy; Garage deduplicates it. This was accepted (D4).

Idea: for a brand-new file, the record exists only to keep the content safe against another instance that saw no file and writes its own. AWS S3 and MinIO support conditional writes (If-None-Match: *). A create-only PUT of the file itself would make that race safe without a record, so new files would not need the second copy until their first overwrite. Garage ignores conditional writes, so it has to stay on the record path.

To do:

  • At boot, probe whether the store honours If-None-Match (alongside the record probe).
  • In that mode, publish new files with a create-only PUT and skip the record for them. Their first overwrite records them, as before #922.
  • Twin tests with the fake in both modes.
  • Measure the storage saved.
From the fix-922 spec §14 (2026-10-16), follow-up to decision D4. **Background:** since v0.6.347 every save records its content before publishing it (#922). On AWS S3 and MinIO the record of each file's current content is a second stored copy; Garage deduplicates it. This was accepted (D4). **Idea:** for a brand-new file, the record exists only to keep the content safe against another instance that saw no file and writes its own. AWS S3 and MinIO support conditional writes (`If-None-Match: *`). A create-only PUT of the file itself would make that race safe without a record, so new files would not need the second copy until their first overwrite. Garage ignores conditional writes, so it has to stay on the record path. **To do:** - At boot, probe whether the store honours `If-None-Match` (alongside the record probe). - In that mode, publish new files with a create-only PUT and skip the record for them. Their first overwrite records them, as before #922. - Twin tests with the fake in both modes. - Measure the storage saved.
Author
Owner

Garage probe (2026-10-17): Garage ignores If-None-Match: * on PUT. When 20 create-only PUTs of one key ran in parallel, all 20 were accepted. On Garage this idea can therefore never apply, and Garage keeps the record path. The idea is only worth doing for AWS S3 and MinIO, and the boot probe has to tell those stores apart.

Garage probe (2026-10-17): Garage ignores `If-None-Match: *` on PUT. When 20 create-only PUTs of one key ran in parallel, all 20 were accepted. On Garage this idea can therefore never apply, and Garage keeps the record path. The idea is only worth doing for AWS S3 and MinIO, and the boot probe has to tell those stores apart.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#949
No description provided.