Storage edge cases from batch A (S3 limits, reserve on S3, quota walks, name length after directory moves) #925

Closed
opened 2026-10-15 21:22:24 +00:00 by Cordy · 2 comments
Owner

Follow-ups from batch A (v0.6.343):

  1. S3 objects over 5 GiB: they cannot be written or have a version made on AWS, because Cairn uses a single PUT and a single CopyObject. Use multipart if AWS over 5 GiB matters. Whether Garage's CopyObject shares blocks is verified only from its source.
  2. S3 with reserveBytes > 0: every write is refused, because S3 reports Free 0. The reserve check also lists the whole bucket.
  3. Quota walks: a cold quota walk has no singleflight, so concurrent requests each walk the tree.
  4. Name length after a directory move: the name-length guard (908-F6) checks only the path it creates. A directory renamed or copied to a deeper path can carry files whose version keys no longer fit, and their first overwrite is refused as a keylimit mint failure.
Follow-ups from batch A (v0.6.343): 1. **S3 objects over 5 GiB:** they cannot be written or have a version made on AWS, because Cairn uses a single PUT and a single CopyObject. Use multipart if AWS over 5 GiB matters. Whether Garage's CopyObject shares blocks is verified only from its source. 2. **S3 with `reserveBytes > 0`:** every write is refused, because S3 reports Free 0. The reserve check also lists the whole bucket. 3. **Quota walks:** a cold quota walk has no singleflight, so concurrent requests each walk the tree. 4. **Name length after a directory move:** the name-length guard (908-F6) checks only the path it creates. A directory renamed or copied to a deeper path can carry files whose version keys no longer fit, and their first overwrite is refused as a `keylimit` mint failure.
Author
Owner

Shipped in v0.6.348 (PR #955), items 2 and 3.

Item 2. reserveBytes now applies only where the storage reports how much free space it has, which means POSIX. On S3 it has no effect, and Cairn logs a WARN once at start. Writes are no longer refused, and the bucket is no longer listed on every write.

Item 3. How quota usage is measured:

  • Concurrent requests share one walk of the tree, and that walk is detached from the request that started it.
  • A cancelled, failed or panicked walk never caches a partial figure.
  • While a refresh runs in the background, the last figure stays in use, plus any writes credited during the refresh.
  • A failed refresh logs a WARN and backs off. A walk the storage never answers is abandoned after about 31 minutes.
  • Usage now also counts the entries POSIX keeps hidden (in-progress uploads and set-aside WebDAV entries).

Still open, waiting for a decision:

  • Item 1: AWS objects over 5 GiB need multipart upload. Does that matter for us?
  • Item 4: after a folder move, the version keys inside can exceed the name limit. Refuse such moves up front, or accept that the first overwrite of an affected file is refused later?

I'll ask both when this issue's batch comes up.

**Shipped in v0.6.348 (PR #955), items 2 and 3.** **Item 2.** `reserveBytes` now applies only where the storage reports how much free space it has, which means POSIX. On S3 it has no effect, and Cairn logs a WARN once at start. Writes are no longer refused, and the bucket is no longer listed on every write. **Item 3.** How quota usage is measured: - Concurrent requests share one walk of the tree, and that walk is detached from the request that started it. - A cancelled, failed or panicked walk never caches a partial figure. - While a refresh runs in the background, the last figure stays in use, plus any writes credited during the refresh. - A failed refresh logs a WARN and backs off. A walk the storage never answers is abandoned after about 31 minutes. - Usage now also counts the entries POSIX keeps hidden (in-progress uploads and set-aside WebDAV entries). **Still open, waiting for a decision:** - **Item 1:** AWS objects over 5 GiB need multipart upload. Does that matter for us? - **Item 4:** after a folder move, the version keys inside can exceed the name limit. Refuse such moves up front, or accept that the first overwrite of an affected file is refused later? I'll ask both when this issue's batch comes up.
Author
Owner

Decision (Nikola, 2026-10-27), items 1 and 4: guards, no multipart in v0.7.

  • Item 1: the store's "too large" answer (AWS and Cloudflare R2 above 5 GiB per object) becomes a named refusal, and the handbook stops implying that files over 5 GiB work everywhere. There is no fixed cap, because MinIO accepts single uploads up to 5 TiB.
  • Item 4: on S3, a folder move whose version keys would no longer fit is refused before the first copy (the driver checks after listAll), so a move never stops halfway. On POSIX the limit, which is about four times S3's, is documented.

Planned for batch R4, size S to M.

**Decision (Nikola, 2026-10-27), items 1 and 4:** guards, no multipart in v0.7. - **Item 1:** the store's "too large" answer (AWS and Cloudflare R2 above 5 GiB per object) becomes a named refusal, and the handbook stops implying that files over 5 GiB work everywhere. There is no fixed cap, because MinIO accepts single uploads up to 5 TiB. - **Item 4:** on S3, a folder move whose version keys would no longer fit is refused before the first copy (the driver checks after `listAll`), so a move never stops halfway. On POSIX the limit, which is about four times S3's, is documented. Planned for batch R4, size S to M.
Cordy closed this issue 2026-10-27 12:22:52 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#925
No description provided.