s3 driver: a range read past the end that gets an empty 206 loops forever #940

Closed
opened 2026-10-16 20:53:21 +00:00 by Cordy · 2 comments
Owner

Found during fix-922 Task 6 (2026-10-16).

What happens: the s3 driver's ranged reader keeps asking for the next range. If an S3-compatible store answers a range that starts past the end of the object with an empty 206 instead of 416, the reader never finishes. On the test fake this happened when two Cairn instances rewrote the shared owner manifest at the same time: the object shrank between the size check and the read, and TestTwinRenameAndWrite on s3-encrypt hung.

Real AWS S3 answers 416, and the fix-922 branch now makes the test fake do the same. Other S3-compatible stores may not. Garage's behaviour is not yet checked.

To do:

  • In the s3 driver, treat a 206 with zero bytes, or a Content-Range past the end, as end of object. Never loop.
  • Add a test using a fake that answers an empty 206.
  • Check Garage's behaviour with the probe.

Relevant for #742 (several instances on one bucket).

Found during fix-922 Task 6 (2026-10-16). **What happens:** the s3 driver's ranged reader keeps asking for the next range. If an S3-compatible store answers a range that starts past the end of the object with an empty `206` instead of `416`, the reader never finishes. On the test fake this happened when two Cairn instances rewrote the shared owner manifest at the same time: the object shrank between the size check and the read, and `TestTwinRenameAndWrite` on s3-encrypt hung. Real AWS S3 answers `416`, and the fix-922 branch now makes the test fake do the same. Other S3-compatible stores may not. Garage's behaviour is not yet checked. **To do:** - In the s3 driver, treat a `206` with zero bytes, or a `Content-Range` past the end, as end of object. Never loop. - Add a test using a fake that answers an empty `206`. - Check Garage's behaviour with the probe. Relevant for #742 (several instances on one bucket).
Cordy closed this issue 2026-10-17 15:47:20 +00:00
Author
Owner

Shipped in v0.6.348 (PR #955), running on both dogfoods.

  • Short object: an empty 206, a 416, or a 206 whose Content-Range starts at a different offset now ends the read with a visible error. It logs WARN s3: object ended before its size with a reason. Before the fix it looped forever or spliced in wrong bytes.
  • Store that ignores Range: if a resumed GET at a non-zero offset comes back as a full 200 with no Content-Range, the read skips forward to the right offset. Previously the whole object was silently spliced in at the resume point.
  • Download marking: this no longer loops when the object turns out short. On S3 the reviewer measured about 80,000 GETs in 3 s before the fix.

Garage's own answer to an out-of-range read hasn't been probed yet. The fix handles every variant either way.

**Shipped in v0.6.348 (PR #955), running on both dogfoods.** - **Short object:** an empty 206, a 416, or a 206 whose `Content-Range` starts at a different offset now ends the read with a visible error. It logs WARN `s3: object ended before its size` with a `reason`. Before the fix it looped forever or spliced in wrong bytes. - **Store that ignores Range:** if a resumed GET at a non-zero offset comes back as a full 200 with no `Content-Range`, the read skips forward to the right offset. Previously the whole object was silently spliced in at the resume point. - **Download marking:** this no longer loops when the object turns out short. On S3 the reviewer measured about 80,000 GETs in 3 s before the fix. Garage's own answer to an out-of-range read hasn't been probed yet. The fix handles every variant either way.
Author
Owner

Live status: not reproducible on the dogfoods. Garage answers range reads correctly, so neither the short object nor an ignored Range happens there. The fix is covered by driver tests against a fake S3 store, one for each misbehaviour: empty 206, 416, a wrong offset, and an ignored Range. On files-bao (v0.6.350, current pod log since start), reads work normally and no WARN s3: object ended before its size appears.

**Live status:** not reproducible on the dogfoods. Garage answers range reads correctly, so neither the short object nor an ignored Range happens there. The fix is covered by driver tests against a fake S3 store, one for each misbehaviour: empty 206, 416, a wrong offset, and an ignored Range. On files-bao (v0.6.350, current pod log since start), reads work normally and no WARN `s3: object ended before its size` appears.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#940
No description provided.