ZIP downloads on S3: one flat listing per subtree instead of one LIST per folder #1121

Open
opened 2026-11-04 21:09:44 +00:00 by Cordy · 0 comments
Owner

zipdl.Build (v0.6.397, #1097) walks with List per folder, so a tree of many small folders on S3 costs one request (or more) per folder before the first byte.
A flat prefix listing (one LIST per 1,000 keys, e.g. the storage.TreeLister added for #1102) for the check would make large selections on S3 much faster; keep the walk order and the skip rules.

`zipdl.Build` (v0.6.397, #1097) walks with `List` per folder, so a tree of many small folders on S3 costs one request (or more) per folder before the first byte. A flat prefix listing (one LIST per 1,000 keys, e.g. the `storage.TreeLister` added for #1102) for the check would make large selections on S3 much faster; keep the walk order and the skip rules.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1121
No description provided.