Folder copy on S3: batch the per-file version checks (cost after #950) #1135

Open
opened 2026-11-06 10:16:33 +00:00 by Cordy · 0 comments
Owner

Since #950 (built in #742 Step 0) a folder copy records every copied file through the full single-file copy path. On S3 a 100-file folder copy measured about 3,100-3,650 requests on the fake (HEAD and LIST checks per file) instead of 129-843 before; the expected cost was one extra CopyObject per file. POSIX is not affected (one hard link per file).

Decision (Nikola, 2026-11-06): optimise before v1.0. Ideas from the build: list the source folder and the destination's version chains once per folder instead of per file; reuse the destination snapshot; skip the per-file manifest re-read when the folder copy already holds the folder's state. Measure on the S3 dogfood (Garage) before and after; keep the #950 twin-race test green.

Since #950 (built in #742 Step 0) a folder copy records every copied file through the full single-file copy path. On S3 a 100-file folder copy measured about 3,100-3,650 requests on the fake (HEAD and LIST checks per file) instead of 129-843 before; the expected cost was one extra CopyObject per file. POSIX is not affected (one hard link per file). Decision (Nikola, 2026-11-06): optimise before v1.0. Ideas from the build: list the source folder and the destination's version chains once per folder instead of per file; reuse the destination snapshot; skip the per-file manifest re-read when the folder copy already holds the folder's state. Measure on the S3 dogfood (Garage) before and after; keep the #950 twin-race test green.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1135
No description provided.