Download a folder or a selection as one ZIP #1097

Closed
opened 2026-11-04 03:10:41 +00:00 by Cordy · 8 comments
Owner

Gap (v0.7 feature-gap research, 2026-11-04, decided by Nikola). Bulk Download starts one browser download per selected file and silently skips folders; a folder row has no Download at all (web/static/index.html:13309-13315, 13478-13486 at 0e0885db). There is no ZIP code in internal/api (VERIFIED).

Competitors. OpenCloud streams ZIP or TAR for any selection (docs); Seafile streams folder ZIPs even in the free edition; Nextcloud and ownCloud 10 zip folders and selections. A tester hits this in the first five minutes.

To research and decide when worked on (playbook). Streaming without temp files; size and file-count limits and how they are shown; ZIP64; file-name encoding (UTF-8 flag, Windows Explorer and macOS Archive Utility); encrypted storage and virus-scan state of the files inside; public links and drop links; held or paused items; audit verb and log line; how a download of a huge space behaves on more than one instance (#742).

Refs: scratch/b7e/backlog/gap-v07/research-v07-gaps.md A1, research-opencloud.md, research-seafile.md.

**Gap (v0.7 feature-gap research, 2026-11-04, decided by Nikola).** Bulk Download starts one browser download per selected file and silently skips folders; a folder row has no Download at all (`web/static/index.html:13309-13315`, `13478-13486` at 0e0885db). There is no ZIP code in `internal/api` (VERIFIED). **Competitors.** OpenCloud streams ZIP or TAR for any selection ([docs](https://docs.opencloud.eu/docs/user/files-and-folders/upload-download-unzip)); Seafile streams folder ZIPs even in the free edition; Nextcloud and ownCloud 10 zip folders and selections. A tester hits this in the first five minutes. **To research and decide when worked on (playbook).** Streaming without temp files; size and file-count limits and how they are shown; ZIP64; file-name encoding (UTF-8 flag, Windows Explorer and macOS Archive Utility); encrypted storage and virus-scan state of the files inside; public links and drop links; held or paused items; audit verb and log line; how a download of a huge space behaves on more than one instance (#742). Refs: `scratch/b7e/backlog/gap-v07/research-v07-gaps.md` A1, research-opencloud.md, research-seafile.md.
Author
Owner

Research done (scratch/b7e/backlog/research-1097.md). Recommended design:

  • Two steps: POST /api/v1/zip with the selection checks it in the person's own view (limits, counts, size, what is left out) and stores a short-lived ticket in the statestore (works on any instance); the browser then downloads GET /api/v1/zip/<id>, same user only.
  • Streamed with Go archive/zip, Store (no compression), ZIP64 and UTF-8 names; every file read through the normal storage stack (decryption, permissions, per-file read audit, leak marks). Names via zipout.SafeEntryName; case-only duplicates renamed; Not included.txt lists skipped and renamed items.
  • A file that cannot be opened is skipped and listed; a failure mid-file or a cancel aborts the download so no truncated ZIP looks complete; work stops when the client disconnects.
  • Admin limits, default 10,000 files and 100 GB (0 = no limit), checked before the first byte: 413 zip-too-many-files / zip-too-large; at most 2 per person and 8 per instance at once, else 429 zip-busy.
  • Unscanned and held files included like single downloads; limited devices already refused by the central gate.
  • Public folder links: "Download all as ZIP" behind every link check; encrypted links stream one age stream decrypted in the page, with a 2 GB in-browser cap and the per-file fallback above it.
  • New access-tier audit verb download-zip; INFO zip download refused, WARN zip download stopped: a file could not be read.

Found on the way: bulk Download in search, Favourites and Recents filters the wrong list (index.html:13480); Content-Disposition is built by string concatenation (internal/api/preview.go:41,44), so a " breaks the header and non-ASCII names get no filename*. Both are fixed in this batch.

Decisions asked of Nikola next.

**Research done** (`scratch/b7e/backlog/research-1097.md`). Recommended design: - Two steps: `POST /api/v1/zip` with the selection checks it in the person's own view (limits, counts, size, what is left out) and stores a short-lived ticket in the statestore (works on any instance); the browser then downloads `GET /api/v1/zip/<id>`, same user only. - Streamed with Go `archive/zip`, Store (no compression), ZIP64 and UTF-8 names; every file read through the normal storage stack (decryption, permissions, per-file `read` audit, leak marks). Names via `zipout.SafeEntryName`; case-only duplicates renamed; `Not included.txt` lists skipped and renamed items. - A file that cannot be opened is skipped and listed; a failure mid-file or a cancel aborts the download so no truncated ZIP looks complete; work stops when the client disconnects. - Admin limits, default 10,000 files and 100 GB (0 = no limit), checked before the first byte: 413 `zip-too-many-files` / `zip-too-large`; at most 2 per person and 8 per instance at once, else 429 `zip-busy`. - Unscanned and held files included like single downloads; limited devices already refused by the central gate. - Public folder links: "Download all as ZIP" behind every link check; encrypted links stream one age stream decrypted in the page, with a 2 GB in-browser cap and the per-file fallback above it. - New access-tier audit verb `download-zip`; INFO `zip download refused`, WARN `zip download stopped: a file could not be read`. Found on the way: bulk Download in search, Favourites and Recents filters the wrong list (`index.html:13480`); `Content-Disposition` is built by string concatenation (`internal/api/preview.go:41,44`), so a `"` breaks the header and non-ASCII names get no `filename*`. Both are fixed in this batch. Decisions asked of Nikola next.
Author
Owner

Decisions (Nikola, 2026-11-04): all recommendations approved. Two-step ticket flow; limits 10,000 files and 100 GB (0 = none), 2 per person and 8 per instance at once; ZIP for every OS, with the manual >4 GiB check in Windows Explorer, macOS Archive Utility, 7-Zip and unzip before release; public folder links get "Download all as ZIP" (2 GB in-browser cap for encrypted links, per-file fallback above); per-file read audit plus download-zip; unscanned files included like single downloads; exact names kept (Windows-safe option after launch). Next: whole-page mockup.

**Decisions (Nikola, 2026-11-04): all recommendations approved.** Two-step ticket flow; limits 10,000 files and 100 GB (0 = none), 2 per person and 8 per instance at once; ZIP for every OS, with the manual >4 GiB check in Windows Explorer, macOS Archive Utility, 7-Zip and unzip before release; public folder links get "Download all as ZIP" (2 GB in-browser cap for encrypted links, per-file fallback above); per-file `read` audit plus `download-zip`; unscanned files included like single downloads; exact names kept (Windows-safe option after launch). Next: whole-page mockup.
Author
Owner

Mockup approved (Nikola, 2026-11-04): 1097-zip-download-mockup.html, frames A to I, its strings, and all nine open points as recommended (limits box in Storage & quotas › Limits; confirm above a fixed 1 GB or when items are left out; silent re-check after 90 s; 2 per public link within 8 per instance; 2 GB cap for encrypted links in v1; no bell for the link creator; Not included.txt in the person's language; a lost slot just fails; the file-row download icon's label names the file). Next: spec and plan (.superpowers/sdd/2026-11-04-zp1097).

**Mockup approved (Nikola, 2026-11-04):** `1097-zip-download-mockup.html`, frames A to I, its strings, and all nine open points as recommended (limits box in Storage & quotas › Limits; confirm above a fixed 1 GB or when items are left out; silent re-check after 90 s; 2 per public link within 8 per instance; 2 GB cap for encrypted links in v1; no bell for the link creator; Not included.txt in the person's language; a lost slot just fails; the file-row download icon's label names the file). Next: spec and plan (`.superpowers/sdd/2026-11-04-zp1097`).
Author
Owner

Spec open points decided (Nikola, 2026-11-04), after research (research-1097-unsettled.md): all verdicts approved. Notably: hard ceiling of 100,000 entries per ZIP (the admin hint names it); Content-Disposition fix at four sites (incl. PDF preview and OCM); one running check per person and a 10-minute ticket sweep; the GB field keeps fractions; FR/IT fixes. Bulk Copy/Move/Favourites/Delete on the wrong list is filed as #1112.

**Spec open points decided (Nikola, 2026-11-04), after research (`research-1097-unsettled.md`):** all verdicts approved. Notably: hard ceiling of 100,000 entries per ZIP (the admin hint names it); Content-Disposition fix at four sites (incl. PDF preview and OCM); one running check per person and a 10-minute ticket sweep; the GB field keeps fractions; FR/IT fixes. Bulk Copy/Move/Favourites/Delete on the wrong list is filed as #1112.
Author
Owner

Strings approved (Nikola, 2026-11-04): pubZipTooManyMore for a public link whose walk stopped early (EN "This folder has more than {limit}, the most one ZIP download can hold. Download the files one by one instead.", DE "Dieser Ordner enthält mehr als {limit}, das Maximum für einen ZIP-Download. Laden Sie die Dateien stattdessen einzeln herunter.", FR/IT alike), and a dative German audit-note key so the note reads "… aus 5 Elementen".

**Strings approved (Nikola, 2026-11-04):** `pubZipTooManyMore` for a public link whose walk stopped early (EN "This folder has more than {limit}, the most one ZIP download can hold. Download the files one by one instead.", DE "Dieser Ordner enthält mehr als {limit}, das Maximum für einen ZIP-Download. Laden Sie die Dateien stattdessen einzeln herunter.", FR/IT alike), and a dative German audit-note key so the note reads "… aus 5 Elementen".
Cordy closed this issue 2026-11-04 20:56:36 +00:00
Author
Owner

v0.6.397 is live on both dogfoods (files.c0rdyceps.ch posix, files-bao.c0rdyceps.ch s3; cairnd listening … version=v0.6.397, both healthy, POST /api/v1/zip without a session answers 401). This issue stays open until the ZIP64 gate below passes (spec §17).

ZIP64 gate, step by step (run on files.c0rdyceps.ch, encryption on)

A. Test data (Mac, Terminal)

  1. mkdir -p ~/zipgate/Gate/Leer && cd ~/zipgate/Gate
  2. dd if=/dev/urandom of=big-5GiB.bin bs=1048576 count=5120 (5 GiB, a few minutes)
  3. printf 'umlaut\n' > "Grüße äöü.txt"; printf 'cjk\n' > "日本語テスト.txt"; printf 'small\n' > "small.txt"
  4. shasum -a 256 big-5GiB.bin "Grüße äöü.txt" "日本語テスト.txt" small.txt > ~/zipgate/expected.sha256

B. Into Cairn. v0.6.397 has no folder upload yet (that is v0.6.398), so: in Personal, New folder Gate, open it, New folder Leer, then Upload the four files (or rclone copy ~/zipgate/Gate cairn:Gate with a WebDAV remote and an app password). Wait until big-5GiB.bin shows 5 GB.

C. Download. Tick Gate, click Download. Expect "Preparing the ZIP…", then the confirm dialog "4 files in 2 folders, 5.0 GB in total." and "Saved as Gate.zip"; click Download.

D. unzip: unzip -t ~/Downloads/Gate.zip → "No errors detected…"; mkdir -p /tmp/g-unzip && unzip -q ~/Downloads/Gate.zip -d /tmp/g-unzip && (cd /tmp/g-unzip/Gate && shasum -a 256 -c ~/zipgate/expected.sha256) → every line OK; Leer is there.

E. 7-Zip: brew install sevenzip (your call), then 7zz t ~/Downloads/Gate.zip → "Everything is Ok" (note any Headers Error or Warnings line); 7zz x ~/Downloads/Gate.zip -o/tmp/g-7z && (cd /tmp/g-7z/Gate && shasum -a 256 -c ~/zipgate/expected.sha256).

F. macOS Archive Utility: double-click Gate.zip in Downloads; no "Unable to expand"/"Error 79"; cd ~/Downloads/Gate && shasum -a 256 -c ~/zipgate/expected.sha256; the two non-ASCII names read correctly in Finder.

G. Windows 11 Explorer: note winver; double-click Gate.zip (four files and Leer, correct names); Extract All… to C:\gate; PowerShell Get-FileHash -Algorithm SHA256 C:\gate\Gate\* | Format-List, compare with expected.sha256 (case-insensitive).

H. Report here a small table: tool, version, test result, checksums match, names correct, warnings. I add the release build's go version.

  • Pass: all four extract big-5GiB.bin with the right SHA-256 and the names read correctly (a 7-Zip header warning with a correct extraction still passes and goes into the handbook's known limits). I close this issue.
  • Fail: I prepare gate-big-nots.zip (no 0x5455 extra) to find the cause, then you decide between the approved contingency (files of 4 GiB or more left out with reason too-large) and a ZIP64 local-header fix before more releases build on this.

Clean up afterwards: delete Gate in Cairn (it goes to Deleted files) and ~/zipgate, ~/Downloads/Gate*, /tmp/g-* on the Mac.

**v0.6.397 is live on both dogfoods** (files.c0rdyceps.ch posix, files-bao.c0rdyceps.ch s3; `cairnd listening … version=v0.6.397`, both healthy, `POST /api/v1/zip` without a session answers 401). This issue stays open until the ZIP64 gate below passes (spec §17). ## ZIP64 gate, step by step (run on files.c0rdyceps.ch, encryption on) **A. Test data (Mac, Terminal)** 1. `mkdir -p ~/zipgate/Gate/Leer && cd ~/zipgate/Gate` 2. `dd if=/dev/urandom of=big-5GiB.bin bs=1048576 count=5120` (5 GiB, a few minutes) 3. `printf 'umlaut\n' > "Grüße äöü.txt"; printf 'cjk\n' > "日本語テスト.txt"; printf 'small\n' > "small.txt"` 4. `shasum -a 256 big-5GiB.bin "Grüße äöü.txt" "日本語テスト.txt" small.txt > ~/zipgate/expected.sha256` **B. Into Cairn.** v0.6.397 has no folder upload yet (that is v0.6.398), so: in Personal, **New folder** `Gate`, open it, **New folder** `Leer`, then **Upload** the four files (or `rclone copy ~/zipgate/Gate cairn:Gate` with a WebDAV remote and an app password). Wait until `big-5GiB.bin` shows 5 GB. **C. Download.** Tick `Gate`, click **Download**. Expect "Preparing the ZIP…", then the confirm dialog "4 files in 2 folders, 5.0 GB in total." and "Saved as Gate.zip"; click **Download**. **D. unzip:** `unzip -t ~/Downloads/Gate.zip` → "No errors detected…"; `mkdir -p /tmp/g-unzip && unzip -q ~/Downloads/Gate.zip -d /tmp/g-unzip && (cd /tmp/g-unzip/Gate && shasum -a 256 -c ~/zipgate/expected.sha256)` → every line OK; `Leer` is there. **E. 7-Zip:** `brew install sevenzip` (your call), then `7zz t ~/Downloads/Gate.zip` → "Everything is Ok" (note any Headers Error or Warnings line); `7zz x ~/Downloads/Gate.zip -o/tmp/g-7z && (cd /tmp/g-7z/Gate && shasum -a 256 -c ~/zipgate/expected.sha256)`. **F. macOS Archive Utility:** double-click `Gate.zip` in Downloads; no "Unable to expand"/"Error 79"; `cd ~/Downloads/Gate && shasum -a 256 -c ~/zipgate/expected.sha256`; the two non-ASCII names read correctly in Finder. **G. Windows 11 Explorer:** note `winver`; double-click `Gate.zip` (four files and `Leer`, correct names); **Extract All…** to `C:\gate`; PowerShell `Get-FileHash -Algorithm SHA256 C:\gate\Gate\* | Format-List`, compare with `expected.sha256` (case-insensitive). **H. Report here** a small table: tool, version, test result, checksums match, names correct, warnings. I add the release build's `go version`. - **Pass:** all four extract `big-5GiB.bin` with the right SHA-256 and the names read correctly (a 7-Zip header warning with a correct extraction still passes and goes into the handbook's known limits). I close this issue. - **Fail:** I prepare `gate-big-nots.zip` (no 0x5455 extra) to find the cause, then you decide between the approved contingency (files of 4 GiB or more left out with reason `too-large`) and a ZIP64 local-header fix before more releases build on this. Clean up afterwards: delete `Gate` in Cairn (it goes to Deleted files) and `~/zipgate`, `~/Downloads/Gate*`, `/tmp/g-*` on the Mac.
Author
Owner

ZIP64 gate results (2026-11-05, run by Claude on Nikola's Mac)

Setup. Real stack: files-bao (v0.6.399, s3, encryption on). files.c0rdyceps.ch could not hold the test: its per-person limit is 2 GiB and its spaces 1 GiB, and raising them is a settings change I don't make on my own. The ZIP writer is the same code on both.

  • Gate/ uploaded through Cairn itself (the app's own TUS upload from the browser): big-5GiB.bin (5,368,709,120 bytes, deterministic content), Grüße äöü.txt, 日本語テスト.txt, small.txt, empty Leer/. Expected SHA-256 computed independently on the Mac from the same generator (the generator was cross-checked byte for byte, including a block boundary).
  • The check answered 4 files, 2 folders, 5,368,709,137 bytes, ask: true; the dialog read "4 files in 2 folders, 5.0 GB in total. Saved as Gate.zip". Downloaded in Edge: Gate.zip, 5,368,710,129 bytes. (Two copies downloaded; they are byte-identical.)
  • Release build toolchain: go1.26.10 (go.mod go 1.26.10, setup-go.sh with GOTOOLCHAIN=local in release.yml).
Tool Version Test Checksums Names Notes
unzip (Info-ZIP) 6.00 (Apple build, ZIP64_SUPPORT) unzip -t: No errors detected big-5GiB.bin and small.txt OK no: the two non-ASCII names fail with "Illegal byte sequence" Apple's unzip is built without UNICODE_SUPPORT and ignores the UTF-8 flag. The archive sets it correctly (flag 0x800 on exactly those two entries, checked with Python's zipfile). Tool limitation, not Cairn's.
7-Zip 26.05 (arm64, Homebrew) 7zz t: Everything is Ok, 1 warning: Headers Error all 4 OK yes The header warning anticipated in spec §17 (golang/go#69415, Go-written ZIP64 data descriptors). Extraction is correct, so it passes; goes into the handbook's known limits.
macOS Archive Utility macOS 26.6.2 expanded, no error all 4 OK yes Leer present. The top Gate folder is dated 30 Nov 1979 (see below).
Windows 11 File Explorer not run Needs a Windows machine. Gate.zip is still in Nikola's Downloads folder for it.

Verdict so far: pass for unzip (ZIP64 data), 7-Zip and Archive Utility. The member over 4 GiB and the members after the 4 GiB mark extract byte-identically. Open: Windows Explorer (steps G on this issue; use the same Gate.zip and compare with expected.sha256, below).

Expected SHA-256:

a4f715ea66600992c085059aa2019ad84baecc63fdb0bd552bec671f820c8002  big-5GiB.bin
eefeabce9a2687ecae740bf791ad4e768b642ec837cc05e9677b25de098e2547  Grüße äöü.txt
9f2c1a03124d90994aa38206721ba5ab10fd03e269fa24860a7f39d362dd9b5f  日本語テスト.txt
4c47b3e816fbe7d40cef9f665ba8f0be1ae68b5e8e7ed70f5b6bab7f70528e8f  small.txt

Found on the way: the top folder entry carries no date (#1133); the handbook gets the 7-Zip header warning and Apple-unzip name note as known limits with the Windows result.

## ZIP64 gate results (2026-11-05, run by Claude on Nikola's Mac) **Setup.** Real stack: files-bao (v0.6.399, s3, encryption on). files.c0rdyceps.ch could not hold the test: its per-person limit is 2 GiB and its spaces 1 GiB, and raising them is a settings change I don't make on my own. The ZIP writer is the same code on both. - `Gate/` uploaded through Cairn itself (the app's own TUS upload from the browser): `big-5GiB.bin` (5,368,709,120 bytes, deterministic content), `Grüße äöü.txt`, `日本語テスト.txt`, `small.txt`, empty `Leer/`. Expected SHA-256 computed independently on the Mac from the same generator (the generator was cross-checked byte for byte, including a block boundary). - The check answered `4 files, 2 folders, 5,368,709,137 bytes, ask: true`; the dialog read "4 files in 2 folders, 5.0 GB in total. Saved as Gate.zip". Downloaded in Edge: `Gate.zip`, 5,368,710,129 bytes. (Two copies downloaded; they are byte-identical.) - Release build toolchain: **go1.26.10** (go.mod `go 1.26.10`, `setup-go.sh` with `GOTOOLCHAIN=local` in release.yml). | Tool | Version | Test | Checksums | Names | Notes | |---|---|---|---|---|---| | unzip (Info-ZIP) | 6.00 (Apple build, `ZIP64_SUPPORT`) | `unzip -t`: No errors detected | big-5GiB.bin and small.txt OK | **no**: the two non-ASCII names fail with "Illegal byte sequence" | Apple's unzip is built without `UNICODE_SUPPORT` and ignores the UTF-8 flag. The archive sets it correctly (flag 0x800 on exactly those two entries, checked with Python's zipfile). Tool limitation, not Cairn's. | | 7-Zip | 26.05 (arm64, Homebrew) | `7zz t`: Everything is Ok, **1 warning: Headers Error** | all 4 OK | yes | The header warning anticipated in spec §17 (golang/go#69415, Go-written ZIP64 data descriptors). Extraction is correct, so it passes; goes into the handbook's known limits. | | macOS Archive Utility | macOS 26.6.2 | expanded, no error | all 4 OK | yes | `Leer` present. The top `Gate` folder is dated 30 Nov 1979 (see below). | | Windows 11 File Explorer | | **not run** | | | Needs a Windows machine. `Gate.zip` is still in Nikola's Downloads folder for it. | **Verdict so far:** pass for unzip (ZIP64 data), 7-Zip and Archive Utility. The member over 4 GiB and the members after the 4 GiB mark extract byte-identically. Open: Windows Explorer (steps G on this issue; use the same `Gate.zip` and compare with `expected.sha256`, below). Expected SHA-256: ``` a4f715ea66600992c085059aa2019ad84baecc63fdb0bd552bec671f820c8002 big-5GiB.bin eefeabce9a2687ecae740bf791ad4e768b642ec837cc05e9677b25de098e2547 Grüße äöü.txt 9f2c1a03124d90994aa38206721ba5ab10fd03e269fa24860a7f39d362dd9b5f 日本語テスト.txt 4c47b3e816fbe7d40cef9f665ba8f0be1ae68b5e8e7ed70f5b6bab7f70528e8f small.txt ``` Found on the way: the top folder entry carries no date (#1133); the handbook gets the 7-Zip header warning and Apple-unzip name note as known limits with the Windows result.
Author
Owner

The Windows Explorer part of the gate is now also listed on #2 (Windows Explorer WebDAV interop), so it runs together with the other Windows tests once a Windows machine is set up.

The Windows Explorer part of the gate is now also listed on #2 (Windows Explorer WebDAV interop), so it runs together with the other Windows tests once a Windows machine is set up.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1097
No description provided.