TUS: unfinished uploads never expire — abandoned partials fill the spool forever #826

Closed
opened 2026-10-01 05:29:32 +00:00 by Cordy · 1 comment
Owner

Found during the #821 research (2026-10-01).

Today: internal/tus has no expiry and no sweep. New() rebuilds every in-progress upload from the spool at startup and keeps it indefinitely. On any persistent /data (both dogfoods, the Helm chart default, docker-compose) a tab closed at 40 % leaves its partial file on the volume forever. Only an ephemeral /data "cleans" it, by losing everything.

Others hit exactly this:

  • Nextcloud: per-user uploads/ growing to 80 GB+ (#19034, #8753); a background job removes chunks after cache_chunk_gc_ttl (1 day), which for a while ignored the setting (#16019); admins raise it for large uploads on slow links.
  • oCIS / OpenCloud: STORAGE_USERS_UPLOAD_EXPIRATION = 24 h, but expired blob + blob.info are not removed automatically; admins run ocis storage-users uploads sessions --clean (#9797, #1227).
  • tusd: no cleanup at all; "run a cronjob" (tusd #176, #435).

Proposal:

  1. Implement the TUS Expiration extension: advertise it in Tus-Extension, set Upload-Expires on create and on each PATCH (deadline = last activity + TTL, so a slow but active upload never expires mid-transfer), and answer an expired upload with 404/410 so clients restart cleanly.
  2. Sweep the spool for entries past their deadline: once at startup (during the existing rebuild) and hourly, deleting data + .json sidecar. Orphans (data without sidecar or vice versa) older than the TTL go too.
  3. Config tus.expiry (Go duration), default 24h (Nextcloud and oCIS defaults); 0 keeps today's never-expire behaviour for anyone who depends on it.
  4. While here: verify that peering staging (peer-blobs/) removes a staged blob whose transfer expires before delivery (transferTTL); fix if not.
  5. Docs: deployment.md (what lives locally / the setting), CHANGELOG.

Server-side only; the one user-visible effect is that resuming after the deadline starts over, which the web client already handles as a fresh upload.

Relates to #821 (the spool stays local by design, so cleanup has to come from expiry, not from location).

Found during the #821 research (2026-10-01). **Today:** `internal/tus` has no expiry and no sweep. `New()` rebuilds every in-progress upload from the spool at startup and keeps it indefinitely. On any persistent `/data` (both dogfoods, the Helm chart default, docker-compose) a tab closed at 40 % leaves its partial file on the volume forever. Only an ephemeral `/data` "cleans" it, by losing everything. **Others hit exactly this:** - Nextcloud: per-user `uploads/` growing to 80 GB+ (#19034, #8753); a background job removes chunks after `cache_chunk_gc_ttl` (1 day), which for a while ignored the setting (#16019); admins raise it for large uploads on slow links. - oCIS / OpenCloud: `STORAGE_USERS_UPLOAD_EXPIRATION` = 24 h, but expired `blob` + `blob.info` are **not** removed automatically; admins run `ocis storage-users uploads sessions --clean` (#9797, #1227). - tusd: no cleanup at all; "run a cronjob" (tusd #176, #435). **Proposal:** 1. Implement the TUS **Expiration** extension: advertise it in `Tus-Extension`, set `Upload-Expires` on create and on each PATCH (deadline = last activity + TTL, so a slow but active upload never expires mid-transfer), and answer an expired upload with 404/410 so clients restart cleanly. 2. **Sweep** the spool for entries past their deadline: once at startup (during the existing rebuild) and hourly, deleting data + `.json` sidecar. Orphans (data without sidecar or vice versa) older than the TTL go too. 3. Config `tus.expiry` (Go duration), default **24h** (Nextcloud and oCIS defaults); `0` keeps today's never-expire behaviour for anyone who depends on it. 4. While here: verify that peering staging (`peer-blobs/`) removes a staged blob whose transfer expires before delivery (`transferTTL`); fix if not. 5. Docs: deployment.md (what lives locally / the setting), CHANGELOG. Server-side only; the one user-visible effect is that resuming after the deadline starts over, which the web client already handles as a fresh upload. Relates to #821 (the spool stays local by design, so cleanup has to come from expiry, not from location).
Author
Owner

Shipped in v0.6.312 (PR #828): Expiration extension with sliding deadline, sweep at boot + hourly (expired uploads, strays, legacy sidecars), tusExpiry default 24h, and the confirmed peering-staging gap fixed (undelivered bodies swept after 24 × transferTTL). Live on both dogfoods (resumable uploads … expiry=24h0m0s).

Shipped in v0.6.312 (PR #828): Expiration extension with sliding deadline, sweep at boot + hourly (expired uploads, strays, legacy sidecars), `tusExpiry` default 24h, and the confirmed peering-staging gap fixed (undelivered bodies swept after 24 × transferTTL). Live on both dogfoods (`resumable uploads … expiry=24h0m0s`).
Cordy closed this issue 2026-10-01 08:53:14 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#826
No description provided.