Cordy/cairn-desktop
1
0
Fork 0
Code Issues 25 Pull requests Projects Releases Packages Wiki Activity Actions

P2-4: Resumable upload via tus #24

New issue
Closed
opened 2026-09-10 17:41:28 +00:00 by Cordy · 1 comment
Cordy commented 2026-09-10 17:41:28 +00:00
Owner
Copy link

Depends on P2-3.

Goal

Upload large files so an interrupted transfer resumes from where it stopped instead of starting over. Cairn already implements the tus protocol server-side.

Files

  • Create: internal/remote/tus.go, internal/remote/tus_test.go

Protocol flow

  1. POST to the tus endpoint with Upload-Length and Tus-Resumable: 1.0.0 → 201 with a Location header identifying the upload.
  2. PATCH to that location with Content-Type: application/offset+octet-stream, Upload-Offset: <n>, and the body chunk → 204 with the new Upload-Offset.
  3. On resume, HEAD the location → current Upload-Offset; continue from there.

Produces

// PutResumable uploads via tus, resuming from a persisted upload URL if one exists.
func (c *Client) PutResumable(path string, r io.ReaderAt, size int64, mod time.Time) (Entry, error)

Note io.ReaderAt, not io.Reader — resuming requires seeking to an arbitrary offset.

Design decisions

  • Threshold. Use simple PUT below ~8 MiB and tus above it. A tus handshake for a 2 KB file is pure overhead. Make the threshold a constant with a comment, not a magic number.
  • Persist the upload URL against the path so a resume survives an application restart, not merely a network blip. Store it in the state DB or a sidecar table keyed by path + size + mtime; invalidate if any of those changed, because the file was edited and the partial upload is now garbage.
  • Chunk size ~4 MiB. Larger wastes re-transmission on failure; smaller wastes round trips.
  • A 409 Conflict on PATCH means the offset is wrong. Re-HEAD and resync rather than retrying blindly.

Steps

  • Write failing tests against a fake tus server:
    • full upload of a multi-chunk file succeeds and content matches
    • a server that drops the connection mid-upload, then a second call resumes from the persisted offset and transfers only the remainder
    • a 409 triggers a HEAD-and-resync rather than an infinite retry
    • files under the threshold take the simple PUT path
  • Run; confirm failure.
  • Implement.
  • Run tests; confirm they pass.
  • Commit: git commit -s -m "feat(remote): resumable uploads via tus"

Acceptance criteria

  • A resumed upload transfers strictly fewer bytes than a fresh one (assert it in the test).
  • An edited file invalidates its stored upload URL rather than resuming into the wrong content.
  • Small files never touch tus.
Depends on P2-3. ## Goal Upload large files so an interrupted transfer resumes from where it stopped instead of starting over. Cairn already implements the tus protocol server-side. ## Files - Create: `internal/remote/tus.go`, `internal/remote/tus_test.go` ## Protocol flow 1. `POST` to the tus endpoint with `Upload-Length` and `Tus-Resumable: 1.0.0` → `201` with a `Location` header identifying the upload. 2. `PATCH` to that location with `Content-Type: application/offset+octet-stream`, `Upload-Offset: <n>`, and the body chunk → `204` with the new `Upload-Offset`. 3. On resume, `HEAD` the location → current `Upload-Offset`; continue from there. ## Produces ```go // PutResumable uploads via tus, resuming from a persisted upload URL if one exists. func (c *Client) PutResumable(path string, r io.ReaderAt, size int64, mod time.Time) (Entry, error) ``` Note `io.ReaderAt`, not `io.Reader` — resuming requires seeking to an arbitrary offset. ## Design decisions - **Threshold.** Use simple `PUT` below ~8 MiB and tus above it. A tus handshake for a 2 KB file is pure overhead. Make the threshold a constant with a comment, not a magic number. - **Persist the upload URL** against the path so a resume survives an application restart, not merely a network blip. Store it in the state DB or a sidecar table keyed by path + size + mtime; invalidate if any of those changed, because the file was edited and the partial upload is now garbage. - **Chunk size** ~4 MiB. Larger wastes re-transmission on failure; smaller wastes round trips. - **A `409 Conflict` on PATCH means the offset is wrong.** Re-`HEAD` and resync rather than retrying blindly. ## Steps - [ ] Write failing tests against a fake tus server: - full upload of a multi-chunk file succeeds and content matches - a server that drops the connection mid-upload, then a second call resumes from the persisted offset and transfers only the remainder - a 409 triggers a HEAD-and-resync rather than an infinite retry - files under the threshold take the simple `PUT` path - [ ] Run; confirm failure. - [ ] Implement. - [ ] Run tests; confirm they pass. - [ ] Commit: `git commit -s -m "feat(remote): resumable uploads via tus"` ## Acceptance criteria - A resumed upload transfers strictly fewer bytes than a fresh one (assert it in the test). - An edited file invalidates its stored upload URL rather than resuming into the wrong content. - Small files never touch tus.
Cordy commented 2026-09-11 06:40:24 +00:00
Author
Owner
Copy link

Done

What was built

  • internal/remote/tus.go: a tus client — PutResumable, chunked PATCH upload, HEAD-and-resync on 409, an UploadStore interface for persisting the upload URL.
  • Client.Put routes to tus at/above an 8 MiB threshold when the reader is a seekable ReaderAt of known size and the Client has a tus endpoint; otherwise plain PUT.
  • internal/state: a new uploads table (path + size + mtime + version) backing UploadStore; invalidated on any change.
  • internal/sync/engine.go: put hashes the file, uploads, then re-Stats and records state only if size/mtime/FileID are unchanged, else Skips ("changed while uploading") per P2-R5.
  • Fix round: F3 stopped tus being routed to a Client with no tus endpoint (regression on non-/dav BaseURLs); F1 made a file changed mid-upload record an unknown-hash row instead of no row, so the next pass re-uploads instead of downloading a torn copy over the user's file; F2 added a content-version token so a same-size/same-mtime replacement during a resume starts a fresh upload instead of resuming into the old bytes.

Tests

  • 14 tus tests against a faithful fake server (tusServer) plus engine and state tests; TDD cycles A (tus client) and B (engine wiring) both RED→GREEN with mutation testing.
  • Live-verified against a throwaway cairnd (cairnd-dev.sh): multi-chunk upload, connection-drop resume (11,534,213 of 20,971,520 bytes sent — strictly fewer), 409 resync, edited-file invalidation, small-file PUT bypass.
  • CI: Forgejo Actions run #22 on commit 19257f4, linux/arm64, go1.25.5 — green. go vet/gofmt/go mod tidy -diff clean on host, windows/amd64 and linux/arm64. Coverage: remote 91.0%, state 87.0%, sync 93.4%, vfs 88.1%, total 91.5%.

Acceptance criteria

  • "A resumed upload transfers strictly fewer bytes than a fresh one" — asserted directly in TestPutResumableResumesAfterTheConnectionDrops and through the engine in TestEngineUploadsLargeFilesThroughTus; confirmed live (11,534,213 < 20,971,520).
  • "An edited file invalidates its stored upload URL rather than resuming into the wrong content" — TestPutResumableStartsAfreshForAnEditedFile (mtime and size changes) and, after the F2 fix, a same-size/same-mtime content replacement too (TestPutResumableStartsAfreshForReplacedContent).
  • "Small files never touch tus" — threshold test asserts no request under /api/v1/tus/; live, a 5-byte file went by PUT.
  • P2-R5 (engine wiring, hash-then-send-then-conditional-record, UploadStore in internal/remote/state, invalidate on size/mtime change) — met; F1 and F2 closed the two corruption paths the ruling's design otherwise left open.
  • P2-R13 (tus endpoint and Upload-Metadata path under Cairn's /dav/home/ mount) — met and tested against the fake and live.
  • P2-R15 (relative Location, HEAD-404 restart, kept spool on failed finalize retried by a zero-length PATCH at offset==length, PROPFIND after every success since PUT/tus return no ETag) — met and tested.

Rulings

  • P2-R1: push to main after clean review; close only once every Actions run for the pushed commit is green; commit subject = issue message, -s, Co-Authored-By.
  • P2-R5: Put routes to tus for a seekable, sized ReaderAt at/above the threshold; the engine hashes, sends, re-Stats, records only if unchanged, else a hashless Skip; UploadStore lives in internal/remote, implemented by internal/state.
  • P2-R13: tus endpoint scheme://host/api/v1/tus/; Upload-Metadata path is BaseURL's path with Cairn's /dav prefix stripped, plus the engine path.
  • P2-R15: server quirks tolerated — MKCOL-then-retry, Depth 0/1 only, PROPFIND after every PUT/tus for the ETag, relative Location, HEAD-404 restart, zero-length PATCH retries a failed finalize.
  • P2-R18: live-server work only via cairnd-dev.sh (throwaway local cairnd; never ~/Cairn, .249, or the owner's credentials).
  • progress.md Ruling F1: a changed-while-uploading row records the server's own write with ContentHash "" instead of no row, amending P2-R5 so the next pass re-uploads with no conflict.
  • progress.md Ruling F2: remote.Upload gains a Version token from an optional reader interface; a mismatch on resume terminates and restarts, closing the same-size/same-mtime replacement gap.
  • progress.md Ruling F6: the re-Stat guard's residual same-mtime-tick window is accepted here, closed in practice by #27's stability wait.

Deferred

  • F4 (tus.go:366): a failed seek-back in resumable falls through to a plain PUT from the end position instead of erroring.
  • F5 (tus.go:309): uploadURL checks only scheme/host, not that the path lies under the tus mount, before use by terminate/HEAD/PATCH.
  • F7 (state.go:48): abandoned uploads leave their uploads row and cairnd spool forever when the local file is later deleted, renamed, or conflict-copied.
  • #26 (ETag-first detection): a future local-stat fast path must not treat a row with ContentHash "" as unchanged.
  • Fix-round residuals: a rename-over between put's two Opens can still slip a resumed upload; an error from re-Stat itself (not just a mismatch) still records nothing; a file deleted mid-upload now gets a row that deletes the server's own-write copy instead of downloading it back.

Implemented and reviewed by Claude (subagent-driven), landed on main after review and green CI.

**Done** - [50eb9122f25f60c678d4a424c0a98c0e43401114](http://192.168.10.245/Cordy/cairn-desktop/commit/50eb9122f25f60c678d4a424c0a98c0e43401114) feat(remote): resumable uploads via tus - [5f6d205c6d05e83b5cb132615a9a35c9d9e7f574](http://192.168.10.245/Cordy/cairn-desktop/commit/5f6d205c6d05e83b5cb132615a9a35c9d9e7f574) fix(remote): keep the plain PUT for a Client without a tus endpoint - [ef1e4bd3d06eae7fa0c5fb10d76b99554e956aae](http://192.168.10.245/Cordy/cairn-desktop/commit/ef1e4bd3d06eae7fa0c5fb10d76b99554e956aae) fix(sync): re-upload a file that changed while uploading - [19257f4c0fe9b1172a675619f4e35d806ada7b39](http://192.168.10.245/Cordy/cairn-desktop/commit/19257f4c0fe9b1172a675619f4e35d806ada7b39) fix(remote): resume a tus upload only for the same content version **What was built** - `internal/remote/tus.go`: a tus client — PutResumable, chunked PATCH upload, HEAD-and-resync on 409, an UploadStore interface for persisting the upload URL. - `Client.Put` routes to tus at/above an 8 MiB threshold when the reader is a seekable ReaderAt of known size and the Client has a tus endpoint; otherwise plain PUT. - `internal/state`: a new `uploads` table (path + size + mtime + version) backing UploadStore; invalidated on any change. - `internal/sync/engine.go`: put hashes the file, uploads, then re-Stats and records state only if size/mtime/FileID are unchanged, else Skips ("changed while uploading") per P2-R5. - Fix round: F3 stopped tus being routed to a Client with no tus endpoint (regression on non-`/dav` BaseURLs); F1 made a file changed mid-upload record an unknown-hash row instead of no row, so the next pass re-uploads instead of downloading a torn copy over the user's file; F2 added a content-version token so a same-size/same-mtime replacement during a resume starts a fresh upload instead of resuming into the old bytes. **Tests** - 14 tus tests against a faithful fake server (tusServer) plus engine and state tests; TDD cycles A (tus client) and B (engine wiring) both RED→GREEN with mutation testing. - Live-verified against a throwaway cairnd (cairnd-dev.sh): multi-chunk upload, connection-drop resume (11,534,213 of 20,971,520 bytes sent — strictly fewer), 409 resync, edited-file invalidation, small-file PUT bypass. - CI: Forgejo Actions run [#22](http://192.168.10.245/Cordy/cairn-desktop/actions/runs/22) on commit 19257f4, linux/arm64, go1.25.5 — green. `go vet`/`gofmt`/`go mod tidy -diff` clean on host, windows/amd64 and linux/arm64. Coverage: remote 91.0%, state 87.0%, sync 93.4%, vfs 88.1%, total 91.5%. **Acceptance criteria** - "A resumed upload transfers strictly fewer bytes than a fresh one" — asserted directly in `TestPutResumableResumesAfterTheConnectionDrops` and through the engine in `TestEngineUploadsLargeFilesThroughTus`; confirmed live (11,534,213 < 20,971,520). - "An edited file invalidates its stored upload URL rather than resuming into the wrong content" — `TestPutResumableStartsAfreshForAnEditedFile` (mtime and size changes) and, after the F2 fix, a same-size/same-mtime content replacement too (`TestPutResumableStartsAfreshForReplacedContent`). - "Small files never touch tus" — threshold test asserts no request under `/api/v1/tus/`; live, a 5-byte file went by PUT. - P2-R5 (engine wiring, hash-then-send-then-conditional-record, UploadStore in internal/remote/state, invalidate on size/mtime change) — met; F1 and F2 closed the two corruption paths the ruling's design otherwise left open. - P2-R13 (tus endpoint and Upload-Metadata path under Cairn's `/dav/home/` mount) — met and tested against the fake and live. - P2-R15 (relative Location, HEAD-404 restart, kept spool on failed finalize retried by a zero-length PATCH at offset==length, PROPFIND after every success since PUT/tus return no ETag) — met and tested. **Rulings** - P2-R1: push to main after clean review; close only once every Actions run for the pushed commit is green; commit subject = issue message, `-s`, Co-Authored-By. - P2-R5: Put routes to tus for a seekable, sized ReaderAt at/above the threshold; the engine hashes, sends, re-Stats, records only if unchanged, else a hashless Skip; UploadStore lives in internal/remote, implemented by internal/state. - P2-R13: tus endpoint `scheme://host/api/v1/tus/`; Upload-Metadata `path` is BaseURL's path with Cairn's `/dav` prefix stripped, plus the engine path. - P2-R15: server quirks tolerated — MKCOL-then-retry, Depth 0/1 only, PROPFIND after every PUT/tus for the ETag, relative Location, HEAD-404 restart, zero-length PATCH retries a failed finalize. - P2-R18: live-server work only via cairnd-dev.sh (throwaway local cairnd; never ~/Cairn, .249, or the owner's credentials). - progress.md Ruling F1: a changed-while-uploading row records the server's own write with ContentHash "" instead of no row, amending P2-R5 so the next pass re-uploads with no conflict. - progress.md Ruling F2: `remote.Upload` gains a `Version` token from an optional reader interface; a mismatch on resume terminates and restarts, closing the same-size/same-mtime replacement gap. - progress.md Ruling F6: the re-Stat guard's residual same-mtime-tick window is accepted here, closed in practice by #27's stability wait. **Deferred** - F4 (tus.go:366): a failed seek-back in `resumable` falls through to a plain PUT from the end position instead of erroring. - F5 (tus.go:309): `uploadURL` checks only scheme/host, not that the path lies under the tus mount, before use by terminate/HEAD/PATCH. - F7 (state.go:48): abandoned uploads leave their `uploads` row and cairnd spool forever when the local file is later deleted, renamed, or conflict-copied. - #26 (ETag-first detection): a future local-stat fast path must not treat a row with `ContentHash ""` as unchanged. - Fix-round residuals: a rename-over between put's two Opens can still slip a resumed upload; an error from re-Stat itself (not just a mismatch) still records nothing; a file deleted mid-upload now gets a row that deletes the server's own-write copy instead of downloading it back. _Implemented and reviewed by Claude (subagent-driven), landed on main after review and green CI._
Cordy closed this issue 2026-09-11 06:40:29 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v0.2.0-integration
v0.1.0-engine
Labels
Clear labels   
data-integrity

Touches correctness where a bug means data loss

  
engine

Sync engine core (internal/sync)

  
platform

OS-specific code, packaging, signing

  
procurement

Manual task — accounts, legal, hardware. Not code.

  
remote

WebDAV / tus / HTTP client

  
scaffold

Repo, CI, tooling

  
ui

Wails shell and frontend

No labels
data-integrity
engine
platform
procurement
remote
scaffold
ui
Milestone
Clear milestone
No items
No milestone
phase-2-integration
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/cairn-desktop#24
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?