WebDAV for sync clients: folder ETags, file id and permissions, keep client file dates, checksums #1099

Closed
opened 2026-11-04 03:10:52 +00:00 by Cordy · 5 comments
Owner

Gap (v0.7 feature-gap research, 2026-11-04, decided by Nikola). At 0e0885db (VERIFIED):

  • PROPFIND sends getetag for files only, never for folders (internal/dav/dav.go:313-319). Sync clients (ownCloud, OpenCloud, Nextcloud desktop; rclone bisync) find changes by comparing folder ETags.
  • No file-id or permissions properties (oc:fileid, oc:permissions or equivalents).
  • PROPPATCH stores nothing and answers 207 (internal/dav/dav.go:139-145), so a client-set modification time is dropped; rclone and import uploads get "now". No X-OC-Mtime handling either.
  • No checksums on upload or in properties.

Why now. Third-party clients and migrations (v0.9 migration guide) depend on it, and the v1.1 desktop app will need the same base. Cairn already has a recursive folder ETag for OCM since #1070, which may be reusable.

To research and decide when worked on. Which properties and headers to support (RFC 4918, ownCloud/Nextcloud conventions as used by rclone and the desktop clients); the cost of folder ETags on large trees and on s3, and cache invalidation across instances (#742); mtime on posix vs s3 (metadata); checksum types; interop tests in CI (rclone, cadaver, Finder, Windows).

Refs: scratch/b7e/backlog/gap-v07/research-v07-gaps.md A3, research-opencloud.md (surprising findings), research-nextcloud-files.md.

**Gap (v0.7 feature-gap research, 2026-11-04, decided by Nikola).** At 0e0885db (VERIFIED): - PROPFIND sends `getetag` for files only, never for folders (`internal/dav/dav.go:313-319`). Sync clients (ownCloud, OpenCloud, Nextcloud desktop; rclone bisync) find changes by comparing folder ETags. - No file-id or permissions properties (`oc:fileid`, `oc:permissions` or equivalents). - PROPPATCH stores nothing and answers 207 (`internal/dav/dav.go:139-145`), so a client-set modification time is dropped; rclone and import uploads get "now". No `X-OC-Mtime` handling either. - No checksums on upload or in properties. **Why now.** Third-party clients and migrations (v0.9 migration guide) depend on it, and the v1.1 desktop app will need the same base. Cairn already has a recursive folder ETag for OCM since #1070, which may be reusable. **To research and decide when worked on.** Which properties and headers to support (RFC 4918, ownCloud/Nextcloud conventions as used by rclone and the desktop clients); the cost of folder ETags on large trees and on s3, and cache invalidation across instances (#742); mtime on posix vs s3 (metadata); checksum types; interop tests in CI (rclone, cadaver, Finder, Windows). Refs: `scratch/b7e/backlog/gap-v07/research-v07-gaps.md` A3, research-opencloud.md (surprising findings), research-nextcloud-files.md.
Author
Owner

Research done (scratch/b7e/backlog/research-1099.md). Most important finding: the documented rclone setup (vendor=other in the sync guide, the Connect a device tile and interop-rclone.yml) makes rclone treat modification times as unsupported and have no hashes, so an edit that keeps the file size is skipped by rclone sync ("Sizes identical / Unchanged skipping", user report on the rclone forum). The Nextcloud and ownCloud desktop apps cannot use Cairn even with every property (they need status.php and more); that is a separate project.

Recommended first version:

  1. Keep client file dates: X-OC-Mtime on PUT, MOVE, COPY and an mtime for TUS and web uploads, stored in the per-folder records listings already show (not on the stored file: posix ETags and versions rely on it, s3 would need a full copy). Dates from 1970-01-02 to now + 24 h; a bad date never fails the upload; X-OC-MTime: accepted only once stored.
  2. PROPPATCH answered per RFC 4918: sets the date for DAV:lastmodified, SAR:lastmodified and Win32LastModifiedTime; other Win32 properties 200 but not kept (Windows copy needs success); protected and unknown properties 403, the rest 424; same checks as a write.
  3. Only ETags decide conditional requests (a client-chosen date must not cause a wrong 304).
  4. oc:id, oc:fileid, oc:permissions when asked (public item id; existing permission logic).
  5. Folder getetag when asked, reusing the #1070 folder tag moved to a shared package (no stored state, holds across instances).
  6. Docs switch to vendor=owncloud and say plainly that the Nextcloud and ownCloud desktop apps do not work.

Checksums after launch (rclone uses SHA-1/MD5; download marking changes bytes of PDF, Office and image files, so a stored checksum would make rclone report corruption).

Decisions asked of Nikola next.

**Research done** (`scratch/b7e/backlog/research-1099.md`). Most important finding: the documented rclone setup (`vendor=other` in the sync guide, the Connect a device tile and `interop-rclone.yml`) makes rclone treat modification times as unsupported and have no hashes, so an edit that keeps the file size is skipped by `rclone sync` ("Sizes identical / Unchanged skipping", user report on the rclone forum). The Nextcloud and ownCloud desktop apps cannot use Cairn even with every property (they need `status.php` and more); that is a separate project. Recommended first version: 1. Keep client file dates: `X-OC-Mtime` on PUT, MOVE, COPY and an `mtime` for TUS and web uploads, stored in the per-folder records listings already show (not on the stored file: posix ETags and versions rely on it, s3 would need a full copy). Dates from 1970-01-02 to now + 24 h; a bad date never fails the upload; `X-OC-MTime: accepted` only once stored. 2. PROPPATCH answered per RFC 4918: sets the date for `DAV:lastmodified`, `SAR:lastmodified` and `Win32LastModifiedTime`; other Win32 properties 200 but not kept (Windows copy needs success); protected and unknown properties 403, the rest 424; same checks as a write. 3. Only ETags decide conditional requests (a client-chosen date must not cause a wrong 304). 4. `oc:id`, `oc:fileid`, `oc:permissions` when asked (public item id; existing permission logic). 5. Folder `getetag` when asked, reusing the #1070 folder tag moved to a shared package (no stored state, holds across instances). 6. Docs switch to `vendor=owncloud` and say plainly that the Nextcloud and ownCloud desktop apps do not work. Checksums after launch (rclone uses SHA-1/MD5; download marking changes bytes of PDF, Office and image files, so a stored checksum would make rclone report corruption). Decisions asked of Nikola next.
Author
Owner

Decisions (Nikola, 2026-11-04): all recommendations approved. Client dates kept in the folder records; strict RFC 4918 PROPPATCH (the three date properties set, other Win32 properties 200 but not kept, unknown/protected 403 with an INFO log); file id, permissions and folder ETags only when asked; docs and CI switch rclone to vendor=owncloud and state that the Nextcloud and ownCloud desktop apps do not work; audit verb for a date change without upload; checksums after launch. No UI frames needed for v1.

**Decisions (Nikola, 2026-11-04): all recommendations approved.** Client dates kept in the folder records; strict RFC 4918 PROPPATCH (the three date properties set, other Win32 properties 200 but not kept, unknown/protected 403 with an INFO log); file id, permissions and folder ETags only when asked; docs and CI switch rclone to `vendor=owncloud` and state that the Nextcloud and ownCloud desktop apps do not work; audit verb for a date change without upload; checksums after launch. No UI frames needed for v1.
Author
Owner

Spec open points decided (Nikola, 2026-11-04), after research (research-1099-unsettled.md): all verdicts approved. Notably: PDF previews get an ETag; downloads send Cache-Control: private, no-cache; add-only backup remotes (--immutable) keep their vendor; the bisync migration recipe is usual run, config update, --resync --resync-mode path1 (dry run first), never --refresh-times; copies keep the original's date (CHANGELOG "read before upgrading"); interop job run once before the tag, TZ=UTC in its script.

**Spec open points decided (Nikola, 2026-11-04), after research (`research-1099-unsettled.md`):** all verdicts approved. Notably: PDF previews get an ETag; downloads send `Cache-Control: private, no-cache`; add-only backup remotes (`--immutable`) keep their vendor; the bisync migration recipe is usual run, `config update`, `--resync --resync-mode path1` (dry run first), never `--refresh-times`; copies keep the original's date (CHANGELOG "read before upgrading"); interop job run once before the tag, `TZ=UTC` in its script.
Author
Owner

Release gate passed: interop-rclone dispatched on branch wd1099 (7329ace0, main v0.6.395 merged in): run #3980, success (vendor=owncloud, TZ=UTC; dates kept, same-size edit synced, rclone touch, server-side move keeps the date, rclone check clean).

PR #1116's first CI run (3979) failed one timing test unrelated to this feature (TestRemoveHalfCopyIdleBound1026, 500 ms batches against a 1 s idle bound under a loaded runner; passes 4/4 locally). Its bound is widened in 501726d6 and CI runs again.

**Release gate passed:** `interop-rclone` dispatched on branch `wd1099` (7329ace0, main v0.6.395 merged in): run #3980, **success** (vendor=owncloud, TZ=UTC; dates kept, same-size edit synced, `rclone touch`, server-side move keeps the date, `rclone check` clean). PR #1116's first CI run (3979) failed one timing test unrelated to this feature (`TestRemoveHalfCopyIdleBound1026`, 500 ms batches against a 1 s idle bound under a loaded runner; passes 4/4 locally). Its bound is widened in 501726d6 and CI runs again.
Cordy closed this issue 2026-11-04 17:00:48 +00:00
Author
Owner

Shipped in v0.6.396 (PR #1116, merge 351b4a69). Both dogfoods run it (0 restarts, no ERROR). Release gate interop-rclone passed (run #3980). Hands-on checks (Windows Explorer folder copy, Finder, Cyberduck, existing bisync migration) are on #1005 ("v0.6.396").

Follow-ups (after launch): checksums (and with them a safe --refresh-times), folder dates, a Nextcloud/ownCloud desktop compatibility layer, an ETag on the OCM grant GET and REST raw downloads, per-caller folder-tag walk slots; all in the spec §12 list and the backlog file. Closing.

Shipped in **v0.6.396** (PR #1116, merge 351b4a69). Both dogfoods run it (0 restarts, no ERROR). Release gate `interop-rclone` passed (run #3980). Hands-on checks (Windows Explorer folder copy, Finder, Cyberduck, existing bisync migration) are on #1005 ("v0.6.396"). Follow-ups (after launch): checksums (and with them a safe `--refresh-times`), folder dates, a Nextcloud/ownCloud desktop compatibility layer, an ETag on the OCM grant GET and REST raw downloads, per-caller folder-tag walk slots; all in the spec §12 list and the backlog file. Closing.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1099
No description provided.