Cordy/cairn-desktop

Fork 0

P4-1: Placeholder abstraction and pin model (no OS code) #44

New issue

Open

opened 2026-09-10 18:08:49 +00:00 by Cordy · 1 comment

Cordy commented

2026-09-10 18:08:49 +00:00

Owner

First task of phase 4. Depends on phase-3-desktop-app.

This is the phase-1 trick applied again: put everything platform-independent behind an
interface first, with a fake implementation, so the hard logic is tested on the arm64 Pi runner
before any OS-specific code exists. Three separate OS integrations follow (P4-2, P4-3, P4-4);
none of them should contain sync logic.

Files

Create: internal/vfs/placeholder.go, internal/vfs/fakeplaceholder.go, internal/vfs/placeholder_test.go
Modify: internal/state/state.go (pin state column)

Produces

// PinState is what the user asked for, independent of what is currently on disk.
type PinState int
const (
	PinAuto     PinState = iota // on-demand; may be evicted to reclaim space
	PinAlways                   // always keep a local copy; never evict
	PinNever                    // dehydrate as soon as it is closed
)

// PlaceholderFS is an FS that can represent files whose content is not local.
type PlaceholderFS interface {
	FS
	// CreatePlaceholder makes a metadata-only entry with no content.
	CreatePlaceholder(path string, size int64, mod time.Time) error
	// Hydrate fetches content for a placeholder, blocking until complete.
	Hydrate(path string, r io.Reader) error
	// Dehydrate discards local content, keeping metadata. Fails if PinAlways.
	Dehydrate(path string) error
	// SetPin records the user's intent for a path.
	SetPin(path string, p PinState) error
	Pin(path string) (PinState, error)
}

FileInfo.Hydrated was added in Task 5 (see its amendment comment). Pin state persists in the
state DB — it is user intent and must survive restarts and re-syncs.

The rules the engine already follows

Both were added upstream so phase 4 adds no engine logic:

A placeholder is never locally modified (issue #26 amendment). You cannot edit bytes that
are not there.
Hydration is not a modification (issue #31 amendment). Opening a file must not cause an
upload.

If implementing an OS provider tempts you to change internal/sync, stop — the design is
wrong somewhere.

The failure mode that defines this feature

Accidental mass hydration. Any code path that reads content for every file — hashing,
indexing, a naive Walk that opens files, an antivirus scanner, a backup tool, Spotlight —
downloads the entire Cairn. The user asked for files-on-demand precisely to avoid that, and
they may be on a metered connection or a 256 GB disk.

Write a test that walks a tree of 1,000 placeholders through a full sync pass and asserts
zero hydrations. This is the single most valuable test in phase 4.

Steps

Define PlaceholderFS and PinState.
Add a pin_state column to the state store; default PinAuto.
Implement FakePlaceholderFS — in-memory, counts hydrations, so tests can assert on them.
Write failing tests: create → placeholder is Hydrated: false; hydrate → true and
content readable; dehydrate → false and content gone; Dehydrate on PinAlways fails;
pin state survives a store reopen; a full sync pass over placeholders causes zero
hydrations and zero operations.
Implement, run, commit: git commit -s -m "feat(vfs): placeholder abstraction and pin model"

Acceptance criteria

A sync pass over a fully dehydrated tree hydrates nothing and uploads nothing.
Pin state persists.
PinAlways files cannot be evicted.
Everything here runs on the Pi runner, with no OS-specific code.

First task of phase 4. Depends on `phase-3-desktop-app`. **This is the phase-1 trick applied again:** put everything platform-independent behind an interface first, with a fake implementation, so the hard logic is tested on the arm64 Pi runner before any OS-specific code exists. Three separate OS integrations follow (P4-2, P4-3, P4-4); none of them should contain sync logic. ## Files - Create: `internal/vfs/placeholder.go`, `internal/vfs/fakeplaceholder.go`, `internal/vfs/placeholder_test.go` - Modify: `internal/state/state.go` (pin state column) ## Produces ```go // PinState is what the user asked for, independent of what is currently on disk. type PinState int const ( PinAuto PinState = iota // on-demand; may be evicted to reclaim space PinAlways // always keep a local copy; never evict PinNever // dehydrate as soon as it is closed ) // PlaceholderFS is an FS that can represent files whose content is not local. type PlaceholderFS interface { FS // CreatePlaceholder makes a metadata-only entry with no content. CreatePlaceholder(path string, size int64, mod time.Time) error // Hydrate fetches content for a placeholder, blocking until complete. Hydrate(path string, r io.Reader) error // Dehydrate discards local content, keeping metadata. Fails if PinAlways. Dehydrate(path string) error // SetPin records the user's intent for a path. SetPin(path string, p PinState) error Pin(path string) (PinState, error) } ``` `FileInfo.Hydrated` was added in Task 5 (see its amendment comment). Pin state persists in the state DB — it is user intent and must survive restarts and re-syncs. ## The rules the engine already follows Both were added upstream so phase 4 adds **no** engine logic: - **A placeholder is never locally modified** (issue #26 amendment). You cannot edit bytes that are not there. - **Hydration is not a modification** (issue #31 amendment). Opening a file must not cause an upload. If implementing an OS provider tempts you to change `internal/sync`, stop — the design is wrong somewhere. ## The failure mode that defines this feature **Accidental mass hydration.** Any code path that reads content for every file — hashing, indexing, a naive `Walk` that opens files, an antivirus scanner, a backup tool, Spotlight — downloads the entire Cairn. The user asked for files-on-demand precisely to avoid that, and they may be on a metered connection or a 256 GB disk. - [ ] Write a test that walks a tree of 1,000 placeholders through a full sync pass and asserts **zero** hydrations. This is the single most valuable test in phase 4. ## Steps - [ ] Define `PlaceholderFS` and `PinState`. - [ ] Add a `pin_state` column to the state store; default `PinAuto`. - [ ] Implement `FakePlaceholderFS` — in-memory, counts hydrations, so tests can assert on them. - [ ] Write failing tests: create → placeholder is `Hydrated: false`; hydrate → `true` and content readable; dehydrate → `false` and content gone; `Dehydrate` on `PinAlways` fails; pin state survives a store reopen; a full sync pass over placeholders causes zero hydrations and zero operations. - [ ] Implement, run, commit: `git commit -s -m "feat(vfs): placeholder abstraction and pin model"` ## Acceptance criteria - A sync pass over a fully dehydrated tree hydrates nothing and uploads nothing. - Pin state persists. - `PinAlways` files cannot be evicted. - Everything here runs on the Pi runner, with no OS-specific code.

Cordy added this to the phase-4-virtual-files milestone

2026-09-10 18:08:49 +00:00

Cordy added the

engine

data-integrity

labels

2026-09-10 18:08:49 +00:00

Cordy referenced this issue

2026-09-10 18:29:51 +00:00

P3-11: Private beta — internal gate, NOT the release #40

Cordy referenced this issue

2026-09-11 01:50:39 +00:00

P2-6: ETag-first change detection and lazy hashing #26

Cordy referenced this issue

2026-09-11 09:27:27 +00:00

P2-6: ETag-first change detection and lazy hashing #26

Cordy referenced this issue

2026-09-11 20:54:48 +00:00

P2-9: Integration tests against a live cairnd in CI #29

Cordy commented

2026-09-11 20:59:05 +00:00

Author

Owner

Amendment — 2026-09-11: what #44 inherits from phase 2 (phase-2 final review X1, X2, and Task 26 F4)

Phase 2 put the placeholder rule into the engine (#26): a file with Hydrated == false is never
opened or hashed, presents as the content last synced, and is never uploaded. Four points from the
phase-2 reviews bind this issue. The first was carried here only in #26's closing comment, so it is
restated here.

1. A renamed placeholder whose FileID is unknown loses its only copy (Task 26 F4)

DetectRenames pairs a delete with an upload only on a known FileID (internal/sync/rename.go:24,
:44-45). Suppose a placeholder is renamed locally and the provider reports no FileID (""). Then:

the old path reads as deleted locally, with its server copy unchanged, so it becomes OpDeleteRemote;
the new path is a new local file, so it becomes OpUpload, which put refuses because the file is a
placeholder (notOnThisDevice, internal/sync/engine.go:726-728).

The server copy is deleted. The bytes are then nowhere, because a placeholder holds none.

Required. A pass never deletes the server copy of a synced path while a placeholder that could be
its renamed self is unpaired. The simplest form is this: when the scan finds a placeholder with no
state row (only a rename or a provider bug makes one), every unpaired OpDeleteRemote in that pass
becomes a Skip naming both paths, until the provider reports a FileID or the new path is hydrated.

Test. Use FakePlaceholderFS. Rename a synced placeholder with FileID "". Assert that no
OpDeleteRemote is applied, that the server still holds the file, and that the pass hydrates
nothing.

2. The content check before a local delete or download never reads a placeholder (final review X1)

If final review X1 lands as proposed, the engine reads a local file before it deletes it or downloads
over it, whenever the state row vouched for that file without a read. That re-read must skip a
placeholder, because a placeholder cannot have been edited here. Extend this issue's "1,000
placeholders, zero hydrations" test with a server delete and a server edit of placeholders. It must
still show zero hydrations: the delete removes the placeholder, and the edit updates it.

3. A dehydrated tree is not an empty folder (final review X2)

If final review X2's guard lands, the engine refuses a pass when one side lists nothing while the
state store holds rows. A tree of placeholders must count as present entries, never as an empty
folder.

4. OSFS on Windows opens every entry during Walk (#45)

fileID (internal/vfs/fileid_windows.go:20-31) takes the FileID through root.Open plus
GetFileInformationByHandle, which opens a handle with read access for every file Walk lists. This
issue names exactly that failure mode: "a naive Walk that opens files". Before #45 ships, confirm
that opening a Cloud Filter placeholder this way neither hydrates nor recalls it. Otherwise take the
FileID from a handle opened with FILE_READ_ATTRIBUTES only. Run the "zero hydrations" test on
Windows through OSFS.Walk, not only through the fake.

## Amendment — 2026-09-11: what #44 inherits from phase 2 (phase-2 final review X1, X2, and Task 26 F4) Phase 2 put the placeholder rule into the engine (#26): a file with `Hydrated == false` is never opened or hashed, presents as the content last synced, and is never uploaded. Four points from the phase-2 reviews bind this issue. The first was carried here only in #26's closing comment, so it is restated here. ### 1. A renamed placeholder whose FileID is unknown loses its only copy (Task 26 F4) `DetectRenames` pairs a delete with an upload only on a known FileID (`internal/sync/rename.go:24`, `:44-45`). Suppose a placeholder is renamed locally and the provider reports no FileID (`""`). Then: - the old path reads as deleted locally, with its server copy unchanged, so it becomes `OpDeleteRemote`; - the new path is a new local file, so it becomes `OpUpload`, which `put` refuses because the file is a placeholder (`notOnThisDevice`, `internal/sync/engine.go:726-728`). The server copy is deleted. The bytes are then nowhere, because a placeholder holds none. **Required.** A pass never deletes the server copy of a synced path while a placeholder that could be its renamed self is unpaired. The simplest form is this: when the scan finds a placeholder with no state row (only a rename or a provider bug makes one), every unpaired `OpDeleteRemote` in that pass becomes a Skip naming both paths, until the provider reports a FileID or the new path is hydrated. **Test.** Use `FakePlaceholderFS`. Rename a synced placeholder with FileID `""`. Assert that no `OpDeleteRemote` is applied, that the server still holds the file, and that the pass hydrates nothing. ### 2. The content check before a local delete or download never reads a placeholder (final review X1) If final review X1 lands as proposed, the engine reads a local file before it deletes it or downloads over it, whenever the state row vouched for that file without a read. That re-read must skip a placeholder, because a placeholder cannot have been edited here. Extend this issue's "1,000 placeholders, zero hydrations" test with a server delete and a server edit of placeholders. It must still show zero hydrations: the delete removes the placeholder, and the edit updates it. ### 3. A dehydrated tree is not an empty folder (final review X2) If final review X2's guard lands, the engine refuses a pass when one side lists nothing while the state store holds rows. A tree of placeholders must count as present entries, never as an empty folder. ### 4. OSFS on Windows opens every entry during Walk (#45) `fileID` (`internal/vfs/fileid_windows.go:20-31`) takes the FileID through `root.Open` plus `GetFileInformationByHandle`, which opens a handle with read access for every file Walk lists. This issue names exactly that failure mode: "a naive `Walk` that opens files". Before #45 ships, confirm that opening a Cloud Filter placeholder this way neither hydrates nor recalls it. Otherwise take the FileID from a handle opened with `FILE_READ_ATTRIBUTES` only. Run the "zero hydrations" test on Windows through `OSFS.Walk`, not only through the fake.