Name search stops silently at 500 folders or 200 results #1102

Closed
opened 2026-11-04 03:11:03 +00:00 by Cordy · 7 comments
Owner

Gap (v0.7 feature-gap research, 2026-11-04, decided by Nikola). Each search visits at most 500 folders, 12 levels deep, and returns at most 200 results (internal/api/search.go:19-24 at 0e0885db, VERIFIED). In a larger tree results are incomplete and the person is not told.

To research and decide when worked on. Tell the person when a limit was hit and offer "search further"; a name index (built from the change feed) vs a smarter walk; s3 listing cost; spaces and shared folders in scope; multi-instance (#742). Full-text content search stays after launch (v1.x).

Refs: scratch/b7e/backlog/gap-v07/research-v07-gaps.md A14, research-opencloud.md.

**Gap (v0.7 feature-gap research, 2026-11-04, decided by Nikola).** Each search visits at most 500 folders, 12 levels deep, and returns at most 200 results (`internal/api/search.go:19-24` at 0e0885db, VERIFIED). In a larger tree results are incomplete and the person is not told. **To research and decide when worked on.** Tell the person when a limit was hit and offer "search further"; a name index (built from the change feed) vs a smarter walk; s3 listing cost; spaces and shared folders in scope; multi-instance (#742). Full-text content search stays after launch (v1.x). Refs: `scratch/b7e/backlog/gap-v07/research-v07-gaps.md` A14, research-opencloud.md.
Author
Owner

Research done (scratch/b7e/backlog/research-1102.md). Not fully silent today: the server returns truncated and the page shows a small grey "(more results exist; refine your search)", which is wrong advice when the 500-folder cap is the cause, and with zero results it also says "No matches". Further: the first 200 matches in walk order win over better ones; each folder costs 4 to 9 requests on s3; names are only lower-cased (NFD and accents do not match); Type and Modified filters only filter the 200 returned; searches typed meanwhile are never cancelled.

Recommended for launch (keeps "no search index"): keep the best 200 instead of the first; budget per backend (posix 2,000 folders, s3 500) plus 3 s; the answer says whether it is complete; Search further continues with a stateless resume token re-checked against access on every step (whole-segment containment, cf. GHSA-p6ph-3jx2-3337); a cheaper listing for search on s3 with a small parallelism and 429 search-busy; accent and Unicode folding as in Find spaces; filters on the server; stale searches cancelled; a visible notice above the results. After launch: an optional name index fed by the change journal, full-text (v1.x).

Decisions asked of Nikola next.

**Research done** (`scratch/b7e/backlog/research-1102.md`). Not fully silent today: the server returns `truncated` and the page shows a small grey "(more results exist; refine your search)", which is wrong advice when the 500-folder cap is the cause, and with zero results it also says "No matches". Further: the first 200 matches in walk order win over better ones; each folder costs 4 to 9 requests on s3; names are only lower-cased (NFD and accents do not match); Type and Modified filters only filter the 200 returned; searches typed meanwhile are never cancelled. Recommended for launch (keeps "no search index"): keep the best 200 instead of the first; budget per backend (posix 2,000 folders, s3 500) plus 3 s; the answer says whether it is complete; **Search further** continues with a stateless resume token re-checked against access on every step (whole-segment containment, cf. GHSA-p6ph-3jx2-3337); a cheaper listing for search on s3 with a small parallelism and 429 `search-busy`; accent and Unicode folding as in Find spaces; filters on the server; stale searches cancelled; a visible notice above the results. After launch: an optional name index fed by the change journal, full-text (v1.x). Decisions asked of Nikola next.
Author
Owner

Decision (Nikola, 2026-11-04): build a name index now, instead of the improved walk with "Search further". This changes the scope: the index design (where it lives, encryption of names at rest, how it is fed and kept correct from the change journal and storage, rebuild, memory, s3, several instances, what search shows while it builds) gets its own research round (research-1102-index.md) before the questions it raises are put to Nikola. The walk improvements that still apply (best matches, accent folding, server-side filters, cancelling stale searches, a visible notice) are kept where the index does not replace them.

**Decision (Nikola, 2026-11-04): build a name index now**, instead of the improved walk with "Search further". This changes the scope: the index design (where it lives, encryption of names at rest, how it is fed and kept correct from the change journal and storage, rebuild, memory, s3, several instances, what search shows while it builds) gets its own research round (`research-1102-index.md`) before the questions it raises are put to Nikola. The walk improvements that still apply (best matches, accent folding, server-side filters, cancelling stale searches, a visible notice) are kept where the index does not replace them.
Author
Owner

Index design approved (Nikola, 2026-11-04), from research-1102-index.md: in-memory index per instance, rebuilt from storage at start (nothing on disk), per home and space, names only; built in the background from a new flat lister with a strict hidden-name filter (folder walk as fallback); kept fresh from the change journal, a jittered hourly full check and repairs on folder listing and on vanished results; every result re-checked through the person's access; accent folding; best 1,000, pages of 50 with Show more; Type/Modified filters on the server; unindexed folders fall back to the bounded walk, marked incomplete; on by default with a switch; cap 2 million entries; one result per item at its best path; admin "Name search" card with "Prepare again" (audited); one index per instance for launch; README "no search index" reworded in the same release. Next: whole-page mockup (search states, admin card).

**Index design approved (Nikola, 2026-11-04)**, from `research-1102-index.md`: in-memory index per instance, rebuilt from storage at start (nothing on disk), per home and space, names only; built in the background from a new flat lister with a strict hidden-name filter (folder walk as fallback); kept fresh from the change journal, a jittered hourly full check and repairs on folder listing and on vanished results; every result re-checked through the person's access; accent folding; best 1,000, pages of 50 with Show more; Type/Modified filters on the server; unindexed folders fall back to the bounded walk, marked incomplete; on by default with a switch; cap 2 million entries; one result per item at its best path; admin "Name search" card with "Prepare again" (audited); one index per instance for launch; README "no search index" reworded in the same release. Next: whole-page mockup (search states, admin card).
Author
Owner

Mockup approved with changes (Nikola, 2026-11-04), after research (research-1102-unsettled.md): the card with "Prepare again" sits in Storage & quotas › Usage, Overview gets a one-line "Name search" row with a status chip (amber for "Partly prepared"); the card refreshes every 3 s in place; "Prepare again" while preparing answers 409 search-index-preparing; paging an incomplete answer on S3 can take seconds. Confirmed: "Load more", "Personal folders and spaces", UTC from/to, capped, the notice only on incomplete answers, no bold highlighting in v0.7. Next: spec and plan (.superpowers/sdd/2026-11-04-ix1102).

**Mockup approved with changes (Nikola, 2026-11-04)**, after research (`research-1102-unsettled.md`): the card with "Prepare again" sits in Storage & quotas › Usage, Overview gets a one-line "Name search" row with a status chip (amber for "Partly prepared"); the card refreshes every 3 s in place; "Prepare again" while preparing answers 409 `search-index-preparing`; paging an incomplete answer on S3 can take seconds. Confirmed: "Load more", "Personal folders and spaces", UTC from/to, `capped`, the notice only on incomplete answers, no bold highlighting in v0.7. Next: spec and plan (`.superpowers/sdd/2026-11-04-ix1102`).
Author
Owner

Spec open points decided (Nikola, 2026-11-04), after research (research-1102-spec-unsettled.md): all confirmed. U1 the 0-byte placeholder removal records a delete in the change feed; U2 instance=<id> in the search-index-rebuild audit note; U3 first build: recently searched folders first, then by name; U4 incomplete and capped shows the "results may be missing" notice; U5 "Being prepared" chip in info blue; U6 no repair from WebDAV listings in v0.7. Also: FR "résultat(s)", IT "risultato/i", IT "Da {time}", IT "cercare in"; the missing writer row (space delete rolled back) and a stronger bypass-inventory guard test.

**Spec open points decided (Nikola, 2026-11-04), after research (`research-1102-spec-unsettled.md`): all confirmed.** U1 the 0-byte placeholder removal records a `delete` in the change feed; U2 `instance=<id>` in the `search-index-rebuild` audit note; U3 first build: recently searched folders first, then by name; U4 incomplete and capped shows the "results may be missing" notice; U5 "Being prepared" chip in info blue; U6 no repair from WebDAV listings in v0.7. Also: FR "résultat(s)", IT "risultato/i", IT "Da {time}", IT "cercare in"; the missing writer row (space delete rolled back) and a stronger bypass-inventory guard test.
Cordy closed this issue 2026-11-04 22:02:04 +00:00
Author
Owner

Shipped in v0.6.399 (PR #1130), live on both dogfoods: cairnd listening … version=v0.6.399, healthy, no ERROR at start.

  • files-bao (s3): search index: prepared roots=19 entries=45 took=378ms
  • files (posix): search index: prepared roots=9 entries=1 took=236ms; /metrics shows 9 roots ready, 0 failed.

One check left before closing: 1 entry over 9 roots on the posix instance is either right (that instance holds almost nothing) or a posix indexing gap. I can't tell from outside: the redeploy signed everyone out and I don't sign in. @Cordy please sign in on files.c0rdyceps.ch and look at Admin › Storage & quotas › Usage › Name search: "Files and folders: n" should roughly match what you have there. Then search for a file name you know exists in a subfolder. If n looks too small, say so and I'll dig in before anything else.

Also decided and built in this release (2026-11-04): a capped answer never says "No matches" (searchCapFiltered); with Last year set, files already older than the bound rank first; API clients are told a page can be empty while more is true. Follow-ups: #1119 (filters for tag: searches), #1131 (a versions test flaked in CI on this PR; unrelated package, filed). Residuals R1 to R8 are in deployment.md.

Shipped in **v0.6.399** (PR #1130), live on both dogfoods: `cairnd listening … version=v0.6.399`, healthy, no ERROR at start. - files-bao (s3): `search index: prepared roots=19 entries=45 took=378ms` - files (posix): `search index: prepared roots=9 entries=1 took=236ms`; `/metrics` shows 9 roots ready, 0 failed. **One check left before closing:** 1 entry over 9 roots on the posix instance is either right (that instance holds almost nothing) or a posix indexing gap. I can't tell from outside: the redeploy signed everyone out and I don't sign in. @Cordy please sign in on files.c0rdyceps.ch and look at Admin › Storage & quotas › Usage › **Name search**: "Files and folders: n" should roughly match what you have there. Then search for a file name you know exists in a subfolder. If n looks too small, say so and I'll dig in before anything else. Also decided and built in this release (2026-11-04): a capped answer never says "No matches" (`searchCapFiltered`); with Last year set, files already older than the bound rank first; API clients are told a page can be empty while `more` is true. Follow-ups: #1119 (filters for `tag:` searches), #1131 (a versions test flaked in CI on this PR; unrelated package, filed). Residuals R1 to R8 are in deployment.md.
Cordy reopened this issue 2026-11-04 22:07:19 +00:00
Author
Owner

Checked live (2026-11-05, signed in as nikola-test):

  • files (posix): Name search is Ready, 9 parts ready, 0 failed, 1 entry. A full walk of everything the account can see finds exactly one file (/spaces/zz-967-test/hello-967.txt) plus space folders and an empty home, and space folders are parts rather than entries. So 1 is correct: that instance holds almost nothing. hello and 967 both find the file, complete: true.
  • files-bao (s3): Ready, 19 parts, 45 entries. A file in a subfolder is found (karpathy → /home/asd/Andrej-karpathy-skills.md), drawio finds both copies, and a name that only exists inside a space's Deleted files store is not found (trash trees stay out of the index).

One difference from v0.6.398 found on the way: a space is no longer found by its own name (folderino finds nothing although the space "Test Ümlauf folderinö" exists). Filed as #1132 (v0.7).

Closing.

**Checked live (2026-11-05, signed in as nikola-test):** - files (posix): Name search is Ready, 9 parts ready, 0 failed, **1 entry**. A full walk of everything the account can see finds exactly one file (`/spaces/zz-967-test/hello-967.txt`) plus space folders and an empty home, and space folders are parts rather than entries. So 1 is correct: that instance holds almost nothing. `hello` and `967` both find the file, `complete: true`. - files-bao (s3): Ready, 19 parts, **45 entries**. A file in a subfolder is found (`karpathy` → `/home/asd/Andrej-karpathy-skills.md`), `drawio` finds both copies, and a name that only exists inside a space's Deleted files store is not found (trash trees stay out of the index). One difference from v0.6.398 found on the way: a space is no longer found by its own name (`folderino` finds nothing although the space "Test Ümlauf folderinö" exists). Filed as #1132 (v0.7). Closing.
Cordy closed this issue 2026-11-05 21:57:57 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1102
No description provided.