v0.6.412: #742 batch B part 2a: uploads on any instance, the shared change feed, nats.go 1.56.0 #1188

Merged
Cordy merged 65 commits from b11v07 into main 2026-11-21 02:13:56 +00:00
Owner

#742 batch B part 2a. Research, decisions, approved mockup, spec/plan, reviews and run logs: .superpowers/sdd/2026-11-20-b11v07/.

Uploads on any instance (decisions-b10 item 5, decisions-b11)

  • Any member accepts any piece and finishes any upload. The upload record (with epoch and piece runs) lives in the shared store. Pieces are write-once objects under .cairn-state/tus/<key>/, always sealed, also with content encryption off.
  • Take-over and finish.
    • A newer request takes an upload over; the older one gets 409 with Upload-Offset.
    • The finish claim is held inside the record. Adopt-on-match means a stored file is never written twice.
    • At most 4 finishes run per member; further finishes are queued, and HEAD answers upload-finishing.
    • A crash mid-finish rolls back to the last piece, and the client's resend finishes it on any member.
  • Quota and errors. Quota reservations are shared and fail closed. Errors are 503, never 404, in outages, maintenance and the switch window.
  • Web client. It pauses and resumes for up to 10 minutes, and every upload error text is localised (12 new strings, approved mockup).
  • Migrate. Unfinished uploads are carried at migrate --to nats (--uploads-from) and handed over at --to local.

Shared change feed

  • One sealed JetStream stream per cluster, with random epochs.
  • Every gap, crash, gone member, forget or stream recreate starts a new epoch, so no change is ever missed silently.
  • 503 during outages, with the cursor kept.
  • The export writes a fresh changes.json, which closes the old-cursor hazard.

Building blocks

  • <prefix>_auth bucket and sealed coord records.
  • One classification table for migrate and backup (dup.*, feed, tus are no longer backed up).
  • The shared features switch on when the cluster's minimum version is raised to v0.6.412.
  • nats.go 1.56.0, a TLS security release.

Fixed, single instance too

  • An at-rest key rotation with an empty stream or bucket kept a member down (nats-server cannot convert empty blocks). Empty blocks now get fresh keys.
  • A restart during a finish could store an upload twice.
  • 409, HEAD and 410 headers now follow the tus spec.
  • Start WARN when the temp folder is on a RAM filesystem while the storage is S3 (OOM risk found on the VMs; host docs fixed).
  • S3 retries are logged.
  • A too-small NATS account refuses at once with the exact formula.
  • A time-of-day test fixed.

Verification

  • 4 waves, 5 fix rounds, and a final review with adversarial probes: 0 Critical or High open.
  • Every package passes; 79 real-main tests; web 324 and a11y 94.
  • VM run on 3 hosts (run-u14.md), all product guards PASS:
    • rolling update and switch;
    • pieces across members (1 and 10 GiB);
    • kill, crash and STOP mid-finish;
    • duplicate PATCH;
    • quota with no overshoot;
    • change feed 132 of 132;
    • migrate both ways;
    • external account limits measured.
#742 batch B part 2a. Research, decisions, approved mockup, spec/plan, reviews and run logs: `.superpowers/sdd/2026-11-20-b11v07/`. **Uploads on any instance** (decisions-b10 item 5, decisions-b11) - **Any member accepts any piece and finishes any upload.** The upload record (with epoch and piece runs) lives in the shared store. Pieces are write-once objects under `.cairn-state/tus/<key>/`, always sealed, also with content encryption off. - **Take-over and finish.** - A newer request takes an upload over; the older one gets 409 with `Upload-Offset`. - The finish claim is held inside the record. Adopt-on-match means a stored file is never written twice. - At most 4 finishes run per member; further finishes are queued, and HEAD answers `upload-finishing`. - A crash mid-finish rolls back to the last piece, and the client's resend finishes it on any member. - **Quota and errors.** Quota reservations are shared and fail closed. Errors are 503, never 404, in outages, maintenance and the switch window. - **Web client.** It pauses and resumes for up to 10 minutes, and every upload error text is localised (12 new strings, approved mockup). - **Migrate.** Unfinished uploads are carried at `migrate --to nats` (`--uploads-from`) and handed over at `--to local`. **Shared change feed** - One sealed JetStream stream per cluster, with random epochs. - Every gap, crash, gone member, forget or stream recreate starts a new epoch, so no change is ever missed silently. - 503 during outages, with the cursor kept. - The export writes a fresh `changes.json`, which closes the old-cursor hazard. **Building blocks** - `<prefix>_auth` bucket and sealed coord records. - One classification table for migrate and backup (`dup.*`, `feed`, `tus` are no longer backed up). - The shared features switch on when the cluster's minimum version is raised to v0.6.412. - nats.go 1.56.0, a TLS security release. **Fixed, single instance too** - An at-rest key rotation with an empty stream or bucket kept a member down (nats-server cannot convert empty blocks). Empty blocks now get fresh keys. - A restart during a finish could store an upload twice. - 409, HEAD and 410 headers now follow the tus spec. - Start WARN when the temp folder is on a RAM filesystem while the storage is S3 (OOM risk found on the VMs; host docs fixed). - S3 retries are logged. - A too-small NATS account refuses at once with the exact formula. - A time-of-day test fixed. **Verification** - 4 waves, 5 fix rounds, and a final review with adversarial probes: 0 Critical or High open. - Every package passes; 79 real-main tests; web 324 and a11y 94. - VM run on 3 hosts (`run-u14.md`), all product guards PASS: - rolling update and switch; - pieces across members (1 and 10 GiB); - kill, crash and STOP mid-finish; - duplicate PATCH; - quota with no overshoot; - change feed 132 of 132; - migrate both ways; - external account limits measured.
- openFeed742c at the coordination start (refusal reason feed-stream), feedStream742c adapter,
  feedShared742c registers the first switch hook (Share retried, changes.json deleted after it)
- W1 review M2: leased pass feed-writers and cluster forget rotate once (crash) for the dirty
  writer marker of a gone member (Journal.ReapWriters); hourly KeepWriter renews the marker
- /api/v1/changes: 503 coord-unavailable (Retry-After 30, cursor unmoved) for polls and first
  contacts during an outage (Journal.HeadErr; local mode unchanged)
- coordination.changeFeed.maxAge/maxBytes; cairn_change_feed_rotations_total{reason};
  cluster status feed line, feed block and the <prefix>_log_changes row
- natsserver: an at-rest key rotation restart re-wraps empty message blocks' key files
  (nats-server 2.15.3 cannot convert an empty block; an empty stream kept the member down)
The shared reservations record counts its writes within a random epoch
and logs the entries each write removes for 2 minutes. A member that reads
a newer record (the quota-res notify, a refresh, the fresh read of every
Reserve, Renew, Release, Rebuild) marks its usage figures for the caps of
the entries that left (the instance figure too; every figure when the log
does not reach back to its copy) as stale: the next lookup waits up to 2 s
for a new walk, and falls back to the old figure plus the bytes that left.
A walk running at the mark is not cached. Own writes mark nothing.
A generic tus client's one-PATCH upload makes 64 MiB pieces; read ahead
whole, 4 finishes held up to 768 MiB. The fetcher now takes a byte budget
of 16 MiB per finish before it reads a piece (given back once the piece
is written), reads into a buffer of the record's length, and hands a
piece larger than the budget over unread, streamed by a pieceReader.
Docs: the read-ahead line in deployment.md, the sizing note.
Verified in nats-server v2.15.3: accountReservation (jetstream.go) counts
replicas x MaxBytes for an account without tiers, one replica's worth for
a tier; checkBytesLimits compares that against the account's max_file and
one replica's worth against each server's own JetStream max_file minus
its reservations (tieredStreamAndReservationCount adds the other streams'
reservations the same way).
Cordy merged commit e147b92bd5 into main 2026-11-21 02:13:56 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1188
No description provided.