Cluster: after migrate --to local the exporting member can stay up (it never reads its own exported record once the others left) #1185

Closed
opened 2026-11-18 20:42:15 +00:00 by Cordy · 2 comments
Owner

Found by CI on PR #1184 (run 4059, TestShrinkThenGrowFirstTimeRealMainFx9a, cmd/cairnd/fx9a742a_realmain_test.go:82: cairn-0 did not exit within 1m30s). Already in v0.6.409 and v0.6.410; timing-dependent. Part of #742 batch B (batch 10), together with #1183.

What happened (VERIFIED from the run log)

  • 20:26:18.680, cairn-0: cluster: migrated from=nats to=local .... The CLI ran on cairn-0's socket, and the handler set the record to exported (adminhandler742a.go:903).
  • 20:26:19.289: cairn-1 and cairn-2 read the exported record on their 1 s poll, log the cluster was exported to single-instance storage, leave and shut down (shutdown complete 20:26:20).
  • 20:26:19.321, cairn-0, 30 ms later: cluster: the cluster record cannot be read; this instance keeps its last state ... the stream leader does not see a majority. From then on it only retries (the legacy fence lease cannot be taken; trying again) and never exits.

Cause (INFERRED)

watchRecord (coordstart742a.go:1056) is the only way into the exported path, including the exporting member's own awaitOthersLeft742a. It enters only by reading the record. The exporting member already knows the outcome, because its Meta.Update(... StateExported) succeeded. But if both other members read and leave before its next poll, the store has no majority and the read never succeeds. The design note ("its server keeps the majority while the other members read the exported record and leave") assumes the exporting member reads first.

Impact

After cairnd migrate --to local on 3 members, the member where the command ran can stay up in maintenance instead of exiting 0. The operator has to stop it by hand. To check: does it still hold or retry the fence while running, and does that affect starting the single instance on the exported state?

Fix direction (to research and review)

  • The handler stores the record it wrote (rt.Record.Store).
  • When a read fails, the watch acts on a stored record that is exported and was changed by this member: enter the exported branch (awaitOthersLeft742a, whose three failed reads end the wait) and stop.
  • Red test: a seam for the record read, failing after the handler's update, so the test is deterministic (not the real-main timing).
Found by CI on PR #1184 (run 4059, `TestShrinkThenGrowFirstTimeRealMainFx9a`, cmd/cairnd/fx9a742a_realmain_test.go:82: `cairn-0 did not exit within 1m30s`). Already in v0.6.409 and v0.6.410; timing-dependent. Part of #742 batch B (batch 10), together with #1183. ## What happened (VERIFIED from the run log) - 20:26:18.680, cairn-0: `cluster: migrated from=nats to=local ...`. The CLI ran on cairn-0's socket, and the handler set the record to exported (`adminhandler742a.go:903`). - 20:26:19.289: cairn-1 and cairn-2 read the exported record on their 1 s poll, log `the cluster was exported to single-instance storage`, leave and shut down (`shutdown complete` 20:26:20). - 20:26:19.321, cairn-0, 30 ms later: `cluster: the cluster record cannot be read; this instance keeps its last state ... the stream leader does not see a majority`. From then on it only retries (`the legacy fence lease cannot be taken; trying again`) and never exits. ## Cause (INFERRED) `watchRecord` (coordstart742a.go:1056) is the only way into the exported path, including the exporting member's own `awaitOthersLeft742a`. It enters only by *reading* the record. The exporting member already knows the outcome, because its `Meta.Update(... StateExported)` succeeded. But if both other members read and leave before its next poll, the store has no majority and the read never succeeds. The design note ("its server keeps the majority while the other members read the exported record and leave") assumes the exporting member reads first. ## Impact After `cairnd migrate --to local` on 3 members, the member where the command ran can stay up in maintenance instead of exiting 0. The operator has to stop it by hand. To check: does it still hold or retry the fence while running, and does that affect starting the single instance on the exported state? ## Fix direction (to research and review) - The handler stores the record it wrote (`rt.Record.Store`). - When a read fails, the watch acts on a stored record that is exported and was changed by this member: enter the exported branch (`awaitOthersLeft742a`, whose three failed reads end the wait) and stop. - Red test: a seam for the record read, failing after the handler's update, so the test is deterministic (not the real-main timing).
Author
Owner

Research: research-b10-r4-lifecycle.md. Decided for batch 10 (v0.6.411), one fix for 3 and 5 members:

  • The handler keeps the record it wrote, so the exporting member acts on its own write.
  • Every member that learns of the export writes its left marker as an acknowledgement. It keeps its NATS server up until every live member has acknowledged (2 s grace, 60 s deadline).
  • Failed reads no longer end the wait.
  • An export is refused while a configured member is not live; the dry run lists it.

The same cause explains the earlier TestMigrateThreeToOneRealMain742a flake.

Research: `research-b10-r4-lifecycle.md`. Decided for batch 10 (v0.6.411), one fix for 3 and 5 members: - The handler keeps the record it wrote, so the exporting member acts on its own write. - Every member that learns of the export writes its left marker as an acknowledgement. It keeps its NATS server up until every live member has acknowledged (2 s grace, 60 s deadline). - Failed reads no longer end the wait. - An export is refused while a configured member is not live; the dry run lists it. The same cause explains the earlier `TestMigrateThreeToOneRealMain742a` flake.
Author
Owner

Shipped in v0.6.411 (PR #1186); live on both dogfoods.

  • The exporting member acts on its own written record.
  • Every member acknowledges with its left marker and keeps NATS up until all live members have acknowledged (2 s grace, 60 s deadline). Failed reads no longer end the wait.
  • An export is refused while a member is not running, and the dry run lists it.

Deterministic red tests reproduced #1185 for 3 and 5 members.

VM run:

  • 3 members: all exit 0 within about 6 s.
  • 5 members on 3 hosts: all exit 0 within 5 s; with one member paused shortly after the start, the export still ends correctly in 10 s.
  • A member hung at the very start: the export is refused before anything is frozen (safe, documented).
Shipped in v0.6.411 (PR #1186); live on both dogfoods. - The exporting member acts on its own written record. - Every member acknowledges with its left marker and keeps NATS up until all live members have acknowledged (2 s grace, 60 s deadline). Failed reads no longer end the wait. - An export is refused while a member is not running, and the dry run lists it. Deterministic red tests reproduced #1185 for 3 and 5 members. VM run: - 3 members: all exit 0 within about 6 s. - 5 members on 3 hosts: all exit 0 within 5 s; with one member paused shortly after the start, the export still ends correctly in 10 s. - A member hung at the very start: the export is refused before anything is frozen (safe, documented).
Cordy closed this issue 2026-11-19 21:01:54 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1185
No description provided.