Cluster: a second process with a member's name can join when its own NATS server disturbs the store (duplicate guard defeats itself) #1183

Closed
opened 2026-11-18 17:58:31 +00:00 by Cordy · 2 comments
Owner

Found in CI on PR #1182 (run 4051, TestDuplicateNameRefusedRealMain742a, cmd/cairnd/cluster742a_realmain_test.go:460). The bug is already in v0.6.409; v0.6.410 does not change it. Part of #742 batch B (batch 10).

What happened (log facts, VERIFIED from the run log)

  • A 3-member embedded cluster runs. A second process starts as cairn-1 (another run) at 17:37:29.
  • Its embedded NATS server carries the same server name. The real members log Remote server has a duplicate name: "cairn-1" for every route attempt, and JetStream on the members turns unavailable on and off: JetStream system temporarily unavailable (code 503), and at 17:37:49 RAFT [_meta_] Detected another leader with higher term, will stepdown.
  • The real cairn-1 cannot write its heartbeat: coord: instance record not written ... 503 (17:37:39).
  • After 30 s the duplicate logs cluster: joined ... members=1 ... leader=cairn-0 and then cairnd listening. The expected refusal cluster: another Cairn process runs as cairn-1 never comes.

Cause (INFERRED)

join() (cmd/cairnd/coordstart742a.go:767) refuses only when it sees a live member record of the same name from another run (dupWindow742a 15 s, waits up to dupWait742a 30 s). The duplicate's own NATS server disturbs the store, so the real member's record ages past the 15 s window and the duplicate sees nothing to refuse. The guard depends on evidence that the duplicate itself destroys. The outcome is timing-dependent: it passed in v0.6.409's CI and in the VM runs.

Impact

Two processes act as the same member: two writers of instance-local paths, double lease holders under one name, and a JetStream meta group disturbed for as long as both run. Most likely on VM or binary installs with a copied config. A StatefulSet keeps names unique, unless an old pod lives on behind a partition.

To research before deciding (playbook Phase 1)

  • NATS 2.15.3 behaviour with duplicate server names in a cluster (route refusal, JetStream meta and peer IDs), and what NATS advises.
  • Options, each with trade-offs:
    • check for the duplicate as a plain client of the peers before starting the embedded server;
    • treat the route-level "duplicate name" refusal as a hard signal;
    • a recently seen record without a left marker counts as live while the store is disturbed;
    • a per-name lease taken before the server starts.
  • External mode: is it affected (no embedded server)?

Tests

  • Make TestDuplicateNameRefusedRealMain742a deterministic.
  • Add a case where the real member's heartbeat is blocked while the duplicate checks.
Found in CI on PR #1182 (run 4051, `TestDuplicateNameRefusedRealMain742a`, cmd/cairnd/cluster742a_realmain_test.go:460). The bug is already in v0.6.409; v0.6.410 does not change it. Part of #742 batch B (batch 10). ## What happened (log facts, VERIFIED from the run log) - A 3-member embedded cluster runs. A second process starts as `cairn-1` (another run) at 17:37:29. - Its embedded NATS server carries the same server name. The real members log `Remote server has a duplicate name: "cairn-1"` for every route attempt, and JetStream on the members turns unavailable on and off: `JetStream system temporarily unavailable (code 503)`, and at 17:37:49 `RAFT [_meta_] Detected another leader with higher term, will stepdown`. - The real `cairn-1` cannot write its heartbeat: `coord: instance record not written ... 503` (17:37:39). - After 30 s the duplicate logs `cluster: joined ... members=1 ... leader=cairn-0` and then `cairnd listening`. The expected refusal `cluster: another Cairn process runs as cairn-1` never comes. ## Cause (INFERRED) `join()` (cmd/cairnd/coordstart742a.go:767) refuses only when it sees a live member record of the same name from another run (`dupWindow742a` 15 s, waits up to `dupWait742a` 30 s). The duplicate's own NATS server disturbs the store, so the real member's record ages past the 15 s window and the duplicate sees nothing to refuse. The guard depends on evidence that the duplicate itself destroys. The outcome is timing-dependent: it passed in v0.6.409's CI and in the VM runs. ## Impact Two processes act as the same member: two writers of instance-local paths, double lease holders under one name, and a JetStream meta group disturbed for as long as both run. Most likely on VM or binary installs with a copied config. A StatefulSet keeps names unique, unless an old pod lives on behind a partition. ## To research before deciding (playbook Phase 1) - NATS 2.15.3 behaviour with duplicate server names in a cluster (route refusal, JetStream meta and peer IDs), and what NATS advises. - Options, each with trade-offs: - check for the duplicate as a plain client of the peers before starting the embedded server; - treat the route-level "duplicate name" refusal as a hard signal; - a recently seen record without a left marker counts as live while the store is disturbed; - a per-name lease taken before the server starts. - External mode: is it affected (no embedded server)? ## Tests - Make `TestDuplicateNameRefusedRealMain742a` deterministic. - Add a case where the real member's heartbeat is blocked while the duplicate checks.
Author
Owner

Research: research-b10-r4-lifecycle.md. Worse than first described: NATS routes accept a second server with a peer's name, and Raft peer ids come from the name, so lost acknowledged writes are possible (INFERRED from nats-server 2.15.3 route.go and raft.go).

Decided for batch 10 (v0.6.411), three layers:

  1. Before the embedded server starts, read each peer's announced server name over the route port. Our own name answering means refuse.
  2. NATS's own duplicate-name refusal is treated as a hard refusal before join.
  3. The record check stays.

A duplicate seen after join raises the bell cluster-member-duplicate and a "duplicate" state in status, and the newer process stops. The handbook line "refused within 30 seconds" gets corrected.

Research: `research-b10-r4-lifecycle.md`. Worse than first described: NATS routes accept a second server with a peer's name, and Raft peer ids come from the name, so lost acknowledged writes are possible (INFERRED from nats-server 2.15.3 route.go and raft.go). Decided for batch 10 (v0.6.411), three layers: 1. Before the embedded server starts, read each peer's announced server name over the route port. Our own name answering means refuse. 2. NATS's own duplicate-name refusal is treated as a hard refusal before join. 3. The record check stays. A duplicate seen after join raises the bell `cluster-member-duplicate` and a "duplicate" state in status, and the newer process stops. The handbook line "refused within 30 seconds" gets corrected.
Author
Owner

Shipped in v0.6.411 (PR #1186); live on both dogfoods.

The three layers:

  1. Route probe before the embedded server starts. It refuses only on a positive own-name answer; no answer, a TLS failure or another cluster is never a refusal.
  2. NATS's own duplicate-name refusal counts as a refusal.
  3. The record check.

After join, the newer process stops without a left marker. The older one shows the state duplicate, the bell item and the audit line cluster-member-duplicate. Detection by the member record no longer depends on clocks.

VM run: a copied member directory was refused 3 ms after joining, a copied config 7 ms after, and a duplicate outside the peers list 44 ms after. The members were never disturbed.

Shipped in v0.6.411 (PR #1186); live on both dogfoods. The three layers: 1. **Route probe before the embedded server starts.** It refuses only on a positive own-name answer; no answer, a TLS failure or another cluster is never a refusal. 2. **NATS's own duplicate-name refusal** counts as a refusal. 3. **The record check.** After join, the newer process stops without a left marker. The older one shows the state `duplicate`, the bell item and the audit line `cluster-member-duplicate`. Detection by the member record no longer depends on clocks. VM run: a copied member directory was refused 3 ms after joining, a copied config 7 ms after, and a duplicate outside the peers list 44 ms after. The members were never disturbed.
Cordy closed this issue 2026-11-19 21:01:50 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1183
No description provided.