Certificates: every rollout shows the previous pod as "not reporting" for 24 h (false amber dot after each update) #1009

Closed
opened 2026-10-26 10:17:24 +00:00 by Cordy · 2 comments
Owner

Seen live after the v0.6.355 rollout (2026-10-26). Administration › Certificates and the rail dot show not-reporting (warning) for cairn-enc-f55db9db7-mlxzc, the v0.6.354 pod that the Recreate rollout replaced. Its entry in tlsstatus.json stays until the 24 h prune.

Instance ids are pod names, so every update, restart or reschedule produces this. The admin learns to ignore the amber dot, which defeats it. The stored Check connection results under "Other instances" (Users & access) show the old pod for the same reason, though there it is only informational.

Fix direction:

  • On graceful shutdown (SIGTERM), an instance removes its own entries: tlsstatus.json (and the bell times keyed to it) and directory-checks.json. A crashed or killed instance still shows "not reporting", which is the case worth warning about.
  • Alternatively or in addition: ignore an entry whose instance stopped reporting within a minute of a new instance starting with the same deployment prefix. This is weaker and relies on naming, so prefer the shutdown hook.
  • Tests: graceful stop removes the entry; a kill keeps it, and it shows after 15 min; two instances, stop one gracefully, and no warning appears.
  • #742: the shutdown write uses WriteShared with read-merge-write.
Seen live after the v0.6.355 rollout (2026-10-26). Administration › Certificates and the rail dot show `not-reporting` (warning) for `cairn-enc-f55db9db7-mlxzc`, the v0.6.354 pod that the `Recreate` rollout replaced. Its entry in `tlsstatus.json` stays until the 24 h prune. Instance ids are pod names, so every update, restart or reschedule produces this. The admin learns to ignore the amber dot, which defeats it. The stored Check connection results under "Other instances" (Users & access) show the old pod for the same reason, though there it is only informational. **Fix direction:** - On graceful shutdown (SIGTERM), an instance removes its own entries: `tlsstatus.json` (and the bell times keyed to it) and `directory-checks.json`. A crashed or killed instance still shows "not reporting", which is the case worth warning about. - Alternatively or in addition: ignore an entry whose instance stopped reporting within a minute of a new instance starting with the same deployment prefix. This is weaker and relies on naming, so prefer the shutdown hook. - Tests: graceful stop removes the entry; a kill keeps it, and it shows after 15 min; two instances, stop one gracefully, and no warning appears. - #742: the shutdown write uses `WriteShared` with read-merge-write.
Cordy referenced this issue from a commit 2026-10-26 10:56:45 +00:00
Author
Owner

Shipped in v0.6.356 (PR #1014), live on both dogfoods since 2026-10-26 11:45.

On SIGTERM/SIGINT an instance now removes its own entries from tlsstatus.json and directory-checks.json. This happens during the HTTP drain (typically 1-2 s, capped at 5 s), with a second check one second later in case another instance wrote the entry back. A killed or crashed instance still shows "not reporting".

As the CHANGELOG says, this rollout itself still shows the old v0.6.355 pod once, because the old version does not remove its entry. Keeping this open until the next rollout confirms it on the dogfoods: after the next version bump, the Certificates page and "Other instances" must show no "not reporting" entry for the replaced pod. This is also tracked in #1005 F.

The related shutdown drain problem under Docker's 10 s stop is #1010.

Shipped in **v0.6.356** (PR #1014), live on both dogfoods since 2026-10-26 11:45. On SIGTERM/SIGINT an instance now removes its own entries from `tlsstatus.json` and `directory-checks.json`. This happens during the HTTP drain (typically 1-2 s, capped at 5 s), with a second check one second later in case another instance wrote the entry back. A killed or crashed instance still shows "not reporting". As the CHANGELOG says, this rollout itself still shows the old v0.6.355 pod once, because the old version does not remove its entry. **Keeping this open until the next rollout confirms it on the dogfoods**: after the next version bump, the Certificates page and "Other instances" must show no "not reporting" entry for the replaced pod. This is also tracked in #1005 F. The related shutdown drain problem under Docker's 10 s stop is #1010.
Author
Owner

Verified live with the v0.6.357 rollout (2026-10-26 17:54). The two v0.6.356 pods (files: started 11:45; files-bao: started 11:45) are gone from tlsstatus.json, so they removed their own entries on shutdown.

The entries still listed are older pods that ran without the fix:

  • files: cairn-enc-859d55f698 (v0.6.355) and cairn-enc-f55db9db7 (v0.6.354)
  • files-bao: cairn-openbao-9868df564 and cairn-openbao-56f5fcbf87

They are pruned by the 24 h rule, at the latest around 2026-10-27 11:45. I also reproduced the removal locally: SIGTERM, then tlsstatus.json is {}. Closing.

**Verified live with the v0.6.357 rollout (2026-10-26 17:54).** The two v0.6.356 pods (files: started 11:45; files-bao: started 11:45) are gone from `tlsstatus.json`, so they removed their own entries on shutdown. The entries still listed are older pods that ran without the fix: - files: `cairn-enc-859d55f698` (v0.6.355) and `cairn-enc-f55db9db7` (v0.6.354) - files-bao: `cairn-openbao-9868df564` and `cairn-openbao-56f5fcbf87` They are pruned by the 24 h rule, at the latest around 2026-10-27 11:45. I also reproduced the removal locally: SIGTERM, then `tlsstatus.json` is `{}`. Closing.
Cordy closed this issue 2026-10-26 18:03:08 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1009
No description provided.