#437 - Licence dropped 60→50 on cairn-enc: one 403 self-revokes forever, silently — retry window, reason surfacing, metrics (pre-launch) - Cordy/Cairn

Cordy commented

2026-09-06 01:55:24 +00:00

Owner

Incident (2026-09, cairn-enc)

Nikola noticed the dogfood dropped from the 60-seat licence to the free tier; the licence page said "licence revoked — running on the free tier". He recovered it by re-pasting the key (2026-09-03 23:40 — the successful check-in right after his account-page login at 23:37).

Root cause (confirmed)

The licence itself was never revoked or expired: server DB shows revoked=0, key valid to 2027-07-29 (the sub is yearly; the monthly one belongs to bao).
The client's check-in received a 403 at some point in the window — almost certainly reason unknown during the licence server's account-era redeploys / JSON→SQLite migration (Aug 24–26; frequent restarts in the journal).
Client behaviour on 403 revoked/unknown: set Revoked=true → cap drops to 50 immediately → checkinRequiredLocked returns false → check-ins stop permanently. No self-heal is possible; only a manual Refresh/SetKey clears it. One transient server-side 403 = a silently bricked licence.

Why it took days to notice, and could not be reconstructed

The server logs no check-ins at all (no successes, no 403s, no reasons); lastSeen is one overwritten value.
Client logs died with ~20 pod restarts (Pis have no persistent journald).
The licence page shows free-tier numbers with one terse line; no since-when, no what-happened, no notification.

Build plan (approved by Nikola; UI parts mockup-first)

Client (this repo):

403 is no longer terminal on first sight. A revoked/unknown refusal keeps the licence inactive (honest) but check-ins continue daily for a revoked-watch window (~14 days, mirroring the #308 expired-watch), so a transient server-side 403 self-heals at the next 200. A genuinely revoked licence gets two weeks of daily 403s, then goes quiet for good.
The drop explains itself. Persist reason + since-when; the Licensing page shows an inactive-state card (revoked / grace exhausted / expired variants — mockup first); an admin bell notification fires on the transition; loud log lines on every state change.
Metrics + alert. Export licensed-state/seat-cap gauges; homelab-config gains CairnLicenseDropped (key installed ∧ not licensed) so the next drop pages within minutes.
TDD red first: a 403-unknown followed by a next-day 200 must recover without manual action (impossible today).

Server: cairn-license-server companion issue — log every check-in with id/outcome/reason, persist last-refusal per licence.

Airgapped deployments are unaffected by the whole class: no check-in URL, no 403, no self-revocation.

## Incident (2026-09, cairn-enc) Nikola noticed the dogfood dropped from the 60-seat licence to the free tier; the licence page said **"licence revoked — running on the free tier"**. He recovered it by re-pasting the key (2026-09-03 23:40 — the successful check-in right after his account-page login at 23:37). ## Root cause (confirmed) - The licence itself was never revoked or expired: server DB shows `revoked=0`, key valid to 2027-07-29 (the sub is **yearly**; the monthly one belongs to bao). - The client's check-in received a **403** at some point in the window — almost certainly reason `unknown` during the licence server's account-era redeploys / JSON→SQLite migration (Aug 24–26; frequent restarts in the journal). - Client behaviour on 403 revoked/unknown: set `Revoked=true` → cap drops to 50 **immediately** → `checkinRequiredLocked` returns false → **check-ins stop permanently**. No self-heal is possible; only a manual Refresh/SetKey clears it. One transient server-side 403 = a silently bricked licence. ## Why it took days to notice, and could not be reconstructed - The server logs **no check-ins at all** (no successes, no 403s, no reasons); `lastSeen` is one overwritten value. - Client logs died with ~20 pod restarts (Pis have no persistent journald). - The licence page shows free-tier numbers with one terse line; no since-when, no what-happened, no notification. ## Build plan (approved by Nikola; UI parts mockup-first) **Client (this repo):** 1. **403 is no longer terminal on first sight.** A `revoked`/`unknown` refusal keeps the licence inactive (honest) but check-ins continue daily for a revoked-watch window (~14 days, mirroring the #308 expired-watch), so a transient server-side 403 self-heals at the next 200. A genuinely revoked licence gets two weeks of daily 403s, then goes quiet for good. 2. **The drop explains itself.** Persist reason + since-when; the Licensing page shows an inactive-state card (revoked / grace exhausted / expired variants — mockup first); an admin bell notification fires on the transition; loud log lines on every state change. 3. **Metrics + alert.** Export licensed-state/seat-cap gauges; homelab-config gains `CairnLicenseDropped` (key installed ∧ not licensed) so the next drop pages within minutes. 4. TDD red first: a 403-unknown followed by a next-day 200 must recover without manual action (impossible today). **Server:** cairn-license-server companion issue — log every check-in with id/outcome/reason, persist last-refusal per licence. Airgapped deployments are unaffected by the whole class: no check-in URL, no 403, no self-revocation.

Cordy referenced this issue

2026-09-06 02:04:24 +00:00

Licensing: "Remove licence" button — clean downgrade to the free tier (after #437's state cards) #438

Cordy referenced this issue from a commit

2026-09-06 02:14:08 +00:00

#437 red: a transient check-in refusal must self-heal, not revoke forever

Cordy referenced this issue from a commit

2026-09-06 02:19:40 +00:00

#437 red (phase B): transitions history + named inactive states

Cordy referenced this issue from a commit

2026-09-06 02:20:23 +00:00

#437 red (phase B): licence gauges on /metrics

Cordy referenced this issue from a commit

2026-09-06 02:23:31 +00:00

#437 one-shot: backend revoked-watch build (red witnessed, then green)

Cordy referenced this issue from a commit

2026-09-06 02:26:54 +00:00

#437: TestForbiddenRevokedStillStops -> TestForbiddenRevokedEntersWatch

forgejo-actions referenced this issue from a commit

2026-09-06 02:28:03 +00:00

#437: revoked is a watch, not a verdict - transient refusals self-heal

Cordy referenced this issue from a commit

2026-09-06 02:34:22 +00:00

#437 one-shot: frontend dropcard + dimmed meter + bell items + i18n ×4

Cordy referenced this issue from a commit

2026-09-06 02:36:41 +00:00

#437 one-shot fix: i18n block built from one-line tuples (YAML indent trap)

Cordy referenced this issue from a commit

2026-09-06 02:40:08 +00:00

#437 one-shot fix: renderLicDrop post-condition is 2 (def + call), not 3

forgejo-actions referenced this issue from a commit

2026-09-06 02:40:25 +00:00

#437 frontend: licence dropcard, dimmed meter, bell items, i18n x4

Cordy referenced this issue from a commit

2026-09-06 02:43:07 +00:00

#437 one-shot: CHANGELOG v0.6.133 + FAQ/ARCHITECTURE revoked-watch wording

forgejo-actions referenced this issue from a commit

2026-09-06 02:43:17 +00:00

#437 docs: CHANGELOG v0.6.133; revoked-watch wording in FAQ and ARCHITECTURE

Cordy referenced this issue from a pull request that will close it,

2026-09-06 02:44:11 +00:00

#437: revoked is a watch, not a verdict — transient refusals self-heal, and the drop explains itself #439

Cordy referenced this issue from a commit

2026-09-06 02:46:39 +00:00

Merge pull request '#437: revoked is a watch, not a verdict — transient refusals self-heal, and the drop explains itself' (#439) from license-drop-437 into main

Cordy closed this issue

2026-09-06 02:46:41 +00:00

Cordy commented

2026-09-06 02:55:38 +00:00

Author

Owner

Shipped in v0.6.133 (PR #439, merged; dogfoods live).

Decision record:

Revocation is a watch, not a verdict: a 403 revoked/unknown drops the licence to the free tier immediately (unchanged) but keeps daily check-ins alive for 14 days (revokedWatchWindow); the first 200 restores the licence with no admin action. Past the watch, check-ins stop for good — same bounded-phone-home shape as the expired watch (#308). Legacy state with the bare Revoked flag re-enters the watch on load.
Licence state persists a capped Transitions history (dropped/restored, reason revoked/grace/expired) — the forensic trail this incident lacked. The bell derives admin-only entries from it (one per state change, click lands on Licensing).
Licence page: one state-driven dropcard per the approved license-inactive-states mockup (refused / unreachable / expired, shared calm line, Check in now); seat meter dims and names the suspended cap. i18n ×4.
/metrics: cairn_license_installed / cairn_license_licensed / cairn_license_seat_cap — absent when unwired, so absent() keeps meaning. homelab-config now alerts CairnLicenseDropped (installed==1 and licensed==0, 5m, critical): the silent month this incident got is no longer possible.
Status API adds inactiveReason / inactiveSince / retryUntil / licSeats.

TDD: TestTransientRefusalSelfHeals replayed the incident red on the runner (server hits stayed 1); TestForbiddenRevokedStillStops — which asserted the incident-causing behaviour — became TestForbiddenRevokedEntersWatch. Full suite green.

Live verify: cairn-enc + cairn-openbao on v0.6.133; files.c0rdyceps.ch/metrics shows installed 1 / licensed 1 / seat_cap 60 (the 60-seat key counting again).

Follow-ups tracked separately: cairn-license-server#40 (log every check-in, persist refusal history, 503-not-403 when storage is unavailable — the server-side half of never letting a redeploy answer "unknown"), #438 (Remove licence button, mockup first).

Shipped in **v0.6.133** (PR #439, merged; dogfoods live). **Decision record:** - Revocation is a **watch, not a verdict**: a 403 revoked/unknown drops the licence to the free tier immediately (unchanged) but keeps daily check-ins alive for **14 days** (`revokedWatchWindow`); the first 200 restores the licence with no admin action. Past the watch, check-ins stop for good — same bounded-phone-home shape as the expired watch (#308). Legacy state with the bare `Revoked` flag re-enters the watch on load. - Licence state persists a capped **Transitions** history (`dropped`/`restored`, reason `revoked`/`grace`/`expired`) — the forensic trail this incident lacked. The bell derives admin-only entries from it (one per state change, click lands on Licensing). - Licence page: one state-driven dropcard per the approved license-inactive-states mockup (refused / unreachable / expired, shared calm line, Check in now); seat meter dims and names the suspended cap. i18n ×4. - `/metrics`: `cairn_license_installed` / `cairn_license_licensed` / `cairn_license_seat_cap` — absent when unwired, so `absent()` keeps meaning. homelab-config now alerts `CairnLicenseDropped` (installed==1 and licensed==0, 5m, critical): the silent month this incident got is no longer possible. - Status API adds `inactiveReason` / `inactiveSince` / `retryUntil` / `licSeats`. **TDD:** `TestTransientRefusalSelfHeals` replayed the incident red on the runner (server hits stayed 1); `TestForbiddenRevokedStillStops` — which asserted the incident-causing behaviour — became `TestForbiddenRevokedEntersWatch`. Full suite green. **Live verify:** cairn-enc + cairn-openbao on v0.6.133; `files.c0rdyceps.ch/metrics` shows installed 1 / licensed 1 / seat_cap 60 (the 60-seat key counting again). Follow-ups tracked separately: cairn-license-server#40 (log every check-in, persist refusal history, 503-not-403 when storage is unavailable — the server-side half of never letting a redeploy answer "unknown"), #438 (Remove licence button, mockup first).

Cordy referenced this issue from a commit

2026-09-06 03:21:34 +00:00

wiki-40 fix: dedupe guard keys on the new heading, not the #40 mention in the #437 entry

Cordy referenced this issue

2026-09-06 04:08:44 +00:00

#438: Remove licence — clean downgrade to the free tier #440

Cordy referenced this issue

2026-09-06 04:18:34 +00:00

Licensing: "Remove licence" button — clean downgrade to the free tier (after #437's state cards) #438

Cordy referenced this issue

2026-09-09 22:03:49 +00:00

Remove nav notification dots; federated/peering deliveries notify via the bell #460