1 Check-in Logging and Refusal History
Cordy edited this page 2026-09-06 03:19:35 +00:00

Check-in logging and refusal history (#40)

Shipped 2026-09-06 (PR #41), deployed via deploy.yml. The server-side half of the 2026-08 licence-drop incident; the client half is Cairn #437 (v0.6.133, the 14-day revoked watch).

Why this exists

In August 2026 cairn-enc's healthy 60-seat licence dropped to the free tier and stayed there for days. Post-incident, the server could tell us nothing: the check-in handler logged no successes, no refusals, no reasons, and lastSeen was a single overwritten value. The most likely cause — a 403 with reason unknown answered while the server was mid-redeploy during the JSON→SQLite migration — could only be inferred, never proven. This change makes the next incident reconstructable from the server alone, and removes the most dangerous refusal entirely.

What is logged now

Every /v1/checkin outcome leaves a journal line:

level=INFO  msg="checkin ok"      id=lic-… org=… seats=… expires=… keyReturned=true
level=WARN  msg="checkin refused" id=lic-… reason=revoked|expired|unknown refusals=N
level=ERROR msg="checkin: store empty or unavailable - answering 503, not 403-unknown" id=lic-…
level=WARN  msg="checkin bad request" remote=…

Read them with:

sudo journalctl -u cairn-license-server --no-pager | grep checkin

Refusal history on the record

Every 403 for a known licence stamps the record, persisted in the SQLite data JSON:

field meaning
lastRefusedAt UTC time of the most recent refused check-in
lastRefusedReason revoked or expired (an unknown id has no record to stamp — the journal covers those)
refusalCount total refusals over the record's lifetime

The history is a trail, not state: a later successful check-in does not clear it. Read it on the VPS:

sudo sqlite3 /var/lib/cairn-license/licenses.db \
  "SELECT id,
          json_extract(data,'$.lastRefusedAt'),
          json_extract(data,'$.lastRefusedReason'),
          json_extract(data,'$.refusalCount'),
          json_extract(data,'$.lastSeen')
   FROM licenses;"

503, not 403-unknown, when the store cannot answer

A check-in that misses the in-memory records is only answered 403 unknown when the store is healthy (DB reachable and at least one licence loaded). An empty or unreachable store — the mid-migration position that refused the incident licence — answers 503 instead. Cairn clients treat 503 like connectivity failure: daily retry on the grace path, no self-revocation, fully self-healing. A production licence server holding zero licences is a migration accident, not a clean slate.

With real licences loaded, an unknown id remains an honest 403 reason=unknown (pinned by TestUnknownIDStillRefusedWhenStoreHealthy).

Interplay with the Cairn client (#437)

Even when a bad 403 does slip out, clients on ≥ v0.6.133 no longer treat it as terminal: revocation is a 14-day watch with daily check-ins, so one wrong answer heals at the next 200. Defence in depth: the server tries never to give the wrong answer (503 guard), and the client survives it if one gets through (revoked watch).

Incident playbook

  1. journalctl … | grep "checkin refused" — who was refused, when, why, how often.
  2. The SQL above — per-licence refusal history plus lastSeen, surviving restarts.
  3. On the instance side, the Cairn licence page names the state (refused / unreachable / expired, with dates), the admin bell keeps drop/restore transitions, and cairn_license_licensed==0 with installed==1 fires the CairnLicenseDropped alert in the homelab.