Start: an identity provider that is slow to answer stops Cairn (OIDC discovery is fatal after one 10 s try) #1137

Closed
opened 2026-11-06 12:13:33 +00:00 by Cordy · 3 comments
Owner

Found in the v0.6.400 live check (2026-11-06): both dogfood pods exited with code 1 on their first start, 10 s after boot, with level=ERROR msg=oidc err="oidc discovery: Get \"https://<issuer>/.well-known/openid-configuration\": context deadline exceeded", and came up on the kubelet's restart.

Code at 6289e07c: internal/auth/oidclib/oidclib.go:303-311 makes one discovery request with a 10 s client timeout; cmd/cairnd/main.go:1176 and :1184 call fatal on its error. So a short IdP or network hiccup at start is a crash, and outside Kubernetes (systemd without Restart=, a plain docker run) Cairn stays down.

To research before a fix (playbook Phase 1): how others handle an unreachable issuer at start (oauth2-proxy, Grafana, Nextcloud user_oidc, oCIS, Keycloak adapters), retry with backoff while not ready versus lazy discovery on first sign-in, what readiness should say meanwhile, and whether local and break-glass sign-in should work while the IdP is unreachable. Restarts also cost a start-lock take-over on S3 state since v0.6.401.

Found in the v0.6.400 live check (2026-11-06): both dogfood pods exited with code 1 on their first start, 10 s after boot, with `level=ERROR msg=oidc err="oidc discovery: Get \"https://<issuer>/.well-known/openid-configuration\": context deadline exceeded"`, and came up on the kubelet's restart. Code at 6289e07c: `internal/auth/oidclib/oidclib.go:303-311` makes one discovery request with a 10 s client timeout; `cmd/cairnd/main.go:1176` and `:1184` call `fatal` on its error. So a short IdP or network hiccup at start is a crash, and outside Kubernetes (systemd without `Restart=`, a plain `docker run`) Cairn stays down. To research before a fix (playbook Phase 1): how others handle an unreachable issuer at start (oauth2-proxy, Grafana, Nextcloud user_oidc, oCIS, Keycloak adapters), retry with backoff while not ready versus lazy discovery on first sign-in, what readiness should say meanwhile, and whether local and break-glass sign-in should work while the IdP is unreachable. Restarts also cost a start-lock take-over on S3 state since v0.6.401.
Author
Owner

Research done (2026-11-11, scratch/b7e/backlog/research-1137-936.md part A). Nextcloud user_oidc, oCIS, Gitea (since PR #16564) and Spring Security 6.5+ all start without the identity provider and retry. Stalwart fails at start and never retries; it is the warning case.

Recommended:

  • Definite configuration faults stay fatal.
  • Otherwise Cairn starts degraded and retries discovery in the background (backoff up to 60 s) and on use, rate-limited.
  • While the provider is down, the sign-in page shows a notice and the local and break-glass form.
  • /healthz stays independent of the identity provider; status shows on an Overview card and a metric.
  • security-standards.md:260 (break-glass without the provider) becomes true and is corrected.

Next: decisions.

**Research done** (2026-11-11, `scratch/b7e/backlog/research-1137-936.md` part A). Nextcloud user_oidc, oCIS, Gitea (since PR #16564) and Spring Security 6.5+ all start without the identity provider and retry. Stalwart fails at start and never retries; it is the warning case. **Recommended:** - Definite configuration faults stay fatal. - Otherwise Cairn starts degraded and retries discovery in the background (backoff up to 60 s) and on use, rate-limited. - While the provider is down, the sign-in page shows a notice and the local and break-glass form. - `/healthz` stays independent of the identity provider; status shows on an Overview card and a metric. - `security-standards.md:260` (break-glass without the provider) becomes true and is corrected. **Next:** decisions.
Author
Owner

Decisions (2026-11-12): approved as researched.

  • Definite configuration faults stay fatal.
  • Otherwise Cairn starts degraded and retries discovery in the background (backoff up to 60 s) and on use.
  • The sign-in page shows a notice plus the local and break-glass form.
  • /healthz stays independent of the identity provider; status shows on an Overview card and a metric.
  • TLS failures during discovery are retried and logged as ERROR.
  • No startup-mode switch.
  • security-standards.md:260 and tls-certificates.md:86 get corrected.

Next: mockups.

**Decisions (2026-11-12):** approved as researched. - Definite configuration faults stay fatal. - Otherwise Cairn starts degraded and retries discovery in the background (backoff up to 60 s) and on use. - The sign-in page shows a notice plus the local and break-glass form. - `/healthz` stays independent of the identity provider; status shows on an Overview card and a metric. - TLS failures during discovery are retried and logged as ERROR. - No startup-mode switch. - `security-standards.md:260` and `tls-certificates.md:86` get corrected. **Next:** mockups.
Author
Owner

Shipped in v0.6.407 (PR #1170), live on both dogfoods (v0.6.407, ready, 0 restarts).

  • An unreachable identity provider at start no longer stops Cairn: it starts degraded, retries discovery, and exits only when the IdP configuration is new or changed (fingerprint record of issuer, client and keys; follows client-key rotation).
  • Break-glass and local sign-in, app passwords, WebDAV and public links keep working; OIDC sign-in is refused until discovery succeeds.
  • Sign-in page notice with "Try single sign-on again", /auth/mode reports "idp":"unreachable" (no-store), Overview card (amber/red), rail dot, provider chips, paused/restored bells (one per outage), idp-unreachable/idp-reachable audit, cairn_oidc_* metrics.

Docs: users-access.md "When the identity provider does not answer", CHANGELOG.

Shipped in **v0.6.407** (PR #1170), live on both dogfoods (v0.6.407, ready, 0 restarts). - An unreachable identity provider at start no longer stops Cairn: it starts degraded, retries discovery, and exits only when the IdP configuration is new or changed (fingerprint record of issuer, client and keys; follows client-key rotation). - Break-glass and local sign-in, app passwords, WebDAV and public links keep working; OIDC sign-in is refused until discovery succeeds. - Sign-in page notice with "Try single sign-on again", `/auth/mode` reports `"idp":"unreachable"` (no-store), Overview card (amber/red), rail dot, provider chips, paused/restored bells (one per outage), `idp-unreachable`/`idp-reachable` audit, `cairn_oidc_*` metrics. Docs: users-access.md "When the identity provider does not answer", CHANGELOG.
Cordy closed this issue 2026-11-12 19:12:09 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1137
No description provided.