Initial Kubernetes deployment #752

Closed
opened 2026-09-25 19:33:41 +00:00 by hyrsh · 3 comments
Collaborator

The documentation lacks a description of how to deploy to Kubernetes for the first time.

If the "config.json" gets pre-generated (copied from example.config.json) and filled with a local user it is read-only and cannot be changed by the application anymore (except there is a k8s API connection with the client library that is allowed to update/create/replace/delete Secrets in the deployed namespace).

There should be an example of all available configs and how to alter them including their location from the applications point of view.

Given this an administrator could pre-create these files and mount them via Secrets/ConfigMaps to the container.

Initial file generation within the Pod is possible but only uncomfortably reachable by someone who really knows how containers and their storage work. Common administrators expect:

  • what configs are there?
  • what do I need to or can change?
  • which ones are crucial?
  • which ones are sensitive?

Static Credentials (e.g. local superuser/admin) for the Deployment should be sourced from Secrets and respective fields into ENV variables since this is also future-proof for injecting secrets with ExternalSecretsOperator (ESO) and an external Vault (like OpenBao). Furthermore it does work with other deployment options (raw binary on a host or docker).

The documentation lacks a description of how to deploy to Kubernetes for the first time. If the "config.json" gets pre-generated (copied from example.config.json) and filled with a local user it is read-only and cannot be changed by the application anymore (except there is a k8s API connection with the client library that is allowed to update/create/replace/delete Secrets in the deployed namespace). There should be an example of all available configs and how to alter them including their location from the applications point of view. Given this an administrator could pre-create these files and mount them via Secrets/ConfigMaps to the container. Initial file generation within the Pod is possible but only uncomfortably reachable by someone who really knows how containers and their storage work. Common administrators expect: - what configs are there? - what do I need to or can change? - which ones are crucial? - which ones are sensitive? Static Credentials (e.g. local superuser/admin) for the Deployment should be sourced from Secrets and respective fields into ENV variables since this is also future-proof for injecting secrets with ExternalSecretsOperator (ESO) and an external Vault (like OpenBao). Furthermore it does work with other deployment options (raw binary on a host or docker).
Owner

Deep research (2026-10-24)

Full write-up: ~/cairn-work/scratch/b7e/backlog/research-752.md. It includes a complete inventory of every config key and CAIRN_* variable, with default, required, sensitive, and where it lives.

The core worry is already handled by the code. Cairn never writes its config file. Runtime changes go to the state stores, not to the config:

  • the first-run admin;
  • local accounts;
  • admin grants;
  • settings;
  • a licence entered in the UI.

The state stores live under <root>/.cairn on posix, or encrypted under .cairn-state/ in the bucket on S3. A read-only config.json mounted from a Secret is therefore the intended model. Env overrides the file, and config is read once at start.

What is missing:

  • Docs: there is no Kubernetes first-deployment page and no settings reference. README and deployment.md point at the Go structs. Nine env vars are undocumented, and example.config.json is prose, so it cannot boot.
  • Chart: it has no env, envFrom or extraVolumes, so the ESO/OpenBao pattern is impossible without a fork. Also:
    • it fails the restricted Pod Security Standard;
    • it has no /tmp volume and no startupProbe;
    • the service-account token is mounted although Cairn never calls the Kubernetes API;
    • appVersion v0.6.131 is the default image tag, and the default registry is a LAN IP.
  • Chart bug: the converter pods carry Cairn's selector labels, so the main Service and the ServiceMonitor also select Gotenberg.
  • Code:
    • There is no *_FILE secret convention, which Vault/OpenBao Agent and CSI need.
    • CAIRN_TUS_EXPIRY is documented but never read.
    • Malformed CAIRN_QUOTA_* values are silently ignored.
    • Misspelled config keys are silently ignored.

What broke elsewhere:

  • Nextcloud and Gitea write their config at runtime, which breaks read-only mounts.
  • Mattermost writes env values back to its config file.
  • Vaultwarden's admin-generated config overrides env.
  • Grafana's _FILE support depends on its entrypoint script.
  • Env secrets never update in a running pod.

Decisions are being asked now.

## Deep research (2026-10-24) Full write-up: `~/cairn-work/scratch/b7e/backlog/research-752.md`. It includes a complete inventory of every config key and `CAIRN_*` variable, with default, required, sensitive, and where it lives. **The core worry is already handled by the code.** Cairn never writes its config file. Runtime changes go to the state stores, not to the config: - the first-run admin; - local accounts; - admin grants; - settings; - a licence entered in the UI. The state stores live under `<root>/.cairn` on posix, or encrypted under `.cairn-state/` in the bucket on S3. A read-only `config.json` mounted from a Secret is therefore the intended model. Env overrides the file, and config is read once at start. **What is missing:** - **Docs:** there is no Kubernetes first-deployment page and no settings reference. README and deployment.md point at the Go structs. Nine env vars are undocumented, and `example.config.json` is prose, so it cannot boot. - **Chart:** it has no `env`, `envFrom` or `extraVolumes`, so the ESO/OpenBao pattern is impossible without a fork. Also: - it fails the restricted Pod Security Standard; - it has no `/tmp` volume and no startupProbe; - the service-account token is mounted although Cairn never calls the Kubernetes API; - `appVersion` v0.6.131 is the default image tag, and the default registry is a LAN IP. - **Chart bug:** the converter pods carry Cairn's selector labels, so the main Service and the ServiceMonitor also select Gotenberg. - **Code:** - There is no `*_FILE` secret convention, which Vault/OpenBao Agent and CSI need. - `CAIRN_TUS_EXPIRY` is documented but never read. - Malformed `CAIRN_QUOTA_*` values are silently ignored. - Misspelled config keys are silently ignored. **What broke elsewhere:** - Nextcloud and Gitea write their config at runtime, which breaks read-only mounts. - Mattermost writes env values back to its config file. - Vaultwarden's admin-generated config overrides env. - Grafana's `_FILE` support depends on its entrypoint script. - Env secrets never update in a running pod. Decisions are being asked now.
Owner

Decision (Nikola, 2026-10-24): file + env + _FILE. Every recommendation is accepted.

Settings and secrets:

  • The non-secret config.json comes from a ConfigMap or Secret. Secrets come from a Secret as env, and every secret variable CAIRN_X also accepts CAIRN_X_FILE.
  • A misspelled config key refuses the start and names the key.
  • The admin set by env stays the break-glass account and is re-applied on every start.
  • Cairn never writes Kubernetes Secrets.
  • Nothing is reloaded live: a change takes a restart. Reloader and the checksum annotation are supported.

Chart:

  • Hardening is on by default (restricted Pod Security Standard, /tmp volume, startupProbe, no service-account token).
  • New values: env, envFrom, extra volumes, and trust and TLS inputs (#740).
  • The converter selector bug is fixed.
  • The image tag follows the release.
  • It stays at 1 replica with no PDB until #742.

Docs:

  • deploy-kubernetes.md and configuration-reference.md, with a CI test that fails when a CAIRN_* variable is missing from the reference.
  • example.config.json will boot as shipped.
  • CAIRN_TUS_EXPIRY is wired up, and malformed CAIRN_QUOTA_* values refuse the start.

This is built in batch 5 (v0.6.352).

**Decision (Nikola, 2026-10-24): file + env + `_FILE`. Every recommendation is accepted.** **Settings and secrets:** - The non-secret `config.json` comes from a ConfigMap or Secret. Secrets come from a Secret as env, and every secret variable `CAIRN_X` also accepts `CAIRN_X_FILE`. - A misspelled config key refuses the start and names the key. - The admin set by env stays the break-glass account and is re-applied on every start. - Cairn never writes Kubernetes Secrets. - Nothing is reloaded live: a change takes a restart. Reloader and the checksum annotation are supported. **Chart:** - Hardening is on by default (restricted Pod Security Standard, `/tmp` volume, startupProbe, no service-account token). - New values: `env`, `envFrom`, extra volumes, and trust and TLS inputs (#740). - The converter selector bug is fixed. - The image tag follows the release. - It stays at 1 replica with no PDB until #742. **Docs:** - `deploy-kubernetes.md` and `configuration-reference.md`, with a CI test that fails when a `CAIRN_*` variable is missing from the reference. - `example.config.json` will boot as shipped. - `CAIRN_TUS_EXPIRY` is wired up, and malformed `CAIRN_QUOTA_*` values refuse the start. This is built in batch 5 (v0.6.352).
Owner

Shipped in v0.6.352 (PR #986). Both dogfoods run it. Live check, 2026-10-24: PASS.

  • Config check: both real dogfood configs passed the new strict key check, so neither has an unknown key, and both started cleanly. Their existing env secrets (Secret-injected) work unchanged.
  • Helm chart 0.5.0: CI ran the real helm lint and helm template over all optional paths, and they pass. Covered:
    • the hardened security context;
    • env, envFrom and extra volumes;
    • trust and TLS inputs, with HTTPS probes;
    • the converter selector fix, with the cairn Deployment selector unchanged, so no deletion is needed on upgrade;
    • image.repository required;
    • no PodDisruptionBudget.
  • Example config: example.config.json boots as shipped (tested).

Not used on the dogfoods (they use their own manifests, not the chart), covered by tests:

  • the _FILE secrets;
  • the chart itself on a cluster.

Docs: docs/handbook/deploy-kubernetes.md and docs/handbook/configuration-reference.md. CI fails if a CAIRN_* variable or config key is missing from the reference.

**Shipped in v0.6.352 (PR #986). Both dogfoods run it. Live check, 2026-10-24: PASS.** - **Config check:** both real dogfood configs passed the new strict key check, so neither has an unknown key, and both started cleanly. Their existing env secrets (Secret-injected) work unchanged. - **Helm chart 0.5.0:** CI ran the real `helm lint` and `helm template` over all optional paths, and they pass. Covered: - the hardened security context; - `env`, `envFrom` and extra volumes; - trust and TLS inputs, with HTTPS probes; - the converter selector fix, with the cairn Deployment selector unchanged, so no deletion is needed on upgrade; - `image.repository` required; - no PodDisruptionBudget. - **Example config:** `example.config.json` boots as shipped (tested). **Not used on the dogfoods (they use their own manifests, not the chart), covered by tests:** - the `_FILE` secrets; - the chart itself on a cluster. **Docs:** `docs/handbook/deploy-kubernetes.md` and `docs/handbook/configuration-reference.md`. CI fails if a `CAIRN_*` variable or config key is missing from the reference.
Cordy closed this issue 2026-10-24 14:56:06 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#752
No description provided.