TLS Config Options for certificate handling #740

Closed
opened 2026-09-25 08:32:51 +00:00 by hyrsh · 3 comments
Collaborator

To address the use of private certificates which are commonly used by our target audience, we need to introduce configuration options to allow injection of this data to a fixed location for our application to consume.

Proposal:

We are introducing two directories. One for a general collection of certificate authorities and one to be used to start the TLS listener if the application needs to directly serve without being behind a proxy.

The fallback mechanism should always be "insecure" with a notification that the TLS handshake or listening could not be established because of not found certificates or invalid ones.

The TLS communication applies to at least four different scenarios:

  • the application serves directly via docker, raw binary or in a pod with a gateway/ingress controller with the passthrough flag enabled
  • peering with other instances
  • sending data via gRPC to other instances
  • authenticating users via LDAP, AD, oauth2 endpoint (with or without OIDC) since these services to not run without TLS

There may be more scenarios but this is what I can think of off the top of my head.

Config Flags and ENV variables:

To implement a standard I would like to make these options available via config flags or environment variables where the variables take precedence over the flags.

The possible flags and variables could be as follows:

  • --ca-dir, for the general CA directory
  • --tls-dir, for the server TLS directory
  • AF_CA_DIR, for the general CA directory
  • AF_TLS_DIR, for the server TLS directory

To make sure that the directories contain the correct data there are two requirements that have to be considered by the users:

  • the CA directory has to contain one certificate per file
  • the TLS directory has to contain exactly one tls.crt and one tls.key file

The language Go offers the possibility to load multiple files to a certificate pool with the crypto/tls function x509.NewCertPool().

Dummy function to load certificates

func createPubCertPool() credentials.TransportCredentials {
    ca_dir := "/default/path/to/cert"                    //this values gets set by AF_CA_DIR or "--ca-dir"
	certPool := x509.NewCertPool()                       //create new cert pool for all valid authority certificates
	caList := GetAllCertFiles(ca_dir)                    //get list of all certs in that directory
	for _, f := range caList {                           //loop over this list
		ca, caErr := ioutil.ReadFile(ca_dir + "/" + f)   //read each file as a whole
		if caErr != nil {
			log.Println(caErr)
		}
		if !certPool.AppendCertsFromPEM(ca) {            //append read file to pool
			log.Println("Could not add certificate "+f)
		}
		ca = nil                                         //reset ca byte array
	}

	config := &tls.Config{                               //set tls config with new root cas
		RootCAs: certPool,
	}
	return credentials.NewTLS(config)                    //return config for grpc dial
}

This dummy function presents loosely how a gRPC CA loading could look like.

For the TLS listener in Go there is just one line that has to be changed:

Dummy code line

http.ListenAndServeTLS(":443", "tls_dir"+"/"+"tls.crt", "tls_dir"+"/"+"tls.key", nil)

The "tls_dir" variable gets set by AF_TLS_DIR or "--tls-dir".

The mechanism for a fallback should always be as follows:

  • certificates tried to be loaded and verify the TLS server certificates (x509 check keypair)
  • if successful serve TLS and use TLS functions
  • if unsuccessful use non-TLS functions and create respective logs and notify the user on the dashboard
  • the notifications must be non-worrying to the user (e.g. "No certificates in CA dir - check if you need it", "No certificate found in TLS dir - cannot serve TLS")

There must be an option to disable the check for the TLS serving of the application, since it can be hosted behind a proxy that already terminates TLS (e.g. in Kubernetes).

To address the use of private certificates which are commonly used by our target audience, we need to introduce configuration options to allow injection of this data to a fixed location for our application to consume. Proposal: We are introducing two directories. One for a general collection of certificate authorities and one to be used to start the TLS listener if the application needs to directly serve without being behind a proxy. The fallback mechanism should always be "insecure" with a notification that the TLS handshake or listening could not be established because of not found certificates or invalid ones. The TLS communication applies to at least four different scenarios: - the application serves directly via docker, raw binary or in a pod with a gateway/ingress controller with the passthrough flag enabled - peering with other instances - sending data via gRPC to other instances - authenticating users via LDAP, AD, oauth2 endpoint (with or without OIDC) since these services to not run without TLS There may be more scenarios but this is what I can think of off the top of my head. Config Flags and ENV variables: To implement a standard I would like to make these options available via config flags or environment variables where the variables take precedence over the flags. The possible flags and variables could be as follows: - --ca-dir, for the general CA directory - --tls-dir, for the server TLS directory - AF_CA_DIR, for the general CA directory - AF_TLS_DIR, for the server TLS directory To make sure that the directories contain the correct data there are two requirements that have to be considered by the users: - the CA directory has to contain one certificate per file - the TLS directory has to contain exactly one tls.crt and one tls.key file The language Go offers the possibility to load multiple files to a certificate pool with the crypto/tls function x509.NewCertPool(). `Dummy function to load certificates` ``` func createPubCertPool() credentials.TransportCredentials { ca_dir := "/default/path/to/cert" //this values gets set by AF_CA_DIR or "--ca-dir" certPool := x509.NewCertPool() //create new cert pool for all valid authority certificates caList := GetAllCertFiles(ca_dir) //get list of all certs in that directory for _, f := range caList { //loop over this list ca, caErr := ioutil.ReadFile(ca_dir + "/" + f) //read each file as a whole if caErr != nil { log.Println(caErr) } if !certPool.AppendCertsFromPEM(ca) { //append read file to pool log.Println("Could not add certificate "+f) } ca = nil //reset ca byte array } config := &tls.Config{ //set tls config with new root cas RootCAs: certPool, } return credentials.NewTLS(config) //return config for grpc dial } ``` This dummy function presents loosely how a gRPC CA loading could look like. For the TLS listener in Go there is just one line that has to be changed: `Dummy code line` `http.ListenAndServeTLS(":443", "tls_dir"+"/"+"tls.crt", "tls_dir"+"/"+"tls.key", nil)` The "tls_dir" variable gets set by AF_TLS_DIR or "--tls-dir". The mechanism for a fallback should always be as follows: - certificates tried to be loaded and verify the TLS server certificates (x509 check keypair) - if successful serve TLS and use TLS functions - if unsuccessful use non-TLS functions and create respective logs and notify the user on the dashboard - the notifications must be non-worrying to the user (e.g. "No certificates in CA dir - check if you need it", "No certificate found in TLS dir - cannot serve TLS") There must be an option to disable the check for the TLS serving of the application, since it can be hosted behind a proxy that already terminates TLS (e.g. in Kubernetes).
Owner

Deep research (2026-10-24)

Full write-up: ~/cairn-work/scratch/b7e/backlog/research-740.md.

Today:

  • Outbound connections. All 14 outbound TLS clients use Go's system trust store:

    • OIDC, LDAP, Keycloak, OpenBao, S3;
    • Collabora, the converter, peering (gRPC and HTTP), OCM;
    • the update check, the licence server and the remote crypto role.

    All of them already refuse anything they cannot verify; the only insecure switch is ldapInsecureSkipVerify. A private CA can be added without code with SSL_CERT_DIR, which adds to the public roots, but nothing documents that. SSL_CERT_FILE would replace the public roots instead and break the licence and update checks.

  • Inbound connections. cairnd only serves plain HTTP. The remote crypto role sends file content and a bearer token over that link with no warning in the docs, which is the strongest reason for native TLS.

  • Bugs:

    • The repo Dockerfile is FROM scratch with no CA bundle, so docker-compose builds fail every outbound HTTPS call. The released image is distroless and has the bundle.
    • LDAP StartTLS lacks a server name; see #741.

The proposal's insecure fallback. OWASP ASVS 5.0 (12.3.1, 12.3.2 and 12.3.4), ASVS 4.0.3 (9.1.1 and 9.2.2) and NIST SP 800-52r2 all forbid falling back to unencrypted or unverified connections. Where others added such a switch, it became the standard forum fix and an attack surface: Mattermost EnableInsecureOutgoingConnections, oCIS OCIS_INSECURE, and Nextcloud LDAP threads. Keycloak refuses to start without TLS unless HTTP is enabled explicitly.

Recommendation:

  • Keep from the proposal: the two directories, renamed CAIRN_CA_DIR and CAIRN_TLS_DIR, and the cert-manager file names tls.crt and tls.key.
  • Trust directory: it adds to the public roots, serves every outbound client through one helper, accepts several certificates per file, and is re-read periodically.
  • Native TLS: it is optional and picks up renewed certificates without a restart. A configured but broken certificate refuses the start; a failed renewal keeps the last good certificate and warns. Peering works over TLS, with HSTS.
  • Settings: minVersion only.
  • Insecure fallback: none.

Decisions are being asked now.

## Deep research (2026-10-24) Full write-up: `~/cairn-work/scratch/b7e/backlog/research-740.md`. **Today:** - **Outbound connections.** All 14 outbound TLS clients use Go's system trust store: - OIDC, LDAP, Keycloak, OpenBao, S3; - Collabora, the converter, peering (gRPC and HTTP), OCM; - the update check, the licence server and the remote crypto role. All of them already refuse anything they cannot verify; the only insecure switch is `ldapInsecureSkipVerify`. A private CA can be added without code with `SSL_CERT_DIR`, which adds to the public roots, but nothing documents that. `SSL_CERT_FILE` would replace the public roots instead and break the licence and update checks. - **Inbound connections.** cairnd only serves plain HTTP. The remote crypto role sends file content and a bearer token over that link with no warning in the docs, which is the strongest reason for native TLS. - **Bugs:** - The repo `Dockerfile` is `FROM scratch` with no CA bundle, so docker-compose builds fail every outbound HTTPS call. The released image is distroless and has the bundle. - LDAP StartTLS lacks a server name; see #741. **The proposal's insecure fallback.** OWASP ASVS 5.0 (12.3.1, 12.3.2 and 12.3.4), ASVS 4.0.3 (9.1.1 and 9.2.2) and NIST SP 800-52r2 all forbid falling back to unencrypted or unverified connections. Where others added such a switch, it became the standard forum fix and an attack surface: Mattermost `EnableInsecureOutgoingConnections`, oCIS `OCIS_INSECURE`, and Nextcloud LDAP threads. Keycloak refuses to start without TLS unless HTTP is enabled explicitly. **Recommendation:** - **Keep from the proposal:** the two directories, renamed `CAIRN_CA_DIR` and `CAIRN_TLS_DIR`, and the cert-manager file names `tls.crt` and `tls.key`. - **Trust directory:** it adds to the public roots, serves every outbound client through one helper, accepts several certificates per file, and is re-read periodically. - **Native TLS:** it is optional and picks up renewed certificates without a restart. A configured but broken certificate refuses the start; a failed renewal keeps the last good certificate and warns. Peering works over TLS, with HSTS. - **Settings:** `minVersion` only. - **Insecure fallback:** none. Decisions are being asked now.
Owner

Decision (Nikola, 2026-10-24): fail closed. Every recommendation is accepted.

  • No insecure fallback. This is the one part of the original proposal that is changed, because OWASP ASVS and NIST SP 800-52 forbid it.
  • Trust directory. CAIRN_CA_DIR / tls.caDir adds to the public roots. It is used by every outbound client through one helper, accepts several certificates per file, and is re-read about every hour. It is the only place to add a CA: there are no per-integration CA paths.
  • Native TLS. CAIRN_TLS_DIR / tls.certDir is optional and holds tls.crt + tls.key (cert-manager names). Renewed certificates are picked up without a restart, the server sends HSTS, and peering works over TLS. If unset, Cairn serves plain HTTP behind a proxy, as today.
  • Broken certificates. A configured but broken certificate refuses the start. A failed renewal keeps the last good certificate and logs a WARN.
  • Settings. tls.minVersion (1.2 by default, or 1.3) is the only crypto setting. Names start with CAIRN_, and there are no command-line flags.
  • Mutual TLS is not part of this issue.
  • Dockerfile. The repo Dockerfile gets a CA bundle.
  • UI. The admin Certificates card follows in a mockup round, filed separately.

This is built in batch 5 (v0.6.352).

**Decision (Nikola, 2026-10-24): fail closed. Every recommendation is accepted.** - **No insecure fallback.** This is the one part of the original proposal that is changed, because OWASP ASVS and NIST SP 800-52 forbid it. - **Trust directory.** `CAIRN_CA_DIR` / `tls.caDir` adds to the public roots. It is used by every outbound client through one helper, accepts several certificates per file, and is re-read about every hour. It is the only place to add a CA: there are no per-integration CA paths. - **Native TLS.** `CAIRN_TLS_DIR` / `tls.certDir` is optional and holds `tls.crt` + `tls.key` (cert-manager names). Renewed certificates are picked up without a restart, the server sends HSTS, and peering works over TLS. If unset, Cairn serves plain HTTP behind a proxy, as today. - **Broken certificates.** A configured but broken certificate refuses the start. A failed renewal keeps the last good certificate and logs a WARN. - **Settings.** `tls.minVersion` (1.2 by default, or 1.3) is the only crypto setting. Names start with `CAIRN_`, and there are no command-line flags. - **Mutual TLS** is not part of this issue. - **Dockerfile.** The repo `Dockerfile` gets a CA bundle. - **UI.** The admin Certificates card follows in a mockup round, filed separately. This is built in batch 5 (v0.6.352).
Owner

Shipped in v0.6.352 (PR #986). Both dogfoods run it. Live check, 2026-10-24: PASS for what the dogfoods use.

  • Startup: both instances log tls: no tls.caDir set; outbound TLS trusts the system CA bundle only and listen with tls=false (behind the proxy, as before).
  • Outbound clients: every one the dogfoods use now goes through the shared trust helper, and all worked:
    • OIDC sign-in (discovery and token exchange) on both instances; Nikola signed in.
    • Keycloak-profile key custody on files: an encrypted write and read back.
    • The OpenBao custody self-check, and an encrypted write and read on files-bao.
    • S3/Garage.
    • The converter and Collabora.

Not used on the dogfoods, covered by tests:

  • A private CA in tls.caDir, including reload, additive roots, and each client trusting a private CA while refusing an unknown one.
  • Native TLS with tls.certDir: the refusals, renewal without restart, peering gRPC over TLS, and HSTS.
  • tls.minVersion, which applies to the listener only; outbound connections always require TLS 1.2 or later.
  • The crypto client never using a proxy.

Docs: docs/handbook/tls-certificates.md, plus the chart values trust.* and tls.*. The admin Certificates card is #985.

**Shipped in v0.6.352 (PR #986). Both dogfoods run it. Live check, 2026-10-24: PASS for what the dogfoods use.** - **Startup:** both instances log `tls: no tls.caDir set; outbound TLS trusts the system CA bundle only` and listen with `tls=false` (behind the proxy, as before). - **Outbound clients:** every one the dogfoods use now goes through the shared trust helper, and all worked: - OIDC sign-in (discovery and token exchange) on both instances; Nikola signed in. - Keycloak-profile key custody on files: an encrypted write and read back. - The OpenBao custody self-check, and an encrypted write and read on files-bao. - S3/Garage. - The converter and Collabora. **Not used on the dogfoods, covered by tests:** - A private CA in `tls.caDir`, including reload, additive roots, and each client trusting a private CA while refusing an unknown one. - Native TLS with `tls.certDir`: the refusals, renewal without restart, peering gRPC over TLS, and HSTS. - `tls.minVersion`, which applies to the listener only; outbound connections always require TLS 1.2 or later. - The crypto client never using a proxy. **Docs:** `docs/handbook/tls-certificates.md`, plus the chart values `trust.*` and `tls.*`. The admin Certificates card is #985.
Cordy closed this issue 2026-10-24 14:56:05 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#740
No description provided.