Dogfoods: Cairn sees cluster node addresses instead of the real client address (externalTrafficPolicy, trustedProxies) #864

Closed
opened 2026-10-08 18:19:20 +00:00 by Cordy · 8 comments
Owner

Found while verifying v0.6.323 (#841) on 2026-10-08.

Today

  • Both dogfood Services (cairn-enc at 192.168.10.249 and cairn-openbao at 192.168.10.247, in homelab-config k3s/cairn/*-dogfood.yaml) are type: LoadBalancer without externalTrafficPolicy, so Kubernetes uses the default, Cluster.
  • With Cluster, kube-proxy replaces the client's source address with the node's own. The pod logs show a LAN Mac as remote=10.42.4.0.
  • Cairn logs at start: "sign-in throttling uses the connection's address; behind a reverse proxy set auth.trustedProxies".

Effect on #841

  • Everyone who comes in through the same node shares one "address":
    • one person's wrong passwords slow down unfamiliar sign-ins for everyone else on that node;
    • spray detection (more than 10 accounts from one address in 15 minutes) can fire from ordinary use;
    • familiar sources (a success in the last 30 days) are effectively per node, not per device.
  • The same applies to the public share-link limiter and to the addresses in the audit log.

Options to discuss

  1. Set externalTrafficPolicy: Local on both Services, through GitOps. MetalLB L2 then announces the IP from the node that runs the pod, and the client address is kept.
    • Trade-off: only that node answers for the IP, and during a reschedule the IP moves with the pod (a short blip).
    • It is an SSA update of an existing defaulted field, so it is safe to apply through Argo.
  2. Public traffic through the VPS (Caddy over WireGuard) arrives from the VPS tunnel address. That address would need to go into auth.trustedProxies (the untracked cairn-enc-config Secret, or env CAIRN_TRUSTED_PROXIES), and Caddy must set X-Forwarded-For. Check whether the dogfoods are published through the VPS at all.
  3. Product side: should Cairn warn louder (an admin Overview hint) when it sees mostly cluster or private node addresses, or when XFF arrives from an untrusted peer? Today there is a one-time WARN in the log.
  4. Docs: deployment.md already covers trustedProxies. Add a Kubernetes section on externalTrafficPolicy: Local and the Helm chart default.

Nikola has follow-up questions; nothing is changed until those are answered.

Found while verifying v0.6.323 (#841) on 2026-10-08. ## Today - Both dogfood Services (`cairn-enc` at 192.168.10.249 and `cairn-openbao` at 192.168.10.247, in homelab-config `k3s/cairn/*-dogfood.yaml`) are `type: LoadBalancer` without `externalTrafficPolicy`, so Kubernetes uses the default, `Cluster`. - With `Cluster`, kube-proxy replaces the client's source address with the node's own. The pod logs show a LAN Mac as `remote=10.42.4.0`. - Cairn logs at start: "sign-in throttling uses the connection's address; behind a reverse proxy set auth.trustedProxies". ## Effect on #841 - Everyone who comes in through the same node shares one "address": - one person's wrong passwords slow down unfamiliar sign-ins for everyone else on that node; - spray detection (more than 10 accounts from one address in 15 minutes) can fire from ordinary use; - familiar sources (a success in the last 30 days) are effectively per node, not per device. - The same applies to the public share-link limiter and to the addresses in the audit log. ## Options to discuss 1. Set `externalTrafficPolicy: Local` on both Services, through GitOps. MetalLB L2 then announces the IP from the node that runs the pod, and the client address is kept. - Trade-off: only that node answers for the IP, and during a reschedule the IP moves with the pod (a short blip). - It is an SSA update of an existing defaulted field, so it is safe to apply through Argo. 2. Public traffic through the VPS (Caddy over WireGuard) arrives from the VPS tunnel address. That address would need to go into `auth.trustedProxies` (the untracked `cairn-enc-config` Secret, or env `CAIRN_TRUSTED_PROXIES`), and Caddy must set X-Forwarded-For. Check whether the dogfoods are published through the VPS at all. 3. Product side: should Cairn warn louder (an admin Overview hint) when it sees mostly cluster or private node addresses, or when XFF arrives from an untrusted peer? Today there is a one-time WARN in the log. 4. Docs: deployment.md already covers `trustedProxies`. Add a Kubernetes section on `externalTrafficPolicy: Local` and the Helm chart default. Nikola has follow-up questions; nothing is changed until those are answered.
Author
Owner

Decisions (Nikola, 2026-10-10): all four points agreed

  1. Dogfoods, LAN traffic: set externalTrafficPolicy: Local on both dogfood Services via GitOps (homelab-config).
  2. Dogfoods, public traffic through the VPS: add the VPS tunnel address to auth.trustedProxies in both config Secrets, and verify that Caddy sends X-Forwarded-For.
  3. Product: customers must not be able to miss this. When Cairn sees mostly cluster or private node addresses, or receives forwarding headers from a peer it doesn't trust, it shows a prominent admin warning: on the Overview, and in the admin bell, not only once in the log. The warning says what is wrong and links to the exact fix for the setup in use.
  4. Docs and defaults: deployment.md gets a section on getting the real client address, covering every deployment type. The Helm chart defaults to externalTrafficPolicy: Local.

Is this only a Kubernetes problem?

No. Kubernetes with the default Cluster policy is the most common case, but the same loss happens in other setups. These need verifying with sources in the research below:

  • Docker Desktop on Mac and Windows: every client looks like the VM gateway.
  • Rootless Docker and Podman, depending on the port driver.
  • Docker Swarm's ingress routing mesh: mode: ingress, unless mode: host is used.
  • The Docker userland proxy in some IPv6 or hairpin cases.
  • Any reverse proxy (Caddy, nginx, Traefik, a cloud load balancer, Cloudflare) unless trustedProxies is set.

Plain docker run / Compose on Linux with the default iptables NAT normally keeps the client address.

Next: deep research on similar cases, pitfalls and how other products detect and warn about this. Findings will be posted here before the spec.

## Decisions (Nikola, 2026-10-10): all four points agreed 1. **Dogfoods, LAN traffic:** set `externalTrafficPolicy: Local` on both dogfood Services via GitOps (homelab-config). 2. **Dogfoods, public traffic through the VPS:** add the VPS tunnel address to `auth.trustedProxies` in both config Secrets, and verify that Caddy sends `X-Forwarded-For`. 3. **Product:** customers must not be able to miss this. When Cairn sees mostly cluster or private node addresses, or receives forwarding headers from a peer it doesn't trust, it shows a prominent admin warning: on the Overview, and in the admin bell, not only once in the log. The warning says what is wrong and links to the exact fix for the setup in use. 4. **Docs and defaults:** deployment.md gets a section on getting the real client address, covering every deployment type. The Helm chart defaults to `externalTrafficPolicy: Local`. ### Is this only a Kubernetes problem? No. Kubernetes with the default `Cluster` policy is the most common case, but the same loss happens in other setups. These need verifying with sources in the research below: - Docker Desktop on Mac and Windows: every client looks like the VM gateway. - Rootless Docker and Podman, depending on the port driver. - Docker Swarm's ingress routing mesh: `mode: ingress`, unless `mode: host` is used. - The Docker userland proxy in some IPv6 or hairpin cases. - Any reverse proxy (Caddy, nginx, Traefik, a cloud load balancer, Cloudflare) unless `trustedProxies` is set. Plain `docker run` / Compose on Linux with the default iptables NAT normally keeps the client address. Next: deep research on similar cases, pitfalls and how other products detect and warn about this. Findings will be posted here before the spec.
Author
Owner

Deep research (2026-10-10)

The full report is in scratch at b7e/864-research.md (46 KB, a URL and a quote for every claim). It covers:

  • which setups lose the client address;
  • how 15 products handle and warn about it;
  • pitfalls, with CVEs and incidents;
  • detection heuristics;
  • an audit of Cairn's code.

Vital findings (these change what we build)

  1. Security: the share-link limiter trusts X-Forwarded-For from anyone, so its limit can be bypassed. X-Forwarded-Proto is trusted from anyone. Cookies are not marked Secure behind a TLS proxy. → #874, which ships first.
  2. k3s's own Traefik loses the address too. Its Service uses the Cluster policy, so Traefik sees 10.x.0.1. The docs must tell customers to set Traefik's Service to Local via HelmChartConfig, not only ours. k3s ServiceLB keeps the address only with Local and when node-external-ip is unset.
  3. Our own docs example is unsafe: deployment.md:405 trusts the whole pod network 10.42.0.0/16. Without a proxy in front, clients can then forge X-Forwarded-For. Fix the docs to trust exact proxy addresses. Cairn should warn at startup if the list contains the pod network, 0.0.0.0/0 or ::/0.
  4. Consistency gaps that remain after the config fix:
    • the request log records r.RemoteAddr;
    • audit events carry no client address at all, except the throttle and spray notes.
  5. Only Nextcloud and Home Assistant detect this at runtime:
    • Nextcloud's Overview check reads "Your remote address was identified as … is brute-force throttled".
    • Home Assistant rejects with 400 any X-Forwarded-For from an untrusted proxy.
    • Neither looks at the address mix over time. Cairn can do better: a "Cairn sees you as X" panel, plus a mix signal over a time window.
  6. Today's warning is too weak: one log line, only on a sign-in attempt, once per process. Detection moves into the HTTP middleware with a sliding window, and feeds the Overview and the admin bell.
  7. Our own spray detection can make a bad proxy setup worse: more than 10 accounts mistyped in 15 minutes from one shared proxy address triggers it. This is Nextcloud's most common support problem (about 20 threads), and GitLab, QNAP+fail2ban and CrowdSec show the same pattern. A spray or limit episode from a private address while no trusted proxy is configured should raise the misconfiguration warning.
  8. Heuristic without false alarms:
    • recognise gateway-style addresses: flannel .0/.1, Docker 172.17.0.1, Podman 10.0.2.100, Docker Desktop 192.168.65.x, loopback, Tailscale 100.64/10;
    • require at least 3 accounts, with 90% or more of traffic from at most 2 private addresses;
    • this spares single-user homes and offices behind one public NAT address.
  9. PROXY protocol must never be auto-detected (the HAProxy spec says so). The docs say to convert it to a header before it reaches Cairn.

Customer setup matrix (for the docs)

# Setup Loses the address? Fix
1 Compose + proxy container (Caddy, Traefik, NPM) yes Give the proxy a static IP; set CAIRN_TRUSTED_PROXIES to that IP
2 Binary + nginx/Caddy on the same host yes Trust 127.0.0.1, ::1; nginx sends XFF and Proto
3 Synology/QNAP reverse proxy yes Add an XFF custom header; trust the NAS/gateway; keep the NAS auto-block off Cairn
4 k3s default (Traefik + ServiceLB) yes, twice Set Traefik's Service to Local via HelmChartConfig; trust the Traefik pod IPs
5 k8s + MetalLB, Cairn LoadBalancer yes, with the Cluster policy Chart defaults to Local; trust nobody
6 k8s + MetalLB + ingress yes, unless the ingress Service is Local Ingress Service Local; trust only the ingress pods
7 VPS edge over WireGuard (our dogfood) yes Local on the target Service; trust the VPS WireGuard /32
8 Cloudflare proxied yes Trust the Cloudflare ranges (keep them updated); firewall the origin
9 Cloudflare Tunnel yes Trust the cloudflared address
10 Plain docker run -p on Linux, IPv4 no none (for IPv6: IPv6 on the bridge, or userland-proxy: false)
11 Rootless Docker yes Use the slirp4netns/pasta port driver, or a proxy
12 Rootless Podman depends Use pasta networking
13 Managed k8s with a cloud load balancer depends Service Local; for ALB, trust the ALB subnets
14 Docker Swarm yes mode: host ports
15 Docker Desktop yes Demo use only; the warning is expected
16 Tailscale Serve/Funnel yes Trust loopback
17 frp/ngrok-style tunnels yes http mode: trust the client; tcp mode: PROXY protocol into a proxy

For the spec

  • Order: #874 first. Then the shared client-address helper for every consumer, the detection with Overview and bell, and the startup warning for trust lists that are too broad. Then docs (matrix, k3s Traefik, PROXY protocol), the Helm default Local, and the dogfood GitOps changes.
  • Decision needed: whether audit events should record the client address. It is personal data under the nDSG/GDPR, with retention implications.
  • Mockup needed: for the Overview warning and the bell item.
## Deep research (2026-10-10) The full report is in scratch at `b7e/864-research.md` (46 KB, a URL and a quote for every claim). It covers: - which setups lose the client address; - how 15 products handle and warn about it; - pitfalls, with CVEs and incidents; - detection heuristics; - an audit of Cairn's code. ### Vital findings (these change what we build) 1. **Security:** the share-link limiter trusts X-Forwarded-For from anyone, so its limit can be bypassed. `X-Forwarded-Proto` is trusted from anyone. Cookies are not marked `Secure` behind a TLS proxy. → **#874**, which ships first. 2. **k3s's own Traefik loses the address too.** Its Service uses the Cluster policy, so Traefik sees `10.x.0.1`. The docs must tell customers to set Traefik's Service to `Local` via HelmChartConfig, not only ours. k3s ServiceLB keeps the address only with `Local` and when `node-external-ip` is unset. 3. **Our own docs example is unsafe:** deployment.md:405 trusts the whole pod network `10.42.0.0/16`. Without a proxy in front, clients can then forge X-Forwarded-For. Fix the docs to trust exact proxy addresses. Cairn should warn at startup if the list contains the pod network, `0.0.0.0/0` or `::/0`. 4. **Consistency gaps** that remain after the config fix: - the request log records `r.RemoteAddr`; - audit events carry no client address at all, except the throttle and spray notes. 5. **Only Nextcloud and Home Assistant detect this at runtime:** - Nextcloud's Overview check reads "Your remote address was identified as … is brute-force throttled". - Home Assistant rejects with 400 any X-Forwarded-For from an untrusted proxy. - Neither looks at the address mix over time. Cairn can do better: a "Cairn sees you as X" panel, plus a mix signal over a time window. 6. **Today's warning is too weak:** one log line, only on a sign-in attempt, once per process. Detection moves into the HTTP middleware with a sliding window, and feeds the Overview and the admin bell. 7. **Our own spray detection can make a bad proxy setup worse:** more than 10 accounts mistyped in 15 minutes from one shared proxy address triggers it. This is Nextcloud's most common support problem (about 20 threads), and GitLab, QNAP+fail2ban and CrowdSec show the same pattern. A spray or limit episode from a private address while no trusted proxy is configured should raise the misconfiguration warning. 8. **Heuristic without false alarms:** - recognise gateway-style addresses: flannel `.0/.1`, Docker `172.17.0.1`, Podman `10.0.2.100`, Docker Desktop `192.168.65.x`, loopback, Tailscale `100.64/10`; - require at least 3 accounts, with 90% or more of traffic from at most 2 private addresses; - this spares single-user homes and offices behind one public NAT address. 9. **PROXY protocol must never be auto-detected** (the HAProxy spec says so). The docs say to convert it to a header before it reaches Cairn. ### Customer setup matrix (for the docs) | # | Setup | Loses the address? | Fix | |---|---|---|---| | 1 | Compose + proxy container (Caddy, Traefik, NPM) | yes | Give the proxy a static IP; set `CAIRN_TRUSTED_PROXIES` to that IP | | 2 | Binary + nginx/Caddy on the same host | yes | Trust `127.0.0.1`, `::1`; nginx sends XFF and Proto | | 3 | Synology/QNAP reverse proxy | yes | Add an XFF custom header; trust the NAS/gateway; keep the NAS auto-block off Cairn | | 4 | k3s default (Traefik + ServiceLB) | yes, twice | Set Traefik's Service to `Local` via HelmChartConfig; trust the Traefik pod IPs | | 5 | k8s + MetalLB, Cairn LoadBalancer | yes, with the Cluster policy | Chart defaults to `Local`; trust nobody | | 6 | k8s + MetalLB + ingress | yes, unless the ingress Service is `Local` | Ingress Service `Local`; trust only the ingress pods | | 7 | VPS edge over WireGuard (our dogfood) | yes | `Local` on the target Service; trust the VPS WireGuard /32 | | 8 | Cloudflare proxied | yes | Trust the Cloudflare ranges (keep them updated); firewall the origin | | 9 | Cloudflare Tunnel | yes | Trust the cloudflared address | | 10 | Plain `docker run -p` on Linux, IPv4 | no | none (for IPv6: IPv6 on the bridge, or `userland-proxy: false`) | | 11 | Rootless Docker | yes | Use the slirp4netns/pasta port driver, or a proxy | | 12 | Rootless Podman | depends | Use pasta networking | | 13 | Managed k8s with a cloud load balancer | depends | Service `Local`; for ALB, trust the ALB subnets | | 14 | Docker Swarm | yes | `mode: host` ports | | 15 | Docker Desktop | yes | Demo use only; the warning is expected | | 16 | Tailscale Serve/Funnel | yes | Trust loopback | | 17 | frp/ngrok-style tunnels | yes | http mode: trust the client; tcp mode: PROXY protocol into a proxy | ### For the spec - **Order:** #874 first. Then the shared client-address helper for every consumer, the detection with Overview and bell, and the startup warning for trust lists that are too broad. Then docs (matrix, k3s Traefik, PROXY protocol), the Helm default `Local`, and the dogfood GitOps changes. - **Decision needed:** whether audit events should record the client address. It is personal data under the nDSG/GDPR, with retention implications. - **Mockup needed:** for the Overview warning and the bell item.
Author
Owner

Research: client address in audit entries, who is responsible (2026-10-11)

This is research, not legal advice. Have a lawyer review it before any of it goes to customers. The full report is in scratch at b7e/864-audit-ip-legal.md.

Verdict on "self-hosted, so it's the customer's obligation"

  • Mostly right. The customer (the employer) is the controller under the GDPR and the nDSG. A vendor with no access to the data is neither controller nor processor (EDPB Guidelines 07/2020, para. 40 and the IT-consultant example). The customer has to inform its employees.
  • Two catches.
    • This only holds while Cairn really has no access. Remote support access would make us a processor, which needs a DPA. Telemetry or license checks that carry personal data would make us controller for that data.
    • The CRA puts duties on the vendor, from 11 Dec 2027. Annex I(2)(l) requires security logging of relevant internal activity "with an opt-out mechanism for the user", plus data minimisation.
  • Employee-monitoring law creates vendor work in practice.
    • Germany: an audit log identifying individuals needs a works-council agreement (BetrVG §87(1)6), whether or not the IP is in it.
    • Switzerland: ArGV 3 Art. 26 and SECO require security logs to be proportionate, announced and justified.
    • Customers can only meet these duties if we document what is logged and why.
  • Competitors. Nextcloud, GitLab, Mattermost, Gitea, Keycloak, Vaultwarden/Bitwarden and Synapse all record client IPs. None truncates them. Privacy controls are retention settings: Synapse 28 days, Gitea 30, Seafile 90, GitLab keeps them forever. Only Nextcloud explicitly says the operator is the controller.

Further Swiss findings

  • DSV Art. 4 can make logging mandatory: logs must identify the person and be kept at least 1 year. Federal bodies must assess their logging in writing by 31 Dec 2026.
  • EDÖB/SECO: IT logs kept for security are fine if proportionate, announced in advance, kept briefly and not analysed per person as routine.

Recommendation

  1. Record the client address in audit entries by default. Resolve it through the trusted-proxy logic, so it is the real client.
  2. Add one setting, audit.ip: full | truncated | off, applied when the event is written. Truncated keeps IPv4 /24 and IPv6 /48. Keep the existing audit retention.
    • Apply it at write time because sealed audit files are hash-chained and never rewritten. Removing IPs later would break tamper-evidence.
    • Truncating at write time gives the same privacy result.
  3. A new handbook page, data-protection-operators.md, containing:
    • who is responsible, and that support access needs a DPA;
    • which events carry an address, and for what purpose;
    • Swiss checklist (ArGV 3 Art. 26, DSV Art. 4);
    • a DE/AT works-council outline;
    • an employee notice template in DE/FR/IT/EN, modelled on the EDÖB model policy;
    • a short DPIA snippet.
  4. CRA, before Dec 2027: document the audit log as our Annex I(2)(l) implementation and the IP setting as the minimisation measure. Decide on the opt-out question for "admin events are always recorded". Track this as its own issue.
## Research: client address in audit entries, who is responsible (2026-10-11) This is research, not legal advice. Have a lawyer review it before any of it goes to customers. The full report is in scratch at `b7e/864-audit-ip-legal.md`. ### Verdict on "self-hosted, so it's the customer's obligation" - **Mostly right.** The customer (the employer) is the controller under the GDPR and the nDSG. A vendor with no access to the data is neither controller nor processor (EDPB Guidelines 07/2020, para. 40 and the IT-consultant example). The customer has to inform its employees. - **Two catches.** - This only holds while Cairn really has no access. Remote support access would make us a processor, which needs a DPA. Telemetry or license checks that carry personal data would make us controller for that data. - The CRA puts duties on the vendor, from 11 Dec 2027. Annex I(2)(l) requires security logging of relevant internal activity "with an opt-out mechanism for the user", plus data minimisation. - **Employee-monitoring law creates vendor work in practice.** - Germany: an audit log identifying individuals needs a works-council agreement (BetrVG §87(1)6), whether or not the IP is in it. - Switzerland: ArGV 3 Art. 26 and SECO require security logs to be proportionate, announced and justified. - Customers can only meet these duties if we document what is logged and why. - **Competitors.** Nextcloud, GitLab, Mattermost, Gitea, Keycloak, Vaultwarden/Bitwarden and Synapse all record client IPs. None truncates them. Privacy controls are retention settings: Synapse 28 days, Gitea 30, Seafile 90, GitLab keeps them forever. Only Nextcloud explicitly says the operator is the controller. ### Further Swiss findings - **DSV Art. 4** can make logging *mandatory*: logs must identify the person and be kept at least 1 year. Federal bodies must assess their logging in writing by 31 Dec 2026. - **EDÖB/SECO:** IT logs kept for security are fine if proportionate, announced in advance, kept briefly and not analysed per person as routine. ### Recommendation 1. Record the client address in audit entries by default. Resolve it through the trusted-proxy logic, so it is the real client. 2. Add one setting, `audit.ip: full | truncated | off`, applied when the event is written. Truncated keeps IPv4 /24 and IPv6 /48. Keep the existing audit retention. - Apply it at write time because sealed audit files are hash-chained and never rewritten. Removing IPs later would break tamper-evidence. - Truncating at write time gives the same privacy result. 3. A new handbook page, **data-protection-operators.md**, containing: - who is responsible, and that support access needs a DPA; - which events carry an address, and for what purpose; - Swiss checklist (ArGV 3 Art. 26, DSV Art. 4); - a DE/AT works-council outline; - an employee notice template in DE/FR/IT/EN, modelled on the EDÖB model policy; - a short DPIA snippet. 4. **CRA, before Dec 2027:** document the audit log as our Annex I(2)(l) implementation and the IP setting as the minimisation measure. Decide on the opt-out question for "admin events are always recorded". Track this as its own issue.
Author
Owner

Decision (Nikola, 2026-10-12): the audit-address recommendation is accepted in full:

  • Audit entries record the client address by default, resolved through the trusted-proxy logic.
  • A new setting audit.ip: full | truncated | off is applied when an entry is written. Truncated keeps IPv4 /24 and IPv6 /48. The existing audit retention applies.
  • A new handbook page data-protection-operators.md: responsibilities, the support-access DPA note, which events carry an address and why, a Swiss checklist, a DE/AT works-council outline, an employee notice template in 4 languages, and a DPIA snippet.
  • The CRA vendor duties are parked as their own issue (milestone "parked until real demand"), with a deep-research reminder.

Order: #874 (security) ships first. After that, #864 and #865.

**Decision (Nikola, 2026-10-12):** the audit-address recommendation is accepted in full: - Audit entries record the client address by default, resolved through the trusted-proxy logic. - A new setting `audit.ip: full | truncated | off` is applied when an entry is written. Truncated keeps IPv4 /24 and IPv6 /48. The existing audit retention applies. - A new handbook page `data-protection-operators.md`: responsibilities, the support-access DPA note, which events carry an address and why, a Swiss checklist, a DE/AT works-council outline, an employee notice template in 4 languages, and a DPIA snippet. - The CRA vendor duties are parked as their own issue (milestone "parked until real demand"), with a deep-research reminder. Order: #874 (security) ships first. After that, #864 and #865.
Author
Owner

Mockups approved (Nikola, 2026-10-12): 864-client-address-warning-mockup.html (the card variants, the bell, the rules) and 864-overview-placement-mockup.html. The placement:

  1. The card goes first in the Overview attention strip, ahead of the update and share-review notices. It can't be dismissed and disappears once the problem is fixed.
  2. A permanent "Client addresses" row in Access, sharing and federation, with a Check, OK or Unsafe chip.
  3. An admin bell item that can't be muted.
  4. An amber dot on Overview in the admin rail.
  5. Red card for a trust list that is too broad, shown ahead of the amber one.

Order: #865 first, then this issue.

**Mockups approved (Nikola, 2026-10-12)**: `864-client-address-warning-mockup.html` (the card variants, the bell, the rules) and `864-overview-placement-mockup.html`. The placement: 1. The card goes first in the Overview attention strip, ahead of the update and share-review notices. It can't be dismissed and disappears once the problem is fixed. 2. A permanent "Client addresses" row in Access, sharing and federation, with a Check, OK or Unsafe chip. 3. An admin bell item that can't be muted. 4. An amber dot on Overview in the admin rail. 5. Red card for a trust list that is too broad, shown ahead of the amber one. Order: #865 first, then this issue.
Cordy closed this issue 2026-10-12 16:09:40 +00:00
Author
Owner

v0.6.331 (PR #881) is shipped and both dogfoods are bumped.

First live signal (cairn-openbao, 2026-10-12 16:15 UTC). I sent about 25 requests through the VPS Caddy (files-bao.c0rdyceps.ch). The detector switched to check/cluster and logged:

WARN client address: Kubernetes hides the client address; set externalTrafficPolicy: Local on Cairn's Service (on k3s also on Traefik's). Do not trust the node address state=check cause=cluster

The request log shows remote=10.42.4.0, the flannel address of the node that received the traffic. This is exactly the case we expected.

Next steps, together with Nikola:

  1. Nikola signs in, and we check the Overview card, the Client addresses row, the rail dot and the bell item.
  2. GitOps: add externalTrafficPolicy: Local to both dogfood Services (homelab-config k3s/cairn/*-dogfood.yaml).
  3. Once the real proxy address appears on the card (proxy-untrusted), Nikola adds it to auth.trustedProxies in both config Secrets, which are not in GitOps.
  4. Confirm that the row shows OK, with "Cairn sees you as via trusted proxy …".
v0.6.331 (PR #881) is shipped and both dogfoods are bumped. **First live signal (cairn-openbao, 2026-10-12 16:15 UTC).** I sent about 25 requests through the VPS Caddy (files-bao.c0rdyceps.ch). The detector switched to `check/cluster` and logged: `WARN client address: Kubernetes hides the client address; set externalTrafficPolicy: Local on Cairn's Service (on k3s also on Traefik's). Do not trust the node address state=check cause=cluster` The request log shows `remote=10.42.4.0`, the flannel address of the node that received the traffic. This is exactly the case we expected. **Next steps, together with Nikola:** 1. Nikola signs in, and we check the Overview card, the Client addresses row, the rail dot and the bell item. 2. GitOps: add `externalTrafficPolicy: Local` to both dogfood Services (`homelab-config k3s/cairn/*-dogfood.yaml`). 3. Once the real proxy address appears on the card (`proxy-untrusted`), Nikola adds it to `auth.trustedProxies` in both config Secrets, which are not in GitOps. 4. Confirm that the row shows OK, with "Cairn sees you as <your public address> via trusted proxy …".
Author
Owner

Live walkthrough on files-bao (Nikola + Claude, 2026-10-12)

  • On v0.6.331, in the "cluster" state, every part showed as designed:
    • the card in the attention strip, first in line, with the copyable externalTrafficPolicy: Local;
    • the row reading "Cairn sees you as 10.42.4.0 · see the warning above" with a Check chip;
    • the amber dot on Overview;
    • the bell item "Cairn can't tell your users apart".
  • GitOps (homelab-config e00415f): both Services set to externalTrafficPolicy: Local. Requests through the VPS then arrived from 15.25.10.1, the WireGuard tunnel, instead of 10.42.4.0.
  • Two design flaws found live, fixed in v0.6.332 (PR #882):
    • R9: old cluster evidence kept the card on "cluster" for up to 24 h after the fix. Newer evidence now replaces it after about 20 requests.
    • R10: a single-user instance could never pass the 3-hop gate, so it never got the proxy card. An authenticated request through the peer now confirms it.
    • Review additions: R11 (probes are ignored) and R12 (an ingress pod replaces the old evidence).
  • Docs drift fixed in the same PR: sharing.md (upload links, password limit, local accounts can be disabled too) and licensing-faq.md (disabled accounts hold no seat).

Homelab note: the WireGuard subnet 15.25.10.0/24 is public address space (part of 15.0.0.0/8), not RFC 1918. The card will therefore mark the proxy as public and add the "check that this is your reverse proxy" note. That is expected. Moving the tunnel to a private range would be a separate homelab task.

Next: Nikola signs in again (the redeploy cleared sessions) and browses through files-bao. The card should switch to "Your reverse proxy isn't trusted yet" and name 15.25.10.1. Then that address goes into the trusted-proxy setting of both dogfoods, and the row should turn OK.

## Live walkthrough on files-bao (Nikola + Claude, 2026-10-12) - **On v0.6.331, in the "cluster" state, every part showed as designed:** - the card in the attention strip, first in line, with the copyable `externalTrafficPolicy: Local`; - the row reading "Cairn sees you as 10.42.4.0 · see the warning above" with a Check chip; - the amber dot on Overview; - the bell item "Cairn can't tell your users apart". - **GitOps (homelab-config e00415f):** both Services set to `externalTrafficPolicy: Local`. Requests through the VPS then arrived from `15.25.10.1`, the WireGuard tunnel, instead of `10.42.4.0`. - **Two design flaws found live, fixed in v0.6.332 (PR #882):** - **R9:** old cluster evidence kept the card on "cluster" for up to 24 h after the fix. Newer evidence now replaces it after about 20 requests. - **R10:** a single-user instance could never pass the 3-hop gate, so it never got the proxy card. An authenticated request through the peer now confirms it. - **Review additions:** R11 (probes are ignored) and R12 (an ingress pod replaces the old evidence). - **Docs drift fixed in the same PR:** sharing.md (upload links, password limit, local accounts can be disabled too) and licensing-faq.md (disabled accounts hold no seat). **Homelab note:** the WireGuard subnet `15.25.10.0/24` is public address space (part of 15.0.0.0/8), not RFC 1918. The card will therefore mark the proxy as public and add the "check that this is your reverse proxy" note. That is expected. Moving the tunnel to a private range would be a separate homelab task. **Next:** Nikola signs in again (the redeploy cleared sessions) and browses through files-bao. The card should switch to "Your reverse proxy isn't trusted yet" and name `15.25.10.1`. Then that address goes into the trusted-proxy setting of both dogfoods, and the row should turn OK.
Author
Owner

Healthy on the dogfoods (2026-10-12)

  • homelab-config 87e4937 sets CAIRN_TRUSTED_PROXIES=15.25.10.1 on both dogfoods, so they trust the VPS Caddy tunnel hop.
  • No Caddy change was needed. The VPS runs Caddy v2.11.3 with a plain reverse_proxy and no trusted_proxies. By default it sends X-Forwarded-For with the real visitor, plus X-Forwarded-Proto: https, and it discards any address a client supplies.
  • Request log: remote=15.25.10.3 peer=15.25.10.1. 15.25.10.3 is Nikola's Mac, which tunnels all its traffic over WireGuard, so that is its real address as Caddy sees it.
  • API: state: ok, you.viaProxy: true. Over 24 h there are 2 accounts from 2 addresses (15.25.10.3, and 85.0.22.213 for the phone).
  • Overview: no card and no rail dot. The row reads "Cairn sees you as 15.25.10.3 via trusted proxy 15.25.10.1 · 2 accounts from 2 addresses in 24 h", with an OK chip.

The full path ran end to end: "cluster", then Local (GitOps), then "proxy not trusted" naming the exact address, then trust (GitOps), then OK.

## Healthy on the dogfoods (2026-10-12) - homelab-config `87e4937` sets `CAIRN_TRUSTED_PROXIES=15.25.10.1` on both dogfoods, so they trust the VPS Caddy tunnel hop. - **No Caddy change was needed.** The VPS runs Caddy v2.11.3 with a plain `reverse_proxy` and no `trusted_proxies`. By default it sends `X-Forwarded-For` with the real visitor, plus `X-Forwarded-Proto: https`, and it discards any address a client supplies. - **Request log:** `remote=15.25.10.3 peer=15.25.10.1`. 15.25.10.3 is Nikola's Mac, which tunnels all its traffic over WireGuard, so that is its real address as Caddy sees it. - **API:** `state: ok`, `you.viaProxy: true`. Over 24 h there are 2 accounts from 2 addresses (`15.25.10.3`, and `85.0.22.213` for the phone). - **Overview:** no card and no rail dot. The row reads "Cairn sees you as 15.25.10.3 via trusted proxy 15.25.10.1 · 2 accounts from 2 addresses in 24 h", with an **OK** chip. The full path ran end to end: "cluster", then `Local` (GitOps), then "proxy not trusted" naming the exact address, then trust (GitOps), then OK.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#864
No description provided.