Peering dogfood: second instance on k3s + live A↔B e2e

Cordy commented

2026-08-04 11:04:52 +00:00

Owner

Peering cannot be dogfooded against itself — we need a second live instance. Deploy cairn-peer on k3s next to the existing enc dogfood (.249): MetalLB has exactly one address left in the pool (192.168.10.250, pool .240–.250), same Keycloak realm is fine (different users make the test more honest — e.g. only zeus321 on the peer), own Garage bucket (cairn-peer), own config secret. Manifests in homelab-config k3s/cairn/.

e2e (one-shot workflow interop-peering.yml, runner as LAN test host, like interop-rclone.yml):

Admin flow via API on both instances: generate key on B for A, register B on A (and reverse).
Send a file A→B to a user that exists on B → arrives in their Inbox, correct bytes, correct owner, encrypted to the B-recipient's key (pull the raw Garage object, prove it's an age file the sender's key can NOT open, recovery can).
Negative matrix: revoked key → refused; unknown recipient → refused; oversized declared size → refused pre-flight; replayed transfer id → idempotent.
Audit trails on both sides show the transfer.

Open questions:

Gateway API: Manuel's condition — "Wenn Kubernetes mit der Gateway API deployed ist könnts klappen." For LAN dogfood a plain MetalLB LoadBalancer suffices; Gateway API (Traefik's GRPCRoute/HTTPRoute) is the story for routing gRPC/h2 PUBLICLY. Do we make Gateway API part of the v0.5 acceptance (install + route the peering port through it on the homelab), or note it as deployment documentation and keep dogfood on LoadBalancer? Depends on #100's transport decision (GRPCRoute only matters if gRPC wins). ➤ Manuel to check "aktuellen Stand" as he said; k3s ships Traefik ≥3 with Gateway API support that must be explicitly enabled.
Public path: real-world peering crosses the internet. Our own public chain is VPS Caddy + WireGuard (CGNAT). gRPC needs end-to-end h2 — verify Caddy's reverse_proxy h2c/grpc config on the VPS, or decide peering rides a dedicated port. Ties into the external-exposure work (#22) — same TLS story files.c0rdyceps.ch needs anyway.
Resource budget: pi cluster runs everything — does cairn-peer get the same 500m/256Mi limits (and does a big transfer trip CPUThrottlingHigh like homelab-config #5)?
Second instance lifetime: keep permanently as the standing peer-test target, or tear down after each e2e run (recommend: keep — it doubles as the upgrade-canary).

Peering cannot be dogfooded against itself — we need a second live instance. Deploy **cairn-peer** on k3s next to the existing enc dogfood (.249): MetalLB has exactly one address left in the pool (192.168.10.250, pool .240–.250), same Keycloak realm is fine (different users make the test more honest — e.g. only `zeus321` on the peer), own Garage bucket (`cairn-peer`), own config secret. Manifests in homelab-config `k3s/cairn/`. **e2e (one-shot workflow `interop-peering.yml`, runner as LAN test host, like `interop-rclone.yml`):** 1. Admin flow via API on both instances: generate key on B for A, register B on A (and reverse). 2. Send a file A→B to a user that exists on B → arrives in their Inbox, correct bytes, correct owner, **encrypted to the B-recipient's key** (pull the raw Garage object, prove it's an age file the sender's key can NOT open, recovery can). 3. Negative matrix: revoked key → refused; unknown recipient → refused; oversized declared size → refused pre-flight; replayed transfer id → idempotent. 4. Audit trails on both sides show the transfer. **Open questions:** - **Gateway API:** Manuel's condition — "Wenn Kubernetes mit der Gateway API deployed ist könnts klappen." For LAN dogfood a plain MetalLB LoadBalancer suffices; Gateway API (Traefik's GRPCRoute/HTTPRoute) is the story for routing gRPC/h2 PUBLICLY. Do we make Gateway API part of the v0.5 acceptance (install + route the peering port through it on the homelab), or note it as deployment documentation and keep dogfood on LoadBalancer? Depends on #100's transport decision (GRPCRoute only matters if gRPC wins). ➤ Manuel to check "aktuellen Stand" as he said; k3s ships Traefik ≥3 with Gateway API support that must be explicitly enabled. - **Public path:** real-world peering crosses the internet. Our own public chain is VPS Caddy + WireGuard (CGNAT). gRPC needs end-to-end h2 — verify Caddy's `reverse_proxy` h2c/grpc config on the VPS, or decide peering rides a dedicated port. Ties into the external-exposure work (#22) — same TLS story `files.c0rdyceps.ch` needs anyway. - Resource budget: pi cluster runs everything — does cairn-peer get the same 500m/256Mi limits (and does a big transfer trip CPUThrottlingHigh like homelab-config #5)? - Second instance lifetime: keep permanently as the standing peer-test target, or tear down after each e2e run (recommend: keep — it doubles as the upgrade-canary).

Cordy added this to the v0.5 — Instance peering (federation v1) milestone

Rows
Columns

Peering dogfood: second instance on k3s + live A↔B e2e #106

First real A↔B transfer completed end to end (2026-08-15 dogfood)