Peering transport: gRPC control plane + HTTP data plane, inbound service #102

Closed
opened 2026-08-04 11:03:36 +00:00 by Cordy · 1 comment
Owner

Rewritten 2026-08-05 — the original text was written before the ADR closed and deferred to "#100's handshake". The design is now decided (IMPLEMENTED-PEERING.md); this issue names it so it is actionable on its own.

Prerequisite #101 is done (registry, admin API, Settings panel). This issue builds the receiving side plus the protocol both sides speak.


Two planes

Control plane — gRPC over TLS. Four RPCs, proto/cairn/peer/v1/peering.proto, generated code committed so customers never run protoc (CI regenerates and drift-checks):

  • Hello(key_id) → challenge — unauthenticated. Unknown key ids still get a well-formed challenge (no oracle).
  • Authenticate(key_id, challenge, proof) → sessionproof = HMAC-SHA256(peerKey, "cairn-peer-auth-v1" ‖ key_id ‖ challenge), constant-time compare, failure indistinguishable from unknown key. The key itself is never transmitted — this replaces the diagram's static hash(pKEY), which would be replayable once observed.
  • PrepareTransfer(session, transfer_id, recipient, file_name, size, content_hash, sender, age_recipient) → upload_url, capability, ttl — verifies the recipient exists locally, runs pre-flight (size vs. quota, concurrency slot), mints the capability.
  • CompleteTransfer(session, transfer_id) → status — verifies the plaintext content hash, triggers delivery (#104), acks.

Data plane — HTTP/2 over TLS. PUT /peer/v1/blob/{transfer_id} with Authorization: Cairn-Capability <token>, body = the age-encrypted stream. Bytes deliberately stay out of protobuf framing.

Serving: preferred single port — the existing HTTPS listener with a handler that routes application/grpc content-types to the gRPC server, everything else to the normal mux (ALPN gives h2 under TLS). Customers expose nothing new. Fallback if a proxy misbehaves: dedicated peering.listen port, documented, not default.

Work items

  1. proto + codegen — service definition, committed generated code, CI drift check. Additive-only evolution rules; protocol field gates behaviour changes.
  2. Key-id lookupX-Cairn-Peer/key_id carries a short id (first bytes of the verifier hash) so the receiver selects the right key without trial verification. internal/peering currently verifies by peer name (VerifyKey(peer, secret)); add id→peer resolution.
  3. Challenge store — short-TTL, single-use, bounded. The challenge is the anti-replay state; no separate nonce cache needed.
  4. Capability tokens — minted and signed by the receiver (HMAC with an instance key), carrying transfer id, accepted size, expiry, single-use. Verified without round-trip. Rationale: authorizes before the first byte instead of requiring an HMAC over a multi-gigabyte body; content integrity comes from content_hash + the age AEAD.
  5. Recipient verification — strictly post-auth, resolving via local users / Keycloak Admin API (spaces.Manager, #94/#96) / LDAP. Rate-limited per peer; every probe hit or miss audited (accepted enumeration exposure, #100 Q4, narrowed by the fact that only allow-listed senders on an authenticated peering can reach it).
  6. Inbound streaming — declared-size enforcement (cut the stream if exceeded), streaming writes only with bounded buffers and read deadlines (#5 OOM lesson applies verbatim to WAN uploads), filename validated as a single path segment (never a path), per-peer concurrency + rate caps, peering.maxFileBytes = min with storage quota.
  7. IdempotencyPrepareTransfer with a known transfer_id returns the existing capability rather than duplicating. No resume in v1 (retry whole file); the transfer id is the handle a later TUS-style resume would need.
  8. Distinct refusal codesUNKNOWN_RECIPIENT / QUOTA_EXCEEDED / TOO_LARGE / HASH_MISMATCH / REFUSED. Peers are admin-trusted and useless errors become support tickets.

Decisions already made — do not re-open here

Transport (option C), handshake (challenge–response + capabilities), directed key pairs, payload encrypted end-to-end with the strong-share pipeline (#19) inside TLS, sender travels as user@instance where the domain is the receiving admin's local alias and is a display hint only — never an authorization input.

Ships with

  • #118 must be actioned in the same PR: this is the commit that adds google.golang.org/grpc to go.mod, so the README dependency line and ARCHITECTURE §1.4 stop being true. Wording is drafted there and awaits Nikola's approval.
  • example.config.json + handbook deployment page gain the peering block.

Open

Whether the reachability "test peering" button (deferred from #101) lands here — there is finally something to test against.

**Rewritten 2026-08-05** — the original text was written before the ADR closed and deferred to "#100's handshake". The design is now decided (`IMPLEMENTED-PEERING.md`); this issue names it so it is actionable on its own. Prerequisite #101 is **done** (registry, admin API, Settings panel). This issue builds the receiving side plus the protocol both sides speak. --- ## Two planes **Control plane — gRPC over TLS.** Four RPCs, `proto/cairn/peer/v1/peering.proto`, generated code **committed** so customers never run `protoc` (CI regenerates and drift-checks): - `Hello(key_id) → challenge` — unauthenticated. Unknown key ids still get a well-formed challenge (no oracle). - `Authenticate(key_id, challenge, proof) → session` — `proof = HMAC-SHA256(peerKey, "cairn-peer-auth-v1" ‖ key_id ‖ challenge)`, constant-time compare, failure indistinguishable from unknown key. The key itself is **never transmitted** — this replaces the diagram's static `hash(pKEY)`, which would be replayable once observed. - `PrepareTransfer(session, transfer_id, recipient, file_name, size, content_hash, sender, age_recipient) → upload_url, capability, ttl` — verifies the recipient exists locally, runs pre-flight (size vs. quota, concurrency slot), mints the capability. - `CompleteTransfer(session, transfer_id) → status` — verifies the plaintext content hash, triggers delivery (#104), acks. **Data plane — HTTP/2 over TLS.** `PUT /peer/v1/blob/{transfer_id}` with `Authorization: Cairn-Capability <token>`, body = the age-encrypted stream. Bytes deliberately stay out of protobuf framing. **Serving:** preferred single port — the existing HTTPS listener with a handler that routes `application/grpc` content-types to the gRPC server, everything else to the normal mux (ALPN gives h2 under TLS). Customers expose nothing new. Fallback if a proxy misbehaves: dedicated `peering.listen` port, documented, not default. ## Work items 1. **proto + codegen** — service definition, committed generated code, CI drift check. Additive-only evolution rules; `protocol` field gates behaviour changes. 2. **Key-id lookup** — `X-Cairn-Peer`/`key_id` carries a short id (first bytes of the verifier hash) so the receiver selects the right key without trial verification. `internal/peering` currently verifies by peer *name* (`VerifyKey(peer, secret)`); add id→peer resolution. 3. **Challenge store** — short-TTL, single-use, bounded. The challenge *is* the anti-replay state; no separate nonce cache needed. 4. **Capability tokens** — minted and signed by the receiver (HMAC with an instance key), carrying transfer id, accepted size, expiry, single-use. Verified without round-trip. Rationale: authorizes *before* the first byte instead of requiring an HMAC over a multi-gigabyte body; content integrity comes from `content_hash` + the age AEAD. 5. **Recipient verification** — strictly post-auth, resolving via local users / Keycloak Admin API (`spaces.Manager`, #94/#96) / LDAP. Rate-limited per peer; every probe hit **or miss** audited (accepted enumeration exposure, #100 Q4, narrowed by the fact that only allow-listed senders on an authenticated peering can reach it). 6. **Inbound streaming** — declared-size enforcement (cut the stream if exceeded), streaming writes only with bounded buffers and read deadlines (#5 OOM lesson applies verbatim to WAN uploads), filename validated as a single path segment (never a path), per-peer concurrency + rate caps, `peering.maxFileBytes` = min with storage quota. 7. **Idempotency** — `PrepareTransfer` with a known `transfer_id` returns the existing capability rather than duplicating. No resume in v1 (retry whole file); the transfer id is the handle a later TUS-style resume would need. 8. **Distinct refusal codes** — `UNKNOWN_RECIPIENT` / `QUOTA_EXCEEDED` / `TOO_LARGE` / `HASH_MISMATCH` / `REFUSED`. Peers are admin-trusted and useless errors become support tickets. ## Decisions already made — do not re-open here Transport (option C), handshake (challenge–response + capabilities), directed key pairs, payload encrypted end-to-end with the strong-share pipeline (#19) inside TLS, `sender` travels as `user@instance` where the domain is the receiving admin's **local alias** and is a display hint only — never an authorization input. ## Ships with - **#118 must be actioned in the same PR**: this is the commit that adds `google.golang.org/grpc` to `go.mod`, so the README dependency line and ARCHITECTURE §1.4 stop being true. Wording is drafted there and awaits Nikola's approval. - `example.config.json` + handbook deployment page gain the `peering` block. ## Open Whether the reachability "test peering" button (deferred from #101) lands here — there is finally something to test against.
Cordy changed title from Peering wire protocol: handshake, recipient verification, inbound transfer service to Peering transport: gRPC control plane + HTTP data plane, inbound service 2026-08-04 23:54:29 +00:00
Cordy closed this issue 2026-08-05 19:33:07 +00:00
Author
Owner

Closed — implementation notes

Merged in PR #122, shipped v0.4.1. Files: internal/peering/{auth,inbound,service,blob,mux,originurl}.go plus peerpb/.

Control plane

proto/cairn/peer/v1/peering.proto, generated code committed so a customer build never runs protoc; protogen.yml regenerates it. Four RPCs on Service.

Two properties the tests pin down, both silent when broken:

  • Hello is not an oracle. An unknown key id gets a challenge of identical length and TTL to a known one, so probing reveals nothing about which peerings exist.
  • Authenticate fails identically for a bad proof and an unknown key. The test compares gRPC code and message string, and the implementation returns one shared package-level error value rather than constructing messages per call — so the two cannot drift apart in a later edit.

Challenges are single-use and spent whether or not the proof verifies, so a captured challenge cannot be brute-forced. That is also why no separate nonce cache exists: the challenge is the anti-replay state.

Capabilities

Minted once per transfer and stored on the record, because §6 requires a repeat PrepareTransfer to return the existing capability — they are single-use, so handing a retrying sender a second token would either race the first or find it spent.

The subject is base64(peer \0 transferID), binding the token to both. Transfer ids come from the sender, so two peers can pick the same one; the data plane recovers the peer from the signed token rather than trusting the URL.

Data plane

PUT /peer/v1/blob/{transfer_id}, Authorization: Cairn-Capability <token>.

Staging reads one byte past the authorization, so an over-long stream is detected rather than silently trimmed to the declared length — which would have staged a corrupt file that still looked the right size. Staged names are sha256(peer \0 id), because both arrive off the wire and neither may become a path component.

Note: the size semantics here were wrong until v0.4.2 — see the Fixed entry there. size is the plaintext size but the wire carries ciphertext, so the original exact-length cap would have cut every real transfer. It passed CI only because the tests uploaded plaintext.

Single port

Mux routes application/grpc* to the gRPC server and everything else to the normal mux. It requires HTTP/2 as well as the content type: gRPC does not exist over 1.1, and without the version check a crafted 1.1 request announcing a gRPC content type would reach a server that cannot speak it.

Also settled here

#118's dependency claims, since this is the commit that put gRPC in go.mod.

## Closed — implementation notes Merged in PR #122, shipped v0.4.1. Files: `internal/peering/{auth,inbound,service,blob,mux,originurl}.go` plus `peerpb/`. ### Control plane `proto/cairn/peer/v1/peering.proto`, generated code **committed** so a customer build never runs `protoc`; `protogen.yml` regenerates it. Four RPCs on `Service`. Two properties the tests pin down, both silent when broken: - **`Hello` is not an oracle.** An unknown key id gets a challenge of identical length and TTL to a known one, so probing reveals nothing about which peerings exist. - **`Authenticate` fails identically** for a bad proof and an unknown key. The test compares gRPC code *and* message string, and the implementation returns one shared package-level `error` value rather than constructing messages per call — so the two cannot drift apart in a later edit. Challenges are single-use and **spent whether or not the proof verifies**, so a captured challenge cannot be brute-forced. That is also why no separate nonce cache exists: the challenge *is* the anti-replay state. ### Capabilities Minted once per transfer and stored on the record, because §6 requires a repeat `PrepareTransfer` to return the *existing* capability — they are single-use, so handing a retrying sender a second token would either race the first or find it spent. The subject is `base64(peer \0 transferID)`, binding the token to **both**. Transfer ids come from the sender, so two peers can pick the same one; the data plane recovers the peer from the signed token rather than trusting the URL. ### Data plane `PUT /peer/v1/blob/{transfer_id}`, `Authorization: Cairn-Capability <token>`. Staging reads **one byte past** the authorization, so an over-long stream is *detected* rather than silently trimmed to the declared length — which would have staged a corrupt file that still looked the right size. Staged names are `sha256(peer \0 id)`, because both arrive off the wire and neither may become a path component. > Note: the size semantics here were wrong until v0.4.2 — see the Fixed entry there. `size` is the plaintext size but the wire carries ciphertext, so the original exact-length cap would have cut every real transfer. It passed CI only because the tests uploaded plaintext. ### Single port `Mux` routes `application/grpc*` to the gRPC server and everything else to the normal mux. It requires **HTTP/2 as well as** the content type: gRPC does not exist over 1.1, and without the version check a crafted 1.1 request announcing a gRPC content type would reach a server that cannot speak it. ### Also settled here #118's dependency claims, since this is the commit that put gRPC in `go.mod`.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#102
No description provided.