Peering ADR: transport, handshake, threat model (gates the v0.5 track) #100
Labels
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Cordy/Cairn#100
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Foundation issue for the peering milestone (Manuel, 2026-07-26 + Peering-Idea.drawio). Deliverable:
PEERING.mddecision record + ARCHITECTURE.md section (and §10 milestone realignment: v0.5 = peering, old v0.5 renumbered to v0.6). Nothing else in the milestone starts before the transport and handshake decisions here are settled with Manuel.v1 scope (from the conversation): one file, one direction, A→B. Every instance stays standalone; peering is an explicit admin-established relationship. "Weitere Funktionen" ride the same channel later — the protocol must leave room, not implement them.
Decision 1 — transport: gRPC vs HTTP/2 streaming. Manuel proposes gRPC. Honest inventory: Cairn has ZERO gRPC/protobuf today — the crypto role (#33) does streaming over plain HTTP and it's proven. Adding
google.golang.org/grpc+ protobuf is the single largest dependency the codebase would take, against the tiny-trust-surface goal (§1.4).Decision 2 — handshake strength. The diagram sends
hash(pKEY-target)+hash(pKEY-source)per message. A static hash is bearer-equivalent: anyone who observes it once can replay it forever. Options:Decision 3 — pairing model. Diagram shows each side generating one key that the OTHER side presents (two directed keys per peering). Confirm this asymmetric-pair model vs a single shared pairing secret. The per-peer key already bounds compromise blast radius nicely — keep that property.
Threat model to write down: user enumeration via the recipient check (see wire-protocol issue), replay, oversized-stream DoS (concurrency caps — remember the #5 OOM lesson), hostile peer after key theft (revocation = delete from registry, instant), filename-based path traversal on receive, and what a compromised peer can and cannot learn (it must learn nothing beyond the addressed recipient's existence).
Open questions (beyond the decisions above):
recipient: ID (group, user))? Groups pull in space-delivery semantics — recommend deferring to v1.1.Draft ADR is on main:
PEERING.md. Six recommendations, seven open questions, ready for Manuel's review. Summary of what it argues and where it pushes back:Positioning — one correction that matters. "Nobody does that shit right now" doesn't survive a fact check: Open Cloud Mesh has existed since ~2015, is implemented by Nextcloud, ownCloud, OpenCloud, CERNBox and Seafile, and reached the IETF standards track in March 2026 (
draft-ietf-ocm-open-cloud-mesh-04). We should not make a falsifiable novelty claim publicly. What IS defensible and genuinely differentiated is the trust model: OCM federates identities and discovers peers; ours is two admins exchanging a key out of band, no discovery, no identity federation, no central broker, and actual bytes moving rather than share notifications. The doc has a comparison table that doubles as the marketing copy. An OCM adapter later would be a strong migration story and nothing here forecloses it (Q7).Transport — I recommend plain HTTP over TLS, not gRPC, for v1. The reasoning in full is in §4; the short version: (1) gRPC-Go + protobuf + codegen would be by far the largest dependency ever added, landing in the network-facing path, against a marketed four-module discipline; (2) for bulk streaming gRPC's advantages invert — length-prefixed protobuf messages, 4 MiB default chunking, double flow control, extra copies, all on 500m-CPU ARM nodes; the industry pattern is control-plane RPC + data-plane HTTP for exactly this reason; (3) gRPC needs end-to-end HTTP/2 through every customer's proxy chain, ours included (Caddy + WireGuard + CGNAT), and fails opaquely, whereas plain HTTP degrades to 1.1 chunked and reproduces with
curl. Manuel is right thatGRPCRouteis GA and mature in 2026 — the point is it becomes a requirement we impose on every customer rather than a capability we need. Where gRPC genuinely wins (schema evolution, typed multi-method surface) is acknowledged, and the escape hatch is explicit: if a real control plane emerges, add gRPC for control and keep bytes on HTTP. Q1 asks Manuel whether gRPC is load-bearing for the vision or was shorthand for "streaming".Handshake — the diagram's literal design needs one change. A static
hash(pKEY)sent every request is bearer-equivalent: observed once, replayable forever, and TLS is terminated by proxies in exactly our customers' environments. Recommendation: HMAC-SHA256 request signing — key never on the wire, timestamp + nonce for replay protection, signature binds the metadata so a captured signature can't be re-pointed at another recipient, stateless verification, no extra round trip. Same idea as the diagram, done so it survives observation.Pairing — keep the diagram exactly. Directed key pairs have a property worth not "simplifying" away: compromising the issuing side yields only verifiers, so a stolen key can impersonate a peer only to the instance that trusted it. A shared secret would make every compromise symmetric.
Payload — reuse the strong-share pipeline (#19). Encrypt end to end inside TLS with a per-transfer ephemeral age recipient, so a TLS-terminating proxy at either end never sees content, and the receiver re-encrypts to the recipient's own key on delivery. Same code path we just shipped and tested.
Net effect: if these hold, v0.5 adds zero dependencies and is a genuinely small build — registry + admin UI, one signed streaming endpoint, a send pipeline, delivery through the existing stack, a dropdown, and the second dogfood instance.
Manuel: Q1 (transport) and Q4 (username enumeration) are the two I'd most like your call on. Nikola: Q5 (paid tier?), Q6 (who may send, and may read-only space members send off-instance?) and Q7 (OCM adapter on the roadmap?).
DECIDED 2026-08-05 (Nikola). Full design:
IMPLEMENTED-PEERING.md.PEERING.mdis kept, marked superseded, so the proposal and the decision can be compared later.Decisions
PrepareRequest; decide before #102 ships.Two draft arguments were contested and are recorded as partly conceded (§0.1 of the implemented design):
Consequences to track: this is the project's first gRPC dependency and the largest ever added. Mitigations in the design: control messages only (four RPCs), generated code committed so customers need no
protoc, data plane independent and replaceable. Documentation debt to settle when it ships — the README's dependency line and ARCHITECTURE §1.4 must record the exception honestly rather than drift, same discipline as the typage bundle.#101 gains the sender allow-list; #102 gains the proto + both planes; #106 now genuinely needs
GRPCRoute. Build order unchanged: #101 → #102 → #103/#104 → #105 → #106.Unblocked — ready to start #101 on your word.
Closed — what was actually built against these decisions
IMPLEMENTED-PEERING.mdis the decision record;PEERING.mdstays as the superseded proposal so the two can be compared. Shipped in v0.4.1 and v0.4.2.proto/cairn/peer/v1/peering.proto; OCM adapter deferred to #117internal/peering/service.go) beside an HTTP/2 data plane (blob.go), sharing one port viamux.goauth.go) then capability tokensregistry.go: we store the peer's public half plus a lookup id, never a shared secretclient.goencrypts to the recipient the receiver minted;deliver.godecrypts and re-encrypts through the storage stackPrepareRequest.recipientis a usernameuser@instanceSendRequest.Sender, treated as a display hint and never an authorization inputCanSend/CanReceive, both fail closedTwo decisions in this ADR were wrong, and implementing them is what proved it
Both are recorded in §0.1 rather than quietly patched:
HMAC-SHA256(peerKey, …)while §4.1 required the receiver to store only a verifier. Those cannot both hold — verifying an HMAC needs the shared secret, so the receiver would have had to keep peer keys in the clear, destroying the exact property directed key pairs exist for. Corrected to Ed25519 signatures.age_recipientinPrepareRequest, travelling sender → receiver. The sender cannot know a recipient the receiver has not minted yet, and a receiver holding only a public key could never have decrypted — #104 would have been unbuildable. Moved toPrepareResponse; field 8 isreserved.Worth noting the pattern for future ADRs: both errors were internally consistent as prose and only surfaced when something tried to execute them. Writing the test first caught each one before any code depended on it.
One decision added after the fact
Peering is post-quantum unconditionally (§5, summary row E) — hybrid ML-KEM768 + X25519 per-transfer recipients, not following the storage
postQuantumsetting. The recipient is minted per transfer and discarded, so there is no data at rest to migrate, and every sender is a Cairn instance where hybrid support predates peering. Wireprotocol = 1now means hybrid.