Fix #5: bound + dedupe argon2id verifications (OOM under Finder WebDAV storms), hardening #6

Merged

Cordy merged 6 commits from fix/issue-5-oom-argon2 into main

2026-07-20 01:25:23 +00:00

Cordy commented

2026-07-20 01:13:25 +00:00

Owner

Diagnosis (evidence from the live pod, 2026-07-20)

lastState.terminated: OOMKilled, exit 137, restartCount 6; pod lived ~3m per life, dying right at Finder mount time
Previous log contains only the startup line — no panic, no graceful-shutdown line → kernel OOM kill, not probe kill or crash
No NFS errors in events; probe failures / rclone "connection refused" were downstream of the dead pod

Root cause: each argon2id verification allocates the full 64 MiB memory parameter (RFC 9106 defaults). Local.Authenticate had no concurrency bound, and Finder's mount opens many parallel connections all presenting the same Basic credentials before the SHA-256 verified cache is warm → N × 64 MiB simultaneous allocations vs the 256Mi cgroup limit → OOM. The limits in k3s/cairn/deployment.yaml were already at the suggested 64Mi/256Mi, so a limit bump alone would not have fixed it. The dummy-hash anti-enumeration path had the same problem, making unauthenticated request storms a remote OOM vector too.

Fix (stdlib-only, no new dependencies)

internal/auth: inflight-dedupe of identical concurrent (username, sha256(password)) verifications — the Finder storm collapses to one argon2 run — plus a process-wide semaphore capping distinct concurrent runs at 2 (~128 MiB worst case, safely inside 256Mi). Unknown-username requests take the same bounded path.
Regression tests: 16-way identical-credential storm (with dedupe-effectiveness timing check), wrong-password storm, unknown-user storm.

Hardening (fix-first doc Step 3)

cmd/cairnd: IdleTimeout: 120s (Finder holds many idle keep-alives). Deliberately no ReadTimeout/WriteTimeout — whole-request deadlines would kill large WebDAV transfers.
internal/obs: Recover middleware, mounted outermost — handler panics now log method/path/stack via slog and answer 500 instead of a stderr-only trace. Re-panics http.ErrAbortHandler.
internal/dav: first test batch for the package — Finder request shapes: PROPFIND variants (allprop/named/empty bodies, missing + infinity Depth), LOCK/UNLOCK sequence, zero-byte AppleDouble ._* PUTs, Range GET, 32-way parallel PROPFIND, MOVE/COPY Overwrite semantics.

No manifest change needed

With argon2 memory bounded, the existing 64Mi request / 256Mi limit is comfortable.

After merge (manual, per manual-guide)

Rebuild + push multi-arch image as v0.0.2 (buildx with the buildkitd.toml registry config), bump the tag in k3s/cairn/deployment.yaml via homelab-config, then rerun the Step 4 interop tests with kubectl get pods -n cairn -w in a second terminal.

Closes #5 (pending retest).

## Diagnosis (evidence from the live pod, 2026-07-20) - `lastState.terminated`: **OOMKilled**, exit 137, restartCount 6; pod lived ~3m per life, dying right at Finder mount time - Previous log contains only the startup line — no panic, no graceful-shutdown line → kernel OOM kill, not probe kill or crash - No NFS errors in events; probe failures / rclone "connection refused" were downstream of the dead pod **Root cause:** each argon2id verification allocates the full 64 MiB memory parameter (RFC 9106 defaults). `Local.Authenticate` had no concurrency bound, and Finder's mount opens many parallel connections all presenting the same Basic credentials *before* the SHA-256 verified cache is warm → N × 64 MiB simultaneous allocations vs the 256Mi cgroup limit → OOM. The limits in `k3s/cairn/deployment.yaml` were already at the suggested 64Mi/256Mi, so a limit bump alone would not have fixed it. The dummy-hash anti-enumeration path had the same problem, making unauthenticated request storms a remote OOM vector too. ## Fix (stdlib-only, no new dependencies) - `internal/auth`: inflight-dedupe of identical concurrent (username, sha256(password)) verifications — the Finder storm collapses to one argon2 run — plus a process-wide semaphore capping distinct concurrent runs at 2 (~128 MiB worst case, safely inside 256Mi). Unknown-username requests take the same bounded path. - Regression tests: 16-way identical-credential storm (with dedupe-effectiveness timing check), wrong-password storm, unknown-user storm. ## Hardening (fix-first doc Step 3) - `cmd/cairnd`: `IdleTimeout: 120s` (Finder holds many idle keep-alives). Deliberately **no** ReadTimeout/WriteTimeout — whole-request deadlines would kill large WebDAV transfers. - `internal/obs`: `Recover` middleware, mounted outermost — handler panics now log method/path/stack via slog and answer 500 instead of a stderr-only trace. Re-panics `http.ErrAbortHandler`. - `internal/dav`: first test batch for the package — Finder request shapes: PROPFIND variants (allprop/named/empty bodies, missing + infinity Depth), LOCK/UNLOCK sequence, zero-byte AppleDouble `._*` PUTs, Range GET, 32-way parallel PROPFIND, MOVE/COPY Overwrite semantics. ## No manifest change needed With argon2 memory bounded, the existing 64Mi request / 256Mi limit is comfortable. ## After merge (manual, per manual-guide) Rebuild + push multi-arch image as `v0.0.2` (buildx with the buildkitd.toml registry config), bump the tag in `k3s/cairn/deployment.yaml` via homelab-config, then rerun the Step 4 interop tests with `kubectl get pods -n cairn -w` in a second terminal. Closes #5 (pending retest).

Cordy added 6 commits

2026-07-20 01:13:26 +00:00

auth: bound and dedupe concurrent argon2id verifications (fixes #5 OOM) 2e46dcb357

Each argon2id verification allocates the full 64 MiB memory parameter.
Finder's WebDAV mount opens many parallel connections that all present
the same Basic credentials before the verified cache is warm; unbounded
concurrent verifications blew the pod's 256Mi cgroup limit -> OOMKilled.

- inflight map dedupes identical concurrent (username, sha256(password))
  verifications: the storm collapses to a single argon2 run
- process-wide semaphore caps distinct concurrent runs at 2 (~128 MiB
  worst case), also closing the unauthenticated-storm OOM vector via the
  dummy-hash anti-enumeration path, which now takes the same bounded path

auth: add credential-storm tests (issue #5 regression coverage) 954dadb688

obs: add outermost panic-recovery middleware (issue #5 hardening) a28a11acf8

A per-request handler panic previously only printed Go's default trace
to stderr (easy to miss) and killed the connection. Recover logs it
through slog with method/path/stack and answers 500.

obs: tests for panic-recovery middleware 3f7d152cb3

dav: add Finder-behavior test batch (issue #5 hardening) ae5559cc3f

First tests for this package. Covers the request shapes macOS Finder
actually sends: PROPFIND variants (allprop bodies, named props, missing
and infinity Depth), the LOCK/UNLOCK sequence, zero-byte AppleDouble
._* PUTs, Range GETs, rapid parallel PROPFINDs, and MOVE/COPY with
Overwrite semantics.

cairnd: add IdleTimeout and panic-recovery middleware (issue #5 hardening)

ci / test-and-build (pull_request) Successful in 7s

Details

fb393b57c3

IdleTimeout reaps idle keep-alive connections (Finder holds many open).
Deliberately no ReadTimeout/WriteTimeout: those are whole-request
deadlines and would kill large WebDAV transfers; ReadHeaderTimeout
already guards the slow-header case.