Cordy/Cairn
2
1
Fork 0
Code Issues 15 Pull requests Projects Releases Packages Wiki Activity Actions

Pod restarts under Finder WebDAV traffic (interop #2 blocker) #5

New issue
Closed
opened 2026-07-20 00:59:32 +00:00 by Cordy · 2 comments
Cordy commented 2026-07-20 00:59:32 +00:00
Owner
Copy link

Observed 2026-07-20 during first dogfood deployment (k3s, 192.168.10.247):

  • macOS Finder connected to /dav via ⌘K: repeated disconnects, eventually browsed files and created one folder, then stalled with Finder error 100060 (timeout).
  • The cairn pod restarted repeatedly during the Finder session — this is the primary bug; the client-side errors are downstream.
  • rclone test afterwards: dial tcp 192.168.10.247:80: connect: connection refused — service down while pod was in CrashLoopBackOff, so no independent signal.

Not yet collected (first step next session):

  • kubectl describe pod -n cairn <pod> → Last State reason (OOMKilled? Error + exit code? Liveness probe failure events?)
  • kubectl logs -n cairn deploy/cairn --previous → panic stack traces / http: panic serving lines vs clean SIGTERM shutdown log (a "shutting down / reason=signal" line means k8s killed us via probe, not a crash)
  • Deployment resource limits + liveness probe spec from k3s/cairn/deployment.yaml in homelab-config

Ranked hypotheses:

  1. Liveness probe kills under load — logs --previous ending in a clean graceful-shutdown line + probe-failure events in describe would confirm. Fix: probe tuning via GitOps.
  2. OOMKilled — Finder opens many parallel connections; if the memory limit is small (Pi cluster), plausible. Fix: raise limits via GitOps.
  3. Genuine panic in internal/dav triggered by a Finder request pattern (PROPFIND variants, LOCK, ._* AppleDouble PUTs, Range) — net/http recovers per-connection panics without killing the process, so this alone shouldn't restart the pod, but stack traces in previous logs would make it a straight code fix + regression test.
  4. NFS stall (TrueNAS cairn-data share) blocking handlers → probe timeout → restart.

Hardening to do regardless of root cause: http.Server currently sets only ReadHeaderTimeout; add sane ReadTimeout/WriteTimeout/IdleTimeout appropriate for large WebDAV transfers, and consider logging recovered handler panics via a recovery middleware so they surface in slog instead of stderr noise.

**Observed 2026-07-20 during first dogfood deployment (k3s, 192.168.10.247):** - macOS Finder connected to `/dav` via ⌘K: repeated disconnects, eventually browsed files and created one folder, then stalled with Finder error 100060 (timeout). - **The cairn pod restarted repeatedly during the Finder session** — this is the primary bug; the client-side errors are downstream. - rclone test afterwards: `dial tcp 192.168.10.247:80: connect: connection refused` — service down while pod was in CrashLoopBackOff, so no independent signal. **Not yet collected (first step next session):** - `kubectl describe pod -n cairn <pod>` → Last State reason (OOMKilled? Error + exit code? Liveness probe failure events?) - `kubectl logs -n cairn deploy/cairn --previous` → panic stack traces / `http: panic serving` lines vs clean SIGTERM shutdown log (a "shutting down / reason=signal" line means k8s killed us via probe, not a crash) - Deployment resource limits + liveness probe spec from `k3s/cairn/deployment.yaml` in homelab-config **Ranked hypotheses:** 1. **Liveness probe kills under load** — logs `--previous` ending in a clean graceful-shutdown line + probe-failure events in describe would confirm. Fix: probe tuning via GitOps. 2. **OOMKilled** — Finder opens many parallel connections; if the memory limit is small (Pi cluster), plausible. Fix: raise limits via GitOps. 3. **Genuine panic in `internal/dav`** triggered by a Finder request pattern (PROPFIND variants, LOCK, `._*` AppleDouble PUTs, Range) — `net/http` recovers per-connection panics without killing the process, so this alone shouldn't restart the pod, but stack traces in previous logs would make it a straight code fix + regression test. 4. **NFS stall** (TrueNAS cairn-data share) blocking handlers → probe timeout → restart. **Hardening to do regardless of root cause:** `http.Server` currently sets only `ReadHeaderTimeout`; add sane `ReadTimeout`/`WriteTimeout`/`IdleTimeout` appropriate for large WebDAV transfers, and consider logging recovered handler panics via a recovery middleware so they surface in slog instead of stderr noise.
Cordy commented 2026-07-20 01:14:53 +00:00
Author
Owner
Copy link

Diagnosed — root cause found and fixed in PR #6 (CI green).

Evidence (live pod, 2026-07-20, via read-only k8s access):

  • lastState.terminated: OOMKilled, exit 137, restartCount 6, each life ~3 min, dying at Finder mount time
  • Previous-container log contains only the startup line — abrupt kernel OOM kill, no panic, no graceful shutdown, no probe-kill
  • No NFS/mount errors in events. Hypothesis-table row 1 (OOM), but with a twist: limits were already 64Mi/256Mi, so the suggested limit raise was not the fix.

Root cause: each argon2id verification allocates its full 64 MiB memory parameter (m=65536 per RFC 9106). Local.Authenticate ran verifications unbounded, and Finder's mount storm presents identical Basic credentials on many parallel connections before the verified-credentials cache is warm → N×64 MiB concurrent allocations → cgroup OOM. The dummy-hash anti-enumeration path made even unauthenticated storms an OOM vector.

Fix (PR #6, stdlib-only): inflight dedupe of identical concurrent verifications (storm collapses to one argon2 run) + semaphore capping distinct concurrent runs at 2 (~128 MiB worst case). Plus Step-3 hardening: IdleTimeout: 120s, outermost slog panic-recovery middleware, and a first internal/dav test batch covering Finder request shapes. Regression tests included (16-way credential storm etc.).

No manifest change needed — existing 256Mi limit is now comfortable.

Remaining to close: merge #6, rebuild/push image v0.0.2, bump tag in homelab-config k3s/cairn/deployment.yaml, rerun Finder/rclone/Cyberduck interop with kubectl get pods -n cairn -w correlating.

**Diagnosed — root cause found and fixed in PR #6 (CI green).** Evidence (live pod, 2026-07-20, via read-only k8s access): - `lastState.terminated: OOMKilled`, exit 137, restartCount 6, each life ~3 min, dying at Finder mount time - Previous-container log contains **only** the startup line — abrupt kernel OOM kill, no panic, no graceful shutdown, no probe-kill - No NFS/mount errors in events. Hypothesis-table row 1 (OOM), but with a twist: limits were **already** 64Mi/256Mi, so the suggested limit raise was not the fix. Root cause: each argon2id verification allocates its full 64 MiB memory parameter (`m=65536` per RFC 9106). `Local.Authenticate` ran verifications unbounded, and Finder's mount storm presents identical Basic credentials on many parallel connections before the verified-credentials cache is warm → N×64 MiB concurrent allocations → cgroup OOM. The dummy-hash anti-enumeration path made even unauthenticated storms an OOM vector. Fix (PR #6, stdlib-only): inflight dedupe of identical concurrent verifications (storm collapses to one argon2 run) + semaphore capping distinct concurrent runs at 2 (~128 MiB worst case). Plus Step-3 hardening: `IdleTimeout: 120s`, outermost slog panic-recovery middleware, and a first `internal/dav` test batch covering Finder request shapes. Regression tests included (16-way credential storm etc.). No manifest change needed — existing 256Mi limit is now comfortable. Remaining to close: merge #6, rebuild/push image `v0.0.2`, bump tag in homelab-config `k3s/cairn/deployment.yaml`, rerun Finder/rclone/Cyberduck interop with `kubectl get pods -n cairn -w` correlating.
Cordy closed this issue 2026-07-20 01:25:25 +00:00
Cordy commented 2026-07-20 01:44:00 +00:00
Author
Owner
Copy link

Retest passed — closing.

v0.0.2 deployed via the new tag-triggered release workflow (ko, built and pushed from the Forgejo runner — no more manual buildx) and rolled out through GitOps. Finder interop retest against the live instance:

  • ⌘K mount of http://192.168.10.247/dav: connected first try, no disconnects
  • Browse (depth-1 PROPFINDs incl. AppleDouble ._* probes), MKCOL + rename (retest-v002), file copy (LOCK/PUT/GET/UNLOCK round-trip, 8.7KB rtf) — all clean in the request log, 0–11ms
  • Pod: Running, 0 restarts throughout — same traffic pattern that OOM-killed v0.0.1 six times
  • Memory under load: 132Mi of the 256Mi limit (bounded argon2 + Go runtime — exactly as predicted for the sem=2 cap)

Remaining interop (rclone, Cyberduck) can ride along whenever; the OOM class is fixed. Follow-ups tracked separately: CairnDown alert rule, Argo webhook for homelab-config didn't fire during the rollout (sync happened via the 3-min poll — worth checking the webhook config).

**Retest passed — closing.** v0.0.2 deployed via the new tag-triggered release workflow (ko, built and pushed from the Forgejo runner — no more manual buildx) and rolled out through GitOps. Finder interop retest against the live instance: - ⌘K mount of `http://192.168.10.247/dav`: connected first try, no disconnects - Browse (depth-1 PROPFINDs incl. AppleDouble `._*` probes), MKCOL + rename (`retest-v002`), file copy (LOCK/PUT/GET/UNLOCK round-trip, 8.7KB rtf) — all clean in the request log, 0–11ms - **Pod: Running, 0 restarts** throughout — same traffic pattern that OOM-killed v0.0.1 six times - Memory under load: 132Mi of the 256Mi limit (bounded argon2 + Go runtime — exactly as predicted for the sem=2 cap) Remaining interop (rclone, Cyberduck) can ride along whenever; the OOM class is fixed. Follow-ups tracked separately: `CairnDown` alert rule, Argo webhook for homelab-config didn't fire during the rollout (sync happened via the 3-min poll — worth checking the webhook config).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.0.3
v0.0.2
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#5
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?