Release state-backend instance lock on graceful shutdown (avoid ~5min stall on redeploy) #190

New issue

Closed

opened 2026-08-10 23:08:00 +00:00 by Cordy · 0 comments

Cordy commented

2026-08-10 23:08:00 +00:00

Owner

Summary

The state-backend instance lock (#153) is only released by stale-timeout expiry (5m after last heartbeat), never proactively on shutdown. As a result, any redeploy where the outgoing pod held the lock stalls the incoming pod for up to 5 minutes.

Observed (dogfood, v0.6.6, 2026-08-10)

During an OIDC-issuer config change + rollout restart of cairn-enc, the new pod crash-looped with:

level=ERROR msg="instance lock: refusing to start" err="state backend is
locked by instance \"cairn-enc-7b687546d8-vrh5s\" (pid 1, last heartbeat
1m24s ago, running since 2026-08-10T23:03:34Z); if two instances share one
backend, fix the configuration - if that instance is gone, its lock goes
stale after 5m0s"

The previous instance had acquired the lock moments before being replaced and did not release it on SIGTERM, so the new instance correctly refused to start (two-writer guard doing its job) — but had to wait out the full 5m staleness window before it could boot.

Impact

Every redeploy/restart where the old pod held the lock incurs up to a 5-minute outage window, not the few seconds a rollout should take.
CrashLoopBackOff makes it worse: the incoming pod's retry interval doubles (10s → … → 300s), so the boot can land well after the lock frees, stretching the effective downtime past 5 minutes.
This affects the normal operational path (config change → restart), so it will bite every operator, not just an edge case.

Proposed fix

On graceful shutdown (SIGTERM/SIGINT), release the instance lock as part of the shutdown sequence — delete/clear the lock object in the state backend (.cairn-state/) held by this instance before exiting. Then a rolling replacement finds a free lock immediately.

Design considerations:

Only release if this instance is the current lock holder (compare instance id / holder token) — never blindly delete a lock that may belong to another writer.
Keep the 5m stale-timeout as the fallback for hard kills (OOM, node loss, SIGKILL) where no graceful release runs — this issue is about adding proactive release, not removing the safety net.
With strategy: Recreate (as the dogfood uses for RWO-on-NFS safety) the old pod is fully terminated before the new one starts, so a clean release on SIGTERM fully closes the gap. With surge-style strategies the guard still (correctly) blocks overlap.
Watch the shutdown ordering: release the lock after the last state write / flush, so nothing writes post-release.

Acceptance

Graceful shutdown releases the holder's lock in the state backend.
A rollout restart (Recreate) brings the new pod Ready in seconds, with no instance lock: refusing to start in the new pod's logs.
A hard-killed instance (no graceful shutdown) still leaves a lock that the next instance can only take after the 5m stale timeout (fallback preserved).

Notes

Filed while waiting out exactly this 5m window on the dogfood. Related: #153 (state-in-backend + instance lock).

## Summary The state-backend instance lock (#153) is only released by **stale-timeout expiry (5m after last heartbeat)**, never proactively on shutdown. As a result, any redeploy where the outgoing pod held the lock stalls the incoming pod for up to 5 minutes. ## Observed (dogfood, v0.6.6, 2026-08-10) During an OIDC-issuer config change + `rollout restart` of `cairn-enc`, the new pod crash-looped with: ``` level=ERROR msg="instance lock: refusing to start" err="state backend is locked by instance \"cairn-enc-7b687546d8-vrh5s\" (pid 1, last heartbeat 1m24s ago, running since 2026-08-10T23:03:34Z); if two instances share one backend, fix the configuration - if that instance is gone, its lock goes stale after 5m0s" ``` The previous instance had acquired the lock moments before being replaced and did not release it on `SIGTERM`, so the new instance correctly refused to start (two-writer guard doing its job) — but had to wait out the full 5m staleness window before it could boot. ## Impact - Every redeploy/restart where the old pod held the lock incurs up to a 5-minute outage window, not the few seconds a rollout should take. - CrashLoopBackOff makes it worse: the incoming pod's retry interval doubles (10s → … → 300s), so the boot can land well *after* the lock frees, stretching the effective downtime past 5 minutes. - This affects the normal operational path (config change → restart), so it will bite every operator, not just an edge case. ## Proposed fix On graceful shutdown (`SIGTERM`/`SIGINT`), release the instance lock as part of the shutdown sequence — delete/clear the lock object in the state backend (`.cairn-state/`) held by this instance before exiting. Then a rolling replacement finds a free lock immediately. Design considerations: - Only release if **this** instance is the current lock holder (compare instance id / holder token) — never blindly delete a lock that may belong to another writer. - Keep the 5m stale-timeout as the fallback for hard kills (OOM, node loss, `SIGKILL`) where no graceful release runs — this issue is about *adding* proactive release, not removing the safety net. - With `strategy: Recreate` (as the dogfood uses for RWO-on-NFS safety) the old pod is fully terminated before the new one starts, so a clean release on `SIGTERM` fully closes the gap. With surge-style strategies the guard still (correctly) blocks overlap. - Watch the shutdown ordering: release the lock *after* the last state write / flush, so nothing writes post-release. ## Acceptance - Graceful shutdown releases the holder's lock in the state backend. - A `rollout restart` (Recreate) brings the new pod `Ready` in seconds, with no `instance lock: refusing to start` in the new pod's logs. - A hard-killed instance (no graceful shutdown) still leaves a lock that the next instance can only take after the 5m stale timeout (fallback preserved). ## Notes Filed while waiting out exactly this 5m window on the dogfood. Related: #153 (state-in-backend + instance lock).