Versions: a save refused after a long wait names its first record, which another instance may already have pruned (no WOPI rescue) #945

Closed
opened 2026-10-16 22:59:08 +00:00 by Cordy · 1 comment
Owner

Found in the fix-922 Task 6 re-review (2026-10-17).

What happens

  1. A save first records its content (record 1), then waits for the commit step.
  2. If the wait is longer than half the record grace (30 minutes), the save makes the record again under a fresh key (record 2), and record 1 is removed.
  3. If the commit step then refuses the save, the Unpublished error still names record 1.
  4. After a wait of at least the grace (1 hour) minus the clock skew, another Cairn instance's prune may already have deleted record 1. The WOPI rescue can then not open it and only logs its guard WARN.

The client still gets a 503, so nothing is lost silently. But the rescue copy that #908 R4 promises is missing.

Fix (from the review)

  • Contract (storage/record.go): Again is also asked after a Commit refusal, outside the lock.
  • Drivers:
    • posix WriteRecorded calls Again before removing the temp file;
    • s3 recordAndPublish and CopyRecorded call it on their error paths.
  • Engine:
    • refuse keeps a *Unpublished on the write;
    • recordAgain answers only while publishing or after such a refusal;
    • recorded updates the ID and unmints the first record.
  • Rig: add a fault that fails the commit's RecordOf or snapshot but not the remake.
  • Tests:
    • the error names an existing record that reads the body;
    • a WOPI rescue variant;
    • a mutant that drops the call on the error path.
Found in the fix-922 Task 6 re-review (2026-10-17). ## What happens 1. A save first records its content (record 1), then waits for the commit step. 2. If the wait is longer than half the record grace (30 minutes), the save makes the record again under a fresh key (record 2), and record 1 is removed. 3. If the commit step then refuses the save, the `Unpublished` error still names record 1. 4. After a wait of at least the grace (1 hour) minus the clock skew, another Cairn instance's prune may already have deleted record 1. The WOPI rescue can then not open it and only logs its guard WARN. The client still gets a 503, so nothing is lost silently. But the rescue copy that #908 R4 promises is missing. ## Fix (from the review) - **Contract** (`storage/record.go`): `Again` is also asked after a Commit refusal, outside the lock. - **Drivers:** - posix `WriteRecorded` calls `Again` before removing the temp file; - s3 `recordAndPublish` and `CopyRecorded` call it on their error paths. - **Engine:** - `refuse` keeps a `*Unpublished` on the write; - `recordAgain` answers only while publishing or after such a refusal; - `recorded` updates the ID and unmints the first record. - **Rig:** add a fault that fails the commit's `RecordOf` or snapshot but not the remake. - **Tests:** - the error names an existing record that reads the body; - a WOPI rescue variant; - a mutant that drops the call on the error path.
Author
Owner

Research for batch 8 (2026-11-12): this is already fixed in v0.6.401 by commit 6f4081e57 (st742 task t10). After a commit refusal the posix and s3 drivers make the record again before discarding the body, the refusal names that fresh record, and the first one is removed, so the WOPI rescue can open it. Tests pass on main and fail with the fix removed.

Decision (Nikola): close as fixed. Batch 8 (v0.6.408) adds the test for a failing remake and changes the s3 copy-refusal remake to copy the first record's content rather than the possibly newer source.

Research for batch 8 (2026-11-12): this is **already fixed in v0.6.401** by commit 6f4081e57 (st742 task t10). After a commit refusal the posix and s3 drivers make the record again before discarding the body, the refusal names that fresh record, and the first one is removed, so the WOPI rescue can open it. Tests pass on main and fail with the fix removed. Decision (Nikola): close as fixed. Batch 8 (v0.6.408) adds the test for a failing remake and changes the s3 copy-refusal remake to copy the first record's content rather than the possibly newer source.
Cordy closed this issue 2026-11-12 23:48:43 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#945
No description provided.