s3: audit log and leak-tracing marks live on the pod volume — move them to the bucket? (research first) #823

Closed
opened 2026-09-29 18:02:21 +00:00 by Cordy · 2 comments
Owner

Follow-up from #820 / PR #822 (2026-09-29).

Today: on the s3 driver, all instance state lives encrypted under .cairn-state/ in the bucket, except two append-only records that stay on the pod's /data volume, documented in deployment.md "What still lives locally":

Path What Loss on a volume-less restart
/data/.cairn/audit/ Audit log, monthly JSONL (if audit.enabled) Access history before the loss is gone
/data/.cairn/marks/ Leak-tracing ledger + its HMAC signing secret (#423) A new secret is minted; copies downloaded before the loss can no longer be verified or traced

The Helm chart calls the volume optional on s3, so an operator following that advice has no audit trail across restarts and silently breaks leak tracing. For a compliance-positioned product that is the wrong default.

Why these two are different from the #820 stores: they are append-only and grow (monthly files, retention in months), whereas every .cairn-state/ document is a small whole-object rewrite. A naive "bind the file" would PUT the whole month on every event.

Research request (before any design)

How do comparable self-hosted / S3-backed systems keep an audit trail (and, where relevant, a signing secret) when the app tier is stateless?

  • Nextcloud (admin_audit → log file or syslog), ownCloud Infinite Scale / OpenCloud (audit service, events via NATS → stdout/file), Seafile, Pydio Cells.
  • MinIO (audit webhook / Kafka targets, no local audit), Garage, Ceph RGW (ops log into a RADOS pool), AWS CloudTrail (append as immutable objects in an S3 bucket, digest files for integrity, Object Lock).
  • HashiCorp Vault / OpenBao audit devices (file / syslog / socket; "fail closed if no audit device can write").
  • Kubernetes-native practice: stdout + a log shipper (Loki, Fluent Bit) vs. app-managed durable storage.

Questions to answer:

  1. Where does the record of truth live: the app's storage backend, a separate sink, or stdout plus the operator's pipeline?
  2. How is append-only achieved on object storage without rewrites (per-event objects, periodic sealed segments, Object Lock / WORM)?
  3. Tamper evidence: hash chains, signed digests (CloudTrail digest files)?
  4. Fail-open vs fail-closed when the audit sink is unavailable.
  5. Where do they keep signing/HMAC secrets for stateless replicas (config/env vs state)? This decides whether the marks secret belongs in config (like the recovery identity) rather than state.

Deliverable of the research: a short comparison table plus a recommendation for Cairn, posted here, before any code. Candidate directions to evaluate, not decided: (a) sealed monthly or daily segments as encrypted objects under .cairn-state/audit/ with an in-volume buffer; (b) the marks secret moves to config/env, the ledger follows the audit design; (c) stdout/syslog sink as an additional option.

Follow-up from #820 / PR #822 (2026-09-29). **Today:** on the s3 driver, all instance state lives encrypted under `.cairn-state/` in the bucket, except two append-only records that stay on the pod's `/data` volume, documented in deployment.md "What still lives locally": | Path | What | Loss on a volume-less restart | |---|---|---| | `/data/.cairn/audit/` | Audit log, monthly JSONL (if `audit.enabled`) | Access history before the loss is gone | | `/data/.cairn/marks/` | Leak-tracing ledger + its HMAC signing secret (#423) | A new secret is minted; copies downloaded before the loss can no longer be verified or traced | The Helm chart calls the volume optional on s3, so an operator following that advice has no audit trail across restarts and silently breaks leak tracing. For a compliance-positioned product that is the wrong default. **Why these two are different from the #820 stores:** they are append-only and grow (monthly files, retention in months), whereas every `.cairn-state/` document is a small whole-object rewrite. A naive "bind the file" would PUT the whole month on every event. ## Research request (before any design) How do comparable self-hosted / S3-backed systems keep an audit trail (and, where relevant, a signing secret) when the app tier is stateless? - **Nextcloud** (`admin_audit` → log file or syslog), **ownCloud Infinite Scale / OpenCloud** (audit service, events via NATS → stdout/file), **Seafile**, **Pydio Cells**. - **MinIO** (audit webhook / Kafka targets, no local audit), **Garage**, **Ceph RGW** (ops log into a RADOS pool), **AWS CloudTrail** (append as immutable objects in an S3 bucket, digest files for integrity, Object Lock). - **HashiCorp Vault / OpenBao** audit devices (file / syslog / socket; "fail closed if no audit device can write"). - Kubernetes-native practice: stdout + a log shipper (Loki, Fluent Bit) vs. app-managed durable storage. Questions to answer: 1. Where does the record of truth live: the app's storage backend, a separate sink, or stdout plus the operator's pipeline? 2. How is append-only achieved on object storage without rewrites (per-event objects, periodic sealed segments, Object Lock / WORM)? 3. Tamper evidence: hash chains, signed digests (CloudTrail digest files)? 4. Fail-open vs fail-closed when the audit sink is unavailable. 5. Where do they keep signing/HMAC secrets for stateless replicas (config/env vs state)? This decides whether the marks secret belongs in config (like the recovery identity) rather than state. **Deliverable of the research:** a short comparison table plus a recommendation for Cairn, posted here, before any code. Candidate directions to evaluate, not decided: (a) sealed monthly or daily segments as encrypted objects under `.cairn-state/audit/` with an in-volume buffer; (b) the marks secret moves to config/env, the ledger follows the audit design; (c) stdout/syslog sink as an additional option.
Author
Owner

Research (2026-10-01)

Scope after #821: on posix the audit log and marks ledger now live in <root>/.cairn on the share, so posix is solved. What remains is s3, where both still sit on the pod's /data.

System Record of truth Append on object storage Tamper evidence Sink down Signing/HMAC secret
AWS CloudTrail Immutable log objects in an S3 bucket, delivered in batches New object per batch, never rewritten Hourly digest files: SHA-256 of each log + signature of the previous digest (hash chain) n/a (managed) Provider-held key pair
HashiCorp Vault / OpenBao Audit devices: file, syslog, socket n/a HMAC'd sensitive fields Fail closed: requests block until one device can write (full disk = outage) In Vault's own barrier (state)
MinIO Shipped out: audit webhook or Kafka target n/a none built in target-side env / KMS
Ceph RGW Ops log into a RADOS pool, or a socket RADOS entries, never trimmed automatically (grows unbounded) none n/a n/a
Nextcloud admin_audit audit.log in the data dir, or syslog/systemd n/a none n/a config.php
oCIS / OpenCloud Audit service: stdout by default, file optional, JSON n/a none n/a config/env
Seafile Pro FileAudit table in the database, pruned by SQL n/a none n/a config

Answers to the five questions:

  1. Where the truth lives: self-hosted products keep it locally (file/DB) and offer a stream (syslog/stdout/webhook) for shipping; CloudTrail is the one that writes it into the object store.
  2. Append on object storage: never append. Buffer, then write sealed segments as new immutable objects (CloudTrail, ~5 min batches). Rewriting a growing month object is what nobody does.
  3. Tamper evidence: CloudTrail's hash chain over segments is the reference; no self-hosted file product has any.
  4. Fail mode: Vault fails closed; everyone else fails open. Cairn today fails open silently (Append drops write errors).
  5. Secrets: kept where the instance's other state is (Vault barrier) or in config/env (MinIO, Gitea). No one regenerates a signing secret on restart, which is what Cairn's marks do today on a volume-less s3 pod.

Recommendation for Cairn (s3 only)

  1. Sealed segments. Keep the local month file as a write-ahead buffer. Every 5 minutes (or 1,000 events, or shutdown) seal the unsealed tail: encrypt with the state recipients and PUT it as a new object .cairn-state/audit/<YYYY-MM>/<start>-<seq>.jsonl.age. Never rewritten. A lost volume loses at most the unsealed tail.
  2. Hash chain. Each segment carries the SHA-256 of the previous segment (CloudTrail-style). Verification tool: cairnd audit-verify. No extra key needed; signing can come later.
  3. Query + retention: the admin query reads sealed segments for the requested months from the bucket plus the local tail; retainMonths deletes whole month prefixes.
  4. Fail mode: stay fail-open (a file server that stops serving because its audit sink hiccups is worse for most deployments), but stop failing silently: metric cairn_audit_write_failures_total + seal backlog gauge, an admin-bell item when sealing has failed for >15 min, and the buffer keeps everything until the bucket is back. Fail-closed stays a possible later setting for strict deployments.
  5. Marks: the HMAC secret becomes a state document (.cairn-state/marks-secret.age, generated once, encrypted like peering keys); the ledger uses the same sealed-segment writer as audit.
  6. Optional, cheap: audit.stdout: true to also emit JSON lines to stdout for Loki/Fluent Bit shippers (the oCIS default).

Posix: unchanged (plain files on the share, #821).

## Research (2026-10-01) **Scope after #821:** on **posix** the audit log and marks ledger now live in `<root>/.cairn` on the share, so posix is solved. What remains is **s3**, where both still sit on the pod's `/data`. | System | Record of truth | Append on object storage | Tamper evidence | Sink down | Signing/HMAC secret | |---|---|---|---|---|---| | AWS CloudTrail | Immutable log objects in an S3 bucket, delivered in batches | New object per batch, never rewritten | Hourly digest files: SHA-256 of each log + signature of the previous digest (hash chain) | n/a (managed) | Provider-held key pair | | HashiCorp Vault / OpenBao | Audit devices: file, syslog, socket | n/a | HMAC'd sensitive fields | **Fail closed**: requests block until one device can write (full disk = outage) | In Vault's own barrier (state) | | MinIO | Shipped out: audit webhook or Kafka target | n/a | none built in | target-side | env / KMS | | Ceph RGW | Ops log into a RADOS pool, or a socket | RADOS entries, **never trimmed automatically** (grows unbounded) | none | n/a | n/a | | Nextcloud `admin_audit` | `audit.log` in the data dir, or syslog/systemd | n/a | none | n/a | config.php | | oCIS / OpenCloud | Audit service: **stdout by default**, file optional, JSON | n/a | none | n/a | config/env | | Seafile Pro | `FileAudit` table in the database, pruned by SQL | n/a | none | n/a | config | Answers to the five questions: 1. **Where the truth lives:** self-hosted products keep it locally (file/DB) and offer a stream (syslog/stdout/webhook) for shipping; CloudTrail is the one that writes it into the object store. 2. **Append on object storage:** never append. Buffer, then write **sealed segments as new immutable objects** (CloudTrail, ~5 min batches). Rewriting a growing month object is what nobody does. 3. **Tamper evidence:** CloudTrail's hash chain over segments is the reference; no self-hosted file product has any. 4. **Fail mode:** Vault fails closed; everyone else fails open. Cairn today fails open **silently** (`Append` drops write errors). 5. **Secrets:** kept where the instance's other state is (Vault barrier) or in config/env (MinIO, Gitea). No one regenerates a signing secret on restart, which is what Cairn's marks do today on a volume-less s3 pod. ## Recommendation for Cairn (s3 only) 1. **Sealed segments.** Keep the local month file as a write-ahead buffer. Every 5 minutes (or 1,000 events, or shutdown) seal the unsealed tail: encrypt with the state recipients and PUT it as a new object `.cairn-state/audit/<YYYY-MM>/<start>-<seq>.jsonl.age`. Never rewritten. A lost volume loses at most the unsealed tail. 2. **Hash chain.** Each segment carries the SHA-256 of the previous segment (CloudTrail-style). Verification tool: `cairnd audit-verify`. No extra key needed; signing can come later. 3. **Query + retention:** the admin query reads sealed segments for the requested months from the bucket plus the local tail; `retainMonths` deletes whole month prefixes. 4. **Fail mode:** stay fail-open (a file server that stops serving because its audit sink hiccups is worse for most deployments), but **stop failing silently**: metric `cairn_audit_write_failures_total` + seal backlog gauge, an admin-bell item when sealing has failed for >15 min, and the buffer keeps everything until the bucket is back. Fail-closed stays a possible later setting for strict deployments. 5. **Marks:** the HMAC secret becomes a state document (`.cairn-state/marks-secret.age`, generated once, encrypted like peering keys); the ledger uses the same sealed-segment writer as audit. 6. Optional, cheap: `audit.stdout: true` to also emit JSON lines to stdout for Loki/Fluent Bit shippers (the oCIS default). Posix: unchanged (plain files on the share, #821).
Author
Owner

Shipped in v0.6.312 (PR #828), recommendation points 1–6. Live: files-bao (s3) logs sealedToBucket=true, files (posix) sealedToBucket=false as designed.

Shipped in v0.6.312 (PR #828), recommendation points 1–6. Live: files-bao (s3) logs `sealedToBucket=true`, files (posix) `sealedToBucket=false` as designed.
Cordy closed this issue 2026-10-01 08:53:11 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#823
No description provided.