Push audit events to a SIEM (syslog / HTTP) (ISO 27001 A.8.15/A.8.16) (research first) #1044

Closed
opened 2026-10-28 23:37:34 +00:00 by Cordy · 4 comments
Owner

Gap found in the #847 customer compliance mapping (2026-10-29): the audit log is a JSONL file (plus optional stdout) the operator collects; there is no push to a SIEM. Research first: syslog (RFC 5424 over TLS, RFC 5425), HTTP/JSON (Splunk HEC, Elastic, OpenSearch, Wazuh, Graylog GELF), CEF/LEEF, delivery guarantees and buffering, filtering by tier, multi-instance (#742). Then decisions and a mockup of the admin config view.

Gap found in the #847 customer compliance mapping (2026-10-29): the audit log is a JSONL file (plus optional stdout) the operator collects; there is no push to a SIEM. Research first: syslog (RFC 5424 over TLS, RFC 5425), HTTP/JSON (Splunk HEC, Elastic, OpenSearch, Wazuh, Graylog GELF), CEF/LEEF, delivery guarantees and buffering, filtering by tier, multi-instance (#742). Then decisions and a mockup of the admin config view.
Author
Owner

Lower priority within v0.7 (feature-gap research, 2026-11-04). ownCloud 10 only writes an audit log file for a forwarder (Splunk) to collect; Cairn already writes JSONL and can mirror the audit log to stdout, which any log shipper can forward. A native syslog/HTTP push stays useful for public sector but is not a launch gap against the closest competitors. Source: scratch/b7e/backlog/gap-v07/research-owncloud10.md.

**Lower priority within v0.7 (feature-gap research, 2026-11-04).** ownCloud 10 only writes an audit log file for a forwarder (Splunk) to collect; Cairn already writes JSONL and can mirror the audit log to stdout, which any log shipper can forward. A native syslog/HTTP push stays useful for public sector but is not a launch gap against the closest competitors. Source: `scratch/b7e/backlog/gap-v07/research-owncloud10.md`.
Author
Owner

Research done (2026-11-11, scratch/b7e/backlog/research-1044.md). No direct competitor pushes audit events: Nextcloud, ownCloud and OpenCloud write to a file or syslog, and the cloud suites offer pull APIs.

Recommended design

  • How it reads: a forwarder that tails the existing audit month files, with a byte cursor saved after the receiver accepts. The audit log is the buffer, so an outage never slows requests.
  • Delivery: at least once, with a stable event id. Each event gets id, sequence, instance and tier added when sent.
  • Transports: syslog over TLS, HTTPS JSON and a Splunk preset. Off by default, up to 3 destinations.
  • Several instances: each s3 instance ships its own files; on a shared posix volume a leased shipper does it.

Findings to fix first or alongside:

  • The s3 audit seal forks with two instances (same segment number).
  • The audit.stdout copy is written under the logger lock, so a blocked stdout stalls audited requests.

Next: decisions.

**Research done** (2026-11-11, `scratch/b7e/backlog/research-1044.md`). No direct competitor pushes audit events: Nextcloud, ownCloud and OpenCloud write to a file or syslog, and the cloud suites offer pull APIs. **Recommended design** - **How it reads:** a forwarder that tails the existing audit month files, with a byte cursor saved after the receiver accepts. The audit log is the buffer, so an outage never slows requests. - **Delivery:** at least once, with a stable event id. Each event gets id, sequence, instance and tier added when sent. - **Transports:** syslog over TLS, HTTPS JSON and a Splunk preset. Off by default, up to 3 destinations. - **Several instances:** each s3 instance ships its own files; on a shared posix volume a leased shipper does it. **Findings to fix first or alongside:** - The s3 audit seal forks with two instances (same segment number). - The `audit.stdout` copy is written under the logger lock, so a blocked stdout stalls audited requests. **Next:** decisions.
Author
Owner

Decisions (2026-11-12)

Built-in push in this batch: a forwarder tailing the audit log. Transports: syslog over TLS, HTTPS JSON, and a Splunk preset. Off by default, up to 3 destinations, at least once with a stable event id. A handbook recipe (stdout plus Fluent Bit or rsyslog) comes with it. The s3 seal fork and the stdout-under-lock stall are fixed in the same batch.

Everything else as researched:

  • Cairn's own JSON line with id, sequence, instance and tier added.
  • Each s3 instance ships its own files; on a shared posix volume one leased shipper does it.
  • The tab is Administration › Audit › Forwarding.
  • A new destination starts from now, with a one-time option to send the last 7 days.
  • Only what is recorded is sent.
  • TLS uses the system CAs or a pinned fingerprint, with an optional client certificate and no "skip verification".
  • File access is never blocked. A bell appears after 15 minutes of lag.
  • No UDP, and plain TCP only behind a warning.

For Nikola / the company: request an IANA private enterprise number for syslog structured data. Until it exists the docs use the example number.

Next: mockups.

**Decisions (2026-11-12)** **Built-in push in this batch:** a forwarder tailing the audit log. Transports: syslog over TLS, HTTPS JSON, and a Splunk preset. Off by default, up to 3 destinations, at least once with a stable event id. A handbook recipe (stdout plus Fluent Bit or rsyslog) comes with it. The s3 seal fork and the stdout-under-lock stall are fixed in the same batch. **Everything else as researched:** - Cairn's own JSON line with id, sequence, instance and tier added. - Each s3 instance ships its own files; on a shared posix volume one leased shipper does it. - The tab is Administration › Audit › Forwarding. - A new destination starts from now, with a one-time option to send the last 7 days. - Only what is recorded is sent. - TLS uses the system CAs or a pinned fingerprint, with an optional client certificate and no "skip verification". - File access is never blocked. A bell appears after 15 minutes of lag. - No UDP, and plain TCP only behind a warning. **For Nikola / the company:** request an IANA private enterprise number for syslog structured data. Until it exists the docs use the example number. **Next:** mockups.
Author
Owner

Shipped in v0.6.407 (PR #1170), live on both dogfoods (v0.6.407, ready, 0 restarts, no new WARN/ERROR; forwarding is off by default, no destination configured).

What shipped:

  • Built-in audit forwarding to up to 3 destinations: syslog over TLS (RFC 5424/5425), HTTPS NDJSON, Splunk HEC; Administration › Audit log › Forwarding or audit.forward; pinned fingerprints or CA trust, client certificates; tokens bound to the endpoint they were typed for.
  • At-least-once delivery with stable event ids; per-instance seal chains on s3; pruned months counted before removal; chain take-over seeds from the instance's own position; skips recorded (audit-forward-skipped).
  • Non-blocking audit.stdout with a dropped counter; six cairn_audit_forward_* metrics with fixed failure stages; pinned certificates on the Certificates page; bells, Overview row, opt-in lag mail.

Open for Nikola: the IANA Private Enterprise Number (syslog STRUCTURED-DATA is - until then). Docs: audit-forwarding.md, deployment.md, CHANGELOG. Follow-ups filed separately.

Shipped in **v0.6.407** (PR #1170), live on both dogfoods (v0.6.407, ready, 0 restarts, no new WARN/ERROR; forwarding is off by default, no destination configured). What shipped: - Built-in audit forwarding to up to 3 destinations: syslog over TLS (RFC 5424/5425), HTTPS NDJSON, Splunk HEC; Administration › Audit log › Forwarding or `audit.forward`; pinned fingerprints or CA trust, client certificates; tokens bound to the endpoint they were typed for. - At-least-once delivery with stable event ids; per-instance seal chains on s3; pruned months counted before removal; chain take-over seeds from the instance's own position; skips recorded (`audit-forward-skipped`). - Non-blocking `audit.stdout` with a dropped counter; six `cairn_audit_forward_*` metrics with fixed failure stages; pinned certificates on the Certificates page; bells, Overview row, opt-in lag mail. Open for Nikola: the IANA Private Enterprise Number (syslog STRUCTURED-DATA is `-` until then). Docs: audit-forwarding.md, deployment.md, CHANGELOG. Follow-ups filed separately.
Cordy closed this issue 2026-11-12 19:12:07 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1044
No description provided.