Upload virus scanning (ClamAV/ICAP) with a "File rejected" notice (research first) #1022

Closed
opened 2026-10-27 00:39:23 +00:00 by Cordy · 4 comments
Owner

From the #695 OpenCloud comparison (item 16b, kept 2026-10-27). OpenCloud ships an antivirus service; Cairn has no upload scanning (grep of internal/ and the handbook).

It is a common public-sector checklist item.

Research first, by the playbook:

  • How others do it: OpenCloud, Nextcloud's antivirus app, Seafile, ownCloud.
  • Scanning before encryption at rest, versus afterwards with the custody key.
  • Large files, and the WebDAV, TUS and REST upload paths.
  • Fail-open versus fail-closed when the scanner is down.
  • Quarantine.
  • A "File rejected" bell item and audit entry.
  • It must be toggleable, since compliance-heavy features stay optional.

Then decisions, a mockup, and a plan.

From the #695 OpenCloud comparison (item 16b, kept 2026-10-27). OpenCloud ships an antivirus service; Cairn has no upload scanning (grep of `internal/` and the handbook). It is a common public-sector checklist item. **Research first, by the playbook:** - How others do it: OpenCloud, Nextcloud's antivirus app, Seafile, ownCloud. - Scanning before encryption at rest, versus afterwards with the custody key. - Large files, and the WebDAV, TUS and REST upload paths. - Fail-open versus fail-closed when the scanner is down. - Quarantine. - A "File rejected" bell item and audit entry. - It must be toggleable, since compliance-heavy features stay optional. Then decisions, a mockup, and a plan.
Author
Owner

Research (2026-10-29)

Market (VERIFIED): OpenCloud/oCIS scan after storing (postprocessing workers download the file again; memory ≈ file size × workers × 8-14, an 844 MB file needed ~7 GB; no streaming, ocis#6803). Nextcloud files_antivirus scans during the upload stream, deletes infected uploads (WebDAV 415), block_unreachable choice, 25 MiB stream default, background rescans. ownCloud 10 scans at final move; ICAP is Enterprise. Seafile Pro: periodic background scan only for sync clients. M365: async, heuristic, flags files and can block downloads.
Voices: false positives are the top complaint (ClamAV docs call auto-delete "a terrible idea"); clamd needs 3-4 GiB RAM; public sector (e.g. BSI OPS.1.1.4 for a state network) expects scanning + reporting.
Standards: BSI OPS.1.1.4 (concept, central reporting; APP.3.2 web server: published files MUST be checked, secondary source), NIST 800-83 §3.4.1, NIS2 CIR 2024/2690 Annex 6.9. Mostly SHOULD for file servers → optional feature fits.

Recommended design: an optional storage decorator (internal/storage/avscan) above quota and above encryption: it streams the plaintext to the scanner while writing and holds back the end of the stream until the verdict. Clean → stored; infected → the write fails and nothing is stored (every driver already discards temp files on read errors; no ciphertext, version, trash entry or quota). One hook covers REST, WebDAV, TUS finish (drop the upload, no retry loop), WOPI saves, OCM, public file drops, peering, version restore. Errors: 422/415 malware-found with the signature name, 503 scan-unavailable, 413 too-large-to-scan. Bell "File rejected" to the uploader (link creator for drops), rate-limited admin bell, admin-tier audit verbs. clamd INSTREAM first (own ~150-line client), ICAP (RFC 3507, configurable header/service) next. Memory bounded by the chunk buffer.

Open questions (recommendations):

  1. Scanner down → fail-closed default, admin can choose fail-open (+ "unscanned" list, scanned later).
  2. Over the scan limit (default 100 MiB, matches clamd) → accept and mark "unscanned" (count for admins), option "refuse"; no partial scans.
  3. Quarantine in v1 → no (nothing is stored; the uploader still has the file); encrypted, time-limited quarantine later.
  4. Show the signature name to the uploader → yes.
  5. Paths → all ingress incl. WOPI, version restore, peering; skip server-generated writes.
  6. Protocols → clamd in v1, ICAP in v0.7.x when a pilot needs a commercial engine.
  7. Toggle → endpoint/limits in config; on/off, failure mode, size policy in admin settings.
  8. False-positive allowlist by SHA-256 (audited) → v1.1.
  9. Background rescans → v1.1, admin-triggered; flag + block download, never auto-delete (works under legal holds).
  10. Peering recipient notice → yes. 11. Optional clamd sidecar in compose/helm (4 GiB note, socket volume).
## Research (2026-10-29) **Market (VERIFIED):** OpenCloud/oCIS scan *after* storing (postprocessing workers download the file again; memory ≈ file size × workers × 8-14, an 844 MB file needed ~7 GB; no streaming, ocis#6803). Nextcloud files_antivirus scans *during* the upload stream, deletes infected uploads (WebDAV 415), `block_unreachable` choice, 25 MiB stream default, background rescans. ownCloud 10 scans at final move; ICAP is Enterprise. Seafile Pro: periodic background scan only for sync clients. M365: async, heuristic, flags files and can block downloads. **Voices:** false positives are the top complaint (ClamAV docs call auto-delete "a terrible idea"); clamd needs 3-4 GiB RAM; public sector (e.g. BSI OPS.1.1.4 for a state network) expects scanning + reporting. **Standards:** BSI OPS.1.1.4 (concept, central reporting; APP.3.2 web server: published files MUST be checked, secondary source), NIST 800-83 §3.4.1, NIS2 CIR 2024/2690 Annex 6.9. Mostly SHOULD for file servers → optional feature fits. **Recommended design:** an optional storage decorator (`internal/storage/avscan`) above quota and above encryption: it streams the plaintext to the scanner while writing and holds back the end of the stream until the verdict. Clean → stored; infected → the write fails and **nothing is stored** (every driver already discards temp files on read errors; no ciphertext, version, trash entry or quota). One hook covers REST, WebDAV, TUS finish (drop the upload, no retry loop), WOPI saves, OCM, public file drops, peering, version restore. Errors: 422/415 `malware-found` with the signature name, 503 `scan-unavailable`, 413 `too-large-to-scan`. Bell "File rejected" to the uploader (link creator for drops), rate-limited admin bell, admin-tier audit verbs. clamd INSTREAM first (own ~150-line client), ICAP (RFC 3507, configurable header/service) next. Memory bounded by the chunk buffer. **Open questions (recommendations):** 1. Scanner down → **fail-closed** default, admin can choose fail-open (+ "unscanned" list, scanned later). 2. Over the scan limit (default 100 MiB, matches clamd) → accept and mark "unscanned" (count for admins), option "refuse"; no partial scans. 3. Quarantine in v1 → **no** (nothing is stored; the uploader still has the file); encrypted, time-limited quarantine later. 4. Show the signature name to the uploader → yes. 5. Paths → all ingress incl. WOPI, version restore, peering; skip server-generated writes. 6. Protocols → clamd in v1, ICAP in v0.7.x when a pilot needs a commercial engine. 7. Toggle → endpoint/limits in config; on/off, failure mode, size policy in admin settings. 8. False-positive allowlist by SHA-256 (audited) → v1.1. 9. Background rescans → v1.1, admin-triggered; flag + block download, never auto-delete (works under legal holds). 10. Peering recipient notice → yes. 11. Optional clamd sidecar in compose/helm (4 GiB note, socket volume).
Author
Owner

Decision (Nikola, 2026-10-29): the recommendations above are approved as the design. Next: whole-page mockup (admin settings + rejection notices), then spec and build.

**Decision (Nikola, 2026-10-29):** the recommendations above are approved as the design. Next: whole-page mockup (admin settings + rejection notices), then spec and build.
Author
Owner

Mockup approved (Nikola, 2026-10-29): design-direction-decision-needed/1022-virus-scanning-mockup.html, all frames, with the recommendations for its 15 open questions (own topic "Virus scanning" under Instance; 422 / WebDAV 415 / 503 / 413 + X-Cairn-Code; remove Finder's 0-byte placeholder from the same credential within 5 min; virus banner stays until closed; bell per file per hour, >3/h folded; public links always refuse what can't be checked; admin bell after 60 s down + "reachable again"; uploader bell when unavailable only for devices, once per hour; Check scanner = PING + VERSION + in-memory EICAR, audited; stale signatures amber after 3 days; pre-existing files not listed as unscanned; "not checked" only in the details pane; signature shown to anonymous uploaders; peering/OCM recipient bell text; DE terms Virenprüfung / Virenscanner / ungeprüft). Next: spec and plan, then build.

**Mockup approved (Nikola, 2026-10-29):** `design-direction-decision-needed/1022-virus-scanning-mockup.html`, all frames, with the recommendations for its 15 open questions (own topic "Virus scanning" under Instance; 422 / WebDAV 415 / 503 / 413 + `X-Cairn-Code`; remove Finder's 0-byte placeholder from the same credential within 5 min; virus banner stays until closed; bell per file per hour, >3/h folded; public links always refuse what can't be checked; admin bell after 60 s down + "reachable again"; uploader bell when unavailable only for devices, once per hour; Check scanner = PING + VERSION + in-memory EICAR, audited; stale signatures amber after 3 days; pre-existing files not listed as unscanned; "not checked" only in the details pane; signature shown to anonymous uploaders; peering/OCM recipient bell text; DE terms Virenprüfung / Virenscanner / ungeprüft). Next: spec and plan, then build.
Author
Owner

Shipped in v0.6.372 (PR #1048). Both dogfoods run v0.6.372; pods ready, no new WARN or ERROR lines at start. No scanner is configured on the dogfoods yet, so the scanning layer is off there.

What is in it:

  • Scanning layer: internal/avscan sits directly after hold. It speaks clamd INSTREAM or ICAP (RFC 3507) and holds back the last 64 KiB chunk until the verdict, so an infected file is never stored. That covers new files and overwrites, on posix and s3, with or without encryption.
  • Every upload path: REST, TUS (refused uploads keep their record until expiry), WebDAV (415, plus the Finder 0-byte placeholder rule), OCM, public drops (strict, with a per-file result line), peering (the blob is discarded), WOPI (no rescue copy for a virus) and version restore.
  • Admin › Virus scanning:
    • status, and Check scanner, which runs EICAR from memory plus a clean probe;
    • fail closed or open, and the over-limit policy;
    • the unscanned files list, which now survives Deleted files and restore.
  • Overview: row and cards, rail dots, outage bells and a shared outage lease for several instances (#742). Bells for people, link creators and peering recipients fold after three per hour.
  • Audit and deployment: audit group scanning (6 verbs, admin tier). Optional clamd sidecar in compose and helm (chart 0.6.1, scanner.clamd.streamMaxLength).
  • CI and docs: a CI end-to-end test against a fake scanner, and a manual interop-clamav workflow. Docs: docs/handbook/virus-scanning.md, plus the configuration reference, deployment and ARCHITECTURE.md.

Open points:

  • The helm sidecar's start command under the restricted Pod Security Standard is not yet verified on a real cluster.
  • The interop-clamav workflow has not run against real ClamAV yet.

The live checks are listed in #1005. Closing.

**Shipped in v0.6.372** (PR #1048). Both dogfoods run v0.6.372; pods ready, no new WARN or ERROR lines at start. No scanner is configured on the dogfoods yet, so the scanning layer is off there. What is in it: - **Scanning layer:** `internal/avscan` sits directly after hold. It speaks clamd INSTREAM or ICAP (RFC 3507) and holds back the last 64 KiB chunk until the verdict, so an infected file is never stored. That covers new files and overwrites, on posix and s3, with or without encryption. - **Every upload path:** REST, TUS (refused uploads keep their record until expiry), WebDAV (415, plus the Finder 0-byte placeholder rule), OCM, public drops (strict, with a per-file result line), peering (the blob is discarded), WOPI (no rescue copy for a virus) and version restore. - **Admin › Virus scanning:** - status, and Check scanner, which runs EICAR from memory plus a clean probe; - fail closed or open, and the over-limit policy; - the unscanned files list, which now survives Deleted files and restore. - **Overview:** row and cards, rail dots, outage bells and a shared outage lease for several instances (#742). Bells for people, link creators and peering recipients fold after three per hour. - **Audit and deployment:** audit group `scanning` (6 verbs, admin tier). Optional clamd sidecar in compose and helm (chart 0.6.1, `scanner.clamd.streamMaxLength`). - **CI and docs:** a CI end-to-end test against a fake scanner, and a manual `interop-clamav` workflow. Docs: `docs/handbook/virus-scanning.md`, plus the configuration reference, deployment and ARCHITECTURE.md. Open points: - The helm sidecar's start command under the restricted Pod Security Standard is not yet verified on a real cluster. - The `interop-clamav` workflow has not run against real ClamAV yet. The live checks are listed in #1005. Closing.
Cordy closed this issue 2026-10-29 11:23:21 +00:00
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Cordy/Cairn#1022
No description provided.