Run against the dev stack with real ClamAV and queue workers, and a code
review looking for ways around the scanner.
Quarantine now stays quarantined until somebody releases the file. A
rescan only touches files people can download, and changes nothing when
the scanner cannot answer or scanning is off. Before, an old infected file
rescanned while clamd restarted went through the "allow" policy and became
downloadable. The daily missing-files check leaves quarantined files alone,
so a storage outage no longer brings one back as a fresh upload.
A file longer than clamd's StreamMaxLength is "too large" again. clamd
answers and hangs up; the next write raised a warning that became an
exception before the answer was read, so the file was recorded as
"scanner down" and retried past the unscannable policy.
The production compose example gives clamd the settings it needs. On its
own defaults an encrypted zip comes back clean. The Test button now sends a
password-protected zip and fails when it is called clean, and says when an
address answers but is not ClamAV.
Saving the settings restarts the queue workers, which kept the old values
in memory. New scan runs --all, as its name says, and is refused while
scans are queued. A retry scheduled for later no longer counts as a scan
in progress.
Also: quarantine respects client scope for listing, release and
notifications; a zip built before a file was quarantined is refused;
public comments and version links skip unavailable files; a client no
longer sees their own quarantined or missing upload; a file whose bytes
return is scanned at once; clamd listens on IPv6 too, so its container
health check passes.
tcp://clamav:3310djlkasjdlk connected happily. PHP reads a port the way
atoi does — the digits at the front, the rest ignored — so an address
with a typo on the end was saved, tested, and reported as working, while
tcp://clamav:33101 went somewhere else and failed. The feedback an
operator got had nothing to do with the mistake they made.
ScannerAddress says what an address is: tcp:// with a host and a port of
1 to 65535 and nothing after it, or unix:// with an absolute path. It is
asked in all three places an address arrives — saving, testing, and
connecting. The third matters because a managed address comes from the
environment and never passes the screen.
The answer names the problem rather than reporting "no answer", which
would be true of any unreachable scanner and would send somebody to look
at their network for a typo.
Checked in a browser with both addresses from the report: each is now
refused on Test and on Save, with the same sentence, and
tcp://clamav:3310 still comes back "Working. ClamAV 1.5.4 detected the
test file".
The Test button asked the scanner on file, which makes it useless at
the moment it is most needed: the first attempt, before anything has
been saved. It now tries what is typed, falling back to the stored
address when the field is empty so the button still answers on a screen
nobody has touched. Nothing is written either way — testing is not
saving.
The address travels as a request field and is applied to the request's
own ScanningConfig, which is scoped so the screen and the scanner it
resolves share one. A preview address beats even a managed one, and is
set in exactly one place.
Quarantine now carries a count in the sidebar, like Comments — in amber
rather than the usual colour, because the others count work waiting and
this one counts something that went wrong. Shown only to whoever holds
the permission to act on it.
Both checked in a browser: an address typed and not saved came back
"No answer from tcp://escrito-a-mano.invalid:3310", the stored one
untouched, and the badge renders amber with the real count.
A row whose bytes are gone was recorded as "the scanner could not be
reached". Wrong on screen, and wrong underneath: that is the one reason
the hourly sweep re-queues, so every orphaned row would have been
rescanned hourly forever.
It is its own state now, `missing`, and withheld rather than offered:
a client who sees a file listed and gets an error on the download is
worse off than one who never saw it. Staff still see it, marked, which
is the point — somebody has to decide what to do about it. The refusal
says what it is ("no longer on the server") instead of sending somebody
looking for a permission that would let them through.
A daily `projectsend:check-missing-files` finds them, whether or not
this installation scans for viruses: it is not a virus question, and an
installation with no scanner has exactly the same problem. It compares
one disk listing against the rows rather than asking "does this exist?"
per file, which on object storage would be a request per file per day.
Files that come back — a remount, a restored backup — are picked up on
the next run and re-checked rather than left for dead.
They are listed beside the orphans, which is the same fault seen from
the other end: bytes with no row, rows with no bytes. The tab carries
the count, each row says where the file should be, and removing one
takes the record with it through the deletion that already exists.
The dashboard says how many there are, and so does
`projectsend:status`, because a fleet-wide jump in this is a storage
fault nothing else in that document would show.
A backfill runs for minutes or hours inside a queue worker, where none
of it is visible. The third tab polls every four seconds and says what
is happening: whether anything is running, how many uploads are held,
how deep the queue is, how many files were checked in the last hour,
and the last twenty verdicts with what each one was. When nothing is
running, that same list is the record of the last run, which is what
somebody opening the tab after the fact came for.
Two things the live screen found that the tests had not:
**A backfill read as "nothing is being scanned."** Re-scanning a file
that already went out unchecked deliberately leaves it available, so it
is never "pending" — and the screen counted only pending files. It
counts the scans queue too, and the two are shown separately, because
"an upload nobody can download yet" and "work the scanner has not
reached" are different facts.
**A file whose bytes are missing was recorded as "the scanner could not
be reached."** Wrong on screen, and worse than wrong in behaviour: that
is the one reason the hourly sweep re-queues, so every orphaned row
would have been rescanned every hour forever. It has its own reason
now, and goes through the same policy as a file the scanner could not
open.
Both tabs also gained the header shortcut to Quarantine, and Quarantine
one back to the settings, each shown only to somebody the destination
will actually let in.
`scan_engine` was always null: the column existed, the client never
filled it. Found by running a real ClamAV against a real upload rather
than by a test, since the fake scanner reports whatever it is told.
Asked once per scanner instance — so once per queue job, and the worker
is recycled hourly — rather than on every scan, which would double the
connections to answer a question that changes daily.
Sharing a file that is still being checked writes the assignment and
says nothing. The announcement goes out when the file becomes
available — a clean scan, a file let through while the scanner was
down, or an administrator releasing it from quarantine — so nobody is
ever sent to a page that refuses them, and a file about to be
quarantined is not announced to everyone before anybody knows.
Recipients are derived from the assignments as they stand at that
moment, not remembered from the moment of sharing: a share taken back
in the meantime produces no email, and one added does. New-version
notices ride the same path, which they had to anyway — the audience
rule re-checks visibility, and a file being scanned is not visible.
Two bugs found while writing the tests, both in the hourly command:
Re-queuing a file marked it pending first. Pending means withheld, so
running --existing over a library that predates scanning would have
hidden every file in it from every client for as long as the backfill
ran, and then announced each one to its recipients a second time when
it came back. The job now knows which state it expects instead, and a
rescan leaves the file downloadable until a verdict actually arrives.
An infected file now goes somewhere rather than nowhere. Staff holding
the new release_quarantined_files permission get a Quarantine screen
listing what was refused, who uploaded it, and what the scanner called
it. They can delete it as they always could, or release it — which
needs a written reason, a password confirmation on top of the
permission, and lands in the activity log under their name.
Only the administrator role holds that permission by default. Deciding
a threat report is wrong is a different judgement from deciding a file
is no longer needed, which is why it is not delete_files.
Two notifications, two audiences: staff who can act on it, and the
person who uploaded it — for whom this is how they learn their own
machine has something on it. The people the file was shared with are
deliberately not told about a file they never received.
`projectsend:scan-files` runs hourly: it re-queues files still waiting,
and re-scans the ones that went out unscanned while the scanner was
unreachable, since it may be back. With --existing it also works
through a library uploaded before scanning was switched on, paced by a
setting so it does not starve today's uploads.
A file that was downloadable before it was caught says so on the
screen, with its download count, because that is the case where
somebody may already have a copy.
Every upload now starts as "being checked" and is not served to anyone
until a scanner has looked at it. Infected files are quarantined: kept
on disk, unreachable, waiting for an administrator.
The scanner is ClamAV, reached over a socket, streaming the file
wherever it is stored — no temporary copy for an S3 or GCS disk. What
the scanner answers is a fact; what it means for the file is this
installation's setting, so ClamAvScanner knows nothing about settings
and ScanPolicy knows nothing about sockets. Three of clamd's own alert
options are what make a file it could not open come back as an answer
rather than as "OK"; the client maps those to "too large" and
"encrypted" instead of to a threat.
Both policies default to letting files through, marked "not scanned",
which is the product owner's decision: a scanner that cannot answer must
not stop people working. Every such file is logged, and the screens that
say so come with the rest of this work.
Withholding is two rules. A file that is not available drops out of the
scopes that answer "what may this person see" — recipients and the
public listings, never the uploader's own copy. And every route that
puts bytes on the wire asks FileAvailability first: download, thumbnail,
preview, share link, the four public routes and both ends of a zip
build. A share link minted before the scan finishes says the file is
still being checked rather than 404ing.
Not yet here, and coming next: the quarantine screen and its permission,
the notifications, the settings screen, the hourly retry, the backfill
for existing libraries, and the Docker service.
ignacionelson