Run against the dev stack with real ClamAV and queue workers, and a code
review looking for ways around the scanner.
Quarantine now stays quarantined until somebody releases the file. A
rescan only touches files people can download, and changes nothing when
the scanner cannot answer or scanning is off. Before, an old infected file
rescanned while clamd restarted went through the "allow" policy and became
downloadable. The daily missing-files check leaves quarantined files alone,
so a storage outage no longer brings one back as a fresh upload.
A file longer than clamd's StreamMaxLength is "too large" again. clamd
answers and hangs up; the next write raised a warning that became an
exception before the answer was read, so the file was recorded as
"scanner down" and retried past the unscannable policy.
The production compose example gives clamd the settings it needs. On its
own defaults an encrypted zip comes back clean. The Test button now sends a
password-protected zip and fails when it is called clean, and says when an
address answers but is not ClamAV.
Saving the settings restarts the queue workers, which kept the old values
in memory. New scan runs --all, as its name says, and is refused while
scans are queued. A retry scheduled for later no longer counts as a scan
in progress.
Also: quarantine respects client scope for listing, release and
notifications; a zip built before a file was quarantined is refused;
public comments and version links skip unavailable files; a client no
longer sees their own quarantined or missing upload; a file whose bytes
return is scanned at once; clamd listens on IPv6 too, so its container
health check passes.
"Scan existing files" sat disabled on an installation with a library of
144 of them, and the hourly backfill would have found none either. Both
looked for `scan_note = 'before_scanning'`, and no file on any upgraded
installation carries it: the migration gives `scan_status` its default
and writes no note, and the v1 import inserts rows the same way. The
feature was inert on exactly the libraries it exists for.
A file with no reason beside its "not scanned" is now what it plainly
is — one nothing has ever looked at — through File::neverScanned(),
which the backfill, the counts and the badge all ask. Reported as
"Uploaded before virus scanning was switched on" rather than as a bare
"Not scanned", which is the one badge somebody would have had to come
and ask about.
Found on the real screen, not by a test. The tests now cover the shape
an upgrade actually produces.
An infected file now goes somewhere rather than nowhere. Staff holding
the new release_quarantined_files permission get a Quarantine screen
listing what was refused, who uploaded it, and what the scanner called
it. They can delete it as they always could, or release it — which
needs a written reason, a password confirmation on top of the
permission, and lands in the activity log under their name.
Only the administrator role holds that permission by default. Deciding
a threat report is wrong is a different judgement from deciding a file
is no longer needed, which is why it is not delete_files.
Two notifications, two audiences: staff who can act on it, and the
person who uploaded it — for whom this is how they learn their own
machine has something on it. The people the file was shared with are
deliberately not told about a file they never received.
`projectsend:scan-files` runs hourly: it re-queues files still waiting,
and re-scans the ones that went out unscanned while the scanner was
unreachable, since it may be back. With --existing it also works
through a library uploaded before scanning was switched on, paced by a
setting so it does not starve today's uploads.
A file that was downloadable before it was caught says so on the
screen, with its download count, because that is the case where
somebody may already have a copy.
Every upload now starts as "being checked" and is not served to anyone
until a scanner has looked at it. Infected files are quarantined: kept
on disk, unreachable, waiting for an administrator.
The scanner is ClamAV, reached over a socket, streaming the file
wherever it is stored — no temporary copy for an S3 or GCS disk. What
the scanner answers is a fact; what it means for the file is this
installation's setting, so ClamAvScanner knows nothing about settings
and ScanPolicy knows nothing about sockets. Three of clamd's own alert
options are what make a file it could not open come back as an answer
rather than as "OK"; the client maps those to "too large" and
"encrypted" instead of to a threat.
Both policies default to letting files through, marked "not scanned",
which is the product owner's decision: a scanner that cannot answer must
not stop people working. Every such file is logged, and the screens that
say so come with the rest of this work.
Withholding is two rules. A file that is not available drops out of the
scopes that answer "what may this person see" — recipients and the
public listings, never the uploader's own copy. And every route that
puts bytes on the wire asks FileAvailability first: download, thumbnail,
preview, share link, the four public routes and both ends of a zip
build. A share link minted before the scan finishes says the file is
still being checked rather than 404ing.
Not yet here, and coming next: the quarantine screen and its permission,
the notifications, the settings screen, the hourly retry, the backfill
for existing libraries, and the Docker service.
The staff library grew filters for uploader, role, public/private, download
count and version. Two of those already existed on /api/v1/files
(`uploaded_by`, `public`); the other four did not, so an integration could
not ask what the screen asks.
Adds `role_id`, `downloads=none|any`, `version=current|outdated` and
`visibility=public|private`.
`visibility` rather than changing `public`, and that is the decision worth
explaining. `public` has always tested the file's own column, and callers
depend on that answer; changing what an existing filter means is breaking
for everyone already sending it, however much better the new meaning is. So
`public` is untouched and `visibility` is added beside it with the
application's own definition -- File::isEffectivelyPublic(), the flag or a
public folder anywhere above the file -- which is what the badge on a staff
row means. The guide says in a sentence which to reach for. Point the
visibility filter at the column instead and the test that separates them
fails, which is the whole point of having both.
That predicate now lives once, as File::scopeEffectivelyPublic(), beside the
isEffectivelyPublic() it has to agree with. It was a private helper on
FoldersController until a second surface wanted it.
`role_id` deliberately carries no identity guard, unlike `uploaded_by` beside
it. A role names nobody: the files in the result are ones the caller may
already read, and learning one came from somebody holding the Client role
narrows to a set they could have guessed. `uploaded_by` is different in kind
-- a non-empty answer confirms exactly the identity the response is
redacting -- which is why only it is guarded. The reasoning is in the code,
because an absent guard sitting next to a present one is the kind of thing a
reader should not have to re-derive.
Tests cover each filter, the public/visibility split, and the client-scoped
negative: every new filter still returns nothing outside the token's own
library, because a filter narrows a library and never widens one.
File::isExpired() documented the rule the whole application is supposed to follow: once past, the file is hidden from clients and the public site but staff keep full access. The second half is not true of a client-scoped staff member. StaffLibraryScope::buildFiles() builds their library as own uploads plus what each assigned client may see, and that second half runs through File::scopeVisibleToClient, which ends in notExpired() -- a client-side rule. So an expired file they held only through a client leaves their library and answers 403 on download, while their own expired upload stays and an unscoped administrator is unaffected. Api\FilesController stated it the same way, "Only the client branch of the visibility rules drops them", which reads as though a staff caller is unaffected when a client-scoped one is reached through that very branch.
This does not change that behaviour. c8078f65 weighed widening it and decided against, because scopeVisibleToClient is the single source of truth for client file access and the highest-stakes function to go changing for a dashboard widget, and relabelled the widget instead. That decision lived in a commit message and one widget's label; nothing in the code said it, and the docblock nearest the rule went on promising the opposite -- which is how the next person re-derives "staff keep full access" and widens the scope to match.
Documentation and characterisation only. isExpired() now states the boundary and why it is where it is, the API comment is corrected, and ExpiredFileStaffAccessTest pins all three cases.
Verified before merging: 3 passed on the trial-merge. The counter-check has to be inverted for a characterisation test -- these pass on unmodified main by construction, so the question is whether they fail when the boundary moves. Deleting the closing notExpired() from scopeVisibleToClient gives 1 failed / 2 passed, and it is the third case, the one carrying the decision, that falls. File.php overlaps #1726 and Api/FilesController.php overlaps #1727, both already merged, and both are intact in the merged tree. scramble:export reproduces the spec unchanged.
Reported and fixed by @denkfabrik-li.
File::isExpired() documents the rule the application is supposed to
follow: once past, the file is hidden from clients and the public site
"but staff keep full access to view, download, and manage it".
The second half is not true of a client-scoped staff member.
StaffLibraryScope::buildFiles() builds their library as their own uploads
union what each assigned client may see, and that second half runs
through File::scopeVisibleToClient, which ends in notExpired() -- a
client-side rule. Measured on main, with a rep holding one client and a
file the administrator uploaded and shared with that client:
before expiry in_library true GET .../download -> 200
after expiry in_library false GET .../download -> 403
the rep's own expired upload in_library true
an unscoped administrator, same expired file in_library true
Api\FilesController says it the same way -- "Only the client branch of
the visibility rules drops them" -- which reads as though a staff caller
is unaffected, when a client-scoped one is reached through that very
branch.
This does not change that behaviour. c8078f65 weighed exactly this and
decided against it: widening it would mean a library query that keeps
expired rows, and scopeVisibleToClient is the single source of truth for
client file access, the highest-stakes function to go changing for a
dashboard widget. The widget was relabelled instead.
That decision lives in a commit message and in one widget's label.
Nothing in the code said it, and the docblock nearest the rule went on
promising the opposite -- which is how the next person re-derives "staff
keep full access" and widens something.
So both comments now state the boundary and why it is where it is, and
ExpiredFileStaffAccessTest makes it executable: an unscoped staff member
keeps an expired file, a client-scoped one keeps their own expired
upload, a client-scoped one loses a client's file when it expires.
Not changed: scopeVisibleToClient, StaffLibraryScope, and the
expired-files widget. If the boundary should move, that is a separate
conversation and a separate change.
Counter-check inverted, since these pass on unmodified main by
construction -- there is no behaviour fix for them to prove. What they
have to do is fail if the boundary moves, so the mutation is the widening
itself. Deleting the closing notExpired() call from scopeVisibleToClient
turns the file red, 1 failed / 2 passed, and it is the third case, the
one carrying the decision, that falls.
Suite 2108 passed / 2 skipped, 11416 assertions, PHPStan level 8 clean.
Measured on base 06c364d2, where main itself is 2105 / 2.
FileDiskCleanup wraps both deletions in one try. The first is the original
upload, on whatever disk the row names; the second is every cached
rendition, always on the local files disk. Storage::disk() throws outright
for a name with no configured driver -- which is the state the original's
disk is in whenever this fails at all -- so the catch swallowed it and the
renditions were never reached.
Nothing looks for them afterwards. OrphanFileScanner skips the rendition
directories on purpose (they are derived artifacts, never orphaned
uploads), so a file whose external disk had been removed or renamed kept
every cached copy of itself, indefinitely, on the disk that was working.
The two attempts are now separate, each with the same tolerance the class
was written for: a storage failure still never turns a delete click into a
500, and the warning is still the report.
While here, the comment in File::booted() that justifies deferring the
byte removal claimed "the worst case is bytes left on disk with no row,
which OrphanFileScanner already finds and reports". Not on this path: the
row is soft-deleted, and knownPaths() counts a trashed row's path as
claimed -- deliberately, so a scan never offers to double-adopt a file
still inside its erasure grace period. The comment now says what actually
happens.
One test: a file whose disk cannot be resolved loses its renditions. It
goes red without the fix, next to the existing test that the delete itself
still succeeds.
File::booted() removed the bytes the moment a row was deleted. For a
single file that is right. Two paths delete files inside a transaction,
though, and both delete many at once: FolderService::delete() takes a
folder's whole subtree, and DeletedAccountContent::cascadeDelete() takes
everything an account uploaded.
Anything that rolls either transaction back puts every row back while the
bytes are already gone. A transaction exists to make a set of writes
undoable, and removing the bytes was the one write in that set that
nothing can undo. The account path is the sharper one: since content
disposal is nested inside the caller's transaction, the write that fails
need not be in this code at all.
The two failure directions are not equal. Bytes gone with the rows
restored leaves rows pointing at nothing and no way back. Rows gone with
the bytes left leaves orphans on disk, which OrphanFileScanner already
exists to find. Defer to the recoverable one.
Three properties this relies on, all of them checked rather than assumed:
without a pending transaction the callback runs immediately, so a single
delete is unchanged; a savepoint committing inside a larger transaction
does not fire it, which is exactly the account case; and the connection
is the row's own rather than whichever is default.
detachOnDelete stays inside the transaction — it repairs the version
chain's pointers, which is database work that must roll back with
everything else.
Client file sharing, rebuilt from the ground up: a private area per
client, resumable uploads, folders, groups and categories, sharing with
expiry dates and download limits, comments, file versions, an activity
log, a REST API, and sixteen languages.
This repository begins here. ProjectSend 2 was developed privately, and
that development history is not published — the previous generation
remains available, with its own history, at projectsend/legacy.
Free software under the GNU General Public License v2, or (at your
option) any later version.
ignacionelson