A row whose bytes are gone was recorded as "the scanner could not be
reached". Wrong on screen, and wrong underneath: that is the one reason
the hourly sweep re-queues, so every orphaned row would have been
rescanned hourly forever.
It is its own state now, `missing`, and withheld rather than offered:
a client who sees a file listed and gets an error on the download is
worse off than one who never saw it. Staff still see it, marked, which
is the point — somebody has to decide what to do about it. The refusal
says what it is ("no longer on the server") instead of sending somebody
looking for a permission that would let them through.
A daily `projectsend:check-missing-files` finds them, whether or not
this installation scans for viruses: it is not a virus question, and an
installation with no scanner has exactly the same problem. It compares
one disk listing against the rows rather than asking "does this exist?"
per file, which on object storage would be a request per file per day.
Files that come back — a remount, a restored backup — are picked up on
the next run and re-checked rather than left for dead.
They are listed beside the orphans, which is the same fault seen from
the other end: bytes with no row, rows with no bytes. The tab carries
the count, each row says where the file should be, and removing one
takes the record with it through the deletion that already exists.
The dashboard says how many there are, and so does
`projectsend:status`, because a fleet-wide jump in this is a storage
fault nothing else in that document would show.
An infected file now goes somewhere rather than nowhere. Staff holding
the new release_quarantined_files permission get a Quarantine screen
listing what was refused, who uploaded it, and what the scanner called
it. They can delete it as they always could, or release it — which
needs a written reason, a password confirmation on top of the
permission, and lands in the activity log under their name.
Only the administrator role holds that permission by default. Deciding
a threat report is wrong is a different judgement from deciding a file
is no longer needed, which is why it is not delete_files.
Two notifications, two audiences: staff who can act on it, and the
person who uploaded it — for whom this is how they learn their own
machine has something on it. The people the file was shared with are
deliberately not told about a file they never received.
`projectsend:scan-files` runs hourly: it re-queues files still waiting,
and re-scans the ones that went out unscanned while the scanner was
unreachable, since it may be back. With --existing it also works
through a library uploaded before scanning was switched on, paced by a
setting so it does not starve today's uploads.
A file that was downloadable before it was caught says so on the
screen, with its download count, because that is the case where
somebody may already have a copy.
Staff can give a client an expiry date on the create and edit screens,
and through /api/v1/clients. When the date passes, the client is refused
at sign-in and on their next request, and their API access ends too.
Files and history stay, and a later date (or none) brings them back.
Access is checked through one predicate, User::maySignIn(), at every
door: sign-in, the web session, API tokens and the two-factor
challenge. An hourly sweep also switches `active` off, so the list,
its filter and seat counts agree. The sweep is not what enforces it,
so a scheduler that is not running cannot keep an account open.
An account cannot be active with a date that has passed. Reactivating
an expired client needs a new date in the same save.
The day-means-end-of-day-where-you-are rule moved out of FileExpiry
into a shared DateInput, so file and account expiry read dates the
same way.
Requested by @Drardollan in #1310.
Reported by @ry2811 as GHSA-6jh6-gvj5-pv8v.
A resumable upload declares its size, and that declaration is what store()
weighs against the maximum file size and the client's storage quota. Only
the assembled file was ever held to it. The parts in between were bounded
one request at a time and never added up, so a client could declare one
byte and then stream parts: ten thousand part numbers at twice a 20 MB
part is about 400 GB, per session, and the number of sessions was not
bounded either. None of it counted against anything, because nothing
becomes a File row until the upload completes and ClientStorageUsage sums
File rows. A client with a 1 MB quota could fill the volume and repeat.
putPart()'s own comment described this defect and treated the per-part cap
as the answer to it: "without a cap here the exposure is a day's worth of
disk". A cap on one request bounds one request. The exposure was a day's
worth of disk multiplied by however many requests somebody cared to make.
Three limits, and each one exists because the other two do not cover it.
A session may not stage more than it declared. The room for a part is
claimed before the body is read — a body's length is not known until it
has arrived, and by then it is on the disk being protected — and the write
is then capped at exactly what was claimed, so an over-long body is cut
off mid-stream as it always was, against a smaller number. The claim is a
read and a conditional update under a per-session lock, the same shape
complete() already uses: the protocol sends parts in parallel and how many
is the client's choice, so an unlocked read lets every part in flight
claim the same room, while an atomic claim alone refuses the honest
parallel upload instead. Whatever the part really weighs is settled back
afterwards, in a finally, or a client's own retries would exhaust a
session with room to spare.
Open sessions count against the quota at the size they declared. A quota
measured against finished files alone is spent twice by opening sessions
one after another — each is told there is room, because the ones before it
have not finished. The cost is that an abandoned transfer holds its share
until it is cancelled or swept, so the sweeper now runs hourly rather than
daily: that gap is now somebody unable to upload, which it was not before.
And a cap on open sessions, because for anyone with no quota to spend —
staff, and clients on an installation that sets none — the session count
is the only thing between a declared size and any multiple of it.
Four tests fail on the unfixed code, and three existing ones had to change:
they declared a tiny size and sent a large part deliberately, to reach the
re-checks at complete(). That route is now closed at putPart(), so they
reach those re-checks the way a real install would instead — the file-size
limit or the quota moving while a long transfer is running, which is the
reason complete() re-asks rather than trusting what store() decided.
The staged-byte total is BIGINT UNSIGNED, and the suite runs SQLite, which
has no unsigned integers. The first version of the bounds read
`staged_bytes + :delta BETWEEN 0 AND size` and raised SQLSTATE 22003 on
MySQL for any refund — in the comparison, so the bound written to prevent
the underflow was the statement that underflowed. Every SQLite test passed
on it. Both bounds are now arranged so the column is never inside a
subtraction, and UploadSessionStagedBytesMysqlTest skips loudly unless the
connection is MySQL. Verified against 8.4, as was the report itself: three
sessions declaring one byte each put 6 MB on the volume of a client with a
1 MB quota before, and nothing at all after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNFU55Tkq6MuEQ73nbbBRx
Adds "Microsoft 365 (OAuth)" to the Email settings provider dropdown.
Selecting it swaps the SMTP form for an app registration (client id,
secret, optional tenant) and a "Connect mailbox" flow: the admin signs
into the mailbox the installation should send as, and outgoing email
goes through Graph sendMail as that mailbox — no password, no app
password, no SMTP AUTH, which Microsoft is winding down.
Delegated flow on purpose: it needs no admin consent and works for
work/school and personal accounts alike. Its one weakness — a grant
can die silently behind a password reset or a Conditional Access
change — is answered by a daily scheduled refresh that keeps the
token alive and, on a dead grant, warns the settings admins once
in-app and on the settings page instead of letting mail stop quietly.
Tokens and the client secret live encrypted in their own row and are
read fresh at send time, never through the boot-config cache. The
stored SMTP transport survives a provider switch untouched.
Failed queue jobs and read notifications both grow with use, and neither
ever shrank on its own. The failed-jobs list waited for somebody to press
"Delete all failed" — a fine tool for a backlog you are looking at, and
the only thing that ever emptied it. Notifications had nothing at all: one
row per recipient per event, kept for the life of the installation, on
what is easily the fastest-growing table here.
Both now have a retention window, set together on the Scheduler screen
under Housekeeping, and a nightly purge that honours it. Thirty days for
failed jobs and ninety for read notifications, and zero means keep
everything — the explicit choice somebody makes when a failure is evidence
rather than debris.
Unread notifications are never deleted, whatever their age. A notification
nobody has looked at is the one row in that table still doing its job, and
somebody back from four months away should find their news rather than a
clean slate. The activity log is untouched by any of this: it is an audit
trail, and it is never pruned.
Two things came out of building it. The API request log purge has been
running nightly since it shipped without ever appearing on the Scheduler
screen — so a failure of it was invisible on the screen that exists to
make failures visible — and there is now a test asserting the screen's
list and the schedule are the same list, because they had already drifted
once and would again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Client file sharing, rebuilt from the ground up: a private area per
client, resumable uploads, folders, groups and categories, sharing with
expiry dates and download limits, comments, file versions, an activity
log, a REST API, and sixteen languages.
This repository begins here. ProjectSend 2 was developed privately, and
that development history is not published — the previous generation
remains available, with its own history, at projectsend/legacy.
Free software under the GNU General Public License v2, or (at your
option) any later version.
ignacionelson