mirror of
https://github.com/projectsend/projectsend.git
synced 2026-09-11 22:38:54 +00:00
main
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ignacionelson
|
6ad26bb61e |
Hold an upload to the size it said it was sending
Reported by @ry2811 as GHSA-6jh6-gvj5-pv8v. A resumable upload declares its size, and that declaration is what store() weighs against the maximum file size and the client's storage quota. Only the assembled file was ever held to it. The parts in between were bounded one request at a time and never added up, so a client could declare one byte and then stream parts: ten thousand part numbers at twice a 20 MB part is about 400 GB, per session, and the number of sessions was not bounded either. None of it counted against anything, because nothing becomes a File row until the upload completes and ClientStorageUsage sums File rows. A client with a 1 MB quota could fill the volume and repeat. putPart()'s own comment described this defect and treated the per-part cap as the answer to it: "without a cap here the exposure is a day's worth of disk". A cap on one request bounds one request. The exposure was a day's worth of disk multiplied by however many requests somebody cared to make. Three limits, and each one exists because the other two do not cover it. A session may not stage more than it declared. The room for a part is claimed before the body is read — a body's length is not known until it has arrived, and by then it is on the disk being protected — and the write is then capped at exactly what was claimed, so an over-long body is cut off mid-stream as it always was, against a smaller number. The claim is a read and a conditional update under a per-session lock, the same shape complete() already uses: the protocol sends parts in parallel and how many is the client's choice, so an unlocked read lets every part in flight claim the same room, while an atomic claim alone refuses the honest parallel upload instead. Whatever the part really weighs is settled back afterwards, in a finally, or a client's own retries would exhaust a session with room to spare. Open sessions count against the quota at the size they declared. A quota measured against finished files alone is spent twice by opening sessions one after another — each is told there is room, because the ones before it have not finished. The cost is that an abandoned transfer holds its share until it is cancelled or swept, so the sweeper now runs hourly rather than daily: that gap is now somebody unable to upload, which it was not before. And a cap on open sessions, because for anyone with no quota to spend — staff, and clients on an installation that sets none — the session count is the only thing between a declared size and any multiple of it. Four tests fail on the unfixed code, and three existing ones had to change: they declared a tiny size and sent a large part deliberately, to reach the re-checks at complete(). That route is now closed at putPart(), so they reach those re-checks the way a real install would instead — the file-size limit or the quota moving while a long transfer is running, which is the reason complete() re-asks rather than trusting what store() decided. The staged-byte total is BIGINT UNSIGNED, and the suite runs SQLite, which has no unsigned integers. The first version of the bounds read `staged_bytes + :delta BETWEEN 0 AND size` and raised SQLSTATE 22003 on MySQL for any refund — in the comparison, so the bound written to prevent the underflow was the statement that underflowed. Every SQLite test passed on it. Both bounds are now arranged so the column is never inside a subtraction, and UploadSessionStagedBytesMysqlTest skips loudly unless the connection is MySQL. Verified against 8.4, as was the report itself: three sessions declaring one byte each put 6 MB on the volume of a client with a 1 MB quota before, and nothing at all after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CNFU55Tkq6MuEQ73nbbBRx |
||
|
ignacionelson
|
eaba7ff633 |
Let an AWS-hosted install authenticate as its own IAM role
Requested by @ToMMy86 in #1773: an install running on ECS, EC2 or EKS already has a role attached, and making it also create an IAM user with a long-lived access key is both extra work and a worse security posture than the one AWS offers. The AWS SDK resolves credentials from its default provider chain whenever none is supplied, and Laravel's FilesystemManager already omits the `credentials` entry when the key and secret are empty — so the upload path needed almost nothing. What blocked it was ours: - `isConfigured()` demanded a key and a secret for S3, so a credential-less row was never "configured" and every upload silently stayed on the local disk. - `access_key` was `required_if:provider,s3` on both the save and the connection test. - `probeS3()` built an explicit `credentials` array, so Test connection would have failed even once uploads worked. An explicit `use_instance_role` column rather than "the key was left blank", because blank already means "keep the credential you have" on this form — neither the secret nor the GCS key file is ever sent back to the browser. Ticking it deletes the stored key and secret rather than leaving them in the row for the next database dump. Unchanged for everyone else: MinIO, Backblaze, Wasabi and any other S3-compatible service still authenticate with a key and secret, and the region is still required — the chain resolves credentials, not regions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmyH342d8MuW3pDuE9mbtS |
||
|
ignacionelson
|
6340b71dca |
Report recent usage and whether the scheduler is alive
projectsend:status could say what an installation holds and how many accounts it has, but nothing about whether anybody was using it. Adds a `usage` block -- downloads split staff/clients/anonymous, uploads, and five allowlisted action counts -- plus `activity.last_client_login_at`, `health.scheduler` and `health.failed_jobs_latest_at`. The scheduler is the one worth having on its own. `health.queues` catches a dead worker; nothing caught a dead scheduler, and its first symptom is not a stalled feature but an expired file that is still downloadable, because the job that was going to remove it stopped running weeks ago. Nothing about the installation looks wrong while that is true. `failed_jobs_latest_at` exists because the count beside it cannot say whether anything is wrong *now*, and reading it as though it could is a category error rather than a threshold wanting tuning. The table is swept daily, so the count spans a retention window -- one the installation chooses, and one that can be set to keep-forever by somebody who treats a failed job as evidence rather than debris. Two identical installations therefore report different numbers, and on a keep-forever one the count grows until any fixed threshold trips. A timestamp is independent of how long rows are kept: 27 failures whose newest is three weeks old is an installation that has been healthy for three weeks and has not been swept yet. `usage` is a rolling window with no lifetime totals, and that is a correctness decision rather than a presentational one: activity_log is never pruned, so a lifetime count over it gets slower every day of the installation's life while a windowed one stays flat. The window is emitted as `window_days` rather than left for the reader to assume. The actions are an allowlist, not a `group by action`. This document leaves the installation and Action gains cases most weeks, so an open group-by would ship new action names outward with nobody having decided they should go -- and some of them (account.erased, two_factor.reset) are somebody's compliance event, not a business metric. It is also ~30x cheaper: five keyed counts ride (action, created_at) while a group-by starts from created_at and reads rows. The scheduler's failure message and the queue exception text are omitted for the same reason; they are the fields here that can carry a path or a stack trace, and a count with a timestamp says "go and look", which is all a watcher is owed. The two indexes ship as a pair and the migration explains at length why. Measured at 2.1M rows: adding (action, created_at) alone fixes the windowed counts and takes last_staff_login_at -- already running hourly on every tenant -- from 0.63s to 7.7s, because the planner switches to it, still needs actor_type, and does a scattered primary-key lookup per row. With both, that query is answered from the index without reading a row at all (0.0004s) and the whole new usage block costs ~70ms. Also documents the keys as a contract, the way `capabilities` already is. This one fails worse: a renamed capability key breaks a comparison somebody is watching, a renamed usage key produces a chart that is silently empty, and nobody gets paged for a flat line. |
||
|
ignacionelson
|
81bb136e9e |
Merge pull request #1750 from denkfabrik-li/fix/mail-oauth-alarm-fires-once
RefreshMailOAuthTokensCommand is the daily refresh and, by its own docblock, the health check that goes with it: a delegated grant can die silently, and for a portal whose password-reset mails ride on this connection that must surface as a warning rather than as a support ticket weeks later. It decided whether to warn from last_error -- but last_error has a second writer. OAuthCodeFlowBroker::refresh() records a dead grant and notifies nobody, and freshAccessToken() reaches it from every send. So on an installation that is actually sending mail the send got there first, the command read the column as "already told them", and the warning never went out. last_error is cleared only by a successful refresh, which a dead grant never has, so it never went out later either. The alarm worked on installations that were not using the mailbox and failed on the ones that were. The anti-nag rule is not the problem and does not change: one notification per broken state is still all anybody gets. The problem is that one column was answering two questions, which the table's own comment describes -- "what the settings page's warning and the admin notification read". The warning wants "is this connection broken", and any writer may answer it, which is why the settings page turning red on a failed send is correct and stays. The notification wants "have the admins been told", and only the notifier can answer that. broken_notified_at is stamped when the command notifies, and the command asks that instead. It is cleared wherever last_error is cleared -- a successful refresh, a disconnect, a changed client id -- and those three sites now call clearFailure() rather than nulling two columns each, because a connection left healthy but still marked "already told them" would go quiet the next time it died, and a fourth caller is exactly how the first one happened. The send path still records the failure and still notifies nobody: a transport is not a place to decide who gets alarmed. Verified before merging: 27 passed on the merged tree, 2 failed / 25 passed with app/ reset and the migration and tests kept. The recovery test is green either way by design. This touches the same command and broker as #1739 and the follow-up to it, so the merged result was read rather than trusted: the refresh reporting sits in the try and the notify guard in the catch, they do not interact, and refreshSerially() re-reads the row before refreshing so the broken_notified_at the catch reads is the stored one -- while a stand-aside throws nothing and never reaches the catch at all. Note for the next release's upgrade notes: this adds a migration, so "nothing to do beyond dropping in the files" no longer holds. Reported and fixed by @denkfabrik-li. |
||
|
ignacionelson
|
479dc61d2d |
Move branding into core, and leave white-labelling behind
Logo and watermark belonged in the private package for one reason: that is where they were written. Nothing about them needs a hosted platform, and an installation wanting its own mark on the pages it serves is the ordinary case rather than the exotic one. They are core's now, and every installation has them. Hiding "Powered by ProjectSend" did not come. That is what a hosted customer pays for, and its gate is not a capability key but the absence of the code: cloud-modules keeps the listener, so an installation without that package holds the column and has nothing able to read it. Flipping an edition variable buys nothing, which was true before and stays true. Core renders the switch where Capability::AttributionHide is held and has no route that can save it -- there is a test asserting exactly that, which fails the day white-labelling quietly becomes free. The migrations move with their original filenames on purpose. A Cloud tenant already ran them under those names, so Laravel skips them there and the table and its data are untouched; a fresh install or a community one runs them from here for the first time. What got better on the way rather than merely moving: The watermark listeners take core's real RenderingImage and ResolvingImageRendering instead of duck-typed `object` payloads, and the tests construct the genuine events rather than anonymous stand-ins that imitated their shape. The package had to do it that way -- it builds with no host present -- so three PHPStan ignore entries existed to describe what the type system could not see. They are gone. ModuleBoundaryTest asserted "branding is cloud-only, and the suite runs as community", which was never what it was testing. It now reads the capability off the route and subtracts it, so the invariant holds for whichever module is installed. The 43 branding strings arrived in all sixteen locales from the package's own catalogues rather than being retranslated, and the package's are pruned to the one string it still uses. A hosted plan without branding subtracts branding.customize and attribution.hide from the instance's environment. The row is never deleted by that: a downgrade is usually an expired card rather than a decision, and wiping somebody's artwork over a billing event is a loss they would find weeks later with no way to know what it used to be. Hiding reverses; deleting does not. |
||
|
denkfabrik-li
|
7be81d3586 |
Tell the admins the mailbox is dead, even when a send noticed first
The daily refresh doubles as the health check for a connected OAuth mailbox, and its own docblock says why that matters: a grant can die silently, "which for a portal whose password-reset mails ride on this connection must surface as a warning, not as a support ticket weeks later". It decided whether to warn by reading last_error -- but the send path writes that column too. OAuthCodeFlowBroker::refresh() records the failure and notifies nobody, and freshAccessToken() reaches it from every send. So on an installation that actually sends mail, the send lands first, the command reads the column as "already told them", and the warning never goes out. last_error is cleared only by a successful refresh, which a dead grant never has, so it never goes out again either. Measured on main, one dead grant, two orders: nobody sends, command first 1 notification, then quiet correct a password-reset mail first 0 ... 0 ... 0 never The alarm worked on installations that were not using the mailbox and failed on the ones that were. The anti-nag rule is not the problem and does not change. The problem is that last_error answers "is this broken", which any writer may set, while the command needs "have the admins been told", which only the notifier can. The table's own comment shows the conflation -- one column described as "what the settings page's warning and the admin notification read". So the notification gets its own column. broken_notified_at is stamped when the command notifies, and cleared wherever last_error is cleared: a successful refresh, a disconnect, a changed client id. The three call sites go through MailOAuthConnection::clearFailure() rather than nulling two columns each, because a connection left marked "already told them" while healthy would go quiet the next time it died -- the same bug in a new place. |
||
|
ignacionelson
|
7cbffefb01 |
Repair the migrated passwords the hasher will not read
An installation brought over from v1 before the migration tool learned to relabel carries $2a$ or $2b$ digests in users.password. All three bcrypt labels name the same algorithm and password_verify() reads any of them, but Laravel's hasher asks password_get_info() first, gets "unknown", and throws before it looks at the password -- so the login form answers 500 for every migrated account while accounts created in v2 sign in fine. Relabelling the stored digest is the whole fix. Four bytes change; salt and digest are the same, so nobody resets anything and there is no mail to send. Guarded on password_get_info() reading bcrypt afterwards, so a truncated row is left visibly broken rather than quietly rewritten to no effect. $2x$ is left alone on purpose -- it asks for the pre-2011 handling of bytes above 127, so relabelling it would lock out anybody whose password is not plain ASCII. The 500 itself is asserted, not just the repair, so nobody removes the migration later on the grounds that bcrypt is bcrypt. Reported by @pabloalvarez44 in #1706. |
||
|
ignacionelson
|
5d99ab94fd |
Say on screen when nothing is building zip downloads
Zip building moved onto its own queue, which a manual install's worker has to be told about. update.sh repairs the service file and Docker is unaffected, so the population left is somebody upgrading by hand who skipped the release note — and for them the failure is the worst shape available. Email keeps going out perfectly. Zip downloads never finish. Nothing in any log says why, because nothing went wrong: the jobs sit on a queue nobody is reading. The person who missed it has no reason to suspect anything, so the notice has to go looking for them. The application cannot see its own worker processes, only whether work gets done, so the question is asked from the other end: was a build requested that no worker ever picked up? That needs a record of when a build *started*, which is what the new zip_downloads.started_at column is — stamped before any of the work, so it says a worker had the row, not that the row succeeded. Two conditions, because either alone cries wolf. A build has waited past five minutes and was never started, *and* no other build is in hand. The second matters because one worker builds one archive at a time: a queue behind a large build is a healthy queue, and its waiting rows look exactly like abandoned ones until you notice something running. "In hand" is bounded by the job's own timeout, so a worker that died holding a build stops counting as alive an hour later. The banner sits beside the stale-code one, on every staff page rather than the dashboard alone, gated on view_system_info for the reason that one already argues: a background worker not picking work up is a fact about the machine, not a feature of an edition. It names the fix rather than the symptom — "your worker command needs --queue=default,zips" — because somebody reading that downloads are not being processed still has to work out what to do about it. Eight tests, covering both halves of the discrimination rather than just the happy one: a queue waiting behind a live build stays quiet, and a build held by a worker that died does not. Translated into all sixteen locales in the same commit, since a release is close and a banner nobody can read is worse than none. Checked on screen as well as in assertions, with a real stalled row on the dev stack: the banner renders, wraps, and reads correctly. |
||
|
denkfabrik-li
|
16787cf697 |
Record which files a zip actually contains
A zip download's row stores what was asked for — some file ids, some folder ids — and the download action resolved that selection a second time, when the archive was collected, to decide what to log as downloaded. The two are not the same thing. Folder contents are resolved against the scope as it stands at that moment, and an archive is written some time before it is fetched. Add a file to the folder in between and it was logged as downloaded without ever having been in the zip. Move one out of the folder and it was handed over without being logged at all. The same goes for a file that expired or otherwise left the requester's scope after the build: its bytes are in the archive either way. Nothing about this is visible to anyone — the download count on the file is simply wrong. The job already walks exactly the set that goes in, and already counted it for file_count. It now keeps the ids rather than a tally, and the download action logs those. count() gives back the number it was keeping before. Rows written before this column existed fall back to resolving the selection, which is what they were built for; the purge command clears them within a day. |
||
|
ignacionelson
|
1b3abd28b7 |
Merge branch 'main' into feature/oauth-mail-providers
Both sides added a .gitignore rule in the same place: this branch's exception for docs/email-oauth.md, and main's block for the local dev TLS material. Keep both. |
||
|
ignacionelson
|
daec0a877e |
Offer Google Cloud Storage as a storage backend
External storage meant S3 and nothing else, which is an odd hole for a product whose users are as likely to be standing on Google Cloud as on AWS — and paying to move bytes between two clouds to use this. The Storage screen now asks which provider first, and the answer decides which fields it shows, which it validates, and which driver the files_external disk resolves to. One disk, not two. files.disk is a stored column, so a third disk name would fragment the data model and make every $file->disk consumer know three names instead of two; the driver is swapped instead. A service account key gets its own encrypted column rather than sharing `secret`, because the two are validated, labelled and displayed differently and one column meaning two things is how that goes wrong later. Three things do not work by simply adding the adapter, and all three fail quietly: Laravel's temporaryUrl() looks for getTemporaryUrl() on the adapter, while League's GCS adapter names it temporaryUrl(), so without the registered callback every download and preview is a 500. The two SDKs spell the signing options differently, and an unrecognised one is dropped in silence — the symptom is a download named after the storage key, not an exception. GoogleCloudStorageDriver translates, so callers keep speaking one vocabulary, and the test asserts on the URL's contents rather than on "a redirect happened", which is what would let it regress. That callback is also re-bound to the FilesystemAdapter before it runs, so the translation is captured before registering rather than called as $this-> `provider` is validated with 'sometimes', not 'required': absent means S3, which is what every payload written before this choice meant, and stops a browser holding a stale bundle from failing to save on a field it cannot see. Verified in a browser as well as in tests — which is how the null provider on an unmigrated row was found, since the suite migrates and never sees that state. |
||
|
denkfabrik-li
|
933eaa2ba4 |
Send mail through Microsoft Graph as an admin-connected mailbox
Adds "Microsoft 365 (OAuth)" to the Email settings provider dropdown. Selecting it swaps the SMTP form for an app registration (client id, secret, optional tenant) and a "Connect mailbox" flow: the admin signs into the mailbox the installation should send as, and outgoing email goes through Graph sendMail as that mailbox — no password, no app password, no SMTP AUTH, which Microsoft is winding down. Delegated flow on purpose: it needs no admin consent and works for work/school and personal accounts alike. Its one weakness — a grant can die silently behind a password reset or a Conditional Access change — is answered by a daily scheduled refresh that keeps the token alive and, on a dead grant, warns the settings admins once in-app and on the settings page instead of letting mail stop quietly. Tokens and the client secret live encrypted in their own row and are read fresh at send time, never through the boot-config cache. The stored SMTP transport survives a provider switch untouched. |
||
|
ignacionelson
|
98597d462d |
Give a deleted folder's name back
Delete a folder called Test and you could never have a folder called Test again. The deletion worked, the folder left the screen, and the name went with it — permanently, with an error that named a collision against a row the interface will not show you and offered nothing to do about it. Files and groups had it too. All three carry a unique index on slug and all three soft-delete, so the trashed row sat in the index holding a name nothing could reach. A public one failed outright at the validator, which checks the table and therefore sees rows the screen does not. A private one failed more quietly: the derived slug stepped around the trashed row into report-2, then report-3, once per deletion, climbing forever. The reservation was deliberate — a trashed row's slug was kept so that restoring it could not land on somebody else's URL. But nothing in this application restores anything. There is no restore() call, no route, no screen; File's own comment says as much. Soft deletes are here so rows can outlive their delete for foreign keys, the activity log and the erasure grace period, never so they can come back. The slug was being held for a page that could not return, and route binding already 404s the trashed row in the meantime. So deleting now hands the slug back, and the database is what makes that a rewrite rather than a gentler lookup: teaching the collision checks to skip trashed rows would leave two rows holding "report", which the unique index rejects whatever the application thinks. The slug moves to report__deleted-42 instead. Underscores are the whole trick — Str::slug() turns them into hyphens and Rules::slug() refuses them outright, so no derived slug and no hand-typed one can ever land on a vacated one. That is a guarantee about the character class rather than a hope about collisions. The format lives in VacatedSlug rather than on the trait because the migration needs it too and a trait constant cannot be reached through the trait's own name — the first version of this was a fatal error waiting for whoever ran migrations. The migration matters as much as the hook: without it the fix only helps installations that have never deleted anything, and every name already buried stays buried. The collision checks still count trashed rows. It costs nothing and keeps them honest about what the index will accept if a row is ever soft-deleted by something that bypasses model events. previous_file_id had this same bug and was fixed this same way, in File::detachOnDelete — a trashed row holding its predecessor's unique slot so the chain could never be re-linked. This is that fix, for the other four unique indexes' worth of the same mistake. users.email is the one left, and is deliberately not in here: an email address is a login identity rather than a URL handle, and freeing it silently is the wrong answer. Fixes #1645 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
ignacionelson
|
6e47d76ba6 |
ProjectSend 2.0.0
Client file sharing, rebuilt from the ground up: a private area per client, resumable uploads, folders, groups and categories, sharing with expiry dates and download limits, comments, file versions, an activity log, a REST API, and sixteen languages. This repository begins here. ProjectSend 2 was developed privately, and that development history is not published — the previous generation remains available, with its own history, at projectsend/legacy. Free software under the GNU General Public License v2, or (at your option) any later version. |