The last piece of the #1687 follow-up. BuildZipDownloadJob allows itself
an hour, every shipped topology runs exactly one worker, and everything
shares the default queue -- so one large archive delayed every
notification email queued behind it. The size cap and the
one-build-per-person rule bounded that in July; they did not remove it.
onQueue('zips') in the constructor rather than at the dispatch site, so a
second caller cannot forget it. Both images grow a worker for it:
compose.yaml gains worker-zips, supervisord gains [program:queue-zips],
and the existing worker in each narrows to --queue=default. --tries=1
there matches the job, which records its own failure rather than being
retried.
The part that needs care is the manual install. A worker whose command
still says plain `queue:work` consumes `default` only, so it would send
email happily and never finish a single zip, with nothing in any log
saying why. INSTALL.md's unit now reads --queue=default,zips -- one
worker watching both, which is right for most installations -- and says
what happens if you leave it off, with the two-worker split offered for
anyone who would rather keep the two kinds of work apart. CHANGELOG
carries it as an upgrade note, since it is something to do rather than
something that was done.
Verified in the dev stack rather than only in a test: dispatched a build
and watched worker-zips take it while the default worker stayed idle.
Follow-up to #1687, which made a zip build report failure honestly. Four
things it passed near, none of them regressions it introduced.
A zip has never had a size limit — only a cap of 10,000 files, which
bounds nothing that costs anything. Ten thousand spreadsheets zip in
seconds; two hundred videos is an hour of stream-copying and an archive
that fills the disk. Bytes are what a build actually costs, so the new
Settings → Downloads screen caps the total size instead, at 2 GB out of
the box. It is a setting rather than a constant because the safe figure
depends on free disk, on whether sources live on a remote disk, and on
the plan a hosted tenant is on — the file count stays fixed, since it is
a foot-gun rail and not a knob anybody needs. The controller measures
the selection at request time and names both numbers when it refuses;
the job measures again, because it re-derives the selection at run time
and a folder can grow while the job waits in the queue.
Every shipped topology runs exactly one queue worker, and everything
shares the default queue, so raising the job timeout to an hour handed
any signed-in person an hour of everyone else's notification mail. There
is now one build in progress per requester and a named throttle bucket
on the endpoint, which had neither. A pending row older than an hour is
treated as abandoned rather than in progress, so a worker killed hard
enough to skip failed() cannot lock somebody out for good. Giving zip
builds their own queue is the structural fix and wants its own change:
it touches compose, supervisord and the systemd unit in INSTALL.md, and
an install that upgrades without changing its worker command would stop
building zips silently.
zip_downloads.requested_by cascades on delete, so removing a user takes
their rows with it and strands every archive they built — invisible to a
purge that walks rows, and to OrphanFileScanner, which skips zips/ on
purpose. The purge now also sweeps files in zips/ that no row explains,
after a day's grace so a build in progress is never taken out from under
itself.
Two smaller things while in here. A build that failed because every file
had already hit its download limit said only that nothing was available,
and dropped the skipped list — the same distinction the store guard goes
out of its way to draw at request time. And a failed close() now logs
libzip's reason, which the @ silencing had been discarding: "the disk is
full" and "the source vanished" are different problems for whoever has
to fix one, while the requester still sees a message with no server
paths in it.
BuildZipDownloadJob deferred every write to ZipArchive::close() but then
marked the row STATUS_READY regardless of the result:
- close() returns false when a source file was deleted between addFile()
and close() (a concurrent staff delete runs FileDiskCleanup at once) or
the disk filled up; the row went ready over an archive libzip never
wrote, and the download controller X-Accel-served a path that isn't there.
- An archive that ended up with no entries (every selected file removed or
its allowance spent before the queued job ran) is written as no file at
all by libzip, yet close() still returns true — again marked ready.
Check both the close() return and the added-entry count, and fail the row
(deleting any partial archive) when either says nothing was written.
The job also had no $tries/$timeout/failed(): a build of up to MAX_FILES
sources runs past the worker's default 60s timeout, and the kill skips the
catch, stranding the row as PENDING while the frontend polls forever. Give
it room, run it once, and add a failed() backstop that fails a row still
pending (leaving an already-resolved one alone).
Finally, purge leftover zips/{id}.zip* by row id: a killed build leaves a
partial archive and libzip temp file with no path recorded, so the path
field alone never cleaned them up.
Client file sharing, rebuilt from the ground up: a private area per
client, resumable uploads, folders, groups and categories, sharing with
expiry dates and download limits, comments, file versions, an activity
log, a REST API, and sixteen languages.
This repository begins here. ProjectSend 2 was developed privately, and
that development history is not published — the previous generation
remains available, with its own history, at projectsend/legacy.
Free software under the GNU General Public License v2, or (at your
option) any later version.
ignacionelson