mirror of
https://github.com/projectsend/projectsend.git
synced 2026-09-16 16:45:07 +00:00
073101d184
Follow-up to #1687, which made a zip build report failure honestly. Four things it passed near, none of them regressions it introduced. A zip has never had a size limit — only a cap of 10,000 files, which bounds nothing that costs anything. Ten thousand spreadsheets zip in seconds; two hundred videos is an hour of stream-copying and an archive that fills the disk. Bytes are what a build actually costs, so the new Settings → Downloads screen caps the total size instead, at 2 GB out of the box. It is a setting rather than a constant because the safe figure depends on free disk, on whether sources live on a remote disk, and on the plan a hosted tenant is on — the file count stays fixed, since it is a foot-gun rail and not a knob anybody needs. The controller measures the selection at request time and names both numbers when it refuses; the job measures again, because it re-derives the selection at run time and a folder can grow while the job waits in the queue. Every shipped topology runs exactly one queue worker, and everything shares the default queue, so raising the job timeout to an hour handed any signed-in person an hour of everyone else's notification mail. There is now one build in progress per requester and a named throttle bucket on the endpoint, which had neither. A pending row older than an hour is treated as abandoned rather than in progress, so a worker killed hard enough to skip failed() cannot lock somebody out for good. Giving zip builds their own queue is the structural fix and wants its own change: it touches compose, supervisord and the systemd unit in INSTALL.md, and an install that upgrades without changing its worker command would stop building zips silently. zip_downloads.requested_by cascades on delete, so removing a user takes their rows with it and strands every archive they built — invisible to a purge that walks rows, and to OrphanFileScanner, which skips zips/ on purpose. The purge now also sweeps files in zips/ that no row explains, after a day's grace so a build in progress is never taken out from under itself. Two smaller things while in here. A build that failed because every file had already hit its download limit said only that nothing was available, and dropped the skipped list — the same distinction the store guard goes out of its way to draw at request time. And a failed close() now logs libzip's reason, which the @ silencing had been discarding: "the disk is full" and "the source vanished" are different problems for whoever has to fix one, while the requester still sees a message with no server paths in it.
99 lines
3.5 KiB
PHP
99 lines
3.5 KiB
PHP
<?php
|
|
|
|
declare(strict_types=1);
|
|
|
|
namespace App\Modules\Files\Console;
|
|
|
|
use App\Modules\Files\Models\ZipDownload;
|
|
use Illuminate\Console\Command;
|
|
use Illuminate\Support\Facades\Storage;
|
|
use League\Flysystem\UnableToRetrieveMetadata;
|
|
|
|
class PurgeZipDownloadsCommand extends Command
|
|
{
|
|
protected $signature = 'projectsend:purge-zip-downloads';
|
|
|
|
protected $description = 'Remove built zip downloads (and their files) older than 24 hours — disposable, regenerable artifacts';
|
|
|
|
public function handle(): int
|
|
{
|
|
$stale = ZipDownload::query()->where('created_at', '<', now()->subDay())->get();
|
|
|
|
// Listed once up front: the loop only deletes, so nothing it does
|
|
// changes what a later row would match.
|
|
$builtZips = collect(Storage::disk('files')->files('zips'));
|
|
|
|
foreach ($stale as $zipDownload) {
|
|
// Every artifact tied to this row's id, not just the recorded
|
|
// path: a build killed before it finished (worker timeout, disk
|
|
// full) leaves a partial archive — and libzip's temp file
|
|
// alongside it — with no path ever written back to the row.
|
|
$artifacts = $builtZips
|
|
->filter(fn (string $path): bool => str_starts_with(basename($path), $zipDownload->id.'.zip'))
|
|
->all();
|
|
|
|
if ($zipDownload->path !== null) {
|
|
$artifacts[] = $zipDownload->path;
|
|
}
|
|
|
|
Storage::disk('files')->delete(array_values(array_unique($artifacts)));
|
|
|
|
$zipDownload->delete();
|
|
}
|
|
|
|
$swept = $this->sweepUnreferenced();
|
|
|
|
$this->info("Purged {$stale->count()} stale zip download(s) and {$swept} unreferenced file(s).");
|
|
|
|
return self::SUCCESS;
|
|
}
|
|
|
|
/**
|
|
* Rows are what the loop above cleans by, so a file whose row is gone
|
|
* is invisible to it — and a row can vanish without its files:
|
|
* zip_downloads.requested_by cascades on delete, so removing a user
|
|
* takes their rows with it and leaves every archive they built behind.
|
|
* Anything already stranded that way before this command learned to
|
|
* look is in the same position.
|
|
*
|
|
* OrphanFileScanner skips zips/ on purpose — this command owns that
|
|
* directory, so closing the gap belongs here.
|
|
*/
|
|
private function sweepUnreferenced(): int
|
|
{
|
|
$disk = Storage::disk('files');
|
|
$cutoff = now()->subDay()->getTimestamp();
|
|
$live = array_flip(ZipDownload::query()->pluck('id')->all());
|
|
$unreferenced = [];
|
|
|
|
foreach ($disk->files('zips') as $path) {
|
|
// Both an archive (12.zip) and libzip's temp beside it
|
|
// (12.zip.aB3xY9) lead with the row id they belong to.
|
|
$id = explode('.', basename($path))[0];
|
|
|
|
if (ctype_digit($id) && isset($live[(int) $id])) {
|
|
continue;
|
|
}
|
|
|
|
try {
|
|
// A day's grace before deleting something no row explains.
|
|
// Nothing here should outlive its row by design, so the
|
|
// wait costs nothing — and it means a file another process
|
|
// has only just put there is never taken out from under it.
|
|
if ($disk->lastModified($path) >= $cutoff) {
|
|
continue;
|
|
}
|
|
} catch (UnableToRetrieveMetadata) {
|
|
// Gone between listing the directory and asking about it.
|
|
continue;
|
|
}
|
|
|
|
$unreferenced[] = $path;
|
|
}
|
|
|
|
$disk->delete($unreferenced);
|
|
|
|
return count($unreferenced);
|
|
}
|
|
}
|