mirror of
https://github.com/projectsend/projectsend.git
synced 2026-10-05 21:01:33 +00:00
veenone
4b30849a88
The header checkbox on Import orphan files selected only the 25 rows on screen, so an install with thousands of stray files had to import them a page at a time. Once a whole page is ticked, the selection bar now offers "Select all N matching files", and "Import all" takes every orphan the search matches, on every page. The import runs in a queued job because it is too slow for a request. Each file is hashed in full and written in three commits, so 5,000 files of 4 MB take about four minutes, and PHP stops a request after 30 s of CPU, around file 1,100. ImportOrphanFilesJob works on the default queue in chunks of about 45 s: each chunk rescans, imports what is still orphaned and queues the next one. That keeps every job inside the worker's 60 s timeout and the queue's 90 s retry_after, so no extra worker is needed, and mail queued in the meantime goes out between chunks. If a run dies part way, the next one picks up what is left. Only one run can be active at a time. OrphanImportProgress keeps its state in the cache and starts a run under a lock. While a run is active, every other import is refused, the per-row button included, so no file is adopted twice. The page polls files/orphans/import-status every 3 s and shows the run as running, finished, failed with the reason, or stalled after 5 minutes without progress, which usually means no worker is listening. Bulk delete still works one page at a time. The adoption itself moved to OrphanFileImporter so the request and the job share it, and the rule for what can be imported now lives in OrphanFileScanner::importable().
63 lines
1.7 KiB
PHP
63 lines
1.7 KiB
PHP
<?php
|
|
|
|
declare(strict_types=1);
|
|
|
|
namespace App\Modules\Files;
|
|
|
|
use App\Models\User;
|
|
use App\Modules\Audit\Action;
|
|
use App\Modules\Files\Models\File;
|
|
use App\Modules\Files\Uploads\StoreUploadedFile;
|
|
use Illuminate\Contracts\Filesystem\Filesystem;
|
|
use Illuminate\Support\Facades\Storage;
|
|
|
|
/**
|
|
* Adopts an orphan in place: a File row for bytes already on disk. Shared
|
|
* by the orphans screen (a ticked selection, in the request) and
|
|
* ImportOrphanFilesJob ("import all", in the background). Which paths are
|
|
* importable is OrphanFileScanner's call.
|
|
*/
|
|
class OrphanFileImporter
|
|
{
|
|
public function __construct(
|
|
private readonly StoreUploadedFile $storeFile,
|
|
) {}
|
|
|
|
public function import(User $user, string $diskName, string $path): File
|
|
{
|
|
$disk = Storage::disk($diskName);
|
|
|
|
return $this->storeFile->create(
|
|
uploader: $user,
|
|
originalName: basename($path),
|
|
path: $path,
|
|
mimeType: $disk->mimeType($path) ?: 'application/octet-stream',
|
|
size: $disk->size($path),
|
|
checksum: $this->checksumOf($disk, $path),
|
|
folderId: null,
|
|
disk: $diskName,
|
|
action: Action::FileImported,
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Streamed rather than hash_file() on a local path — the only way to
|
|
* checksum a file that might live on a non-local disk (S3 has no
|
|
* local filesystem path to hand hash_file()).
|
|
*/
|
|
private function checksumOf(Filesystem $disk, string $path): string
|
|
{
|
|
$stream = $disk->readStream($path);
|
|
|
|
if ($stream === null) {
|
|
return '';
|
|
}
|
|
|
|
$context = hash_init('sha256');
|
|
hash_update_stream($context, $stream);
|
|
fclose($stream);
|
|
|
|
return hash_final($context);
|
|
}
|
|
}
|