The header checkbox on Import orphan files selected only the 25 rows on
screen, so an install with thousands of stray files had to import them a
page at a time. Once a whole page is ticked, the selection bar now offers
"Select all N matching files", and "Import all" takes every orphan the
search matches, on every page.
The import runs in a queued job because it is too slow for a request.
Each file is hashed in full and written in three commits, so 5,000 files
of 4 MB take about four minutes, and PHP stops a request after 30 s of
CPU, around file 1,100. ImportOrphanFilesJob works on the default queue in
chunks of about 45 s: each chunk rescans, imports what is still orphaned
and queues the next one. That keeps every job inside the worker's 60 s
timeout and the queue's 90 s retry_after, so no extra worker is needed,
and mail queued in the meantime goes out between chunks. If a run dies
part way, the next one picks up what is left.
Only one run can be active at a time. OrphanImportProgress keeps its state
in the cache and starts a run under a lock. While a run is active, every
other import is refused, the per-row button included, so no file is
adopted twice. The page polls files/orphans/import-status every 3 s and
shows the run as running, finished, failed with the reason, or stalled
after 5 minutes without progress, which usually means no worker is
listening.
Bulk delete still works one page at a time. The adoption itself moved to
OrphanFileImporter so the request and the job share it, and the rule for
what can be imported now lives in OrphanFileScanner::importable().
A row whose bytes are gone was recorded as "the scanner could not be
reached". Wrong on screen, and wrong underneath: that is the one reason
the hourly sweep re-queues, so every orphaned row would have been
rescanned hourly forever.
It is its own state now, `missing`, and withheld rather than offered:
a client who sees a file listed and gets an error on the download is
worse off than one who never saw it. Staff still see it, marked, which
is the point — somebody has to decide what to do about it. The refusal
says what it is ("no longer on the server") instead of sending somebody
looking for a permission that would let them through.
A daily `projectsend:check-missing-files` finds them, whether or not
this installation scans for viruses: it is not a virus question, and an
installation with no scanner has exactly the same problem. It compares
one disk listing against the rows rather than asking "does this exist?"
per file, which on object storage would be a request per file per day.
Files that come back — a remount, a restored backup — are picked up on
the next run and re-checked rather than left for dead.
They are listed beside the orphans, which is the same fault seen from
the other end: bytes with no row, rows with no bytes. The tab carries
the count, each row says where the file should be, and removing one
takes the record with it through the deletion that already exists.
The dashboard says how many there are, and so does
`projectsend:status`, because a fleet-wide jump in this is a storage
fault nothing else in that document would show.
Client file sharing, rebuilt from the ground up: a private area per
client, resumable uploads, folders, groups and categories, sharing with
expiry dates and download limits, comments, file versions, an activity
log, a REST API, and sixteen languages.
This repository begins here. ProjectSend 2 was developed privately, and
that development history is not published — the previous generation
remains available, with its own history, at projectsend/legacy.
Free software under the GNU General Public License v2, or (at your
option) any later version.
veenone