Report storage, health and what packages loaded in projectsend:status

Five more facts for whatever watches an installation from outside the
container, and one seam so a package can add its own.

Storage is the one that was about to be wrong. It is summed from the rows
that record it, not measured on the volume: measuring the directory was
correct until external storage went live and silently stopped being, since
an upload that resolves to a bucket leaves nothing on disk to measure. A
figure taken from the filesystem freezes while the account keeps filling,
and on a managed installation that figure is what a customer is shown and
billed against. `by_disk` splits the same sum by where the bytes went,
which is the only way to see what is still sitting locally from before a
cutover. Trashed files are excluded because they hold no bytes -- File's
deleted hook takes them.

Health is what a container cannot show from outside. A queue worker dying
is invisible to anything watching the process: it is still up, and zips
quietly stop building while mail stops going out. Same for a deploy whose
migrations failed -- the application answers every request and is a schema
behind. An unreachable queue reports null rather than zero, because an
unreachable Redis is not an empty queue and reading the second as the
first is how a dead worker looks healthy.

The two-factor enforcement setting is echoed back the way EnforceTwoFactor
reads it, fallback included: reporting a stricter rule than the middleware
actually applies would be worse than reporting none.

And ResolvingInstallationStatus, so a package can report what core cannot
know. The managed storage backend and the version of the package providing
it live in cloud-modules, which this repository must not reference, and a
platform that writes eight environment variables only ever knows what it
asked for. Those came apart once: a bucket provisioned, a token minted,
every variable correct, and an image whose copy of the package predated
the module that reads them. Files went to local disk with the
configuration sitting perfectly right beside them.

Two shapes are cast to objects deliberately. An empty PHP array encodes as
[], so an installation with no packages -- or holding no files -- would
answer a map-shaped field with a list, and a reader unmarshalling it
breaks on the day it happens to be empty rather than the day it is
written. There is a test for each.

Requested by the ProjectSend Cloud control plane, whose storage figure
stops growing the moment a tenant's uploads start reaching the bucket.

app/Modules/Platform/Installation/Console/StatusCommand.php

@@ -6,11 +6,21 @@ namespace App\Modules\Platform\Installation\Console;
 
 use App\Modules\Audit\Action;
 use App\Modules\Audit\ActivityLog;
+use App\Modules\Files\Models\File;
+use App\Modules\Identity\TwoFactor\TwoFactorEnforcement;
 use App\Modules\Identity\UserType;
 use App\Modules\Platform\Capabilities\CapabilityRegistry;
+use App\Modules\Platform\Installation\Events\ResolvingInstallationStatus;
 use App\Modules\Platform\Seats\SeatAllowance;
+use App\Modules\Platform\Settings\Setting;
+use App\Modules\Platform\Settings\Settings;
 use Illuminate\Console\Command;
+use Illuminate\Database\Migrations\Migrator;
 use Illuminate\Support\Carbon;
+use Illuminate\Support\Facades\DB;
+use Illuminate\Support\Facades\Event;
+use Illuminate\Support\Facades\Queue;
+use Throwable;
 
 /**
  * What this installation is, as a fact rather than a screen.
@@ -54,6 +64,36 @@ use Illuminate\Support\Carbon;
  * erasure anonymises entries rather than deleting them (`actor_type`
  * survives on purpose — see AccountEraser), so the answer does not change
  * when the person who gave it is forgotten.
+ *
+ * ### Storage is the application's number, not the disk's
+ *
+ * `storage.bytes` is what this installation holds, summed from the rows
+ * that record it. Measuring the directory instead was correct until
+ * external storage went live, and silently stopped being: an upload that
+ * resolves to a bucket leaves nothing on the volume to measure, so a
+ * figure taken from the filesystem freezes while the account keeps
+ * filling. `by_disk` is the same sum split by where the bytes went, which
+ * is the only way to see what is still sitting on local disk from before
+ * a cutover.
+ *
+ * Trashed files are excluded because they hold no bytes: File's `deleted`
+ * hook removes them, so a soft-deleted row is a record of something that
+ * is gone rather than something still costing anything.
+ *
+ * ### Health is what a container cannot show from outside
+ *
+ * A tenant's queue worker dying is invisible to anything watching the
+ * container: it is still up, and zips quietly stop building while mail
+ * stops going out. Same for migrations that failed after a deploy — the
+ * application answers every request and is a schema behind. Neither is a
+ * secret; both are already visible to anyone who can open the database,
+ * which is anyone who can run this command.
+ *
+ * ### What core cannot answer
+ *
+ * `modules` is filled by whatever packages are installed, through
+ * ResolvingInstallationStatus. A platform that provisioned a bucket knows
+ * what it asked for; only the installation knows what loaded.
  */
 class StatusCommand extends Command
 {
@@ -61,7 +101,7 @@ class StatusCommand extends Command
 
     protected $description = 'Report this installation\'s version, edition, capabilities and seat usage';
 
-    public function handle(CapabilityRegistry $capabilities, SeatAllowance $seats): int
+    public function handle(CapabilityRegistry $capabilities, SeatAllowance $seats, Settings $settings): int
     {
         $status = [
             'version' => (string) config('projectsend.version'),
@@ -89,6 +129,23 @@ class StatusCommand extends Command
                 // two is how a broken probe reads as a dormant fleet.
                 'last_staff_login_at' => $this->lastStaffLoginAt(),
             ],
+            'storage' => $this->storage(),
+            'health' => $this->health(),
+            'settings' => [
+                // Echoed back rather than assumed: an operator writes the
+                // environment variable, and this is the installation
+                // saying what it actually applied. Read the way
+                // EnforceTwoFactor reads it, down to what an unreadable
+                // value falls back to -- reporting a stricter answer than
+                // the middleware enforces would be worse than reporting
+                // none at all.
+                'two_factor_enforcement' => $this->enforcement($settings),
+            ],
+            // Cast so an installation with no packages emits {} rather
+            // than [] -- an empty PHP array encodes as a list, and a
+            // reader unmarshalling a map breaks on the day it happens to
+            // be empty rather than on the day it is written.
+            'modules' => (object) $this->modules(),
         ];
 
         if ($this->option('json')) {
@@ -102,10 +159,128 @@ class StatusCommand extends Command
         $this->line('Staff seats:  '.$this->seatLine($status['seats']['staff']));
         $this->line('Clients:      '.$this->seatLine($status['seats']['clients']));
         $this->line('Last staff login: '.($status['activity']['last_staff_login_at'] ?? 'never'));
+        $this->line('Storage:      '.number_format($status['storage']['bytes']).' bytes in '.$status['storage']['files'].' files');
+        $this->line('Health:       '.$status['health']['pending_migrations'].' migrations pending, '
+            .$status['health']['failed_jobs'].' failed jobs, '
+            .array_sum(array_filter($status['health']['queues'], 'is_int')).' queued');
 
         return self::SUCCESS;
     }
 
+    private function enforcement(Settings $settings): string
+    {
+        $value = $settings->get(Setting::TwoFactorEnforcement);
+
+        $enforcement = (is_string($value) ? TwoFactorEnforcement::tryFrom($value) : null)
+            ?? TwoFactorEnforcement::None;
+
+        return $enforcement->value;
+    }
+
+    /**
+     * What this installation holds, from the rows that record it.
+     *
+     * @return array{bytes: int, files: int, by_disk: object}
+     */
+    private function storage(): array
+    {
+        $perDisk = File::query()
+            ->groupBy('disk')
+            ->selectRaw('disk, sum(size) as bytes, count(*) as files')
+            ->get();
+
+        return [
+            'bytes' => (int) $perDisk->sum(fn (File $row): int => (int) $row->getAttribute('bytes')),
+            'files' => (int) $perDisk->sum(fn (File $row): int => (int) $row->getAttribute('files')),
+            // Keyed by disk name rather than a list, because the reader
+            // wants one of them by name — "how much is still local" — and
+            // not to walk a list looking for it.
+            // Same reason as `modules`: an installation holding no files
+            // at all must still answer with a map.
+            'by_disk' => (object) $perDisk
+                ->mapWithKeys(fn (File $row): array => [
+                    (string) $row->getAttribute('disk') => [
+                        'bytes' => (int) $row->getAttribute('bytes'),
+                        'files' => (int) $row->getAttribute('files'),
+                    ],
+                ])->all(),
+        ];
+    }
+
+    /**
+     * @return array{pending_migrations: int, failed_jobs: int, queues: array<string, int|null>}
+     */
+    private function health(): array
+    {
+        return [
+            'pending_migrations' => $this->pendingMigrations(),
+            'failed_jobs' => $this->failedJobs(),
+            // The two this application actually runs workers for. A depth
+            // is not a fault on its own -- a busy installation has one --
+            // but a depth that only ever grows is a worker that died, and
+            // nothing outside the container can see the difference.
+            'queues' => [
+                'default' => $this->queueDepth('default'),
+                'zips' => $this->queueDepth('zips'),
+            ],
+        ];
+    }
+
+    private function pendingMigrations(): int
+    {
+        /** @var Migrator $migrator */
+        $migrator = app('migrator');
+
+        // Every path, not just database/migrations: a package registers
+        // its own, and a package migration left unrun is exactly the kind
+        // of half-deploy this is here to report.
+        $files = $migrator->getMigrationFiles(array_merge([database_path('migrations')], $migrator->paths()));
+
+        return count(array_diff(array_keys($files), $migrator->getRepository()->getRan()));
+    }
+
+    private function failedJobs(): int
+    {
+        $table = config('queue.failed.table');
+
+        if (! is_string($table) || $table === '') {
+            return 0;
+        }
+
+        return DB::table($table)->count();
+    }
+
+    /**
+     * Null rather than a crash when the queue cannot be reached, and null
+     * rather than zero: an unreachable Redis is not an empty queue, and a
+     * reader watching for a worker that died would read the second as
+     * everything being fine.
+     *
+     * This command is a probe, and a probe that dies on one unreachable
+     * dependency tells the reader nothing about the facts it could still
+     * have answered.
+     */
+    private function queueDepth(string $queue): ?int
+    {
+        try {
+            return Queue::size($queue);
+        } catch (Throwable) {
+            return null;
+        }
+    }
+
+    /**
+     * @return array<string, string|int|bool|null>
+     */
+    private function modules(): array
+    {
+        $event = new ResolvingInstallationStatus;
+
+        Event::dispatch($event);
+
+        return $event->facts;
+    }
+
     /**
      * The most recent interactive staff sign-in, or null if there has
      * never been one.

app/Modules/Platform/Installation/Events/ResolvingInstallationStatus.php

@@ -0,0 +1,48 @@
+<?php
+
+declare(strict_types=1);
+
+namespace App\Modules\Platform\Installation\Events;
+
+/**
+ * "What else is worth knowing about this installation?" — asked once,
+ * by `projectsend:status`, of whatever packages happen to be installed.
+ *
+ * Core cannot answer for them. A managed installation's storage backend
+ * and the version of the package providing it live in
+ * projectsend/cloud-modules, which this repository is public and must
+ * not reference; a control plane still has to be able to observe them,
+ * and observing is exactly what that command is for.
+ *
+ * The distinction this exists to preserve: a platform writing eight
+ * environment variables knows what it *asked for*. Only the installation
+ * knows what actually loaded. Those came apart once — a bucket was
+ * provisioned and a token minted while the container ignored both,
+ * because its image predated the module that reads them, and the
+ * configuration sitting beside the files looked perfectly correct.
+ *
+ * Listened to by *string* class name from a package, same as every
+ * other hook here — see docs/extension-points-architecture.md.
+ */
+final class ResolvingInstallationStatus
+{
+    /**
+     * What listeners have reported, keyed by name.
+     *
+     * Scalars and null only: this is serialised to JSON for a reader
+     * that is not this application, and a shape it has to walk is a
+     * shape it has to be taught. Null is a real answer — "asked, and
+     * the thing is not here" — and it must survive to the document
+     * rather than being dropped, for the reason the whole file's null
+     * handling exists: absent and "nothing to report" are different
+     * facts, and a reader that cannot tell them apart guesses.
+     *
+     * @var array<string, string|int|bool|null>
+     */
+    public array $facts = [];
+
+    public function report(string $key, string|int|bool|null $value): void
+    {
+        $this->facts[$key] = $value;
+    }
+}

tests/Feature/Platform/StatusCommandTest.php

@@ -3,9 +3,14 @@
 declare(strict_types=1);
 
 use App\Models\User;
+use App\Modules\Files\Models\File;
 use App\Modules\Platform\Capabilities\Edition;
+use App\Modules\Platform\Installation\Events\ResolvingInstallationStatus;
+use App\Modules\Platform\Settings\Setting;
+use App\Modules\Platform\Settings\Settings;
 use Illuminate\Support\Facades\Artisan;
 use Illuminate\Support\Facades\Auth;
+use Illuminate\Support\Facades\Event;
 
 /**
  * The probe a reconciler reads instead of being given a shell one-liner
@@ -15,13 +20,16 @@ beforeEach(function () {
     $this->admin = User::factory()->create();
 });
 
-function statusJson(): array
+function statusJson(bool $assoc = true): array
 {
     // Capturing the command's own output rather than asserting on lines,
     // because the contract here is the document and not the wording.
     Artisan::call('projectsend:status', ['--json' => true]);
 
-    return json_decode(Artisan::output(), true, flags: JSON_THROW_ON_ERROR);
+    // Decoded as objects where the shape itself is under test: {} and []
+    // are the same array once an associative decode has flattened them,
+    // and telling them apart is the point of those cases.
+    return (array) json_decode(Artisan::output(), $assoc, flags: JSON_THROW_ON_ERROR);
 }
 
 test('it reports the version, the edition and the capabilities that edition grants', function () {
@@ -133,3 +141,94 @@ test('the human form says never rather than nothing', function () {
         ->expectsOutputToContain('Last staff login: never')
         ->assertSuccessful();
 });
+
+// ------------------------------------------------- what the disk cannot say
+
+/**
+ * Storage is summed from the rows, not measured on the volume.
+ *
+ * Measuring the directory was right until external storage went live and
+ * silently stopped being: an upload that resolves to a bucket leaves
+ * nothing on the volume, so a figure taken from the filesystem freezes
+ * while the account keeps filling.
+ */
+test('storage is what the installation holds, wherever the bytes went', function () {
+    File::factory()->create(['size' => 100, 'disk' => 'files']);
+    File::factory()->create(['size' => 250, 'disk' => 'files']);
+    File::factory()->create(['size' => 1000, 'disk' => 'files_external']);
+
+    $storage = statusJson()['storage'];
+
+    expect($storage['bytes'])->toBe(1350)
+        ->and($storage['files'])->toBe(3)
+        // Split by disk, which is the only way to see what is still
+        // sitting locally from before a cutover.
+        ->and($storage['by_disk'])->toBe([
+            'files' => ['bytes' => 350, 'files' => 2],
+            'files_external' => ['bytes' => 1000, 'files' => 1],
+        ]);
+});
+
+test('a trashed file is not still costing anything', function () {
+    // File's `deleted` hook takes the bytes off disk, so a soft-deleted
+    // row records something that is gone rather than something held.
+    $file = File::factory()->create(['size' => 500, 'disk' => 'files']);
+    File::factory()->create(['size' => 100, 'disk' => 'files']);
+
+    $file->delete();
+
+    expect(statusJson()['storage']['bytes'])->toBe(100);
+});
+
+test('an installation holding nothing still answers with a map', function () {
+    // An empty PHP array encodes as [], and a reader unmarshalling a map
+    // breaks on the day it happens to be empty rather than the day it is
+    // written.
+    expect(json_encode(statusJson(false)['storage']->by_disk))->toBe('{}');
+});
+
+test('it reports the health a container cannot show from outside', function () {
+    // A queue worker dying is invisible to anything watching the
+    // container: it is still up, and zips quietly stop building.
+    $health = statusJson()['health'];
+
+    expect($health)->toHaveKeys(['pending_migrations', 'failed_jobs', 'queues'])
+        ->and($health['pending_migrations'])->toBe(0)
+        ->and($health['failed_jobs'])->toBe(0)
+        ->and($health['queues'])->toHaveKeys(['default', 'zips']);
+});
+
+test('the enforcement setting is echoed back as applied', function () {
+    app(Settings::class)->set(Setting::TwoFactorEnforcement, 'all');
+
+    expect(statusJson()['settings']['two_factor_enforcement'])->toBe('all');
+});
+
+test('an unreadable enforcement value reports none, not something stricter', function () {
+    // Read the way EnforceTwoFactor reads it: reporting a stricter answer
+    // than the middleware actually enforces is worse than reporting none.
+    app(Settings::class)->set(Setting::TwoFactorEnforcement, 'everybody-ish');
+
+    expect(statusJson()['settings']['two_factor_enforcement'])->toBe('none');
+});
+
+// --------------------------------------------- what core cannot answer alone
+
+test('a package can report what core has no way to know', function () {
+    // The managed storage backend and the version of the package that
+    // provides it live outside this repository. A platform knows what it
+    // asked for; only the installation knows what loaded.
+    Event::listen(ResolvingInstallationStatus::class, function (ResolvingInstallationStatus $event): void {
+        $event->report('cloud_modules', '1.1.0');
+        $event->report('managed_storage', 's3 bucket "psc-rebels"');
+    });
+
+    expect(statusJson()['modules'])->toBe([
+        'cloud_modules' => '1.1.0',
+        'managed_storage' => 's3 bucket "psc-rebels"',
+    ]);
+});
+
+test('an installation running no packages answers with a map, not a list', function () {
+    expect(json_encode(statusJson(false)['modules']))->toBe('{}');
+});