fix: distinguish failed image-update checks from "up to date" (#1470)

* fix: distinguish failed image-update checks from "up to date"

The image-update detector collapsed every failure (registry unreachable,
missing auth, rate limit, unresolved local digest) into hasUpdate:false and
dropped the captured reason, so a failed check was indistinguishable from a
current image and never raised a notification, even while a manual stack
update still pulled a newer image.

Detection now records a tri-state per stack (ok / partial / failed) with the
failure reason, exposed via a new GET /api/image-updates/detail (the boolean
GET / is unchanged so fleet aggregation is unaffected). A fully-failed check
preserves the last known has_update, so a transient outage neither erases a
real update nor flaps the notification state. The sidebar shows a muted
"couldn't check" indicator with the reason on hover, and the Update board
lists stacks whose check failed in a "could not be checked" advisory.

Detector hardening: the manifest digest lookup issues HEAD first (falling back
to GET) so it no longer draws down Docker Hub's anonymous pull-rate budget, and
local RepoDigest matching is normalized so official library/* images resolve
their digest instead of falling through to a silent "no update".

* fix: preserve confirmed updates through partial checks; tighten failure surfacing

Address review findings on the tri-state image-update detection:

- A partial check (some images errored) no longer erases a previously
  confirmed update; only a fully-ok check can lower has_update, so a single
  image's registry blip cannot drop the stack's update and re-fire the
  notification on recovery. Adds a regression test.
- The image-level catch stores getErrorMessage(e) rather than raw String(e),
  since that value surfaces verbatim in the sidebar tooltip and readiness
  advisory.
- useImageUpdates and the readiness detail fetch now log unexpected non-ok
  responses instead of silently leaving stale state.
- Remove an unused checkFailedCount derivation (the row indicator is driven by
  the checkStatus prop).
- Reword the recordStackCheckFailure docstring and the HEAD-first comment.

backend/src/services/DatabaseService.ts

@@ -24,6 +24,21 @@ export interface GlobalSetting {
     value: string;
 }
 
+/**
+ * Per-stack image-update check outcome. 'ok' = every checkable image was
+ * reached; 'partial' = some checkable images errored; 'failed' = no checkable
+ * image could be reached (status undeterminable). Distinguishes a failed check
+ * from a confirmed "up to date".
+ */
+export type StackCheckStatus = 'ok' | 'partial' | 'failed';
+
+export interface StackUpdateDetail {
+    hasUpdate: boolean;
+    checkStatus: StackCheckStatus;
+    lastError: string | null;
+    checkedAt: number;
+}
+
 export interface StackAlert {
     id?: number;
     stack_name: string;
@@ -920,6 +935,8 @@ export class DatabaseService {
         node_id INTEGER NOT NULL DEFAULT 0,
         stack_name TEXT NOT NULL,
         has_update INTEGER DEFAULT 0,
+        check_status TEXT NOT NULL DEFAULT 'ok',
+        last_error TEXT,
         checked_at INTEGER NOT NULL,
         PRIMARY KEY (node_id, stack_name)
       );
@@ -1548,6 +1565,15 @@ export class DatabaseService {
           `);
         }
 
+        // Tri-state image-update check outcome. Must run AFTER the composite-PK
+        // recreate above (that block recreates the table from the original four
+        // columns, so columns added earlier would be dropped). 'ok' = every
+        // checkable image was reached; the detector records 'failed'/'partial'
+        // plus a reason when registry checks could not determine status, so a
+        // failed check is no longer indistinguishable from "up to date".
+        maybeAddCol('stack_update_status', 'check_status', "TEXT NOT NULL DEFAULT 'ok'");
+        maybeAddCol('stack_update_status', 'last_error', 'TEXT');
+
         // Drop legacy SSH/TLS columns from pre-0.7 databases (no longer read or written)
         const legacyCols = ['host', 'port', 'ssh_port', 'ssh_user', 'ssh_password', 'ssh_key', 'tls_ca', 'tls_cert', 'tls_key'];
         for (const col of legacyCols) {
@@ -3161,12 +3187,42 @@ export class DatabaseService {
 
     // --- Stack Update Status ---
 
-    public upsertStackUpdateStatus(nodeId: number, stackName: string, hasUpdate: boolean, checkedAt: number): void {
+    public upsertStackUpdateStatus(
+        nodeId: number,
+        stackName: string,
+        hasUpdate: boolean,
+        checkedAt: number,
+        checkStatus: StackCheckStatus = 'ok',
+        lastError: string | null = null,
+    ): void {
         this.db.prepare(
-            `INSERT INTO stack_update_status (node_id, stack_name, has_update, checked_at)
-             VALUES (?, ?, ?, ?)
-             ON CONFLICT(node_id, stack_name) DO UPDATE SET has_update = excluded.has_update, checked_at = excluded.checked_at`
-        ).run(nodeId, stackName, hasUpdate ? 1 : 0, checkedAt);
+            `INSERT INTO stack_update_status (node_id, stack_name, has_update, check_status, last_error, checked_at)
+             VALUES (?, ?, ?, ?, ?, ?)
+             ON CONFLICT(node_id, stack_name) DO UPDATE SET
+               has_update = excluded.has_update,
+               check_status = excluded.check_status,
+               last_error = excluded.last_error,
+               checked_at = excluded.checked_at`
+        ).run(nodeId, stackName, hasUpdate ? 1 : 0, checkStatus, lastError, checkedAt);
+    }
+
+    /**
+     * Record a fully-failed check (no checkable image could be reached) without
+     * touching has_update, so a transient registry outage cannot erase a real
+     * update or flap the notification state. On an existing row it updates only
+     * check_status / last_error / checked_at, leaving has_update intact; a
+     * first-ever failed check inserts a row with has_update = 0 so the stack
+     * still appears with its failure reason.
+     */
+    public recordStackCheckFailure(nodeId: number, stackName: string, lastError: string, checkedAt: number): void {
+        this.db.prepare(
+            `INSERT INTO stack_update_status (node_id, stack_name, has_update, check_status, last_error, checked_at)
+             VALUES (?, ?, 0, 'failed', ?, ?)
+             ON CONFLICT(node_id, stack_name) DO UPDATE SET
+               check_status = 'failed',
+               last_error = excluded.last_error,
+               checked_at = excluded.checked_at`
+        ).run(nodeId, stackName, lastError, checkedAt);
     }
 
     public getStackUpdateStatus(nodeId?: number): Record<string, boolean> {
@@ -3180,6 +3236,27 @@ export class DatabaseService {
         return result;
     }
 
+    /**
+     * Rich per-stack update status (hasUpdate + check outcome + reason) for the
+     * sidebar/readiness UI. GET /api/image-updates stays the boolean map so the
+     * cross-version fleet aggregation contract is unaffected.
+     */
+    public getStackUpdateDetail(nodeId: number): Record<string, StackUpdateDetail> {
+        const rows = this.db.prepare(
+            'SELECT stack_name, has_update, check_status, last_error, checked_at FROM stack_update_status WHERE node_id = ?'
+        ).all(nodeId) as Array<{ stack_name: string; has_update: number; check_status: string | null; last_error: string | null; checked_at: number }>;
+        const result: Record<string, StackUpdateDetail> = {};
+        for (const row of rows) {
+            result[row.stack_name] = {
+                hasUpdate: row.has_update === 1,
+                checkStatus: (row.check_status === 'failed' || row.check_status === 'partial') ? row.check_status : 'ok',
+                lastError: row.last_error,
+                checkedAt: row.checked_at,
+            };
+        }
+        return result;
+    }
+
     public clearStackUpdateStatus(nodeId: number, stackName: string): void {
         this.db.prepare('DELETE FROM stack_update_status WHERE node_id = ? AND stack_name = ?').run(nodeId, stackName);
     }