feat(auto-heal): restart crashed containers and harden the heal loop (#1258)

* feat(auto-heal): restart crashed containers and harden the heal loop

Auto-Heal now restarts containers that crash (non-zero exit) and stay down
past the policy threshold, in addition to those that fail their Docker
healthcheck. Crash detection reuses the container event classifier so a
container that exits cleanly or that an operator stopped is never restarted;
only classified crashes set the heal signal.

Also hardens the existing loop:
- A paid controlling instance refreshes proxied remotes' entitlement on a
  background interval so a remote node's policies keep evaluating between
  operator visits instead of lapsing a few minutes after the sheet was last
  opened. A node that stays unreachable surfaces a warning.
- Overlapping policies (all-services plus a service-specific one) restart a
  given container at most once per evaluation pass, so the hourly cap holds.
- A failed restart now counts toward the cooldown and hourly cap, so a broken
  setup is retried on the cooldown interval rather than every poll.
- Diagnostic logging behind developer mode for evaluation, heal decisions,
  timing, and lease refresh.

* docs(auto-heal): document crash healing and refresh troubleshooting

Cover the two heal conditions (unhealthy and crashed), note that clean exits
and operator stops are never restarted and that crash healing acts on crashes
observed while Sencho is running, and update the troubleshooting and tab
visibility entries accordingly.

* fix(auto-heal): close stale crash-signal race and harden lease refresh

A crash signal could outlive the crash it described. The exit classifier is
deferred 500ms, so an immediate restart could let it stamp the crash marker
after the container was already running, and a later clean or operator-initiated
exit did not clear it; the next poll could then restart a container that had
exited cleanly. Now a clean or intentional exit always clears the marker, a die
that a start has superseded is not stamped, and the die's own time is captured at
arrival rather than at the deferred classification so the supersede check is
accurate.

Also:
- Crash state survives the event service's idle-prune window, so crash healing
  works for any configured threshold rather than only short ones.
- An exited or dead container is matched before any health-text parsing, so it
  can never fall into the healthcheck path.
- A remote with no reachable proxy target counts toward the lease-refresh
  failure warning instead of being silently skipped.

backend/src/services/DockerEventService.ts

@@ -33,6 +33,13 @@ export interface ContainerHealthSnapshot {
     healthStatus?: 'healthy' | 'unhealthy' | 'starting';
     unhealthySince?: number;
     lastKillAt?: number;
+    /**
+     * Timestamp (ms) of the last exit classified as a crash or OOM kill, cleared
+     * when the container next starts. Set independently of the crash-alert toggle
+     * so Auto-Heal can distinguish a crash (heal-worthy) from an operator stop or
+     * clean exit (which are never stamped here).
+     */
+    crashedAt?: number;
 }
 
 /** Grace window after a `die` before classifying, to absorb out-of-order kill events. */
@@ -85,6 +92,8 @@ interface InternalContainerState extends ContainerLifecycleState {
     lastActivityAt: number;
     healthStatus?: 'healthy' | 'unhealthy' | 'starting';
     unhealthySince?: number;
+    crashedAt?: number;
+    lastStartAt?: number;
 }
 
 interface DockerEventPayload {
@@ -427,12 +436,15 @@ export class DockerEventService {
     }
 
     private onDie(id: string, event: DockerEventPayload): void {
+        // Capture the die time at arrival, not inside the deferred classifier, so a
+        // start that races in during the grace window is correctly seen as later.
+        const dieAt = this.eventTimeMs(event);
         // Defer classification to absorb out-of-order kill events.
         const existing = this.pendingDieTimers.get(id);
         if (existing) clearTimeout(existing);
         const timer = setTimeout(() => {
             this.pendingDieTimers.delete(id);
-            void this.classifyDie(id, event);
+            void this.classifyDie(id, event, dieAt);
         }, DIE_GRACE_WINDOW_MS);
         this.pendingDieTimers.set(id, timer);
     }
@@ -478,8 +490,10 @@ export class DockerEventService {
         state.lastKillAt = undefined;
         state.oomPending = undefined;
         state.lastCrashAlertAt = undefined;
+        state.crashedAt = undefined;
         state.unhealthySince = undefined;
         state.healthStatus = 'starting';
+        state.lastStartAt = Date.now();
         state.lastActivityAt = Date.now();
     }
 
@@ -492,7 +506,7 @@ export class DockerEventService {
         }
     }
 
-    private async classifyDie(id: string, event: DockerEventPayload): Promise<void> {
+    private async classifyDie(id: string, event: DockerEventPayload, dieAt: number): Promise<void> {
         const state = this.getOrCreateState(id, event);
         const exitCodeStr = event.Actor?.Attributes?.exitCode;
         const parsedExit = exitCodeStr !== undefined ? parseInt(exitCodeStr, 10) : undefined;
@@ -500,7 +514,7 @@ export class DockerEventService {
         const now = Date.now();
 
         let classification = classifyDie(
-            { at: this.eventTimeMs(event), exitCode },
+            { at: dieAt, exitCode },
             { lastKillAt: state.lastKillAt, oomPending: state.oomPending },
         );
 
@@ -508,7 +522,25 @@ export class DockerEventService {
         state.oomPending = undefined;
         state.lastActivityAt = now;
 
-        if (classification === 'intentional' || classification === 'clean') return;
+        // A clean or intentional exit clears any prior crash signal, so a stale
+        // crash cannot outlive a later graceful stop and be mistaken for a fresh one.
+        if (classification === 'intentional' || classification === 'clean') {
+            state.crashedAt = undefined;
+            return;
+        }
+
+        // If the container started again strictly after this die occurred, the die
+        // is superseded (the container recovered). Do not stamp a crash signal or
+        // alert for it; the classification is deferred 500ms, so an immediate
+        // restart can race ahead of this handler. A start at the same instant is
+        // treated as preceding a genuine re-crash, not superseding it.
+        if (state.lastStartAt !== undefined && state.lastStartAt > dieAt) return;
+
+        // Stamp the heal signal for Auto-Heal before the alert dedup/toggle gates
+        // below. This must be independent of whether a crash alert is dispatched,
+        // so Auto-Heal still sees the crash when crash alerts are disabled or
+        // rate-suppressed. Cleared on the next `start` or a later clean exit.
+        state.crashedAt = now;
 
         // Dedup early: crashloops repeatedly reach this point with exit 137,
         // and the OOM fallback below issues a Docker inspect. Skipping the
@@ -678,6 +710,10 @@ export class DockerEventService {
         if (this.containerState.size === 0) return;
         const cutoff = Date.now() - STATE_STALE_AFTER_MS;
         for (const [id, state] of this.containerState) {
+            // Keep state for a container with an unresolved crash so Auto-Heal can
+            // still act on it past the default stale window (policy thresholds run
+            // up to 24h). It is cleared on `start` and removed on `destroy`.
+            if (state.crashedAt !== undefined) continue;
             if (state.lastActivityAt < cutoff) {
                 this.containerState.delete(id);
             }
@@ -759,6 +795,7 @@ export class DockerEventService {
             healthStatus: s.healthStatus,
             unhealthySince: s.unhealthySince,
             lastKillAt: s.lastKillAt,
+            crashedAt: s.crashedAt,
         };
     }
 }