fix(docker-events): harden crash detection against edge cases (#591)

* fix(docker-events): harden crash detection against edge cases

- Isolate per-container failures in the reconcile path so one failed
  container inspect cannot abort classification for the rest of the
  batch after a reconnect.
- Fall back to inspecting State.OOMKilled when a container exits with
  code 137 and no oom event preceded the die, so cgroup OOM kills that
  lose the oom event are still classified correctly. The dedup check
  runs before the fallback so crashloops do not hammer the daemon.
- Guard the intentional-kill window against wildly future-dated Docker
  timestamps by switching to a signed age comparison with a bounded
  negative-skew tolerance, so clock skew cannot flip a genuine crash
  into an intentional stop.
- Align diagnostic logging with the codebase [Service:diag] convention
  and add one informational lifecycle log on boot and shutdown so
  operators running in developer mode can confirm the watcher started.
- New tests cover gap-inspect isolation, the OOM inspect fallback (both
  success and failure paths), duplicate die events collapsing within
  the grace window, and clock-skew bounds on the classifier.

* docs(alerts): add troubleshooting entries for crash detection toggle and rate limits

backend/src/services/ContainerLifecycleClassifier.ts

@@ -14,6 +14,14 @@ export type Classification = 'intentional' | 'clean' | 'crash' | 'oom';
 /** Window (ms) after a `kill` event within which a subsequent `die` is considered intentional. */
 export const INTENTIONAL_KILL_WINDOW_MS = 60_000;
 
+/**
+ * Maximum negative age (ms) tolerated when matching a kill to a later die.
+ * Absorbs small out-of-order deliveries and minor clock skew, but rejects
+ * wildly future-dated Docker timestamps that could otherwise flip a genuine
+ * crash into an "intentional" classification.
+ */
+export const MAX_NEGATIVE_SKEW_MS = 10_000;
+
 export interface ContainerLifecycleState {
     /** Timestamp (ms) of the most recent `kill` event for this container, if any. */
     lastKillAt?: number;
@@ -44,11 +52,12 @@ export function classifyDie(
     if (state.oomPending) return 'oom';
 
     if (typeof state.lastKillAt === 'number') {
-        // Use absolute age so out-of-order deliveries (kill arrives slightly
-        // after die) still classify as intentional. DockerEventService's 500ms
-        // die grace window makes this realistically bounded.
-        const age = Math.abs(input.at - state.lastKillAt);
-        if (age <= INTENTIONAL_KILL_WINDOW_MS) {
+        // Use signed age so a die with a wildly future-dated timestamp
+        // (clock skew / bad container clock) cannot match a past kill.
+        // Allow modest negative skew so out-of-order deliveries still
+        // classify as intentional within DockerEventService's grace window.
+        const age = input.at - state.lastKillAt;
+        if (age >= -MAX_NEGATIVE_SKEW_MS && age <= INTENTIONAL_KILL_WINDOW_MS) {
             return 'intentional';
         }
     }