fix(docker-events): harden crash detection against edge cases (#591)

* fix(docker-events): harden crash detection against edge cases

- Isolate per-container failures in the reconcile path so one failed
  container inspect cannot abort classification for the rest of the
  batch after a reconnect.
- Fall back to inspecting State.OOMKilled when a container exits with
  code 137 and no oom event preceded the die, so cgroup OOM kills that
  lose the oom event are still classified correctly. The dedup check
  runs before the fallback so crashloops do not hammer the daemon.
- Guard the intentional-kill window against wildly future-dated Docker
  timestamps by switching to a signed age comparison with a bounded
  negative-skew tolerance, so clock skew cannot flip a genuine crash
  into an intentional stop.
- Align diagnostic logging with the codebase [Service:diag] convention
  and add one informational lifecycle log on boot and shutdown so
  operators running in developer mode can confirm the watcher started.
- New tests cover gap-inspect isolation, the OOM inspect fallback (both
  success and failure paths), duplicate die events collapsing within
  the grace window, and clock-skew bounds on the classifier.

* docs(alerts): add troubleshooting entries for crash detection toggle and rate limits
This commit is contained in:
Anso
2026-04-14 14:37:28 -04:00
committed by GitHub
parent f062fa6aa2
commit 44a89d9d2e
6 changed files with 224 additions and 17 deletions
@@ -10,6 +10,7 @@ import {
classifyDie,
classifyGapExit,
INTENTIONAL_KILL_WINDOW_MS,
MAX_NEGATIVE_SKEW_MS,
} from '../services/ContainerLifecycleClassifier';
describe('classifyDie', () => {
@@ -87,6 +88,28 @@ describe('classifyDie', () => {
);
expect(result).toBe('crash');
});
it('rejects a die with a wildly future-dated timestamp (clock skew)', () => {
// Die timestamp 120s in the future while the kill happened "now":
// a real out-of-order delivery is bounded by the 500ms grace window,
// so 120s of future skew must not match an earlier intentional kill.
const result = classifyDie(
{ at: now + 120_000, exitCode: 1 },
{ lastKillAt: now },
);
expect(result).toBe('crash');
});
it('accepts a die with modest negative skew within the tolerance window', () => {
// Kill happened slightly after the die (bounded out-of-order delivery
// plus normal clock skew, under MAX_NEGATIVE_SKEW_MS). Still intentional.
const skew = MAX_NEGATIVE_SKEW_MS - 1_000;
const result = classifyDie(
{ at: now, exitCode: 1 },
{ lastKillAt: now + skew },
);
expect(result).toBe('intentional');
});
});
describe('classifyGapExit', () => {