Use authoritative Docker OOM evidence

This commit is contained in:
rcourtman
2026-07-15 17:55:26 +01:00
parent e499f92360
commit e3382c8bcb
32 changed files with 234 additions and 81 deletions
@@ -170,6 +170,12 @@ percent because that is the collector's source evidence. Lifecycle surfaces may
transport that report, but must not reinterpret it as host-capacity utilization;
monitoring and unified resources own the normalized CPU contract used by
history, alerts, and canonical app-container metrics.
The same module authors the container OOM evidence boundary from Docker inspect.
Current agents must serialize a non-null `OOMKilled` boolean for inspected
containers, preserving explicit false; absence is reserved for older or
reduced-fidelity report producers. Lifecycle transport must not synthesize OOM
state from exit code 137 or collapse absent and false, because monitoring and
alerts depend on that distinction to fail closed.
Inside-guest Docker / Podman visibility is also a privacy boundary: full
Docker / Podman inventory may come from a guest-local agent or another explicit
guest reporting path. LXC Docker inventory may also come from the Proxmox host
@@ -21,6 +21,12 @@ Docker and Podman container CPU thresholds evaluate host-capacity-normalized
CPU percent, not Docker's runtime-native per-core percent. Alert metadata may
carry the raw per-core value and reporting host CPU count for evidence, but the
threshold value and canonical `cpuPercent` metadata remain normalized.
Docker and Podman OOM alerts require authoritative runtime evidence: the
container must be stopped (`exited` or `dead`) and its reported `OOMKilled`
state must be explicitly true. Exit code 137 alone is only SIGKILL evidence;
explicit false and unavailable/legacy OOM state both fail closed without an OOM
alert. Recovery clears an existing OOM alert when the authoritative predicate
is no longer true.
## Canonical Files
@@ -421,7 +427,10 @@ Docker alert evaluation now lives in `internal/alerts/docker.go`. That file
owns Docker host connectivity, container state and health, container metric
projection, service gap/update-state checks, image-update timing, and Docker
tracking cleanup; future Docker alert behavior should extend that resource
checker owner rather than expanding the central Manager file.
checker owner rather than expanding the central Manager file. It must not keep
shadow last-exit-code state or infer an OOM kill from exit 137; the accepted
container model's nullable runtime-authored `OOMKilled` field is the sole OOM
classification input.
PBS alert evaluation now lives in `internal/alerts/pbs.go`. That file owns PBS
connectivity normalization, PBS metric projection, PBS metric cleanup, and PBS
offline lifecycle handling; future PBS alert behavior should extend that
@@ -61,6 +61,14 @@ per-core CPU percent, but monitoring-owned history and alert threshold
evaluation use host-capacity-normalized CPU percent when host CPU capacity is
known. Raw runtime CPU remains alert/resource metadata, not the canonical
threshold value.
Docker and Podman container OOM state is runtime-authored evidence, not an
exit-code inference. Current agents must publish Docker inspect's `OOMKilled`
boolean for every inspected container, including explicit `false`; monitoring
must preserve that nullable boolean through report ingest, internal/frontend
models, and unified resources. An absent value means an older or reduced-fidelity
report did not provide the evidence and must remain distinguishable from both
confirmed OOM and confirmed non-OOM state. Exit code 137 proves only SIGKILL and
must not be promoted into OOM truth by monitoring.
Proxmox read-state rehydration is the inverse boundary: canonical
unified-resource CPU metrics are 0..100 percentages, while legacy
`models.Node.CPU`, `models.VM.CPU`, and `models.Container.CPU` remain Proxmox
@@ -216,6 +224,11 @@ resource health.
and Docker container CPU alerts must pass through the shared normalized
capacity helper so an 80% threshold means 80% of the reporting host capacity,
not 0.8 of one core on a multi-core host.
Container OOM evidence must come from the inspected runtime state. The report
wire field is nullable for compatibility with older agents, but a current
collector must set it to the exact Docker inspect boolean even when false;
report ingest and model conversion must clone and preserve the pointer so
concurrent state replacement cannot alter previously accepted evidence.
8. Add or change Proxmox Ceph compatibility payload decoding through `pkg/proxmox/ceph.go`
9. Add or change Proxmox ZFS compatibility payload decoding and vdev-role normalization through `pkg/proxmox/zfs.go`
10. Add or change mock chart synthesis, seeded history continuity, or mock-owned
@@ -115,6 +115,11 @@ storage/recovery-adjacent only because `DockerData` is a shared resource
payload. Raw per-core CPU evidence and normalized capacity CPU must not be used
as backup coverage, restore readiness, storage-health, or recovery-point
signals.
The same dependency rule applies to nullable Docker `OOMKilled` evidence.
Storage and recovery consumers may preserve and display it as runtime workload
context, but must not reinterpret confirmed OOM, confirmed non-OOM, missing
evidence, or exit code 137 as backup damage, protection health, restore
readiness, or a recovery-point signal.
Local subscription-agent settings in `internal/api/ai_handlers.go` remain an
ai-runtime transport choice even when Patrol observes storage or recovery
@@ -1236,7 +1241,8 @@ recovery scope, or a storage/recovery-owned secret source.
only; they must not reinterpret it as backup freshness, recovery
point recency, or protection cadence.
Docker / Podman `DockerData` container lifecycle, Podman metadata, and
cumulative block I/O totals remain unified-resource runtime context.
cumulative block I/O totals, including nullable runtime-authored OOM state,
remain unified-resource runtime context.
Storage and recovery may use those fields only as workload description
when linking to an owning runtime/platform page; they must not reinterpret
container block I/O totals as backup throughput, recovery-point evidence,
@@ -528,7 +528,10 @@ engine storage rows must stay host-scoped with table proof hooks so browser
proof can distinguish a populated disk-usage tab from an empty fixture.
Runtime container detail payloads must preserve the agent-reported lifecycle
timestamps, Podman pod/compose/auto-update/user-namespace metadata, and
cumulative block I/O totals on `DockerData`; frontend detail summaries and
cumulative block I/O totals on `DockerData`. They must also preserve nullable,
runtime-authored `OOMKilled` evidence without converting absent state or exit
code 137 into a positive classification; typed Docker views must return an
independent copy of that value. Frontend detail summaries and
Docker page search consume those backend-authored fields before falling back to
legacy labels.
Docker network rows must consume canonical runtime attachment relationships,