The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
The pct exec socket probe that discovers Docker inside LXC guests
failed invisibly: errors logged at debug only, and a failed probe never
set DockerCheckedAt, so the guest stayed in first-check state and was
re-probed on every poll forever. On a node where lxc-attach could not
enter unprivileged guests that meant ~49 pct execs per minute of
steady-state churn, with nothing at default log level to say why the
Proxmox page showed no Docker.
Track consecutive probe failures per guest in the monitor. The first
failure of a streak, and the point where the backoff reaches its cap,
log at warn with the guest, node, and error; repeats stay at debug.
Failed probes retry with exponential backoff from one minute to a
thirty-minute ceiling, a success clears the streak with an info log,
and reconfiguring the checker resets all streaks so command execution
enabled from the UI at runtime still retries immediately. Entries not
refreshed for a day belong to deleted guests and are pruned.
PULSE_PROXMOX_GUEST_DOCKER_INVENTORY_VMIDS now gates the socket probe
as well as inventory collection. It previously only limited inventory,
so opting into specific guests still probed every running LXC on every
cycle; now guests outside the allowlist are never pct exec'd at all.
Subsystem contract and configuration docs updated to match.
Force stale negative LXC Docker detections to re-run after the Docker checker is configured so explicit Proxmox guest Docker inventory repopulates after backend restarts.
The Proxmox-side LXC Docker inventory poll occasionally produces an
ok=true report with zero CONTAINER lines but a valid PS_OK marker:
docker ps -a inside the LXC genuinely returned an empty list for
that one call, typically while watchtower is briefly recreating a
container or the dockerd is mid-restart. ApplyDockerReport then
wipes the host's container list, every connected WebSocket client
sees every row for that host flash blank, and the next successful
poll 10 s later restores it.
Before calling ApplyDockerReport, check whether the host already
has containers tracked in state. If yes and the new report has
zero containers, log at debug and skip the apply, counting the
poll as skipped rather than collected. A host that is genuinely
empty still applies on first sight (no previous containers means
no previous state to protect).
Add regression tests for both branches: a populated host receiving
an empty report keeps its container list; a freshly-seen empty
host still gets created.
The Proxmox-LXC Docker hosts row was blank (no CPU / MEM-used /
Disk) and the display name appended a noisy "(LXC <vmid>)" suffix,
because the inventory script only collects Docker daemon metadata,
not host resource usage. Pulse already polls the same data for the
underlying LXC via the PVE cluster/resources endpoint, so the
Docker host row should mirror what the rest of Pulse already knows.
Before applying each LXC inventory report, enrich the Host info
with CPU / Memory / Disk / Uptime taken from the source LXC's
existing models.Container. Drop the "(LXC <vmid>)" suffix from
the synthesized display name. On the frontend, detect the
proxmox-lxc-docker: hostSourceId prefix and emit a styled
"LXC <vmid>" badge into the System column instead of the
filtered-out runtime-only "docker" badge.
The Proxmox-side LXC Docker inventory script masked `docker ps`
failures with `|| true`, so when `pct exec` was truncated, slow, or
`docker ps` itself transiently failed, the parser produced an
ok=true report with `containers=[]` and `ApplyDockerReport` blanked
the host's container list. Containers then reappeared on the next
successful poll, producing the visible flicker on the Docker page.
Emit a `PS_OK` marker only when `docker ps` exits cleanly, and
refuse to apply the report when the marker is absent. A genuine
"host has zero containers" state still applies (PS_OK present,
zero CONTAINER lines), but a truncated or failed poll now skips
apply and lets the previous container list survive the inter-poll
gap. Add regression tests for both paths.
Adds automatic Docker detection for Proxmox LXC containers:
- New HasDocker and DockerCheckedAt fields on Container model
- Docker socket check via connected agents on first run, restart, or start
- Parallel checking with timeouts for efficiency
- Caches results and only re-checks after state transitions
This enables the AI to know which LXC containers are Docker hosts
for better infrastructure guidance.
rcourtman