mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 18:45:53 +00:00
rcourtman
4c7b1a2434
The guest Docker socket probe hung minipc hard enough to need a power cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and pveproxy starved. Three bugs chained, each fixed here: 1. Dispatcher re-issued a probe while the previous one was still executing. The poll cycle's enrichment context had expired, so ExecuteCommand dispatched, returned the context error 50ms later, and the next 3s cycle sent the identical command again — unbounded concurrency against a host that was slow to begin with. The monitoring dispatcher now takes a per-guest in-flight claim before dispatching probe or inventory commands (completed probes release it; abandoned ones hold it for a 2-minute window), and both dispatch paths bail out under a dead context. 2. The host agent never got the July process-leak fix: 45480a5cc landed only on pulse/v6-release, so main-line agents killed just the direct shell on timeout, orphaning pct exec → lxc-attach children and blocking Wait on their inherited pipes (10s timeouts reported as 300s+ durations). Port it: run each command in its own process group, SIGKILL the group on cancel, bound Wait with WaitDelay, and treat ErrWaitDelay after a clean exit as success. 3. Server-side abandonment never reached the agent. ExecuteCommand and ReadFile now refuse to dispatch under an already-expired context, and send a best-effort cancel_command when they stop waiting; the agent cancels the in-flight execution (killing its process group) and reports "command canceled". Older agents ignore the unknown message type. Also add a per-node circuit breaker: three consecutive command failures on one node suspend all Docker probe/inventory dispatch to it on the existing 1m→30m backoff schedule, so a host-level stall (NFS flapping) stops the probing entirely instead of failing guest by guest. Regression tests simulate the storm without hardware: a never-returning executor is not re-issued across poll cycles, an expired context dispatches nothing and records no failure, abandoned probes hold their claim, the breaker blocks new guests on a failing node, and the agent kills the whole process group on timeout and on server-issued cancel. Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged