The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
RecoverExecutingActions existed with full test coverage but had no
production caller, so any typed action mid-dispatch across a server
restart (container update, start/stop/restart, host update, storage
cleanup) stayed in the executing state forever and sat in the Actions
inbox as live work, even after the agent persisted its terminal durable
receipt. Reproduced live on the dev instance with a Docker container
update (act_bf77dfe860ad3d8e4e0a91dc8eb83b44).
The router now runs a bounded, serialized recovery pass per organization
from a startup background worker, and again whenever an agent
(re)registers on the agentexec command server via a new registration
notifier, because a receipt-pending attempt can only be reconciled while
the owning agent is connected. Both triggers reuse the existing
query-only reconciliation semantics; nothing gains a resend authority.
Task 07 owns this residual; the api-contracts and agent-lifecycle
subsystem contracts now record the production trigger. The
rg-07-durable-delivery gate suite stays green, and a new router-level
test pins that a receipt-pending executing action completes from the
agent receipt without a second dispatch.
1. Enforce monitoring:read scope on WebSocket upgrades
- Prevents low-privilege tokens (e.g. host-agent:report) from accessing
full infra state via requestData on the main WebSocket.
2. Enforce agent token binding to prevent impersonation
- Added Metadata field to APITokenRecord to support bound_agent_id
- Updated agentexec server to validate token-to-agent binding if present
- Prevents agent:exec tokens from registering as arbitrary agent IDs
Richard Courtman