The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
The rejection told operators to re-run the agent installer, but the
token minted by the original install command cannot gain the exec scope
after the fact and install tokens are single use, so re-running the
same command loops forever (issues #1586, #1564). Name the actual
recovery step instead.
Contract-Neutral: reworded operator-facing rejection copy (#1586, #1564); no behavior or contract change
RecoverExecutingActions existed with full test coverage but had no
production caller, so any typed action mid-dispatch across a server
restart (container update, start/stop/restart, host update, storage
cleanup) stayed in the executing state forever and sat in the Actions
inbox as live work, even after the agent persisted its terminal durable
receipt. Reproduced live on the dev instance with a Docker container
update (act_bf77dfe860ad3d8e4e0a91dc8eb83b44).
The router now runs a bounded, serialized recovery pass per organization
from a startup background worker, and again whenever an agent
(re)registers on the agentexec command server via a new registration
notifier, because a receipt-pending attempt can only be reconciled while
the owning agent is connected. Both triggers reuse the existing
query-only reconciliation semantics; nothing gains a resend authority.
Task 07 owns this residual; the api-contracts and agent-lifecycle
subsystem contracts now record the production trigger. The
rg-07-durable-delivery gate suite stays green, and a new router-level
test pins that a receipt-pending executing action completes from the
agent receipt without a second dispatch.
v6.1.0-rc.1 retired the legacy update endpoints before a replacement
existed, so the UI's Update button failed with an internal-jargon 410
(issue #1564). This lands the replacement end to end: update_container
is a typed agentexec operation with its own strict codec, durable
receipts, and a request digest bound to the image digest the plan
observed; the unified agent bridges execution to the Docker module's
existing pull/backup/recreate/verify/rollback implementation (which now
reports rollback attempt and outcome); and the container action
executor plans, dispatches, and reconciles the operation with declared
backup/rollback compensation truth. Containers advertise an
admin-approval update capability while an image update with a stated
current digest is detected. The legacy endpoints stay retired but
return actionable copy.
Proven live against a Colima daemon: single-container update, the
issue-1564 shared-network-namespace update, and the full UI journey
(Update button, governed review, approve, run) all completed with the
namespace preserved and the backup retained.
An agent enrolled for metrics but whose token the server doesn't recognise (or
that lacks the agent:exec scope, or is bound to a different agent) was rejected
on the command-exec WebSocket with a bare 'Invalid token' and — for the
token-not-found case — no server log at all. The agent then retried forever,
logging only 'Invalid token', so the operator had no signal that discovery
deep-scan was failing or why. (Confirmed live: delly/minipc agents pointed at a
backend that didn't recognise their token retried thousands of times; discovery
abstained for every guest as a result.)
- agentexec/server.go: the registration-rejection message the agent logs
verbatim now says 'agent token not authorized for command execution — re-run
the agent installer to enroll an agent:exec-scoped token'.
- api/agent_exec_token_binding.go: the previously-silent token-not-recognised
branch now logs the specific reason with the agent hostname.
Contract-neutral: same rejection behaviour, just legible. Regression test:
TestHandleWebSocket_RejectionMessageIsActionable. Verified live end-to-end.
Discovery wraps every probe in `docker exec <container> sh -c '...'`.
The agentexec command policy lists `^docker\s+exec\s` as RequireApproval
(a sound default for user-driven docker exec) and Discovery has no path
to mint or supply an ApprovalID. Result: every probe was rejected, the
scanner returned empty CommandOutputs, and the AI fell back to
"Unknown Infrastructure Resource" at confidence 0. The Discovery sub-tab
rendered empty after a "successful" run.
Add a Trusted bool to ExecuteCommandPayload on both the server-facing
agentexec type and the agent's wire struct. When set, the approval gate
is skipped on both ends and the server does not attempt to auto-mint an
approval grant (which would fail with "approval id is required").
PolicyBlock still applies; this is not a way to run arbitrary commands.
Only the discoveryCommandAdapter sets Trusted=true. The flag is never
populated from a deserialised HTTP body or any user-driven path. Patrol
fixes, Assistant remediation, and AI tool calls continue to flow through
the governed approval-record path with a real ApprovalID.
Contracts: amend agent-lifecycle Completion Obligations and Current
State to document the lone exception to the on-agent approval rail, and
amend ai-runtime to fence the Trusted flag to the discovery adapter
only.
1. Enforce monitoring:read scope on WebSocket upgrades
- Prevents low-privilege tokens (e.g. host-agent:report) from accessing
full infra state via requestData on the main WebSocket.
2. Enforce agent token binding to prevent impersonation
- Added Metadata field to APITokenRecord to support bound_agent_id
- Updated agentexec server to validate token-to-agent binding if present
- Prevents agent:exec tokens from registering as arbitrary agent IDs
- Move deadline/pong handler setup BEFORE registering agent in map
- Use writeMu mutex consistently for all WebSocket writes
- Prevents race between registration response and ExecuteCommand calls
- Fixes flaky TestExecuteCommand_RoundTripViaWebSocket in CI
Keep only the simple AI-powered approach:
- set_resource_url tool lets AI save discovered URLs
- Users ask AI directly: 'Find URLs for my containers'
- AI uses its intelligence to discover and set URLs
Removed:
- URLDiscoveryService (rigid port scanning)
- Bulk discovery API endpoints
- Frontend discovery button
The AI itself is smart enough to iterate through resources
and discover URLs when asked.
- Add AI service with Anthropic, OpenAI, and Ollama providers
- Add AI chat UI component with streaming responses
- Add AI settings page for configuration
- Add agent exec framework for command execution
- Add API endpoints for AI chat and configuration
rcourtman