An action that reached executing had exactly one way out: a terminal
operation receipt from the agent. Expiry skips executing rows on purpose,
and reconciliation bailed for every non-terminal query answer, so a Docker
update whose agent restarted mid-pull sat in executing forever with no
operator route out of it.
The agent-side receipt store rewrites accepted and started receipts to
interrupted on every Open(), and nothing can move an interrupted or
tombstoned receipt back to terminal. An identity-correlated answer of
either kind is therefore proof the operation will never report, and the
three executors now settle the action immediately as inconclusive with a
message telling the operator Pulse cannot confirm the effect and the
resource needs checking by hand.
A not_found answer, or a receipt still merely accepted or started, may
still be completed, so those keep waiting and only settle once the
dispatch attempt is older than one hour, the same threshold the
pulse-intelligence telemetry already uses to call an executing action
stuck. Every typed operation timeout is far shorter than that, so an
in-flight mutation is never cut short. A transport error answering the
query is still not evidence and preserves receipt_pending unchanged.
RecoverExecutingActions drives all of this on the existing two-minute
recovery loop, so rows already wedged before the upgrade heal themselves
without anyone touching them.
For the residue that reconciliation cannot reach, an agent that was
reinstalled or a legacy executing row with no dispatch attempt, POST
/api/actions/{id}/force-fail writes the same inconclusive terminal truth
under an operator attribution. It never touches the transport, refuses
anything already terminal, and is gated on admin plus settings:write on
top of the execute capability check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two diagnosability gaps found by exercising the docker-update vertical
live with a real agent, each of which turns 'approved action cannot
execute' into a dead end with no visible cause:
- The durable operation-receipt store failing to open (e.g. unwritable
state dir) silently registered the agent with receipt version 0; the
only symptom was a server-side claim that the agent was too old. The
agent now logs the real cause and state dir loudly at startup.
- The server readiness copy asserted 'older agent version' for any
receipt-version mismatch. It now names both causes (old version or
unavailable state directory) and points at the agent logs.
(The related docker collect-cycle watchdog landed separately in
a0f75b1bb with a hung-daemon reproducer.)
Contract-Neutral: agent-side diagnosability: receipt-store failure warning + honest readiness copy; no public contract delta
'This agent does not support durable typed-operation receipts' told users
nothing they could act on. Say the agent is on an older version and how to
resolve it. Applies to docker container, host update and host storage
cleanup executors.
v6.1.0-rc.1 retired the legacy update endpoints before a replacement
existed, so the UI's Update button failed with an internal-jargon 410
(issue #1564). This lands the replacement end to end: update_container
is a typed agentexec operation with its own strict codec, durable
receipts, and a request digest bound to the image digest the plan
observed; the unified agent bridges execution to the Docker module's
existing pull/backup/recreate/verify/rollback implementation (which now
reports rollback attempt and outcome); and the container action
executor plans, dispatches, and reconciles the operation with declared
backup/rollback compensation truth. Containers advertise an
admin-approval update capability while an image update with a stated
current digest is detected. The legacy endpoints stay retired but
return actionable copy.
Proven live against a Colima daemon: single-container update, the
issue-1564 shared-network-namespace update, and the full UI journey
(Update button, governed review, approve, run) all completed with the
namespace preserved and the backup retained.
Richard Courtman