An action that reached executing had exactly one way out: a terminal
operation receipt from the agent. Expiry skips executing rows on purpose,
and reconciliation bailed for every non-terminal query answer, so a Docker
update whose agent restarted mid-pull sat in executing forever with no
operator route out of it.
The agent-side receipt store rewrites accepted and started receipts to
interrupted on every Open(), and nothing can move an interrupted or
tombstoned receipt back to terminal. An identity-correlated answer of
either kind is therefore proof the operation will never report, and the
three executors now settle the action immediately as inconclusive with a
message telling the operator Pulse cannot confirm the effect and the
resource needs checking by hand.
A not_found answer, or a receipt still merely accepted or started, may
still be completed, so those keep waiting and only settle once the
dispatch attempt is older than one hour, the same threshold the
pulse-intelligence telemetry already uses to call an executing action
stuck. Every typed operation timeout is far shorter than that, so an
in-flight mutation is never cut short. A transport error answering the
query is still not evidence and preserves receipt_pending unchanged.
RecoverExecutingActions drives all of this on the existing two-minute
recovery loop, so rows already wedged before the upgrade heal themselves
without anyone touching them.
For the residue that reconciliation cannot reach, an agent that was
reinstalled or a legacy executing row with no dispatch attempt, POST
/api/actions/{id}/force-fail writes the same inconclusive terminal truth
under an operator attribution. It never touches the transport, refuses
anything already terminal, and is gated on admin plus settings:write on
top of the execute capability check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A routine Docker container update took five clicks across two in-row
confirmations and a two-phase Approve then Run dialog, and the review
screen said "Rollback declared: No" while its own intended-change text
promised automatic backup and rollback (#1564 feedback).
- Capabilities can declare rollback support; the planner carries it
into plan.rollbackAvailable instead of hard-coding false, so the
container update review now truthfully says rollback is available.
- Container update is classed AutoAuthorizeLowRisk (it ships backup and
rollback; restart was already low_risk). The action projection
exposes the read-time capability class plus resolved blast-radius
names, none of which enter plan identity.
- The review dialog collapses pending approval for low-risk
capabilities into one Approve and run control that records the
approval and dispatches execution from a single confirmation; other
capabilities keep the two-phase flow. Blast radius renders resource
names with IDs demoted.
- The in-row Update button plans on first click; the dialog is the
confirmation surface, so the Confirm hop and its cancel affordance
are gone.
Routine update ceremony is now Update, Approve and run, Close, with
the durable action record unchanged.
Production telemetry (30d) showed 18 of 26 approved governed-action
attempts never reaching verified success. The dominant stranding path:
HandleExecuteAction dispatched under the HTTP request context, so a
reverse proxy or browser dropping the connection mid-operation (docker
image pulls run minutes; proxies cut at 60-100s) cancelled the committed
dispatch and left the action executing forever — recovery only ran at
restart or agent re-registration, which never fires while the agent
stays healthily connected.
- Execute now dispatches under context.WithoutCancel: a committed
dispatch runs to its terminal state regardless of the caller's
connection, bounded by the per-operation transport timeout.
- A standing 2-minute recovery loop re-drives durable-dispatch
reconciliation so any abandoned executing action (timeout, send
failure, crash) converges from the agent's durable receipt instead
of waiting for a reconnect that may never come.
Verified live: docker-update executed with the client aborting at 3s;
the agent completed pull+recreate+verify and the action reached
completed/verified with the replacement container running.
Contract-Neutral: behavioral fix: detach approved-action dispatch from client ctx + periodic recovery loop; no public contract delta
Proposals now require their full correlation identity before anything
persists: a Submit without a finding ID or investigation ID is refused,
so a planned action can never lose the deterministic link back to its
Patrol finding. Pinned in the plan-only broker contract test.
The persisted-state transition callback becomes org-scoped
(OnActionTransition func(orgID, record)) and is wired through
ResourceHandlers.SetActionTransitionPublisher into the shared lifecycle
service, so a multi-tenant Patrol reconciler can key per-tenant stores
and can never apply a transition to the wrong tenant. Publication still
strictly follows persistence; code-standards pins guard the org-scoped
signature and wiring.
pkg/aicontracts/action_broker.go gains machine ownership: a shared
ai-runtime/api-contracts registry boundary (owned_files plus a sorted
shared_ownerships entry) with path policies proving through
pkg/aicontracts/contracts_test.go, and Shared Boundaries entries
inserted in canonical sorted order in both contracts.
Slice 2b-1 of the typed-lifecycle ratchet; the Patrol reconciler itself
lands with the orchestrator wiring now that ai_handlers.go is free.
Planning, approval decisions, and execution for typed resource actions
move out of the HTTP handlers in internal/api/actions.go into a new
internal/actionlifecycle.Service owned by api-contracts. The REST
handlers become thin decode/actor/error-mapping adapters over the one
shared service, and ResourceHandlers.ActionLifecycle() exposes the same
service for in-process consumers, so a future Patrol action broker
inherits identical resource lookup, availability checks, plan hashing,
audit persistence, remediation locks, plan-drift revalidation,
execution, and terminal publication instead of loopback HTTP or a
parallel lifecycle.
Behavior is preserved: same status codes, error codes, and audit/
lifecycle persistence ordering, backed by the existing api contract
tests plus new fail-closed proofs for the service itself (unknown
resource/capability, availability refusal, unapproved execution,
remediation lock, plan drift, missing executor, missing store).
Contract text in api-contracts, agent-lifecycle, and storage-recovery
now names the service alongside actions.go and planner.go; the
subsystem registry owns internal/actionlifecycle/ under api-contracts
with a dedicated path policy; the code-standards and contract source
pins follow the moved invariants; and the subsystem_lookup line-number
pin shifts with the api-contracts canonical-files list insertion.
This is the first slice of making the typed action lifecycle the only
autonomous execution route for Patrol, Assistant, and MCP.
Manifest-backed MCP tools, prompts, and resources with surface affordance contracts; agent capability manifest and governance projection; API contract tests and capability route projection; operations-loop and intelligence-funnel telemetry; release-control subsystem documentation, registry, and tooling; licensing and configuration.
Ensure dry-run-only actions fail with action_dry_run_only before executor availability checks, and bridge action completion verification projection through router payload mapping.
Closes the last known gap in the agent substrate. The three
action endpoints (POST /api/actions/plan, /api/actions/{id}/decision,
/api/actions/{id}/execute) previously emitted the platform-wide
APIError shape (stable code under "code", human under "error").
The agent surface uses the inverted shape (stable code under
"error", human under "message"), so adding action capabilities
to the manifest as-is would have forced agents to remember which
envelope each capability uses.
The slice refactors actions.go to emit the agent-stable envelope
across all 42 writeErrorResponse call sites. writeJSONError gains
a writeJSONErrorWithDetails sibling so the 13 calls that pass
field-level reasons (validation failures) preserve that
information under a new optional `details` field. The action
endpoints' JSON shape becomes:
{"error": "<stable_code>", "message": "<human>",
"details"?: {"<field>": "<reason>"}}
Frontend impact: zero. Verified that no frontend code consumes
the three action endpoints; the refactor is API-only.
Three new manifest entries (plan_action, decide_action,
execute_action) under a new "action" category, with their
declared error codes pinned per capability. Internal-failure 5xx
codes (audit-store outages, encode failures) are not declared
per capability; agents branch on 5xx generically.
TestContract_AgentSurfaceErrorCodesMatchManifestDeclarations now
audits actions.go alongside the existing two handler files, with
a documented internal-only allowlist for the 5xx codes.
The TestAgentSubstrate_ActionEndpointsEmitAgentStableEnvelope e2e
test exercises one error path through each endpoint via the actual
HTTP boundary, asserting the agent-stable envelope reaches the
wire and the legacy APIError fields (code, status_code, timestamp)
do NOT — drift back would mean the refactor regressed.
The TestContract_ActionDryRunOnlyExecutionErrorJSONSnapshot pin
is updated to match the new envelope shape; the manifest's
category allowlist gains "action".
api-contracts.md documents the new envelope (with details map),
the action governance loop's place in the substrate, the
ai:execute scope distinction from monitoring:write, and the
"manifest projection has a footnote" trade-off: bringing an
existing endpoint into the agent surface may require migrating
its error envelope, but the substrate keeps a single envelope
contract rather than carrying a translation wrapper layer.
agent-lifecycle.md and storage-recovery.md document the action
surface joining the agent paradigm and its zero-new-persistence
posture respectively. AGENT_SUBSTRATE.md's "what it does not do
yet" no longer lists the action surface; it now reflects the
real outstanding items (consumer feedback, an in-Pulse agent
integrations panel, a distribution path for pulse-mcp).
Richard Courtman