Refuse dispatch when action readiness is lost

This commit is contained in:
rcourtman
2026-07-20 21:44:59 +01:00
parent 1eaffa9838
commit 9f9e02e811
17 changed files with 420 additions and 25 deletions
+2 -2
View File
@@ -250,7 +250,7 @@ the live set.
- `plan_action` (Plan action, `POST /api/actions/plan`, scope `actions:plan`, mode `write`, approval `action_plan`): Plan an action against a resource. The planner validates the request, looks up the capability on the resource, checks executor-owned live availability, and returns an ActionPlan with the approval policy, blast radius, plan hash, and preflight summary. The plan is persisted to the audit history at the planned/pending state only after the live availability check passes, so subsequent decide_action and execute_action calls can reference it by id. Plan-and-execute is a two-step flow when the resulting plan requires approval, one-step otherwise.
- `decide_action` (Decide action, `POST /api/actions/{actionId}/decision`, scope `actions:approve`, mode `write`, approval `action_plan`): Record an approval decision (approved or rejected) on a previously planned action. The actor is taken from the authenticated identity; an explicit reason can be passed in the body. An exact retry returns the authoritative persisted decision without adding an approval or lifecycle event; a conflicting retry fails closed.
- `execute_action` (Execute action, `POST /api/actions/{actionId}/execute`, scope `actions:execute`, mode `write`, approval `action_plan`): Execute a previously planned and (when required) approved action. Returns the persisted audit record with the execution result attached. Refuses with stable codes when the action is in the wrong lifecycle state (action_not_approved, action_already_executing, action_execution_final, action_dry_run_only, action_plan_expired), when the approved plan no longer matches the current resource/capability contract (action_plan_drift), when the target is operator-locked against automated remediation (resource_remediation_locked), or when the API instance has no executor wired (action_executor_unavailable). action.completed SSE events fire on every terminal state so agents watching the stream do not need to poll this endpoint after dispatch.
- `execute_action` (Execute action, `POST /api/actions/{actionId}/execute`, scope `actions:execute`, mode `write`, approval `action_plan`): Execute a previously planned and (when required) approved action. Returns the persisted audit record with the execution result attached. Refuses with stable codes when the action is in the wrong lifecycle state (action_not_approved, action_already_executing, action_execution_final, action_dry_run_only, action_plan_expired), when executor-owned live readiness is no longer available (action_execution_unavailable), when the approved plan no longer matches the current resource/capability contract (action_plan_drift), when the target is operator-locked against automated remediation (resource_remediation_locked), or when the API instance has no executor wired (action_executor_unavailable). Both human and automatic policy execution recheck readiness before dispatch admission. action.completed SSE events fire on every terminal state so agents watching the stream do not need to poll this endpoint after dispatch.
**Provisioning (infrastructure onboarding):**
@@ -321,7 +321,7 @@ Capability-specific stable codes are advertised by the manifest:
- `resolve_finding`: `invalid_finding_request`, `finding_not_found`, `finding_action_not_allowed`, and `patrol_unavailable`
- `plan_action`: `invalid_action_request`, `mock_mode_enabled`, `action_actor_unavailable`, `resource_not_found`, `capability_not_found`, and `action_execution_unavailable`
- `decide_action`: `mock_mode_enabled`, `missing_id`, `invalid_id`, `invalid_action_decision`, `action_not_found`, `action_not_pending`, `action_plan_expired`, `action_plan_identity_mismatch`, `action_actor_unavailable`, `action_approval_forbidden`, `action_step_up_unavailable`, `action_decision_conflict`, `action_separation_required`, and `action_replan_required`
- `execute_action`: `mock_mode_enabled`, `missing_id`, `invalid_id`, `invalid_action_execution`, `action_not_found`, `action_not_approved`, `action_already_executing`, `action_execution_final`, `action_dry_run_only`, `action_plan_expired`, `action_plan_drift`, `action_plan_identity_mismatch`, `resource_remediation_locked`, `action_executor_unavailable`, `action_actor_unavailable`, `action_execution_forbidden`, `action_not_executing`, and `action_replan_required`
- `execute_action`: `mock_mode_enabled`, `missing_id`, `invalid_id`, `invalid_action_execution`, `action_not_found`, `action_not_approved`, `action_already_executing`, `action_execution_final`, `action_dry_run_only`, `action_plan_expired`, `action_execution_unavailable`, `action_plan_drift`, `action_plan_identity_mismatch`, `resource_remediation_locked`, `action_executor_unavailable`, `action_actor_unavailable`, `action_execution_forbidden`, `action_not_executing`, and `action_replan_required`
<!-- pulse-mcp-errors:end -->
Cross-cutting codes from the auth / multi-tenant middleware
@@ -1955,6 +1955,20 @@ Agent` secondary handoff against the live setup wizard instead of relying
## Current State
### Governed action readiness remains outside agent lifecycle authority
The canonical Actions lifecycle may ask an executor-owned
`AvailabilityChecker` whether an already-planned capability is still reachable
immediately before human or automatic policy dispatch admission. Agent
connectivity and command-agent loss are read-only readiness evidence at this
boundary: an explicit unavailable result produces the stable
`action_execution_unavailable` refusal, a terminal failed action audit and
lifecycle event, and the normal action-completed publication without creating
a dispatch attempt or issuing an agent command. The check does not enroll,
reconnect, reconfigure, update, or otherwise mutate an agent, and it cannot
replace canonical planning, approval, policy authorization, dispatch receipt,
or verification.
Shared `internal/api/ai_handlers.go` now projects separate Patrol investigation
evidence-call and model-response budgets/counters. Those fields remain adjacent
AI-runtime/API evidence only: agent enrollment, dispatch attempts, liveness,
@@ -3901,6 +3901,18 @@ unassessed until deterministic remediation evals exist.
## Current State
### Agent surfaces expose execution-time readiness refusal
The canonical capability manifest and generated Pulse MCP documentation expose
`action_execution_unavailable` on `execute_action` as a stable permanent
refusal. Assistant and MCP clients may branch on that code after an approved
plan loses executor-owned live readiness, but an adapter cannot retry around
the terminal refusal, substitute a different capability, or dispatch through a
provider-local path. The bounded human reason and provider reason code are
diagnostic evidence only; the shared manifest remains the error-vocabulary
owner, and canonical Actions remains the sole mutation plane for both human and
automatic policy execution.
First-session assistant discoverability is now a contract concern. Successful
first-time provider setup (the Provider & Models setup modal) must open the
Assistant drawer and refresh the session `assistantEnabled` capability in
@@ -3500,6 +3500,23 @@ returning prompts, tool transcripts, credentials, or infrastructure data.
## Current State
### Action execution revalidates live readiness
`POST /api/actions/{actionId}/execute` routes through the transport-independent
Actions lifecycle, which revalidates the approved plan against the current
canonical resource and then asks the optional executor-owned
`AvailabilityChecker` for live readiness before entering `executing`, creating
a dispatch attempt, or calling the executor. The same gate applies to
`ExecuteUnderPolicy`, so an automatic broker cannot bypass it. A resource that
disappears remains `action_plan_drift`; an explicitly unavailable capability
returns HTTP `409` with shared code `action_execution_unavailable` and bounded
`resourceId`, `capabilityName`, `reasonCode`, and `reason` details. Pulse
persists a terminal failed/no-effect audit and lifecycle event and publishes
the normal completion notification. Executors without the optional checker,
and checkers returning an empty readiness result, preserve the existing
compatibility path. Registry or readiness-check infrastructure failures remain
nonterminal internal errors rather than false permanent refusals.
The public Patrol investigation boundary now carries independent
`max_turns` and `max_evidence_calls` request limits and returns `model_turns`,
`evidence_calls`, and total `tool_calls`. Persisted investigation sessions keep
@@ -1838,6 +1838,18 @@ that Safe auto-fix or Autopilot remediation is verified.
## Current State
### Recovery actions retain execution-time readiness checks
Recovery-oriented capabilities admitted to canonical Actions are subject to
the same execution-time `AvailabilityChecker` gate as every other governed
action. Losing executor or agent reachability after planning or approval
produces a persisted `action_execution_unavailable` terminal no-effect refusal
before a dispatch attempt for both human and automatic policy execution.
Recovery Assurance remains a separate deterministic domain: backup freshness,
protection posture, restore-chain evidence, and recoverability cannot satisfy
action authorization or live executor readiness, and the readiness refusal
cannot be rewritten as proof that a restore is or is not recoverable.
Shared `internal/api/ai_handlers.go` now projects separate Patrol investigation
evidence-call and model-response budgets/counters. Storage and recovery may use
those values only as investigation cost/load evidence; they do not establish
@@ -1538,6 +1538,25 @@ AI-only summary payloads, or page-local heuristics.
## Current State
### Governed action live-readiness refusal
The unified action model owns `ErrActionExecutionUnavailable` as a permanent
pre-dispatch refusal with canonical reason code
`action_execution_unavailable`. The transport-independent lifecycle rebuilds
the current canonical resource before asking the executor-owned readiness
checker; a missing resource remains plan drift, while an explicit unavailable
result records a bounded reason on the failed/no-effect audit and lifecycle
event. Empty readiness or an executor without the optional checker preserves
compatibility, but a checker that names the capability and reports it
unavailable fails closed before either human or automatic policy admission.
This refusal cannot alter the approved plan, select a replacement executor, or
grant mutation authority from resource health alone.
`internal/unifiedresources/actions_test.go`,
`internal/actionlifecycle/service_test.go`, `internal/api/actions_test.go`,
`internal/api/contract_test.go`, and
`internal/agentcapabilities/manifest_test.go` prove the domain transition,
both admission paths, HTTP/completion result, and shared manifest declaration.
Resource detail drawer discovery-tab Suspense fallbacks now compose the
frontend-primitives `DiscoveryLoadingFallback` template. `ResourceDetailDrawer`
and `ResourceDetailDrawerOverviewTab` own when the Discovery tab is available,
+86 -5
View File
@@ -39,8 +39,9 @@ type DispatchBinder interface {
BindActionDispatch(ctx context.Context, record unified.ActionAuditRecord, attempt unified.ActionDispatchAttempt) (unified.ActionDispatchAttempt, error)
}
// AvailabilityChecker lets an executor contribute live readiness checks
// before Pulse advertises or persists an executable action plan.
// AvailabilityChecker lets an executor contribute read-only live readiness
// checks before Pulse persists a plan and again immediately before dispatch
// admission. It must not mutate the target or its agent connection.
type AvailabilityChecker interface {
CheckActionAvailable(ctx context.Context, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness
}
@@ -269,8 +270,9 @@ func (e *CapabilityNotFoundError) Error() string {
func (e *CapabilityNotFoundError) Unwrap() error { return actionplanner.ErrCapabilityNotFound }
// AvailabilityRefusedError reports that the executor's live readiness check
// refused the action before a plan was persisted.
// AvailabilityRefusedError reports that an executor-owned live readiness
// check explicitly refused the action. During planning no audit is persisted;
// during execution the refusal is a permanent terminal no-effect result.
type AvailabilityRefusedError struct {
ResourceID string
CapabilityName string
@@ -282,9 +284,21 @@ func (e *AvailabilityRefusedError) Error() string {
if reason == "" {
reason = "action execution is unavailable"
}
return fmt.Sprintf("action %s on %s unavailable: %s", e.CapabilityName, e.ResourceID, reason)
return fmt.Sprintf("%s: %s", unified.ErrActionExecutionUnavailable, reason)
}
func (e *AvailabilityRefusedError) Unwrap() error { return unified.ErrActionExecutionUnavailable }
// AvailabilityCheckError wraps an infrastructure failure while obtaining the
// live resource or executor readiness. Explicit unavailability is represented
// by AvailabilityRefusedError and must not be wrapped as transient.
type AvailabilityCheckError struct{ Err error }
func (e *AvailabilityCheckError) Error() string {
return fmt.Sprintf("action execution availability check: %v", e.Err)
}
func (e *AvailabilityCheckError) Unwrap() error { return e.Err }
// PersistError wraps a storage write failure at a named lifecycle stage.
type PersistError struct {
Op string
@@ -945,6 +959,18 @@ func (s *Service) Execute(ctx context.Context, orgID, actionID string, actor uni
}
return unified.ActionAuditRecord{}, &PolicyCheckError{Err: err}
}
if err := s.ValidateExecutionAvailable(ctx, orgID, record); err != nil {
if unified.IsPermanentActionExecutionRefusal(err) {
failed, persistErr := RecordRefusedExecution(store, record, actorID, now, err)
if persistErr != nil {
return unified.ActionAuditRecord{}, &PersistError{Op: "unavailable action execution refusal", Err: persistErr}
}
s.publishTransition(orgID, failed)
s.publishCompleted(failed)
return failed, err
}
return unified.ActionAuditRecord{}, &AvailabilityCheckError{Err: err}
}
started, startEvent, err := unified.BeginActionExecution(record, actorID, now)
if err != nil {
@@ -1066,6 +1092,13 @@ func (s *Service) beginPolicyExecution(ctx context.Context, orgID, actionID, act
failed, refuseErr := s.refusePolicyAdmission(store, orgID, record, actor, now, unified.ErrActionPolicyAuthorizationInvalid)
return failed, store, false, refuseErr
}
if err := s.ValidateExecutionAvailable(ctx, orgID, record); err != nil {
if unified.IsPermanentActionExecutionRefusal(err) {
failed, refuseErr := s.refusePolicyAdmission(store, orgID, record, actor, now, err)
return failed, store, false, refuseErr
}
return unified.ActionAuditRecord{}, store, false, &AvailabilityCheckError{Err: err}
}
started, approvedEvent, startEvent, err := unified.BeginPolicyActionExecution(record, unified.ActionApprovalRecord{Actor: actor, Reason: reason}, lease, now)
if err != nil {
failed, refuseErr := s.refusePolicyAdmission(store, orgID, record, actor, now, err)
@@ -1309,6 +1342,54 @@ func (s *Service) ValidatePlanFresh(orgID string, record unified.ActionAuditReco
return nil
}
const (
actionAvailabilityReasonCodeMaxRunes = 128
actionAvailabilityReasonMaxRunes = 512
)
// ValidateExecutionAvailable resolves the current canonical resource and asks
// the optional executor-owned readiness checker immediately before admission.
// An absent checker or an empty readiness result preserves compatibility.
func (s *Service) ValidateExecutionAvailable(ctx context.Context, orgID string, record unified.ActionAuditRecord) error {
checker, ok := s.Executor.(AvailabilityChecker)
if !ok {
return nil
}
normalized, err := unified.NormalizeActionAuditRecord(record)
if err != nil {
return fmt.Errorf("%w: %v", unified.ErrActionPlanDrift, err)
}
registry, err := s.registry(orgID)
if err != nil {
return err
}
resource, ok := registry.Get(normalized.Request.ResourceID)
if !ok || resource == nil {
return fmt.Errorf("%w: resource %q is no longer present", unified.ErrActionPlanDrift, normalized.Request.ResourceID)
}
readiness := checker.CheckActionAvailable(ctx, normalized.Request, *resource)
readiness.Name = strings.TrimSpace(readiness.Name)
readiness.ReasonCode = boundedActionAvailabilityText(readiness.ReasonCode, actionAvailabilityReasonCodeMaxRunes)
readiness.Reason = boundedActionAvailabilityText(readiness.Reason, actionAvailabilityReasonMaxRunes)
if readiness.Name == "" || readiness.Available {
return nil
}
return &AvailabilityRefusedError{
ResourceID: normalized.Request.ResourceID,
CapabilityName: normalized.Request.CapabilityName,
Readiness: readiness,
}
}
func boundedActionAvailabilityText(value string, maxRunes int) string {
value = strings.TrimSpace(value)
runes := []rune(value)
if maxRunes <= 0 || len(runes) <= maxRunes {
return value
}
return strings.TrimSpace(string(runes[:maxRunes-1])) + "…"
}
// validateExecutionPolicy enforces operator-set per-resource policy at the
// dispatch decision point, currently the NeverAutoRemediate lock.
func validateExecutionPolicy(store Store, record unified.ActionAuditRecord) error {
+72
View File
@@ -555,6 +555,40 @@ func TestPolicyAdmissionCommitsApprovalAndExecutingAtomicallySQLite(t *testing.T
runPolicyAdmissionCommitsAtomically(t, store)
}
func TestExecuteUnderPolicyRefusesLostReadinessBeforeAutomaticAdmission(t *testing.T) {
now := time.Now().UTC()
store := unified.NewMemoryStore()
executor := &stubExecutor{
result: &unified.ExecutionResult{Success: true},
readiness: &unified.ResourceActionReadiness{Name: "restart", Available: true},
}
service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), executor)
service.Now = func() time.Time { return now }
plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
if err != nil {
t.Fatal(err)
}
executor.readiness = &unified.ResourceActionReadiness{
Name: "restart",
Available: false,
ReasonCode: "command_agent_disconnected",
Reason: "command agent disconnected",
}
failed, err := service.ExecuteUnderPolicy(context.Background(), "default", plan.ActionID, "pulse_patrol_policy", func(_ context.Context, record unified.ActionAuditRecord, at time.Time) (unified.ActionPolicyAuthorizationLease, string, error) {
return policyTestLease(record, at), "bounded policy", nil
})
if !errors.Is(err, unified.ErrActionExecutionUnavailable) || failed.State != unified.ActionStateFailed {
t.Fatalf("failed=%#v err=%v", failed, err)
}
if executor.calls != 0 || len(failed.Approvals) != 0 {
t.Fatalf("executor calls=%d approvals=%#v, want no dispatch or policy approval", executor.calls, failed.Approvals)
}
if _, found, getErr := store.GetActionDispatchAttempt(plan.ActionID); getErr != nil || found {
t.Fatalf("dispatch attempt found=%v err=%v, want none", found, getErr)
}
}
func runPolicyBarrierRevocations(t *testing.T, storeFactory func(t *testing.T) unified.ResourceStore) {
cases := []struct {
name string
@@ -1175,6 +1209,44 @@ func TestExecuteRunsApprovedActionToTerminalAudit(t *testing.T) {
}
}
func TestExecuteRefusesLostReadinessBeforeAdmission(t *testing.T) {
now := time.Now().UTC()
env := newServiceEnv(t, testResource(now, unified.ApprovalNone))
env.executor.readiness = &unified.ResourceActionReadiness{Name: "restart", Available: true}
plan, err := env.service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
if err != nil {
t.Fatalf("Plan: %v", err)
}
env.executor.readiness = &unified.ResourceActionReadiness{
Name: "restart",
Available: false,
ReasonCode: "command_agent_disconnected",
Reason: "command agent disconnected",
}
failed, err := env.service.Execute(context.Background(), "default", plan.ActionID, testActionActor("requester", "default"), "")
var refused *AvailabilityRefusedError
if !errors.As(err, &refused) || !errors.Is(err, unified.ErrActionExecutionUnavailable) {
t.Fatalf("error = %v, want AvailabilityRefusedError wrapping ErrActionExecutionUnavailable", err)
}
if failed.State != unified.ActionStateFailed || failed.Result == nil ||
failed.Result.ActionResultV2 == nil ||
failed.Result.ActionResultV2.Execution.ReasonCode != "action_execution_unavailable" ||
!strings.HasPrefix(failed.Result.ErrorMessage, "action_execution_unavailable: command agent disconnected") {
t.Fatalf("failed audit = %#v", failed)
}
if env.executor.calls != 0 {
t.Fatalf("executor calls = %d, want no dispatch", env.executor.calls)
}
if len(env.completed) != 1 || env.completed[0].State != unified.ActionStateFailed {
t.Fatalf("completion publisher observed %#v", env.completed)
}
if _, found, getErr := env.store.GetActionDispatchAttempt(plan.ActionID); getErr != nil || found {
t.Fatalf("dispatch attempt found=%v err=%v, want none", found, getErr)
}
}
func TestExecuteRefusesUnapprovedActionWithoutDispatch(t *testing.T) {
now := time.Now().UTC()
env := newServiceEnv(t, testResource(now, unified.ApprovalAdmin))
+1
View File
@@ -34,6 +34,7 @@ const (
AgentErrCodeActionExecutionForbidden = "action_execution_forbidden"
AgentErrCodeActionNotExecuting = "action_not_executing"
AgentErrCodeActionDryRunOnly = "action_dry_run_only"
AgentErrCodeActionReadinessCheckFailed = "action_execution_availability_failed"
AgentErrCodeActionPlanDrift = "action_plan_drift"
AgentErrCodeActionPlanIdentityMismatch = "action_plan_identity_mismatch"
AgentErrCodeResourceRemediationLocked = "resource_remediation_locked"
+2 -1
View File
@@ -640,6 +640,7 @@ var (
AgentErrCodeActionExecutionFinal,
AgentErrCodeActionDryRunOnly,
AgentErrCodeActionPlanExpired,
AgentErrCodeActionExecutionUnavailable,
AgentErrCodeActionPlanDrift,
AgentErrCodeActionPlanIdentityMismatch,
AgentErrCodeResourceRemediationLocked,
@@ -1069,7 +1070,7 @@ var canonicalManifest = Manifest{
{
Name: ExecuteActionCapabilityName,
Title: "Execute action",
Description: "Execute a previously planned and (when required) approved action. Returns the persisted audit record with the execution result attached. Refuses with stable codes when the action is in the wrong lifecycle state (action_not_approved, action_already_executing, action_execution_final, action_dry_run_only, action_plan_expired), when the approved plan no longer matches the current resource/capability contract (action_plan_drift), when the target is operator-locked against automated remediation (resource_remediation_locked), or when the API instance has no executor wired (action_executor_unavailable). action.completed SSE events fire on every terminal state so agents watching the stream do not need to poll this endpoint after dispatch.",
Description: "Execute a previously planned and (when required) approved action. Returns the persisted audit record with the execution result attached. Refuses with stable codes when the action is in the wrong lifecycle state (action_not_approved, action_already_executing, action_execution_final, action_dry_run_only, action_plan_expired), when executor-owned live readiness is no longer available (action_execution_unavailable), when the approved plan no longer matches the current resource/capability contract (action_plan_drift), when the target is operator-locked against automated remediation (resource_remediation_locked), or when the API instance has no executor wired (action_executor_unavailable). Both human and automatic policy execution recheck readiness before dispatch admission. action.completed SSE events fire on every terminal state so agents watching the stream do not need to poll this endpoint after dispatch.",
Category: "action",
Method: http.MethodPost,
Path: ActionExecutionCapabilityPath,
@@ -512,6 +512,7 @@ func TestCanonicalManifestPinsPulseMCPResolvedOperationsLoopCapabilities(t *test
AgentErrCodeActionExecutionFinal,
AgentErrCodeActionDryRunOnly,
AgentErrCodeActionPlanExpired,
AgentErrCodeActionExecutionUnavailable,
AgentErrCodeActionPlanDrift,
AgentErrCodeActionPlanIdentityMismatch,
AgentErrCodeResourceRemediationLocked,
+14 -1
View File
@@ -28,7 +28,7 @@ const maxPendingActionAudits = 100
type ActionExecutor = actionlifecycle.Executor
// ActionAvailabilityChecker is the API-facing name for the canonical
// pre-plan readiness contract owned by internal/actionlifecycle.
// plan-time and pre-dispatch readiness contract owned by actionlifecycle.
type ActionAvailabilityChecker = actionlifecycle.AvailabilityChecker
type actionDecisionRequest struct {
@@ -650,15 +650,26 @@ func writeActionExecuteError(w http.ResponseWriter, err error) {
var persist *actionlifecycle.PersistError
var freshness *actionlifecycle.FreshnessCheckError
var policy *actionlifecycle.PolicyCheckError
var availability *actionlifecycle.AvailabilityRefusedError
var availabilityCheck *actionlifecycle.AvailabilityCheckError
switch {
case errors.Is(err, actionlifecycle.ErrExecutorUnavailable):
writeJSONError(w, http.StatusNotImplemented, agentcapabilities.AgentErrCodeActionExecutorUnavailable, "No action executor is configured for this API instance")
case errors.As(err, &availability):
writeJSONErrorWithDetails(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionExecutionUnavailable, "Action execution is unavailable", map[string]string{
"resourceId": availability.ResourceID,
"capabilityName": availability.CapabilityName,
"reasonCode": availability.Readiness.ReasonCode,
"reason": firstNonEmpty(availability.Readiness.Reason, "action execution is unavailable"),
})
case errors.As(err, &persist):
writeActionExecutionPersistError(w, err)
case errors.As(err, &freshness):
writeJSONError(w, http.StatusInternalServerError, "action_plan_validation_failed", sanitizeErrorForClient(err, "Failed to validate action plan freshness"))
case errors.As(err, &policy):
writeJSONError(w, http.StatusInternalServerError, "action_policy_validation_failed", sanitizeErrorForClient(err, "Failed to validate action policy"))
case errors.As(err, &availabilityCheck):
writeJSONError(w, http.StatusInternalServerError, agentcapabilities.AgentErrCodeActionReadinessCheckFailed, sanitizeErrorForClient(err, "Failed to validate action execution availability"))
default:
writeActionExecutionApplyError(w, err)
}
@@ -703,6 +714,8 @@ func writeActionExecutionApplyError(w http.ResponseWriter, err error) {
writeJSONError(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionPlanExpired, "Action plan has expired")
case errors.Is(err, unified.ErrActionDryRunOnly):
writeJSONError(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionDryRunOnly, "Action plan is dry-run only and cannot be executed")
case errors.Is(err, unified.ErrActionExecutionUnavailable):
writeJSONError(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionExecutionUnavailable, "Action execution is unavailable")
case errors.Is(err, unified.ErrActionPlanDrift):
writeJSONError(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionPlanDrift, "Action plan no longer matches the current resource contract; re-plan before executing")
case errors.Is(err, unified.ErrResourceRemediationLocked):
+124 -4
View File
@@ -63,10 +63,11 @@ func boundActionTestDecisionApproval(actionID, planHash, subject string, outcome
}
type stubActionExecutor struct {
result *unified.ExecutionResult
err error
received unified.ActionAuditRecord
calls int
result *unified.ExecutionResult
err error
readiness *unified.ResourceActionReadiness
received unified.ActionAuditRecord
calls int
}
func (s *stubActionExecutor) ExecuteAction(_ context.Context, record unified.ActionAuditRecord) (*unified.ExecutionResult, error) {
@@ -75,6 +76,13 @@ func (s *stubActionExecutor) ExecuteAction(_ context.Context, record unified.Act
return s.result, s.err
}
func (s *stubActionExecutor) CheckActionAvailable(_ context.Context, _ unified.ActionRequest, _ unified.Resource) unified.ResourceActionReadiness {
if s.readiness == nil {
return unified.ResourceActionReadiness{}
}
return *s.readiness
}
func TestHandlePlanActionBindsActorAndPlanHashToAuthenticatedOrg(t *testing.T) {
now := time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC)
h := newActionTestResourceHandlers(t, &config.Config{DataPath: t.TempDir()})
@@ -749,6 +757,118 @@ func TestHandleExecuteActionRunsApprovedPlanThroughExecutor(t *testing.T) {
}
}
func TestHandleExecuteActionReturnsStableLostReadinessRefusal(t *testing.T) {
now := time.Date(2026, 7, 20, 20, 0, 0, 0, time.UTC)
h := newActionTestResourceHandlers(t, &config.Config{DataPath: t.TempDir()})
h.SetStateProvider(resourceUnifiedSeedProvider{
snapshot: models.StateSnapshot{LastUpdate: now},
resources: []unified.Resource{
{
ID: "vm:42",
Type: unified.ResourceTypeVM,
Name: "web-42",
Status: unified.StatusWarning,
LastSeen: now,
UpdatedAt: now,
Sources: []unified.DataSource{unified.SourceProxmox},
Capabilities: []unified.ResourceCapability{
{
Name: "restart",
Type: unified.CapabilityTypeCommon,
Description: "Restart the VM",
MinimumApprovalLevel: unified.ApprovalAdmin,
InternalHandler: "proxmox.vm.restart",
Params: []unified.CapabilityParam{
{Name: "mode", Type: "string", Required: true, Enum: []string{"graceful", "force"}},
},
},
},
},
},
})
executor := &stubActionExecutor{
result: &unified.ExecutionResult{Success: true, Output: "should not run"},
readiness: &unified.ResourceActionReadiness{Name: "restart", Available: true},
}
h.SetActionExecutor(executor)
published := make(chan unified.ActionAuditRecord, 1)
h.SetActionCompletedPublisher(func(record unified.ActionAuditRecord) { published <- record })
planRec := httptest.NewRecorder()
planReq := httptest.NewRequest(http.MethodPost, "/api/actions/plan", bytes.NewBufferString(`{
"requestId":"agent-run-readiness",
"resourceId":"vm:42",
"capabilityName":"restart",
"params":{"mode":"graceful"},
"reason":"Recover after confirmed outage"
}`))
h.HandlePlanAction(planRec, actionHandlerTestRequest(planReq, ""))
if planRec.Code != http.StatusOK {
t.Fatalf("plan status = %d, body=%s", planRec.Code, planRec.Body.String())
}
var plan unified.ActionPlan
if err := json.Unmarshal(planRec.Body.Bytes(), &plan); err != nil {
t.Fatalf("decode plan response: %v", err)
}
decisionRec := httptest.NewRecorder()
decisionReq := httptest.NewRequest(http.MethodPost, "/api/actions/"+plan.ActionID+"/decision", bytes.NewBufferString(`{"outcome":"approved"}`))
decisionReq.SetPathValue("id", plan.ActionID)
h.HandleDecideAction(decisionRec, actionHandlerTestRequest(decisionReq, "operator@example.com"))
if decisionRec.Code != http.StatusOK {
t.Fatalf("decision status = %d, body=%s", decisionRec.Code, decisionRec.Body.String())
}
executor.readiness = &unified.ResourceActionReadiness{
Name: "restart",
Available: false,
ReasonCode: "command_agent_disconnected",
Reason: "Proxmox node command agent is not connected.",
}
executeRec := httptest.NewRecorder()
executeReq := httptest.NewRequest(http.MethodPost, "/api/actions/"+plan.ActionID+"/execute", bytes.NewBufferString(`{}`))
executeReq.SetPathValue("id", plan.ActionID)
h.HandleExecuteAction(executeRec, actionHandlerTestRequest(executeReq, "operator@example.com"))
if executeRec.Code != http.StatusConflict {
t.Fatalf("execute status = %d, want %d, body=%s", executeRec.Code, http.StatusConflict, executeRec.Body.String())
}
if executor.calls != 0 {
t.Fatalf("executor calls = %d, want no dispatch", executor.calls)
}
var envelope struct {
Error string `json:"error"`
Details map[string]string `json:"details"`
}
if err := json.Unmarshal(executeRec.Body.Bytes(), &envelope); err != nil {
t.Fatalf("decode execution error: %v", err)
}
if envelope.Error != "action_execution_unavailable" ||
envelope.Details["resourceId"] != "vm:42" ||
envelope.Details["capabilityName"] != "restart" ||
envelope.Details["reasonCode"] != "command_agent_disconnected" ||
envelope.Details["reason"] != "Proxmox node command agent is not connected." {
t.Fatalf("execution error = %#v", envelope)
}
store, err := h.getStore("default")
if err != nil {
t.Fatalf("get store: %v", err)
}
audit, ok, err := store.GetActionAudit(plan.ActionID)
if err != nil || !ok || audit.State != unified.ActionStateFailed || audit.Result == nil ||
!strings.HasPrefix(audit.Result.ErrorMessage, "action_execution_unavailable:") {
t.Fatalf("persisted audit = %#v ok=%v err=%v", audit, ok, err)
}
select {
case completed := <-published:
if completed.ID != plan.ActionID || completed.State != unified.ActionStateFailed {
t.Fatalf("published completion = %#v", completed)
}
default:
t.Fatal("expected terminal completion publication")
}
}
func TestHandleExecuteActionRejectsStalePlanBeforeExecutor(t *testing.T) {
now := time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC)
resource := unified.Resource{
+33 -12
View File
@@ -14655,7 +14655,7 @@ func TestContract_ActionDryRunOnlyExecutionErrorJSONSnapshot(t *testing.T) {
}
}
func TestContract_APIActionExecutionRevalidatesPlanFreshness(t *testing.T) {
func TestContract_APIActionExecutionRevalidatesPlanAndLiveReadiness(t *testing.T) {
source, err := os.ReadFile(filepath.Join("..", "actionlifecycle", "service.go"))
if err != nil {
t.Fatalf("read actionlifecycle service source: %v", err)
@@ -14664,25 +14664,33 @@ func TestContract_APIActionExecutionRevalidatesPlanFreshness(t *testing.T) {
for _, snippet := range []string{
"if unified.IsPermanentActionExecutionRefusal(err)",
"if err := s.ValidatePlanFresh(orgID, record); err != nil",
"if err := s.ValidateExecutionAvailable(ctx, orgID, record); err != nil",
"errors.Is(err, unified.ErrActionPlanDrift)",
"RecordRefusedExecution(store, record, actorID, now, err)",
"unified.RefuseActionExecution(record, reason, actor, now)",
} {
if !strings.Contains(src, snippet) {
t.Fatalf("actionlifecycle service must pin execute plan freshness guard snippet %q", snippet)
t.Fatalf("actionlifecycle service must pin execute plan and readiness guard snippet %q", snippet)
}
}
if strings.Index(src, "if err := s.ValidatePlanFresh(orgID, record); err != nil") >
strings.Index(src, "started, startEvent, err := unified.BeginActionExecution(record, actorID, now)") {
t.Fatal("Execute must validate plan freshness before entering executing state or calling the executor")
beginIndex := strings.Index(src, "started, startEvent, err := unified.BeginActionExecution(record, actorID, now)")
planIndex := strings.Index(src, "if err := s.ValidatePlanFresh(orgID, record); err != nil")
readinessIndex := strings.Index(src, "if err := s.ValidateExecutionAvailable(ctx, orgID, record); err != nil")
if planIndex < 0 || readinessIndex < 0 || beginIndex < 0 || planIndex > readinessIndex || readinessIndex > beginIndex {
t.Fatal("Execute must validate plan freshness and executor-owned live readiness before entering executing state or dispatching")
}
adapterSource, err := os.ReadFile("actions.go")
if err != nil {
t.Fatalf("read actions.go: %v", err)
}
if !strings.Contains(string(adapterSource), "writeJSONError(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionPlanDrift") {
t.Fatal("actions.go must map plan drift to a 409 conflict with the canonical drift error code")
for _, snippet := range []string{
"writeJSONError(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionPlanDrift",
"writeJSONErrorWithDetails(w, http.StatusConflict, agentcapabilities.AgentErrCodeActionExecutionUnavailable",
} {
if !strings.Contains(string(adapterSource), snippet) {
t.Fatalf("actions.go must preserve the canonical execution refusal mapping %q", snippet)
}
}
}
@@ -14695,7 +14703,7 @@ func readAgentCapabilitiesManifestSource(t *testing.T) string {
return string(source)
}
func TestContract_ExecuteActionCapabilityDeclaresPlanExpired(t *testing.T) {
func TestContract_ExecuteActionCapabilityDeclaresPermanentRefusals(t *testing.T) {
manifest := agentcapabilities.CanonicalManifest()
var executeAction agentcapabilities.Capability
for _, cap := range manifest.Capabilities {
@@ -14710,6 +14718,9 @@ func TestContract_ExecuteActionCapabilityDeclaresPlanExpired(t *testing.T) {
if !stringSliceContains(executeAction.ErrorCodes, agentcapabilities.AgentErrCodeActionPlanExpired) {
t.Error("execute_action manifest must declare action_plan_expired so agents can branch on permanent expired-plan refusals")
}
if !stringSliceContains(executeAction.ErrorCodes, agentcapabilities.AgentErrCodeActionExecutionUnavailable) {
t.Error("execute_action manifest must declare action_execution_unavailable so agents can branch on execution-time live-readiness refusal")
}
}
func TestContract_ResourceTimelineEndpointsIncludeRelatedChanges(t *testing.T) {
@@ -19247,10 +19258,6 @@ func TestContract_AgentSurfaceErrorEnvelopeUsesSharedAgentCapabilitiesType(t *te
"Error string",
"Message string",
"Details map[string]string",
`AgentErrCodeResourceNotFound = "resource_not_found"`,
`AgentErrCodeOperatorStateNotSet = "operator_state_not_set"`,
`AgentErrCodeInvalidActionRequest = "invalid_action_request"`,
`AgentErrCodeActionExecutionUnavailable = "action_execution_unavailable"`,
"func NewErrorEnvelope(",
"func DecodeErrorEnvelope(",
} {
@@ -19258,6 +19265,18 @@ func TestContract_AgentSurfaceErrorEnvelopeUsesSharedAgentCapabilitiesType(t *te
t.Errorf("agentcapabilities must own the shared agent error envelope; missing %s", fragment)
}
}
for constant, value := range map[string]string{
"AgentErrCodeResourceNotFound": "resource_not_found",
"AgentErrCodeOperatorStateNotSet": "operator_state_not_set",
"AgentErrCodeInvalidActionRequest": "invalid_action_request",
"AgentErrCodeActionExecutionUnavailable": "action_execution_unavailable",
"AgentErrCodeActionReadinessCheckFailed": "action_execution_availability_failed",
} {
pattern := regexp.MustCompile(`\b` + regexp.QuoteMeta(constant) + `\s*=\s*"` + regexp.QuoteMeta(value) + `"`)
if !pattern.Match(agentCapabilitiesSource) {
t.Errorf("agentcapabilities must bind %s to %q", constant, value)
}
}
}
// TestContract_AgentCapabilitiesManifestDeclaresProvisioningSurface pins the
@@ -20082,6 +20101,7 @@ func TestContract_AgentSurfaceErrorCodesMatchManifestDeclarations(t *testing.T)
"resource_registry_unavailable": true,
agentcapabilities.AgentErrCodeRawCommandRetired: true,
}
internalOnlyCodes[agentcapabilities.AgentErrCodeActionReadinessCheckFailed] = true
agentErrorConstantValues := map[string]string{
"AgentErrCodeResourceNotFound": agentcapabilities.AgentErrCodeResourceNotFound,
@@ -20113,6 +20133,7 @@ func TestContract_AgentSurfaceErrorCodesMatchManifestDeclarations(t *testing.T)
"AgentErrCodeActionExecutionForbidden": agentcapabilities.AgentErrCodeActionExecutionForbidden,
"AgentErrCodeActionNotExecuting": agentcapabilities.AgentErrCodeActionNotExecuting,
"AgentErrCodeActionDryRunOnly": agentcapabilities.AgentErrCodeActionDryRunOnly,
"AgentErrCodeActionReadinessCheckFailed": agentcapabilities.AgentErrCodeActionReadinessCheckFailed,
"AgentErrCodeActionPlanDrift": agentcapabilities.AgentErrCodeActionPlanDrift,
"AgentErrCodeActionPlanIdentityMismatch": agentcapabilities.AgentErrCodeActionPlanIdentityMismatch,
"AgentErrCodeResourceRemediationLocked": agentcapabilities.AgentErrCodeResourceRemediationLocked,
+7
View File
@@ -571,6 +571,7 @@ var (
ErrActionPlanExpired = errors.New("action plan expired")
ErrActionPlanNotExpired = errors.New("action plan has not expired")
ErrActionDryRunOnly = errors.New("action plan is dry-run only")
ErrActionExecutionUnavailable = errors.New("action execution unavailable")
ErrActionExecutionRefusal = errors.New("action execution refusal is not a permanent terminal refusal")
ErrInvalidApprovalOutcome = errors.New("invalid approval outcome")
ErrActionAuditAlreadyExists = errors.New("action audit already exists")
@@ -1064,6 +1065,12 @@ func permanentActionExecutionRefusalMessage(reason error) (string, string, bool)
return "action_plan_expired", "action_plan_expired: action plan has expired; re-plan before executing", true
case errors.Is(reason, ErrActionDryRunOnly):
return "action_dry_run_only", "action_dry_run_only: action plan is dry-run only and cannot be executed", true
case errors.Is(reason, ErrActionExecutionUnavailable):
message := strings.TrimSpace(strings.TrimPrefix(reason.Error(), ErrActionExecutionUnavailable.Error()+":"))
if message == "" {
message = "action execution is unavailable; re-plan before executing"
}
return "action_execution_unavailable", "action_execution_unavailable: " + message, true
case errors.Is(reason, ErrResourceRemediationLocked):
return "resource_remediation_locked", "resource_remediation_locked: resource is operator-locked against automated remediation", true
case errors.Is(reason, ErrActionPolicyAuthorizationExpired):
@@ -2,6 +2,7 @@ package unifiedresources
import (
"errors"
"fmt"
"strings"
"testing"
"time"
@@ -367,6 +368,7 @@ func TestRefuseActionExecutionRecordsPermanentRefusal(t *testing.T) {
{name: "plan drift", reason: ErrActionPlanDrift, wantCode: "plan_drift", wantPrefix: "plan_drift:"},
{name: "expired", reason: ErrActionPlanExpired, wantCode: "action_plan_expired", wantPrefix: "action_plan_expired:"},
{name: "dry run only", reason: ErrActionDryRunOnly, wantCode: "action_dry_run_only", wantPrefix: "action_dry_run_only:"},
{name: "execution unavailable", reason: fmt.Errorf("%w: command agent disconnected", ErrActionExecutionUnavailable), wantCode: "action_execution_unavailable", wantPrefix: "action_execution_unavailable: command agent disconnected"},
{name: "emergency stop", reason: ErrActionEmergencyStop, wantCode: "action_emergency_stop", wantPrefix: "action_emergency_stop:"},
} {
t.Run(tc.name, func(t *testing.T) {
@@ -924,6 +924,8 @@ func TestActionExecutionContractStaysAPIOwned(t *testing.T) {
"type AvailabilityChecker interface",
"CheckActionAvailable(ctx context.Context, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness",
"func (s *Service) ValidatePlanFresh(orgID string, record unified.ActionAuditRecord) error",
"func (s *Service) ValidateExecutionAvailable(ctx context.Context, orgID string, record unified.ActionAuditRecord) error",
"ErrActionExecutionUnavailable",
"func (s *Service) ExecuteUnderPolicy(",
"func RecordRefusedExecution(store Store, record unified.ActionAuditRecord",
"func (s *Service) publishCompleted(record unified.ActionAuditRecord)",