mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
feat(telemetry): attribute approved-action failures to cause buckets
The 30d window said 18 of 26 approved action attempts failed but carried zero cause information. Every approved attempt that is not a verified success now lands in exactly one content-free bucket: - pre_dispatch: terminally refused before dispatch (plan drift, expiry, emergency stop, policy authorization) - execution: dispatched execution failed or ended inconclusive - unverified: execution succeeded but outcome verification was not confirmed (reads as success in the UI, counted as failure here) - stuck_executing: still executing over an hour after dispatch plus one sanitized machine reason code for the most recent failure. Successes, failure buckets, and recent in-flight attempts partition the attempt count, so the next window explains its own gap. RefuseActionExecution now persists the specific refusal code (plan_drift, action_plan_expired, ...) as the canonical execution reason code instead of the generic pre_dispatch_refused, so audit truth and telemetry distinguish refusal causes without message parsing. Privacy disclosures updated in both PRIVACY.md copies; the adoption report script surfaces the new counters. Contract-Neutral: additive content-free telemetry counters + specific pre-dispatch refusal reason codes; privacy disclosures updated in-commit per security-privacy extension point
This commit is contained in:
@@ -142,6 +142,11 @@ Every field is listed below with the reason it exists. Nothing else is included
|
||||
| Pulse Intelligence approved action decisions 30d | `1` | Count approved governed action decisions in the current 30-day telemetry window without sending approvers, reasons, command text, targets, or action IDs |
|
||||
| Pulse Intelligence approved action attempts 30d | `1` | Count approved governed action attempts in the current 30-day telemetry window without sending action output, command text, or verification detail |
|
||||
| Pulse Intelligence approved action successes 30d | `1` | Count approved governed actions that completed successfully in the current 30-day telemetry window without sending action output, command text, resource IDs, actors, reasons, or verification detail |
|
||||
| Pulse Intelligence approved action failures (pre-dispatch) 30d | `1` | Count approved governed actions refused before dispatch (for example an expired or drifted plan) in the current 30-day telemetry window without sending action output, command text, resource IDs, actors, or reasons |
|
||||
| Pulse Intelligence approved action failures (execution) 30d | `1` | Count approved governed actions whose dispatched execution failed in the current 30-day telemetry window without sending action output, error text, command text, resource IDs, or actors |
|
||||
| Pulse Intelligence approved action failures (unverified) 30d | `1` | Count approved governed actions that executed but whose outcome verification was not confirmed in the current 30-day telemetry window without sending verification evidence, action output, command text, resource IDs, or actors |
|
||||
| Pulse Intelligence approved action stuck executing 30d | `1` | Count approved governed actions abandoned in the executing state in the current 30-day telemetry window without sending action output, command text, resource IDs, or actors |
|
||||
| Pulse Intelligence approved action last failure reason 30d | `plan_drift` | See one fixed machine reason code for the most recent approved-action failure in the current 30-day telemetry window without sending error text, action output, command text, resource IDs, or actors |
|
||||
|
||||
#### Server-side handling and retention
|
||||
|
||||
|
||||
@@ -142,6 +142,11 @@ Every field is listed below with the reason it exists. Nothing else is included
|
||||
| Pulse Intelligence approved action decisions 30d | `1` | Count approved governed action decisions in the current 30-day telemetry window without sending approvers, reasons, command text, targets, or action IDs |
|
||||
| Pulse Intelligence approved action attempts 30d | `1` | Count approved governed action attempts in the current 30-day telemetry window without sending action output, command text, or verification detail |
|
||||
| Pulse Intelligence approved action successes 30d | `1` | Count approved governed actions that completed successfully in the current 30-day telemetry window without sending action output, command text, resource IDs, actors, reasons, or verification detail |
|
||||
| Pulse Intelligence approved action failures (pre-dispatch) 30d | `1` | Count approved governed actions refused before dispatch (for example an expired or drifted plan) in the current 30-day telemetry window without sending action output, command text, resource IDs, actors, or reasons |
|
||||
| Pulse Intelligence approved action failures (execution) 30d | `1` | Count approved governed actions whose dispatched execution failed in the current 30-day telemetry window without sending action output, error text, command text, resource IDs, or actors |
|
||||
| Pulse Intelligence approved action failures (unverified) 30d | `1` | Count approved governed actions that executed but whose outcome verification was not confirmed in the current 30-day telemetry window without sending verification evidence, action output, command text, resource IDs, or actors |
|
||||
| Pulse Intelligence approved action stuck executing 30d | `1` | Count approved governed actions abandoned in the executing state in the current 30-day telemetry window without sending action output, command text, resource IDs, or actors |
|
||||
| Pulse Intelligence approved action last failure reason 30d | `plan_drift` | See one fixed machine reason code for the most recent approved-action failure in the current 30-day telemetry window without sending error text, action output, command text, resource IDs, or actors |
|
||||
|
||||
#### Server-side handling and retention
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -41,7 +42,11 @@ func (r *Router) GetPulseIntelligenceActionTelemetry(since time.Time) telemetry.
|
||||
log.Warn().Err(err).Str("org_id", orgID).Msg("Unable to query action audit telemetry summary")
|
||||
continue
|
||||
}
|
||||
recordsByID := make(map[string]unifiedresources.ActionAuditRecord, len(records))
|
||||
for _, record := range records {
|
||||
if actionID := strings.TrimSpace(record.ID); actionID != "" {
|
||||
recordsByID[actionID] = record
|
||||
}
|
||||
snapshot.ActionPlans30d++
|
||||
if pulseIntelligenceActionRequiresApproval(record) {
|
||||
snapshot.ApprovalRequests30d++
|
||||
@@ -71,11 +76,95 @@ func (r *Router) GetPulseIntelligenceActionTelemetry(since time.Time) telemetry.
|
||||
snapshot.ApprovedActionDecisions30d += len(approvedDecisionIDs)
|
||||
snapshot.ApprovedActionAttempts30d += len(approvedAttemptIDs)
|
||||
snapshot.ApprovedActionSuccesses30d += len(approvedSuccessIDs)
|
||||
accumulatePulseIntelligenceApprovedActionFailures(&snapshot, store, orgID, approvedAttemptIDs, approvedSuccessIDs, recordsByID, time.Now().UTC())
|
||||
}
|
||||
|
||||
return snapshot
|
||||
}
|
||||
|
||||
// pulseIntelligenceStuckExecutingThreshold separates an in-flight dispatch
|
||||
// from an abandoned one. The longest legitimate typed dispatch transport wait
|
||||
// is 30 minutes, so an executing record untouched for longer is stuck.
|
||||
const pulseIntelligenceStuckExecutingThreshold = time.Hour
|
||||
|
||||
// pulseIntelligenceReasonCodePattern bounds exported failure reason codes to
|
||||
// closed machine-code shape so telemetry stays content-free even if a future
|
||||
// executor misuses the reason-code field.
|
||||
var pulseIntelligenceReasonCodePattern = regexp.MustCompile(`^[a-z0-9_.-]{1,64}$`)
|
||||
|
||||
// accumulatePulseIntelligenceApprovedActionFailures attributes every approved
|
||||
// attempt that is not a success to one cause bucket, and records the machine
|
||||
// reason code of the most recent failure.
|
||||
func accumulatePulseIntelligenceApprovedActionFailures(snapshot *telemetry.PulseIntelligenceActionSnapshot, store unifiedresources.ResourceStore, orgID string, attemptIDs, successIDs map[string]struct{}, recordsByID map[string]unifiedresources.ActionAuditRecord, now time.Time) {
|
||||
var lastFailureAt time.Time
|
||||
for actionID := range attemptIDs {
|
||||
if _, ok := successIDs[actionID]; ok {
|
||||
continue
|
||||
}
|
||||
record, ok := recordsByID[actionID]
|
||||
if !ok {
|
||||
fetched, found, err := store.GetActionAudit(actionID)
|
||||
if err != nil || !found {
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Str("org_id", orgID).Msg("Unable to resolve action audit for failure-cause telemetry summary")
|
||||
}
|
||||
continue
|
||||
}
|
||||
record = fetched
|
||||
}
|
||||
cause, reason := pulseIntelligenceApprovedActionFailureCause(record, now)
|
||||
switch cause {
|
||||
case "pre_dispatch":
|
||||
snapshot.ApprovedActionFailuresPreDispatch30d++
|
||||
case "execution":
|
||||
snapshot.ApprovedActionFailuresExecution30d++
|
||||
case "unverified":
|
||||
snapshot.ApprovedActionFailuresUnverified30d++
|
||||
case "stuck_executing":
|
||||
snapshot.ApprovedActionStuckExecuting30d++
|
||||
default:
|
||||
continue
|
||||
}
|
||||
if record.UpdatedAt.After(lastFailureAt) {
|
||||
lastFailureAt = record.UpdatedAt
|
||||
snapshot.ApprovedActionLastFailureReason30d = reason
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pulseIntelligenceApprovedActionFailureCause classifies an approved attempt
|
||||
// that is not a verified success into a coarse cause bucket plus the specific
|
||||
// machine reason code. A recently-executing record returns no cause: it is
|
||||
// still in flight and may yet succeed.
|
||||
func pulseIntelligenceApprovedActionFailureCause(record unifiedresources.ActionAuditRecord, now time.Time) (string, string) {
|
||||
switch record.State {
|
||||
case unifiedresources.ActionStateExecuting:
|
||||
if record.UpdatedAt.IsZero() || now.Sub(record.UpdatedAt) >= pulseIntelligenceStuckExecutingThreshold {
|
||||
return "stuck_executing", "stuck_executing"
|
||||
}
|
||||
return "", ""
|
||||
case unifiedresources.ActionStateFailed:
|
||||
truth := unifiedresources.CanonicalActionResultV2(record)
|
||||
if truth.Execution.Status == unifiedresources.ActionExecutionNotRun {
|
||||
return "pre_dispatch", pulseIntelligenceSanitizedReasonCode(truth.Execution.ReasonCode, "pre_dispatch_refused")
|
||||
}
|
||||
return "execution", pulseIntelligenceSanitizedReasonCode(truth.Execution.ReasonCode, "execution_failed")
|
||||
case unifiedresources.ActionStateCompleted:
|
||||
truth := unifiedresources.CanonicalActionResultV2(record)
|
||||
return "unverified", pulseIntelligenceSanitizedReasonCode(truth.Verification.ReasonCode, "verification_unconfirmed")
|
||||
default:
|
||||
return "", ""
|
||||
}
|
||||
}
|
||||
|
||||
func pulseIntelligenceSanitizedReasonCode(code, fallback string) string {
|
||||
code = strings.TrimSpace(code)
|
||||
if pulseIntelligenceReasonCodePattern.MatchString(code) {
|
||||
return code
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func (r *Router) pulseIntelligenceTelemetryOrgIDs() []string {
|
||||
if r == nil || r.multiTenant == nil {
|
||||
return []string{"default"}
|
||||
|
||||
@@ -241,6 +241,136 @@ func TestGetPulseIntelligenceActionTelemetry_RequiresVerifiedOutcomeForApprovedS
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetPulseIntelligenceActionTelemetry_AttributesApprovedActionFailureCauses(t *testing.T) {
|
||||
// Cause classification measures staleness against the wall clock, so this
|
||||
// test anchors records to real time instead of a fixed date.
|
||||
now := time.Now().UTC()
|
||||
since := now.Add(-telemetry.PulseIntelligenceTelemetryWindow)
|
||||
dataDir := t.TempDir()
|
||||
router := &Router{
|
||||
resourceHandlers: NewResourceHandlers(&config.Config{DataPath: dataDir}),
|
||||
}
|
||||
store, err := router.resourceHandlers.getStore("default")
|
||||
if err != nil {
|
||||
t.Fatalf("getStore: %v", err)
|
||||
}
|
||||
|
||||
approvedAt := func(at time.Time) []unifiedresources.ActionApprovalRecord {
|
||||
return []unifiedresources.ActionApprovalRecord{{
|
||||
Outcome: unifiedresources.OutcomeApproved,
|
||||
Method: unifiedresources.MethodUI,
|
||||
Timestamp: at,
|
||||
Actor: "operator",
|
||||
}}
|
||||
}
|
||||
|
||||
// Approved, then terminally refused before dispatch (plan drift).
|
||||
refused := pulseTelemetryActionRecord("refused-pre-dispatch", now.Add(-3*time.Hour), unifiedresources.ActionStateApproved, true, approvedAt(now.Add(-3*time.Hour)))
|
||||
refused.Plan.ExpiresAt = now.Add(time.Hour)
|
||||
if err := store.RecordActionAudit(refused); err != nil {
|
||||
t.Fatalf("RecordActionAudit(refused): %v", err)
|
||||
}
|
||||
if _, err := actionlifecycle.RecordRefusedExecution(store, refused, "operator", now.Add(-170*time.Minute), unifiedresources.ErrActionPlanDrift); err != nil {
|
||||
t.Fatalf("RecordRefusedExecution(refused): %v", err)
|
||||
}
|
||||
|
||||
// Approved, dispatched, execution failed on the agent.
|
||||
execFailed := pulseTelemetryActionRecord("exec-failed", now.Add(-2*time.Hour), unifiedresources.ActionStateFailed, true, approvedAt(now.Add(-2*time.Hour)))
|
||||
execFailed.Result = &unifiedresources.ExecutionResult{Success: false, ErrorMessage: "image pull failed"}
|
||||
if err := store.RecordActionAudit(execFailed); err != nil {
|
||||
t.Fatalf("RecordActionAudit(execFailed): %v", err)
|
||||
}
|
||||
|
||||
// Approved, execution succeeded, but no verification evidence confirmed it.
|
||||
unverified := pulseTelemetryActionRecord("completed-unverified", now.Add(-100*time.Minute), unifiedresources.ActionStateCompleted, true, approvedAt(now.Add(-100*time.Minute)))
|
||||
unverified.Result = &unifiedresources.ExecutionResult{Success: true}
|
||||
if err := store.RecordActionAudit(unverified); err != nil {
|
||||
t.Fatalf("RecordActionAudit(unverified): %v", err)
|
||||
}
|
||||
|
||||
// Approved and abandoned in executing state well past the dispatch window.
|
||||
stuck := pulseTelemetryActionRecord("stuck-executing", now.Add(-5*time.Hour), unifiedresources.ActionStateExecuting, true, approvedAt(now.Add(-5*time.Hour)))
|
||||
if err := store.RecordActionAudit(stuck); err != nil {
|
||||
t.Fatalf("RecordActionAudit(stuck): %v", err)
|
||||
}
|
||||
|
||||
// Approved and still legitimately in flight: no failure bucket.
|
||||
inFlight := pulseTelemetryActionRecord("in-flight", now.Add(-5*time.Minute), unifiedresources.ActionStateExecuting, true, approvedAt(now.Add(-5*time.Minute)))
|
||||
if err := store.RecordActionAudit(inFlight); err != nil {
|
||||
t.Fatalf("RecordActionAudit(inFlight): %v", err)
|
||||
}
|
||||
|
||||
// Approved and verified success: not a failure.
|
||||
verified := pulseTelemetryActionRecord("verified-success", now.Add(-90*time.Minute), unifiedresources.ActionStateCompleted, true, approvedAt(now.Add(-90*time.Minute)))
|
||||
verified.Result = &unifiedresources.ExecutionResult{Success: true}
|
||||
verified.VerificationOutcome = unifiedresources.VerificationOutcome{Status: unifiedresources.VerificationVerified}
|
||||
if err := store.RecordActionAudit(verified); err != nil {
|
||||
t.Fatalf("RecordActionAudit(verified): %v", err)
|
||||
}
|
||||
|
||||
got := router.GetPulseIntelligenceActionTelemetry(since)
|
||||
|
||||
if got.ApprovedActionAttempts30d != 6 {
|
||||
t.Fatalf("ApprovedActionAttempts30d = %d, want 6", got.ApprovedActionAttempts30d)
|
||||
}
|
||||
if got.ApprovedActionSuccesses30d != 1 {
|
||||
t.Fatalf("ApprovedActionSuccesses30d = %d, want 1", got.ApprovedActionSuccesses30d)
|
||||
}
|
||||
if got.ApprovedActionFailuresPreDispatch30d != 1 {
|
||||
t.Fatalf("ApprovedActionFailuresPreDispatch30d = %d, want 1", got.ApprovedActionFailuresPreDispatch30d)
|
||||
}
|
||||
if got.ApprovedActionFailuresExecution30d != 1 {
|
||||
t.Fatalf("ApprovedActionFailuresExecution30d = %d, want 1", got.ApprovedActionFailuresExecution30d)
|
||||
}
|
||||
if got.ApprovedActionFailuresUnverified30d != 1 {
|
||||
t.Fatalf("ApprovedActionFailuresUnverified30d = %d, want 1", got.ApprovedActionFailuresUnverified30d)
|
||||
}
|
||||
if got.ApprovedActionStuckExecuting30d != 1 {
|
||||
t.Fatalf("ApprovedActionStuckExecuting30d = %d, want 1", got.ApprovedActionStuckExecuting30d)
|
||||
}
|
||||
// The completed-unverified record is the most recent failure; the legacy
|
||||
// row carries no canonical reason code, so the sanitized fallback applies.
|
||||
if got.ApprovedActionLastFailureReason30d != "verification_unconfirmed" {
|
||||
t.Fatalf("ApprovedActionLastFailureReason30d = %q, want %q", got.ApprovedActionLastFailureReason30d, "verification_unconfirmed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetPulseIntelligenceActionTelemetry_LastFailureReasonUsesCanonicalReasonCode(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
since := now.Add(-telemetry.PulseIntelligenceTelemetryWindow)
|
||||
dataDir := t.TempDir()
|
||||
router := &Router{
|
||||
resourceHandlers: NewResourceHandlers(&config.Config{DataPath: dataDir}),
|
||||
}
|
||||
store, err := router.resourceHandlers.getStore("default")
|
||||
if err != nil {
|
||||
t.Fatalf("getStore: %v", err)
|
||||
}
|
||||
|
||||
refused := pulseTelemetryActionRecord("refused-plan-drift", now.Add(-time.Hour), unifiedresources.ActionStateApproved, true, []unifiedresources.ActionApprovalRecord{{
|
||||
Outcome: unifiedresources.OutcomeApproved,
|
||||
Method: unifiedresources.MethodUI,
|
||||
Timestamp: now.Add(-time.Hour),
|
||||
Actor: "operator",
|
||||
}})
|
||||
refused.Plan.ExpiresAt = now.Add(time.Hour)
|
||||
if err := store.RecordActionAudit(refused); err != nil {
|
||||
t.Fatalf("RecordActionAudit(refused): %v", err)
|
||||
}
|
||||
if _, err := actionlifecycle.RecordRefusedExecution(store, refused, "operator", now.Add(-30*time.Minute), unifiedresources.ErrActionPlanDrift); err != nil {
|
||||
t.Fatalf("RecordRefusedExecution(refused): %v", err)
|
||||
}
|
||||
|
||||
got := router.GetPulseIntelligenceActionTelemetry(since)
|
||||
|
||||
if got.ApprovedActionFailuresPreDispatch30d != 1 {
|
||||
t.Fatalf("ApprovedActionFailuresPreDispatch30d = %d, want 1", got.ApprovedActionFailuresPreDispatch30d)
|
||||
}
|
||||
if got.ApprovedActionLastFailureReason30d != "plan_drift" {
|
||||
t.Fatalf("ApprovedActionLastFailureReason30d = %q, want %q", got.ApprovedActionLastFailureReason30d, "plan_drift")
|
||||
}
|
||||
}
|
||||
|
||||
func pulseTelemetryActionRecord(
|
||||
id string,
|
||||
createdAt time.Time,
|
||||
|
||||
@@ -237,6 +237,15 @@ type Ping struct {
|
||||
PulseIntelligenceApprovedActionDecisions30d int `json:"pulse_intelligence_approved_action_decisions_30d"`
|
||||
PulseIntelligenceApprovedActionAttempts30d int `json:"pulse_intelligence_approved_action_attempts_30d"`
|
||||
PulseIntelligenceApprovedActionSuccesses30d int `json:"pulse_intelligence_approved_action_successes_30d"`
|
||||
|
||||
// Cause-coded approved-action failure counters. Together with successes
|
||||
// and still-in-flight attempts these partition the attempt count, so the
|
||||
// attempt/success gap is attributable without exporting action content.
|
||||
PulseIntelligenceApprovedActionFailuresPreDispatch30d int `json:"pulse_intelligence_approved_action_failures_pre_dispatch_30d"`
|
||||
PulseIntelligenceApprovedActionFailuresExecution30d int `json:"pulse_intelligence_approved_action_failures_execution_30d"`
|
||||
PulseIntelligenceApprovedActionFailuresUnverified30d int `json:"pulse_intelligence_approved_action_failures_unverified_30d"`
|
||||
PulseIntelligenceApprovedActionStuckExecuting30d int `json:"pulse_intelligence_approved_action_stuck_executing_30d"`
|
||||
PulseIntelligenceApprovedActionLastFailureReason30d string `json:"pulse_intelligence_approved_action_last_failure_reason_30d,omitempty"`
|
||||
}
|
||||
|
||||
// Snapshot holds the dynamic state gathered at ping time.
|
||||
@@ -338,11 +347,18 @@ type Snapshot struct {
|
||||
PulseIntelligenceApprovedActionDecisions30d int
|
||||
PulseIntelligenceApprovedActionAttempts30d int
|
||||
PulseIntelligenceApprovedActionSuccesses30d int
|
||||
PulseIntelligenceApprovedActionFailuresPreDispatch30d int
|
||||
PulseIntelligenceApprovedActionFailuresExecution30d int
|
||||
PulseIntelligenceApprovedActionFailuresUnverified30d int
|
||||
PulseIntelligenceApprovedActionStuckExecuting30d int
|
||||
PulseIntelligenceApprovedActionLastFailureReason30d string
|
||||
}
|
||||
|
||||
// PulseIntelligenceActionSnapshot is the action-governance portion of the
|
||||
// Pulse Intelligence telemetry loop. It is intentionally count-only so callers
|
||||
// can aggregate local audit records without exporting action details.
|
||||
// can aggregate local audit records without exporting action details. The
|
||||
// failure-cause fields carry only closed machine reason codes, never command
|
||||
// text, resource identifiers, or output.
|
||||
type PulseIntelligenceActionSnapshot struct {
|
||||
ActionPlans30d int
|
||||
ApprovalRequests30d int
|
||||
@@ -350,6 +366,23 @@ type PulseIntelligenceActionSnapshot struct {
|
||||
ApprovedActionDecisions30d int
|
||||
ApprovedActionAttempts30d int
|
||||
ApprovedActionSuccesses30d int
|
||||
|
||||
// ApprovedActionFailuresPreDispatch30d counts approved attempts refused
|
||||
// terminally before dispatch (plan drift, expiry, emergency stop, policy
|
||||
// authorization).
|
||||
ApprovedActionFailuresPreDispatch30d int
|
||||
// ApprovedActionFailuresExecution30d counts approved attempts whose
|
||||
// dispatched execution failed or ended inconclusive.
|
||||
ApprovedActionFailuresExecution30d int
|
||||
// ApprovedActionFailuresUnverified30d counts approved attempts that
|
||||
// completed execution but whose outcome verification was not confirmed.
|
||||
ApprovedActionFailuresUnverified30d int
|
||||
// ApprovedActionStuckExecuting30d counts approved attempts still in the
|
||||
// executing state well past any legitimate dispatch window.
|
||||
ApprovedActionStuckExecuting30d int
|
||||
// ApprovedActionLastFailureReason30d is the machine reason code of the
|
||||
// most recent approved-action failure, sanitized to a closed code shape.
|
||||
ApprovedActionLastFailureReason30d string
|
||||
}
|
||||
|
||||
// ApplyUpdateTelemetrySnapshot adds content-free update funnel counters from
|
||||
@@ -833,6 +866,11 @@ func applySnapshot(base Ping, fn SnapshotFunc) Ping {
|
||||
ping.PulseIntelligenceApprovedActionDecisions30d = s.PulseIntelligenceApprovedActionDecisions30d
|
||||
ping.PulseIntelligenceApprovedActionAttempts30d = s.PulseIntelligenceApprovedActionAttempts30d
|
||||
ping.PulseIntelligenceApprovedActionSuccesses30d = s.PulseIntelligenceApprovedActionSuccesses30d
|
||||
ping.PulseIntelligenceApprovedActionFailuresPreDispatch30d = s.PulseIntelligenceApprovedActionFailuresPreDispatch30d
|
||||
ping.PulseIntelligenceApprovedActionFailuresExecution30d = s.PulseIntelligenceApprovedActionFailuresExecution30d
|
||||
ping.PulseIntelligenceApprovedActionFailuresUnverified30d = s.PulseIntelligenceApprovedActionFailuresUnverified30d
|
||||
ping.PulseIntelligenceApprovedActionStuckExecuting30d = s.PulseIntelligenceApprovedActionStuckExecuting30d
|
||||
ping.PulseIntelligenceApprovedActionLastFailureReason30d = s.PulseIntelligenceApprovedActionLastFailureReason30d
|
||||
return ping
|
||||
}
|
||||
|
||||
|
||||
@@ -1004,7 +1004,7 @@ func RefuseActionExecution(record ActionAuditRecord, reason error, actor string,
|
||||
} else {
|
||||
now = now.UTC()
|
||||
}
|
||||
message, ok := permanentActionExecutionRefusalMessage(reason)
|
||||
code, message, ok := permanentActionExecutionRefusalMessage(reason)
|
||||
if !ok {
|
||||
return ActionAuditRecord{}, ActionLifecycleEvent{}, fmt.Errorf("%w: %v", ErrActionExecutionRefusal, reason)
|
||||
}
|
||||
@@ -1015,7 +1015,7 @@ func RefuseActionExecution(record ActionAuditRecord, reason error, actor string,
|
||||
|
||||
record.State = ActionStateFailed
|
||||
record.UpdatedAt = now
|
||||
record.Result = KnownNoEffectResult("pre_dispatch_refused", message, record.Plan.RollbackAvailable)
|
||||
record.Result = KnownNoEffectResult(code, message, record.Plan.RollbackAvailable)
|
||||
normalized, err := NormalizeActionAuditRecord(record)
|
||||
if err != nil {
|
||||
return ActionAuditRecord{}, ActionLifecycleEvent{}, err
|
||||
@@ -1037,32 +1037,36 @@ func RefuseActionExecution(record ActionAuditRecord, reason error, actor string,
|
||||
// IsPermanentActionExecutionRefusal reports whether err represents a
|
||||
// non-dispatchable execution attempt that should terminally fail the audit.
|
||||
func IsPermanentActionExecutionRefusal(err error) bool {
|
||||
_, ok := permanentActionExecutionRefusalMessage(err)
|
||||
_, _, ok := permanentActionExecutionRefusalMessage(err)
|
||||
return ok
|
||||
}
|
||||
|
||||
func permanentActionExecutionRefusalMessage(reason error) (string, bool) {
|
||||
// permanentActionExecutionRefusalMessage returns the stable machine reason
|
||||
// code and the human refusal message for a permanent pre-dispatch refusal.
|
||||
// The code is persisted as the canonical execution reason code, so telemetry
|
||||
// and audit consumers can distinguish refusal causes without message parsing.
|
||||
func permanentActionExecutionRefusalMessage(reason error) (string, string, bool) {
|
||||
switch {
|
||||
case errors.Is(reason, ErrActionPlanDrift):
|
||||
return "plan_drift: action plan no longer matches the current resource contract; re-plan before executing", true
|
||||
return "plan_drift", "plan_drift: action plan no longer matches the current resource contract; re-plan before executing", true
|
||||
case errors.Is(reason, ErrActionPlanExpired):
|
||||
return "action_plan_expired: action plan has expired; re-plan before executing", true
|
||||
return "action_plan_expired", "action_plan_expired: action plan has expired; re-plan before executing", true
|
||||
case errors.Is(reason, ErrActionDryRunOnly):
|
||||
return "action_dry_run_only: action plan is dry-run only and cannot be executed", true
|
||||
return "action_dry_run_only", "action_dry_run_only: action plan is dry-run only and cannot be executed", true
|
||||
case errors.Is(reason, ErrResourceRemediationLocked):
|
||||
return "resource_remediation_locked: resource is operator-locked against automated remediation", true
|
||||
return "resource_remediation_locked", "resource_remediation_locked: resource is operator-locked against automated remediation", true
|
||||
case errors.Is(reason, ErrActionPolicyAuthorizationExpired):
|
||||
return "policy_authorization_expired: automatic authority expired before dispatch", true
|
||||
return "policy_authorization_expired", "policy_authorization_expired: automatic authority expired before dispatch", true
|
||||
case errors.Is(reason, ErrActionPolicyAuthorizationInvalid):
|
||||
return "policy_authorization_invalid: automatic authority is missing, unreadable, or malformed", true
|
||||
return "policy_authorization_invalid", "policy_authorization_invalid: automatic authority is missing, unreadable, or malformed", true
|
||||
case errors.Is(reason, ErrActionPolicyAuthorizationRevoked):
|
||||
return "policy_authorization_revoked: automatic authority changed before dispatch", true
|
||||
return "policy_authorization_revoked", "policy_authorization_revoked: automatic authority changed before dispatch", true
|
||||
case errors.Is(reason, ErrActionEmergencyStop):
|
||||
return "action_emergency_stop: action dispatch is stopped by the operator", true
|
||||
return "action_emergency_stop", "action_emergency_stop: action dispatch is stopped by the operator", true
|
||||
case errors.Is(reason, ErrActionReplanRequired):
|
||||
return "action_replan_required: legacy action authority is unbound; re-plan before deciding or executing", true
|
||||
return "action_replan_required", "action_replan_required: legacy action authority is unbound; re-plan before deciding or executing", true
|
||||
default:
|
||||
return "", false
|
||||
return "", "", false
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -361,11 +361,13 @@ func TestRefuseActionExecutionRecordsPermanentRefusal(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
reason error
|
||||
wantCode string
|
||||
wantPrefix string
|
||||
}{
|
||||
{name: "plan drift", reason: ErrActionPlanDrift, wantPrefix: "plan_drift:"},
|
||||
{name: "expired", reason: ErrActionPlanExpired, wantPrefix: "action_plan_expired:"},
|
||||
{name: "dry run only", reason: ErrActionDryRunOnly, wantPrefix: "action_dry_run_only:"},
|
||||
{name: "plan drift", reason: ErrActionPlanDrift, wantCode: "plan_drift", wantPrefix: "plan_drift:"},
|
||||
{name: "expired", reason: ErrActionPlanExpired, wantCode: "action_plan_expired", wantPrefix: "action_plan_expired:"},
|
||||
{name: "dry run only", reason: ErrActionDryRunOnly, wantCode: "action_dry_run_only", wantPrefix: "action_dry_run_only:"},
|
||||
{name: "emergency stop", reason: ErrActionEmergencyStop, wantCode: "action_emergency_stop", wantPrefix: "action_emergency_stop:"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
updated, event, err := RefuseActionExecution(record, tc.reason, " operator@example.com ", now)
|
||||
@@ -378,6 +380,10 @@ func TestRefuseActionExecutionRecordsPermanentRefusal(t *testing.T) {
|
||||
if !strings.HasPrefix(updated.Result.ErrorMessage, tc.wantPrefix) {
|
||||
t.Fatalf("ErrorMessage = %q, want prefix %q", updated.Result.ErrorMessage, tc.wantPrefix)
|
||||
}
|
||||
truth := CanonicalActionResultV2(updated)
|
||||
if truth.Execution.Status != ActionExecutionNotRun || truth.Execution.ReasonCode != tc.wantCode {
|
||||
t.Fatalf("execution truth = %+v, want not_run with reason code %q", truth.Execution, tc.wantCode)
|
||||
}
|
||||
if event.ActionID != updated.ID || event.State != ActionStateFailed || event.Actor != "operator@example.com" || event.Message != updated.Result.ErrorMessage {
|
||||
t.Fatalf("lifecycle event = %#v, updated result = %#v", event, updated.Result)
|
||||
}
|
||||
|
||||
@@ -37,6 +37,11 @@ func applyPulseIntelligenceTelemetrySnapshot(
|
||||
snap.PulseIntelligenceApprovedActionDecisions30d = actionSnapshot.ApprovedActionDecisions30d
|
||||
snap.PulseIntelligenceApprovedActionAttempts30d = actionSnapshot.ApprovedActionAttempts30d
|
||||
snap.PulseIntelligenceApprovedActionSuccesses30d = actionSnapshot.ApprovedActionSuccesses30d
|
||||
snap.PulseIntelligenceApprovedActionFailuresPreDispatch30d = actionSnapshot.ApprovedActionFailuresPreDispatch30d
|
||||
snap.PulseIntelligenceApprovedActionFailuresExecution30d = actionSnapshot.ApprovedActionFailuresExecution30d
|
||||
snap.PulseIntelligenceApprovedActionFailuresUnverified30d = actionSnapshot.ApprovedActionFailuresUnverified30d
|
||||
snap.PulseIntelligenceApprovedActionStuckExecuting30d = actionSnapshot.ApprovedActionStuckExecuting30d
|
||||
snap.PulseIntelligenceApprovedActionLastFailureReason30d = actionSnapshot.ApprovedActionLastFailureReason30d
|
||||
|
||||
applyPulseIntelligenceAdoptionSnapshot(snap)
|
||||
}
|
||||
|
||||
@@ -261,6 +261,22 @@ PULSE_INTELLIGENCE_COUNT_FIELDS = (
|
||||
("pulse_intelligence_approved_action_decisions_30d", "Approved action decisions 30d"),
|
||||
("pulse_intelligence_approved_action_attempts_30d", "Approved action attempts 30d"),
|
||||
("pulse_intelligence_approved_action_successes_30d", "Approved action successes 30d"),
|
||||
(
|
||||
"pulse_intelligence_approved_action_failures_pre_dispatch_30d",
|
||||
"Approved action failures (pre-dispatch refusal) 30d",
|
||||
),
|
||||
(
|
||||
"pulse_intelligence_approved_action_failures_execution_30d",
|
||||
"Approved action failures (execution) 30d",
|
||||
),
|
||||
(
|
||||
"pulse_intelligence_approved_action_failures_unverified_30d",
|
||||
"Approved action failures (completed unverified) 30d",
|
||||
),
|
||||
(
|
||||
"pulse_intelligence_approved_action_stuck_executing_30d",
|
||||
"Approved action attempts stuck executing 30d",
|
||||
),
|
||||
)
|
||||
PULSE_INTELLIGENCE_OUTCOME_COHORTS = (
|
||||
(
|
||||
|
||||
Reference in New Issue
Block a user