Add powered-off alert tolerance

This commit is contained in:
rcourtman
2026-07-24 12:18:26 +01:00
parent 20b7d6788d
commit 9da4d4e966
26 changed files with 1065 additions and 85 deletions
+20
View File
@@ -396,6 +396,26 @@ Pulse uses a powerful alerting engine with hysteresis (separate trigger/clear th
**Managed via UI**: Alerts → Thresholds
### VM and container powered-off tolerance
Open **Alerts → Thresholds → Alert intent & grace** to configure how long a
Proxmox VM or LXC container may remain stopped before Pulse raises its
powered-off alert.
- Leave the VM/container default blank to inherit the existing policy. An
installation with no applicable policy keeps the legacy two-poll behavior.
- Set it to `0` to alert on the first authoritative stopped observation.
- Set a positive number of seconds (for example, `300`) to tolerate a short
stop regardless of polling cadence. Per-resource values override the
VM/container default; a blank per-resource value inherits it.
- **Extend offline grace during Proxmox backups** applies only while Pulse has
fresh matching evidence of an active backup. The maximum deferral is a hard
cap, so a stale or stuck backup cannot hide a sustained outage.
This tolerance delays activation; it does not disable powered-off monitoring.
Use the existing guest offline-alert toggle when a guest should never produce
powered-off alerts.
<details>
<summary><strong>Manual Configuration (JSON)</strong></summary>
@@ -5066,6 +5066,11 @@ Neither an expected-transient decision nor an indeterminate UDP outcome may be
translated into an agent action. Any later customer-infrastructure mutation
still requires the canonical Actions planner, approval, executor, receipt,
audit, and verification boundaries.
Powered-off tolerance is evaluated from the Pulse server's monotonic elapsed
time and the existing VM/LXC observation stream. It does not accept agent time,
poll count, a new report field, or remote configuration authority, and a guest
resource-type default cannot change node or host-agent connectivity
confirmations.
### Server-side host deletion and re-enrollment authority
@@ -1283,6 +1283,27 @@ same resolver but restores pending state before returning, so it is read-only.
Invalid documents, persistence failures, or revision conflicts must leave the
prior in-memory and durable policy active.
Resource-type inheritance is field-wise from general to specific before the
canonical-resource rule: for example `guest` supplies a VM/LXC default and
`vm` or `system-container` may replace individual fields. A guest policy must
not leak into node or agent connectivity. Omitted grace preserves inherited or
factory behavior, explicit zero is a deliberate no-wait policy, and a positive
powered-off grace is duration authority rather than a poll-count threshold.
The no-policy powered-state path retains the legacy two-confirmation contract
for migration compatibility; an applicable powered-state duration activates
on the first stopped observation at or after elapsed eligibility.
Runtime eligibility uses server receipt time plus process-monotonic elapsed
progress, never client/provider timestamps or poll counts. Duplicate reports
advance no time, delayed polls count actual elapsed time, and wall-clock
forward/backward changes do not change the decision. Pending state persists
accumulated elapsed progress. Restart re-baselines the monotonic clock without
counting unobserved process downtime, while legacy timestamp-only pending files
are imported once into the elapsed representation. Recovery, explicit guest
suppression, disabling offline alerts, and tracking cleanup remove both
durable pending state and its transient monotonic baseline before a later
outage can start.
Operator maintenance and intentionally-offline state are read only through the
canonical unified-resource identity. Backup-aware offline deferral consumes
fresh, matching, active task evidence, applies the configured post-backup grace,
@@ -1291,6 +1312,9 @@ finished, or mismatched backup evidence cannot suppress an outage. This policy
changes alert activation only: notification delivery, recovery assurance, and
customer-infrastructure mutation retain their existing owners.
`internal/alerts/intent_policy_test.go` proves precedence, normalization,
operator and backup contexts, preview immutability, restart continuity, and
first-match lifecycle identity.
`internal/alerts/intent_policy_test.go` and
`internal/alerts/powered_off_tolerance_test.go` prove field-wise type
precedence, normalization, exact zero/positive duration boundaries, VM/LXC
coverage, duplicate and delayed reports, wall-clock changes, suppression
reset, migration/restart continuity, backup hard-cap behavior, preview
immutability, and first-match lifecycle identity.
@@ -8247,7 +8247,14 @@ default, resource-type, and canonical-resource rules. A stale revision returns
`409 Conflict`; invalid policy or preview input returns `400`; unavailable
runtime or persistence ownership returns `503`. A save failure restores the
previous in-memory document and returns `500`. Preview is bounded and read-only
and cannot advance pending grace state.
and cannot advance pending grace state. Both write and preview transports
accept exactly one JSON object, reject unknown fields, and validate the signal
against the same stable policy vocabulary as persistence. JSON coercion,
trailing objects, unsupported signals, fractional durations, negative
durations, and values above the 30-day bound fail with `400` rather than being
silently rounded or ignored. Tenant selection remains request-context owned;
policy reads, preview, and writes never fall back to another organization's
manager once an organization context is present.
Availability targets add `udp` plus `response_required` and
`open_or_filtered` modes. Request and expected-response payloads remain
@@ -8257,7 +8264,7 @@ clients. A silent open-or-filtered result is `indeterminate`, not successful
reachability and not a transport failure that may be promoted to an outage.
`internal/api/alerts_endpoints_test.go` proves scopes, revision conflicts,
rollback, and preview behavior.
rollback, exact decoding, invalid-signal rejection, and preview behavior.
`frontend-modern/src/api/__tests__/alertIntentPolicies.test.ts` and the
availability target API and settings tests prove the canonical browser routes
and additive UDP wire fields.
@@ -5263,6 +5263,16 @@ server-owned document after success. Preview renders clear,
expected-transient, pending-grace, and would-activate as distinct states and
never presents preview as a write.
The powered-off default is presented and persisted as the `guest` resource-type
rule so VM and LXC resources inherit it without changing node or agent
connectivity. Blank means inherit, `0` explicitly means no wait, and the UI
states both meanings next to the control. Duration fields accept only base-10
whole seconds from zero through 30 days; an enabled backup hard cap must be
positive. Invalid, fractional, negative, or oversized values remain local,
show an actionable error, and issue no API write. Disabling backup extension is
an explicit `enabled: false` rule and remains separate from disabling a guest's
powered-off alerts.
Availability controls expose UDP mode, request payload, and optional expected
response only where valid for the selected protocol. Unified-resource
presentation keeps `indeterminate` visibly distinct from reachable and
@@ -5273,6 +5283,7 @@ does not infer detector, operator-intent, or recovery truth.
The focused proofs are
`frontend-modern/src/features/alerts/__tests__/AlertIntentPolicyPanel.test.tsx`,
`frontend-modern/src/features/alerts/__tests__/ThresholdsTab.test.tsx`,
`tests/integration/tests/85-powered-off-tolerance.spec.ts`,
`frontend-modern/src/components/Settings/ConnectionEditor/__tests__/AvailabilityTargetSlot.test.tsx`,
and
`frontend-modern/src/utils/__tests__/availabilityProbePresentation.test.ts`.
@@ -2402,6 +2402,11 @@ evidence fails closed for alert deferral, while PBS and provider recovery
evidence continues through its independent mapper and posture rules. An alert
grace period or hard-cap expiry cannot be interpreted as backup success or
failure and grants no restore or infrastructure mutation authority.
The alert runtime measures its base powered-off tolerance and backup hard cap
from server-owned monotonic elapsed time. Backup task timestamps may qualify
fresh matching context but cannot advance, reset, or extend that clock beyond
the cap. A restart conservatively resumes persisted alert progress without
claiming process downtime as backup coverage or recovery evidence.
### Physical-disk collection truth