The alert engine fired powered-off alerts for every stopped VM and
container regardless of whether the guest was configured to autostart.
This meant intentionally-stopped guests (onboot=0) generated alarm
fatigue — in the live lab, 7 of 12 alerts were noise from stopped VMs
that are deliberately off.
Now the alert engine checks the Proxmox onboot setting before firing:
- onboot=true → stopped is unexpected, alert (preserved behavior)
- onboot=false → stopped is expected, suppress
- onboot=nil → unknown, alert (preserved behavior)
For containers, onboot is parsed from the already-fetched container
config in enrichContainerMetadata (no new API call). For VMs, the
config is fetched only for stopped VMs via GetVMConfig (one extra call
per stopped VM, running VMs are unaffected).
Models: add OnBoot *bool to VM and Container.
Alerts: add onboot-aware suppression branch in CheckGuest.
Monitoring: add parseProxmoxOnBoot helper and fetchVMOnBoot.
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
Clears the dupl pairs outside internal/api:
- internal/alerts/pmg.go: the total/deferred/hold per-node queue checks
share evaluatePMGNodeQueueAlert; the historical short-circuit (a
below-threshold clear or invalid spec skips the node's remaining
checks) is preserved via the helper's skip-node return.
- internal/vmware/provider.go: the cloneInventory* family rides generic
cloneSliceWith / cloneShallowSlice helpers instead of fourteen copies
of the same nil/make/loop scaffold.
- internal/vmware/client.go + client_signals.go: the byte-identical
Automation and VI/JSON fetchers delegate to one getSessionScopedJSON.
- internal/vmware/fixtures.go: added to the dupl exclude list in
.golangci.yml — literal mock fixture catalogs are the same category as
the existing internal/mock/ exclusion.
- pkg/licensing/license_server_client.go: Activate and
ExchangeLegacyLicense share postActivationRequest (idempotent POST +
shared activation response decode).
- pkg/proxmox/client.go: LXC/VM RRD fetches share getGuestRRDData.
- pkg/pulsecli/actions.go is intentionally left for the api-contracts
slice.
Full test suites pass for internal/alerts, internal/vmware,
pkg/licensing, pkg/proxmox.
Follow-up to a85ec40a4 (#19), caught by independent re-review. Ceph pool
storage ("<instance>-ceph-pool-<name>") is raised/cleared by the separate
Ceph poll path (checkCephPoolStorage) and is NOT in the allStorage
inventory passed to SyncStorageAlertsForInstance — which runs before
pollCephCluster. Without a guard the sweep cleared the ceph pool usage
alert every storage poll and pollCephCluster re-raised it every cycle
(flapping: notification spam + StartTime reset). Skip any alert whose
ResourceID contains '-ceph-pool-'. Adds a regression test.
Back-port v5 fix 117ec901d to v6, adapted to v6's config subpackage.
Adds NormalizePBSDefaults (called from UpdateConfig alongside the other
Normalize*Defaults) and pbs.cpu/pbs.memory to ValidateHysteresisThresholds.
PBS was the only monitored resource type whose thresholds were not
normalized on UpdateConfig, so user-supplied configs with nil/negative
PBS thresholds were never defaulted and a Trigger=0 'disable' sentinel
did not clear the Clear value. Adds regression test.
Back-port v5 fix a4834ed80 to v6. hostDisplayName now appends
'(Host Agent)' (or returns 'Host Agent' for an unnamed agent) when the
host is linked to a Proxmox node/VM/container, and the disk-temperature
resource name uses hostDisplayName(host) instead of host.DisplayName.
Without this, a host agent running alongside the Proxmox node/VM it is
linked to produced alerts with an identical name, making them
indistinguishable. Adapted to v6's 'Agent' fallback. Adds test cases.
Back-port v5 fix 398ef8117 to v6, adapted to v6's canonical alert keying.
pollStorageWithNodes now calls SyncStorageAlertsForInstance after the
per-storage checks, clearing storage-inventory alerts whose ResourceID is
no longer present. Matches by ResourceID (every v6 storage alert sets
ResourceID=storage.ID) rather than reconstructed alert keys, so it is
robust to v6's composite alert-state IDs (e.g. ...::metric-threshold:usage).
Conservative: live storage keeps its alert (ResourceID still valid),
non-storage alerts are untouched, Ceph alias IDs are treated as valid.
Without this, a storage that disappears or changes identity left its
alert visible until the multi-day stale-alert TTL. Adds unit + poll
integration regression tests.
Back-port v5 fix 9fb76579c to v6. CheckBackupsWithInventory now filters
same-VMID guest candidates by backup kind (qemu/lxc) before attributing
a backup, in both the canonical-key lookup and the bare-VMID secondary
path. Without this, a vm backup with vmid=101 was attributed to an lxc
container with the same vmid (or vice-versa), suppressing the
orphaned-backup alert and attaching the wrong guest identity.
Adds guestMatchesBackupType/filterGuestsByBackupType helpers and a
regression test.
Verified via deadcode/ts-prune detection + 317-agent cross-repo verification swarm.
Gated: go build + go vet + test-compile (pulse module), tsc --noEmit (full project
incl tests), npm run lint. Net -2549 lines across 161 files.
Apply max alerts/hour to cooldown and critical re-notification paths so active alerts cannot bypass the UI delivery cap.
Persist LastNotified on active alert records when dispatching, including clone dispatches after reload.
Refs #1444
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
Seeds AlertConfig.DiskFillByType in defaultAlertConfig() with the three
forward-compatible keys (nvme 92/87, sata 90/85, hdd 85/80). Adds
NormalizeDiskFillByType to lowercase keys on load, seed defaults when
the map is nil, and reset non-positive trigger/clear values to the
default for that key. NormalizeAgentDefaults invokes the new helper so
it runs through the existing UpdateConfig normalization chain. Round
trip tests cover nil seed, customized survival, negative reset, and
mixed case normalization, plus a defaultAlertConfig seed assertion.
Only the "nvme" key is consulted by the host evaluation today (the
inference helper only matches /dev/nvme*); "sata" and "hdd" are
forward-compatible placeholders for when the agent protocol carries
hardware type explicitly.
Inserts a per-type threshold lookup before the global AgentDefaults.Disk
fallback in CheckHost's disk evaluation. inferDiskHardwareType maps the
disk device path to "nvme" (only NVMe is inferable today); when the
DiskFillByType map carries a matching key, that hysteresis threshold is
used, otherwise evaluation falls through to the existing global default.
Storage-type unified_eval branch is unchanged.
Tests prove: (a) NVMe device at 91% does not alert when DiskFillByType
sets the nvme trigger to 92; (b) the same device at 93% alerts at the
nvme threshold; (c) /dev/sda1 falls back to the global threshold; (d)
the storage-type alert branch is not regressed.
Adds DiskFillByType map to AlertConfig (keys: nvme, sata, hdd ->
HysteresisThreshold). Adds inferDiskHardwareType helper that returns
"nvme" for /dev/nvme* device paths (case-insensitive) and "" for
anything else, since sata vs hdd cannot be reliably inferred from
device paths today. Unit tests cover the four input shapes (nvme,
sata, empty, mixed case).
Substrate only. No wire-in to host alert evaluation yet.
When the alert manager suppresses an alert because it is flapping, today
the suppression is silent: the operator sees no alert and no diagnosis.
That trains people not to trust Pulse on flapping resources -- the
detector did its job, but the user just sees absence.
Add a one-shot Manager callback that fires exactly when a tracking key
crosses from quiet into the flapping cooldown window. The callback is
the hook downstream surfaces (patrol, findings) will use to explain
what is flapping and why Pulse stopped notifying.
Lock-safety: checkFlappingLocked now returns (suppress, justTransitioned).
The caller (dispatchAlert) reads the transition signal, then dispatches
the callback from a fresh goroutine. The alerts manager mutex is never
held while the callback runs, so subscribers may take their own locks
or re-enter the alerts package without deadlock.
Subsequent dispatches inside the cooldown window remain silent: the
transition flag is only set on the call that flips flappingActive from
false to true.
A Proxmox host wedged on a ZFS deadlock yesterday took the cluster API poll
with it (context deadline exceeded). The unified connections aggregator
flipped the Connection from active to stale to unreachable, and the
Settings / Infrastructure page rendered the right badges, but no top-nav
alert ever fired because nothing was actively notifying off that derived
state. Patrol's deterministic triage flagged it every minute, but its LLM
investigation stage has been broken since 2026-02-26 so flags never
escalated into user-visible findings. Result: a 3 hour outage I only
noticed because I happened to open Settings.
This wires an active notification off the same connection state the
Settings badges already use:
- internal/alerts/connection.go: new CheckConnection +
clearConnectionDegradedAlert that fire connection-degraded after three
consecutive stale or unreachable observations. Severity scales: stale
warning, unreachable / unauthorized critical. Clear runs through the
same recovery-confirmation gate as clearNodeOfflineAlert so a single
flap back to active doesn't silently resolve a real outage. Paused,
disabled, and non-platform connections are no-ops.
- internal/api/connections_alerts.go: snapshot translator that turns
api.Connection into the narrow alerts.ConnectionSnapshot view. Keeping
the snapshot type inside the alerts package preserves the existing
api -> monitoring import direction; the monitor would have cycled if
it called back into api directly.
- internal/monitoring: new SetConnectionsSnapshotLister hook + a
per-tick checkConnectionAlerts call in the main poll loop, alongside
the existing evaluate*Agents passes.
- internal/api/router.go: register the lister closure on r.monitor so
the alerts loop sees the same Connection rows the HTTP handler does.
- internal/alerts/specs/types.go: add "connection" to the migration
bridge list of accepted ResourceTypes, alongside node / docker-host /
proxmox-disk / etc. The connection concept doesn't have a canonical
unified resource type yet; this matches the existing pattern for
alert-keyed resources that aren't first-class canonical.
Test coverage in internal/alerts/connection_test.go covers active never
fires, three stale observations escalate from pending to warning,
unreachable escalates warning to critical, unauthorized fires critical
cold, paused / disabled / agent never fire, recovery confirmation gate,
and a stale flap during recovery resets the gate.
TestResourceAlertSpecValidateAllowsConnectionMigrationBridgeType mirrors
the existing migration-bridge proof tests for the new type.
CleanupAlertsForNodes removes alerts whose Node isn't in
existingNodes. That map is built upstream only from state.Nodes
(Proxmox nodes) and state.PBSInstances (PBS instances) — no agent
resources. Agent-sourced alerts (Unraid, standalone Linux hosts,
TrueNAS, anything reached via Pulse Agent) typically have Node=""
or an agent UUID, so they fall into the cleanup branch every
cycle. Then the next poll re-creates the alert as new, calls
AddAlert, and appends a fresh history row. Observed in the wild:
3,980 alert history entries in 7 days, with the same canonical
alert ID (e.g. "Unraid array running without parity protection")
appearing every 30 seconds.
Adds a carve-out for ResourceID prefixes starting with "agent:",
matching the existing pattern for "docker-" / "docker:" and
"pbs-" / "pbs-offline". Locks in the behaviour with a new subtest
that mixes agent-sourced and Proxmox-sourced alerts and asserts
that only the legitimately stale Proxmox alert is removed.
Delete the residual internal/alerts/alerts.go catch-all and move its remaining ownership groups into config_facade, model, constants, metric_hooks, manager, default_config, lifecycle, and escalation files.
Keep compatibility aliases in the alerts package, preserve the type-alias identity test, add a default-config pointer isolation regression test, and record the new ownership split in the alerts subsystem contract.
Proof: git diff --check -- docs/release-control/v6/internal/subsystems/alerts.md internal/alerts
Proof: go test ./internal/alerts/... -run 'TestDefaultAlertConfigUsesIndependentBackupAlertOrphanedPointer|TestTypeAliasIdentity|TestSetMetricHooks|TestSetLicenseCheckerStoresChecker|TestHistoryManager_Stop|TestEscalationDisabledWhenAlertsDisabled|TestEscalationDisabledWhenActivationPending|TestEscalationDisabledWhenActivationSnoozed|TestEscalationSkipsWhenScheduleDisabled|TestEscalationSkipsAcknowledgedAlerts|TestEscalationAdvancesLevels|TestEscalationDoesNotRepeatSameLevel|TestEscalationUsesCallback' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move active alert TTL cleanup, auto-ack cleanup, node cleanup, and full active-state reset into internal/alerts/active_cleanup.go.
Record active_cleanup.go in the alerts subsystem contract and broaden ClearActiveAlerts coverage for alias, Docker, recovery, and canonical acknowledgement maps.
Proof: go test ./internal/alerts/... -run 'TestClearActiveAlertsEmptyMaps|TestClearActiveAlertsWithExistingAlerts|TestCleanup|TestCleanupAlertsForNodes' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move active alert acknowledgement, manual clear, recovery clear, state preservation, resolved registration, and no-lock removal helpers into internal/alerts/active_lifecycle.go.
Record active_lifecycle.go in the alerts subsystem contract and add a canonical-alias manual-clear characterization test.
Proof: go test ./internal/alerts/... -run 'TestClearAlertMarksResolutionAndReturnsStatus|TestClearAlertByCanonicalAliasRemovesActiveState|TestAddRecentlyResolvedUsesCanonicalStorageKey|TestAcknowledgeAlertNotFound|TestUnacknowledgeAlertSuccess|TestUnacknowledgeAlertByCanonicalAlias' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move active alert save/load, secure storage handling, startup restore migration, periodic persistence, and stale tracking cleanup into dedicated alerts package owners.
Record active_persistence.go and tracking_cleanup.go in the alerts subsystem contract and add a load-time file-permission hardening characterization test.
Proof: go test ./internal/alerts/... -run 'TestCleanupStaleMaps|TestLoadActiveAlerts|TestLoadActiveAlertsHardensExistingFilePermissions|TestSaveAndLoadActiveAlerts_UsesManagerDataDirAndSecurePermissions|TestSaveActiveAlertsBackfillsCanonicalIdentityOnDiskWithoutMutatingLiveAlert' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move alert configuration normalization, activation migration, global-disable cleanup, active-alert reevaluation, and threshold override cloning into internal/alerts/config_runtime.go.
Record config_runtime.go as the alert config runtime owner in the alerts subsystem contract and add an activation-state preservation characterization test.
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move guest metric projection, per-disk guest metric evaluation, powered-off lifecycle alerts, Pulse tag controls, relaxed thresholds, and guest suppression cleanup into internal/alerts/guest.go.
Record guest.go as the Proxmox guest alert owner in the alerts subsystem contract and add a Pulse-tag characterization test.
Proof: go test ./internal/alerts/...
Move storage-health reason normalization, ZFS assessment helpers, active health alert value lookup, and canonical health-assessment alert synchronization into internal/alerts/health_assessment.go.
Record health_assessment.go as the shared storage-health assessment owner in the alerts subsystem contract and add a reason-code characterization test.
Proof: go test ./internal/alerts/...
Move threshold lookup, per-metric delay resolution, legacy metric alert lifecycle, metric options, and shared metric key helpers into internal/alerts/metric_runtime.go.
Record metric_runtime.go as the shared metric-threshold runtime owner in the alerts subsystem contract and add a metric-options characterization test.
Proof: go test ./internal/alerts/...
Move Proxmox disk canonical identity, disk health assessment, known-firmware suppression, and SSD wearout alerting into internal/alerts/disk_health.go.
Keep the Manager API unchanged while recording disk_health.go as the Proxmox disk-health checker owner in the alerts subsystem contract.
Proof: go test ./internal/alerts/...
Move snapshot age and size evaluation, backup rollup age evaluation, inventory readiness, namespace disambiguation, template matching, and backup/snapshot cleanup into internal/alerts/backup_snapshot.go.
Keep the generic async active-alert save helper in the central package because canonical metric migration still shares it, and record backup_snapshot.go as the backup/snapshot owner in the alerts subsystem contract.
Proof: go test ./internal/alerts/...
Move host-agent identity, metric projection, disk/SMART/RAID/Unraid health handling, cleanup, and offline lifecycle into internal/alerts/host.go.
Keep shared health-assessment evaluation package-level for now because storage ZFS and host SMART/RAID still share that bridge, while recording host.go as the host checker owner in the alerts subsystem contract.
Proof: go test ./internal/alerts/...
Move Proxmox node metric, temperature, offline lifecycle, host-agent deduplication, and node display-name cache support into internal/alerts/node.go.
Keep the Manager API unchanged while recording the node checker owner in the alerts subsystem contract and adding a focused display-name cache key characterization.
Proof: go test ./internal/alerts/...
Move PBS connectivity and metric evaluation into internal/alerts/pbs.go, and move storage connectivity, usage, and ZFS health evaluation into internal/alerts/storage.go.
Keep the Manager API unchanged while recording PBS and storage as resource-checker owners in the alerts subsystem contract, with focused characterization tests for PBS offline normalization and ZFS device labels.
Proof: go test ./internal/alerts/...
Move Docker host connectivity, container state and health, metric projection, service gap/update-state checks, image update timing, and Docker tracking cleanup into internal/alerts/docker.go.
Keep the Manager API unchanged while recording Docker as the resource-checker owner and strengthening Docker resource ID normalization proof.
Proof: go test ./internal/alerts/...
Move PMG connectivity, queue, per-node queue, quarantine, and anomaly evaluation into internal/alerts/pmg.go while keeping the Manager API unchanged.
Record PMG as the resource-checker owner in the alerts contract and add a PMG connection-health normalization proof through CheckPMG.
Proof: go test ./internal/alerts/...
Proof: go test ./internal/monitoring
Move active-alert projection, sorting, metadata coercion, recently resolved reads, history wrappers, and notify-existing redispatch into internal/alerts/read_model.go.
Keep the Manager API unchanged while recording the read-side alerts contract owner and strengthening resolved-alert clone proof.
Proof: go test ./internal/alerts/...
Move alert dispatch, flapping suppression, quiet-hours suppression, monitor-only suppression, cooldown, and rate-limit policy into internal/alerts/notification_policy.go.
Keep Manager behavior and public API unchanged while recording the new alerts contract owner and adding monitor-only dispatch proof.
Proof: go test ./internal/alerts/...; go test -json ./internal/api -count=1; go test ./internal/api ./internal/monitoring ./internal/ai/... ./internal/websocket (internal/api did not reproduce on isolated rerun; other packages passed in the broad run).
Extract alert config types, normalization, and identity helpers into internal/alerts/config while preserving the existing alerts package API through aliases and wrappers.
Move Manager callback lifecycle state into a same-package callbackBus, keeping public Set/Subscribe methods unchanged.
Harden metrics SQLite artifacts to owner-only permissions and cover permissive umask behavior.
Proof: go test -json ./internal/api -count=1; go test ./internal/alerts/... ./internal/monitoring ./internal/ai/... ./internal/websocket ./internal/config ./pkg/metrics; go test ./internal/alerts/... ./pkg/metrics
Preserve the most severe Proxmox connection member state so offline members roll up correctly.
Move history-chart tooltips beside the hovered point when space allows and resolve PBS metric alerts against PBS thresholds during config reevaluation.
Refs #1441
Refs #1452
Refs https://github.com/rcourtman/Pulse/discussions/1448
rcourtman