Seeds AlertConfig.DiskFillByType in defaultAlertConfig() with the three
forward-compatible keys (nvme 92/87, sata 90/85, hdd 85/80). Adds
NormalizeDiskFillByType to lowercase keys on load, seed defaults when
the map is nil, and reset non-positive trigger/clear values to the
default for that key. NormalizeAgentDefaults invokes the new helper so
it runs through the existing UpdateConfig normalization chain. Round
trip tests cover nil seed, customized survival, negative reset, and
mixed case normalization, plus a defaultAlertConfig seed assertion.
Only the "nvme" key is consulted by the host evaluation today (the
inference helper only matches /dev/nvme*); "sata" and "hdd" are
forward-compatible placeholders for when the agent protocol carries
hardware type explicitly.
Inserts a per-type threshold lookup before the global AgentDefaults.Disk
fallback in CheckHost's disk evaluation. inferDiskHardwareType maps the
disk device path to "nvme" (only NVMe is inferable today); when the
DiskFillByType map carries a matching key, that hysteresis threshold is
used, otherwise evaluation falls through to the existing global default.
Storage-type unified_eval branch is unchanged.
Tests prove: (a) NVMe device at 91% does not alert when DiskFillByType
sets the nvme trigger to 92; (b) the same device at 93% alerts at the
nvme threshold; (c) /dev/sda1 falls back to the global threshold; (d)
the storage-type alert branch is not regressed.
Adds DiskFillByType map to AlertConfig (keys: nvme, sata, hdd ->
HysteresisThreshold). Adds inferDiskHardwareType helper that returns
"nvme" for /dev/nvme* device paths (case-insensitive) and "" for
anything else, since sata vs hdd cannot be reliably inferred from
device paths today. Unit tests cover the four input shapes (nvme,
sata, empty, mixed case).
Substrate only. No wire-in to host alert evaluation yet.
When the alert manager suppresses an alert because it is flapping, today
the suppression is silent: the operator sees no alert and no diagnosis.
That trains people not to trust Pulse on flapping resources -- the
detector did its job, but the user just sees absence.
Add a one-shot Manager callback that fires exactly when a tracking key
crosses from quiet into the flapping cooldown window. The callback is
the hook downstream surfaces (patrol, findings) will use to explain
what is flapping and why Pulse stopped notifying.
Lock-safety: checkFlappingLocked now returns (suppress, justTransitioned).
The caller (dispatchAlert) reads the transition signal, then dispatches
the callback from a fresh goroutine. The alerts manager mutex is never
held while the callback runs, so subscribers may take their own locks
or re-enter the alerts package without deadlock.
Subsequent dispatches inside the cooldown window remain silent: the
transition flag is only set on the call that flips flappingActive from
false to true.
A Proxmox host wedged on a ZFS deadlock yesterday took the cluster API poll
with it (context deadline exceeded). The unified connections aggregator
flipped the Connection from active to stale to unreachable, and the
Settings / Infrastructure page rendered the right badges, but no top-nav
alert ever fired because nothing was actively notifying off that derived
state. Patrol's deterministic triage flagged it every minute, but its LLM
investigation stage has been broken since 2026-02-26 so flags never
escalated into user-visible findings. Result: a 3 hour outage I only
noticed because I happened to open Settings.
This wires an active notification off the same connection state the
Settings badges already use:
- internal/alerts/connection.go: new CheckConnection +
clearConnectionDegradedAlert that fire connection-degraded after three
consecutive stale or unreachable observations. Severity scales: stale
warning, unreachable / unauthorized critical. Clear runs through the
same recovery-confirmation gate as clearNodeOfflineAlert so a single
flap back to active doesn't silently resolve a real outage. Paused,
disabled, and non-platform connections are no-ops.
- internal/api/connections_alerts.go: snapshot translator that turns
api.Connection into the narrow alerts.ConnectionSnapshot view. Keeping
the snapshot type inside the alerts package preserves the existing
api -> monitoring import direction; the monitor would have cycled if
it called back into api directly.
- internal/monitoring: new SetConnectionsSnapshotLister hook + a
per-tick checkConnectionAlerts call in the main poll loop, alongside
the existing evaluate*Agents passes.
- internal/api/router.go: register the lister closure on r.monitor so
the alerts loop sees the same Connection rows the HTTP handler does.
- internal/alerts/specs/types.go: add "connection" to the migration
bridge list of accepted ResourceTypes, alongside node / docker-host /
proxmox-disk / etc. The connection concept doesn't have a canonical
unified resource type yet; this matches the existing pattern for
alert-keyed resources that aren't first-class canonical.
Test coverage in internal/alerts/connection_test.go covers active never
fires, three stale observations escalate from pending to warning,
unreachable escalates warning to critical, unauthorized fires critical
cold, paused / disabled / agent never fire, recovery confirmation gate,
and a stale flap during recovery resets the gate.
TestResourceAlertSpecValidateAllowsConnectionMigrationBridgeType mirrors
the existing migration-bridge proof tests for the new type.
CleanupAlertsForNodes removes alerts whose Node isn't in
existingNodes. That map is built upstream only from state.Nodes
(Proxmox nodes) and state.PBSInstances (PBS instances) — no agent
resources. Agent-sourced alerts (Unraid, standalone Linux hosts,
TrueNAS, anything reached via Pulse Agent) typically have Node=""
or an agent UUID, so they fall into the cleanup branch every
cycle. Then the next poll re-creates the alert as new, calls
AddAlert, and appends a fresh history row. Observed in the wild:
3,980 alert history entries in 7 days, with the same canonical
alert ID (e.g. "Unraid array running without parity protection")
appearing every 30 seconds.
Adds a carve-out for ResourceID prefixes starting with "agent:",
matching the existing pattern for "docker-" / "docker:" and
"pbs-" / "pbs-offline". Locks in the behaviour with a new subtest
that mixes agent-sourced and Proxmox-sourced alerts and asserts
that only the legitimately stale Proxmox alert is removed.
Delete the residual internal/alerts/alerts.go catch-all and move its remaining ownership groups into config_facade, model, constants, metric_hooks, manager, default_config, lifecycle, and escalation files.
Keep compatibility aliases in the alerts package, preserve the type-alias identity test, add a default-config pointer isolation regression test, and record the new ownership split in the alerts subsystem contract.
Proof: git diff --check -- docs/release-control/v6/internal/subsystems/alerts.md internal/alerts
Proof: go test ./internal/alerts/... -run 'TestDefaultAlertConfigUsesIndependentBackupAlertOrphanedPointer|TestTypeAliasIdentity|TestSetMetricHooks|TestSetLicenseCheckerStoresChecker|TestHistoryManager_Stop|TestEscalationDisabledWhenAlertsDisabled|TestEscalationDisabledWhenActivationPending|TestEscalationDisabledWhenActivationSnoozed|TestEscalationSkipsWhenScheduleDisabled|TestEscalationSkipsAcknowledgedAlerts|TestEscalationAdvancesLevels|TestEscalationDoesNotRepeatSameLevel|TestEscalationUsesCallback' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move active alert TTL cleanup, auto-ack cleanup, node cleanup, and full active-state reset into internal/alerts/active_cleanup.go.
Record active_cleanup.go in the alerts subsystem contract and broaden ClearActiveAlerts coverage for alias, Docker, recovery, and canonical acknowledgement maps.
Proof: go test ./internal/alerts/... -run 'TestClearActiveAlertsEmptyMaps|TestClearActiveAlertsWithExistingAlerts|TestCleanup|TestCleanupAlertsForNodes' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move active alert acknowledgement, manual clear, recovery clear, state preservation, resolved registration, and no-lock removal helpers into internal/alerts/active_lifecycle.go.
Record active_lifecycle.go in the alerts subsystem contract and add a canonical-alias manual-clear characterization test.
Proof: go test ./internal/alerts/... -run 'TestClearAlertMarksResolutionAndReturnsStatus|TestClearAlertByCanonicalAliasRemovesActiveState|TestAddRecentlyResolvedUsesCanonicalStorageKey|TestAcknowledgeAlertNotFound|TestUnacknowledgeAlertSuccess|TestUnacknowledgeAlertByCanonicalAlias' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move active alert save/load, secure storage handling, startup restore migration, periodic persistence, and stale tracking cleanup into dedicated alerts package owners.
Record active_persistence.go and tracking_cleanup.go in the alerts subsystem contract and add a load-time file-permission hardening characterization test.
Proof: go test ./internal/alerts/... -run 'TestCleanupStaleMaps|TestLoadActiveAlerts|TestLoadActiveAlertsHardensExistingFilePermissions|TestSaveAndLoadActiveAlerts_UsesManagerDataDirAndSecurePermissions|TestSaveActiveAlertsBackfillsCanonicalIdentityOnDiskWithoutMutatingLiveAlert' -count=1
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move alert configuration normalization, activation migration, global-disable cleanup, active-alert reevaluation, and threshold override cloning into internal/alerts/config_runtime.go.
Record config_runtime.go as the alert config runtime owner in the alerts subsystem contract and add an activation-state preservation characterization test.
Proof: go test ./internal/alerts/... -count=1
Proof: go test ./internal/api -run Alert -count=1
Move guest metric projection, per-disk guest metric evaluation, powered-off lifecycle alerts, Pulse tag controls, relaxed thresholds, and guest suppression cleanup into internal/alerts/guest.go.
Record guest.go as the Proxmox guest alert owner in the alerts subsystem contract and add a Pulse-tag characterization test.
Proof: go test ./internal/alerts/...
Move storage-health reason normalization, ZFS assessment helpers, active health alert value lookup, and canonical health-assessment alert synchronization into internal/alerts/health_assessment.go.
Record health_assessment.go as the shared storage-health assessment owner in the alerts subsystem contract and add a reason-code characterization test.
Proof: go test ./internal/alerts/...
Move threshold lookup, per-metric delay resolution, legacy metric alert lifecycle, metric options, and shared metric key helpers into internal/alerts/metric_runtime.go.
Record metric_runtime.go as the shared metric-threshold runtime owner in the alerts subsystem contract and add a metric-options characterization test.
Proof: go test ./internal/alerts/...
Move Proxmox disk canonical identity, disk health assessment, known-firmware suppression, and SSD wearout alerting into internal/alerts/disk_health.go.
Keep the Manager API unchanged while recording disk_health.go as the Proxmox disk-health checker owner in the alerts subsystem contract.
Proof: go test ./internal/alerts/...
Move snapshot age and size evaluation, backup rollup age evaluation, inventory readiness, namespace disambiguation, template matching, and backup/snapshot cleanup into internal/alerts/backup_snapshot.go.
Keep the generic async active-alert save helper in the central package because canonical metric migration still shares it, and record backup_snapshot.go as the backup/snapshot owner in the alerts subsystem contract.
Proof: go test ./internal/alerts/...
Move host-agent identity, metric projection, disk/SMART/RAID/Unraid health handling, cleanup, and offline lifecycle into internal/alerts/host.go.
Keep shared health-assessment evaluation package-level for now because storage ZFS and host SMART/RAID still share that bridge, while recording host.go as the host checker owner in the alerts subsystem contract.
Proof: go test ./internal/alerts/...
Move Proxmox node metric, temperature, offline lifecycle, host-agent deduplication, and node display-name cache support into internal/alerts/node.go.
Keep the Manager API unchanged while recording the node checker owner in the alerts subsystem contract and adding a focused display-name cache key characterization.
Proof: go test ./internal/alerts/...
Move PBS connectivity and metric evaluation into internal/alerts/pbs.go, and move storage connectivity, usage, and ZFS health evaluation into internal/alerts/storage.go.
Keep the Manager API unchanged while recording PBS and storage as resource-checker owners in the alerts subsystem contract, with focused characterization tests for PBS offline normalization and ZFS device labels.
Proof: go test ./internal/alerts/...
Move Docker host connectivity, container state and health, metric projection, service gap/update-state checks, image update timing, and Docker tracking cleanup into internal/alerts/docker.go.
Keep the Manager API unchanged while recording Docker as the resource-checker owner and strengthening Docker resource ID normalization proof.
Proof: go test ./internal/alerts/...
Move PMG connectivity, queue, per-node queue, quarantine, and anomaly evaluation into internal/alerts/pmg.go while keeping the Manager API unchanged.
Record PMG as the resource-checker owner in the alerts contract and add a PMG connection-health normalization proof through CheckPMG.
Proof: go test ./internal/alerts/...
Proof: go test ./internal/monitoring
Move active-alert projection, sorting, metadata coercion, recently resolved reads, history wrappers, and notify-existing redispatch into internal/alerts/read_model.go.
Keep the Manager API unchanged while recording the read-side alerts contract owner and strengthening resolved-alert clone proof.
Proof: go test ./internal/alerts/...
Move alert dispatch, flapping suppression, quiet-hours suppression, monitor-only suppression, cooldown, and rate-limit policy into internal/alerts/notification_policy.go.
Keep Manager behavior and public API unchanged while recording the new alerts contract owner and adding monitor-only dispatch proof.
Proof: go test ./internal/alerts/...; go test -json ./internal/api -count=1; go test ./internal/api ./internal/monitoring ./internal/ai/... ./internal/websocket (internal/api did not reproduce on isolated rerun; other packages passed in the broad run).
Extract alert config types, normalization, and identity helpers into internal/alerts/config while preserving the existing alerts package API through aliases and wrappers.
Move Manager callback lifecycle state into a same-package callbackBus, keeping public Set/Subscribe methods unchanged.
Harden metrics SQLite artifacts to owner-only permissions and cover permissive umask behavior.
Proof: go test -json ./internal/api -count=1; go test ./internal/alerts/... ./internal/monitoring ./internal/ai/... ./internal/websocket ./internal/config ./pkg/metrics; go test ./internal/alerts/... ./pkg/metrics
Preserve the most severe Proxmox connection member state so offline members roll up correctly.
Move history-chart tooltips beside the hovered point when space allows and resolve PBS metric alerts against PBS thresholds during config reevaluation.
Refs #1441
Refs #1452
Refs https://github.com/rcourtman/Pulse/discussions/1448
Parse the /proc/mdstat operation keyword for mdadm arrays and propagate it through host reports, models, unified resources, monitoring views, alert metadata, and AI storage summaries.
Treat recovery and reshape as rebuild signals while silencing routine check and resync maintenance, with fallback rebuild detection only when no mdstat operation is available.
Tests cover mdstat operation parsing plus recovery, check, and resync alert behavior.
Fixes#1446
Fix snapshot, backup, powered-off, and config-reevaluation guest threshold resolution by routing them through shared guest alert context instead of override-only fallback paths.\n\nFixes #1418
The 10ms goroutine drain pause was insufficient under full parallel
test suite load, causing intermittent failures in
TestPulseMonitorOnlySkipsDispatchButRetainsAlert.
rcourtman