mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-26 21:47:20 +00:00
Compose retained disk policy and attribution work with current main
Preserve exact retained Core ancestry while retaining the accepted Assistant target, webhook confidentiality, log-viewer and installer repairs. Qualify the composed source against the original seven-package, inventory-control and real embed/API/Patrol recipe; source acceptance remains separate from native recovery. Change-source: pulse-maintainer
This commit is contained in:
@@ -9259,6 +9259,18 @@ carrying a prefixed attribute or namespace declaration named like a SAML
|
||||
attribute is refused. No agent
|
||||
registration, enrolment, install, update or removal path changed.
|
||||
|
||||
### Mock-mode read views fold linked fixture agents
|
||||
|
||||
`internal/monitoring/monitor.go` changed only so the mock-mode unified view
|
||||
applies the operator's manual links (unified-resources contract, "Mock-mode
|
||||
unified view applies operator links"). A fixture agent the operator linked
|
||||
into a guest now folds into that guest in mock read views, as a linked agent
|
||||
already does in live mode, so host snapshots built from them
|
||||
(`HostsSnapshot`) stop listing it while the guest carries its facet. Agent
|
||||
registration, enrolment, report admission, removal, continuity and the
|
||||
fixture agent set itself are unchanged, and the view writes nothing to the
|
||||
resource store.
|
||||
|
||||
### Demo write guard ignores websocket upgrade headers
|
||||
|
||||
`internal/api/demo_middleware.go` changed only so a `POST`, `PUT`, `PATCH` or
|
||||
@@ -9321,3 +9333,14 @@ machine setting. Cleanup restores only the captured prior setting, even when
|
||||
owned server teardown fails; an unconfirmed stop fails and retains the handle
|
||||
rather than reporting clean teardown. Real Windows execution and full terminal
|
||||
cleanup evidence still have to be returned by the existing native CI job.
|
||||
|
||||
### Reconciled node-agent relink fold bookkeeping
|
||||
|
||||
A declared node-agent join can consume one split row before its newer
|
||||
operator relink is applied. The relink records that side's ID and source in
|
||||
the current holder-bound `ManualLinkFold` index, so identity-pin succession
|
||||
keeps the operator decision and the resources API seed retains the same fold
|
||||
for report-merge. No synthetic telemetry, identity key, enrollment or service
|
||||
link is added. `TestOperatorSplitOverridesProxmoxNodeAgentLink` checks that
|
||||
the monitor and its re-ingested API seed retain the fold across all split
|
||||
shapes, then survive pin persistence and a repeated split.
|
||||
|
||||
@@ -422,6 +422,17 @@ linked node or guest) replaces the per-type trigger, and an override or agent
|
||||
default that switches the host's alerts off leaves no heat to judge. A disk no
|
||||
agent reports gets the per-type policy. The global alerts switch and the agent
|
||||
alert-type switch stop `CheckHost` without changing this policy, as before.
|
||||
A TrueNAS disk (`alerts.IsTrueNASDiskResource`, the disks the unified evaluator
|
||||
alerts on as `truenas-disk`) skips that walk, because its TrueNAS system carries
|
||||
a synthetic `Agent.AgentID` that would pass for a host agent:
|
||||
`physicalDiskTemperatureLimits` resolves it through
|
||||
`ThresholdProvider.GetTrueNASDiskTemperatureThreshold(resourceID, diskType)`
|
||||
(the alert manager's `TrueNASDiskTemperatureThreshold`, the `truenas-disk`
|
||||
tiers of `effectiveAlertPolicyNoLock`), so the disk's own override, then the
|
||||
TrueNAS-wide TrueNAS Disks value, then the per-type policy decide, as for its
|
||||
temperature alert, and a disabled override or TrueNAS Disks default leaves no
|
||||
heat. A host override under the system's synthetic ID no longer reaches its
|
||||
disks, and Disable all TrueNAS silences the alert without changing this policy.
|
||||
The run state carries the provider (`patrolRuntimeState.thresholdProvider`). With no provider the factory alert
|
||||
configuration applies. A disk is hot from its trigger: a triage warning, a disk
|
||||
issue and an AI chat "needing attention" entry, as its temperature alert and
|
||||
@@ -432,9 +443,13 @@ verification unknown rather than recovered. Scoped runs keep the provider, and a
|
||||
provider set before Patrol starts is handed to it. Below the trigger Patrol flags nothing, so an NVMe at 63C stays
|
||||
quiet and a SATA disk at 56C is flagged. Proofs:
|
||||
`internal/ai/patrol_disk_temperature_test.go` (a real registry with one
|
||||
overridden agent in `TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride`),
|
||||
`TestAlertThresholdAdapter_DiskTemperatureFollowsAlertPolicy` and
|
||||
`TestAlertThresholdAdapter_DiskTemperatureHonoursHostOverrides` in
|
||||
overridden agent in `TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride`,
|
||||
and TrueNAS fixture disks under a per-disk override, a TrueNAS-wide value and a
|
||||
host override on the system ID, one with a retained reading, in
|
||||
`TestPatrolJudgesTrueNASDisksByTheirAlertTiers`),
|
||||
`TestAlertThresholdAdapter_DiskTemperatureFollowsAlertPolicy`,
|
||||
`TestAlertThresholdAdapter_DiskTemperatureHonoursHostOverrides` and
|
||||
`TestAlertThresholdAdapter_TrueNASDiskTemperatureFollowsItsAlertTiers` in
|
||||
`internal/ai/alert_threshold_adapter_test.go`.
|
||||
|
||||
Retained summaries disclose that point and bucket timestamps describe returned
|
||||
|
||||
@@ -4475,3 +4475,27 @@ resubmission and concurrent response encoding/update under the race detector.
|
||||
Existing snapshot, persistence, partial-update and threshold tests remain
|
||||
required. This is source ownership acceptance, not installed notification or
|
||||
guest-agent recovery evidence.
|
||||
|
||||
### TrueNAS disk heat policy for Patrol and Assistant
|
||||
|
||||
`Manager.TrueNASDiskTemperatureThreshold(resourceID, diskType)` resolves the
|
||||
`truenas-disk` tiers of `effectiveAlertPolicyNoLock`: the disk's canonical or
|
||||
identity-resolved override, then TrueNAS Disks defaults, then disk-type policy.
|
||||
A disabled disk/default or non-positive trigger leaves no heat policy.
|
||||
Global/platform alert switches silence alert evaluation without changing this
|
||||
policy, matching existing agent-disk semantics. Callers receive a copy; config
|
||||
normalisation, saved Off/custom/inherited values and alert severity are unchanged.
|
||||
`IsTrueNASDiskResource` uses the evaluator's classification, including merged
|
||||
agent/TrueNAS disks. Patrol/Assistant do not walk those disks to a synthetic
|
||||
TrueNAS system agent and accidentally apply an unrelated host override.
|
||||
Retained temperature readings remain unjudged, not affirmative recovery.
|
||||
|
||||
Verification: `TestTrueNASDiskTemperatureThresholdMatchesTrueNASDiskAlerts`
|
||||
and `TestIsTrueNASDiskResourceMatchesTheUnifiedEvaluator` in
|
||||
`internal/alerts/threshold_resolution_shared_test.go` cover tier, identity,
|
||||
Off/disabled, untyped, clone and nil-manager compatibility.
|
||||
`TestAlertThresholdAdapter_TrueNASDiskTemperatureFollowsItsAlertTiers` and
|
||||
`TestPatrolJudgesTrueNASDisksByTheirAlertTiers` cover adapter, Patrol triage,
|
||||
verification and Assistant context while preserving agent-host overrides.
|
||||
Frontend table/drawer tiers remain a separate Web-owned change; this backend
|
||||
repair does not establish browser or native appliance acceptance.
|
||||
|
||||
@@ -6502,3 +6502,21 @@ and REST row counts with and without the split, and
|
||||
`TestBroadcastPresentationCoalescePrefersTheListingReadState` pins the
|
||||
listing read state over an older store generation and the store's exclusions
|
||||
for a mock-style view.
|
||||
|
||||
### Reconciled node-agent relink fold bookkeeping
|
||||
|
||||
A declared node-agent join can consume one split row before its newer
|
||||
operator relink is applied. The relink records that side's ID and source in
|
||||
the current holder-bound `ManualLinkFold` index, so identity-pin succession
|
||||
keeps the operator decision and the resources API seed retains the same fold
|
||||
for report-merge. No synthetic telemetry, identity key, enrollment or service
|
||||
link is added. `TestOperatorSplitOverridesProxmoxNodeAgentLink` checks that
|
||||
the monitor and its re-ingested API seed retain the fold across all split
|
||||
shapes, then survive pin persistence and a repeated split.
|
||||
|
||||
Mock-link lifecycle validation drives the owning ingest/rebuild boundary
|
||||
after unlink and relink: read hydration intentionally reuses a registry
|
||||
generation younger than two seconds. The decision takes effect in the next
|
||||
loaded generation, not retroactively in the listing already served. The
|
||||
fixture seed keeps default thresholds while the configured live adapter
|
||||
retains its four-minute Proxmox threshold.
|
||||
|
||||
@@ -6950,11 +6950,11 @@ of durable state through it.
|
||||
as before. With links each monitor builds its own linked snapshot whenever
|
||||
the fixture data or links change (concurrent cache misses can each build
|
||||
one).
|
||||
- Exclusions are not carried, as before: the fixture graph makes its own
|
||||
merges before the view seeds it, so fixtures that identity matching merged
|
||||
stay merged ("Unlink replaces the pair's operator link"). Presentation
|
||||
coalescing still honours them, because the broadcast takes the resource
|
||||
store adapter's exclusions for every store-less view.
|
||||
- Exclusions are not carried by the fixture registry, as before. The
|
||||
broadcast's presentation coalescing consults the monitor adapter's
|
||||
exclusions, while the resources API's `ListForPresentation` consults its
|
||||
own store-backed registry's exclusions. Neither can undo an automatic
|
||||
identity merge already made inside the store-free fixture graph.
|
||||
- An agent linked into a guest leaves the agent listings (`Hosts()` lists
|
||||
agent-type rows only) and the guest carries its facet, as in live mode.
|
||||
|
||||
@@ -7362,9 +7362,11 @@ agent's SMART inventory as the node's disk fallback when the Proxmox disk
|
||||
query fails, the agent's disk-exclude patterns on the node's disks, and the
|
||||
node's linked agent that guest discovery, agent deployment and service
|
||||
discovery act on.
|
||||
Physical disks inside one host still join through
|
||||
`resolveLinkedPhysicalDisk` without reading exclusions; a node split puts the
|
||||
two sides' disks under different parents, so they no longer meet there.
|
||||
Physical disks inside one host join through `resolveLinkedPhysicalDisk`,
|
||||
which now honours their operator exclusions ("Operator split separates disks
|
||||
inside a linked host" above, upstream #2748). A node split also puts the two
|
||||
sides' disks under different parents, so they no longer meet there. The node
|
||||
split and the disk split remain independent operator decisions.
|
||||
Cluster- and hostname-derived IDs carry no source, so an exclusion recorded
|
||||
against another resource that derives the node's own ID (a Docker Swarm host
|
||||
reporting no machine key, named like the node in a swarm named like its
|
||||
@@ -7395,11 +7397,12 @@ already has the link applied, so it cannot split the pair itself: an unlink
|
||||
shows on REST, `/api/state` and the websocket from the monitor's next
|
||||
registry rebuild, the same generation in which a new link reaches the
|
||||
broadcast. REST applies a new link at once, because its own registry applies
|
||||
the store's links on top of the seed. The mock-mode view applies the links
|
||||
the resource store's current generation loaded ("Mock-mode unified view
|
||||
applies operator links"), so mock mode follows the same timing. Mock fixtures
|
||||
that identity matching merged stay merged on every surface: the fixture graph
|
||||
is built without the store's exclusions.
|
||||
the store's links on top of the seed. The mock-mode view now applies the
|
||||
current monitor generation's operator links too ("Mock-mode unified view
|
||||
applies operator links" above), so a mock unlink likewise needs that
|
||||
monitor rebuild to remove the fold from the seed. Mock fixtures that identity
|
||||
matching merged stay merged on every surface: the fixture graph is built
|
||||
without the store's exclusions.
|
||||
|
||||
The websocket broadcast coalesced host views with
|
||||
`CoalescePresentationHostResources`, which honours no exclusions, while the
|
||||
@@ -7508,3 +7511,27 @@ system into a VM across repeated record ingests, and
|
||||
`TestManualLinkFoldRepeatedPairKeepsEarlierSources` refolds a side with fewer
|
||||
sources than it first brought. `TestResourceAPIReportMergeExcludesRegistryLinkFolds`
|
||||
keeps the handler on the registry's fold record.
|
||||
|
||||
### Reconciled node-agent relink fold bookkeeping
|
||||
|
||||
A declared node-agent join can consume one split row before its newer
|
||||
operator relink is applied. The relink records that side's ID and source in
|
||||
the current holder-bound `ManualLinkFold` index, so identity-pin succession
|
||||
keeps the operator decision and the resources API seed retains the same fold
|
||||
for report-merge. No synthetic telemetry, identity key, enrollment or service
|
||||
link is added. `TestOperatorSplitOverridesProxmoxNodeAgentLink` checks that
|
||||
the monitor and its re-ingested API seed retain the fold across all split
|
||||
shapes, then survive pin persistence and a repeated split.
|
||||
|
||||
### TrueNAS disk policy consumers use the disk's alert identity
|
||||
|
||||
The AI/Patrol physical-disk consumers classify TrueNAS disks through
|
||||
`alerts.IsTrueNASDiskResource`, the same classifier as unified alert evaluation.
|
||||
They resolve temperature policy under the disk's canonical resource ID, not its
|
||||
parent system's synthetic agent ID. Merged agent/TrueNAS disks retain that
|
||||
classification; ordinary agent disks still follow their reporting host's policy.
|
||||
Registry identity, source precedence and retained-reading semantics are unchanged.
|
||||
`TestIsTrueNASDiskResourceMatchesTheUnifiedEvaluator` and
|
||||
`TestPatrolJudgesTrueNASDisksByTheirAlertTiers` exercise the unified registry
|
||||
projection and its consumer boundaries. Table/drawer frontend acceptance remains
|
||||
with Web, not established by these backend tests.
|
||||
|
||||
@@ -79,6 +79,26 @@ func (a *AlertThresholdAdapter) GetDiskTemperatureThreshold(host alerts.DiskTemp
|
||||
} else {
|
||||
threshold = alerts.DefaultDiskTemperatureThreshold(diskType)
|
||||
}
|
||||
return diskTemperatureTriggerAndClear(threshold)
|
||||
}
|
||||
|
||||
// GetTrueNASDiskTemperatureThreshold returns the temperature alert trigger and
|
||||
// clear values for one TrueNAS disk, resolved exactly as its temperature alert
|
||||
// resolves them (alerts.Manager.TrueNASDiskTemperatureThreshold): the disk's
|
||||
// own override, then the TrueNAS-wide value, then the per-type policy.
|
||||
func (a *AlertThresholdAdapter) GetTrueNASDiskTemperatureThreshold(resourceID, diskType string) (float64, float64) {
|
||||
var threshold *alerts.HysteresisThreshold
|
||||
if a.manager != nil {
|
||||
threshold = a.manager.TrueNASDiskTemperatureThreshold(resourceID, diskType)
|
||||
} else {
|
||||
threshold = alerts.DefaultDiskTemperatureThreshold(diskType)
|
||||
}
|
||||
return diskTemperatureTriggerAndClear(threshold)
|
||||
}
|
||||
|
||||
// diskTemperatureTriggerAndClear unpacks a disk temperature threshold, with a
|
||||
// nil or non-positive trigger meaning off (0, 0).
|
||||
func diskTemperatureTriggerAndClear(threshold *alerts.HysteresisThreshold) (float64, float64) {
|
||||
if threshold == nil || threshold.Trigger <= 0 {
|
||||
return 0, 0
|
||||
}
|
||||
|
||||
@@ -108,6 +108,39 @@ func TestAlertThresholdAdapter_DiskTemperatureFollowsAlertPolicy(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A TrueNAS disk reaches Patrol through its own alert tiers: its override,
|
||||
// then the TrueNAS-wide value, then the per-type policy.
|
||||
func TestAlertThresholdAdapter_TrueNASDiskTemperatureFollowsItsAlertTiers(t *testing.T) {
|
||||
if trigger, clear := NewAlertThresholdAdapter(nil).GetTrueNASDiskTemperatureThreshold("disk-any", "nvme"); trigger != 70 || clear != 65 {
|
||||
t.Fatalf("no manager: nvme = %v/%v, want the factory 70/65", trigger, clear)
|
||||
}
|
||||
|
||||
mgr := alerts.NewManager()
|
||||
t.Cleanup(mgr.Stop)
|
||||
a := NewAlertThresholdAdapter(mgr)
|
||||
if trigger, clear := a.GetTrueNASDiskTemperatureThreshold("disk-plain", "sata"); trigger != 55 || clear != 50 {
|
||||
t.Fatalf("no TrueNAS-wide value: sata = %v/%v, want the per-type 55/50", trigger, clear)
|
||||
}
|
||||
|
||||
cfg := mgr.GetConfig()
|
||||
cfg.TrueNASDiskDefaults.Temperature = &alerts.HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
cfg.Overrides = map[string]alerts.ThresholdConfig{
|
||||
"disk-raised": {Temperature: &alerts.HysteresisThreshold{Trigger: 75, Clear: 70}},
|
||||
"disk-off": {Disabled: true},
|
||||
}
|
||||
mgr.UpdateConfig(cfg)
|
||||
for id, want := range map[string][2]float64{
|
||||
"disk-raised": {75, 70},
|
||||
"disk-plain": {62, 57},
|
||||
"disk-off": {0, 0},
|
||||
} {
|
||||
trigger, clear := a.GetTrueNASDiskTemperatureThreshold(id, "nvme")
|
||||
if trigger != want[0] || clear != want[1] {
|
||||
t.Errorf("%s nvme = %v/%v, want %v/%v", id, trigger, clear, want[0], want[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A host's Disk Temp override reaches Patrol for that host's disks only, and a
|
||||
// host whose alerts are switched off has no disk heat policy at all.
|
||||
func TestAlertThresholdAdapter_DiskTemperatureHonoursHostOverrides(t *testing.T) {
|
||||
|
||||
@@ -73,6 +73,12 @@ func (m *mockThresholdProvider) GetDiskTemperatureThreshold(_ alerts.DiskTempera
|
||||
return 0, 0
|
||||
}
|
||||
|
||||
// GetTrueNASDiskTemperatureThreshold judges a TrueNAS disk by the same
|
||||
// per-type table, as a TrueNAS disk with no override or TrueNAS-wide value is.
|
||||
func (m *mockThresholdProvider) GetTrueNASDiskTemperatureThreshold(_ string, diskType string) (float64, float64) {
|
||||
return m.GetDiskTemperatureThreshold(alerts.DiskTemperatureHost{}, diskType)
|
||||
}
|
||||
|
||||
type mockUnifiedResourceProvider struct {
|
||||
UnifiedResourceProvider
|
||||
getAllFunc func() []unifiedresources.Resource
|
||||
|
||||
@@ -71,6 +71,12 @@ type ThresholdProvider interface {
|
||||
// agent reports. A host with no ID means no agent reports the disk. A zero
|
||||
// trigger means disk temperature alerting is off for it.
|
||||
GetDiskTemperatureThreshold(host alerts.DiskTemperatureHost, diskType string) (trigger, clear float64)
|
||||
// GetTrueNASDiskTemperatureThreshold returns the disk temperature alert
|
||||
// trigger and clear values (°C) for one TrueNAS disk, given its resource
|
||||
// ID and type: the disk's own override, then the TrueNAS-wide value, then
|
||||
// the per-type policy, as its temperature alert resolves them. A zero
|
||||
// trigger means disk temperature alerting is off for it.
|
||||
GetTrueNASDiskTemperatureThreshold(resourceID, diskType string) (trigger, clear float64)
|
||||
}
|
||||
|
||||
// AlertResolver provides the ability to review and resolve alerts
|
||||
|
||||
@@ -2491,7 +2491,7 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool)
|
||||
temperature: tools.SplitDiskTemperature(r.PhysicalDisk.Temperature, r.PhysicalDisk.Collection),
|
||||
smartEvidence: unifiedPhysicalDiskSMARTIssueParts(r.PhysicalDisk.SMART),
|
||||
}
|
||||
row.temperatureLimits = snap.diskTemperatureLimits(physicalDiskTemperatureHost(r, owners), row.diskType)
|
||||
row.temperatureLimits = physicalDiskTemperatureLimits(snap.thresholdProvider, r, owners)
|
||||
rows = append(rows, row)
|
||||
}
|
||||
return rows
|
||||
|
||||
@@ -22,12 +22,53 @@ type diskTemperatureLimits struct {
|
||||
// that host's Disk Temp override applies, or from the factory alert
|
||||
// configuration when Patrol has no threshold provider.
|
||||
func diskTemperatureLimitsFor(provider ThresholdProvider, host alerts.DiskTemperatureHost, diskType string) diskTemperatureLimits {
|
||||
var trigger, clear float64
|
||||
if provider != nil {
|
||||
trigger, clear = provider.GetDiskTemperatureThreshold(host, diskType)
|
||||
} else if threshold := alerts.DefaultDiskTemperatureThreshold(diskType); threshold != nil {
|
||||
trigger, clear = threshold.Trigger, threshold.Clear
|
||||
return newDiskTemperatureLimits(provider.GetDiskTemperatureThreshold(host, diskType))
|
||||
}
|
||||
return factoryDiskTemperatureLimits(diskType)
|
||||
}
|
||||
|
||||
// trueNASDiskTemperatureLimitsFor resolves the policy for one TrueNAS disk the
|
||||
// way its temperature alert does, from the user's alert configuration, so the
|
||||
// disk's own override and the TrueNAS-wide value apply, or from the factory
|
||||
// alert configuration when Patrol has no threshold provider.
|
||||
func trueNASDiskTemperatureLimitsFor(provider ThresholdProvider, resourceID, diskType string) diskTemperatureLimits {
|
||||
if provider != nil {
|
||||
return newDiskTemperatureLimits(provider.GetTrueNASDiskTemperatureThreshold(resourceID, diskType))
|
||||
}
|
||||
return factoryDiskTemperatureLimits(diskType)
|
||||
}
|
||||
|
||||
// physicalDiskTemperatureLimits resolves the policy for one physical disk
|
||||
// resource under the alert that judges its heat. A TrueNAS disk has its own
|
||||
// temperature alert, judged by the disk's override, the TrueNAS-wide value and
|
||||
// the per-type policy; its TrueNAS system carries a synthetic agent ID, so the
|
||||
// host walk would wrongly treat that system as a host agent. Any other disk is
|
||||
// judged as a SMART disk of the host agent that reports it.
|
||||
func physicalDiskTemperatureLimits(provider ThresholdProvider, disk unifiedresources.Resource, owners map[string]unifiedresources.Resource) diskTemperatureLimits {
|
||||
diskType := ""
|
||||
if disk.PhysicalDisk != nil {
|
||||
diskType = disk.PhysicalDisk.DiskType
|
||||
}
|
||||
if alerts.IsTrueNASDiskResource(disk) {
|
||||
return trueNASDiskTemperatureLimitsFor(provider, disk.ID, diskType)
|
||||
}
|
||||
return diskTemperatureLimitsFor(provider, physicalDiskTemperatureHost(disk, owners), diskType)
|
||||
}
|
||||
|
||||
// factoryDiskTemperatureLimits is the policy for a disk of the given type
|
||||
// under the factory alert configuration.
|
||||
func factoryDiskTemperatureLimits(diskType string) diskTemperatureLimits {
|
||||
if threshold := alerts.DefaultDiskTemperatureThreshold(diskType); threshold != nil {
|
||||
return newDiskTemperatureLimits(threshold.Trigger, threshold.Clear)
|
||||
}
|
||||
return diskTemperatureLimits{}
|
||||
}
|
||||
|
||||
// newDiskTemperatureLimits builds limits from an alert trigger and clear
|
||||
// value. A non-positive trigger is off; a clear value missing or above the
|
||||
// trigger leaves no band below it.
|
||||
func newDiskTemperatureLimits(trigger, clear float64) diskTemperatureLimits {
|
||||
if trigger <= 0 {
|
||||
return diskTemperatureLimits{}
|
||||
}
|
||||
@@ -72,7 +113,8 @@ func physicalDiskOwnerIndex(urp UnifiedResourceProvider) map[string]unifiedresou
|
||||
// disk, the host its disk temperature alerts are evaluated under: the agent on
|
||||
// the machine the disk is parented to or, for a disk parented to a storage
|
||||
// pool (Unraid array and cache disks), the agent on the pool's machine. A disk
|
||||
// no agent reports gets the hostless policy.
|
||||
// no agent reports gets the hostless policy. TrueNAS disks do not come here;
|
||||
// physicalDiskTemperatureLimits judges them by their own alert.
|
||||
func physicalDiskTemperatureHost(disk unifiedresources.Resource, owners map[string]unifiedresources.Resource) alerts.DiskTemperatureHost {
|
||||
parentID := disk.ParentID
|
||||
for hops := 0; parentID != nil && hops < 2; hops++ {
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
@@ -343,3 +344,147 @@ func TestPhysicalDiskTemperatureHostFollowsTheParentChain(t *testing.T) {
|
||||
t.Errorf("parentless disk: host = %+v, want hostless", got)
|
||||
}
|
||||
}
|
||||
|
||||
// trueNASHeatRegistry is a TrueNAS system whose pool holds four disks beside
|
||||
// the two overridden host agents of overriddenAgentDisksRegistry: nvme0n1 at
|
||||
// 72C, nvme1n1 at 63C, SATA sda at 60C and SATA sdb, whose 80C reading is
|
||||
// retained from before standby. The user saved a TrueNAS-wide 62C (clear 57),
|
||||
// raised nvme0n1's own trigger to 75C (clear 70), and left a 50C Disk Temp
|
||||
// override under the TrueNAS system's synthetic agent ID. It returns the disk
|
||||
// resources by name.
|
||||
func trueNASHeatRegistry(t *testing.T) (*unifiedresources.ResourceRegistry, *AlertThresholdAdapter, map[string]unifiedresources.Resource) {
|
||||
t.Helper()
|
||||
registry, _ := overriddenAgentDisksRegistry(t)
|
||||
now := time.Now()
|
||||
disk := func(name, transport string, temperature int) truenas.Disk {
|
||||
return truenas.Disk{
|
||||
ID: "disk-" + name, Name: name, Pool: "tank", Status: "ONLINE", Model: "Model " + name,
|
||||
Serial: "SERIAL-" + strings.ToUpper(name), Temperature: temperature, Transport: transport,
|
||||
}
|
||||
}
|
||||
records := truenas.FixtureRecords(truenas.FixtureSnapshot{
|
||||
CollectedAt: now,
|
||||
System: truenas.SystemInfo{Hostname: "truenas-main", Healthy: true, CollectedAt: now},
|
||||
Pools: []truenas.Pool{{ID: "pool-tank", Name: "tank", Status: "ONLINE"}},
|
||||
Disks: []truenas.Disk{
|
||||
disk("nvme0n1", "nvme", 72), disk("nvme1n1", "nvme", 63),
|
||||
disk("sda", "sata", 60), disk("sdb", "sata", 80),
|
||||
},
|
||||
})
|
||||
for i := range records {
|
||||
if meta := records[i].Resource.PhysicalDisk; meta != nil && records[i].Resource.Name == "sdb" {
|
||||
meta.Collection = &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("truenas", "disk is in standby")}
|
||||
}
|
||||
}
|
||||
registry.IngestRecords(unifiedresources.SourceTrueNAS, records)
|
||||
|
||||
adapter := unifiedresources.NewUnifiedAIAdapter(registry)
|
||||
disks := make(map[string]unifiedresources.Resource)
|
||||
for _, r := range adapter.GetByType(unifiedresources.ResourceTypePhysicalDisk) {
|
||||
if alerts.IsTrueNASDiskResource(r) {
|
||||
disks[r.Name] = r
|
||||
}
|
||||
}
|
||||
systemAgentID := ""
|
||||
for _, r := range adapter.GetAll() {
|
||||
if r.Agent != nil && r.Agent.Platform == "truenas" {
|
||||
systemAgentID = r.Agent.AgentID
|
||||
}
|
||||
}
|
||||
if len(disks) != 4 || systemAgentID == "" {
|
||||
t.Fatalf("TrueNAS disks = %v, system agent ID = %q; want four disks under one system", disks, systemAgentID)
|
||||
}
|
||||
|
||||
mgr := alerts.NewManager()
|
||||
t.Cleanup(mgr.Stop)
|
||||
cfg := mgr.GetConfig()
|
||||
cfg.TrueNASDiskDefaults.Temperature = &alerts.HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
cfg.Overrides = map[string]alerts.ThresholdConfig{
|
||||
"agent-cool": {DiskTemperature: &alerts.HysteresisThreshold{Trigger: 80, Clear: 75}},
|
||||
systemAgentID: {DiskTemperature: &alerts.HysteresisThreshold{Trigger: 50, Clear: 45}},
|
||||
disks["nvme0n1"].ID: {Temperature: &alerts.HysteresisThreshold{Trigger: 75, Clear: 70}},
|
||||
}
|
||||
mgr.UpdateConfig(cfg)
|
||||
return registry, NewAlertThresholdAdapter(mgr), disks
|
||||
}
|
||||
|
||||
// Patrol judges a TrueNAS disk by its own temperature alert's tiers, not as a
|
||||
// disk of the TrueNAS system's synthetic host agent: nvme0n1's override keeps
|
||||
// it quiet at 72C, the TrueNAS-wide 62C flags nvme1n1 at 63C (under the NVMe
|
||||
// trigger of 70C) and spares SATA sda at 60C (over the SATA trigger of 55C),
|
||||
// and the 50C host override under the system ID reaches none of them. A
|
||||
// retained reading stays unjudged. Agent disks keep their host's override.
|
||||
func TestPatrolJudgesTrueNASDisksByTheirAlertTiers(t *testing.T) {
|
||||
registry, provider, disks := trueNASHeatRegistry(t)
|
||||
state := newPatrolRuntimeStateWithProviders(models.StateSnapshot{}, nil, unifiedresources.NewUnifiedAIAdapter(registry))
|
||||
state.thresholdProvider = provider
|
||||
|
||||
type want struct {
|
||||
limits diskTemperatureLimits
|
||||
issue bool
|
||||
}
|
||||
expected := map[string]want{
|
||||
disks["nvme0n1"].ID: {limits: diskTemperatureLimits{trigger: 75, clear: 70}},
|
||||
disks["nvme1n1"].ID: {limits: diskTemperatureLimits{trigger: 62, clear: 57}, issue: true},
|
||||
disks["sda"].ID: {limits: diskTemperatureLimits{trigger: 62, clear: 57}},
|
||||
disks["sdb"].ID: {limits: diskTemperatureLimits{trigger: 62, clear: 57}},
|
||||
}
|
||||
rows := patrolPhysicalDiskRows(state, nil)
|
||||
if len(rows) != 6 {
|
||||
t.Fatalf("rows = %+v, want four TrueNAS disks and two agent disks", rows)
|
||||
}
|
||||
for _, row := range rows {
|
||||
w, trueNAS := expected[row.id]
|
||||
if !trueNAS {
|
||||
// overriddenAgentDisksRegistry: agent-cool's 76C NVMe under its 80C
|
||||
// override, agent-plain's 72C NVMe over the NVMe trigger.
|
||||
w = want{limits: diskTemperatureLimits{trigger: 70, clear: 65}, issue: true}
|
||||
if row.temperature.Collected == 76 {
|
||||
w = want{limits: diskTemperatureLimits{trigger: 80, clear: 75}}
|
||||
}
|
||||
}
|
||||
if row.temperatureLimits != w.limits {
|
||||
t.Errorf("disk %s at %dC: limits = %+v, want %+v", row.id, row.temperature.Collected, row.temperatureLimits, w.limits)
|
||||
}
|
||||
if got := patrolPhysicalDiskHealthIssue(row); got != w.issue {
|
||||
t.Errorf("disk %s at %dC: health issue = %v, want %v", row.id, row.temperature.Collected, got, w.issue)
|
||||
}
|
||||
}
|
||||
|
||||
flags := triageDiskHealthChecksState(state, nil)
|
||||
for _, name := range []string{"nvme0n1", "sda", "sdb"} {
|
||||
if flag := triageFindFlag(flags, func(f TriageFlag) bool {
|
||||
return f.ResourceID == disks[name].ID && strings.Contains(f.Reason, "Disk temperature")
|
||||
}); flag != nil {
|
||||
t.Errorf("%s: unexpected temperature flag %+v", name, *flag)
|
||||
}
|
||||
}
|
||||
if flag := triageFindFlag(flags, func(f TriageFlag) bool {
|
||||
return f.ResourceID == disks["nvme1n1"].ID && strings.Contains(f.Reason, "Disk temperature")
|
||||
}); flag == nil || flag.Threshold != 62 {
|
||||
t.Errorf("nvme1n1: temperature flag = %+v, want one at the TrueNAS-wide 62C", flag)
|
||||
}
|
||||
|
||||
if _, err := verifyMetricRecoveredState(state, PatrolThresholds{}, "disk-high", disks["sdb"].ID, "physical_disk"); !errors.Is(err, aicontracts.ErrVerificationUnknown) {
|
||||
t.Errorf("retained 80C: err = %v, want ErrVerificationUnknown", err)
|
||||
}
|
||||
|
||||
s := &Service{}
|
||||
s.SetUnifiedResourceProvider(unifiedresources.NewUnifiedAIAdapter(registry))
|
||||
s.SetPatrolThresholdProvider(provider)
|
||||
_, section, _ := strings.Cut(s.buildUnifiedResourceContext(), "Physical Disks Needing Attention")
|
||||
section, _, _ = strings.Cut(section, "\n\n")
|
||||
for _, reading := range []string{"Temp: 63C", "Temp: 72C"} {
|
||||
if !strings.Contains(section, reading) {
|
||||
t.Errorf("attention section = %q, want the disk at %s", section, reading)
|
||||
}
|
||||
}
|
||||
for _, reading := range []string{"Temp: 60C", "Temp: 76C", "Temp: 80C"} {
|
||||
if strings.Contains(section, reading) {
|
||||
t.Errorf("attention section = %q, lists a disk at %s", section, reading)
|
||||
}
|
||||
}
|
||||
if strings.Count(section, "Temp: 72C") != 1 {
|
||||
t.Errorf("attention section = %q, want only agent-plain's 72C disk, not TrueNAS nvme0n1 under its 75C override", section)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,9 @@ func (m MockThresholdProvider) GetStorageThreshold() float64 { return m.Stor
|
||||
func (m MockThresholdProvider) GetDiskTemperatureThreshold(alerts.DiskTemperatureHost, string) (float64, float64) {
|
||||
return 0, 0
|
||||
}
|
||||
func (m MockThresholdProvider) GetTrueNASDiskTemperatureThreshold(string, string) (float64, float64) {
|
||||
return 0, 0
|
||||
}
|
||||
|
||||
func TestCalculatePatrolThresholds_Default(t *testing.T) {
|
||||
// Test default behavior (exact mode)
|
||||
|
||||
@@ -417,7 +417,7 @@ func (s *Service) buildUnifiedResourceContextForModel(destinationModel string) s
|
||||
health = strings.ToUpper(strings.TrimSpace(disk.PhysicalDisk.Health))
|
||||
// A retained last-known reading is not current heat.
|
||||
collected := tools.SplitDiskTemperature(disk.PhysicalDisk.Temperature, disk.PhysicalDisk.Collection).Collected
|
||||
hot = diskTemperatureLimitsFor(thresholdProvider, physicalDiskTemperatureHost(disk, owners), disk.PhysicalDisk.DiskType).hot(collected)
|
||||
hot = physicalDiskTemperatureLimits(thresholdProvider, disk, owners).hot(collected)
|
||||
}
|
||||
if disk.Status != unifiedresources.StatusOnline || (health != "" && health != "PASSED" && health != "UNKNOWN") || hot {
|
||||
attention = append(attention, disk)
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
func boolPtr(v bool) *bool {
|
||||
@@ -2135,3 +2136,239 @@ func TestHostDiskTemperatureThresholdMatchesCheckHostOverrides(t *testing.T) {
|
||||
t.Fatalf("nil manager host policy = %+v, want the factory nvme 70", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TrueNASDiskTemperatureThreshold is the disk heat policy for a TrueNAS disk,
|
||||
// so it must agree with the disk's temperature alert on every tier: the
|
||||
// disk's own override, then the TrueNAS-wide value, then the per-type policy.
|
||||
// A host override under the TrueNAS system's synthetic agent ID is not one of
|
||||
// them. Disable all TrueNAS silences the alert without changing the policy,
|
||||
// as the agent switches do for agent disks.
|
||||
func TestTrueNASDiskTemperatureThresholdMatchesTrueNASDiskAlerts(t *testing.T) {
|
||||
disk := trueNASTemperatureDisk("nvme0n1", "nvme", 0)
|
||||
|
||||
const canonicalDiskID = "physical-disk:canonical-nvme0n1"
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
mutate func(*AlertConfig)
|
||||
setup func(*Manager)
|
||||
want *HysteresisThreshold
|
||||
off bool
|
||||
silenced bool
|
||||
}{
|
||||
{name: "per-type policy", want: &HysteresisThreshold{Trigger: 70, Clear: 65}},
|
||||
{
|
||||
name: "raised per-type trigger",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.DiskTempByType = map[string]HysteresisThreshold{"nvme": {Trigger: 75, Clear: 70}}
|
||||
},
|
||||
want: &HysteresisThreshold{Trigger: 75, Clear: 70},
|
||||
},
|
||||
{
|
||||
name: "TrueNAS-wide value",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
},
|
||||
want: &HysteresisThreshold{Trigger: 62, Clear: 57},
|
||||
},
|
||||
{
|
||||
name: "per-disk override beats the TrueNAS-wide value",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
cfg.Overrides = map[string]ThresholdConfig{disk.ID: {Temperature: &HysteresisThreshold{Trigger: 75, Clear: 70}}}
|
||||
},
|
||||
want: &HysteresisThreshold{Trigger: 75, Clear: 70},
|
||||
},
|
||||
{
|
||||
name: "per-disk override without a clear value",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.Overrides = map[string]ThresholdConfig{disk.ID: {Temperature: &HysteresisThreshold{Trigger: 75}}}
|
||||
},
|
||||
want: &HysteresisThreshold{Trigger: 75, Clear: 70},
|
||||
},
|
||||
{
|
||||
name: "per-disk override under the canonical identity",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.Overrides = map[string]ThresholdConfig{canonicalDiskID: {Temperature: &HysteresisThreshold{Trigger: 78, Clear: 73}}}
|
||||
},
|
||||
setup: func(m *Manager) {
|
||||
m.SetResourceIntentIdentityResolver(func(resourceID string) (string, bool) {
|
||||
return canonicalDiskID, resourceID == disk.ID
|
||||
})
|
||||
},
|
||||
want: &HysteresisThreshold{Trigger: 78, Clear: 73},
|
||||
},
|
||||
{
|
||||
name: "host override under the TrueNAS system ID does not reach its disks",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.Overrides = map[string]ThresholdConfig{"truenas-main": {DiskTemperature: &HysteresisThreshold{Trigger: 50, Clear: 45}}}
|
||||
},
|
||||
want: &HysteresisThreshold{Trigger: 70, Clear: 65},
|
||||
},
|
||||
{
|
||||
name: "per-disk override switches the disk off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.Overrides = map[string]ThresholdConfig{disk.ID: {Temperature: &HysteresisThreshold{}}}
|
||||
},
|
||||
off: true,
|
||||
},
|
||||
{
|
||||
name: "per-disk override disables the disk's alerts",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.Overrides = map[string]ThresholdConfig{disk.ID: {Disabled: true}}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "TrueNAS-wide off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{}
|
||||
},
|
||||
off: true,
|
||||
},
|
||||
{
|
||||
name: "TrueNAS Disks defaults disabled",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Disabled = true
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "agent Disk Temp default off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{}
|
||||
},
|
||||
off: true,
|
||||
},
|
||||
{
|
||||
name: "Disable all TrueNAS keeps the policy",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.DisableAllTrueNAS = true
|
||||
},
|
||||
want: &HysteresisThreshold{Trigger: 70, Clear: 65},
|
||||
silenced: true,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
if tc.mutate != nil {
|
||||
tc.mutate(&cfg)
|
||||
}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
if tc.setup != nil {
|
||||
tc.setup(m)
|
||||
}
|
||||
|
||||
got := m.TrueNASDiskTemperatureThreshold(" "+disk.ID+" ", " NVMe")
|
||||
switch {
|
||||
case tc.off:
|
||||
if got == nil || got.Trigger > 0 {
|
||||
t.Fatalf("TrueNASDiskTemperatureThreshold = %+v, want a switched-off threshold", got)
|
||||
}
|
||||
case (got == nil) != (tc.want == nil) || (got != nil && *got != *tc.want):
|
||||
t.Fatalf("TrueNASDiskTemperatureThreshold = %+v, want %+v", got, tc.want)
|
||||
}
|
||||
|
||||
var alert *Alert
|
||||
fires := func(temperature int) bool {
|
||||
reading := disk
|
||||
meta := *disk.PhysicalDisk
|
||||
meta.Temperature = temperature
|
||||
reading.PhysicalDisk = &meta
|
||||
checkTrueNASTemperatureDisk(t, m, reading)
|
||||
var firing bool
|
||||
alert, firing = trueNASDiskTemperatureAlert(t, m, reading)
|
||||
return firing
|
||||
}
|
||||
if tc.silenced {
|
||||
if fires(99) {
|
||||
t.Fatalf("alert fired at 99C with TrueNAS alerts disabled, active: %v", alertKeys(m))
|
||||
}
|
||||
return
|
||||
}
|
||||
if got == nil || got.Trigger <= 0 {
|
||||
if fires(99) {
|
||||
t.Fatalf("alert fired at 99C on a disk the policy judges no heat for, active: %v", alertKeys(m))
|
||||
}
|
||||
return
|
||||
}
|
||||
trigger := int(got.Trigger)
|
||||
if fires(trigger - 1) {
|
||||
t.Fatalf("alert fired at %dC, under the policy trigger %v", trigger-1, got.Trigger)
|
||||
}
|
||||
if !fires(trigger) {
|
||||
t.Fatalf("alert stayed quiet at the policy trigger %dC, active: %v", trigger, alertKeys(m))
|
||||
}
|
||||
// The alert recovers at the clear value Patrol reads.
|
||||
if alert.Threshold != got.Trigger || alert.Metadata["clearThreshold"] != got.Clear {
|
||||
t.Fatalf("alert trigger/clear = %v/%v, want the policy %v/%v", alert.Threshold, alert.Metadata["clearThreshold"], got.Trigger, got.Clear)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A disk whose type TrueNAS never reported follows the agent Disk Temp
|
||||
// default, as its alert does.
|
||||
t.Run("untyped disk", func(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 58, Clear: 53}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
untypedDisk := trueNASTemperatureDisk("sdx", "", 58)
|
||||
if got := m.TrueNASDiskTemperatureThreshold(untypedDisk.ID, ""); got == nil || got.Trigger != 58 || got.Clear != 53 {
|
||||
t.Fatalf("untyped disk = %+v, want the agent default 58/53", got)
|
||||
}
|
||||
checkTrueNASTemperatureDisk(t, m, untypedDisk)
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, untypedDisk); !firing {
|
||||
t.Fatalf("untyped disk at the agent default 58C raised no alert, active: %v", alertKeys(m))
|
||||
}
|
||||
})
|
||||
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
m.TrueNASDiskTemperatureThreshold(disk.ID, "nvme").Trigger = 1
|
||||
if got := m.TrueNASDiskTemperatureThreshold(disk.ID, "nvme"); got == nil || got.Trigger != 70 {
|
||||
t.Fatalf("caller mutation reached the config: %+v", got)
|
||||
}
|
||||
var nilManager *Manager
|
||||
if got := nilManager.TrueNASDiskTemperatureThreshold(disk.ID, "sata"); got == nil || got.Trigger != 55 {
|
||||
t.Fatalf("nil manager TrueNAS disk policy = %+v, want the factory sata 55", got)
|
||||
}
|
||||
}
|
||||
|
||||
// IsTrueNASDiskResource names the disks the unified evaluator judges as
|
||||
// TrueNAS disks, the ones TrueNASDiskTemperatureThreshold resolves.
|
||||
func TestIsTrueNASDiskResourceMatchesTheUnifiedEvaluator(t *testing.T) {
|
||||
agentDisk := unifiedresources.Resource{
|
||||
ID: "physical-disk:host-1/nvme0n1", Type: unifiedresources.ResourceTypePhysicalDisk,
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceAgent},
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{DiskType: "nvme"},
|
||||
}
|
||||
trueNASFacetDisk := agentDisk
|
||||
trueNASFacetDisk.TrueNAS = &unifiedresources.TrueNASData{}
|
||||
mergedDisk := agentDisk
|
||||
mergedDisk.Sources = []unifiedresources.DataSource{unifiedresources.SourceAgent, unifiedresources.SourceTrueNAS}
|
||||
noMetaDisk := trueNASTemperatureDisk("sdy", "sata", 40)
|
||||
noMetaDisk.PhysicalDisk = nil
|
||||
trueNASSystem := unifiedresources.Resource{
|
||||
ID: "agent:truenas-main", Type: unifiedresources.ResourceTypeAgent,
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceTrueNAS},
|
||||
}
|
||||
for name, tc := range map[string]struct {
|
||||
resource unifiedresources.Resource
|
||||
want bool
|
||||
}{
|
||||
"TrueNAS-sourced disk": {resource: trueNASTemperatureDisk("sda", "sata", 40), want: true},
|
||||
"TrueNAS facet disk": {resource: trueNASFacetDisk, want: true},
|
||||
"agent and TrueNAS": {resource: mergedDisk, want: true},
|
||||
"no disk metadata": {resource: noMetaDisk, want: true},
|
||||
"agent disk": {resource: agentDisk},
|
||||
"TrueNAS system": {resource: trueNASSystem},
|
||||
} {
|
||||
if got := IsTrueNASDiskResource(tc.resource); got != tc.want {
|
||||
t.Errorf("%s: IsTrueNASDiskResource = %v, want %v", name, got, tc.want)
|
||||
}
|
||||
input, ok := UnifiedResourceInputFromResource(tc.resource)
|
||||
if evaluated := ok && input.Type == "truenas-disk"; evaluated != tc.want {
|
||||
t.Errorf("%s: unified evaluator judges it as a TrueNAS disk = %v, want %v", name, evaluated, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
// trueNASDiskTemperatureDefaultNoLock returns the default tier of a TrueNAS
|
||||
// disk's temperature threshold. A TrueNAS-wide value the user saved under
|
||||
// TrueNAS Disks applies to every TrueNAS disk, as a host Disk Temp override
|
||||
@@ -16,3 +22,39 @@ func (m *Manager) trueNASDiskTemperatureDefaultNoLock(diskType string, typeUnkno
|
||||
}
|
||||
return cloneThreshold(m.hostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature, false, diskType))
|
||||
}
|
||||
|
||||
// IsTrueNASDiskResource reports whether the unified evaluator judges a
|
||||
// resource as a TrueNAS disk (truenas-disk), whose temperature alert resolves
|
||||
// its threshold through TrueNASDiskTemperatureThreshold rather than through
|
||||
// the host agent that reports it.
|
||||
func IsTrueNASDiskResource(resource unifiedresources.Resource) bool {
|
||||
typeKey, ok := unifiedAlertResourceType(resource)
|
||||
return ok && typeKey == "truenas-disk"
|
||||
}
|
||||
|
||||
// TrueNASDiskTemperatureThreshold is the disk heat policy for one TrueNAS
|
||||
// disk, resolved by the same effectiveAlertPolicyNoLock tiers its temperature
|
||||
// alert uses: an override stored under the disk's resource ID, then the
|
||||
// TrueNAS-wide TrueNAS Disks value, then the per-type policy. An override or
|
||||
// TrueNAS Disks default that switches the disk's alerts off leaves no
|
||||
// threshold (nil); a non-positive trigger is off too. Like
|
||||
// HostDiskTemperatureThreshold it ignores the global and per-platform alert
|
||||
// switches, which silence the alert without changing the policy. A nil
|
||||
// manager resolves the factory configuration, where no TrueNAS-wide value is
|
||||
// set.
|
||||
func (m *Manager) TrueNASDiskTemperatureThreshold(resourceID, diskType string) *HysteresisThreshold {
|
||||
if m == nil {
|
||||
return DefaultDiskTemperatureThreshold(diskType)
|
||||
}
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
thresholds := m.effectiveAlertPolicyNoLock(alertPolicyQuery{
|
||||
TypeKey: "truenas-disk",
|
||||
ResourceID: unifiedresources.CanonicalResourceID(resourceID),
|
||||
DiskType: strings.ToLower(strings.TrimSpace(diskType)),
|
||||
}).Thresholds
|
||||
if thresholds.Disabled {
|
||||
return nil
|
||||
}
|
||||
return cloneThreshold(thresholds.Temperature)
|
||||
}
|
||||
|
||||
@@ -71,6 +71,9 @@ func (concurrencyThresholdProvider) GetStorageThreshold() float64 { return 92
|
||||
func (concurrencyThresholdProvider) GetDiskTemperatureThreshold(alerts.DiskTemperatureHost, string) (float64, float64) {
|
||||
return 70, 65
|
||||
}
|
||||
func (concurrencyThresholdProvider) GetTrueNASDiskTemperatureThreshold(string, string) (float64, float64) {
|
||||
return 70, 65
|
||||
}
|
||||
|
||||
type concurrencyMetricsHistoryProvider struct{}
|
||||
|
||||
|
||||
@@ -162,6 +162,9 @@ func (stubThresholdProvider) GetStorageThreshold() float64 { return 92 }
|
||||
func (stubThresholdProvider) GetDiskTemperatureThreshold(alerts.DiskTemperatureHost, string) (float64, float64) {
|
||||
return 70, 65
|
||||
}
|
||||
func (stubThresholdProvider) GetTrueNASDiskTemperatureThreshold(string, string) (float64, float64) {
|
||||
return 70, 65
|
||||
}
|
||||
|
||||
type stubMetricsHistoryProvider struct{}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package monitoring
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -27,6 +28,8 @@ func TestCrossInstallationIdentitySurvivesStandaloneAddition(t *testing.T) {
|
||||
{"classified member-hostname coincidence", "https://pmx1:8006", true, false},
|
||||
{"unclassified FQDN connection and distinct guest domains", "https://pmx1.remote.example:8006", false, true},
|
||||
{"classified FQDN connection and distinct guest domains", "https://pmx1.remote.example:8006", true, true},
|
||||
{"unclassified FQDN connection and identical short guest names", "https://pmx1.remote.example:8006", false, false},
|
||||
{"classified FQDN connection and identical short guest names", "https://pmx1.remote.example:8006", true, false},
|
||||
} {
|
||||
name, classified := tc.name, tc.classified
|
||||
clusterName := ""
|
||||
@@ -80,6 +83,45 @@ func TestCrossInstallationIdentitySurvivesStandaloneAddition(t *testing.T) {
|
||||
t.Fatal("initial Docker report absent from read state")
|
||||
}
|
||||
homeCanonicalID := adapter.VMs()[0].ID()
|
||||
initialHosts := m.state.GetSnapshot().Hosts
|
||||
if len(initialHosts) != 1 || initialHosts[0].LinkedVMID != homeGuestID {
|
||||
t.Fatalf("initial automatic guest link = %+v, want %s", initialHosts, homeGuestID)
|
||||
}
|
||||
initialDocker := m.state.GetSnapshot().DockerHosts
|
||||
initialDockerHostID := adapter.DockerHosts()[0].ID()
|
||||
initialWorkloadID := adapter.DockerContainers()[0].ID()
|
||||
assertUnchangedDocker := func(stage string) {
|
||||
t.Helper()
|
||||
// No new Docker report has arrived: a report after link clearing
|
||||
// could restore deleted inventory and hide the actual regression.
|
||||
if got := m.state.GetSnapshot().DockerHosts; !reflect.DeepEqual(got, initialDocker) {
|
||||
t.Fatalf("%s changed the independent Docker snapshot: got %+v, want %+v", stage, got, initialDocker)
|
||||
}
|
||||
if hosts, workloads := adapter.DockerHosts(), adapter.DockerContainers(); len(hosts) != 1 || len(workloads) != 1 ||
|
||||
hosts[0].ID() != initialDockerHostID || workloads[0].ID() != initialWorkloadID {
|
||||
t.Fatalf("%s lost Docker source identities: hosts=%+v workloads=%+v", stage, hosts, workloads)
|
||||
}
|
||||
listed, _, thresholds := adapter.GetAllWithMetricsTargetsAndStaleThresholds()
|
||||
broadcast, ok := adapter.CoalesceForPresentation(listed, thresholds)
|
||||
if !ok {
|
||||
t.Fatal("store-backed Docker presentation was not available")
|
||||
}
|
||||
for surface, resources := range map[string][]unifiedresources.Resource{"API list": listed, "broadcast": broadcast} {
|
||||
var machineID string
|
||||
var workloads []unifiedresources.Resource
|
||||
for _, resource := range resources {
|
||||
if resource.Type == unifiedresources.ResourceTypeAgent && resource.Docker != nil && resource.Docker.HostSourceID == initialDocker[0].ID {
|
||||
machineID = resource.ID
|
||||
}
|
||||
if resource.Type == unifiedresources.ResourceTypeAppContainer && resource.Docker != nil && resource.Docker.ContainerID == "home-workload" {
|
||||
workloads = append(workloads, resource)
|
||||
}
|
||||
}
|
||||
if machineID == "" || len(workloads) != 1 || workloads[0].ID != initialWorkloadID || workloads[0].ParentID == nil || *workloads[0].ParentID != machineID {
|
||||
t.Fatalf("%s %s detached Docker workload from its reporting machine: machine=%s workloads=%+v", stage, surface, machineID, workloads)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
standalone := config.PVEInstance{Name: "Hetzner pmx1", Host: tc.host}
|
||||
m.config.PVEInstances = append(m.config.PVEInstances, standalone)
|
||||
@@ -100,9 +142,19 @@ func TestCrossInstallationIdentitySurvivesStandaloneAddition(t *testing.T) {
|
||||
// not discard the agent, its Docker inventory, or the original VM's data.
|
||||
hostReport.Timestamp = now.Add(time.Second)
|
||||
dockerReport.Timestamp = now.Add(time.Second)
|
||||
assertUnchangedDocker("provider addition without agent reports")
|
||||
if _, err := m.ApplyHostReport(hostReport, token); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hosts := m.state.GetSnapshot().Hosts
|
||||
wantLink := ""
|
||||
if tc.fqdnGuests {
|
||||
wantLink = homeGuestID
|
||||
}
|
||||
if len(hosts) != 1 || hosts[0].LinkedVMID != wantLink || hosts[0].LinkedNodeID != "" || hosts[0].LinkedContainerID != "" {
|
||||
t.Fatalf("post-addition guest link = %+v, want %q with no node/container guess", hosts, wantLink)
|
||||
}
|
||||
assertUnchangedDocker("host-only report after automatic guest-link reconciliation")
|
||||
if _, err := m.ApplyDockerReport(dockerReport, token); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -5925,6 +5925,71 @@ func TestMockUnifiedViewAppliesOperatorManualLinks(t *testing.T) {
|
||||
if len(after) != len(before) {
|
||||
t.Fatalf("mock view wrote %d change rows into the durable store", len(after)-len(before))
|
||||
}
|
||||
|
||||
// Reconcile the mock link proposal with the current latest-decision
|
||||
// rule: a fresh monitor must restore the link, and unlink/relink must
|
||||
// invalidate the seed cache without a fixture tick. Mock seeds retain
|
||||
// their own default thresholds, not the live adapter's configured ones.
|
||||
restarted := &Monitor{
|
||||
state: models.NewState(),
|
||||
config: &config.Config{PVEPollingInterval: 2 * time.Minute},
|
||||
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)),
|
||||
alertManager: alerts.NewManager(),
|
||||
}
|
||||
t.Cleanup(restarted.alertManager.Stop)
|
||||
adapter := restarted.resourceStore.(*unifiedresources.MonitorAdapter)
|
||||
assertMockLink := func(wantLinked bool) time.Time {
|
||||
t.Helper()
|
||||
frontend := restarted.BuildBroadcastFrontendState()
|
||||
var vmFound, agentFound, vmHasAgent bool
|
||||
for _, resource := range frontend.Resources {
|
||||
if resource.ID == vmID {
|
||||
vmFound, vmHasAgent = true, len(resource.Agent) > 0
|
||||
}
|
||||
if resource.ID == agentID {
|
||||
agentFound = true
|
||||
}
|
||||
}
|
||||
if !vmFound || vmHasAgent != wantLinked || agentFound == wantLinked {
|
||||
t.Fatalf("mock link=%t: vm=%t vmAgent=%t standaloneAgent=%t", wantLinked, vmFound, vmHasAgent, agentFound)
|
||||
}
|
||||
seed, freshness, thresholds := restarted.UnifiedResourceSnapshotWithStaleThresholds()
|
||||
if len(thresholds) != 0 {
|
||||
t.Fatalf("mock seed borrowed live thresholds: %v", thresholds)
|
||||
}
|
||||
if got := adapter.StaleThresholds()[unifiedresources.SourceProxmox]; got != 4*time.Minute {
|
||||
t.Fatalf("live adapter threshold = %v, want configured four minutes", got)
|
||||
}
|
||||
rest := unifiedresources.NewRegistryWithStaleThresholds(store, thresholds)
|
||||
rest.IngestResources(seed)
|
||||
_, hasStandaloneAgent := rest.Get(agentID)
|
||||
if hasStandaloneAgent == wantLinked {
|
||||
t.Fatalf("REST seed link=%t: standalone agent=%t", wantLinked, hasStandaloneAgent)
|
||||
}
|
||||
if got := mock.FixtureDataVersion(); got != version {
|
||||
t.Fatalf("fixture version moved from %d to %d during operator lifecycle", version, got)
|
||||
}
|
||||
return freshness
|
||||
}
|
||||
restartFreshness := assertMockLink(true)
|
||||
if err := store.AddExclusion(unifiedresources.ResourceExclusion{ResourceA: vmID, ResourceB: agentID}); err != nil {
|
||||
t.Fatalf("unlink: %v", err)
|
||||
}
|
||||
// Reads deliberately reuse a generation younger than two seconds.
|
||||
// Apply the owning ingest/rebuild boundary instead of waiting for that
|
||||
// throttle or claiming a recorded decision changes a previous listing.
|
||||
restarted.updateResourceStore(restarted.currentStateWithScope())
|
||||
unlinkFreshness := assertMockLink(false)
|
||||
if !unlinkFreshness.After(restartFreshness) {
|
||||
t.Fatal("unlink must advance the mock seed's freshness")
|
||||
}
|
||||
if err := store.AddLink(unifiedresources.ResourceLink{ResourceA: agentID, ResourceB: vmID, PrimaryID: vmID}); err != nil {
|
||||
t.Fatalf("relink: %v", err)
|
||||
}
|
||||
restarted.updateResourceStore(restarted.currentStateWithScope())
|
||||
if relinkFreshness := assertMockLink(true); !relinkFreshness.After(unlinkFreshness) {
|
||||
t.Fatal("relink must advance the mock seed's freshness")
|
||||
}
|
||||
}
|
||||
|
||||
// TestMockModeDiscardsRealHostReports pins the push-side of the mock clean
|
||||
|
||||
@@ -290,8 +290,8 @@ func TestOperatorSplitOverridesProxmoxNodeAgentLink(t *testing.T) {
|
||||
t.Helper()
|
||||
for rebuild := 1; rebuild <= rebuilds; rebuild++ {
|
||||
adapter.PopulateFromSnapshot(snapshot)
|
||||
listed := adapter.GetAll()
|
||||
broadcast, ok := adapter.CoalesceForPresentation(listed, nil)
|
||||
listed, _, thresholds := adapter.GetAllWithMetricsTargetsAndStaleThresholds()
|
||||
broadcast, ok := adapter.CoalesceForPresentation(listed, thresholds)
|
||||
if !ok {
|
||||
t.Fatal("store-backed adapter did not coalesce with its exclusions")
|
||||
}
|
||||
@@ -351,18 +351,25 @@ func TestOperatorSplitOverridesProxmoxNodeAgentLink(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assertViews("after relink", 2, nodeAgentRows{joined: merged.joined})
|
||||
// The joined row records the relinked row it took in, so a
|
||||
// report-merge of it names the relink's own pair.
|
||||
folded := want.node
|
||||
if folded == merged.joined {
|
||||
folded = want.agent
|
||||
}
|
||||
recorded := false
|
||||
for _, fold := range adapter.currentRegistry().ManualLinkFolds(merged.joined) {
|
||||
recorded = recorded || (fold.HolderID == merged.joined && fold.FoldedID == folded)
|
||||
}
|
||||
if !recorded {
|
||||
t.Fatalf("joined %s records no fold of relinked %s: %+v", merged.joined, folded, adapter.currentRegistry().ManualLinkFolds(merged.joined))
|
||||
// The inferred join can consume the node row before manual
|
||||
// links run. Its fold must still reach current succession and
|
||||
// the API seed, rather than an obsolete registry-wide ID set.
|
||||
seeded := NewRegistry(store)
|
||||
seeded.IngestResources(adapter.GetAll())
|
||||
for name, registry := range map[string]*ResourceRegistry{"monitor": adapter.currentRegistry(), "API seed": seeded} {
|
||||
found := false
|
||||
for _, fold := range registry.ManualLinkFolds(merged.joined) {
|
||||
nodeFold := fold.HolderID == want.agent && fold.FoldedID == want.node && slices.Contains(fold.Sources, SourceProxmox)
|
||||
agentFold := fold.HolderID == want.node && fold.FoldedID == want.agent && slices.Contains(fold.Sources, SourceAgent)
|
||||
// A machine-keyless agent rejoins under the node's ID,
|
||||
// so the same pair is folded in the opposite direction.
|
||||
if nodeFold || agentFold {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("%s lost the relink's pair fold: %+v", name, registry.ManualLinkFolds(merged.joined))
|
||||
}
|
||||
}
|
||||
// A rebuild that meets the agent first orders the joined row's
|
||||
// hostnames the agent's way; persisting its pins must not
|
||||
|
||||
Reference in New Issue
Block a user