Keep retained disk temperatures out of AI, Patrol and reports

Normalization keeps a disk's last-known temperature when the current
observation did not collect one (a disk in standby, a host agent past
its reporting lease) and marks it non-available. Several readers still
took PhysicalDisk.Temperature at face value. The AI chat context listed
a standby disk as needing attention for heat. Patrol raised health
issues and triage flags from the retained value. The AI query and
disk-health tools handed it to the model as the disk's temperature.
Performance reports and the reporting runtime snapshot tabulated it as
measured, and the disk temperature charts padded a series out to now
with it.

Each of them now goes through diskinventory.TemperatureCollected. AI
surfaces keep a retained value as context, labelled last known with
its reason. Reports and the charts leave it out. monitor.go's private
copy of the predicate is replaced by the shared helper.
This commit is contained in:
rcourtman
2026-10-06 22:48:26 +01:00
parent d9ffb39ce8
commit fb7dd955ed
25 changed files with 525 additions and 63 deletions
@@ -760,6 +760,11 @@ offline the unified-resources adapter reports them `unavailable` with the same
reason (`models.HostAgentStoppedReportingReason`) and leaves them out of the
disk's risk, and the metrics-history API does not return any such retained
disk temperature as a live point.
Downstream readers keep that distinction too. The disk temperature charts do
not pad a series to now with the retained value. The AI chat context, Patrol and
the AI disk tools present it only as a last-known value with its reason. The
performance report and the reporting runtime snapshot leave it out of their
disk tables.
An enabled availability target assigned to a host agent creates an
agent-lifecycle lease for that exact target/agent pairing. First assignment
@@ -288,6 +288,24 @@ exposing the source-specific ambiguous `wearout` name, and distinguishes it from
`smart.percentageUsed` as consumed endurance. The regression proof is
`internal/ai/tools/physical_disk_evidence_test.go`.
Disk temperatures reach the model as current readings only when they were
collected now. `tools.SplitDiskTemperature` classifies a physical disk
temperature with `diskinventory.TemperatureCollected`. The physical-disk
projection's `temperature` and the `disk_health` SMART row's `temperature`
carry only a reading the current observation collected. A value normalization
retained instead (a disk in standby, a host agent past its reporting lease)
moves to `last_known_temperature` with `last_known_temperature_reason`. The
same split feeds the AI chat context and Patrol. The "Physical Disks Needing
Attention" list and Patrol's disk issue gate, triage flags and finding
verification judge heat on the collected reading alone. Their text renders a
retained value as "last known 41C (disk is in standby)". Proofs:
`TestDiskToolsReportRetainedTemperatureAsLastKnown` in
`internal/ai/tools/physical_disk_evidence_test.go`,
`TestBuildUnifiedResourceContextJudgesDiskHeatOnCollectedTemperature` in
`internal/ai/resource_context_test.go` and
`TestPatrolPhysicalDiskRowsJudgeHeatOnCollectedTemperature` in
`internal/ai/patrol_ai_more_test.go`.
Retained summaries disclose that point and bucket timestamps describe returned
history inside the requested window. Their spacing does not measure collection
uptime or explain missing history. Collector lifecycle and retention settings
@@ -6332,6 +6332,14 @@ stored one out to `now`, only from a temperature collected now; without one the
stored samples are returned as they are. An empty mock range can still fall
through to the monitor's generic demo chart series for that disk ID, which is
not derived from the disk's reading.
Reports follow the same rule. A performance report's disk table
(`enrichNodeReport` in `internal/api/metrics_reporting_handlers.go`) and the
reporting runtime snapshot's disks (`internal/api/reporting_runtime_snapshot.go`)
take a disk temperature through `reportDiskTemperature`. It keeps a reading only
when `diskinventory.TemperatureCollected` holds, so a retained value is reported
as 0 (no reading) rather than tabulated as measured. Proof:
`TestContract_ReportsOmitRetainedDiskTemperatures` in
`internal/api/contract_test.go`.
That same metrics-history contract also owns Kubernetes pod identity
normalization. `/api/metrics-store/history` must accept legacy bare pod IDs
such as `cluster-1:pod:pod-1`, canonicalize them onto the unified pod metrics
@@ -3815,6 +3815,15 @@ consumers outside the history writers: the metrics-history API serves a disk
series from the disk's reading or pads a stored one, only from a collected
temperature, so a retained standby or silent-agent reading never reappears as a
point at the current time.
The SMART history writers in `internal/monitoring/monitor.go` and
`monitor_agents.go` call that helper rather than a private copy of the rule,
and `GetPhysicalDiskTemperatureCharts` pads a short disk temperature series out
to now only with a collected reading: with a retained one the stored samples
stay as they are, and a disk with none gets no series. Mock mode can still
substitute its generic demo series for that disk ID before this step; that
series is not derived from the disk's reading. Proof:
`TestDiskTemperatureChartsPadOnlyWithCollectedReading` in
`internal/monitoring/monitor_metrics_slo_test.go`.
Unified-resource physical-disk round trips must retain named
`StorageGroup` membership rather than degrading it to the generic `Used`
filesystem label.
@@ -6779,3 +6779,6 @@ and the Unraid physical-disk adapter marks an inventory temperature kept past
the host agent's lease as no longer collected. Both change presentation of
values Pulse already holds: no metrics write, retention window, backup,
migration or recovery authority is added or moved.
The performance report and reporting runtime snapshot handlers apply the same
test before tabulating a disk temperature. That changes only which held value a
report shows, not any storage or recovery path.
+12 -9
View File
@@ -1927,8 +1927,11 @@ type patrolPhysicalDiskRow struct {
sizeBytes int64
devPath, model string
health, status string
wearout, temperature int
smartEvidence []string
wearout int
// temperature.Collected is the only temperature Patrol may judge as
// heat; a retained last-known value is shown as context.
temperature tools.DiskTemperature
smartEvidence []string
}
type patrolPrecomputeNodeSource struct {
@@ -2472,7 +2475,7 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool)
health: health,
status: status,
wearout: r.PhysicalDisk.Wearout,
temperature: r.PhysicalDisk.Temperature,
temperature: tools.SplitDiskTemperature(r.PhysicalDisk.Temperature, r.PhysicalDisk.Collection),
smartEvidence: unifiedPhysicalDiskSMARTIssueParts(r.PhysicalDisk.SMART),
})
}
@@ -2513,7 +2516,7 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool)
health: health,
status: status,
wearout: d.Wearout,
temperature: d.Temperature,
temperature: tools.SplitDiskTemperature(d.Temperature, d.Collection),
smartEvidence: modelPhysicalDiskSMARTIssueParts(d.SmartAttributes),
})
}
@@ -2531,7 +2534,7 @@ func patrolPhysicalDiskHealthIssue(row patrolPhysicalDiskRow) bool {
(row.wearout == 0 &&
(strings.EqualFold(row.diskType, "ssd") || strings.EqualFold(row.diskType, "nvme")))) &&
row.wearout < 20) ||
row.temperature > 55 ||
row.temperature.Collected > 55 ||
len(row.smartEvidence) > 0
}
@@ -2959,8 +2962,8 @@ func (p *PatrolService) seedResourceInventoryState(snap patrolRuntimeState, scop
wear = fmt.Sprintf("%d%%", row.wearout)
}
temp := "—"
if row.temperature > 0 {
temp = fmt.Sprintf("%dC", row.temperature)
if text := row.temperature.Format("C"); text != "" {
temp = text
}
sb.WriteString(fmt.Sprintf("| %s | %s | %s | %s | %s | %s | %s | %s |\n",
diskName, node, diskType, size, row.health, wear, temp, row.status))
@@ -3470,8 +3473,8 @@ func (p *PatrolService) seedHealthAndAlertsState(snap patrolRuntimeState, scoped
wearout = fmt.Sprintf("%d%%", row.wearout)
}
temp := "—"
if row.temperature > 0 {
temp = fmt.Sprintf("%d°C", row.temperature)
if text := row.temperature.Format("°C"); text != "" {
temp = text
}
smartEvidence := "—"
if len(row.smartEvidence) > 0 {
+66
View File
@@ -17,6 +17,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
)
type noExecutorChatService struct{}
@@ -1054,6 +1055,71 @@ func TestPatrolPhysicalDiskHealthIssueDistinguishesKnownZeroWearout(t *testing.T
}
}
// A retained last-known temperature (a disk in standby, a host agent past its
// reporting lease) is context, not heat. Patrol's disk issue gate, triage flags
// and finding verification judge only a collected reading, and both seed
// tables label the retained value as last known.
func TestPatrolPhysicalDiskRowsJudgeHeatOnCollectedTemperature(t *testing.T) {
standby := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}
collected := &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}
unified := newPatrolRuntimeState(models.StateSnapshot{})
unified.unifiedResourceProvider = &mockUnifiedResourceProvider{
getByTypeFunc: func(t unifiedresources.ResourceType) []unifiedresources.Resource {
if t != unifiedresources.ResourceTypePhysicalDisk {
return nil
}
return []unifiedresources.Resource{
{ID: "disk-standby", Name: "standby", Type: unifiedresources.ResourceTypePhysicalDisk, Status: unifiedresources.StatusOnline, ParentName: "tower",
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{DevPath: "/dev/sda", Model: "WDC WD80EFAX", DiskType: "sata", Health: "PASSED", Wearout: -1, Temperature: 61, Collection: standby}},
{ID: "disk-hot", Name: "hot", Type: unifiedresources.ResourceTypePhysicalDisk, Status: unifiedresources.StatusOnline, ParentName: "tower",
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{DevPath: "/dev/sdb", Model: "WDC WD80EFAX", DiskType: "sata", Health: "PASSED", Wearout: -1, Temperature: 58, Collection: collected}},
}
},
}
legacy := newPatrolRuntimeState(models.StateSnapshot{PhysicalDisks: []models.PhysicalDisk{
{ID: "disk-standby", Node: "tower", DevPath: "/dev/sda", Model: "WDC WD80EFAX", Type: "sata", Health: "PASSED", Wearout: -1, Temperature: 61, Collection: standby},
{ID: "disk-hot", Node: "tower", DevPath: "/dev/sdb", Model: "WDC WD80EFAX", Type: "sata", Health: "PASSED", Wearout: -1, Temperature: 58, Collection: collected},
}})
for name, snap := range map[string]patrolRuntimeState{"unified": unified, "state": legacy} {
t.Run(name, func(t *testing.T) {
byID := map[string]patrolPhysicalDiskRow{}
for _, row := range patrolPhysicalDiskRows(snap, nil) {
byID[row.id] = row
}
if patrolPhysicalDiskHealthIssue(byID["disk-standby"]) {
t.Fatalf("a standby disk's retained 61C is not a health issue: %+v", byID["disk-standby"])
}
if !patrolPhysicalDiskHealthIssue(byID["disk-hot"]) {
t.Fatalf("a collected 58C is a health issue: %+v", byID["disk-hot"])
}
flags := triageDiskHealthChecksState(snap, nil)
if len(flags) != 1 || !strings.Contains(flags[0].Reason, "Disk temperature 58") {
t.Fatalf("triage flags = %+v, want only the collected 58C disk", flags)
}
if verification, ok := patrolLookupPhysicalDiskVerificationState(snap, "disk-standby"); !ok || verification.temperature != 0 {
t.Fatalf("standby disk verification = %+v (found %v), want no temperature", verification, ok)
}
})
}
ps := NewPatrolService(nil, nil)
cfg := DefaultPatrolConfig()
now := time.Now()
health := ps.seedHealthAndAlertsState(unified, nil, cfg, now)
for _, part := range []string{"| last known 61°C (disk is in standby) |", "| 58°C |"} {
if !strings.Contains(health, part) {
t.Fatalf("disk health table missing %q:\n%s", part, health)
}
}
inventory := ps.seedResourceInventoryState(unified, nil, cfg, now, false, nil)
for _, part := range []string{"| last known 61C (disk is in standby) |", "| 58C |"} {
if !strings.Contains(inventory, part) {
t.Fatalf("physical disk inventory missing %q:\n%s", part, inventory)
}
}
}
func TestSeedHealthAndAlerts_WithIssues(t *testing.T) {
ps := NewPatrolService(nil, nil)
cfg := DefaultPatrolConfig()
+1 -1
View File
@@ -2463,7 +2463,7 @@ func patrolVisitPhysicalDiskVerification(snap patrolRuntimeState, visit patrolPh
if !visit([]string{disk.id, disk.name, disk.devPath, disk.model}, patrolPhysicalDiskVerification{
health: strings.TrimSpace(disk.health),
wearout: disk.wearout,
temperature: disk.temperature,
temperature: disk.temperature.Collected,
}) {
return true
}
+1 -1
View File
@@ -345,7 +345,7 @@ func triageDiskHealthChecksState(snap patrolRuntimeState, scopedSet map[string]b
flags := make([]TriageFlag, 0)
for _, disk := range patrolPhysicalDiskRows(snap, scopedSet) {
flags = append(flags, triagePhysicalDiskFlags(disk.id, disk.name, disk.health, disk.wearout, disk.temperature)...)
flags = append(flags, triagePhysicalDiskFlags(disk.id, disk.name, disk.health, disk.wearout, disk.temperature.Collected)...)
}
return flags
+5 -3
View File
@@ -5,6 +5,7 @@ import (
"sort"
"strings"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rs/zerolog/log"
)
@@ -412,7 +413,8 @@ func (s *Service) buildUnifiedResourceContextForModel(destinationModel string) s
temperature := 0
if disk.PhysicalDisk != nil {
health = strings.ToUpper(strings.TrimSpace(disk.PhysicalDisk.Health))
temperature = disk.PhysicalDisk.Temperature
// A retained last-known reading is not current heat.
temperature = tools.SplitDiskTemperature(disk.PhysicalDisk.Temperature, disk.PhysicalDisk.Collection).Collected
}
if disk.Status != unifiedresources.StatusOnline || (health != "" && health != "PASSED" && health != "UNKNOWN") || temperature >= 50 {
attention = append(attention, disk)
@@ -431,8 +433,8 @@ func (s *Service) buildUnifiedResourceContextForModel(destinationModel string) s
if value := strings.TrimSpace(disk.PhysicalDisk.Health); value != "" {
health = value
}
if disk.PhysicalDisk.Temperature > 0 {
temperature = fmt.Sprintf(", Temp: %dC", disk.PhysicalDisk.Temperature)
if text := tools.SplitDiskTemperature(disk.PhysicalDisk.Temperature, disk.PhysicalDisk.Collection).Format("C"); text != "" {
temperature = ", Temp: " + text
}
}
healthSummary := ""
+51
View File
@@ -8,6 +8,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
)
func TestBuildUnifiedResourceContext_NilProvider(t *testing.T) {
@@ -741,6 +742,56 @@ func TestBuildUnifiedResourceContextIncludesTrueNASResources(t *testing.T) {
}
}
// A retained disk temperature (a disk in standby, a host agent past its
// reporting lease) is not current heat: it must not put a disk on the
// attention list, and a disk listed for another reason shows it as last known.
func TestBuildUnifiedResourceContextJudgesDiskHeatOnCollectedTemperature(t *testing.T) {
disk := func(id string, status unifiedresources.ResourceStatus, temperature int, collection diskinventory.FieldStatus) unifiedresources.Resource {
return unifiedresources.Resource{
ID: id, Name: id, Type: unifiedresources.ResourceTypePhysicalDisk, Status: status, ParentName: "tower",
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{
Health: "PASSED", Wearout: -1, Temperature: temperature,
Collection: &diskinventory.CollectionStatus{Temperature: collection},
},
}
}
disks := []unifiedresources.Resource{
disk("standby-disk", unifiedresources.StatusOnline, 58, diskinventory.Unavailable("smartctl", "disk is in standby")),
disk("silent-disk", unifiedresources.StatusOffline, 52, diskinventory.Unavailable("smartctl", "host agent stopped reporting")),
disk("hot-disk", unifiedresources.StatusOnline, 55, diskinventory.Available("smartctl")),
}
s := &Service{unifiedResourceProvider: &mockUnifiedResourceProvider{
getStatsFunc: func() unifiedresources.ResourceStats {
return unifiedresources.ResourceStats{Total: len(disks), ByType: map[unifiedresources.ResourceType]int{unifiedresources.ResourceTypePhysicalDisk: len(disks)}}
},
getByTypeFunc: func(resourceType unifiedresources.ResourceType) []unifiedresources.Resource {
if resourceType == unifiedresources.ResourceTypePhysicalDisk {
return disks
}
return nil
},
}}
got := s.buildUnifiedResourceContext()
_, attention, found := strings.Cut(got, "**Physical Disks Needing Attention:**")
if !found {
t.Fatalf("expected disk attention section, got %q", got)
}
attention, _, _ = strings.Cut(attention, "\n\n")
if entries := strings.Count(attention, "\n- "); entries != 2 {
t.Fatalf("attention entries = %d, want the offline and the collected-hot disk only: %q", entries, attention)
}
if strings.Contains(attention, "58C") {
t.Fatalf("a standby disk's retained 58C must not flag it for heat: %q", attention)
}
if !strings.Contains(attention, "(PASSED), Temp: last known 52C (host agent stopped reporting) [offline]") {
t.Fatalf("expected the offline disk's retained temperature as last known: %q", attention)
}
if !strings.Contains(attention, "(PASSED), Temp: 55C [online]") {
t.Fatalf("expected the collected 55C disk on the attention list: %q", attention)
}
}
func TestBuildUnifiedResourceContext_TruncatesLargeContext(t *testing.T) {
largeName := strings.Repeat("a", 60000)
+13 -2
View File
@@ -1222,7 +1222,12 @@ type SMARTDiskSummary struct {
Device string `json:"device"`
Model string `json:"model"`
Health string `json:"health"`
Temperature int `json:"temperature,omitempty"`
Temperature int `json:"temperature,omitempty"` // Celsius, only when collected now
// LastKnownTemperature is a temperature kept from an earlier observation
// when the current one did not collect it (a disk in standby, a host
// agent past its reporting lease); LastKnownTemperatureReason says why.
LastKnownTemperature int `json:"last_known_temperature,omitempty"`
LastKnownTemperatureReason string `json:"last_known_temperature_reason,omitempty"`
}
// RAIDArraySummary is a summarized RAID array
@@ -2045,10 +2050,16 @@ type PhysicalDiskSummary struct {
Risk *unifiedresources.PhysicalDiskRisk `json:"risk,omitempty"`
SMART *unifiedresources.SMARTMeta `json:"smart,omitempty"`
LifeRemainingPercent *int `json:"life_remaining_percent,omitempty"` // SSD life remaining percent (0-100), nil when unavailable
Temperature *int `json:"temperature,omitempty"` // Celsius, nil when unavailable
Temperature *int `json:"temperature,omitempty"` // Celsius, nil unless collected now
RPM *int `json:"rpm,omitempty"` // 0 for SSDs, nil when unavailable
Used string `json:"used,omitempty"`
LastChecked time.Time `json:"last_checked,omitempty"`
// LastKnownTemperature is a temperature kept from an earlier observation
// when the current one did not collect it (a disk in standby, a host
// agent past its reporting lease); LastKnownTemperatureReason says why.
LastKnownTemperature *int `json:"last_known_temperature,omitempty"`
LastKnownTemperatureReason string `json:"last_known_temperature_reason,omitempty"`
}
// ========== Host RAID Types ==========
+54
View File
@@ -0,0 +1,54 @@
package tools
import (
"fmt"
"strings"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
)
// DiskTemperature splits a physical disk temperature by its collection state
// for AI context. Collected is a reading the current observation took, and it
// is the only temperature that may be judged as heat. LastKnown is a value
// normalization kept without collecting it now (a disk in standby, a host
// agent past its reporting lease), with the collection state's reason in
// Reason. It is history, never evidence that the disk is hot now.
type DiskTemperature struct {
Collected int
LastKnown int
Reason string
}
// SplitDiskTemperature classifies a disk temperature with
// diskinventory.TemperatureCollected.
func SplitDiskTemperature(temperature int, collection *diskinventory.CollectionStatus) DiskTemperature {
if diskinventory.TemperatureCollected(temperature, collection) {
return DiskTemperature{Collected: temperature}
}
if temperature <= 0 {
return DiskTemperature{}
}
split := DiskTemperature{LastKnown: temperature}
if collection != nil {
split.Reason = strings.TrimSpace(collection.Temperature.Reason)
}
return split
}
// Format renders the temperature with unit for AI text: a collected reading as
// the value ("41C"), a retained one as "last known 41C (disk is in standby)",
// and "" when the disk has no temperature.
func (t DiskTemperature) Format(unit string) string {
switch {
case t.Collected > 0:
return fmt.Sprintf("%d%s", t.Collected, unit)
case t.LastKnown > 0:
text := fmt.Sprintf("last known %d%s", t.LastKnown, unit)
if t.Reason != "" {
text += " (" + t.Reason + ")"
}
return text
default:
return ""
}
}
@@ -9,6 +9,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/storagehealth"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/stretchr/testify/require"
)
@@ -111,3 +112,106 @@ func TestPhysicalDiskEvidencePreservesCanonicalRisk(t *testing.T) {
}
}
// Normalization keeps a disk's last-known temperature when the current
// observation did not collect one (a disk in standby, a host agent past its
// reporting lease). The tools must not hand that value to the model as the
// disk's temperature: it moves to last_known_temperature with its reason.
func TestDiskToolsReportRetainedTemperatureAsLastKnown(t *testing.T) {
standby := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}
silent := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "host agent stopped reporting")}
collected := &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}
resources := []unifiedresources.Resource{
{ID: "disk-standby", Name: "Standby disk", Type: unifiedresources.ResourceTypePhysicalDisk,
Status: unifiedresources.StatusOnline, ParentName: "node-one",
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{Health: "PASSED", Wearout: -1, Temperature: 61, Collection: standby}},
{ID: "disk-live", Name: "Live disk", Type: unifiedresources.ResourceTypePhysicalDisk,
Status: unifiedresources.StatusOnline, ParentName: "node-one",
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{Health: "PASSED", Wearout: -1, Temperature: 38, Collection: collected}},
}
diskProvider := &mockDiskHealthProvider{}
diskProvider.On("GetHosts").Return([]*unifiedresources.HostView{
newHostView("host-tower", "tower", "agent-tower", "tower", &unifiedresources.HostSensorMeta{
SMART: []unifiedresources.HostSMARTMeta{
{Device: "/dev/sda", Model: "WDC WD80EFAX", Health: "PASSED", Temperature: 58, Collection: silent},
{Device: "/dev/sdb", Model: "WDC WD80EFAX", Health: "PASSED", Temperature: 36, Collection: collected},
},
}, nil, nil),
})
executor := NewPulseToolExecutor(ExecutorConfig{
StateProvider: &mockStateProvider{state: models.StateSnapshot{}},
UnifiedResourceProvider: &stubUnifiedResourceProvider{resources: resources},
DiskHealthProvider: diskProvider,
ControlLevel: ControlLevelReadOnly,
})
for _, tool := range []struct {
name string
args map[string]interface{}
key string
}{
{"pulse_metrics", map[string]interface{}{"type": "disks"}, "disks"},
{"pulse_query", map[string]interface{}{"action": "list", "type": "physical-disks"}, "physical_disks"},
} {
t.Run(tool.name, func(t *testing.T) {
result, err := executor.ExecuteTool(context.Background(), tool.name, tool.args)
require.NoError(t, err)
require.False(t, result.IsError, "%+v", result.Content)
var payload map[string]json.RawMessage
require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &payload))
var disks []PhysicalDiskSummary
require.NoError(t, json.Unmarshal(payload[tool.key], &disks))
byID := map[string]PhysicalDiskSummary{}
for _, disk := range disks {
byID[disk.ID] = disk
}
retained := byID["disk-standby"]
require.Nil(t, retained.Temperature, "a retained reading is not the disk's temperature")
require.NotNil(t, retained.LastKnownTemperature)
require.Equal(t, 61, *retained.LastKnownTemperature)
require.Equal(t, "disk is in standby", retained.LastKnownTemperatureReason)
live := byID["disk-live"]
require.NotNil(t, live.Temperature)
require.Equal(t, 38, *live.Temperature)
require.Nil(t, live.LastKnownTemperature)
require.NotContains(t, string(payload[tool.key]), `"temperature":61`)
})
}
t.Run("pulse_storage disk_health", func(t *testing.T) {
result, err := executor.ExecuteTool(context.Background(), "pulse_storage", map[string]interface{}{"type": "disk_health"})
require.NoError(t, err)
require.False(t, result.IsError, "%+v", result.Content)
var response DiskHealthResponse
require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &response))
require.Len(t, response.Hosts, 1)
require.Len(t, response.Hosts[0].SMART, 2)
silentDisk, liveDisk := response.Hosts[0].SMART[0], response.Hosts[0].SMART[1]
require.Zero(t, silentDisk.Temperature)
require.Equal(t, 58, silentDisk.LastKnownTemperature)
require.Equal(t, "host agent stopped reporting", silentDisk.LastKnownTemperatureReason)
require.Equal(t, 36, liveDisk.Temperature)
require.Zero(t, liveDisk.LastKnownTemperature)
require.NotContains(t, result.Content[0].Text, `"temperature":58`)
})
}
func TestDiskTemperatureFormatsRetainedReadingAsLastKnown(t *testing.T) {
cases := []struct {
temperature int
collection *diskinventory.CollectionStatus
want string
}{
{41, nil, "41C"},
{41, &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}, "41C"},
{41, &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}, "last known 41C (disk is in standby)"},
{41, &diskinventory.CollectionStatus{Temperature: diskinventory.Missing("smartctl", "")}, "last known 41C"},
{0, &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}, ""},
{0, nil, ""},
}
for _, tc := range cases {
split := SplitDiskTemperature(tc.temperature, tc.collection)
require.Equal(t, tc.want, split.Format("C"), "%d %+v", tc.temperature, tc.collection)
require.Equal(t, diskinventory.TemperatureCollected(tc.temperature, tc.collection), split.Collected > 0)
}
}
+6 -3
View File
@@ -3583,9 +3583,12 @@ func physicalDiskSummaryFromResource(resource unifiedresources.Resource) Physica
wearout := pd.Wearout
summary.LifeRemainingPercent = &wearout
}
if pd.Temperature > 0 {
temp := pd.Temperature
summary.Temperature = &temp
temperature := SplitDiskTemperature(pd.Temperature, pd.Collection)
if temperature.Collected > 0 {
summary.Temperature = &temperature.Collected
} else if temperature.LastKnown > 0 {
summary.LastKnownTemperature = &temperature.LastKnown
summary.LastKnownTemperatureReason = temperature.Reason
}
if pd.RPM > 0 {
rpm := pd.RPM
+7 -4
View File
@@ -402,11 +402,14 @@ func (e *PulseToolExecutor) executeGetDiskHealth(_ context.Context, _ map[string
// SMART data
if sensors := host.Sensors(); sensors != nil {
for _, disk := range sensors.SMART {
temperature := SplitDiskTemperature(disk.Temperature, disk.Collection)
hostHealth.SMART = append(hostHealth.SMART, SMARTDiskSummary{
Device: disk.Device,
Model: disk.Model,
Health: disk.Health,
Temperature: disk.Temperature,
Device: disk.Device,
Model: disk.Model,
Health: disk.Health,
Temperature: temperature.Collected,
LastKnownTemperature: temperature.LastKnown,
LastKnownTemperatureReason: temperature.Reason,
})
}
}
+58
View File
@@ -5228,6 +5228,64 @@ func TestContract_PerformanceReportTransportUsesCatalogDefinition(t *testing.T)
}
}
// Normalization keeps a disk's last-known temperature when the current
// observation did not collect one (a disk in standby, a host agent past its
// reporting lease). Reports present their disk tables as measured, so the
// performance report and the reporting runtime snapshot leave that value out.
func TestContract_ReportsOmitRetainedDiskTemperatures(t *testing.T) {
engine := &stubReportingEngine{data: []byte("report"), contentType: "application/pdf"}
original := reporting.GetEngine()
reporting.SetEngine(engine)
t.Cleanup(func() { reporting.SetEngine(original) })
state := models.NewState()
state.Nodes = []models.Node{{ID: "node-1", Name: "node-a", Status: "online"}}
disk := func(id, devPath string, temperature int, collection diskinventory.FieldStatus) unifiedresources.Resource {
return unifiedresources.Resource{
ID: id, Type: unifiedresources.ResourceTypePhysicalDisk, Name: id, ParentName: "node-a",
Identity: unifiedresources.ResourceIdentity{Hostnames: []string{"node-a"}},
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{
DevPath: devPath, DiskType: "hdd", Health: "PASSED", Wearout: -1, Temperature: temperature,
Collection: &diskinventory.CollectionStatus{Temperature: collection},
},
}
}
monitor := newReportingMonitorForTest(t, state, []unifiedresources.Resource{
disk("disk-standby", "/dev/sda", 61, diskinventory.Unavailable("smartctl", "disk is in standby")),
disk("disk-silent", "/dev/sdb", 57, diskinventory.Unavailable("smartctl", "host agent stopped reporting")),
disk("disk-live", "/dev/sdc", 38, diskinventory.Available("smartctl")),
})
handler := NewReportingHandlers(newReportingMTMForTest(t, monitor), nil)
want := map[string]int{"/dev/sda": 0, "/dev/sdb": 0, "/dev/sdc": 38}
req := httptest.NewRequest(http.MethodGet, "/api/reporting?format=pdf&resourceType=node&resourceId=node-1", nil)
req = req.WithContext(context.WithValue(req.Context(), OrgIDContextKey, "default"))
rec := httptest.NewRecorder()
handler.HandleGenerateReport(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", rec.Code, rec.Body.String())
}
reported := map[string]int{}
for _, d := range engine.lastReq.Disks {
reported[d.Device] = d.Temperature
}
if !reflect.DeepEqual(reported, want) {
t.Fatalf("performance report disk temperatures = %v, want %v", reported, want)
}
snapshot, ok := handler.getRuntimeStateSnapshot(context.Background(), "default")
if !ok {
t.Fatal("expected runtime snapshot to be available")
}
snapshotted := map[string]int{}
for _, d := range snapshot.Disks {
snapshotted[d.Device] = d.Temperature
}
if !reflect.DeepEqual(snapshotted, want) {
t.Fatalf("reporting runtime snapshot disk temperatures = %v, want %v", snapshotted, want)
}
}
func TestContract_ReportingRequestCarriesEntitledReportBranding(t *testing.T) {
engine := &stubReportingEngine{data: []byte("report"), contentType: "application/pdf"}
original := reporting.GetEngine()
+14 -1
View File
@@ -20,6 +20,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/rcourtman/pulse-go-rewrite/pkg/reporting"
"github.com/rs/zerolog/log"
)
@@ -882,13 +883,25 @@ func (h *ReportingHandlers) enrichNodeReport(req *reporting.MetricReportRequest,
Type: pd.DiskType,
Size: pd.SizeBytes,
Health: pd.Health,
Temperature: pd.Temperature,
Temperature: reportDiskTemperature(*pd),
WearLevel: pd.Wearout,
})
}
}
}
// reportDiskTemperature returns the disk temperature a report may tabulate:
// one the current observation collected. Normalization may keep a last-known
// temperature it did not collect (a disk in standby, a host agent past its
// reporting lease); a report presents its values as measured, so that one is
// left out (0, rendered as no reading) rather than shown as current.
func reportDiskTemperature(pd unifiedresources.PhysicalDiskMeta) int {
if !diskinventory.TemperatureCollected(pd.Temperature, pd.Collection) {
return 0
}
return pd.Temperature
}
// enrichVMReport adds VM-specific data to the report request
func (h *ReportingHandlers) enrichVMReport(ctx context.Context, orgID string, req *reporting.MetricReportRequest, snapshot reportingEnrichmentSnapshot, start, end time.Time) {
// Find the VM
+1 -1
View File
@@ -191,7 +191,7 @@ func (h *ReportingHandlers) getRuntimeStateSnapshot(ctx context.Context, orgID s
Type: disk.DiskType,
Size: disk.SizeBytes,
Health: disk.Health,
Temperature: disk.Temperature,
Temperature: reportDiskTemperature(*disk),
WearLevel: disk.Wearout,
})
}
@@ -319,8 +319,9 @@ func hasIssue1595Source(sources []unifiedresources.DataSource, want unifiedresou
}
// An agent report's SMART row may carry a temperature its collector did not
// collect this time (retained across a standby or failed probe). Only a
// collected temperature becomes a history sample.
// collect this time (retained across a standby or failed probe, or kept after
// the agent's reporting lease expired). Only a temperature
// diskinventory.TemperatureCollected accepts becomes a history sample.
func TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures(t *testing.T) {
storeConfig := metrics.DefaultConfig(t.TempDir())
storeConfig.FlushInterval = time.Hour
@@ -337,6 +338,10 @@ func TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures(t *testing.T) {
{Device: "sdb", Serial: "LEGACY1", Temperature: 38},
{Device: "sdc", Serial: "RETAINED1", Temperature: 36,
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}},
{Device: "sdd", Serial: "SILENT1", Temperature: 44,
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", models.HostAgentStoppedReportingReason)}},
{Device: "sde", Serial: "MISSING1", Temperature: 39,
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Missing("smartctl", "temperature attribute absent")}},
}}}
now := time.Now()
m.writeHostSMARTMetrics(host, now)
@@ -349,6 +354,8 @@ func TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures(t *testing.T) {
{host.Sensors.SMART[0], 1},
{host.Sensors.SMART[1], 1},
{host.Sensors.SMART[2], 0},
{host.Sensors.SMART[3], 0},
{host.Sensors.SMART[4], 0},
} {
id := unifiedresources.HostSMARTDiskSourceID(host, tc.disk)
points, err := store.Query("disk", id, "smart_temp", now.Add(-time.Minute), now.Add(time.Minute), 0)
+2 -18
View File
@@ -939,7 +939,7 @@ func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
return
}
if diskTemperatureCollected(disk.Temperature, disk.Collection) && m.metricsHistory != nil {
if diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) && m.metricsHistory != nil {
m.metricsHistory.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), now)
}
@@ -953,22 +953,6 @@ func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
}
}
// diskTemperatureCollected reports whether a disk temperature was collected by
// its current observation. Normalization may carry a last-known temperature
// when it was not (standby, an agent past its reporting lease); that value must
// not be recorded as a new history sample. A temperature without collection
// state predates the contract and keeps the old behavior.
func diskTemperatureCollected(temperature int, collection *diskinventory.CollectionStatus) bool {
if temperature <= 0 {
return false
}
if collection == nil {
return true
}
state := collection.Temperature.State
return state == "" || state == diskinventory.FieldAvailable
}
// smartMetricStoreWrites builds the persisted SMART writes for one physical
// disk observation. The caller owns batching so the unified sync can anchor the
// observation time and drop exact replays (#1966).
@@ -989,7 +973,7 @@ func (m *Monitor) smartMetricStoreWrites(disk models.PhysicalDisk, resourceID st
})
}
if diskTemperatureCollected(disk.Temperature, disk.Collection) {
if diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) {
appendWrite("smart_temp", float64(disk.Temperature))
}
+2 -1
View File
@@ -26,6 +26,7 @@ import (
agentsdocker "github.com/rcourtman/pulse-go-rewrite/pkg/agents/docker"
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters"
pkglicensing "github.com/rcourtman/pulse-go-rewrite/pkg/licensing"
"github.com/rs/zerolog"
@@ -4086,7 +4087,7 @@ func (m *Monitor) writeHostSMARTMetrics(host models.Host, now time.Time) {
continue
}
if diskTemperatureCollected(disk.Temperature, disk.Collection) {
if diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) {
m.metricsStore.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), now)
}
+29 -15
View File
@@ -8,6 +8,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
)
@@ -267,11 +268,13 @@ func (m *Monitor) GetPhysicalDiskTemperatureCharts(duration time.Duration) map[s
// Phase 1: Collect disk metadata and resource IDs.
type diskMeta struct {
resourceID string
name string
node string
instance string
temperature int
resourceID string
name string
node string
instance string
// collectedTemperature is the reading the current observation took,
// or 0 when the disk only retains a last-known value.
collectedTemperature int
}
var disks []diskMeta
for _, disk := range readState.PhysicalDisks() {
@@ -294,12 +297,16 @@ func (m *Monitor) GetPhysicalDiskTemperatureCharts(duration time.Duration) map[s
if name == "" {
name = strings.TrimSpace(disk.DevPath())
}
collectedTemperature := 0
if diskinventory.TemperatureCollected(disk.Temperature(), disk.Collection()) {
collectedTemperature = disk.Temperature()
}
disks = append(disks, diskMeta{
resourceID: resourceID,
name: name,
node: strings.TrimSpace(disk.Node()),
instance: strings.TrimSpace(disk.Instance()),
temperature: disk.Temperature(),
resourceID: resourceID,
name: name,
node: strings.TrimSpace(disk.Node()),
instance: strings.TrimSpace(disk.Instance()),
collectedTemperature: collectedTemperature,
})
}
@@ -349,12 +356,19 @@ func (m *Monitor) GetPhysicalDiskTemperatureCharts(duration time.Duration) map[s
// Sparklines require >= 2 points. If the store returned 0 or 1 points
// but the disk has a live temperature reading, pad to 2 points so the
// chart can render (flat line at current temperature).
// chart can render (flat line at current temperature). A retained
// last-known temperature (standby, a silent host agent) is not a
// reading taken now, so the stored samples stay as they are and a
// disk with none has no series.
if len(tempPoints) < 2 {
now := time.Now()
tempPoints = []MetricPoint{
{Timestamp: now.Add(-60 * time.Second), Value: float64(d.temperature)},
{Timestamp: now, Value: float64(d.temperature)},
if d.collectedTemperature > 0 {
now := time.Now()
tempPoints = []MetricPoint{
{Timestamp: now.Add(-60 * time.Second), Value: float64(d.collectedTemperature)},
{Timestamp: now, Value: float64(d.collectedTemperature)},
}
} else if len(tempPoints) == 0 {
continue
}
}
@@ -11,6 +11,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
)
@@ -671,6 +672,50 @@ func TestSLO_GetPhysicalDiskTemperatureCharts_WithNativeHistoryFallback(t *testi
assertLatencySLO(t, "GetPhysicalDiskTemperatureCharts(native-history fallback)", latencies, target)
}
// The disk temperature charts pad a short series to now with the disk's
// current reading. A retained last-known temperature (a disk in standby, a host
// agent past its reporting lease) is not a reading taken now: the samples
// stored while the disk was live stay as they are, and a disk with none gets no
// series rather than a flat line at the retained value.
func TestDiskTemperatureChartsPadOnlyWithCollectedReading(t *testing.T) {
now := time.Now().UTC()
standby := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}
silent := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", models.HostAgentStoppedReportingReason)}
collected := &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}
registry := unifiedresources.NewRegistry(nil)
registry.IngestSnapshot(models.StateSnapshot{PhysicalDisks: []models.PhysicalDisk{
{ID: "disk-standby", Node: "node-a", DevPath: "/dev/sda", Model: "WDC WD80EFAX", Serial: "STANDBY-NO-HISTORY", Temperature: 41, Collection: standby, LastChecked: now},
{ID: "disk-silent", Node: "node-a", DevPath: "/dev/sdb", Model: "WDC WD80EFAX", Serial: "SILENT-ONE-SAMPLE", Temperature: 43, Collection: silent, LastChecked: now},
{ID: "disk-live", Node: "node-a", DevPath: "/dev/sdc", Model: "WDC WD80EFAX", Serial: "LIVE-NO-HISTORY", Temperature: 38, Collection: collected, LastChecked: now},
}})
m := &Monitor{
metricsHistory: NewMetricsHistory(1024, 24*time.Hour),
state: models.NewState(),
resourceStore: unifiedresources.NewMonitorAdapter(registry),
}
sampledAt := now.Add(-20 * time.Minute)
m.metricsHistory.AddDiskMetric("SILENT-ONE-SAMPLE", "smart_temp", 44, sampledAt)
charts := m.GetPhysicalDiskTemperatureCharts(time.Hour)
if entry, ok := charts["STANDBY-NO-HISTORY"]; ok {
t.Fatalf("a standby disk with no stored samples got a series from its retained reading: %+v", entry.Temperature)
}
silentEntry, ok := charts["SILENT-ONE-SAMPLE"]
if !ok || len(silentEntry.Temperature) != 1 || silentEntry.Temperature[0].Value != 44 ||
!silentEntry.Temperature[0].Timestamp.Equal(sampledAt) {
t.Fatalf("silent agent disk series = %+v (found %v), want only its stored 44C sample at %s", silentEntry.Temperature, ok, sampledAt)
}
live, ok := charts["LIVE-NO-HISTORY"]
if !ok || len(live.Temperature) != 2 {
t.Fatalf("live disk series = %+v (found %v), want the padded 2-point sparkline", live.Temperature, ok)
}
for _, point := range live.Temperature {
if point.Value != 38 {
t.Fatalf("live disk padded with %.0f, want its collected 38C", point.Value)
}
}
}
func TestSLO_GetDiskMetricsForChart_WithNativeStoreFallback(t *testing.T) {
skipMonitoringSLOUnderRace(t)
suppressMonitoringTestLogs(t)
@@ -536,7 +536,7 @@ func TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease(t *test
if reporting.Temperature != 38 || reporting.Collection.Temperature != diskinventory.Available(hostAgentLegacySource) {
t.Fatalf("legacy agent temperature not recorded as collected: temp=%d collection=%+v", reporting.Temperature, reporting.Collection)
}
if !diskTemperatureCollected(reporting.Temperature, reporting.Collection) {
if !diskinventory.TemperatureCollected(reporting.Temperature, reporting.Collection) {
t.Fatal("a reporting legacy agent's temperature must reach history")
}
@@ -549,7 +549,7 @@ func TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease(t *test
if got.Temperature != 38 || got.Collection.Temperature.State != diskinventory.FieldUnavailable {
t.Fatalf("%s: silent legacy agent temperature presented as collected: temp=%d collection=%+v", name, got.Temperature, got.Collection)
}
if diskTemperatureCollected(got.Temperature, got.Collection) {
if diskinventory.TemperatureCollected(got.Temperature, got.Collection) {
t.Fatalf("%s: a silent legacy agent's retained temperature must not reach history", name)
}
}