mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-23 20:25:56 +00:00
Keep retained disk temperatures out of AI, Patrol and reports
Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. Several readers still took PhysicalDisk.Temperature at face value. The AI chat context listed a standby disk as needing attention for heat. Patrol raised health issues and triage flags from the retained value. The AI query and disk-health tools handed it to the model as the disk's temperature. Performance reports and the reporting runtime snapshot tabulated it as measured, and the disk temperature charts padded a series out to now with it. Each of them now goes through diskinventory.TemperatureCollected. AI surfaces keep a retained value as context, labelled last known with its reason. Reports and the charts leave it out. monitor.go's private copy of the predicate is replaced by the shared helper.
This commit is contained in:
@@ -760,6 +760,11 @@ offline the unified-resources adapter reports them `unavailable` with the same
|
||||
reason (`models.HostAgentStoppedReportingReason`) and leaves them out of the
|
||||
disk's risk, and the metrics-history API does not return any such retained
|
||||
disk temperature as a live point.
|
||||
Downstream readers keep that distinction too. The disk temperature charts do
|
||||
not pad a series to now with the retained value. The AI chat context, Patrol and
|
||||
the AI disk tools present it only as a last-known value with its reason. The
|
||||
performance report and the reporting runtime snapshot leave it out of their
|
||||
disk tables.
|
||||
|
||||
An enabled availability target assigned to a host agent creates an
|
||||
agent-lifecycle lease for that exact target/agent pairing. First assignment
|
||||
|
||||
@@ -288,6 +288,24 @@ exposing the source-specific ambiguous `wearout` name, and distinguishes it from
|
||||
`smart.percentageUsed` as consumed endurance. The regression proof is
|
||||
`internal/ai/tools/physical_disk_evidence_test.go`.
|
||||
|
||||
Disk temperatures reach the model as current readings only when they were
|
||||
collected now. `tools.SplitDiskTemperature` classifies a physical disk
|
||||
temperature with `diskinventory.TemperatureCollected`. The physical-disk
|
||||
projection's `temperature` and the `disk_health` SMART row's `temperature`
|
||||
carry only a reading the current observation collected. A value normalization
|
||||
retained instead (a disk in standby, a host agent past its reporting lease)
|
||||
moves to `last_known_temperature` with `last_known_temperature_reason`. The
|
||||
same split feeds the AI chat context and Patrol. The "Physical Disks Needing
|
||||
Attention" list and Patrol's disk issue gate, triage flags and finding
|
||||
verification judge heat on the collected reading alone. Their text renders a
|
||||
retained value as "last known 41C (disk is in standby)". Proofs:
|
||||
`TestDiskToolsReportRetainedTemperatureAsLastKnown` in
|
||||
`internal/ai/tools/physical_disk_evidence_test.go`,
|
||||
`TestBuildUnifiedResourceContextJudgesDiskHeatOnCollectedTemperature` in
|
||||
`internal/ai/resource_context_test.go` and
|
||||
`TestPatrolPhysicalDiskRowsJudgeHeatOnCollectedTemperature` in
|
||||
`internal/ai/patrol_ai_more_test.go`.
|
||||
|
||||
Retained summaries disclose that point and bucket timestamps describe returned
|
||||
history inside the requested window. Their spacing does not measure collection
|
||||
uptime or explain missing history. Collector lifecycle and retention settings
|
||||
|
||||
@@ -6332,6 +6332,14 @@ stored one out to `now`, only from a temperature collected now; without one the
|
||||
stored samples are returned as they are. An empty mock range can still fall
|
||||
through to the monitor's generic demo chart series for that disk ID, which is
|
||||
not derived from the disk's reading.
|
||||
Reports follow the same rule. A performance report's disk table
|
||||
(`enrichNodeReport` in `internal/api/metrics_reporting_handlers.go`) and the
|
||||
reporting runtime snapshot's disks (`internal/api/reporting_runtime_snapshot.go`)
|
||||
take a disk temperature through `reportDiskTemperature`. It keeps a reading only
|
||||
when `diskinventory.TemperatureCollected` holds, so a retained value is reported
|
||||
as 0 (no reading) rather than tabulated as measured. Proof:
|
||||
`TestContract_ReportsOmitRetainedDiskTemperatures` in
|
||||
`internal/api/contract_test.go`.
|
||||
That same metrics-history contract also owns Kubernetes pod identity
|
||||
normalization. `/api/metrics-store/history` must accept legacy bare pod IDs
|
||||
such as `cluster-1:pod:pod-1`, canonicalize them onto the unified pod metrics
|
||||
|
||||
@@ -3815,6 +3815,15 @@ consumers outside the history writers: the metrics-history API serves a disk
|
||||
series from the disk's reading or pads a stored one, only from a collected
|
||||
temperature, so a retained standby or silent-agent reading never reappears as a
|
||||
point at the current time.
|
||||
The SMART history writers in `internal/monitoring/monitor.go` and
|
||||
`monitor_agents.go` call that helper rather than a private copy of the rule,
|
||||
and `GetPhysicalDiskTemperatureCharts` pads a short disk temperature series out
|
||||
to now only with a collected reading: with a retained one the stored samples
|
||||
stay as they are, and a disk with none gets no series. Mock mode can still
|
||||
substitute its generic demo series for that disk ID before this step; that
|
||||
series is not derived from the disk's reading. Proof:
|
||||
`TestDiskTemperatureChartsPadOnlyWithCollectedReading` in
|
||||
`internal/monitoring/monitor_metrics_slo_test.go`.
|
||||
Unified-resource physical-disk round trips must retain named
|
||||
`StorageGroup` membership rather than degrading it to the generic `Used`
|
||||
filesystem label.
|
||||
|
||||
@@ -6779,3 +6779,6 @@ and the Unraid physical-disk adapter marks an inventory temperature kept past
|
||||
the host agent's lease as no longer collected. Both change presentation of
|
||||
values Pulse already holds: no metrics write, retention window, backup,
|
||||
migration or recovery authority is added or moved.
|
||||
The performance report and reporting runtime snapshot handlers apply the same
|
||||
test before tabulating a disk temperature. That changes only which held value a
|
||||
report shows, not any storage or recovery path.
|
||||
|
||||
@@ -1927,8 +1927,11 @@ type patrolPhysicalDiskRow struct {
|
||||
sizeBytes int64
|
||||
devPath, model string
|
||||
health, status string
|
||||
wearout, temperature int
|
||||
smartEvidence []string
|
||||
wearout int
|
||||
// temperature.Collected is the only temperature Patrol may judge as
|
||||
// heat; a retained last-known value is shown as context.
|
||||
temperature tools.DiskTemperature
|
||||
smartEvidence []string
|
||||
}
|
||||
|
||||
type patrolPrecomputeNodeSource struct {
|
||||
@@ -2472,7 +2475,7 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool)
|
||||
health: health,
|
||||
status: status,
|
||||
wearout: r.PhysicalDisk.Wearout,
|
||||
temperature: r.PhysicalDisk.Temperature,
|
||||
temperature: tools.SplitDiskTemperature(r.PhysicalDisk.Temperature, r.PhysicalDisk.Collection),
|
||||
smartEvidence: unifiedPhysicalDiskSMARTIssueParts(r.PhysicalDisk.SMART),
|
||||
})
|
||||
}
|
||||
@@ -2513,7 +2516,7 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool)
|
||||
health: health,
|
||||
status: status,
|
||||
wearout: d.Wearout,
|
||||
temperature: d.Temperature,
|
||||
temperature: tools.SplitDiskTemperature(d.Temperature, d.Collection),
|
||||
smartEvidence: modelPhysicalDiskSMARTIssueParts(d.SmartAttributes),
|
||||
})
|
||||
}
|
||||
@@ -2531,7 +2534,7 @@ func patrolPhysicalDiskHealthIssue(row patrolPhysicalDiskRow) bool {
|
||||
(row.wearout == 0 &&
|
||||
(strings.EqualFold(row.diskType, "ssd") || strings.EqualFold(row.diskType, "nvme")))) &&
|
||||
row.wearout < 20) ||
|
||||
row.temperature > 55 ||
|
||||
row.temperature.Collected > 55 ||
|
||||
len(row.smartEvidence) > 0
|
||||
}
|
||||
|
||||
@@ -2959,8 +2962,8 @@ func (p *PatrolService) seedResourceInventoryState(snap patrolRuntimeState, scop
|
||||
wear = fmt.Sprintf("%d%%", row.wearout)
|
||||
}
|
||||
temp := "—"
|
||||
if row.temperature > 0 {
|
||||
temp = fmt.Sprintf("%dC", row.temperature)
|
||||
if text := row.temperature.Format("C"); text != "" {
|
||||
temp = text
|
||||
}
|
||||
sb.WriteString(fmt.Sprintf("| %s | %s | %s | %s | %s | %s | %s | %s |\n",
|
||||
diskName, node, diskType, size, row.health, wear, temp, row.status))
|
||||
@@ -3470,8 +3473,8 @@ func (p *PatrolService) seedHealthAndAlertsState(snap patrolRuntimeState, scoped
|
||||
wearout = fmt.Sprintf("%d%%", row.wearout)
|
||||
}
|
||||
temp := "—"
|
||||
if row.temperature > 0 {
|
||||
temp = fmt.Sprintf("%d°C", row.temperature)
|
||||
if text := row.temperature.Format("°C"); text != "" {
|
||||
temp = text
|
||||
}
|
||||
smartEvidence := "—"
|
||||
if len(row.smartEvidence) > 0 {
|
||||
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
)
|
||||
|
||||
type noExecutorChatService struct{}
|
||||
@@ -1054,6 +1055,71 @@ func TestPatrolPhysicalDiskHealthIssueDistinguishesKnownZeroWearout(t *testing.T
|
||||
}
|
||||
}
|
||||
|
||||
// A retained last-known temperature (a disk in standby, a host agent past its
|
||||
// reporting lease) is context, not heat. Patrol's disk issue gate, triage flags
|
||||
// and finding verification judge only a collected reading, and both seed
|
||||
// tables label the retained value as last known.
|
||||
func TestPatrolPhysicalDiskRowsJudgeHeatOnCollectedTemperature(t *testing.T) {
|
||||
standby := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}
|
||||
collected := &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}
|
||||
unified := newPatrolRuntimeState(models.StateSnapshot{})
|
||||
unified.unifiedResourceProvider = &mockUnifiedResourceProvider{
|
||||
getByTypeFunc: func(t unifiedresources.ResourceType) []unifiedresources.Resource {
|
||||
if t != unifiedresources.ResourceTypePhysicalDisk {
|
||||
return nil
|
||||
}
|
||||
return []unifiedresources.Resource{
|
||||
{ID: "disk-standby", Name: "standby", Type: unifiedresources.ResourceTypePhysicalDisk, Status: unifiedresources.StatusOnline, ParentName: "tower",
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{DevPath: "/dev/sda", Model: "WDC WD80EFAX", DiskType: "sata", Health: "PASSED", Wearout: -1, Temperature: 61, Collection: standby}},
|
||||
{ID: "disk-hot", Name: "hot", Type: unifiedresources.ResourceTypePhysicalDisk, Status: unifiedresources.StatusOnline, ParentName: "tower",
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{DevPath: "/dev/sdb", Model: "WDC WD80EFAX", DiskType: "sata", Health: "PASSED", Wearout: -1, Temperature: 58, Collection: collected}},
|
||||
}
|
||||
},
|
||||
}
|
||||
legacy := newPatrolRuntimeState(models.StateSnapshot{PhysicalDisks: []models.PhysicalDisk{
|
||||
{ID: "disk-standby", Node: "tower", DevPath: "/dev/sda", Model: "WDC WD80EFAX", Type: "sata", Health: "PASSED", Wearout: -1, Temperature: 61, Collection: standby},
|
||||
{ID: "disk-hot", Node: "tower", DevPath: "/dev/sdb", Model: "WDC WD80EFAX", Type: "sata", Health: "PASSED", Wearout: -1, Temperature: 58, Collection: collected},
|
||||
}})
|
||||
|
||||
for name, snap := range map[string]patrolRuntimeState{"unified": unified, "state": legacy} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
byID := map[string]patrolPhysicalDiskRow{}
|
||||
for _, row := range patrolPhysicalDiskRows(snap, nil) {
|
||||
byID[row.id] = row
|
||||
}
|
||||
if patrolPhysicalDiskHealthIssue(byID["disk-standby"]) {
|
||||
t.Fatalf("a standby disk's retained 61C is not a health issue: %+v", byID["disk-standby"])
|
||||
}
|
||||
if !patrolPhysicalDiskHealthIssue(byID["disk-hot"]) {
|
||||
t.Fatalf("a collected 58C is a health issue: %+v", byID["disk-hot"])
|
||||
}
|
||||
flags := triageDiskHealthChecksState(snap, nil)
|
||||
if len(flags) != 1 || !strings.Contains(flags[0].Reason, "Disk temperature 58") {
|
||||
t.Fatalf("triage flags = %+v, want only the collected 58C disk", flags)
|
||||
}
|
||||
if verification, ok := patrolLookupPhysicalDiskVerificationState(snap, "disk-standby"); !ok || verification.temperature != 0 {
|
||||
t.Fatalf("standby disk verification = %+v (found %v), want no temperature", verification, ok)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
ps := NewPatrolService(nil, nil)
|
||||
cfg := DefaultPatrolConfig()
|
||||
now := time.Now()
|
||||
health := ps.seedHealthAndAlertsState(unified, nil, cfg, now)
|
||||
for _, part := range []string{"| last known 61°C (disk is in standby) |", "| 58°C |"} {
|
||||
if !strings.Contains(health, part) {
|
||||
t.Fatalf("disk health table missing %q:\n%s", part, health)
|
||||
}
|
||||
}
|
||||
inventory := ps.seedResourceInventoryState(unified, nil, cfg, now, false, nil)
|
||||
for _, part := range []string{"| last known 61C (disk is in standby) |", "| 58C |"} {
|
||||
if !strings.Contains(inventory, part) {
|
||||
t.Fatalf("physical disk inventory missing %q:\n%s", part, inventory)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedHealthAndAlerts_WithIssues(t *testing.T) {
|
||||
ps := NewPatrolService(nil, nil)
|
||||
cfg := DefaultPatrolConfig()
|
||||
|
||||
@@ -2463,7 +2463,7 @@ func patrolVisitPhysicalDiskVerification(snap patrolRuntimeState, visit patrolPh
|
||||
if !visit([]string{disk.id, disk.name, disk.devPath, disk.model}, patrolPhysicalDiskVerification{
|
||||
health: strings.TrimSpace(disk.health),
|
||||
wearout: disk.wearout,
|
||||
temperature: disk.temperature,
|
||||
temperature: disk.temperature.Collected,
|
||||
}) {
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -345,7 +345,7 @@ func triageDiskHealthChecksState(snap patrolRuntimeState, scopedSet map[string]b
|
||||
flags := make([]TriageFlag, 0)
|
||||
|
||||
for _, disk := range patrolPhysicalDiskRows(snap, scopedSet) {
|
||||
flags = append(flags, triagePhysicalDiskFlags(disk.id, disk.name, disk.health, disk.wearout, disk.temperature)...)
|
||||
flags = append(flags, triagePhysicalDiskFlags(disk.id, disk.name, disk.health, disk.wearout, disk.temperature.Collected)...)
|
||||
}
|
||||
|
||||
return flags
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"
|
||||
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
@@ -412,7 +413,8 @@ func (s *Service) buildUnifiedResourceContextForModel(destinationModel string) s
|
||||
temperature := 0
|
||||
if disk.PhysicalDisk != nil {
|
||||
health = strings.ToUpper(strings.TrimSpace(disk.PhysicalDisk.Health))
|
||||
temperature = disk.PhysicalDisk.Temperature
|
||||
// A retained last-known reading is not current heat.
|
||||
temperature = tools.SplitDiskTemperature(disk.PhysicalDisk.Temperature, disk.PhysicalDisk.Collection).Collected
|
||||
}
|
||||
if disk.Status != unifiedresources.StatusOnline || (health != "" && health != "PASSED" && health != "UNKNOWN") || temperature >= 50 {
|
||||
attention = append(attention, disk)
|
||||
@@ -431,8 +433,8 @@ func (s *Service) buildUnifiedResourceContextForModel(destinationModel string) s
|
||||
if value := strings.TrimSpace(disk.PhysicalDisk.Health); value != "" {
|
||||
health = value
|
||||
}
|
||||
if disk.PhysicalDisk.Temperature > 0 {
|
||||
temperature = fmt.Sprintf(", Temp: %dC", disk.PhysicalDisk.Temperature)
|
||||
if text := tools.SplitDiskTemperature(disk.PhysicalDisk.Temperature, disk.PhysicalDisk.Collection).Format("C"); text != "" {
|
||||
temperature = ", Temp: " + text
|
||||
}
|
||||
}
|
||||
healthSummary := ""
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
|
||||
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
)
|
||||
|
||||
func TestBuildUnifiedResourceContext_NilProvider(t *testing.T) {
|
||||
@@ -741,6 +742,56 @@ func TestBuildUnifiedResourceContextIncludesTrueNASResources(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A retained disk temperature (a disk in standby, a host agent past its
|
||||
// reporting lease) is not current heat: it must not put a disk on the
|
||||
// attention list, and a disk listed for another reason shows it as last known.
|
||||
func TestBuildUnifiedResourceContextJudgesDiskHeatOnCollectedTemperature(t *testing.T) {
|
||||
disk := func(id string, status unifiedresources.ResourceStatus, temperature int, collection diskinventory.FieldStatus) unifiedresources.Resource {
|
||||
return unifiedresources.Resource{
|
||||
ID: id, Name: id, Type: unifiedresources.ResourceTypePhysicalDisk, Status: status, ParentName: "tower",
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{
|
||||
Health: "PASSED", Wearout: -1, Temperature: temperature,
|
||||
Collection: &diskinventory.CollectionStatus{Temperature: collection},
|
||||
},
|
||||
}
|
||||
}
|
||||
disks := []unifiedresources.Resource{
|
||||
disk("standby-disk", unifiedresources.StatusOnline, 58, diskinventory.Unavailable("smartctl", "disk is in standby")),
|
||||
disk("silent-disk", unifiedresources.StatusOffline, 52, diskinventory.Unavailable("smartctl", "host agent stopped reporting")),
|
||||
disk("hot-disk", unifiedresources.StatusOnline, 55, diskinventory.Available("smartctl")),
|
||||
}
|
||||
s := &Service{unifiedResourceProvider: &mockUnifiedResourceProvider{
|
||||
getStatsFunc: func() unifiedresources.ResourceStats {
|
||||
return unifiedresources.ResourceStats{Total: len(disks), ByType: map[unifiedresources.ResourceType]int{unifiedresources.ResourceTypePhysicalDisk: len(disks)}}
|
||||
},
|
||||
getByTypeFunc: func(resourceType unifiedresources.ResourceType) []unifiedresources.Resource {
|
||||
if resourceType == unifiedresources.ResourceTypePhysicalDisk {
|
||||
return disks
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}}
|
||||
|
||||
got := s.buildUnifiedResourceContext()
|
||||
_, attention, found := strings.Cut(got, "**Physical Disks Needing Attention:**")
|
||||
if !found {
|
||||
t.Fatalf("expected disk attention section, got %q", got)
|
||||
}
|
||||
attention, _, _ = strings.Cut(attention, "\n\n")
|
||||
if entries := strings.Count(attention, "\n- "); entries != 2 {
|
||||
t.Fatalf("attention entries = %d, want the offline and the collected-hot disk only: %q", entries, attention)
|
||||
}
|
||||
if strings.Contains(attention, "58C") {
|
||||
t.Fatalf("a standby disk's retained 58C must not flag it for heat: %q", attention)
|
||||
}
|
||||
if !strings.Contains(attention, "(PASSED), Temp: last known 52C (host agent stopped reporting) [offline]") {
|
||||
t.Fatalf("expected the offline disk's retained temperature as last known: %q", attention)
|
||||
}
|
||||
if !strings.Contains(attention, "(PASSED), Temp: 55C [online]") {
|
||||
t.Fatalf("expected the collected 55C disk on the attention list: %q", attention)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildUnifiedResourceContext_TruncatesLargeContext(t *testing.T) {
|
||||
largeName := strings.Repeat("a", 60000)
|
||||
|
||||
|
||||
@@ -1222,7 +1222,12 @@ type SMARTDiskSummary struct {
|
||||
Device string `json:"device"`
|
||||
Model string `json:"model"`
|
||||
Health string `json:"health"`
|
||||
Temperature int `json:"temperature,omitempty"`
|
||||
Temperature int `json:"temperature,omitempty"` // Celsius, only when collected now
|
||||
// LastKnownTemperature is a temperature kept from an earlier observation
|
||||
// when the current one did not collect it (a disk in standby, a host
|
||||
// agent past its reporting lease); LastKnownTemperatureReason says why.
|
||||
LastKnownTemperature int `json:"last_known_temperature,omitempty"`
|
||||
LastKnownTemperatureReason string `json:"last_known_temperature_reason,omitempty"`
|
||||
}
|
||||
|
||||
// RAIDArraySummary is a summarized RAID array
|
||||
@@ -2045,10 +2050,16 @@ type PhysicalDiskSummary struct {
|
||||
Risk *unifiedresources.PhysicalDiskRisk `json:"risk,omitempty"`
|
||||
SMART *unifiedresources.SMARTMeta `json:"smart,omitempty"`
|
||||
LifeRemainingPercent *int `json:"life_remaining_percent,omitempty"` // SSD life remaining percent (0-100), nil when unavailable
|
||||
Temperature *int `json:"temperature,omitempty"` // Celsius, nil when unavailable
|
||||
Temperature *int `json:"temperature,omitempty"` // Celsius, nil unless collected now
|
||||
RPM *int `json:"rpm,omitempty"` // 0 for SSDs, nil when unavailable
|
||||
Used string `json:"used,omitempty"`
|
||||
LastChecked time.Time `json:"last_checked,omitempty"`
|
||||
|
||||
// LastKnownTemperature is a temperature kept from an earlier observation
|
||||
// when the current one did not collect it (a disk in standby, a host
|
||||
// agent past its reporting lease); LastKnownTemperatureReason says why.
|
||||
LastKnownTemperature *int `json:"last_known_temperature,omitempty"`
|
||||
LastKnownTemperatureReason string `json:"last_known_temperature_reason,omitempty"`
|
||||
}
|
||||
|
||||
// ========== Host RAID Types ==========
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
)
|
||||
|
||||
// DiskTemperature splits a physical disk temperature by its collection state
|
||||
// for AI context. Collected is a reading the current observation took, and it
|
||||
// is the only temperature that may be judged as heat. LastKnown is a value
|
||||
// normalization kept without collecting it now (a disk in standby, a host
|
||||
// agent past its reporting lease), with the collection state's reason in
|
||||
// Reason. It is history, never evidence that the disk is hot now.
|
||||
type DiskTemperature struct {
|
||||
Collected int
|
||||
LastKnown int
|
||||
Reason string
|
||||
}
|
||||
|
||||
// SplitDiskTemperature classifies a disk temperature with
|
||||
// diskinventory.TemperatureCollected.
|
||||
func SplitDiskTemperature(temperature int, collection *diskinventory.CollectionStatus) DiskTemperature {
|
||||
if diskinventory.TemperatureCollected(temperature, collection) {
|
||||
return DiskTemperature{Collected: temperature}
|
||||
}
|
||||
if temperature <= 0 {
|
||||
return DiskTemperature{}
|
||||
}
|
||||
split := DiskTemperature{LastKnown: temperature}
|
||||
if collection != nil {
|
||||
split.Reason = strings.TrimSpace(collection.Temperature.Reason)
|
||||
}
|
||||
return split
|
||||
}
|
||||
|
||||
// Format renders the temperature with unit for AI text: a collected reading as
|
||||
// the value ("41C"), a retained one as "last known 41C (disk is in standby)",
|
||||
// and "" when the disk has no temperature.
|
||||
func (t DiskTemperature) Format(unit string) string {
|
||||
switch {
|
||||
case t.Collected > 0:
|
||||
return fmt.Sprintf("%d%s", t.Collected, unit)
|
||||
case t.LastKnown > 0:
|
||||
text := fmt.Sprintf("last known %d%s", t.LastKnown, unit)
|
||||
if t.Reason != "" {
|
||||
text += " (" + t.Reason + ")"
|
||||
}
|
||||
return text
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/storagehealth"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
@@ -111,3 +112,106 @@ func TestPhysicalDiskEvidencePreservesCanonicalRisk(t *testing.T) {
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// Normalization keeps a disk's last-known temperature when the current
|
||||
// observation did not collect one (a disk in standby, a host agent past its
|
||||
// reporting lease). The tools must not hand that value to the model as the
|
||||
// disk's temperature: it moves to last_known_temperature with its reason.
|
||||
func TestDiskToolsReportRetainedTemperatureAsLastKnown(t *testing.T) {
|
||||
standby := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}
|
||||
silent := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "host agent stopped reporting")}
|
||||
collected := &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}
|
||||
resources := []unifiedresources.Resource{
|
||||
{ID: "disk-standby", Name: "Standby disk", Type: unifiedresources.ResourceTypePhysicalDisk,
|
||||
Status: unifiedresources.StatusOnline, ParentName: "node-one",
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{Health: "PASSED", Wearout: -1, Temperature: 61, Collection: standby}},
|
||||
{ID: "disk-live", Name: "Live disk", Type: unifiedresources.ResourceTypePhysicalDisk,
|
||||
Status: unifiedresources.StatusOnline, ParentName: "node-one",
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{Health: "PASSED", Wearout: -1, Temperature: 38, Collection: collected}},
|
||||
}
|
||||
diskProvider := &mockDiskHealthProvider{}
|
||||
diskProvider.On("GetHosts").Return([]*unifiedresources.HostView{
|
||||
newHostView("host-tower", "tower", "agent-tower", "tower", &unifiedresources.HostSensorMeta{
|
||||
SMART: []unifiedresources.HostSMARTMeta{
|
||||
{Device: "/dev/sda", Model: "WDC WD80EFAX", Health: "PASSED", Temperature: 58, Collection: silent},
|
||||
{Device: "/dev/sdb", Model: "WDC WD80EFAX", Health: "PASSED", Temperature: 36, Collection: collected},
|
||||
},
|
||||
}, nil, nil),
|
||||
})
|
||||
executor := NewPulseToolExecutor(ExecutorConfig{
|
||||
StateProvider: &mockStateProvider{state: models.StateSnapshot{}},
|
||||
UnifiedResourceProvider: &stubUnifiedResourceProvider{resources: resources},
|
||||
DiskHealthProvider: diskProvider,
|
||||
ControlLevel: ControlLevelReadOnly,
|
||||
})
|
||||
|
||||
for _, tool := range []struct {
|
||||
name string
|
||||
args map[string]interface{}
|
||||
key string
|
||||
}{
|
||||
{"pulse_metrics", map[string]interface{}{"type": "disks"}, "disks"},
|
||||
{"pulse_query", map[string]interface{}{"action": "list", "type": "physical-disks"}, "physical_disks"},
|
||||
} {
|
||||
t.Run(tool.name, func(t *testing.T) {
|
||||
result, err := executor.ExecuteTool(context.Background(), tool.name, tool.args)
|
||||
require.NoError(t, err)
|
||||
require.False(t, result.IsError, "%+v", result.Content)
|
||||
var payload map[string]json.RawMessage
|
||||
require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &payload))
|
||||
var disks []PhysicalDiskSummary
|
||||
require.NoError(t, json.Unmarshal(payload[tool.key], &disks))
|
||||
byID := map[string]PhysicalDiskSummary{}
|
||||
for _, disk := range disks {
|
||||
byID[disk.ID] = disk
|
||||
}
|
||||
retained := byID["disk-standby"]
|
||||
require.Nil(t, retained.Temperature, "a retained reading is not the disk's temperature")
|
||||
require.NotNil(t, retained.LastKnownTemperature)
|
||||
require.Equal(t, 61, *retained.LastKnownTemperature)
|
||||
require.Equal(t, "disk is in standby", retained.LastKnownTemperatureReason)
|
||||
live := byID["disk-live"]
|
||||
require.NotNil(t, live.Temperature)
|
||||
require.Equal(t, 38, *live.Temperature)
|
||||
require.Nil(t, live.LastKnownTemperature)
|
||||
require.NotContains(t, string(payload[tool.key]), `"temperature":61`)
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("pulse_storage disk_health", func(t *testing.T) {
|
||||
result, err := executor.ExecuteTool(context.Background(), "pulse_storage", map[string]interface{}{"type": "disk_health"})
|
||||
require.NoError(t, err)
|
||||
require.False(t, result.IsError, "%+v", result.Content)
|
||||
var response DiskHealthResponse
|
||||
require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &response))
|
||||
require.Len(t, response.Hosts, 1)
|
||||
require.Len(t, response.Hosts[0].SMART, 2)
|
||||
silentDisk, liveDisk := response.Hosts[0].SMART[0], response.Hosts[0].SMART[1]
|
||||
require.Zero(t, silentDisk.Temperature)
|
||||
require.Equal(t, 58, silentDisk.LastKnownTemperature)
|
||||
require.Equal(t, "host agent stopped reporting", silentDisk.LastKnownTemperatureReason)
|
||||
require.Equal(t, 36, liveDisk.Temperature)
|
||||
require.Zero(t, liveDisk.LastKnownTemperature)
|
||||
require.NotContains(t, result.Content[0].Text, `"temperature":58`)
|
||||
})
|
||||
}
|
||||
|
||||
func TestDiskTemperatureFormatsRetainedReadingAsLastKnown(t *testing.T) {
|
||||
cases := []struct {
|
||||
temperature int
|
||||
collection *diskinventory.CollectionStatus
|
||||
want string
|
||||
}{
|
||||
{41, nil, "41C"},
|
||||
{41, &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}, "41C"},
|
||||
{41, &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}, "last known 41C (disk is in standby)"},
|
||||
{41, &diskinventory.CollectionStatus{Temperature: diskinventory.Missing("smartctl", "")}, "last known 41C"},
|
||||
{0, &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}, ""},
|
||||
{0, nil, ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
split := SplitDiskTemperature(tc.temperature, tc.collection)
|
||||
require.Equal(t, tc.want, split.Format("C"), "%d %+v", tc.temperature, tc.collection)
|
||||
require.Equal(t, diskinventory.TemperatureCollected(tc.temperature, tc.collection), split.Collected > 0)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3583,9 +3583,12 @@ func physicalDiskSummaryFromResource(resource unifiedresources.Resource) Physica
|
||||
wearout := pd.Wearout
|
||||
summary.LifeRemainingPercent = &wearout
|
||||
}
|
||||
if pd.Temperature > 0 {
|
||||
temp := pd.Temperature
|
||||
summary.Temperature = &temp
|
||||
temperature := SplitDiskTemperature(pd.Temperature, pd.Collection)
|
||||
if temperature.Collected > 0 {
|
||||
summary.Temperature = &temperature.Collected
|
||||
} else if temperature.LastKnown > 0 {
|
||||
summary.LastKnownTemperature = &temperature.LastKnown
|
||||
summary.LastKnownTemperatureReason = temperature.Reason
|
||||
}
|
||||
if pd.RPM > 0 {
|
||||
rpm := pd.RPM
|
||||
|
||||
@@ -402,11 +402,14 @@ func (e *PulseToolExecutor) executeGetDiskHealth(_ context.Context, _ map[string
|
||||
// SMART data
|
||||
if sensors := host.Sensors(); sensors != nil {
|
||||
for _, disk := range sensors.SMART {
|
||||
temperature := SplitDiskTemperature(disk.Temperature, disk.Collection)
|
||||
hostHealth.SMART = append(hostHealth.SMART, SMARTDiskSummary{
|
||||
Device: disk.Device,
|
||||
Model: disk.Model,
|
||||
Health: disk.Health,
|
||||
Temperature: disk.Temperature,
|
||||
Device: disk.Device,
|
||||
Model: disk.Model,
|
||||
Health: disk.Health,
|
||||
Temperature: temperature.Collected,
|
||||
LastKnownTemperature: temperature.LastKnown,
|
||||
LastKnownTemperatureReason: temperature.Reason,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5228,6 +5228,64 @@ func TestContract_PerformanceReportTransportUsesCatalogDefinition(t *testing.T)
|
||||
}
|
||||
}
|
||||
|
||||
// Normalization keeps a disk's last-known temperature when the current
|
||||
// observation did not collect one (a disk in standby, a host agent past its
|
||||
// reporting lease). Reports present their disk tables as measured, so the
|
||||
// performance report and the reporting runtime snapshot leave that value out.
|
||||
func TestContract_ReportsOmitRetainedDiskTemperatures(t *testing.T) {
|
||||
engine := &stubReportingEngine{data: []byte("report"), contentType: "application/pdf"}
|
||||
original := reporting.GetEngine()
|
||||
reporting.SetEngine(engine)
|
||||
t.Cleanup(func() { reporting.SetEngine(original) })
|
||||
|
||||
state := models.NewState()
|
||||
state.Nodes = []models.Node{{ID: "node-1", Name: "node-a", Status: "online"}}
|
||||
disk := func(id, devPath string, temperature int, collection diskinventory.FieldStatus) unifiedresources.Resource {
|
||||
return unifiedresources.Resource{
|
||||
ID: id, Type: unifiedresources.ResourceTypePhysicalDisk, Name: id, ParentName: "node-a",
|
||||
Identity: unifiedresources.ResourceIdentity{Hostnames: []string{"node-a"}},
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{
|
||||
DevPath: devPath, DiskType: "hdd", Health: "PASSED", Wearout: -1, Temperature: temperature,
|
||||
Collection: &diskinventory.CollectionStatus{Temperature: collection},
|
||||
},
|
||||
}
|
||||
}
|
||||
monitor := newReportingMonitorForTest(t, state, []unifiedresources.Resource{
|
||||
disk("disk-standby", "/dev/sda", 61, diskinventory.Unavailable("smartctl", "disk is in standby")),
|
||||
disk("disk-silent", "/dev/sdb", 57, diskinventory.Unavailable("smartctl", "host agent stopped reporting")),
|
||||
disk("disk-live", "/dev/sdc", 38, diskinventory.Available("smartctl")),
|
||||
})
|
||||
handler := NewReportingHandlers(newReportingMTMForTest(t, monitor), nil)
|
||||
want := map[string]int{"/dev/sda": 0, "/dev/sdb": 0, "/dev/sdc": 38}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/reporting?format=pdf&resourceType=node&resourceId=node-1", nil)
|
||||
req = req.WithContext(context.WithValue(req.Context(), OrgIDContextKey, "default"))
|
||||
rec := httptest.NewRecorder()
|
||||
handler.HandleGenerateReport(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
reported := map[string]int{}
|
||||
for _, d := range engine.lastReq.Disks {
|
||||
reported[d.Device] = d.Temperature
|
||||
}
|
||||
if !reflect.DeepEqual(reported, want) {
|
||||
t.Fatalf("performance report disk temperatures = %v, want %v", reported, want)
|
||||
}
|
||||
|
||||
snapshot, ok := handler.getRuntimeStateSnapshot(context.Background(), "default")
|
||||
if !ok {
|
||||
t.Fatal("expected runtime snapshot to be available")
|
||||
}
|
||||
snapshotted := map[string]int{}
|
||||
for _, d := range snapshot.Disks {
|
||||
snapshotted[d.Device] = d.Temperature
|
||||
}
|
||||
if !reflect.DeepEqual(snapshotted, want) {
|
||||
t.Fatalf("reporting runtime snapshot disk temperatures = %v, want %v", snapshotted, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContract_ReportingRequestCarriesEntitledReportBranding(t *testing.T) {
|
||||
engine := &stubReportingEngine{data: []byte("report"), contentType: "application/pdf"}
|
||||
original := reporting.GetEngine()
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/reporting"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
@@ -882,13 +883,25 @@ func (h *ReportingHandlers) enrichNodeReport(req *reporting.MetricReportRequest,
|
||||
Type: pd.DiskType,
|
||||
Size: pd.SizeBytes,
|
||||
Health: pd.Health,
|
||||
Temperature: pd.Temperature,
|
||||
Temperature: reportDiskTemperature(*pd),
|
||||
WearLevel: pd.Wearout,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reportDiskTemperature returns the disk temperature a report may tabulate:
|
||||
// one the current observation collected. Normalization may keep a last-known
|
||||
// temperature it did not collect (a disk in standby, a host agent past its
|
||||
// reporting lease); a report presents its values as measured, so that one is
|
||||
// left out (0, rendered as no reading) rather than shown as current.
|
||||
func reportDiskTemperature(pd unifiedresources.PhysicalDiskMeta) int {
|
||||
if !diskinventory.TemperatureCollected(pd.Temperature, pd.Collection) {
|
||||
return 0
|
||||
}
|
||||
return pd.Temperature
|
||||
}
|
||||
|
||||
// enrichVMReport adds VM-specific data to the report request
|
||||
func (h *ReportingHandlers) enrichVMReport(ctx context.Context, orgID string, req *reporting.MetricReportRequest, snapshot reportingEnrichmentSnapshot, start, end time.Time) {
|
||||
// Find the VM
|
||||
|
||||
@@ -191,7 +191,7 @@ func (h *ReportingHandlers) getRuntimeStateSnapshot(ctx context.Context, orgID s
|
||||
Type: disk.DiskType,
|
||||
Size: disk.SizeBytes,
|
||||
Health: disk.Health,
|
||||
Temperature: disk.Temperature,
|
||||
Temperature: reportDiskTemperature(*disk),
|
||||
WearLevel: disk.Wearout,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -319,8 +319,9 @@ func hasIssue1595Source(sources []unifiedresources.DataSource, want unifiedresou
|
||||
}
|
||||
|
||||
// An agent report's SMART row may carry a temperature its collector did not
|
||||
// collect this time (retained across a standby or failed probe). Only a
|
||||
// collected temperature becomes a history sample.
|
||||
// collect this time (retained across a standby or failed probe, or kept after
|
||||
// the agent's reporting lease expired). Only a temperature
|
||||
// diskinventory.TemperatureCollected accepts becomes a history sample.
|
||||
func TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures(t *testing.T) {
|
||||
storeConfig := metrics.DefaultConfig(t.TempDir())
|
||||
storeConfig.FlushInterval = time.Hour
|
||||
@@ -337,6 +338,10 @@ func TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures(t *testing.T) {
|
||||
{Device: "sdb", Serial: "LEGACY1", Temperature: 38},
|
||||
{Device: "sdc", Serial: "RETAINED1", Temperature: 36,
|
||||
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}},
|
||||
{Device: "sdd", Serial: "SILENT1", Temperature: 44,
|
||||
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", models.HostAgentStoppedReportingReason)}},
|
||||
{Device: "sde", Serial: "MISSING1", Temperature: 39,
|
||||
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Missing("smartctl", "temperature attribute absent")}},
|
||||
}}}
|
||||
now := time.Now()
|
||||
m.writeHostSMARTMetrics(host, now)
|
||||
@@ -349,6 +354,8 @@ func TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures(t *testing.T) {
|
||||
{host.Sensors.SMART[0], 1},
|
||||
{host.Sensors.SMART[1], 1},
|
||||
{host.Sensors.SMART[2], 0},
|
||||
{host.Sensors.SMART[3], 0},
|
||||
{host.Sensors.SMART[4], 0},
|
||||
} {
|
||||
id := unifiedresources.HostSMARTDiskSourceID(host, tc.disk)
|
||||
points, err := store.Query("disk", id, "smart_temp", now.Add(-time.Minute), now.Add(time.Minute), 0)
|
||||
|
||||
@@ -939,7 +939,7 @@ func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
|
||||
return
|
||||
}
|
||||
|
||||
if diskTemperatureCollected(disk.Temperature, disk.Collection) && m.metricsHistory != nil {
|
||||
if diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) && m.metricsHistory != nil {
|
||||
m.metricsHistory.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
}
|
||||
|
||||
@@ -953,22 +953,6 @@ func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
|
||||
}
|
||||
}
|
||||
|
||||
// diskTemperatureCollected reports whether a disk temperature was collected by
|
||||
// its current observation. Normalization may carry a last-known temperature
|
||||
// when it was not (standby, an agent past its reporting lease); that value must
|
||||
// not be recorded as a new history sample. A temperature without collection
|
||||
// state predates the contract and keeps the old behavior.
|
||||
func diskTemperatureCollected(temperature int, collection *diskinventory.CollectionStatus) bool {
|
||||
if temperature <= 0 {
|
||||
return false
|
||||
}
|
||||
if collection == nil {
|
||||
return true
|
||||
}
|
||||
state := collection.Temperature.State
|
||||
return state == "" || state == diskinventory.FieldAvailable
|
||||
}
|
||||
|
||||
// smartMetricStoreWrites builds the persisted SMART writes for one physical
|
||||
// disk observation. The caller owns batching so the unified sync can anchor the
|
||||
// observation time and drop exact replays (#1966).
|
||||
@@ -989,7 +973,7 @@ func (m *Monitor) smartMetricStoreWrites(disk models.PhysicalDisk, resourceID st
|
||||
})
|
||||
}
|
||||
|
||||
if diskTemperatureCollected(disk.Temperature, disk.Collection) {
|
||||
if diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) {
|
||||
appendWrite("smart_temp", float64(disk.Temperature))
|
||||
}
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ import (
|
||||
agentsdocker "github.com/rcourtman/pulse-go-rewrite/pkg/agents/docker"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters"
|
||||
pkglicensing "github.com/rcourtman/pulse-go-rewrite/pkg/licensing"
|
||||
"github.com/rs/zerolog"
|
||||
@@ -4086,7 +4087,7 @@ func (m *Monitor) writeHostSMARTMetrics(host models.Host, now time.Time) {
|
||||
continue
|
||||
}
|
||||
|
||||
if diskTemperatureCollected(disk.Temperature, disk.Collection) {
|
||||
if diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) {
|
||||
m.metricsStore.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
}
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
|
||||
)
|
||||
|
||||
@@ -267,11 +268,13 @@ func (m *Monitor) GetPhysicalDiskTemperatureCharts(duration time.Duration) map[s
|
||||
|
||||
// Phase 1: Collect disk metadata and resource IDs.
|
||||
type diskMeta struct {
|
||||
resourceID string
|
||||
name string
|
||||
node string
|
||||
instance string
|
||||
temperature int
|
||||
resourceID string
|
||||
name string
|
||||
node string
|
||||
instance string
|
||||
// collectedTemperature is the reading the current observation took,
|
||||
// or 0 when the disk only retains a last-known value.
|
||||
collectedTemperature int
|
||||
}
|
||||
var disks []diskMeta
|
||||
for _, disk := range readState.PhysicalDisks() {
|
||||
@@ -294,12 +297,16 @@ func (m *Monitor) GetPhysicalDiskTemperatureCharts(duration time.Duration) map[s
|
||||
if name == "" {
|
||||
name = strings.TrimSpace(disk.DevPath())
|
||||
}
|
||||
collectedTemperature := 0
|
||||
if diskinventory.TemperatureCollected(disk.Temperature(), disk.Collection()) {
|
||||
collectedTemperature = disk.Temperature()
|
||||
}
|
||||
disks = append(disks, diskMeta{
|
||||
resourceID: resourceID,
|
||||
name: name,
|
||||
node: strings.TrimSpace(disk.Node()),
|
||||
instance: strings.TrimSpace(disk.Instance()),
|
||||
temperature: disk.Temperature(),
|
||||
resourceID: resourceID,
|
||||
name: name,
|
||||
node: strings.TrimSpace(disk.Node()),
|
||||
instance: strings.TrimSpace(disk.Instance()),
|
||||
collectedTemperature: collectedTemperature,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -349,12 +356,19 @@ func (m *Monitor) GetPhysicalDiskTemperatureCharts(duration time.Duration) map[s
|
||||
|
||||
// Sparklines require >= 2 points. If the store returned 0 or 1 points
|
||||
// but the disk has a live temperature reading, pad to 2 points so the
|
||||
// chart can render (flat line at current temperature).
|
||||
// chart can render (flat line at current temperature). A retained
|
||||
// last-known temperature (standby, a silent host agent) is not a
|
||||
// reading taken now, so the stored samples stay as they are and a
|
||||
// disk with none has no series.
|
||||
if len(tempPoints) < 2 {
|
||||
now := time.Now()
|
||||
tempPoints = []MetricPoint{
|
||||
{Timestamp: now.Add(-60 * time.Second), Value: float64(d.temperature)},
|
||||
{Timestamp: now, Value: float64(d.temperature)},
|
||||
if d.collectedTemperature > 0 {
|
||||
now := time.Now()
|
||||
tempPoints = []MetricPoint{
|
||||
{Timestamp: now.Add(-60 * time.Second), Value: float64(d.collectedTemperature)},
|
||||
{Timestamp: now, Value: float64(d.collectedTemperature)},
|
||||
}
|
||||
} else if len(tempPoints) == 0 {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
|
||||
)
|
||||
|
||||
@@ -671,6 +672,50 @@ func TestSLO_GetPhysicalDiskTemperatureCharts_WithNativeHistoryFallback(t *testi
|
||||
assertLatencySLO(t, "GetPhysicalDiskTemperatureCharts(native-history fallback)", latencies, target)
|
||||
}
|
||||
|
||||
// The disk temperature charts pad a short series to now with the disk's
|
||||
// current reading. A retained last-known temperature (a disk in standby, a host
|
||||
// agent past its reporting lease) is not a reading taken now: the samples
|
||||
// stored while the disk was live stay as they are, and a disk with none gets no
|
||||
// series rather than a flat line at the retained value.
|
||||
func TestDiskTemperatureChartsPadOnlyWithCollectedReading(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
standby := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}
|
||||
silent := &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", models.HostAgentStoppedReportingReason)}
|
||||
collected := &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}
|
||||
registry := unifiedresources.NewRegistry(nil)
|
||||
registry.IngestSnapshot(models.StateSnapshot{PhysicalDisks: []models.PhysicalDisk{
|
||||
{ID: "disk-standby", Node: "node-a", DevPath: "/dev/sda", Model: "WDC WD80EFAX", Serial: "STANDBY-NO-HISTORY", Temperature: 41, Collection: standby, LastChecked: now},
|
||||
{ID: "disk-silent", Node: "node-a", DevPath: "/dev/sdb", Model: "WDC WD80EFAX", Serial: "SILENT-ONE-SAMPLE", Temperature: 43, Collection: silent, LastChecked: now},
|
||||
{ID: "disk-live", Node: "node-a", DevPath: "/dev/sdc", Model: "WDC WD80EFAX", Serial: "LIVE-NO-HISTORY", Temperature: 38, Collection: collected, LastChecked: now},
|
||||
}})
|
||||
m := &Monitor{
|
||||
metricsHistory: NewMetricsHistory(1024, 24*time.Hour),
|
||||
state: models.NewState(),
|
||||
resourceStore: unifiedresources.NewMonitorAdapter(registry),
|
||||
}
|
||||
sampledAt := now.Add(-20 * time.Minute)
|
||||
m.metricsHistory.AddDiskMetric("SILENT-ONE-SAMPLE", "smart_temp", 44, sampledAt)
|
||||
|
||||
charts := m.GetPhysicalDiskTemperatureCharts(time.Hour)
|
||||
if entry, ok := charts["STANDBY-NO-HISTORY"]; ok {
|
||||
t.Fatalf("a standby disk with no stored samples got a series from its retained reading: %+v", entry.Temperature)
|
||||
}
|
||||
silentEntry, ok := charts["SILENT-ONE-SAMPLE"]
|
||||
if !ok || len(silentEntry.Temperature) != 1 || silentEntry.Temperature[0].Value != 44 ||
|
||||
!silentEntry.Temperature[0].Timestamp.Equal(sampledAt) {
|
||||
t.Fatalf("silent agent disk series = %+v (found %v), want only its stored 44C sample at %s", silentEntry.Temperature, ok, sampledAt)
|
||||
}
|
||||
live, ok := charts["LIVE-NO-HISTORY"]
|
||||
if !ok || len(live.Temperature) != 2 {
|
||||
t.Fatalf("live disk series = %+v (found %v), want the padded 2-point sparkline", live.Temperature, ok)
|
||||
}
|
||||
for _, point := range live.Temperature {
|
||||
if point.Value != 38 {
|
||||
t.Fatalf("live disk padded with %.0f, want its collected 38C", point.Value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSLO_GetDiskMetricsForChart_WithNativeStoreFallback(t *testing.T) {
|
||||
skipMonitoringSLOUnderRace(t)
|
||||
suppressMonitoringTestLogs(t)
|
||||
|
||||
@@ -536,7 +536,7 @@ func TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease(t *test
|
||||
if reporting.Temperature != 38 || reporting.Collection.Temperature != diskinventory.Available(hostAgentLegacySource) {
|
||||
t.Fatalf("legacy agent temperature not recorded as collected: temp=%d collection=%+v", reporting.Temperature, reporting.Collection)
|
||||
}
|
||||
if !diskTemperatureCollected(reporting.Temperature, reporting.Collection) {
|
||||
if !diskinventory.TemperatureCollected(reporting.Temperature, reporting.Collection) {
|
||||
t.Fatal("a reporting legacy agent's temperature must reach history")
|
||||
}
|
||||
|
||||
@@ -549,7 +549,7 @@ func TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease(t *test
|
||||
if got.Temperature != 38 || got.Collection.Temperature.State != diskinventory.FieldUnavailable {
|
||||
t.Fatalf("%s: silent legacy agent temperature presented as collected: temp=%d collection=%+v", name, got.Temperature, got.Collection)
|
||||
}
|
||||
if diskTemperatureCollected(got.Temperature, got.Collection) {
|
||||
if diskinventory.TemperatureCollected(got.Temperature, got.Collection) {
|
||||
t.Fatalf("%s: a silent legacy agent's retained temperature must not reach history", name)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user