gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-23 20:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix MD RAID spare counts without hiding member failures

Browse Source 
Retain configured RAID members separately from source-native totals through collection, reports and canonical read views. Correct the healthy legacy mdadm tuple from #2369 without subtracting spares from mdstat requirements or suppressing real deficits, failures and recovery warnings. Preserve observed zero-active counts through fallback, and verify collector, wire, ingestion, health and canonical activation/resolution boundaries.

Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-01 19:56:47 +01:00
parent 2ae06bb2f3
commit 14d3dc3b6c
27 changed files with 703 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+17
View File
@@ -15,6 +15,23 @@
## Purpose
### MD RAID required members and spares — issue #2369
Host RAID reports carry optional `requiredDevices`: the configured member count
from mdadm's `Raid Devices` or mdstat's required/active bitmap, excluding spares.
Zero/omission means unknown. `totalDevices` retains its existing source-native
meaning: attached devices including spares for mdadm, bitmap width for mdstat.
The existing QNAP sparse-role normalization applies to both bitmap counts.
Invalid or negative detail counts fall back to the kernel report, and missing
detail requirements may be enriched from that report. This is read-only
collection, not permission to change array membership or perform recovery.
Collector-to-wire/health/alert tests cover healthy spares, missing and failed
members, mdstat-only and malformed-detail fallback, and maintenance/recovery.
Report serialization and `buildReport` must preserve both counts independently.
An explicit zero active count under a known requirement is retained; neither
device-token inference nor a different kernel fallback probe may fill it in.
The collector/report controls exercise both zero-active boundaries.
### Continuity-aware broadcast read ownership
Broadcast state takes its single resource list from the current continuity-aware
docs/release-control/v6/internal/subsystems/ai-runtime.md
+10
View File
@@ -25,6 +25,16 @@ that same result. Successful reads retain their content and execution provenance
## Purpose
### RAID configured-member read evidence — issue #2369
The existing host RAID status tool preserves optional `required_devices` from
the canonical host view alongside source-native total, active, working, failed
and spare counts. Omission means the configured requirement is unknown, not
zero members. A total that includes spares is not evidence of a member deficit;
the shared health owner remains authoritative for risk. Tool serialization tests
check the separate counts. This read-only projection creates no feature, route,
array-repair grant, backup verdict or recovery confirmation.
### Ollama Basic Auth continuity
Patrol's runtime provider factory must carry the persisted Ollama username and
password into streaming requests without trimming password bytes.
docs/release-control/v6/internal/subsystems/alerts.md
+15
View File
@@ -15,6 +15,21 @@
## Purpose
### RAID spare-count lifecycle — issue #2369
Canonical host RAID alerts use the shared required-member health assessment,
not attached-device arithmetic. Known configured counts appear as
`raidRequiredDevices` metadata alongside unchanged total/active/failed/spare
observations. A healthy spare creates no incident; an actual missing member or
failed device still activates critical health. Scrub/resync/recovery and transient
operation expiry cannot mask a static deficit. Restoring all required members
with a spare resolves the existing incident through the same canonical state
identity, once, without a duplicate fire or resolve.
`TestRAIDSpareCanonicalActivationAndRecovery` checks event-ledger activation,
continuity and resolution; collector and ingestion tests check both health and
canonical alert severity. Routine scrub/resync remains silent, and genuine
recovery/reshape keeps its warning when no critical deficit exists.
### Retained occurrence re-fires preserve incident truth
A reducer re-fire inside the retention window keeps its original occurrence
docs/release-control/v6/internal/subsystems/monitoring.md
+17
View File
@@ -17,6 +17,23 @@
## Purpose
### RAID count provenance across ingestion — issue #2369
Agent ingestion, host snapshots and canonical read-state projection retain
`requiredDevices` separately from source-native `totalDevices` and spare counts.
Health compares active members with a known configured requirement, including
explicit zero-active deficits; a spare cannot fill a missing active role.
Collector fallback preserves an explicit zero active count under a known
requirement rather than substituting members from another probe.
Legacy reports retain conservative total-based detection except for an exact
clean/active, zero-failure tuple where total equals working and active plus
spares. That corroborated mdadm tuple is healthy; arbitrary subtraction from
mdstat totals is forbidden. Missing or unreconciled evidence does not gain
that exception. Explicit degradation/failures remain critical and scrub/resync
never hide a member deficit. `TestRAIDRequiredMembersReportAndReadState` pins
the wire/ingestion/snapshot/readback and canonical alert/risk paths; storage
health tests pin legacy compatibility and reconstruction/maintenance boundaries.
### Single-pass live broadcast projection — issue #2199
A frontend broadcast lists the continuity-aware read store once, coalesces host
docs/release-control/v6/internal/subsystems/storage-recovery.md
+13
View File
@@ -21,6 +21,19 @@
## Purpose
### RAID required-member count evidence — issue #2369
Canonical host RAID metadata carries optional `requiredDevices`, excluding
spares, separately from source-native attached/bitmap totals. Array protection
risk follows the shared required-member assessment rather than assuming that
every attached spare is a missing active member. Missing configured members,
failed devices and explicit degraded state remain critical; maintenance and
spare presence cannot erase those risks. A healthy spare is not recovery proof,
backup coverage or storage mutation authority. Collector, ingestion and
canonical event-ledger tests distinguish that correction from genuine member
recovery, which resolves the existing health incident only after the reported
active set again meets its requirement.
Incident history retains canonical storage-related risk, source identity and
separate observation/occurrence timestamps. A recorded alert closure or operator
note is not a restore result. Canonical read failure remains unavailable and
docs/release-control/v6/internal/subsystems/unified-resources.md
+12
View File
@@ -23,6 +23,18 @@ and sort the complete canonical change table while startup and ingestion wait.
## Purpose
### Canonical RAID configured-member evidence — issue #2369
Host RAID metadata and read views retain optional `requiredDevices` (configured
members excluding spares) without rewriting source-native `totalDevices`.
Clones preserve both values and isolate caller mutation. The shared storage
health assessment owns per-array risk and aggregate protection posture: a full
member set with a hot spare must not become critical, while a configured member
deficit remains critical even if active plus spares equals the attached count.
Adapter, clone and monitoring readback tests pin these boundaries. Count
evidence does not establish backup coverage, native array recovery or any
storage mutation authority; the field adds no resource or action identity.
### Bounded confidence/priority work in broadcast grouping — issue #2199
The general identity matcher retains every confidence, reason, ordering and
internal/ai/tools/data_types.go
+15 -14
View File
@@ -2092,20 +2092,21 @@ func (s HostRAIDSummary) NormalizeCollections() HostRAIDSummary {
// HostRAIDArraySummary summarizes a RAID array
type HostRAIDArraySummary struct {
Device string `json:"device"`
Name string `json:"name,omitempty"`
Level string `json:"level"` // raid0, raid1, raid5, etc.
State string `json:"state"` // clean, degraded, rebuilding
TotalDevices int `json:"total_devices"`
ActiveDevices int `json:"active_devices"`
WorkingDevices int `json:"working_devices"`
FailedDevices int `json:"failed_devices"`
SpareDevices int `json:"spare_devices"`
UUID string `json:"uuid,omitempty"`
RebuildPercent float64 `json:"rebuild_percent,omitempty"`
RebuildSpeed string `json:"rebuild_speed,omitempty"`
Operation string `json:"operation,omitempty"`
Devices []HostRAIDDeviceSummary `json:"devices"`
Device string `json:"device"`
Name string `json:"name,omitempty"`
Level string `json:"level"` // raid0, raid1, raid5, etc.
State string `json:"state"` // clean, degraded, rebuilding
RequiredDevices int `json:"required_devices,omitempty"` // configured members excluding spares; zero means unknown
TotalDevices int `json:"total_devices"`
ActiveDevices int `json:"active_devices"`
WorkingDevices int `json:"working_devices"`
FailedDevices int `json:"failed_devices"`
SpareDevices int `json:"spare_devices"`
UUID string `json:"uuid,omitempty"`
RebuildPercent float64 `json:"rebuild_percent,omitempty"`
RebuildSpeed string `json:"rebuild_speed,omitempty"`
Operation string `json:"operation,omitempty"`
Devices []HostRAIDDeviceSummary `json:"devices"`
}
func (s HostRAIDArraySummary) NormalizeCollections() HostRAIDArraySummary {
internal/ai/tools/infrastructure_low_coverage_test.go
+13 -6
View File
@@ -1013,12 +1013,14 @@ func TestExecuteGetHostRAIDStatus(t *testing.T) {
nil,
[]unifiedresources.HostRAIDMeta{
{
Device: "/dev/md0",
Level: "raid1",
State: "clean",
TotalDevices: 2,
ActiveDevices: 2,
WorkingDevices: 2,
Device: "/dev/md0",
Level: "raid1",
State: "clean",
RequiredDevices: 2,
TotalDevices: 3,
ActiveDevices: 2,
WorkingDevices: 3,
SpareDevices: 1,
Devices: []unifiedresources.HostRAIDDeviceMeta{
{Device: "/dev/sda", State: "active", Slot: 0},
},
@@ -1042,6 +1044,11 @@ func TestExecuteGetHostRAIDStatus(t *testing.T) {
require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &resp))
require.Len(t, resp.Hosts, 1)
assert.Equal(t, "node1", resp.Hosts[0].Hostname)
require.Len(t, resp.Hosts[0].Arrays, 1)
assert.Equal(t, 2, resp.Hosts[0].Arrays[0].RequiredDevices)
assert.Equal(t, 3, resp.Hosts[0].Arrays[0].TotalDevices)
assert.Equal(t, 1, resp.Hosts[0].Arrays[0].SpareDevices)
assert.Contains(t, result.Content[0].Text, `"required_devices":2`)
}
func TestExecuteGetHostCephDetails(t *testing.T) {
internal/ai/tools/tools_storage.go
+15 -14
View File
@@ -1328,20 +1328,21 @@ func (e *PulseToolExecutor) executeGetHostRAIDStatus(_ context.Context, args map
}
arrays = append(arrays, HostRAIDArraySummary{
Device: raid.Device,
Name: raid.Name,
Level: raid.Level,
State: raid.State,
TotalDevices: raid.TotalDevices,
ActiveDevices: raid.ActiveDevices,
WorkingDevices: raid.WorkingDevices,
FailedDevices: raid.FailedDevices,
SpareDevices: raid.SpareDevices,
UUID: raid.UUID,
RebuildPercent: raid.RebuildPercent,
RebuildSpeed: raid.RebuildSpeed,
Operation: raid.Operation,
Devices: devices,
Device: raid.Device,
Name: raid.Name,
Level: raid.Level,
State: raid.State,
RequiredDevices: raid.RequiredDevices,
TotalDevices: raid.TotalDevices,
ActiveDevices: raid.ActiveDevices,
WorkingDevices: raid.WorkingDevices,
FailedDevices: raid.FailedDevices,
SpareDevices: raid.SpareDevices,
UUID: raid.UUID,
RebuildPercent: raid.RebuildPercent,
RebuildSpeed: raid.RebuildSpeed,
Operation: raid.Operation,
Devices: devices,
})
}
internal/alerts/canonical_stateful_test.go
+55
View File
@@ -54,6 +54,61 @@ func TestCanonicalLifecycleCopiesValidatedAlertCorrelation(t *testing.T) {
}
}
func TestRAIDSpareCanonicalActivationAndRecovery(t *testing.T) {
m := newEventLogManager(t)
cfg := m.GetConfig()
cfg.TimeThresholds = map[string]int{}
m.UpdateConfig(cfg)
host := models.Host{ID: "raid-host", Hostname: "linux-raid", Status: "online", RAID: []models.HostRAIDArray{{
Device: "/dev/md1", Level: "raid5", State: "clean", TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1,
}}}
id := buildCanonicalStateID("agent:raid-host/raid:md1", "agent:raid-host/raid:md1-health")
m.CheckHost(host)
if testHasActiveAlert(t, m, id) || len(queryAlertEvents(t, m, eventlog.Filter{})) != 0 {
t.Fatal("legacy healthy spare report activated an incident")
}
// An unreplaced required member is critical, even when the attached spare
// makes the old total/active arithmetic look like a healthy spare tuple.
host.RAID[0].RequiredDevices = 4
host.RAID[0].TotalDevices = 4
host.RAID[0].ActiveDevices = 3
host.RAID[0].WorkingDevices = 4
m.CheckHost(host)
active := testRequireActiveAlert(t, m, id)
if active.Level != AlertLevelCritical || active.Metadata["raidRequiredDevices"] != 4 {
t.Fatalf("deficit alert lost required count or severity: %+v", active)
}
for _, operation := range []string{"check", "resync", "recovery"} {
host.RAID[0].Operation = operation
host.RAID[0].RebuildPercent = 25
m.CheckHost(host)
if got := testRequireActiveAlert(t, m, id); got.Level != AlertLevelCritical || !got.StartTime.Equal(active.StartTime) {
t.Fatalf("maintenance/recovery masked or reopened required-member deficit: %+v", got)
}
}
m.HandleHostTelemetryExpired(host)
if got := testRequireActiveAlert(t, m, id); got.Level != AlertLevelCritical {
t.Fatal("transient operation expiry removed a static member deficit")
}
host.RAID[0].TotalDevices = 5
host.RAID[0].ActiveDevices = 4
host.RAID[0].WorkingDevices = 5
host.RAID[0].Operation = ""
host.RAID[0].RebuildPercent = 0
m.CheckHost(host)
m.CheckHost(host)
if testHasActiveAlert(t, m, id) {
t.Fatal("full member recovery plus spare did not resolve the incident")
}
for kind, want := range map[string]int{eventlog.TypeFired: 1, eventlog.TypeResolved: 1} {
if got := len(queryAlertEvents(t, m, eventlog.Filter{Types: []string{kind}})); got != want {
t.Fatalf("canonical %s events=%d, want %d", kind, got, want)
}
}
}
func TestStatefulAlertReFireCooldown(t *testing.T) {
t.Run("re-fire within cooldown does not create duplicate history entry", func(t *testing.T) {
m := newTestManager(t)
internal/alerts/host.go
+3
View File
@@ -522,6 +522,9 @@ func (m *Manager) CheckHost(host models.Host) {
raidMetadata["raidDevice"] = array.Device
raidMetadata["raidLevel"] = array.Level
raidMetadata["raidState"] = array.State
if array.RequiredDevices > 0 {
raidMetadata["raidRequiredDevices"] = array.RequiredDevices
}
raidMetadata["raidTotalDevices"] = array.TotalDevices
raidMetadata["raidActiveDevices"] = array.ActiveDevices
raidMetadata["raidFailedDevices"] = array.FailedDevices
internal/hostagent/agent_metrics_test.go
+24 -8
View File
@@ -226,12 +226,13 @@ func TestBuildReport(t *testing.T) {
mc.raidArraysFn = func(ctx context.Context) ([]agentshost.RAIDArray, error) {
return []agentshost.RAIDArray{
{
Device: "/dev/md0",
Level: "raid10",
State: "active",
TotalDevices: 4,
ActiveDevices: 4,
WorkingDevices: 4,
Device: "/dev/md0",
Level: "raid10",
State: "active",
RequiredDevices: 4,
TotalDevices: 4,
ActiveDevices: 4,
WorkingDevices: 4,
Devices: []agentshost.RAIDDevice{
{Device: "/dev/sda1", State: "active sync", Slot: 0},
{Device: "/dev/sdb1", State: "active sync", Slot: 1},
@@ -254,7 +255,7 @@ func TestBuildReport(t *testing.T) {
if array.Device != "/dev/md0" || array.Level != "raid10" || array.State != "active" {
t.Fatalf("unexpected RAID array summary: %+v", array)
}
if array.TotalDevices != 4 || array.ActiveDevices != 4 || len(array.Devices) != 4 {
if array.RequiredDevices != 4 || array.TotalDevices != 4 || array.ActiveDevices != 4 || len(array.Devices) != 4 {
t.Fatalf("unexpected RAID array topology: %+v", array)
}
mc.raidArraysFn = nil
@@ -278,12 +279,27 @@ func TestBuildReport(t *testing.T) {
if array.Device != "/dev/md13" || array.State != "active" {
t.Fatalf("unexpected QNAP RAID array summary: %+v", array)
}
if array.TotalDevices != 2 || array.ActiveDevices != 2 || array.WorkingDevices != 2 || array.FailedDevices != 0 {
if array.RequiredDevices != 2 || array.TotalDevices != 2 || array.ActiveDevices != 2 || array.WorkingDevices != 2 || array.FailedDevices != 0 {
t.Fatalf("unexpected QNAP RAID array counts: %+v", array)
}
mc.raidArraysFn = nil
})
t.Run("RAID collection preserves explicit zero active bitmap", func(t *testing.T) {
mc.raidArraysFn = func(ctx context.Context) ([]agentshost.RAIDArray, error) {
return parseMDStatArrays("md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sdd1[3]\n 1024000 blocks super 1.2 [4/0] [____]\n"), nil
}
report, err := agent.buildReport(context.Background())
if err != nil || len(report.RAID) != 1 {
t.Fatalf("buildReport: raid=%+v err=%v", report.RAID, err)
}
array := report.RAID[0]
if array.RequiredDevices != 4 || array.ActiveDevices != 0 || array.FailedDevices != 4 {
t.Fatalf("explicit zero active members became fabricated healthy counts: %+v", array)
}
mc.raidArraysFn = nil
})
// Test case 4: Ceph collection
t.Run("Ceph collection", func(t *testing.T) {
mc.cephStatusFn = func(ctx context.Context) (*CephClusterStatus, error) {
internal/hostagent/mdadm.go
+20 -2
View File
@@ -213,6 +213,7 @@ func parseMDStatArraySection(deviceName string, section []string) agentshost.RAI
total, _ := strconv.Atoi(matches[1])
active, _ := strconv.Atoi(matches[2])
array.TotalDevices = total
array.RequiredDevices = total
array.ActiveDevices = active
}
@@ -246,7 +247,7 @@ func parseMDStatArraySection(deviceName string, section []string) agentshost.RAI
workingDevices++
}
}
if array.ActiveDevices == 0 {
if array.ActiveDevices == 0 && array.RequiredDevices == 0 {
array.ActiveDevices = workingDevices - spareDevices
}
if array.TotalDevices == 0 {
@@ -257,6 +258,7 @@ func parseMDStatArraySection(deviceName string, section []string) agentshost.RAI
// as [24/2] [UU______________________], while their active members use
// roles 24 and 25. Those unused bitmap positions are not failed disks.
array.TotalDevices = array.ActiveDevices
array.RequiredDevices = array.ActiveDevices
}
if array.TotalDevices > array.ActiveDevices && !strings.Contains(array.State, "degraded") {
array.State = appendMDStatState(array.State, "degraded")
@@ -344,6 +346,10 @@ func appendMDStatState(state, suffix string) string {
}
func mergeMDStatFallback(array, fallback agentshost.RAIDArray) agentshost.RAIDArray {
// A detail report with a configured requirement can legitimately have no
// active members. Do not turn that observed zero into kernel-fallback
// members from a different probe in the same collection cycle.
detailRequirementKnown := array.RequiredDevices > 0
if strings.TrimSpace(array.Device) == "" {
array.Device = fallback.Device
}
@@ -353,10 +359,13 @@ func mergeMDStatFallback(array, fallback agentshost.RAIDArray) agentshost.RAIDAr
if strings.TrimSpace(array.State) == "" {
array.State = fallback.State
}
if array.RequiredDevices == 0 {
array.RequiredDevices = fallback.RequiredDevices
}
if array.TotalDevices == 0 {
array.TotalDevices = fallback.TotalDevices
}
if array.ActiveDevices == 0 {
if array.ActiveDevices == 0 && !detailRequirementKnown {
array.ActiveDevices = fallback.ActiveDevices
}
if array.WorkingDevices == 0 {
@@ -482,6 +491,12 @@ func parseMdadmDetail(device, output string) (agentshost.RAIDArray, error) {
array.Level = strings.ToLower(value)
case "State":
array.State = strings.ToLower(value)
case "Raid Devices":
requiredDevices, err := parseIntField(device, key, value)
if err != nil {
return host.RAIDArray{}, err
}
array.RequiredDevices = requiredDevices
case "Total Devices":
totalDevices, err := parseIntField(device, key, value)
if err != nil {
@@ -550,6 +565,9 @@ func parseIntField(device, key, value string) (int, error) {
if err != nil {
return 0, fmt.Errorf("parse %s for %s from %q: %w", key, device, value, err)
}
if n < 0 {
return 0, fmt.Errorf("parse %s for %s: device count must not be negative", key, device)
}
return n, nil
}
internal/hostagent/mdadm_spare_test.go
+157
View File
@@ -0,0 +1,157 @@
//go:build !windows
package hostagent
import (
"context"
"encoding/json"
"fmt"
"strings"
"testing"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/storagehealth"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
)
// These are synthetic devices, retaining #2369's count tuple but no host
// identity. Both command and kernel reads are replaced; no array is touched.
const healthyMDStatWithSpare = `md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4](S)
1024000 blocks super 1.2 level 5, 512k chunk [4/4] [UUUU]
`
func mdadmSpareDetail(required string, total, active, working, failed, spare int, state string) string {
var detail strings.Builder
fmt.Fprintf(&detail, `Raid Level : raid5
Raid Devices : %s
Total Devices : %d
State : %s
Active Devices : %d
Working Devices : %d
Failed Devices : %d
Spare Devices : %d
Number Major Minor RaidDevice State
`, required, total, state, active, working, failed, spare)
for i := 0; i < active; i++ {
fmt.Fprintf(&detail, "%d 8 %d %d active sync /dev/sd%c1\n", i, i*16, i, 'a'+i)
}
if spare > 0 {
detail.WriteString("4 8 64 - spare /dev/sde1\n")
}
if failed > 0 {
detail.WriteString("5 8 80 - faulty /dev/sdf1\n")
}
return detail.String()
}
func TestIssue2369MdadmRequiredMemberCount(t *testing.T) {
withReadProcMDStat(t, func() ([]byte, error) { return nil, nil })
array, err := parseMdadmDetail("/dev/md1", mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean"))
if err != nil {
t.Fatal(err)
}
if array.RequiredDevices != 4 || array.TotalDevices != 5 || array.ActiveDevices != 4 || array.WorkingDevices != 5 || array.FailedDevices != 0 || array.SpareDevices != 1 || len(array.Devices) != 5 {
t.Fatalf("required members must remain separate from attached devices: %+v", array)
}
if array.Devices[4].State != "spare" {
t.Fatalf("spare topology was lost: %+v", array.Devices)
}
}
func TestRAIDCollectorReportHealthAndAlert(t *testing.T) {
degradedMDStat := `md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sde1[4](S)
1024000 blocks super 1.2 level 5, 512k chunk [4/3] [UUU_]
`
tests := []struct {
name string
mdstat string
detail string
required int
total int
active int
want storagehealth.RiskLevel
}{
{"mdadm clean four plus spare", healthyMDStatWithSpare, mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean"), 4, 5, 4, storagehealth.RiskHealthy},
{"mdadm missing member with spare", degradedMDStat, mdadmSpareDetail("4", 4, 3, 4, 0, 1, "clean"), 4, 4, 3, storagehealth.RiskCritical},
{"mdadm failed member with spare", degradedMDStat, mdadmSpareDetail("4", 5, 3, 4, 1, 1, "clean"), 4, 5, 3, storagehealth.RiskCritical},
{"known zero active detail cannot borrow kernel members", healthyMDStatWithSpare, mdadmSpareDetail("4", 1, 0, 1, 0, 1, "active"), 4, 1, 0, storagehealth.RiskCritical},
{"known zero active bitmap cannot borrow device tokens", "md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4](S)\n 1024000 blocks super 1.2 [4/0] [____]\n", "", 4, 4, 0, storagehealth.RiskCritical},
{"mdstat-only clean spare", healthyMDStatWithSpare, "", 4, 4, 4, storagehealth.RiskHealthy},
{"mdstat-only missing member with spare", degradedMDStat, "", 4, 4, 3, storagehealth.RiskCritical},
{"malformed detail retains kernel deficit", degradedMDStat, mdadmSpareDetail("invalid", 4, 3, 4, 0, 1, "clean"), 4, 4, 3, storagehealth.RiskCritical},
{"negative detail retains kernel deficit", degradedMDStat, mdadmSpareDetail("-1", 4, 3, 4, 0, 1, "clean"), 4, 4, 3, storagehealth.RiskCritical},
{"older detail gets kernel requirement", healthyMDStatWithSpare, strings.ReplaceAll(mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean"), "Raid Devices : 4\n", ""), 4, 5, 4, storagehealth.RiskHealthy},
{"scrub with spare", healthyMDStatWithSpare + " check = 25.0% (...) speed=50000K/sec\n", mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean"), 4, 5, 4, storagehealth.RiskHealthy},
{"resync with spare", healthyMDStatWithSpare + " resync = 25.0% (...) speed=50000K/sec\n", "", 4, 4, 4, storagehealth.RiskHealthy},
{"recovery with spare remains warning", healthyMDStatWithSpare + " recovery = 25.0% (...) speed=50000K/sec\n", mdadmSpareDetail("4", 5, 4, 5, 0, 1, "active"), 4, 5, 4, storagehealth.RiskWarning},
{"degraded scrub remains critical", degradedMDStat + " check = 25.0% (...) speed=50000K/sec\n", "", 4, 4, 3, storagehealth.RiskCritical},
{"QNAP sparse role normalization", "md13 : active raid1 sdb4[25] sda4[24]\n 458880 blocks super 1.0 [24/2] [UU______________________]\n", "", 2, 2, 2, storagehealth.RiskHealthy},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
withReadProcMDStat(t, func() ([]byte, error) { return []byte(tc.mdstat), nil })
withResolveMdadmBinary(t, func() (string, error) {
if tc.detail == "" {
return "", fmt.Errorf("mdadm unavailable")
}
return "/usr/sbin/mdadm", nil
})
withRunCommandOutput(t, func(_ context.Context, _ string, args ...string) ([]byte, error) {
if len(args) == 1 && args[0] == "--version" {
return []byte("mdadm"), nil
}
if len(args) != 2 || args[0] != "--detail" || args[1] != "/dev/md1" {
t.Fatalf("unexpected command arguments: %v", args)
}
return []byte(tc.detail), nil
})
arrays, err := CollectRAIDArrays(context.Background())
if err != nil || len(arrays) != 1 {
t.Fatalf("CollectRAIDArrays: arrays=%+v err=%v", arrays, err)
}
payload, err := json.Marshal(agentshost.Report{RAID: arrays})
if err != nil {
t.Fatal(err)
}
var report struct {
RAID []models.HostRAIDArray `json:"raid"`
}
if err := json.Unmarshal(payload, &report); err != nil {
t.Fatal(err)
}
array := report.RAID[0]
if array.RequiredDevices != tc.required || array.TotalDevices != tc.total || array.ActiveDevices != tc.active {
t.Fatalf("report counts=%+v, want required=%d total=%d active=%d", array, tc.required, tc.total, tc.active)
}
assessment := storagehealth.AssessHostRAIDArray(array)
if assessment.Level != tc.want {
t.Fatalf("collector/report/health assessment=%+v, want %s", assessment, tc.want)
}
manager := alerts.NewManagerWithDataDir(t.TempDir())
t.Cleanup(manager.Stop)
cfg := manager.GetConfig()
cfg.Enabled = true
cfg.ActivationState = alerts.ActivationActive
cfg.Schedule.QuietHours.Enabled = false
cfg.TimeThresholds = map[string]int{}
manager.UpdateConfig(cfg)
manager.CheckHost(models.Host{ID: "raid-fixture", Hostname: "linux-fixture", Status: "online", RAID: report.RAID})
var raidAlerts []alerts.Alert
for _, alert := range manager.GetActiveAlerts() {
if alert.Type == "raid" {
raidAlerts = append(raidAlerts, alert)
}
}
if tc.want == storagehealth.RiskHealthy {
if len(raidAlerts) != 0 {
t.Fatalf("healthy collector report activated RAID alert: %+v", raidAlerts)
}
} else {
if len(raidAlerts) != 1 || string(raidAlerts[0].Level) != string(tc.want) || raidAlerts[0].Metadata["raidRequiredDevices"] != tc.required {
t.Fatalf("RAID alert=%+v, want exactly one %s alert with required count", raidAlerts, tc.want)
}
}
})
}
}
internal/models/metrics_types_test.go
+21
View File
@@ -19,6 +19,27 @@ func TestMetricPoint_ZeroValue(t *testing.T) {
}
}
func TestHostRAIDRequiredDevicesWireContract(t *testing.T) {
for _, required := range []int{0, 4} {
array := HostRAIDArray{Device: "/dev/md1", RequiredDevices: required, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1}
payload, err := json.Marshal(array.NormalizeCollections())
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(payload), `"requiredDevices"`) != (required > 0) {
t.Fatalf("known/unknown required count not preserved: %s", payload)
}
var decoded HostRAIDArray
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatal(err)
}
cloned := cloneHostRAIDArrays([]HostRAIDArray{decoded})
if cloned[0].RequiredDevices != required || cloned[0].TotalDevices != 5 || cloned[0].SpareDevices != 1 || cloned[0].Devices == nil {
t.Fatalf("model wire/clone changed count meaning: %+v", cloned)
}
}
}
func TestAlertLastSeenWireContract(t *testing.T) {
withoutLastSeen, err := json.Marshal(Alert{ID: "alert-1"})
if err != nil {
internal/models/models.go
+15 -13
View File
@@ -702,19 +702,21 @@ type SMARTAttributes struct {
// HostRAIDArray represents an mdadm RAID array on a host.
type HostRAIDArray struct {
Device string `json:"device"`
Name string `json:"name,omitempty"`
Level string `json:"level"`
State string `json:"state"`
TotalDevices int `json:"totalDevices"`
ActiveDevices int `json:"activeDevices"`
WorkingDevices int `json:"workingDevices"`
FailedDevices int `json:"failedDevices"`
SpareDevices int `json:"spareDevices"`
UUID string `json:"uuid,omitempty"`
Devices []HostRAIDDevice `json:"devices"`
RebuildPercent float64 `json:"rebuildPercent"`
RebuildSpeed string `json:"rebuildSpeed,omitempty"`
Device string `json:"device"`
Name string `json:"name,omitempty"`
Level string `json:"level"`
State string `json:"state"`
// RequiredDevices excludes spares; zero denotes an older or unknown count.
RequiredDevices int `json:"requiredDevices,omitempty"`
TotalDevices int `json:"totalDevices"`
ActiveDevices int `json:"activeDevices"`
WorkingDevices int `json:"workingDevices"`
FailedDevices int `json:"failedDevices"`
SpareDevices int `json:"spareDevices"`
UUID string `json:"uuid,omitempty"`
Devices []HostRAIDDevice `json:"devices"`
RebuildPercent float64 `json:"rebuildPercent"`
RebuildSpeed string `json:"rebuildSpeed,omitempty"`
// Operation is the in-progress sync action from /proc/mdstat
// ("recovery", "resync", "check", or "reshape"); empty when idle.
Operation string `json:"operation,omitempty"`
internal/monitoring/monitor.go
+15 -14
View File
@@ -3960,20 +3960,21 @@ func hostRAIDFromReadStateView(raid []unifiedresources.HostRAIDMeta) []models.Ho
})
}
out = append(out, models.HostRAIDArray{
Device: entry.Device,
Name: entry.Name,
Level: entry.Level,
State: entry.State,
TotalDevices: entry.TotalDevices,
ActiveDevices: entry.ActiveDevices,
WorkingDevices: entry.WorkingDevices,
FailedDevices: entry.FailedDevices,
SpareDevices: entry.SpareDevices,
UUID: entry.UUID,
Devices: devices,
RebuildPercent: entry.RebuildPercent,
RebuildSpeed: entry.RebuildSpeed,
Operation: entry.Operation,
Device: entry.Device,
Name: entry.Name,
Level: entry.Level,
State: entry.State,
RequiredDevices: entry.RequiredDevices,
TotalDevices: entry.TotalDevices,
ActiveDevices: entry.ActiveDevices,
WorkingDevices: entry.WorkingDevices,
FailedDevices: entry.FailedDevices,
SpareDevices: entry.SpareDevices,
UUID: entry.UUID,
Devices: devices,
RebuildPercent: entry.RebuildPercent,
RebuildSpeed: entry.RebuildSpeed,
Operation: entry.Operation,
})
}
return out
internal/monitoring/monitor_agents.go
+15 -14
View File
@@ -3370,20 +3370,21 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config.
rebuildSpeed = ""
}
raid = append(raid, models.HostRAIDArray{
Device: array.Device,
Name: array.Name,
Level: array.Level,
State: array.State,
TotalDevices: array.TotalDevices,
ActiveDevices: array.ActiveDevices,
WorkingDevices: array.WorkingDevices,
FailedDevices: array.FailedDevices,
SpareDevices: array.SpareDevices,
UUID: array.UUID,
Devices: devices,
RebuildPercent: rebuildPercent,
RebuildSpeed: rebuildSpeed,
Operation: operation,
Device: array.Device,
Name: array.Name,
Level: array.Level,
State: array.State,
RequiredDevices: array.RequiredDevices,
TotalDevices: array.TotalDevices,
ActiveDevices: array.ActiveDevices,
WorkingDevices: array.WorkingDevices,
FailedDevices: array.FailedDevices,
SpareDevices: array.SpareDevices,
UUID: array.UUID,
Devices: devices,
RebuildPercent: rebuildPercent,
RebuildSpeed: rebuildSpeed,
Operation: operation,
})
}
internal/monitoring/monitor_host_agents_test.go
+76
View File
@@ -68,6 +68,82 @@ func TestNewMonitorRoutesStartupCustomSensorWarningBeforeStart(t *testing.T) {
}
}
func TestRAIDRequiredMembersReportAndReadState(t *testing.T) {
for _, tc := range []struct {
name string
required int
total int
active int
working int
want storagehealth.RiskLevel
}{
{"reported legacy clean spare", 0, 5, 4, 5, storagehealth.RiskHealthy},
{"new mdadm clean spare", 4, 5, 4, 5, storagehealth.RiskHealthy},
{"new mdstat clean spare", 4, 4, 4, 5, storagehealth.RiskHealthy},
{"count deficit despite clean state and spare", 4, 4, 3, 4, storagehealth.RiskCritical},
} {
t.Run(tc.name, func(t *testing.T) {
manager := alerts.NewManagerWithDataDir(t.TempDir())
t.Cleanup(manager.Stop)
cfg := manager.GetConfig()
cfg.Enabled = true
cfg.ActivationState = alerts.ActivationActive
cfg.TimeThresholds = map[string]int{}
manager.UpdateConfig(cfg)
monitor := &Monitor{
state: models.NewState(), alertManager: manager,
hostTokenBindings: make(map[string]string), config: &config.Config{}, rateTracker: NewRateTracker(),
}
report := agentshost.Report{
Agent: agentshost.AgentInfo{ID: "raid-agent", Version: "test", IntervalSeconds: 30},
Host: agentshost.HostInfo{ID: "raid-host", Hostname: "linux-raid", Platform: "linux"},
Timestamp: time.Now().UTC(),
RAID: []agentshost.RAIDArray{{Device: "/dev/md1", Level: "raid5", State: "clean", RequiredDevices: tc.required, TotalDevices: tc.total, ActiveDevices: tc.active, WorkingDevices: tc.working, SpareDevices: 1}},
}
payload, err := json.Marshal(report)
if err != nil {
t.Fatal(err)
}
var decoded agentshost.Report
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatal(err)
}
host, err := monitor.ApplyHostReport(decoded, nil)
if err != nil {
t.Fatal(err)
}
stored := monitor.state.GetSnapshot().Hosts
if len(stored) != 1 || len(stored[0].RAID) != 1 {
t.Fatalf("stored host RAID=%+v", stored)
}
record := unifiedresources.HostIngestRecord(stored[0])
view := unifiedresources.NewHostView(&record.Resource)
readback := hostRAIDFromReadStateView(view.RAID())
for _, array := range []models.HostRAIDArray{host.RAID[0], stored[0].RAID[0], readback[0]} {
if array.RequiredDevices != tc.required || array.TotalDevices != tc.total || array.ActiveDevices != tc.active || array.SpareDevices != 1 {
t.Fatalf("ingest/state/canonical readback changed count meaning: %+v", array)
}
if got := storagehealth.AssessHostRAIDArray(array); got.Level != tc.want {
t.Fatalf("assessment after round trip=%+v, want %s", got, tc.want)
}
}
var raidAlerts []alerts.Alert
for _, alert := range manager.GetActiveAlerts() {
if alert.Type == "raid" {
raidAlerts = append(raidAlerts, alert)
}
}
if tc.want == storagehealth.RiskHealthy {
if len(raidAlerts) != 0 || record.Resource.Agent.RAID[0].Risk != nil {
t.Fatalf("healthy spare produced alert or canonical risk: alerts=%+v risk=%+v", raidAlerts, record.Resource.Agent.RAID[0].Risk)
}
} else if len(raidAlerts) != 1 || raidAlerts[0].Level != alerts.AlertLevelCritical || record.Resource.Agent.RAID[0].Risk == nil || record.Resource.Agent.RAID[0].Risk.Level != storagehealth.RiskCritical {
t.Fatalf("real deficit lost alert or canonical risk: alerts=%+v risk=%+v", raidAlerts, record.Resource.Agent.RAID[0].Risk)
}
})
}
}
func TestHostZFSPoolsFromAgentDisksPreservesDatasetFacts(t *testing.T) {
got := hostZFSPoolsFromAgentDisks([]agentshost.Disk{
{Device: "/", Type: "ext4"},
internal/storagehealth/topology.go
+17 -1
View File
@@ -30,7 +30,23 @@ func AssessHostRAIDArray(array models.HostRAIDArray) Assessment {
(operation == "" && (strings.Contains(stateLower, "recover") ||
(array.RebuildPercent > 0 && !strings.Contains(stateLower, "clean") && !strings.Contains(stateLower, "check")))))
if strings.Contains(stateLower, "degraded") || array.FailedDevices > 0 || (array.TotalDevices > 0 && array.ActiveDevices > 0 && array.ActiveDevices < array.TotalDevices) {
requiredDevices := array.RequiredDevices
if requiredDevices <= 0 {
requiredDevices = array.TotalDevices
// Older mdadm reports include spares in TotalDevices, unlike mdstat's
// required/active bitmap. Accept only a corroborated healthy tuple;
// subtracting spares from every legacy total would hide missing members.
if (stateLower == "clean" || stateLower == "active") && array.FailedDevices == 0 &&
array.SpareDevices > 0 && array.ActiveDevices > 0 &&
array.TotalDevices > array.ActiveDevices &&
array.TotalDevices-array.ActiveDevices == array.SpareDevices &&
array.WorkingDevices == array.TotalDevices {
requiredDevices = array.ActiveDevices
}
}
memberDeficit := requiredDevices > 0 && array.ActiveDevices >= 0 && array.ActiveDevices < requiredDevices &&
(array.ActiveDevices > 0 || array.RequiredDevices > 0)
if strings.Contains(stateLower, "degraded") || array.FailedDevices > 0 || memberDeficit {
summary := fmt.Sprintf("RAID array %s is degraded", array.Device)
if array.FailedDevices > 0 {
summary = fmt.Sprintf("RAID array %s has %d failed device(s)", array.Device, array.FailedDevices)
internal/storagehealth/topology_test.go
+52
View File
@@ -23,6 +23,58 @@ func TestAssessHostRAIDArrayDegraded(t *testing.T) {
}
}
func TestIssue2369CleanArrayWithSpare(t *testing.T) {
// The count tuple supplied in #2369, without the new field: servers must
// also correct the alert while an older collector is still installed.
got := AssessHostRAIDArray(models.HostRAIDArray{
Device: "/dev/md1", Level: "raid5", State: "clean",
TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1,
})
if got.Level != RiskHealthy || len(got.Reasons) != 0 {
t.Fatalf("healthy four-member array plus spare: level=%s reasons=%+v", got.Level, got.Reasons)
}
}
func TestRAIDRequiredMemberHealth(t *testing.T) {
tests := []struct {
name string
array models.HostRAIDArray
want RiskLevel
degraded bool
rebuilding bool
}{
{"mdadm healthy spare", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskHealthy, false, false},
{"legacy active healthy spare", models.HostRAIDArray{State: "active", TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskHealthy, false, false},
{"legacy mdstat healthy spare", models.HostRAIDArray{State: "active", TotalDevices: 4, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskHealthy, false, false},
{"missing member not replaced by spare", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 4, SpareDevices: 1}, RiskCritical, true, false},
{"mdstat deficit without failed counter", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 4, SpareDevices: 1}, RiskCritical, true, false},
{"legacy mdstat deficit", models.HostRAIDArray{State: "active, degraded", TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 4, FailedDevices: 1, SpareDevices: 1}, RiskCritical, true, false},
{"legacy unreconciled count deficit", models.HostRAIDArray{State: "clean", TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 3, SpareDevices: 1}, RiskCritical, true, false},
{"legacy negative total cannot corroborate spare", models.HostRAIDArray{State: "clean", TotalDevices: -1, ActiveDevices: 4, WorkingDevices: -1, SpareDevices: 1}, RiskHealthy, false, false},
{"legacy missing state is not healthy corroboration", models.HostRAIDArray{TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskCritical, true, false},
{"failed member despite spare", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 6, ActiveDevices: 4, WorkingDevices: 5, FailedDevices: 1, SpareDevices: 1}, RiskCritical, true, false},
{"explicit degraded state wins", models.HostRAIDArray{State: "clean, degraded", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskCritical, true, false},
{"zero active known members", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 1, WorkingDevices: 1, SpareDevices: 1}, RiskCritical, true, false},
{"legacy unknown active count", models.HostRAIDArray{State: "active", TotalDevices: 4}, RiskHealthy, false, false},
{"healthy spare under scrub", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "check", RebuildPercent: 25}, RiskHealthy, false, false},
{"healthy spare under resync", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "resync", RebuildPercent: 25}, RiskHealthy, false, false},
{"spare under recovery stays warning", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "recovery", RebuildPercent: 25}, RiskWarning, false, true},
{"spare under reshape stays warning", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "reshape", RebuildPercent: 25}, RiskWarning, false, true},
{"degraded recovery stays critical", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, SpareDevices: 1, Operation: "recovery", RebuildPercent: 25}, RiskCritical, true, true},
{"scrub cannot mask deficit", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, SpareDevices: 1, Operation: "check", RebuildPercent: 25}, RiskCritical, true, false},
{"offline array stays critical", models.HostRAIDArray{State: "offline", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1}, RiskCritical, false, false},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
tc.array.Device = "/dev/md1"
got := AssessHostRAIDArray(tc.array)
if got.Level != tc.want || hasReasonCode(got, "raid_degraded") != tc.degraded || hasReasonCode(got, "raid_rebuilding") != tc.rebuilding {
t.Fatalf("assessment=%+v, want level=%s degraded=%v rebuilding=%v", got, tc.want, tc.degraded, tc.rebuilding)
}
})
}
}
func TestAssessHostRAIDArrayRebuilding(t *testing.T) {
assessment := AssessHostRAIDArray(models.HostRAIDArray{
Device: "/dev/md3",
internal/unifiedresources/adapters.go
+16 -15
View File
@@ -360,21 +360,22 @@ func resourceFromHost(host models.Host) (Resource, ResourceIdentity) {
}
assessment := storagehealth.AssessHostRAIDArray(r)
raid[i] = HostRAIDMeta{
Device: r.Device,
Name: r.Name,
Level: r.Level,
State: r.State,
TotalDevices: r.TotalDevices,
ActiveDevices: r.ActiveDevices,
WorkingDevices: r.WorkingDevices,
FailedDevices: r.FailedDevices,
SpareDevices: r.SpareDevices,
UUID: r.UUID,
Devices: devices,
RebuildPercent: r.RebuildPercent,
RebuildSpeed: r.RebuildSpeed,
Operation: r.Operation,
Risk: storageRiskFromAssessment(assessment),
Device: r.Device,
Name: r.Name,
Level: r.Level,
State: r.State,
RequiredDevices: r.RequiredDevices,
TotalDevices: r.TotalDevices,
ActiveDevices: r.ActiveDevices,
WorkingDevices: r.WorkingDevices,
FailedDevices: r.FailedDevices,
SpareDevices: r.SpareDevices,
UUID: r.UUID,
Devices: devices,
RebuildPercent: r.RebuildPercent,
RebuildSpeed: r.RebuildSpeed,
Operation: r.Operation,
Risk: storageRiskFromAssessment(assessment),
}
if !isInternalHostRAIDDevice(r.Device) {
storageAssessments = append(storageAssessments, assessment)
internal/unifiedresources/adapters_test.go
+24
View File
@@ -41,6 +41,30 @@ func TestResourceFromProxmoxNodeIncludesTemperature(t *testing.T) {
}
}
func TestHostRAIDRequiredMembersCanonicalRisk(t *testing.T) {
for _, active := range []int{4, 3} {
host := models.Host{ID: "raid-source", Hostname: "linux", Status: "online", RAID: []models.HostRAIDArray{{
Device: "/dev/md2", Level: "raid5", State: "clean", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: active, WorkingDevices: 5, SpareDevices: 1,
}}}
record := HostIngestRecord(host)
arrays := NewHostView(&record.Resource).RAID()
if len(arrays) != 1 || arrays[0].RequiredDevices != 4 || arrays[0].TotalDevices != 5 || arrays[0].SpareDevices != 1 {
t.Fatalf("canonical RAID count provenance=%+v", arrays)
}
if active == 4 {
if arrays[0].Risk != nil || record.Resource.Agent.StorageRisk != nil || record.Resource.Agent.ProtectionReduced {
t.Fatalf("healthy spare reduced protection: %+v", record.Resource.Agent)
}
} else if arrays[0].Risk == nil || arrays[0].Risk.Level != storagehealth.RiskCritical || !record.Resource.Agent.ProtectionReduced {
t.Fatalf("member deficit with spare lost protection risk: %+v", record.Resource.Agent)
}
arrays[0].RequiredDevices = 1
if record.Resource.Agent.RAID[0].RequiredDevices != 4 {
t.Fatal("read view mutated canonical required count")
}
}
}
func TestResourceFromProxmoxNodeIncludesNetworkInterfaces(t *testing.T) {
node := models.Node{
ID: "mock-cluster-pve1", Name: "pve1", Status: "online",
internal/unifiedresources/clone_test.go
+8 -1
View File
@@ -613,7 +613,7 @@ func TestCloneAgentData_DeepIsolation(t *testing.T) {
LoadAverage: []float64{0.5, 1.0, 1.5},
DiskExclude: []string{"/dev/sda"},
RAID: []HostRAIDMeta{
{Device: "/dev/md0", Level: "raid1", Risk: &StorageRisk{Level: storagehealth.RiskHealthy}},
{Device: "/dev/md0", Level: "raid1", RequiredDevices: 2, TotalDevices: 3, SpareDevices: 1, Risk: &StorageRisk{Level: storagehealth.RiskHealthy}},
},
PackageUpdates: &AgentPackageUpdateMeta{Packages: []AgentPackageUpdate{{Name: "openssl"}}},
StorageCleanup: &AgentStorageCleanupMeta{Provider: "apt-package-cache", ReclaimableBytes: 512},
@@ -634,6 +634,13 @@ func TestCloneAgentData_DeepIsolation(t *testing.T) {
if original.RAID[0].Risk.Level == storagehealth.RiskCritical {
t.Error("mutating cloned RAID risk should not affect original")
}
if cloned.RAID[0].RequiredDevices != 2 || cloned.RAID[0].TotalDevices != 3 || cloned.RAID[0].SpareDevices != 1 {
t.Fatalf("clone lost RAID count provenance: %+v", cloned.RAID[0])
}
cloned.RAID[0].RequiredDevices = 1
if original.RAID[0].RequiredDevices != 2 {
t.Fatal("mutating cloned RAID required count changed the source")
}
cloned.PackageUpdates.Packages[0].Name = "mutated"
cloned.StorageCleanup.Provider = "mutated"
if original.PackageUpdates.Packages[0].Name != "openssl" || original.StorageCleanup.Provider != "apt-package-cache" {
internal/unifiedresources/types.go
+16 -15
View File
@@ -707,21 +707,22 @@ type HostRAIDDeviceMeta struct {
// HostRAIDMeta describes a RAID array.
type HostRAIDMeta struct {
Device string `json:"device"`
Name string `json:"name,omitempty"`
Level string `json:"level"`
State string `json:"state"`
TotalDevices int `json:"totalDevices"`
ActiveDevices int `json:"activeDevices"`
WorkingDevices int `json:"workingDevices"`
FailedDevices int `json:"failedDevices"`
SpareDevices int `json:"spareDevices"`
UUID string `json:"uuid,omitempty"`
Devices []HostRAIDDeviceMeta `json:"devices,omitempty"`
RebuildPercent float64 `json:"rebuildPercent,omitempty"`
RebuildSpeed string `json:"rebuildSpeed,omitempty"`
Operation string `json:"operation,omitempty"`
Risk *StorageRisk `json:"risk,omitempty"`
Device string `json:"device"`
Name string `json:"name,omitempty"`
Level string `json:"level"`
State string `json:"state"`
RequiredDevices int `json:"requiredDevices,omitempty"` // configured members, excluding spares; zero means unknown
TotalDevices int `json:"totalDevices"`
ActiveDevices int `json:"activeDevices"`
WorkingDevices int `json:"workingDevices"`
FailedDevices int `json:"failedDevices"`
SpareDevices int `json:"spareDevices"`
UUID string `json:"uuid,omitempty"`
Devices []HostRAIDDeviceMeta `json:"devices,omitempty"`
RebuildPercent float64 `json:"rebuildPercent,omitempty"`
RebuildSpeed string `json:"rebuildSpeed,omitempty"`
Operation string `json:"operation,omitempty"`
Risk *StorageRisk `json:"risk,omitempty"`
}
// HostUnraidDiskMeta describes a disk's role and state inside an Unraid array.
pkg/agents/host/report.go
+16 -13
View File
@@ -462,19 +462,22 @@ type SMARTAttributes struct {
// RAIDArray represents an mdadm RAID array.
type RAIDArray struct {
Device string `json:"device"` // e.g., /dev/md0
Name string `json:"name,omitempty"` // Array name if set
Level string `json:"level"` // RAID level: raid0, raid1, raid5, raid6, raid10
State string `json:"state"` // clean, active, degraded, recovering, resyncing, etc.
TotalDevices int `json:"totalDevices"` // Total number of devices in array
ActiveDevices int `json:"activeDevices"` // Number of active devices
WorkingDevices int `json:"workingDevices"` // Number of working devices
FailedDevices int `json:"failedDevices"` // Number of failed devices
SpareDevices int `json:"spareDevices"` // Number of spare devices
UUID string `json:"uuid,omitempty"` // Array UUID
Devices []RAIDDevice `json:"devices"` // Individual devices in array
RebuildPercent float64 `json:"rebuildPercent"` // Rebuild/resync progress (0-100)
RebuildSpeed string `json:"rebuildSpeed,omitempty"` // Rebuild speed (e.g., "50000K/sec")
Device string `json:"device"` // e.g., /dev/md0
Name string `json:"name,omitempty"` // Array name if set
Level string `json:"level"` // RAID level: raid0, raid1, raid5, raid6, raid10
State string `json:"state"` // clean, active, degraded, recovering, resyncing, etc.
// RequiredDevices is the configured member count (mdadm Raid Devices or
// mdstat's required/active bitmap), excluding spares. Zero means unknown.
RequiredDevices int `json:"requiredDevices,omitempty"`
TotalDevices int `json:"totalDevices"` // mdadm attached devices (including spares); mdstat bitmap width
ActiveDevices int `json:"activeDevices"` // Number of active devices
WorkingDevices int `json:"workingDevices"` // Number of working devices
FailedDevices int `json:"failedDevices"` // Number of failed devices
SpareDevices int `json:"spareDevices"` // Number of spare devices
UUID string `json:"uuid,omitempty"` // Array UUID
Devices []RAIDDevice `json:"devices"` // Individual devices in array
RebuildPercent float64 `json:"rebuildPercent"` // Rebuild/resync progress (0-100)
RebuildSpeed string `json:"rebuildSpeed,omitempty"` // Rebuild speed (e.g., "50000K/sec")
// Operation reports the in-progress sync action from /proc/mdstat:
// "recovery", "resync", "check", or "reshape". Empty when idle.
Operation string `json:"operation,omitempty"`
pkg/agents/host/report_test.go
+26
View File
@@ -18,6 +18,32 @@ func TestLegacyReportJSONDefaultsOperationReceiptProtocolUnsupported(t *testing.
}
}
func TestRAIDRequiredDevicesWireCompatibility(t *testing.T) {
for _, required := range []int{0, 4} {
report := Report{RAID: []RAIDArray{{Device: "/dev/md1", RequiredDevices: required, TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}}}
data, err := json.Marshal(report)
if err != nil {
t.Fatal(err)
}
var fields struct {
RAID []map[string]any `json:"raid"`
}
if err := json.Unmarshal(data, &fields); err != nil {
t.Fatal(err)
}
if value, present := fields.RAID[0]["requiredDevices"]; present != (required != 0) || (present && value != float64(required)) {
t.Fatalf("required member presence/value = %v/%v in %s", present, value, data)
}
var got Report
if err := json.Unmarshal(data, &got); err != nil {
t.Fatal(err)
}
if got.RAID[0].RequiredDevices != required || got.RAID[0].TotalDevices != 5 || got.RAID[0].SpareDevices != 1 {
t.Fatalf("RAID count round trip = %+v", got.RAID)
}
}
}
func TestReportSequenceIDRoundTrip(t *testing.T) {
value := FormatReportSequenceID("0123456789abcdef", 42)
if value != "v1:0123456789abcdef:42" {