diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index f8b2d1cb37..7c7ba657d3 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -15,6 +15,23 @@ ## Purpose +### MD RAID required members and spares — issue #2369 + +Host RAID reports carry optional `requiredDevices`: the configured member count +from mdadm's `Raid Devices` or mdstat's required/active bitmap, excluding spares. +Zero/omission means unknown. `totalDevices` retains its existing source-native +meaning: attached devices including spares for mdadm, bitmap width for mdstat. +The existing QNAP sparse-role normalization applies to both bitmap counts. +Invalid or negative detail counts fall back to the kernel report, and missing +detail requirements may be enriched from that report. This is read-only +collection, not permission to change array membership or perform recovery. +Collector-to-wire/health/alert tests cover healthy spares, missing and failed +members, mdstat-only and malformed-detail fallback, and maintenance/recovery. +Report serialization and `buildReport` must preserve both counts independently. +An explicit zero active count under a known requirement is retained; neither +device-token inference nor a different kernel fallback probe may fill it in. +The collector/report controls exercise both zero-active boundaries. + ### Continuity-aware broadcast read ownership Broadcast state takes its single resource list from the current continuity-aware diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md index c493f84f22..70d798f6ac 100644 --- a/docs/release-control/v6/internal/subsystems/ai-runtime.md +++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md @@ -25,6 +25,16 @@ that same result. Successful reads retain their content and execution provenance ## Purpose +### RAID configured-member read evidence — issue #2369 + +The existing host RAID status tool preserves optional `required_devices` from +the canonical host view alongside source-native total, active, working, failed +and spare counts. Omission means the configured requirement is unknown, not +zero members. A total that includes spares is not evidence of a member deficit; +the shared health owner remains authoritative for risk. Tool serialization tests +check the separate counts. This read-only projection creates no feature, route, +array-repair grant, backup verdict or recovery confirmation. + ### Ollama Basic Auth continuity Patrol's runtime provider factory must carry the persisted Ollama username and password into streaming requests without trimming password bytes. diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md index 2a6710c950..bc7c4bc4a5 100644 --- a/docs/release-control/v6/internal/subsystems/alerts.md +++ b/docs/release-control/v6/internal/subsystems/alerts.md @@ -15,6 +15,21 @@ ## Purpose +### RAID spare-count lifecycle — issue #2369 + +Canonical host RAID alerts use the shared required-member health assessment, +not attached-device arithmetic. Known configured counts appear as +`raidRequiredDevices` metadata alongside unchanged total/active/failed/spare +observations. A healthy spare creates no incident; an actual missing member or +failed device still activates critical health. Scrub/resync/recovery and transient +operation expiry cannot mask a static deficit. Restoring all required members +with a spare resolves the existing incident through the same canonical state +identity, once, without a duplicate fire or resolve. +`TestRAIDSpareCanonicalActivationAndRecovery` checks event-ledger activation, +continuity and resolution; collector and ingestion tests check both health and +canonical alert severity. Routine scrub/resync remains silent, and genuine +recovery/reshape keeps its warning when no critical deficit exists. + ### Retained occurrence re-fires preserve incident truth A reducer re-fire inside the retention window keeps its original occurrence diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index 95abd650ac..f8151068f2 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -17,6 +17,23 @@ ## Purpose +### RAID count provenance across ingestion — issue #2369 + +Agent ingestion, host snapshots and canonical read-state projection retain +`requiredDevices` separately from source-native `totalDevices` and spare counts. +Health compares active members with a known configured requirement, including +explicit zero-active deficits; a spare cannot fill a missing active role. +Collector fallback preserves an explicit zero active count under a known +requirement rather than substituting members from another probe. +Legacy reports retain conservative total-based detection except for an exact +clean/active, zero-failure tuple where total equals working and active plus +spares. That corroborated mdadm tuple is healthy; arbitrary subtraction from +mdstat totals is forbidden. Missing or unreconciled evidence does not gain +that exception. Explicit degradation/failures remain critical and scrub/resync +never hide a member deficit. `TestRAIDRequiredMembersReportAndReadState` pins +the wire/ingestion/snapshot/readback and canonical alert/risk paths; storage +health tests pin legacy compatibility and reconstruction/maintenance boundaries. + ### Single-pass live broadcast projection — issue #2199 A frontend broadcast lists the continuity-aware read store once, coalesces host diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index 7816344107..987a1e1836 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -21,6 +21,19 @@ ## Purpose +### RAID required-member count evidence — issue #2369 + +Canonical host RAID metadata carries optional `requiredDevices`, excluding +spares, separately from source-native attached/bitmap totals. Array protection +risk follows the shared required-member assessment rather than assuming that +every attached spare is a missing active member. Missing configured members, +failed devices and explicit degraded state remain critical; maintenance and +spare presence cannot erase those risks. A healthy spare is not recovery proof, +backup coverage or storage mutation authority. Collector, ingestion and +canonical event-ledger tests distinguish that correction from genuine member +recovery, which resolves the existing health incident only after the reported +active set again meets its requirement. + Incident history retains canonical storage-related risk, source identity and separate observation/occurrence timestamps. A recorded alert closure or operator note is not a restore result. Canonical read failure remains unavailable and diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md index 04d35708f2..70ad3c7455 100644 --- a/docs/release-control/v6/internal/subsystems/unified-resources.md +++ b/docs/release-control/v6/internal/subsystems/unified-resources.md @@ -23,6 +23,18 @@ and sort the complete canonical change table while startup and ingestion wait. ## Purpose +### Canonical RAID configured-member evidence — issue #2369 + +Host RAID metadata and read views retain optional `requiredDevices` (configured +members excluding spares) without rewriting source-native `totalDevices`. +Clones preserve both values and isolate caller mutation. The shared storage +health assessment owns per-array risk and aggregate protection posture: a full +member set with a hot spare must not become critical, while a configured member +deficit remains critical even if active plus spares equals the attached count. +Adapter, clone and monitoring readback tests pin these boundaries. Count +evidence does not establish backup coverage, native array recovery or any +storage mutation authority; the field adds no resource or action identity. + ### Bounded confidence/priority work in broadcast grouping — issue #2199 The general identity matcher retains every confidence, reason, ordering and diff --git a/internal/ai/tools/data_types.go b/internal/ai/tools/data_types.go index 31f14762b0..db3360534a 100644 --- a/internal/ai/tools/data_types.go +++ b/internal/ai/tools/data_types.go @@ -2092,20 +2092,21 @@ func (s HostRAIDSummary) NormalizeCollections() HostRAIDSummary { // HostRAIDArraySummary summarizes a RAID array type HostRAIDArraySummary struct { - Device string `json:"device"` - Name string `json:"name,omitempty"` - Level string `json:"level"` // raid0, raid1, raid5, etc. - State string `json:"state"` // clean, degraded, rebuilding - TotalDevices int `json:"total_devices"` - ActiveDevices int `json:"active_devices"` - WorkingDevices int `json:"working_devices"` - FailedDevices int `json:"failed_devices"` - SpareDevices int `json:"spare_devices"` - UUID string `json:"uuid,omitempty"` - RebuildPercent float64 `json:"rebuild_percent,omitempty"` - RebuildSpeed string `json:"rebuild_speed,omitempty"` - Operation string `json:"operation,omitempty"` - Devices []HostRAIDDeviceSummary `json:"devices"` + Device string `json:"device"` + Name string `json:"name,omitempty"` + Level string `json:"level"` // raid0, raid1, raid5, etc. + State string `json:"state"` // clean, degraded, rebuilding + RequiredDevices int `json:"required_devices,omitempty"` // configured members excluding spares; zero means unknown + TotalDevices int `json:"total_devices"` + ActiveDevices int `json:"active_devices"` + WorkingDevices int `json:"working_devices"` + FailedDevices int `json:"failed_devices"` + SpareDevices int `json:"spare_devices"` + UUID string `json:"uuid,omitempty"` + RebuildPercent float64 `json:"rebuild_percent,omitempty"` + RebuildSpeed string `json:"rebuild_speed,omitempty"` + Operation string `json:"operation,omitempty"` + Devices []HostRAIDDeviceSummary `json:"devices"` } func (s HostRAIDArraySummary) NormalizeCollections() HostRAIDArraySummary { diff --git a/internal/ai/tools/infrastructure_low_coverage_test.go b/internal/ai/tools/infrastructure_low_coverage_test.go index 7b5e4c85d4..736eb58d16 100644 --- a/internal/ai/tools/infrastructure_low_coverage_test.go +++ b/internal/ai/tools/infrastructure_low_coverage_test.go @@ -1013,12 +1013,14 @@ func TestExecuteGetHostRAIDStatus(t *testing.T) { nil, []unifiedresources.HostRAIDMeta{ { - Device: "/dev/md0", - Level: "raid1", - State: "clean", - TotalDevices: 2, - ActiveDevices: 2, - WorkingDevices: 2, + Device: "/dev/md0", + Level: "raid1", + State: "clean", + RequiredDevices: 2, + TotalDevices: 3, + ActiveDevices: 2, + WorkingDevices: 3, + SpareDevices: 1, Devices: []unifiedresources.HostRAIDDeviceMeta{ {Device: "/dev/sda", State: "active", Slot: 0}, }, @@ -1042,6 +1044,11 @@ func TestExecuteGetHostRAIDStatus(t *testing.T) { require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &resp)) require.Len(t, resp.Hosts, 1) assert.Equal(t, "node1", resp.Hosts[0].Hostname) + require.Len(t, resp.Hosts[0].Arrays, 1) + assert.Equal(t, 2, resp.Hosts[0].Arrays[0].RequiredDevices) + assert.Equal(t, 3, resp.Hosts[0].Arrays[0].TotalDevices) + assert.Equal(t, 1, resp.Hosts[0].Arrays[0].SpareDevices) + assert.Contains(t, result.Content[0].Text, `"required_devices":2`) } func TestExecuteGetHostCephDetails(t *testing.T) { diff --git a/internal/ai/tools/tools_storage.go b/internal/ai/tools/tools_storage.go index 12a70eafc2..988b5d71bf 100644 --- a/internal/ai/tools/tools_storage.go +++ b/internal/ai/tools/tools_storage.go @@ -1328,20 +1328,21 @@ func (e *PulseToolExecutor) executeGetHostRAIDStatus(_ context.Context, args map } arrays = append(arrays, HostRAIDArraySummary{ - Device: raid.Device, - Name: raid.Name, - Level: raid.Level, - State: raid.State, - TotalDevices: raid.TotalDevices, - ActiveDevices: raid.ActiveDevices, - WorkingDevices: raid.WorkingDevices, - FailedDevices: raid.FailedDevices, - SpareDevices: raid.SpareDevices, - UUID: raid.UUID, - RebuildPercent: raid.RebuildPercent, - RebuildSpeed: raid.RebuildSpeed, - Operation: raid.Operation, - Devices: devices, + Device: raid.Device, + Name: raid.Name, + Level: raid.Level, + State: raid.State, + RequiredDevices: raid.RequiredDevices, + TotalDevices: raid.TotalDevices, + ActiveDevices: raid.ActiveDevices, + WorkingDevices: raid.WorkingDevices, + FailedDevices: raid.FailedDevices, + SpareDevices: raid.SpareDevices, + UUID: raid.UUID, + RebuildPercent: raid.RebuildPercent, + RebuildSpeed: raid.RebuildSpeed, + Operation: raid.Operation, + Devices: devices, }) } diff --git a/internal/alerts/canonical_stateful_test.go b/internal/alerts/canonical_stateful_test.go index 8f7f41e9aa..d76ced7073 100644 --- a/internal/alerts/canonical_stateful_test.go +++ b/internal/alerts/canonical_stateful_test.go @@ -54,6 +54,61 @@ func TestCanonicalLifecycleCopiesValidatedAlertCorrelation(t *testing.T) { } } +func TestRAIDSpareCanonicalActivationAndRecovery(t *testing.T) { + m := newEventLogManager(t) + cfg := m.GetConfig() + cfg.TimeThresholds = map[string]int{} + m.UpdateConfig(cfg) + host := models.Host{ID: "raid-host", Hostname: "linux-raid", Status: "online", RAID: []models.HostRAIDArray{{ + Device: "/dev/md1", Level: "raid5", State: "clean", TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1, + }}} + id := buildCanonicalStateID("agent:raid-host/raid:md1", "agent:raid-host/raid:md1-health") + m.CheckHost(host) + if testHasActiveAlert(t, m, id) || len(queryAlertEvents(t, m, eventlog.Filter{})) != 0 { + t.Fatal("legacy healthy spare report activated an incident") + } + + // An unreplaced required member is critical, even when the attached spare + // makes the old total/active arithmetic look like a healthy spare tuple. + host.RAID[0].RequiredDevices = 4 + host.RAID[0].TotalDevices = 4 + host.RAID[0].ActiveDevices = 3 + host.RAID[0].WorkingDevices = 4 + m.CheckHost(host) + active := testRequireActiveAlert(t, m, id) + if active.Level != AlertLevelCritical || active.Metadata["raidRequiredDevices"] != 4 { + t.Fatalf("deficit alert lost required count or severity: %+v", active) + } + for _, operation := range []string{"check", "resync", "recovery"} { + host.RAID[0].Operation = operation + host.RAID[0].RebuildPercent = 25 + m.CheckHost(host) + if got := testRequireActiveAlert(t, m, id); got.Level != AlertLevelCritical || !got.StartTime.Equal(active.StartTime) { + t.Fatalf("maintenance/recovery masked or reopened required-member deficit: %+v", got) + } + } + m.HandleHostTelemetryExpired(host) + if got := testRequireActiveAlert(t, m, id); got.Level != AlertLevelCritical { + t.Fatal("transient operation expiry removed a static member deficit") + } + + host.RAID[0].TotalDevices = 5 + host.RAID[0].ActiveDevices = 4 + host.RAID[0].WorkingDevices = 5 + host.RAID[0].Operation = "" + host.RAID[0].RebuildPercent = 0 + m.CheckHost(host) + m.CheckHost(host) + if testHasActiveAlert(t, m, id) { + t.Fatal("full member recovery plus spare did not resolve the incident") + } + for kind, want := range map[string]int{eventlog.TypeFired: 1, eventlog.TypeResolved: 1} { + if got := len(queryAlertEvents(t, m, eventlog.Filter{Types: []string{kind}})); got != want { + t.Fatalf("canonical %s events=%d, want %d", kind, got, want) + } + } +} + func TestStatefulAlertReFireCooldown(t *testing.T) { t.Run("re-fire within cooldown does not create duplicate history entry", func(t *testing.T) { m := newTestManager(t) diff --git a/internal/alerts/host.go b/internal/alerts/host.go index 30a5bbdd7f..cc2969046f 100644 --- a/internal/alerts/host.go +++ b/internal/alerts/host.go @@ -522,6 +522,9 @@ func (m *Manager) CheckHost(host models.Host) { raidMetadata["raidDevice"] = array.Device raidMetadata["raidLevel"] = array.Level raidMetadata["raidState"] = array.State + if array.RequiredDevices > 0 { + raidMetadata["raidRequiredDevices"] = array.RequiredDevices + } raidMetadata["raidTotalDevices"] = array.TotalDevices raidMetadata["raidActiveDevices"] = array.ActiveDevices raidMetadata["raidFailedDevices"] = array.FailedDevices diff --git a/internal/hostagent/agent_metrics_test.go b/internal/hostagent/agent_metrics_test.go index 9c70c0ba11..c457a1701e 100644 --- a/internal/hostagent/agent_metrics_test.go +++ b/internal/hostagent/agent_metrics_test.go @@ -226,12 +226,13 @@ func TestBuildReport(t *testing.T) { mc.raidArraysFn = func(ctx context.Context) ([]agentshost.RAIDArray, error) { return []agentshost.RAIDArray{ { - Device: "/dev/md0", - Level: "raid10", - State: "active", - TotalDevices: 4, - ActiveDevices: 4, - WorkingDevices: 4, + Device: "/dev/md0", + Level: "raid10", + State: "active", + RequiredDevices: 4, + TotalDevices: 4, + ActiveDevices: 4, + WorkingDevices: 4, Devices: []agentshost.RAIDDevice{ {Device: "/dev/sda1", State: "active sync", Slot: 0}, {Device: "/dev/sdb1", State: "active sync", Slot: 1}, @@ -254,7 +255,7 @@ func TestBuildReport(t *testing.T) { if array.Device != "/dev/md0" || array.Level != "raid10" || array.State != "active" { t.Fatalf("unexpected RAID array summary: %+v", array) } - if array.TotalDevices != 4 || array.ActiveDevices != 4 || len(array.Devices) != 4 { + if array.RequiredDevices != 4 || array.TotalDevices != 4 || array.ActiveDevices != 4 || len(array.Devices) != 4 { t.Fatalf("unexpected RAID array topology: %+v", array) } mc.raidArraysFn = nil @@ -278,12 +279,27 @@ func TestBuildReport(t *testing.T) { if array.Device != "/dev/md13" || array.State != "active" { t.Fatalf("unexpected QNAP RAID array summary: %+v", array) } - if array.TotalDevices != 2 || array.ActiveDevices != 2 || array.WorkingDevices != 2 || array.FailedDevices != 0 { + if array.RequiredDevices != 2 || array.TotalDevices != 2 || array.ActiveDevices != 2 || array.WorkingDevices != 2 || array.FailedDevices != 0 { t.Fatalf("unexpected QNAP RAID array counts: %+v", array) } mc.raidArraysFn = nil }) + t.Run("RAID collection preserves explicit zero active bitmap", func(t *testing.T) { + mc.raidArraysFn = func(ctx context.Context) ([]agentshost.RAIDArray, error) { + return parseMDStatArrays("md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sdd1[3]\n 1024000 blocks super 1.2 [4/0] [____]\n"), nil + } + report, err := agent.buildReport(context.Background()) + if err != nil || len(report.RAID) != 1 { + t.Fatalf("buildReport: raid=%+v err=%v", report.RAID, err) + } + array := report.RAID[0] + if array.RequiredDevices != 4 || array.ActiveDevices != 0 || array.FailedDevices != 4 { + t.Fatalf("explicit zero active members became fabricated healthy counts: %+v", array) + } + mc.raidArraysFn = nil + }) + // Test case 4: Ceph collection t.Run("Ceph collection", func(t *testing.T) { mc.cephStatusFn = func(ctx context.Context) (*CephClusterStatus, error) { diff --git a/internal/hostagent/mdadm.go b/internal/hostagent/mdadm.go index 67a3993845..7d1bc1154a 100644 --- a/internal/hostagent/mdadm.go +++ b/internal/hostagent/mdadm.go @@ -213,6 +213,7 @@ func parseMDStatArraySection(deviceName string, section []string) agentshost.RAI total, _ := strconv.Atoi(matches[1]) active, _ := strconv.Atoi(matches[2]) array.TotalDevices = total + array.RequiredDevices = total array.ActiveDevices = active } @@ -246,7 +247,7 @@ func parseMDStatArraySection(deviceName string, section []string) agentshost.RAI workingDevices++ } } - if array.ActiveDevices == 0 { + if array.ActiveDevices == 0 && array.RequiredDevices == 0 { array.ActiveDevices = workingDevices - spareDevices } if array.TotalDevices == 0 { @@ -257,6 +258,7 @@ func parseMDStatArraySection(deviceName string, section []string) agentshost.RAI // as [24/2] [UU______________________], while their active members use // roles 24 and 25. Those unused bitmap positions are not failed disks. array.TotalDevices = array.ActiveDevices + array.RequiredDevices = array.ActiveDevices } if array.TotalDevices > array.ActiveDevices && !strings.Contains(array.State, "degraded") { array.State = appendMDStatState(array.State, "degraded") @@ -344,6 +346,10 @@ func appendMDStatState(state, suffix string) string { } func mergeMDStatFallback(array, fallback agentshost.RAIDArray) agentshost.RAIDArray { + // A detail report with a configured requirement can legitimately have no + // active members. Do not turn that observed zero into kernel-fallback + // members from a different probe in the same collection cycle. + detailRequirementKnown := array.RequiredDevices > 0 if strings.TrimSpace(array.Device) == "" { array.Device = fallback.Device } @@ -353,10 +359,13 @@ func mergeMDStatFallback(array, fallback agentshost.RAIDArray) agentshost.RAIDAr if strings.TrimSpace(array.State) == "" { array.State = fallback.State } + if array.RequiredDevices == 0 { + array.RequiredDevices = fallback.RequiredDevices + } if array.TotalDevices == 0 { array.TotalDevices = fallback.TotalDevices } - if array.ActiveDevices == 0 { + if array.ActiveDevices == 0 && !detailRequirementKnown { array.ActiveDevices = fallback.ActiveDevices } if array.WorkingDevices == 0 { @@ -482,6 +491,12 @@ func parseMdadmDetail(device, output string) (agentshost.RAIDArray, error) { array.Level = strings.ToLower(value) case "State": array.State = strings.ToLower(value) + case "Raid Devices": + requiredDevices, err := parseIntField(device, key, value) + if err != nil { + return host.RAIDArray{}, err + } + array.RequiredDevices = requiredDevices case "Total Devices": totalDevices, err := parseIntField(device, key, value) if err != nil { @@ -550,6 +565,9 @@ func parseIntField(device, key, value string) (int, error) { if err != nil { return 0, fmt.Errorf("parse %s for %s from %q: %w", key, device, value, err) } + if n < 0 { + return 0, fmt.Errorf("parse %s for %s: device count must not be negative", key, device) + } return n, nil } diff --git a/internal/hostagent/mdadm_spare_test.go b/internal/hostagent/mdadm_spare_test.go new file mode 100644 index 0000000000..8f580cbc0d --- /dev/null +++ b/internal/hostagent/mdadm_spare_test.go @@ -0,0 +1,157 @@ +//go:build !windows + +package hostagent + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "testing" + + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" + "github.com/rcourtman/pulse-go-rewrite/internal/models" + "github.com/rcourtman/pulse-go-rewrite/internal/storagehealth" + agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host" +) + +// These are synthetic devices, retaining #2369's count tuple but no host +// identity. Both command and kernel reads are replaced; no array is touched. +const healthyMDStatWithSpare = `md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4](S) + 1024000 blocks super 1.2 level 5, 512k chunk [4/4] [UUUU] +` + +func mdadmSpareDetail(required string, total, active, working, failed, spare int, state string) string { + var detail strings.Builder + fmt.Fprintf(&detail, `Raid Level : raid5 +Raid Devices : %s +Total Devices : %d +State : %s +Active Devices : %d +Working Devices : %d +Failed Devices : %d +Spare Devices : %d +Number Major Minor RaidDevice State +`, required, total, state, active, working, failed, spare) + for i := 0; i < active; i++ { + fmt.Fprintf(&detail, "%d 8 %d %d active sync /dev/sd%c1\n", i, i*16, i, 'a'+i) + } + if spare > 0 { + detail.WriteString("4 8 64 - spare /dev/sde1\n") + } + if failed > 0 { + detail.WriteString("5 8 80 - faulty /dev/sdf1\n") + } + return detail.String() +} + +func TestIssue2369MdadmRequiredMemberCount(t *testing.T) { + withReadProcMDStat(t, func() ([]byte, error) { return nil, nil }) + array, err := parseMdadmDetail("/dev/md1", mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean")) + if err != nil { + t.Fatal(err) + } + if array.RequiredDevices != 4 || array.TotalDevices != 5 || array.ActiveDevices != 4 || array.WorkingDevices != 5 || array.FailedDevices != 0 || array.SpareDevices != 1 || len(array.Devices) != 5 { + t.Fatalf("required members must remain separate from attached devices: %+v", array) + } + if array.Devices[4].State != "spare" { + t.Fatalf("spare topology was lost: %+v", array.Devices) + } +} + +func TestRAIDCollectorReportHealthAndAlert(t *testing.T) { + degradedMDStat := `md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sde1[4](S) + 1024000 blocks super 1.2 level 5, 512k chunk [4/3] [UUU_] +` + tests := []struct { + name string + mdstat string + detail string + required int + total int + active int + want storagehealth.RiskLevel + }{ + {"mdadm clean four plus spare", healthyMDStatWithSpare, mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean"), 4, 5, 4, storagehealth.RiskHealthy}, + {"mdadm missing member with spare", degradedMDStat, mdadmSpareDetail("4", 4, 3, 4, 0, 1, "clean"), 4, 4, 3, storagehealth.RiskCritical}, + {"mdadm failed member with spare", degradedMDStat, mdadmSpareDetail("4", 5, 3, 4, 1, 1, "clean"), 4, 5, 3, storagehealth.RiskCritical}, + {"known zero active detail cannot borrow kernel members", healthyMDStatWithSpare, mdadmSpareDetail("4", 1, 0, 1, 0, 1, "active"), 4, 1, 0, storagehealth.RiskCritical}, + {"known zero active bitmap cannot borrow device tokens", "md1 : active raid5 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4](S)\n 1024000 blocks super 1.2 [4/0] [____]\n", "", 4, 4, 0, storagehealth.RiskCritical}, + {"mdstat-only clean spare", healthyMDStatWithSpare, "", 4, 4, 4, storagehealth.RiskHealthy}, + {"mdstat-only missing member with spare", degradedMDStat, "", 4, 4, 3, storagehealth.RiskCritical}, + {"malformed detail retains kernel deficit", degradedMDStat, mdadmSpareDetail("invalid", 4, 3, 4, 0, 1, "clean"), 4, 4, 3, storagehealth.RiskCritical}, + {"negative detail retains kernel deficit", degradedMDStat, mdadmSpareDetail("-1", 4, 3, 4, 0, 1, "clean"), 4, 4, 3, storagehealth.RiskCritical}, + {"older detail gets kernel requirement", healthyMDStatWithSpare, strings.ReplaceAll(mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean"), "Raid Devices : 4\n", ""), 4, 5, 4, storagehealth.RiskHealthy}, + {"scrub with spare", healthyMDStatWithSpare + " check = 25.0% (...) speed=50000K/sec\n", mdadmSpareDetail("4", 5, 4, 5, 0, 1, "clean"), 4, 5, 4, storagehealth.RiskHealthy}, + {"resync with spare", healthyMDStatWithSpare + " resync = 25.0% (...) speed=50000K/sec\n", "", 4, 4, 4, storagehealth.RiskHealthy}, + {"recovery with spare remains warning", healthyMDStatWithSpare + " recovery = 25.0% (...) speed=50000K/sec\n", mdadmSpareDetail("4", 5, 4, 5, 0, 1, "active"), 4, 5, 4, storagehealth.RiskWarning}, + {"degraded scrub remains critical", degradedMDStat + " check = 25.0% (...) speed=50000K/sec\n", "", 4, 4, 3, storagehealth.RiskCritical}, + {"QNAP sparse role normalization", "md13 : active raid1 sdb4[25] sda4[24]\n 458880 blocks super 1.0 [24/2] [UU______________________]\n", "", 2, 2, 2, storagehealth.RiskHealthy}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + withReadProcMDStat(t, func() ([]byte, error) { return []byte(tc.mdstat), nil }) + withResolveMdadmBinary(t, func() (string, error) { + if tc.detail == "" { + return "", fmt.Errorf("mdadm unavailable") + } + return "/usr/sbin/mdadm", nil + }) + withRunCommandOutput(t, func(_ context.Context, _ string, args ...string) ([]byte, error) { + if len(args) == 1 && args[0] == "--version" { + return []byte("mdadm"), nil + } + if len(args) != 2 || args[0] != "--detail" || args[1] != "/dev/md1" { + t.Fatalf("unexpected command arguments: %v", args) + } + return []byte(tc.detail), nil + }) + arrays, err := CollectRAIDArrays(context.Background()) + if err != nil || len(arrays) != 1 { + t.Fatalf("CollectRAIDArrays: arrays=%+v err=%v", arrays, err) + } + payload, err := json.Marshal(agentshost.Report{RAID: arrays}) + if err != nil { + t.Fatal(err) + } + var report struct { + RAID []models.HostRAIDArray `json:"raid"` + } + if err := json.Unmarshal(payload, &report); err != nil { + t.Fatal(err) + } + array := report.RAID[0] + if array.RequiredDevices != tc.required || array.TotalDevices != tc.total || array.ActiveDevices != tc.active { + t.Fatalf("report counts=%+v, want required=%d total=%d active=%d", array, tc.required, tc.total, tc.active) + } + assessment := storagehealth.AssessHostRAIDArray(array) + if assessment.Level != tc.want { + t.Fatalf("collector/report/health assessment=%+v, want %s", assessment, tc.want) + } + manager := alerts.NewManagerWithDataDir(t.TempDir()) + t.Cleanup(manager.Stop) + cfg := manager.GetConfig() + cfg.Enabled = true + cfg.ActivationState = alerts.ActivationActive + cfg.Schedule.QuietHours.Enabled = false + cfg.TimeThresholds = map[string]int{} + manager.UpdateConfig(cfg) + manager.CheckHost(models.Host{ID: "raid-fixture", Hostname: "linux-fixture", Status: "online", RAID: report.RAID}) + var raidAlerts []alerts.Alert + for _, alert := range manager.GetActiveAlerts() { + if alert.Type == "raid" { + raidAlerts = append(raidAlerts, alert) + } + } + if tc.want == storagehealth.RiskHealthy { + if len(raidAlerts) != 0 { + t.Fatalf("healthy collector report activated RAID alert: %+v", raidAlerts) + } + } else { + if len(raidAlerts) != 1 || string(raidAlerts[0].Level) != string(tc.want) || raidAlerts[0].Metadata["raidRequiredDevices"] != tc.required { + t.Fatalf("RAID alert=%+v, want exactly one %s alert with required count", raidAlerts, tc.want) + } + } + }) + } +} diff --git a/internal/models/metrics_types_test.go b/internal/models/metrics_types_test.go index 7dc45cf5b8..64b5755a09 100644 --- a/internal/models/metrics_types_test.go +++ b/internal/models/metrics_types_test.go @@ -19,6 +19,27 @@ func TestMetricPoint_ZeroValue(t *testing.T) { } } +func TestHostRAIDRequiredDevicesWireContract(t *testing.T) { + for _, required := range []int{0, 4} { + array := HostRAIDArray{Device: "/dev/md1", RequiredDevices: required, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1} + payload, err := json.Marshal(array.NormalizeCollections()) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(payload), `"requiredDevices"`) != (required > 0) { + t.Fatalf("known/unknown required count not preserved: %s", payload) + } + var decoded HostRAIDArray + if err := json.Unmarshal(payload, &decoded); err != nil { + t.Fatal(err) + } + cloned := cloneHostRAIDArrays([]HostRAIDArray{decoded}) + if cloned[0].RequiredDevices != required || cloned[0].TotalDevices != 5 || cloned[0].SpareDevices != 1 || cloned[0].Devices == nil { + t.Fatalf("model wire/clone changed count meaning: %+v", cloned) + } + } +} + func TestAlertLastSeenWireContract(t *testing.T) { withoutLastSeen, err := json.Marshal(Alert{ID: "alert-1"}) if err != nil { diff --git a/internal/models/models.go b/internal/models/models.go index 46d9e0e728..08530261fb 100644 --- a/internal/models/models.go +++ b/internal/models/models.go @@ -702,19 +702,21 @@ type SMARTAttributes struct { // HostRAIDArray represents an mdadm RAID array on a host. type HostRAIDArray struct { - Device string `json:"device"` - Name string `json:"name,omitempty"` - Level string `json:"level"` - State string `json:"state"` - TotalDevices int `json:"totalDevices"` - ActiveDevices int `json:"activeDevices"` - WorkingDevices int `json:"workingDevices"` - FailedDevices int `json:"failedDevices"` - SpareDevices int `json:"spareDevices"` - UUID string `json:"uuid,omitempty"` - Devices []HostRAIDDevice `json:"devices"` - RebuildPercent float64 `json:"rebuildPercent"` - RebuildSpeed string `json:"rebuildSpeed,omitempty"` + Device string `json:"device"` + Name string `json:"name,omitempty"` + Level string `json:"level"` + State string `json:"state"` + // RequiredDevices excludes spares; zero denotes an older or unknown count. + RequiredDevices int `json:"requiredDevices,omitempty"` + TotalDevices int `json:"totalDevices"` + ActiveDevices int `json:"activeDevices"` + WorkingDevices int `json:"workingDevices"` + FailedDevices int `json:"failedDevices"` + SpareDevices int `json:"spareDevices"` + UUID string `json:"uuid,omitempty"` + Devices []HostRAIDDevice `json:"devices"` + RebuildPercent float64 `json:"rebuildPercent"` + RebuildSpeed string `json:"rebuildSpeed,omitempty"` // Operation is the in-progress sync action from /proc/mdstat // ("recovery", "resync", "check", or "reshape"); empty when idle. Operation string `json:"operation,omitempty"` diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go index 685015a54c..ce0c51b77f 100644 --- a/internal/monitoring/monitor.go +++ b/internal/monitoring/monitor.go @@ -3960,20 +3960,21 @@ func hostRAIDFromReadStateView(raid []unifiedresources.HostRAIDMeta) []models.Ho }) } out = append(out, models.HostRAIDArray{ - Device: entry.Device, - Name: entry.Name, - Level: entry.Level, - State: entry.State, - TotalDevices: entry.TotalDevices, - ActiveDevices: entry.ActiveDevices, - WorkingDevices: entry.WorkingDevices, - FailedDevices: entry.FailedDevices, - SpareDevices: entry.SpareDevices, - UUID: entry.UUID, - Devices: devices, - RebuildPercent: entry.RebuildPercent, - RebuildSpeed: entry.RebuildSpeed, - Operation: entry.Operation, + Device: entry.Device, + Name: entry.Name, + Level: entry.Level, + State: entry.State, + RequiredDevices: entry.RequiredDevices, + TotalDevices: entry.TotalDevices, + ActiveDevices: entry.ActiveDevices, + WorkingDevices: entry.WorkingDevices, + FailedDevices: entry.FailedDevices, + SpareDevices: entry.SpareDevices, + UUID: entry.UUID, + Devices: devices, + RebuildPercent: entry.RebuildPercent, + RebuildSpeed: entry.RebuildSpeed, + Operation: entry.Operation, }) } return out diff --git a/internal/monitoring/monitor_agents.go b/internal/monitoring/monitor_agents.go index 461f6b1d22..2a1d74ee88 100644 --- a/internal/monitoring/monitor_agents.go +++ b/internal/monitoring/monitor_agents.go @@ -3370,20 +3370,21 @@ func (m *Monitor) ApplyHostReport(report agentshost.Report, tokenRecord *config. rebuildSpeed = "" } raid = append(raid, models.HostRAIDArray{ - Device: array.Device, - Name: array.Name, - Level: array.Level, - State: array.State, - TotalDevices: array.TotalDevices, - ActiveDevices: array.ActiveDevices, - WorkingDevices: array.WorkingDevices, - FailedDevices: array.FailedDevices, - SpareDevices: array.SpareDevices, - UUID: array.UUID, - Devices: devices, - RebuildPercent: rebuildPercent, - RebuildSpeed: rebuildSpeed, - Operation: operation, + Device: array.Device, + Name: array.Name, + Level: array.Level, + State: array.State, + RequiredDevices: array.RequiredDevices, + TotalDevices: array.TotalDevices, + ActiveDevices: array.ActiveDevices, + WorkingDevices: array.WorkingDevices, + FailedDevices: array.FailedDevices, + SpareDevices: array.SpareDevices, + UUID: array.UUID, + Devices: devices, + RebuildPercent: rebuildPercent, + RebuildSpeed: rebuildSpeed, + Operation: operation, }) } diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.go index 264f4b9c4b..97e335410a 100644 --- a/internal/monitoring/monitor_host_agents_test.go +++ b/internal/monitoring/monitor_host_agents_test.go @@ -68,6 +68,82 @@ func TestNewMonitorRoutesStartupCustomSensorWarningBeforeStart(t *testing.T) { } } +func TestRAIDRequiredMembersReportAndReadState(t *testing.T) { + for _, tc := range []struct { + name string + required int + total int + active int + working int + want storagehealth.RiskLevel + }{ + {"reported legacy clean spare", 0, 5, 4, 5, storagehealth.RiskHealthy}, + {"new mdadm clean spare", 4, 5, 4, 5, storagehealth.RiskHealthy}, + {"new mdstat clean spare", 4, 4, 4, 5, storagehealth.RiskHealthy}, + {"count deficit despite clean state and spare", 4, 4, 3, 4, storagehealth.RiskCritical}, + } { + t.Run(tc.name, func(t *testing.T) { + manager := alerts.NewManagerWithDataDir(t.TempDir()) + t.Cleanup(manager.Stop) + cfg := manager.GetConfig() + cfg.Enabled = true + cfg.ActivationState = alerts.ActivationActive + cfg.TimeThresholds = map[string]int{} + manager.UpdateConfig(cfg) + monitor := &Monitor{ + state: models.NewState(), alertManager: manager, + hostTokenBindings: make(map[string]string), config: &config.Config{}, rateTracker: NewRateTracker(), + } + report := agentshost.Report{ + Agent: agentshost.AgentInfo{ID: "raid-agent", Version: "test", IntervalSeconds: 30}, + Host: agentshost.HostInfo{ID: "raid-host", Hostname: "linux-raid", Platform: "linux"}, + Timestamp: time.Now().UTC(), + RAID: []agentshost.RAIDArray{{Device: "/dev/md1", Level: "raid5", State: "clean", RequiredDevices: tc.required, TotalDevices: tc.total, ActiveDevices: tc.active, WorkingDevices: tc.working, SpareDevices: 1}}, + } + payload, err := json.Marshal(report) + if err != nil { + t.Fatal(err) + } + var decoded agentshost.Report + if err := json.Unmarshal(payload, &decoded); err != nil { + t.Fatal(err) + } + host, err := monitor.ApplyHostReport(decoded, nil) + if err != nil { + t.Fatal(err) + } + stored := monitor.state.GetSnapshot().Hosts + if len(stored) != 1 || len(stored[0].RAID) != 1 { + t.Fatalf("stored host RAID=%+v", stored) + } + record := unifiedresources.HostIngestRecord(stored[0]) + view := unifiedresources.NewHostView(&record.Resource) + readback := hostRAIDFromReadStateView(view.RAID()) + for _, array := range []models.HostRAIDArray{host.RAID[0], stored[0].RAID[0], readback[0]} { + if array.RequiredDevices != tc.required || array.TotalDevices != tc.total || array.ActiveDevices != tc.active || array.SpareDevices != 1 { + t.Fatalf("ingest/state/canonical readback changed count meaning: %+v", array) + } + if got := storagehealth.AssessHostRAIDArray(array); got.Level != tc.want { + t.Fatalf("assessment after round trip=%+v, want %s", got, tc.want) + } + } + var raidAlerts []alerts.Alert + for _, alert := range manager.GetActiveAlerts() { + if alert.Type == "raid" { + raidAlerts = append(raidAlerts, alert) + } + } + if tc.want == storagehealth.RiskHealthy { + if len(raidAlerts) != 0 || record.Resource.Agent.RAID[0].Risk != nil { + t.Fatalf("healthy spare produced alert or canonical risk: alerts=%+v risk=%+v", raidAlerts, record.Resource.Agent.RAID[0].Risk) + } + } else if len(raidAlerts) != 1 || raidAlerts[0].Level != alerts.AlertLevelCritical || record.Resource.Agent.RAID[0].Risk == nil || record.Resource.Agent.RAID[0].Risk.Level != storagehealth.RiskCritical { + t.Fatalf("real deficit lost alert or canonical risk: alerts=%+v risk=%+v", raidAlerts, record.Resource.Agent.RAID[0].Risk) + } + }) + } +} + func TestHostZFSPoolsFromAgentDisksPreservesDatasetFacts(t *testing.T) { got := hostZFSPoolsFromAgentDisks([]agentshost.Disk{ {Device: "/", Type: "ext4"}, diff --git a/internal/storagehealth/topology.go b/internal/storagehealth/topology.go index d18f5a0b83..86a23e827f 100644 --- a/internal/storagehealth/topology.go +++ b/internal/storagehealth/topology.go @@ -30,7 +30,23 @@ func AssessHostRAIDArray(array models.HostRAIDArray) Assessment { (operation == "" && (strings.Contains(stateLower, "recover") || (array.RebuildPercent > 0 && !strings.Contains(stateLower, "clean") && !strings.Contains(stateLower, "check"))))) - if strings.Contains(stateLower, "degraded") || array.FailedDevices > 0 || (array.TotalDevices > 0 && array.ActiveDevices > 0 && array.ActiveDevices < array.TotalDevices) { + requiredDevices := array.RequiredDevices + if requiredDevices <= 0 { + requiredDevices = array.TotalDevices + // Older mdadm reports include spares in TotalDevices, unlike mdstat's + // required/active bitmap. Accept only a corroborated healthy tuple; + // subtracting spares from every legacy total would hide missing members. + if (stateLower == "clean" || stateLower == "active") && array.FailedDevices == 0 && + array.SpareDevices > 0 && array.ActiveDevices > 0 && + array.TotalDevices > array.ActiveDevices && + array.TotalDevices-array.ActiveDevices == array.SpareDevices && + array.WorkingDevices == array.TotalDevices { + requiredDevices = array.ActiveDevices + } + } + memberDeficit := requiredDevices > 0 && array.ActiveDevices >= 0 && array.ActiveDevices < requiredDevices && + (array.ActiveDevices > 0 || array.RequiredDevices > 0) + if strings.Contains(stateLower, "degraded") || array.FailedDevices > 0 || memberDeficit { summary := fmt.Sprintf("RAID array %s is degraded", array.Device) if array.FailedDevices > 0 { summary = fmt.Sprintf("RAID array %s has %d failed device(s)", array.Device, array.FailedDevices) diff --git a/internal/storagehealth/topology_test.go b/internal/storagehealth/topology_test.go index 01ea69283a..1a28d60fa7 100644 --- a/internal/storagehealth/topology_test.go +++ b/internal/storagehealth/topology_test.go @@ -23,6 +23,58 @@ func TestAssessHostRAIDArrayDegraded(t *testing.T) { } } +func TestIssue2369CleanArrayWithSpare(t *testing.T) { + // The count tuple supplied in #2369, without the new field: servers must + // also correct the alert while an older collector is still installed. + got := AssessHostRAIDArray(models.HostRAIDArray{ + Device: "/dev/md1", Level: "raid5", State: "clean", + TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1, + }) + if got.Level != RiskHealthy || len(got.Reasons) != 0 { + t.Fatalf("healthy four-member array plus spare: level=%s reasons=%+v", got.Level, got.Reasons) + } +} + +func TestRAIDRequiredMemberHealth(t *testing.T) { + tests := []struct { + name string + array models.HostRAIDArray + want RiskLevel + degraded bool + rebuilding bool + }{ + {"mdadm healthy spare", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskHealthy, false, false}, + {"legacy active healthy spare", models.HostRAIDArray{State: "active", TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskHealthy, false, false}, + {"legacy mdstat healthy spare", models.HostRAIDArray{State: "active", TotalDevices: 4, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskHealthy, false, false}, + {"missing member not replaced by spare", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 4, SpareDevices: 1}, RiskCritical, true, false}, + {"mdstat deficit without failed counter", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 4, SpareDevices: 1}, RiskCritical, true, false}, + {"legacy mdstat deficit", models.HostRAIDArray{State: "active, degraded", TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 4, FailedDevices: 1, SpareDevices: 1}, RiskCritical, true, false}, + {"legacy unreconciled count deficit", models.HostRAIDArray{State: "clean", TotalDevices: 4, ActiveDevices: 3, WorkingDevices: 3, SpareDevices: 1}, RiskCritical, true, false}, + {"legacy negative total cannot corroborate spare", models.HostRAIDArray{State: "clean", TotalDevices: -1, ActiveDevices: 4, WorkingDevices: -1, SpareDevices: 1}, RiskHealthy, false, false}, + {"legacy missing state is not healthy corroboration", models.HostRAIDArray{TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskCritical, true, false}, + {"failed member despite spare", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 6, ActiveDevices: 4, WorkingDevices: 5, FailedDevices: 1, SpareDevices: 1}, RiskCritical, true, false}, + {"explicit degraded state wins", models.HostRAIDArray{State: "clean, degraded", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}, RiskCritical, true, false}, + {"zero active known members", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 1, WorkingDevices: 1, SpareDevices: 1}, RiskCritical, true, false}, + {"legacy unknown active count", models.HostRAIDArray{State: "active", TotalDevices: 4}, RiskHealthy, false, false}, + {"healthy spare under scrub", models.HostRAIDArray{State: "clean", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "check", RebuildPercent: 25}, RiskHealthy, false, false}, + {"healthy spare under resync", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "resync", RebuildPercent: 25}, RiskHealthy, false, false}, + {"spare under recovery stays warning", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "recovery", RebuildPercent: 25}, RiskWarning, false, true}, + {"spare under reshape stays warning", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1, Operation: "reshape", RebuildPercent: 25}, RiskWarning, false, true}, + {"degraded recovery stays critical", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, SpareDevices: 1, Operation: "recovery", RebuildPercent: 25}, RiskCritical, true, true}, + {"scrub cannot mask deficit", models.HostRAIDArray{State: "active", RequiredDevices: 4, TotalDevices: 4, ActiveDevices: 3, SpareDevices: 1, Operation: "check", RebuildPercent: 25}, RiskCritical, true, false}, + {"offline array stays critical", models.HostRAIDArray{State: "offline", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: 4, SpareDevices: 1}, RiskCritical, false, false}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + tc.array.Device = "/dev/md1" + got := AssessHostRAIDArray(tc.array) + if got.Level != tc.want || hasReasonCode(got, "raid_degraded") != tc.degraded || hasReasonCode(got, "raid_rebuilding") != tc.rebuilding { + t.Fatalf("assessment=%+v, want level=%s degraded=%v rebuilding=%v", got, tc.want, tc.degraded, tc.rebuilding) + } + }) + } +} + func TestAssessHostRAIDArrayRebuilding(t *testing.T) { assessment := AssessHostRAIDArray(models.HostRAIDArray{ Device: "/dev/md3", diff --git a/internal/unifiedresources/adapters.go b/internal/unifiedresources/adapters.go index b3e9096cc6..9d360e2ece 100644 --- a/internal/unifiedresources/adapters.go +++ b/internal/unifiedresources/adapters.go @@ -360,21 +360,22 @@ func resourceFromHost(host models.Host) (Resource, ResourceIdentity) { } assessment := storagehealth.AssessHostRAIDArray(r) raid[i] = HostRAIDMeta{ - Device: r.Device, - Name: r.Name, - Level: r.Level, - State: r.State, - TotalDevices: r.TotalDevices, - ActiveDevices: r.ActiveDevices, - WorkingDevices: r.WorkingDevices, - FailedDevices: r.FailedDevices, - SpareDevices: r.SpareDevices, - UUID: r.UUID, - Devices: devices, - RebuildPercent: r.RebuildPercent, - RebuildSpeed: r.RebuildSpeed, - Operation: r.Operation, - Risk: storageRiskFromAssessment(assessment), + Device: r.Device, + Name: r.Name, + Level: r.Level, + State: r.State, + RequiredDevices: r.RequiredDevices, + TotalDevices: r.TotalDevices, + ActiveDevices: r.ActiveDevices, + WorkingDevices: r.WorkingDevices, + FailedDevices: r.FailedDevices, + SpareDevices: r.SpareDevices, + UUID: r.UUID, + Devices: devices, + RebuildPercent: r.RebuildPercent, + RebuildSpeed: r.RebuildSpeed, + Operation: r.Operation, + Risk: storageRiskFromAssessment(assessment), } if !isInternalHostRAIDDevice(r.Device) { storageAssessments = append(storageAssessments, assessment) diff --git a/internal/unifiedresources/adapters_test.go b/internal/unifiedresources/adapters_test.go index 3f9c2e4630..b06013ade9 100644 --- a/internal/unifiedresources/adapters_test.go +++ b/internal/unifiedresources/adapters_test.go @@ -41,6 +41,30 @@ func TestResourceFromProxmoxNodeIncludesTemperature(t *testing.T) { } } +func TestHostRAIDRequiredMembersCanonicalRisk(t *testing.T) { + for _, active := range []int{4, 3} { + host := models.Host{ID: "raid-source", Hostname: "linux", Status: "online", RAID: []models.HostRAIDArray{{ + Device: "/dev/md2", Level: "raid5", State: "clean", RequiredDevices: 4, TotalDevices: 5, ActiveDevices: active, WorkingDevices: 5, SpareDevices: 1, + }}} + record := HostIngestRecord(host) + arrays := NewHostView(&record.Resource).RAID() + if len(arrays) != 1 || arrays[0].RequiredDevices != 4 || arrays[0].TotalDevices != 5 || arrays[0].SpareDevices != 1 { + t.Fatalf("canonical RAID count provenance=%+v", arrays) + } + if active == 4 { + if arrays[0].Risk != nil || record.Resource.Agent.StorageRisk != nil || record.Resource.Agent.ProtectionReduced { + t.Fatalf("healthy spare reduced protection: %+v", record.Resource.Agent) + } + } else if arrays[0].Risk == nil || arrays[0].Risk.Level != storagehealth.RiskCritical || !record.Resource.Agent.ProtectionReduced { + t.Fatalf("member deficit with spare lost protection risk: %+v", record.Resource.Agent) + } + arrays[0].RequiredDevices = 1 + if record.Resource.Agent.RAID[0].RequiredDevices != 4 { + t.Fatal("read view mutated canonical required count") + } + } +} + func TestResourceFromProxmoxNodeIncludesNetworkInterfaces(t *testing.T) { node := models.Node{ ID: "mock-cluster-pve1", Name: "pve1", Status: "online", diff --git a/internal/unifiedresources/clone_test.go b/internal/unifiedresources/clone_test.go index 6f6b837be0..1bd583e334 100644 --- a/internal/unifiedresources/clone_test.go +++ b/internal/unifiedresources/clone_test.go @@ -613,7 +613,7 @@ func TestCloneAgentData_DeepIsolation(t *testing.T) { LoadAverage: []float64{0.5, 1.0, 1.5}, DiskExclude: []string{"/dev/sda"}, RAID: []HostRAIDMeta{ - {Device: "/dev/md0", Level: "raid1", Risk: &StorageRisk{Level: storagehealth.RiskHealthy}}, + {Device: "/dev/md0", Level: "raid1", RequiredDevices: 2, TotalDevices: 3, SpareDevices: 1, Risk: &StorageRisk{Level: storagehealth.RiskHealthy}}, }, PackageUpdates: &AgentPackageUpdateMeta{Packages: []AgentPackageUpdate{{Name: "openssl"}}}, StorageCleanup: &AgentStorageCleanupMeta{Provider: "apt-package-cache", ReclaimableBytes: 512}, @@ -634,6 +634,13 @@ func TestCloneAgentData_DeepIsolation(t *testing.T) { if original.RAID[0].Risk.Level == storagehealth.RiskCritical { t.Error("mutating cloned RAID risk should not affect original") } + if cloned.RAID[0].RequiredDevices != 2 || cloned.RAID[0].TotalDevices != 3 || cloned.RAID[0].SpareDevices != 1 { + t.Fatalf("clone lost RAID count provenance: %+v", cloned.RAID[0]) + } + cloned.RAID[0].RequiredDevices = 1 + if original.RAID[0].RequiredDevices != 2 { + t.Fatal("mutating cloned RAID required count changed the source") + } cloned.PackageUpdates.Packages[0].Name = "mutated" cloned.StorageCleanup.Provider = "mutated" if original.PackageUpdates.Packages[0].Name != "openssl" || original.StorageCleanup.Provider != "apt-package-cache" { diff --git a/internal/unifiedresources/types.go b/internal/unifiedresources/types.go index 6fdff83064..ee5e4a1452 100644 --- a/internal/unifiedresources/types.go +++ b/internal/unifiedresources/types.go @@ -707,21 +707,22 @@ type HostRAIDDeviceMeta struct { // HostRAIDMeta describes a RAID array. type HostRAIDMeta struct { - Device string `json:"device"` - Name string `json:"name,omitempty"` - Level string `json:"level"` - State string `json:"state"` - TotalDevices int `json:"totalDevices"` - ActiveDevices int `json:"activeDevices"` - WorkingDevices int `json:"workingDevices"` - FailedDevices int `json:"failedDevices"` - SpareDevices int `json:"spareDevices"` - UUID string `json:"uuid,omitempty"` - Devices []HostRAIDDeviceMeta `json:"devices,omitempty"` - RebuildPercent float64 `json:"rebuildPercent,omitempty"` - RebuildSpeed string `json:"rebuildSpeed,omitempty"` - Operation string `json:"operation,omitempty"` - Risk *StorageRisk `json:"risk,omitempty"` + Device string `json:"device"` + Name string `json:"name,omitempty"` + Level string `json:"level"` + State string `json:"state"` + RequiredDevices int `json:"requiredDevices,omitempty"` // configured members, excluding spares; zero means unknown + TotalDevices int `json:"totalDevices"` + ActiveDevices int `json:"activeDevices"` + WorkingDevices int `json:"workingDevices"` + FailedDevices int `json:"failedDevices"` + SpareDevices int `json:"spareDevices"` + UUID string `json:"uuid,omitempty"` + Devices []HostRAIDDeviceMeta `json:"devices,omitempty"` + RebuildPercent float64 `json:"rebuildPercent,omitempty"` + RebuildSpeed string `json:"rebuildSpeed,omitempty"` + Operation string `json:"operation,omitempty"` + Risk *StorageRisk `json:"risk,omitempty"` } // HostUnraidDiskMeta describes a disk's role and state inside an Unraid array. diff --git a/pkg/agents/host/report.go b/pkg/agents/host/report.go index 8e27177aaf..0cacbeadfe 100644 --- a/pkg/agents/host/report.go +++ b/pkg/agents/host/report.go @@ -462,19 +462,22 @@ type SMARTAttributes struct { // RAIDArray represents an mdadm RAID array. type RAIDArray struct { - Device string `json:"device"` // e.g., /dev/md0 - Name string `json:"name,omitempty"` // Array name if set - Level string `json:"level"` // RAID level: raid0, raid1, raid5, raid6, raid10 - State string `json:"state"` // clean, active, degraded, recovering, resyncing, etc. - TotalDevices int `json:"totalDevices"` // Total number of devices in array - ActiveDevices int `json:"activeDevices"` // Number of active devices - WorkingDevices int `json:"workingDevices"` // Number of working devices - FailedDevices int `json:"failedDevices"` // Number of failed devices - SpareDevices int `json:"spareDevices"` // Number of spare devices - UUID string `json:"uuid,omitempty"` // Array UUID - Devices []RAIDDevice `json:"devices"` // Individual devices in array - RebuildPercent float64 `json:"rebuildPercent"` // Rebuild/resync progress (0-100) - RebuildSpeed string `json:"rebuildSpeed,omitempty"` // Rebuild speed (e.g., "50000K/sec") + Device string `json:"device"` // e.g., /dev/md0 + Name string `json:"name,omitempty"` // Array name if set + Level string `json:"level"` // RAID level: raid0, raid1, raid5, raid6, raid10 + State string `json:"state"` // clean, active, degraded, recovering, resyncing, etc. + // RequiredDevices is the configured member count (mdadm Raid Devices or + // mdstat's required/active bitmap), excluding spares. Zero means unknown. + RequiredDevices int `json:"requiredDevices,omitempty"` + TotalDevices int `json:"totalDevices"` // mdadm attached devices (including spares); mdstat bitmap width + ActiveDevices int `json:"activeDevices"` // Number of active devices + WorkingDevices int `json:"workingDevices"` // Number of working devices + FailedDevices int `json:"failedDevices"` // Number of failed devices + SpareDevices int `json:"spareDevices"` // Number of spare devices + UUID string `json:"uuid,omitempty"` // Array UUID + Devices []RAIDDevice `json:"devices"` // Individual devices in array + RebuildPercent float64 `json:"rebuildPercent"` // Rebuild/resync progress (0-100) + RebuildSpeed string `json:"rebuildSpeed,omitempty"` // Rebuild speed (e.g., "50000K/sec") // Operation reports the in-progress sync action from /proc/mdstat: // "recovery", "resync", "check", or "reshape". Empty when idle. Operation string `json:"operation,omitempty"` diff --git a/pkg/agents/host/report_test.go b/pkg/agents/host/report_test.go index 4d085356e7..c87df049a8 100644 --- a/pkg/agents/host/report_test.go +++ b/pkg/agents/host/report_test.go @@ -18,6 +18,32 @@ func TestLegacyReportJSONDefaultsOperationReceiptProtocolUnsupported(t *testing. } } +func TestRAIDRequiredDevicesWireCompatibility(t *testing.T) { + for _, required := range []int{0, 4} { + report := Report{RAID: []RAIDArray{{Device: "/dev/md1", RequiredDevices: required, TotalDevices: 5, ActiveDevices: 4, WorkingDevices: 5, SpareDevices: 1}}} + data, err := json.Marshal(report) + if err != nil { + t.Fatal(err) + } + var fields struct { + RAID []map[string]any `json:"raid"` + } + if err := json.Unmarshal(data, &fields); err != nil { + t.Fatal(err) + } + if value, present := fields.RAID[0]["requiredDevices"]; present != (required != 0) || (present && value != float64(required)) { + t.Fatalf("required member presence/value = %v/%v in %s", present, value, data) + } + var got Report + if err := json.Unmarshal(data, &got); err != nil { + t.Fatal(err) + } + if got.RAID[0].RequiredDevices != required || got.RAID[0].TotalDevices != 5 || got.RAID[0].SpareDevices != 1 { + t.Fatalf("RAID count round trip = %+v", got.RAID) + } + } +} + func TestReportSequenceIDRoundTrip(t *testing.T) { value := FormatReportSequenceID("0123456789abcdef", 42) if value != "v1:0123456789abcdef:42" {