diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index fdbf6c7ff6..37121d33d5 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -2770,6 +2770,13 @@ agent inventory, registration state, or command-channel readiness. recovered. That is report presentation only: the report never decides agent linkage, ownership of a metric, or agent freshness, and it adds no lifecycle state. + Reports on an `agent` resource (`internal/api/reporting_subject_alerts.go`) + read the agent's host ID from the unified resource's agent payload, only + when the agent source is present, to attach that agent's `agent:` + alerts and component children beside the linked node's alerts. The + report reads linkage the registry already resolved; it never links, + unlinks, identifies or authorizes an agent, and it adds no lifecycle + state. The same isolation rule applies to Patrol investigation-record propagation through shared AI intelligence handlers and `internal/api/router.go`: lifecycle surfaces may observe the resulting resource context, but they must diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index ece6cf813f..3e788f882d 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -9673,14 +9673,45 @@ aggregate, health card, table and resource blocks, and the fleet heuristic recommends checking the agent it moved to. "Triggered and resolved" counts recoveries only, and the report narrator's payload marks a covered handover with `successor_alert_listed` so the model describes the condition once. -Reports requested as `agent`, which is how the Reports picker addresses every -Proxmox node and standalone agent in v6, carry no alert rows yet: -`enrichReportRequest` enriches node, VM and container reports only, so an -agent report's deterministic verdict ignores its machine's alerts (HEALTHY, -or NO DATA without metrics). The rule above reaches those reports once they -attach the machine's alerts. The fleet heuristic offers its all-clear -pattern and "No fleet-wide action required" only while no alert, moved -ones included, is active. +The fleet heuristic offers its all-clear pattern and "No fleet-wide action +required" only while no alert, moved ones included, is active. +Reports requested as `agent`, which is how the Reports picker addresses +every Proxmox node and standalone Pulse agent in v6, attach the machine's +alerts through `enrichAgentReport` in +`internal/api/reporting_subject_alerts.go`. `reportAlertScopeFor` builds a +subject's alert identities from its unified resource, never from host or +node names, and an alert belongs to the report of each resource whose +identity it was raised under. For a machine that is the requested unified +ID, the metrics-target ID, the linked Proxmox node's source ID +(`-`, the node's own alerts) and, when the agent source is +present, the Pulse agent's `agent:` alerts with every component child +the agent raises under that identity (`agent:/disk:...`, disk +temperature, SMART, RAID, the Unraid array, custom sensors). When the alert +engine records a node alert's handover to the agent (`moved_to_agent`), the +moved alert and the agent's own alert for that metric land in one report, +where the rule above counts them once +(`TestContract_AgentReportCountsAMovedNodeAlertOnce`). Guests, containers, +Proxmox storage pools and physical disks raise alerts under their own +identities, so those stay on their own reports, and a same-named node in +another cluster never leaks in +(`TestContract_AgentReportAttachesTheMachinesOwnAlerts`). The one alert +raised under two identities is the agent's Unraid array alert +(`agent:/storage:unraid-array`), which appears on both the machine's +and the array's report. The agent report also fills resource details from +the resource's Proxmox and agent payloads and lists the storage pools and +physical disks whose unified parent is the machine through the node +report's row builders, so a disk shows only a collected temperature, +coloured by the alert thresholds. Every other type +without a dedicated enricher attaches the alerts in its scope: the unified +and metrics-target IDs; `alerts.DockerHostResourceID`, +`DockerContainerResourceID` and `DockerServiceResourceID` for Docker +runtimes, containers (by name when they have no ID) and Swarm services; a Proxmox storage pool's source ID with its ZFS pool and device +children; and the Proxmox health and wearout alerts on a disk's device-path +reference (`ProxmoxPhysicalDiskAlertResourceID`) only when the alert's +recorded `disk_serial` is the disk's, because a replacement disk in the same +slot takes over the path (#2627) +(`TestContract_UnenrichedReportTypesAttachTheSubjectsAlerts`). The legacy +`node` enricher keeps its name-based match for direct API callers. Multi-resource fleet reports (`engine.GenerateMulti`) now also carry an optional fleet-level narrative through a distinct `pkg/reporting.FleetNarrator` interface, kept separate from the diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index ce49337a20..e4cb41db47 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -1493,6 +1493,14 @@ recovery scope, or a storage/recovery-owned secret source. handover resolution (moved to a Pulse agent, not recovered). That changes how a report states alert health only; it opens no storage, backup or recovery path, and report backups still come from the recovery store. + Machine (`agent`) reports now list the storage pools and physical disks + whose unified parent is the machine, through the same row builders as + node reports. Storage reports attach their pool's own alerts (the Proxmox + source ID plus ZFS pool and device children, or the agent's alert on an + Unraid array), and physical-disk reports attach the Proxmox health and + wearout alerts that record the disk's serial. All of it reads the existing unified storage + and disk projections only; it opens no storage, backup or recovery path + and grants no storage action. Update-plan readiness payloads and apply-route readiness enforcement are adjacent shared API context only. Storage and recovery surfaces may observe the resulting update state if a future settings flow links to recovery diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go index 1adcbc7d63..dc240e657e 100644 --- a/internal/api/contract_test.go +++ b/internal/api/contract_test.go @@ -5049,6 +5049,361 @@ func TestContract_VMInventoryExportCSVHeaders(t *testing.T) { } } +// reportAlertResourceIDs lists a report request's alert rows by resource ID, +// sorted, so a test can state exactly which alerts a report attributes to +// its subject. +func reportAlertResourceIDs(req reporting.MetricReportRequest) []string { + ids := make([]string, 0, len(req.Alerts)) + for _, alert := range req.Alerts { + ids = append(ids, alert.ResourceID) + } + sort.Strings(ids) + return ids +} + +// Every Proxmox node and standalone Pulse agent is an `agent` resource in v6, +// and the Reports picker requests it as one. Its report must carry the +// machine's own alerts, matched through the identities its unified resource +// carries: the linked node's source ID, the agent's "agent:" alerts and +// their component children. Guests, storage pools and same-named nodes in +// another cluster are not the machine. +func TestContract_AgentReportAttachesTheMachinesOwnAlerts(t *testing.T) { + now := time.Now() + start := now.Add(-24 * time.Hour) + machineID := "agent-west-a" + memoryTotal, diskTotal := int64(64<<30), int64(1<<40) + storageTotal, storageUsed := int64(2<<40), int64(1<<40) + snapshot := reportingEnrichmentSnapshot{ + Resources: []unifiedresources.Resource{ + { + ID: machineID, + Type: unifiedresources.ResourceTypeAgent, + Name: "West Production A", + Status: unifiedresources.StatusOnline, + Uptime: 86400, + Sources: []unifiedresources.DataSource{unifiedresources.SourceProxmox, unifiedresources.SourceAgent}, + Identity: unifiedresources.ResourceIdentity{ + IPAddresses: []string{"192.0.2.10"}, + }, + Metrics: &unifiedresources.ResourceMetrics{ + Memory: &unifiedresources.MetricValue{Total: &memoryTotal}, + Disk: &unifiedresources.MetricValue{Total: &diskTotal}, + }, + Proxmox: &unifiedresources.ProxmoxData{ + SourceID: "cluster-a-pve1", + NodeName: "pve1", + Instance: "cluster-a", + HostURL: "https://pve1.example:8006", + PVEVersion: "8.2.4", + KernelVersion: "6.8.12-1-pve", + ClusterName: "cluster-a", + CPUInfo: &unifiedresources.CPUInfo{Model: "Xeon E5-2680", Cores: 16, Sockets: 2}, + }, + Agent: &unifiedresources.AgentData{ + AgentID: "host-pve1", + OSName: "Debian GNU/Linux", + OSVersion: "12", + }, + }, + { + ID: "storage-west-a-zfs", + Type: unifiedresources.ResourceTypeStorage, + Name: "local-zfs", + Status: unifiedresources.StatusOnline, + ParentID: &machineID, + Metrics: &unifiedresources.ResourceMetrics{ + Disk: &unifiedresources.MetricValue{Total: &storageTotal, Used: &storageUsed, Percent: 50}, + }, + Storage: &unifiedresources.StorageMeta{Type: "zfspool", Content: "images"}, + Proxmox: &unifiedresources.ProxmoxData{SourceID: "cluster-a-pve1-local-zfs"}, + }, + { + ID: "physical-disk-west-a-nvme0", + Type: unifiedresources.ResourceTypePhysicalDisk, + ParentID: &machineID, + PhysicalDisk: &unifiedresources.PhysicalDiskMeta{DevPath: "/dev/nvme0n1", Model: "Samsung 990", DiskType: "nvme", Health: "PASSED", Temperature: 41}, + }, + { + // A disk in standby keeps its last-known temperature, which a + // report must not tabulate as measured. + ID: "physical-disk-west-a-sdb", + Type: unifiedresources.ResourceTypePhysicalDisk, + ParentID: &machineID, + PhysicalDisk: &unifiedresources.PhysicalDiskMeta{ + DevPath: "/dev/sdb", DiskType: "hdd", Health: "PASSED", Temperature: 52, + Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.FieldStatus{State: diskinventory.FieldUnavailable, Reason: "standby"}, + }, + }, + }, + { + ID: "vm-west-a-101", + Type: unifiedresources.ResourceTypeVM, + ParentID: &machineID, + }, + }, + ActiveAlerts: []models.Alert{ + // The node's own alert, keyed by the Proxmox source ID. + {ID: "node-cpu", Type: "cpu", Level: "critical", ResourceID: "cluster-a-pve1", Node: "pve1", StartTime: now.Add(-time.Hour)}, + // The agent's own alert and a filesystem it reports. + {ID: "agent-memory", Type: "memory", Level: "warning", ResourceID: "agent:host-pve1", Node: "pve1", StartTime: now.Add(-time.Hour)}, + {ID: "agent-disk", Type: "disk", Level: "warning", ResourceID: "agent:host-pve1/disk:/", Node: "pve1", StartTime: now.Add(-time.Hour)}, + // The agent raises its Unraid array alert under its own identity, + // so it belongs to the machine as well as to the array's report. + {ID: "agent-array", Type: "unraid-array", Level: "warning", ResourceID: "agent:host-pve1/storage:unraid-array", Node: "pve1", StartTime: now.Add(-time.Hour)}, + // A provider incident keyed by the unified ID. + {ID: "incident", Type: "resource-incident", Level: "warning", ResourceID: machineID, StartTime: now.Add(-time.Hour)}, + // Not the machine: a guest, a storage pool, a same-named node in + // another cluster, and an agent whose host ID shares a prefix. + {ID: "guest", Type: "powered-off", Level: "warning", ResourceID: "cluster-a-pve1-101", Node: "pve1", StartTime: now.Add(-time.Hour)}, + {ID: "pool", Type: "zfs-pool-errors", Level: "warning", ResourceID: "cluster-a-pve1-local-zfs/zfs-pool:local-zfs", Node: "pve1", StartTime: now.Add(-time.Hour)}, + {ID: "other-cluster", Type: "cpu", Level: "critical", ResourceID: "cluster-b-pve1", Node: "pve1", StartTime: now.Add(-time.Hour)}, + {ID: "other-agent", Type: "memory", Level: "critical", ResourceID: "agent:host-pve10", Node: "pve10", StartTime: now.Add(-time.Hour)}, + }, + RecentlyResolved: []models.ResolvedAlert{ + { + Alert: models.Alert{ID: "node-temp", Type: "temperature", Level: "warning", ResourceID: "cluster-a-pve1", Node: "pve1", StartTime: now.Add(-3 * time.Hour)}, + ResolvedTime: now.Add(-2 * time.Hour), + }, + { + Alert: models.Alert{ID: "node-old", Type: "cpu", Level: "warning", ResourceID: "cluster-a-pve1", Node: "pve1", StartTime: now.Add(-50 * time.Hour)}, + ResolvedTime: now.Add(-48 * time.Hour), + }, + }, + } + + req := reporting.MetricReportRequest{ResourceType: "agent", ResourceID: machineID} + NewReportingHandlers(nil, nil).enrichReportRequest(context.Background(), "default", &req, snapshot, start, now) + + wantAlerts := []string{"agent-west-a", "agent:host-pve1", "agent:host-pve1/disk:/", "agent:host-pve1/storage:unraid-array", "cluster-a-pve1", "cluster-a-pve1"} + if got := reportAlertResourceIDs(req); !reflect.DeepEqual(got, wantAlerts) { + t.Fatalf("agent report alerts = %v, want %v", got, wantAlerts) + } + for _, alert := range req.Alerts { + if alert.Type == "temperature" && (alert.ResolvedTime == nil || !alert.Recovered()) { + t.Fatalf("node alert resolved inside the window must read as a recovery, got %+v", alert) + } + } + + res := req.Resource + if res == nil { + t.Fatal("agent report carries no resource details") + } + if res.Name != "West Production A" || res.Host != "https://pve1.example:8006" || res.PVEVersion != "8.2.4" || + res.KernelVersion != "6.8.12-1-pve" || res.CPUModel != "Xeon E5-2680" || res.CPUCores != 16 || res.CPUSockets != 2 || + res.MemoryTotal != memoryTotal || res.DiskTotal != diskTotal || res.Uptime != 86400 || + res.OSName != "Debian GNU/Linux" || res.OSVersion != "12" || res.ClusterName != "cluster-a" || + !reflect.DeepEqual(res.IPAddresses, []string{"192.0.2.10"}) { + t.Fatalf("agent report resource details = %+v", res) + } + if len(req.Storage) != 1 || req.Storage[0].Name != "local-zfs" || req.Storage[0].Type != "zfspool" || req.Storage[0].UsagePerc != 50 { + t.Fatalf("agent report storage = %+v", req.Storage) + } + // Disk rows share the node report's builder: only a collected temperature + // is tabulated, coloured by the disk temperature alert thresholds. + if len(req.Disks) != 2 { + t.Fatalf("agent report disks = %+v", req.Disks) + } + nvme, standby := req.Disks[0], req.Disks[1] + if nvme.Device != "/dev/nvme0n1" || nvme.Health != "PASSED" || nvme.Temperature != 41 || + nvme.TemperatureCritical <= 0 || nvme.TemperatureWarning <= 0 || nvme.TemperatureWarning > nvme.TemperatureCritical { + t.Fatalf("collected disk row = %+v", nvme) + } + if standby.Device != "/dev/sdb" || standby.Temperature != 0 { + t.Fatalf("a retained disk temperature must not be reported as measured, got %+v", standby) + } +} + +// A node alert that moved to the machine's Pulse agent and the agent's own +// alert for that metric both belong to the machine's report, and the report +// verdict counts the condition once. +func TestContract_AgentReportCountsAMovedNodeAlertOnce(t *testing.T) { + now := time.Now() + machineID := "agent-west-a" + snapshot := reportingEnrichmentSnapshot{ + Resources: []unifiedresources.Resource{{ + ID: machineID, + Type: unifiedresources.ResourceTypeAgent, + Name: "West Production A", + Sources: []unifiedresources.DataSource{unifiedresources.SourceProxmox, unifiedresources.SourceAgent}, + Proxmox: &unifiedresources.ProxmoxData{SourceID: "cluster-a-pve1", NodeName: "pve1"}, + Agent: &unifiedresources.AgentData{AgentID: "host-pve1"}, + }}, + ActiveAlerts: []models.Alert{{ + ID: "agent-memory", Type: "memory", Level: "warning", ResourceID: "agent:host-pve1", + Message: "Agent memory at 94%", StartTime: now.Add(-time.Minute), + }}, + RecentlyResolved: []models.ResolvedAlert{{ + Alert: models.Alert{ + ID: "node-memory", Type: "memory", Level: "warning", ResourceID: "cluster-a-pve1", + Message: "Node memory at 95%", StartTime: now.Add(-time.Hour), + Resolution: &models.AlertResolution{ + Reason: "moved_to_agent", + SuccessorResourceID: "agent:host-pve1", + SuccessorName: "pve1 (Host Agent)", + }, + }, + ResolvedTime: now.Add(-2 * time.Minute), + }}, + } + + req := reporting.MetricReportRequest{ResourceType: "agent", ResourceID: machineID} + NewReportingHandlers(nil, nil).enrichReportRequest(context.Background(), "default", &req, snapshot, now.Add(-24*time.Hour), now) + + if got := reportAlertResourceIDs(req); !reflect.DeepEqual(got, []string{"agent:host-pve1", "cluster-a-pve1"}) { + t.Fatalf("agent report alerts = %v", got) + } + narrative, err := reporting.HeuristicNarrator{}.Narrate(context.Background(), reporting.NarrativeInput{Alerts: req.Alerts}) + if err != nil { + t.Fatal(err) + } + if narrative.HealthStatus != "WARNING" || narrative.HealthMessage != "1 warning detected - review recommended" { + t.Fatalf("moved node alert and its successor must count once, got %q / %q", narrative.HealthStatus, narrative.HealthMessage) + } +} + +// Report types without a dedicated enricher still list the alerts raised on +// their subject: the unified ID, the metrics-target ID, and the IDs each +// alert producer builds for a Docker runtime, a container or a storage pool's +// ZFS components. Workloads under a runtime stay with their own reports. +func TestContract_UnenrichedReportTypesAttachTheSubjectsAlerts(t *testing.T) { + now := time.Now() + resources := []unifiedresources.Resource{ + { + // Docker runtimes are agent-typed internally and travel as + // docker-host. + ID: "agent-docker-1", + Type: unifiedresources.ResourceTypeAgent, + Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker}, + Docker: &unifiedresources.DockerData{HostSourceID: "docker-host-1"}, + }, + { + ID: "app-container-web", + Type: unifiedresources.ResourceTypeAppContainer, + Name: "web", + Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker}, + Docker: &unifiedresources.DockerData{HostSourceID: "docker-host-1", ContainerID: "web-1"}, + }, + { + // A container reported without an ID is referenced by its name. + ID: "app-container-sidecar", + Type: unifiedresources.ResourceTypeAppContainer, + Name: "sidecar", + Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker}, + Docker: &unifiedresources.DockerData{HostSourceID: "docker-host-1"}, + }, + { + ID: "storage-zfs", + Type: unifiedresources.ResourceTypeStorage, + Sources: []unifiedresources.DataSource{unifiedresources.SourceProxmox}, + Proxmox: &unifiedresources.ProxmoxData{SourceID: "cluster-a-pve1-local-zfs"}, + }, + { + ID: "k8s-cluster-prod", + Type: unifiedresources.ResourceTypeK8sCluster, + }, + { + ID: "pmg-eu", + Type: unifiedresources.ResourceTypePMG, + MetricsTarget: &unifiedresources.MetricsTarget{ResourceType: "agent", ResourceID: "pmg-main"}, + }, + { + ID: "docker-service-web", + Type: unifiedresources.ResourceTypeDockerService, + Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker}, + Docker: &unifiedresources.DockerData{HostSourceID: "docker-host-1", ServiceID: "svc-1"}, + }, + { + // A service without an ID is named by the adapter's resource + // name, which the producer falls back to. + ID: "docker-service-api", + Type: unifiedresources.ResourceTypeDockerService, + Name: "api", + Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker}, + Docker: &unifiedresources.DockerData{HostSourceID: "docker-host-1"}, + }, + { + // A Pulse agent reports the Unraid array as storage keyed by + // its own source ID, which is the metrics target. + ID: "storage-unraid", + Type: unifiedresources.ResourceTypeStorage, + Sources: []unifiedresources.DataSource{unifiedresources.SourceAgent}, + MetricsTarget: &unifiedresources.MetricsTarget{ResourceType: "storage", ResourceID: "host-1/storage:unraid-array"}, + }, + { + ID: "physical-disk-sda", + Type: unifiedresources.ResourceTypePhysicalDisk, + Sources: []unifiedresources.DataSource{unifiedresources.SourceProxmox}, + Proxmox: &unifiedresources.ProxmoxData{Instance: "cluster-a", NodeName: "pve1"}, + PhysicalDisk: &unifiedresources.PhysicalDiskMeta{DevPath: "/dev/sda", Serial: "SER-B"}, + }, + } + active := func(resourceID string) models.Alert { + return models.Alert{ID: resourceID + "-alert", Type: "status", Level: "warning", ResourceID: resourceID, StartTime: now.Add(-time.Hour)} + } + // Proxmox disk alerts reference the device path, which a replacement disk + // in the same slot takes over, and record the serial of the disk they + // were raised on. + diskRef := unifiedresources.ProxmoxPhysicalDiskAlertResourceID("cluster-a", "pve1", "/dev/sda") + diskAlert := func(id, serial string) models.Alert { + alert := active(diskRef) + alert.ID, alert.Type, alert.Message = id, "disk-health", id + alert.Metadata = map[string]interface{}{"disk_serial": serial} + return alert + } + snapshot := reportingEnrichmentSnapshot{ + Resources: resources, + ActiveAlerts: []models.Alert{ + active(alerts.DockerHostResourceID("docker-host-1")), + active(alerts.DockerContainerResourceID("docker-host-1", "web-1", "web")), + active(alerts.DockerContainerResourceID("docker-host-1", "", "sidecar")), + active("cluster-a-pve1-local-zfs"), + active("cluster-a-pve1-local-zfs/zfs-pool:local-zfs"), + active("cluster-a-pve1-local-zfs/zfs-pool:local-zfs/device:sda2"), + active("cluster-a-pve1-local-zfs2"), + active("k8s-cluster-prod"), + active("pmg-main"), + active(alerts.DockerServiceResourceID("docker-host-1", "svc-1", "web")), + active(alerts.DockerServiceResourceID("docker-host-1", "", "api")), + active("agent:host-1/storage:unraid-array"), + diskAlert("disk-health-replacement", "SER-B"), + diskAlert("disk-health-predecessor", "SER-A"), + }, + } + + for _, tc := range []struct { + resourceType string + resourceID string + want []string + }{ + {"docker-host", "agent-docker-1", []string{alerts.DockerHostResourceID("docker-host-1")}}, + {"app-container", "app-container-web", []string{alerts.DockerContainerResourceID("docker-host-1", "web-1", "web")}}, + {"app-container", "app-container-sidecar", []string{alerts.DockerContainerResourceID("docker-host-1", "", "sidecar")}}, + {"storage", "storage-zfs", []string{ + "cluster-a-pve1-local-zfs", + "cluster-a-pve1-local-zfs/zfs-pool:local-zfs", + "cluster-a-pve1-local-zfs/zfs-pool:local-zfs/device:sda2", + }}, + {"k8s", "k8s-cluster-prod", []string{"k8s-cluster-prod"}}, + {"pmg", "pmg-eu", []string{"pmg-main"}}, + {"app-container", "docker-service-web", []string{alerts.DockerServiceResourceID("docker-host-1", "svc-1", "web")}}, + {"app-container", "docker-service-api", []string{alerts.DockerServiceResourceID("docker-host-1", "", "api")}}, + {"storage", "storage-unraid", []string{"agent:host-1/storage:unraid-array"}}, + {"disk", "physical-disk-sda", []string{diskRef}}, + } { + req := reporting.MetricReportRequest{ResourceType: tc.resourceType, ResourceID: tc.resourceID} + NewReportingHandlers(nil, nil).enrichReportRequest(context.Background(), "default", &req, snapshot, now.Add(-24*time.Hour), now) + if got := reportAlertResourceIDs(req); !reflect.DeepEqual(got, tc.want) { + t.Fatalf("%s report alerts = %v, want %v", tc.resourceType, got, tc.want) + } + // Only the alert naming the replacement disk's serial is its own; + // its predecessor's alert on the same path stays off its report. + if tc.resourceType == "disk" && req.Alerts[0].Message != "disk-health-replacement" { + t.Fatalf("disk report alert = %+v", req.Alerts[0]) + } + } +} + func TestContract_ReportingCatalogJSONSnapshot(t *testing.T) { handler := NewReportingHandlers(nil, nil) req := httptest.NewRequest(http.MethodGet, "/api/admin/reports/catalog", nil) diff --git a/internal/api/metrics_reporting_handlers.go b/internal/api/metrics_reporting_handlers.go index 08e4c7b240..201c3b110e 100644 --- a/internal/api/metrics_reporting_handlers.go +++ b/internal/api/metrics_reporting_handlers.go @@ -694,6 +694,10 @@ func (h *ReportingHandlers) enrichReportRequest(ctx context.Context, orgID strin h.enrichVMReport(ctx, orgID, req, snapshot, start, end) case "system-container", "oci-container": h.enrichContainerReport(ctx, orgID, req, snapshot, start, end) + case "agent": + h.enrichAgentReport(req, snapshot, start, end) + default: + attachReportAlerts(req, snapshot, reportAlertScopeFor(req, findReportSubjectResource(req, snapshot)), start, end) } } @@ -849,41 +853,7 @@ func (h *ReportingHandlers) enrichNodeReport(req *reporting.MetricReportRequest, if storageNode != node.Name { continue } - - var total, used, available int64 - var usagePerc float64 - if r.Metrics != nil && r.Metrics.Disk != nil { - if r.Metrics.Disk.Total != nil { - total = *r.Metrics.Disk.Total - } - if r.Metrics.Disk.Used != nil { - used = *r.Metrics.Disk.Used - } - if total > 0 { - available = total - used - } - usagePerc = r.Metrics.Disk.Percent - if usagePerc == 0 && total > 0 { - usagePerc = (float64(used) / float64(total)) * 100 - } - } - - var storageType, content string - if r.Storage != nil { - storageType = r.Storage.Type - content = r.Storage.Content - } - - req.Storage = append(req.Storage, reporting.StorageInfo{ - Name: r.Name, - Type: storageType, - Status: string(r.Status), - Total: total, - Used: used, - Available: available, - UsagePerc: usagePerc, - Content: content, - }) + req.Storage = append(req.Storage, reportStorageInfo(r)) case unifiedresources.ResourceTypePhysicalDisk: if r.PhysicalDisk == nil { continue @@ -895,20 +865,7 @@ func (h *ReportingHandlers) enrichNodeReport(req *reporting.MetricReportRequest, if diskNode != node.Name { continue } - pd := r.PhysicalDisk - temperatureWarning, temperatureCritical := reportDiskTemperatureThresholds(snapshot.alertManager, pd.DiskType) - req.Disks = append(req.Disks, reporting.DiskInfo{ - Device: pd.DevPath, - Model: pd.Model, - Serial: pd.Serial, - Type: pd.DiskType, - Size: pd.SizeBytes, - Health: pd.Health, - Temperature: reportDiskTemperature(*pd), - WearLevel: pd.Wearout, - TemperatureWarning: temperatureWarning, - TemperatureCritical: temperatureCritical, - }) + req.Disks = append(req.Disks, reportDiskInfo(*r.PhysicalDisk, snapshot.alertManager)) } } } diff --git a/internal/api/reporting_subject_alerts.go b/internal/api/reporting_subject_alerts.go new file mode 100644 index 0000000000..8a722ecfa5 --- /dev/null +++ b/internal/api/reporting_subject_alerts.go @@ -0,0 +1,363 @@ +package api + +import ( + "slices" + "strings" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" + "github.com/rcourtman/pulse-go-rewrite/internal/models" + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" + "github.com/rcourtman/pulse-go-rewrite/pkg/reporting" +) + +// reportAlertScope is the set of alert resource IDs a report attributes to +// its subject. It is built from the identities the subject's unified +// resource carries, never from host or node names: an alert belongs to the +// report of each resource whose identity it was raised under. A machine's +// report therefore leaves out the guests, containers, storage pools and +// disks it hosts, whose alerts are raised under their own identities and +// appear on their own reports. +type reportAlertScope struct { + ids map[string]struct{} + // childPrefixes match component alerts raised under the subject's own + // alert identity, such as "agent:/disk:/" for a filesystem the + // agent reports. + childPrefixes []string + // serialGatedIDs are path-shaped references the subject owns only for + // alerts that record its serial: a replacement disk in the same slot + // takes over a Proxmox disk's device path (#2627). + serialGatedIDs map[string]string +} + +func (s *reportAlertScope) addID(id string) { + if id = strings.TrimSpace(id); id != "" { + s.ids[id] = struct{}{} + } +} + +func (s *reportAlertScope) addChildPrefix(parentID string) { + if parentID = strings.TrimSpace(parentID); parentID != "" { + s.childPrefixes = append(s.childPrefixes, parentID+"/") + } +} + +// matchesAlert reports whether an alert belongs to the subject. +func (s reportAlertScope) matchesAlert(alert models.Alert) bool { + if s.matches(alert.ResourceID) { + return true + } + serial, gated := s.serialGatedIDs[strings.TrimSpace(alert.ResourceID)] + if !gated { + return false + } + recorded, _ := alert.Metadata["disk_serial"].(string) + return strings.TrimSpace(recorded) != "" && strings.EqualFold(strings.TrimSpace(recorded), serial) +} + +func (s reportAlertScope) matches(resourceID string) bool { + resourceID = strings.TrimSpace(resourceID) + if resourceID == "" { + return false + } + if _, ok := s.ids[resourceID]; ok { + return true + } + for _, prefix := range s.childPrefixes { + if strings.HasPrefix(resourceID, prefix) { + return true + } + } + return false +} + +// reportAlertScopeFor returns the alert identities of a report subject: +// - the requested unified ID and the resource's metrics-target ID, which +// key provider incidents and the alerts evaluated on unified resources +// (vSphere hosts, TrueNAS systems, Kubernetes, PBS, PMG); +// - for a machine, its linked Proxmox node's source ID +// ("-"), which keys the node's own alerts, and its +// Pulse agent's "agent:" alerts with their component children +// ("agent:/disk:", disk temperature, SMART, RAID, the Unraid +// array, custom sensors). A node alert that moved to the agent and the +// agent's own alert for that metric therefore land in the same report, +// where pkg/reporting counts them once; +// - for a Docker runtime, container or Swarm service, the IDs the Docker +// alert producer builds (alerts.DockerHostResourceID, +// DockerContainerResourceID, DockerServiceResourceID); +// - for a storage pool, its source ID and the pool's ZFS components +// ("/zfs-pool:/device:"), or for an agent-reported +// Unraid array, the agent alert raised on it; +// - for a Proxmox physical disk, the alerts on its device-path reference +// that record its serial. +// +// resource may be nil when the request names no live unified resource; the +// scope then holds only the request's own IDs. +func reportAlertScopeFor(req *reporting.MetricReportRequest, resource *unifiedresources.Resource) reportAlertScope { + scope := reportAlertScope{ids: make(map[string]struct{}), serialGatedIDs: make(map[string]string)} + if req != nil { + scope.addID(req.ResourceID) + scope.addID(req.MetricsResourceID) + } + if resource == nil { + return scope + } + scope.addID(resource.ID) + switch unifiedresources.CanonicalResourceType(resource.Type) { + case unifiedresources.ResourceTypeAgent: + if resource.Proxmox != nil { + scope.addID(resource.Proxmox.SourceID) + } + if agentID := reportHostAgentID(*resource); agentID != "" { + // Alert evaluation names a Pulse agent "agent:" + // (alerts.hostResourceID), the same reference the canonical + // identity indexes for per-resource policy (#1497). + scope.addID("agent:" + agentID) + scope.addChildPrefix("agent:" + agentID) + } + if resource.Docker != nil && strings.TrimSpace(resource.Docker.HostSourceID) != "" { + scope.addID(alerts.DockerHostResourceID(resource.Docker.HostSourceID)) + } + case unifiedresources.ResourceTypeAppContainer: + if docker := resource.Docker; docker != nil && strings.TrimSpace(docker.HostSourceID) != "" { + // A container without an ID is referenced by its name, which + // the container adapter carries as the resource name. + scope.addID(alerts.DockerContainerResourceID(docker.HostSourceID, docker.ContainerID, resource.Name)) + } + case unifiedresources.ResourceTypeDockerService: + if docker := resource.Docker; docker != nil && strings.TrimSpace(docker.HostSourceID) != "" { + // The service adapter carries the service name as the resource + // name; the producer falls back to it when the ID is empty. + serviceName := docker.ServiceName + if strings.TrimSpace(serviceName) == "" { + serviceName = resource.Name + } + scope.addID(alerts.DockerServiceResourceID(docker.HostSourceID, docker.ServiceID, serviceName)) + } + case unifiedresources.ResourceTypeStorage: + if resource.Proxmox != nil { + scope.addID(resource.Proxmox.SourceID) + scope.addChildPrefix(resource.Proxmox.SourceID) + } + // A Pulse agent reports an Unraid array as storage keyed + // "/storage:unraid-array", which is then the metrics target, and + // raises the array's alert under its own identity as + // "agent:/storage:unraid-array". That alert belongs to both the + // array's report and its machine's. + if req != nil && reportStorageReportedOnlyByAgent(*resource) && strings.TrimSpace(req.MetricsResourceID) != "" { + scope.addID("agent:" + strings.TrimSpace(req.MetricsResourceID)) + } + case unifiedresources.ResourceTypePhysicalDisk: + if px, disk := resource.Proxmox, resource.PhysicalDisk; px != nil && disk != nil && + strings.TrimSpace(px.Instance) != "" && strings.TrimSpace(px.NodeName) != "" && + strings.TrimSpace(disk.DevPath) != "" && strings.TrimSpace(disk.Serial) != "" { + // Proxmox disk health and wearout alerts are raised on the + // disk's device-path reference and record the disk's serial + // (disk_serial). The path is not hardware identity, so an alert + // counts for this disk only when it names this disk's serial. + ref := unifiedresources.ProxmoxPhysicalDiskAlertResourceID(px.Instance, px.NodeName, disk.DevPath) + scope.serialGatedIDs[ref] = strings.TrimSpace(disk.Serial) + } + } + return scope +} + +// reportStorageReportedOnlyByAgent reports whether a storage resource comes +// from a Pulse agent alone, so its metrics target is the agent's source ID. +func reportStorageReportedOnlyByAgent(resource unifiedresources.Resource) bool { + if !slices.Contains(resource.Sources, unifiedresources.SourceAgent) { + return false + } + for _, source := range resource.Sources { + if source != unifiedresources.SourceAgent { + return false + } + } + return true +} + +// reportHostAgentID is the host ID of the Pulse agent reporting a machine. +// vSphere hosts and TrueNAS systems also fill AgentData, so the agent source +// must be present before the ID names a Pulse agent. +func reportHostAgentID(resource unifiedresources.Resource) string { + if resource.Agent == nil || !slices.Contains(resource.Sources, unifiedresources.SourceAgent) { + return "" + } + return strings.TrimSpace(resource.Agent.AgentID) +} + +// findReportSubjectResource returns the unified resource the request names. +func findReportSubjectResource(req *reporting.MetricReportRequest, snapshot reportingEnrichmentSnapshot) *unifiedresources.Resource { + if req == nil { + return nil + } + for i := range snapshot.Resources { + if snapshot.Resources[i].ID == req.ResourceID { + return &snapshot.Resources[i] + } + } + return nil +} + +// attachReportAlerts adds the subject's open alerts and the alerts resolved +// inside the report window, keeping each alert's resolution. +func attachReportAlerts(req *reporting.MetricReportRequest, snapshot reportingEnrichmentSnapshot, scope reportAlertScope, start, end time.Time) { + for _, alert := range snapshot.ActiveAlerts { + if scope.matchesAlert(alert) { + req.Alerts = append(req.Alerts, reportActiveAlertInfo(alert)) + } + } + for _, resolved := range snapshot.RecentlyResolved { + if scope.matchesAlert(resolved.Alert) && + resolved.ResolvedTime.After(start) && resolved.ResolvedTime.Before(end) { + req.Alerts = append(req.Alerts, reportResolvedAlertInfo(resolved)) + } + } +} + +// enrichAgentReport fills a report on a machine: every Proxmox node and +// standalone Pulse agent is a unified `agent` resource in v6. Details come +// from the resource's Proxmox and agent payloads, storage pools and physical +// disks from the resources whose unified parent is the machine, and alerts +// from reportAlertScopeFor. +func (h *ReportingHandlers) enrichAgentReport(req *reporting.MetricReportRequest, snapshot reportingEnrichmentSnapshot, start, end time.Time) { + resource := findReportSubjectResource(req, snapshot) + if resource != nil { + req.Resource = reportMachineResourceInfo(*resource) + for i := range snapshot.Resources { + child := &snapshot.Resources[i] + if child.ParentID == nil || *child.ParentID != resource.ID { + continue + } + switch unifiedresources.CanonicalResourceType(child.Type) { + case unifiedresources.ResourceTypeStorage: + req.Storage = append(req.Storage, reportStorageInfo(*child)) + case unifiedresources.ResourceTypePhysicalDisk: + if child.PhysicalDisk != nil { + req.Disks = append(req.Disks, reportDiskInfo(*child.PhysicalDisk, snapshot.alertManager)) + } + } + } + } + attachReportAlerts(req, snapshot, reportAlertScopeFor(req, resource), start, end) +} + +// reportMachineResourceInfo projects a machine's unified payloads onto the +// report's resource details. The Proxmox API is authoritative for PVE +// facts; the Pulse agent adds the operating system. +func reportMachineResourceInfo(resource unifiedresources.Resource) *reporting.ResourceInfo { + info := &reporting.ResourceInfo{ + Name: resource.Name, + Status: string(resource.Status), + Uptime: resource.Uptime, + IPAddresses: append([]string(nil), resource.Identity.IPAddresses...), + Tags: append([]string(nil), resource.Tags...), + } + if resource.Temperature != nil { + temp := *resource.Temperature + info.Temperature = &temp + } + if metrics := resource.Metrics; metrics != nil { + if metrics.Memory != nil && metrics.Memory.Total != nil { + info.MemoryTotal = *metrics.Memory.Total + } + if metrics.Disk != nil && metrics.Disk.Total != nil { + info.DiskTotal = *metrics.Disk.Total + } + } + if px := resource.Proxmox; px != nil { + info.Host = px.HostURL + info.Instance = px.Instance + info.KernelVersion = px.KernelVersion + info.PVEVersion = px.PVEVersion + info.ClusterName = px.ClusterName + info.IsCluster = px.IsClusterMember + info.LoadAverage = append([]float64(nil), px.LoadAverage...) + if info.Uptime == 0 { + info.Uptime = px.Uptime + } + if px.CPUInfo != nil { + info.CPUModel = px.CPUInfo.Model + info.CPUCores = px.CPUInfo.Cores + info.CPUSockets = px.CPUInfo.Sockets + } + } + if agent := resource.Agent; agent != nil && reportHostAgentID(resource) != "" { + info.OSName = agent.OSName + info.OSVersion = agent.OSVersion + if info.KernelVersion == "" { + info.KernelVersion = agent.KernelVersion + } + if info.CPUCores == 0 { + info.CPUCores = agent.CPUCount + } + if len(info.LoadAverage) == 0 { + info.LoadAverage = append([]float64(nil), agent.LoadAverage...) + } + if info.Uptime == 0 { + info.Uptime = agent.UptimeSeconds + } + if info.MemoryTotal == 0 && agent.Memory != nil { + info.MemoryTotal = agent.Memory.Total + } + } + return info +} + +// reportStorageInfo is the report row for a storage pool resource. +func reportStorageInfo(r unifiedresources.Resource) reporting.StorageInfo { + var total, used, available int64 + var usagePerc float64 + if r.Metrics != nil && r.Metrics.Disk != nil { + if r.Metrics.Disk.Total != nil { + total = *r.Metrics.Disk.Total + } + if r.Metrics.Disk.Used != nil { + used = *r.Metrics.Disk.Used + } + if total > 0 { + available = total - used + } + usagePerc = r.Metrics.Disk.Percent + if usagePerc == 0 && total > 0 { + usagePerc = (float64(used) / float64(total)) * 100 + } + } + + var storageType, content string + if r.Storage != nil { + storageType = r.Storage.Type + content = r.Storage.Content + } + + return reporting.StorageInfo{ + Name: r.Name, + Type: storageType, + Status: string(r.Status), + Total: total, + Used: used, + Available: available, + UsagePerc: usagePerc, + Content: content, + } +} + +// reportDiskInfo is the report row for a physical disk resource. Its +// temperature is the collected reading only (reportDiskTemperature), coloured +// by the tenant's disk temperature alert thresholds; a nil manager means the +// factory alert configuration. +func reportDiskInfo(pd unifiedresources.PhysicalDiskMeta, manager *alerts.Manager) reporting.DiskInfo { + temperatureWarning, temperatureCritical := reportDiskTemperatureThresholds(manager, pd.DiskType) + return reporting.DiskInfo{ + Device: pd.DevPath, + Model: pd.Model, + Serial: pd.Serial, + Type: pd.DiskType, + Size: pd.SizeBytes, + Health: pd.Health, + Temperature: reportDiskTemperature(pd), + WearLevel: pd.Wearout, + TemperatureWarning: temperatureWarning, + TemperatureCritical: temperatureCritical, + } +}