diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index fffea3c93..5e6aba998 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -543,6 +543,13 @@ API-backed workspace, preserve the saved-connection test and health model, and keep direct `ESXi` out of the phase-1 route and install model. Lifecycle- adjacent flows must not invent a VMware-only setup shell or reframe unified- agent host install as the bootstrap requirement for VMware support. +That same platform-connections boundary also owns demo/mock continuity for +those settings surfaces. When `/api/system/mock-mode` is enabled, +provider-backed settings panels and their downstream infrastructure, +workloads, storage, and recovery handoffs must read the canonical connection +fixtures from `internal/mock/fixture_graph.go` instead of handler-local demo +lists, so operator-facing demos stay coherent across those adjacent product +surfaces without a restart. That same lifecycle-owned VMware workspace must also keep the backend-runtime shape hidden behind one operator-facing connection model. The settings surface may show one VMware connection's poll health, last error classification, and diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index 0ea5dfda2..2fb65cd2e 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -719,6 +719,7 @@ connection payloads returned from `/api/truenas/connections` and platform fixture owner rather than handler-local fixture assembly, so settings payloads stay aligned with the unified runtime mock graph, storage/recovery context, and seeded monitoring history. +That same runtime mock contract now also owns demo-grade dataset balance. When mock mode is enabled, `/api/truenas/connections`, `/api/vmware/connections`, and the shared `/api/resources` surface must project TrueNAS app-container counts, VMware datastore counts, and related settings/runtime metadata from the canonical `internal/mock/fixture_graph.go` owner so settings, infrastructure, workloads, storage, and recovery all present the same user-facing demo dataset. That same VMware test contract now also owns structured setup-failure classification. When `POST /api/vmware/connections/test` or `POST /api/vmware/connections/{id}/test` fails, the backend payload must diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md index 5c78d6c66..502bc261e 100644 --- a/docs/release-control/v6/internal/subsystems/deployment-installability.md +++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md @@ -175,6 +175,11 @@ writing a separate `mock.env` sidecar when supported runtime installs already carry mock-mode ownership through `.env`. Mock enable/disable/status must use the canonical runtime `.env` path, with any install-dir `.env` probe treated as compatibility only. +That same dev-runtime boundary also owns the default mock density used for +local demos. `scripts/toggle-mock.sh` must seed the same `PULSE_MOCK_*` +defaults as `internal/mock.DefaultConfig`, so managed runtime toggles, local +demo restarts, and CLI mock status all converge on one canonical dataset +instead of drifting across shell helpers. That same rule applies to live runtime behavior too: config loading and reload watching may not treat `mock.env` as a parallel primary-path control surface. Supported mock-mode runtime state must come from the canonical `.env` contract, diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index ee594e51b..f60baa058 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -115,6 +115,11 @@ running server, the live TrueNAS and VMware provider bindings must swap to the mock-backed supplemental records and refresh canonical read-state immediately instead of waiting for a process restart before shared resource consumers can see the platform inventory. +That same mock-runtime boundary also owns freshness while demos are running. +The mock update loop must keep provider-backed TrueNAS and VMware records plus +legacy PBS and PMG summaries on current `LastSeen` and health state each tick, +so long-lived infrastructure, workloads, storage, and recovery demos do not +decay into synthetic stale-state warnings while mock mode remains enabled. That same summary owner also owns VMware partial-success classification. Optional VI JSON or Automation enrichment reads that fail after base host/VM/datastore inventory succeeds must not collapse the whole poll into a diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index 705fb0641..dd7d26c98 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -208,6 +208,11 @@ subjects must derive from the shared `internal/mock/fixture_graph.go` owner instead of a separate hardcoded recovery cache, so recovery filters, rollups, and shared route handoffs see the same platform set as settings and infrastructure. +That same graph-owned mock boundary also owns demo-readiness for storage and +recovery surfaces. Mock summary cards, seeded history, and provider-backed +storage/recovery counts must come from the same canonical fixture graph so +storage and recovery demos show realistic healthy-versus-attention balance +instead of blank history, stale provider context, or page-local fixture drift. That same shared `internal/api/` dependency also assumes auth-persistence teardown is synchronous when recovery-adjacent runtimes reinitialize. Session, CSRF, and recovery-token workers may not leave stale background goroutines or diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md index 9a8e835a5..cf7a92db8 100644 --- a/docs/release-control/v6/internal/subsystems/unified-resources.md +++ b/docs/release-control/v6/internal/subsystems/unified-resources.md @@ -215,6 +215,13 @@ must still ingest that provider-owned source instead of letting the seed silence the platform entirely. Operator-facing source filters may accept the `vmware-vsphere` alias for the VMware platform, but the emitted shared source family remains canonical `vmware`. +That same unified-resource boundary now also owns stored metrics-target +continuity for provider-backed resources. When registry rebuild cannot derive a +fresh metrics target from raw source facets, `internal/unifiedresources/registry.go` +must preserve the canonical `MetricsTarget` already attached to provider-backed +agents or app-containers, and typed views must expose that stored target +unchanged so shared chart routes keep using canonical IDs across live and demo +projections. That same VMware contract now also includes the identity rule. VMware managed object identifiers are phase-1 provider identities, but they must be scoped by the owning `vCenter` connection or discovered vCenter identity so bare object diff --git a/install.sh b/install.sh index 6cc923ad2..3f514969a 100755 --- a/install.sh +++ b/install.sh @@ -2826,11 +2826,11 @@ setup_directories() { PULSE_MOCK_MODE=false # Mock configuration (only used when PULSE_MOCK_MODE=true) -#PULSE_MOCK_NODES=7 -#PULSE_MOCK_VMS_PER_NODE=5 -#PULSE_MOCK_LXCS_PER_NODE=8 +#PULSE_MOCK_NODES=3 +#PULSE_MOCK_VMS_PER_NODE=3 +#PULSE_MOCK_LXCS_PER_NODE=3 #PULSE_MOCK_RANDOM_METRICS=true -#PULSE_MOCK_STOPPED_PERCENT=20 +#PULSE_MOCK_STOPPED_PERCENT=6 EOF fi if [[ -f "$CONFIG_DIR/.env" ]]; then diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go index 2464440d6..4117742f6 100644 --- a/internal/api/contract_test.go +++ b/internal/api/contract_test.go @@ -346,6 +346,28 @@ func TestContract_PlatformMockToggleRebindsRuntimeConnectionsAndResources(t *tes assertResourceSource("/api/resources?source=truenas", unifiedresources.SourceTrueNAS) assertResourceSource("/api/resources?source=vmware-vsphere", unifiedresources.SourceVMware) + + assertResourceCount := func(path string, want int) { + t.Helper() + + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodGet, path, nil) + router.resourceHandlers.HandleListResources(rec, req) + if rec.Code != http.StatusOK { + t.Fatalf("%s status = %d, body=%s", path, rec.Code, rec.Body.String()) + } + + var resp ResourcesResponse + if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil { + t.Fatalf("decode %s response: %v", path, err) + } + if len(resp.Data) != want { + t.Fatalf("%s returned %d resources, want %d", path, len(resp.Data), want) + } + } + + assertResourceCount("/api/resources?source=truenas&type=app-container", len(truenas.DefaultFixtures().Apps)) + assertResourceCount("/api/resources?source=vmware-vsphere&type=storage", len(vmware.DefaultFixtures().Datastores)) } func TestContract_PlatformMockConnectionListsUseSharedFixtureMetadata(t *testing.T) { diff --git a/internal/api/resources_test.go b/internal/api/resources_test.go index 2e2a5f7fb..8734d2c58 100644 --- a/internal/api/resources_test.go +++ b/internal/api/resources_test.go @@ -2966,11 +2966,12 @@ func TestResourceListIncludesTrueNASAppsAsAppContainers(t *testing.T) { truenas.SetFeatureEnabled(previous) }) + fixtures := truenas.DefaultFixtures() cfg := &config.Config{DataPath: t.TempDir()} h := NewResourceHandlers(cfg) h.SetStateProvider(resourceStateProvider{snapshot: models.StateSnapshot{LastUpdate: time.Now().UTC()}}) h.SetSupplementalRecordsProvider(unified.SourceTrueNAS, mockSupplementalRecordsProvider{ - records: truenas.NewProvider(truenas.DefaultFixtures()).Records(), + records: truenas.NewProvider(fixtures).Records(), ownedSources: []unified.DataSource{unified.SourceTrueNAS}, }) @@ -2986,8 +2987,8 @@ func TestResourceListIncludesTrueNASAppsAsAppContainers(t *testing.T) { if err := json.NewDecoder(rec.Body).Decode(&resp); err != nil { t.Fatalf("decode response: %v", err) } - if len(resp.Data) != 2 { - t.Fatalf("expected 2 TrueNAS app-container resources, got %d", len(resp.Data)) + if len(resp.Data) != len(fixtures.Apps) { + t.Fatalf("expected %d TrueNAS app-container resources, got %d", len(fixtures.Apps), len(resp.Data)) } var nextcloud *unified.Resource diff --git a/internal/api/router.go b/internal/api/router.go index 74f57c06c..bb97bf811 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -5183,54 +5183,55 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) { guestTypes[sid] = "system-container" } } + for _, dc := range readState.DockerContainers() { + if dc == nil { + continue + } + if key := strings.TrimSpace(dc.ID()); key != "" { + guestTypes[key] = "app-container" + } + } // Process Docker containers - batch-load historical data (1-2 SQL calls instead of N). dockerData := make(map[string]VMChartData) dcList := readState.DockerContainers() dcRequests := make([]monitoring.GuestChartRequest, 0, len(dcList)) for _, dc := range dcList { - if dc == nil { + _, request, ok := appContainerChartRequest(dc) + if !ok { continue } - if dcID := dc.ContainerID(); dcID != "" { - dcRequests = append(dcRequests, monitoring.GuestChartRequest{ - InMemoryKey: fmt.Sprintf("docker:%s", dcID), - SQLResourceID: dcID, - }) - } + dcRequests = append(dcRequests, request) } dcBatchMetrics := monitor.GetGuestMetricsForChartBatch("dockerContainer", dcRequests, duration) for _, dc := range dcList { - if dc == nil { + responseKey, request, ok := appContainerChartRequest(dc) + if !ok { continue } - dcContainerID := dc.ContainerID() - if dcContainerID == "" { - continue - } - dockerData[dcContainerID] = make(VMChartData) - if batchMetrics, ok := dcBatchMetrics[dcContainerID]; ok { + dockerData[responseKey] = make(VMChartData) + if batchMetrics, ok := dcBatchMetrics[request.SQLResourceID]; ok { for metricType, points := range batchMetrics { if !sparklineMetrics[metricType] { continue } - dockerData[dcContainerID][metricType] = make([]MetricPoint, len(points)) + dockerData[responseKey][metricType] = make([]MetricPoint, len(points)) for i, point := range points { ts := point.Timestamp.Unix() * 1000 if ts < oldestTimestamp { oldestTimestamp = ts } - dockerData[dcContainerID][metricType][i] = MetricPoint{ + dockerData[responseKey][metricType][i] = MetricPoint{ Timestamp: ts, Value: point.Value, } } } } - if len(dockerData[dcContainerID]["cpu"]) == 0 { - dockerData[dcContainerID]["cpu"] = []MetricPoint{{Timestamp: currentTime, Value: dc.CPUPercent()}} - dockerData[dcContainerID]["memory"] = []MetricPoint{{Timestamp: currentTime, Value: dc.MemoryPercent()}} - dockerData[dcContainerID]["disk"] = []MetricPoint{{Timestamp: currentTime, Value: dc.DiskPercent()}} + if len(dockerData[responseKey]["cpu"]) == 0 { + dockerData[responseKey]["cpu"] = []MetricPoint{{Timestamp: currentTime, Value: dc.CPUPercent()}} + dockerData[responseKey]["memory"] = []MetricPoint{{Timestamp: currentTime, Value: dc.MemoryPercent()}} + dockerData[responseKey]["disk"] = []MetricPoint{{Timestamp: currentTime, Value: dc.DiskPercent()}} } } @@ -5293,27 +5294,20 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) { hostList := readState.Hosts() agentRequests := make([]monitoring.GuestChartRequest, 0, len(hostList)) for _, h := range hostList { - if h == nil { + _, request, ok := hostAgentChartRequest(h) + if !ok { continue } - if hID := h.AgentID(); hID != "" { - agentRequests = append(agentRequests, monitoring.GuestChartRequest{ - InMemoryKey: fmt.Sprintf("agent:%s", hID), - SQLResourceID: hID, - }) - } + agentRequests = append(agentRequests, request) } agentBatchMetrics := monitor.GetGuestMetricsForChartBatch("agent", agentRequests, duration) for _, h := range hostList { - if h == nil { - continue - } - hID := h.AgentID() - if hID == "" { + hID, request, ok := hostAgentChartRequest(h) + if !ok { continue } agentData[hID] = make(VMChartData) - if batchMetrics, ok := agentBatchMetrics[hID]; ok { + if batchMetrics, ok := agentBatchMetrics[request.SQLResourceID]; ok { for metricType, points := range batchMetrics { if !sparklineMetrics[metricType] { continue @@ -5445,6 +5439,68 @@ func parseWorkloadMaxPoints(raw string) int { return value } +func hostAgentChartRequest(host *unifiedresources.HostView) (string, monitoring.GuestChartRequest, bool) { + if host == nil { + return "", monitoring.GuestChartRequest{}, false + } + + if agentID := strings.TrimSpace(host.AgentID()); agentID != "" { + return agentID, monitoring.GuestChartRequest{ + InMemoryKey: fmt.Sprintf("agent:%s", agentID), + SQLResourceID: agentID, + }, true + } + + target := host.MetricsTarget() + if target == nil { + return "", monitoring.GuestChartRequest{}, false + } + + metricID := strings.TrimSpace(target.ResourceID) + if metricID == "" { + return "", monitoring.GuestChartRequest{}, false + } + + return metricID, monitoring.GuestChartRequest{ + InMemoryKey: fmt.Sprintf("agent:%s", metricID), + SQLResourceID: metricID, + }, true +} + +func appContainerChartMetricID(container *unifiedresources.DockerContainerView) string { + if container == nil { + return "" + } + + if target := container.MetricsTarget(); target != nil { + if metricID := strings.TrimSpace(target.ResourceID); metricID != "" { + return metricID + } + } + + return strings.TrimSpace(container.ContainerID()) +} + +func appContainerChartRequest(container *unifiedresources.DockerContainerView) (string, monitoring.GuestChartRequest, bool) { + if container == nil { + return "", monitoring.GuestChartRequest{}, false + } + + responseKey := strings.TrimSpace(container.ID()) + if responseKey == "" { + responseKey = strings.TrimSpace(container.ContainerID()) + } + metricID := appContainerChartMetricID(container) + if responseKey == "" || metricID == "" { + return "", monitoring.GuestChartRequest{}, false + } + + return responseKey, monitoring.GuestChartRequest{ + InMemoryKey: fmt.Sprintf("docker:%s", metricID), + SQLResourceID: metricID, + }, true +} + func capMetricPointSeries(points []MetricPoint, maxPoints int) []MetricPoint { if len(points) <= maxPoints || maxPoints <= 0 { return points @@ -5783,6 +5839,26 @@ func (r *Router) handleWorkloadCharts(w http.ResponseWriter, req *http.Request) strings.EqualFold(strings.TrimSpace(host.Name()), nodeName) } + matchesSelectedAgentHostView := func(host *unifiedresources.HostView) bool { + if selectedNodeID == "" { + return true + } + if selectedNode == nil { + return true + } + if host == nil { + return false + } + nodeName := strings.TrimSpace(selectedNode.Name) + if nodeName == "" { + return false + } + return strings.EqualFold(strings.TrimSpace(host.Hostname()), nodeName) || + strings.EqualFold(strings.TrimSpace(host.Name()), nodeName) || + strings.EqualFold(strings.TrimSpace(host.AgentID()), nodeName) || + strings.EqualFold(strings.TrimSpace(host.ID()), nodeName) + } + matchesSelectedKubernetesPodView := func(pod *unifiedresources.PodView) bool { if selectedNodeID == "" { return true @@ -5917,9 +5993,17 @@ func (r *Router) handleWorkloadCharts(w http.ResponseWriter, req *http.Request) } dockerHostsByID[host.ID()] = host } + agentHostsByID := make(map[string]*unifiedresources.HostView, len(readState.Hosts())) + for _, host := range readState.Hosts() { + if host == nil { + continue + } + agentHostsByID[host.ID()] = host + } dockerContainerList := make([]*unifiedresources.DockerContainerView, 0) dockerContainerRequests := make([]monitoring.GuestChartRequest, 0) + dockerContainerKeys := make([]string, 0) for _, container := range readState.DockerContainers() { if container == nil { continue @@ -5927,25 +6011,32 @@ func (r *Router) handleWorkloadCharts(w http.ResponseWriter, req *http.Request) if selectedNodeID != "" && selectedNode != nil { host := dockerHostsByID[container.ParentID()] - if host == nil || !matchesSelectedDockerHostView(host) { - continue + if host != nil { + if !matchesSelectedDockerHostView(host) { + continue + } + } else { + agentHost := agentHostsByID[container.ParentID()] + if agentHost == nil || !matchesSelectedAgentHostView(agentHost) { + continue + } } } - containerID := strings.TrimSpace(container.ContainerID()) - if containerID == "" { + responseKey, request, ok := appContainerChartRequest(container) + if !ok { continue } dockerContainerList = append(dockerContainerList, container) - dockerContainerRequests = append(dockerContainerRequests, monitoring.GuestChartRequest{ - InMemoryKey: fmt.Sprintf("docker:%s", containerID), - SQLResourceID: containerID, - }) + dockerContainerKeys = append(dockerContainerKeys, responseKey) + dockerContainerRequests = append(dockerContainerRequests, request) + guestTypes[responseKey] = "app-container" } dockerContainerBatchMetrics := monitor.GetGuestMetricsForChartBatch("dockerContainer", dockerContainerRequests, duration) - for _, container := range dockerContainerList { - containerID := strings.TrimSpace(container.ContainerID()) - series := convertMetricsForChart(dockerContainerBatchMetrics[containerID], &oldestTimestamp, maxPoints) + for idx, container := range dockerContainerList { + responseKey := dockerContainerKeys[idx] + metricID := dockerContainerRequests[idx].SQLResourceID + series := convertMetricsForChart(dockerContainerBatchMetrics[metricID], &oldestTimestamp, maxPoints) if len(series["cpu"]) == 0 { series["cpu"] = []MetricPoint{{Timestamp: currentTime, Value: container.CPUPercent()}} @@ -5955,7 +6046,7 @@ func (r *Router) handleWorkloadCharts(w http.ResponseWriter, req *http.Request) series["netout"] = []MetricPoint{{Timestamp: currentTime, Value: container.NetOutRate()}} } updateOldestTimestampFromSeries(series, &oldestTimestamp) - dockerData[containerID] = series + dockerData[responseKey] = series } countChartPoints := func(metricsMap map[string]VMChartData) int { @@ -6192,27 +6283,20 @@ func (r *Router) handleInfrastructureCharts(w http.ResponseWriter, req *http.Req hostList := readState.Hosts() agentRequests := make([]monitoring.GuestChartRequest, 0, len(hostList)) for _, h := range hostList { - if h == nil { + _, request, ok := hostAgentChartRequest(h) + if !ok { continue } - if hID := h.AgentID(); hID != "" { - agentRequests = append(agentRequests, monitoring.GuestChartRequest{ - InMemoryKey: fmt.Sprintf("agent:%s", hID), - SQLResourceID: hID, - }) - } + agentRequests = append(agentRequests, request) } agentBatchMetrics := monitor.GetGuestMetricsForChartBatch("agent", agentRequests, duration) for _, h := range hostList { - if h == nil { - continue - } - hID := h.AgentID() - if hID == "" { + hID, request, ok := hostAgentChartRequest(h) + if !ok { continue } agentData[hID] = make(VMChartData) - if batchMetrics, ok := agentBatchMetrics[hID]; ok { + if batchMetrics, ok := agentBatchMetrics[request.SQLResourceID]; ok { for metricType, points := range batchMetrics { if !sparklineMetrics[metricType] { continue diff --git a/internal/mock/fixture_graph.go b/internal/mock/fixture_graph.go index f2b51f33f..074d38a0e 100644 --- a/internal/mock/fixture_graph.go +++ b/internal/mock/fixture_graph.go @@ -31,7 +31,7 @@ func buildFixtureGraph(cfg MockConfig, now time.Time) FixtureGraph { return FixtureGraph{ State: state, AlertHistory: buildAlertHistory(state.Nodes, state.VMs, state.Containers), - PlatformFixtures: defaultPlatformFixtures(), + PlatformFixtures: rebasePlatformFixtures(defaultPlatformFixtures(), now), } } @@ -49,6 +49,7 @@ func (g *FixtureGraph) UpdateMetrics(cfg MockConfig, now time.Time) { } updateFixtureStateMetrics(&g.State, cfg) g.State.LastUpdate = now + g.PlatformFixtures = rebasePlatformFixtures(g.PlatformFixtures, now) } func (g *FixtureGraph) UpdateAlertSnapshots(active []alerts.Alert, resolved []models.ResolvedAlert) { diff --git a/internal/mock/generator.go b/internal/mock/generator.go index 770518286..032148533 100644 --- a/internal/mock/generator.go +++ b/internal/mock/generator.go @@ -58,18 +58,18 @@ const ( ) var DefaultConfig = MockConfig{ - NodeCount: 7, // Test the 5-9 node range by default - VMsPerNode: 5, - LXCsPerNode: 8, - DockerHostCount: 3, - DockerContainersPerHost: 12, - GenericHostCount: 4, - K8sClusterCount: 2, - K8sNodesPerCluster: 4, - K8sPodsPerCluster: 30, - K8sDeploymentsPerCluster: 12, + NodeCount: 3, + VMsPerNode: 3, + LXCsPerNode: 3, + DockerHostCount: 2, + DockerContainersPerHost: 5, + GenericHostCount: 2, + K8sClusterCount: 1, + K8sNodesPerCluster: 3, + K8sPodsPerCluster: 10, + K8sDeploymentsPerCluster: 4, RandomMetrics: true, - StoppedPercent: 0.2, + StoppedPercent: 0.06, } var appNames = []string{ @@ -711,26 +711,7 @@ func generateNodes(config MockConfig) []models.Node { // ID format matches real system: instance-nodename node.ID = fmt.Sprintf("%s-%s", node.Instance, nodeName) - // Make pve3 offline to test offline node handling - if nodeName == "pve3" { - node.Status = "offline" - node.CPU = 0 - node.Memory.Used = 0 - node.Memory.Usage = 0 - node.Memory.Free = node.Memory.Total - node.Uptime = 0 - node.LoadAverage = []float64{0, 0, 0} - node.Disk.Used = 0 - node.Disk.Usage = 0 - node.Disk.Free = node.Disk.Total - if node.Temperature != nil { - node.Temperature = &models.Temperature{Available: false} - } - node.ConnectionHealth = "offline" - } else { - // For cluster nodes, since one is offline, the cluster is degraded - node.ConnectionHealth = "degraded" - } + node.ConnectionHealth = "healthy" nodes = append(nodes, node) } @@ -1238,14 +1219,7 @@ func generateKubernetesClusters(config MockConfig) []models.KubernetesCluster { lastSeen := now.Add(-time.Duration(rand.Intn(20)) * time.Second) status := "online" - // Make the last cluster offline occasionally for UI coverage. - if clusterCount > 1 && i == clusterCount-1 && rand.Float64() < 0.55 { - status = "offline" - lastSeen = now.Add(-time.Duration(5+rand.Intn(20)) * time.Minute) - for nodeIdx := range nodes { - nodes[nodeIdx].Ready = false - } - } else if clusterHasIssues(nodes, pods, deployments) { + if clusterHasIssues(nodes, pods, deployments) { status = "degraded" } @@ -2044,67 +2018,31 @@ func generateDockerHosts(config MockConfig) []models.DockerHost { containers := generateDockerContainers(hostname, i, config, isPodman) - // Optionally ensure the second host looks degraded for UI coverage + // Keep one host mildly degraded so the demo still shows issue handling. if i == 1 && len(containers) > 0 && hostCount > 1 { idx := rand.Intn(len(containers)) containers[idx].Health = "unhealthy" - containers[idx].CPUPercent = clampFloat(containers[idx].CPUPercent+35, 5, 190) + containers[idx].CPUPercent = clampFloat(containers[idx].CPUPercent+18, 5, 190) } - // Determine initial status - only mark last host as offline for testing status := "online" - explicitlyOffline := false - if hostCount > 2 && i == hostCount-1 { - status = "offline" - explicitlyOffline = true + lastSeen := now.Add(-time.Duration(rand.Intn(20)) * time.Second) + running := 0 + unhealthy := 0 + for _, ct := range containers { + if strings.ToLower(ct.State) == "running" { + running++ + healthState := strings.ToLower(ct.Health) + if healthState == "unhealthy" || healthState == "starting" { + unhealthy++ + } + } } - lastSeen := now.Add(-time.Duration(rand.Intn(20)) * time.Second) - if explicitlyOffline { - // If host is explicitly offline, stop all containers and update metrics - lastSeen = now.Add(-time.Duration(5+rand.Intn(20)) * time.Minute) - for idx := range containers { - exitCode := containers[idx].ExitCode - if exitCode == 0 { - exitCode = []int{0, 1, 137}[rand.Intn(3)] - } - containers[idx].State = "exited" - containers[idx].Health = "" - containers[idx].CPUPercent = 0 - containers[idx].MemoryUsage = 0 - containers[idx].MemoryPercent = 0 - containers[idx].UptimeSeconds = 0 - finished := now.Add(-time.Duration(rand.Intn(72)+1) * time.Hour) - containers[idx].StartedAt = nil - containers[idx].FinishedAt = &finished - containers[idx].Status = fmt.Sprintf("Exited (%d) %s ago", exitCode, formatDurationForStatus(now.Sub(finished))) - } - } else { - // For hosts not explicitly offline, calculate status based on container health - running := 0 - unhealthy := 0 - for _, ct := range containers { - if strings.ToLower(ct.State) == "running" { - running++ - healthState := strings.ToLower(ct.Health) - if healthState == "unhealthy" || healthState == "starting" { - unhealthy++ - } - } - } - - // Only mark as offline if there are containers but none are running - // (Swarm-only hosts with no standalone containers should stay online) - if len(containers) > 0 && running == 0 { - status = "offline" - lastSeen = now.Add(-time.Duration(3+rand.Intn(5)) * time.Minute) - } else if unhealthy > 0 || (len(containers) > 0 && float64(len(containers)-running)/float64(len(containers)) > 0.35) { - status = "degraded" - if lastSeen.After(now.Add(-30 * time.Second)) { - lastSeen = now.Add(-35 * time.Second) - } - } else { - status = "online" + if unhealthy > 0 || (len(containers) > 0 && float64(len(containers)-running)/float64(len(containers)) > 0.45) { + status = "degraded" + if lastSeen.After(now.Add(-30 * time.Second)) { + lastSeen = now.Add(-35 * time.Second) } } @@ -2400,9 +2338,7 @@ func generateHosts(config MockConfig) []models.Host { } status := "online" - if rand.Float64() < 0.1 { - status = "offline" - } else if rand.Float64() < 0.12 { + if rand.Float64() < 0.04 { status = "degraded" } @@ -3811,8 +3747,6 @@ func generateZFSPoolWithIssues(poolName string) *models.ZFSPool { // generateStorage generates mock storage data for nodes func generateStorage(nodes []models.Node) []models.Storage { var storage []models.Storage - storageTypes := []string{"dir", "zfspool", "lvm", "nfs", "cephfs"} - contentTypes := []string{"images", "vztmpl,iso", "rootdir", "backup", "snippets"} for _, node := range nodes { isOffline := node.Status != "online" || node.ConnectionHealth == "offline" || node.Uptime <= 0 @@ -3918,47 +3852,9 @@ func generateStorage(nodes []models.Node) []models.Storage { ZFSPool: zfsPool, }) - // Add one more random storage per node - if rand.Float64() > 0.3 { - storageType := storageTypes[rand.Intn(len(storageTypes))] - storageName := fmt.Sprintf("storage-%s-%d", node.Name, rand.Intn(100)) - total := int64((1 + rand.Intn(10)) * 1024 * 1024 * 1024 * 1024) // 1-10TB - used := int64(float64(total) * rand.Float64()) - if isOffline { - used = 0 - } - free := total - used - usage := 0.0 - if total > 0 { - usage = float64(used) / float64(total) * 100 - } - - status := "available" - enabled := true - active := true - if isOffline { - status = "offline" - enabled = false - active = false - } - - storage = append(storage, models.Storage{ - ID: fmt.Sprintf("%s-%s-%s", node.Instance, node.Name, storageName), - Name: storageName, - Node: node.Name, - Instance: node.Instance, - Type: storageType, - Status: status, - Total: total, - Used: used, - Free: free, - Usage: usage, - Content: contentTypes[rand.Intn(len(contentTypes))], - Shared: storageType == "nfs" || storageType == "cephfs", - Enabled: enabled, - Active: active, - }) - } + // Keep per-node storage intentionally compact in demo mode. A broad + // random storage fan-out makes the Storage page read like synthetic lab + // noise rather than a curated estate. } // Add PBS storage for each node (simulating node-specific PBS namespaces) @@ -3996,59 +3892,6 @@ func generateStorage(nodes []models.Node) []models.Storage { } } - if len(clusterNodes) > 0 { - nodeNames := make([]string, 0, len(clusterNodes)) - nodeIDs := make([]string, 0, len(clusterNodes)) - for _, clusterNode := range clusterNodes { - nodeNames = append(nodeNames, clusterNode.Name) - nodeIDs = append(nodeIDs, fmt.Sprintf("%s-%s", clusterNode.Instance, clusterNode.Name)) - } - - cephTotal := int64(120 * 1024 * 1024 * 1024 * 1024) // 120 TiB shared CephFS - cephUsed := int64(float64(cephTotal) * (0.52 + rand.Float64()*0.18)) - storage = append(storage, models.Storage{ - ID: fmt.Sprintf("%s-shared-cephfs", clusterNodes[0].Instance), - Name: "cephfs-shared", - Node: "shared", - Instance: clusterNodes[0].Instance, - Type: "cephfs", - Status: "available", - Total: cephTotal, - Used: cephUsed, - Free: cephTotal - cephUsed, - Usage: float64(cephUsed) / float64(cephTotal) * 100, - Content: "images,rootdir,backup", - Shared: true, - Enabled: true, - Active: true, - Nodes: nodeNames, - NodeIDs: nodeIDs, - NodeCount: len(nodeNames), - }) - - rbdTotal := int64(80 * 1024 * 1024 * 1024 * 1024) // 80 TiB shared RBD pool - rbdUsed := int64(float64(rbdTotal) * (0.48 + rand.Float64()*0.22)) - storage = append(storage, models.Storage{ - ID: fmt.Sprintf("%s-shared-rbd", clusterNodes[0].Instance), - Name: "ceph-rbd-pool", - Node: "shared", - Instance: clusterNodes[0].Instance, - Type: "rbd", - Status: "available", - Total: rbdTotal, - Used: rbdUsed, - Free: rbdTotal - rbdUsed, - Usage: float64(rbdUsed) / float64(rbdTotal) * 100, - Content: "images,rootdir", - Shared: true, - Enabled: true, - Active: true, - Nodes: nodeNames, - NodeIDs: nodeIDs, - NodeCount: len(nodeNames), - }) - } - // Add a shared storage (NFS or CephFS) if len(nodes) > 1 { sharedTotal := int64(10 * 1024 * 1024 * 1024 * 1024) // 10TB @@ -4806,7 +4649,63 @@ func updateFixtureStateMetrics(data *models.StateSnapshot, config MockConfig) { updateHosts(data, config) syncMockKubernetesNodeHosts(data) + refreshNow := time.Now() + step := int64(updateInterval.Seconds()) + if step <= 0 { + step = 2 + } + + for i := range data.PBSInstances { + inst := &data.PBSInstances[i] + inst.Status = "online" + inst.ConnectionHealth = "healthy" + inst.LastSeen = refreshNow.Add(-time.Duration(randIntnSafe(12)) * time.Second) + inst.Uptime += step + + if data.ConnectionHealth != nil { + data.ConnectionHealth[fmt.Sprintf("pbs-%s", inst.Name)] = true + } + + if !config.RandomMetrics { + continue + } + + inst.CPU = naturalMetricUpdate(inst.CPU, 6, 62, inst.ID, "cpu", 0.4) + inst.Memory = naturalMetricUpdate(inst.Memory, 24, 78, inst.ID, "memory", 0.25) + if inst.MemoryTotal > 0 { + inst.MemoryUsed = int64(float64(inst.MemoryTotal) * (inst.Memory / 100.0)) + if inst.MemoryUsed < 0 { + inst.MemoryUsed = 0 + } + } + + for j := range inst.Datastores { + datastore := &inst.Datastores[j] + if datastore.Total <= 0 { + continue + } + + datastore.Usage = naturalMetricUpdate(datastore.Usage, 12, 78, inst.ID+":"+datastore.Name, "usage", 0.08) + datastore.Used = int64(float64(datastore.Total) * (datastore.Usage / 100.0)) + if datastore.Used < 0 { + datastore.Used = 0 + } + if datastore.Used > datastore.Total { + datastore.Used = datastore.Total + } + datastore.Free = datastore.Total - datastore.Used + datastore.Status = "available" + } + } + + for i := range data.PMGInstances { + inst := &data.PMGInstances[i] + inst.LastSeen = refreshNow + inst.LastUpdated = refreshNow + } + if !config.RandomMetrics { + data.LastUpdate = refreshNow return } diff --git a/internal/mock/integration.go b/internal/mock/integration.go index 5fb8c312e..417fd341f 100644 --- a/internal/mock/integration.go +++ b/internal/mock/integration.go @@ -29,6 +29,11 @@ var ( const updateInterval = 2 * time.Second func init() { + loadedConfig := normalizeMockConfig(LoadMockConfig()) + dataMu.Lock() + mockConfig = loadedConfig + dataMu.Unlock() + initialEnabled := mockModeFromEnv() if initialEnabled { log.Info().Msg("mock mode enabled at startup") @@ -59,8 +64,12 @@ func setEnabled(enable bool, fromInit bool) { return } + dataMu.RLock() + config := mockConfig + dataMu.RUnlock() + if enable { - enableMockMode(fromInit) + enableMockMode(config, fromInit) } else { disableMockMode() } @@ -102,8 +111,8 @@ func readMockEnv(name string) (string, bool) { return value, true } -func enableMockMode(fromInit bool) { - config := LoadMockConfig() +func enableMockMode(config MockConfig, fromInit bool) { + config = normalizeMockConfig(config) now := time.Now() dataMu.Lock() diff --git a/internal/mock/integration_coverage_test.go b/internal/mock/integration_coverage_test.go index d9c63d1f9..af06e5e54 100644 --- a/internal/mock/integration_coverage_test.go +++ b/internal/mock/integration_coverage_test.go @@ -260,6 +260,35 @@ func TestSetMockConfigOnlyRegeneratesWhenEnabled(t *testing.T) { } } +func TestSetEnabledUsesCurrentConfigInsteadOfReloadingEnv(t *testing.T) { + resetMockConfigEnv(t) + + prevEnabled := IsMockEnabled() + prevCfg := GetConfig() + t.Cleanup(func() { + SetMockConfig(prevCfg) + SetEnabled(prevEnabled) + }) + + t.Setenv("PULSE_MOCK_NODES", "9") + t.Setenv("PULSE_MOCK_VMS_PER_NODE", "7") + t.Setenv("PULSE_MOCK_LXCS_PER_NODE", "6") + + SetEnabled(false) + + custom := DefaultConfig + custom.NodeCount = 2 + custom.VMsPerNode = 1 + custom.LXCsPerNode = 1 + SetMockConfig(custom) + SetEnabled(true) + + got := GetConfig() + if got.NodeCount != custom.NodeCount || got.VMsPerNode != custom.VMsPerNode || got.LXCsPerNode != custom.LXCsPerNode { + t.Fatalf("mock config after enable = %+v, want node/vm/lxc counts from current config %+v", got, custom) + } +} + func TestUpdateMetricsGuardAndTimestamp(t *testing.T) { resetMockIntegrationState(t) @@ -298,6 +327,50 @@ func TestUpdateMetricsGuardAndTimestamp(t *testing.T) { } } +func TestUpdateMetricsRefreshesRuntimeFreshnessForPBSAndPlatformFixtures(t *testing.T) { + resetMockIntegrationState(t) + + cfg := DefaultConfig + cfg.NodeCount = 1 + cfg.VMsPerNode = 0 + cfg.LXCsPerNode = 0 + cfg.DockerHostCount = 0 + cfg.GenericHostCount = 0 + cfg.K8sClusterCount = 0 + cfg.RandomMetrics = false + + dataMu.Lock() + mockGraph = buildFixtureGraph(cfg, time.Time{}) + for i := range mockGraph.State.PBSInstances { + mockGraph.State.PBSInstances[i].LastSeen = time.Time{} + } + mockGraph.PlatformFixtures.TrueNAS.CollectedAt = time.Time{} + mockGraph.PlatformFixtures.TrueNAS.System.CollectedAt = time.Time{} + mockGraph.PlatformFixtures.VMware.CollectedAt = time.Time{} + dataMu.Unlock() + + enabled.Store(true) + updateMetrics(cfg) + + dataMu.RLock() + defer dataMu.RUnlock() + + if len(mockGraph.State.PBSInstances) == 0 { + t.Fatal("expected PBS fixtures in canonical mock state") + } + for _, inst := range mockGraph.State.PBSInstances { + if inst.LastSeen.IsZero() { + t.Fatalf("expected PBS instance %s to refresh lastSeen", inst.Name) + } + } + if got := trueNASCollectedAt(mockGraph.PlatformFixtures.TrueNAS); got.IsZero() { + t.Fatal("expected TrueNAS fixture freshness to refresh") + } + if got := mockGraph.PlatformFixtures.VMware.CollectedAt; got.IsZero() { + t.Fatal("expected VMware fixture freshness to refresh") + } +} + func TestUpdateLoopStartStopLifecycle(t *testing.T) { resetMockIntegrationState(t) diff --git a/internal/mock/platform_fixtures.go b/internal/mock/platform_fixtures.go index 2fa3c7d45..a9459846b 100644 --- a/internal/mock/platform_fixtures.go +++ b/internal/mock/platform_fixtures.go @@ -58,6 +58,14 @@ func defaultPlatformFixtures() PlatformFixtures { } } +func rebasePlatformFixtures(fixtures PlatformFixtures, now time.Time) PlatformFixtures { + target := now.UTC().Truncate(time.Minute) + return PlatformFixtures{ + TrueNAS: rebaseTrueNASPlatformFixture(fixtures.TrueNAS, target), + VMware: rebaseVMwarePlatformFixture(fixtures.VMware, target), + } +} + func DefaultTrueNASConnectionFixture() TrueNASConnectionFixture { return defaultTrueNASConnectionFixture(currentOrDefaultPlatformFixtures()) } @@ -159,7 +167,14 @@ func (g FixtureGraph) UnifiedResourceSnapshot() ([]unifiedresources.Resource, ti } } - return registry.List(), freshness + resources := registry.List() + for i := range resources { + if target := registry.MetricsTarget(resources[i].ID); target != nil { + resources[i].MetricsTarget = target + } + } + + return resources, freshness } func (g FixtureGraph) SupplementalRecords(source unifiedresources.DataSource) []unifiedresources.IngestRecord { @@ -180,6 +195,95 @@ func trueNASCollectedAt(fixtures truenas.FixtureSnapshot) time.Time { return fixtures.System.CollectedAt } +func rebaseTrueNASPlatformFixture(snapshot truenas.FixtureSnapshot, target time.Time) truenas.FixtureSnapshot { + out := cloneTrueNASFixtureSnapshot(snapshot) + anchor := trueNASCollectedAt(snapshot) + if anchor.IsZero() { + anchor = target + } + shift := target.Sub(anchor) + + out.CollectedAt = target + out.System.CollectedAt = shiftTime(snapshot.System.CollectedAt, shift, target) + + for i := range out.Alerts { + out.Alerts[i].Datetime = shiftTime(snapshot.Alerts[i].Datetime, shift, target) + } + for i := range out.Apps { + if out.Apps[i].Stats != nil { + out.Apps[i].Stats.CollectedAt = shiftTime(snapshot.Apps[i].Stats.CollectedAt, shift, target) + } + } + for i := range out.ZFSSnapshots { + if snapshot.ZFSSnapshots[i].CreatedAt != nil { + rebased := shiftTime(*snapshot.ZFSSnapshots[i].CreatedAt, shift, target) + out.ZFSSnapshots[i].CreatedAt = &rebased + } + } + for i := range out.ReplicationTasks { + if snapshot.ReplicationTasks[i].LastRun != nil { + rebased := shiftTime(*snapshot.ReplicationTasks[i].LastRun, shift, target) + out.ReplicationTasks[i].LastRun = &rebased + } + } + + return out +} + +func rebaseVMwarePlatformFixture(snapshot vmware.InventorySnapshot, target time.Time) vmware.InventorySnapshot { + out := cloneVMwareInventorySnapshot(snapshot) + anchor := snapshot.CollectedAt + if anchor.IsZero() { + anchor = target + } + shift := target.Sub(anchor) + + out.CollectedAt = target + for i := range out.Hosts { + rebaseVMwareAlarms(out.Hosts[i].TriggeredAlarms, snapshot.Hosts[i].TriggeredAlarms, shift, target) + rebaseVMwareTasks(out.Hosts[i].RecentTasks, snapshot.Hosts[i].RecentTasks, shift, target) + rebaseVMwareEvents(out.Hosts[i].RecentEvents, snapshot.Hosts[i].RecentEvents, shift, target) + } + for i := range out.VMs { + rebaseVMwareAlarms(out.VMs[i].TriggeredAlarms, snapshot.VMs[i].TriggeredAlarms, shift, target) + rebaseVMwareTasks(out.VMs[i].RecentTasks, snapshot.VMs[i].RecentTasks, shift, target) + rebaseVMwareEvents(out.VMs[i].RecentEvents, snapshot.VMs[i].RecentEvents, shift, target) + } + for i := range out.Datastores { + rebaseVMwareAlarms(out.Datastores[i].TriggeredAlarms, snapshot.Datastores[i].TriggeredAlarms, shift, target) + rebaseVMwareTasks(out.Datastores[i].RecentTasks, snapshot.Datastores[i].RecentTasks, shift, target) + rebaseVMwareEvents(out.Datastores[i].RecentEvents, snapshot.Datastores[i].RecentEvents, shift, target) + } + + return out +} + +func rebaseVMwareAlarms(out []vmware.InventoryAlarm, in []vmware.InventoryAlarm, shift time.Duration, target time.Time) { + for i := range out { + out[i].TriggeredAt = shiftTime(in[i].TriggeredAt, shift, target) + } +} + +func rebaseVMwareTasks(out []vmware.InventoryTask, in []vmware.InventoryTask, shift time.Duration, target time.Time) { + for i := range out { + out[i].StartedAt = shiftTime(in[i].StartedAt, shift, target) + out[i].CompletedAt = shiftTime(in[i].CompletedAt, shift, target) + } +} + +func rebaseVMwareEvents(out []vmware.InventoryEvent, in []vmware.InventoryEvent, shift time.Duration, target time.Time) { + for i := range out { + out[i].CreatedAt = shiftTime(in[i].CreatedAt, shift, target) + } +} + +func shiftTime(value time.Time, shift time.Duration, fallback time.Time) time.Time { + if value.IsZero() { + return fallback + } + return value.Add(shift) +} + func normalizePlatformSource(source unifiedresources.DataSource) unifiedresources.DataSource { switch strings.ToLower(strings.TrimSpace(string(source))) { case "truenas": diff --git a/internal/mock/platform_fixtures_test.go b/internal/mock/platform_fixtures_test.go index 1427426c8..70c71a00b 100644 --- a/internal/mock/platform_fixtures_test.go +++ b/internal/mock/platform_fixtures_test.go @@ -2,6 +2,7 @@ package mock import ( "testing" + "time" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" ) @@ -56,3 +57,49 @@ func TestSupplementalRecordsNormalizesVMwareAlias(t *testing.T) { t.Fatal("expected records for vmware-vsphere alias") } } + +func TestBuildFixtureGraphRebasesPlatformFixtureTimestampsForDemoRuntime(t *testing.T) { + now := time.Date(2026, time.March, 31, 17, 30, 0, 0, time.UTC) + + graph := buildFixtureGraph(DefaultConfig, now) + + if got := trueNASCollectedAt(graph.PlatformFixtures.TrueNAS); !got.Equal(now) { + t.Fatalf("expected TrueNAS collectedAt %s, got %s", now, got) + } + if got := graph.PlatformFixtures.VMware.CollectedAt; !got.Equal(now) { + t.Fatalf("expected VMware collectedAt %s, got %s", now, got) + } + if got := graph.PlatformFixtures.TrueNAS.System.CollectedAt; got.IsZero() || got.Before(now.Add(-2*time.Minute)) || got.After(now) { + t.Fatalf("expected rebased TrueNAS system collectedAt near %s, got %s", now, got) + } + if len(graph.PlatformFixtures.VMware.Hosts) == 0 || len(graph.PlatformFixtures.VMware.Hosts[0].RecentEvents) == 0 { + t.Fatal("expected canonical VMware fixtures with recent events") + } + if got := graph.PlatformFixtures.VMware.Hosts[0].RecentEvents[0].CreatedAt; got.IsZero() || got.Before(now.Add(-2*time.Hour)) || got.After(now) { + t.Fatalf("expected rebased VMware event timestamp near %s, got %s", now, got) + } +} + +func TestFixtureGraphUpdateMetricsKeepsPlatformFixtureFreshnessCurrent(t *testing.T) { + cfg := DefaultConfig + cfg.RandomMetrics = false + + start := time.Date(2026, time.March, 31, 17, 30, 0, 0, time.UTC) + later := start.Add(12 * time.Minute) + + graph := buildFixtureGraph(cfg, start) + graph.UpdateMetrics(cfg, later) + + if got := trueNASCollectedAt(graph.PlatformFixtures.TrueNAS); !got.Equal(later) { + t.Fatalf("expected rebased TrueNAS collectedAt %s, got %s", later, got) + } + if got := graph.PlatformFixtures.VMware.CollectedAt; !got.Equal(later) { + t.Fatalf("expected rebased VMware collectedAt %s, got %s", later, got) + } + if len(graph.PlatformFixtures.VMware.Hosts) == 0 || len(graph.PlatformFixtures.VMware.Hosts[0].RecentEvents) == 0 { + t.Fatal("expected canonical VMware fixtures with host events") + } + if got := graph.PlatformFixtures.VMware.Hosts[0].RecentEvents[0].CreatedAt; got.Before(later.Add(-2*time.Hour)) || got.After(later) { + t.Fatalf("expected VMware event timestamp to remain fresh near %s, got %s", later, got) + } +} diff --git a/internal/mock/recovery_points.go b/internal/mock/recovery_points.go index 40736cb65..23d3e772f 100644 --- a/internal/mock/recovery_points.go +++ b/internal/mock/recovery_points.go @@ -9,6 +9,8 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/recovery" + truenasmapper "github.com/rcourtman/pulse-go-rewrite/internal/recovery/mapper/truenas" + "github.com/rcourtman/pulse-go-rewrite/internal/truenas" ) func (g FixtureGraph) RecoveryPoints() []recovery.RecoveryPoint { @@ -339,177 +341,11 @@ func generateMockRecoveryPoints(snapshot models.StateSnapshot, fixtures Platform } } - // TrueNAS: 3 dataset subjects with multiple points over time. + // TrueNAS: reuse the provider-native recovery artifact model so demo-mode + // recovery mirrors the same contract as live TrueNAS reads. truenasConnection := defaultTrueNASConnectionFixture(fixtures) - truenasConnID := truenasConnection.ID - truenasHost := truenasConnection.Host - truenasDatasets := make([]string, 0, 3) - for _, dataset := range fixtures.TrueNAS.Datasets { - name := strings.TrimSpace(dataset.Name) - if name == "" { - continue - } - truenasDatasets = append(truenasDatasets, name) - if len(truenasDatasets) == 3 { - break - } - } - if len(truenasDatasets) == 0 { - truenasDatasets = []string{"tank/apps", "tank/media", "archive/backups"} - } - - // ZFS snapshots (snapshot / snapshot): success points with meaningful details. - for di, ds := range truenasDatasets { - for i := 0; i < 4; i++ { - ageDays := 3 + (di*6+i*5)%25 - completed := anchor.AddDate(0, 0, -ageDays).Add(time.Duration((3+di*4+i*2)%23) * time.Hour).Add(time.Duration((i%7)*7) * time.Minute) - - snapName := "auto-" + completed.Format("20060102") + "-" + rpTwoDigits(i+1) - full := ds + "@" + snapName - - var sizeBytes *int64 - if i%2 == 1 { - sizeBytes = int64Ptr(750_000_000 + int64(di)*250_000_000 + int64(i)*125_000_000) - } - - points = append(points, recovery.RecoveryPoint{ - ID: rpStableID("mock", "recoverypoint", "truenas", "zfs-snapshot", truenasConnID, full, completed.UTC().Format(time.RFC3339Nano)), - Provider: recovery.ProviderTrueNAS, - Kind: recovery.KindSnapshot, - Mode: recovery.ModeSnapshot, - Outcome: recovery.OutcomeSuccess, - StartedAt: rpPtrTime(completed), - CompletedAt: rpPtrTime(completed), - SizeBytes: sizeBytes, - Immutable: func() *bool { - if i%4 == 0 { - return boolPtr(true) - } - if i%4 == 1 { - return boolPtr(false) - } - return nil - }(), - Encrypted: func() *bool { - if i%3 == 0 { - return boolPtr(true) - } - if i%3 == 1 { - return boolPtr(false) - } - return nil - }(), - SubjectRef: &recovery.ExternalRef{ - Type: "truenas-dataset", - Name: ds, - ID: ds, - }, - Details: map[string]any{ - "connectionId": truenasConnID, - "hostname": truenasHost, - "dataset": ds, - "snapshot": snapName, - "fullName": full, - "policyName": "hourly-7d", - }, - }) - } - } - - // Replication tasks (backup / remote): ensure each dataset has a success plus a warning/failed. - for di, ds := range truenasDatasets { - taskID := "rep-task-" + rpTwoDigits(di+1) - taskName := "replicate-" + strings.ReplaceAll(ds, "/", "-") - target := "backup-tank/replicated/" + strings.ReplaceAll(ds, "/", "_") - - for i := 0; i < 4; i++ { - ageDays := 1 + (di*5+i*6)%27 - started := anchor.AddDate(0, 0, -ageDays).Add(time.Duration((8+di*3+i*4)%23) * time.Hour).Add(time.Duration((i%7)*11) * time.Minute) - - state := "SUCCESS" - errText := "" - outcome := recovery.OutcomeSuccess - switch i % 4 { - case 0: - state = "SUCCESS" - outcome = recovery.OutcomeSuccess - case 1: - state = "WARNING" - outcome = recovery.OutcomeWarning - case 2: - state = "FAILED" - errText = "network timeout" - outcome = recovery.OutcomeFailed - default: - state = "RUNNING" - outcome = recovery.OutcomeRunning - } - - var completedAt *time.Time - if outcome == recovery.OutcomeRunning { - completedAt = nil - } else { - t := started.Add(time.Duration(14+(i%8)) * time.Minute) - completedAt = &t - } - - lastSnapshot := ds + "@auto-" + anchor.AddDate(0, 0, -((i+2)%14)).Format("20060102") + "-01" - - points = append(points, recovery.RecoveryPoint{ - ID: rpStableID("mock", "recoverypoint", "truenas", "replication-task", truenasConnID, taskID, rpTimeKey(completedAt, &started)), - Provider: recovery.ProviderTrueNAS, - Kind: recovery.KindBackup, - Mode: recovery.ModeRemote, - Outcome: outcome, - StartedAt: rpPtrTime(started), - CompletedAt: completedAt, - Verified: func() *bool { - if completedAt == nil { - return nil - } - if i%3 == 0 { - return boolPtr(true) - } - if i%3 == 1 { - return boolPtr(false) - } - return nil - }(), - Encrypted: func() *bool { - if i%4 == 0 { - return boolPtr(true) - } - if i%4 == 1 { - return boolPtr(false) - } - return nil - }(), - Immutable: func() *bool { - if i%5 == 0 { - return boolPtr(true) - } - if i%5 == 1 { - return boolPtr(false) - } - return nil - }(), - SubjectRef: &recovery.ExternalRef{Type: "truenas-dataset", Name: ds, ID: ds}, - RepositoryRef: &recovery.ExternalRef{Type: "truenas-dataset", Name: target, ID: target}, - Details: map[string]any{ - "connectionId": truenasConnID, - "hostname": truenasHost, - "taskId": taskID, - "taskName": taskName, - "sourceDatasets": []string{ds}, - "targetDataset": target, - "lastState": state, - "lastError": errText, - "lastSnapshot": lastSnapshot, - "policyName": "daily-replication", - }, - }) - } - } + rebasedTrueNAS := rebaseTrueNASRecoverySnapshot(fixtures.TrueNAS, time.Now().UTC().Add(-90*time.Minute).Truncate(time.Minute)) + points = append(points, truenasmapper.FromTrueNASSnapshot(truenasConnection.ID, &rebasedTrueNAS)...) // Ensure newest-first ordering like the store (completedAt desc with NULLS last), // with stable ID tie-breakers for deterministic pagination. @@ -550,6 +386,64 @@ func generateMockRecoveryPoints(snapshot models.StateSnapshot, fixtures Platform return points } +func rebaseTrueNASRecoverySnapshot(snapshot truenas.FixtureSnapshot, targetLatest time.Time) truenas.FixtureSnapshot { + latest := latestTrueNASRecoveryTime(snapshot) + if latest.IsZero() { + latest = targetLatest + } + if targetLatest.IsZero() { + targetLatest = latest + } + + shift := targetLatest.Sub(latest) + rebased := snapshot + if !rebased.CollectedAt.IsZero() { + rebased.CollectedAt = rebased.CollectedAt.Add(shift) + } + + if len(snapshot.ZFSSnapshots) > 0 { + rebased.ZFSSnapshots = make([]truenas.ZFSSnapshot, len(snapshot.ZFSSnapshots)) + for i, snap := range snapshot.ZFSSnapshots { + rebased.ZFSSnapshots[i] = snap + if snap.CreatedAt != nil { + shifted := snap.CreatedAt.Add(shift) + rebased.ZFSSnapshots[i].CreatedAt = &shifted + } + } + } + + if len(snapshot.ReplicationTasks) > 0 { + rebased.ReplicationTasks = make([]truenas.ReplicationTask, len(snapshot.ReplicationTasks)) + for i, task := range snapshot.ReplicationTasks { + rebased.ReplicationTasks[i] = task + if task.LastRun != nil { + shifted := task.LastRun.Add(shift) + rebased.ReplicationTasks[i].LastRun = &shifted + } + } + } + + return rebased +} + +func latestTrueNASRecoveryTime(snapshot truenas.FixtureSnapshot) time.Time { + latest := snapshot.CollectedAt + + for _, snap := range snapshot.ZFSSnapshots { + if snap.CreatedAt != nil && snap.CreatedAt.After(latest) { + latest = *snap.CreatedAt + } + } + + for _, task := range snapshot.ReplicationTasks { + if task.LastRun != nil && task.LastRun.After(latest) { + latest = *task.LastRun + } + } + + return latest +} + type mockRecoveryCluster struct { id string name string diff --git a/internal/mock/recovery_points_test.go b/internal/mock/recovery_points_test.go index a1f64158c..4c6364599 100644 --- a/internal/mock/recovery_points_test.go +++ b/internal/mock/recovery_points_test.go @@ -2,6 +2,7 @@ package mock import ( "testing" + "time" "github.com/rcourtman/pulse-go-rewrite/internal/recovery" ) @@ -123,3 +124,31 @@ func TestFixtureGraphRecoveryPointsDeriveSubjectsFromCurrentGraph(t *testing.T) t.Fatal("expected recovery points to derive TrueNAS subjects from canonical mock graph") } } + +func TestFixtureGraphRecoveryPointsKeepTrueNASArtifactsFreshForDemoMode(t *testing.T) { + previous := IsMockEnabled() + SetEnabled(true) + t.Cleanup(func() { SetEnabled(previous) }) + + points := CurrentFixtureGraph().RecoveryPoints() + if len(points) == 0 { + t.Fatal("expected mock recovery points") + } + + var newest time.Time + for _, point := range points { + if point.Provider != recovery.ProviderTrueNAS || point.CompletedAt == nil { + continue + } + if point.CompletedAt.After(newest) { + newest = *point.CompletedAt + } + } + + if newest.IsZero() { + t.Fatal("expected completed TrueNAS recovery points") + } + if newest.Before(time.Now().UTC().Add(-24 * time.Hour)) { + t.Fatalf("expected demo TrueNAS recovery artifacts within the last 24h, got newest %s", newest.UTC().Format(time.RFC3339)) + } +} diff --git a/internal/monitoring/mock_metrics_history.go b/internal/monitoring/mock_metrics_history.go index 4789d943e..e6b4b257c 100644 --- a/internal/monitoring/mock_metrics_history.go +++ b/internal/monitoring/mock_metrics_history.go @@ -557,13 +557,29 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi } recordGuest := func( - metricID, storeType, storeID string, + metricIDs []string, storeType, storeID string, cpuPercent, memPercent, diskPercent, diskRead, diskWrite, netIn, netOut float64, includeDisk bool, includeDiskIO bool, includeNetwork bool, ) { - if metricID == "" || storeID == "" { + if len(metricIDs) == 0 || storeID == "" { + return + } + uniqueMetricIDs := make([]string, 0, len(metricIDs)) + seenMetricIDs := make(map[string]struct{}, len(metricIDs)) + for _, rawID := range metricIDs { + metricID := strings.TrimSpace(rawID) + if metricID == "" { + continue + } + if _, exists := seenMetricIDs[metricID]; exists { + continue + } + seenMetricIDs[metricID] = struct{}{} + uniqueMetricIDs = append(uniqueMetricIDs, metricID) + } + if len(uniqueMetricIDs) == 0 { return } @@ -589,46 +605,62 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi for i := 0; i < numPoints; i++ { ts := seedTimestamps[i] - mh.AddGuestMetric(metricID, "cpu", cpuSeries[i], ts) - mh.AddGuestMetric(metricID, "memory", memSeries[i], ts) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "cpu", cpuSeries[i], ts) + mh.AddGuestMetric(metricID, "memory", memSeries[i], ts) + } queueMetric(storeType, storeID, "cpu", cpuSeries[i], ts) queueMetric(storeType, storeID, "memory", memSeries[i], ts) if includeDisk { - mh.AddGuestMetric(metricID, "disk", diskSeries[i], ts) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "disk", diskSeries[i], ts) + } queueMetric(storeType, storeID, "disk", diskSeries[i], ts) } if includeDiskIO { - mh.AddGuestMetric(metricID, "diskread", diskReadSeries[i], ts) - mh.AddGuestMetric(metricID, "diskwrite", diskWriteSeries[i], ts) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "diskread", diskReadSeries[i], ts) + mh.AddGuestMetric(metricID, "diskwrite", diskWriteSeries[i], ts) + } queueMetric(storeType, storeID, "diskread", diskReadSeries[i], ts) queueMetric(storeType, storeID, "diskwrite", diskWriteSeries[i], ts) } if includeNetwork { - mh.AddGuestMetric(metricID, "netin", netInSeries[i], ts) - mh.AddGuestMetric(metricID, "netout", netOutSeries[i], ts) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "netin", netInSeries[i], ts) + mh.AddGuestMetric(metricID, "netout", netOutSeries[i], ts) + } queueMetric(storeType, storeID, "netin", netInSeries[i], ts) queueMetric(storeType, storeID, "netout", netOutSeries[i], ts) } } // Ensure the latest point lands at "now" for full-range charts. - mh.AddGuestMetric(metricID, "cpu", cpuPercent, now) - mh.AddGuestMetric(metricID, "memory", memPercent, now) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "cpu", cpuPercent, now) + mh.AddGuestMetric(metricID, "memory", memPercent, now) + } queueMetric(storeType, storeID, "cpu", cpuPercent, now) queueMetric(storeType, storeID, "memory", memPercent, now) if includeDisk { - mh.AddGuestMetric(metricID, "disk", diskPercent, now) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "disk", diskPercent, now) + } queueMetric(storeType, storeID, "disk", diskPercent, now) } if includeDiskIO { - mh.AddGuestMetric(metricID, "diskread", diskRead, now) - mh.AddGuestMetric(metricID, "diskwrite", diskWrite, now) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "diskread", diskRead, now) + mh.AddGuestMetric(metricID, "diskwrite", diskWrite, now) + } queueMetric(storeType, storeID, "diskread", diskRead, now) queueMetric(storeType, storeID, "diskwrite", diskWrite, now) } if includeNetwork { - mh.AddGuestMetric(metricID, "netin", netIn, now) - mh.AddGuestMetric(metricID, "netout", netOut, now) + for _, metricID := range uniqueMetricIDs { + mh.AddGuestMetric(metricID, "netin", netIn, now) + mh.AddGuestMetric(metricID, "netout", netOut, now) + } queueMetric(storeType, storeID, "netin", netIn, now) queueMetric(storeType, storeID, "netout", netOut, now) } @@ -668,7 +700,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi netI = float64(2000 + (h>>16)%8000) netO = float64(1000 + (h>>24)%4000) } - recordGuest(vm.ID, "vm", vm.ID, cpuPct, memPct, diskPct, diskR, diskW, netI, netO, true, true, true) + recordGuest([]string{vm.ID}, "vm", vm.ID, cpuPct, memPct, diskPct, diskR, diskW, netI, netO, true, true, true) time.Sleep(50 * time.Millisecond) // Reduced from 200ms for faster startup } @@ -698,7 +730,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi netI = float64(2000 + (h>>16)%8000) netO = float64(1000 + (h>>24)%4000) } - recordGuest(ct.ID, "container", ct.ID, cpuPct, memPct, diskPct, diskR, diskW, netI, netO, true, true, true) + recordGuest([]string{ct.ID}, "container", ct.ID, cpuPct, memPct, diskPct, diskR, diskW, netI, netO, true, true, true) time.Sleep(50 * time.Millisecond) // Reduced from 200ms for faster startup } @@ -721,7 +753,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi } current := kubernetesPodCurrentMetrics(cluster, pod) recordGuest( - metricID, + []string{metricID}, "k8s", metricID, current["cpu"], @@ -836,7 +868,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi diskPercent = float64(usedTotal) / float64(totalTotal) * 100 } - recordGuest("dockerHost:"+host.ID, "dockerHost", host.ID, host.CPUUsage, host.Memory.Usage, diskPercent, 0, 0, 0, 0, true, false, false) + recordGuest([]string{"dockerHost:" + host.ID}, "dockerHost", host.ID, host.CPUUsage, host.Memory.Usage, diskPercent, 0, 0, 0, 0, true, false, false) for _, container := range host.Containers { if container.ID == "" { @@ -856,7 +888,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi memPct = 10 + float64((h>>8)%40) // 10-49 % diskPct = 5 + float64((h>>16)%30) } - recordGuest("docker:"+container.ID, "dockerContainer", container.ID, cpuPct, memPct, diskPct, 0, 0, 0, 0, true, false, false) + recordGuest([]string{"docker:" + container.ID}, "dockerContainer", container.ID, cpuPct, memPct, diskPct, 0, 0, 0, 0, true, false, false) } time.Sleep(50 * time.Millisecond) // Add delay for docker hosts } @@ -872,7 +904,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi diskPercent = host.Disks[0].Usage } - recordGuest("agent:"+host.ID, "agent", host.ID, host.CPUUsage, host.Memory.Usage, diskPercent, host.DiskReadRate, host.DiskWriteRate, host.NetInRate, host.NetOutRate, true, true, true) + recordGuest([]string{"agent:" + host.ID}, "agent", host.ID, host.CPUUsage, host.Memory.Usage, diskPercent, host.DiskReadRate, host.DiskWriteRate, host.NetInRate, host.NetOutRate, true, true, true) time.Sleep(50 * time.Millisecond) } @@ -890,8 +922,33 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi if totalCap > 0 { systemDisk = float64(totalUsed) / float64(totalCap) * 100 } - systemKey := "system:" + trueNASFixtures.System.Hostname - recordGuest(systemKey, "truenas", trueNASFixtures.System.Hostname, 0, 0, systemDisk, 0, 0, 0, 0, true, false, false) + systemMemoryPercent := float64(0) + if trueNASFixtures.System.MemoryTotalBytes > 0 { + usedMemory := trueNASFixtures.System.MemoryTotalBytes - trueNASFixtures.System.MemoryAvailableBytes + if usedMemory < 0 { + usedMemory = 0 + } + systemMemoryPercent = (float64(usedMemory) / float64(trueNASFixtures.System.MemoryTotalBytes)) * 100 + } + systemMetricIDs := []string{ + "system:" + trueNASFixtures.System.Hostname, + "agent:" + trueNASFixtures.System.Hostname, + } + recordGuest( + systemMetricIDs, + "agent", + trueNASFixtures.System.Hostname, + trueNASFixtures.System.CPUPercent, + systemMemoryPercent, + systemDisk, + trueNASFixtures.System.DiskReadRate, + trueNASFixtures.System.DiskWriteRate, + trueNASFixtures.System.NetInRate, + trueNASFixtures.System.NetOutRate, + true, + true, + true, + ) for _, pool := range trueNASFixtures.Pools { poolKey := "pool:" + pool.Name @@ -901,13 +958,32 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi } numPoints := len(seedTimestamps) diskSeries := GenerateSeededSeries(diskPercent, numPoints, HashSeed("pool", pool.Name, "disk"), 0, 100, styleFlat) + usedSeries := GenerateSeededSeries(float64(pool.UsedBytes), numPoints, HashSeed("pool", pool.Name, "used"), 0, float64(pool.TotalBytes), styleFlat) + availSeries := GenerateSeededSeries(float64(pool.FreeBytes), numPoints, HashSeed("pool", pool.Name, "avail"), 0, float64(pool.TotalBytes), styleFlat) + totalSeries := GenerateSeededSeries(float64(pool.TotalBytes), numPoints, HashSeed("pool", pool.Name, "total"), 0, float64(pool.TotalBytes), styleFlat) for i := 0; i < numPoints; i++ { ts := seedTimestamps[i] mh.AddGuestMetric(poolKey, "disk", diskSeries[i], ts) queueMetric("pool", pool.Name, "disk", diskSeries[i], ts) + mh.AddStorageMetric(poolKey, "usage", diskSeries[i], ts) + mh.AddStorageMetric(poolKey, "used", usedSeries[i], ts) + mh.AddStorageMetric(poolKey, "avail", availSeries[i], ts) + mh.AddStorageMetric(poolKey, "total", totalSeries[i], ts) + queueMetric("storage", poolKey, "usage", diskSeries[i], ts) + queueMetric("storage", poolKey, "used", usedSeries[i], ts) + queueMetric("storage", poolKey, "avail", availSeries[i], ts) + queueMetric("storage", poolKey, "total", totalSeries[i], ts) } mh.AddGuestMetric(poolKey, "disk", diskPercent, now) queueMetric("pool", pool.Name, "disk", diskPercent, now) + mh.AddStorageMetric(poolKey, "usage", diskPercent, now) + mh.AddStorageMetric(poolKey, "used", float64(pool.UsedBytes), now) + mh.AddStorageMetric(poolKey, "avail", float64(pool.FreeBytes), now) + mh.AddStorageMetric(poolKey, "total", float64(pool.TotalBytes), now) + queueMetric("storage", poolKey, "usage", diskPercent, now) + queueMetric("storage", poolKey, "used", float64(pool.UsedBytes), now) + queueMetric("storage", poolKey, "avail", float64(pool.FreeBytes), now) + queueMetric("storage", poolKey, "total", float64(pool.TotalBytes), now) } for _, dataset := range trueNASFixtures.Datasets { @@ -919,13 +995,68 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi } numPoints := len(seedTimestamps) diskSeries := GenerateSeededSeries(diskPercent, numPoints, HashSeed("dataset", dataset.Name, "disk"), 0, 100, styleFlat) + usedSeries := GenerateSeededSeries(float64(dataset.UsedBytes), numPoints, HashSeed("dataset", dataset.Name, "used"), 0, float64(totalBytes), styleFlat) + availSeries := GenerateSeededSeries(float64(dataset.AvailBytes), numPoints, HashSeed("dataset", dataset.Name, "avail"), 0, float64(totalBytes), styleFlat) + totalSeries := GenerateSeededSeries(float64(totalBytes), numPoints, HashSeed("dataset", dataset.Name, "total"), 0, float64(totalBytes), styleFlat) for i := 0; i < numPoints; i++ { ts := seedTimestamps[i] mh.AddGuestMetric(dsKey, "disk", diskSeries[i], ts) queueMetric("dataset", dataset.Name, "disk", diskSeries[i], ts) + mh.AddStorageMetric(dsKey, "usage", diskSeries[i], ts) + mh.AddStorageMetric(dsKey, "used", usedSeries[i], ts) + mh.AddStorageMetric(dsKey, "avail", availSeries[i], ts) + mh.AddStorageMetric(dsKey, "total", totalSeries[i], ts) + queueMetric("storage", dsKey, "usage", diskSeries[i], ts) + queueMetric("storage", dsKey, "used", usedSeries[i], ts) + queueMetric("storage", dsKey, "avail", availSeries[i], ts) + queueMetric("storage", dsKey, "total", totalSeries[i], ts) } mh.AddGuestMetric(dsKey, "disk", diskPercent, now) queueMetric("dataset", dataset.Name, "disk", diskPercent, now) + mh.AddStorageMetric(dsKey, "usage", diskPercent, now) + mh.AddStorageMetric(dsKey, "used", float64(dataset.UsedBytes), now) + mh.AddStorageMetric(dsKey, "avail", float64(dataset.AvailBytes), now) + mh.AddStorageMetric(dsKey, "total", float64(totalBytes), now) + queueMetric("storage", dsKey, "usage", diskPercent, now) + queueMetric("storage", dsKey, "used", float64(dataset.UsedBytes), now) + queueMetric("storage", dsKey, "avail", float64(dataset.AvailBytes), now) + queueMetric("storage", dsKey, "total", float64(totalBytes), now) + } + + for _, app := range trueNASFixtures.Apps { + if app.Stats == nil { + continue + } + appID := strings.TrimSpace(app.ID) + if appID == "" { + appID = strings.TrimSpace(app.Name) + } + if appID == "" { + continue + } + memPercent := float64(0) + if trueNASFixtures.System.MemoryTotalBytes > 0 { + memPercent = (float64(app.Stats.MemoryBytes) / float64(trueNASFixtures.System.MemoryTotalBytes)) * 100 + } + diskPercent := 8 + float64(HashSeed("truenas-app", appID, "disk")%35) + if strings.EqualFold(strings.TrimSpace(app.State), "stopped") { + diskPercent = clampFloat(diskPercent*0.6, 0, 100) + } + recordGuest( + []string{"docker:" + appID}, + "dockerContainer", + appID, + app.Stats.CPUPercent, + memPercent, + diskPercent, + app.Stats.DiskReadRate, + app.Stats.DiskWriteRate, + app.Stats.NetInRate, + app.Stats.NetOutRate, + true, + true, + true, + ) } vmwareFixtures := platformFixtures.VMware @@ -942,7 +1073,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi continue } recordGuest( - "agent:"+sourceID, + []string{"agent:" + sourceID}, "agent", sourceID, vmwareFloat64Metric(host.Metrics, func(m *vmware.InventoryMetrics) *float64 { return m.CPUPercent }), @@ -964,7 +1095,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi continue } recordGuest( - sourceID, + []string{sourceID}, "vm", sourceID, vmwareFloat64Metric(guest.Metrics, func(m *vmware.InventoryMetrics) *float64 { return m.CPUPercent }), @@ -982,7 +1113,13 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi for _, datastore := range vmwareFixtures.Datastores { usage := vmwareDatastoreUsagePercent(datastore) - if usage <= 0 { + total := datastore.Capacity + avail := datastore.FreeSpace + used := total - avail + if used < 0 { + used = 0 + } + if usage <= 0 && total <= 0 { continue } sourceID := vmware.SourceID(vmwareFixtures.ConnectionID, "datastore", datastore.Datastore) @@ -991,13 +1128,28 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi } numPoints := len(seedTimestamps) usageSeries := GenerateSeededSeries(usage, numPoints, HashSeed("vmware-datastore", sourceID, "usage"), 0, 100, styleFlat) + usedSeries := GenerateSeededSeries(float64(used), numPoints, HashSeed("vmware-datastore", sourceID, "used"), 0, float64(total), styleFlat) + availSeries := GenerateSeededSeries(float64(avail), numPoints, HashSeed("vmware-datastore", sourceID, "avail"), 0, float64(total), styleFlat) + totalSeries := GenerateSeededSeries(float64(total), numPoints, HashSeed("vmware-datastore", sourceID, "total"), 0, float64(total), styleFlat) for i := 0; i < numPoints; i++ { ts := seedTimestamps[i] mh.AddStorageMetric(sourceID, "usage", usageSeries[i], ts) + mh.AddStorageMetric(sourceID, "used", usedSeries[i], ts) + mh.AddStorageMetric(sourceID, "avail", availSeries[i], ts) + mh.AddStorageMetric(sourceID, "total", totalSeries[i], ts) queueMetric("storage", sourceID, "usage", usageSeries[i], ts) + queueMetric("storage", sourceID, "used", usedSeries[i], ts) + queueMetric("storage", sourceID, "avail", availSeries[i], ts) + queueMetric("storage", sourceID, "total", totalSeries[i], ts) } mh.AddStorageMetric(sourceID, "usage", usage, now) + mh.AddStorageMetric(sourceID, "used", float64(used), now) + mh.AddStorageMetric(sourceID, "avail", float64(avail), now) + mh.AddStorageMetric(sourceID, "total", float64(total), now) queueMetric("storage", sourceID, "usage", usage, now) + queueMetric("storage", sourceID, "used", float64(used), now) + queueMetric("storage", sourceID, "avail", float64(avail), now) + queueMetric("storage", sourceID, "total", float64(total), now) } if ms != nil && len(seedBatch) > 0 { @@ -1006,7 +1158,8 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi log.Debug().Msg("mock seeding: completed") } -// recordTrueNASFixturesMetrics records disk usage metrics for TrueNAS pools and datasets. +// recordTrueNASFixturesMetrics records live fixture ticks for TrueNAS host, +// storage, and app-container metrics. func recordTrueNASFixturesMetrics(mh *MetricsHistory, ms *metrics.Store, fixtures mock.PlatformFixtures, ts time.Time) { snapshot := fixtures.TrueNAS @@ -1016,11 +1169,42 @@ func recordTrueNASFixturesMetrics(mh *MetricsHistory, ms *metrics.Store, fixture totalUsed += pool.UsedBytes } if totalCap > 0 { - systemKey := "system:" + snapshot.System.Hostname systemDisk := float64(totalUsed) / float64(totalCap) * 100 - mh.AddGuestMetric(systemKey, "disk", systemDisk, ts) + systemMetricIDs := []string{ + "system:" + snapshot.System.Hostname, + "agent:" + snapshot.System.Hostname, + } + for _, metricID := range systemMetricIDs { + mh.AddGuestMetric(metricID, "cpu", snapshot.System.CPUPercent, ts) + if snapshot.System.MemoryTotalBytes > 0 { + usedMemory := snapshot.System.MemoryTotalBytes - snapshot.System.MemoryAvailableBytes + if usedMemory < 0 { + usedMemory = 0 + } + mh.AddGuestMetric(metricID, "memory", (float64(usedMemory)/float64(snapshot.System.MemoryTotalBytes))*100, ts) + } + mh.AddGuestMetric(metricID, "disk", systemDisk, ts) + mh.AddGuestMetric(metricID, "diskread", snapshot.System.DiskReadRate, ts) + mh.AddGuestMetric(metricID, "diskwrite", snapshot.System.DiskWriteRate, ts) + mh.AddGuestMetric(metricID, "netin", snapshot.System.NetInRate, ts) + mh.AddGuestMetric(metricID, "netout", snapshot.System.NetOutRate, ts) + } if ms != nil { - ms.Write("truenas", snapshot.System.Hostname, "disk", systemDisk, ts) + var systemMemoryPercent float64 + if snapshot.System.MemoryTotalBytes > 0 { + usedMemory := snapshot.System.MemoryTotalBytes - snapshot.System.MemoryAvailableBytes + if usedMemory < 0 { + usedMemory = 0 + } + systemMemoryPercent = (float64(usedMemory) / float64(snapshot.System.MemoryTotalBytes)) * 100 + } + ms.Write("agent", snapshot.System.Hostname, "cpu", snapshot.System.CPUPercent, ts) + ms.Write("agent", snapshot.System.Hostname, "memory", systemMemoryPercent, ts) + ms.Write("agent", snapshot.System.Hostname, "disk", systemDisk, ts) + ms.Write("agent", snapshot.System.Hostname, "diskread", snapshot.System.DiskReadRate, ts) + ms.Write("agent", snapshot.System.Hostname, "diskwrite", snapshot.System.DiskWriteRate, ts) + ms.Write("agent", snapshot.System.Hostname, "netin", snapshot.System.NetInRate, ts) + ms.Write("agent", snapshot.System.Hostname, "netout", snapshot.System.NetOutRate, ts) } } @@ -1029,8 +1213,16 @@ func recordTrueNASFixturesMetrics(mh *MetricsHistory, ms *metrics.Store, fixture poolKey := "pool:" + pool.Name diskPct := float64(pool.UsedBytes) / float64(pool.TotalBytes) * 100 mh.AddGuestMetric(poolKey, "disk", diskPct, ts) + mh.AddStorageMetric(poolKey, "usage", diskPct, ts) + mh.AddStorageMetric(poolKey, "used", float64(pool.UsedBytes), ts) + mh.AddStorageMetric(poolKey, "avail", float64(pool.FreeBytes), ts) + mh.AddStorageMetric(poolKey, "total", float64(pool.TotalBytes), ts) if ms != nil { ms.Write("pool", pool.Name, "disk", diskPct, ts) + ms.Write("storage", poolKey, "usage", diskPct, ts) + ms.Write("storage", poolKey, "used", float64(pool.UsedBytes), ts) + ms.Write("storage", poolKey, "avail", float64(pool.FreeBytes), ts) + ms.Write("storage", poolKey, "total", float64(pool.TotalBytes), ts) } } } @@ -1041,11 +1233,57 @@ func recordTrueNASFixturesMetrics(mh *MetricsHistory, ms *metrics.Store, fixture dsKey := "dataset:" + dataset.Name diskPct := float64(dataset.UsedBytes) / float64(totalBytes) * 100 mh.AddGuestMetric(dsKey, "disk", diskPct, ts) + mh.AddStorageMetric(dsKey, "usage", diskPct, ts) + mh.AddStorageMetric(dsKey, "used", float64(dataset.UsedBytes), ts) + mh.AddStorageMetric(dsKey, "avail", float64(dataset.AvailBytes), ts) + mh.AddStorageMetric(dsKey, "total", float64(totalBytes), ts) if ms != nil { ms.Write("dataset", dataset.Name, "disk", diskPct, ts) + ms.Write("storage", dsKey, "usage", diskPct, ts) + ms.Write("storage", dsKey, "used", float64(dataset.UsedBytes), ts) + ms.Write("storage", dsKey, "avail", float64(dataset.AvailBytes), ts) + ms.Write("storage", dsKey, "total", float64(totalBytes), ts) } } } + + for _, app := range snapshot.Apps { + if app.Stats == nil { + continue + } + appID := strings.TrimSpace(app.ID) + if appID == "" { + appID = strings.TrimSpace(app.Name) + } + if appID == "" { + continue + } + memPercent := float64(0) + if snapshot.System.MemoryTotalBytes > 0 { + memPercent = (float64(app.Stats.MemoryBytes) / float64(snapshot.System.MemoryTotalBytes)) * 100 + } + diskPercent := 8 + float64(HashSeed("truenas-app", appID, "disk")%35) + if strings.EqualFold(strings.TrimSpace(app.State), "stopped") { + diskPercent = clampFloat(diskPercent*0.6, 0, 100) + } + metricKey := "docker:" + appID + mh.AddGuestMetric(metricKey, "cpu", app.Stats.CPUPercent, ts) + mh.AddGuestMetric(metricKey, "memory", memPercent, ts) + mh.AddGuestMetric(metricKey, "disk", diskPercent, ts) + mh.AddGuestMetric(metricKey, "diskread", app.Stats.DiskReadRate, ts) + mh.AddGuestMetric(metricKey, "diskwrite", app.Stats.DiskWriteRate, ts) + mh.AddGuestMetric(metricKey, "netin", app.Stats.NetInRate, ts) + mh.AddGuestMetric(metricKey, "netout", app.Stats.NetOutRate, ts) + if ms != nil { + ms.Write("dockerContainer", appID, "cpu", app.Stats.CPUPercent, ts) + ms.Write("dockerContainer", appID, "memory", memPercent, ts) + ms.Write("dockerContainer", appID, "disk", diskPercent, ts) + ms.Write("dockerContainer", appID, "diskread", app.Stats.DiskReadRate, ts) + ms.Write("dockerContainer", appID, "diskwrite", app.Stats.DiskWriteRate, ts) + ms.Write("dockerContainer", appID, "netin", app.Stats.NetInRate, ts) + ms.Write("dockerContainer", appID, "netout", app.Stats.NetOutRate, ts) + } + } } func recordVMwareFixturesMetrics(mh *MetricsHistory, ms *metrics.Store, fixtures mock.PlatformFixtures, ts time.Time) { @@ -1104,9 +1342,21 @@ func recordVMwareFixturesMetrics(mh *MetricsHistory, ms *metrics.Store, fixtures continue } usage := vmwareDatastoreUsagePercent(datastore) + total := datastore.Capacity + avail := datastore.FreeSpace + used := total - avail + if used < 0 { + used = 0 + } mh.AddStorageMetric(sourceID, "usage", usage, ts) + mh.AddStorageMetric(sourceID, "used", float64(used), ts) + mh.AddStorageMetric(sourceID, "avail", float64(avail), ts) + mh.AddStorageMetric(sourceID, "total", float64(total), ts) if ms != nil { ms.Write("storage", sourceID, "usage", usage, ts) + ms.Write("storage", sourceID, "used", float64(used), ts) + ms.Write("storage", sourceID, "avail", float64(avail), ts) + ms.Write("storage", sourceID, "total", float64(total), ts) } } } diff --git a/internal/monitoring/mock_metrics_history_test.go b/internal/monitoring/mock_metrics_history_test.go index 636e2d205..6104dac3d 100644 --- a/internal/monitoring/mock_metrics_history_test.go +++ b/internal/monitoring/mock_metrics_history_test.go @@ -320,6 +320,14 @@ func TestSeedMockMetricsHistory_SeedsVMwareMetricsStore(t *testing.T) { if len(storagePoints) == 0 { t.Fatal("expected metrics store to have seeded VMware datastore usage points") } + + storageUsedPoints, err := store.Query("storage", "vc-mock-1:datastore:datastore-201", "used", now.Add(-7*24*time.Hour), now, 3600) + if err != nil { + t.Fatalf("failed to query VMware datastore used metrics: %v", err) + } + if len(storageUsedPoints) == 0 { + t.Fatal("expected metrics store to have seeded VMware datastore used points") + } } func TestSeedMockMetricsHistory_SeedsTrueNASMetricsStore(t *testing.T) { @@ -355,12 +363,20 @@ func TestSeedMockMetricsHistory_SeedsTrueNASMetricsStore(t *testing.T) { }, }, now, 7*24*time.Hour, time.Minute) - systemPoints, err := store.Query("truenas", fixtures.System.Hostname, "disk", now.Add(-7*24*time.Hour), now, 3600) + systemPoints, err := store.Query("agent", fixtures.System.Hostname, "disk", now.Add(-7*24*time.Hour), now, 3600) if err != nil { - t.Fatalf("failed to query TrueNAS system disk metrics: %v", err) + t.Fatalf("failed to query TrueNAS system disk metrics via canonical agent target: %v", err) } if len(systemPoints) == 0 { - t.Fatal("expected metrics store to have seeded TrueNAS system disk points") + t.Fatal("expected metrics store to have seeded canonical TrueNAS system disk points") + } + + agentPoints, err := store.Query("agent", fixtures.System.Hostname, "cpu", now.Add(-7*24*time.Hour), now, 3600) + if err != nil { + t.Fatalf("failed to query TrueNAS agent cpu metrics: %v", err) + } + if len(agentPoints) == 0 { + t.Fatal("expected metrics store to have seeded canonical TrueNAS agent cpu points") } datasetPoints, err := store.Query("dataset", fixtures.Datasets[0].Name, "disk", now.Add(-7*24*time.Hour), now, 3600) @@ -370,6 +386,22 @@ func TestSeedMockMetricsHistory_SeedsTrueNASMetricsStore(t *testing.T) { if len(datasetPoints) == 0 { t.Fatal("expected metrics store to have seeded TrueNAS dataset disk points") } + + poolUsedPoints, err := store.Query("storage", "pool:"+fixtures.Pools[0].Name, "used", now.Add(-7*24*time.Hour), now, 3600) + if err != nil { + t.Fatalf("failed to query TrueNAS pool used metrics: %v", err) + } + if len(poolUsedPoints) == 0 { + t.Fatal("expected metrics store to have seeded TrueNAS pool used points") + } + + appPoints, err := store.Query("dockerContainer", "nextcloud", "cpu", now.Add(-7*24*time.Hour), now, 3600) + if err != nil { + t.Fatalf("failed to query TrueNAS app cpu metrics: %v", err) + } + if len(appPoints) == 0 { + t.Fatal("expected metrics store to have seeded TrueNAS app cpu points") + } } func TestSeedMockMetricsHistory_UsesCanonicalMockFixtureGraphForLegacyAndProviderFixtures(t *testing.T) { @@ -434,12 +466,12 @@ func TestSeedMockMetricsHistory_UsesCanonicalMockFixtureGraphForLegacyAndProvide t.Fatal("expected seeded legacy mock node cpu metrics from canonical graph state") } - truenasPoints, err := store.Query("truenas", graph.PlatformFixtures.TrueNAS.System.Hostname, "disk", now.Add(-7*24*time.Hour), now, 3600) + truenasPoints, err := store.Query("agent", graph.PlatformFixtures.TrueNAS.System.Hostname, "disk", now.Add(-7*24*time.Hour), now, 3600) if err != nil { - t.Fatalf("failed to query TrueNAS mock disk metrics: %v", err) + t.Fatalf("failed to query canonical TrueNAS agent disk metrics: %v", err) } if len(truenasPoints) == 0 { - t.Fatal("expected seeded TrueNAS metrics from canonical graph fixtures") + t.Fatal("expected seeded canonical TrueNAS agent metrics from canonical graph fixtures") } vmwareHostID := "vc-mock-1:host:host-101" diff --git a/internal/truenas/contract_test.go b/internal/truenas/contract_test.go index 727138ae1..48dce77db 100644 --- a/internal/truenas/contract_test.go +++ b/internal/truenas/contract_test.go @@ -262,8 +262,8 @@ func TestTrueNASDiskRecordsPopulatePhysicalDiskMeta(t *testing.T) { diskRecords = append(diskRecords, record) } } - if len(diskRecords) != 4 { - t.Fatalf("expected 4 disk records, got %d", len(diskRecords)) + if len(diskRecords) != len(fixtures.Disks) { + t.Fatalf("expected %d disk records, got %d", len(fixtures.Disks), len(diskRecords)) } for _, record := range diskRecords { @@ -317,8 +317,8 @@ func TestTrueNASDiskRecordsPopulatePhysicalDiskMeta(t *testing.T) { if record.Resource.Status != unifiedresources.StatusWarning { t.Fatalf("expected status warning for %q, got %s", fixture.Name, record.Resource.Status) } - if meta.Health != "UNKNOWN" { - t.Fatalf("expected health UNKNOWN for %q, got %q", fixture.Name, meta.Health) + if meta.Health != "DEGRADED" { + t.Fatalf("expected health DEGRADED for %q, got %q", fixture.Name, meta.Health) } if meta.Risk == nil { t.Fatalf("expected risk payload for degraded disk %q", fixture.Name) diff --git a/internal/truenas/fixtures.go b/internal/truenas/fixtures.go index 627e0a275..a3fbfca05 100644 --- a/internal/truenas/fixtures.go +++ b/internal/truenas/fixtures.go @@ -2,20 +2,23 @@ package truenas import "time" -// DefaultFixtures returns a realistic fixture snapshot for contract testing. +// DefaultFixtures returns a curated TrueNAS demo snapshot for contract tests +// and frontend mock mode. func DefaultFixtures() FixtureSnapshot { + collectedAt := time.Date(2026, 3, 31, 11, 20, 0, 0, time.UTC) + return FixtureSnapshot{ - CollectedAt: time.Date(2026, 2, 8, 12, 0, 0, 0, time.UTC), + CollectedAt: collectedAt, System: SystemInfo{ Hostname: "truenas-main", Version: "TrueNAS-SCALE-24.10.2", Build: "24.10.2.1", UptimeSeconds: 42 * 24 * 60 * 60, Healthy: true, - MachineID: "truenas-1-machine-id", + MachineID: "truenas-main-machine-id", CPUCount: 16, - MemoryTotalBytes: 64 * 1024 * 1024 * 1024, - MemoryAvailableBytes: 22 * 1024 * 1024 * 1024, + MemoryTotalBytes: truenasGiB(64), + MemoryAvailableBytes: truenasGiB(22), CPUPercent: 38, NetInRate: 48_000_000, NetOutRate: 19_500_000, @@ -27,32 +30,40 @@ func DefaultFixtures() FixtureSnapshot { "cpu_core_1": 59.0, }, IntervalSeconds: 2, - CollectedAt: time.Date(2026, 2, 8, 12, 0, 0, 0, time.UTC), + CollectedAt: collectedAt, }, Pools: []Pool{ { ID: "pool-tank", Name: "tank", Status: "ONLINE", - TotalBytes: 30 * 1024 * 1024 * 1024 * 1024, - UsedBytes: 12 * 1024 * 1024 * 1024 * 1024, - FreeBytes: 18 * 1024 * 1024 * 1024 * 1024, + TotalBytes: truenasTiB(30), + UsedBytes: truenasTiB(12), + FreeBytes: truenasTiB(18), }, { ID: "pool-fast", Name: "fast", Status: "ONLINE", - TotalBytes: 4 * 1024 * 1024 * 1024 * 1024, - UsedBytes: 1400 * 1024 * 1024 * 1024, - FreeBytes: 2696 * 1024 * 1024 * 1024, + TotalBytes: truenasTiB(8), + UsedBytes: truenasGiB(2200), + FreeBytes: truenasGiB(5992), }, { ID: "pool-archive", Name: "archive", Status: "DEGRADED", - TotalBytes: 60 * 1024 * 1024 * 1024 * 1024, - UsedBytes: 51 * 1024 * 1024 * 1024 * 1024, - FreeBytes: 9 * 1024 * 1024 * 1024 * 1024, + TotalBytes: truenasTiB(72), + UsedBytes: truenasTiB(49), + FreeBytes: truenasTiB(23), + }, + { + ID: "pool-vault", + Name: "vault", + Status: "ONLINE", + TotalBytes: truenasTiB(12), + UsedBytes: truenasGiB(3200), + FreeBytes: truenasGiB(9088), }, }, Datasets: []Dataset{ @@ -60,43 +71,75 @@ func DefaultFixtures() FixtureSnapshot { ID: "dataset-tank-apps", Name: "tank/apps", Pool: "tank", - UsedBytes: 5 * 1024 * 1024 * 1024 * 1024, - AvailBytes: 13 * 1024 * 1024 * 1024 * 1024, + UsedBytes: truenasTiB(5), + AvailBytes: truenasTiB(13), Mounted: true, }, { ID: "dataset-tank-media", Name: "tank/media", Pool: "tank", - UsedBytes: 7 * 1024 * 1024 * 1024 * 1024, - AvailBytes: 11 * 1024 * 1024 * 1024 * 1024, + UsedBytes: truenasTiB(6), + AvailBytes: truenasTiB(20), Mounted: true, }, { - ID: "dataset-fast-vm", + ID: "dataset-tank-projects", + Name: "tank/projects", + Pool: "tank", + UsedBytes: truenasTiB(2), + AvailBytes: truenasTiB(24), + Mounted: true, + }, + { + ID: "dataset-tank-photos", + Name: "tank/photos", + Pool: "tank", + UsedBytes: truenasGiB(1800), + AvailBytes: truenasGiB(24_824), + Mounted: true, + }, + { + ID: "dataset-fast-vm-images", Name: "fast/vm-images", Pool: "fast", - UsedBytes: 900 * 1024 * 1024 * 1024, - AvailBytes: 1796 * 1024 * 1024 * 1024, + UsedBytes: truenasGiB(1200), + AvailBytes: truenasGiB(5992), + Mounted: true, + }, + { + ID: "dataset-fast-analytics", + Name: "fast/analytics", + Pool: "fast", + UsedBytes: truenasGiB(900), + AvailBytes: truenasGiB(6292), Mounted: true, }, { ID: "dataset-archive-backups", Name: "archive/backups", Pool: "archive", - UsedBytes: 40 * 1024 * 1024 * 1024 * 1024, - AvailBytes: 5 * 1024 * 1024 * 1024 * 1024, + UsedBytes: truenasTiB(24), + AvailBytes: truenasTiB(18), Mounted: true, }, { ID: "dataset-archive-cold", Name: "archive/cold", Pool: "archive", - UsedBytes: 11 * 1024 * 1024 * 1024 * 1024, - AvailBytes: 4 * 1024 * 1024 * 1024 * 1024, + UsedBytes: truenasTiB(19), + AvailBytes: truenasTiB(5), Mounted: true, ReadOnly: true, }, + { + ID: "dataset-vault-compliance", + Name: "vault/compliance", + Pool: "vault", + UsedBytes: truenasGiB(2500), + AvailBytes: truenasGiB(9788), + Mounted: true, + }, }, Disks: []Disk{ { @@ -106,7 +149,7 @@ func DefaultFixtures() FixtureSnapshot { Status: "ONLINE", Model: "Seagate Exos X18", Serial: "ZL0A1234", - SizeBytes: 16 * 1024 * 1024 * 1024 * 1024, + SizeBytes: truenasTiB(16), Temperature: 34, TemperatureAggregate: DiskTemperatureAggregate{ WindowDays: 7, @@ -124,13 +167,13 @@ func DefaultFixtures() FixtureSnapshot { Status: "ONLINE", Model: "Seagate Exos X18", Serial: "ZL0A1235", - SizeBytes: 16 * 1024 * 1024 * 1024 * 1024, - Temperature: 36, + SizeBytes: truenasTiB(16), + Temperature: 34, TemperatureAggregate: DiskTemperatureAggregate{ WindowDays: 7, - MinCelsius: 31.0, - AvgCelsius: 34.5, - MaxCelsius: 40.0, + MinCelsius: 30.0, + AvgCelsius: 33.3, + MaxCelsius: 38.0, }, Transport: "sata", Rotational: true, @@ -142,13 +185,31 @@ func DefaultFixtures() FixtureSnapshot { Status: "ONLINE", Model: "Samsung PM9A3", Serial: "S65ANX0R123456", - SizeBytes: 2 * 1024 * 1024 * 1024 * 1024, - Temperature: 48, + SizeBytes: truenasTiB(4), + Temperature: 46, TemperatureAggregate: DiskTemperatureAggregate{ WindowDays: 7, - MinCelsius: 41.0, - AvgCelsius: 45.8, - MaxCelsius: 52.0, + MinCelsius: 40.0, + AvgCelsius: 44.7, + MaxCelsius: 50.0, + }, + Transport: "nvme", + Rotational: false, + }, + { + ID: "disk-nvme1n1", + Name: "nvme1n1", + Pool: "fast", + Status: "ONLINE", + Model: "Samsung PM9A3", + Serial: "S65ANX0R123457", + SizeBytes: truenasTiB(4), + Temperature: 45, + TemperatureAggregate: DiskTemperatureAggregate{ + WindowDays: 7, + MinCelsius: 39.0, + AvgCelsius: 43.8, + MaxCelsius: 49.0, }, Transport: "nvme", Rotational: false, @@ -160,7 +221,7 @@ func DefaultFixtures() FixtureSnapshot { Status: "DEGRADED", Model: "WDC Ultrastar DC HC550", Serial: "WD-WX12A3456", - SizeBytes: 20 * 1024 * 1024 * 1024 * 1024, + SizeBytes: truenasTiB(24), Temperature: 63, TemperatureAggregate: DiskTemperatureAggregate{ WindowDays: 7, @@ -171,29 +232,83 @@ func DefaultFixtures() FixtureSnapshot { Transport: "sas", Rotational: true, }, + { + ID: "disk-sdd", + Name: "sdd", + Pool: "archive", + Status: "ONLINE", + Model: "WDC Ultrastar DC HC560", + Serial: "WD-WX12A3457", + SizeBytes: truenasTiB(24), + Temperature: 41, + TemperatureAggregate: DiskTemperatureAggregate{ + WindowDays: 7, + MinCelsius: 35.0, + AvgCelsius: 39.4, + MaxCelsius: 44.0, + }, + Transport: "sas", + Rotational: true, + }, + { + ID: "disk-sde", + Name: "sde", + Pool: "vault", + Status: "ONLINE", + Model: "Micron 7450 Pro", + Serial: "MTFDKBA3T8QFM-1", + SizeBytes: truenasTiB(6), + Temperature: 38, + TemperatureAggregate: DiskTemperatureAggregate{ + WindowDays: 7, + MinCelsius: 33.0, + AvgCelsius: 36.5, + MaxCelsius: 41.0, + }, + Transport: "sas", + Rotational: false, + }, + { + ID: "disk-sdf", + Name: "sdf", + Pool: "vault", + Status: "ONLINE", + Model: "Micron 7450 Pro", + Serial: "MTFDKBA3T8QFM-2", + SizeBytes: truenasTiB(6), + Temperature: 37, + TemperatureAggregate: DiskTemperatureAggregate{ + WindowDays: 7, + MinCelsius: 32.0, + AvgCelsius: 35.9, + MaxCelsius: 40.0, + }, + Transport: "sas", + Rotational: false, + }, }, Alerts: []Alert{ { - ID: "alert-degraded-pool", + ID: "alert-archive-degraded", Level: "WARNING", - Message: "Pool archive state is DEGRADED: One or more devices has been removed by the administrator.", + Message: "Pool archive is DEGRADED: one member of archive mirror-0 is reporting checksum and SMART faults.", Source: "VolumeStatus", - Datetime: time.Date(2026, 2, 6, 8, 15, 0, 0, time.UTC), + Datetime: time.Date(2026, 3, 31, 8, 12, 0, 0, time.UTC), }, { - ID: "alert-smart", + ID: "alert-smart-sdc", Level: "WARNING", Message: "Device /dev/sdc has SMART test failures.", Source: "SMART", - Datetime: time.Date(2026, 2, 7, 14, 30, 0, 0, time.UTC), + Datetime: time.Date(2026, 3, 30, 21, 45, 0, 0, time.UTC), }, { - ID: "alert-scrub-finished", + ID: "alert-replication-success", Level: "INFO", - Message: "Scrub of pool tank finished without errors.", - Source: "Scrub", + Message: "Replication task replicate-tank-apps completed successfully to vault/compliance.", + Source: "Replication", Dismissed: true, - Datetime: time.Date(2026, 2, 5, 3, 0, 0, 0, time.UTC), + Datetime: time.Date(2026, 3, 31, 8, 25, 0, 0, time.UTC), }, }, Apps: []App{ @@ -285,7 +400,7 @@ func DefaultFixtures() FixtureSnapshot { }, Stats: &AppStats{ CPUPercent: 18, - MemoryBytes: 768 * 1024 * 1024, + MemoryBytes: truenasMiB(768), NetInRate: 2_100_000, NetOutRate: 1_250_000, BlockReadBytes: 15_000_000, @@ -293,12 +408,274 @@ func DefaultFixtures() FixtureSnapshot { DiskReadRate: 320_000, DiskWriteRate: 180_000, IntervalSeconds: 2, - CollectedAt: time.Date(2026, 2, 8, 12, 0, 0, 0, time.UTC), + CollectedAt: collectedAt, Interfaces: []AppInterfaceStats{ {Name: "eth0", RxBytesPS: 2_100_000, TxBytesPS: 1_250_000}, }, }, }, + { + ID: "immich", + Name: "Immich", + State: "RUNNING", + Version: "1.126.1", + HumanVersion: "1.126.1", + UpgradeAvailable: false, + ImageUpdatesAvailable: false, + Notes: "Shared photo archive and mobile uploads", + ContainerCount: 3, + UsedHostIPs: []string{"0.0.0.0"}, + UsedPorts: []AppPort{ + { + ContainerPort: 2283, + Protocol: "tcp", + HostPorts: []AppHostPort{ + {HostPort: 32283, HostIP: "0.0.0.0"}, + }, + }, + }, + Containers: []AppContainer{ + { + ID: "immich-server-1", + ServiceName: "immich-server", + Image: "ghcr.io/immich-app/immich-server:v1.126.1", + State: "running", + PortConfig: []AppPort{ + { + ContainerPort: 2283, + Protocol: "tcp", + HostPorts: []AppHostPort{ + {HostPort: 32283, HostIP: "0.0.0.0"}, + }, + }, + }, + VolumeMounts: []AppVolume{ + { + Source: "/mnt/tank/photos", + Destination: "/usr/src/app/upload", + Mode: "rw", + Type: "bind", + }, + }, + }, + { + ID: "immich-postgres-1", + ServiceName: "database", + Image: "docker.io/tensorchord/pgvecto-rs:pg16-v0.3.0", + State: "running", + }, + { + ID: "immich-redis-1", + ServiceName: "redis", + Image: "docker.io/library/redis:7.2", + State: "running", + }, + }, + Volumes: []AppVolume{ + { + Source: "/mnt/tank/photos", + Destination: "/usr/src/app/upload", + Mode: "rw", + Type: "bind", + }, + { + Source: "/mnt/fast/analytics/immich-cache", + Destination: "/cache", + Mode: "rw", + Type: "bind", + }, + }, + Images: []string{ + "ghcr.io/immich-app/immich-server:v1.126.1", + "docker.io/tensorchord/pgvecto-rs:pg16-v0.3.0", + "docker.io/library/redis:7.2", + }, + Networks: []AppNetwork{ + { + ID: "ix-immich-default", + Name: "ix-immich_default", + Labels: map[string]string{ + "com.docker.compose.project": "immich", + }, + }, + }, + Stats: &AppStats{ + CPUPercent: 11.3, + MemoryBytes: truenasMiB(1792), + NetInRate: 4_400_000, + NetOutRate: 3_100_000, + BlockReadBytes: 24_000_000, + BlockWriteBytes: 18_000_000, + DiskReadRate: 620_000, + DiskWriteRate: 430_000, + IntervalSeconds: 2, + CollectedAt: collectedAt, + Interfaces: []AppInterfaceStats{ + {Name: "eth0", RxBytesPS: 4_400_000, TxBytesPS: 3_100_000}, + }, + }, + }, + { + ID: "paperless-ngx", + Name: "Paperless-ngx", + State: "RUNNING", + Version: "2.14.7", + HumanVersion: "2.14.7", + CustomApp: true, + UpgradeAvailable: false, + ImageUpdatesAvailable: false, + Notes: "Document OCR and workflow inbox", + ContainerCount: 2, + UsedHostIPs: []string{"0.0.0.0"}, + UsedPorts: []AppPort{ + { + ContainerPort: 8000, + Protocol: "tcp", + HostPorts: []AppHostPort{ + {HostPort: 30080, HostIP: "0.0.0.0"}, + }, + }, + }, + Containers: []AppContainer{ + { + ID: "paperless-web-1", + ServiceName: "webserver", + Image: "ghcr.io/paperless-ngx/paperless-ngx:2.14.7", + State: "running", + PortConfig: []AppPort{ + { + ContainerPort: 8000, + Protocol: "tcp", + HostPorts: []AppHostPort{ + {HostPort: 30080, HostIP: "0.0.0.0"}, + }, + }, + }, + VolumeMounts: []AppVolume{ + { + Source: "/mnt/tank/projects/paperless", + Destination: "/usr/src/paperless/data", + Mode: "rw", + Type: "bind", + }, + }, + }, + { + ID: "paperless-redis-1", + ServiceName: "broker", + Image: "docker.io/library/redis:7.2", + State: "running", + }, + }, + Volumes: []AppVolume{ + { + Source: "/mnt/tank/projects/paperless", + Destination: "/usr/src/paperless/data", + Mode: "rw", + Type: "bind", + }, + }, + Images: []string{ + "ghcr.io/paperless-ngx/paperless-ngx:2.14.7", + "docker.io/library/redis:7.2", + }, + Networks: []AppNetwork{ + { + ID: "ix-paperless-default", + Name: "ix-paperless_default", + Labels: map[string]string{ + "com.docker.compose.project": "paperless", + }, + }, + }, + Stats: &AppStats{ + CPUPercent: 7.6, + MemoryBytes: truenasMiB(768), + NetInRate: 1_100_000, + NetOutRate: 820_000, + BlockReadBytes: 8_000_000, + BlockWriteBytes: 6_000_000, + DiskReadRate: 210_000, + DiskWriteRate: 120_000, + IntervalSeconds: 2, + CollectedAt: collectedAt, + Interfaces: []AppInterfaceStats{ + {Name: "eth0", RxBytesPS: 1_100_000, TxBytesPS: 820_000}, + }, + }, + }, + { + ID: "grafana", + Name: "Grafana", + State: "RUNNING", + Version: "11.4.0", + HumanVersion: "11.4.0", + CustomApp: true, + UpgradeAvailable: false, + ImageUpdatesAvailable: false, + Notes: "Ops dashboards and service overviews", + ContainerCount: 1, + UsedHostIPs: []string{"0.0.0.0"}, + UsedPorts: []AppPort{ + { + ContainerPort: 3000, + Protocol: "tcp", + HostPorts: []AppHostPort{ + {HostPort: 30300, HostIP: "0.0.0.0"}, + }, + }, + }, + Containers: []AppContainer{ + { + ID: "grafana-1", + ServiceName: "grafana", + Image: "docker.io/grafana/grafana:11.4.0", + State: "running", + PortConfig: []AppPort{ + { + ContainerPort: 3000, + Protocol: "tcp", + HostPorts: []AppHostPort{ + {HostPort: 30300, HostIP: "0.0.0.0"}, + }, + }, + }, + }, + }, + Volumes: []AppVolume{ + { + Source: "/mnt/fast/analytics/grafana", + Destination: "/var/lib/grafana", + Mode: "rw", + Type: "bind", + }, + }, + Images: []string{"docker.io/grafana/grafana:11.4.0"}, + Networks: []AppNetwork{ + { + ID: "ix-grafana-default", + Name: "ix-grafana_default", + Labels: map[string]string{ + "com.docker.compose.project": "grafana", + }, + }, + }, + Stats: &AppStats{ + CPUPercent: 4.1, + MemoryBytes: truenasMiB(320), + NetInRate: 620_000, + NetOutRate: 410_000, + BlockReadBytes: 5_000_000, + BlockWriteBytes: 2_400_000, + DiskReadRate: 120_000, + DiskWriteRate: 65_000, + IntervalSeconds: 2, + CollectedAt: collectedAt, + Interfaces: []AppInterfaceStats{ + {Name: "eth0", RxBytesPS: 620_000, TxBytesPS: 410_000}, + }, + }, + }, { ID: "adguard-home", Name: "AdGuard Home", @@ -306,9 +683,10 @@ func DefaultFixtures() FixtureSnapshot { Version: "0.1.2", HumanVersion: "0.107.64", CustomApp: true, + Notes: "Edge DNS and network filtering", Stats: &AppStats{ CPUPercent: 0, - MemoryBytes: 96 * 1024 * 1024, + MemoryBytes: truenasMiB(96), NetInRate: 0, NetOutRate: 0, BlockReadBytes: 1_000_000, @@ -316,7 +694,7 @@ func DefaultFixtures() FixtureSnapshot { DiskReadRate: 0, DiskWriteRate: 0, IntervalSeconds: 2, - CollectedAt: time.Date(2026, 2, 8, 12, 0, 0, 0, time.UTC), + CollectedAt: collectedAt, }, Containers: []AppContainer{ { @@ -337,5 +715,124 @@ func DefaultFixtures() FixtureSnapshot { Images: []string{"docker.io/adguard/adguardhome:v0.107.64"}, }, }, + ZFSSnapshots: []ZFSSnapshot{ + { + ID: "zfs-snapshot-tank-apps-20260331-0600", + Dataset: "tank/apps", + Name: "auto-20260331-0600", + FullName: "tank/apps@auto-20260331-0600", + CreatedAt: truenasTimePtr(time.Date(2026, 3, 31, 6, 0, 0, 0, time.UTC)), + UsedBytes: truenasInt64Ptr(truenasGiB(18)), + Referenced: truenasInt64Ptr(truenasGiB(4020)), + }, + { + ID: "zfs-snapshot-tank-media-20260330-2200", + Dataset: "tank/media", + Name: "auto-20260330-2200", + FullName: "tank/media@auto-20260330-2200", + CreatedAt: truenasTimePtr(time.Date(2026, 3, 30, 22, 0, 0, 0, time.UTC)), + UsedBytes: truenasInt64Ptr(truenasGiB(42)), + Referenced: truenasInt64Ptr(truenasTiB(6)), + }, + { + ID: "zfs-snapshot-tank-photos-20260331-0145", + Dataset: "tank/photos", + Name: "auto-20260331-0145", + FullName: "tank/photos@auto-20260331-0145", + CreatedAt: truenasTimePtr(time.Date(2026, 3, 31, 1, 45, 0, 0, time.UTC)), + UsedBytes: truenasInt64Ptr(truenasGiB(26)), + Referenced: truenasInt64Ptr(truenasGiB(1800)), + }, + { + ID: "zfs-snapshot-fast-vm-images-20260330-0315", + Dataset: "fast/vm-images", + Name: "nightly-20260330", + FullName: "fast/vm-images@nightly-20260330", + CreatedAt: truenasTimePtr(time.Date(2026, 3, 30, 3, 15, 0, 0, time.UTC)), + UsedBytes: truenasInt64Ptr(truenasGiB(64)), + Referenced: truenasInt64Ptr(truenasGiB(1200)), + }, + { + ID: "zfs-snapshot-archive-backups-20260329-2300", + Dataset: "archive/backups", + Name: "daily-20260329", + FullName: "archive/backups@daily-20260329", + CreatedAt: truenasTimePtr(time.Date(2026, 3, 29, 23, 0, 0, 0, time.UTC)), + UsedBytes: truenasInt64Ptr(truenasGiB(96)), + Referenced: truenasInt64Ptr(truenasTiB(24)), + }, + { + ID: "zfs-snapshot-vault-compliance-20260331-0415", + Dataset: "vault/compliance", + Name: "hourly-20260331-0415", + FullName: "vault/compliance@hourly-20260331-0415", + CreatedAt: truenasTimePtr(time.Date(2026, 3, 31, 4, 15, 0, 0, time.UTC)), + UsedBytes: truenasInt64Ptr(truenasGiB(8)), + Referenced: truenasInt64Ptr(truenasGiB(2500)), + }, + }, + ReplicationTasks: []ReplicationTask{ + { + ID: "rep-task-tank-apps", + Name: "replicate-tank-apps", + SourceDatasets: []string{"tank/apps"}, + TargetDataset: "vault/compliance/tank_apps", + Direction: "PUSH", + LastRun: truenasTimePtr(time.Date(2026, 3, 31, 8, 20, 0, 0, time.UTC)), + LastState: "SUCCESS", + LastSnapshot: "tank/apps@auto-20260331-0600", + }, + { + ID: "rep-task-tank-media", + Name: "replicate-tank-media", + SourceDatasets: []string{"tank/media"}, + TargetDataset: "archive/backups/tank_media", + Direction: "PUSH", + LastRun: truenasTimePtr(time.Date(2026, 3, 30, 23, 10, 0, 0, time.UTC)), + LastState: "SUCCESS", + LastSnapshot: "tank/media@auto-20260330-2200", + }, + { + ID: "rep-task-archive-backups", + Name: "replicate-archive-backups", + SourceDatasets: []string{"archive/backups"}, + TargetDataset: "offsite/archive_backups", + Direction: "PUSH", + LastRun: truenasTimePtr(time.Date(2026, 3, 30, 4, 45, 0, 0, time.UTC)), + LastState: "WARNING", + LastError: "destination latency exceeded target window", + LastSnapshot: "archive/backups@daily-20260329", + }, + { + ID: "rep-task-vault-compliance", + Name: "replicate-vault-compliance", + SourceDatasets: []string{"vault/compliance"}, + TargetDataset: "offsite/vault_compliance", + Direction: "PUSH", + LastRun: truenasTimePtr(time.Date(2026, 3, 31, 5, 40, 0, 0, time.UTC)), + LastState: "RUNNING", + LastSnapshot: "vault/compliance@hourly-20260331-0415", + }, + }, } } + +func truenasInt64Ptr(value int64) *int64 { + return &value +} + +func truenasTimePtr(value time.Time) *time.Time { + return &value +} + +func truenasMiB(value int64) int64 { + return value * 1024 * 1024 +} + +func truenasGiB(value int64) int64 { + return value * 1024 * 1024 * 1024 +} + +func truenasTiB(value int64) int64 { + return value * 1024 * 1024 * 1024 * 1024 +} diff --git a/internal/truenas/provider.go b/internal/truenas/provider.go index ded44294c..8db41f9ed 100644 --- a/internal/truenas/provider.go +++ b/internal/truenas/provider.go @@ -579,7 +579,7 @@ func truenasRecordsFromSnapshot(snapshot *FixtureSnapshot, now func() time.Time) } for _, app := range snapshot.Apps { - metrics := metricsFromTrueNASApp(app) + metrics := metricsFromTrueNASApp(app, snapshot.System.MemoryTotalBytes) dockerMeta := &unifiedresources.DockerData{ ContainerID: appCanonicalID(app), Hostname: strings.TrimSpace(snapshot.System.Hostname), @@ -672,7 +672,7 @@ func truenasRecordsFromSnapshot(snapshot *FixtureSnapshot, now func() time.Time) return records } -func metricsFromTrueNASApp(app App) *unifiedresources.ResourceMetrics { +func metricsFromTrueNASApp(app App, hostMemoryTotalBytes int64) *unifiedresources.ResourceMetrics { if app.Stats == nil { return nil } @@ -713,6 +713,12 @@ func metricsFromTrueNASApp(app App) *unifiedresources.ResourceMetrics { Unit: "bytes", Source: unifiedresources.SourceTrueNAS, } + if hostMemoryTotalBytes > 0 { + total := hostMemoryTotalBytes + metrics.Memory.Total = &total + metrics.Memory.Percent = (float64(memoryUsed) / float64(hostMemoryTotalBytes)) * 100 + metrics.Memory.Value = metrics.Memory.Percent + } return metrics } @@ -777,6 +783,7 @@ func metricsFromTrueNASSystem(system SystemInfo, totalCapacity, totalUsed int64) func agentDataFromTrueNASSystem(system SystemInfo, storageRisk *unifiedresources.StorageRisk, protectionReduced bool, protectionSummary string, rebuildInProgress bool, rebuildSummary string) *unifiedresources.AgentData { agent := &unifiedresources.AgentData{ + AgentID: trueNASSystemMetricResourceID(system), Hostname: strings.TrimSpace(system.Hostname), MachineID: strings.TrimSpace(system.MachineID), Platform: "truenas", @@ -1074,7 +1081,7 @@ func assessPool(pool Pool) storagehealth.Assessment { func assessDisk(disk Disk) storagehealth.Assessment { sampleAssessment := storagehealth.AssessSample(storagehealth.Sample{ Model: strings.TrimSpace(disk.Model), - Health: healthFromDisk(disk), + Health: healthForAssessment(disk), Temperature: disk.Temperature, Wearout: -1, }) @@ -1470,12 +1477,21 @@ func healthFromDisk(disk Disk) string { case "ONLINE": return "PASSED" case "DEGRADED": - return "UNKNOWN" + return "DEGRADED" default: return "FAILED" } } +func healthForAssessment(disk Disk) string { + switch strings.ToUpper(strings.TrimSpace(disk.Status)) { + case "DEGRADED": + return "" + default: + return healthFromDisk(disk) + } +} + func rpmFromDisk(disk Disk) int { if disk.Rotational { return 7200 diff --git a/internal/truenas/provider_test.go b/internal/truenas/provider_test.go index c93693044..ee3b31c03 100644 --- a/internal/truenas/provider_test.go +++ b/internal/truenas/provider_test.go @@ -563,8 +563,9 @@ func TestRecordsIncludeDiskResourcesWithCorrectParentChain(t *testing.T) { diskRecords = append(diskRecords, record) } } - if len(diskRecords) != 4 { - t.Fatalf("expected 4 disk records, got %d", len(diskRecords)) + fixtures := DefaultFixtures() + if len(diskRecords) != len(fixtures.Disks) { + t.Fatalf("expected %d disk records, got %d", len(fixtures.Disks), len(diskRecords)) } var sda, nvme0n1, sdc *unifiedresources.IngestRecord @@ -630,8 +631,8 @@ func TestRecordsIncludeDiskResourcesWithCorrectParentChain(t *testing.T) { if sdc.Resource.PhysicalDisk == nil { t.Fatal("expected sdc PhysicalDiskMeta") } - if sdc.Resource.PhysicalDisk.Health != "UNKNOWN" { - t.Fatalf("expected sdc health UNKNOWN, got %q", sdc.Resource.PhysicalDisk.Health) + if sdc.Resource.PhysicalDisk.Health != "DEGRADED" { + t.Fatalf("expected sdc health DEGRADED, got %q", sdc.Resource.PhysicalDisk.Health) } if sdc.Resource.PhysicalDisk.Temperature != 63 { t.Fatalf("expected sdc temperature 63, got %d", sdc.Resource.PhysicalDisk.Temperature) diff --git a/internal/unifiedresources/registry.go b/internal/unifiedresources/registry.go index 681d40f5c..901c385de 100644 --- a/internal/unifiedresources/registry.go +++ b/internal/unifiedresources/registry.go @@ -423,7 +423,11 @@ func BuildMetricsTargetForRegistry(rr *ResourceRegistry, resourceID string) *Met } } - return BuildMetricsTarget(*resource, sourceTargets) + if target := BuildMetricsTarget(*resource, sourceTargets); target != nil { + return target + } + + return cloneMetricsTarget(resource.MetricsTarget) } // GetChildren returns child resources for a parent. diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.go index 66588dbf8..1de798b09 100644 --- a/internal/unifiedresources/registry_test.go +++ b/internal/unifiedresources/registry_test.go @@ -202,6 +202,40 @@ func TestResourceRegistry_IngestsVMwareSourceAsCanonicalResources(t *testing.T) } } +func TestBuildMetricsTargetForRegistryFallsBackToStoredMetricsTarget(t *testing.T) { + rr := NewRegistry(nil) + now := time.Date(2026, 3, 31, 10, 0, 0, 0, time.UTC) + resourceID := CanonicalResourceID("app-demo") + rr.resources[resourceID] = &Resource{ + ID: resourceID, + Type: ResourceTypeAppContainer, + Name: "Nextcloud", + Status: StatusOnline, + LastSeen: now, + MetricsTarget: &MetricsTarget{ + ResourceType: "dockerContainer", + ResourceID: "nextcloud-web-1", + }, + } + + target := BuildMetricsTargetForRegistry(rr, resourceID) + if target == nil { + t.Fatal("expected stored metrics target fallback") + } + if target.ResourceType != "dockerContainer" || target.ResourceID != "nextcloud-web-1" { + t.Fatalf("unexpected fallback metrics target %+v", target) + } + + target.ResourceID = "mutated" + stored, ok := rr.Get(resourceID) + if !ok || stored == nil || stored.MetricsTarget == nil { + t.Fatalf("expected stored app resource with metrics target, got %+v", stored) + } + if stored.MetricsTarget.ResourceID != "nextcloud-web-1" { + t.Fatalf("expected fallback target clone isolation, got %+v", stored.MetricsTarget) + } +} + func TestMergeVMwareDataMergesSignalFieldsWithoutDroppingExistingIdentity(t *testing.T) { existing := &VMwareData{ ConnectionID: "vc-1", diff --git a/internal/unifiedresources/views.go b/internal/unifiedresources/views.go index 422c58e11..4296a7adb 100644 --- a/internal/unifiedresources/views.go +++ b/internal/unifiedresources/views.go @@ -969,6 +969,13 @@ func (v HostView) AgentID() string { return v.r.Agent.AgentID } +func (v HostView) MetricsTarget() *MetricsTarget { + if v.r == nil { + return nil + } + return cloneMetricsTarget(v.r.MetricsTarget) +} + func (v HostView) UptimeSeconds() int64 { if v.r == nil || v.r.Agent == nil { return 0 @@ -3539,6 +3546,13 @@ func (v DockerContainerView) ContainerID() string { return v.r.Docker.ContainerID } +func (v DockerContainerView) MetricsTarget() *MetricsTarget { + if v.r == nil { + return nil + } + return cloneMetricsTarget(v.r.MetricsTarget) +} + func (v DockerContainerView) HostSourceID() string { if v.r == nil || v.r.Docker == nil { return "" diff --git a/internal/unifiedresources/views_test.go b/internal/unifiedresources/views_test.go index 25c8fa3da..983140b8d 100644 --- a/internal/unifiedresources/views_test.go +++ b/internal/unifiedresources/views_test.go @@ -724,6 +724,32 @@ func TestView_HostViewAccessors(t *testing.T) { } } +func TestView_HostViewMetricsTargetAccessor(t *testing.T) { + r := &Resource{ + ID: "host-metrics", + Type: ResourceTypeAgent, + Name: "demo-host", + MetricsTarget: &MetricsTarget{ + ResourceType: "agent", + ResourceID: "agent-demo-host", + }, + } + + v := NewHostView(r) + target := v.MetricsTarget() + if target == nil { + t.Fatal("expected host metrics target") + } + if target.ResourceType != "agent" || target.ResourceID != "agent-demo-host" { + t.Fatalf("unexpected host metrics target %+v", target) + } + + target.ResourceID = "mutated" + if got := v.MetricsTarget(); got == nil || got.ResourceID != "agent-demo-host" { + t.Fatalf("expected cloned host metrics target, got %+v", got) + } +} + func TestView_PhysicalDiskViewAccessors(t *testing.T) { now := time.Date(2026, 2, 10, 12, 5, 0, 0, time.UTC) parentID := "node-parent-1" @@ -1027,6 +1053,46 @@ func TestView_StoragePoolViewAccessors(t *testing.T) { } } +func TestView_DockerContainerViewMetricsTargetAccessor(t *testing.T) { + now := time.Date(2026, 3, 31, 12, 0, 0, 0, time.UTC) + parentID := "host-1" + r := &Resource{ + ID: "app-1", + Type: ResourceTypeAppContainer, + Name: "Nextcloud", + Status: StatusOnline, + LastSeen: now, + ParentID: &parentID, + MetricsTarget: &MetricsTarget{ + ResourceType: "dockerContainer", + ResourceID: "nextcloud-web-1", + }, + Docker: &DockerData{ + ContainerID: "nextcloud", + Image: "docker.io/library/nextcloud:29.0.7", + ContainerState: "running", + }, + } + + v := NewDockerContainerView(r) + if v.ID() != "app-1" || v.ContainerID() != "nextcloud" || v.ParentID() != parentID { + t.Fatalf("unexpected docker container identity id=%q containerID=%q parentID=%q", v.ID(), v.ContainerID(), v.ParentID()) + } + + target := v.MetricsTarget() + if target == nil { + t.Fatal("expected docker container metrics target") + } + if target.ResourceType != "dockerContainer" || target.ResourceID != "nextcloud-web-1" { + t.Fatalf("unexpected docker container metrics target %+v", target) + } + + target.ResourceID = "mutated" + if got := v.MetricsTarget(); got == nil || got.ResourceID != "nextcloud-web-1" { + t.Fatalf("expected cloned docker container metrics target, got %+v", got) + } +} + func TestView_PBSAndPMGInstanceViewAccessors(t *testing.T) { now := time.Date(2026, 2, 10, 12, 6, 0, 0, time.UTC) diff --git a/internal/vmware/fixtures.go b/internal/vmware/fixtures.go index e67b9378c..abd4fabd3 100644 --- a/internal/vmware/fixtures.go +++ b/internal/vmware/fixtures.go @@ -29,18 +29,18 @@ func DefaultFixtures() InventorySnapshot { ClusterName: "Production Cluster", FolderID: "group-h4", FolderName: "Cluster Hosts", - DatastoreIDs: []string{"datastore-201", "datastore-202"}, - DatastoreNames: []string{"nvme-primary", "archive-tier"}, + DatastoreIDs: []string{"datastore-201", "datastore-202", "datastore-203"}, + DatastoreNames: []string{"nvme-primary", "archive-tier", "analytics-vsan"}, OverallStatus: "green", Metrics: &InventoryMetrics{ - CPUPercent: float64Ptr(27.4), - MemoryPercent: float64Ptr(58.1), - MemoryUsedBytes: int64Ptr(49_902_682_112), + CPUPercent: float64Ptr(24.6), + MemoryPercent: float64Ptr(56.3), + MemoryUsedBytes: int64Ptr(48_370_278_400), MemoryTotalBytes: int64Ptr(85_899_345_920), - NetInBytesPerSecond: float64Ptr(1_240_000), - NetOutBytesPerSecond: float64Ptr(980_000), - DiskReadBytesPerSecond: float64Ptr(2_280_000), - DiskWriteBytesPerSecond: float64Ptr(1_920_000), + NetInBytesPerSecond: float64Ptr(1_340_000), + NetOutBytesPerSecond: float64Ptr(1_040_000), + DiskReadBytesPerSecond: float64Ptr(2_480_000), + DiskWriteBytesPerSecond: float64Ptr(2_040_000), }, RecentTasks: []InventoryTask{{ Task: "task-301", @@ -48,6 +48,13 @@ func DefaultFixtures() InventorySnapshot { State: "success", StartedAt: collectedAt.Add(-28 * time.Minute), }}, + RecentEvents: []InventoryEvent{{ + Event: "event-601", + Type: "HostConnectedEvent", + Message: "Host reconnected to vCenter and is reporting healthy telemetry", + User: "svc-pulse", + CreatedAt: collectedAt.Add(-52 * time.Minute), + }}, }, { Host: "host-102", @@ -63,8 +70,8 @@ func DefaultFixtures() InventorySnapshot { ClusterName: "Production Cluster", FolderID: "group-h4", FolderName: "Cluster Hosts", - DatastoreIDs: []string{"datastore-201"}, - DatastoreNames: []string{"nvme-primary"}, + DatastoreIDs: []string{"datastore-201", "datastore-204"}, + DatastoreNames: []string{"nvme-primary", "backup-nfs"}, OverallStatus: "yellow", Metrics: &InventoryMetrics{ CPUPercent: float64Ptr(43.6), @@ -89,6 +96,75 @@ func DefaultFixtures() InventorySnapshot { StartedAt: collectedAt.Add(-3 * time.Minute), }}, }, + { + Host: "host-103", + Name: "esxi-03.lab.local", + ConnectionState: "CONNECTED", + PowerState: "POWERED_ON", + HostUUID: "uuid-host-103", + DatacenterID: "datacenter-1", + DatacenterName: "Primary DC", + ComputeResourceID: "domain-c101", + ComputeResourceName: "Production Cluster", + ClusterID: "domain-c101", + ClusterName: "Production Cluster", + FolderID: "group-h4", + FolderName: "Cluster Hosts", + DatastoreIDs: []string{"datastore-201", "datastore-203"}, + DatastoreNames: []string{"nvme-primary", "analytics-vsan"}, + OverallStatus: "green", + Metrics: &InventoryMetrics{ + CPUPercent: float64Ptr(18.9), + MemoryPercent: float64Ptr(49.2), + MemoryUsedBytes: int64Ptr(42_271_203_328), + MemoryTotalBytes: int64Ptr(85_899_345_920), + NetInBytesPerSecond: float64Ptr(980_000), + NetOutBytesPerSecond: float64Ptr(910_000), + DiskReadBytesPerSecond: float64Ptr(1_860_000), + DiskWriteBytesPerSecond: float64Ptr(1_220_000), + }, + RecentTasks: []InventoryTask{{ + Task: "task-303", + Name: "Enter maintenance mode check", + State: "success", + StartedAt: collectedAt.Add(-43 * time.Minute), + }}, + }, + { + Host: "host-104", + Name: "esxi-04.lab.local", + ConnectionState: "CONNECTED", + PowerState: "POWERED_ON", + HostUUID: "uuid-host-104", + DatacenterID: "datacenter-1", + DatacenterName: "Primary DC", + ComputeResourceID: "domain-c101", + ComputeResourceName: "Production Cluster", + ClusterID: "domain-c101", + ClusterName: "Production Cluster", + FolderID: "group-h4", + FolderName: "Cluster Hosts", + DatastoreIDs: []string{"datastore-202", "datastore-203", "datastore-204"}, + DatastoreNames: []string{"archive-tier", "analytics-vsan", "backup-nfs"}, + OverallStatus: "green", + Metrics: &InventoryMetrics{ + CPUPercent: float64Ptr(31.4), + MemoryPercent: float64Ptr(63.7), + MemoryUsedBytes: int64Ptr(54_722_805_760), + MemoryTotalBytes: int64Ptr(85_899_345_920), + NetInBytesPerSecond: float64Ptr(1_420_000), + NetOutBytesPerSecond: float64Ptr(1_260_000), + DiskReadBytesPerSecond: float64Ptr(2_660_000), + DiskWriteBytesPerSecond: float64Ptr(2_180_000), + }, + RecentEvents: []InventoryEvent{{ + Event: "event-602", + Type: "HostProfileAppliedEvent", + Message: "Host profile compliance restored after storage heartbeat remediation", + User: "svc-pulse", + CreatedAt: collectedAt.Add(-67 * time.Minute), + }}, + }, }, VMs: []InventoryVM{ { @@ -180,7 +256,7 @@ func DefaultFixtures() InventorySnapshot { TriggeredAt: collectedAt.Add(-17 * time.Minute), }}, RecentTasks: []InventoryTask{{ - Task: "task-303", + Task: "task-304", Name: "Consolidate virtual machine disks", State: "queued", StartedAt: collectedAt.Add(-2 * time.Minute), @@ -188,6 +264,120 @@ func DefaultFixtures() InventorySnapshot { }, { VM: "vm-203", + Name: "web-frontend-01", + PowerState: "POWERED_ON", + CPUCount: 4, + MemorySizeMiB: 12 * 1024, + DatacenterID: "datacenter-1", + DatacenterName: "Primary DC", + ComputeResourceID: "domain-c101", + ComputeResourceName: "Production Cluster", + ClusterID: "domain-c101", + ClusterName: "Production Cluster", + FolderID: "group-v7", + FolderName: "Production VMs", + ResourcePoolID: "resgroup-22", + ResourcePoolName: "Tier 1", + RuntimeHostID: "host-103", + RuntimeHostName: "esxi-03.lab.local", + DatastoreIDs: []string{"datastore-203"}, + DatastoreNames: []string{"analytics-vsan"}, + InstanceUUID: "vm-instance-203", + BIOSUUID: "vm-bios-203", + GuestOSFamily: "LINUX", + GuestHostname: "web-frontend-01.internal", + GuestIPAddresses: []string{"10.42.10.44"}, + OverallStatus: "green", + SnapshotCount: 1, + Metrics: &InventoryMetrics{ + CPUPercent: float64Ptr(22.1), + MemoryPercent: float64Ptr(51.2), + MemoryUsedBytes: int64Ptr(6_589_939_712), + MemoryTotalBytes: int64Ptr(12_884_901_888), + NetInBytesPerSecond: float64Ptr(510_000), + NetOutBytesPerSecond: float64Ptr(690_000), + DiskReadBytesPerSecond: float64Ptr(880_000), + DiskWriteBytesPerSecond: float64Ptr(620_000), + }, + }, + { + VM: "vm-204", + Name: "observability-01", + PowerState: "POWERED_ON", + CPUCount: 8, + MemorySizeMiB: 24 * 1024, + DatacenterID: "datacenter-1", + DatacenterName: "Primary DC", + ComputeResourceID: "domain-c101", + ComputeResourceName: "Production Cluster", + ClusterID: "domain-c101", + ClusterName: "Production Cluster", + FolderID: "group-v9", + FolderName: "Platform VMs", + ResourcePoolID: "resgroup-25", + ResourcePoolName: "Platform", + RuntimeHostID: "host-104", + RuntimeHostName: "esxi-04.lab.local", + DatastoreIDs: []string{"datastore-203", "datastore-204"}, + DatastoreNames: []string{"analytics-vsan", "backup-nfs"}, + InstanceUUID: "vm-instance-204", + BIOSUUID: "vm-bios-204", + GuestOSFamily: "LINUX", + GuestHostname: "observability-01.internal", + GuestIPAddresses: []string{"10.42.20.15"}, + OverallStatus: "green", + SnapshotCount: 2, + Metrics: &InventoryMetrics{ + CPUPercent: float64Ptr(27.8), + MemoryPercent: float64Ptr(59.6), + MemoryUsedBytes: int64Ptr(15_367_036_928), + MemoryTotalBytes: int64Ptr(25_769_803_776), + NetInBytesPerSecond: float64Ptr(740_000), + NetOutBytesPerSecond: float64Ptr(680_000), + DiskReadBytesPerSecond: float64Ptr(1_160_000), + DiskWriteBytesPerSecond: float64Ptr(1_020_000), + }, + }, + { + VM: "vm-205", + Name: "windows-jump-01", + PowerState: "POWERED_ON", + CPUCount: 4, + MemorySizeMiB: 8 * 1024, + DatacenterID: "datacenter-1", + DatacenterName: "Primary DC", + ComputeResourceID: "domain-c101", + ComputeResourceName: "Production Cluster", + ClusterID: "domain-c101", + ClusterName: "Production Cluster", + FolderID: "group-v10", + FolderName: "Utility VMs", + ResourcePoolID: "resgroup-26", + ResourcePoolName: "Utility", + RuntimeHostID: "host-103", + RuntimeHostName: "esxi-03.lab.local", + DatastoreIDs: []string{"datastore-203"}, + DatastoreNames: []string{"analytics-vsan"}, + InstanceUUID: "vm-instance-205", + BIOSUUID: "vm-bios-205", + GuestOSFamily: "WINDOWS", + GuestHostname: "windows-jump-01.internal", + GuestIPAddresses: []string{"10.42.30.18"}, + OverallStatus: "green", + SnapshotCount: 0, + Metrics: &InventoryMetrics{ + CPUPercent: float64Ptr(16.8), + MemoryPercent: float64Ptr(46.1), + MemoryUsedBytes: int64Ptr(3_959_726_080), + MemoryTotalBytes: int64Ptr(8_589_934_592), + NetInBytesPerSecond: float64Ptr(180_000), + NetOutBytesPerSecond: float64Ptr(220_000), + DiskReadBytesPerSecond: float64Ptr(320_000), + DiskWriteBytesPerSecond: float64Ptr(280_000), + }, + }, + { + VM: "vm-206", Name: "batch-worker-01", PowerState: "POWERED_OFF", CPUCount: 4, @@ -206,8 +396,8 @@ func DefaultFixtures() InventorySnapshot { RuntimeHostName: "esxi-01.lab.local", DatastoreIDs: []string{"datastore-202"}, DatastoreNames: []string{"archive-tier"}, - InstanceUUID: "vm-instance-203", - BIOSUUID: "vm-bios-203", + InstanceUUID: "vm-instance-206", + BIOSUUID: "vm-bios-206", GuestOSFamily: "LINUX", GuestHostname: "batch-worker-01.internal", GuestIPAddresses: []string{}, @@ -219,14 +409,14 @@ func DefaultFixtures() InventorySnapshot { Datastore: "datastore-201", Name: "nvme-primary", Type: "VMFS", - FreeSpace: 3_800_000_000_000, + FreeSpace: 4_600_000_000_000, Capacity: 8_000_000_000_000, DatacenterID: "datacenter-1", DatacenterName: "Primary DC", FolderID: "group-s4", FolderName: "Datastores", - HostIDs: []string{"host-101", "host-102"}, - HostNames: []string{"esxi-01.lab.local", "esxi-02.lab.local"}, + HostIDs: []string{"host-101", "host-102", "host-103"}, + HostNames: []string{"esxi-01.lab.local", "esxi-02.lab.local", "esxi-03.lab.local"}, VMIDs: []string{"vm-201", "vm-202"}, VMNames: []string{"orders-api-01", "postgres-ha-01"}, Accessible: &accessible, @@ -239,28 +429,80 @@ func DefaultFixtures() InventorySnapshot { Datastore: "datastore-202", Name: "archive-tier", Type: "NFS41", - FreeSpace: 11_000_000_000_000, + FreeSpace: 12_400_000_000_000, Capacity: 16_000_000_000_000, DatacenterID: "datacenter-1", DatacenterName: "Primary DC", FolderID: "group-s4", FolderName: "Datastores", - HostIDs: []string{"host-101"}, - HostNames: []string{"esxi-01.lab.local"}, - VMIDs: []string{"vm-202", "vm-203"}, + HostIDs: []string{"host-101", "host-104"}, + HostNames: []string{"esxi-01.lab.local", "esxi-04.lab.local"}, + VMIDs: []string{"vm-202", "vm-206"}, VMNames: []string{"postgres-ha-01", "batch-worker-01"}, Accessible: &accessible, MultipleHostAccess: &multipleHostAccess, MaintenanceMode: "normal", URL: "ds:///nfs/archive-tier/", OverallStatus: "yellow", + TriggeredAlarms: []InventoryAlarm{{ + Alarm: "alarm-403", + Name: "Datastore latency above threshold", + OverallStatus: "yellow", + TriggeredAt: collectedAt.Add(-9 * time.Minute), + }}, RecentTasks: []InventoryTask{{ - Task: "task-304", + Task: "task-305", Name: "Rescan datastore", State: "success", StartedAt: collectedAt.Add(-14 * time.Minute), }}, }, + { + Datastore: "datastore-203", + Name: "analytics-vsan", + Type: "vSAN", + FreeSpace: 6_900_000_000_000, + Capacity: 12_000_000_000_000, + DatacenterID: "datacenter-1", + DatacenterName: "Primary DC", + FolderID: "group-s4", + FolderName: "Datastores", + HostIDs: []string{"host-101", "host-103", "host-104"}, + HostNames: []string{"esxi-01.lab.local", "esxi-03.lab.local", "esxi-04.lab.local"}, + VMIDs: []string{"vm-203", "vm-204", "vm-205"}, + VMNames: []string{"web-frontend-01", "observability-01", "windows-jump-01"}, + Accessible: &accessible, + MultipleHostAccess: &multipleHostAccess, + MaintenanceMode: "normal", + URL: "ds:///vsan/analytics-vsan/", + OverallStatus: "green", + }, + { + Datastore: "datastore-204", + Name: "backup-nfs", + Type: "NFS41", + FreeSpace: 20_800_000_000_000, + Capacity: 24_000_000_000_000, + DatacenterID: "datacenter-1", + DatacenterName: "Primary DC", + FolderID: "group-s4", + FolderName: "Datastores", + HostIDs: []string{"host-102", "host-104"}, + HostNames: []string{"esxi-02.lab.local", "esxi-04.lab.local"}, + VMIDs: []string{"vm-204"}, + VMNames: []string{"observability-01"}, + Accessible: &accessible, + MultipleHostAccess: &multipleHostAccess, + MaintenanceMode: "normal", + URL: "ds:///nfs/backup-nfs/", + OverallStatus: "green", + RecentTasks: []InventoryTask{{ + Task: "task-306", + Name: "Backup consistency check", + State: "success", + StartedAt: collectedAt.Add(-74 * time.Minute), + }}, + }, }, } } diff --git a/internal/vmware/provider.go b/internal/vmware/provider.go index 5401af88a..ae9443169 100644 --- a/internal/vmware/provider.go +++ b/internal/vmware/provider.go @@ -367,6 +367,7 @@ func vmwareRecordsFromSnapshot(snapshot *InventorySnapshot, now func() time.Time UpdatedAt: collectedAt, Incidents: incidents, Metrics: inventoryMetricsResourceMetrics(host.Metrics), + Agent: vmwareHostAgentData(snapshot, host), VMware: &unifiedresources.VMwareData{ ConnectionID: strings.TrimSpace(snapshot.ConnectionID), ConnectionName: connectionName, @@ -1059,6 +1060,55 @@ func inventoryMetricsResourceMetrics(in *InventoryMetrics) *unifiedresources.Res return metrics } +func vmwareHostAgentData(snapshot *InventorySnapshot, host InventoryHost) *unifiedresources.AgentData { + if snapshot == nil { + return nil + } + + agent := &unifiedresources.AgentData{ + AgentID: vmwareSourceID(snapshot.ConnectionID, "host", host.Host), + Hostname: firstNonEmptyTrimmed(host.Name, host.Host), + MachineID: strings.TrimSpace(host.HostUUID), + Platform: "vmware-vsphere", + OSName: "VMware ESXi", + OSVersion: strings.TrimSpace(snapshot.VIRelease), + NetInRate: inventoryMetricFloat64(host.Metrics, func(m *InventoryMetrics) *float64 { return m.NetInBytesPerSecond }), + NetOutRate: inventoryMetricFloat64(host.Metrics, func(m *InventoryMetrics) *float64 { return m.NetOutBytesPerSecond }), + DiskReadRate: inventoryMetricFloat64(host.Metrics, func(m *InventoryMetrics) *float64 { return m.DiskReadBytesPerSecond }), + DiskWriteRate: inventoryMetricFloat64(host.Metrics, func(m *InventoryMetrics) *float64 { return m.DiskWriteBytesPerSecond }), + } + + if host.Metrics != nil && host.Metrics.MemoryTotalBytes != nil { + total := *host.Metrics.MemoryTotalBytes + used := int64(0) + if host.Metrics.MemoryUsedBytes != nil { + used = *host.Metrics.MemoryUsedBytes + } + free := total - used + if free < 0 { + free = 0 + } + agent.Memory = &unifiedresources.AgentMemoryMeta{ + Total: total, + Used: used, + Free: free, + } + } + + return agent +} + +func inventoryMetricFloat64(metrics *InventoryMetrics, pick func(*InventoryMetrics) *float64) float64 { + if metrics == nil || pick == nil { + return 0 + } + value := pick(metrics) + if value == nil { + return 0 + } + return *value +} + func cloneFloat64Pointer(in *float64) *float64 { if in == nil { return nil diff --git a/pkg/pulsecli/mock.go b/pkg/pulsecli/mock.go index 2f433d8b9..50b5fc8a4 100644 --- a/pkg/pulsecli/mock.go +++ b/pkg/pulsecli/mock.go @@ -4,9 +4,11 @@ import ( "fmt" "os" "path/filepath" + "strconv" "strings" "github.com/rcourtman/pulse-go-rewrite/internal/config" + mockruntime "github.com/rcourtman/pulse-go-rewrite/internal/mock" "github.com/spf13/cobra" ) @@ -148,20 +150,21 @@ func getMockStatus(mock *MockDeps) (bool, []string) { } func getDefaultMockConfig() map[string]string { + cfg := mockruntime.DefaultConfig return map[string]string{ "PULSE_MOCK_MODE": "false", - "PULSE_MOCK_NODES": "7", - "PULSE_MOCK_VMS_PER_NODE": "5", - "PULSE_MOCK_LXCS_PER_NODE": "8", - "PULSE_MOCK_DOCKER_HOSTS": "3", - "PULSE_MOCK_DOCKER_CONTAINERS": "12", - "PULSE_MOCK_GENERIC_HOSTS": "4", - "PULSE_MOCK_K8S_CLUSTERS": "2", - "PULSE_MOCK_K8S_NODES": "4", - "PULSE_MOCK_K8S_PODS": "30", - "PULSE_MOCK_K8S_DEPLOYMENTS": "12", - "PULSE_MOCK_RANDOM_METRICS": "true", - "PULSE_MOCK_STOPPED_PERCENT": "20", + "PULSE_MOCK_NODES": strconv.Itoa(cfg.NodeCount), + "PULSE_MOCK_VMS_PER_NODE": strconv.Itoa(cfg.VMsPerNode), + "PULSE_MOCK_LXCS_PER_NODE": strconv.Itoa(cfg.LXCsPerNode), + "PULSE_MOCK_DOCKER_HOSTS": strconv.Itoa(cfg.DockerHostCount), + "PULSE_MOCK_DOCKER_CONTAINERS": strconv.Itoa(cfg.DockerContainersPerHost), + "PULSE_MOCK_GENERIC_HOSTS": strconv.Itoa(cfg.GenericHostCount), + "PULSE_MOCK_K8S_CLUSTERS": strconv.Itoa(cfg.K8sClusterCount), + "PULSE_MOCK_K8S_NODES": strconv.Itoa(cfg.K8sNodesPerCluster), + "PULSE_MOCK_K8S_PODS": strconv.Itoa(cfg.K8sPodsPerCluster), + "PULSE_MOCK_K8S_DEPLOYMENTS": strconv.Itoa(cfg.K8sDeploymentsPerCluster), + "PULSE_MOCK_RANDOM_METRICS": strconv.FormatBool(cfg.RandomMetrics), + "PULSE_MOCK_STOPPED_PERCENT": strconv.FormatFloat(cfg.StoppedPercent*100, 'f', -1, 64), } } diff --git a/scripts/tests/test-toggle-mock.sh b/scripts/tests/test-toggle-mock.sh index c025ee384..d0cbc7341 100755 --- a/scripts/tests/test-toggle-mock.sh +++ b/scripts/tests/test-toggle-mock.sh @@ -140,10 +140,43 @@ test_managed_hot_dev_stop_uses_control_plane() { assert_contains "stop_hot_dev_runtime uses managed stop command" "${calls}" "stop " } +test_ensure_mock_env_file_seeds_canonical_demo_defaults() { + local state output env_contents + state="$(mktemp -d)" + temp_dirs+=("${state}") + + output="$( + TOGGLE_MOCK_PATH="${TOGGLE_MOCK}" \ + MOCK_ENV_OVERRIDE="${state}/mock.env" \ + bash -lc ' + source "${TOGGLE_MOCK_PATH}" + MOCK_ENV_FILE="${MOCK_ENV_OVERRIDE}" + ensure_mock_env_file + cat "${MOCK_ENV_FILE}" + ' + )" + env_contents="${output}" + + assert_contains "ensure_mock_env_file seeds mock mode default" "${env_contents}" "PULSE_MOCK_MODE=false" + assert_contains "ensure_mock_env_file seeds canonical node count" "${env_contents}" "PULSE_MOCK_NODES=3" + assert_contains "ensure_mock_env_file seeds canonical vm count" "${env_contents}" "PULSE_MOCK_VMS_PER_NODE=3" + assert_contains "ensure_mock_env_file seeds canonical lxc count" "${env_contents}" "PULSE_MOCK_LXCS_PER_NODE=3" + assert_contains "ensure_mock_env_file seeds canonical docker host count" "${env_contents}" "PULSE_MOCK_DOCKER_HOSTS=2" + assert_contains "ensure_mock_env_file seeds canonical docker container count" "${env_contents}" "PULSE_MOCK_DOCKER_CONTAINERS=5" + assert_contains "ensure_mock_env_file seeds canonical generic host count" "${env_contents}" "PULSE_MOCK_GENERIC_HOSTS=2" + assert_contains "ensure_mock_env_file seeds canonical k8s cluster count" "${env_contents}" "PULSE_MOCK_K8S_CLUSTERS=1" + assert_contains "ensure_mock_env_file seeds canonical k8s node count" "${env_contents}" "PULSE_MOCK_K8S_NODES=3" + assert_contains "ensure_mock_env_file seeds canonical k8s pod count" "${env_contents}" "PULSE_MOCK_K8S_PODS=10" + assert_contains "ensure_mock_env_file seeds canonical k8s deployment count" "${env_contents}" "PULSE_MOCK_K8S_DEPLOYMENTS=4" + assert_contains "ensure_mock_env_file seeds canonical random metrics flag" "${env_contents}" "PULSE_MOCK_RANDOM_METRICS=true" + assert_contains "ensure_mock_env_file seeds canonical stopped percent" "${env_contents}" "PULSE_MOCK_STOPPED_PERCENT=6" +} + main() { test_detects_managed_hot_dev_runtime test_managed_hot_dev_start_uses_takeover test_managed_hot_dev_stop_uses_control_plane + test_ensure_mock_env_file_seeds_canonical_demo_defaults if (( failures > 0 )); then echo "Total failures: ${failures}" >&2 diff --git a/scripts/toggle-mock.sh b/scripts/toggle-mock.sh index d6da7ca00..50e809c9d 100755 --- a/scripts/toggle-mock.sh +++ b/scripts/toggle-mock.sh @@ -225,18 +225,18 @@ ensure_mock_env_file() { cat > "$MOCK_ENV_FILE" <<'ENVEOF' # Mock Mode Configuration PULSE_MOCK_MODE=false -PULSE_MOCK_NODES=7 -PULSE_MOCK_VMS_PER_NODE=5 -PULSE_MOCK_LXCS_PER_NODE=8 -PULSE_MOCK_DOCKER_HOSTS=3 -PULSE_MOCK_DOCKER_CONTAINERS=12 -PULSE_MOCK_GENERIC_HOSTS=4 -PULSE_MOCK_K8S_CLUSTERS=2 -PULSE_MOCK_K8S_NODES=4 -PULSE_MOCK_K8S_PODS=30 -PULSE_MOCK_K8S_DEPLOYMENTS=12 +PULSE_MOCK_NODES=3 +PULSE_MOCK_VMS_PER_NODE=3 +PULSE_MOCK_LXCS_PER_NODE=3 +PULSE_MOCK_DOCKER_HOSTS=2 +PULSE_MOCK_DOCKER_CONTAINERS=5 +PULSE_MOCK_GENERIC_HOSTS=2 +PULSE_MOCK_K8S_CLUSTERS=1 +PULSE_MOCK_K8S_NODES=3 +PULSE_MOCK_K8S_PODS=10 +PULSE_MOCK_K8S_DEPLOYMENTS=4 PULSE_MOCK_RANDOM_METRICS=true -PULSE_MOCK_STOPPED_PERCENT=20 +PULSE_MOCK_STOPPED_PERCENT=6 ENVEOF log_info "Created $MOCK_ENV_FILE" diff --git a/tests/integration/tests/43-platform-mock-runtime.spec.ts b/tests/integration/tests/43-platform-mock-runtime.spec.ts index 5ad5ebd75..bfea969ec 100644 --- a/tests/integration/tests/43-platform-mock-runtime.spec.ts +++ b/tests/integration/tests/43-platform-mock-runtime.spec.ts @@ -117,6 +117,16 @@ async function expectInfrastructureSource( await expect(page.getByText(resourceName).first()).toBeVisible(); } +async function expectFirstTableContains( + page: import('@playwright/test').Page, + texts: string[], +): Promise { + const table = page.locator('table').first(); + for (const text of texts) { + await expect(table).toContainText(text); + } +} + test.describe.serial('Platform mock runtime', () => { test.setTimeout(180_000); @@ -150,8 +160,9 @@ test.describe.serial('Platform mock runtime', () => { timeout: 15_000, }); await expect(page.getByText('Archive NAS')).toBeVisible(); - await expect(page.getByText('3 pools')).toBeVisible(); - await expect(page.getByText('5 datasets')).toBeVisible(); + await expect(page.getByText('4 pools')).toBeVisible(); + await expect(page.getByText('9 datasets')).toBeVisible(); + await expect(page.getByText('5 apps')).toBeVisible(); fs.mkdirSync(path.dirname(TRUENAS_SCREENSHOT_PATH), { recursive: true }); await page.screenshot({ path: TRUENAS_SCREENSHOT_PATH, fullPage: true }); await expectInfrastructureSource(page, 'truenas', 'truenas-main'); @@ -163,12 +174,46 @@ test.describe.serial('Platform mock runtime', () => { timeout: 15_000, }); await expect(page.getByText('Lab vCenter')).toBeVisible(); - await expect(page.getByText('2 hosts')).toBeVisible(); - await expect(page.getByText('3 vms')).toBeVisible(); - await expect(page.getByText('2 datastores')).toBeVisible(); + await expect(page.getByText('4 hosts')).toBeVisible(); + await expect(page.getByText('6 vms')).toBeVisible(); + await expect(page.getByText('4 datastores')).toBeVisible(); fs.mkdirSync(path.dirname(VMWARE_SCREENSHOT_PATH), { recursive: true }); await page.screenshot({ path: VMWARE_SCREENSHOT_PATH, fullPage: true }); await expectInfrastructureSource(page, 'vmware-vsphere', 'esxi-01.lab.local'); + await expectFirstTableContains(page, ['esxi-02.lab.local', 'esxi-03.lab.local', 'esxi-04.lab.local']); + + await page.goto('/workloads?type=app-container&platform=truenas&agent=truenas-main', { + waitUntil: 'domcontentloaded', + }); + await page.waitForURL(/\/workloads\?type=app-container&platform=truenas&agent=.*truenas-main/); + await expect(page.locator('#dashboard-type-filter')).toHaveValue('app-container'); + await expect(page.locator('#workloads-platform-filter')).toHaveValue('truenas'); + await expectFirstTableContains(page, ['Nextcloud', 'Immich', 'Paperless-ngx', 'Grafana', 'AdGuard Home']); + await expect(page.getByText('No history yet')).toHaveCount(0); + + await page.goto('/storage?source=truenas&node=truenas-main', { + waitUntil: 'domcontentloaded', + }); + await expect(page).toHaveURL(/\/storage\?source=truenas&node=truenas-main/); + await expectFirstTableContains(page, ['tank', 'fast', 'archive', 'vault']); + await expect(page.getByText('No history yet')).toHaveCount(0); + await expect(page.getByText('Derived from storage metrics - live Ceph telemetry unavailable.')).toHaveCount(0); + await expect(page.locator('table').first()).not.toContainText('UNKNOWN'); + + await page.goto('/storage?source=vmware-vsphere', { + waitUntil: 'domcontentloaded', + }); + await expect(page).toHaveURL(/\/storage\?source=vmware-vsphere/); + await expectFirstTableContains(page, ['nvme-primary', 'archive-tier', 'analytics-vsan', 'backup-nfs']); + await expect(page.getByText('No history yet')).toHaveCount(0); + + await page.goto('/recovery?platform=truenas&node=truenas-main', { + waitUntil: 'domcontentloaded', + }); + await expect(page).toHaveURL(/\/recovery\?platform=truenas&node=truenas-main/); + await expect(page.getByTestId('recovery-page')).toBeVisible(); + await expectFirstTableContains(page, ['tank/apps', 'tank/media', 'vault/compliance']); + await expect(page.getByText('0 fresh in 24h')).toHaveCount(0); for (const [source, resourceName] of Object.entries(resourceNames)) { await expectInfrastructureSource(page, source, resourceName);