mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
Unify mock chart history timelines
This commit is contained in:
@@ -67,6 +67,9 @@ that already belong to the canonical system.
|
||||
named boundary-only exception still owns it. Do not wait for the current
|
||||
slice to have authored the replacement before cleaning up clearly obsolete
|
||||
old-way internals in the surface it is already governing.
|
||||
Do not preserve or polish legacy-primary internal paths once the canonical
|
||||
replacement exists and the remaining obligation is only boundary
|
||||
compatibility.
|
||||
12. Prefer the largest coherent slice.
|
||||
When a claimed lane is already moving through one clear behavior arc on one
|
||||
surface, prefer the largest same-surface slice that still has one coherent
|
||||
|
||||
@@ -38,6 +38,8 @@ These rules are retrospective.
|
||||
When an existing lane is touched, reviewed, or reopened, judge it against the
|
||||
same modernization bar as new lane work rather than assuming earlier work
|
||||
closed it permanently.
|
||||
If a superseded or obsolete internal path remains in the primary governed
|
||||
surface, that is drift. Retire it or record the exception.
|
||||
|
||||
## Required Operating Files
|
||||
|
||||
|
||||
@@ -133,6 +133,7 @@ management, and fleet control surfaces.
|
||||
9. Preserve the dedicated backend-owned `relay:mobile:access` capability and its governed backward-compatible route inventory plus the shared helper call sites around it, so lifecycle-adjacent setup and install flows do not widen the mobile device credential back into general AI chat/execute scope ownership.
|
||||
10. Preserve shipped security-doc guidance in shared lifecycle setup helpers so `internal/api/config_setup_handlers.go` and adjacent install/setup runtime paths point operators at the running build's local security documentation route rather than GitHub `main` links.
|
||||
11. Keep shared `internal/api/router.go` workload-chart downsampling presentation-only: when that router caps mixed-cadence workload history into equal-time buckets for operator-facing cards, lifecycle-adjacent setup and fleet surfaces must not reuse the shaped chart samples as heartbeat, enrollment, or last-seen authority.
|
||||
That same presentation-only boundary must preserve canonical millisecond timestamps when it serializes chart points, so lifecycle-adjacent first-host and fleet surfaces do not misread rounded chart samples as duplicate or restarted heartbeat evidence.
|
||||
|
||||
## Forbidden Paths
|
||||
|
||||
|
||||
@@ -175,6 +175,7 @@ Own canonical runtime payload shapes between backend and frontend.
|
||||
22. Keep post-boot AI enablement contract-backed on the shared AI/mobile approval surface: `internal/api/ai_handler.go`, `internal/api/ai_handlers.go`, `internal/api/router_routes_ai_relay.go`, and `internal/api/contract_test.go` must turn the governed approvals-list API into the canonical empty-list payload as soon as settings-driven AI enablement succeeds, rather than leaving that surface on `503 Approval store not initialized` until some separate startup-only side effect happens.
|
||||
23. Keep infrastructure summary chart transport contract-backed on the shared API surface: `internal/api/router.go`, `internal/api/contract_test.go`, and frontend infrastructure summary consumers must normalize long-range mixed-cadence history into equal-time summary buckets before shipping the infrastructure charts API payload, so 7-day and 30-day summary cards do not expose compressed right-edge tails just because recent samples arrive at a finer storage resolution.
|
||||
24. Keep long-range workload chart transport time-proportional on the shared API surface: `internal/api/router.go`, `internal/api/contract_test.go`, and workload chart consumers must cap mixed-cadence workload history by equal-time buckets rather than raw point index for the per-workload and aggregate workload chart APIs, so 7-day and 30-day workload cards do not bunch recent samples at the right edge just because recent telemetry is stored more densely.
|
||||
25. Keep chart timestamp precision canonical on that same shared API surface: when `internal/api/router.go` serializes monitoring history into infrastructure or workload chart payloads, it must preserve canonical millisecond timestamps from the shared monitoring timeline instead of rounding through whole-second conversion, so seeded mock history and live appends collapse onto one operator-visible timeline instead of appearing as duplicated tail samples.
|
||||
|
||||
## Forbidden Paths
|
||||
|
||||
@@ -207,37 +208,37 @@ Own canonical runtime payload shapes between backend and frontend.
|
||||
2. Update frontend API types in the same slice
|
||||
3. Route runtime changes through the explicit API-contract proof policies in `registry.json`; default fallback proof routing is not allowed
|
||||
4. Update this contract when canonical payload ownership changes
|
||||
5. Keep mock recovery handlers graph-first: `/api/recovery/rollups`, `/api/recovery/points`, `/api/recovery/series`, and `/api/recovery/facets` must derive mock data from `internal/mock/fixture_graph.go` and its graph-owned `RecoveryPoints()` projection instead of route-local mock caches or legacy helper exports
|
||||
5. Keep `/api/resources` policy metadata aligned across backend payload tests and canonical frontend resource consumers whenever sensitivity or routing fields change
|
||||
6. Keep Patrol status payloads explicit enough that the frontend can present blocked runtime state without treating a previously healthy summary snapshot as current runtime truth, and keep Patrol recency semantics explicit in transport by reserving `last_patrol_at` for completed full patrols while exposing any Patrol activity separately through `last_activity_at`
|
||||
and the scoped-trigger status payload on that same Patrol status surface, so queued scoped work, busy-mode state, and per-source enablement (`alert` versus `anomaly`) stay transport-backed instead of being inferred by page-local heuristics
|
||||
and the split Patrol trigger settings contract, so `patrol_alert_triggers_enabled` and `patrol_anomaly_triggers_enabled` are the canonical AI settings fields while legacy `patrol_event_triggers_enabled` remains a compatibility aggregate rather than the primary control surface
|
||||
7. Keep Patrol summary payload consumers aligned on one assessment hierarchy: transport-driven Patrol summary surfaces may show supporting counts and outcomes, but the canonical assessment and verification states must remain singular and not be repeated as a second compact verdict strip
|
||||
8. Treat Patrol summary supporting metrics as readouts, not reinterpretations: when frontend consumers derive cards such as active findings, criticals, warnings, or fixes from the canonical payloads, those cards must stay numeric and must not synthesize new assessment labels like `Issues detected` or verification labels like `Partial verification` beneath the primary summary contract
|
||||
9. Treat active Patrol runtime transport as compatible with factual activity surfaces: when the runtime is currently running, frontend consumers may surface in-progress activity context, but they must not replace the activity strip with a second assessment verdict derived from runtime state alone
|
||||
10. Treat Patrol recency as a singular transport-driven fact: once header metadata, verification copy, or the findings footer already present the governed Patrol timing context, frontend summary consumers must not derive an extra timing pill from the same payloads inside the primary summary card
|
||||
11. Treat Patrol findings counts as a singular supporting surface as well: when the summary shell already exposes count cards for active findings, warnings, criticals, and fixes, the primary assessment card must not repeat those same payload-derived counts as secondary badges
|
||||
12. Treat Patrol schedule and recency as header-owned metadata on the main Patrol page: findings empty-state consumers should not receive or restate `next_patrol_at`, `last_patrol_at`, `last_activity_at`, or interval timing once those transport fields are already presented by the primary header and verification shell
|
||||
13. Keep recovery payload filters canonical across `/api/recovery/rollups`, `/api/recovery/points`, `/api/recovery/series`, and `/api/recovery/facets`: when `internal/api/recovery_handlers.go` adds a governed recovery filter or display field such as provider-neutral `itemType`, the same normalized transport must land across all four endpoints and the contract tests must pin both outbound payload shape and accepted query aliases in the same slice
|
||||
14. Keep recovery platform-query vocabulary canonical across that same `/api/recovery/*` surface: operator-facing transport must emit `platform` as the canonical query field, accepted legacy `provider` aliases must remain compatibility-only input, and `internal/api/contract_test.go` must pin that fallback behavior in the same slice as any handler change
|
||||
15. Keep recovery payload platform vocabulary canonical across that same `/api/recovery/*` surface: point payloads must expose `platform`, rollup payloads must expose `platforms`, and any compatibility `provider` / `providers` aliases must remain secondary fallback fields rather than replacing the shared response model
|
||||
16. Keep recovery linked-resource vocabulary canonical across that same `/api/recovery/*` surface: points and rollups must expose `itemResourceId` as the canonical linked-resource field, accepted legacy `subjectResourceId` aliases must remain compatibility-only input or secondary payload fields, and the shared proof surface must pin that normalization in the same slice as any handler change
|
||||
17. Keep recovery external item-reference vocabulary canonical across that same `/api/recovery/*` surface: point and rollup payloads must expose `itemRef` as the canonical external item-reference field, accepted legacy `subjectRef` aliases must remain compatibility-only secondary payload fields, and the shared proof surface must pin that normalization in the same slice as any handler change
|
||||
18. Keep first-host lookup completion explicit on the shared install-state API
|
||||
8. Keep Patrol verification and activity facts unified on one transport-backed secondary status area: when frontend consumers combine Patrol status payloads (`runtime_state`, `last_patrol_at`, `last_activity_at`, `trigger_status`) with run-history transport, the latest run result, activity mix, scoped-trigger state, and circuit-breaker context must read as one supporting explanation beneath the primary assessment instead of being re-expanded into a separate full-width status strip plus duplicate summary layers
|
||||
9. Treat Patrol summary supporting metrics as readouts, not reinterpretations: when frontend consumers derive cards such as active findings, criticals, warnings, or fixes from the canonical payloads, those cards must stay numeric and must not synthesize new assessment labels like `Issues detected` or verification labels like `Partial verification` beneath the primary summary contract
|
||||
10. Treat active Patrol runtime transport as compatible with factual activity surfaces: when the runtime is currently running, frontend consumers may surface in-progress activity context, but they must not replace the activity strip with a second assessment verdict derived from runtime state alone
|
||||
11. Treat Patrol recency as a singular transport-driven fact: once header metadata, verification copy, or the findings footer already present the governed Patrol timing context, frontend summary consumers must not derive an extra timing pill from the same payloads inside the primary summary card
|
||||
12. Treat Patrol findings counts as a singular supporting surface as well: when the summary shell already exposes count cards for active findings, warnings, criticals, and fixes, the primary assessment card must not repeat those same payload-derived counts as secondary badges
|
||||
13. Treat Patrol schedule and recency as header-owned metadata on the main Patrol page: findings empty-state consumers should not receive or restate `next_patrol_at`, `last_patrol_at`, `last_activity_at`, or interval timing once those transport fields are already presented by the primary header and verification shell
|
||||
14. Keep recovery payload filters canonical across `/api/recovery/rollups`, `/api/recovery/points`, `/api/recovery/series`, and `/api/recovery/facets`: when `internal/api/recovery_handlers.go` adds a governed recovery filter or display field such as provider-neutral `itemType`, the same normalized transport must land across all four endpoints and the contract tests must pin both outbound payload shape and accepted query aliases in the same slice
|
||||
15. Keep recovery platform-query vocabulary canonical across that same `/api/recovery/*` surface: operator-facing transport must emit `platform` as the canonical query field, accepted legacy `provider` aliases must remain compatibility-only input, and `internal/api/contract_test.go` must pin that fallback behavior in the same slice as any handler change
|
||||
16. Keep recovery payload platform vocabulary canonical across that same `/api/recovery/*` surface: point payloads must expose `platform`, rollup payloads must expose `platforms`, and any compatibility `provider` / `providers` aliases must remain secondary fallback fields rather than replacing the shared response model
|
||||
17. Keep recovery linked-resource vocabulary canonical across that same `/api/recovery/*` surface: points and rollups must expose `itemResourceId` as the canonical linked-resource field, accepted legacy `subjectResourceId` aliases must remain compatibility-only input or secondary payload fields, and the shared proof surface must pin that normalization in the same slice as any handler change
|
||||
18. Keep recovery external item-reference vocabulary canonical across that same `/api/recovery/*` surface: point and rollup payloads must expose `itemRef` as the canonical external item-reference field, accepted legacy `subjectRef` aliases must remain compatibility-only secondary payload fields, and the shared proof surface must pin that normalization in the same slice as any handler change
|
||||
19. Keep first-host lookup completion explicit on the shared install-state API
|
||||
boundary: when
|
||||
`frontend-modern/src/components/Settings/useInfrastructureInstallState.tsx`
|
||||
receives a successful connected-agent lookup result, the canonical install
|
||||
flow must expose direct navigation into `/dashboard` and
|
||||
`/settings/infrastructure/operations` rather than leaving the operator on a
|
||||
transport-only status readout.
|
||||
19. Keep the shared first-host detection contract explicit on `/api/state` as
|
||||
20. Keep the shared first-host detection contract explicit on `/api/state` as
|
||||
used by
|
||||
`frontend-modern/src/components/Settings/useInfrastructureInstallState.tsx`:
|
||||
the canonical `connectedInfrastructure` projection must stay suitable for
|
||||
detecting the first active reporting system during install so brand-new
|
||||
operators can receive the first success handoff without typing a hostname
|
||||
or agent ID.
|
||||
20. Keep the shared first-run install-token transport explicit on
|
||||
21. Keep the shared first-run install-token transport explicit on
|
||||
`/api/security/tokens` as used by
|
||||
`frontend-modern/src/components/Settings/useInfrastructureInstallState.tsx`:
|
||||
once quick setup has produced the setup handoff credentials, the canonical
|
||||
@@ -248,7 +249,7 @@ Own canonical runtime payload shapes between backend and frontend.
|
||||
install-state surface must describe that prepared token path consistently
|
||||
with the live runtime behavior rather than directing the operator to create
|
||||
another install token manually.
|
||||
21. Keep connected-infrastructure surface vocabulary canonical across the
|
||||
22. Keep connected-infrastructure surface vocabulary canonical across the
|
||||
shared `/api/state` and reporting/install consumers:
|
||||
`frontend-modern/src/types/api.ts` must treat `truenas` as a first-class
|
||||
connected-infrastructure surface kind, and connected-infrastructure
|
||||
@@ -260,7 +261,7 @@ Own canonical runtime payload shapes between backend and frontend.
|
||||
(`agent`, `docker`, `kubernetes`) and platform-connections-managed
|
||||
surfaces (`proxmox`, `pbs`, `pmg`, `truenas`) instead of collapsing them
|
||||
into one uninstall/stop-monitoring model.
|
||||
22. Keep API-backed first-target onboarding canonical on that same shared
|
||||
23. Keep API-backed first-target onboarding canonical on that same shared
|
||||
infrastructure-settings boundary:
|
||||
`frontend-modern/src/components/Settings/infrastructureOperationsModel.tsx`,
|
||||
`frontend-modern/src/components/Settings/useInfrastructureInstallState.tsx`,
|
||||
@@ -715,13 +716,6 @@ inventory through `source=truenas` and `source=vmware-vsphere`. Shared query
|
||||
parsing may accept `vmware-vsphere` as the operator-facing VMware alias, but
|
||||
the emitted canonical resource source remains the shared `vmware` source
|
||||
family rather than a second backend source key.
|
||||
That same runtime mock contract now also owns fixture authority. Mock
|
||||
connection payloads returned from `/api/truenas/connections` and
|
||||
`/api/vmware/connections` must derive from the shared `internal/mock/`
|
||||
platform fixture owner rather than handler-local fixture assembly, so settings
|
||||
payloads stay aligned with the unified runtime mock graph, storage/recovery
|
||||
context, and seeded monitoring history.
|
||||
That same runtime mock contract now also owns demo-grade dataset balance. When mock mode is enabled, `/api/truenas/connections`, `/api/vmware/connections`, and the shared `/api/resources` surface must project TrueNAS app-container counts, VMware datastore counts, and related settings/runtime metadata from the canonical `internal/mock/fixture_graph.go` owner so settings, infrastructure, workloads, storage, and recovery all present the same user-facing demo dataset.
|
||||
That same VMware test contract now also owns structured setup-failure
|
||||
classification. When `POST /api/vmware/connections/test` or
|
||||
`POST /api/vmware/connections/{id}/test` fails, the backend payload must
|
||||
|
||||
@@ -120,6 +120,13 @@ The mock update loop must keep provider-backed TrueNAS and VMware records plus
|
||||
legacy PBS and PMG summaries on current `LastSeen` and health state each tick,
|
||||
so long-lived infrastructure, workloads, storage, and recovery demos do not
|
||||
decay into synthetic stale-state warnings while mock mode remains enabled.
|
||||
That same demo-owned mock boundary also owns chart continuity. Seeded mock
|
||||
history and runtime mock sampling must be projections of the same canonical
|
||||
metric timeline, so changing chart ranges feels like zooming one history
|
||||
window instead of stitching a second live tail onto the end of seeded
|
||||
sparklines. Monitoring must not let provider-owned mock resources receive a
|
||||
duplicate generic unified-resource writer that appends a divergent recent tail
|
||||
after the canonical mock sampler has already seeded and extended that series.
|
||||
That same summary owner also owns VMware partial-success classification.
|
||||
Optional VI JSON or Automation enrichment reads that fail after base
|
||||
host/VM/datastore inventory succeeds must not collapse the whole poll into a
|
||||
|
||||
@@ -218,6 +218,12 @@ query only if the runtime preserves ascending timestamps plus correct
|
||||
avg/min/max bucket aggregates within every returned resource/metric series, and
|
||||
the hot-path proof keeps latency, ordering, and aggregate correctness guarded
|
||||
together in the explicit metrics SLO surface. That protected surface should
|
||||
That same workload-summary hot path now also owns chart-cache invalidation
|
||||
whenever the shaped timeline contract changes. `frontend-modern/src/components/Workloads/WorkloadsSummary.tsx`
|
||||
must version-bust cached summary payloads in the same slice that changes
|
||||
workload chart bucket semantics or timestamp precision, so operators are not
|
||||
served stale mixed-cadence chart shapes after the backend timeline model has
|
||||
already been corrected.
|
||||
stay on one ordered index scan plus Go-side bucket aggregation rather than
|
||||
forcing SQLite to `GROUP BY` computed buckets through a temp B-tree on the
|
||||
fleet-scale dashboard path.
|
||||
|
||||
@@ -215,6 +215,12 @@ recovery surfaces. Mock summary cards, seeded history, and provider-backed
|
||||
storage/recovery counts must come from the same canonical fixture graph so
|
||||
storage and recovery demos show realistic healthy-versus-attention balance
|
||||
instead of blank history, stale provider context, or page-local fixture drift.
|
||||
That same adjacent chart boundary now also assumes seeded and live mock
|
||||
storage timelines are one continuous series. Disk-temperature, pool-usage,
|
||||
used-capacity, and available-space cards may consume shaped chart payloads for
|
||||
presentation, but those payloads must still reflect one canonical mock metric
|
||||
timeline instead of a seeded seven-day sparkline with a second live tail
|
||||
stitched on afterward.
|
||||
That same shared `internal/api/` dependency also assumes auth-persistence
|
||||
teardown is synchronous when recovery-adjacent runtimes reinitialize. Session,
|
||||
CSRF, and recovery-token workers may not leave stale background goroutines or
|
||||
|
||||
@@ -144,6 +144,37 @@ describe('WorkloadsSummary performance behavior', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('ignores stale cache versions while waiting for live fetch', async () => {
|
||||
const staleWorkloadId = 'cluster-a:pve1:stale';
|
||||
localStorage.setItem(
|
||||
'pulse.workloadsSummaryCharts.default::1h::__all__',
|
||||
JSON.stringify({
|
||||
version: 0,
|
||||
range: '1h',
|
||||
nodeScope: '',
|
||||
cachedAt: Date.now(),
|
||||
data: makeChartsResponse([staleWorkloadId]).data,
|
||||
dockerData: {},
|
||||
}),
|
||||
);
|
||||
mockGetWorkloadCharts.mockImplementationOnce(() => new Promise(() => {}));
|
||||
|
||||
render(() => (
|
||||
<WorkloadsSummary
|
||||
timeRange="1h"
|
||||
fallbackGuestCounts={{ total: 1, running: 1, stopped: 0 }}
|
||||
fallbackSnapshots={makeSnapshots(['cluster-a:pve1:fresh'])}
|
||||
/>
|
||||
));
|
||||
|
||||
await waitFor(() => {
|
||||
expect(mockGetWorkloadCharts).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
expect(screen.queryByTestId('sparkline')).not.toBeInTheDocument();
|
||||
expect(screen.getAllByTestId('sparkline-skeleton')).toHaveLength(4);
|
||||
});
|
||||
|
||||
it('requests fewer chart points for large workload sets', async () => {
|
||||
mockGetWorkloadCharts.mockResolvedValueOnce(makeChartsResponse([]));
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ const WORKLOAD_COLORS = [
|
||||
];
|
||||
|
||||
const WORKLOADS_SUMMARY_CACHE_PREFIX = 'pulse.workloadsSummaryCharts.';
|
||||
const WORKLOADS_SUMMARY_CACHE_VERSION = 3;
|
||||
const WORKLOADS_SUMMARY_CACHE_VERSION = 5;
|
||||
const WORKLOADS_SUMMARY_CACHE_MAX_AGE_MS = 5 * 60_000;
|
||||
const WORKLOADS_SUMMARY_CACHE_MAX_POINTS_PER_SERIES = 360;
|
||||
const WORKLOADS_SUMMARY_CACHE_MAX_CHARS = 900_000;
|
||||
@@ -407,7 +407,8 @@ export const WorkloadsSummary: Component<WorkloadsSummaryProps> = (props) => {
|
||||
const selectedRange = createMemo<TimeRange>(() => props.timeRange || '1h');
|
||||
const selectedNodeScope = createMemo(() => props.selectedNodeId?.trim() || '');
|
||||
const activeScopeKey = createMemo(
|
||||
() => `${orgScope()}::${selectedRange()}::${selectedNodeScope()}`,
|
||||
() =>
|
||||
`${WORKLOADS_SUMMARY_CACHE_VERSION}::${orgScope()}::${selectedRange()}::${selectedNodeScope()}`,
|
||||
);
|
||||
const hasCurrentRangeData = createMemo(() => loadedScopeKey() === activeScopeKey());
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
|
||||
)
|
||||
|
||||
// TestChartResponseTypes verifies the ChartResponse struct fields
|
||||
@@ -196,6 +197,31 @@ func TestMetricPointStructure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvertMetricsForChart_PreservesMillisecondTimestamps(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
oldest := time.Now().UnixMilli()
|
||||
ts := time.Date(2026, time.March, 31, 12, 0, 0, 789_000_000, time.UTC)
|
||||
|
||||
converted := convertMetricsForChart(
|
||||
map[string][]monitoring.MetricPoint{
|
||||
"memory": {
|
||||
{Timestamp: ts, Value: 42},
|
||||
},
|
||||
},
|
||||
&oldest,
|
||||
180,
|
||||
)
|
||||
|
||||
points := converted["memory"]
|
||||
if len(points) != 1 {
|
||||
t.Fatalf("expected 1 converted point, got %d", len(points))
|
||||
}
|
||||
if points[0].Timestamp != ts.UnixMilli() {
|
||||
t.Fatalf("expected timestamp %d, got %d", ts.UnixMilli(), points[0].Timestamp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTimeRangeConversion(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -318,12 +344,21 @@ func TestNormalizeInfrastructureSummaryMetricPointSeriesAveragesPlateauMetrics(t
|
||||
durationMillis := duration.Milliseconds()
|
||||
firstBucketEnd := windowStart.UnixMilli() + durationMillis/int64(bucketCount)
|
||||
|
||||
points := []MetricPoint{
|
||||
{Timestamp: windowStart.UnixMilli() + 1_000, Value: 20},
|
||||
{Timestamp: windowStart.UnixMilli() + 30_000, Value: 40},
|
||||
{Timestamp: firstBucketEnd - 1_000, Value: 80},
|
||||
{Timestamp: now.UnixMilli(), Value: 55},
|
||||
points := make([]MetricPoint, 0, bucketCount+2)
|
||||
points = append(points,
|
||||
MetricPoint{Timestamp: windowStart.UnixMilli() + 1_000, Value: 20},
|
||||
MetricPoint{Timestamp: windowStart.UnixMilli() + 30_000, Value: 40},
|
||||
MetricPoint{Timestamp: firstBucketEnd - 1_000, Value: 80},
|
||||
)
|
||||
for bucketIndex := 1; bucketIndex < bucketCount-1; bucketIndex++ {
|
||||
bucketStart := windowStart.UnixMilli() + (int64(bucketIndex)*durationMillis)/int64(bucketCount)
|
||||
bucketEnd := windowStart.UnixMilli() + (int64(bucketIndex+1)*durationMillis)/int64(bucketCount)
|
||||
points = append(points, MetricPoint{
|
||||
Timestamp: bucketStart + (bucketEnd-bucketStart)/2,
|
||||
Value: float64(20 + bucketIndex),
|
||||
})
|
||||
}
|
||||
points = append(points, MetricPoint{Timestamp: now.UnixMilli(), Value: 55})
|
||||
|
||||
normalized := normalizeInfrastructureSummaryMetricPointSeries(points, "memory", duration, now.UnixMilli())
|
||||
if len(normalized) < 2 {
|
||||
|
||||
@@ -88,6 +88,21 @@ func TestContract_WireAIChatDependencies_WiresTrueNASAppActionProvider(t *testin
|
||||
}
|
||||
}
|
||||
|
||||
func TestContract_ChartMetricPointsPreserveMillisecondPrecision(t *testing.T) {
|
||||
pointTime := time.Date(2026, time.March, 31, 12, 0, 0, 987_000_000, time.UTC)
|
||||
|
||||
converted := monitorPointsToAPI([]monitoring.MetricPoint{{
|
||||
Timestamp: pointTime,
|
||||
Value: 42,
|
||||
}})
|
||||
if len(converted) != 1 {
|
||||
t.Fatalf("expected one converted point, got %d", len(converted))
|
||||
}
|
||||
if converted[0].Timestamp != pointTime.UnixMilli() {
|
||||
t.Fatalf("expected millisecond timestamp %d, got %d", pointTime.UnixMilli(), converted[0].Timestamp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContract_TrueNASConnectionsDisabledMessageIsExplicit(t *testing.T) {
|
||||
setTrueNASFeatureForTest(t, false)
|
||||
handler, _, _ := newTrueNASHandlersForTest(t, nil)
|
||||
@@ -580,6 +595,59 @@ func TestContract_WorkloadsSummaryChartsNormalizeLongRangeMixedCadence(t *testin
|
||||
}
|
||||
}
|
||||
|
||||
func TestContract_GenerateStyledMockSeries_UsesTimestampBasedCurve(t *testing.T) {
|
||||
now := time.Date(2026, time.March, 31, 12, 0, 0, 0, time.UTC).UnixMilli()
|
||||
|
||||
coarse := generateStyledMockSeries(
|
||||
now,
|
||||
time.Hour,
|
||||
7,
|
||||
51.9,
|
||||
0,
|
||||
100,
|
||||
"history-mock",
|
||||
"orion-2-f54579833f9c",
|
||||
"memory",
|
||||
)
|
||||
fine := generateStyledMockSeries(
|
||||
now,
|
||||
time.Hour,
|
||||
13,
|
||||
51.9,
|
||||
0,
|
||||
100,
|
||||
"history-mock",
|
||||
"orion-2-f54579833f9c",
|
||||
"memory",
|
||||
)
|
||||
|
||||
if len(coarse) != 7 || len(fine) != 13 {
|
||||
t.Fatalf("unexpected synthetic series lengths coarse=%d fine=%d", len(coarse), len(fine))
|
||||
}
|
||||
|
||||
for i, point := range coarse {
|
||||
fineIndex := i * 2
|
||||
if fine[fineIndex].Timestamp != point.Timestamp {
|
||||
t.Fatalf(
|
||||
"expected shared timestamp at coarse[%d]=%d to match fine[%d]=%d",
|
||||
i,
|
||||
point.Timestamp,
|
||||
fineIndex,
|
||||
fine[fineIndex].Timestamp,
|
||||
)
|
||||
}
|
||||
if fine[fineIndex].Value != point.Value {
|
||||
t.Fatalf(
|
||||
"expected shared timestamp value at coarse[%d]=%f to match fine[%d]=%f",
|
||||
i,
|
||||
point.Value,
|
||||
fineIndex,
|
||||
fine[fineIndex].Value,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestContract_PlatformMockToggleRebindsRuntimeConnectionsAndResources(t *testing.T) {
|
||||
t.Setenv("PULSE_MOCK_MODE", "false")
|
||||
prevMock := mock.IsMockEnabled()
|
||||
|
||||
+24
-20
@@ -4958,7 +4958,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
chartData := make(map[string]VMChartData)
|
||||
nodeData := make(map[string]NodeChartData)
|
||||
|
||||
currentTime := time.Now().Unix() * 1000 // JavaScript timestamp format
|
||||
currentTime := time.Now().UnixMilli() // JavaScript timestamp format
|
||||
oldestTimestamp := currentTime
|
||||
|
||||
// Process VMs - batch-load historical data (1-2 SQL calls instead of N).
|
||||
@@ -4989,7 +4989,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
chartData[vid][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -5037,7 +5037,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
chartData[cid][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -5083,7 +5083,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
if usagePoints, found := batchMetrics["usage"]; found && len(usagePoints) > 0 {
|
||||
storageData[sid]["disk"] = make([]MetricPoint, len(usagePoints))
|
||||
for i, point := range usagePoints {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -5131,7 +5131,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
nodeData[nid][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -5217,7 +5217,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
dockerData[responseKey][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -5267,7 +5267,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
dockerHostData[dhID][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -5314,7 +5314,7 @@ func (r *Router) handleCharts(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
agentData[hID][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -5808,7 +5808,7 @@ func convertMetricsForChart(
|
||||
}
|
||||
points := make([]MetricPoint, len(metricPoints))
|
||||
for i, point := range metricPoints {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < *oldestTimestamp {
|
||||
*oldestTimestamp = ts
|
||||
}
|
||||
@@ -5893,7 +5893,6 @@ func generateStyledMockSeries(
|
||||
) []MetricPoint {
|
||||
seed := monitoring.HashSeed(seedPrefix, resourceID, metricType)
|
||||
style := mockMetricStyle(metricType)
|
||||
values := monitoring.GenerateSeededSeries(current, numPoints, seed, min, max, style)
|
||||
|
||||
durationMillis := int64(duration / time.Millisecond)
|
||||
if durationMillis <= 0 {
|
||||
@@ -5904,6 +5903,11 @@ func generateStyledMockSeries(
|
||||
step = 1
|
||||
}
|
||||
startMillis := nowMillis - durationMillis
|
||||
timestamps := make([]time.Time, numPoints)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
timestamps[i] = time.UnixMilli(startMillis + int64(i)*step)
|
||||
}
|
||||
values := monitoring.GenerateSeededMetricSeriesForTimestamps(current, timestamps, seed, min, max, metricType, style)
|
||||
points := make([]MetricPoint, numPoints)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
points[i] = MetricPoint{
|
||||
@@ -6045,7 +6049,7 @@ func (r *Router) handleWorkloadCharts(w http.ResponseWriter, req *http.Request)
|
||||
primarySourceHint = "store_or_memory_fallback"
|
||||
}
|
||||
|
||||
currentTime := time.Now().Unix() * 1000
|
||||
currentTime := time.Now().UnixMilli()
|
||||
oldestTimestamp := currentTime
|
||||
|
||||
var selectedNode *models.Node
|
||||
@@ -6409,7 +6413,7 @@ func (r *Router) handleInfrastructureCharts(w http.ResponseWriter, req *http.Req
|
||||
primarySourceHint = "store_or_memory_fallback"
|
||||
}
|
||||
|
||||
currentTime := time.Now().Unix() * 1000
|
||||
currentTime := time.Now().UnixMilli()
|
||||
oldestTimestamp := currentTime
|
||||
|
||||
// Process Nodes - batch-load historical data (1-2 SQL calls instead of N×5).
|
||||
@@ -6443,7 +6447,7 @@ func (r *Router) handleInfrastructureCharts(w http.ResponseWriter, req *http.Req
|
||||
}
|
||||
nodeData[nid][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -6510,7 +6514,7 @@ func (r *Router) handleInfrastructureCharts(w http.ResponseWriter, req *http.Req
|
||||
}
|
||||
dockerHostData[dhID][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -6558,7 +6562,7 @@ func (r *Router) handleInfrastructureCharts(w http.ResponseWriter, req *http.Req
|
||||
}
|
||||
agentData[hID][metricType] = make([]MetricPoint, len(points))
|
||||
for i, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts < oldestTimestamp {
|
||||
oldestTimestamp = ts
|
||||
}
|
||||
@@ -6810,7 +6814,7 @@ func appendWorkloadMetricPoints(
|
||||
) int {
|
||||
added := 0
|
||||
for _, point := range points {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts <= 0 {
|
||||
continue
|
||||
}
|
||||
@@ -6845,14 +6849,14 @@ func mergeWorkloadNetworkPoints(
|
||||
) []monitoring.MetricPoint {
|
||||
totals := make(map[int64]float64)
|
||||
for _, point := range netIn {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts <= 0 {
|
||||
continue
|
||||
}
|
||||
totals[ts] += clampNonNegativeWorkloadValue(point.Value)
|
||||
}
|
||||
for _, point := range netOut {
|
||||
ts := point.Timestamp.Unix() * 1000
|
||||
ts := point.Timestamp.UnixMilli()
|
||||
if ts <= 0 {
|
||||
continue
|
||||
}
|
||||
@@ -7086,7 +7090,7 @@ func (r *Router) handleWorkloadsSummaryCharts(w http.ResponseWriter, req *http.R
|
||||
primarySourceHint = "store_or_memory_fallback"
|
||||
}
|
||||
|
||||
currentTime := time.Now().Unix() * 1000
|
||||
currentTime := time.Now().UnixMilli()
|
||||
currentTimeTime := time.UnixMilli(currentTime)
|
||||
oldestTimestamp := currentTime
|
||||
buckets := make(map[int64]*workloadSummaryBuckets)
|
||||
@@ -7711,7 +7715,7 @@ func monitorPointsToAPI(points []monitoring.MetricPoint) []MetricPoint {
|
||||
}
|
||||
out := make([]MetricPoint, len(points))
|
||||
for i, p := range points {
|
||||
out[i] = MetricPoint{Timestamp: p.Timestamp.Unix() * 1000, Value: p.Value}
|
||||
out[i] = MetricPoint{Timestamp: p.Timestamp.UnixMilli(), Value: p.Value}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
+369
-284
File diff suppressed because it is too large
Load Diff
@@ -1,8 +1,10 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
)
|
||||
@@ -120,6 +122,19 @@ func TestBuildFixtureStateIncludesHostAgents(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeMockBlendWeight_ComposesAcrossUpdateInterval(t *testing.T) {
|
||||
perMinuteWeight := 0.22
|
||||
perTickWeight := normalizeMockBlendWeight(perMinuteWeight, updateInterval, time.Minute)
|
||||
compounded := 1 - math.Pow(1-perTickWeight, float64(time.Minute/updateInterval))
|
||||
|
||||
if perTickWeight >= perMinuteWeight {
|
||||
t.Fatalf("expected per-tick weight %.6f to be less than per-minute weight %.6f", perTickWeight, perMinuteWeight)
|
||||
}
|
||||
if math.Abs(compounded-perMinuteWeight) > 0.01 {
|
||||
t.Fatalf("expected compounded weight %.6f to stay close to %.6f", compounded, perMinuteWeight)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildFixtureStateLinksAllNodesToHostAgents(t *testing.T) {
|
||||
cfg := DefaultConfig
|
||||
cfg.NodeCount = 7
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -59,10 +60,10 @@ func defaultPlatformFixtures() PlatformFixtures {
|
||||
}
|
||||
|
||||
func rebasePlatformFixtures(fixtures PlatformFixtures, now time.Time) PlatformFixtures {
|
||||
target := now.UTC().Truncate(time.Minute)
|
||||
target := now.UTC()
|
||||
return PlatformFixtures{
|
||||
TrueNAS: rebaseTrueNASPlatformFixture(fixtures.TrueNAS, target),
|
||||
VMware: rebaseVMwarePlatformFixture(fixtures.VMware, target),
|
||||
TrueNAS: rebaseTrueNASPlatformFixture(refreshTrueNASPlatformFixture(fixtures.TrueNAS, target), target),
|
||||
VMware: rebaseVMwarePlatformFixture(refreshVMwarePlatformFixture(fixtures.VMware, target), target),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -258,6 +259,244 @@ func rebaseVMwarePlatformFixture(snapshot vmware.InventorySnapshot, target time.
|
||||
return out
|
||||
}
|
||||
|
||||
func refreshTrueNASPlatformFixture(snapshot truenas.FixtureSnapshot, at time.Time) truenas.FixtureSnapshot {
|
||||
out := cloneTrueNASFixtureSnapshot(snapshot)
|
||||
hostname := strings.TrimSpace(out.System.Hostname)
|
||||
if hostname != "" {
|
||||
out.System.CPUPercent = SampleMetric("agent", hostname, "cpu", at)
|
||||
out.System.DiskReadRate = SampleMetric("agent", hostname, "diskread", at)
|
||||
out.System.DiskWriteRate = SampleMetric("agent", hostname, "diskwrite", at)
|
||||
out.System.NetInRate = SampleMetric("agent", hostname, "netin", at)
|
||||
out.System.NetOutRate = SampleMetric("agent", hostname, "netout", at)
|
||||
if out.System.MemoryTotalBytes > 0 {
|
||||
memoryPercent := SampleMetric("agent", hostname, "memory", at)
|
||||
usedBytes := bytesFromPercent(out.System.MemoryTotalBytes, memoryPercent)
|
||||
out.System.MemoryAvailableBytes = out.System.MemoryTotalBytes - usedBytes
|
||||
if out.System.MemoryAvailableBytes < 0 {
|
||||
out.System.MemoryAvailableBytes = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for i := range out.Pools {
|
||||
usage := SampleMetric("storage", "pool:"+strings.TrimSpace(out.Pools[i].Name), "usage", at)
|
||||
applyTrueNASCapacityUsage(&out.Pools[i].UsedBytes, &out.Pools[i].FreeBytes, out.Pools[i].TotalBytes, usage)
|
||||
}
|
||||
|
||||
for i := range out.Datasets {
|
||||
totalBytes := out.Datasets[i].UsedBytes + out.Datasets[i].AvailBytes
|
||||
usage := SampleMetric("storage", "dataset:"+strings.TrimSpace(out.Datasets[i].Name), "usage", at)
|
||||
usedBytes := bytesFromPercent(totalBytes, usage)
|
||||
out.Datasets[i].UsedBytes = usedBytes
|
||||
out.Datasets[i].AvailBytes = totalBytes - usedBytes
|
||||
if out.Datasets[i].AvailBytes < 0 {
|
||||
out.Datasets[i].AvailBytes = 0
|
||||
}
|
||||
}
|
||||
|
||||
for i := range out.Disks {
|
||||
resourceID := trueNASDiskMetricID(out.Disks[i])
|
||||
if resourceID == "" {
|
||||
continue
|
||||
}
|
||||
out.Disks[i].Temperature = int(math.Round(SampleMetric("disk", resourceID, "smart_temp", at)))
|
||||
refreshTrueNASDiskAggregate(&out.Disks[i].TemperatureAggregate, out.Disks[i].Temperature)
|
||||
}
|
||||
|
||||
for i := range out.Apps {
|
||||
stats := out.Apps[i].Stats
|
||||
if stats == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
appID := strings.TrimSpace(out.Apps[i].ID)
|
||||
if appID == "" {
|
||||
appID = strings.TrimSpace(out.Apps[i].Name)
|
||||
}
|
||||
if appID == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
stats.CollectedAt = at
|
||||
stopped := strings.EqualFold(strings.TrimSpace(out.Apps[i].State), "stopped")
|
||||
if stopped {
|
||||
stats.CPUPercent = 0
|
||||
stats.NetInRate = 0
|
||||
stats.NetOutRate = 0
|
||||
stats.DiskReadRate = 0
|
||||
stats.DiskWriteRate = 0
|
||||
stats.BlockReadBytes = 0
|
||||
stats.BlockWriteBytes = 0
|
||||
if out.System.MemoryTotalBytes > 0 {
|
||||
idlePercent := clampFloat(SampleMetric("dockerContainer", appID, "memory", at)*0.12, 0.5, 8)
|
||||
stats.MemoryBytes = bytesFromPercent(out.System.MemoryTotalBytes, idlePercent)
|
||||
}
|
||||
} else {
|
||||
stats.CPUPercent = SampleMetric("dockerContainer", appID, "cpu", at)
|
||||
stats.NetInRate = SampleMetric("dockerContainer", appID, "netin", at)
|
||||
stats.NetOutRate = SampleMetric("dockerContainer", appID, "netout", at)
|
||||
stats.DiskReadRate = SampleMetric("dockerContainer", appID, "diskread", at)
|
||||
stats.DiskWriteRate = SampleMetric("dockerContainer", appID, "diskwrite", at)
|
||||
stats.BlockReadBytes = int64(math.Round(stats.DiskReadRate * math.Max(1, float64(stats.IntervalSeconds))))
|
||||
stats.BlockWriteBytes = int64(math.Round(stats.DiskWriteRate * math.Max(1, float64(stats.IntervalSeconds))))
|
||||
if out.System.MemoryTotalBytes > 0 {
|
||||
memoryPercent := SampleMetric("dockerContainer", appID, "memory", at)
|
||||
stats.MemoryBytes = bytesFromPercent(out.System.MemoryTotalBytes, memoryPercent)
|
||||
}
|
||||
}
|
||||
refreshTrueNASAppInterfaces(stats)
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
func refreshVMwarePlatformFixture(snapshot vmware.InventorySnapshot, at time.Time) vmware.InventorySnapshot {
|
||||
out := cloneVMwareInventorySnapshot(snapshot)
|
||||
for i := range out.Hosts {
|
||||
refreshVMwareInventoryMetrics(out.Hosts[i].Metrics, "agent", vmware.SourceID(out.ConnectionID, "host", out.Hosts[i].Host), at)
|
||||
}
|
||||
for i := range out.VMs {
|
||||
resourceID := vmware.SourceID(out.ConnectionID, "vm", out.VMs[i].VM)
|
||||
refreshVMwareInventoryMetrics(out.VMs[i].Metrics, "vm", resourceID, at)
|
||||
if strings.EqualFold(strings.TrimSpace(out.VMs[i].PowerState), "powered_off") && out.VMs[i].Metrics != nil {
|
||||
*out.VMs[i].Metrics.CPUPercent = 0
|
||||
*out.VMs[i].Metrics.NetInBytesPerSecond = 0
|
||||
*out.VMs[i].Metrics.NetOutBytesPerSecond = 0
|
||||
*out.VMs[i].Metrics.DiskReadBytesPerSecond = 0
|
||||
*out.VMs[i].Metrics.DiskWriteBytesPerSecond = 0
|
||||
}
|
||||
}
|
||||
for i := range out.Datastores {
|
||||
total := out.Datastores[i].Capacity
|
||||
if total <= 0 {
|
||||
continue
|
||||
}
|
||||
usage := SampleMetric("storage", vmware.SourceID(out.ConnectionID, "datastore", out.Datastores[i].Datastore), "usage", at)
|
||||
used := bytesFromPercent(total, usage)
|
||||
out.Datastores[i].FreeSpace = total - used
|
||||
if out.Datastores[i].FreeSpace < 0 {
|
||||
out.Datastores[i].FreeSpace = 0
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func applyTrueNASCapacityUsage(usedBytes *int64, freeBytes *int64, totalBytes int64, usagePercent float64) {
|
||||
if usedBytes == nil || freeBytes == nil || totalBytes <= 0 {
|
||||
return
|
||||
}
|
||||
used := bytesFromPercent(totalBytes, usagePercent)
|
||||
*usedBytes = used
|
||||
*freeBytes = totalBytes - used
|
||||
if *freeBytes < 0 {
|
||||
*freeBytes = 0
|
||||
}
|
||||
}
|
||||
|
||||
func bytesFromPercent(totalBytes int64, usagePercent float64) int64 {
|
||||
if totalBytes <= 0 {
|
||||
return 0
|
||||
}
|
||||
usage := clampFloat(usagePercent, 0, 100)
|
||||
used := int64(math.Round((float64(totalBytes) * usage) / 100.0))
|
||||
if used < 0 {
|
||||
return 0
|
||||
}
|
||||
if used > totalBytes {
|
||||
return totalBytes
|
||||
}
|
||||
return used
|
||||
}
|
||||
|
||||
func refreshTrueNASDiskAggregate(aggregate *truenas.DiskTemperatureAggregate, current int) {
|
||||
if aggregate == nil {
|
||||
return
|
||||
}
|
||||
currentFloat := float64(current)
|
||||
if aggregate.WindowDays <= 0 {
|
||||
aggregate.WindowDays = 7
|
||||
}
|
||||
if aggregate.MinCelsius <= 0 || aggregate.MinCelsius > currentFloat {
|
||||
aggregate.MinCelsius = math.Max(25, currentFloat-4)
|
||||
}
|
||||
if aggregate.MaxCelsius <= 0 || aggregate.MaxCelsius < currentFloat {
|
||||
aggregate.MaxCelsius = math.Min(95, currentFloat+5)
|
||||
}
|
||||
aggregate.AvgCelsius = clampFloat(currentFloat-0.8, aggregate.MinCelsius, aggregate.MaxCelsius)
|
||||
}
|
||||
|
||||
func trueNASDiskMetricID(disk truenas.Disk) string {
|
||||
resourceID := strings.TrimSpace(disk.Serial)
|
||||
if resourceID == "" {
|
||||
resourceID = strings.TrimSpace(disk.ID)
|
||||
}
|
||||
if resourceID == "" {
|
||||
resourceID = strings.TrimSpace(disk.Name)
|
||||
}
|
||||
return resourceID
|
||||
}
|
||||
|
||||
func refreshTrueNASAppInterfaces(stats *truenas.AppStats) {
|
||||
if stats == nil || len(stats.Interfaces) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
var totalRx float64
|
||||
var totalTx float64
|
||||
for _, iface := range stats.Interfaces {
|
||||
totalRx += iface.RxBytesPS
|
||||
totalTx += iface.TxBytesPS
|
||||
}
|
||||
|
||||
for i := range stats.Interfaces {
|
||||
rxShare := 1 / float64(len(stats.Interfaces))
|
||||
txShare := 1 / float64(len(stats.Interfaces))
|
||||
if totalRx > 0 {
|
||||
rxShare = stats.Interfaces[i].RxBytesPS / totalRx
|
||||
}
|
||||
if totalTx > 0 {
|
||||
txShare = stats.Interfaces[i].TxBytesPS / totalTx
|
||||
}
|
||||
stats.Interfaces[i].RxBytesPS = stats.NetInRate * rxShare
|
||||
stats.Interfaces[i].TxBytesPS = stats.NetOutRate * txShare
|
||||
}
|
||||
}
|
||||
|
||||
func refreshVMwareInventoryMetrics(metrics *vmware.InventoryMetrics, resourceClass, resourceID string, at time.Time) {
|
||||
if metrics == nil || strings.TrimSpace(resourceID) == "" {
|
||||
return
|
||||
}
|
||||
|
||||
memoryTotal := int64(0)
|
||||
if metrics.MemoryTotalBytes != nil {
|
||||
memoryTotal = *metrics.MemoryTotalBytes
|
||||
}
|
||||
|
||||
*ensureFloat64Ptr(&metrics.CPUPercent) = SampleMetric(resourceClass, resourceID, "cpu", at)
|
||||
*ensureFloat64Ptr(&metrics.MemoryPercent) = SampleMetric(resourceClass, resourceID, "memory", at)
|
||||
*ensureFloat64Ptr(&metrics.NetInBytesPerSecond) = SampleMetric(resourceClass, resourceID, "netin", at)
|
||||
*ensureFloat64Ptr(&metrics.NetOutBytesPerSecond) = SampleMetric(resourceClass, resourceID, "netout", at)
|
||||
*ensureFloat64Ptr(&metrics.DiskReadBytesPerSecond) = SampleMetric(resourceClass, resourceID, "diskread", at)
|
||||
*ensureFloat64Ptr(&metrics.DiskWriteBytesPerSecond) = SampleMetric(resourceClass, resourceID, "diskwrite", at)
|
||||
if memoryTotal > 0 {
|
||||
*ensureInt64Ptr(&metrics.MemoryUsedBytes) = bytesFromPercent(memoryTotal, *metrics.MemoryPercent)
|
||||
}
|
||||
}
|
||||
|
||||
func ensureFloat64Ptr(target **float64) *float64 {
|
||||
if *target == nil {
|
||||
*target = new(float64)
|
||||
}
|
||||
return *target
|
||||
}
|
||||
|
||||
func ensureInt64Ptr(target **int64) *int64 {
|
||||
if *target == nil {
|
||||
*target = new(int64)
|
||||
}
|
||||
return *target
|
||||
}
|
||||
|
||||
func rebaseVMwareAlarms(out []vmware.InventoryAlarm, in []vmware.InventoryAlarm, shift time.Duration, target time.Time) {
|
||||
for i := range out {
|
||||
out[i].TriggeredAt = shiftTime(in[i].TriggeredAt, shift, target)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package mock
|
||||
|
||||
import (
|
||||
"math"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -103,3 +104,44 @@ func TestFixtureGraphUpdateMetricsKeepsPlatformFixtureFreshnessCurrent(t *testin
|
||||
t.Fatalf("expected VMware event timestamp to remain fresh near %s, got %s", later, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildFixtureGraphRefreshesPlatformFixtureMetricsFromCanonicalModel(t *testing.T) {
|
||||
now := time.Date(2026, time.March, 31, 17, 30, 45, 0, time.UTC)
|
||||
|
||||
graph := buildFixtureGraph(DefaultConfig, now)
|
||||
|
||||
system := graph.PlatformFixtures.TrueNAS.System
|
||||
if got, want := system.CPUPercent, SampleMetric("agent", system.Hostname, "cpu", now); math.Abs(got-want) > 1e-9 {
|
||||
t.Fatalf("expected refreshed TrueNAS system cpu %.6f, got %.6f", want, got)
|
||||
}
|
||||
|
||||
app := graph.PlatformFixtures.TrueNAS.Apps[1]
|
||||
if app.Stats == nil {
|
||||
t.Fatal("expected refreshed TrueNAS app stats")
|
||||
}
|
||||
if got, want := app.Stats.CPUPercent, SampleMetric("dockerContainer", app.ID, "cpu", now); math.Abs(got-want) > 1e-9 {
|
||||
t.Fatalf("expected refreshed TrueNAS app cpu %.6f, got %.6f", want, got)
|
||||
}
|
||||
if got, want := app.Stats.MemoryBytes, bytesFromPercent(system.MemoryTotalBytes, SampleMetric("dockerContainer", app.ID, "memory", now)); got != want {
|
||||
t.Fatalf("expected refreshed TrueNAS app memory bytes %d, got %d", want, got)
|
||||
}
|
||||
|
||||
disk := graph.PlatformFixtures.TrueNAS.Disks[0]
|
||||
if got, want := disk.Temperature, int(math.Round(SampleMetric("disk", disk.Serial, "smart_temp", now))); got != want {
|
||||
t.Fatalf("expected refreshed TrueNAS disk temperature %d, got %d", want, got)
|
||||
}
|
||||
|
||||
host := graph.PlatformFixtures.VMware.Hosts[0]
|
||||
if host.Metrics == nil || host.Metrics.CPUPercent == nil {
|
||||
t.Fatal("expected refreshed VMware host metrics")
|
||||
}
|
||||
if got, want := *host.Metrics.CPUPercent, SampleMetric("agent", "vc-mock-1:host:host-101", "cpu", now); math.Abs(got-want) > 1e-9 {
|
||||
t.Fatalf("expected refreshed VMware host cpu %.6f, got %.6f", want, got)
|
||||
}
|
||||
|
||||
datastore := graph.PlatformFixtures.VMware.Datastores[0]
|
||||
wantFree := datastore.Capacity - bytesFromPercent(datastore.Capacity, SampleMetric("storage", "vc-mock-1:datastore:"+datastore.Datastore, "usage", now))
|
||||
if datastore.FreeSpace != wantFree {
|
||||
t.Fatalf("expected refreshed VMware datastore free space %d, got %d", wantFree, datastore.FreeSpace)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
@@ -88,6 +89,28 @@ func TestMonitoringRuntimeAvoidsLegacyMockPartialHelpers(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMockOwnedUnifiedMetricSyncSkipsProviderWriters(t *testing.T) {
|
||||
previous := mock.IsMockEnabled()
|
||||
mock.SetEnabled(true)
|
||||
t.Cleanup(func() { mock.SetEnabled(previous) })
|
||||
|
||||
if !shouldSkipMockOwnedUnifiedMetricSync(unifiedresources.Resource{
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceTrueNAS},
|
||||
}) {
|
||||
t.Fatal("expected TrueNAS mock-owned resources to skip generic unified metric sync")
|
||||
}
|
||||
if !shouldSkipMockOwnedUnifiedMetricSync(unifiedresources.Resource{
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceVMware},
|
||||
}) {
|
||||
t.Fatal("expected VMware mock-owned resources to skip generic unified metric sync")
|
||||
}
|
||||
if shouldSkipMockOwnedUnifiedMetricSync(unifiedresources.Resource{
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker},
|
||||
}) {
|
||||
t.Fatal("expected non-provider mock resources to keep generic unified metric sync")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectedInfrastructureUsesSharedTopLevelSystemResolver(t *testing.T) {
|
||||
data, err := os.ReadFile("connected_infrastructure.go")
|
||||
if err != nil {
|
||||
|
||||
@@ -30,12 +30,18 @@ type StorageMetrics struct {
|
||||
Avail []MetricPoint `json:"avail"`
|
||||
}
|
||||
|
||||
// MetricsHistory maintains historical metrics for all guests and nodes
|
||||
// DiskMetrics holds historical metrics for a single physical disk.
|
||||
type DiskMetrics struct {
|
||||
Temperature []MetricPoint `json:"smart_temp"`
|
||||
}
|
||||
|
||||
// MetricsHistory maintains historical metrics for guests, nodes, storage, and disks.
|
||||
type MetricsHistory struct {
|
||||
mu sync.RWMutex
|
||||
guestMetrics map[string]*GuestMetrics // key: guestID
|
||||
nodeMetrics map[string]*GuestMetrics // key: nodeID
|
||||
storageMetrics map[string]*StorageMetrics // key: storageID
|
||||
diskMetrics map[string]*DiskMetrics // key: disk metrics resource ID
|
||||
maxDataPoints int
|
||||
retentionTime time.Duration
|
||||
}
|
||||
@@ -46,6 +52,7 @@ func NewMetricsHistory(maxDataPoints int, retentionTime time.Duration) *MetricsH
|
||||
guestMetrics: make(map[string]*GuestMetrics),
|
||||
nodeMetrics: make(map[string]*GuestMetrics),
|
||||
storageMetrics: make(map[string]*StorageMetrics),
|
||||
diskMetrics: make(map[string]*DiskMetrics),
|
||||
maxDataPoints: maxDataPoints,
|
||||
retentionTime: retentionTime,
|
||||
}
|
||||
@@ -59,6 +66,7 @@ func (mh *MetricsHistory) Reset() {
|
||||
mh.guestMetrics = make(map[string]*GuestMetrics)
|
||||
mh.nodeMetrics = make(map[string]*GuestMetrics)
|
||||
mh.storageMetrics = make(map[string]*StorageMetrics)
|
||||
mh.diskMetrics = make(map[string]*DiskMetrics)
|
||||
}
|
||||
|
||||
// AddGuestMetric adds a metric value for a guest
|
||||
@@ -123,8 +131,13 @@ func (mh *MetricsHistory) AddNodeMetric(nodeID string, metricType string, value
|
||||
|
||||
// appendMetric appends a metric point and maintains max data points and retention
|
||||
func (mh *MetricsHistory) appendMetric(metrics []MetricPoint, point MetricPoint) []MetricPoint {
|
||||
// Append new point
|
||||
metrics = append(metrics, point)
|
||||
// Keep a single canonical value per timestamp so chart consumers never
|
||||
// have to guess which duplicate tail point to render.
|
||||
if len(metrics) > 0 && metrics[len(metrics)-1].Timestamp.Equal(point.Timestamp) {
|
||||
metrics[len(metrics)-1] = point
|
||||
} else {
|
||||
metrics = append(metrics, point)
|
||||
}
|
||||
|
||||
// Remove old points beyond retention time
|
||||
cutoffTime := time.Now().Add(-mh.retentionTime)
|
||||
@@ -313,6 +326,44 @@ func (mh *MetricsHistory) GetAllStorageMetrics(storageID string, duration time.D
|
||||
return result
|
||||
}
|
||||
|
||||
// AddDiskMetric adds a metric value for a physical disk.
|
||||
func (mh *MetricsHistory) AddDiskMetric(resourceID string, metricType string, value float64, timestamp time.Time) {
|
||||
mh.mu.Lock()
|
||||
defer mh.mu.Unlock()
|
||||
|
||||
if _, exists := mh.diskMetrics[resourceID]; !exists {
|
||||
mh.diskMetrics[resourceID] = &DiskMetrics{}
|
||||
}
|
||||
|
||||
metrics := mh.diskMetrics[resourceID]
|
||||
point := MetricPoint{Value: value, Timestamp: timestamp}
|
||||
|
||||
switch metricType {
|
||||
case "smart_temp":
|
||||
metrics.Temperature = mh.appendMetric(metrics.Temperature, point)
|
||||
}
|
||||
}
|
||||
|
||||
// GetDiskMetrics returns historical metrics for a physical disk.
|
||||
func (mh *MetricsHistory) GetDiskMetrics(resourceID string, metricType string, duration time.Duration) []MetricPoint {
|
||||
mh.mu.RLock()
|
||||
defer mh.mu.RUnlock()
|
||||
|
||||
metrics, exists := mh.diskMetrics[resourceID]
|
||||
if !exists {
|
||||
return []MetricPoint{}
|
||||
}
|
||||
|
||||
cutoffTime := time.Now().Add(-duration)
|
||||
|
||||
switch metricType {
|
||||
case "smart_temp":
|
||||
return filterMetricsByTime(metrics.Temperature, cutoffTime)
|
||||
default:
|
||||
return []MetricPoint{}
|
||||
}
|
||||
}
|
||||
|
||||
// Cleanup removes old data points beyond retention time and deletes
|
||||
// map entries for resources that have no remaining data points.
|
||||
// This prevents unbounded memory growth when containers/VMs are deleted.
|
||||
@@ -321,7 +372,7 @@ func (mh *MetricsHistory) Cleanup() {
|
||||
defer mh.mu.Unlock()
|
||||
|
||||
cutoffTime := time.Now().Add(-mh.retentionTime)
|
||||
var guestsRemoved, nodesRemoved, storageRemoved int
|
||||
var guestsRemoved, nodesRemoved, storageRemoved, disksRemoved int
|
||||
|
||||
// Cleanup guest metrics and remove empty entries
|
||||
for key, metrics := range mh.guestMetrics {
|
||||
@@ -368,15 +419,25 @@ func (mh *MetricsHistory) Cleanup() {
|
||||
}
|
||||
}
|
||||
|
||||
for key, metrics := range mh.diskMetrics {
|
||||
metrics.Temperature = mh.cleanupMetrics(metrics.Temperature, cutoffTime)
|
||||
if len(metrics.Temperature) == 0 {
|
||||
delete(mh.diskMetrics, key)
|
||||
disksRemoved++
|
||||
}
|
||||
}
|
||||
|
||||
// Log cleanup activity at debug level
|
||||
if guestsRemoved > 0 || nodesRemoved > 0 || storageRemoved > 0 {
|
||||
if guestsRemoved > 0 || nodesRemoved > 0 || storageRemoved > 0 || disksRemoved > 0 {
|
||||
log.Debug().
|
||||
Int("guestsRemoved", guestsRemoved).
|
||||
Int("nodesRemoved", nodesRemoved).
|
||||
Int("storageRemoved", storageRemoved).
|
||||
Int("disksRemoved", disksRemoved).
|
||||
Int("guestsRemaining", len(mh.guestMetrics)).
|
||||
Int("nodesRemaining", len(mh.nodeMetrics)).
|
||||
Int("storageRemaining", len(mh.storageMetrics)).
|
||||
Int("disksRemaining", len(mh.diskMetrics)).
|
||||
Msg("Cleaned up stale metrics history entries")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -135,6 +135,19 @@ func TestAppendMetric(t *testing.T) {
|
||||
wantFirst: 30.0,
|
||||
wantLast: 30.0,
|
||||
},
|
||||
{
|
||||
name: "same timestamp replaces latest point",
|
||||
maxDataPoints: 10,
|
||||
retentionTime: time.Hour,
|
||||
existing: []MetricPoint{
|
||||
{Value: 10.0, Timestamp: now.Add(-5 * time.Minute)},
|
||||
{Value: 20.0, Timestamp: now},
|
||||
},
|
||||
newPoint: MetricPoint{Value: 30.0, Timestamp: now},
|
||||
wantLen: 2,
|
||||
wantFirst: 10.0,
|
||||
wantLast: 30.0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
|
||||
@@ -10,7 +10,9 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mockmodel"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/vmware"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
|
||||
"github.com/rs/zerolog/log"
|
||||
@@ -283,6 +285,47 @@ func GenerateSeededSeries(current float64, points int, seed uint64, min, max flo
|
||||
return raw
|
||||
}
|
||||
|
||||
func GenerateSeededSeriesForTimestamps(
|
||||
current float64,
|
||||
timestamps []time.Time,
|
||||
seed uint64,
|
||||
min float64,
|
||||
max float64,
|
||||
style SeriesStyle,
|
||||
) []float64 {
|
||||
var mappedStyle mockmodel.SeriesStyle
|
||||
switch style {
|
||||
case stylePlateau:
|
||||
mappedStyle = mockmodel.StylePlateau
|
||||
case styleFlat:
|
||||
mappedStyle = mockmodel.StyleFlat
|
||||
default:
|
||||
mappedStyle = mockmodel.StyleSpiky
|
||||
}
|
||||
return mockmodel.SeriesForTimestamps(current, timestamps, seed, min, max, mappedStyle)
|
||||
}
|
||||
|
||||
func GenerateSeededMetricSeriesForTimestamps(
|
||||
current float64,
|
||||
timestamps []time.Time,
|
||||
seed uint64,
|
||||
min float64,
|
||||
max float64,
|
||||
metricType string,
|
||||
style SeriesStyle,
|
||||
) []float64 {
|
||||
var mappedStyle mockmodel.SeriesStyle
|
||||
switch style {
|
||||
case stylePlateau:
|
||||
mappedStyle = mockmodel.StylePlateau
|
||||
case styleFlat:
|
||||
mappedStyle = mockmodel.StyleFlat
|
||||
default:
|
||||
mappedStyle = mockmodel.StyleSpiky
|
||||
}
|
||||
return mockmodel.SeriesForMetricTimestamps(current, timestamps, seed, min, max, metricType, mappedStyle)
|
||||
}
|
||||
|
||||
// generateSpikySeries produces a low baseline with occasional sharp spikes —
|
||||
// matching how real CPU and I/O metrics behave (mostly idle, with bursts).
|
||||
func generateSpikySeries(current float64, points int, seed uint64, min, max, span float64, rng *rand.Rand) []float64 {
|
||||
@@ -468,14 +511,6 @@ func buildTieredTimestamps(now time.Time, totalDuration time.Duration) []time.Ti
|
||||
}
|
||||
}
|
||||
|
||||
// Add "now" as the final point
|
||||
if len(timestamps) > 0 {
|
||||
last := timestamps[len(timestamps)-1]
|
||||
if !last.Equal(now) {
|
||||
timestamps = append(timestamps, now)
|
||||
}
|
||||
}
|
||||
|
||||
return timestamps
|
||||
}
|
||||
|
||||
@@ -490,13 +525,17 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
|
||||
// Build a tiered timestamp list so short time ranges (1h, 4h) have dense
|
||||
// data without needing an API-level fallback layer.
|
||||
// Last 2h: 30s intervals (~240 points)
|
||||
// Last 2h: 1min intervals (~120 points)
|
||||
// 2h–24h: 2min intervals (~660 points)
|
||||
// 24h–90d: ~65min intervals (~1920 points)
|
||||
// Total: ~2820 points per metric per resource.
|
||||
// The current "now" point is appended explicitly by each recorder so the
|
||||
// seed and live sampler share one canonical terminal timestamp.
|
||||
seedTimestamps := buildTieredTimestamps(now, seedDuration)
|
||||
const seedBatchSize = 5000
|
||||
|
||||
numPoints := len(seedTimestamps)
|
||||
generateSeries := func(metricType string, current float64, seed uint64, min, max float64, style SeriesStyle) []float64 {
|
||||
return GenerateSeededMetricSeriesForTimestamps(current, seedTimestamps, seed, min, max, metricType, style)
|
||||
}
|
||||
var seedBatch []metrics.WriteMetric
|
||||
queueMetric := func(resourceType, resourceID, metricType string, value float64, ts time.Time) {
|
||||
if ms == nil {
|
||||
@@ -526,16 +565,51 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
seedBatch = seedBatch[:0]
|
||||
}
|
||||
}
|
||||
recordStorageTimeline := func(storageID string, currentUsed, currentTotal float64) {
|
||||
if strings.TrimSpace(storageID) == "" || currentTotal <= 0 || numPoints == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
series := mockmodel.StorageCapacitySeriesForTimestamps(
|
||||
currentUsed,
|
||||
currentTotal,
|
||||
seedTimestamps,
|
||||
mock.MetricSeed("storage", storageID, "usage"),
|
||||
)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
mh.AddStorageMetric(storageID, "usage", series.Usage[i], ts)
|
||||
mh.AddStorageMetric(storageID, "used", series.Used[i], ts)
|
||||
mh.AddStorageMetric(storageID, "avail", series.Avail[i], ts)
|
||||
mh.AddStorageMetric(storageID, "total", series.Total[i], ts)
|
||||
queueMetric("storage", storageID, "usage", series.Usage[i], ts)
|
||||
queueMetric("storage", storageID, "used", series.Used[i], ts)
|
||||
queueMetric("storage", storageID, "avail", series.Avail[i], ts)
|
||||
queueMetric("storage", storageID, "total", series.Total[i], ts)
|
||||
}
|
||||
|
||||
last := numPoints - 1
|
||||
mh.AddStorageMetric(storageID, "usage", series.Usage[last], now)
|
||||
mh.AddStorageMetric(storageID, "used", series.Used[last], now)
|
||||
mh.AddStorageMetric(storageID, "avail", series.Avail[last], now)
|
||||
mh.AddStorageMetric(storageID, "total", series.Total[last], now)
|
||||
queueMetric("storage", storageID, "usage", series.Usage[last], now)
|
||||
queueMetric("storage", storageID, "used", series.Used[last], now)
|
||||
queueMetric("storage", storageID, "avail", series.Avail[last], now)
|
||||
queueMetric("storage", storageID, "total", series.Total[last], now)
|
||||
}
|
||||
|
||||
recordNode := func(node models.Node) {
|
||||
if node.ID == "" {
|
||||
return
|
||||
}
|
||||
|
||||
numPoints := len(seedTimestamps)
|
||||
cpuSeries := GenerateSeededSeries(node.CPU*100, numPoints, HashSeed("node", node.ID, "cpu"), 5, 85, styleSpiky)
|
||||
memSeries := GenerateSeededSeries(node.Memory.Usage, numPoints, HashSeed("node", node.ID, "memory"), 10, 85, stylePlateau)
|
||||
diskSeries := GenerateSeededSeries(node.Disk.Usage, numPoints, HashSeed("node", node.ID, "disk"), 5, 95, styleFlat)
|
||||
cpuMin, cpuMax := mock.MetricBounds("node", "cpu")
|
||||
memMin, memMax := mock.MetricBounds("node", "memory")
|
||||
diskMin, diskMax := mock.MetricBounds("node", "disk")
|
||||
cpuSeries := generateSeries("cpu", node.CPU*100, mock.MetricSeed("node", node.ID, "cpu"), cpuMin, cpuMax, styleSpiky)
|
||||
memSeries := generateSeries("memory", node.Memory.Usage, mock.MetricSeed("node", node.ID, "memory"), memMin, memMax, stylePlateau)
|
||||
diskSeries := generateSeries("disk", node.Disk.Usage, mock.MetricSeed("node", node.ID, "disk"), diskMin, diskMax, styleFlat)
|
||||
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
@@ -583,24 +657,27 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
return
|
||||
}
|
||||
|
||||
numPoints := len(seedTimestamps)
|
||||
cpuSeries := GenerateSeededSeries(cpuPercent, numPoints, HashSeed(storeType, storeID, "cpu"), 0, 100, styleSpiky)
|
||||
memSeries := GenerateSeededSeries(memPercent, numPoints, HashSeed(storeType, storeID, "memory"), 0, 100, stylePlateau)
|
||||
cpuMin, cpuMax := mock.MetricBounds(storeType, "cpu")
|
||||
memMin, memMax := mock.MetricBounds(storeType, "memory")
|
||||
diskMin, diskMax := mock.MetricBounds(storeType, "disk")
|
||||
readMin, readMax := mock.MetricBounds(storeType, "diskread")
|
||||
writeMin, writeMax := mock.MetricBounds(storeType, "diskwrite")
|
||||
netInMin, netInMax := mock.MetricBounds(storeType, "netin")
|
||||
netOutMin, netOutMax := mock.MetricBounds(storeType, "netout")
|
||||
cpuSeries := generateSeries("cpu", cpuPercent, mock.MetricSeed(storeType, storeID, "cpu"), cpuMin, cpuMax, styleSpiky)
|
||||
memSeries := generateSeries("memory", memPercent, mock.MetricSeed(storeType, storeID, "memory"), memMin, memMax, stylePlateau)
|
||||
var diskSeries []float64
|
||||
if includeDisk {
|
||||
diskSeries = GenerateSeededSeries(diskPercent, numPoints, HashSeed(storeType, storeID, "disk"), 0, 100, styleFlat)
|
||||
}
|
||||
ioMax := func(value float64) float64 {
|
||||
return math.Max(value*1.8, 1)
|
||||
diskSeries = generateSeries("disk", diskPercent, mock.MetricSeed(storeType, storeID, "disk"), diskMin, diskMax, styleFlat)
|
||||
}
|
||||
var diskReadSeries, diskWriteSeries, netInSeries, netOutSeries []float64
|
||||
if includeDiskIO {
|
||||
diskReadSeries = GenerateSeededSeries(diskRead, numPoints, HashSeed(storeType, storeID, "diskread"), 0, ioMax(diskRead), styleSpiky)
|
||||
diskWriteSeries = GenerateSeededSeries(diskWrite, numPoints, HashSeed(storeType, storeID, "diskwrite"), 0, ioMax(diskWrite), styleSpiky)
|
||||
diskReadSeries = generateSeries("diskread", diskRead, mock.MetricSeed(storeType, storeID, "diskread"), readMin, readMax, styleSpiky)
|
||||
diskWriteSeries = generateSeries("diskwrite", diskWrite, mock.MetricSeed(storeType, storeID, "diskwrite"), writeMin, writeMax, styleSpiky)
|
||||
}
|
||||
if includeNetwork {
|
||||
netInSeries = GenerateSeededSeries(netIn, numPoints, HashSeed(storeType, storeID, "netin"), 0, ioMax(netIn), styleSpiky)
|
||||
netOutSeries = GenerateSeededSeries(netOut, numPoints, HashSeed(storeType, storeID, "netout"), 0, ioMax(netOut), styleSpiky)
|
||||
netInSeries = generateSeries("netin", netIn, mock.MetricSeed(storeType, storeID, "netin"), netInMin, netInMax, styleSpiky)
|
||||
netOutSeries = generateSeries("netout", netOut, mock.MetricSeed(storeType, storeID, "netout"), netOutMin, netOutMax, styleSpiky)
|
||||
}
|
||||
|
||||
for i := 0; i < numPoints; i++ {
|
||||
@@ -776,18 +853,8 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
if storage.ID == "" {
|
||||
continue
|
||||
}
|
||||
numPoints := len(seedTimestamps)
|
||||
usageSeries := GenerateSeededSeries(storage.Usage, numPoints, HashSeed("storage", storage.ID, "usage"), 0, 100, styleFlat)
|
||||
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
mh.AddStorageMetric(storage.ID, "usage", usageSeries[i], ts)
|
||||
queueMetric("storage", storage.ID, "usage", usageSeries[i], ts)
|
||||
}
|
||||
|
||||
// Ensure the latest point lands at "now" for full-range charts.
|
||||
mh.AddStorageMetric(storage.ID, "usage", storage.Usage, now)
|
||||
queueMetric("storage", storage.ID, "usage", storage.Usage, now)
|
||||
_, used, total, _ := normalizedStorageCapacityMetrics(storage.Total, storage.Used, storage.Free, storage.Usage)
|
||||
recordStorageTimeline(storage.ID, used, total)
|
||||
time.Sleep(50 * time.Millisecond) // Reduced from 200ms for faster startup
|
||||
}
|
||||
|
||||
@@ -801,21 +868,16 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
continue
|
||||
}
|
||||
|
||||
numPoints := len(seedTimestamps)
|
||||
tempSeries := GenerateSeededSeries(
|
||||
float64(disk.Temperature),
|
||||
numPoints,
|
||||
HashSeed("disk", resourceID, "smart_temp"),
|
||||
25,
|
||||
95,
|
||||
styleFlat,
|
||||
)
|
||||
tempMin, tempMax := mock.MetricBounds("disk", "smart_temp")
|
||||
tempSeries := generateSeries("smart_temp", float64(disk.Temperature), mock.MetricSeed("disk", resourceID, "smart_temp"), tempMin, tempMax, styleFlat)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
mh.AddDiskMetric(resourceID, "smart_temp", tempSeries[i], ts)
|
||||
queueMetric("disk", resourceID, "smart_temp", tempSeries[i], ts)
|
||||
}
|
||||
|
||||
// Ensure the latest point lands at "now" for full-range charts.
|
||||
mh.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
queueMetric("disk", resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
}
|
||||
|
||||
@@ -832,15 +894,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
continue
|
||||
}
|
||||
|
||||
numPoints := len(seedTimestamps)
|
||||
usageSeries := GenerateSeededSeries(
|
||||
cluster.UsagePercent,
|
||||
numPoints,
|
||||
HashSeed("ceph", cephID, "usage"),
|
||||
0,
|
||||
100,
|
||||
styleFlat,
|
||||
)
|
||||
usageSeries := generateSeries("usage", cluster.UsagePercent, mock.MetricSeed("ceph", cephID, "usage"), 0, 100, styleFlat)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
queueMetric("ceph", cephID, "usage", usageSeries[i], ts)
|
||||
@@ -956,34 +1010,15 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
if pool.TotalBytes > 0 {
|
||||
diskPercent = float64(pool.UsedBytes) / float64(pool.TotalBytes) * 100
|
||||
}
|
||||
numPoints := len(seedTimestamps)
|
||||
diskSeries := GenerateSeededSeries(diskPercent, numPoints, HashSeed("pool", pool.Name, "disk"), 0, 100, styleFlat)
|
||||
usedSeries := GenerateSeededSeries(float64(pool.UsedBytes), numPoints, HashSeed("pool", pool.Name, "used"), 0, float64(pool.TotalBytes), styleFlat)
|
||||
availSeries := GenerateSeededSeries(float64(pool.FreeBytes), numPoints, HashSeed("pool", pool.Name, "avail"), 0, float64(pool.TotalBytes), styleFlat)
|
||||
totalSeries := GenerateSeededSeries(float64(pool.TotalBytes), numPoints, HashSeed("pool", pool.Name, "total"), 0, float64(pool.TotalBytes), styleFlat)
|
||||
diskSeries := generateSeries("usage", diskPercent, mock.MetricSeed("pool", pool.Name, "disk"), 0, 100, styleFlat)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
mh.AddGuestMetric(poolKey, "disk", diskSeries[i], ts)
|
||||
queueMetric("pool", pool.Name, "disk", diskSeries[i], ts)
|
||||
mh.AddStorageMetric(poolKey, "usage", diskSeries[i], ts)
|
||||
mh.AddStorageMetric(poolKey, "used", usedSeries[i], ts)
|
||||
mh.AddStorageMetric(poolKey, "avail", availSeries[i], ts)
|
||||
mh.AddStorageMetric(poolKey, "total", totalSeries[i], ts)
|
||||
queueMetric("storage", poolKey, "usage", diskSeries[i], ts)
|
||||
queueMetric("storage", poolKey, "used", usedSeries[i], ts)
|
||||
queueMetric("storage", poolKey, "avail", availSeries[i], ts)
|
||||
queueMetric("storage", poolKey, "total", totalSeries[i], ts)
|
||||
}
|
||||
mh.AddGuestMetric(poolKey, "disk", diskPercent, now)
|
||||
queueMetric("pool", pool.Name, "disk", diskPercent, now)
|
||||
mh.AddStorageMetric(poolKey, "usage", diskPercent, now)
|
||||
mh.AddStorageMetric(poolKey, "used", float64(pool.UsedBytes), now)
|
||||
mh.AddStorageMetric(poolKey, "avail", float64(pool.FreeBytes), now)
|
||||
mh.AddStorageMetric(poolKey, "total", float64(pool.TotalBytes), now)
|
||||
queueMetric("storage", poolKey, "usage", diskPercent, now)
|
||||
queueMetric("storage", poolKey, "used", float64(pool.UsedBytes), now)
|
||||
queueMetric("storage", poolKey, "avail", float64(pool.FreeBytes), now)
|
||||
queueMetric("storage", poolKey, "total", float64(pool.TotalBytes), now)
|
||||
recordStorageTimeline(poolKey, float64(pool.UsedBytes), float64(pool.TotalBytes))
|
||||
}
|
||||
|
||||
for _, dataset := range trueNASFixtures.Datasets {
|
||||
@@ -993,34 +1028,34 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
if totalBytes > 0 {
|
||||
diskPercent = float64(dataset.UsedBytes) / float64(totalBytes) * 100
|
||||
}
|
||||
numPoints := len(seedTimestamps)
|
||||
diskSeries := GenerateSeededSeries(diskPercent, numPoints, HashSeed("dataset", dataset.Name, "disk"), 0, 100, styleFlat)
|
||||
usedSeries := GenerateSeededSeries(float64(dataset.UsedBytes), numPoints, HashSeed("dataset", dataset.Name, "used"), 0, float64(totalBytes), styleFlat)
|
||||
availSeries := GenerateSeededSeries(float64(dataset.AvailBytes), numPoints, HashSeed("dataset", dataset.Name, "avail"), 0, float64(totalBytes), styleFlat)
|
||||
totalSeries := GenerateSeededSeries(float64(totalBytes), numPoints, HashSeed("dataset", dataset.Name, "total"), 0, float64(totalBytes), styleFlat)
|
||||
diskSeries := generateSeries("usage", diskPercent, mock.MetricSeed("dataset", dataset.Name, "disk"), 0, 100, styleFlat)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
mh.AddGuestMetric(dsKey, "disk", diskSeries[i], ts)
|
||||
queueMetric("dataset", dataset.Name, "disk", diskSeries[i], ts)
|
||||
mh.AddStorageMetric(dsKey, "usage", diskSeries[i], ts)
|
||||
mh.AddStorageMetric(dsKey, "used", usedSeries[i], ts)
|
||||
mh.AddStorageMetric(dsKey, "avail", availSeries[i], ts)
|
||||
mh.AddStorageMetric(dsKey, "total", totalSeries[i], ts)
|
||||
queueMetric("storage", dsKey, "usage", diskSeries[i], ts)
|
||||
queueMetric("storage", dsKey, "used", usedSeries[i], ts)
|
||||
queueMetric("storage", dsKey, "avail", availSeries[i], ts)
|
||||
queueMetric("storage", dsKey, "total", totalSeries[i], ts)
|
||||
}
|
||||
mh.AddGuestMetric(dsKey, "disk", diskPercent, now)
|
||||
queueMetric("dataset", dataset.Name, "disk", diskPercent, now)
|
||||
mh.AddStorageMetric(dsKey, "usage", diskPercent, now)
|
||||
mh.AddStorageMetric(dsKey, "used", float64(dataset.UsedBytes), now)
|
||||
mh.AddStorageMetric(dsKey, "avail", float64(dataset.AvailBytes), now)
|
||||
mh.AddStorageMetric(dsKey, "total", float64(totalBytes), now)
|
||||
queueMetric("storage", dsKey, "usage", diskPercent, now)
|
||||
queueMetric("storage", dsKey, "used", float64(dataset.UsedBytes), now)
|
||||
queueMetric("storage", dsKey, "avail", float64(dataset.AvailBytes), now)
|
||||
queueMetric("storage", dsKey, "total", float64(totalBytes), now)
|
||||
recordStorageTimeline(dsKey, float64(dataset.UsedBytes), float64(totalBytes))
|
||||
}
|
||||
|
||||
for _, disk := range trueNASFixtures.Disks {
|
||||
if disk.Temperature <= 0 {
|
||||
continue
|
||||
}
|
||||
resourceID := trueNASDiskMetricsResourceID(disk)
|
||||
if resourceID == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
tempSeries := generateSeries("smart_temp", float64(disk.Temperature), mock.MetricSeed("disk", resourceID, "smart_temp"), 25, 95, styleFlat)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
mh.AddDiskMetric(resourceID, "smart_temp", tempSeries[i], ts)
|
||||
queueMetric("disk", resourceID, "smart_temp", tempSeries[i], ts)
|
||||
}
|
||||
mh.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
queueMetric("disk", resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
}
|
||||
|
||||
for _, app := range trueNASFixtures.Apps {
|
||||
@@ -1126,30 +1161,7 @@ func seedMockMetricsHistory(mh *MetricsHistory, ms *metrics.Store, graph mock.Fi
|
||||
if sourceID == "" {
|
||||
continue
|
||||
}
|
||||
numPoints := len(seedTimestamps)
|
||||
usageSeries := GenerateSeededSeries(usage, numPoints, HashSeed("vmware-datastore", sourceID, "usage"), 0, 100, styleFlat)
|
||||
usedSeries := GenerateSeededSeries(float64(used), numPoints, HashSeed("vmware-datastore", sourceID, "used"), 0, float64(total), styleFlat)
|
||||
availSeries := GenerateSeededSeries(float64(avail), numPoints, HashSeed("vmware-datastore", sourceID, "avail"), 0, float64(total), styleFlat)
|
||||
totalSeries := GenerateSeededSeries(float64(total), numPoints, HashSeed("vmware-datastore", sourceID, "total"), 0, float64(total), styleFlat)
|
||||
for i := 0; i < numPoints; i++ {
|
||||
ts := seedTimestamps[i]
|
||||
mh.AddStorageMetric(sourceID, "usage", usageSeries[i], ts)
|
||||
mh.AddStorageMetric(sourceID, "used", usedSeries[i], ts)
|
||||
mh.AddStorageMetric(sourceID, "avail", availSeries[i], ts)
|
||||
mh.AddStorageMetric(sourceID, "total", totalSeries[i], ts)
|
||||
queueMetric("storage", sourceID, "usage", usageSeries[i], ts)
|
||||
queueMetric("storage", sourceID, "used", usedSeries[i], ts)
|
||||
queueMetric("storage", sourceID, "avail", availSeries[i], ts)
|
||||
queueMetric("storage", sourceID, "total", totalSeries[i], ts)
|
||||
}
|
||||
mh.AddStorageMetric(sourceID, "usage", usage, now)
|
||||
mh.AddStorageMetric(sourceID, "used", float64(used), now)
|
||||
mh.AddStorageMetric(sourceID, "avail", float64(avail), now)
|
||||
mh.AddStorageMetric(sourceID, "total", float64(total), now)
|
||||
queueMetric("storage", sourceID, "usage", usage, now)
|
||||
queueMetric("storage", sourceID, "used", float64(used), now)
|
||||
queueMetric("storage", sourceID, "avail", float64(avail), now)
|
||||
queueMetric("storage", sourceID, "total", float64(total), now)
|
||||
recordStorageTimeline(sourceID, float64(used), float64(total))
|
||||
}
|
||||
|
||||
if ms != nil && len(seedBatch) > 0 {
|
||||
@@ -1247,6 +1259,21 @@ func recordTrueNASFixturesMetrics(mh *MetricsHistory, ms *metrics.Store, fixture
|
||||
}
|
||||
}
|
||||
|
||||
for _, disk := range snapshot.Disks {
|
||||
if disk.Temperature <= 0 {
|
||||
continue
|
||||
}
|
||||
resourceID := trueNASDiskMetricsResourceID(disk)
|
||||
if resourceID == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
mh.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), ts)
|
||||
if ms != nil {
|
||||
ms.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), ts)
|
||||
}
|
||||
}
|
||||
|
||||
for _, app := range snapshot.Apps {
|
||||
if app.Stats == nil {
|
||||
continue
|
||||
@@ -1581,13 +1608,17 @@ func recordMockStateToMetricsHistory(mh *MetricsHistory, ms *metrics.Store, grap
|
||||
if storage.ID == "" || storage.Status != "available" {
|
||||
continue
|
||||
}
|
||||
mh.AddStorageMetric(storage.ID, "usage", storage.Usage, ts)
|
||||
mh.AddStorageMetric(storage.ID, "used", float64(storage.Used), ts)
|
||||
mh.AddStorageMetric(storage.ID, "total", float64(storage.Total), ts)
|
||||
mh.AddStorageMetric(storage.ID, "avail", float64(storage.Free), ts)
|
||||
usage, used, total, avail := normalizedStorageCapacityMetrics(storage.Total, storage.Used, storage.Free, storage.Usage)
|
||||
mh.AddStorageMetric(storage.ID, "usage", usage, ts)
|
||||
mh.AddStorageMetric(storage.ID, "used", used, ts)
|
||||
mh.AddStorageMetric(storage.ID, "total", total, ts)
|
||||
mh.AddStorageMetric(storage.ID, "avail", avail, ts)
|
||||
|
||||
if ms != nil {
|
||||
ms.Write("storage", storage.ID, "usage", storage.Usage, ts)
|
||||
ms.Write("storage", storage.ID, "usage", usage, ts)
|
||||
ms.Write("storage", storage.ID, "used", used, ts)
|
||||
ms.Write("storage", storage.ID, "total", total, ts)
|
||||
ms.Write("storage", storage.ID, "avail", avail, ts)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1600,6 +1631,7 @@ func recordMockStateToMetricsHistory(mh *MetricsHistory, ms *metrics.Store, grap
|
||||
continue
|
||||
}
|
||||
|
||||
mh.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), ts)
|
||||
if ms != nil {
|
||||
ms.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), ts)
|
||||
}
|
||||
@@ -1723,6 +1755,44 @@ func diskMetricsResourceID(disk models.PhysicalDisk) string {
|
||||
return resourceID
|
||||
}
|
||||
|
||||
func normalizedStorageCapacityMetrics(total, used, free int64, usage float64) (float64, float64, float64, float64) {
|
||||
totalValue := float64(total)
|
||||
usedValue := float64(used)
|
||||
freeValue := float64(free)
|
||||
|
||||
if totalValue > 0 {
|
||||
if free >= 0 {
|
||||
derivedUsed := float64(total - free)
|
||||
if derivedUsed >= 0 && derivedUsed <= totalValue {
|
||||
usedValue = derivedUsed
|
||||
freeValue = totalValue - usedValue
|
||||
}
|
||||
} else {
|
||||
if usedValue < 0 {
|
||||
usedValue = 0
|
||||
}
|
||||
if usedValue > totalValue {
|
||||
usedValue = totalValue
|
||||
}
|
||||
freeValue = totalValue - usedValue
|
||||
}
|
||||
usage = (usedValue / totalValue) * 100
|
||||
}
|
||||
|
||||
return clampFloat(usage, 0, 100), usedValue, totalValue, math.Max(0, freeValue)
|
||||
}
|
||||
|
||||
func trueNASDiskMetricsResourceID(disk truenas.Disk) string {
|
||||
resourceID := strings.TrimSpace(disk.Serial)
|
||||
if resourceID == "" {
|
||||
resourceID = strings.TrimSpace(disk.ID)
|
||||
}
|
||||
if resourceID == "" {
|
||||
resourceID = strings.TrimSpace(disk.Name)
|
||||
}
|
||||
return resourceID
|
||||
}
|
||||
|
||||
func (m *Monitor) startMockMetricsSampler(ctx context.Context) {
|
||||
if ctx == nil || m == nil {
|
||||
log.Debug().Msg("mock metrics sampler: nil context or monitor")
|
||||
@@ -1768,7 +1838,6 @@ func (m *Monitor) startMockMetricsSampler(ctx context.Context) {
|
||||
// Keep mock trend generation in-memory only so production history in the
|
||||
// persistent metrics store remains untouched while mock mode is active.
|
||||
seedMockMetricsHistory(m.metricsHistory, nil, graph, time.Now(), seedDuration, cfg.SampleInterval)
|
||||
recordMockStateToMetricsHistory(m.metricsHistory, nil, graph, time.Now())
|
||||
|
||||
m.mockMetricsWg.Add(1)
|
||||
go func() {
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/vmware"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
|
||||
)
|
||||
@@ -18,6 +19,20 @@ func fixtureGraphWithState(state models.StateSnapshot) mock.FixtureGraph {
|
||||
return mock.FixtureGraph{State: state}
|
||||
}
|
||||
|
||||
func TestBuildTieredTimestamps_LeavesTerminalNowToRecorders(t *testing.T) {
|
||||
now := time.Date(2026, time.March, 31, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
timestamps := buildTieredTimestamps(now, time.Hour)
|
||||
if len(timestamps) == 0 {
|
||||
t.Fatal("expected tiered timestamps")
|
||||
}
|
||||
|
||||
last := timestamps[len(timestamps)-1]
|
||||
if !last.Before(now) {
|
||||
t.Fatalf("expected seed timestamps to stop before now, got %v with now=%v", last, now)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedMockMetricsHistory_PopulatesSeries(t *testing.T) {
|
||||
now := time.Now()
|
||||
|
||||
@@ -59,6 +74,14 @@ func TestSeedMockMetricsHistory_PopulatesSeries(t *testing.T) {
|
||||
Usage: 42,
|
||||
},
|
||||
},
|
||||
PhysicalDisks: []models.PhysicalDisk{
|
||||
{
|
||||
ID: "disk-1",
|
||||
Node: "node-1",
|
||||
Serial: "SERIAL-LOCAL-1",
|
||||
Temperature: 41,
|
||||
},
|
||||
},
|
||||
DockerHosts: []models.DockerHost{
|
||||
{
|
||||
ID: "host-1",
|
||||
@@ -108,6 +131,82 @@ func TestSeedMockMetricsHistory_PopulatesSeries(t *testing.T) {
|
||||
if got, want := dockerCPU[len(dockerCPU)-1].Value, state.DockerHosts[0].Containers[0].CPUPercent; math.Abs(got-want) > 1e-9 {
|
||||
t.Fatalf("expected last docker cpu point to match current, got=%v want=%v", got, want)
|
||||
}
|
||||
|
||||
storageMetrics := mh.GetAllStorageMetrics("local", time.Hour)
|
||||
if len(storageMetrics["usage"]) < 10 || len(storageMetrics["used"]) < 10 || len(storageMetrics["avail"]) < 10 {
|
||||
t.Fatalf("expected seeded storage capacity history, got usage=%d used=%d avail=%d", len(storageMetrics["usage"]), len(storageMetrics["used"]), len(storageMetrics["avail"]))
|
||||
}
|
||||
last := len(storageMetrics["usage"]) - 1
|
||||
if got, want := storageMetrics["used"][last].Value, float64(state.Storage[0].Used); math.Abs(got-want) > 1e-9 {
|
||||
t.Fatalf("expected last storage used point to match current, got=%v want=%v", got, want)
|
||||
}
|
||||
if got, want := storageMetrics["avail"][last].Value, float64(state.Storage[0].Free); math.Abs(got-want) > 1e-9 {
|
||||
t.Fatalf("expected last storage avail point to match current, got=%v want=%v", got, want)
|
||||
}
|
||||
for i := range storageMetrics["usage"] {
|
||||
if diff := math.Abs(storageMetrics["used"][i].Value + storageMetrics["avail"][i].Value - float64(state.Storage[0].Total)); diff > 0.001 {
|
||||
t.Fatalf("expected storage used+avail to equal total at index %d, diff=%f", i, diff)
|
||||
}
|
||||
}
|
||||
|
||||
diskTemps := mh.GetDiskMetrics("SERIAL-LOCAL-1", "smart_temp", time.Hour)
|
||||
if len(diskTemps) < 10 {
|
||||
t.Fatalf("expected seeded disk temperature history, got %d points", len(diskTemps))
|
||||
}
|
||||
if got, want := diskTemps[len(diskTemps)-1].Value, float64(state.PhysicalDisks[0].Temperature); math.Abs(got-want) > 1e-9 {
|
||||
t.Fatalf("expected last disk temp point to match current, got=%v want=%v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedMockMetricsHistory_AppendsSingleTerminalNowPoint(t *testing.T) {
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
|
||||
state := models.StateSnapshot{
|
||||
DockerHosts: []models.DockerHost{
|
||||
{
|
||||
ID: "host-1",
|
||||
Status: "online",
|
||||
CPUUsage: 22.5,
|
||||
Memory: models.Memory{Usage: 58, Total: 16 * 1024 * 1024 * 1024},
|
||||
Disks: []models.Disk{
|
||||
{Total: 1000, Used: 600, Usage: 60},
|
||||
},
|
||||
Containers: []models.DockerContainer{
|
||||
{
|
||||
ID: "cont-1",
|
||||
State: "running",
|
||||
CPUPercent: 3.3,
|
||||
MemoryPercent: 11.2,
|
||||
WritableLayerBytes: 10,
|
||||
RootFilesystemBytes: 100,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
mh := NewMetricsHistory(1000, 24*time.Hour)
|
||||
seedMockMetricsHistory(mh, nil, fixtureGraphWithState(state), now, time.Hour, 30*time.Second)
|
||||
|
||||
memorySeries := mh.GetGuestMetrics("docker:cont-1", "memory", time.Hour)
|
||||
if len(memorySeries) < 2 {
|
||||
t.Fatalf("expected seeded docker memory points, got %d", len(memorySeries))
|
||||
}
|
||||
|
||||
last := memorySeries[len(memorySeries)-1]
|
||||
if !last.Timestamp.Equal(now) {
|
||||
t.Fatalf("expected terminal docker memory timestamp %v, got %v", now, last.Timestamp)
|
||||
}
|
||||
|
||||
nowCount := 0
|
||||
for _, point := range memorySeries {
|
||||
if point.Timestamp.Equal(now) {
|
||||
nowCount++
|
||||
}
|
||||
}
|
||||
if nowCount != 1 {
|
||||
t.Fatalf("expected exactly one terminal now point, got %d", nowCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedMockMetricsHistory_PopulatesKubernetesPodSeries(t *testing.T) {
|
||||
@@ -395,6 +494,11 @@ func TestSeedMockMetricsHistory_SeedsTrueNASMetricsStore(t *testing.T) {
|
||||
t.Fatal("expected metrics store to have seeded TrueNAS pool used points")
|
||||
}
|
||||
|
||||
diskTempPoints := mh.GetDiskMetrics(fixtures.Disks[0].Serial, "smart_temp", 7*24*time.Hour)
|
||||
if len(diskTempPoints) == 0 {
|
||||
t.Fatal("expected in-memory history to have seeded TrueNAS disk temperature points")
|
||||
}
|
||||
|
||||
appPoints, err := store.Query("dockerContainer", "nextcloud", "cpu", now.Add(-7*24*time.Hour), now, 3600)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to query TrueNAS app cpu metrics: %v", err)
|
||||
@@ -557,6 +661,108 @@ func TestStartMockMetricsSampler_DoesNotClearExistingMetricsStoreData(t *testing
|
||||
}
|
||||
}
|
||||
|
||||
func TestStartMockMetricsSampler_SeedsCanonicalMockResourceHistory(t *testing.T) {
|
||||
previousEnabled := mock.IsMockEnabled()
|
||||
previousConfig := mock.GetConfig()
|
||||
t.Cleanup(func() {
|
||||
mock.SetEnabled(false)
|
||||
mock.SetMockConfig(previousConfig)
|
||||
if previousEnabled {
|
||||
mock.SetEnabled(true)
|
||||
mock.SetMockConfig(previousConfig)
|
||||
}
|
||||
})
|
||||
|
||||
cfg := mock.DefaultConfig
|
||||
cfg.NodeCount = 3
|
||||
cfg.DockerHostCount = 2
|
||||
cfg.DockerContainersPerHost = 5
|
||||
cfg.RandomMetrics = true
|
||||
|
||||
mock.SetEnabled(false)
|
||||
mock.SetMockConfig(cfg)
|
||||
mock.SetEnabled(true)
|
||||
|
||||
resources, _ := mock.UnifiedResourceSnapshot()
|
||||
if len(resources) == 0 {
|
||||
t.Fatal("expected canonical mock unified resources")
|
||||
}
|
||||
registry := unifiedresources.NewRegistry(nil)
|
||||
registry.IngestResources(resources)
|
||||
|
||||
monitor := &Monitor{
|
||||
metricsHistory: NewMetricsHistory(1000, 24*time.Hour),
|
||||
state: models.NewState(),
|
||||
resourceStore: unifiedresources.NewMonitorAdapter(registry),
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
monitor.startMockMetricsSampler(ctx)
|
||||
t.Cleanup(func() { monitor.stopMockMetricsSampler() })
|
||||
|
||||
graph := mock.CurrentFixtureGraph()
|
||||
if len(graph.State.PhysicalDisks) == 0 {
|
||||
t.Fatal("expected proxmox physical disks in canonical mock graph")
|
||||
}
|
||||
if len(graph.State.DockerHosts) == 0 || len(graph.State.DockerHosts[0].Containers) == 0 {
|
||||
t.Fatal("expected docker app containers in canonical mock graph")
|
||||
}
|
||||
|
||||
proxmoxDiskID := strings.TrimSpace(graph.State.PhysicalDisks[0].Serial)
|
||||
if proxmoxDiskID == "" {
|
||||
proxmoxDiskID = strings.TrimSpace(graph.State.PhysicalDisks[0].ID)
|
||||
}
|
||||
if proxmoxDiskID == "" {
|
||||
t.Fatal("expected proxmox physical disk metric id")
|
||||
}
|
||||
|
||||
diskPoints := monitor.metricsHistory.GetDiskMetrics(proxmoxDiskID, "smart_temp", 7*24*time.Hour)
|
||||
if got := len(diskPoints); got < 300 {
|
||||
t.Fatalf("expected seeded in-memory proxmox disk history, got %d points for %q", got, proxmoxDiskID)
|
||||
}
|
||||
|
||||
diskCharts := monitor.GetPhysicalDiskTemperatureCharts(7 * 24 * time.Hour)
|
||||
diskChart, ok := diskCharts[proxmoxDiskID]
|
||||
if !ok {
|
||||
t.Fatalf("expected disk chart for %q, got keys=%v", proxmoxDiskID, keysDiskCharts(diskCharts))
|
||||
}
|
||||
if got := len(diskChart.Temperature); got < 300 {
|
||||
t.Fatalf("expected seeded proxmox disk chart history, got %d points for %q", got, proxmoxDiskID)
|
||||
}
|
||||
|
||||
dockerMetricID := strings.TrimSpace(graph.State.DockerHosts[0].Containers[0].ID)
|
||||
if dockerMetricID == "" {
|
||||
t.Fatal("expected docker app container metric id")
|
||||
}
|
||||
|
||||
workloadMetrics := monitor.GetGuestMetricsForChartBatch(
|
||||
"dockerContainer",
|
||||
[]GuestChartRequest{{
|
||||
InMemoryKey: "docker:" + dockerMetricID,
|
||||
SQLResourceID: dockerMetricID,
|
||||
}},
|
||||
7*24*time.Hour,
|
||||
)
|
||||
cpuPoints := workloadMetrics[dockerMetricID]["cpu"]
|
||||
memoryPoints := workloadMetrics[dockerMetricID]["memory"]
|
||||
if got := len(cpuPoints); got < 300 {
|
||||
t.Fatalf("expected seeded docker app cpu history, got %d points for %q", got, dockerMetricID)
|
||||
}
|
||||
if got := len(memoryPoints); got < 300 {
|
||||
t.Fatalf("expected seeded docker app memory history, got %d points for %q", got, dockerMetricID)
|
||||
}
|
||||
}
|
||||
|
||||
func keysDiskCharts(charts map[string]DiskChartEntry) []string {
|
||||
keys := make([]string, 0, len(charts))
|
||||
for key := range charts {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
func TestGenerateSeededSeries_Deterministic(t *testing.T) {
|
||||
seed := HashSeed("node", "deterministic", "cpu")
|
||||
seriesA := GenerateSeededSeries(57.3, 240, seed, 0, 100, styleSpiky)
|
||||
@@ -615,3 +821,102 @@ func TestGenerateSeededSeries_SpikyProducesSpikes(t *testing.T) {
|
||||
t.Fatalf("expected some spike events above 40; only got %d", spikeCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateSeededSeriesForTimestamps_StableAcrossOverlappingWindows(t *testing.T) {
|
||||
now := time.Date(2026, time.March, 31, 12, 0, 0, 0, time.UTC)
|
||||
seed := HashSeed("dockerContainer", "orion-2-f54579833f9c", "memory")
|
||||
|
||||
fullWindow := make([]time.Time, 0, 25)
|
||||
for ts := now.Add(-24 * time.Hour); !ts.After(now); ts = ts.Add(time.Hour) {
|
||||
fullWindow = append(fullWindow, ts)
|
||||
}
|
||||
recentWindow := append([]time.Time(nil), fullWindow[len(fullWindow)-7:]...)
|
||||
|
||||
fullSeries := GenerateSeededSeriesForTimestamps(51.9, fullWindow, seed, 0, 100, stylePlateau)
|
||||
recentSeries := GenerateSeededSeriesForTimestamps(51.9, recentWindow, seed, 0, 100, stylePlateau)
|
||||
|
||||
if len(recentSeries) != len(recentWindow) {
|
||||
t.Fatalf("expected %d recent points, got %d", len(recentWindow), len(recentSeries))
|
||||
}
|
||||
|
||||
offset := len(fullSeries) - len(recentSeries)
|
||||
for i := range recentSeries {
|
||||
if fullSeries[offset+i] != recentSeries[i] {
|
||||
t.Fatalf(
|
||||
"overlapping timestamp mismatch at index %d: full=%f recent=%f",
|
||||
i,
|
||||
fullSeries[offset+i],
|
||||
recentSeries[i],
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateSeededMetricSeriesForTimestamps_StableAcrossOverlappingWindows(t *testing.T) {
|
||||
now := time.Date(2026, time.March, 31, 12, 0, 0, 0, time.UTC)
|
||||
seed := HashSeed("dockerContainer", "nebula-1", "netin")
|
||||
|
||||
fullWindow := make([]time.Time, 0, 25)
|
||||
for ts := now.Add(-24 * time.Hour); !ts.After(now); ts = ts.Add(time.Hour) {
|
||||
fullWindow = append(fullWindow, ts)
|
||||
}
|
||||
recentWindow := append([]time.Time(nil), fullWindow[len(fullWindow)-7:]...)
|
||||
|
||||
fullSeries := GenerateSeededMetricSeriesForTimestamps(320, fullWindow, seed, 0, 1200, "netin", styleSpiky)
|
||||
recentSeries := GenerateSeededMetricSeriesForTimestamps(320, recentWindow, seed, 0, 1200, "netin", styleSpiky)
|
||||
|
||||
if len(recentSeries) != len(recentWindow) {
|
||||
t.Fatalf("expected %d recent points, got %d", len(recentWindow), len(recentSeries))
|
||||
}
|
||||
|
||||
offset := len(fullSeries) - len(recentSeries)
|
||||
for i := range recentSeries {
|
||||
if fullSeries[offset+i] != recentSeries[i] {
|
||||
t.Fatalf(
|
||||
"overlapping metric timestamp mismatch at index %d: full=%f recent=%f",
|
||||
i,
|
||||
fullSeries[offset+i],
|
||||
recentSeries[i],
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateSeededMetricSeriesForTimestamps_UsesSameTimelineAsMockRuntime(t *testing.T) {
|
||||
now := time.Date(2026, time.March, 31, 12, 0, 0, 0, time.UTC)
|
||||
timestamps := make([]time.Time, 0, 25)
|
||||
for ts := now.Add(-24 * time.Hour); !ts.After(now); ts = ts.Add(time.Hour) {
|
||||
timestamps = append(timestamps, ts)
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
resourceType string
|
||||
resourceID string
|
||||
metricType string
|
||||
style SeriesStyle
|
||||
}{
|
||||
{resourceType: "dockerContainer", resourceID: "orion-2-f54579833f9c", metricType: "memory", style: stylePlateau},
|
||||
{resourceType: "disk", resourceID: "SERIAL-LOCAL-1", metricType: "smart_temp", style: styleFlat},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
min, max := mock.MetricBounds(tc.resourceType, tc.metricType)
|
||||
current := mock.SampleMetric(tc.resourceType, tc.resourceID, tc.metricType, now)
|
||||
seed := mock.MetricSeed(tc.resourceType, tc.resourceID, tc.metricType)
|
||||
series := GenerateSeededMetricSeriesForTimestamps(current, timestamps, seed, min, max, tc.metricType, tc.style)
|
||||
|
||||
for i, ts := range timestamps {
|
||||
want := mock.SampleMetric(tc.resourceType, tc.resourceID, tc.metricType, ts)
|
||||
if diff := math.Abs(series[i] - want); diff > 1e-9 {
|
||||
t.Fatalf(
|
||||
"expected seeded %s/%s series to match runtime timeline at index %d: got=%f want=%f",
|
||||
tc.resourceType,
|
||||
tc.metricType,
|
||||
i,
|
||||
series[i],
|
||||
want,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -664,7 +664,8 @@ func hostsFromReadState(readState unifiedresources.ReadState) []models.Host {
|
||||
return out
|
||||
}
|
||||
|
||||
// writeSMARTMetrics writes SMART attribute metrics to the persistent metrics store for a single disk.
|
||||
// writeSMARTMetrics writes SMART temperature history to the in-memory chart
|
||||
// buffer and persists SMART attributes when the metrics store is enabled.
|
||||
func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
|
||||
resourceID := unifiedresources.PhysicalDiskMetricID(disk)
|
||||
if resourceID == "" {
|
||||
@@ -673,11 +674,16 @@ func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
|
||||
|
||||
// Temperature (always write if > 0)
|
||||
if disk.Temperature > 0 {
|
||||
m.metricsStore.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
if m.metricsHistory != nil {
|
||||
m.metricsHistory.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
}
|
||||
if m.metricsStore != nil {
|
||||
m.metricsStore.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), now)
|
||||
}
|
||||
}
|
||||
|
||||
attrs := disk.SmartAttributes
|
||||
if attrs == nil {
|
||||
if attrs == nil || m.metricsStore == nil {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -4238,6 +4244,9 @@ func (m *Monitor) syncUnifiedAgentMetrics(store ResourceStoreInterface) {
|
||||
if resource.Type != unifiedresources.ResourceTypeAgent || resource.Metrics == nil {
|
||||
continue
|
||||
}
|
||||
if shouldSkipMockOwnedUnifiedMetricSync(resource) {
|
||||
continue
|
||||
}
|
||||
if monitorHasSource(resource.Sources, unifiedresources.SourceAgent) ||
|
||||
monitorHasSource(resource.Sources, unifiedresources.SourceProxmox) ||
|
||||
monitorHasSource(resource.Sources, unifiedresources.SourceDocker) {
|
||||
@@ -4342,6 +4351,9 @@ func (m *Monitor) syncUnifiedVMMetrics(store ResourceStoreInterface) {
|
||||
if resource.Type != unifiedresources.ResourceTypeVM || resource.Metrics == nil {
|
||||
continue
|
||||
}
|
||||
if shouldSkipMockOwnedUnifiedMetricSync(resource) {
|
||||
continue
|
||||
}
|
||||
|
||||
hasNativeVMWriter := false
|
||||
for _, source := range resource.Sources {
|
||||
@@ -4451,6 +4463,9 @@ func (m *Monitor) syncUnifiedStorageMetrics(store ResourceStoreInterface) {
|
||||
if resource.Type != unifiedresources.ResourceTypeStorage || resource.Metrics == nil || resource.Metrics.Disk == nil {
|
||||
continue
|
||||
}
|
||||
if shouldSkipMockOwnedUnifiedMetricSync(resource) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Native Proxmox storage already writes to history during the storage poller.
|
||||
if resource.Storage != nil && resource.Storage.Platform == "" {
|
||||
@@ -4529,6 +4544,9 @@ func (m *Monitor) syncUnifiedPhysicalDiskMetrics(store ResourceStoreInterface) {
|
||||
if resource.Type != unifiedresources.ResourceTypePhysicalDisk || resource.PhysicalDisk == nil {
|
||||
continue
|
||||
}
|
||||
if shouldSkipMockOwnedUnifiedMetricSync(resource) {
|
||||
continue
|
||||
}
|
||||
|
||||
hasNativeWriter := false
|
||||
for _, source := range resource.Sources {
|
||||
@@ -4602,6 +4620,9 @@ func (m *Monitor) syncUnifiedAppContainerMetrics(store ResourceStoreInterface) {
|
||||
if resource.Type != unifiedresources.ResourceTypeAppContainer || resource.Metrics == nil {
|
||||
continue
|
||||
}
|
||||
if shouldSkipMockOwnedUnifiedMetricSync(resource) {
|
||||
continue
|
||||
}
|
||||
hasDockerSource := false
|
||||
for _, source := range resource.Sources {
|
||||
if source == unifiedresources.SourceDocker {
|
||||
@@ -4695,6 +4716,21 @@ func (m *Monitor) syncUnifiedAppContainerMetrics(store ResourceStoreInterface) {
|
||||
}
|
||||
}
|
||||
|
||||
func shouldSkipMockOwnedUnifiedMetricSync(resource unifiedresources.Resource) bool {
|
||||
if !mock.IsMockEnabled() {
|
||||
return false
|
||||
}
|
||||
|
||||
for _, source := range resource.Sources {
|
||||
switch source {
|
||||
case unifiedresources.SourceTrueNAS, unifiedresources.SourceVMware:
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// getUnifiedResourcesForBroadcast retrieves all resources from the store.
|
||||
// Returns nil if no resource store is configured.
|
||||
func (m *Monitor) getUnifiedResourcesForBroadcast() []unifiedresources.Resource {
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/storagehealth"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
|
||||
@@ -591,6 +592,44 @@ func TestSyncUnifiedAppContainerMetricsRecordsTrueNASHistory(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSyncUnifiedAppContainerMetricsSkipsMockOwnedTrueNASHistoryWhenMockEnabled(t *testing.T) {
|
||||
previousFeature := truenas.IsFeatureEnabled()
|
||||
truenas.SetFeatureEnabled(true)
|
||||
t.Cleanup(func() {
|
||||
truenas.SetFeatureEnabled(previousFeature)
|
||||
})
|
||||
|
||||
previousMock := mock.IsMockEnabled()
|
||||
mock.SetEnabled(true)
|
||||
t.Cleanup(func() {
|
||||
mock.SetEnabled(previousMock)
|
||||
})
|
||||
|
||||
cfg := metrics.DefaultConfig(t.TempDir())
|
||||
store, err := metrics.NewStore(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("metrics.NewStore() error = %v", err)
|
||||
}
|
||||
defer func() { _ = store.Close() }()
|
||||
|
||||
resourceStore := unifiedresources.NewMonitorAdapter(nil)
|
||||
resourceStore.PopulateSnapshotAndSupplemental(models.StateSnapshot{}, map[unifiedresources.DataSource][]unifiedresources.IngestRecord{
|
||||
unifiedresources.SourceTrueNAS: truenas.NewProvider(truenas.DefaultFixtures()).Records(),
|
||||
})
|
||||
|
||||
monitor := &Monitor{
|
||||
resourceStore: resourceStore,
|
||||
metricsHistory: NewMetricsHistory(1024, 24*time.Hour),
|
||||
metricsStore: store,
|
||||
}
|
||||
|
||||
monitor.syncUnifiedAppContainerMetrics(resourceStore)
|
||||
|
||||
if got := len(monitor.GetGuestMetrics("docker:nextcloud", time.Hour)["cpu"]); got != 0 {
|
||||
t.Fatalf("expected mock-owned TrueNAS app history to be skipped, got %d cpu points", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSyncUnifiedAgentMetricsRecordsTrueNASHostHistory(t *testing.T) {
|
||||
previous := truenas.IsFeatureEnabled()
|
||||
truenas.SetFeatureEnabled(true)
|
||||
@@ -649,6 +688,44 @@ func TestSyncUnifiedAgentMetricsRecordsTrueNASHostHistory(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSyncUnifiedAgentMetricsSkipsMockOwnedProviderHistoryWhenMockEnabled(t *testing.T) {
|
||||
previousTrueNAS := truenas.IsFeatureEnabled()
|
||||
truenas.SetFeatureEnabled(true)
|
||||
t.Cleanup(func() {
|
||||
truenas.SetFeatureEnabled(previousTrueNAS)
|
||||
})
|
||||
|
||||
previousMock := mock.IsMockEnabled()
|
||||
mock.SetEnabled(true)
|
||||
t.Cleanup(func() {
|
||||
mock.SetEnabled(previousMock)
|
||||
})
|
||||
|
||||
cfg := metrics.DefaultConfig(t.TempDir())
|
||||
store, err := metrics.NewStore(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("metrics.NewStore() error = %v", err)
|
||||
}
|
||||
defer func() { _ = store.Close() }()
|
||||
|
||||
resourceStore := unifiedresources.NewMonitorAdapter(nil)
|
||||
resourceStore.PopulateSnapshotAndSupplemental(models.StateSnapshot{}, map[unifiedresources.DataSource][]unifiedresources.IngestRecord{
|
||||
unifiedresources.SourceTrueNAS: truenas.NewProvider(truenas.DefaultFixtures()).Records(),
|
||||
})
|
||||
|
||||
monitor := &Monitor{
|
||||
resourceStore: resourceStore,
|
||||
metricsHistory: NewMetricsHistory(1024, 24*time.Hour),
|
||||
metricsStore: store,
|
||||
}
|
||||
|
||||
monitor.syncUnifiedAgentMetrics(resourceStore)
|
||||
|
||||
if got := len(monitor.GetGuestMetrics("agent:truenas-main", time.Hour)["cpu"]); got != 0 {
|
||||
t.Fatalf("expected mock-owned TrueNAS host history to be skipped, got %d cpu points", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSyncUnifiedAgentMetricsRecordsVMwareHostHistory(t *testing.T) {
|
||||
previous := vmware.IsFeatureEnabled()
|
||||
vmware.SetFeatureEnabled(true)
|
||||
@@ -872,6 +949,53 @@ func TestSyncUnifiedPhysicalDiskMetricsRecordsTrueNASDiskHistory(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSyncUnifiedStorageAndDiskMetricsSkipMockOwnedTrueNASHistoryWhenMockEnabled(t *testing.T) {
|
||||
previousFeature := truenas.IsFeatureEnabled()
|
||||
truenas.SetFeatureEnabled(true)
|
||||
t.Cleanup(func() {
|
||||
truenas.SetFeatureEnabled(previousFeature)
|
||||
})
|
||||
|
||||
previousMock := mock.IsMockEnabled()
|
||||
mock.SetEnabled(true)
|
||||
t.Cleanup(func() {
|
||||
mock.SetEnabled(previousMock)
|
||||
})
|
||||
|
||||
cfg := metrics.DefaultConfig(t.TempDir())
|
||||
store, err := metrics.NewStore(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("metrics.NewStore() error = %v", err)
|
||||
}
|
||||
defer func() { _ = store.Close() }()
|
||||
|
||||
resourceStore := unifiedresources.NewMonitorAdapter(nil)
|
||||
resourceStore.PopulateSnapshotAndSupplemental(models.StateSnapshot{}, map[unifiedresources.DataSource][]unifiedresources.IngestRecord{
|
||||
unifiedresources.SourceTrueNAS: truenas.NewProvider(truenas.DefaultFixtures()).Records(),
|
||||
})
|
||||
|
||||
monitor := &Monitor{
|
||||
resourceStore: resourceStore,
|
||||
metricsHistory: NewMetricsHistory(1024, 24*time.Hour),
|
||||
metricsStore: store,
|
||||
}
|
||||
|
||||
monitor.syncUnifiedStorageMetrics(resourceStore)
|
||||
monitor.syncUnifiedPhysicalDiskMetrics(resourceStore)
|
||||
|
||||
if got := len(monitor.GetStorageMetrics("pool:tank", time.Hour)["usage"]); got != 0 {
|
||||
t.Fatalf("expected mock-owned TrueNAS storage history to be skipped, got %d usage points", got)
|
||||
}
|
||||
|
||||
points, err := store.Query("disk", "WD-WX12A3456", "smart_temp", time.Now().Add(-time.Hour), time.Now(), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("store.Query() error = %v", err)
|
||||
}
|
||||
if len(points) != 0 {
|
||||
t.Fatalf("expected mock-owned TrueNAS disk history to be skipped, got %d points", len(points))
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildBroadcastFrontendStatePrefersLiveAlertManagerOverSnapshotAlerts(t *testing.T) {
|
||||
alertManager := alerts.NewManagerWithDataDir(t.TempDir())
|
||||
defer alertManager.Stop()
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { expect, test as base } from '@playwright/test';
|
||||
|
||||
import { createAuthenticatedStorageState, getMockMode, setMockMode } from './helpers';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
type WorkerFixtures = {
|
||||
authStorageStatePath: string;
|
||||
};
|
||||
|
||||
type MetricPoint = {
|
||||
timestamp: number;
|
||||
value: number;
|
||||
};
|
||||
|
||||
type ChartData = {
|
||||
memory?: MetricPoint[];
|
||||
};
|
||||
|
||||
type WorkloadChartsResponse = {
|
||||
data?: Record<string, ChartData>;
|
||||
dockerData?: Record<string, ChartData>;
|
||||
};
|
||||
|
||||
const WORKLOADS_SCREENSHOT_PATH = path.resolve(
|
||||
__dirname,
|
||||
'..',
|
||||
'..',
|
||||
'tmp',
|
||||
'workloads-summary-1h-memory-tail.png',
|
||||
);
|
||||
|
||||
let mockModeWasEnabled: boolean | null = null;
|
||||
|
||||
const test = base.extend<{}, WorkerFixtures>({
|
||||
storageState: async ({ authStorageStatePath }, use) => {
|
||||
await use(authStorageStatePath);
|
||||
},
|
||||
authStorageStatePath: [async ({ browser }, use, workerInfo) => {
|
||||
const storageStatePath = path.resolve(
|
||||
__dirname,
|
||||
'..',
|
||||
'..',
|
||||
'tmp',
|
||||
'playwright-auth',
|
||||
`workloads-memory-tail-${workerInfo.project.name}.json`,
|
||||
);
|
||||
fs.mkdirSync(path.dirname(storageStatePath), { recursive: true });
|
||||
await createAuthenticatedStorageState(browser, storageStatePath);
|
||||
try {
|
||||
await use(storageStatePath);
|
||||
} finally {
|
||||
fs.rmSync(storageStatePath, { force: true });
|
||||
}
|
||||
}, { scope: 'worker' }],
|
||||
});
|
||||
|
||||
async function ensureMockModeEnabled(page: import('@playwright/test').Page): Promise<void> {
|
||||
const state = await getMockMode(page);
|
||||
if (mockModeWasEnabled === null) {
|
||||
mockModeWasEnabled = state.enabled;
|
||||
}
|
||||
if (!state.enabled) {
|
||||
await setMockMode(page, true);
|
||||
}
|
||||
}
|
||||
|
||||
async function dismissWhatsNewModal(page: import('@playwright/test').Page): Promise<void> {
|
||||
const modalTitle = page.getByText('Welcome to the New Navigation!');
|
||||
if (!(await modalTitle.isVisible().catch(() => false))) {
|
||||
return;
|
||||
}
|
||||
await page.getByRole('button', { name: "Let's go" }).click();
|
||||
await expect(modalTitle).toHaveCount(0);
|
||||
}
|
||||
|
||||
function average(values: number[]): number {
|
||||
if (values.length === 0) return 0;
|
||||
return values.reduce((sum, value) => sum + value, 0) / values.length;
|
||||
}
|
||||
|
||||
function memoryTailDeltas(payload: WorkloadChartsResponse): number[] {
|
||||
return [
|
||||
...Object.values(payload.data || {}).map((chartData) => chartData.memory || []),
|
||||
...Object.values(payload.dockerData || {}).map((chartData) => chartData.memory || []),
|
||||
]
|
||||
.map((points) => points.slice().sort((a, b) => a.timestamp - b.timestamp))
|
||||
.filter((points) => points.length >= 8)
|
||||
.map((points) => {
|
||||
const previous = points.slice(-8, -2).map((point) => point.value);
|
||||
const tail = points.slice(-2).map((point) => point.value);
|
||||
return Math.abs(average(tail) - average(previous));
|
||||
})
|
||||
.sort((a, b) => a - b);
|
||||
}
|
||||
|
||||
function percentile(sortedValues: number[], ratio: number): number {
|
||||
if (sortedValues.length === 0) return 0;
|
||||
const boundedRatio = Math.min(1, Math.max(0, ratio));
|
||||
const index = Math.min(
|
||||
sortedValues.length - 1,
|
||||
Math.floor((sortedValues.length - 1) * boundedRatio),
|
||||
);
|
||||
return sortedValues[index];
|
||||
}
|
||||
|
||||
test.describe.serial('Workloads memory tail', () => {
|
||||
test.setTimeout(180_000);
|
||||
|
||||
test.afterAll(async ({ browser }) => {
|
||||
if (mockModeWasEnabled === null) {
|
||||
return;
|
||||
}
|
||||
|
||||
const context = await browser.newContext();
|
||||
const page = await context.newPage();
|
||||
try {
|
||||
const current = await getMockMode(page);
|
||||
if (current.enabled !== mockModeWasEnabled) {
|
||||
await setMockMode(page, mockModeWasEnabled);
|
||||
}
|
||||
} finally {
|
||||
await context.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('keeps 1h workload memory tails visually stable on the live page', async ({ page }, testInfo) => {
|
||||
test.skip(testInfo.project.name.startsWith('mobile-'), 'Desktop runtime proof');
|
||||
|
||||
await ensureMockModeEnabled(page);
|
||||
|
||||
await page.goto('/workloads', { waitUntil: 'domcontentloaded' });
|
||||
await page.evaluate(() => window.localStorage.clear());
|
||||
|
||||
const responsePromise = page.waitForResponse((response) => {
|
||||
const url = response.url();
|
||||
return response.request().method() === 'GET' &&
|
||||
url.includes('/api/charts/workloads?') &&
|
||||
url.includes('range=1h');
|
||||
});
|
||||
|
||||
await page.reload({ waitUntil: 'domcontentloaded' });
|
||||
await expect(page.getByTestId('workloads-summary')).toBeVisible();
|
||||
await dismissWhatsNewModal(page);
|
||||
|
||||
const response = await responsePromise;
|
||||
expect(response.ok()).toBeTruthy();
|
||||
|
||||
const payload = (await response.json()) as WorkloadChartsResponse;
|
||||
const tailDeltas = memoryTailDeltas(payload);
|
||||
expect(percentile(tailDeltas, 0.95)).toBeLessThan(6);
|
||||
expect(tailDeltas[tailDeltas.length - 1]).toBeLessThan(8);
|
||||
|
||||
fs.mkdirSync(path.dirname(WORKLOADS_SCREENSHOT_PATH), { recursive: true });
|
||||
await page.getByTestId('workloads-summary').screenshot({ path: WORKLOADS_SCREENSHOT_PATH });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,169 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { expect, test as base } from '@playwright/test';
|
||||
|
||||
import {
|
||||
apiRequest,
|
||||
createAuthenticatedStorageState,
|
||||
getMockMode,
|
||||
setMockMode,
|
||||
} from './helpers';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
type WorkerFixtures = {
|
||||
authStorageStatePath: string;
|
||||
};
|
||||
|
||||
type MetricPoint = {
|
||||
timestamp: number;
|
||||
value: number;
|
||||
};
|
||||
|
||||
type StorageSeries = {
|
||||
usage?: MetricPoint[];
|
||||
used?: MetricPoint[];
|
||||
avail?: MetricPoint[];
|
||||
};
|
||||
|
||||
type DiskSeries = {
|
||||
temperature?: MetricPoint[];
|
||||
};
|
||||
|
||||
type StorageChartsResponse = {
|
||||
pools?: Record<string, StorageSeries>;
|
||||
disks?: Record<string, DiskSeries>;
|
||||
};
|
||||
|
||||
const ARTIFACTS_DIR = path.resolve(__dirname, '..', '..', 'tmp', 'storage-summary-continuity');
|
||||
|
||||
const test = base.extend<{}, WorkerFixtures>({
|
||||
storageState: async ({ authStorageStatePath }, use) => {
|
||||
await use(authStorageStatePath);
|
||||
},
|
||||
authStorageStatePath: [async ({ browser }, use, workerInfo) => {
|
||||
const storageStatePath = path.resolve(
|
||||
__dirname,
|
||||
'..',
|
||||
'..',
|
||||
'tmp',
|
||||
'playwright-auth',
|
||||
`storage-summary-continuity-${workerInfo.project.name}.json`,
|
||||
);
|
||||
fs.mkdirSync(path.dirname(storageStatePath), { recursive: true });
|
||||
await createAuthenticatedStorageState(browser, storageStatePath);
|
||||
try {
|
||||
await use(storageStatePath);
|
||||
} finally {
|
||||
fs.rmSync(storageStatePath, { force: true });
|
||||
}
|
||||
}, { scope: 'worker' }],
|
||||
});
|
||||
|
||||
function average(values: number[]): number {
|
||||
if (values.length === 0) return 0;
|
||||
return values.reduce((sum, value) => sum + value, 0) / values.length;
|
||||
}
|
||||
|
||||
function diskTemperatureSeries(payload: StorageChartsResponse): MetricPoint[][] {
|
||||
return Object.values(payload.disks ?? {})
|
||||
.map((disk) => disk.temperature ?? [])
|
||||
.map((points) => points.slice().sort((a, b) => a.timestamp - b.timestamp))
|
||||
.filter((points) => points.length >= 8);
|
||||
}
|
||||
|
||||
function worstTemperatureTailDelta(payload: StorageChartsResponse): number {
|
||||
return diskTemperatureSeries(payload)
|
||||
.map((points) => {
|
||||
const previous = points.slice(-8, -2).map((point) => point.value);
|
||||
const tail = points.slice(-2).map((point) => point.value);
|
||||
return Math.abs(average(tail) - average(previous));
|
||||
})
|
||||
.reduce((worst, delta) => Math.max(worst, delta), 0);
|
||||
}
|
||||
|
||||
test.describe.serial('Storage summary chart continuity', () => {
|
||||
test.setTimeout(180_000);
|
||||
|
||||
test('renders coherent storage summary histories across the live storage page', async ({
|
||||
page,
|
||||
}, testInfo) => {
|
||||
test.skip(testInfo.project.name.startsWith('mobile-'), 'Desktop runtime proof');
|
||||
|
||||
const mockMode = await getMockMode(page);
|
||||
if (!mockMode.enabled) {
|
||||
await setMockMode(page, true);
|
||||
}
|
||||
|
||||
fs.mkdirSync(ARTIFACTS_DIR, { recursive: true });
|
||||
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem('pulse_whats_new_v2_shown', 'true');
|
||||
});
|
||||
await page.goto('/storage', { waitUntil: 'domcontentloaded' });
|
||||
await expect(page).toHaveURL(/\/storage/);
|
||||
await expect(page.getByTestId('storage-summary')).toBeVisible();
|
||||
await expect(page.getByText('Pool Usage')).toBeVisible();
|
||||
await expect(page.getByText('Disk Temperature')).toBeVisible();
|
||||
|
||||
await page.getByTestId('storage-summary').screenshot({
|
||||
path: path.resolve(ARTIFACTS_DIR, 'storage-summary-1h.png'),
|
||||
});
|
||||
|
||||
const oneHourResponse = await apiRequest(page, '/api/storage-charts?range=60');
|
||||
expect(oneHourResponse.ok()).toBeTruthy();
|
||||
const oneHourPayload = (await oneHourResponse.json()) as StorageChartsResponse;
|
||||
const oneHourPools = Object.values(oneHourPayload.pools ?? {}) as StorageSeries[];
|
||||
const oneHourDisks = Object.values(oneHourPayload.disks ?? {}) as DiskSeries[];
|
||||
expect(oneHourPools.length).toBeGreaterThan(0);
|
||||
expect(oneHourDisks.length).toBeGreaterThan(0);
|
||||
for (const pool of oneHourPools) {
|
||||
expect((pool.usage ?? []).length).toBeGreaterThanOrEqual(30);
|
||||
expect((pool.used ?? []).length).toBeGreaterThanOrEqual(30);
|
||||
expect((pool.avail ?? []).length).toBeGreaterThanOrEqual(30);
|
||||
}
|
||||
for (const disk of oneHourDisks) {
|
||||
expect((disk.temperature ?? []).length).toBeGreaterThanOrEqual(30);
|
||||
}
|
||||
|
||||
const sevenDayResponse = await apiRequest(page, '/api/storage-charts?range=10080');
|
||||
expect(sevenDayResponse.ok()).toBeTruthy();
|
||||
const sevenDayPayload = (await sevenDayResponse.json()) as StorageChartsResponse;
|
||||
const sevenDayPools = Object.values(sevenDayPayload.pools ?? {}) as StorageSeries[];
|
||||
const sevenDayDisks = Object.values(sevenDayPayload.disks ?? {}) as DiskSeries[];
|
||||
expect(sevenDayPools.length).toBeGreaterThan(0);
|
||||
expect(sevenDayDisks.length).toBeGreaterThan(0);
|
||||
for (const pool of sevenDayPools) {
|
||||
expect((pool.usage ?? []).length).toBeGreaterThanOrEqual(300);
|
||||
expect((pool.used ?? []).length).toBeGreaterThanOrEqual(300);
|
||||
expect((pool.avail ?? []).length).toBeGreaterThanOrEqual(300);
|
||||
}
|
||||
for (const disk of sevenDayDisks) {
|
||||
expect((disk.temperature ?? []).length).toBeGreaterThanOrEqual(300);
|
||||
}
|
||||
expect(worstTemperatureTailDelta(sevenDayPayload)).toBeLessThan(3);
|
||||
|
||||
const sevenDayResponsePromise = page.waitForResponse((response) => {
|
||||
const url = response.url();
|
||||
return response.request().method() === 'GET' &&
|
||||
url.includes('/api/storage-charts?') &&
|
||||
url.includes('range=10080');
|
||||
});
|
||||
await page.getByRole('button', { name: '7d', exact: true }).click();
|
||||
await sevenDayResponsePromise;
|
||||
await page.waitForTimeout(1000);
|
||||
await page.getByTestId('storage-summary').screenshot({
|
||||
path: path.resolve(ARTIFACTS_DIR, 'storage-summary-7d.png'),
|
||||
});
|
||||
|
||||
fs.writeFileSync(
|
||||
path.resolve(ARTIFACTS_DIR, 'storage-summary-1h.json'),
|
||||
JSON.stringify(oneHourPayload, null, 2),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.resolve(ARTIFACTS_DIR, 'storage-summary-7d.json'),
|
||||
JSON.stringify(sevenDayPayload, null, 2),
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user