Judge resources API freshness by the monitor's thresholds (#2745)

Supersedes #2737, rebuilt on current main because its merge with main
conflicted in four files (`monitor.go`, `registry.go`,
`canonical_guardrails_test.go`, `code_standards_test.go`) and a
conflicted merge cannot pass the commit hooks.

The resources API rebuilt its registry from the monitor's unified seed
with the registry default stale thresholds, while the monitor judges
each source by its configured polling cadence (Proxmox/PBS/PMG: max(2x
polling interval, default)). Presentation coalescing used the defaults
too, in both the websocket broadcast and `/api/resources`.

Reproduced before the fix (Proxmox polled every two minutes, read 90
seconds after its poll, agent silent for 75 seconds):

| | websocket, before | `/api/resources`, before | both, after |
|---|---|---|---|
| Proxmox sighting | online | stale | online |
| node status | online | warning | online |
| CPU on a node whose agent keeps its own row (coalesced by hostname) |
70% (silent agent) | 70% (silent agent) | 20% (node's poll) |

## Change

- The seed carries the thresholds its registry generation was judged by.
`MonitorAdapter` records them per generation (`StaleThresholds`,
`GetAllWithMetricsTargetsAndStaleThresholds`). Thresholds set since the
last rebuild wait for the next one, and overlays and live supplemental
refreshes of a generation are judged by its thresholds. A view built
from mock fixtures or a bare snapshot reports none (the defaults it was
judged by).
- `Monitor.UnifiedResourceSnapshotWithStaleThresholds` and
`MultiTenantStateProvider.UnifiedResourceSnapshotWithStaleThresholdsForTenant`
hand seed, freshness and thresholds over from one view.
- `resourceapi` builds its registry with
`NewRegistryWithStaleThresholds` (ingests and stale passes that pass
none use them) and keys its cache on its own copy of the thresholds.
- The coalesce judges metric freshness with `metricSourceStale`, the
rule the stale pass and merge already share. The broadcast and
`ListForPresentation` pass their view's thresholds.
- Conflict resolution with main's exclusion-aware coalesce
(#2725/#2739): `ListForPresentation` passes both its exclusion filter
and its construction thresholds;
`MonitorAdapter.CoalesceForPresentation(resources, thresholds)` takes
the listing's thresholds next to the exclusions, and
`Monitor.coalesceResourcesForPresentation` passes the broadcast view's,
falling back to `CoalescePresentationHostResourcesWithStaleThresholds`.

Not changed here: Patrol's scoped runtime state still rebuilds
registries with the defaults, and PBS/PMG thresholds read the config
rather than runtime polling overrides. Both predate this change.

## Verification

- New: `TestContract_ResourceListJudgesFreshnessLikeTheMonitorBroadcast`
(default monitor, tenant monitor, monitor without a resource store, each
against the same monitor's broadcast),
`TestRegistrySeededFromMonitorJudgesFreshnessByItsThresholds`,
`TestMonitorAdapterReportsTheThresholdsItsGenerationWasJudgedBy`,
`TestUnifiedViewCarriesItsGenerationsStaleThresholds`,
`TestBroadcastCoalescedHostJudgesFreshnessByConfiguredThresholds`,
`TestResourceListRebuildsWhenSeedStaleThresholdsChange`, plus source
guardrails.
- Negative controls: each new test fails with its runtime piece reverted
(base `resources.go`, coalesce by defaults, providers dropping
thresholds, cache keeping the provider's map, adapter reporting pending
thresholds, overlay judged by pending thresholds).
- Full `internal/unifiedresources`, `internal/api/resourceapi` and
`internal/monitoring` suites pass. Re-run on the rebuilt tree: full
`internal/unifiedresources`, `internal/api/resourceapi` and
`internal/monitoring` suites pass, plus the stale-threshold,
resource-list and contract tests in `internal/api`. The full
`internal/api` suite passes except
`TestServerInfoEndpointReportsDevelopment`, which fails only because the
test tree had no `.git`.
This commit is contained in:
rcourtman
2026-10-08 09:19:38 +01:00
committed by GitHub
21 changed files with 902 additions and 64 deletions
@@ -8746,6 +8746,17 @@ nodes/controller members, and confirmed inventory removal.
`TestPhysicalDiskReadbackSourceIDFallback` covers missing source metadata.
This is synthetic runtime evidence, not USB hardware or reporter acceptance.
### Coalesced host readings follow configured source freshness
The websocket broadcast coalesces a Proxmox node with a Pulse agent that keeps
its own registry row, and now judges which source's readings the row shows by
the stale thresholds the view's registry used, the monitor's configured ones
on the resource store (`internal/monitoring/monitor.go`).
This changes which readings a coalesced row shows, not agent admission,
registration, linking, heartbeat or removal. Proof:
`TestBroadcastCoalescedHostJudgesFreshnessByConfiguredThresholds` in
`internal/monitoring/monitor_host_agents_test.go`.
### Quick security setup preserves unrelated settings
Authenticated force setup in `internal/api/security_setup_fix.go` retains the
@@ -9570,6 +9570,22 @@ That same resource-handler seed contract must also stay on canonical unified
resource ownership for tenant-scoped requests: once a tenant state provider
implements `UnifiedResourceSnapshotForTenant`, `/api/resources` may not fall
back to raw tenant `StateSnapshot` seeding when that unified seed is empty.
The seed contract carries source freshness too. A provider that implements
`UnifiedResourceSnapshotWithStaleThresholds` (the default monitor) or
`UnifiedResourceSnapshotWithStaleThresholdsForTenant` (the tenant state
provider) returns the seed together with the stale thresholds its registry
generation judged it by, captured with the listing, and the registry
`/api/resources` rebuilds is built with them. Its stale pass and
presentation coalesce then judge as the websocket broadcast does: a Proxmox node
polled every two minutes stays online with a current sighting between polls
instead of reading stale with warning status. The registry cache is keyed by
its own copy of those thresholds as well as the freshness marker. A provider without those
methods leaves the registry defaults, and the resources API reads them
through optional interface checks, so `internal/api/router_helpers.go`
asserts the monitor and the tenant state provider implement them. Proof:
`TestContract_ResourceListJudgesFreshnessLikeTheMonitorBroadcast` (default
and tenant monitors on a resource store, and a monitor without one, each
against the same monitor's broadcast).
That same mock/runtime contract now also governs chart payloads under
`internal/api/chartapi/service.go`: when demo or mock presentation is enabled,
`/api/charts`, `/api/charts/infrastructure`, and `/api/storage-charts` must
@@ -1193,6 +1193,18 @@ Monitoring owns source freshness cadence for Proxmox, PBS, and PMG resources:
the stale threshold is derived from the configured polling interval with a
minimum floor, so API-facing resource status must not degrade merely because a
healthy source is between normal poll cycles.
Each unified view carries the thresholds its listed registry generation was
judged by, read with the listing from the same generation: the resource
store's configured ones, or none (the defaults) for a view built from mock
fixtures or a bare state snapshot.
`UnifiedResourceSnapshotWithStaleThresholds` hands them, with the seed, to
consumers that rebuild a registry from the monitor's resources (the resources
API reads each tenant monitor's), and the websocket broadcast coalesces split
host rows with them, so a coalesced row prefers readings by the same
freshness rule and thresholds as the view's own merges. Regression coverage:
`TestBroadcastCoalescedHostJudgesFreshnessByConfiguredThresholds` and
`TestUnifiedViewCarriesItsGenerationsStaleThresholds` in
`internal/monitoring/monitor_host_agents_test.go`.
PBS and PMG configured instances also have one monitoring-owned runtime
resource identity constructor. Poll publication, connection status, setup and
auto-registration checks, canonical alias resolution, and alert-policy bridges
@@ -6506,6 +6506,15 @@ unchanged and older builds ignore the fields. `internal/api/ai_handlers.go`
reads them for the usage export only. No new file, directory, retention
window, backup, migration or recovery authority is introduced.
### Resource freshness thresholds open no storage or recovery path
`internal/api/resourceapi/resources.go` builds the `/api/resources` registry
with the stale thresholds its seed was judged by, and
`internal/api/router_helpers.go` hands a tenant monitor's seed and thresholds
to it.
Both are read-path changes to source freshness and presentation; no storage,
retention, backup or recovery path is added or moved.
### Webhook configuration never persists the API mask
`internal/api/alerting/notifications.go` resolves each `***REDACTED***`
@@ -1615,6 +1615,45 @@ the stale pass's own threshold rule, so the merge's freshness gate never
calls a source current that the stale pass marks stale, or the reverse. The
gate decides only where no stronger rule does: a hypervisor-managed guest
keeps its platform CPU even while that source is stale.
The resources API's registry rebuild is resource seeding too, and the seed
carries the thresholds its own registry generation judged it by. The monitor
adapter records them per generation (`MonitorAdapter.StaleThresholds`):
thresholds set since a rebuild wait for the next one, and the read-state
overlays and live supplemental refreshes of a generation are judged by that
generation's thresholds, since a listing already marked stale cannot be
cleared by re-judging it. A monitor view lists the store's current
generation together with those thresholds
(`GetAllWithMetricsTargetsAndStaleThresholds` reads both from one registry
pointer); a view the monitor builds from mock fixtures or a bare state
snapshot was judged by the defaults and carries none. `UnifiedResourceSnapshotWithStaleThresholds`
(per tenant, `UnifiedResourceSnapshotWithStaleThresholdsForTenant`) hands the
view's listing, freshness marker and thresholds to the resources API, which
builds its registry with `NewRegistryWithStaleThresholds`: every ingest or
stale pass on that registry that passes no thresholds of its own uses them,
and a cached registry built with other thresholds is rebuilt even at the same
freshness marker. Presentation coalescing picks the source of a merged host
row's readings with the metric merge's own rule (`metricSourceStale`):
`ListForPresentation` passes the thresholds its registry was constructed
with, and the websocket broadcast passes its view's, so on mock and snapshot
views both follow the defaults and still agree. Before this the resources API judged by
the defaults everywhere, so a Proxmox node polled every two minutes read
stale, with warning status, in `/api/resources` ninety seconds after its poll
while the broadcast called it online, and both views showed a silent agent's
readings on a node whose agent keeps its own registry row. Patrol's scoped
runtime state still rebuilds registries with the defaults. Regression
coverage: `TestRegistrySeededFromMonitorJudgesFreshnessByItsThresholds` and
`TestRegistryIngestThresholdsOverrideConstructionThresholds` in
`registry_merge_policy_test.go`,
`TestMonitorAdapterReportsTheThresholdsItsGenerationWasJudgedBy` in
`monitor_adapter_read_state_test.go`,
`TestUnifiedViewCarriesItsGenerationsStaleThresholds` in
`internal/monitoring/monitor_host_agents_test.go`,
`TestResourceAPIRebuildsWithTheSeedingMonitorStaleThresholds` in
`code_standards_test.go`,
`TestResourceListRebuildsWhenSeedStaleThresholdsChange` in
`internal/api/resourceapi/resources_test.go`, and
`TestContract_ResourceListJudgesFreshnessLikeTheMonitorBroadcast` in
`internal/api/contract_test.go`.
Service-discovery readiness is a unified-resource payload contract, not a
drawer-local decoration. Resource list/detail payloads that expose a
+176
View File
@@ -14983,6 +14983,182 @@ func TestContract_ResourceListUsesTenantStateProviderAtStartup(t *testing.T) {
}
}
// The resources API rebuilds its registry from a monitor's unified seed, so it
// judges source freshness by the thresholds that seed's registry judged it by,
// for the default monitor and a tenant monitor alike. A Proxmox estate polled
// every two minutes, read ninety seconds after its last poll, lists as the
// websocket broadcast shows it: every node online on a current poll, with the
// node's own readings over those of an agent silent for seventy-five seconds.
// With the registry defaults the API called each poll stale (warning), and
// both views showed the silent agent's readings on a node whose agent keeps
// its own row. A monitor without a resource store builds its view with the
// defaults, which call the ninety-second polls stale, and both views then
// follow the defaults: no row takes a reading from a source its own sighting
// calls stale over a live one.
func TestContract_ResourceListJudgesFreshnessLikeTheMonitorBroadcast(t *testing.T) {
now := time.Now().UTC()
polledMonitor := func(t *testing.T, withStore bool) *monitoring.Monitor {
t.Helper()
monitor, state, _ := newTestMonitor(t)
setUnexportedField(t, monitor, "config", &config.Config{PVEPollingInterval: 2 * time.Minute})
polled := now.Add(-90 * time.Second)
agentSilent := now.Add(-75 * time.Second)
state.UpdateNodesForInstance("homelab", []models.Node{
{
ID: "homelab-pve1", Name: "pve1", Instance: "homelab", Status: "online",
LinkedAgentID: "host-pve1", CPU: 0.12,
Memory: models.Memory{Total: 64 << 30, Used: 16 << 30, Free: 48 << 30, Usage: 25},
LastSeen: polled,
},
{
ID: "homelab-pve2", Name: "pve2", Instance: "homelab", Status: "online", CPU: 0.30,
Memory: models.Memory{Total: 64 << 30, Used: 32 << 30, Free: 32 << 30, Usage: 50},
LastSeen: polled,
},
{
ID: "homelab-pve3", Name: "pve3", Instance: "homelab", Status: "online", CPU: 0.20,
Memory: models.Memory{Total: 64 << 30, Used: 8 << 30, Free: 56 << 30, Usage: 12.5},
LastSeen: polled,
},
})
state.UpsertHost(models.Host{
ID: "host-pve1", MachineID: "machine-pve1", Hostname: "pve1", LinkedNodeID: "homelab-pve1",
Status: "online", CPUUsage: 80,
Memory: models.Memory{Total: 64 << 30, Used: 48 << 30, Free: 16 << 30, Usage: 75},
LastSeen: agentSilent, IntervalSeconds: 30,
})
state.UpsertHost(models.Host{
ID: "host-pve3", MachineID: "machine-pve3", Hostname: "pve3",
Status: "online", CPUUsage: 70,
Memory: models.Memory{Total: 64 << 30, Used: 40 << 30, Free: 24 << 30, Usage: 62.5},
LastSeen: agentSilent, IntervalSeconds: 30,
})
if withStore {
adapter := unifiedresources.NewMonitorAdapter(nil)
monitor.SetResourceStore(adapter)
adapter.PopulateFromSnapshot(state.GetSnapshot())
}
return monitor
}
type nodeRow struct {
status string
cpu, memory float64
}
nodeNames := map[string]bool{"pve1": true, "pve2": true, "pve3": true}
broadcastRows := func(t *testing.T, monitor *monitoring.Monitor) map[string]nodeRow {
t.Helper()
rows := make(map[string]nodeRow)
for _, resource := range monitor.BuildBroadcastFrontendState().Resources {
if !nodeNames[resource.Name] {
continue
}
if _, dup := rows[resource.Name]; dup {
t.Fatalf("broadcast shows node %s twice", resource.Name)
}
row := nodeRow{status: resource.Status}
if row.status == "degraded" {
// The broadcast spells a host's warning status as degraded.
row.status = string(unifiedresources.StatusWarning)
}
if resource.CPU != nil {
row.cpu = resource.CPU.Current
}
if resource.Memory != nil {
row.memory = resource.Memory.Current
}
rows[resource.Name] = row
}
return rows
}
// listedRows also returns each row's Proxmox sighting, which the
// broadcast payload does not carry.
listedRows := func(t *testing.T, h *ResourceHandlers, orgID string) (map[string]nodeRow, map[string]string) {
t.Helper()
req := httptest.NewRequest(http.MethodGet, "/api/resources?type=agent", nil)
if orgID != "" {
req = req.WithContext(context.WithValue(req.Context(), OrgIDContextKey, orgID))
}
rec := httptest.NewRecorder()
h.HandleListResources(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rec.Code, rec.Body.String())
}
var response ResourcesResponse
if err := json.NewDecoder(rec.Body).Decode(&response); err != nil {
t.Fatalf("decode response: %v", err)
}
rows := make(map[string]nodeRow)
sightings := make(map[string]string)
for _, resource := range response.Data {
if !nodeNames[resource.Name] {
continue
}
if _, dup := rows[resource.Name]; dup {
t.Fatalf("resources API lists node %s twice", resource.Name)
}
row := nodeRow{status: string(resource.Status)}
if resource.Metrics != nil && resource.Metrics.CPU != nil {
row.cpu = resource.Metrics.CPU.Percent
}
if resource.Metrics != nil && resource.Metrics.Memory != nil {
row.memory = resource.Metrics.Memory.Percent
}
rows[resource.Name] = row
sightings[resource.Name] = resource.SourceStatus[unifiedresources.SourceProxmox].Status
}
return rows, sightings
}
current := map[string]nodeRow{
"pve1": {status: "online", cpu: 12, memory: 25},
"pve2": {status: "online", cpu: 30, memory: 50},
"pve3": {status: "online", cpu: 20, memory: 12.5},
}
check := func(t *testing.T, monitor *monitoring.Monitor, h *ResourceHandlers, orgID string, want map[string]nodeRow, wantSighting string) {
t.Helper()
broadcast := broadcastRows(t, monitor)
if !reflect.DeepEqual(broadcast, want) {
t.Fatalf("broadcast node rows = %+v, want %+v", broadcast, want)
}
listed, sightings := listedRows(t, h, orgID)
for name, sighting := range sightings {
if sighting != wantSighting {
t.Errorf("resources API %s Proxmox sighting = %q, want %q", name, sighting, wantSighting)
}
}
if !reflect.DeepEqual(listed, broadcast) {
t.Fatalf("resources API node rows = %+v, want the broadcast's %+v", listed, broadcast)
}
}
t.Run("default monitor", func(t *testing.T) {
monitor := polledMonitor(t, true)
h := NewResourceHandlers(&config.Config{DataPath: t.TempDir()})
h.SetStateProvider(monitor)
check(t, monitor, h, "", current, "online")
})
t.Run("tenant monitor", func(t *testing.T) {
tenant := polledMonitor(t, true)
defaultMonitor, _, _ := newTestMonitor(t)
mtm := &monitoring.MultiTenantMonitor{}
setUnexportedField(t, mtm, "monitors", map[string]*monitoring.Monitor{"acme": tenant})
h := NewResourceHandlers(&config.Config{DataPath: t.TempDir()})
h.SetStateProvider(defaultMonitor)
h.SetTenantStateProvider(NewMultiTenantStateProvider(mtm, defaultMonitor))
check(t, tenant, h, "acme", current, "online")
})
t.Run("monitor without a resource store", func(t *testing.T) {
monitor := polledMonitor(t, false)
h := NewResourceHandlers(&config.Config{DataPath: t.TempDir()})
h.SetStateProvider(monitor)
check(t, monitor, h, "", map[string]nodeRow{
"pve1": {status: "warning", cpu: 80, memory: 75},
"pve2": {status: "warning", cpu: 30, memory: 50},
"pve3": {status: "warning", cpu: 70, memory: 62.5},
}, "stale")
})
}
func TestContract_ResourceCapabilitiesJSONSnapshot(t *testing.T) {
payload := struct {
ResourceID string `json:"resourceId"`
+49 -7
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"io"
"maps"
"net/http"
"sort"
"strconv"
@@ -66,6 +67,22 @@ type UnifiedResourceSnapshotProvider interface {
UnifiedResourceSnapshot() ([]unified.Resource, time.Time)
}
// UnifiedResourceSnapshotWithStaleThresholdsProvider supplies the default
// tenant's unified-resource seed with the stale thresholds the seed's registry
// judged its sightings by (nil: the registry defaults). The registry rebuilt
// from the seed runs its own stale pass, merges and presentation coalesce,
// which must judge by the same thresholds or a source the monitor calls
// current between configured polls reads stale here.
type UnifiedResourceSnapshotWithStaleThresholdsProvider interface {
UnifiedResourceSnapshotWithStaleThresholds() ([]unified.Resource, time.Time, map[unified.DataSource]time.Duration)
}
// TenantUnifiedResourceSnapshotWithStaleThresholdsProvider is the
// tenant-scoped form of UnifiedResourceSnapshotWithStaleThresholdsProvider.
type TenantUnifiedResourceSnapshotWithStaleThresholdsProvider interface {
UnifiedResourceSnapshotWithStaleThresholdsForTenant(orgID string) ([]unified.Resource, time.Time, map[unified.DataSource]time.Duration)
}
// ResourceDiscoveryReadinessProvider projects service-discovery state onto a
// unified resource without coupling the resource API to discovery storage.
type ResourceDiscoveryReadinessProvider interface {
@@ -77,6 +94,9 @@ type registrySeed struct {
resources []unified.Resource
lastUpdate time.Time
unifiedSource bool
// staleThresholds are those the seed's registry judged it by; nil
// means the registry defaults.
staleThresholds map[unified.DataSource]time.Duration
}
// SupplementalRecordsProvider provides out-of-band ingest records for a specific source.
@@ -1087,7 +1107,8 @@ func (h *QueryService) buildRegistry(orgID string) (*unified.ResourceRegistry, e
h.cacheMu.Lock()
entry, ok := h.registryCache[key]
if ok && entry.registry != nil && !seed.lastUpdate.IsZero() && entry.lastUpdate.Equal(seed.lastUpdate) {
if ok && entry.registry != nil && !seed.lastUpdate.IsZero() && entry.lastUpdate.Equal(seed.lastUpdate) &&
maps.Equal(entry.staleThresholds, seed.staleThresholds) {
h.cacheMu.Unlock()
return entry.registry, nil
}
@@ -1100,7 +1121,11 @@ func (h *QueryService) buildRegistry(orgID string) (*unified.ResourceRegistry, e
}
h.supplementalMu.RUnlock()
registry := unified.NewRegistry(store)
// The rebuild runs its own stale pass, merges and presentation coalesce,
// so it judges freshness by the thresholds the seed was judged by; with
// the defaults a Proxmox node polled every two minutes read stale here
// between polls while the monitor called it current.
registry := unified.NewRegistryWithStaleThresholds(store, seed.staleThresholds)
ownedSources := supplementalSnapshotOwnedSources(supplementalProviders, orgID)
supplementalSources := sortedSupplementalSources(supplementalProviders)
if seed.unifiedSource {
@@ -1143,7 +1168,7 @@ func (h *QueryService) buildRegistry(orgID string) (*unified.ResourceRegistry, e
if h.registryCache == nil {
h.registryCache = make(map[string]registryCacheEntry)
}
h.registryCache[key] = registryCacheEntry{registry: registry, lastUpdate: seed.lastUpdate}
h.registryCache[key] = registryCacheEntry{registry: registry, lastUpdate: seed.lastUpdate, staleThresholds: seed.staleThresholds}
h.cacheMu.Unlock()
return registry, nil
@@ -1182,14 +1207,28 @@ func (h *QueryService) registrySeed(orgID string) (registrySeed, error) {
if h.tenantStateProvider == nil {
return seed, errors.New("tenant state provider unavailable")
}
resources, lastUpdate := h.tenantStateProvider.UnifiedResourceSnapshotForTenant(orgID)
seed.resources = resources
seed.lastUpdate = lastUpdate
if provider, ok := any(h.tenantStateProvider).(TenantUnifiedResourceSnapshotWithStaleThresholdsProvider); ok {
var thresholds map[unified.DataSource]time.Duration
seed.resources, seed.lastUpdate, thresholds = provider.UnifiedResourceSnapshotWithStaleThresholdsForTenant(orgID)
seed.staleThresholds = maps.Clone(thresholds)
} else {
seed.resources, seed.lastUpdate = h.tenantStateProvider.UnifiedResourceSnapshotForTenant(orgID)
}
seed.unifiedSource = true
return seed, nil
}
if provider, ok := any(h.stateProvider).(UnifiedResourceSnapshotProvider); ok {
if provider, ok := any(h.stateProvider).(UnifiedResourceSnapshotWithStaleThresholdsProvider); ok {
resources, lastUpdate, thresholds := provider.UnifiedResourceSnapshotWithStaleThresholds()
if len(resources) > 0 || !lastUpdate.IsZero() {
seed.resources = resources
seed.lastUpdate = lastUpdate
seed.unifiedSource = true
// Owned copy: the cache compares it with the next seed's.
seed.staleThresholds = maps.Clone(thresholds)
return seed, nil
}
} else if provider, ok := any(h.stateProvider).(UnifiedResourceSnapshotProvider); ok {
resources, lastUpdate := provider.UnifiedResourceSnapshot()
if len(resources) > 0 || !lastUpdate.IsZero() {
seed.resources = resources
@@ -1655,6 +1694,9 @@ type StorageIncidentSection struct {
type registryCacheEntry struct {
registry *unified.ResourceRegistry
lastUpdate time.Time
// staleThresholds are those the registry was built with; a seed judged
// by others needs a rebuild even at the same lastUpdate.
staleThresholds map[unified.DataSource]time.Duration
// rawList and presentation are lazily built once per registry generation
// and SHARED between requests. Both the slices and the nested data of
// their elements are read-only: take a flat copy (flatCopyResources)
@@ -694,6 +694,72 @@ func TestResourceListInvalidatesUnifiedSeedCacheOnFreshnessChange(t *testing.T)
}
}
type staleThresholdSeedProvider struct {
resources []unified.Resource
freshness time.Time
thresholds map[unified.DataSource]time.Duration
}
func (p *staleThresholdSeedProvider) ReadSnapshot() models.StateSnapshot {
return models.StateSnapshot{}
}
func (p *staleThresholdSeedProvider) UnifiedResourceSnapshotWithStaleThresholds() ([]unified.Resource, time.Time, map[unified.DataSource]time.Duration) {
out := make([]unified.Resource, len(p.resources))
copy(out, p.resources)
return out, p.freshness, p.thresholds
}
// A cached registry was built with one seed's stale thresholds; a seed judged
// by others needs a rebuild even when its freshness marker is unchanged. The
// provider changes its threshold map in place, so the cache must keep its own
// copy to notice.
func TestResourceListRebuildsWhenSeedStaleThresholdsChange(t *testing.T) {
now := time.Now().UTC()
polled := now.Add(-90 * time.Second)
provider := &staleThresholdSeedProvider{
resources: []unified.Resource{{
ID: "agent-pve2", Type: unified.ResourceTypeAgent, Name: "pve2", Status: unified.StatusOnline,
LastSeen: polled, UpdatedAt: polled,
Sources: []unified.DataSource{unified.SourceProxmox},
SourceStatus: map[unified.DataSource]unified.SourceStatus{unified.SourceProxmox: {Status: "online", LastSeen: polled}},
Proxmox: &unified.ProxmoxData{NodeName: "pve2", Instance: "homelab"},
Identity: unified.ResourceIdentity{Hostnames: []string{"pve2"}},
}},
freshness: now,
thresholds: map[unified.DataSource]time.Duration{unified.SourceProxmox: 4 * time.Minute},
}
h := NewQueryService(&config.Config{DataPath: t.TempDir()})
h.SetStateProvider(provider)
list := func(t *testing.T) unified.Resource {
t.Helper()
rec := httptest.NewRecorder()
h.HandleListResources(rec, httptest.NewRequest(http.MethodGet, "/api/resources?type=agent", nil))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rec.Code, rec.Body.String())
}
var response ResourcesResponse
if err := json.NewDecoder(rec.Body).Decode(&response); err != nil {
t.Fatalf("decode response: %v", err)
}
if len(response.Data) != 1 {
t.Fatalf("resources = %#v, want the one node", response.Data)
}
return response.Data[0]
}
if got := list(t); got.Status != unified.StatusOnline || got.SourceStatus[unified.SourceProxmox].Status != "online" {
t.Fatalf("under a four-minute threshold: status %q, Proxmox sighting %q; want online for both",
got.Status, got.SourceStatus[unified.SourceProxmox].Status)
}
provider.thresholds[unified.SourceProxmox] = 60 * time.Second
if got := list(t); got.SourceStatus[unified.SourceProxmox].Status != "stale" {
t.Fatalf("under sixty seconds at the same freshness: Proxmox sighting %q, want stale; the cached registry was reused",
got.SourceStatus[unified.SourceProxmox].Status)
}
}
func TestResourceListMergesOneSidedLinkedHostWhenHostnameCorroborates(t *testing.T) {
now := time.Now().UTC()
node := models.Node{
+20
View File
@@ -5,6 +5,7 @@ import (
"net/http"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/api/resourceapi"
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rs/zerolog/log"
@@ -80,6 +81,25 @@ func (p *MultiTenantStateProvider) UnifiedResourceSnapshotForTenant(orgID string
return monitor.UnifiedResourceSnapshot()
}
// The resources API reads both seeds-with-thresholds through optional
// interface checks, so a signature drift would silently restore the registry
// defaults.
var (
_ resourceapi.UnifiedResourceSnapshotWithStaleThresholdsProvider = (*monitoring.Monitor)(nil)
_ resourceapi.TenantUnifiedResourceSnapshotWithStaleThresholdsProvider = (*MultiTenantStateProvider)(nil)
)
// UnifiedResourceSnapshotWithStaleThresholdsForTenant returns a tenant's
// unified-resource seed with the stale thresholds its registry judged it by,
// read from one monitor in one call.
func (p *MultiTenantStateProvider) UnifiedResourceSnapshotWithStaleThresholdsForTenant(orgID string) ([]unifiedresources.Resource, time.Time, map[unifiedresources.DataSource]time.Duration) {
monitor := p.monitorForTenant(orgID)
if monitor == nil {
return nil, time.Time{}, nil
}
return monitor.UnifiedResourceSnapshotWithStaleThresholds()
}
// ResourceHealthAlertsForTenant returns only the minimal alert facts needed by
// the resource health projection, resolved from the same tenant monitor as the
// resource seed.
@@ -650,8 +650,9 @@ func TestBroadcastResourceProjectionCoalescesSplitHostIdentities(t *testing.T) {
for _, snippet := range []string{
"metricsTargetResolver := unifiedView.metricsTargets",
"metricsTargetResolver = broadcastMetricsTargetResolver(unifiedView.readState)",
"broadcastResources := m.coalesceResourcesForPresentation(unifiedView.readState, unifiedView.resources)",
"if coalesced, ok := coalescer.CoalesceForPresentation(resources); ok {",
"broadcastResources := m.coalesceResourcesForPresentation(unifiedView.readState, unifiedView.resources, unifiedView.staleThresholds)",
"if coalesced, ok := coalescer.CoalesceForPresentation(resources, staleThresholds); ok {",
"return unifiedresources.CoalescePresentationHostResourcesWithStaleThresholds(resources, staleThresholds)",
"broadcastFrontendResources, broadcastCatalogs := convertPresentationResourcesForBroadcast(broadcastResources)",
"attachBroadcastMetricsTargetsInPlace(broadcastResources, metricsTargetResolver)",
"frontendState.CapabilityCatalog = broadcastCatalogs.capabilities",
+22 -10
View File
@@ -4479,7 +4479,9 @@ func (m *Monitor) buildBroadcastFrontendStateFromSnapshotWithClock(snapshot mode
if metricsTargetResolver == nil {
metricsTargetResolver = broadcastMetricsTargetResolver(unifiedView.readState)
}
broadcastResources := m.coalesceResourcesForPresentation(unifiedView.readState, unifiedView.resources)
// Coalescing picks each merged metric's source by freshness, judged by
// the thresholds the view's own registry judged its sightings by.
broadcastResources := m.coalesceResourcesForPresentation(unifiedView.readState, unifiedView.resources, unifiedView.staleThresholds)
// Coalescing owns the outer slice. Decorate that one projection in place,
// not three full-resource copies; nested store data is still read-only.
healthAlerts := resourceHealthAlerts(frontendState.ActiveAlerts)
@@ -4505,7 +4507,7 @@ func (m *Monitor) buildBroadcastFrontendStateFromSnapshotWithClock(snapshot mode
// presentationCoalescingResourceStore is implemented by resource stores whose
// registry can hold the operator's merge exclusions (MonitorAdapter).
type presentationCoalescingResourceStore interface {
CoalesceForPresentation(resources []unifiedresources.Resource) ([]unifiedresources.Resource, bool)
CoalesceForPresentation(resources []unifiedresources.Resource, thresholds map[unifiedresources.DataSource]time.Duration) ([]unifiedresources.Resource, bool)
}
// coalesceResourcesForPresentation applies the presentation host coalesce with
@@ -4515,10 +4517,16 @@ type presentationCoalescingResourceStore interface {
// the rows supplies them when its registry is store-backed, as the resource
// store's adapter and a host-continuity overlay are; the mock view and other
// views built from an unified list carry none, so the resource store's
// adapter supplies them instead.
func (m *Monitor) coalesceResourcesForPresentation(readState unifiedresources.ReadState, resources []unifiedresources.Resource) []unifiedresources.Resource {
// adapter supplies them instead. staleThresholds are those the listing's
// registry generation judged its sightings by; the coalesce judges each
// merged metric's freshness by them.
func (m *Monitor) coalesceResourcesForPresentation(
readState unifiedresources.ReadState,
resources []unifiedresources.Resource,
staleThresholds map[unifiedresources.DataSource]time.Duration,
) []unifiedresources.Resource {
if coalescer, ok := readState.(presentationCoalescingResourceStore); ok {
if coalesced, ok := coalescer.CoalesceForPresentation(resources); ok {
if coalesced, ok := coalescer.CoalesceForPresentation(resources, staleThresholds); ok {
return coalesced
}
}
@@ -4527,12 +4535,12 @@ func (m *Monitor) coalesceResourcesForPresentation(readState unifiedresources.Re
store := m.resourceStore
m.mu.RUnlock()
if coalescer, ok := store.(presentationCoalescingResourceStore); ok {
if coalesced, ok := coalescer.CoalesceForPresentation(resources); ok {
if coalesced, ok := coalescer.CoalesceForPresentation(resources, staleThresholds); ok {
return coalesced
}
}
}
return unifiedresources.CoalescePresentationHostResources(resources)
return unifiedresources.CoalescePresentationHostResourcesWithStaleThresholds(resources, staleThresholds)
}
// GetLiveStateSnapshot returns the underlying monitor state snapshot without
@@ -5086,6 +5094,9 @@ type monitorUnifiedStateView struct {
readState unifiedresources.ReadState
freshness time.Time
metricsTargets MetricsTargetResourceStore
// staleThresholds are those the listed resources' registry generation
// was judged by; nil means the registry defaults.
staleThresholds map[unifiedresources.DataSource]time.Duration
}
type unifiedResourceReadStateLister interface {
@@ -5155,9 +5166,10 @@ func (m *Monitor) unifiedStateViewWithStandaloneHostContinuity(view monitorUnifi
return view
}
resources, targets := unifiedProjectionResources(lister)
resources, targets, thresholds := unifiedProjectionResourcesWithStaleThresholds(lister)
view.resources = resources
view.metricsTargets = targets
view.staleThresholds = thresholds
if view.freshness.IsZero() {
view.freshness = latestUnifiedResourceLastSeen(resources)
}
@@ -5263,11 +5275,11 @@ func unifiedResourceFreshness(store ResourceStoreInterface, state *models.State)
// associated freshness marker. In mock mode it returns the shared mock
// unified-resource fixture graph rather than the live resource store.
func (m *Monitor) UnifiedResourceSnapshot() ([]unifiedresources.Resource, time.Time) {
view := m.currentUnifiedStateView()
// REST /api/resources seeds its registry from this snapshot. Apply the
// same user-metadata hydration (container customUrl) as the websocket
// broadcast path, so the two payload shapes cannot drift.
return m.applyPersistedMetadataToUnifiedResources(view.resources), view.freshness
resources, freshness, _ := m.UnifiedResourceSnapshotWithStaleThresholds()
return resources, freshness
}
// GetUnifiedReadState returns a typed unified read-state provider when the
@@ -279,6 +279,83 @@ func TestBroadcastSlimmingKeepsHostAgentSupersededIdentityResolvable(t *testing.
}
}
// The broadcast coalesces an agent that keeps its own registry row into its
// Proxmox node by hostname, and picks each reading's source by freshness
// judged by the configured thresholds, as the registry's own merges are. A
// node polled every two minutes, read ninety seconds after its poll, shows its
// own readings rather than those of its agent, silent for seventy-five
// seconds. Polled at the default cadence, the node is stale at ninety seconds
// too, so the agent's readings stand.
func TestBroadcastCoalescedHostJudgesFreshnessByConfiguredThresholds(t *testing.T) {
now := time.Now().UTC()
for _, tc := range []struct {
name string
interval time.Duration
cpu, memory float64
}{
{"two-minute polls", 2 * time.Minute, 20, 12.5},
{"default polls", 10 * time.Second, 70, 62.5},
} {
t.Run(tc.name, func(t *testing.T) {
m := &Monitor{state: models.NewState(), config: &config.Config{PVEPollingInterval: tc.interval}}
m.state.UpdateNodesForInstance("homelab", []models.Node{{
ID: "homelab-pve3", Name: "pve3", Instance: "homelab", Status: "online", CPU: 0.20,
Memory: models.Memory{Total: 64 << 30, Used: 8 << 30, Free: 56 << 30, Usage: 12.5},
LastSeen: now.Add(-90 * time.Second),
}})
m.state.UpsertHost(models.Host{
ID: "host-pve3", MachineID: "machine-pve3", Hostname: "pve3", Status: "online", CPUUsage: 70,
Memory: models.Memory{Total: 64 << 30, Used: 40 << 30, Free: 24 << 30, Usage: 62.5},
LastSeen: now.Add(-75 * time.Second), IntervalSeconds: 30,
})
m.SetResourceStore(unifiedresources.NewMonitorAdapter(nil))
var rows []models.ResourceFrontend
for _, resource := range m.BuildBroadcastFrontendState().Resources {
if resource.Name == "pve3" {
rows = append(rows, resource)
}
}
if len(rows) != 1 {
t.Fatalf("broadcast shows %d pve3 rows, want the node and its agent coalesced into one", len(rows))
}
if rows[0].CPU == nil || rows[0].Memory == nil {
t.Fatalf("pve3 lost its readings: cpu %+v memory %+v", rows[0].CPU, rows[0].Memory)
}
if rows[0].CPU.Current != tc.cpu || rows[0].Memory.Current != tc.memory {
t.Fatalf("pve3 cpu %.1f%%, memory %.1f%%; want %.1f%% and %.1f%%",
rows[0].CPU.Current, rows[0].Memory.Current, tc.cpu, tc.memory)
}
})
}
}
// A monitor view pairs its listing with the thresholds that registry
// generation was judged by. Thresholds set on the store since its last
// rebuild belong to the next generation, so the view and the seed it hands
// the resources API keep the ones the listing was judged by.
func TestUnifiedViewCarriesItsGenerationsStaleThresholds(t *testing.T) {
now := time.Now().UTC()
m := &Monitor{state: models.NewState(), config: &config.Config{PVEPollingInterval: 2 * time.Minute}}
m.state.UpdateNodesForInstance("homelab", []models.Node{{
ID: "homelab-pve2", Name: "pve2", Instance: "homelab", Status: "online", CPU: 0.30,
Memory: models.Memory{Total: 64 << 30, Used: 32 << 30, Free: 32 << 30, Usage: 50},
LastSeen: now.Add(-90 * time.Second),
}})
adapter := unifiedresources.NewMonitorAdapter(nil)
m.SetResourceStore(adapter)
adapter.PopulateFromSnapshot(m.state.GetSnapshot())
adapter.SetStaleThresholds(map[unifiedresources.DataSource]time.Duration{unifiedresources.SourceProxmox: time.Minute})
resources, _, thresholds := m.UnifiedResourceSnapshotWithStaleThresholds()
if got := thresholds[unifiedresources.SourceProxmox]; got != 4*time.Minute {
t.Fatalf("seed thresholds = %v, want the four minutes its generation was judged by", thresholds)
}
if len(resources) != 1 || resources[0].SourceStatus[unifiedresources.SourceProxmox].Status != "online" {
t.Fatalf("seed = %+v, want pve2 with the Proxmox sighting its generation judged online", resources)
}
}
func TestHostAgentCorrelationKeepsSameNamedProxmoxProvidersDistinct(t *testing.T) {
now := time.Now()
state := &models.State{
@@ -6926,6 +7003,10 @@ func (s *broadcastProjectionCountingStore) GetAllWithMetricsTargets() ([]unified
s.reads++
return s.MonitorAdapter.GetAllWithMetricsTargets()
}
func (s *broadcastProjectionCountingStore) GetAllWithMetricsTargetsAndStaleThresholds() ([]unifiedresources.Resource, map[string]unifiedresources.MetricsTarget, map[unifiedresources.DataSource]time.Duration) {
s.reads++
return s.MonitorAdapter.GetAllWithMetricsTargetsAndStaleThresholds()
}
func (*broadcastProjectionCountingStore) TryReplaceRegistryForRead(models.StateSnapshot, time.Duration, func() map[unifiedresources.DataSource][]unifiedresources.IngestRecord) bool {
return false
}
@@ -7079,10 +7160,10 @@ func TestBroadcastPresentationCoalescePrefersTheListingReadState(t *testing.T) {
overlay := unifiedresources.NewRegistry(store)
overlay.IngestResources(resources)
listing := unifiedresources.NewMonitorAdapter(overlay)
if got := len(m.coalesceResourcesForPresentation(listing, listing.GetAll())); got != 2 {
if got := len(m.coalesceResourcesForPresentation(listing, listing.GetAll(), nil)); got != 2 {
t.Fatalf("coalesced %d rows with the listing's exclusion, want 2", got)
}
if got := len(m.coalesceResourcesForPresentation(nil, listing.GetAll())); got != 1 {
if got := len(m.coalesceResourcesForPresentation(nil, listing.GetAll(), nil)); got != 1 {
t.Fatalf("without a listing read state the store generation decides: %d rows, want 1", got)
}
@@ -7090,7 +7171,7 @@ func TestBroadcastPresentationCoalescePrefersTheListingReadState(t *testing.T) {
// operator decisions, so the resource store's adapter supplies them.
mockView := monitorUnifiedStateViewFromResources(resources, now)
withSplit := &Monitor{resourceStore: listing}
if got := len(withSplit.coalesceResourcesForPresentation(mockView.readState, mockView.resources)); got != 2 {
if got := len(withSplit.coalesceResourcesForPresentation(mockView.readState, mockView.resources, mockView.staleThresholds)); got != 2 {
t.Fatalf("mock-style view coalesced %d rows, want the store's split to keep 2", got)
}
}
@@ -57,6 +57,38 @@ func resourceStaleThresholdsForConfig(
return thresholds
}
// UnifiedResourceSnapshotWithStaleThresholds is UnifiedResourceSnapshot
// plus the stale thresholds the snapshot's registry judged its sightings by.
// A consumer that rebuilds a registry from the snapshot, as the resources API
// does, judges by the same thresholds, so its stale pass and presentation
// agree with the monitor's views.
func (m *Monitor) UnifiedResourceSnapshotWithStaleThresholds() ([]unifiedresources.Resource, time.Time, map[unifiedresources.DataSource]time.Duration) {
view := m.currentUnifiedStateView()
return m.applyPersistedMetadataToUnifiedResources(view.resources), view.freshness, view.staleThresholds
}
// unifiedResourceThresholdProjectionLister lists a read state with the stale
// thresholds the listed registry generation was judged by (MonitorAdapter,
// and any wrapper embedding one).
type unifiedResourceThresholdProjectionLister interface {
GetAllWithMetricsTargetsAndStaleThresholds() ([]unifiedresources.Resource, map[string]unifiedresources.MetricsTarget, map[unifiedresources.DataSource]time.Duration)
}
// unifiedProjectionResourcesWithStaleThresholds lists a read state with the
// thresholds the listed generation was judged by: the resource store's
// configured ones, or nil (the registry defaults) for a read state that
// reports none, such as a view built from mock fixtures or a bare snapshot.
func unifiedProjectionResourcesWithStaleThresholds(
lister unifiedResourceReadStateLister,
) ([]unifiedresources.Resource, MetricsTargetResourceStore, map[unifiedresources.DataSource]time.Duration) {
if projection, ok := lister.(unifiedResourceThresholdProjectionLister); ok {
resources, targets, thresholds := projection.GetAllWithMetricsTargetsAndStaleThresholds()
return resources, projectionMetricsTargets(targets), thresholds
}
resources, targets := unifiedProjectionResources(lister)
return resources, targets, nil
}
func (m *Monitor) resourceStaleThresholds() map[unifiedresources.DataSource]time.Duration {
if m == nil {
return ResourceStaleThresholdsForConfig(nil)
@@ -729,6 +729,34 @@ func TestResourceAPIUsesCanonicalTenantUnifiedSeed(t *testing.T) {
}
}
// The resources API rebuilds its registry from a monitor's seed, so the
// rebuild must judge source freshness by the thresholds that seed was judged
// by, delivered with the seed in one call, and must not reuse a registry built
// with other thresholds. A bare NewRegistry falls back to the defaults and
// marks a slowly polled source stale between its configured polls.
func TestResourceAPIRebuildsWithTheSeedingMonitorStaleThresholds(t *testing.T) {
data, err := os.ReadFile(filepath.Join("..", "api", "resourceapi", "resources.go"))
if err != nil {
t.Fatalf("failed to read resources.go: %v", err)
}
source := string(data)
for _, snippet := range []string{
"registry := unified.NewRegistryWithStaleThresholds(store, seed.staleThresholds)",
"seed.resources, seed.lastUpdate, thresholds = provider.UnifiedResourceSnapshotWithStaleThresholdsForTenant(orgID)",
"resources, lastUpdate, thresholds := provider.UnifiedResourceSnapshotWithStaleThresholds()",
"seed.staleThresholds = maps.Clone(thresholds)",
"maps.Equal(entry.staleThresholds, seed.staleThresholds)",
} {
if !strings.Contains(source, snippet) {
t.Fatalf("internal/api/resourceapi/resources.go must contain %q", snippet)
}
}
if strings.Contains(source, "unified.NewRegistry(store)") {
t.Fatal("internal/api/resourceapi/resources.go must not rebuild a tenant registry with the default stale thresholds")
}
}
// Report-merge undoes an operator link through the registry's fold record. A
// candidate ID derived from the merged resource's type names neither side of
// a link, and the resources API's registry, seeded from the monitor's
+57 -7
View File
@@ -26,7 +26,11 @@ type MonitorAdapter struct {
activeAlerts []models.Alert
lastRebuiltAt time.Time
staleThresholds map[DataSource]time.Duration
legacyHistory legacyHistoryBackfill
// registryStaleThresholds are the thresholds the current registry
// generation was built and judged with. staleThresholds may already hold
// newer ones, which reach the registry at the next rebuild.
registryStaleThresholds map[DataSource]time.Duration
legacyHistory legacyHistoryBackfill
overlays overlayReadStateCache
}
@@ -64,8 +68,9 @@ func NewMonitorAdapterWithStaleThresholds(registry *ResourceRegistry, staleThres
}
return &MonitorAdapter{
registry: registry,
staleThresholds: cloneStaleThresholds(staleThresholds),
registry: registry,
staleThresholds: cloneStaleThresholds(staleThresholds),
registryStaleThresholds: cloneStaleThresholds(staleThresholds),
}
}
@@ -80,6 +85,20 @@ func (a *MonitorAdapter) SetStaleThresholds(staleThresholds map[DataSource]time.
a.mu.Unlock()
}
// StaleThresholds returns the source freshness thresholds the current
// registry generation was judged by; nil means the registry defaults.
// Thresholds set since that rebuild are not reported until the next one
// judges a generation by them.
func (a *MonitorAdapter) StaleThresholds() map[DataSource]time.Duration {
if a == nil {
return nil
}
a.mu.RLock()
thresholds := cloneStaleThresholds(a.registryStaleThresholds)
a.mu.RUnlock()
return thresholds
}
func (a *MonitorAdapter) currentStaleThresholds() map[DataSource]time.Duration {
if a == nil {
return nil
@@ -117,7 +136,10 @@ func readStateWithRecords(readState ReadState, source DataSource, records []Inge
return readState
}
registry := adapter.currentRegistry()
// The overlay re-judges the generation's listing, so it takes the
// thresholds that generation was judged by, not ones set since: those
// cannot clear a stale verdict the listing already carries.
registry, thresholds := adapter.currentRegistryAndStaleThresholds()
if registry == nil {
return readState
}
@@ -133,7 +155,6 @@ func readStateWithRecords(readState ReadState, source DataSource, records []Inge
}
cloned := NewRegistry(registry.store)
thresholds := adapter.currentStaleThresholds()
cloned.IngestResourcesWithStaleThresholds(registry.List(), thresholds)
cloned.ingestRecords(source, records, onlyMissing, thresholds)
overlay := NewMonitorAdapterWithStaleThresholds(cloned, thresholds)
@@ -184,6 +205,19 @@ func (c *overlayReadStateCache) store(registry *ResourceRegistry, rebuiltAt time
c.overlay = overlay
}
// currentRegistryAndStaleThresholds reads the current registry generation
// together with the thresholds it was judged by.
func (a *MonitorAdapter) currentRegistryAndStaleThresholds() (*ResourceRegistry, map[DataSource]time.Duration) {
if a == nil {
return nil, nil
}
a.mu.RLock()
registry := a.registry
thresholds := cloneStaleThresholds(a.registryStaleThresholds)
a.mu.RUnlock()
return registry, thresholds
}
func (a *MonitorAdapter) currentRegistry() *ResourceRegistry {
if a == nil {
return nil
@@ -496,6 +530,7 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, re
a.mu.Lock()
a.registry = rebuilt
a.registryStaleThresholds = staleThresholds
a.activeAlerts = append([]models.Alert(nil), snapshot.ActiveAlerts...)
a.lastRebuiltAt = rebuiltAt
a.mu.Unlock()
@@ -571,6 +606,19 @@ func (a *MonitorAdapter) GetAllWithMetricsTargets() ([]Resource, map[string]Metr
return registry.ListWithMetricsTargets()
}
// GetAllWithMetricsTargetsAndStaleThresholds is GetAllWithMetricsTargets
// plus the stale thresholds the listed registry generation was judged by,
// read with the registry pointer, so a rebuild publishing meanwhile cannot
// pair one generation's listing with another's thresholds.
func (a *MonitorAdapter) GetAllWithMetricsTargetsAndStaleThresholds() ([]Resource, map[string]MetricsTarget, map[DataSource]time.Duration) {
registry, thresholds := a.currentRegistryAndStaleThresholds()
if registry == nil {
return nil, nil, nil
}
resources, targets := registry.ListWithMetricsTargets()
return resources, targets, thresholds
}
func (a *MonitorAdapter) unifiedAIAdapter() *UnifiedAIAdapter {
registry := a.currentRegistry()
if registry == nil {
@@ -684,12 +732,14 @@ func (a *MonitorAdapter) PopulateSupplementalRecords(source DataSource, records
a.mutationMu.Lock()
defer a.mutationMu.Unlock()
registry := a.currentRegistry()
// Records merge into the current generation, so they are judged by the
// thresholds it was; thresholds set since wait for the next rebuild.
registry, thresholds := a.currentRegistryAndStaleThresholds()
if registry == nil || len(records) == 0 || strings.TrimSpace(string(source)) == "" {
return
}
before := registry.List()
registry.IngestRecordsWithStaleThresholds(source, records, a.currentStaleThresholds())
registry.IngestRecordsWithStaleThresholds(source, records, thresholds)
rebuiltAt := time.Now().UTC()
recordRegistryChanges(registry.store, before, registry.List(), rebuiltAt, nil, SourcePlatformEvent, changeSourceAdapterForDataSource(source))
registry.PersistIdentityPins()
@@ -628,6 +628,61 @@ func TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds(t *testing.T) {
})
}
// The adapter reports the thresholds its current registry generation was
// judged by. Thresholds set since then wait for the rebuild that judges a
// generation by them, so a consumer rebuilding from the current listing never
// re-judges it by thresholds it was not judged by. A read-state overlay of
// that generation is judged by the same thresholds: re-judging its listing by
// newer, shorter ones would mark polls stale that the generation calls
// current.
func TestMonitorAdapterReportsTheThresholdsItsGenerationWasJudgedBy(t *testing.T) {
snapshot := staleThresholdEstateSnapshot(time.Now().UTC())
configured := map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute}
adapter := NewMonitorAdapterWithStaleThresholds(NewRegistry(nil), configured)
adapter.PopulateFromSnapshot(snapshot)
if got := adapter.StaleThresholds(); got[SourceProxmox] != 4*time.Minute {
t.Fatalf("after the rebuild StaleThresholds = %v, want the four minutes it was judged by", got)
}
adapter.SetStaleThresholds(map[DataSource]time.Duration{SourceProxmox: 60 * time.Second})
if got := adapter.StaleThresholds(); got[SourceProxmox] != 4*time.Minute {
t.Fatalf("before the next rebuild StaleThresholds = %v, want the current generation's four minutes", got)
}
for name, row := range presentedNodeRows(t, adapter.GetAll()) {
if poll := row.SourceStatus[SourceProxmox]; poll.Status != "online" {
t.Fatalf("current generation %s Proxmox sighting = %q, want online as judged", name, poll.Status)
}
}
overlay, ok := ReadStateWithRecords(adapter, SourceAgent, []IngestRecord{{
SourceID: "host-elsewhere",
Resource: Resource{
Type: ResourceTypeAgent, Name: "elsewhere", Status: StatusOnline, LastSeen: time.Now().UTC(),
Agent: &AgentData{AgentID: "host-elsewhere", Hostname: "elsewhere"},
},
Identity: ResourceIdentity{MachineID: "machine-elsewhere", Hostnames: []string{"elsewhere"}},
}}).(*MonitorAdapter)
if !ok || overlay == adapter {
t.Fatal("ReadStateWithRecords did not build an overlay")
}
if got := overlay.StaleThresholds(); got[SourceProxmox] != 4*time.Minute {
t.Fatalf("overlay StaleThresholds = %v, want the generation's four minutes", got)
}
overlayRows := presentedNodeRows(t, overlay.GetAll())
if len(overlayRows) != 3 {
t.Fatalf("overlay lists %d Proxmox node rows, want 3", len(overlayRows))
}
for name, row := range overlayRows {
if poll := row.SourceStatus[SourceProxmox]; poll.Status != "online" {
t.Fatalf("overlay %s Proxmox sighting = %q, want online as the generation judged it", name, poll.Status)
}
}
adapter.PopulateFromSnapshot(snapshot)
if got := adapter.StaleThresholds(); got[SourceProxmox] != 60*time.Second {
t.Fatalf("after the next rebuild StaleThresholds = %v, want the sixty seconds it was judged by", got)
}
}
func TestHostContinuityCannotOverwriteCurrentResource(t *testing.T) {
now := time.Now().UTC()
for _, mode := range []string{"machine-identity", "source-identity", "future-timestamp", "provider-only"} {
@@ -11,7 +11,18 @@ import (
// monitored host when a runtime/platform view and the Pulse agent view share a
// canonical hostname.
func CoalescePresentationHostResources(resources []Resource) []Resource {
return CoalescePresentationHostResourcesWithExclusions(resources, nil)
return coalescePresentationHostResources(resources, nil, nil)
}
// CoalescePresentationHostResourcesWithStaleThresholds applies the
// presentation host coalesce, judging which source's metrics are current by
// the caller's stale thresholds: those the registry that produced the
// resources judged its sightings by.
func CoalescePresentationHostResourcesWithStaleThresholds(
resources []Resource,
thresholds map[DataSource]time.Duration,
) []Resource {
return coalescePresentationHostResources(resources, nil, thresholds)
}
// CoalescePresentationHostResourcesWithExclusions applies the presentation
@@ -20,9 +31,17 @@ func CoalescePresentationHostResourcesWithExclusions(
resources []Resource,
excluded func(left, right Resource) bool,
) []Resource {
coalesced := coalescePresentationHostResourcesOnce(resources, excluded)
return coalescePresentationHostResources(resources, excluded, nil)
}
func coalescePresentationHostResources(
resources []Resource,
excluded func(left, right Resource) bool,
thresholds map[DataSource]time.Duration,
) []Resource {
coalesced := coalescePresentationHostResourcesOnce(resources, excluded, thresholds)
for len(coalesced) < len(resources) {
next := coalescePresentationHostResourcesOnce(coalesced, excluded)
next := coalescePresentationHostResourcesOnce(coalesced, excluded, thresholds)
if len(next) == len(coalesced) {
return refreshPresentationProxmoxChildActionAgents(next)
}
@@ -35,6 +54,7 @@ func CoalescePresentationHostResourcesWithExclusions(
func coalescePresentationHostResourcesOnce(
resources []Resource,
excluded func(left, right Resource) bool,
thresholds map[DataSource]time.Duration,
) []Resource {
if len(resources) == 0 {
return resources
@@ -74,7 +94,7 @@ func coalescePresentationHostResourcesOnce(
if !shouldMergePresentationHostResources(existing, resource) {
continue
}
mergedResource := mergePresentationHostResources(existing, resource)
mergedResource := mergePresentationHostResources(existing, resource, thresholds)
coalesced[existingIndex] = mergedResource
addPresentationParentRedirect(parentRedirects, existing.ID, mergedResource.ID)
addPresentationParentRedirect(parentRedirects, resource.ID, mergedResource.ID)
@@ -414,7 +434,7 @@ func mergePresentationSources(left, right []DataSource) []DataSource {
return merged
}
func mergePresentationHostResources(left, right Resource) Resource {
func mergePresentationHostResources(left, right Resource, thresholds map[DataSource]time.Duration) Resource {
primary, secondary := left, right
if preferPresentationHostPrimary(right, left) {
primary, secondary = right, left
@@ -467,8 +487,12 @@ func mergePresentationHostResources(left, right Resource) Resource {
for source, status := range primary.SourceStatus {
combinedStatus[source] = status
}
// Judged by the producing registry's thresholds and the rule its
// stale pass and metric merge read, so where both rows carry a
// reading this merge prefers a current source over a stale one as
// the registry's own merges do.
stale := func(source DataSource) bool {
return presentationSourceStale(now, combinedStatus, source)
return metricSourceStale(now, combinedStatus, source, thresholds)
}
merged.Metrics = mergePresentationMetrics(merged.Metrics, secondary.Metrics, stale)
}
@@ -568,24 +592,6 @@ func uniquePresentationStrings(values []string) []string {
return unique
}
// presentationSourceStale reports whether a source's most recent report is
// older than its stale threshold. A zero/unknown last-seen is treated as NOT
// stale so coalescing never demotes a source on missing information.
func presentationSourceStale(now time.Time, status map[DataSource]SourceStatus, source DataSource) bool {
if status == nil {
return false
}
st, ok := status[source]
if !ok || st.LastSeen.IsZero() {
return false
}
threshold, ok := defaultStaleThresholds[source]
if !ok {
threshold = 60 * time.Second
}
return now.Sub(st.LastSeen) > threshold
}
// mergePresentationMetric keeps the primary (left) value, except that a metric
// from a stale source must never win over a metric from a live one. Without
// this, a Proxmox node whose Pulse Agent has gone offline shows the agent's
@@ -390,7 +390,7 @@ func TestMonitorAdapterCoalesceForPresentationHonoursExclusions(t *testing.T) {
if split {
want = 2
}
rows, ok := adapter.CoalesceForPresentation(adapter.GetAll())
rows, ok := adapter.CoalesceForPresentation(adapter.GetAll(), nil)
if !ok || len(rows) != want {
t.Fatalf("split=%v: adapter coalesced %d rows (ok=%v), want %d", split, len(rows), ok, want)
}
@@ -403,7 +403,7 @@ func TestMonitorAdapterCoalesceForPresentationHonoursExclusions(t *testing.T) {
// rather than coalescing as if none existed.
storeless := NewRegistry(nil)
storeless.IngestResources(resources)
if _, ok := NewMonitorAdapter(storeless).CoalesceForPresentation(resources); ok {
if _, ok := NewMonitorAdapter(storeless).CoalesceForPresentation(resources, nil); ok {
t.Fatal("a store-less adapter claimed to carry operator decisions")
}
}
@@ -1,5 +1,7 @@
package unifiedresources
import "time"
// presentationExclusionFilter reports whether the presentation host coalesce
// must keep two resources apart because the operator split the pair (unlink
// or report-merge). It reads a copy of the registry's exclusions, so callers
@@ -31,14 +33,15 @@ func (rr *ResourceRegistry) presentationExclusionFilter() func(left, right Resou
// listed from this adapter's current registry generation, honouring the
// operator's merge exclusions through the same filter that
// ResourceRegistry.ListForPresentation uses, so the websocket broadcast keeps
// a pair the operator split apart as the resources API does. It reports
// false, leaving resources untouched, when the registry has no store and so
// carries no operator decisions: the mock view and other read states built
// from an already-unified list.
func (a *MonitorAdapter) CoalesceForPresentation(resources []Resource) ([]Resource, bool) {
// a pair the operator split apart as the resources API does. Metric
// freshness is judged by thresholds: those the listing's registry generation
// judged its sightings by. It reports false, leaving resources untouched,
// when the registry has no store and so carries no operator decisions: the
// mock view and other read states built from an already-unified list.
func (a *MonitorAdapter) CoalesceForPresentation(resources []Resource, thresholds map[DataSource]time.Duration) ([]Resource, bool) {
registry := a.currentRegistry()
if registry == nil || registry.store == nil {
return resources, false
}
return CoalescePresentationHostResourcesWithExclusions(resources, registry.presentationExclusionFilter()), true
return coalescePresentationHostResources(resources, registry.presentationExclusionFilter(), thresholds), true
}
+30 -3
View File
@@ -177,6 +177,12 @@ type ResourceRegistry struct {
// stale pass do. nil outside an ingest and for callers without
// thresholds, where all three use the defaults.
ingestStaleThresholds map[DataSource]time.Duration
// staleThresholds are the freshness thresholds of the monitor whose
// resources this registry holds, fixed at construction. An ingest or
// stale pass without thresholds of its own and the presentation listing
// judge sightings by them; nil means the defaults.
staleThresholds map[DataSource]time.Duration
}
// agentNodeCandidate pairs a resources-map key with its entry so the indexed
@@ -215,6 +221,13 @@ func (rr *ResourceRegistry) ensureCanonicalMetadataLocked() {
// NewRegistry creates a new registry using the provided store for overrides.
func NewRegistry(store ResourceStore) *ResourceRegistry {
return NewRegistryWithStaleThresholds(store, nil)
}
// NewRegistryWithStaleThresholds creates a registry that judges source
// freshness by the given thresholds wherever a caller passes none, for a
// registry rebuilt from a monitor's resources, such as the resources API's.
func NewRegistryWithStaleThresholds(store ResourceStore, thresholds map[DataSource]time.Duration) *ResourceRegistry {
rr := &ResourceRegistry{
resources: make(map[string]*Resource),
bySource: make(map[DataSource]map[string]string),
@@ -222,6 +235,7 @@ func NewRegistry(store ResourceStore) *ResourceRegistry {
store: store,
exclusions: make(map[string]struct{}),
canonicalMetadataDirty: true,
staleThresholds: cloneStaleThresholds(thresholds),
}
rr.bySource[SourceProxmox] = make(map[string]string)
@@ -275,6 +289,7 @@ func (rr *ResourceRegistry) IngestSnapshotWithStaleThresholds(snapshot models.St
}
func (rr *ResourceRegistry) ingestSnapshot(snapshot models.StateSnapshot, thresholds map[DataSource]time.Duration) {
thresholds = rr.thresholdsOrOwn(thresholds)
rr.mu.Lock()
rr.ingestStaleThresholds = thresholds
rr.mu.Unlock()
@@ -658,6 +673,7 @@ func (rr *ResourceRegistry) IngestRecordsWithStaleThresholds(source DataSource,
}
func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRecord, onlyMissing bool, thresholds map[DataSource]time.Duration) {
thresholds = rr.thresholdsOrOwn(thresholds)
rr.mu.Lock()
rr.ingestStaleThresholds = thresholds
rr.mu.Unlock()
@@ -916,6 +932,7 @@ func (rr *ResourceRegistry) IngestResourcesWithStaleThresholds(resources []Resou
}
func (rr *ResourceRegistry) ingestResources(resources []Resource, thresholds map[DataSource]time.Duration) {
thresholds = rr.thresholdsOrOwn(thresholds)
seededIDs := make([]string, 0, len(resources))
for _, incoming := range resources {
resource := cloneResourcePtr(&incoming)
@@ -1503,10 +1520,11 @@ func (rr *ResourceRegistry) listMaterialized(withTargets bool) ([]Resource, map[
// ListForPresentation returns resources in the canonical API/broadcast
// presentation shape, including top-level host coalescing that respects manual
// merge exclusions.
// merge exclusions and judges source freshness by the thresholds the registry
// was constructed with (not those an individual ingest passed).
func (rr *ResourceRegistry) ListForPresentation() []Resource {
resources := rr.List()
return CoalescePresentationHostResourcesWithExclusions(resources, rr.presentationExclusionFilter())
return coalescePresentationHostResources(resources, rr.presentationExclusionFilter(), rr.staleThresholds)
}
// ListByType returns all resources of the provided type.
@@ -1865,13 +1883,22 @@ func (rr *ResourceRegistry) Stats() ResourceStats {
}
// MarkStale marks sources as stale based on last seen timestamps.
// If thresholds is nil, default thresholds are used.
// If thresholds is nil, the registry's own thresholds are used.
func (rr *ResourceRegistry) MarkStale(now time.Time, thresholds map[DataSource]time.Duration) {
thresholds = rr.thresholdsOrOwn(thresholds)
rr.mu.Lock()
defer rr.mu.Unlock()
rr.markStaleLocked(now, thresholds)
}
// thresholdsOrOwn returns a caller's thresholds, else the registry's own.
func (rr *ResourceRegistry) thresholdsOrOwn(thresholds map[DataSource]time.Duration) map[DataSource]time.Duration {
if thresholds != nil {
return thresholds
}
return rr.staleThresholds
}
// sourceSightingStatus derives the per-source delivery status from the last
// time that source actually delivered the resource. A zero sighting means the
// source has never delivered it (synthesized offline placeholders, instances
@@ -356,6 +356,158 @@ func TestMetricMergeFreshnessMatchesTheStalePass(t *testing.T) {
}
}
// staleThresholdEstateSnapshot is a Proxmox estate polled every two minutes,
// read ninety seconds after its last poll: a node merged with its auto-linked
// agent, silent for seventy-five seconds; a node with no agent; and a node
// whose agent the registry keeps as its own row, which presentation coalesces
// into the node by hostname.
func staleThresholdEstateSnapshot(now time.Time) models.StateSnapshot {
polled := now.Add(-90 * time.Second)
agentSilent := now.Add(-75 * time.Second)
return models.StateSnapshot{
Nodes: []models.Node{
{
ID: "homelab-pve1", Name: "pve1", Instance: "homelab", Status: "online",
LinkedAgentID: "host-pve1", CPU: 0.12,
Memory: models.Memory{Total: 64 << 30, Used: 16 << 30, Free: 48 << 30, Usage: 25},
LastSeen: polled,
},
{
ID: "homelab-pve2", Name: "pve2", Instance: "homelab", Status: "online", CPU: 0.30,
Memory: models.Memory{Total: 64 << 30, Used: 32 << 30, Free: 32 << 30, Usage: 50},
LastSeen: polled,
},
{
ID: "homelab-pve3", Name: "pve3", Instance: "homelab", Status: "online", CPU: 0.20,
Memory: models.Memory{Total: 64 << 30, Used: 8 << 30, Free: 56 << 30, Usage: 12.5},
LastSeen: polled,
},
},
Hosts: []models.Host{
{
ID: "host-pve1", MachineID: "machine-pve1", Hostname: "pve1", LinkedNodeID: "homelab-pve1",
Status: "online", CPUUsage: 80,
Memory: models.Memory{Total: 64 << 30, Used: 48 << 30, Free: 16 << 30, Usage: 75},
LastSeen: agentSilent, IntervalSeconds: 30,
},
{
ID: "host-pve3", MachineID: "machine-pve3", Hostname: "pve3",
Status: "online", CPUUsage: 70,
Memory: models.Memory{Total: 64 << 30, Used: 40 << 30, Free: 24 << 30, Usage: 62.5},
LastSeen: agentSilent, IntervalSeconds: 30,
},
},
LastUpdate: now,
}
}
// presentedNodeRows indexes the coalesced rows that carry a Proxmox sighting
// by name.
func presentedNodeRows(t *testing.T, resources []Resource) map[string]Resource {
t.Helper()
rows := make(map[string]Resource)
for _, resource := range resources {
if _, ok := resource.SourceStatus[SourceProxmox]; !ok {
continue
}
if _, dup := rows[resource.Name]; dup {
t.Fatalf("two presented rows carry Proxmox node %q", resource.Name)
}
rows[resource.Name] = resource
}
return rows
}
// A registry rebuilt from the monitor's resources, as the resources API
// rebuilds one, judges source freshness by the monitor's configured
// thresholds: its stale pass, and the presentation coalesce of both views,
// agree with the monitor's own registry. With the defaults the rebuild called
// every node's poll stale at ninety seconds (warning status), and both views
// showed the silent agent's readings on the node whose agent stays its own
// row.
func TestRegistrySeededFromMonitorJudgesFreshnessByItsThresholds(t *testing.T) {
now := time.Now().UTC()
thresholds := map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute}
snapshot := staleThresholdEstateSnapshot(now)
adapter := NewMonitorAdapterWithStaleThresholds(NewRegistry(nil), thresholds)
adapter.PopulateFromSnapshot(snapshot)
adapter.PopulateFromSnapshot(snapshot)
broadcast := presentedNodeRows(t, CoalescePresentationHostResourcesWithStaleThresholds(adapter.GetAll(), thresholds))
seeded := NewRegistryWithStaleThresholds(nil, thresholds)
seeded.IngestResources(adapter.GetAll())
api := presentedNodeRows(t, seeded.ListForPresentation())
want := map[string]struct {
cpu, memory float64
agent bool
}{
"pve1": {cpu: 12, memory: 25, agent: true},
"pve2": {cpu: 30, memory: 50},
"pve3": {cpu: 20, memory: 12.5, agent: true},
}
for view, rows := range map[string]map[string]Resource{"broadcast": broadcast, "resources API": api} {
if len(rows) != len(want) {
t.Fatalf("%s presents %d node rows, want %d", view, len(rows), len(want))
}
for name, expected := range want {
row, ok := rows[name]
if !ok {
t.Fatalf("%s lost node %s", view, name)
}
if poll := row.SourceStatus[SourceProxmox]; poll.Status != "online" {
t.Errorf("%s %s Proxmox sighting = %q, want online: ninety seconds is within the configured four minutes", view, name, poll.Status)
}
if row.Status != StatusOnline {
t.Errorf("%s %s status = %q, want online", view, name, row.Status)
}
if agent, ok := row.SourceStatus[SourceAgent]; ok != expected.agent || (ok && agent.Status != "stale") {
t.Errorf("%s %s agent sighting = %+v (present %v), want a stale one present %v", view, name, agent, ok, expected.agent)
}
if row.Metrics == nil || row.Metrics.CPU == nil || row.Metrics.Memory == nil {
t.Fatalf("%s %s lost its readings: %+v", view, name, row.Metrics)
}
if cpu, memory := row.Metrics.CPU, row.Metrics.Memory; cpu.Source != SourceProxmox || cpu.Percent != expected.cpu ||
memory.Source != SourceProxmox || memory.Percent != expected.memory {
t.Errorf("%s %s cpu %.1f%% from %s, memory %.1f%% from %s; want the current poll's %.1f%% and %.1f%%",
view, name, cpu.Percent, cpu.Source, memory.Percent, memory.Source, expected.cpu, expected.memory)
}
}
}
// The fixture sits where the thresholds decide: a rebuild judged by the
// defaults calls the same polls stale.
defaults := NewRegistry(nil)
defaults.IngestResources(adapter.GetAll())
defaultRows := presentedNodeRows(t, defaults.ListForPresentation())
if len(defaultRows) != len(want) {
t.Fatalf("default-threshold rebuild presents %d node rows, want %d", len(defaultRows), len(want))
}
for name, row := range defaultRows {
if poll := row.SourceStatus[SourceProxmox]; poll.Status != "stale" {
t.Fatalf("default-threshold rebuild %s Proxmox sighting = %q, want stale; the fixture no longer exercises the thresholds", name, poll.Status)
}
}
}
// An ingest that passes thresholds of its own keeps them over the registry's.
func TestRegistryIngestThresholdsOverrideConstructionThresholds(t *testing.T) {
now := time.Now().UTC()
snapshot := staleThresholdEstateSnapshot(now)
registry := NewRegistryWithStaleThresholds(nil, map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute})
registry.IngestSnapshotWithStaleThresholds(snapshot, map[DataSource]time.Duration{SourceProxmox: 60 * time.Second})
rows := presentedNodeRows(t, registry.List())
if len(rows) != 3 {
t.Fatalf("registry lists %d Proxmox node rows, want 3", len(rows))
}
for name, row := range rows {
if poll := row.SourceStatus[SourceProxmox]; poll.Status != "stale" {
t.Fatalf("%s Proxmox sighting = %q, want stale under the ingest's own sixty seconds", name, poll.Status)
}
}
}
// Docker hosts hold a shorter lease than the registry's Docker stale
// threshold, so a silent Docker host was offline for a while and then
// flipped to warning once its sighting went stale.