In mock mode the unified view is keyed on the fixture data version, which
advances on every metric tick, and building it for the default
1,776-resource estate costs about half a second of CPU (many seconds on a
starved process). #2778 moved the metrics-target lookup onto the fixture
structure revision; the other callers that read only what the estate
lists and how it connects still rebuilt the whole view after each tick.
At startup they made 42 view requests, all misses, and with no client
connected the monitor tick alone rebuilt the estate about every nine
seconds.
Callers that read identity and topology now share the view built for the
current structure revision and operator links (currentStructureUnifiedStateView,
GetUnifiedStructureReadState): the Docker-alert prune, the workload chart
prewarm, the demo Patrol resolver, and the host-agent evaluation, which
also lists nodes only when an offline host names a linked node. The
connection-degraded alert feed keeps only platform rows, so it no longer
reads the hosts, their desired-config fingerprints or the PBS instances
that only feed agent rows and a host alias its snapshot drops.
The mock storage chart is not one of them: a linked pool takes its
capacity from whichever source was seen last, which a tick can move
without a structure change. It keeps the data-version view, read once per
request instead of once per uncached pool. The per-node polling lookups
of a monitor that keeps real polling in mock mode (linkedHostForNode,
getHostAgentTemperatureForNode, carriedTemperatureOutlivesAgentLease)
read a host's status, disks and sensors, which a tick moves, so they stay
on the data-version view.
On pulse-dev (one pinned core, host load 30 to 110, probe builds): a
start built the estate view twice before the listener answered with the
wiring change alone and once with this; with no client connected the
process built the estate 33 times in 299 s and used 12.3 s of CPU over a
180 s window, against twice and 3.3 s with this change.
After a restart, an agent an operator linked into a live resource, such
as an agent inside a vSphere VM, exists only through saved-host
continuity until it reports again. Three consumers then disagreed about
what a reference to it meant. The read state kept the saved agent as its
own offline row and resolved references to it there. The resources API,
which seeds a registry from that listing and re-applies the links,
folded it into the VM and merged its stale telemetry. Alert intent
resolved through the published registry, where the agent is absent, so
its agent:<host ID> alerts kept their literal reference. Maintenance or
NeverAutoRemediate set on the VM reached neither the agent's alerts nor
its Patrol findings in that window, while the API said it applied.
An operator link names one identity; telemetry separation is the only
reason the saved row stays apart. Link application now holds a saved
row beside the link's primary instead of merging it, in every registry.
A reference that resolves to the saved row (its canonical ID,
agent:<host ID>, its machine ID or an alias it shares with the primary)
resolves to the primary, as a folded row's references do, while each
row keeps its own telemetry. An in-memory marker on rows continuity
introduced rides clones, so the resources API's registry holds them the
same way. A saved primary leaves a live member its own identity.
Alert intent now resolves references and ancestors through the published
registry with saved hosts overlaid, the read state Patrol resolves
through. That also lets intent set on an unlinked saved host reach its
offline alert. The resolver runs under the alert manager's lock once
per metric check, so the overlay is kept per published registry
generation, publication builds it before alerts are evaluated, and a
build that overlaps a mock-mode switch is discarded.
SetMockMode(false) switched the mock package off and then cleared active
alerts, but every path that evaluates mode-dependent data checked the
mode once, up front. A mock alert pass (started from the ticker with go,
so passes overlap), a frontend or ingest refresh evaluating a GetState
snapshot, a backup-age check over fixture recovery rollups, or a
connection check over the mock ledger could already hold fixture data
when the switch happened. Its evaluations then landed after the clear,
in live mode, where nothing evaluates or resolves fixture resources, so
the alerts stayed until the 24-hour stale sweep.
A pass-level lock around checkMockAlerts was tried and reverted: it left
the other paths open and made the mock-mode API wait behind a whole pass
of synchronous event-log writes.
The monitor now owns a mock-mode epoch fence. The mock pass, backup-age
and snapshot checks, connection checks, Docker pruning and the
resource-store refresh take a scope before reading anything
mode-dependent and send each alert-manager call through it. SetMockMode,
now serialized, flips the mode, ends the epoch, waits only for calls
already admitted, and then clears. Unified metrics are evaluated one
resource per call, so the wait covers single calls, not passes.
Registry rebuilds are tracked rather than waited on. A refresh evaluates
the registry it read back only if no rebuild from an ended epoch
overlapped it, and the readers that open or remove alerts from the
inventory use current state until a clean rebuild in the new epoch
replaces the previous mode's registry. The mock supplemental fixture
sources now serve nothing outside mock mode: the router swaps them for
the real pollers only after the switch, refreshing between sources, so
a single switch to live mode used to raise fixture vSphere incidents.
The new tests park a real evaluation at a fixed point (a held lock or a
blocking supplier), switch modes and release it. Against the unfenced
code the mock pass left about 40 fixture alerts, the frontend refresh 5
vSphere incidents and the backup check 137 backup-age alerts in live
mode, and a mock-mode connection observation counted toward a live
connection's confirmation. The drain, republish and pruning tests fail
against a fence without its wait, overlap check or registry tracking.
Switching mock mode with several tenant monitors running remains
uncovered and is noted in the monitoring contract.
A threshold alert stays open after its reading drops below the trigger:
it holds until the reading reaches the clear level, then must stay there
for the recovery delay. While it held, the evaluator returned without
touching the alert, so Value, Message and LastSeen stayed at the last
breach. On 2026-10-06 minipc's row and guests said "Node temperature at
80.0°C" while the node table read a green 72°C, with nothing explaining
why the warning was still there (also #2068: a disk alert that stayed
open after usage dropped, with no sign of why).
The evaluator now records a volatile MetricStatus on every evaluation:
the value it compared (the average for rolling-average rules, with the
latest sample beside it), the phase (breaching, latched or recovering),
the trigger and clear levels, the recovery delay and the reducer's
monotonic recovery progress. Value, Message and LastSeen keep meaning
the last breach, and the hold branch still never notifies. The status is
stripped from the active-state checkpoint, the recovery mirror and the
degraded recovery projection, so a held alert adds no durable writes. It
reaches the websocket state through the active-alert projection.
Stale-alert cleanup and the retention TTLs now use the latest
observation. Before, an alert held below its trigger for over a day had
an old LastSeen and was auto-resolved as "resource no longer being
monitored" while it was still being evaluated.
Surfaces format the status through one helper and never re-derive the
phase. Node, guest and Mail Gateway drawer rows lead with the reading
now and say what clears the alert, with the last breach on hover. The
Proxmox TEMP cell keeps its own open temperature alert's tone and
explains it on hover instead of reading green beside a warning row.
Storage rows stop saying "Over N%" when usage is back under the limit,
and the Assistant handoff no longer labels the breach value as current.
Refs #2068
Rejected active-alert writes still reached history, firing events and
notification dispatch. Make admission explicit across detector and restore
paths so repeated suppressed observations cannot create those side effects.
Reconcile restored alerts when persisted resource policy attaches at startup,
reconcile again after native aliases become canonical, and refresh shared
state before those alerts reach clients.
Preserve occurrence identity when retained alerts re-fire, including durable
history reconstruction and delayed lifecycle replay.
Keep retained table and card components mounted when scrolling shifts the
visible window, so policy editors do not lose unsaved state. Show actual
future maintenance times and the paused-attention effect.
Keep synthetic host machine identities stable across restarts so saved
operator policy remains attached during verification.
Qualify server update and rollback instructions by helper ownership, and
stop signed installer recipes before execution if a download or signature
check fails.
Refs #2237
The #1601 follow-on: per-container alert overrides were keyed by Docker
container ID (docker:{host}/{containerID}), which changes on every
recreate, so each image update silently re-armed alerts the user had
switched off and left a dead entry behind in alerts.json — the unbounded
growth that pushed the reporter's config past the old 64KB body cap
(raised in 38434a513). The v6 thresholds UI additionally wrote keys from
the unified hash id (docker:{host}/app-container-{16hex}), which the
evaluator never read at all.
Overrides now key on stable identity, docker:{host}/{containerName}:
- The evaluator resolves the name key first and falls back to the legacy
container-ID key so pre-migration entries keep working
(evaluateDockerContainer, checkDockerContainerState, the
container-update resolver, and reevaluateActiveAlertsLocked).
- MigrateDockerContainerOverrideKeys runs in the monitor sync next to
MigrateCanonicalOverrideKeys, driven by the unified resource snapshot:
it re-homes live legacy-ID and unified-hash keys onto the name key and
prunes orphaned ID-shaped entries, ending the per-update orphan
accumulation. Name-keyed entries for absent containers are kept so a
recreate under the same name still honours them.
- The UI candidate chain (single implementation in alertOverridesModel)
now leads with docker:{host}/{name} and trails the container-ID, short
ID, unified-hash and slash-tail forms, so rows bind pre-existing
overrides of every historical shape and the next save re-homes them.
Rows carry overrideStorageId/overrideIdCandidates so toggle,
connectivity, offline-state, edit and remove all write the stable key.
- The ignored-containers card copy now documents the wildcard forms
(runner-*, *-dev, *staging*) shipped in b5fa6a9af, under the title
"Ignored container patterns".
Contract deltas: alerts, frontend-primitives, monitoring, and
unified-resources now pin the name-keyed override identity, the single
frontend candidate-chain owner, the sync-cadence migration, and the
resource-facet-backed table identity respectively.
Verified live against a mock instance: a UI toggle persists
docker:{host}/loki and binds back after reload, and seeded
legacy/hash/orphan keys converge to name keys on disk within two sync
ticks. go test ./internal/alerts/... ./internal/monitoring/... green;
recreate survival pinned in
TestDockerContainerOverrideSurvivesContainerRecreate.
Refs #1601
A guest's unified canonical ID hashed its node-scoped source ID
(instance:node:vmid), so a live migration to another cluster node
re-minted the resource and orphaned every operator-owned row keyed by
the old ID: explicit availability check links (fail-closed by design,
the reported symptom in #1669), alert overrides, operator state, action
audits, manual links, and recovery subjects. VMIDs are unique within a
cluster, so guests now derive their canonical ID from instance+VMID
("proxmox-guest:<instance>:<vmid>") and keep it across migrations. The
guest-metadata half of #1669 was fixed separately at the metadata-store
layer.
Existing installs converge through record-declared succession: ingest
declares the retired node-scoped IDs superseded for every node the
instance currently knows (current names plus native aliases), so rows
orphaned by pre-upgrade migrations also re-key. Successions are now
recorded durably in a canonical_id_successions table, which memoizes
the re-key (steady-state rebuilds re-declare the same eras every tick
without touching SQL) and lets change-journal reads merge retired guest
eras the way pin EraIDs do for hosts. The succession re-key also covers
manual link/exclusion rows.
Availability links resolve retired canonical IDs and old-node source
triples through a registry superseded index plus guest-triple parsing
(persistence keys only, ambiguity fails closed), and the stored
LinkedResourceID re-homes to the current canonical ID on the
alert-migration cadence. Recovery subjects converge on the same
derivation via CanonicalSubjectResourceID, the mapper's registry-miss
fallback, node-independent external guest keys, and the store's startup
backfill, which also sweeps posture rows stranded under retired subject
keys. Metrics history and frontend row identity key off the node-scoped
source ID and are deliberately unchanged.
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
rcourtman