URLs saved through the resource drawer historically landed in the docker
store under the runtime container key, which any stable record (including
an intentionally empty cleared one) outranks in the unified customUrl
projection, and which orphans on container recreation. On report ingest,
copy those records into the stable app-container guest key when it is
missing (cleared links stay cleared), healing saves stranded before the
drawer moved to the stable identity. Also read the runtime key before its
copy-if-missing container-name snapshot so the freshest write wins among
the docker-store fallbacks.
Refs #1556
The websocket broadcast path applies docker metadata (container
customUrl) via applyDockerMetadataToUnifiedResources, but the REST
/api/resources registry seed read the raw unified state view, so the
two payloads drifted: a container web-interface URL saved in the
drawer never appeared in REST-hydrated tables. Apply the same
hydration at the UnifiedResourceSnapshot provider boundary.
Force stale negative LXC Docker detections to re-run after the Docker checker is configured so explicit Proxmox guest Docker inventory repopulates after backend restarts.
getConfiguredHostIPs() and the new discoveryConfigSnapshot() now deep-copy config under a brief RLock and release it before resolving Proxmox/PBS/PMG hostnames via the injectable lookupConfiguredHostIP seam, so slow or blocked DNS can no longer stall monitor writers. The discovery IPBlocklist is the deduplicated merge of the operator-configured blocklist and the resolved configured-host IPs. Adds lock-freedom and merge coverage tests; updates the monitoring contract obligation.
The unified-resource adapter flattened models.ZFSPool to four scalars
(state + R/W/C totals), so scan status and the per-device report never
reached the frontend. The storage detail's ZFS card, the usage bar's
scrub/resilver overlay and ZFS tooltip, and ZFS-device-to-disk matching
were all dead code on the unified path: a degraded pool rendered as a
one-word State cell with no way to see which device failed or that a
resilver was running. Found by the v5-parity audit of the pools table
(same pipeline disease as #1471 on the disks table).
- Add StorageMeta.ZFSPool and populate it in resourceFromStorage
(normalized copy, not aliased).
- Serialize it in monitorStoragePlatformData so the /api/state path
carries it alongside the existing flat scalars.
- Map it through normalizeStorageMeta and the storage record adapter
(meta first, flat platformData fallback).
- Render the per-device report (name, vdev type, state, R/W/C errors,
message) in the detail's ZFS Pool card - the v6 home for what v5
showed via the health-map dots and the degraded-pool warning row.
- Pin the end-to-end path in code_standards_test and update the
unified-resources, monitoring, storage-recovery, and
frontend-primitives contracts.
Verified live in mock mode: degraded pool now shows the resilver scan,
pool error totals, per-device states, and the bar pulse overlay.
The liveness handler called SchedulerHealth(), which refreshes provider
instance caches under the monitor's write lock and snapshots queues and
breakers on every probe — it only needed the dead-letter count. Add
Monitor.DeadLetterCount() (queue-local mutex only) and use it.
The hot-dev watchdog killed the backend after 2 missed 3s probes; on a
dev box running race-detector test suites and builds that is routine
CPU starvation, and each kill destroyed any in-flight Assistant chat
stream (observed repeatedly during the chat-feel audit). Default the
miss threshold to 4 (~20s) — genuine wedges still die quickly.
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.
- models.Storage gains LastSeen (omitzero), stamped where entries are
built: the PVE storage poll (poll start time, including synthesized
cluster-shared entries; preserved entries for unpolled nodes keep their
old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
pool projection (cluster LastUpdated), and the mock generator (offline
mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
wholesale with each host report, so the host report timestamp is the
container sighting. This matches every other docker sub-resource adapter
(services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
type pbs) to SourcePBS, parented to the PBS instance. Keying them
SourceProxmox would judge their freshness against the 60s Proxmox stale
threshold while PBS polls every 60s by default, flapping healthy
datastores stale between polls; SourcePBS carries the cadence-matched
120s threshold. PVE-reported pbs-typed storage.cfg backends stay
SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
reports the honest sighting; mock refresh re-stamps available storage on
each simulated poll.
The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
Ingest stamped every resource's SourceStatus "online" and replaced a zero
LastSeen with time.Now(), so a source that never delivered a resource
(synthesized offline PVE placeholders, never-polled instances) reported a
fresh sighting and a healthy source forever: markStaleLocked skips zero
LastSeen entries, so the stamp was permanently exempt from stale-marking.
That fabricated freshness fed monitor previous-state reads (the
resurrection loop mitigated in 8372a22c5), the frontend lastSeen display,
the monitored-systems ledger, and AI/agent context.
- ingest/mergeInto/IngestResources/presentation coalesce: stamp per-source
delivery status from the actual sighting ("unknown" when the source has
never delivered, "online" otherwise) and preserve zero LastSeen instead
of substituting ingest time.
- replaceRegistry: run MarkStale over the fully assembled registry. The
IngestSnapshot stale pass ran before record sources (TrueNAS, VMware)
were ingested, leaving them stamped "online" at any age.
- monitorLastSeenUnix: report 0 for never-seen instead of time.Now(), so
the frontend renders a dash rather than "just now".
- connected infrastructure: clamp zero LastSeen to 0 ms instead of the
negative UnixMilli of the zero time.
preserveOrExpireNodes keeps requiring a real online sighting for grace;
the registry no longer manufactures the timestamps that made that guard
necessary, so the 8372a22c5 failure mode is impossible by construction.
Two regressions surfaced by generating a real-mode agent report after a
backend restart:
- Report subject lookups used Monitor.GetUnifiedResources (the raw
resource store), but the raw store's canonical IDs depend on per-boot
ingest order for merged-source hosts: after a restart the same host
resolved to a different agent-<hash> than the one the UI and
/api/state advertise, so reports lost the resource name, availability,
and metrics translation entirely. Subject enrichment now reads
Monitor.UnifiedResourceSnapshot and MetricsTargetForResource resolves
through GetUnifiedReadStateOrSnapshot first (raw store as fallback) -
the same re-ingested registry every other read surface uses.
- The performance summary card grid positions cards absolutely and never
paginated: an agent host reporting 8+ metric families walked off the
page bottom, fought fpdf's auto page break, and scattered one orphan
element per page (a 7-day delly report rendered 18 pages, ten of them
near-blank). The grid now starts a new page before a row that will not
fit; the same report renders 9 pages with intact cards.
Verified live: real-mode delly report shows name, 288 data points,
availability, charts, and correctly paginated cards. The underlying
canonical-ID instability (raw store vs re-ingested view, and the
resource_changes journal fragmenting across boot eras) is a separate
root issue tracked for its own fix.
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.
Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.
Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).
Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
Performance reports answered 'what were the averages' but never 'was my
infrastructure up' - the question a managed-service client reads a
monthly report for. Reports now carry an Availability summary derived
from the recorded resource change timeline (state_transition entries
keyed by the canonical unified ID):
- uptime percent over the observed portion of the window, outage count,
total downtime, and longest outage, rendered in the executive summary
with an explicit semantics note; fleet summaries gain a per-resource
Uptime column and CSV exports gain availability header lines
- absent/unknown spans are unobserved time: excluded from the uptime
math entirely and disclosed as coverage, never counted as downtime.
The journal records a registry absence for every monitor restart, so
treating gaps as outages would invent fleet-wide downtime every time
the operator restarts Pulse
- warning states count as up (the resource is reachable and serving);
the uptime label clamps rounding so any real downtime can never
display as a clean 100%
- resources with no timeline render no availability section at all
rather than a fabricated number
Verified live against a real 7-day window: uptime/outage/downtime
figures reconcile with the raw resource_changes journal.
Performance reports were structurally disconnected from the v6 ID
space: the UI (and any API caller working from /api/state) addresses
resources by canonical unified ID, while the metrics store is keyed by
each platform's native source ID (the resource's metricsTarget). The
engine queried the store with the unified ID verbatim, so every report
rendered 'Data Points: 0' regardless of how much history existed, and
covers showed raw hash IDs a report reader cannot map to a machine.
- MetricReportRequest gains MetricsResourceID: handlers resolve the
unified ID through the tenant monitor's resource store (new
Monitor.MetricsTargetForResource accessor; the registry computes
targets on demand, they are not persisted on snapshot structs) and
the engine uses it for store queries only. Recovery points and
Patrol findings stay keyed by the unified ID.
- Legacy snapshot models and their alerts are keyed by the metrics
target ID, so enrichment now matches either ID space and resource
names/status resolve again on covers, headers, and fleet rows.
- Fleet summaries mirror the single-report guard: zero data points
across the fleet renders a muted NO DATA card instead of a green
HEALTHY 'All systems operating normally' - false reassurance is the
worst failure mode for a client-facing stability report.
- Em dashes in PDF-bound literals become hyphens; fpdf core fonts are
cp1252 and rendered them as mojibake.
recordTaskResult reset the failure counters on a successful poll but left
LastErrorAt/LastErrorMessage/LastErrorCategory in place forever. The
connections aggregator surfaces that field as a live error banner on
Settings -> Infrastructure and derives the Unauthorized state from it, so
one transient cluster outage (or startup blip) pinned a red "no healthy
nodes available" banner onto a connection that polls fine, and a single
past 401 could pin a healthy connection to Unauthorized.
LastError now means "current outstanding failure since the last success":
recorded on failure, cleared on success. Connections that are genuinely
failing (no successful poll) keep their banner.
Refs #1493
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Back-port v5 fix 22f8dd3bb to v6. Adds normalizeSMARTDeviceIdentifier
(strips a trailing ' [scsi]'/'[ata]'-style type annotation and the /dev/
prefix) and uses it in the device-path match for both mergeNVMeTempsIntoDisks
and mergeHostAgentSMARTIntoDisks, with a non-empty guard. Without it, disks
whose SMART entries carry a typed label never matched by device path, so
their temperature stayed 0. Function signatures/call sites are unchanged
(canonical guardrail still passes). Adds a unit test.
VsphereHostsTable was missing the Version (ESXi build) and Uptime
columns that Proxmox / Docker / Kubernetes / TrueNAS host
equivalents carry. The data was already piped through the
canonical projection — ESXi version on resource.agent.osVersion
(e.g. "8.0.3"), host uptime on resource.uptime (lifted from
InventoryMetrics.UptimeSeconds via the sys.uptime.latest
PerformanceManager counter wired in 23ea4e487 this morning) —
the table just didn't render columns for them.
VsphereHostsTable now renders Version and Uptime cells, ordered
Version-adjacent-to-Host and Uptime before vCenter to mirror the
Proxmox Nodes layout. Column widths trimmed to fit. Uptime uses
the shared formatUptime util in condensed form (e.g. "163d")
with the full label as the cell title attribute. Detail row
colspan bumps 9 to 11 to keep the drawer spanning the full row.
Plumbing the canonical Resource.Uptime through to the frontend
required two fallback extensions surfaced by the column audit:
- useUnifiedResources.ts toResource: the uptime fallback chain
ended on platform-specific carve-outs (agent.uptimeSeconds,
proxmox.uptime, pbs/pmg/kubernetes); vSphere populates only
the canonical Resource.Uptime, so the chain has to land on
v2.uptime. Same shape as the workloads-hook fix in e5b31f484.
- internal/monitoring/monitor.go monitorUptime: the websocket
broadcast converter walked the same platform-specific chain
and silently dropped vSphere host/VM uptime, then merge-
clobbered the REST-loaded value once WS reconnect replayed
the broadcast. Add the canonical resource.Uptime fallback so
the broadcast payload carries uptime for VMware-backed rows
consistently with the REST contract. Carve-outs still take
precedence so existing platforms keep prior behavior.
Verified: vSphere overview Hosts table now renders esxi-01..07
with Version 8.0.3 and Uptime 147d-167d cells. No regression in
Proxmox / Docker / Kubernetes / TrueNAS uptime paths.
Contracts:
- monitoring.md documents the canonical Resource.Uptime fallback
in monitorUptime and the carve-out precedence.
- unified-resources.md adds the same canonical-uptime fallback
rule to the toResource consumer-side contract.
- storage-recovery.md amends rule 29 to call out that the
canonical Resource.Uptime fallback is descriptive host/VM
uptime only and must not be reinterpreted as backup recency
or recovery cadence.
Proofs:
- internal/monitoring/canonical_guardrails_test.go locks the
monitorUptime fallback contract: canonical Resource.Uptime is
surfaced when no carve-out is set, carve-outs take precedence,
nil when nothing populates.
- frontend-modern/src/hooks/__tests__/useUnifiedResources.test.ts
asserts the toResource v2.uptime fallback for a vSphere-shaped
payload with no platform-specific uptime carve-out.
Project vCenter network inventory through canonical resources and add the vSphere Networks table backed by vCenter network topology. Align resource presentation coalescing so state and resource APIs share the same host contract.
monitorFrontendStatus's "pod" branch was re-deriving status from
Kubernetes.PodPhase alone, so a CrashLoopBackOff pod (Phase=Running
with unready containers) was reported as "running" even though
statusFromKubernetesPod had already classified it as StatusWarning.
Drop the unconditional phase=running -> "running" mapping and fall
through to the unified-status switch, which surfaces StatusWarning as
"degraded". Phase-based mappings for pending/unknown/succeeded/failed
are kept since they still cover edge cases where the unified status
is StatusOnline but phase carries useful information.
Adds two regression cases in TestMonitorFrontendAndMetricHelpers:
- pod warning running degraded (CrashLoopBackOff)
- pod online running running (healthy baseline)
Ensure unified resource snapshots include recent standalone host-agent continuity so Infrastructure does not briefly undercount connected systems after login or restart.
A Proxmox host wedged on a ZFS deadlock yesterday took the cluster API poll
with it (context deadline exceeded). The unified connections aggregator
flipped the Connection from active to stale to unreachable, and the
Settings / Infrastructure page rendered the right badges, but no top-nav
alert ever fired because nothing was actively notifying off that derived
state. Patrol's deterministic triage flagged it every minute, but its LLM
investigation stage has been broken since 2026-02-26 so flags never
escalated into user-visible findings. Result: a 3 hour outage I only
noticed because I happened to open Settings.
This wires an active notification off the same connection state the
Settings badges already use:
- internal/alerts/connection.go: new CheckConnection +
clearConnectionDegradedAlert that fire connection-degraded after three
consecutive stale or unreachable observations. Severity scales: stale
warning, unreachable / unauthorized critical. Clear runs through the
same recovery-confirmation gate as clearNodeOfflineAlert so a single
flap back to active doesn't silently resolve a real outage. Paused,
disabled, and non-platform connections are no-ops.
- internal/api/connections_alerts.go: snapshot translator that turns
api.Connection into the narrow alerts.ConnectionSnapshot view. Keeping
the snapshot type inside the alerts package preserves the existing
api -> monitoring import direction; the monitor would have cycled if
it called back into api directly.
- internal/monitoring: new SetConnectionsSnapshotLister hook + a
per-tick checkConnectionAlerts call in the main poll loop, alongside
the existing evaluate*Agents passes.
- internal/api/router.go: register the lister closure on r.monitor so
the alerts loop sees the same Connection rows the HTTP handler does.
- internal/alerts/specs/types.go: add "connection" to the migration
bridge list of accepted ResourceTypes, alongside node / docker-host /
proxmox-disk / etc. The connection concept doesn't have a canonical
unified resource type yet; this matches the existing pattern for
alert-keyed resources that aren't first-class canonical.
Test coverage in internal/alerts/connection_test.go covers active never
fires, three stale observations escalate from pending to warning,
unreachable escalates warning to critical, unauthorized fires critical
cold, paused / disabled / agent never fire, recovery confirmation gate,
and a stale flap during recovery resets the gate.
TestResourceAlertSpecValidateAllowsConnectionMigrationBridgeType mirrors
the existing migration-bridge proof tests for the new type.
Retire runtime/API/UI monitored-system volume enforcement now that infrastructure monitoring is no longer capped.
Keep only legacy metadata scrubbing and purchase-start compatibility for old max_monitored_systems references.
Rename the remaining preview surface to monitored-system impact and make previews explanatory rather than save-blocking.
Update subsystem contracts and RA7 evidence for the caps-retired invariant.
Refs #1447
Use the canonical Docker host identity for token bindings after a matched reconnect, preserve previous/current identity aliases for recreated containers, and keep rejecting token reuse by genuinely different Docker hosts.
Record the RC3 release-gate evidence for the Docker-in-LXC reconnect path.
Parse the /proc/mdstat operation keyword for mdadm arrays and propagate it through host reports, models, unified resources, monitoring views, alert metadata, and AI storage summaries.
Treat recovery and reshape as rebuild signals while silencing routine check and resync maintenance, with fallback rebuild detection only when no mdstat operation is available.
Tests cover mdstat operation parsing plus recovery, check, and resync alert behavior.
Fixes#1446
rcourtman