f62f35e24 restored the v5 used | cache | free memory split for Proxmox
nodes and guests, but standalone host agents still reported a flat
used/free pair, so the Machines page memory bar could not show the
reclaimable segment. Flagged by the Machines page v5 parity audit.
- Host agent reports cacheBytes (gopsutil Available minus Free); the
ZFS ARC adjustment recomputes free so used + cache + free still
covers the total.
- ApplyHostReport maps the field into models.Memory.Cache and clamps
inconsistent or older-agent reports so used + cache never exceeds
total.
- AgentMemoryMeta carries cache onto unified resources so the frontend
agent payload exposes it.
- Mock generic hosts split a third of non-used pages as cache, and the
node-linked host conversion now holds the invariant instead of
stacking the node's cache on top of a recomputed free.
- Contracts: monitoring, unified-resources, and storage-recovery now
document the split (also covering the f62f35e24 node/guest surface,
which landed without contract deltas).
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
The mock random-metrics updater recomputes Used/Free from the sampled
percentage but left Cache at its generation-time value, so a drifting
node could show used+cache > total and a 102% 'Shown in Proxmox' row.
Clamp the cache into the non-used pages in applyMemoryUsage, and teach
the memory-bar presentation to clamp defensively so a momentarily
inconsistent snapshot can never render segments past 100%.
v5 modeled memory as used | cache | free (Memory.Cache, 'reclaimable
buff/cache') and the memory bar's tooltip carried a 'Shown in Proxmox'
row explaining why Pulse's percentage reads lower than the Proxmox UI's
cache-inclusive number — a recurring support question. The v6 rebuild
deleted the field from the backend model, so the split and the
reconciliation vanished product-wide. Flagged by the Proxmox overview
parity audit.
Backend: re-add Memory.Cache; split it out via a shared
splitReclaimableMemory helper at the node resolver (node status reports
truly-free directly) and the VM builder (when guest meminfo reported
free pages); transport as proxmox.memoryCache on unified resources
alongside swap/balloon; mock mode populates plausible cache for nodes
and VMs.
Frontend: cache prop on StackedMemoryBar with the v5 muted-amber
segment between active and balloon, tooltip rows for reclaimable cache,
truly-free Free (balloon-capped), and the 'Shown in Proxmox'
reconciliation; guest and node memory adapters normalize free to
truly-free at the boundary; guest and node drawers grow a Reclaimable
cache row; the Proxmox nodes table passes cache and node swap through.
The agent collects QoSClass and the legacy models carry it, but
resourceFromKubernetesPod dropped it, so no v6 surface could render
what v5's pod expansion showed. Additive optional field; proof added
to the kubernetes registry ingest test.
Found by the v5-parity audit of the Kubernetes page.
OpenCode shows how full the model's context is next to the composer;
Pulse showed raw token counts with no sense of scale. The done event
now carries context_limit_tokens (providers.ContextWindowTokens for
the selected route) and the drawer's last-turn summary renders input
tokens as a percentage of the window: '8,500 tokens (6% of context)',
with the full breakdown in the hover detail. Without the limit the
summary renders exactly as before. Pinned in the plain-answer corpus
scenario.
After a tool round, the next model turn can reason server-side for tens
of seconds without streaming a single event. The footer kept the last
tool's status counting up ('Reading inventory. (40s)') while the model
was already composing the answer — observed live on the OpenRouter
reasoning route. OpenCode always shows a thinking state here; Pulse now
emits a model_processing workflow state at the start of every post-tool
turn ('Working on the response with the gathered results.'), which
model_thinking upgrades when reasoning deltas actually arrive.
Pinned by a new interaction-quality corpus scenario.
The unified-resource adapter flattened models.ZFSPool to four scalars
(state + R/W/C totals), so scan status and the per-device report never
reached the frontend. The storage detail's ZFS card, the usage bar's
scrub/resilver overlay and ZFS tooltip, and ZFS-device-to-disk matching
were all dead code on the unified path: a degraded pool rendered as a
one-word State cell with no way to see which device failed or that a
resilver was running. Found by the v5-parity audit of the pools table
(same pipeline disease as #1471 on the disks table).
- Add StorageMeta.ZFSPool and populate it in resourceFromStorage
(normalized copy, not aliased).
- Serialize it in monitorStoragePlatformData so the /api/state path
carries it alongside the existing flat scalars.
- Map it through normalizeStorageMeta and the storage record adapter
(meta first, flat platformData fallback).
- Render the per-device report (name, vdev type, state, R/W/C errors,
message) in the detail's ZFS Pool card - the v6 home for what v5
showed via the health-map dots and the degraded-pool warning row.
- Pin the end-to-end path in code_standards_test and update the
unified-resources, monitoring, storage-recovery, and
frontend-primitives contracts.
Verified live in mock mode: degraded pool now shows the resilver scan,
pool error totals, per-device states, and the bar pulse overlay.
Session titles were the truncated first prompt, so the picker filled
with 'Reply with exactly: OK' and five identical 'hows esphome' rows.
OpenCode background-generates a short title after the first exchange;
Pulse now does the equivalent: once the done event has reached the
client, the chat runtime asks the configured chat model for a <=50-char
title from the first user message and reply, then renames the session.
- Runs only on the first exchange; any customized title is left alone,
with a re-check before persisting to lose races against a concurrent
rename. Failure or timeout keeps the placeholder.
- Title requests pass the model-boundary sanitizer like normal turns
and record usage as assistant_session_title.
- Synchronous after done (no user-visible latency, no goroutine racing
service teardown), bounded at 20s for slow reasoning routes.
Verified by tests end-to-end (hook through Rename); live verification
on a real provider route pending while the dev backend sits in mock
assistant mode, which intercepts turns before the agentic path.
Nodes that set up SSH temperature monitoring before v6.0.0-rc.6 have an
authorized_keys entry locked to `sensors -j`, so the pulse-sensors wrapper
never runs and SMART (SATA/SAS) disk temperatures silently never arrive;
only NVMe shows via kernel hwmon (what the #1471 reporters hit). The server
already detected the legacy payload but only logged it.
- models.Temperature gains LegacySensorsFormat, set by parseSensorsJSON
when the payload is raw sensors -j, and preserved across the host-agent
merge; it rides the existing temperatureDetails projection on unified
node resources.
- New PlatformOutdatedSensorSetupNotice on the Proxmox page, data-gated:
collection succeeded AND payload was legacy AND the node has SATA/SAS
disks without temperatures, so nodes whose disk temps arrive via a host
agent stay clean. Copy points at re-running the node setup script, which
replaces the locked key entry and installs the wrapper.
- monitoring.md contract extended with the legacy-format marker behavior.
The liveness handler called SchedulerHealth(), which refreshes provider
instance caches under the monitor's write lock and snapshots queues and
breakers on every probe — it only needed the dead-letter count. Add
Monitor.DeadLetterCount() (queue-local mutex only) and use it.
The hot-dev watchdog killed the backend after 2 missed 3s probes; on a
dev box running race-detector test suites and builds that is routine
CPU starvation, and each kill destroyed any in-flight Assistant chat
stream (observed repeatedly during the chat-feel audit). Default the
miss threshold to 4 (~20s) — genuine wedges still die quickly.
GET /api/ai/sessions fully re-parsed every session transcript on every
call, roughly three times over (handoff refresh List, per-session action
loads, final List). On a long-lived install the endpoint took tens of
seconds, pegged the CPU, and the dev watchdog kill-looped the backend.
- SessionStore.List() now serves summaries from a per-file cache
validated by (modTime, size); only changed files are re-read, with a
raw-skip decode that never deep-parses message payloads.
- The cache persists as .sessions_index.json next to the session files,
so the first list after a restart is index-speed too (18ms on a 662MB
legacy directory vs 23s before).
- writeSession/Delete keep cache and index coherent; external edits
self-heal via the (modTime, size) check.
- ListSessions refreshes handoff action status only for the 20 newest
sessions that carry actions, matching what the recent-sessions UI can
surface; the send path already refreshes the active session on load.
- ai-runtime contract updated: listing-cost clause and bounded-refresh
clause replace the unbounded refresh promise.
A full local go test run on 2026-06-11 failed pkg/metrics while vite
builds were saturating the machine. Reproduced under synthetic 8-core
load: only the TestSLO_* wall-clock p95 assertions failed, never the
functional tests. A latency budget measured on a shared dev machine
cannot distinguish host contention from a code regression; load storms
inflated medians up to ~4x with no code change, so no absolute
threshold separates the two.
Route the latency SLO assertions (pkg/metrics, internal/api,
internal/monitoring) and the load/stress perf assertions (internal/api)
through helpers that keep strict enforcement on GitHub Actions runners
(controlled environment, existing hosted-runner envelopes unchanged)
and skip locally on overrun, printing the full p50/p95/p99
distribution in the skip message. A local pass still means the budget
was genuinely met. Error-response and correctness checks remain hard
failures. CI -race runs are unaffected: these tests already skip under
the race detector.
Verified: pkg/metrics, internal/api, and internal/monitoring all pass
with 8 CPU burners saturating the host, the scenario that previously
turned TestSLO_RollupTierBatchedFleet and four other budget tests red.
TestLogHandlers_HandleStreamLogs_SendsEvents raced the handler
goroutine against a broadcaster write and a 10ms sleep. The global
broadcaster is shared with every concurrently running test and drops
messages to full subscriber channels, so under CI load the test's
line could be dropped and the body stayed empty; it never failed
locally because the flood only exists during a full package run on a
slow runner (first CI execution of the Go steps in ~100 runs exposed
it).
Write the line into broadcaster history first and run the handler
synchronously with a pre-cancelled context: Subscribe replays history
before the live select loop, so subscription, SSE formatting, and
context cancellation are all exercised with no goroutine, no sleep,
and no dependence on concurrent traffic. Live-channel delivery stays
covered by the broadcaster's own package tests.
Two failures surfaced once CI reached the Go steps for the first time
in ~100 runs:
- TestParityStorageFields predates models.Storage.LastSeen (e2a036ce2,
zero = never seen, no fabricated stamps), so the fixture ingested as
never-seen and LastSeen() was honestly zero. Stamp the fixture like
the real pollers do and assert exact round-trip, matching the docker
host parity test.
- internal/api alone takes ~10m under -race on a fast machine, so the
10m per-binary budget set when the package was smaller times out on
CI runners. Raise to 25m. The package has no sleep-debt (all test
sleeps are milliseconds); it is legitimately heavy.
Mock fixtures never set LastSeen on PVE nodes, VMs, LXC containers, or
storage. The registry used to paper over that by replacing zero sightings
with ingest time; since 53faa4e46 preserves zero ("never seen") and stamps
those sources "unknown", mock mode rendered its whole PVE estate with dash
last-seen and unknown source freshness.
Stamp sightings in updateFixtureStateMetricsAt, which runs at generation
and on every refresh tick, before the RandomMetrics gate so static-metrics
fixtures stay fresh too. Online nodes and everything they host get the
refresh time (a poll delivers its full inventory, stopped guests
included); anything on an offline node keeps its old stamp, with zero
backdated ten minutes so the UI shows a stale sighting rather than
"never".
workloadSummaryStatusIsRunning treated every status except StatusOnline as
not running, so a guest at StatusWarning landed in the Stopped bucket.
Warning is an attention state on a running workload: statusFromGuest maps
degraded guest states to it (power-off maps to StatusOffline), and since
53faa4e46 the registry stale pass degrades online resources with stale
source sightings to it as well. That stale pass surfaced the bug as
provider-backed (VMware) VMs counting Total:1 Running:0 Stopped:1 in
TestContract_WorkloadChartsUseCanonicalWorkloadIDsForProviderBackedVMs and
TestHandleWorkloadsSummaryCharts_UsesCanonicalWorkloadIDsForVMwareVMs,
which seeded sightings older than the VMware stale threshold. Both pass
again with warning counted as running.
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.
- models.Storage gains LastSeen (omitzero), stamped where entries are
built: the PVE storage poll (poll start time, including synthesized
cluster-shared entries; preserved entries for unpolled nodes keep their
old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
pool projection (cluster LastUpdated), and the mock generator (offline
mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
wholesale with each host report, so the host report timestamp is the
container sighting. This matches every other docker sub-resource adapter
(services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
type pbs) to SourcePBS, parented to the PBS instance. Keying them
SourceProxmox would judge their freshness against the 60s Proxmox stale
threshold while PBS polls every 60s by default, flapping healthy
datastores stale between polls; SourcePBS carries the cadence-matched
120s threshold. PVE-reported pbs-typed storage.cfg backends stay
SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
reports the honest sighting; mock refresh re-stamps available storage on
each simulated poll.
The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
Ingest stamped every resource's SourceStatus "online" and replaced a zero
LastSeen with time.Now(), so a source that never delivered a resource
(synthesized offline PVE placeholders, never-polled instances) reported a
fresh sighting and a healthy source forever: markStaleLocked skips zero
LastSeen entries, so the stamp was permanently exempt from stale-marking.
That fabricated freshness fed monitor previous-state reads (the
resurrection loop mitigated in 8372a22c5), the frontend lastSeen display,
the monitored-systems ledger, and AI/agent context.
- ingest/mergeInto/IngestResources/presentation coalesce: stamp per-source
delivery status from the actual sighting ("unknown" when the source has
never delivered, "online" otherwise) and preserve zero LastSeen instead
of substituting ingest time.
- replaceRegistry: run MarkStale over the fully assembled registry. The
IngestSnapshot stale pass ran before record sources (TrueNAS, VMware)
were ingested, leaving them stamped "online" at any age.
- monitorLastSeenUnix: report 0 for never-seen instead of time.Now(), so
the frontend renders a dash rather than "just now".
- connected infrastructure: clamp zero LastSeen to 0 ms instead of the
negative UnixMilli of the zero time.
preserveOrExpireNodes keeps requiring a real online sighting for grace;
the registry no longer manufactures the timestamps that made that guard
necessary, so the 8372a22c5 failure mode is impossible by construction.
A configured PVE instance that was already down when Pulse started had
no node state, so the unreachable-poll path returned without writing
anything and the instance was invisible on the Proxmox overview until
its first successful poll. Synthesize offline node entries from the
instance config in that case: cluster configs get one row per
discovered member endpoint with the same node ID a live poll would
assign, standalone instances get a single row named after the
configured instance. The next successful poll replaces them wholesale.
Also stop the offline grace policy from resurrecting such placeholders.
Previous node state round-trips through the unified registry, which
stamps zero LastSeen values with the ingest time, so a never-seen
placeholder came back "recently seen" every cycle and flipped to a
permanent online/degraded state. Grace now requires a real online
sighting: a nodeLastOnline entry from this process, or a previous
snapshot that itself reports the node online with a recent poll
timestamp.
Verified live on the dev stack: a synthetic instance pointing at an
unreachable TEST-NET address appears as an offline node row on the
Proxmox overview immediately after a fresh-state backend start and
stays offline past the grace and stale windows.
Fixes#1433
Two silent-downgrade paths in the v5→v6 migration are now visible and
self-healing:
License load/decrypt failure (was: one log line, no UI state): when
license.enc exists but cannot be read, getTenantComponents now persists
a terminal commercial_migration state with reason
persisted_license_unreadable, so the licence panel and global banner
tell the customer to re-enter their v5 key instead of leaving them to
discover missing Pro features.
Failed startup exchange (was: once per process, Community until manual
restart): exchange failures classified as pending — license-server
blips, DNS failures, rate limits — now schedule a background retry loop
with backoff (30s → 30m cap) that re-attempts the exchange until it
succeeds, hits a terminal classification, or the org's service activates
through another path. The loop stops cleanly on manual activation and
StopAllBackgroundLoops.
Registers the new files in the subsystem registry (cloud-paid
commercial-migration verification policy + shared ownership) and pins
both behaviors in the cloud-paid and api-contracts contracts.
Canonical IDs for merged-source hosts (PVE node + pulse-agent) were minted
from whichever identity keys the creating record happened to carry: the
agent record knows the machine ID, the Proxmox node record only knows
cluster+hostname. The registry rebuilds from scratch every tick, so a boot
window where the agent had not checked in yet minted a cluster-keyed ID
(agent-7a62... for delly) while steady state minted a machine-keyed one
(agent-bdd4...). Every restart re-ran the race, fragmenting the
resource_changes journal into per-boot eras (9.4k vs 6.1k rows for the
same host) and silently truncating report availability and UI timelines.
Fix, in the layer that owns identity:
- Persist identity pins (canonical_id <-> machine_id/dmi/cluster/hostname)
in the previously schema-only resource_identities table, written by the
store-backed registry after monitor-adapter rebuilds, diff-aware so
steady-state ticks cost no writes.
- Complete weak incoming identities from the pins before matching and ID
derivation, so a node-only boot window derives the same machine-keyed
canonical ID as steady state. Derivation itself is unchanged; ephemeral
nil-store registries behave exactly as before.
- Expand change-journal reads (Get/Count families, SQLite and memory) to
the full era set recomputed from the pinned identity keys, healing
historical journals at query time with no row migration. Reads keyed by
a stale era ID resolve to the same merged timeline.
Regression tests cover both ingest orders, restart simulation via the
monitor adapter, era ID derivation, and era-merged journal reads on both
store implementations. Contracts updated: unified-resources obligation 25
(durable identity pins), monitoring obligation 10 (adapter rebuild
persistence).
v5 documented PORT as a legacy alias for FRONTEND_PORT. v6 dropped the
fallback silently, so an install configured with PORT=8080 upgrades to
v6 and comes up on 7655 with no explanation, unreachable behind its
existing port mapping or reverse proxy.
Honor PORT again at lowest precedence (FRONTEND_PORT wins when both are
set) and log a deprecation warning at startup, mirroring the existing
legacy DISABLE_AUTH handling. Document PORT as deprecated in
CONFIGURATION.md and note the compatibility behavior in the v6
changelog.
Two regressions surfaced by generating a real-mode agent report after a
backend restart:
- Report subject lookups used Monitor.GetUnifiedResources (the raw
resource store), but the raw store's canonical IDs depend on per-boot
ingest order for merged-source hosts: after a restart the same host
resolved to a different agent-<hash> than the one the UI and
/api/state advertise, so reports lost the resource name, availability,
and metrics translation entirely. Subject enrichment now reads
Monitor.UnifiedResourceSnapshot and MetricsTargetForResource resolves
through GetUnifiedReadStateOrSnapshot first (raw store as fallback) -
the same re-ingested registry every other read surface uses.
- The performance summary card grid positions cards absolutely and never
paginated: an agent host reporting 8+ metric families walked off the
page bottom, fought fpdf's auto page break, and scattered one orphan
element per page (a 7-day delly report rendered 18 pages, ten of them
near-blank). The grid now starts a new page before a row that will not
fit; the same report renders 9 pages with intact cards.
Verified live: real-mode delly report shows name, 288 data points,
availability, charts, and correctly paginated cards. The underlying
canonical-ID instability (raw store vs re-ingested view, and the
resource_changes journal fragmenting across boot eras) is a separate
root issue tracked for its own fix.
Mock fixture IDs churned across backend restarts, breaking resource
identity continuity in mock mode: k8s pod and deployment names picked
their namespace and prefix with rand.Intn per boot, the ceph FSID was
pure rand.Int63n, and generic agent hosts randomized both platform
profile (which feeds the host ID) and hostname. Every restart re-keyed
those unified resources and re-seeded ~31k orphan metric rows that
lingered for the full 90d mock retention.
Derive all of them from mockStableChoice/mockStableDecimalString over
stable inputs (cluster ID, item ordinal, instance name) instead,
matching the generator's existing stable-ID idiom and its documented
contract. Add a regression test that builds the fixture graph twice
and requires every identity set to match.
Verified live: two mock-mode boots now share all 185 distinct metric
resource IDs and the second boot's backfill seeds 0 rows (was ~31k).
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.
Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.
Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).
Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
Real-mode agent host metrics are written to the metrics store keyed by
host.ID (monitor_agents.go writes "agent"/host.ID, pinned by a canonical
guardrail), and ingestHost registers that same ID as the SourceAgent
mapping. But BuildMetricsTarget preferred the Proxmox/VMware source ID
for agent resources, so a host that is also a Proxmox node (delly,
minipc, pi) advertised an "agent"-typed metrics target (e.g.
"homelab-delly") that nothing ever writes. Every reader that trusts the
registry target queried zero rows: performance reports resolved via
MetricsResourceID rendered "Data Points: 0", and the resource drawer
history endpoint fell back to in-memory node history (losing
temperature and disk I/O series the agent records).
Flip the priority: the agent source wins for agent resources, platform
sources (Proxmox, VMware, TrueNAS, Docker) remain fallbacks for hosts
without a reporting agent. This matches the write path exactly - the
bulk charts feed (hostAgentChartRequest) already preferred the agent ID,
and pure-agent hosts already resolved to it, so merged hosts simply
become consistent. No history is orphaned: "agent" store rows were
always keyed by host.ID, and node-poller rows ("node"/<node.ID>) were
never reachable through the "agent"-typed target.
Verified live: all five real agents now resolve metricsTarget to their
agent UUID, /api/metrics-store/history returns store rows for the new
target, and the delly performance report renders 312 data points where
it previously rendered zero.
Regression tests: BuildMetricsTarget prefers the agent source for
merged-source resources, and a registry ingest test asserts the merged
node+agent resource resolves its target to host.ID - the agent store
write key.
Performance reports answered 'what were the averages' but never 'was my
infrastructure up' - the question a managed-service client reads a
monthly report for. Reports now carry an Availability summary derived
from the recorded resource change timeline (state_transition entries
keyed by the canonical unified ID):
- uptime percent over the observed portion of the window, outage count,
total downtime, and longest outage, rendered in the executive summary
with an explicit semantics note; fleet summaries gain a per-resource
Uptime column and CSV exports gain availability header lines
- absent/unknown spans are unobserved time: excluded from the uptime
math entirely and disclosed as coverage, never counted as downtime.
The journal records a registry absence for every monitor restart, so
treating gaps as outages would invent fleet-wide downtime every time
the operator restarts Pulse
- warning states count as up (the resource is reachable and serving);
the uptime label clamps rounding so any real downtime can never
display as a clean 100%
- resources with no timeline render no availability section at all
rather than a fabricated number
Verified live against a real 7-day window: uptime/outage/downtime
figures reconcile with the raw resource_changes journal.
Performance reports were structurally disconnected from the v6 ID
space: the UI (and any API caller working from /api/state) addresses
resources by canonical unified ID, while the metrics store is keyed by
each platform's native source ID (the resource's metricsTarget). The
engine queried the store with the unified ID verbatim, so every report
rendered 'Data Points: 0' regardless of how much history existed, and
covers showed raw hash IDs a report reader cannot map to a machine.
- MetricReportRequest gains MetricsResourceID: handlers resolve the
unified ID through the tenant monitor's resource store (new
Monitor.MetricsTargetForResource accessor; the registry computes
targets on demand, they are not persisted on snapshot structs) and
the engine uses it for store queries only. Recovery points and
Patrol findings stay keyed by the unified ID.
- Legacy snapshot models and their alerts are keyed by the metrics
target ID, so enrichment now matches either ID space and resource
names/status resolve again on covers, headers, and fleet rows.
- Fleet summaries mirror the single-report guard: zero data points
across the fleet renders a muted NO DATA card instead of a green
HEALTHY 'All systems operating normally' - false reassurance is the
worst failure mode for a client-facing stability report.
- Em dashes in PDF-bound literals become hyphens; fpdf core fonts are
cp1252 and rendered them as mojibake.
Cluster endpoints were discovered once at add time and never re-read:
detectClusterMembership returned immediately for instances already marked
IsCluster, so when a cluster moved subnets Pulse kept dialing the dead
per-node addresses for failover and kept displaying them on Settings ->
Infrastructure, even though monitoring still worked through the main host
fallback. The manual /refresh-cluster endpoint exists but is not wired to
any UI control, so nothing ever corrected the stored addresses.
The 5-minute cluster re-check now also covers cluster instances: it
re-reads /cluster/status through the configured main host, and when a
node's address or the node set actually changed (volatile fields like
Online/LastSeen are ignored) it updates the stored endpoints, persists the
config, and rebuilds the failover client so polling moves to the new
addresses. User-managed fields (IP overrides, guest URLs, fingerprints)
are preserved by the existing discovery helpers, and fields the cluster
status API omits are inherited from the stored endpoint rather than
erased.
Refs #1493
recordTaskResult reset the failure counters on a successful poll but left
LastErrorAt/LastErrorMessage/LastErrorCategory in place forever. The
connections aggregator surfaces that field as a live error banner on
Settings -> Infrastructure and derives the Unauthorized state from it, so
one transient cluster outage (or startup blip) pinned a red "no healthy
nodes available" banner onto a connection that polls fine, and a single
past 401 could pin a healthy connection to Unauthorized.
LastError now means "current outstanding failure since the last success":
recorded on failure, cleared on success. Connections that are genuinely
failing (no successful poll) keep their banner.
Refs #1493
pollPVEInstance returned early on a GetNodes error, so a host that
stopped answering entirely (shutdown, network loss) kept its last
successful snapshot in state forever: node row frozen at online with
stale uptime. The existing 60s offline grace policy only ran for
cluster-reported peer-down and empty-poll cycles, never for a whole
instance going dark.
Route the poll error path through the same grace policy: nodes of an
unreachable instance stay online (connection degraded) within the
grace window and flip to offline with cleared uptime and CPU once it
lapses, matching the cluster-reported node-down behavior.
Verified live: registered a synthetic PVE instance, killed it, and the
node flips to offline one poll cycle after the grace period.
Fixes#1441
On the #1483 reporter's Proxmox node a SATA SSD (INTEL SSDSC2BW240A4) never
appeared in Pulse even though smartctl reads it fine, because the agent's
discovery and probing both had single points of failure:
- Discovery trusted smartctl --scan-open exclusively whenever it returned at
least one target. The scan silently omits any device it fails to open at
scan time (the failure is only a #-comment in its output), so one bad open
hid a real disk while its neighbours were listed.
- A scan-provided -d type got exactly one probe attempt on Linux. A type whose
full query (-i -A -H) fails or returns nothing dropped the disk silently.
- A probe yielding no usable SMART data dropped the disk entirely instead of
reporting the identity the kernel can prove.
Fixes, all in the agent collector:
- Union discovery: the kernel block device list (/sys/block, lsblk fallback)
is the ground truth for which disks exist; smartctl --scan-open only
contributes device-type hints. Any physical disk the scan misses gets an
untyped probe target.
- Untyped retry: a typed Linux probe that errors, yields no usable data, or
fails to open (exit bit 2, distinct from -n standby,3's exit 3) retries with
smartctl auto-detection before giving up. Multiplexed controller members
(megaraid, cciss, ...) are exempt since dropping -d would probe the array.
- Identity-only fallback: when every probe fails but the disk exists in
/sys/block with nonzero capacity, report device/model/serial/size with
health UNKNOWN instead of hiding it. No SMART data is fabricated;
multiplexed array paths and zero-capacity media are excluded.
- Exclusion follow-through: --disk-exclude now also matches the canonical
post-refine name (nvme0n1), not just the scan label (nvme0).
Regression tests use smartctl/lsblk fixtures captured from PVE 9.1.9 hosts
and the issue report. Verified live on two PVE nodes: NVMe keyed by
namespace with byte-exact pvesh sizes, SATA disk reported with full identity.
Completes the #1483 fix started in bd20069c6 (namespace devpath, authoritative
sizes, merge downgrade guard). Closes#1483.
The variable signs hosted entitlement leases; the trial-activation name is
left over from the retired trial era and reads as trial machinery to an
operator generating their licensing root key. The provider MSP bundle has
no installed base yet, so the canonical rename is free today and frozen
the moment the first design partner installs.
- Control plane reads CP_ENTITLEMENT_SIGNING_PRIVATE_KEY first and falls
back to CP_TRIAL_ACTIVATION_PRIVATE_KEY, so existing Pulse-hosted cloud
deployments (deploy/cloud, hibernated snapshot) keep working unchanged.
- deploy/provider-msp (.env.example, compose, setup.sh), MSP.md, and the
install-test pins use the canonical name; error messages name it too.
- deploy/cloud intentionally keeps the legacy name: that stack historically
signed hosted trial activations, and its snapshot predates the rename.
rcourtman