Nodes that set up SSH temperature monitoring before v6.0.0-rc.6 have an
authorized_keys entry locked to `sensors -j`, so the pulse-sensors wrapper
never runs and SMART (SATA/SAS) disk temperatures silently never arrive;
only NVMe shows via kernel hwmon (what the #1471 reporters hit). The server
already detected the legacy payload but only logged it.
- models.Temperature gains LegacySensorsFormat, set by parseSensorsJSON
when the payload is raw sensors -j, and preserved across the host-agent
merge; it rides the existing temperatureDetails projection on unified
node resources.
- New PlatformOutdatedSensorSetupNotice on the Proxmox page, data-gated:
collection succeeded AND payload was legacy AND the node has SATA/SAS
disks without temperatures, so nodes whose disk temps arrive via a host
agent stay clean. Copy points at re-running the node setup script, which
replaces the locked key entry and installs the wrapper.
- monitoring.md contract extended with the legacy-format marker behavior.
The v6 physical disks table regressed against its v5 ancestor: device
path was demoted to a tooltip (identical models became indistinguishable),
SSD wearout vanished from the surface entirely despite flowing all the way
into the presentation layer, the Belongs column rendered empty for every
disk without a ZFS pool mapping, and a Source badge spent a column saying
PVE on every row. Flagged from the side-by-side screenshots in #1471.
- Add a Device column (mono devPath, md+).
- Add a Life column rendering wearout with v5-style thresholds (md+).
- Fall back Belongs to Proxmox's usage string (ZFS / ext4 / BIOS boot)
when no pool mapping exists; searchable via the existing haystack.
- Demote the Source badge to the row expansion header next to the
serial; drop the table column.
- Render missing temperatures as muted dashes instead of green ones
(getTemperatureTextClass(0) returned the healthy tone).
- Tighten rows from 38px to 32px to match the platform table standard.
A preview/tooling Vite instance on a non-default port gets its
WebSocket upgrades rejected by the backend's dev origin allowlist
(http://localhost:5173,7655). PULSE_DEV_PROXY_ORIGIN rewrites the
proxied Origin header to an allowed one; unset, nothing changes.
The dedup landed in isRenderableStreamEvent, but that predicate only
feeds the has-any-events gate; the model_switch Match rendered on its
own condition, so live turns still showed the duplicate row. Extract
shouldRenderModelRouteEvent, use it at the Match itself, and compare
display labels instead of raw ids — the done event rewrites
message.model to a provider-reported id that can differ in raw form
while reading identically to the user.
The liveness handler called SchedulerHealth(), which refreshes provider
instance caches under the monitor's write lock and snapshots queues and
breakers on every probe — it only needed the dead-letter count. Add
Monitor.DeadLetterCount() (queue-local mutex only) and use it.
The hot-dev watchdog killed the backend after 2 missed 3s probes; on a
dev box running race-detector test suites and builds that is routine
CPU starvation, and each kill destroyed any in-flight Assistant chat
stream (observed repeatedly during the chat-feel audit). Default the
miss threshold to 4 (~20s) — genuine wedges still die quickly.
Every assistant turn rendered a 'Using <route>' status row that repeats
the route badge in the turn header, so a restored session read as a
column of duplicate route chips. Keep the row only when it adds
information: the header has no route attribution, the selected route
differs from the turn's final model, or a later mid-turn switch makes
the starting route part of the story.
The completed tool row measured from tool_start, which providers emit
when the model begins streaming tool arguments. On slow reasoning
routes that ran minutes ahead of execution: the live audit showed
'QUERY completed 2m 8s' for a tool the backend executed in under a
second, billing the model's thinking time to Pulse's tools.
Track runningAt (first 'running' phase, which the agentic loop emits at
actual execution) on pending tools, carry it onto the completed stream
event, and prefer it for the duration stamp. The live pending row keeps
counting from arg-streaming start because its phase label ('Receiving
tool input') correctly names what is being timed.
The retest-comment workflow fired on 'reopened' as well as 'opened', and
canPostRetestComment gated on the issue author's association, not the
reopen actor's. Since non-collaborator reporters cannot reopen
maintainer-closed issues, a reopened event is in practice always a
deliberate maintainer decision, and the bot would post version-retest
boilerplate that contradicts the maintainer's own comment while planting
the needs-retest auto-close marker on an issue they chose to keep open
(seen on #1471).
Restrict the trigger and the guard to 'opened' and pin the reopened skip
with a test.
The availability map computes a fall-through reason string for every
command whether or not it is disabled, and commandWithAvailability
copied it through unconditionally. Enabled menu items then displayed
false statements: '/fork ... Forking is already running.', '/copy ...
Requires transcript content.' on a session with a full transcript.
Carry the reason only when the command is actually disabled.
Live audit against the OpenCode bar found two ways the Assistant could
pin a permanent 'Generating response' footer with nothing in flight:
- loadSession left isStreaming undefined on restored messages, which the
activity dock reads as 'still streaming' (isStreaming !== false). Any
restored session whose transcript ends in an assistant message showed
a phantom active turn; new prompts then queued behind it forever and
stop could not recover. Restored messages are now always settled.
- A stream that closed without a terminal event (backend restart, proxy
drop) synthesized a silent 'done', pretending the turn finished; and
the 300s inactivity ceiling meant a dead connection counted up under
'Generating response' for five minutes. The close now surfaces as a
recoverable interruption error, and inactivity trips at 30s, backed
by the backend's 5s SSE heartbeats.
GET /api/ai/sessions fully re-parsed every session transcript on every
call, roughly three times over (handoff refresh List, per-session action
loads, final List). On a long-lived install the endpoint took tens of
seconds, pegged the CPU, and the dev watchdog kill-looped the backend.
- SessionStore.List() now serves summaries from a per-file cache
validated by (modTime, size); only changed files are re-read, with a
raw-skip decode that never deep-parses message payloads.
- The cache persists as .sessions_index.json next to the session files,
so the first list after a restart is index-speed too (18ms on a 662MB
legacy directory vs 23s before).
- writeSession/Delete keep cache and index coherent; external edits
self-heal via the (modTime, size) check.
- ListSessions refreshes handoff action status only for the 20 newest
sessions that carry actions, matching what the recent-sessions UI can
surface; the send path already refreshes the active session on load.
- ai-runtime contract updated: listing-cost clause and bounded-refresh
clause replace the unbounded refresh promise.
A full local go test run on 2026-06-11 failed pkg/metrics while vite
builds were saturating the machine. Reproduced under synthetic 8-core
load: only the TestSLO_* wall-clock p95 assertions failed, never the
functional tests. A latency budget measured on a shared dev machine
cannot distinguish host contention from a code regression; load storms
inflated medians up to ~4x with no code change, so no absolute
threshold separates the two.
Route the latency SLO assertions (pkg/metrics, internal/api,
internal/monitoring) and the load/stress perf assertions (internal/api)
through helpers that keep strict enforcement on GitHub Actions runners
(controlled environment, existing hosted-runner envelopes unchanged)
and skip locally on overrun, printing the full p50/p95/p99
distribution in the skip message. A local pass still means the budget
was genuinely met. Error-response and correctness checks remain hard
failures. CI -race runs are unaffected: these tests already skip under
the race detector.
Verified: pkg/metrics, internal/api, and internal/monitoring all pass
with 8 CPU burners saturating the host, the scenario that previously
turned TestSLO_RollupTierBatchedFleet and four other budget tests red.
TestLogHandlers_HandleStreamLogs_SendsEvents raced the handler
goroutine against a broadcaster write and a 10ms sleep. The global
broadcaster is shared with every concurrently running test and drops
messages to full subscriber channels, so under CI load the test's
line could be dropped and the body stayed empty; it never failed
locally because the flood only exists during a full package run on a
slow runner (first CI execution of the Go steps in ~100 runs exposed
it).
Write the line into broadcaster history first and run the handler
synchronously with a pre-cancelled context: Subscribe replays history
before the live select loop, so subscription, SSE formatting, and
context cancellation are all exercised with no goroutine, no sleep,
and no dependence on concurrent traffic. Live-channel delivery stays
covered by the broadcaster's own package tests.
Two failures surfaced once CI reached the Go steps for the first time
in ~100 runs:
- TestParityStorageFields predates models.Storage.LastSeen (e2a036ce2,
zero = never seen, no fabricated stamps), so the fixture ingested as
never-seen and LastSeen() was honestly zero. Stamp the fixture like
the real pollers do and assert exact round-trip, matching the docker
host parity test.
- internal/api alone takes ~10m under -race on a fast machine, so the
10m per-binary budget set when the package was smaller times out on
CI runners. Raise to 25m. The package has no sleep-debt (all test
sleeps are milliseconds); it is legitimately heavy.
The bundle-size checker groups dist chunks by basename, so the lazy
AI Chat chunk (src/components/AI/Chat/index.tsx) and the SetupWizard
chunk (index.ts) were summed into the entry's 'index' budget line.
The budget breach that surfaced once the unit tests were fixed was
that conflation, not an entry regression: since the Jun 1 baseline
(d2639fada) the real entry grew 81.4 -> 90.7 kB gzip (+11.5%, within
the 15% threshold) while the lazy Chat chunk grew 22.2 -> 61.8 kB
with the deliberate Assistant UX modernization.
Name dynamic-entry chunks after their facade directory (Chat,
SetupWizard) so 'index' means only the real entry, and refresh the
baseline to record current per-chunk truth, including Docker's
deliberate platform-page growth (15.9 -> 18.8 kB).
Mock fixtures never set LastSeen on PVE nodes, VMs, LXC containers, or
storage. The registry used to paper over that by replacing zero sightings
with ingest time; since 53faa4e46 preserves zero ("never seen") and stamps
those sources "unknown", mock mode rendered its whole PVE estate with dash
last-seen and unknown source freshness.
Stamp sightings in updateFixtureStateMetricsAt, which runs at generation
and on every refresh tick, before the RandomMetrics gate so static-metrics
fixtures stay fresh too. Online nodes and everything they host get the
refresh time (a poll delivers its full inventory, stopped guests
included); anything on an offline node keeps its old stamp, with zero
backdated ten minutes so the UI shows a stale sighting rather than
"never".
workloadSummaryStatusIsRunning treated every status except StatusOnline as
not running, so a guest at StatusWarning landed in the Stopped bucket.
Warning is an attention state on a running workload: statusFromGuest maps
degraded guest states to it (power-off maps to StatusOffline), and since
53faa4e46 the registry stale pass degrades online resources with stale
source sightings to it as well. That stale pass surfaced the bug as
provider-backed (VMware) VMs counting Total:1 Running:0 Stopped:1 in
TestContract_WorkloadChartsUseCanonicalWorkloadIDsForProviderBackedVMs and
TestHandleWorkloadsSummaryCharts_UsesCanonicalWorkloadIDsForVMwareVMs,
which seeded sightings older than the VMware stale threshold. Both pass
again with warning counted as running.
26c7ca910 added the Pulse for MSPs entry to docs/README.md and the
deprecated PORT alias row landed in docs/CONFIGURATION.md without
re-syncing the shipped copies under frontend-modern/public/docs,
which the docsLinks vitest contract enforces byte-for-byte.
Four Assistant refactors (3f8525a7c, 3c3ca42c2, 29ebaf788, 1fc36a790)
moved code to new owner files without updating the source-contract
tests, and the hooks never run vitest, so CI has been red since Jun 4:
- formatIdentifierLabel moved from MessageItem.tsx to activeTurnStatus.ts
and from ToolExecutionBlock.tsx to toolPresentation.ts; the boundaries
contract now asserts on the new owners (ad-hoc-regex guards on the
components remain).
- The model-selector empty state moved to shared AIModelPicker.tsx;
the contract follows it and keeps the no-hardcoded-copy guard on both.
- useGuestDrawerState's discovery query was prettier-wrapped across
lines; the workloads contract now uses a whitespace-tolerant toMatch.
- ModelSelector's empty-name fallback deliberately renders a humanized
'Anthropic: Claude Test' route label since the OpenCode alignment;
the test expectation now matches the shipped behavior.
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.
- models.Storage gains LastSeen (omitzero), stamped where entries are
built: the PVE storage poll (poll start time, including synthesized
cluster-shared entries; preserved entries for unpolled nodes keep their
old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
pool projection (cluster LastUpdated), and the mock generator (offline
mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
wholesale with each host report, so the host report timestamp is the
container sighting. This matches every other docker sub-resource adapter
(services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
type pbs) to SourcePBS, parented to the PBS instance. Keying them
SourceProxmox would judge their freshness against the 60s Proxmox stale
threshold while PBS polls every 60s by default, flapping healthy
datastores stale between polls; SourcePBS carries the cadence-matched
120s threshold. PVE-reported pbs-typed storage.cfg backends stay
SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
reports the honest sighting; mock refresh re-stamps available storage on
each simulated poll.
The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
Ingest stamped every resource's SourceStatus "online" and replaced a zero
LastSeen with time.Now(), so a source that never delivered a resource
(synthesized offline PVE placeholders, never-polled instances) reported a
fresh sighting and a healthy source forever: markStaleLocked skips zero
LastSeen entries, so the stamp was permanently exempt from stale-marking.
That fabricated freshness fed monitor previous-state reads (the
resurrection loop mitigated in 8372a22c5), the frontend lastSeen display,
the monitored-systems ledger, and AI/agent context.
- ingest/mergeInto/IngestResources/presentation coalesce: stamp per-source
delivery status from the actual sighting ("unknown" when the source has
never delivered, "online" otherwise) and preserve zero LastSeen instead
of substituting ingest time.
- replaceRegistry: run MarkStale over the fully assembled registry. The
IngestSnapshot stale pass ran before record sources (TrueNAS, VMware)
were ingested, leaving them stamped "online" at any age.
- monitorLastSeenUnix: report 0 for never-seen instead of time.Now(), so
the frontend renders a dash rather than "just now".
- connected infrastructure: clamp zero LastSeen to 0 ms instead of the
negative UnixMilli of the zero time.
preserveOrExpireNodes keeps requiring a real online sighting for grace;
the registry no longer manufactures the timestamps that made that guard
necessary, so the 8372a22c5 failure mode is impossible by construction.
A configured PVE instance that was already down when Pulse started had
no node state, so the unreachable-poll path returned without writing
anything and the instance was invisible on the Proxmox overview until
its first successful poll. Synthesize offline node entries from the
instance config in that case: cluster configs get one row per
discovered member endpoint with the same node ID a live poll would
assign, standalone instances get a single row named after the
configured instance. The next successful poll replaces them wholesale.
Also stop the offline grace policy from resurrecting such placeholders.
Previous node state round-trips through the unified registry, which
stamps zero LastSeen values with the ingest time, so a never-seen
placeholder came back "recently seen" every cycle and flipped to a
permanent online/degraded state. Grace now requires a real online
sighting: a nodeLastOnline entry from this process, or a previous
snapshot that itself reports the node online with a recent poll
timestamp.
Verified live on the dev stack: a synthetic instance pointing at an
unreachable TEST-NET address appears as an offline node row on the
Proxmox overview immediately after a fresh-state backend start and
stays offline past the grace and stale windows.
Fixes#1433
The only surface explaining an unresolved v5→v6 license migration was an
inline notice buried in Settings → Billing → Plan, so a paying customer
whose migration stalled discovered it as silently missing Pro features.
Mount a global banner in the app shell (same slot as the demo and update
banners) whenever commercial posture carries a pending or failed
commercial_migration state. It reuses getCommercialMigrationNotice for
copy, links to the licence panel, dismisses per session and per state
(a pending→failed transition resurfaces it), re-checks posture every
minute while pending so the backend's background retry clears it
without a reload, and loads commercial posture itself — the app shell
only loads posture when upgrade prompts are enabled, and a migration
warning is service degradation, not an upgrade nag.
Verified with vitest (component + presentation suites) and live against
the dev instance with an undecryptable license.enc fixture.
Two silent-downgrade paths in the v5→v6 migration are now visible and
self-healing:
License load/decrypt failure (was: one log line, no UI state): when
license.enc exists but cannot be read, getTenantComponents now persists
a terminal commercial_migration state with reason
persisted_license_unreadable, so the licence panel and global banner
tell the customer to re-enter their v5 key instead of leaving them to
discover missing Pro features.
Failed startup exchange (was: once per process, Community until manual
restart): exchange failures classified as pending — license-server
blips, DNS failures, rate limits — now schedule a background retry loop
with backoff (30s → 30m cap) that re-attempts the exchange until it
succeeds, hits a terminal classification, or the org's service activates
through another path. The loop stops cleanly on manual activation and
StopAllBackgroundLoops.
Registers the new files in the subsystem registry (cloud-paid
commercial-migration verification policy + shared ownership) and pins
both behaviors in the cloud-paid and api-contracts contracts.
v5.0.0 derived the license.enc AES key as sha256("pulse-license-" +
material) where material was the raw /etc/machine-id file content —
trailing newline included — with hostname and a fixed dev fallback as
alternates. v6 trims machine-id before hashing and dropped both
fallbacks, so licenses activated on v5.0.0 (before .license-key shipped
in v5.0.1) could never be decrypted and the December 2025 lifetime
cohort silently booted as Community.
Collect the raw v5 key materials at construction and try them verbatim
as legacy sha256 candidates during decrypt, alongside the existing
trimmed HKDF/legacy derivations. The write path is unchanged: persistent
random key + HKDF only. Pins the cloud-paid persistence contract to the
full v5.0.x compatibility-loader material set.
Go half of the licensing-policy client work from the v6 upgrade sweep
(license findings 5 and 6); the copy and docs landed in 3a51429a5 and
the license-server side lands in pulse-pro.
The grant refresh loop treated every HTTP 401 as revocation and wiped
the licence and activation file immediately, but the server returns 401
for token-not-found and token-expired as well as revocation, so a
single spurious 401 dropped a paying customer to Community until a
successful re-exchange. Activation state is now cleared only when the
401 carries an explicit revocation code (TOKEN_REVOKED,
INSTALLATION_REVOKED, LICENSE_REVOKED); any other 401 keeps the licence
and retries, letting the grant's 72h expiry plus the 7-day grace window
govern degradation. The bump_license_version immediate-refresh path in
the revocation poller had the same any-401 wipe and gets the same
guard.
ClassifyLegacyExchangeError now maps a 409 MAX_INSTALLATIONS exchange
error to failed/exchange_installation_limit with the
free_installation_slot action instead of the retryable
pending/exchange_conflict state, since retrying can never succeed until
a slot is freed server-side. Other 409s keep the pending conflict
semantics.
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Canonical IDs for merged-source hosts (PVE node + pulse-agent) were minted
from whichever identity keys the creating record happened to carry: the
agent record knows the machine ID, the Proxmox node record only knows
cluster+hostname. The registry rebuilds from scratch every tick, so a boot
window where the agent had not checked in yet minted a cluster-keyed ID
(agent-7a62... for delly) while steady state minted a machine-keyed one
(agent-bdd4...). Every restart re-ran the race, fragmenting the
resource_changes journal into per-boot eras (9.4k vs 6.1k rows for the
same host) and silently truncating report availability and UI timelines.
Fix, in the layer that owns identity:
- Persist identity pins (canonical_id <-> machine_id/dmi/cluster/hostname)
in the previously schema-only resource_identities table, written by the
store-backed registry after monitor-adapter rebuilds, diff-aware so
steady-state ticks cost no writes.
- Complete weak incoming identities from the pins before matching and ID
derivation, so a node-only boot window derives the same machine-keyed
canonical ID as steady state. Derivation itself is unchanged; ephemeral
nil-store registries behave exactly as before.
- Expand change-journal reads (Get/Count families, SQLite and memory) to
the full era set recomputed from the pinned identity keys, healing
historical journals at query time with no row migration. Reads keyed by
a stale era ID resolve to the same merged timeline.
Regression tests cover both ingest orders, restart simulation via the
monitor adapter, era ID derivation, and era-merged journal reads on both
store implementations. Contracts updated: unified-resources obligation 25
(durable identity pins), monitoring obligation 10 (adapter rebuild
persistence).
Docs and frontend half of the licensing-policy client work from the v6
upgrade sweep (license findings 5 and 6); the Go classifier and grant
refresh fix land separately once the shared pre-commit guard unblocks.
A 409 MAX_INSTALLATIONS during v5 key migration rendered as 'another v6
activation handoff is still settling. Retry activation from this
instance.' Retrying can never succeed until a slot is freed server-side,
so that copy stranded users. The notice now says the key is already
active on its maximum number of v6 installations and points at
support@pulserelay.pro to free a slot, via the new
exchange_installation_limit reason and free_installation_slot action.
docs/UPGRADE_v6.md gains a 'Breaking Change: Paid Licensing Requires
Connectivity' section: v5 validated keys fully offline, v6 needs
periodic connectivity to license.pulserelay.pro, with roughly 10 days of
offline tolerance (72 hour grant plus 7 day grace) before paid features
drop to Community, plus air-gap guidance and the migrated-key
installation cap. docs/releases/V6_CHANGELOG.md gains the matching
breaking-change bullet.
Note: the two frontend files also carry two small additive hunks from
the parallel persisted-license-unreadable slice (a copy case and its
test, inert until its Go side lands); they share these files and are
included to avoid splitting them mid-flight.
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Two GA-blocking bugs in the auto-update install path:
The pulse-update.service unit was rendered through an unquoted heredoc
containing $${PULSE_SERVICE_NAME}, which bash expands to the installer's
PID, so the unit shipped with ExecCondition checking a service like
"24757{PULSE_SERVICE_NAME}". The condition always failed and systemd
silently skipped every scheduled run, for fresh installs and upgrades
alike. The heredoc now renders the detected service name directly:
ExecCondition=/bin/sh -c 'systemctl is-active --quiet pulse'.
Upgrades also never refreshed the updater assets: a v5 box with
auto-updates enabled keeps pulse-update.timer and autoUpdateEnabled=true,
so the update/reinstall flows never re-ran setup_auto_updates and the
v5.1-pinned /usr/local/bin/pulse-auto-update.sh stayed in place, logging
"Already running latest version" forever instead of selecting v6
releases. The asset-install half of setup_auto_updates is now a shared
install_auto_update_assets(), and a new refresh_auto_updates() rewrites
the helper script and units unconditionally whenever the timer already
exists, without touching system.json or the timer's enabled state. All
five install flows (update, reinstall, --version, --source, fresh) are
wired.
Tests now render the real unit and execute the rendered ExecCondition
against a recording systemctl stub instead of asserting source-text
fragments, pin refresh_auto_updates behavior (stale helper replaced,
system.json and enablement untouched), and pin the call-site wiring.
The deployment-installability contract records both invariants.
Note: rewritten message only — the original local commit carried a
parallel agent's licensing commit message due to a shared-index race;
the tree is byte-identical to the hook-verified original.
v5 documented PORT as a legacy alias for FRONTEND_PORT. v6 dropped the
fallback silently, so an install configured with PORT=8080 upgrades to
v6 and comes up on 7655 with no explanation, unreachable behind its
existing port mapping or reverse proxy.
Honor PORT again at lowest precedence (FRONTEND_PORT wins when both are
set) and log a deprecation warning at startup, mirroring the existing
legacy DISABLE_AUTH handling. Document PORT as deprecated in
CONFIGURATION.md and note the compatibility behavior in the v6
changelog.
Two regressions surfaced by generating a real-mode agent report after a
backend restart:
- Report subject lookups used Monitor.GetUnifiedResources (the raw
resource store), but the raw store's canonical IDs depend on per-boot
ingest order for merged-source hosts: after a restart the same host
resolved to a different agent-<hash> than the one the UI and
/api/state advertise, so reports lost the resource name, availability,
and metrics translation entirely. Subject enrichment now reads
Monitor.UnifiedResourceSnapshot and MetricsTargetForResource resolves
through GetUnifiedReadStateOrSnapshot first (raw store as fallback) -
the same re-ingested registry every other read surface uses.
- The performance summary card grid positions cards absolutely and never
paginated: an agent host reporting 8+ metric families walked off the
page bottom, fought fpdf's auto page break, and scattered one orphan
element per page (a 7-day delly report rendered 18 pages, ten of them
near-blank). The grid now starts a new page before a row that will not
fit; the same report renders 9 pages with intact cards.
Verified live: real-mode delly report shows name, 288 data points,
availability, charts, and correctly paginated cards. The underlying
canonical-ID instability (raw store vs re-ingested view, and the
resource_changes journal fragmenting across boot eras) is a separate
root issue tracked for its own fix.
Mock fixture IDs churned across backend restarts, breaking resource
identity continuity in mock mode: k8s pod and deployment names picked
their namespace and prefix with rand.Intn per boot, the ceph FSID was
pure rand.Int63n, and generic agent hosts randomized both platform
profile (which feeds the host ID) and hostname. Every restart re-keyed
those unified resources and re-seeded ~31k orphan metric rows that
lingered for the full 90d mock retention.
Derive all of them from mockStableChoice/mockStableDecimalString over
stable inputs (cluster ID, item ordinal, instance name) instead,
matching the generator's existing stable-ID idiom and its documented
contract. Add a regression test that builds the fixture graph twice
and requires every identity set to match.
Verified live: two mock-mode boots now share all 185 distinct metric
resource IDs and the second boot's backfill seeds 0 rows (was ~31k).
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.
Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.
Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).
Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
Real-mode agent host metrics are written to the metrics store keyed by
host.ID (monitor_agents.go writes "agent"/host.ID, pinned by a canonical
guardrail), and ingestHost registers that same ID as the SourceAgent
mapping. But BuildMetricsTarget preferred the Proxmox/VMware source ID
for agent resources, so a host that is also a Proxmox node (delly,
minipc, pi) advertised an "agent"-typed metrics target (e.g.
"homelab-delly") that nothing ever writes. Every reader that trusts the
registry target queried zero rows: performance reports resolved via
MetricsResourceID rendered "Data Points: 0", and the resource drawer
history endpoint fell back to in-memory node history (losing
temperature and disk I/O series the agent records).
Flip the priority: the agent source wins for agent resources, platform
sources (Proxmox, VMware, TrueNAS, Docker) remain fallbacks for hosts
without a reporting agent. This matches the write path exactly - the
bulk charts feed (hostAgentChartRequest) already preferred the agent ID,
and pure-agent hosts already resolved to it, so merged hosts simply
become consistent. No history is orphaned: "agent" store rows were
always keyed by host.ID, and node-poller rows ("node"/<node.ID>) were
never reachable through the "agent"-typed target.
Verified live: all five real agents now resolve metricsTarget to their
agent UUID, /api/metrics-store/history returns store rows for the new
target, and the delly performance report renders 312 data points where
it previously rendered zero.
Regression tests: BuildMetricsTarget prefers the agent source for
merged-source resources, and a registry ingest test asserts the merged
node+agent resource resolves its target to host.ID - the agent store
write key.
A design-partner review of real output called the PDFs out as thin, and
the page renders agreed: fleet reports spent one nearly blank A4 page
per resource, metric cards showed raw store keys with unformatted
values ('diskread 880000.00'), and value cells overlapped the adjacent
column for long readings.
- Fleet per-resource blocks now flow several to a page with separators,
breaking only when the next block will not fit (a 6-resource report
drops from 8 pages to 3); each block gains an availability line
- Rate metrics get display names and humane units (Disk Read 859.38
KiB/s instead of diskread 880000.00); byte units are self-describing
so the old +unit suffix no longer renders '12.00 GiBbytes'
- Metric card stat columns use bounded cells so long values cannot run
under the Avg/Samples labels
- Single reports merge Resource Details and Performance Summary onto a
shared page when they fit instead of stranding a third of a page of
content on each of two pages
Verified by regenerating branded fleet and single reports and reviewing
every rendered page.
Performance reports answered 'what were the averages' but never 'was my
infrastructure up' - the question a managed-service client reads a
monthly report for. Reports now carry an Availability summary derived
from the recorded resource change timeline (state_transition entries
keyed by the canonical unified ID):
- uptime percent over the observed portion of the window, outage count,
total downtime, and longest outage, rendered in the executive summary
with an explicit semantics note; fleet summaries gain a per-resource
Uptime column and CSV exports gain availability header lines
- absent/unknown spans are unobserved time: excluded from the uptime
math entirely and disclosed as coverage, never counted as downtime.
The journal records a registry absence for every monitor restart, so
treating gaps as outages would invent fleet-wide downtime every time
the operator restarts Pulse
- warning states count as up (the resource is reachable and serving);
the uptime label clamps rounding so any real downtime can never
display as a clean 100%
- resources with no timeline render no availability section at all
rather than a fabricated number
Verified live against a real 7-day window: uptime/outage/downtime
figures reconcile with the raw resource_changes journal.
Free-form strings entering the PDF generator (AI narrative prose, resource
names, alert messages, brand display names) were written to fpdf core fonts
as raw UTF-8, and the cp1252-decoding fonts rendered em dashes and curly
quotes as mojibake. Generate and GenerateMulti now run every string field
reachable from ReportData/MultiReportData through fpdf's cp1252 translator
once before rendering, so write sites stay encoding-free. The translator is
built per call: fpdf's closure reuses an internal buffer and the generator
is shared across concurrent requests. Runes outside cp1252 degrade to '.'.
Tests render AI-shaped narratives with em dashes and curly quotes for both
the single-resource and fleet paths and assert the extracted content
streams decode without mojibake.
Performance reports were structurally disconnected from the v6 ID
space: the UI (and any API caller working from /api/state) addresses
resources by canonical unified ID, while the metrics store is keyed by
each platform's native source ID (the resource's metricsTarget). The
engine queried the store with the unified ID verbatim, so every report
rendered 'Data Points: 0' regardless of how much history existed, and
covers showed raw hash IDs a report reader cannot map to a machine.
- MetricReportRequest gains MetricsResourceID: handlers resolve the
unified ID through the tenant monitor's resource store (new
Monitor.MetricsTargetForResource accessor; the registry computes
targets on demand, they are not persisted on snapshot structs) and
the engine uses it for store queries only. Recovery points and
Patrol findings stay keyed by the unified ID.
- Legacy snapshot models and their alerts are keyed by the metrics
target ID, so enrichment now matches either ID space and resource
names/status resolve again on covers, headers, and fleet rows.
- Fleet summaries mirror the single-report guard: zero data points
across the fleet renders a muted NO DATA card instead of a green
HEALTHY 'All systems operating normally' - false reassurance is the
worst failure mode for a client-facing stability report.
- Em dashes in PDF-bound literals become hyphens; fpdf core fonts are
cp1252 and rendered them as mojibake.
In dev/demo builds HasFeature returned the implicit dev grant verdict
unconditionally, so features excluded from that grant (white_label,
multi_user, unlimited, env-gated multi_tenant) were denied even when an
explicitly activated, signature-verified license carried them - while
/api/license/status reported them active from the same claims. That
inconsistency made branded report rendering impossible to exercise in
dev builds. Excluded features now fall through to real entitlement
evaluation, so an activated license behaves the same in dev as in
release builds; the no-license dev posture is unchanged.
A cluster has one API connection, but every member node row repeated the
connection-level "API" badge next to the node's discovered URL. In the
rc.6 stacked layout that read as per-node API keys to a real user (#1493:
"API keys ??? I would say URL!!!"), and it still implied per-node
credentials in the flattened table.
Member rows now only badge sources that are actually node-local: an
attached Pulse Agent. API-only members show a muted dash whose tooltip
explains the node is monitored through the cluster's single API
connection. The cluster row itself keeps its API badge under the Method
header, and standalone connections (which do own their credentials) are
unchanged.
Refs #1493
Cluster endpoints were discovered once at add time and never re-read:
detectClusterMembership returned immediately for instances already marked
IsCluster, so when a cluster moved subnets Pulse kept dialing the dead
per-node addresses for failover and kept displaying them on Settings ->
Infrastructure, even though monitoring still worked through the main host
fallback. The manual /refresh-cluster endpoint exists but is not wired to
any UI control, so nothing ever corrected the stored addresses.
The 5-minute cluster re-check now also covers cluster instances: it
re-reads /cluster/status through the configured main host, and when a
node's address or the node set actually changed (volatile fields like
Online/LastSeen are ignored) it updates the stored endpoints, persists the
config, and rebuilds the failover client so polling moves to the new
addresses. User-managed fields (IP overrides, guest URLs, fingerprints)
are preserved by the existing discovery helpers, and fields the cluster
status API omits are inherited from the stored endpoint rather than
erased.
Refs #1493
recordTaskResult reset the failure counters on a successful poll but left
LastErrorAt/LastErrorMessage/LastErrorCategory in place forever. The
connections aggregator surfaces that field as a live error banner on
Settings -> Infrastructure and derives the Unauthorized state from it, so
one transient cluster outage (or startup blip) pinned a red "no healthy
nodes available" banner onto a connection that polls fine, and a single
past 401 could pin a healthy connection to Unauthorized.
LastError now means "current outstanding failure since the last success":
recorded on failure, cleared on success. Connections that are genuinely
failing (no successful poll) keep their banner.
Refs #1493
pollPVEInstance returned early on a GetNodes error, so a host that
stopped answering entirely (shutdown, network loss) kept its last
successful snapshot in state forever: node row frozen at online with
stale uptime. The existing 60s offline grace policy only ran for
cluster-reported peer-down and empty-poll cycles, never for a whole
instance going dark.
Route the poll error path through the same grace policy: nodes of an
unreachable instance stay online (connection degraded) within the
grace window and flip to offline with cleared uptime and CPU once it
lapses, matching the cluster-reported node-down behavior.
Verified live: registered a synthetic PVE instance, killed it, and the
node flips to offline one poll cycle after the grace period.
Fixes#1441
On the #1483 reporter's Proxmox node a SATA SSD (INTEL SSDSC2BW240A4) never
appeared in Pulse even though smartctl reads it fine, because the agent's
discovery and probing both had single points of failure:
- Discovery trusted smartctl --scan-open exclusively whenever it returned at
least one target. The scan silently omits any device it fails to open at
scan time (the failure is only a #-comment in its output), so one bad open
hid a real disk while its neighbours were listed.
- A scan-provided -d type got exactly one probe attempt on Linux. A type whose
full query (-i -A -H) fails or returns nothing dropped the disk silently.
- A probe yielding no usable SMART data dropped the disk entirely instead of
reporting the identity the kernel can prove.
Fixes, all in the agent collector:
- Union discovery: the kernel block device list (/sys/block, lsblk fallback)
is the ground truth for which disks exist; smartctl --scan-open only
contributes device-type hints. Any physical disk the scan misses gets an
untyped probe target.
- Untyped retry: a typed Linux probe that errors, yields no usable data, or
fails to open (exit bit 2, distinct from -n standby,3's exit 3) retries with
smartctl auto-detection before giving up. Multiplexed controller members
(megaraid, cciss, ...) are exempt since dropping -d would probe the array.
- Identity-only fallback: when every probe fails but the disk exists in
/sys/block with nonzero capacity, report device/model/serial/size with
health UNKNOWN instead of hiding it. No SMART data is fabricated;
multiplexed array paths and zero-capacity media are excluded.
- Exclusion follow-through: --disk-exclude now also matches the canonical
post-refine name (nvme0n1), not just the scan label (nvme0).
Regression tests use smartctl/lsblk fixtures captured from PVE 9.1.9 hosts
and the issue report. Verified live on two PVE nodes: NVMe keyed by
namespace with byte-exact pvesh sizes, SATA disk reported with full identity.
Completes the #1483 fix started in bd20069c6 (namespace devpath, authoritative
sizes, merge downgrade guard). Closes#1483.
Polling direction (provider to client site, 8006/8007) versus agent
check-in direction, per-client steps with the guided pveum setup command,
privilege-separated token caveat, TOFU certificate pinning, and the
overlapping-RFC1918 note that container-per-client isolation makes moot.
The install proof passed green this morning while every provisioned
workspace ran unlicensed; the entitlement gap survived because nothing
asserted lease health. The workspace proof now reads each workspace's
provisioned billing state and verifies the lease exactly the way a
release-build client runtime will: against the hosted entitlement trust
root, through the provider MSP license chain, requiring white_label.
A present license with an unverifiable lease fails the proof with the
specific reason; the environment-fallback plan (dev, no license) reports
entitlement_skipped_reason=no_provider_msp_license instead of asserting.
Proof and install-proof workspace output lines gain
entitlement_lease_checked/verified, entitlement_white_label, and the
skip reason.
The variable signs hosted entitlement leases; the trial-activation name is
left over from the retired trial era and reads as trial machinery to an
operator generating their licensing root key. The provider MSP bundle has
no installed base yet, so the canonical rename is free today and frozen
the moment the first design partner installs.
- Control plane reads CP_ENTITLEMENT_SIGNING_PRIVATE_KEY first and falls
back to CP_TRIAL_ACTIVATION_PRIVATE_KEY, so existing Pulse-hosted cloud
deployments (deploy/cloud, hibernated snapshot) keep working unchanged.
- deploy/provider-msp (.env.example, compose, setup.sh), MSP.md, and the
install-test pins use the canonical name; error messages name it too.
- deploy/cloud intentionally keeps the legacy name: that stack historically
signed hosted trial activations, and its snapshot predates the rename.
- MSP.md now leads with deploy/provider-msp/ (compose stack, setup.sh,
upgrade.sh, run-install-proof.sh), documents the HTTPS requirement
(__Host- portal session cookie) and the pulse.provider-msp.role labels
workspace provisioning requires, and explains the licence/lease chain
including licence-expiry behavior.
- setup.sh derives and prints the lease signing public key the provider
MSP licence must bind (also via --print-lease-signing-public-key), and
the missing-licence error now includes it with request instructions.
- .env.example documents the CP_TRIAL_ACTIVATION_PRIVATE_KEY binding.
rcourtman