On a Proxmox node, physical disks collected by the host agent were keyed by
the NVMe controller (e.g. "nvme0 [nvme]") instead of the namespace, reported
sizeBytes 0 (or a stale filesystem-usage value), and flickered as the agent
reading intermittently replaced the authoritative Proxmox disks/list reading.
Root causes:
- smartctl --scan-open reports NVMe disks by their controller char device
(/dev/nvme0), and that scan label became the reported devPath.
- DiskSMART carried no capacity, so the server backfilled size by matching the
SMART device against host filesystem-usage entries, which never match a whole
partitioned/LVM/ZFS disk, leaving size 0.
- The unified-resource merge let the agent's controller label overwrite the
canonical Proxmox /dev/... devPath.
Fixes:
- The agent now reports the canonical block device (an NVMe controller resolves
to its namespace) and the authoritative capacity from /sys/block, with the
smartctl user_capacity / nvme_total_capacity as a cross-platform fallback.
Disks behind multiplexing controllers (megaraid, cciss, areca) keep their
disambiguating label and smartctl-reported size.
- SizeBytes flows through the agent report, host model, and adapter; the
filesystem-usage match is demoted to a legacy fallback.
- The merge keeps a canonical /dev/<device> devPath and never downgrades it to
a scan label, so an un-updated agent can no longer corrupt Proxmox data.
Refs #1483.
Prefer QEMU guest-agent MemAvailable when Proxmox reports saturated VM memory without guest free fields.
Add regression coverage for the issue #1319 Windows fsinfo volume payload so usable C/E/F volumes remain counted while System Reserved partitions are skipped.
Refs #1319
Fixes#1341
Forward-port of the release/5.1 fix (ce1607694).
When a Pulse host-agent reports Ceph data, ApplyHostReport upserts the
cluster into state but only the Proxmox-API polling path ran
checkCephPoolStorage. Users with agent-reported Ceph (instance prefix
"agent:hostname") saved per-pool overrides under agent-prefixed IDs
that the alert manager never evaluated. The threshold appeared to
save (and showed Custom in the UI), but the polling cycle was
checking a different storage ID, so the alert silently stayed
dormant. Run checkCephPoolStorage right after the agent upsert so
the override key actually drives evaluation.
Two real bugs that surfaced once the 20m test timeout let the
internal/api and internal/monitoring packages run to completion.
cloneVMwareInventoryMetrics omitted four fields:
Commit 23ea4e487 (Surface vSphere VM uptime and guest disk usage)
added UptimeSeconds, DiskUsedBytes, DiskTotalBytes, and DiskPercent
to vmware.InventoryMetrics but did not extend
cloneVMwareInventoryMetrics. The clone left those pointer fields
aliased to the source struct, so the mock fixture refresh path
(refreshVMwareInventoryMetrics writing through metrics.UptimeSeconds
via ensureInt64Ptr) and the snapshot read path
(inventoryUptimeSeconds dereferencing metrics.UptimeSeconds) raced
on the same heap-allocated int64.
TestMonitorBuildBroadcastFrontendStateUsesCanonicalMockUnifiedResources
exemption:
The test asserts broadcast state does not publish the lowercase-
hyphenated legacy docker host label so canonical docker hosts
surface their human-readable DisplayName. Commit 89abed099
(2026-05-24) added the docker-swarm-node resource type whose Name
is the swarm node hostname (matching how Docker Swarm identifies
node members), which collided with the legacy-label rejection.
Refine the assertion to apply only to host-type resources
(docker-host, agent, node).
handleWorkloadCharts: add a 3s TTL response cache + per-key singleflight
so repeated sparkline polls from dashboards share work. TestSLO_WorkloadCharts
p95 drops from 188ms (over the 90ms target) to 838µs because most polls
hit cache or coalesce. Cold-call cost is unchanged; the win is dedup of
the steady-state polling pattern.
Cache + singleflight state live on the Router struct, not as globals, so
tenants don't cross-contaminate and tests don't pollute each other. Same
treatment for stateComputeGroup added in the previous commit.
monitor_metrics.mergeMetricHistory: drop the defensive cloneMetricPointMap
of base and cloneMetricSeries of candidate inside the merge. Every caller
already passes a freshly-owned map (from filterChartMetricMap,
GetAllGuestMetrics, GetAllStorageMetrics, queryStore*), so the internal
clones were pure waste. Caller-side cloneMetricPointMap is removed in
the three sites where the input is owned. ~6% allocation reduction on
BenchmarkHandleWorkloadCharts_StoreBacked.
Contract-neutral: no endpoint, response body, header, or wire-format
change.
Closes the only API-coverage gap from the Docker / Kubernetes IA
maturity review: Roles, ClusterRoles, RoleBindings, and
ClusterRoleBindings now flow from the Kubernetes agent through the
canonical resource registry into the Kubernetes platform-page
Configuration tab.
Agent: pkg/agents/kubernetes/report.go gains four new report struct
types that carry summary counts plus subject-kind sets; individual
subject names and full PolicyRule contents are deliberately omitted
so Pulse stays a "what permissions exist where" surface, not an RBAC
enumeration tool. internal/kubernetesagent/agent.go gains four
collectors that call rbacv1.RoleList/ClusterRoleList/etc. through the
existing runKubernetesCallWithRetry wrapper, matching the
ServiceAccount collector's RBAC-forbidden retry pattern.
Canonical: internal/models mirrors with NormalizeCollections coverage;
convert* funcs in internal/monitoring/kubernetes_agents.go translate
agent report -> model; ResourceTypeK8sRole / K8sClusterRole /
K8sRoleBinding / K8sClusterRoleBinding join the canonical type set;
registry ingest* + adapter resourceFrom* functions emit one Resource
per RBAC object with ruleCount / roleKind / roleName / subjectCount /
subjectKinds / aggregationLabels on the K8s meta; search mapping in
internal/api/resources.go and the privacy allow-list in
internal/api/org_handlers.go pick up the four new type tokens; the
K8s privacy category in unifiedresources/policy_metadata.go classifies
them like the rest of K8s.
Frontend: ResourceType union + ResourceKubernetesMeta carry the new
kinds and RBAC summary fields; KubernetesPageSurface query asks for
them; the page model buckets them into the Configuration group;
KubernetesConfigTable renders Role / ClusterRole rule counts and the
aggregated flag, plus RoleBinding / ClusterRoleBinding role refs and
"N subjects · Kind1, Kind2 +overflow" subject summaries.
Curated demo seeds per-namespace Roles + RoleBindings plus an
aggregated ClusterRole + ClusterRoleBinding for pulse-demo-monitoring
in each cluster so the Configuration tab renders 18 RBAC rows across
the three demo clusters.
Contracts updated for the canonical-shape guard: monitoring,
api-contracts, unified-resources, frontend-primitives,
organization-settings (canonical) plus agent-lifecycle and
storage-recovery (dependent via Extension Points). Verification
proofs extended: kubernetes_registry_test.go, kubernetes_agents_test.go,
agent_inventory_test.go (new TestCollectRBACInventoryReportsSummaryCountsOnly
that pins the subject-name-omission contract), demo_scenarios_test.go,
adapter_coverage_test.go, contract_test.go, org_handlers_test.go,
resourceIdentity.test.ts, reportingResourceTypes.test.ts,
KubernetesConfigTable.test.tsx, and the
subsystem_lookup_test.py line-anchor bumps that the contract edits
shifted (api-contracts 246 -> 253, organization-settings 92 -> 93).
Verified:
- go build ./internal/... ./cmd/... clean
- go test ./internal/unifiedresources/..., ./internal/mock/...,
./internal/kubernetesagent/..., ./internal/api/...,
the K8s subset of ./internal/monitoring/... all clean (three
pre-existing unrelated monitoring failures noted earlier remain
unchanged by this commit)
- npm run type-check, lint:eslint, lint:theme,
lint:canonical-platforms clean
- vitest: 70 K8s frontend tests pass including the new RBAC render
coverage in KubernetesConfigTable.test.tsx
- browser proof on /kubernetes/configuration: 36 config rows
including 18 RBAC rows across three clusters; ClusterRole
"pulse-demo-monitoring" shows "12 rules · Aggregated";
ClusterRoleBinding shows "3 subjects · Group, ServiceAccount +1"
Collect native Kubernetes config, policy, and autoscaling objects.
Project the new resource types through API filters, unified resources, mock fixtures, and Kubernetes tabs.
Keep Secret inventory metadata-only and route k8s-secret policy as restricted local-only.
VsphereHostsTable was missing the Version (ESXi build) and Uptime
columns that Proxmox / Docker / Kubernetes / TrueNAS host
equivalents carry. The data was already piped through the
canonical projection — ESXi version on resource.agent.osVersion
(e.g. "8.0.3"), host uptime on resource.uptime (lifted from
InventoryMetrics.UptimeSeconds via the sys.uptime.latest
PerformanceManager counter wired in 23ea4e487 this morning) —
the table just didn't render columns for them.
VsphereHostsTable now renders Version and Uptime cells, ordered
Version-adjacent-to-Host and Uptime before vCenter to mirror the
Proxmox Nodes layout. Column widths trimmed to fit. Uptime uses
the shared formatUptime util in condensed form (e.g. "163d")
with the full label as the cell title attribute. Detail row
colspan bumps 9 to 11 to keep the drawer spanning the full row.
Plumbing the canonical Resource.Uptime through to the frontend
required two fallback extensions surfaced by the column audit:
- useUnifiedResources.ts toResource: the uptime fallback chain
ended on platform-specific carve-outs (agent.uptimeSeconds,
proxmox.uptime, pbs/pmg/kubernetes); vSphere populates only
the canonical Resource.Uptime, so the chain has to land on
v2.uptime. Same shape as the workloads-hook fix in e5b31f484.
- internal/monitoring/monitor.go monitorUptime: the websocket
broadcast converter walked the same platform-specific chain
and silently dropped vSphere host/VM uptime, then merge-
clobbered the REST-loaded value once WS reconnect replayed
the broadcast. Add the canonical resource.Uptime fallback so
the broadcast payload carries uptime for VMware-backed rows
consistently with the REST contract. Carve-outs still take
precedence so existing platforms keep prior behavior.
Verified: vSphere overview Hosts table now renders esxi-01..07
with Version 8.0.3 and Uptime 147d-167d cells. No regression in
Proxmox / Docker / Kubernetes / TrueNAS uptime paths.
Contracts:
- monitoring.md documents the canonical Resource.Uptime fallback
in monitorUptime and the carve-out precedence.
- unified-resources.md adds the same canonical-uptime fallback
rule to the toResource consumer-side contract.
- storage-recovery.md amends rule 29 to call out that the
canonical Resource.Uptime fallback is descriptive host/VM
uptime only and must not be reinterpreted as backup recency
or recovery cadence.
Proofs:
- internal/monitoring/canonical_guardrails_test.go locks the
monitorUptime fallback contract: canonical Resource.Uptime is
surfaced when no carve-out is set, carve-outs take precedence,
nil when nothing populates.
- frontend-modern/src/hooks/__tests__/useUnifiedResources.test.ts
asserts the toResource v2.uptime fallback for a vSphere-shaped
payload with no platform-specific uptime carve-out.
Project vCenter network inventory through canonical resources and add the vSphere Networks table backed by vCenter network topology. Align resource presentation coalescing so state and resource APIs share the same host contract.
Add a global resource timeline endpoint for provider activity and wire vSphere Activity to VMware timeline changes. Seed mock VMware activity through the same supplemental-change path and keep the relevant resource contract tests current.
Add VM and share counts to the TrueNAS observed summary contract and systems table.
Keep TrueNAS overview inventory table-first with a platform guardrail.
Scope TrueNAS child source IDs under the system source key so duplicate pool, dataset, app, VM, share, and disk names from different appliances do not merge.
Use the resulting hierarchy for per-system TrueNAS overview counts and scoped mock metric targets.
monitorFrontendStatus's "pod" branch was re-deriving status from
Kubernetes.PodPhase alone, so a CrashLoopBackOff pod (Phase=Running
with unready containers) was reported as "running" even though
statusFromKubernetesPod had already classified it as StatusWarning.
Drop the unconditional phase=running -> "running" mapping and fall
through to the unified-status switch, which surfaces StatusWarning as
"degraded". Phase-based mappings for pending/unknown/succeeded/failed
are kept since they still cover edge cases where the unified status
is StatusOnline but phase carries useful information.
Adds two regression cases in TestMonitorFrontendAndMetricHelpers:
- pod warning running degraded (CrashLoopBackOff)
- pod online running running (healthy baseline)
The Proxmox-side LXC Docker inventory poll occasionally produces an
ok=true report with zero CONTAINER lines but a valid PS_OK marker:
docker ps -a inside the LXC genuinely returned an empty list for
that one call, typically while watchtower is briefly recreating a
container or the dockerd is mid-restart. ApplyDockerReport then
wipes the host's container list, every connected WebSocket client
sees every row for that host flash blank, and the next successful
poll 10 s later restores it.
Before calling ApplyDockerReport, check whether the host already
has containers tracked in state. If yes and the new report has
zero containers, log at debug and skip the apply, counting the
poll as skipped rather than collected. A host that is genuinely
empty still applies on first sight (no previous containers means
no previous state to protect).
Add regression tests for both branches: a populated host receiving
an empty report keeps its container list; a freshly-seen empty
host still gets created.
The Proxmox-LXC Docker hosts row was blank (no CPU / MEM-used /
Disk) and the display name appended a noisy "(LXC <vmid>)" suffix,
because the inventory script only collects Docker daemon metadata,
not host resource usage. Pulse already polls the same data for the
underlying LXC via the PVE cluster/resources endpoint, so the
Docker host row should mirror what the rest of Pulse already knows.
Before applying each LXC inventory report, enrich the Host info
with CPU / Memory / Disk / Uptime taken from the source LXC's
existing models.Container. Drop the "(LXC <vmid>)" suffix from
the synthesized display name. On the frontend, detect the
proxmox-lxc-docker: hostSourceId prefix and emit a styled
"LXC <vmid>" badge into the System column instead of the
filtered-out runtime-only "docker" badge.
The Proxmox-side LXC Docker inventory script masked `docker ps`
failures with `|| true`, so when `pct exec` was truncated, slow, or
`docker ps` itself transiently failed, the parser produced an
ok=true report with `containers=[]` and `ApplyDockerReport` blanked
the host's container list. Containers then reappeared on the next
successful poll, producing the visible flicker on the Docker page.
Emit a `PS_OK` marker only when `docker ps` exits cleanly, and
refuse to apply the report when the marker is absent. A genuine
"host has zero containers" state still applies (PS_OK present,
zero CONTAINER lines), but a truncated or failed poll now skips
apply and lets the previous container list survive the inter-poll
gap. Add regression tests for both paths.
Two specific platform-page quality issues from the audit:
1. **/kubernetes/overview only had 1 cluster.** Bumping the K8s cluster
count past 1 had been deferred because the prior
monitor-broadcast equivalence test compared the raw snapshot count
to the broadcast count exactly, and broadcast's
`coalesceBroadcastResources` + second-pass coalesce inside
`convertResourcesForBroadcast` legitimately drops merge candidates
that the raw snapshot keeps. Switch the test to compare against
the canonical snapshot count within a ±5% tolerance so future
fixture bumps stay green without loosening any of the test's
exact-name and exact-identity assertions. With that in place, bump
`K8sClusterCount` 1 → 3 in `internal/mock/generator.go`,
`scripts/toggle-mock.sh`, and the matching
`scripts/tests/test-toggle-mock.sh` so the canonical mock estate
ships with production + staging + edge clusters end-to-end.
Live mock survey: k8s-cluster: 3, k8s-deployment: 42, pod: 120,
plus 15 K8s nodes merged onto their agent hosts.
2. **/vmware/storage looked empty under platform-page chrome.**
`resolveStorageSourceKey` was reading only `storage.type` (the
on-disk technology like `vsan`, `vmfs`, `nfs41`, `zfs-pool`) and
never consulted `storage.platform` (the canonical platform key
like `vmware-vsphere` or `truenas`). Source filter chip options
were therefore generated as `vsan`, `vmfs`, `nfs41`, etc., and
`forcedSourceFilter='vmware-vsphere'` had nothing to match.
Prefer the canonical `storage.platform` tag when set, so VMware
datastores group under `vmware-vsphere`, TrueNAS pools group under
`truenas`, PBS datastores under `proxmox-pbs`, etc., for both the
chip options and the embedded platform-page filter.
Browser verification (Playwright, chromium, live mock-mode dev runtime):
- 9 tests pass.
Targeted vitest:
- `src/features/storageBackups` + `src/utils/__tests__/sourcePlatforms.test.ts` +
`src/components/Storage/__tests__/storageSourceOptions.test.ts` (31
files / 141 tests) green.
Go tests:
- `go test ./internal/mock/... ./internal/monitoring/... ./internal/vmware/...`
all green.
Contracts updated:
- `monitoring.md` Shared Boundaries: new K8s multi-cluster default,
±5% tolerance for the broadcast equivalence assertion.
- `deployment-installability.md` Shared Boundaries: toggle-mock.sh /
DefaultConfig parity updated for the K8sClusterCount=3 baseline.
Mock pages were sparse: 3 Proxmox nodes × 3 VMs × 3 LXCs, 2 Docker
hosts × 5 containers, 1 K8s cluster × 3 nodes × 10 pods × 4
deployments. That populated platform pages with handfuls of rows
rather than table density that exercises sorting, grouping, drawers,
and responsive layout.
Bump `internal/mock/generator.go::DefaultConfig` to target a mature
small-to-mid homelab / SMB environment:
- NodeCount: 3 → 5 (matches the curated demo scenario's pve1..pve5
regional naming)
- VMsPerNode: 3 → 6
- LXCsPerNode: 3 → 8
- DockerHostCount: 2 → 5
- DockerContainersPerHost: 5 → 14
- GenericHostCount: 2 → 4
- K8sClusterCount: 1 (unchanged; the curated demo and broadcast
coalesce tests assume a single cluster identity)
- K8sNodesPerCluster: 3 → 5
- K8sPodsPerCluster: 10 → 40
- K8sDeploymentsPerCluster: 4 → 14
Resource survey under the new defaults (live mock backend):
- TOTAL 307 unique resources (was ~50-100)
- app-container: 75, storage: 55, system-container: 44, pod: 40,
vm: 31, physical_disk: 19, agent: 15, k8s-deployment: 14,
docker-host: 5, network-endpoint: 5, pmg: 2, pbs: 1, k8s-cluster: 1
Platform pages now feel populated under mock mode:
- /docker/overview: 5 hosts (was 2)
- /docker/containers: 75 containers (was 13)
- /kubernetes/nodes: 5 (was 3)
- /kubernetes/pods: 40 (was 10)
- /kubernetes/deployments: 14 (was 4)
`internal/mock/demo_scenarios.go` extended to season `local`,
`local-zfs`, and per-node iso/service-pool storage names for pve6 and
beyond, so future NodeCount bumps don't regress the curated demo into
generic "service-pool" labels (a test guard explicitly forbids that
alias). A new `TestDemoScenarioStorageNamingHandlesScaledNodeCount`
covers the scaled-NodeCount path.
`internal/monitoring/monitor_unified_state_test.go` updated to compare
the broadcast count against the coalesced snapshot count rather than
the raw snapshot count — the broadcast path merges resources that
share a canonical host key (K8s nodes onto linked agent hosts), so
larger fixture sizes legitimately produce more merge candidates, and
the prior raw-equality assertion would have broken on any future
fixture growth too. The test still asserts every canonical name and
mock identity it checked before.
`scripts/toggle-mock.sh` (`mock_default_entries`) and the matching
`scripts/tests/test-toggle-mock.sh` assertions are aligned with the
new defaults so `npm run mock:edit` and per-dev `.env` seeding match
the canonical baseline.
Contracts updated:
- `monitoring.md` Shared Boundaries: records the new DefaultConfig
target sizes and the requirement that demo-scenario seasoning stay
aligned with NodeCount changes.
- `deployment-installability.md` Shared Boundaries: records that
`mock_default_entries()` in toggle-mock.sh must stay aligned with
`internal/mock.DefaultConfig` so CLI/toggle/runtime mock densities
never drift apart.
Targeted Go tests:
- `go test ./internal/mock/...` green
- `go test ./internal/monitoring/...` green
Playwright (chromium, live mock-mode dev runtime):
- 9 tests, all pass; populated assertions now hit dense tables (5
hosts, 14+ containers, 40 pods, etc.).
Known remaining fixture gaps (canonical adapter, not config):
- VMware fixture inventory in `internal/vmware/fixtures.go` is
hardcoded at 4 hosts / 6 VMs / 4 datastores; not scaled in this
commit.
- TrueNAS fixture inventory in `internal/truenas/fixtures.go` is
similarly hardcoded; not scaled in this commit.
Ensure unified resource snapshots include recent standalone host-agent continuity so Infrastructure does not briefly undercount connected systems after login or restart.
Keep desired config fingerprints as response metadata derived from the signed command and settings payload.
Use merged agent profile settings when building remote config fingerprints.
A Proxmox host wedged on a ZFS deadlock yesterday took the cluster API poll
with it (context deadline exceeded). The unified connections aggregator
flipped the Connection from active to stale to unreachable, and the
Settings / Infrastructure page rendered the right badges, but no top-nav
alert ever fired because nothing was actively notifying off that derived
state. Patrol's deterministic triage flagged it every minute, but its LLM
investigation stage has been broken since 2026-02-26 so flags never
escalated into user-visible findings. Result: a 3 hour outage I only
noticed because I happened to open Settings.
This wires an active notification off the same connection state the
Settings badges already use:
- internal/alerts/connection.go: new CheckConnection +
clearConnectionDegradedAlert that fire connection-degraded after three
consecutive stale or unreachable observations. Severity scales: stale
warning, unreachable / unauthorized critical. Clear runs through the
same recovery-confirmation gate as clearNodeOfflineAlert so a single
flap back to active doesn't silently resolve a real outage. Paused,
disabled, and non-platform connections are no-ops.
- internal/api/connections_alerts.go: snapshot translator that turns
api.Connection into the narrow alerts.ConnectionSnapshot view. Keeping
the snapshot type inside the alerts package preserves the existing
api -> monitoring import direction; the monitor would have cycled if
it called back into api directly.
- internal/monitoring: new SetConnectionsSnapshotLister hook + a
per-tick checkConnectionAlerts call in the main poll loop, alongside
the existing evaluate*Agents passes.
- internal/api/router.go: register the lister closure on r.monitor so
the alerts loop sees the same Connection rows the HTTP handler does.
- internal/alerts/specs/types.go: add "connection" to the migration
bridge list of accepted ResourceTypes, alongside node / docker-host /
proxmox-disk / etc. The connection concept doesn't have a canonical
unified resource type yet; this matches the existing pattern for
alert-keyed resources that aren't first-class canonical.
Test coverage in internal/alerts/connection_test.go covers active never
fires, three stale observations escalate from pending to warning,
unreachable escalates warning to critical, unauthorized fires critical
cold, paused / disabled / agent never fire, recovery confirmation gate,
and a stale flap during recovery resets the gate.
TestResourceAlertSpecValidateAllowsConnectionMigrationBridgeType mirrors
the existing migration-bridge proof tests for the new type.
rcourtman