An admin-browser update read does not establish access for the monitoring token. Cover the proxied target path, configured token, denial propagation and healthy endpoint retention. Remove the unsupported Sys.Audit diagnostic hint: endpoint access is determined by Proxmox, not general inventory access.
Change-source: pulse-maintainer
Contract-Neutral: Diagnostic wording and method-comment correction plus synthetic regression coverage only; no collection logic, public API, permission policy, freshness or counter semantics change.
Treat node-scoped HTTP 595 responses as debug-level resource unavailability instead of repeated authentication warnings. Preserve warnings and returned errors for real credential failures.
Refs #1794.
Contract-Neutral: corrects internal log severity without changing API, resource, or extension contracts
Change-source: pulse-maintainer
An explicit connection address override on a cluster member was silently
discarded when VerifySSL was enabled and the member had no per-endpoint
fingerprint: the hostname-for-TLS preference displaced the operator's
address, so overriding an undialable discovered hostname changed nothing.
The override now wins in every TLS mode, and the hostname preference
applies only to auto-discovered addresses.
Failed endpoint recovery attempts also logged their cause at debug level
only, leaving the recurring 'No endpoints recovered' warning without a
reason. The warning now carries per-endpoint failure reasons and the
sanitized error is stored so the UI health status shows it too.
Refs #1665
Contract-Neutral: behavioral bug fix refs #1665: cluster member IPOverride honored in all TLS modes, recovery failure reasons surfaced; no public contract delta
A node joining a PVE cluster after setup could never be trusted when the
primary was fingerprint-pinned (#1664). Two independent gaps: cluster
discovery validation only relaxed TLS when client construction failed,
but a pinned-fingerprint mismatch surfaces from the first API call, so
the member was judged not-a-Proxmox-node and its captured fingerprint
discarded; and the cluster client's TOFU refresh only ran for endpoints
that already had a per-endpoint fingerprint, handling rotation but never
first trust. Validation now retries with the member's own captured
fingerprint before rejecting it, TOFU capture runs on first use, and
discovery failures now distinguish DNS, refused, timeout, and TLS
causes in the endpoint error instead of one generic message.
Contract-Neutral: cluster discovery TLS validation bug fix; no wire contract or payload change
Recorded PVE 8 and PVE 9 guest rrddata responses (fixtures under
pkg/proxmox/testdata/rrd/) prove guest RRD never carries the cache-aware
memused/memavailable columns — they exist only in node RRD — so every
consumer branch reading them was dead code that #1634's listing fallback
(7d7d2b6a3) had already routed around.
Drop the two fields from GuestRRDPoint (now time/maxmem only, matching
the recordings), delete the dead VM RRD memory fallback and its
getVMRRDMetrics/getVMRRDMemory helpers plus the vmRRDMemCache they fed,
remove the pointless per-poll guest RRD fetch from the LXC memory path,
and retire the guest RRD lookups from PVEClientInterface. VMMemoryRaw
loses its never-populated RRD diagnostic fields, and guest reliability
scoring no longer treats the node-only rrd-* sources as trusted guest
evidence. The knownDeadGuestRRDFields allowlist in the fixture
alignment test is gone; a new reflection guard in
code_standards_test.go keeps GuestRRDPoint pinned to recorded columns,
and cleanupRRDCache pruning of the guest-agent meminfo cache gains
direct coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Issue #1634 happened because GuestRRDPoint declared memused/memavailable
columns that real PVE guest rrddata responses never contain (they exist
only in node RRD), and every test mocked the fictional columns, so CI
validated the assumption instead of the API.
Add recorded fixtures for the guest and node rrddata endpoints in both
generations: PVE 9 responses captured live from a pve-manager 9.2.3
cluster via pvesh, and PVE 8 responses recovered from the host's
pre-migration pve2-vm/pve2-node RRD databases, serialized the way
PVE::RRD::create_rrd_data emits them. Decode tests in pkg/proxmox now
replay these recordings through the real client and fail whenever a
NodeRRDPoint or GuestRRDPoint field references a column absent from
every recorded response; the two known-dead guest fields are held in an
explicit allowlist that also fails if they are removed or ever start
appearing in recordings.
Companion to 7d7d2b6a3, which restored the LXC listing fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scheduled multi-guest vzdump jobs run under a single UPID whose VMID slot
is empty, so pollBackupTasks stored them with VMID 0 and the guest-centric
backups coverage view dropped them entirely: only individually backed-up
guests ever showed task status. (Regressed with the v6.0.0 guest-centric
redesign, which removed the flat task table that used to render job runs.)
pollBackupTasks now fetches the job task's log and parses the per-guest
markers ("Starting Backup of VM", "Finished Backup of VM (duration)",
"Backup of VM failed - reason") into synthetic per-guest BackupTask
entries. Their IDs embed the parent UPID, keeping them stable across polls
and distinct from individually-run backups; per-guest times are
reconstructed from the job start plus the printed durations. Finished
jobs' logs are immutable, so results are cached per instance|UPID and each
finished run is fetched at most once, with a per-cycle fetch cap so a
historical backlog trickles in without stalling the backup poll budget.
The task listing now uses source=all + typefilter=vzdump, so running jobs
are visible too: guests covered by an in-progress job get a "running"
synthetic task, which also feeds resolveBackupIntentContext and
suppresses offline/backup alerts for guests the job is actively backing
up. The frontend needs no changes - synthetic tasks carry real VMIDs and
flow through the existing coverage model, recovery mapper, and alert
intent paths.
Contract: monitoring.md completion obligation 13 records the per-guest
synthesis boundary; proofs land in monitor_backup_job_tasks_test.go,
monitor_alert_intent_test.go, and cluster_client_api_test.go.
Reported by Johannes Strasser (support thread "PBS Bug").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ceph Quincy and later (including Squid on PVE 9) dropped the monmap
mons array and the mgrmap active_name/standbys arrays from ceph status
output, replacing them with monmap.num_mons and mgrmap.num_standbys,
with quorum membership reported at the top level of the payload. Both
the host agent parser and the Proxmox API path only understood the
legacy arrays, so modern clusters showed 0 monitors and undercounted
managers.
- hostagent: read num_mons/num_standbys and top-level quorum data,
taking the largest available signal, and base the mon/mgr service
rows on the same counts
- pkg/proxmox: decode mgrmap num_standbys and top-level
quorum_names/quorum on CephStatus
- monitoring: fall back to the new fields when counting MON/MGR
daemons, and log Ceph 401/403 failures at warn level with a hint to
grant Sys.Audit on / instead of hiding them at debug
- models: prefer the larger non-zero MON/MGR counts when merging Ceph
cluster records from multiple sources
Fixes#1626, Refs discussion #1290
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep the configured cluster URL as primary and recover discovered member
endpoints asynchronously while it is healthy. Reset member reachability
evidence when the effective address changes.
Refs #1437 and #1493
Clears the dupl pairs outside internal/api:
- internal/alerts/pmg.go: the total/deferred/hold per-node queue checks
share evaluatePMGNodeQueueAlert; the historical short-circuit (a
below-threshold clear or invalid spec skips the node's remaining
checks) is preserved via the helper's skip-node return.
- internal/vmware/provider.go: the cloneInventory* family rides generic
cloneSliceWith / cloneShallowSlice helpers instead of fourteen copies
of the same nil/make/loop scaffold.
- internal/vmware/client.go + client_signals.go: the byte-identical
Automation and VI/JSON fetchers delegate to one getSessionScopedJSON.
- internal/vmware/fixtures.go: added to the dupl exclude list in
.golangci.yml — literal mock fixture catalogs are the same category as
the existing internal/mock/ exclusion.
- pkg/licensing/license_server_client.go: Activate and
ExchangeLegacyLicense share postActivationRequest (idempotent POST +
shared activation response decode).
- pkg/proxmox/client.go: LXC/VM RRD fetches share getGuestRRDData.
- pkg/pulsecli/actions.go is intentionally left for the api-contracts
slice.
Full test suites pass for internal/alerts, internal/vmware,
pkg/licensing, pkg/proxmox.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Prefer QEMU guest-agent MemAvailable when Proxmox reports saturated VM memory without guest free fields.
Add regression coverage for the issue #1319 Windows fsinfo volume payload so usable C/E/F volumes remain counted while System Reserved partitions are skipped.
Refs #1319
When every cluster endpoint failed health, getHealthyClient wrapped
the failure as `no healthy nodes available in cluster X (all N
endpoints unreachable: [...])`, dropping the per-endpoint reason from
cc.lastError. The connections aggregator's auth-error regex
(401/403/unauthorized/forbidden/authentication/...) only sees the
outer message, so a token rejected with 401 on every endpoint of a
clustered PVE connection surfaced as `state: "unreachable"` /
`adapterHealth: "blocked"` instead of `state: "unauthorized"` /
`credentialStatus: "invalid"` — the same Settings → Connections
brokenness the rest of today's commits set out to remove.
Single-node `pve:pi` already classified the same kind of failure
correctly because its error came straight from the per-instance
client; only the cluster wrapper masked it.
Surface each unhealthy endpoint's already-sanitized reason in the
outer error. The "no healthy nodes available" prefix is preserved so
existing callers that test for it (monitor_polling_storage.go,
internal cluster_client passthroughs, existing tests) keep working.
Add a regression test covering both shapes:
- all endpoints failed auth → wrapped error contains
"Authentication failed" so the aggregator regex now matches.
- endpoint with no recorded reason → wrapped error includes the
fallback "no recorded reason" text rather than a bare URL.
Pulse was generating tag colours from a hash of the tag name instead
of using the colours configured in Proxmox. Now polls /cluster/options
once per PVE instance and merges the tag-style colour map into state,
which the frontend uses as the first-priority colour source for tag
badges. Falls back to the existing special-tag and hash-based colours
when Proxmox hasn't set a custom colour for a tag.
Proxmox status.Mem includes page cache as "used" memory, inflating
reported VM usage. The existing fallbacks (balloon meminfo, RRD, linked
host agent) were frequently unavailable, causing most VMs to fall
through to the inflated status-mem source.
Adds a new last-resort fallback that reads /proc/meminfo via the QEMU
guest agent file-read endpoint to get accurate MemAvailable. Results
are cached (60s positive, 5min negative backoff for unsupported VMs).
Also fixes: RRD memavailable fallback missing from traditional polling
path, cache key collisions in multi-PVE setups, FreeMem underflow
guard inconsistency, and integer overflow in kB-to-bytes conversion.
Backport of v6 commits a87c9950 and 347d7db1.
Part 1 (a87c9950): Wrap the four guest agent c.get() errors with
fmt.Errorf("guest agent ...: %w", err) so isVMSpecificError() correctly
scopes them to the VM rather than the cluster endpoint.
Part 2 (347d7db1): Replace the 20+ pattern blocklist in
executeWithFailover with an allowlist via isEndpointConnectivityError().
Only true TCP/DNS/TLS failures mark an endpoint unhealthy. Any HTTP
response from Proxmox — including 500 — proves the node is reachable
and returns the error without affecting endpoint health.
Linux VM page cache (#1270): QEMU VM memory now falls back to Proxmox
RRD's memavailable metric (which excludes reclaimable page cache) when
the qemu-guest-agent doesn't provide MemInfo.Available. Previously the
fallback was detailedStatus.Mem (total - MemFree), inflating usage to
80%+ on VMs with normal Linux page cache. Mirrors the existing LXC
rrd-memavailable path.
FreeBSD ZFS ARC (#1264, #1051): The host agent now reads
kstat.zfs.misc.arcstats.size via SysctlRaw on FreeBSD and subtracts
the ARC size from reported memory usage. ZFS ARC is reclaimable under
memory pressure (like Linux SReclaimable) but gopsutil counts it as
wired/non-reclaimable, causing false 90%+ memory alerts on TrueNAS
and FreeBSD hosts. Build-tagged so it compiles cleanly on all platforms.
Fixes#1270Fixes#1264Fixes#1051
(cherry picked from commit 94502f83ff9ffc6da28aaadc946a2f7d8b4e9bac)
- Add PendingUpdates and PendingUpdatesCheckedAt fields to Node model
- Add GetNodePendingUpdates method to Proxmox client (calls /nodes/{node}/apt/update)
- Add 30-minute polling cache to avoid excessive API calls
- Add pendingUpdates to frontend Node type
- Add color-coded badge in NodeSummaryTable (yellow: 1-9, orange: 10+)
- Update test stubs for interface compliance
Requires Sys.Audit permission on Proxmox API token to read apt updates.
- Add comprehensive tests for internal/api/config_handlers.go (Phases 1-3)
- Improve test coverage for AI tools, chat service, and session management
- Enhance alert and notification tests (ResolvedAlert, Webhook)
- Add frontend unit tests for utils (searchHistory, tagColors, temperature, url)
- Add proximity client API tests
Previously, when some cluster endpoints were unreachable (e.g., backup
nodes intentionally offline), the cluster was marked as "degraded" even
though the Proxmox cluster itself was healthy and had quorum.
Now the connection health check queries the Proxmox cluster's actual
quorum status. A cluster is only marked "degraded" if it has lost
quorum (not enough votes for consensus), which is the actual indicator
of cluster instability.
This means:
- Cluster with quorum + some nodes offline = "healthy"
- Cluster without quorum = "degraded" (warning)
- All endpoints down = "error"
Fixes#1085
When discovering cluster nodes, Pulse now automatically prefers IPs
on the same subnet as the initial connection. This fixes the common
issue where Pulse used internal cluster network IPs (e.g., 172.x.x.x)
instead of management network IPs (e.g., 10.x.x.x).
How it works:
1. Extract subnet from initial connection URL (assumes /24 for IPv4)
2. For each discovered node, query /nodes/{node}/network for all IPs
3. If cluster-reported IP is on a different subnet, find an IP on
the preferred subnet and set it as IPOverride
4. Manual IPOverride settings are preserved and take precedence
This eliminates the need for manual IPOverride configuration in most
multi-network Proxmox setups.
Refs #929, #1066
Changes:
1. Add MAX_POLL_TIMEOUT env var for large Proxmox clusters that need
more than 3 minutes for polling (default: 3m, minimum: 30s)
2. Handle external Ceph storage gracefully - don't mark nodes unhealthy
when Proxmox returns 'binary not installed' (e.g., for Ceph not
managed by Proxmox)
Related to #965
pulse-triage[bot]