Recorded PVE 8 and PVE 9 guest rrddata responses (fixtures under
pkg/proxmox/testdata/rrd/) prove guest RRD never carries the cache-aware
memused/memavailable columns — they exist only in node RRD — so every
consumer branch reading them was dead code that #1634's listing fallback
(7d7d2b6a3) had already routed around.
Drop the two fields from GuestRRDPoint (now time/maxmem only, matching
the recordings), delete the dead VM RRD memory fallback and its
getVMRRDMetrics/getVMRRDMemory helpers plus the vmRRDMemCache they fed,
remove the pointless per-poll guest RRD fetch from the LXC memory path,
and retire the guest RRD lookups from PVEClientInterface. VMMemoryRaw
loses its never-populated RRD diagnostic fields, and guest reliability
scoring no longer treats the node-only rrd-* sources as trusted guest
evidence. The knownDeadGuestRRDFields allowlist in the fixture
alignment test is gone; a new reflection guard in
code_standards_test.go keeps GuestRRDPoint pinned to recorded columns,
and cleanupRRDCache pruning of the guest-agent meminfo cache gains
direct coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Issue #1634 happened because GuestRRDPoint declared memused/memavailable
columns that real PVE guest rrddata responses never contain (they exist
only in node RRD), and every test mocked the fictional columns, so CI
validated the assumption instead of the API.
Add recorded fixtures for the guest and node rrddata endpoints in both
generations: PVE 9 responses captured live from a pve-manager 9.2.3
cluster via pvesh, and PVE 8 responses recovered from the host's
pre-migration pve2-vm/pve2-node RRD databases, serialized the way
PVE::RRD::create_rrd_data emits them. Decode tests in pkg/proxmox now
replay these recordings through the real client and fail whenever a
NodeRRDPoint or GuestRRDPoint field references a column absent from
every recorded response; the two known-dead guest fields are held in an
explicit allowlist that also fails if they are removed or ever start
appearing in recordings.
Companion to 7d7d2b6a3, which restored the LXC listing fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scheduled multi-guest vzdump jobs run under a single UPID whose VMID slot
is empty, so pollBackupTasks stored them with VMID 0 and the guest-centric
backups coverage view dropped them entirely: only individually backed-up
guests ever showed task status. (Regressed with the v6.0.0 guest-centric
redesign, which removed the flat task table that used to render job runs.)
pollBackupTasks now fetches the job task's log and parses the per-guest
markers ("Starting Backup of VM", "Finished Backup of VM (duration)",
"Backup of VM failed - reason") into synthetic per-guest BackupTask
entries. Their IDs embed the parent UPID, keeping them stable across polls
and distinct from individually-run backups; per-guest times are
reconstructed from the job start plus the printed durations. Finished
jobs' logs are immutable, so results are cached per instance|UPID and each
finished run is fetched at most once, with a per-cycle fetch cap so a
historical backlog trickles in without stalling the backup poll budget.
The task listing now uses source=all + typefilter=vzdump, so running jobs
are visible too: guests covered by an in-progress job get a "running"
synthetic task, which also feeds resolveBackupIntentContext and
suppresses offline/backup alerts for guests the job is actively backing
up. The frontend needs no changes - synthetic tasks carry real VMIDs and
flow through the existing coverage model, recovery mapper, and alert
intent paths.
Contract: monitoring.md completion obligation 13 records the per-guest
synthesis boundary; proofs land in monitor_backup_job_tasks_test.go,
monitor_alert_intent_test.go, and cluster_client_api_test.go.
Reported by Johannes Strasser (support thread "PBS Bug").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ceph Quincy and later (including Squid on PVE 9) dropped the monmap
mons array and the mgrmap active_name/standbys arrays from ceph status
output, replacing them with monmap.num_mons and mgrmap.num_standbys,
with quorum membership reported at the top level of the payload. Both
the host agent parser and the Proxmox API path only understood the
legacy arrays, so modern clusters showed 0 monitors and undercounted
managers.
- hostagent: read num_mons/num_standbys and top-level quorum data,
taking the largest available signal, and base the mon/mgr service
rows on the same counts
- pkg/proxmox: decode mgrmap num_standbys and top-level
quorum_names/quorum on CephStatus
- monitoring: fall back to the new fields when counting MON/MGR
daemons, and log Ceph 401/403 failures at warn level with a hint to
grant Sys.Audit on / instead of hiding them at debug
- models: prefer the larger non-zero MON/MGR counts when merging Ceph
cluster records from multiple sources
Fixes#1626, Refs discussion #1290
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep the configured cluster URL as primary and recover discovered member
endpoints asynchronously while it is healthy. Reset member reachability
evidence when the effective address changes.
Refs #1437 and #1493
Clears the dupl pairs outside internal/api:
- internal/alerts/pmg.go: the total/deferred/hold per-node queue checks
share evaluatePMGNodeQueueAlert; the historical short-circuit (a
below-threshold clear or invalid spec skips the node's remaining
checks) is preserved via the helper's skip-node return.
- internal/vmware/provider.go: the cloneInventory* family rides generic
cloneSliceWith / cloneShallowSlice helpers instead of fourteen copies
of the same nil/make/loop scaffold.
- internal/vmware/client.go + client_signals.go: the byte-identical
Automation and VI/JSON fetchers delegate to one getSessionScopedJSON.
- internal/vmware/fixtures.go: added to the dupl exclude list in
.golangci.yml — literal mock fixture catalogs are the same category as
the existing internal/mock/ exclusion.
- pkg/licensing/license_server_client.go: Activate and
ExchangeLegacyLicense share postActivationRequest (idempotent POST +
shared activation response decode).
- pkg/proxmox/client.go: LXC/VM RRD fetches share getGuestRRDData.
- pkg/pulsecli/actions.go is intentionally left for the api-contracts
slice.
Full test suites pass for internal/alerts, internal/vmware,
pkg/licensing, pkg/proxmox.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Prefer QEMU guest-agent MemAvailable when Proxmox reports saturated VM memory without guest free fields.
Add regression coverage for the issue #1319 Windows fsinfo volume payload so usable C/E/F volumes remain counted while System Reserved partitions are skipped.
Refs #1319
When every cluster endpoint failed health, getHealthyClient wrapped
the failure as `no healthy nodes available in cluster X (all N
endpoints unreachable: [...])`, dropping the per-endpoint reason from
cc.lastError. The connections aggregator's auth-error regex
(401/403/unauthorized/forbidden/authentication/...) only sees the
outer message, so a token rejected with 401 on every endpoint of a
clustered PVE connection surfaced as `state: "unreachable"` /
`adapterHealth: "blocked"` instead of `state: "unauthorized"` /
`credentialStatus: "invalid"` — the same Settings → Connections
brokenness the rest of today's commits set out to remove.
Single-node `pve:pi` already classified the same kind of failure
correctly because its error came straight from the per-instance
client; only the cluster wrapper masked it.
Surface each unhealthy endpoint's already-sanitized reason in the
outer error. The "no healthy nodes available" prefix is preserved so
existing callers that test for it (monitor_polling_storage.go,
internal cluster_client passthroughs, existing tests) keep working.
Add a regression test covering both shapes:
- all endpoints failed auth → wrapped error contains
"Authentication failed" so the aggregator regex now matches.
- endpoint with no recorded reason → wrapped error includes the
fallback "no recorded reason" text rather than a bare URL.
Pulse was generating tag colours from a hash of the tag name instead
of using the colours configured in Proxmox. Now polls /cluster/options
once per PVE instance and merges the tag-style colour map into state,
which the frontend uses as the first-priority colour source for tag
badges. Falls back to the existing special-tag and hash-based colours
when Proxmox hasn't set a custom colour for a tag.
Proxmox status.Mem includes page cache as "used" memory, inflating
reported VM usage. The existing fallbacks (balloon meminfo, RRD, linked
host agent) were frequently unavailable, causing most VMs to fall
through to the inflated status-mem source.
Adds a new last-resort fallback that reads /proc/meminfo via the QEMU
guest agent file-read endpoint to get accurate MemAvailable. Results
are cached (60s positive, 5min negative backoff for unsupported VMs).
Also fixes: RRD memavailable fallback missing from traditional polling
path, cache key collisions in multi-PVE setups, FreeMem underflow
guard inconsistency, and integer overflow in kB-to-bytes conversion.
Backport of v6 commits a87c9950 and 347d7db1.
Part 1 (a87c9950): Wrap the four guest agent c.get() errors with
fmt.Errorf("guest agent ...: %w", err) so isVMSpecificError() correctly
scopes them to the VM rather than the cluster endpoint.
Part 2 (347d7db1): Replace the 20+ pattern blocklist in
executeWithFailover with an allowlist via isEndpointConnectivityError().
Only true TCP/DNS/TLS failures mark an endpoint unhealthy. Any HTTP
response from Proxmox — including 500 — proves the node is reachable
and returns the error without affecting endpoint health.
Linux VM page cache (#1270): QEMU VM memory now falls back to Proxmox
RRD's memavailable metric (which excludes reclaimable page cache) when
the qemu-guest-agent doesn't provide MemInfo.Available. Previously the
fallback was detailedStatus.Mem (total - MemFree), inflating usage to
80%+ on VMs with normal Linux page cache. Mirrors the existing LXC
rrd-memavailable path.
FreeBSD ZFS ARC (#1264, #1051): The host agent now reads
kstat.zfs.misc.arcstats.size via SysctlRaw on FreeBSD and subtracts
the ARC size from reported memory usage. ZFS ARC is reclaimable under
memory pressure (like Linux SReclaimable) but gopsutil counts it as
wired/non-reclaimable, causing false 90%+ memory alerts on TrueNAS
and FreeBSD hosts. Build-tagged so it compiles cleanly on all platforms.
Fixes#1270Fixes#1264Fixes#1051
(cherry picked from commit 94502f83ff9ffc6da28aaadc946a2f7d8b4e9bac)
- Add PendingUpdates and PendingUpdatesCheckedAt fields to Node model
- Add GetNodePendingUpdates method to Proxmox client (calls /nodes/{node}/apt/update)
- Add 30-minute polling cache to avoid excessive API calls
- Add pendingUpdates to frontend Node type
- Add color-coded badge in NodeSummaryTable (yellow: 1-9, orange: 10+)
- Update test stubs for interface compliance
Requires Sys.Audit permission on Proxmox API token to read apt updates.
- Add comprehensive tests for internal/api/config_handlers.go (Phases 1-3)
- Improve test coverage for AI tools, chat service, and session management
- Enhance alert and notification tests (ResolvedAlert, Webhook)
- Add frontend unit tests for utils (searchHistory, tagColors, temperature, url)
- Add proximity client API tests
Previously, when some cluster endpoints were unreachable (e.g., backup
nodes intentionally offline), the cluster was marked as "degraded" even
though the Proxmox cluster itself was healthy and had quorum.
Now the connection health check queries the Proxmox cluster's actual
quorum status. A cluster is only marked "degraded" if it has lost
quorum (not enough votes for consensus), which is the actual indicator
of cluster instability.
This means:
- Cluster with quorum + some nodes offline = "healthy"
- Cluster without quorum = "degraded" (warning)
- All endpoints down = "error"
Fixes#1085
When discovering cluster nodes, Pulse now automatically prefers IPs
on the same subnet as the initial connection. This fixes the common
issue where Pulse used internal cluster network IPs (e.g., 172.x.x.x)
instead of management network IPs (e.g., 10.x.x.x).
How it works:
1. Extract subnet from initial connection URL (assumes /24 for IPv4)
2. For each discovered node, query /nodes/{node}/network for all IPs
3. If cluster-reported IP is on a different subnet, find an IP on
the preferred subnet and set it as IPOverride
4. Manual IPOverride settings are preserved and take precedence
This eliminates the need for manual IPOverride configuration in most
multi-network Proxmox setups.
Refs #929, #1066
Changes:
1. Add MAX_POLL_TIMEOUT env var for large Proxmox clusters that need
more than 3 minutes for polling (default: 3m, minimum: 30s)
2. Handle external Ceph storage gracefully - don't mark nodes unhealthy
when Proxmox returns 'binary not installed' (e.g., for Ceph not
managed by Proxmox)
Related to #965
- Add comprehensive debug logging to diagnose replication status fetch failures
- Handle both array and single-object response formats from Proxmox API
- Log raw response body for easier debugging
- Log success/failure for each enrichment step
This helps diagnose issue #992 where replication last/next sync times aren't
showing. The logging will reveal if the API call is failing, returning empty
data, or returning data in an unexpected format.
Related to #992
Ensures that LinkedHostAgentId, CommandsEnabled, IsLegacy, and LinkedNodeId
are correctly propagated to the frontend. This prevents regressions of the
bugs fixed for #952 and #971.
- Add persistent volume mounts for Go/npm caches (faster rebuilds)
- Add shell config with helpful aliases and custom prompt
- Add comprehensive devcontainer documentation
- Add pre-commit hooks for Go formatting and linting
- Use go-version-file in CI workflows instead of hardcoded versions
- Simplify docker compose commands with --wait flag
- Add gitignore entries for devcontainer auth files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The /cluster/replication endpoint only returns job configuration (guest,
schedule, source, target), not status data (last_sync, next_sync,
duration, fail_count, state).
This fix enriches each replication job with status from the per-node
endpoint /nodes/{node}/replication/{id}/status to get timing and state
data needed for proper UI display.
Added integration tests to verify:
- Status endpoint is called and data is merged correctly
- Graceful handling when status endpoint fails
Fixes#992
PBS storage content queries with encrypted backups can take 10-20+ seconds
to enumerate. The previous 30s timeout was causing intermittent failures
when polling backup data from PBS storage configured in PVE.
This increases the timeout to 60s to accommodate slow PBS backends while
still preventing indefinite hangs on unavailable NFS/network storage.
When a PVE cluster has unique self-signed certificates on each node, Pulse
would mark secondary nodes as unhealthy because only the primary node's
fingerprint was used for all connections.
Now, during cluster discovery, Pulse captures each node's TLS fingerprint
and uses it when connecting to that specific node. This enables
"Trust On First Use" (TOFU) for clusters with unique per-node certs.
Changes:
- Add Fingerprint field to ClusterEndpoint config
- Add FetchFingerprint() to tlsutil for capturing node certs
- validateNodeAPI() now captures and returns fingerprints during discovery
- NewClusterClient() accepts endpointFingerprints map for per-node certs
- All client creation paths use per-endpoint fingerprints when available
Related to #879
courtmanr@gmail.com