Backport from v6 (88d5865a8). Recovery webhook notifications were using
the firing PayloadTemplate which services like Telegram, Teams, Discord
etc. silently rejected as malformed. Now uses a three-tier template
pipeline matching the firing path:
- Tier 1: Custom user template (if configured)
- Tier 2: Service-specific ResolvedPayloadTemplate (Discord green embed,
Telegram chat_id+text, Slack header blocks, Teams MessageCard/Adaptive,
PagerDuty event_action:"resolve", Pushover, Gotify, Mattermost)
- Tier 3: Generic JSON fallback (backward compatible)
Also adds Event, ResolvedAt, ResolvedAtISO fields to WebhookPayloadData.
The backend only allows one command per host at a time. The "Update All"
button was firing requests in parallel chunks, causing the second
container per host to fail with 400. Group targets by host and process
them sequentially within each host while still updating different hosts
in parallel.
Recovery notifications for Discord, Slack, Teams, PagerDuty, and other
service webhooks were sending a generic JSON payload that lacked the
required format (e.g. Discord needs `embeds`, Slack needs `blocks`),
causing resolved notifications to silently fail.
- Add `prepareResolvedWebhookData` to build template data with Level="resolved"
- Route resolved webhooks through service-specific templates with full
URL rendering, Telegram ChatID extraction, and PagerDuty routing_key
- Custom user templates take precedence over built-in service templates
- Return errors on service template failures instead of falling back to
generic payloads that endpoints would reject
- Fix PagerDuty template to send event_action="resolve" for resolved alerts
Fire a warning alert immediately when a backup's guest no longer exists
in inventory, without requiring age thresholds to be breached. The
existing alertOrphaned toggle and ignoreVMIDs UI control this feature
with no frontend changes needed.
The isRunning prop used a `cpuPercent > 0` gate that treated idle
containers (0% CPU) as not-running, causing the bar to flip between
a percentage and an em-dash on every poll cycle. Remove the value
guard so visibility depends only on container running state, matching
how memory, disk, and restart columns already behave.
Replace first-match-only logic in upsertMentionResource with a
union-merge algorithm that collects all matching keys, merges losers
into a canonical winner, and re-points aliases. This fixes the case
where a host agent bridges a VM and a DockerHost but only the first
alias match was merged, leaving a duplicate entry in the picker.
The v4 installer added mount entries for /run/pulse-sensor-proxy to LXC
container configs. After upgrading to v5 and rebooting, /run (tmpfs) is
wiped and the container fails to start. The installer now detects and
removes these stale mp<N> and lxc.mount.entry references automatically
when run on a PVE host, and the upgrade docs include manual fix steps.
When Proxmox /nodes/{node}/status returns only total/used/free without
available/buffers/cached, EffectiveAvailable() returns Free (non-zero),
causing the RRD fallback gate to be skipped. This results in inflated
node memory where cache/buffers are counted as "used."
Widen the RRD fallback condition from requiring effectiveAvailable == 0
to triggering whenever missingCacheMetrics is true. Add negative caching
for failed RRD lookups (2-minute backoff) to avoid repeated retries.
HostFrontend was missing LinkedVmId and LinkedContainerId fields, so the
frontend dedup aliases for VM/container agents resolved to undefined and
never matched. Also add .trim() to getStatusColor and default host agent
status to 'online' to fix grey status dots.
When an agent re-registers with the same token, the DHCP matching case
would overwrite the Host field with the agent's local IP — even if the
user had edited it to a public URL or different IP. Now agent source
re-registrations always preserve the existing host, while non-agent
DHCP updates still work. Adds 5 regression tests covering hostname
preservation, public-IP preservation, agent DHCP, non-agent DHCP, and
PBS parity.
Three fixes for remaining mention autocomplete issues:
- Status dots now correctly show green/red/yellow for online/offline/
degraded statuses (previously only handled running/stopped/paused)
- Docker hosts merge with their host agent via agentId cross-reference
- VMs and LXC containers merge with host agents running inside them
via linkedVmId/linkedContainerId backend ID aliases
Users can now pass --env KEY=VALUE (repeatable) to the install script to
inject custom environment variables into the agent's service file. Useful
for KUBECONFIG and similar paths not auto-detected by the installer.
The Settings UI adds a textarea for entering env vars that get appended
to the generated install command. Both frontend and script validate key
format and reject unsafe value characters.
When a PVE node has a linked host agent (or vice versa), they now merge
into a single mention resource instead of appearing as duplicate entries.
Uses alias cross-referencing via both linkedHostAgentId and linkedNodeId
(node-backend-id) to handle one-way and two-way links.
Two changes to prevent duplicates in Settings > Virtual Environment:
1. Install script: only clear Proxmox state files on fresh installs,
not upgrades. Previously every install forced re-registration.
2. Auto-register dedup: match agent re-registrations by server name
when both the existing entry and new request have Pulse-created
tokens (pulse-monitor@pam!pulse-*). This catches the case where
the agent creates a new token after state files are cleared.
Add pencil icon + link column to the hosts overview table and the
docker unified table (containers and services), matching the existing
VM/guest URL column pattern. Uses the shared UrlEditPopover component
and existing metadata APIs. No backend changes needed.
The agent gate only allowed temperature collection on Linux (lm-sensors).
FreeBSD exposes CPU and ACPI thermal zone temperatures via sysctl
(dev.cpu.N.temperature, hw.acpi.thermal.tzN.temperature). Parse sysctl
output directly in Go without shell involvement.
Recovery notifications were silently disabled for users with pre-5.1.12
configs because the NotifyOnResolve bool field defaults to false when
absent from JSON. Use a *bool probe to detect missing field and default
to true.
Patrol trigger queue filled with warnings when the patrol loop wasn't
running. Gate TriggerPatrolForAlert on p.running and clear the flag
via defer when the loop exits.
When editing an existing webhook, header values are masked as
***REDACTED*** for security. The "Test" button in the edit form
sent these redacted values to the webhook endpoint, causing auth
failures (HTTP 403) on services like ntfy.sh that require tokens.
The test button outside the edit form worked because it used the
server-side saved config with real header values.
Fix: frontend now includes the webhook ID in test payloads for
existing webhooks, and the backend TestWebhook handler merges
redacted values with the saved originals before sending the test
(same logic already used by UpdateWebhook).
The inline URL edit was removed from the hosts table in 5.1.2 and
the only way to set a host URL was buried in the Discovery tab
(which requires AI features and a successful discovery scan).
Adds a Web Interface URL field directly in the System card of the
host drawer's Overview tab - always accessible without AI/Pro.
Users can add, edit, and remove URLs with inline editing.
Also fixes the Discovery tab using GuestMetadataAPI instead of
HostMetadataAPI for host URLs, which caused saved URLs to not
appear in the hosts table.
Two remaining issues from #1255 after the 5.1.10 fixes:
1. OIDC/SAML provider edit fields appeared blank because the GET
endpoint returned a flattened response while the frontend reads
nested oidc/saml objects. Now returns the full provider config
with secrets redacted (client secret, SP private key).
2. SSO users didn't appear in Settings > Users because RBAC entries
were only created when group-role mappings matched. Now ensures
every SSO user is registered in RBAC on login, even without
role mappings.
Also fixes: SAML SP private key and certificate lost on edit (no
preservation logic existed), OIDC client secret preservation
hardened to check actual secret presence not just flag.
Proxmox status.Mem includes page cache as "used" memory, inflating
reported VM usage. The existing fallbacks (balloon meminfo, RRD, linked
host agent) were frequently unavailable, causing most VMs to fall
through to the inflated status-mem source.
Adds a new last-resort fallback that reads /proc/meminfo via the QEMU
guest agent file-read endpoint to get accurate MemAvailable. Results
are cached (60s positive, 5min negative backoff for unsupported VMs).
Also fixes: RRD memavailable fallback missing from traditional polling
path, cache key collisions in multi-PVE setups, FreeMem underflow
guard inconsistency, and integer overflow in kB-to-bytes conversion.
The --disk-exclude agent flag only filtered local metric collection but
had no effect on server-side Proxmox disk health and SSD wearout alerts,
which poll the Proxmox API directly. Users excluding disks (e.g.
--disk-exclude sda) still received alerts for those disks.
Agent now sends its DiskExclude patterns in each report. The server
stores them on the Host model and consults them during Proxmox disk
polling — excluded disks get a synthetic healthy status passed to
CheckDiskHealth so any existing alerts clear immediately.
Also adds FreeBSD pseudo-filesystem types (fdescfs, devfs, linprocfs,
linsysfs) to the virtual FS filter and /var/run/ to special mount
prefixes, fixing false disk-full alerts on FreeBSD for fdescfs mounts.
When no guest agent MemInfo or RRD data is available, prefer the linked
Pulse host agent's memory (read from /proc/meminfo via gopsutil, which
excludes page cache) over Proxmox's status.Mem (total - free, inflated
by reclaimable cache). Applied to both efficient and traditional polling
paths. Diagnostic fields added to VMMemoryRaw for visibility.
The --cacert flag was only used for curl during installation. On systems
with custom CA certificates (e.g. TrueNAS CORE with certs in
/etc/certificates/CA), the agent process had no way to trust the custom
CA and users had to fall back to --insecure.
Set SSL_CERT_FILE in the agent's runtime environment when --cacert is
provided. Go's crypto/x509 reads this natively, so the agent trusts the
custom CA without any binary changes. All service types are covered:
systemd, upstart, launchd, FreeBSD rc.d, OpenRC, SysV init, and Unraid.
Also validates the --cacert path at install time: directories and missing
paths now fail early with a clear message instead of silently proceeding.
registerWithPulse() was a one-shot call at agent startup — if it failed
(timing, transient network, Pulse not ready), the agent silently continued
as a generic Host forever. Wrap the HTTP POST in a retry loop with
exponential backoff (5s, 10s, 20s, 40s, 60s) and distinguish 4xx errors
(no retry) from 5xx/network errors (retry).
Probe ~/.docker/run/docker.sock for RuntimeDocker and RuntimeAuto
before falling back to /var/run/docker.sock. This lets the agent
connect on macOS without requiring DOCKER_HOST to be set manually.
Ref #1200
saveToDisk used os.WriteFile which doesn't sync to disk before the
atomic rename. On CI runners with aggressive filesystem caching this
can leave the destination file with zero bytes, causing
TestKnowledgeStore_SaveLoad to fail with "unexpected end of JSON input".
syscall.SysctlRaw is Darwin-only in Go's standard library; FreeBSD
requires the equivalent from golang.org/x/sys/unix. This fixes the
Docker cross-compilation build failure for the freebsd/amd64 target.
(cherry picked from commit 5fe16c75a075b817f90b7192d8270a7bd6677017)
Extends the TrueNAS SCALE installer to also support TrueNAS CORE
(FreeBSD-based). The installer auto-detects the platform and configures
the appropriate service manager: systemd for SCALE, rc.d for CORE.
- Rename is_truenas_scale() to is_truenas() with FreeBSD detection
- Add FreeBSD rc.d service script generation with placeholder substitution
- Add FreeBSD bootstrap script for Init/Shutdown task persistence
- Split install/uninstall paths by OS throughout the TrueNAS block
- Add --cacert <path> flag for custom CA bundles (wired to curl only,
not passed to the agent binary)
- Fix --cacert incorrectly mapping to --insecure in exec args
- Fix missing closing quote on RCSCRIPT_LINK in FreeBSD bootstrap
- Fix unreachable echo after exit 0 in FreeBSD bootstrap
Co-authored-by: wilddev65 <wilddev65@users.noreply.github.com>
(cherry picked from commit affdbaeebaf2b1135431b232593122f464c6bb53)
Backport of v6 commits a87c9950 and 347d7db1.
Part 1 (a87c9950): Wrap the four guest agent c.get() errors with
fmt.Errorf("guest agent ...: %w", err) so isVMSpecificError() correctly
scopes them to the VM rather than the cluster endpoint.
Part 2 (347d7db1): Replace the 20+ pattern blocklist in
executeWithFailover with an allowlist via isEndpointConnectivityError().
Only true TCP/DNS/TLS failures mark an endpoint unhealthy. Any HTTP
response from Proxmox — including 500 — proves the node is reachable
and returns the error without affecting endpoint health.
Windows VMs and VMs without qemu-guest-agent triggered an uncached
GetVMRRDData HTTP call on every poll cycle. Add vmRRDMemCache using the
same read-through cache pattern as nodeRRDMemCache (shared rrdCacheMu,
same TTL, same cleanup path).
(cherry picked from commit 582f16004a0f275de4c458e5d288be70eee613e4)
Add per-container custom URL inputs to the expanded Docker agent row in
Settings → Agents. URLs are stored via DockerMetadataAPI with the key
format {hostId}:container:{containerId}, matching the backend migration
that preserves URLs across container image updates.
(cherry picked from commit 0babb2e546a0a1edaf3210073e3d5b6b7655239b)
rcourtman