The pollPVEInstance error path returned without touching node state, so
a host that stopped answering kept showing its last online snapshot
forever (#1441, reported against v5 in #1135), and a host that was
already down when Pulse started never appeared on the dashboard at all
(#1433).
Route the poll error through the same grace policy as the existing
empty-node fallback: nodes seen within the grace window keep their
prior state with degraded connection health, then flip to offline with
cleared uptime and CPU once the window lapses. An instance with no node
state yet gets the synthesized offline entry from removeFailedPVENode
so it still shows on the dashboard.
Backport of the v6 fixes 02955619f and 8372a22c5.
The agent self-updater's HTTP client followed redirects with no
CheckRedirect policy. Go's net/http strips Authorization/Cookie on a
cross-host redirect but forwards custom headers such as X-API-Token, so a
cross-origin 30x from the operator-configured PulseURL (open redirect,
compromised/hijacked server, DNS/BGP hijack, or a malicious provisioning
URL) leaked the agent API token to the redirect target (CWE-200/CWE-522).
Backport of the v6 fix: refuse to follow any redirect rather than
re-sending the token. Covers both the getServerVersion version-check path
and the binary-download path, with regression tests asserting the redirect
target is never hit.
Reported by tonghuaroot.
Follow-up to the #1341 dual-source fix. In a Proxmox cluster the
Proxmox-API instance name (cluster/connection name) differs from the
host-agent's node hostname, so a pool's two source identities share no
prefix (e.g. `prodcluster-ceph-pool-X` vs `agent:pve5-ceph-pool-X`). The
per-pool override is saved under one identity. The previous fix only
toggled the `agent:` prefix, which bridges a single node but not a
cluster, so when the deduped winner was the API copy the row found no
override and showed the storage default. With the cluster's API Ceph
source intermittently dropping (the same clustered environment as #1319),
the winner oscillated and the threshold appeared to flap between the
custom value (50%) and the default (80%) on the polling cadence.
Carry every source instance for an FSID as `CephCluster.InstanceAliases`
(populated in DedupeCephClusters) and emit the pool ID under each as
`Storage.AliasIDs`. Override resolution (alert evaluation and the
thresholds UI) now tries the primary ID plus every alias, so a per-pool
override resolves regardless of which source currently wins. This mirrors
v6's source-independent override identity. Non-breaking: existing keys
resolve via aliases, no re-entry needed.
Follow-up to the #1341 dual-source fix. When the Proxmox API identity
wins the FSID dedup, proactively clear any usage alert still active under
this agent's own pool IDs so a pre-fix duplicate clears on the next agent
report rather than aging out over the ~24h stale-alert window. Agent-only
clusters (where the agent identity is the winner) are untouched.
Fixes#1341 (reopened)
When the same Ceph cluster is reported by both the Proxmox API poller
(instance "pve5") and a Pulse host-agent (instance "agent:pve5"), it
landed in state twice under two pool-ID namespaces
("pve5-ceph-pool-X" vs "agent:pve5-ceph-pool-X"). The frontend collapsed
them via an FSID dedup whose winner was chosen by a fluctuating
completeness score, but alert evaluation ran on the raw, un-deduped list
(and ce1607694 added a third check on the raw agent cluster). So:
- the threshold UI row identity flipped between the two pool IDs as the
dedup winner oscillated, making a saved per-pool override appear to
revert between the custom value and the storage default; and
- both pool IDs were alert-checked under independent lifecycles,
producing duplicate, flapping active alerts.
Make alert evaluation and the frontend consume the same deterministic
FSID-deduplicated Ceph view (DedupeCephClusters), preferring the
authoritative non-agent identity so a pool has exactly one ID. Route the
agent-report and mock alert paths through GetDedupedCephClusters, and
honor the legacy "agent:"-prefixed override key for the surviving pool so
existing thresholds keep firing without manual re-entry. Retires the
ce1607694 agent-only check path.
Centralize the QEMU guest-agent MemAvailable fallback on the 5.1 branch and record the direct guest-agent value in diagnostic snapshots.
Add regression coverage for the issue #1319 saturated Linux memory payload and Windows fsinfo volume payload.
Refs #1319
Fixes#1341
When a Pulse host-agent reports Ceph data, ApplyHostReport upserts the
cluster into state but only the Proxmox-API polling path ran
cephPoolAlertStorageTargets. Users with agent-reported Ceph (instance
prefix "agent:hostname") saved per-pool overrides under
agent-prefixed IDs that the alert manager never evaluated. The
threshold appeared to save (and showed Custom in the UI), but the
polling cycle was checking a different storage ID, so the alert
silently stayed dormant. Run CheckStorage for each pool right after
the agent upsert so the override key actually drives evaluation.
Refs #1341
UpdateAlertConfig used to log SaveAlertConfig failures and still tell
the client "saved successfully", leaving the in-memory state with the
new override but the on-disk file untouched. On the next config reload
or process restart, the override silently vanished and the user saw
their threshold "revert" with no surfaced error. Return HTTP 500 with
the persistence error so the frontend can show a real save-failed
toast instead of false confidence.
Refs #1341
buildAlertsDiagnostic previously emitted only boolean flags (legacy
thresholds, missing cooldown/grouping window). Override keys and their
trigger values were absent, so triaging a support case where the user
suspects an override mismatch required asking them to paste
data/alerts.json from inside their container. Add an Overrides slice
that names each persisted key with its thresholds and disabled flags.
Sanitize mode in the frontend redacts the keys to override-N while
keeping thresholds visible, so a public export still shows the alert
shape without leaking instance names that may be hostnames.
Refs #1341
The 5.1.32 fix in 6f3bea32f only tested the storage default path for
synthetic Ceph pool entries. Reporter on #1341 sets a 50% per-pool
override on a pool at ~61% usage and still doesn't get an alert; lock
the override path against future regressions so this remains a config
question rather than a code regression.
Normalize diagnostics collection fields to empty arrays before encoding and harden the sanitized GitHub export path against null arrays so empty v5 installs can still produce issue attachments.
Refs #1454
shouldNotifyAfterCooldown previously returned true on every call when
Schedule.Cooldown was 0 or negative, which the alert evaluation loop
runs on every metric tick. With cooldown disabled, an active alert was
re-notified on each tick.
The UI labels cooldown=0 as "Disabled," so the intuitive contract is
"do not re-notify," not "re-notify continuously." Treat <=0 as
"first-time only": fire the initial notification, then suppress
subsequent re-notifications until the alert clears or the cooldown is
configured to a positive value. Level escalation re-notifications
remain handled at the call site and are unaffected.
Tests cover all three branches: first-time fire with cooldown=0,
re-notification suppression with cooldown=0 (named regression guard
for #1444), and the same behavior for negative values.
When the snapshot-polling budget runs out mid-loop, or a single guest's
GetVMSnapshots/GetContainerSnapshots call returns an error, the polling
function used to early-return without writing any state. That meant:
1. snapshots successfully fetched for earlier guests in the same
cycle were thrown away, and
2. on the next successful cycle, the freshly-polled snapshots
replaced the entire instance's snapshot list — wiping out any
snapshots whose owning VM had failed to respond this round.
For users with a busy production cluster (many guests, intermittent
per-VM API failures), this manifests as "new snapshots never appear
in the Backups tab" because the failing VM keeps blanking the list
the moment a successful poll lands (#1437).
Now we read the previous snapshots for the instance up front, track
which guests we successfully polled this cycle, and at the end merge
the fresh data with previously-known snapshots for any guest we
couldn't reach. Successfully-polled guests get their fresh data so
new snapshots appear; failed guests keep their last-known list so
transient errors do not blank state. The early-return on deadline is
removed so the merge runs even on partial-failure cycles.
Tests cover the carry-forward path: a fresh successful poll for one
VM lands a new snapshot, and a concurrent failed poll for a second
VM preserves its previously-known snapshot rather than dropping it.
The qemu-guest-agent's get-fsinfo cannot reliably report ZFS mounts on
some guest configurations (notably Proxmox Backup Server), so VMs that
have ZFS-formatted partitions show only their EXT4 root and datastore
in the VM Overview FILESYSTEMS card while the much larger ZFS dataset
holding the actual backups is missing entirely (Fixes#1438).
The unified pulse-agent running inside the same guest already has
direct OS-level visibility into every mounted filesystem, including
ZFS, and Pulse already knows the link between the host agent and its
guest via Host.LinkedVMID / Host.LinkedContainerID (set in
findLinkedProxmoxEntity by hostname match).
GetState now calls StateSnapshot.MergeLinkedHostDisksIntoGuests after
producing the snapshot. For each Host with a linked VM or container,
that helper:
1. appends host-agent disks to the guest's Disks slice, deduped by
mountpoint (qemu-guest-agent entries take precedence so we don't
overwrite per-VM-perspective values), and
2. updates the guest's aggregate Disk.{Total,Used,Free,Usage} to
include the newly-added partitions so the row total stays
consistent with the partitions visible in the FILESYSTEMS card.
The merge runs on a defensive copy of the disks slice to avoid
mutating the underlying State slice that GetSnapshot shallow-copies.
Tests cover the happy path (PBS-shaped fixture mirroring the issue
screenshots), the no-link no-op, container linking, empty-mountpoint
filtering, and the slice-isolation invariant.
Ollama keeps the loaded model in RAM for 5 minutes by default after
each request, and every new request refreshes that 5-minute window.
Pulse never passed keep_alive, so any Ollama traffic (Patrol, alert
analysis, AI chat) within 5 minutes of the previous request kept the
model warm — and on a server with continuous Pulse activity that
meant the model never unloaded, even with Patrol set to a 24-hour
interval (Fixes#1425).
Pass keep_alive=30s on every Chat and ChatStream request. Short
enough that the model unloads shortly after a Patrol burst or
one-shot analysis ends, long enough to span the small gaps between
sequential calls within a single analysis session (so the model is
not reloaded mid-burst).
Tests assert that both the streaming and non-streaming Chat paths
include the keep_alive field in the Ollama request body.
Distinguish a real rebuild ("recovery" after disk replacement) from
routine maintenance ("check" data scrubs, "resync" after unclean
shutdown) using the in-progress sync action from /proc/mdstat. The
mdadm --detail State field does not reliably surface scrub state on
all kernel/distribution combinations (notably Synology DSM), which is
why scheduled scrubs were firing "RAID array is rebuilding" warnings
every 30 seconds (Fixes#1446).
The mdadm parser now extracts the operation keyword from the
/proc/mdstat progress line and surfaces it as RAIDArray.Operation
alongside the existing speed parse. The alert layer treats "recovery"
and "reshape" as rebuild signals; "check" and "resync" are treated as
maintenance and do not fire an alert. Stringy State matching is kept
as a backstop for arrays without a /proc/mdstat progress line, but
"resync" alone in State no longer counts as a rebuild signal.
Threaded the new field through the host-agent report, the resources
converter, and the monitor's models conversion. Added /proc/mdstat
parser tests covering recovery/check/resync/reshape/idle, and
end-to-end alert tests for recovery (alerts), check (silent scrub),
and resync (silent maintenance).
When the email config passed to sendHTMLEmailWithError differs from the
manager's persisted config (a test send with edited but unsaved
settings), build a fresh manager so stale Username, Password,
AuthRequired, SMTPHost, SMTPPort, TLS, StartTLS, or Provider fields
cannot leak into the SMTP exchange. The shared production manager is
left untouched.
Without this, a relay-mode test (port 25, no credentials) on a
deployment that previously had authenticated SMTP saved would still
attempt AUTH and fail with "AUTH not available" because the manager's
old AuthRequired and credentials persisted (Fixes#1440).
When the configs match, the existing reuse path is preserved so the
production manager's rate limiter keeps working across grouped sends.
Address review feedback:
- Extract shared tool-call finalization into emitFinalToolCalls closure
to eliminate duplication between [DONE] and EOF-fallback paths
- Build tool calls in deterministic index order (sorted)
- Normalize stopReason consistently in both paths
- Add unit tests:
- TestOpenAIClient_ChatStream_ToolCallWithSimultaneousEOF: verifies
tool calls are parsed when Read returns n>0 and io.EOF together
- TestOpenAIClient_ChatStream_ToolCallWithoutDONE: verifies fallback
emission when stream ends without [DONE]
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The read loop in ChatStream breaks immediately on io.EOF without
processing remaining buffered data. Per Go's io.Reader contract,
Read may return both n > 0 and io.EOF simultaneously, so the final
bytes (which may contain tool call deltas and [DONE]) are silently
discarded.
This causes the agentic loop to see tool_calls=0 even though the
model correctly produced tool calls in the stream.
Changes:
- Process pendingData when EOF is received before breaking
- Add fallback: emit accumulated tool calls if [DONE] was never
reached (server closed connection early)
Fixes#1411
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On QNAP, /usr/local/bin is a tiny RAM disk that gets wiped on every
reboot. The install wrapper stores the real binary under
${QNAP_VOL}/.pulse-agent/<name> and a boot script copies it back into
/usr/local/bin. Without refreshing the stored copy, auto-updates applied
to the RAM disk were silently reverted on the next reboot.
Mirror the Unraid persistence pattern: after the atomic in-place swap,
when running on QNAP, rewrite the stored binary via a temp-file rename.
Skip when the running binary already is the persistent copy (fallback
mode, where the rename step already updated it).
The infra discovery service auto-started with a hardcoded 5-minute
ticker the moment the AI service initialized, regardless of the user's
Patrol schedule. Each tick called AnalyzeForDiscovery, which hit the
Ollama chat endpoint and reset Ollama's keep_alive (5 min default), so
the model never had a chance to unload between requests.
Default the discovery interval to 24h and align it with the user's
Patrol preset (GetPatrolInterval) when the AI service constructs the
discovery service. With Patrol at its 6h default, the LLM now sits idle
long enough for Ollama to release it.
Fixes#1425
Apply quiet-hours suppression to escalation notifications so offline and other suppressed categories do not bypass the normal notification rules during escalation.
Fixes#1398.
Prefer Podman's reported CPU percentage from the compat stats payload and fall back to Podman's wall-clock calculation instead of Docker's multi-core normalization.
Fixes#1391.
rcourtman