1830 Commits

Author SHA1 Message Date
rcourtman 18944b0350 Fix resolved notifications for state alerts
Refs #1350
2026-05-24 23:27:45 +01:00
rcourtman da38263ea4 Prefer linked host-agent VM disk inventory
Refs #1438
2026-05-24 23:07:24 +01:00
rcourtman 469a8e1bdc Count Ceph monitors from detailed monmap data
Refs #1290
2026-05-24 22:32:58 +01:00
rcourtman 8046b5035d Prefer Proxmox guest-agent privileges
Refs #1466
2026-05-24 22:28:29 +01:00
rcourtman 117ec901da Clarify PBS alert threshold identity
Refs #1448
2026-05-24 22:13:58 +01:00
rcourtman a8f6dbd8b4 Prefer host-agent temperature data over SSH fallback
Refs #1473
2026-05-24 22:04:36 +01:00
rcourtman 8600706da3 Fix FreeBSD agent restart handling
Refs #1457
2026-05-24 21:51:11 +01:00
rcourtman 0dca8a0375 Keep snapshot polling independent of backup scans
Refs #1437
2026-05-24 21:41:56 +01:00
rcourtman 6f3bea32ff Fix Ceph pool alert threshold targets
Refs #1341
2026-05-24 21:35:46 +01:00
rcourtman 2dffa40379 Fix v5 user-reported monitoring and auth issues
Refs #1452

Refs #1455

Refs #1467

Refs #1471

Refs #1472

Refs #1477

Refs #1478

Refs #1479
2026-05-24 21:07:55 +01:00
rcourtman e503863c9e Fix guest metadata migration race
Refs #1435
2026-05-17 09:43:42 +01:00
rcourtman 8337cbc4c9 Fix v5 diagnostics GitHub export
Normalize diagnostics collection fields to empty arrays before encoding and harden the sanitized GitHub export path against null arrays so empty v5 installs can still produce issue attachments.

Refs #1454
2026-05-03 19:12:24 +01:00
rcourtman 3d3b1a9642 Stop re-notification spam when alert cooldown is disabled (Fixes #1444)
shouldNotifyAfterCooldown previously returned true on every call when
Schedule.Cooldown was 0 or negative, which the alert evaluation loop
runs on every metric tick. With cooldown disabled, an active alert was
re-notified on each tick.

The UI labels cooldown=0 as "Disabled," so the intuitive contract is
"do not re-notify," not "re-notify continuously." Treat <=0 as
"first-time only": fire the initial notification, then suppress
subsequent re-notifications until the alert clears or the cooldown is
configured to a positive value. Level escalation re-notifications
remain handled at the call site and are unaffected.

Tests cover all three branches: first-time fire with cooldown=0,
re-notification suppression with cooldown=0 (named regression guard
for #1444), and the same behavior for negative values.
2026-05-01 15:04:27 +01:00
rcourtman 4a5e234c12 Carry forward previous snapshots for guests we cannot poll this cycle
When the snapshot-polling budget runs out mid-loop, or a single guest's
GetVMSnapshots/GetContainerSnapshots call returns an error, the polling
function used to early-return without writing any state. That meant:

  1. snapshots successfully fetched for earlier guests in the same
     cycle were thrown away, and
  2. on the next successful cycle, the freshly-polled snapshots
     replaced the entire instance's snapshot list — wiping out any
     snapshots whose owning VM had failed to respond this round.

For users with a busy production cluster (many guests, intermittent
per-VM API failures), this manifests as "new snapshots never appear
in the Backups tab" because the failing VM keeps blanking the list
the moment a successful poll lands (#1437).

Now we read the previous snapshots for the instance up front, track
which guests we successfully polled this cycle, and at the end merge
the fresh data with previously-known snapshots for any guest we
couldn't reach. Successfully-polled guests get their fresh data so
new snapshots appear; failed guests keep their last-known list so
transient errors do not blank state. The early-return on deadline is
removed so the merge runs even on partial-failure cycles.

Tests cover the carry-forward path: a fresh successful poll for one
VM lands a new snapshot, and a concurrent failed poll for a second
VM preserves its previously-known snapshot rather than dropping it.
2026-04-30 11:43:01 +01:00
rcourtman a53de0fc53 Surface unified-agent filesystems in linked VM/container Overview
The qemu-guest-agent's get-fsinfo cannot reliably report ZFS mounts on
some guest configurations (notably Proxmox Backup Server), so VMs that
have ZFS-formatted partitions show only their EXT4 root and datastore
in the VM Overview FILESYSTEMS card while the much larger ZFS dataset
holding the actual backups is missing entirely (Fixes #1438).

The unified pulse-agent running inside the same guest already has
direct OS-level visibility into every mounted filesystem, including
ZFS, and Pulse already knows the link between the host agent and its
guest via Host.LinkedVMID / Host.LinkedContainerID (set in
findLinkedProxmoxEntity by hostname match).

GetState now calls StateSnapshot.MergeLinkedHostDisksIntoGuests after
producing the snapshot. For each Host with a linked VM or container,
that helper:

  1. appends host-agent disks to the guest's Disks slice, deduped by
     mountpoint (qemu-guest-agent entries take precedence so we don't
     overwrite per-VM-perspective values), and
  2. updates the guest's aggregate Disk.{Total,Used,Free,Usage} to
     include the newly-added partitions so the row total stays
     consistent with the partitions visible in the FILESYSTEMS card.

The merge runs on a defensive copy of the disks slice to avoid
mutating the underlying State slice that GetSnapshot shallow-copies.
Tests cover the happy path (PBS-shaped fixture mirroring the issue
screenshots), the no-link no-op, container linking, empty-mountpoint
filtering, and the slice-isolation invariant.
2026-04-30 11:24:47 +01:00
rcourtman 5c65f65a90 Pass keep_alive=30s to Ollama so the model unloads between Patrol runs
Ollama keeps the loaded model in RAM for 5 minutes by default after
each request, and every new request refreshes that 5-minute window.
Pulse never passed keep_alive, so any Ollama traffic (Patrol, alert
analysis, AI chat) within 5 minutes of the previous request kept the
model warm — and on a server with continuous Pulse activity that
meant the model never unloaded, even with Patrol set to a 24-hour
interval (Fixes #1425).

Pass keep_alive=30s on every Chat and ChatStream request. Short
enough that the model unloads shortly after a Patrol burst or
one-shot analysis ends, long enough to span the small gaps between
sequential calls within a single analysis session (so the model is
not reloaded mid-burst).

Tests assert that both the streaming and non-streaming Chat paths
include the keep_alive field in the Ollama request body.
2026-04-30 10:59:04 +01:00
rcourtman 012c25d604 Use /proc/mdstat operation type to gate RAID rebuilding alerts
Distinguish a real rebuild ("recovery" after disk replacement) from
routine maintenance ("check" data scrubs, "resync" after unclean
shutdown) using the in-progress sync action from /proc/mdstat. The
mdadm --detail State field does not reliably surface scrub state on
all kernel/distribution combinations (notably Synology DSM), which is
why scheduled scrubs were firing "RAID array is rebuilding" warnings
every 30 seconds (Fixes #1446).

The mdadm parser now extracts the operation keyword from the
/proc/mdstat progress line and surfaces it as RAIDArray.Operation
alongside the existing speed parse. The alert layer treats "recovery"
and "reshape" as rebuild signals; "check" and "resync" are treated as
maintenance and do not fire an alert. Stringy State matching is kept
as a backstop for arrays without a /proc/mdstat progress line, but
"resync" alone in State no longer counts as a rebuild signal.

Threaded the new field through the host-agent report, the resources
converter, and the monitor's models conversion. Added /proc/mdstat
parser tests covering recovery/check/resync/reshape/idle, and
end-to-end alert tests for recovery (alerts), check (silent scrub),
and resync (silent maintenance).
2026-04-30 10:37:47 +01:00
rcourtman 0464bdbad0 Stop test-config sends from leaking stale auth into shared SMTP manager
When the email config passed to sendHTMLEmailWithError differs from the
manager's persisted config (a test send with edited but unsaved
settings), build a fresh manager so stale Username, Password,
AuthRequired, SMTPHost, SMTPPort, TLS, StartTLS, or Provider fields
cannot leak into the SMTP exchange. The shared production manager is
left untouched.

Without this, a relay-mode test (port 25, no credentials) on a
deployment that previously had authenticated SMTP saved would still
attempt AUTH and fail with "AUTH not available" because the manager's
old AuthRequired and credentials persisted (Fixes #1440).

When the configs match, the existing reuse path is preserved so the
production manager's rate limiter keeps working across grouped sends.
2026-04-30 10:28:47 +01:00
kanylbullen 4557fb8159 Refactor: extract emitFinalToolCalls helper, add EOF tests
Address review feedback:
- Extract shared tool-call finalization into emitFinalToolCalls closure
  to eliminate duplication between [DONE] and EOF-fallback paths
- Build tool calls in deterministic index order (sorted)
- Normalize stopReason consistently in both paths
- Add unit tests:
  - TestOpenAIClient_ChatStream_ToolCallWithSimultaneousEOF: verifies
    tool calls are parsed when Read returns n>0 and io.EOF together
  - TestOpenAIClient_ChatStream_ToolCallWithoutDONE: verifies fallback
    emission when stream ends without [DONE]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-30 10:05:59 +01:00
kanylbullen c9bbe8b3a8 Fix SSE stream parser dropping tool calls on EOF
The read loop in ChatStream breaks immediately on io.EOF without
processing remaining buffered data. Per Go's io.Reader contract,
Read may return both n > 0 and io.EOF simultaneously, so the final
bytes (which may contain tool call deltas and [DONE]) are silently
discarded.

This causes the agentic loop to see tool_calls=0 even though the
model correctly produced tool calls in the stream.

Changes:
- Process pendingData when EOF is received before breaking
- Add fallback: emit accumulated tool calls if [DONE] was never
  reached (server closed connection early)

Fixes #1411

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-30 10:05:59 +01:00
rcourtman 7e4d4e07bf Persist QNAP agent updates to data volume (Fixes #1420)
On QNAP, /usr/local/bin is a tiny RAM disk that gets wiped on every
reboot. The install wrapper stores the real binary under
${QNAP_VOL}/.pulse-agent/<name> and a boot script copies it back into
/usr/local/bin. Without refreshing the stored copy, auto-updates applied
to the RAM disk were silently reverted on the next reboot.

Mirror the Unraid persistence pattern: after the atomic in-place swap,
when running on QNAP, rewrite the stored binary via a temp-file rename.
Skip when the running binary already is the persistent copy (fallback
mode, where the rename step already updated it).
2026-04-17 11:44:17 +01:00
rcourtman e1011230b9 Align infra discovery with Patrol interval
The infra discovery service auto-started with a hardcoded 5-minute
ticker the moment the AI service initialized, regardless of the user's
Patrol schedule. Each tick called AnalyzeForDiscovery, which hit the
Ollama chat endpoint and reset Ollama's keep_alive (5 min default), so
the model never had a chance to unload between requests.

Default the discovery interval to 24h and align it with the user's
Patrol preset (GetPatrolInterval) when the AI service constructs the
discovery service. With Patrol at its 6h default, the LLM now sits idle
long enough for Ollama to release it.

Fixes #1425
2026-04-17 11:10:14 +01:00
rcourtman 4de1c3745a Preflight disk space before Pulse updates 2026-04-15 20:56:58 +01:00
rcourtman 81661a934a Move v5 maintenance flow onto release/5.1 2026-04-14 18:34:41 +01:00
rcourtman dfbe2eb873 Suppress noisy recovery notifications 2026-04-13 14:40:12 +01:00
rcourtman 19b2a4e4c4 Clear stale guest per-disk alerts 2026-04-13 14:20:54 +01:00
rcourtman 1f0dfd60fc Lock SAML metadata public URL refresh 2026-04-13 13:48:27 +01:00
rcourtman 9fb76579cc Fix backup type-aware orphan detection 2026-04-13 11:54:46 +01:00
rcourtman 3981df57a2 Detect NAS host vendors from platform files 2026-04-13 11:25:27 +01:00
rcourtman 754aa0e39c Fix linked host agent threshold overrides 2026-04-12 22:47:34 +01:00
rcourtman 5f3a4b79ba Fix oversized AI discovery responses 2026-04-12 22:33:48 +01:00
rcourtman 95409985b5 Normalize vendor-managed NAS RAID arrays 2026-04-12 22:20:04 +01:00
rcourtman a86c7120cf Debounce recovery for poll-driven offline alerts 2026-04-12 22:04:10 +01:00
rcourtman 005f64182f Respect quiet hours for escalation alerts
Apply quiet-hours suppression to escalation notifications so offline and other suppressed categories do not bypass the normal notification rules during escalation.

Fixes #1398.
2026-04-12 21:29:32 +01:00
rcourtman 22f38c84d9 Fix Podman container CPU calculation
Prefer Podman's reported CPU percentage from the compat stats payload and fall back to Podman's wall-clock calculation instead of Docker's multi-core normalization.

Fixes #1391.
2026-04-09 16:17:12 +01:00
rcourtman 30eb9d7847 Fix repeated Docker update recovery alerts
Preserve Docker container update alerts and first-seen tracking when update status is temporarily unavailable or the registry check fails.

Fixes #1394.
2026-04-09 15:59:15 +01:00
rcourtman 205376594b Fail closed on unknown AI read commands 2026-04-09 14:33:54 +01:00
rcourtman a4834ed80f Disambiguate linked host agent alert names 2026-04-07 10:50:52 +01:00
rcourtman 7db110e15a Preserve Proxmox node disk source over storage fallback 2026-04-05 23:35:54 +01:00
rcourtman 398ef8117b Clear stale storage alerts on inventory changes 2026-04-05 23:35:54 +01:00
rcourtman edc5a99d9b Block unspecified webhook addresses 2026-04-05 23:35:54 +01:00
rcourtman 22f8dd3bbb Normalize typed SMART device labels for temp merge 2026-04-05 23:35:54 +01:00
rcourtman b83f717ce7 Prefer root mountpoint for ZFS summaries 2026-04-05 23:35:54 +01:00
rcourtman 678f528961 Preserve OIDC group fields on SSO updates 2026-04-05 23:35:54 +01:00
rcourtman af486b6f55 Default legacy alert schedule settings 2026-04-05 23:35:54 +01:00
rcourtman b524659f03 Add patrol trigger disabled-state coverage 2026-04-05 23:35:54 +01:00
rcourtman 48bdfdc30b Use node-local counts for node-scoped swarm services 2026-04-05 23:35:54 +01:00
rcourtman aa69bd2d8e Add resource alert stats regression coverage 2026-04-05 23:35:54 +01:00
rcourtman 1430b52b97 Hydrate shared Ceph storage usage before alerts 2026-04-05 23:35:54 +01:00
rcourtman 0c2de2938f Fix RC upgrade ordering and OPNsense SMART temps 2026-04-05 23:35:54 +01:00