Commit Graph

1645 Commits

Author SHA1 Message Date
rcourtman 89577fe533 Fix OIDC token refresh bypass and guard AISettingsHandler nil path
The applyAuthContextHeaders early-return in CheckAuth skipped the OIDC
token refresh block, causing long-lived OIDC sessions to expire instead
of auto-refreshing. Move the refresh trigger into extractAndStoreAuthContext
so it fires at the middleware level before CheckAuth's early return.

Also add a nil guard on mtPersistence in AISettingsHandler.GetAIService
for non-default org paths, preventing a potential panic if background
code carries a non-default org context in v5 single-tenant mode.
2026-03-06 11:05:01 +00:00
rcourtman 743ef17b79 Fix AI and config profile handlers broken in v5 single-tenant mode
The single-tenant lockdown (499ab812e) set mtPersistence to nil but
only patched AISettingsHandler with a legacy fallback. AIHandler (chat
service) and ConfigProfileHandler were missed, so AI features (Patrol,
Chat) failed with "chat service not available" and config profiles
would panic on nil dereference. Wire legacy persistence into both
handlers and add the same fallback to ProfileSuggestionHandler.

Fixes #1322
2026-03-06 11:05:01 +00:00
rcourtman 6618db7799 Fix v5 single-tenant router test setup 2026-03-05 23:58:11 +00:00
rcourtman 499ab812e3 Fix post-release regressions and lock v5 to single-tenant runtime 2026-03-05 23:46:35 +00:00
rcourtman 464d3f8486 Fix stale queued notification delivery 2026-03-05 23:46:35 +00:00
rcourtman 0493fb78bf fix(agent): detect rootless Docker/Podman sockets for other users (#1200)
When the agent runs as root, os.Getuid() returns 0 so it only probes
/run/user/0/docker.sock. Rootless Docker installs live at
/run/user/1000/docker.sock (or similar). Glob /run/user/*/docker.sock
and /run/user/*/podman/podman.sock to discover sockets for all users.
2026-03-05 10:01:43 +00:00
rcourtman 10872c8ca8 fix(patrol): remove noisy per-alert log when patrol is disabled (#1258)
The alert callback logged at Info level for every alert regardless of
whether patrol was enabled. TriggerPatrolForAlert already has an
enabled/running guard and its own debug logging.
2026-03-05 10:01:43 +00:00
rcourtman 72be883f4e fix(proxmox): prevent broken TLS config on auto-register fingerprint failure (#1303)
When FetchFingerprint fails during agent auto-registration, set verifySSL
based on whether a fingerprint was captured rather than hardcoding true.
Also heal already-broken nodes (verifySSL=true with empty fingerprint) on
legacy re-register to prevent permanent connection failures with self-signed
Proxmox certs.
2026-03-05 10:01:43 +00:00
rcourtman a3fcaafbdb test(notifications): remove queue stats race with background processor 2026-03-03 21:37:27 +00:00
rcourtman ac83074fc2 fix(hostmetrics): skip network mounts before usage probe (#1313) 2026-03-03 20:27:41 +00:00
rcourtman 8818a740e2 fix(proxmox): prevent setup-script token drift and add lifecycle integration tests (#1312) 2026-03-03 20:11:01 +00:00
rcourtman a8e562034e fix(ai): restore dismissed patrol findings and add regression tests 2026-03-03 19:53:55 +00:00
rcourtman a4571f580b fix(monitoring): harden VM memory selection and flag repeated VM usage 2026-03-03 16:19:17 +00:00
rcourtman ff9dc34687 Fix offline host visibility/alerting across restarts (#1311) 2026-03-03 15:43:29 +00:00
rcourtman b38488f2da fix(proxmox): stabilize pulse monitor token lifecycle 2026-03-03 10:57:19 +00:00
rcourtman 510ec999ab fix(api): store TLS fingerprint during auto-registration (#1303)
The legacy auto-register endpoint captured TLS fingerprints via
FetchFingerprint() but never persisted them to the node config. Nodes
with self-signed certs registered via the agent would fail with
"x509: certificate signed by unknown authority" on subsequent polls.

Store the fingerprint in all add/update paths for both PVE and PBS,
guard updates against empty-fingerprint clobber when FetchFingerprint
fails, and pass the fingerprint to cluster detection configs.
2026-03-02 14:07:18 +00:00
rcourtman 10a4e994b6 fix(api): return 404 from undismiss endpoint for invalid finding IDs (#1300)
HandleUndismissFinding now checks both patrol and unified stores
before returning. Returns 404 with error message when the finding
is not found or not dismissed, instead of silently returning success.
2026-03-02 11:48:23 +00:00
rcourtman 60bdc9a101 fix(memory): skip meminfo-derived when balloon lacks cache metrics (#1302)
When the balloon driver reports Free but not Buffers or Cached, the
meminfo-derived fallback computed memAvailable = Free alone, counting
all reclaimable page cache as used memory. This caused Linux VMs to
show wildly inflated usage (e.g. 93% when actual is 21%).

Now meminfo-derived requires at least one cache metric (Buffers > 0
or Cached > 0) before trusting the value. When missing, the code
falls through to RRD/guest-agent/Total-Used fallbacks which provide
accurate cache-aware data. Both efficient and traditional polling
paths are now consistent.
2026-03-02 11:48:18 +00:00
rcourtman d43dfbc490 feat(ui): add host removal action to hosts table
Add an actions menu to the hosts overview with a "Remove host from
Pulse" button. Includes permission checks (requires settings:write
scope), confirmation handling, and a security regression test for
the delete endpoint scope enforcement.
2026-03-01 23:28:33 +00:00
rcourtman 5bd0563283 test(providers): update Ollama integration tests for timeout parameter 2026-03-01 23:28:16 +00:00
rcourtman d46b5fc84b fix(ai): route OpenRouter slash-delimited models to OpenAI provider (#1296)
createProviderForModel() only handled "provider:model" colon format.
Models like "google/gemini-2.5-flash" or "google/gemini-2.0-flash:free"
(OpenRouter format) failed because the colon split produced invalid
provider names.

Now uses config.ParseModelString() which correctly detects slash-
delimited models as OpenRouter (routed via OpenAI-compatible API).
2026-03-01 22:29:45 +00:00
rcourtman 2fcddecf80 feat(api): add POST /api/ai/patrol/undismiss endpoint to revert suppressed findings (#1300)
The Undismiss() method existed on FindingsStore but was never exposed
via the API. Users who dismissed findings as "not_an_issue" had no way
to revert them.

- Add HandleUndismissFinding handler and route
- Add Undismiss() to UnifiedStore for parity with FindingsStore
- Also remove matching explicit suppression rules on undismiss
2026-03-01 22:29:36 +00:00
rcourtman 027fd9932c fix(proxmox): make monitor reload synchronous after auto-registration (#1303)
Auto-register was running the monitor reload in a background goroutine,
so the HTTP response was sent before the poller picked up the new node.
If reload failed or was slow, the node appeared in Settings > Proxmox
(reads config from disk) but not on the main Proxmox tab (reads from
active polling state).

Changed both auto-register paths to reload synchronously, matching the
manual add path (HandleAddNode).
2026-03-01 21:04:20 +00:00
rcourtman d852964696 fix(ai): record patrol and QuickAnalysis token usage in cost store for budget enforcement
Patrol runs, evaluation passes, and QuickAnalysis calls were consuming
LLM tokens without recording them in the cost store. This made the
cost_budget_usd_30d budget setting ineffective since enforceBudget()
never saw patrol spend.

- Add RecordUsage() to ai.Service for thread-safe cost recording
- Add recordPatrolUsage() helper to PatrolService, called on both
  success and error paths for main patrol and evaluation pass
- Record QuickAnalysis token usage in cost store
- Return partial PatrolResponse (with token counts) on error instead
  of nil, so callers can always record consumed tokens
- Propagate partial response through chat_service_adapter on error
2026-03-01 19:19:47 +00:00
rcourtman b1ff7e006f fix(ui): show PULSE_PUBLIC_URL value in settings and expand node tables to full width (#1305, #1304)
Expose PublicURL from runtime config in the system settings API response
so the frontend displays the actual value instead of the placeholder when
the env var is set.

Add w-full to PVE, PBS, and PMG node tables so they expand to fill the
container in full-width mode.
2026-03-01 14:42:30 +00:00
rcourtman c575c7e295 fix(patrol): rename wearout JSON field to ssd_life_remaining_pct (#1300)
The AI also receives disk data via tool calls (pulse_metrics type="disks"),
not just the patrol context table. The raw JSON field "wearout" was
ambiguous — rename to "ssd_life_remaining_pct" so the field name itself
communicates that 100 = healthy.
2026-02-27 23:12:27 +00:00
rcourtman 3006f51b60 fix(patrol): clarify wearout semantics so AI knows 100% = healthy (#1300)
The patrol context table header said "Wearout" and the tool returned a raw
"wearout" JSON field with no indication that 100 = full life remaining.
The AI interpreted "wearout: 100" as fully worn out and raised false
"100% Disk Wearout" findings on healthy NVMe drives.

Rename the patrol table column to "SSD Life Remaining (100%=new)" and
update the data type comment to clarify the semantics.
2026-02-27 23:05:02 +00:00
rcourtman 7530b66254 fix(setup): escape printf %s in Sprintf template to fix format verb count (#1297)
The printf '%s\n' calls in shell code within the Go Sprintf template
were being counted as format verbs, causing a build failure (10 verbs
but 9 args). Using %%s produces literal %s in the output.
2026-02-27 14:44:41 +00:00
rcourtman 62225e0c12 fix(alerts): scope orphaned backup detection per PVE instance to prevent false positives (#1286)
The previous hasLiveInventory guard was a single boolean — if any PVE
instance had at least one live guest, orphan detection ran for all
instances. In multi-instance clusters with staggered polling, backups
from instances whose VMs hadn't been polled yet appeared orphaned,
producing false positive alerts with 0m duration.

Replace the global boolean with a per-instance map. PVE storage backups
now only run orphan detection when their specific instance has live
inventory. PBS/PMG backups (which span instances) retain the "any
instance has live guests" check.
2026-02-27 13:32:15 +00:00
rcourtman 4c7a79cecb fix(setup): preserve SSH authorized_keys symlink on Proxmox and fix key entry quoting (#1297)
The PVE setup script had three bugs in the temperature monitoring SSH key setup:

- Nested double quotes in SSH_SENSORS_KEY_ENTRY broke the bash string, causing
  "No such file or directory" errors for the key options
- The grep/mv pattern to update authorized_keys destroyed the symlink that
  Proxmox maintains from /root/.ssh/authorized_keys to /etc/pve/priv/
- The uninstall path grepped for "# pulse-managed-key" but keys were tagged
  "# pulse-sensors", so uninstall never cleaned up sensor keys

Fixes: resolve symlinks with readlink -f before operating, create temp files in
/tmp with mv-then-cp fallback for cross-device moves, escape inner quotes, and
broaden the uninstall filter to match all pulse-prefixed keys.
2026-02-27 13:23:03 +00:00
rcourtman 9aee8fa293 fix(ui): add Pro badge to Reporting tab and reduce patrol trigger log noise (#1285, #1258)
Show "Pro" badge on the Reporting settings tab so users know upfront
that advanced reporting requires a Pro license, rather than discovering
it after filling out the form.

Downgrade patrol trigger queue-full and rejection messages from Warn to
Debug — these are normal rate-limiting behavior, not actionable warnings.
2026-02-26 21:09:13 +00:00
rcourtman af712006c9 fix(ai): allow Gemini and other models via OpenRouter without false provider warning (#1296)
Model name detection used substring matching (.includes('gemini')) which
falsely required Gemini provider config for OpenRouter model IDs like
"google/gemini-2.5-flash". Now only known provider prefixes are treated
as explicit delimiters, slash-containing names route to OpenAI (OpenRouter
convention), and colons in model names (e.g. "llama3.2:latest") are no
longer misinterpreted as provider prefixes.
2026-02-26 20:49:10 +00:00
rcourtman fa519cd8ce fix(alerts): prevent false positive orphaned backup alerts during startup race (#1286)
Backup polling goroutines can snapshot state before VM/container polling
populates the guest inventory. When guestsByVMID is empty, every backup
appears orphaned. Gate orphan detection on hasLiveInventory (at least one
guest with non-empty ResourceID) and preserve existing orphan alerts when
inventory becomes unavailable.
2026-02-26 20:49:10 +00:00
rcourtman eb2397d99a fix(notifications): route escalation notifications to selected channels only (#1259)
Escalation was calling SendAlert() which always sends to all enabled
channels, ignoring the per-level channel selection (email/webhook/all).

Add SendAlertToChannels() that snapshots only the requested channel
configs and uses a distinct "_escalation" queue type so the dequeue
handler skips cooldown writes — preventing interference with the alert
manager's own re-notify cadence.
2026-02-26 20:49:10 +00:00
rcourtman 77bd2e70d9 fix(notifications): add service-specific resolved webhook templates (#1259)
Backport from v6 (88d5865a8). Recovery webhook notifications were using
the firing PayloadTemplate which services like Telegram, Teams, Discord
etc. silently rejected as malformed. Now uses a three-tier template
pipeline matching the firing path:
- Tier 1: Custom user template (if configured)
- Tier 2: Service-specific ResolvedPayloadTemplate (Discord green embed,
  Telegram chat_id+text, Slack header blocks, Teams MessageCard/Adaptive,
  PagerDuty event_action:"resolve", Pushover, Gotify, Mattermost)
- Tier 3: Generic JSON fallback (backward compatible)

Also adds Event, ResolvedAt, ResolvedAtISO fields to WebhookPayloadData.
2026-02-24 23:28:33 +00:00
rcourtman 24f5b1cb31 fix(patrol): cap per-run tokens and reset patrol session history 2026-02-24 11:29:47 +00:00
rcourtman 82ccb662f9 fix(notifications): use service-specific templates for resolved webhooks (#1068)
Recovery notifications for Discord, Slack, Teams, PagerDuty, and other
service webhooks were sending a generic JSON payload that lacked the
required format (e.g. Discord needs `embeds`, Slack needs `blocks`),
causing resolved notifications to silently fail.

- Add `prepareResolvedWebhookData` to build template data with Level="resolved"
- Route resolved webhooks through service-specific templates with full
  URL rendering, Telegram ChatID extraction, and PagerDuty routing_key
- Custom user templates take precedence over built-in service templates
- Return errors on service template failures instead of falling back to
  generic payloads that endpoints would reject
- Fix PagerDuty template to send event_action="resolve" for resolved alerts
2026-02-24 10:49:52 +00:00
rcourtman 4dc09a1240 feat(alerts): add dedicated backup-orphaned alert type (#1286)
Fire a warning alert immediately when a backup's guest no longer exists
in inventory, without requiring age thresholds to be breached. The
existing alertOrphaned toggle and ignoreVMIDs UI control this feature
with no frontend changes needed.
2026-02-24 09:07:43 +00:00
rcourtman 32746e2d2a fix(monitoring): use RRD memavailable fallback when PVE node cache metrics missing (#1270)
When Proxmox /nodes/{node}/status returns only total/used/free without
available/buffers/cached, EffectiveAvailable() returns Free (non-zero),
causing the RRD fallback gate to be skipped. This results in inflated
node memory where cache/buffers are counted as "used."

Widen the RRD fallback condition from requiring effectiveAvailable == 0
to triggering whenever missingCacheMetrics is true. Add negative caching
for failed RRD lookups (2-minute backoff) to avoid repeated retries.
2026-02-21 22:47:20 +00:00
rcourtman 1170da6a57 fix(ai): serialize linkedVmId/linkedContainerId and harden mention status (#1252)
HostFrontend was missing LinkedVmId and LinkedContainerId fields, so the
frontend dedup aliases for VM/container agents resolved to undefined and
never matched. Also add .trim() to getStatusColor and default host agent
status to 'online' to fix grey status dots.
2026-02-21 22:00:43 +00:00
rcourtman b445f8d8fa fix(agent): preserve user-configured host URL during agent re-registration (#1283)
When an agent re-registers with the same token, the DHCP matching case
would overwrite the Host field with the agent's local IP — even if the
user had edited it to a public URL or different IP. Now agent source
re-registrations always preserve the existing host, while non-agent
DHCP updates still work. Adds 5 regression tests covering hostname
preservation, public-IP preservation, agent DHCP, non-agent DHCP, and
PBS parity.
2026-02-21 12:46:02 +00:00
rcourtman 1d07c1cd30 fix(agent): prevent duplicate PVE entries on agent re-registration (#1245)
Two changes to prevent duplicates in Settings > Virtual Environment:

1. Install script: only clear Proxmox state files on fresh installs,
   not upgrades. Previously every install forced re-registration.

2. Auto-register dedup: match agent re-registrations by server name
   when both the existing entry and new request have Pulse-created
   tokens (pulse-monitor@pam!pulse-*). This catches the case where
   the agent creates a new token after state files are cleared.
2026-02-20 19:38:03 +00:00
rcourtman dacf0f86c4 fix(agent): collect temperature on FreeBSD via sysctl (#1254)
The agent gate only allowed temperature collection on Linux (lm-sensors).
FreeBSD exposes CPU and ACPI thermal zone temperatures via sysctl
(dev.cpu.N.temperature, hw.acpi.thermal.tzN.temperature). Parse sysctl
output directly in Go without shell involvement.
2026-02-20 19:00:40 +00:00
rcourtman 706502c22d fix(alerts): default NotifyOnResolve to true and prevent patrol queue spam (#1259, #1258)
Recovery notifications were silently disabled for users with pre-5.1.12
configs because the NotifyOnResolve bool field defaults to false when
absent from JSON. Use a *bool probe to detect missing field and default
to true.

Patrol trigger queue filled with warnings when the patrol loop wasn't
running. Gate TriggerPatrolForAlert on p.running and clear the flag
via defer when the loop exits.
2026-02-20 17:56:41 +00:00
rcourtman 6c78d78fc7 fix(webhooks): test button in edit form sends redacted headers (#1273)
When editing an existing webhook, header values are masked as
***REDACTED*** for security. The "Test" button in the edit form
sent these redacted values to the webhook endpoint, causing auth
failures (HTTP 403) on services like ntfy.sh that require tokens.

The test button outside the edit form worked because it used the
server-side saved config with real header values.

Fix: frontend now includes the webhook ID in test payloads for
existing webhooks, and the backend TestWebhook handler merges
redacted values with the saved originals before sending the test
(same logic already used by UpdateWebhook).
2026-02-20 13:31:52 +00:00
rcourtman afb4dcc2bd fix(sso): return full provider config for edit form and register SSO users in RBAC (#1255)
Two remaining issues from #1255 after the 5.1.10 fixes:

1. OIDC/SAML provider edit fields appeared blank because the GET
   endpoint returned a flattened response while the frontend reads
   nested oidc/saml objects. Now returns the full provider config
   with secrets redacted (client secret, SP private key).

2. SSO users didn't appear in Settings > Users because RBAC entries
   were only created when group-role mappings matched. Now ensures
   every SSO user is registered in RBAC on login, even without
   role mappings.

Also fixes: SAML SP private key and certificate lost on edit (no
preservation logic existed), OIDC client secret preservation
hardened to check actual secret presence not just flag.
2026-02-20 13:31:52 +00:00
rcourtman 0ae2806f18 fix(memory): add guest agent /proc/meminfo fallback to avoid VM memory inflation (#1270)
Proxmox status.Mem includes page cache as "used" memory, inflating
reported VM usage. The existing fallbacks (balloon meminfo, RRD, linked
host agent) were frequently unavailable, causing most VMs to fall
through to the inflated status-mem source.

Adds a new last-resort fallback that reads /proc/meminfo via the QEMU
guest agent file-read endpoint to get accurate MemAvailable. Results
are cached (60s positive, 5min negative backoff for unsupported VMs).

Also fixes: RRD memavailable fallback missing from traditional polling
path, cache key collisions in multi-PVE setups, FreeMem underflow
guard inconsistency, and integer overflow in kB-to-bytes conversion.
2026-02-20 13:31:52 +00:00
rcourtman 8c7d507ea4 fix(alerts): make --disk-exclude suppress Proxmox SSD wear/health alerts (#1142)
The --disk-exclude agent flag only filtered local metric collection but
had no effect on server-side Proxmox disk health and SSD wearout alerts,
which poll the Proxmox API directly. Users excluding disks (e.g.
--disk-exclude sda) still received alerts for those disks.

Agent now sends its DiskExclude patterns in each report. The server
stores them on the Host model and consults them during Proxmox disk
polling — excluded disks get a synthetic healthy status passed to
CheckDiskHealth so any existing alerts clear immediately.

Also adds FreeBSD pseudo-filesystem types (fdescfs, devfs, linprocfs,
linsysfs) to the virtual FS filter and /var/run/ to special mount
prefixes, fixing false disk-full alerts on FreeBSD for fdescfs mounts.
2026-02-20 13:31:52 +00:00
rcourtman fb7582c7e4 fix(memory): use linked Pulse host agent memory to avoid VM inflation (#1270)
When no guest agent MemInfo or RRD data is available, prefer the linked
Pulse host agent's memory (read from /proc/meminfo via gopsutil, which
excludes page cache) over Proxmox's status.Mem (total - free, inflated
by reclaimable cache). Applied to both efficient and traditional polling
paths. Diagnostic fields added to VMMemoryRaw for visibility.
2026-02-19 19:04:19 +00:00
rcourtman 00afaec2ae fix(agent): add retry with backoff to Proxmox auto-registration (#1267, #1269, #1261, #1268)
registerWithPulse() was a one-shot call at agent startup — if it failed
(timing, transient network, Pulse not ready), the agent silently continued
as a generic Host forever. Wrap the HTTP POST in a retry loop with
exponential backoff (5s, 10s, 20s, 40s, 60s) and distinguish 4xx errors
(no retry) from 5xx/network errors (retry).
2026-02-18 16:05:40 +00:00