When the SQLite resource database is corrupted (malformed disk image),
NewSQLiteResourceStore now backs up the corrupted file to
*.corrupted.<timestamp> and recreates a fresh database. Without this,
every /api/resources request returned 500 with no recovery path
until the admin manually deleted the file.
Resource data is derived from monitor state and repopulated on the
next poll cycle; user-authored metadata (links, notes) in the corrupted
file is preserved in the backup.
The refreshSuggestedAvailabilityProbeFromState method existed but was
never called, so existing discovery records never received availability
probe suggestions. This adds backfillAvailabilitySuggestions which:
- Triggers from SetReadState (goroutine) and runDiscoveryLoop (ticker)
- Retries with exponential backoff when the state snapshot is empty,
waiting for the monitor to populate Proxmox data before proceeding
- Matches containers by VMID only (was VMID+Node, which failed in
clusters where the discovery targetID differs from the container's
node)
Also extends SuggestAvailabilityProbe with a hostname fallback: when
ServiceType is empty (deep scan not yet run), checks the discovery's
Hostname against webServiceDefaults/tcpServiceDefaults. This covers
containers like jellyfin, grafana, frigate, esphome, zigbee2mqtt,
homeassistant, mqtt, etc. in environments where background AI is
disabled.
Adds zigbee2mqtt and ntfy to webServiceDefaults.
writeHostPhysicalDiskIOMetrics gated ALL per-disk I/O metrics on
host.Sensors.SMART being non-empty. When the agent's SMART collection
fails (smartctl not installed, LXC container can't see /sys/block),
DiskIO data from gopsutil is valid but silently discarded.
Fix: remove the SMART requirement. When no SMART match is found for a
DiskIO entry, try matching against Proxmox API physical disks by device
name (via LinkedNodeID), then fall back to hostID:device as the metric
resource ID. This matches the fallback scheme already used by
HostSMARTDiskSourceID and PhysicalDiskMetaMetricID.
Refs #1487
TCP probes to fast local-network services can complete in under 1ms,
causing latency.Milliseconds() to return 0. Combined with omitempty on
the poller status struct, this made some probes show 'Online' with no
latency in the settings panel. Now successful probes always report at
least 1ms.
The critical (severity-escalation) threshold was hardcoded as
Trigger + 10 for all metric types. For percentage metrics (cpu,
memory, disk, usage) with high triggers, this made critical
escalation unreachable: a CPU trigger of 95% produced a critical
threshold of 105%, which is impossible for a 0-100% metric.
Fix: add computeCriticalThreshold() helper that caps the critical
threshold at 99 for percentage metrics. Non-percentage metrics
(temperature, diskRead, diskWrite, networkIn, networkOut) keep
the Trigger + 10 offset unchanged.
Applied to all three code paths that derive critical from trigger:
- buildCanonicalMetricSpec (canonical evaluation)
- checkMetric new alert creation (legacy path)
- checkMetric existing alert update (legacy path)
When a monitor reload was triggered by node auto-registration, the
reloadFunc in server.go recreated the monitor (and its notification
manager) but never re-applied system settings. The new notification
manager started with an empty webhook private CIDR allowlist, causing
webhook notifications to private IPs to fail until the allowlist was
manually re-saved in Settings.
Fix: call router.ReloadSystemSettings() at the end of reloadFunc,
after the new monitor references are set. This re-applies all
persisted system settings — including the webhook CIDR allowlist —
to the freshly created notification manager.
Refs #1507
The resource_changes and action_audits tables grew without bound
because no retention mechanism existed. In production deployments with
frequent monitoring cycles, the database could reach multiple GB within
weeks.
Add a background goroutine that runs every 6 hours and deletes:
- resource_changes older than 30 days
- action_audits older than 90 days
The goroutine starts in NewSQLiteResourceStore and is stopped via the
retentionStop channel in Close(). The first prune runs on the first
ticker interval (6h after startup), not immediately, so test fixtures
with historical timestamps are not affected.
Fixes#1496.
All TCP and HTTP availability probes were failing with 'no route to host'
because the Pulse process's Go networking stack cannot reach local subnet
IPs when a Tailscale system extension is active. The ICMP probe already
worked because it uses exec.Command('ping'). Apply the same pattern:
- probeTCP: try Go net.Dialer first, fall back to exec.Command('nc')
- probeHTTP: try Go http.Client first, fall back to exec.Command('curl')
- Both fallbacks use OS-appropriate timeout flags (macOS: -G, Linux: -w)
- HTTP probes now skip TLS verification (availability check, not identity)
- Add AvailabilityProbeHTTPS protocol constant and wire it through
Validate, HTTPURL, Normalize, and the probe switch
- Replace bulky TooltipPortal tooltip on AvailabilityProbeCell with a
clean native title attribute ('14 ms - checked 3s ago')
Before: 0/20 probes green (all 'no route to host')
After: 14/20 probes green (6 legitimately down hosts)
Discovery already identifies services and their default ports. Now it
also suggests an availability probe configuration for each discovered
service with a known web interface (webServiceDefaults) or TCP service
(tcpServiceDefaults). The suggestion appears in the resource drawer as a
card with a 'Monitor availability' button. On approval, it calls the
existing POST /api/availability-targets API — one canonical system, no
second management surface.
Backend:
- New SuggestAvailabilityProbe() generates protocol/port/path from the
same webServiceDefaults map used for URL suggestions, with a
tcpServiceDefaults fallback for databases and message brokers
- New AvailabilityProbeSuggestion type on ResourceDiscovery
- Wired into both discovery paths (DiscoverResource + Docker background)
- Cached discoveries get the suggestion via refreshSuggestedAvailabilityProbe
on read, so existing data picks it up without re-discovery
Frontend:
- AvailabilityProbeSuggestionCard in GuestDrawerOverview with one-click
creation via AvailabilityTargetsAPI.create()
- Card hidden when the resource already has an availability facet
- Added 'https' to AvailabilityProbeProtocol type (backend already
supports it)
TestMonitorPollGuestSnapshots_RefreshesStaleCanonicalStoreForClusterGuest
passed context.Background() to pollGuestSnapshots, which creates an
internal 60s snapshot timeout. The backupStorageTimeoutSnapshotClient
mock blocks GetStorage until context cancellation, so the test wasted
60s waiting for the timeout to expire. Pass a 200ms deadline context
so pollGuestSnapshots caps its budget down — test now runs in 0.2s
instead of 60s and no longer times out under -race.
chooseStatus() had a special case for SourceAvailability that always
returned 'incoming', overriding SourceAgent (priority 3) and Proxmox
(priority 2). When an availability probe reported offline, it overwrote
the Proxmox API's online status on every poll cycle, causing node-level
CPU/Mem/Disk to show — because isOnline() gated metric rendering.
Also fix markStaleLocked() to recompute status from remaining fresh
sources via aggregateStatus() instead of blindly downgrading online to
warning. This ensures correct status when high-priority sources go stale
but lower-priority sources are still fresh.
SyncStorageAlertsForInstance matched alert.ResourceID against valid
storage IDs (plain inventory IDs like 'inst-node-local-zfs'). But ZFS
health assessment alerts carry a composite spec resource ID (e.g.
'inst-node-local-zfs/zfs-pool:data') set via params.Spec.ResourceID in
evaluateCanonicalStatefulAlert. Every ZFS alert was immediately cleared
by the sync sweep right after creation, then re-fired on the next poll —
an infinite flapping loop every ~10 seconds.
Fix: when the exact resource ID doesn't match, check whether the prefix
before the first '/' is a valid storage ID. Storage IDs use hyphens
(never slashes), so the prefix cleanly extracts the inventory ID from
the composite spec resource ID.
Added two test cases:
- preserves zfs alerts with composite spec resource ids
- clears zfs alerts when storage is removed (ensures fix doesn't
prevent legitimate cleanup)
The /api/health endpoint gated scheduler health on
DeadLetterCount()==0, causing a 503 whenever any single monitored
node became unreachable. Dead-lettered tasks are the scheduler
working as designed — after repeated failures the task is moved to
the dead-letter queue to prevent infinite retries.
In the live lab this manifested as a persistent 503 on /api/health
because the Pi node (192.168.0.2) times out on every poll cycle,
dead-lettering its task while the other 20+ tasks continue normally.
The scheduler is healthy when the monitor is running. Dead-letter
counts remain available as diagnostic info via the
/api/monitoring/scheduler/health endpoint.
1. Regenerate pulse-mcp README from canonical manifest (doc drift)
2. Add 31 missing Pulse Intelligence telemetry fields to both PRIVACY.md
copies to match current Ping struct JSON tags
3. Rebuild portal frontend bundle to update build_manifest.json hash
4. Update action execution contract test to match current code structure
(handler wrapped with withExternalAgentCapabilityActivity, error codes
referenced via agentcapabilities constants instead of literal strings)
Full Go test suite now passes clean: 126 packages, 0 failures.
Agentless availability checks (ICMP/TCP/HTTP) were always minting
standalone network-endpoint resources, leaving them disconnected from
the known Proxmox/Docker guest they actually monitor. This made
availability evidence invisible on the platform resource row where the
user expects it, per the performance-and-scalability bounded-row
contract.
Backend (unified-resources ingest):
- Add LinkedResourceID field to AvailabilityData and AvailabilityTarget
- resolveAvailabilityLink: explicit link first, then exact-IP unambiguous
correlation; skip hostname-only (lossy); guard against overwriting a
different target's facet
- Unlinked/unmatched probes still mint network-endpoint (fallback)
Frontend:
- Relax getAvailabilityProbePresentation for any resource with availability
- Add compact protocol badge to UnifiedResourceHostTableCard name cell
- Add optional 'Link to resource' field to availability target form
- Add linkedResourceId to frontend types
Contracts: api-contracts, unified-resources, monitoring,
performance-and-scalability, storage-recovery.
Governance: coverage_gap + candidate_lane in status.json.
parseProxmoxOnBoot returned nil when the onboot key was absent from a
valid config, but Proxmox's default is onboot=0 (do not start on boot).
This caused stopped VMs without an explicit onboot line (like
windows-runner and tails-anon) to still generate powered-off alerts
despite not being configured to autostart.
Now returns false for absent keys in non-empty configs. nil is reserved
for empty/failed config fetches and unrecognised values, preserving the
'unknown' fallback for genuine uncertainty.
TestAlertLifecycleCanonicalChangesRemainWritable used a hardcoded date
(2026-03-20) that is now >90 days old, exceeding the incident store's
maxAge retention. trimLocked() evicted the incident shell immediately
after creation, causing GetTimelineByAlertIdentifier to return nil.
Switch to time.Now().UTC().Add(-1h) so the test never goes stale.
The alert engine fired powered-off alerts for every stopped VM and
container regardless of whether the guest was configured to autostart.
This meant intentionally-stopped guests (onboot=0) generated alarm
fatigue — in the live lab, 7 of 12 alerts were noise from stopped VMs
that are deliberately off.
Now the alert engine checks the Proxmox onboot setting before firing:
- onboot=true → stopped is unexpected, alert (preserved behavior)
- onboot=false → stopped is expected, suppress
- onboot=nil → unknown, alert (preserved behavior)
For containers, onboot is parsed from the already-fetched container
config in enrichContainerMetadata (no new API call). For VMs, the
config is fetched only for stopped VMs via GetVMConfig (one extra call
per stopped VM, running VMs are unaffected).
Models: add OnBoot *bool to VM and Container.
Alerts: add onboot-aware suppression branch in CheckGuest.
Monitoring: add parseProxmoxOnBoot helper and fetchVMOnBoot.
Force stale negative LXC Docker detections to re-run after the Docker checker is configured so explicit Proxmox guest Docker inventory repopulates after backend restarts.
Treat browser-encoded comma-separated resource type lists the same as literal comma filters so Docker and platform pages keep canonical inventory rows.
The alert AI button now offers a manual targeted Patrol check alongside the
existing context-only Pulse Assistant handoff, so the take-the-lead intent routes
to Patrol (the investigator) instead of overloading Assistant (the explainer).
POST /api/ai/patrol/run (HandleForcePatrol) accepts an optional scope body and
routes through the same TriggerScopedPatrol engine and scoped run record as
automatic alert-triggered work, not a new trigger route; empty body keeps the
legacy fleet-wide run. Scoped runs honour Patrol readiness but bypass the
full-run cadence gate, consistent with the existing scoped-cadence rule, and
carry resource identity only (no command or remediation payload).
Frontend: triggerPatrolRun(scope?) and a split AI button (patrolOption) on the
alert overview card whose menu runs the scoped check; icon/history rows stay
single-purpose. Contract updates land in the touched/dependent subsystem docs
(ai-runtime, api-contracts, agent-lifecycle, storage-recovery, frontend-primitives,
security-privacy, patrol-intelligence, alerts); subsystem_lookup_test line
expectation refreshed for the api-contracts shift.
Z.ai keys on the coding subscription get 429 'Insufficient balance' on the standard /api/paas/v4 endpoint. Add a per-provider BaseURLField to the Z.ai registry entry, a ZaiBaseURL override on AIConfig (returned by GetBaseURLForProvider when set, else the standard default), the handler request/response/apply plumbing, and a 'Custom Base URL' field on the Z.ai provider card so operators can point at /api/coding/paas/v4. Mirrors the existing OpenAI custom-base-URL override; the standard endpoint remains the default for pay-as-you-go users. Adds config/handler/frontend proofs and updates the ai-runtime, api-contracts, frontend-primitives, agent-lifecycle, and storage-recovery contracts.
Add a registry-backed provider model for runtime metadata and chat-compatible transports.
Support Z.ai, Groq, Mistral, Cerebras, Together, and Fireworks through the shared chat-compatible client path while keeping native providers on their existing adapters. Expose provider metadata through settings and render the expanded provider set in the frontend.
getConfiguredHostIPs() and the new discoveryConfigSnapshot() now deep-copy config under a brief RLock and release it before resolving Proxmox/PBS/PMG hostnames via the injectable lookupConfiguredHostIP seam, so slow or blocked DNS can no longer stall monitor writers. The discovery IPBlocklist is the deduplicated merge of the operator-configured blocklist and the resolved configured-host IPs. Adds lock-freedom and merge coverage tests; updates the monitoring contract obligation.
Manifest-backed MCP tools, prompts, and resources with surface affordance contracts; agent capability manifest and governance projection; API contract tests and capability route projection; operations-loop and intelligence-funnel telemetry; release-control subsystem documentation, registry, and tooling; licensing and configuration.
f62f35e24 restored the v5 used | cache | free memory split for Proxmox
nodes and guests, but standalone host agents still reported a flat
used/free pair, so the Machines page memory bar could not show the
reclaimable segment. Flagged by the Machines page v5 parity audit.
- Host agent reports cacheBytes (gopsutil Available minus Free); the
ZFS ARC adjustment recomputes free so used + cache + free still
covers the total.
- ApplyHostReport maps the field into models.Memory.Cache and clamps
inconsistent or older-agent reports so used + cache never exceeds
total.
- AgentMemoryMeta carries cache onto unified resources so the frontend
agent payload exposes it.
- Mock generic hosts split a third of non-used pages as cache, and the
node-linked host conversion now holds the invariant instead of
stacking the node's cache on top of a recomputed free.
- Contracts: monitoring, unified-resources, and storage-recovery now
document the split (also covering the f62f35e24 node/guest surface,
which landed without contract deltas).
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
The mock random-metrics updater recomputes Used/Free from the sampled
percentage but left Cache at its generation-time value, so a drifting
node could show used+cache > total and a 102% 'Shown in Proxmox' row.
Clamp the cache into the non-used pages in applyMemoryUsage, and teach
the memory-bar presentation to clamp defensively so a momentarily
inconsistent snapshot can never render segments past 100%.
rcourtman