The deterministic capacity forecast queried utilization history using the
canonical resource ID (storage-<hash>), but the monitoring layer records
metrics under the metrics-target/source ID (MetricsTarget.ResourceID). For
most storage sources these two IDs diverge, so the forecast query always
found nothing and the feature stayed dormant even for pools with full
ingestion history (Proxmox, TrueNAS, PBS, agent-backed storage).
This splits the two ID roles in the precompute path: a new metricsID field
on patrolStoragePoolRow / patrolPrecomputeStorageSource carries the
metrics-target ID (resolved from SourceID() in the readState branch, or
MetricsTarget.ResourceID in the unified-resource branch) for history
queries, while the canonical id is still used for stamp-matching findings.
Adds a test proving the forecast fires when history exists only under the
metrics-target ID and that the resulting forecast resourceID stays
canonical so StampCapacityForecasts can still match it.
Contract (ai-runtime): the forecast must query history under the
metrics-target ID (StoragePoolView.SourceID()), not the canonical ID; the
forecast's own resourceID stays canonical for stamp-matching.
Refs lane: protection-posture-attention-queue
Capacity forecasts (days-to-full, current usage, trend) were previously
computed deterministically but only fed into LLM prompt text. They never
reached the finding as structured data, so the frontend could not render a
verified urgency signal and the model's speculation was the only thing the
operator saw.
This persists CapacityForecast on Finding (marshal-mirror pair) and stamps
it post-analysis via a service join (StampCapacityForecasts). The forecast
filter now also keeps stable-high (>=80%) pools so the deterministic "no
fill trend" reading wins over model speculation, and fixes isQuiet wrongly
treating stable (-1) trends as filling.
Frontend maps the forecast through the UnifiedFinding view model and renders
a deterministic urgency line (Filling up / Stable / days-to-full · % used)
in the expanded finding detail.
Note: forecasts only populate for resources whose usage is ingested into
metrics history (Proxmox/Ceph storage, nodes, guests). Agent-host storage
(unraid pools) is not yet ingested as a time-series and remains a follow-up
to activate the feature for those pools.
Refs lane: protection-posture-attention-queue
When a Proxmox node has an agent installed, SMART/temperature metrics are
written by the host agent under the agent's disk source ID. But
BuildMetricsTarget preferred the Proxmox source first, and Proxmox API
does not expose detailed SMART data. The mismatch meant the frontend
queried a metrics key that nothing writes, so disk metrics appeared empty.
Move the Agent source check ahead of Proxmox/TrueNAS for
ResourceTypePhysicalDisk, mirroring the existing pattern for
ResourceTypeAgent where agent source already wins over platform sources.
When a disk has a serial number, it is used as the canonical metric ID
regardless of source (PreferredPhysicalDiskMetricID), so the reorder
only affects disks without serials — exactly the case where source
priority determines the key.
Refs #1487
The unified_resources.db grew without bound (2GB reported) because:
1. No VACUUM: DELETE freed rows internally but never shrank the file.
Added auto_vacuum(INCREMENTAL) to the DSN for new databases, plus a
one-time migrateAutoVacuum() that converts existing databases.
reclaimFreePages() now runs after each prune cycle to return freed
pages to the OS via PRAGMA incremental_vacuum.
2. Missing retention: action_lifecycle_events, export_audits, and
loop_reports had no retention at all. Added 90-day retention for
lifecycle/export audits and 30-day for loop_reports, matching the
existing action_audits/resource_changes cadence.
3. Slow cleanup cadence: the retention loop ran every 6h and never on
startup. Reduced to hourly and added an initial prune 30s after
startup so a restart with a bloated DB starts recovering immediately.
Mirrors the proven pattern from metrics.db (auto_vacuum INCREMENTAL +
incremental_vacuum + WAL checkpoint).
Refs #1496
getDockerCommandPayload returned dispatched commands on every report
fetch, causing the agent to re-execute check-updates on every poll
cycle. When the ack also failed, the report was buffered and retried,
creating an infinite loop.
- Only return command payload on the queued->dispatched transition;
subsequent fetches return nil (agent already received it).
- Don't propagate ack errors from handleCheckUpdatesCommand; the report
was delivered and check-updates is fire-and-forget. Command expires
if ack never succeeds.
Refs #1504
When the Proxmox API's meminfo/status payload omits Available, Buffers,
and Cached (common for QEMU guest-agent and node status responses), the
code derived 'available' from Free alone — producing used = Total-Free
which counts reclaimable page cache as used memory (e.g. 94% instead of
the correct 76%).
Guest path (deriveGuestMemInfoAvailable): return 0 when cache metrics
are completely missing and only Free is available, so resolveGuestStatusMemory
tries better sources: guest-agent /proc/meminfo file-read (returns
MemAvailable), RRD memavailable, or the linked host agent.
Node path (resolveNodeMemory): always try RRD memavailable when cache
metrics are missing, not only when effectiveAvailable == 0 — previously
a non-zero Free value blocked the RRD fallback.
Refs #1501
sendViaProviderWithAddresses mutated the shared e.config.Username for
provider-specific defaults (SendGrid, Postmark, SparkPost, Resend).
If concurrent goroutines sent email simultaneously, this was a data
race on the config struct.
Move the resolution into negotiateAuth via a local variable
(resolveProviderUsername helper) so the shared config is never mutated.
Unraid's mdResyncAction field can retain its last value (e.g. "check")
after a parity check is canceled, causing Pulse to report a stale sync
action indefinitely. The mdResync/mdResyncPos field is the authoritative
indicator: it drops to 0 when no resync is running. Gate SyncAction on
a non-zero position so the alert clears once the sync actually stops.
Refs #1485
Add deduplicateHistory() to HistoryManager that collapses consecutive
same-alert entries within a 5-minute window. The flapping/re-fire bugs
(lifecycle and stateful paths) created many duplicate history entries
before the cooldown fixes were deployed. This cleanup runs on startup
after loadHistory and periodically alongside cleanOldEntries, so both
existing noise and any future edge-case duplicates are handled.
On this install: 332 entries → 130 (60% reduction).
handleLogin rejected all credentials when ALLOW_ADMIN_BYPASS=1 because
it validated against config AuthUser/AuthPass directly without checking
the bypass flag. This made dev-mode browser testing impossible after a
backend restart with bypass env vars — the API middleware accepted all
requests but the login page could never obtain a session cookie.
When bypass is enabled, accept any credentials and create a session as
'admin'.
Stateful alerts (ZFS pool/device, storage topology) were creating
duplicate history entries every poll cycle because SyncStorageAlertsForInstance
clears alerts between evaluations, making the existing-alert check miss.
Add the same 5-minute re-fire cooldown used in the lifecycle path: when a
stateful alert re-fires within 5 minutes of resolution, reactivate the
original (preserving StartTime) and update the existing history entry's
LastSeen instead of appending a new entry.
When an alert fires, resolves, and re-fires within 5 minutes (the
recently-resolved retention window), the previous implementation created
a new history entry for each fire cycle. For a flapping connection like
an unreachable Proxmox node, this produced hundreds of duplicate entries
in a single night — 302 identical 'pi unreachable' alerts in one case.
The fix checks the recently-resolved map before creating a new history
entry. If the alert was resolved within the cooldown window, it
reactivates the original alert (preserving StartTime) and updates the
existing history entry's LastSeen instead of appending a new one.
The patrol_report_finding tool descriptions for evidence, impact, and
recommendation were too permissive, causing the LLM to frequently omit
them. Future patrol runs will produce better-scaffolded findings.
1. Rewrite tool descriptions to emphasize evidence as a trust anchor
that should always be included, and impact as expected whenever the
data supports it.
2. Add 'Authoring Evidence' section to the patrol system prompt with
concrete examples, matching the existing 'Authoring Impact' section.
3. Add test verifying trust scaffolding guidance is in the system prompt.
Addresses checklist L48-50 (trust scaffolding).
1. Add Impact field to findingView struct — was completely missing from
the unified findings API response serialization.
2. Expand generateImpact() to cover all ~25 alert types the system
produces. Previously only 7 had curated impact text.
3. Fix naming mismatches: poweredOff/nodeOffline never matched actual
alert types powered-off/host-offline.
4. Expand TypeCategoryMap with all missing alert types.
Existing alerts kept their stale message from when they first fired.
Now the message is regenerated on every evaluation cycle so label
fixes and wording improvements take effect immediately.
When the SQLite resource database is corrupted (malformed disk image),
NewSQLiteResourceStore now backs up the corrupted file to
*.corrupted.<timestamp> and recreates a fresh database. Without this,
every /api/resources request returned 500 with no recovery path
until the admin manually deleted the file.
Resource data is derived from monitor state and repopulated on the
next poll cycle; user-authored metadata (links, notes) in the corrupted
file is preserved in the backup.
The refreshSuggestedAvailabilityProbeFromState method existed but was
never called, so existing discovery records never received availability
probe suggestions. This adds backfillAvailabilitySuggestions which:
- Triggers from SetReadState (goroutine) and runDiscoveryLoop (ticker)
- Retries with exponential backoff when the state snapshot is empty,
waiting for the monitor to populate Proxmox data before proceeding
- Matches containers by VMID only (was VMID+Node, which failed in
clusters where the discovery targetID differs from the container's
node)
Also extends SuggestAvailabilityProbe with a hostname fallback: when
ServiceType is empty (deep scan not yet run), checks the discovery's
Hostname against webServiceDefaults/tcpServiceDefaults. This covers
containers like jellyfin, grafana, frigate, esphome, zigbee2mqtt,
homeassistant, mqtt, etc. in environments where background AI is
disabled.
Adds zigbee2mqtt and ntfy to webServiceDefaults.
writeHostPhysicalDiskIOMetrics gated ALL per-disk I/O metrics on
host.Sensors.SMART being non-empty. When the agent's SMART collection
fails (smartctl not installed, LXC container can't see /sys/block),
DiskIO data from gopsutil is valid but silently discarded.
Fix: remove the SMART requirement. When no SMART match is found for a
DiskIO entry, try matching against Proxmox API physical disks by device
name (via LinkedNodeID), then fall back to hostID:device as the metric
resource ID. This matches the fallback scheme already used by
HostSMARTDiskSourceID and PhysicalDiskMetaMetricID.
Refs #1487
TCP probes to fast local-network services can complete in under 1ms,
causing latency.Milliseconds() to return 0. Combined with omitempty on
the poller status struct, this made some probes show 'Online' with no
latency in the settings panel. Now successful probes always report at
least 1ms.
The critical (severity-escalation) threshold was hardcoded as
Trigger + 10 for all metric types. For percentage metrics (cpu,
memory, disk, usage) with high triggers, this made critical
escalation unreachable: a CPU trigger of 95% produced a critical
threshold of 105%, which is impossible for a 0-100% metric.
Fix: add computeCriticalThreshold() helper that caps the critical
threshold at 99 for percentage metrics. Non-percentage metrics
(temperature, diskRead, diskWrite, networkIn, networkOut) keep
the Trigger + 10 offset unchanged.
Applied to all three code paths that derive critical from trigger:
- buildCanonicalMetricSpec (canonical evaluation)
- checkMetric new alert creation (legacy path)
- checkMetric existing alert update (legacy path)
When a monitor reload was triggered by node auto-registration, the
reloadFunc in server.go recreated the monitor (and its notification
manager) but never re-applied system settings. The new notification
manager started with an empty webhook private CIDR allowlist, causing
webhook notifications to private IPs to fail until the allowlist was
manually re-saved in Settings.
Fix: call router.ReloadSystemSettings() at the end of reloadFunc,
after the new monitor references are set. This re-applies all
persisted system settings — including the webhook CIDR allowlist —
to the freshly created notification manager.
Refs #1507
The resource_changes and action_audits tables grew without bound
because no retention mechanism existed. In production deployments with
frequent monitoring cycles, the database could reach multiple GB within
weeks.
Add a background goroutine that runs every 6 hours and deletes:
- resource_changes older than 30 days
- action_audits older than 90 days
The goroutine starts in NewSQLiteResourceStore and is stopped via the
retentionStop channel in Close(). The first prune runs on the first
ticker interval (6h after startup), not immediately, so test fixtures
with historical timestamps are not affected.
Fixes#1496.
All TCP and HTTP availability probes were failing with 'no route to host'
because the Pulse process's Go networking stack cannot reach local subnet
IPs when a Tailscale system extension is active. The ICMP probe already
worked because it uses exec.Command('ping'). Apply the same pattern:
- probeTCP: try Go net.Dialer first, fall back to exec.Command('nc')
- probeHTTP: try Go http.Client first, fall back to exec.Command('curl')
- Both fallbacks use OS-appropriate timeout flags (macOS: -G, Linux: -w)
- HTTP probes now skip TLS verification (availability check, not identity)
- Add AvailabilityProbeHTTPS protocol constant and wire it through
Validate, HTTPURL, Normalize, and the probe switch
- Replace bulky TooltipPortal tooltip on AvailabilityProbeCell with a
clean native title attribute ('14 ms - checked 3s ago')
Before: 0/20 probes green (all 'no route to host')
After: 14/20 probes green (6 legitimately down hosts)
Discovery already identifies services and their default ports. Now it
also suggests an availability probe configuration for each discovered
service with a known web interface (webServiceDefaults) or TCP service
(tcpServiceDefaults). The suggestion appears in the resource drawer as a
card with a 'Monitor availability' button. On approval, it calls the
existing POST /api/availability-targets API — one canonical system, no
second management surface.
Backend:
- New SuggestAvailabilityProbe() generates protocol/port/path from the
same webServiceDefaults map used for URL suggestions, with a
tcpServiceDefaults fallback for databases and message brokers
- New AvailabilityProbeSuggestion type on ResourceDiscovery
- Wired into both discovery paths (DiscoverResource + Docker background)
- Cached discoveries get the suggestion via refreshSuggestedAvailabilityProbe
on read, so existing data picks it up without re-discovery
Frontend:
- AvailabilityProbeSuggestionCard in GuestDrawerOverview with one-click
creation via AvailabilityTargetsAPI.create()
- Card hidden when the resource already has an availability facet
- Added 'https' to AvailabilityProbeProtocol type (backend already
supports it)
TestMonitorPollGuestSnapshots_RefreshesStaleCanonicalStoreForClusterGuest
passed context.Background() to pollGuestSnapshots, which creates an
internal 60s snapshot timeout. The backupStorageTimeoutSnapshotClient
mock blocks GetStorage until context cancellation, so the test wasted
60s waiting for the timeout to expire. Pass a 200ms deadline context
so pollGuestSnapshots caps its budget down — test now runs in 0.2s
instead of 60s and no longer times out under -race.
chooseStatus() had a special case for SourceAvailability that always
returned 'incoming', overriding SourceAgent (priority 3) and Proxmox
(priority 2). When an availability probe reported offline, it overwrote
the Proxmox API's online status on every poll cycle, causing node-level
CPU/Mem/Disk to show — because isOnline() gated metric rendering.
Also fix markStaleLocked() to recompute status from remaining fresh
sources via aggregateStatus() instead of blindly downgrading online to
warning. This ensures correct status when high-priority sources go stale
but lower-priority sources are still fresh.
SyncStorageAlertsForInstance matched alert.ResourceID against valid
storage IDs (plain inventory IDs like 'inst-node-local-zfs'). But ZFS
health assessment alerts carry a composite spec resource ID (e.g.
'inst-node-local-zfs/zfs-pool:data') set via params.Spec.ResourceID in
evaluateCanonicalStatefulAlert. Every ZFS alert was immediately cleared
by the sync sweep right after creation, then re-fired on the next poll —
an infinite flapping loop every ~10 seconds.
Fix: when the exact resource ID doesn't match, check whether the prefix
before the first '/' is a valid storage ID. Storage IDs use hyphens
(never slashes), so the prefix cleanly extracts the inventory ID from
the composite spec resource ID.
Added two test cases:
- preserves zfs alerts with composite spec resource ids
- clears zfs alerts when storage is removed (ensures fix doesn't
prevent legitimate cleanup)
The /api/health endpoint gated scheduler health on
DeadLetterCount()==0, causing a 503 whenever any single monitored
node became unreachable. Dead-lettered tasks are the scheduler
working as designed — after repeated failures the task is moved to
the dead-letter queue to prevent infinite retries.
In the live lab this manifested as a persistent 503 on /api/health
because the Pi node (192.168.0.2) times out on every poll cycle,
dead-lettering its task while the other 20+ tasks continue normally.
The scheduler is healthy when the monitor is running. Dead-letter
counts remain available as diagnostic info via the
/api/monitoring/scheduler/health endpoint.
1. Regenerate pulse-mcp README from canonical manifest (doc drift)
2. Add 31 missing Pulse Intelligence telemetry fields to both PRIVACY.md
copies to match current Ping struct JSON tags
3. Rebuild portal frontend bundle to update build_manifest.json hash
4. Update action execution contract test to match current code structure
(handler wrapped with withExternalAgentCapabilityActivity, error codes
referenced via agentcapabilities constants instead of literal strings)
Full Go test suite now passes clean: 126 packages, 0 failures.
Agentless availability checks (ICMP/TCP/HTTP) were always minting
standalone network-endpoint resources, leaving them disconnected from
the known Proxmox/Docker guest they actually monitor. This made
availability evidence invisible on the platform resource row where the
user expects it, per the performance-and-scalability bounded-row
contract.
Backend (unified-resources ingest):
- Add LinkedResourceID field to AvailabilityData and AvailabilityTarget
- resolveAvailabilityLink: explicit link first, then exact-IP unambiguous
correlation; skip hostname-only (lossy); guard against overwriting a
different target's facet
- Unlinked/unmatched probes still mint network-endpoint (fallback)
Frontend:
- Relax getAvailabilityProbePresentation for any resource with availability
- Add compact protocol badge to UnifiedResourceHostTableCard name cell
- Add optional 'Link to resource' field to availability target form
- Add linkedResourceId to frontend types
Contracts: api-contracts, unified-resources, monitoring,
performance-and-scalability, storage-recovery.
Governance: coverage_gap + candidate_lane in status.json.
rcourtman