Snapshot recovery must preserve identity, lifecycle events and acknowledgement fields. Canonical query coverage uses a fresh observation window on each read and is deliberately not checkpointed.
Change-source: pulse-maintainer
Eight imported snapshots caused eight filtered history queries and thirty-six identity reads although replay discards each result. Share the unchanged occurrence repair with a write-only materialisation entrypoint; keep EnsureAlertOccurrence and user queries on their existing canonical read-repair path.
Preserve snapshot fallback, bounded count and age retention, checkpoint recovery and read-error behaviour. Shutdown worker ownership and the reported memory baseline remain separate outcomes.
Contract-Neutral: Internal replay cost correction only; alert occurrences, canonical history, checkpoint and retention formats, user reads and notification policy are unchanged.
Change-source: pulse-maintainer
Additively integrate exact upstream 1a094646d7 while retaining starting reviewed 19a8358a49. Keep reviewed UUID incident/event IDs and upstream non-wrapping atomic sequences for other memory IDs. Compose upstream shared SMART/Unraid temperatures with reviewed pending context and observation-gap interruption; pin Unraid-only continuity and both grace paths. Retain all browser, completion and security validation limits until evidenced.
Change-source: pulse-maintainer
Each tenant monitor owns its own IncidentStore, but incident and
incident-event IDs came from package-level int64 counters bumped with ++
under nothing but the calling store's mutex. Two tenants recording
alerts at once raced on them: go test -race failed
TestLeavingMockModeClearsEveryRunningTenantMonitor on PR 2698 at
incidents.go:1753. The remediation record counter had the same shape,
and remediation logs are created per org.
All six internal/ai/memory ID generators now mint through one atomic
helper. The numeric suffix also stops wrapping at 1000: even with atomic
counters, one generator minting more than a thousand IDs in a
wall-clock second repeated an ID, and incident notes find their incident
by ID. The prefix and timestamp layout is unchanged, and intToString
lost its last callers.
ids_test.go mints 2,400 IDs from every generator on 16 goroutines and
records alerts into two tenant stores concurrently. Against the old
files it reports DATA RACE at incidents.go:1753/1760 and
remediation.go:355, and every generator mints duplicate or wrapped IDs.
Separate tenant stores previously raced on process-wide counters and reused identifiers every thousand events in one second. Use the existing UUID dependency without mutable generator state. Cover concurrent bursts, cross-store lifecycle and by-ID note ownership, plus legacy JSON references and reloads.
Contract-Neutral: Correct uniqueness of existing opaque incident and event references; prefixes and persisted legacy references remain supported without API or schema changes.
Change-source: pulse-maintainer
PVE disk health and wearout alerts reference the disk by instance, node
and device path, so a physical disk's history never showed its own SMART
and wearout alerts. Binding the path reference failed review: a binding
belongs to the reference string, so any disk that takes over the path
inherits every row and every read of it.
Own each lifecycle row instead. The alert now records the disk's WWN
next to its serial, and the history writer puts the row under the disk
the registry knows by that hardware identity, wherever it sits now. The
registry only translates identity into a canonical ID, so a generation
that still places another disk at the path cannot claim the row, and no
check precedes a later write. A disk not yet in inventory, or removed,
gets the ID the registry mints for its identity. Rows without usable or
unambiguous identity stay under the reference, as do rows journaled
before this change; journal rows are never rewritten.
Reads by the reference still return every row raised under it, through
the alert identifier index, and the incident projection keeps owned rows
under their alert reference, so the Alerts history Resource action and
incident timelines show the same occurrences as before. A read by the
disk returns its own occurrences with their saved shells and notes.
A node alert handed to its Pulse agent closes with an AlertResolution
(#2580), and notifications already say it moved. Every in-app reader
still treated the close as a recovery: Patrol's detail wore a green
Resolved badge over a timeline labelled "Recovery evidence", the alert
history listed the row as resolved and its incident timeline said
"Alert resolved: Node memory at 95%", and the assistant, Patrol's seed
prompt and the pulse_alerts tool saw a plain resolve because
GetRecentlyResolved dropped the resolution.
Operational trust gains a non-recovery cause, ownership_transferred,
which may only enter resolved and only with evidence. The resolve
transition uses it whenever the alert carries a moved_to_agent
resolution, so a handover during a snooze also stops counting as
resolved while snoozed.
models.Alert gains Resolution (reason, successor and the engine's
summary), copied by cloneAlert and filled by GetRecentlyResolved, so
the state snapshot, websocket, assistant context, Patrol seed and the
resolved-alerts tool all report where the alert went. The resolved
resource change carries the summary as its reason with alert_resolution
metadata, and incident memory titles the close with it.
In the UI, the history row reads "moved to agent" with the account in
its title and on the phone card, and Patrol's detail shows a Moved
badge with the handover sentence under the summary.
Rejected active-alert writes still reached history, firing events and
notification dispatch. Make admission explicit across detector and restore
paths so repeated suppressed observations cannot create those side effects.
Reconcile restored alerts when persisted resource policy attaches at startup,
reconcile again after native aliases become canonical, and refresh shared
state before those alerts reach clients.
Preserve occurrence identity when retained alerts re-fire, including durable
history reconstruction and delayed lifecycle replay.
Keep retained table and card components mounted when scrolling shifts the
visible window, so policy editors do not lose unsaved state. Show actual
future maintenance times and the paused-attention effect.
Keep synthetic host machine identities stable across restarts so saved
operator policy remains attached during verification.
Qualify server update and rollback instructions by helper ownership, and
stop signed installer recipes before execution if a download or signature
check fails.
Refs #2237
Patrol stopped calling DetectChanges when it moved to agentic execution
(1b1c9bb2a3). Recent-change context comes from the unified-resource
timeline, and the detector survives only as a fallback reader of the
legacy ai_changes.json that older releases wrote. The previous change
removed the detector's unjoined save; the detection itself was still
there, kept alive only by tests.
Remove DetectChanges, its snapshot diffing, ResourceSnapshot (and its
internal/ai alias), change-ID generation, the description formatters and
the detector's lock. The history is now loaded once and never modified,
so reads need no lock. The read APIs, trimming and legacy loading are
unchanged.
Tests that used detection to seed the fallback now write a legacy
ai_changes.json and load it, the same path production reads. With
loading broken, 10 of the 12 re-seeded internal/ai tests fail; the other
two assert that the fallback is absent or loses to the canonical
timeline. Two formatter comparisons now reject empty output, which let
them pass against an empty detector. Tests of the detection logic, its
heap stability and ID uniqueness go with it.
DetectChanges launched an untracked goroutine to rewrite ai_changes.json.
Tests backed by a temp directory slept 20ms or 100ms hoping it had
finished; when the goroutine started late it wrote into a directory that
cleanup was removing, which is the latent TempDir race already fixed for
IncidentStore and KnowledgeStore.
Nothing in the runtime has called DetectChanges since patrol moved to
agentic execution (1b1c9bb2a3). Recent-change context comes from the
unified-resource timeline, and the detector is only its read fallback, so
the save goroutine was reachable only from tests. Joining it would harden
a dead legacy writer, so remove it along with saveToDisk and the unused
save-scheduling fields. The detector still loads legacy history at
construction; detected changes stay in memory.
Replace the sleeping tests with TestChangeDetector_LegacyHistoryIsReadOnly,
which fails 210/300 runs against the old async writer. With the save
delayed 0-150ms, 349 of 600 old-code saves ran after their test's
directory was removed; none remain. Also drop two sleeps after
RemediationLog.Log, which already persists before returning.
Every IncidentStore mutation queues a coalesced saveAsync goroutine
that later runs MkdirAll on the data directory and writes its .tmp
file. Nothing tracked those goroutines, so a t.TempDir-backed test
could finish while one was still queued. The save then either wrote
into the directory while TempDir's RemoveAll was deleting it (CI:
"unlinkat ...: directory not empty" in
TestIncidentQueryCanonicalOccurrencesAndNotes/memory, run 35932105100)
or recreated the directory after cleanup and leaked it. The test's
synchronous saveToDisk did not help: it cannot join a save that is
already queued.
Track queued saves with a WaitGroup and add an unexported flush(),
mirroring KnowledgeStore (47cadcc188). The disk-backed incident tests
now join through flush in place of polling, a 20ms sleep or waiting
for a save count. The canonical-occurrences test reloads from
RecordNote's own asynchronous save instead of a manual checkpoint.
TestIncidentStoreFlushJoinsQueuedSave holds the save lock to prove
flush waits for a queued save that has not started.
With a 0-3ms delay injected before each save, the old code leaked a
recreated data directory in 233 of 300 runs; with this change, none.
Checkpoint timing, coalescing and the stored format are unchanged.
The hosted backend shard and local repetition showed TempDir cleanup racing RecordNote persistence. Wait for the existing asynchronous save in this test so validation cannot fail after assertions pass; runtime behaviour is unchanged.
Change-source: pulse-maintainer
Filter canonical history before selecting occurrences, preserve source evidence
and expose bounded reads and failures. Reconcile duplicate saved shells without
splitting one alert lifecycle, and keep note identity and canonical risk intact.
Carry attributed operator notes into Assistant. Preserve mobile investigations
across layout changes, transfer composer focus on handoff and keep long event
text readable. Record scoped qualification and its unresolved wider limits.
Refs #1782
Retain unordered and subsecond recurrence boundaries through incident-store reconstruction; assert unchanged projected identity and zero checkpoint replacements after replay. Canonical events remain in memory, with no durable event-store or installed write-rate claim.
Change-source: pulse-maintainer
The real monitoring lifecycle dispatcher reproduces a closed incident reopening when its next occurrence fires: canonical projection selected all later events for the same alert. Bound retained-shell projections to their exact start and the next retained start, keeping subsecond recurrence and historical acknowledgement separate. Add in-memory callback coverage with both projections attached; twenty race repetitions pass along with focused incident tests. No retention migration, canonical-only fallback rewrite, notification-delivery claim or aggregate write-byte claim.
Change-source: pulse-maintainer
Issue #1966 reports distinct incident IDs for one occurrence. Deterministic lifecycle replay creates eleven shells for one resolved start, including after JSON restart; open-only matching also lets an old resolution close a newer occurrence. Match exact lifecycle starts and retain closed identity on replay, without changing legacy zero-start matching or retention. Add recurrence, restart and canonical-shell regression coverage; focused incident race tests pass for twenty repetitions. This does not establish total write savings or migrate existing duplicates.
Change-source: pulse-maintainer
Repeated alert lifecycle evaluations retain one occurrence but still replace the whole incident file. Compare bounded existing bytes before replacement, preserving changed-state saves and retry after failure or file loss.
Focused regression reproduced 21 replacements for one initial save plus 20 unchanged evaluations before the fix. Repeated race-enabled incident tests cover unchanged checkpoints, metadata, restart, resolution, recurrence and failed-write retry. This does not establish the cause of all writes or duplicate incidents reported in #1966.
Change-source: pulse-maintainer
Issue #1892 reports successful replication information requiring acknowledgement. Preserve provider information on resources while excluding TrueNAS INFO-level conditions from active alert synchronisation. Warning conditions still activate and clear when downgraded to information.
Resource-incident timeline events contain numeric placeholders, not threshold evidence. Avoid displaying these as a fictitious 0 >= 0 trigger while retaining numeric metric formatting. Focused regressions reproduce both failures and pass with these changes.
Change-source: pulse-maintainer
Projection replay walked the entire alert event log on every boot, twice
(monitor start and the resource-store boundary), synchronously on the
serving path, and every replayed event queued a full incident-store JSON
rewrite. A 64k-event log made startup exceed the dev supervisor's
health-kill window, so the backend restarted forever and never served.
The event log now owns per-consumer projection watermarks in
alert_store_meta, WalkOldest takes an AfterID cursor, and the single
catch-up pass runs in the background at the canonical resource-store
boundary with periodic watermark checkpoints. The watermark only
advances when both the incident store and the canonical change recorder
are attached, so partial-surface passes repair without marking events
applied, and lowering the watermark forces a full repair replay.
Incident-store saves now coalesce: a burst of mutations queues one
whole-store serialization instead of one per event, live paths included.
Registers the alert-lifecycle-replay-startup-scalability coverage gap
and amends the alerts and monitoring subsystem contracts accordingly.
Log and MarkRolledBack saved through fire-and-forget goroutines, so a
write could land after the owner tore down the data dir and the newest
record could be dropped entirely on process exit. CI hit the teardown
race as TempDir cleanup failures in internal/ai/memory, and the mtime
quiescence helper cannot see a goroutine that has not written yet. The
single production write site runs after a remediation command has
already executed, so the small synchronous file write costs nothing
observable. Mutation now happens under the lock, persistence after
release, still serialized by persistMu.
Contract-Neutral: behavioral bug fix: remediation log fire-and-forget saves raced owner teardown and could drop the newest record on exit; no public contract delta
Adds branch-coverage tests for eight packages whose target functions were
measured at 0% before this change. Every named target was verified to move
by running each package's coverage with and without the new file.
- internal/ai/eval: all 36 Scenario constructors and the four PatrolScenario
constructors 0% -> 100%. These are catalog-invariant tests, not literal
echoes: unique names, populated required fields, runnable assertions, tool
references checked against the agentcapabilities registry, and exact
assertion-count deltas across the env-gated conditional appends. A parity
test scans scenarios.go itself, so adding a constructor without registering
it in the table now fails rather than silently going untested.
- cmd/pulse-control-plane: nine MSP and tenant-runtime print helpers
0% -> 100%, covering the nil, empty-slice and optional-field arms.
- internal/ai/memory: RemediationLog GetByID, MarkRolledBack and
GetRollbackable 0% -> 100%, pinning the overwrite-vs-preserve contract on
RollbackInfo and each falsy arm of the rollbackable predicate.
- internal/alerts/config: AlertConfig.UnmarshalJSON 0% -> 90% and
NormalizeAlertConfigAliases 52.9% -> 94.1%.
- internal/config: RunMigrationIfNeeded 0% -> 100%, copyFile 0% -> 88.9%.
- internal/mock: AvailabilityFixtures, FixtureGraph.SupplementalChanges and
generateMockHostRate 0% -> 100%.
- internal/api: testProxmoxPlatformConnection 0% -> 100% through its injected
connect func, so no network is involved.
- internal/servicediscovery: needsDeepScan 0% -> 100% across every return arm
including the confidence boundary.
No source file is modified. Adversarial review found no rejects; two findings
were acted on before committing, replacing a circular catalog-count assertion
with the real source-parity scan and reducing an AllPatrolScenarios test that
compared the function against the same constructors it calls to the ordering
and completeness signal that is actually independent.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Contract-Neutral: test-only branch coverage, no source or contract change
Clears the ten dupl pairs across internal/ai:
- patrol_intelligence.go, tools_query.go, tools_storage.go: VM and LXC
system-container paths collapse into generics over read-state view
method subsets (gatherGuestIntelligenceFromViews, canonicalGuestGetResult
+ guestViewGetResult, addCanonicalGuestSearchMatches +
addGuestViewSearchMatches, appendGuestDiskSummaries).
- tools_file.go: append/write share executeFileMutation driven by
fileMutationSpec (approval-command text, shell redirect, verification
strategy stay per-action and verbatim).
- tools_kubernetes.go: deployment restart / pod delete share
executeKubernetesResourceAction driven by kubernetesResourceAction.
- providers/anthropic.go + anthropic_oauth.go: message conversion shared
via convertMessagesToAnthropic (the OAuth copy was annotated 'same as
regular client').
- memory/changes.go + memory/remediation.go: history loading shared via
the generic loadMemoryHistory in memory/paths.go (10 MiB cap, sort,
missing-file semantics preserved via a found flag).
- findings.go and unified/alerts.go: Finding/findingJSON and
UnifiedFinding/unifiedFindingJSON are deliberate marshal-mirror twins
(AlertIdentifier json:"-" vs alert_identifier round-trip); merging
would break every public literal. Suppressed with nolint:dupl and
enforced instead by new reflect-based mirror-sync tests.
Contract Extension Points name the shared helpers and the mirror
invariant. Full ./internal/ai/... test tree passes.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
Major new AI capabilities for infrastructure monitoring:
Investigation System:
- Autonomous finding investigation with configurable autonomy levels
- Investigation orchestrator with rate limiting and guardrails
- Safety checks for read-only mode enforcement
- Chat-based investigation with approval workflows
Forecasting & Remediation:
- Trend forecasting for resource capacity planning
- Remediation engine for generating fix proposals
- Circuit breaker for AI operation protection
Unified Findings:
- Unified store bridging alerts and AI findings
- Correlation and root cause analysis
- Incident coordinator with metrics recording
New Frontend:
- AI Intelligence page with patrol controls
- Investigation drawer for finding details
- Unified findings panel with actions
Supporting Infrastructure:
- Learning store for user preference tracking
- Proxmox event ingestion and correlation
- Enhanced patrol with investigation triggers
- Add freebsd-amd64 and freebsd-arm64 to normalizeUnifiedAgentArch()
so the download endpoint serves FreeBSD binaries when requested
- Add FreeBSD/pfSense/OPNsense platform option to agent setup UI
with note about bash installation requirement
- Add FreeBSD test cases to unified_agent_test.go
Fixes installation on pfSense/OPNsense where users were getting 404
errors because the backend didn't recognize the freebsd-amd64 arch
parameter from install.sh.
- Add persistent volume mounts for Go/npm caches (faster rebuilds)
- Add shell config with helpful aliases and custom prompt
- Add comprehensive devcontainer documentation
- Add pre-commit hooks for Go formatting and linting
- Use go-version-file in CI workflows instead of hardcoded versions
- Simplify docker compose commands with --wait flag
- Add gitignore entries for devcontainer auth files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Backend:
- Enhanced buildEnrichedResourceContext to ALWAYS show learned baselines with
status indicators (normal/elevated/anomaly) instead of only when anomalous
- This makes Pulse Pro's 'moat' visible - users can see the AI understands
their infrastructure's normal behavior patterns
- Added baseline import to service.go
Frontend (user changes):
- Added incident event type filtering with toggle buttons
- Added resource incident panel to view all incidents for a resource
- Added timeline expand/collapse functionality in alert history
- Added incident note saving with proper incidentId tracking
- Added startedAt parameter for proper incident timeline loading
pulse-triage[bot]