Each tenant monitor owns its own IncidentStore, but incident and
incident-event IDs came from package-level int64 counters bumped with ++
under nothing but the calling store's mutex. Two tenants recording
alerts at once raced on them: go test -race failed
TestLeavingMockModeClearsEveryRunningTenantMonitor on PR 2698 at
incidents.go:1753. The remediation record counter had the same shape,
and remediation logs are created per org.
All six internal/ai/memory ID generators now mint through one atomic
helper. The numeric suffix also stops wrapping at 1000: even with atomic
counters, one generator minting more than a thousand IDs in a
wall-clock second repeated an ID, and incident notes find their incident
by ID. The prefix and timestamp layout is unchanged, and intToString
lost its last callers.
ids_test.go mints 2,400 IDs from every generator on 16 goroutines and
records alerts into two tenant stores concurrently. Against the old
files it reports DATA RACE at incidents.go:1753/1760 and
remediation.go:355, and every generator mints duplicate or wrapped IDs.
Patrol stopped calling DetectChanges when it moved to agentic execution
(1b1c9bb2a3). Recent-change context comes from the unified-resource
timeline, and the detector survives only as a fallback reader of the
legacy ai_changes.json that older releases wrote. The previous change
removed the detector's unjoined save; the detection itself was still
there, kept alive only by tests.
Remove DetectChanges, its snapshot diffing, ResourceSnapshot (and its
internal/ai alias), change-ID generation, the description formatters and
the detector's lock. The history is now loaded once and never modified,
so reads need no lock. The read APIs, trimming and legacy loading are
unchanged.
Tests that used detection to seed the fallback now write a legacy
ai_changes.json and load it, the same path production reads. With
loading broken, 10 of the 12 re-seeded internal/ai tests fail; the other
two assert that the fallback is absent or loses to the canonical
timeline. Two formatter comparisons now reject empty output, which let
them pass against an empty detector. Tests of the detection logic, its
heap stability and ID uniqueness go with it.
DetectChanges launched an untracked goroutine to rewrite ai_changes.json.
Tests backed by a temp directory slept 20ms or 100ms hoping it had
finished; when the goroutine started late it wrote into a directory that
cleanup was removing, which is the latent TempDir race already fixed for
IncidentStore and KnowledgeStore.
Nothing in the runtime has called DetectChanges since patrol moved to
agentic execution (1b1c9bb2a3). Recent-change context comes from the
unified-resource timeline, and the detector is only its read fallback, so
the save goroutine was reachable only from tests. Joining it would harden
a dead legacy writer, so remove it along with saveToDisk and the unused
save-scheduling fields. The detector still loads legacy history at
construction; detected changes stay in memory.
Replace the sleeping tests with TestChangeDetector_LegacyHistoryIsReadOnly,
which fails 210/300 runs against the old async writer. With the save
delayed 0-150ms, 349 of 600 old-code saves ran after their test's
directory was removed; none remain. Also drop two sleeps after
RemediationLog.Log, which already persists before returning.
Every IncidentStore mutation queues a coalesced saveAsync goroutine
that later runs MkdirAll on the data directory and writes its .tmp
file. Nothing tracked those goroutines, so a t.TempDir-backed test
could finish while one was still queued. The save then either wrote
into the directory while TempDir's RemoveAll was deleting it (CI:
"unlinkat ...: directory not empty" in
TestIncidentQueryCanonicalOccurrencesAndNotes/memory, run 35932105100)
or recreated the directory after cleanup and leaked it. The test's
synchronous saveToDisk did not help: it cannot join a save that is
already queued.
Track queued saves with a WaitGroup and add an unexported flush(),
mirroring KnowledgeStore (47cadcc188). The disk-backed incident tests
now join through flush in place of polling, a 20ms sleep or waiting
for a save count. The canonical-occurrences test reloads from
RecordNote's own asynchronous save instead of a manual checkpoint.
TestIncidentStoreFlushJoinsQueuedSave holds the save lock to prove
flush waits for a queued save that has not started.
With a 0-3ms delay injected before each save, the old code leaked a
recreated data directory in 233 of 300 runs; with this change, none.
Checkpoint timing, coalescing and the stored format are unchanged.
Major new AI capabilities for infrastructure monitoring:
Investigation System:
- Autonomous finding investigation with configurable autonomy levels
- Investigation orchestrator with rate limiting and guardrails
- Safety checks for read-only mode enforcement
- Chat-based investigation with approval workflows
Forecasting & Remediation:
- Trend forecasting for resource capacity planning
- Remediation engine for generating fix proposals
- Circuit breaker for AI operation protection
Unified Findings:
- Unified store bridging alerts and AI findings
- Correlation and root cause analysis
- Incident coordinator with metrics recording
New Frontend:
- AI Intelligence page with patrol controls
- Investigation drawer for finding details
- Unified findings panel with actions
Supporting Infrastructure:
- Learning store for user preference tracking
- Proxmox event ingestion and correlation
- Enhanced patrol with investigation triggers
- Add persistent volume mounts for Go/npm caches (faster rebuilds)
- Add shell config with helpful aliases and custom prompt
- Add comprehensive devcontainer documentation
- Add pre-commit hooks for Go formatting and linting
- Use go-version-file in CI workflows instead of hardcoded versions
- Simplify docker compose commands with --wait flag
- Add gitignore entries for devcontainer auth files
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
rcourtman