AI Discovery ran a separate model-backed service scanner beside Patrol
and the Assistant: its own store, scheduler, model route, API, tool and
drawer UI. No telemetry counter ever measured its use (the telemetry
discovery_enabled flag OR'd it with network subnet discovery), the
maintainer judged it a flop, and the context it produced is already
available to the models through pulse_query, pulse_read and the
unified-resource context sections.
This deletes internal/servicediscovery, internal/ai/infradiscovery, the
pulse_discovery tool, the /api/discovery routes, the mock Discovery
fixtures, the Discovery tabs in the resource, guest, node and Docker host
drawers, the identified-service cards, readiness badges, provenance
markers, suggested web-interface URLs, the Workloads AI Context column,
the chat "Discovery is off" hint and the AI Service Context settings page.
The old settings route redirects to Assistant settings. The model-bound
sanitizer allow-list existed only to pass Discovery context through, so it
goes too, along with the CopyableCodeRow primitive that only the
Discovery tab used.
Network subnet discovery (/api/discover, the Infrastructure discovery
settings) and the discoveryTarget agent coordinates are separate features
and stay. Stored discovery_enabled, discovery_interval_hours and
discovery_model AI settings are ignored on load; the settings API decoder
is lenient, so older frontends saving those fields still succeed.
Since 4edf5f8265, ChatRequest in internal/api/ai_handler.go has no
autonomous_mode field: Go's decoder drops it and HandleChat passes
approval-required mode into the chat service for every /api/ai/chat
request. The browser still serialized autonomous_mode from its send
options, and the eval runner kept a per-step override that sent it too,
so both read as if the client chose the execution mode while the server
ignored it.
Drop the field from AIChatAPI.chat, SendMessageOptions,
ChatMessageRequestContext and the clone, retry and restore helpers, and
Step.AutonomousMode from internal/ai/eval. aiChatStore.context's
autonomousMode stays as a drawer-only flag, so scoped handoffs still show
"Ask first" and "Approval required before any action." Tests now assert
the request carries no autonomous_mode; the backend test proving a stale
client's autonomous_mode:true cannot elevate a request stays.
Contract prose that described autonomous_mode:false as a per-request
override set by the browser now describes the server-set mode and the
drawer disclosure.
A paying operator asked the Assistant to reboot five Proxmox VMs matching
a name pattern (GitHub #1782, support mail 2026-08-26 and 2026-08-29). The
model resolved the VMs and then ended with a report that invented a
prerequisite: a QEMU guest agent on 6.3.2, a "discovery binding" on stable
6.4.0. It never planned the action. Three defects made the governed path
fail whenever the model did try it, and nothing refused the prose ending
when it did not:
- pulse_control handed the session-scoped id (vm:<node>:<vmid>) to the
action lifecycle, whose registry keys on canonical unified ids, so a
Proxmox guest plan could never resolve.
- pulse_control gated the action on the legacy per-executor action list,
which never carried the canonical "reboot" capability Proxmox guests
advertise, so "reboot" was refused as not permitted before planning.
- A reference absent from the session context was refused with "resource
discovery is required" even when the unified inventory resolved it.
pulse_control now binds its target to the canonical unified resource
(session alias first, then a unique inventory match, refusing ambiguity
with candidate ids and naming the pulse_query recovery on a miss), passes
the canonical id to the planner, and answers "not available" only from
the resource's current advertised capabilities. The FSM ordering block
and the shared operating instructions state that a recoverable block is
not a limitation to report, and the instructions require the governed
action tool for advertised capabilities. The agentic loop adds a bounded
advertised-action gate: when the operator asked for a lifecycle action,
pulse_control was offered but never submitted, and a session-resolved
resource advertises the action, a tool-free final answer is refused once
with the exact per-target calls.
Covered by tools and loop unit tests (the #1782 transcript against a
scripted provider fails on the previous code with the two exact errors
above), a prompt-contract test, and the live eval scenario
ProxmoxBulkLifecycleActionScenario.
Adds branch-coverage tests for eight packages whose target functions were
measured at 0% before this change. Every named target was verified to move
by running each package's coverage with and without the new file.
- internal/ai/eval: all 36 Scenario constructors and the four PatrolScenario
constructors 0% -> 100%. These are catalog-invariant tests, not literal
echoes: unique names, populated required fields, runnable assertions, tool
references checked against the agentcapabilities registry, and exact
assertion-count deltas across the env-gated conditional appends. A parity
test scans scenarios.go itself, so adding a constructor without registering
it in the table now fails rather than silently going untested.
- cmd/pulse-control-plane: nine MSP and tenant-runtime print helpers
0% -> 100%, covering the nil, empty-slice and optional-field arms.
- internal/ai/memory: RemediationLog GetByID, MarkRolledBack and
GetRollbackable 0% -> 100%, pinning the overwrite-vs-preserve contract on
RollbackInfo and each falsy arm of the rollbackable predicate.
- internal/alerts/config: AlertConfig.UnmarshalJSON 0% -> 90% and
NormalizeAlertConfigAliases 52.9% -> 94.1%.
- internal/config: RunMigrationIfNeeded 0% -> 100%, copyFile 0% -> 88.9%.
- internal/mock: AvailabilityFixtures, FixtureGraph.SupplementalChanges and
generateMockHostRate 0% -> 100%.
- internal/api: testProxmoxPlatformConnection 0% -> 100% through its injected
connect func, so no network is involved.
- internal/servicediscovery: needsDeepScan 0% -> 100% across every return arm
including the confidence boundary.
No source file is modified. Adversarial review found no rejects; two findings
were acted on before committing, replacing a circular catalog-count assertion
with the real source-parity scan and reducing an AllPatrolScenarios test that
compared the function against the same constructors it calls to the ordering
and completeness signal that is actually independent.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Contract-Neutral: test-only branch coverage, no source or contract change
Five new branch-coverage tests taking previously unreached functions from zero
to covered, with no source or existing test touched.
cmd/pulse-control-plane: the MSP status failure lookup across exact, absent,
case-differing and substring inputs, and the status state ordering proved
deterministic across repeated runs over the same map so a random map iteration
order cannot pass by luck.
internal/ai/eval: the signal coverage assertion driven through both the lazy
quality evaluation path and the supplied-quality path, at the minimum rate
boundary and either side of it, asserting the formatted message rather than
only the pass or fail outcome, plus the approval write command builder.
internal/api: the setup script server name derivation including the fallback
arm, the patrol autonomy level validator, the approval risk assessment across
every level it can return, and the two typed error messages.
internal/hostagent: the token already exists classifier over each phrase it
recognises, a wrapped error carrying the phrase, and near-miss messages that
must not match.
internal/monitoring: the limited temperature buffer including its truncation
boundary and its aliasing behaviour, and the mock series generators asserted on
determinism under a fixed seed and on their value bounds.
Reported for follow-up, not fixed here: the node escalation arm in the approval
risk assessment is unreachable for the current command set, generatePlateauSeries
panics on a negative point count and leaves its tail unclamped, and the limited
temperature buffer returns a slice that aliases its internal storage.
Contract-Neutral: test-only branch coverage, no contract surface touched
Manifest-backed MCP tools, prompts, and resources with surface affordance contracts; agent capability manifest and governance projection; API contract tests and capability route projection; operations-loop and intelligence-funnel telemetry; release-control subsystem documentation, registry, and tooling; licensing and configuration.
- Add unit tests for internal/ai/eval package
- Validate configuration, retry logic, and custom SSE parsing
- Enables coverage for eval framework without requiring live Pulse server
Refactor patrol eval runner to use a dual approach:
1. Poll GET /api/ai/patrol/status until Running=false (primary signal)
2. Best-effort SSE stream connection for tool event visibility
Changes:
- Add status polling loop with configurable timeout
- Make SSE stream optional (may not connect in time)
- Add Completed flag to PatrolRunResult
- Improve assertion error messages
- Add new scenarios and assertions
This is more reliable than relying solely on SSE stream which
may timeout waiting for headers during slow patrol initialization.
Add comprehensive patrol evaluation framework:
- patrol.go: Runner for patrol scenarios with streaming support
- patrol_assertions.go: Assertions for tool usage, findings, timing
- patrol_scenarios.go: Scenarios for basic, investigation, finding quality
- eval_test.go: Unit tests for patrol eval runner
Scenarios:
- patrol-basic: Verifies patrol completes with tools and findings
- patrol-investigation: Ensures investigation before reporting
- patrol-finding-quality: Validates finding structure and evidence
Run with: go run ./cmd/eval -scenario patrol
- Add retry logic for transient failures (phantom, stream, empty response)
- Add environment variable overrides for infrastructure naming
- Add JSON report output per scenario
- Expand assertions with new validation types
- Add more comprehensive test scenarios
- Add docs/EVAL.md with usage documentation
The eval harness now better handles flaky AI responses and provides
detailed reports for debugging.
rcourtman