mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
fix(patrol): preserve refusals and expose measured summary evidence
A provider policy refusal was classified as a connection failure, while summary tools mixed high-utilisation heuristics with unchecked health claims. Preserve explicit refusals before tool recovery and provide retained metrics with source scope, observation timestamps and bucket extrema for diagnosis. Record the live qualification limits and the shared temporal tier-query gap.
This commit is contained in:
rcourtman
@@ -244,3 +244,70 @@ and collapse, output scrolling, final-answer pixels, Escape, session history
|
||||
and reload. The actual cached readiness result and actual HTTP 409 gate were
|
||||
verified after the final source changes. Local receipts bind source hashes,
|
||||
transcripts and screenshots under the homelab resource-browser artifact set.
|
||||
|
||||
|
||||
## Evidence interpretation continuation, 2026-09-05
|
||||
|
||||
The provider refusal is now a first-class `provider_refusal` diagnostic. The
|
||||
subscription transport preserves the terminal refusal before either structured
|
||||
completion or a preceding tool call can be recovered. Both process exit paths
|
||||
are covered. Saved legacy evidence is corrected only from a complete terminal
|
||||
JSON envelope with `stop_reason=refusal`, preserving the original time and probe
|
||||
counts. This produces clear policy/support guidance without a new provider
|
||||
request, weakened CLI restrictions or a verified Patrol mode. The real saved
|
||||
result was inspected in desktop/mobile settings and the manual endpoint still
|
||||
returned HTTP 409.
|
||||
|
||||
The model-facing summary no longer invokes the report narrator. Its canonical
|
||||
`MetricEvidenceProvider` contract reads the report engine's retained data and
|
||||
returns units, extrema, means, latest values, point counts and observation
|
||||
timestamps. Extrema include recorded bucket ranges and identify their buckets.
|
||||
Means and latest values can be bucket averages. The response names alerts,
|
||||
findings, disk health, backups and topology as not queried, requiring their own
|
||||
tools before health or causal conclusions. A 92% memory reading is returned as
|
||||
evidence without a heuristic critical or healthy label. Existing export/report
|
||||
narratives and health cards remain separate behavior and have not been qualified
|
||||
or repaired by this change.
|
||||
|
||||
The first live summary investigation correctly separated high utilisation from
|
||||
proven memory pressure and disclosed the short, stale retained window. It still
|
||||
dismissed a recorded historical temperature maximum because the current sensor
|
||||
reading was lower. That is an incorrect inference, not proof that the recorded
|
||||
peak is an artefact. The summary now explicitly distinguishes preserved bucket
|
||||
extrema from bucket averages and includes the extrema's bucket timestamps.
|
||||
Unsupported workload-change inference and assumptions about a previously
|
||||
working agent channel remain model-reasoning concerns.
|
||||
|
||||
A separate local fixture reproduced the shared history query coverage defect.
|
||||
It wrote CPU=10 in the minute tier at 21:37 and CPU=20 in raw data at 22:16.
|
||||
Both 24-hour `Store.Query` and `Store.QueryAll` returned only the older point.
|
||||
A two-hour query returned the newer raw point. The preferred non-empty tier
|
||||
therefore hides newer evidence. Temporal tier reconciliation, including batch
|
||||
queries and aggregation semantics, is recorded in the performance-and-scalability
|
||||
contract under the existing outcome gap. The evidence tool exposes its returned
|
||||
coverage and cannot make the underlying query complete. This is separate from
|
||||
missing observations that were never collected.
|
||||
|
||||
The final live read-only investigation completed in 195 seconds with ten visible
|
||||
tool controls. It resolved the named resource, preserved the extrema bucket
|
||||
timestamps, explained the difference between extrema and plotted averages,
|
||||
disclosed the 83-point / 82-minute returned span and roughly 45-minute age,
|
||||
and distinguished high utilisation from demonstrated memory pressure. The
|
||||
failed host pressure read was correctly treated as unavailable evidence.
|
||||
However, the answer still speculated about a sensor/startup artefact, inferred
|
||||
monitoring restart from coincident timestamps, claimed continuous coverage from
|
||||
minute spacing, and called the returned span a retention limit. None of those
|
||||
claims was established by the tool results. Recommending simply waiting a day
|
||||
is insufficient given the independently reproduced tier query defect. This
|
||||
is a bounded evidence-contract pass, with diagnosis qualification still open.
|
||||
|
||||
Final browser inspection covered `/patrol` and
|
||||
`/settings/pulse-intelligence/patrol` at 1440x1000 and 390x1000. The interaction
|
||||
matrix included Assistant submission/completion, keyboard and pointer expansion
|
||||
and collapse, scrollable tool outputs through their final entries, final answer
|
||||
pixels, Escape, session-history selection and reload. It also covered the actual
|
||||
cached provider-refusal settings and blocked manual execution, with no repeated
|
||||
provider readiness probe. The final saved-session pixel pass followed a correction
|
||||
to the tool's descriptive governance metadata. Runtime source hashes, transcript
|
||||
and screenshots are retained in the local homelab evidence-trust artifact set.
|
||||
These receipts do not qualify autonomous Patrol or infrastructure mutations.
|
||||
|
||||
@@ -5648,11 +5648,15 @@
|
||||
"status": "partial",
|
||||
"completion": {
|
||||
"state": "bounded-residual",
|
||||
"summary": "Performance is at the current release floor, but additional scalability headroom and non-blocking performance polish remain post-RC follow-up.",
|
||||
"summary": "Performance is at the current release floor with bounded residuals. Retained queries still require temporal tier reconciliation so an existing aggregate does not hide newer raw evidence. Scalability headroom and performance follow-up remain tracked separately.",
|
||||
"tracking": [
|
||||
{
|
||||
"kind": "lane-followup",
|
||||
"id": "performance-post-rc-headroom"
|
||||
},
|
||||
{
|
||||
"kind": "lane-followup",
|
||||
"id": "retained-metric-query-coverage"
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -9726,6 +9730,19 @@
|
||||
"pulse-pro"
|
||||
],
|
||||
"subsystem_ids": []
|
||||
},
|
||||
{
|
||||
"id": "retained-metric-query-coverage",
|
||||
"summary": "Repair temporal coverage across retained metric tiers. A 24-hour Query or QueryAll currently returns an older non-empty aggregate tier even when a newer raw point exists. Reconcile bucket precedence, extrema and downsampling consistently across single-series, all-series and batch reads. The model-facing evidence tool reports the returned timestamps but does not make this shared-store query complete. Required for trustworthy Patrol and Assistant history, tracked with patrol-assistant-customer-outcome-qualification.",
|
||||
"owner": "project-owner",
|
||||
"status": "planned",
|
||||
"recorded_at": "2026-09-05",
|
||||
"lane_ids": [
|
||||
"L10"
|
||||
],
|
||||
"subsystem_ids": [
|
||||
"performance-and-scalability"
|
||||
]
|
||||
}
|
||||
],
|
||||
"coverage_gaps": [
|
||||
@@ -10201,17 +10218,19 @@
|
||||
},
|
||||
{
|
||||
"id": "patrol-assistant-customer-outcome-qualification",
|
||||
"summary": "The 2026-09-05 product review found that Explain with Assistant opened a blank generic conversation and the attention workbench discarded richer canonical finding context. The explicit explanation dispatcher and canonical handoff repair address that frontend break, with scripted browser proof and regression tests. They do not qualify model reasoning, real infrastructure actions, or repeated customer value. Latest-report, monitoring-active, multi-ping telemetry excluding dev and deployment proof contained 127 paid installs, 71 with Patrol enabled and 23 with Assistant calls. Fourteen reported verified resolutions were concentrated in one install. Paid includes all non-free tiers, and activity cooccurrence is not a linked successful journey. Schema 17 outcome/provider/cost fields had no adoption in that review. Three real jobs remain to qualify across named provider/model/version configurations: unhealthy service diagnosis, backup or capacity risk, and supported VM/LXC plan, approval and verified result. Repeatability must be established across independent paid customer environments without pooling free-tier or ineligible installs. The same-day real maintainer homelab evaluation with Claude Opus 5 exposed missing Proxmox temperature-tool evidence, contradictory summary health, incomplete visible prose, alert-count confusion and an inaccurately classified provider refusal during Patrol readiness. The shared temperature projection and premature within-window evidence compaction are repaired, while the documented reasoning, completion, readiness and end-to-end outcome failures remain unqualified. The continuation repaired canonical resource lookup, operation-versus-disk filtering, Proxmox CPU topology and retained resource-scoped history across backend restarts. Unscoped summary/baseline modernization, wear semantics, disk risk explanation and complete retained-window coverage remain open. A named-resource repeat exposed overly restrictive current_resource recovery guidance, now corrected without weakening the attachment boundary. Live testing also exposed warning-based API execution after an incomplete provider check. Runtime and API now enforce the selected-mode verdict, including saved incomplete results, and the repaired live endpoint returns HTTP 409. One read-only test run started before the gate correction and was stopped, so it is not qualification evidence.",
|
||||
"summary": "The 2026-09-05 product review found that Explain with Assistant opened a blank generic conversation and the attention workbench discarded richer canonical finding context. The explicit explanation dispatcher and canonical handoff repair address that frontend break, with scripted browser proof and regression tests. They do not qualify model reasoning, real infrastructure actions, or repeated customer value. Latest-report, monitoring-active, multi-ping telemetry excluding dev and deployment proof contained 127 paid installs, 71 with Patrol enabled and 23 with Assistant calls. Fourteen reported verified resolutions were concentrated in one install. Paid includes all non-free tiers, and activity cooccurrence is not a linked successful journey. Schema 17 outcome/provider/cost fields had no adoption in that review. Three real jobs remain to qualify across named provider/model/version configurations: unhealthy service diagnosis, backup or capacity risk, and supported VM/LXC plan, approval and verified result. Repeatability must be established across independent paid customer environments without pooling free-tier or ineligible installs. The same-day real maintainer homelab evaluation with Claude Opus 5 exposed missing Proxmox temperature-tool evidence, contradictory summary health, incomplete visible prose, alert-count confusion and an inaccurately classified provider refusal during Patrol readiness. The shared temperature projection and premature within-window evidence compaction are repaired, while the documented reasoning, completion, readiness and end-to-end outcome failures remain unqualified. The continuation repaired canonical resource lookup, operation-versus-disk filtering, Proxmox CPU topology and retained resource-scoped history across backend restarts. Unscoped summary/baseline modernization, wear semantics, disk risk explanation and complete retained-window coverage remain open. A named-resource repeat exposed overly restrictive current_resource recovery guidance, now corrected without weakening the attachment boundary. Live testing also exposed warning-based API execution after an incomplete provider check. Runtime and API now enforce the selected-mode verdict, including saved incomplete results, and the repaired live endpoint returns HTTP 409. One read-only test run started before the gate correction and was stopped, so it is not qualification evidence. The evidence interpretation continuation replaces model-facing heuristic summaries with retained statistics, explicit source scope, real observation coverage and preserved bucket extrema/timestamps. A typed provider_refusal diagnosis now survives CLI exit and legacy cache decoding without granting readiness or repeating the refused request. The final real summary distinguished utilisation from demonstrated pressure and explained extrema versus bucket averages, but still speculated about sensor artefacts and monitoring restarts, inferred continuous coverage from minute spacing, and conflated returned history with retained history. Reasoning remains unqualified. A local fixture confirmed that 24-hour Query and QueryAll can hide newer raw points when a preferred aggregate tier is non-empty. Shared temporal tier reconciliation, including batch queries, is required in L10 before claiming current requested-window history.",
|
||||
"owner": "project-owner",
|
||||
"status": "planned",
|
||||
"recorded_at": "2026-09-05",
|
||||
"lane_ids": [
|
||||
"L6"
|
||||
"L6",
|
||||
"L10"
|
||||
],
|
||||
"subsystem_ids": [
|
||||
"ai-runtime",
|
||||
"api-contracts",
|
||||
"patrol-intelligence"
|
||||
"patrol-intelligence",
|
||||
"performance-and-scalability"
|
||||
],
|
||||
"proposed_resolution": "lane-expansion",
|
||||
"coverage_impact": 4,
|
||||
@@ -10221,6 +10240,11 @@
|
||||
"path": "docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md",
|
||||
"kind": "file"
|
||||
},
|
||||
{
|
||||
"repo": "pulse",
|
||||
"path": "docs/release-control/v6/internal/subsystems/performance-and-scalability.md",
|
||||
"kind": "file"
|
||||
},
|
||||
{
|
||||
"repo": "pulse",
|
||||
"path": "internal/ai/qualification",
|
||||
@@ -10358,7 +10382,8 @@
|
||||
"recorded_at": "2026-09-05",
|
||||
"target_id": "v6-product-lane-expansion",
|
||||
"current_lane_ids": [
|
||||
"L6"
|
||||
"L6",
|
||||
"L10"
|
||||
],
|
||||
"coverage_gap_ids": [
|
||||
"patrol-assistant-customer-outcome-qualification"
|
||||
@@ -10366,7 +10391,8 @@
|
||||
"subsystem_ids": [
|
||||
"ai-runtime",
|
||||
"api-contracts",
|
||||
"patrol-intelligence"
|
||||
"patrol-intelligence",
|
||||
"performance-and-scalability"
|
||||
],
|
||||
"demand_evidence": [
|
||||
"user-direction: 2026-09-05 prioritise dependable Patrol and Assistant outcomes as core Pulse Pro value",
|
||||
|
||||
@@ -36,8 +36,8 @@ metrics target. The retained SQLite metrics store supplies the requested window
|
||||
across backend restarts. An available store error remains an error, with no
|
||||
silent downgrade to a shorter in-memory history. The in-memory-only adapter
|
||||
remains a compatibility boundary for installations without a retained store.
|
||||
The unscoped summary and baseline paths still use their existing providers and
|
||||
are a modernization residual, not proof of retained-history coverage.
|
||||
Unscoped `pulse_metrics` performance and baseline paths still use their existing
|
||||
providers and are a modernization residual, not proof of retained-history coverage.
|
||||
`TestPerformanceMetricsRetainedAcrossRestart` verifies native store coordinates,
|
||||
canonical response identity, node/agent/guest families, restart retention and
|
||||
store failure. The `node` get alias resolves to canonical `agent`, whose CPU
|
||||
@@ -50,8 +50,17 @@ suitability unassessed. Completed tool/context evidence remains available, and
|
||||
an explicit provider-error guard prevents authorizing Patrol. Provider errors
|
||||
remain distinct from measured latency failures. The exact legacy unfinished-probe latency verdict is corrected when persisted
|
||||
evidence is decoded, preserving its timestamp, provider cause and completed
|
||||
checks without a new provider request or granting a verified mode. Provider refusal taxonomy and
|
||||
supported subscription readiness remain unresolved qualification work.
|
||||
checks without a new provider request or granting a verified mode.
|
||||
|
||||
An explicit Claude terminal `stop_reason=refusal` becomes the typed provider
|
||||
refusal error before any tool call or final answer can be recovered, on either
|
||||
successful or unsuccessful process exit. Runtime and connection diagnostics
|
||||
project `provider_refusal` with policy/support guidance rather than network,
|
||||
latency or billing guidance. A saved legacy CLI envelope is reclassified only
|
||||
when its complete terminal JSON contains that explicit signal. Initial probe
|
||||
counts and the original evaluation time survive, the CLI dump is removed from
|
||||
the displayed refusal detail, and no provider request or authorization is
|
||||
created. Supported subscription Patrol execution remains unqualified.
|
||||
|
||||
Within an active investigation, tool observations are preserved across provider
|
||||
turns while the full request fits the model context window. Turn age is not a
|
||||
@@ -689,6 +698,8 @@ cheap local detection into model-owned diagnosis and governed action.
|
||||
|
||||
## Canonical Files
|
||||
|
||||
- `pkg/reporting/evidence.go`: measured retained-history evidence contract for model-facing summaries, with no report narrator or heuristic health judgement.
|
||||
|
||||
1. `internal/ai/`
|
||||
1g. `internal/ai/patrol_objectives.go`
|
||||
1e. `internal/ai/attention.go`
|
||||
@@ -7084,14 +7095,12 @@ narrative honestly retrospective on Patrol's work and prevents
|
||||
silent shadow-classification competing with Patrol's detection
|
||||
rules.
|
||||
|
||||
The same reporting synthesis layer is now exposed to Pulse
|
||||
Assistant as a first-class chat tool, `pulse_summarize`. The tool
|
||||
wraps the engine's `NarrativeFor` and `FleetNarrativeFor` entry
|
||||
points (single-resource and fleet modes selected by an `action`
|
||||
parameter) so an operator can ask "what's been happening with
|
||||
pve1 this week" or "where should I look across my fleet" and get
|
||||
a structured retrospective answer in chat rather than having to
|
||||
generate, download, and read a PDF.
|
||||
`pulse_summarize` is the model-facing retained-metric evidence tool. It uses
|
||||
`pkg/reporting.MetricEvidenceProvider`, implemented by `ReportEngine` in
|
||||
`pkg/reporting/evidence.go`, to read the same retained store as reports without
|
||||
invoking a narrator. The capable model in the active conversation owns synthesis.
|
||||
Report-only narratives and PDF health cards are separate reporting behavior,
|
||||
not evidence of machine health or a substitute for an investigation.
|
||||
|
||||
The tool is self-targeting: `action=fleet` with `resource_ids`
|
||||
omitted enumerates the known fleet from the executor's unified
|
||||
@@ -7123,40 +7132,27 @@ Every remaining error path in the tool tells the model to
|
||||
enumerate or retry and explicitly forbids asking the operator for
|
||||
resource IDs. `TestSummarizeTool_FleetEnumeratesWhenIDsOmitted`,
|
||||
`TestSummarizeTool_FleetResolvesNamesAndTranslatesMetricsIDs`, and
|
||||
the compile-time `summarizeMetricsTargetResolver` pin on the
|
||||
monitor adapter hold this behavior. The tool is read-only (no
|
||||
approval gate, no control-level requirement) and returns a JSON
|
||||
envelope carrying the narrative source, health status, observations
|
||||
or outliers, recommendations, and provenance disclaimer. v1 always
|
||||
returns heuristic narrative; the AI narrator wiring through the
|
||||
chat session is a focused follow-up that adds `Narrator`,
|
||||
`FleetNarrator`, and `FindingsProvider` plumbing to the executor
|
||||
configuration so the tool inherits the same per-tenant AI service
|
||||
the report PDF endpoint already uses. Reporting therefore expands
|
||||
from an export-shaped feature into a first-class capability
|
||||
Assistant can compose with — the underlying engine surface stays
|
||||
unchanged.
|
||||
the compile-time `resourceMetricsTargetResolver` pin on the monitor adapter
|
||||
hold this behavior.
|
||||
|
||||
That follow-up has now landed. `chat.Config` carries three optional
|
||||
fields (`ReportNarrator`, `ReportFleetNarrator`,
|
||||
`ReportFindingsProvider`) which are threaded through to
|
||||
`tools.ExecutorConfig` and stored on `PulseToolExecutor`. The
|
||||
`pulse_summarize` tool reads them when building requests so the
|
||||
engine sees a populated narrator when the tenant's AI service is
|
||||
configured. The router installs a `SetReportNarratorResolver`
|
||||
closure on the chat handler that mirrors the reporting handler's
|
||||
pattern: it asks the AISettingsHandler for the per-tenant
|
||||
`ai.Service` and, when that service has `Enabled=true`, returns it
|
||||
as the implementation for all three roles (Service satisfies
|
||||
`reporting.Narrator`, `reporting.FleetNarrator`, and
|
||||
`reporting.FindingsProvider` already). An unconfigured tenant still
|
||||
sees the heuristic fallback — the tool never errors on missing AI,
|
||||
matching the report PDF's graceful-degradation posture. AI-narrated
|
||||
chat synthesis therefore uses the same provider, sanitizer, model
|
||||
selection, cost ledger (report_narrative / report_narrative_fleet
|
||||
use-cases), and budget gate the report PDF endpoint already
|
||||
enforces — there is exactly one canonical synthesis path for both
|
||||
surfaces.
|
||||
Both modes return statistics with units, retained point counts, first/latest
|
||||
observation timestamps and the largest gap between returned points. Means are
|
||||
unweighted means of retained values, which may themselves be retention
|
||||
aggregates. Extrema retain bucket minima/maxima. Neither the requested range
|
||||
nor the first-to-last span proves continuous coverage. Empty metrics remain an
|
||||
empty evidence map. Store errors and unavailable evidence capabilities return
|
||||
errors, never a healthy verdict or heuristic fallback. Fleet reads fail visibly
|
||||
if a selected resource query fails rather than silently dropping that resource.
|
||||
|
||||
The response explicitly names alerts, findings, disk health, backups and topology
|
||||
as not queried. Models must use their corresponding tools to collect those
|
||||
sources. The evidence tool does not emit health labels, scored outliers,
|
||||
recommendations or a second model's conclusions. Report narrator dependencies
|
||||
remain available for report generation but are not invoked by `pulse_summarize`.
|
||||
`TestSummarizeToolReturnsHighUtilizationAsEvidenceWithoutDiagnosis` verifies that
|
||||
92% memory remains a measured reading and is not labelled critical or healthy.
|
||||
`TestMetricEvidenceRetainsObservedCoverageAcrossRestart` verifies retained
|
||||
coordinates, sparse timestamps, statistics, empty results and query failures.
|
||||
|
||||
The same canonical AI runtime now also records user-chat token
|
||||
usage to the cost ledger. `chat.Service.ExecuteStream` was a
|
||||
|
||||
@@ -15,6 +15,19 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
The open `patrol-assistant-customer-outcome-qualification` gap includes retained
|
||||
query coverage in `pkg/metrics/store.go`. A 24-hour `Query` returns the first
|
||||
non-empty resolution tier, and `QueryAll` fills missing metric names rather than
|
||||
missing times. A fixture with a minute-tier CPU point at 21:37 and a raw point
|
||||
at 22:16 returns only 21:37 for both 24-hour APIs, while a two-hour query returns
|
||||
22:16. The new model-facing evidence contract discloses the returned timestamps
|
||||
but does not repair this shared-store defect. Canonical follow-up must reconcile
|
||||
temporal coverage across tiers for single-series, all-series and batch queries,
|
||||
with explicit bucket precedence, extrema and downsampling semantics. Do not
|
||||
claim complete or current requested-window coverage from the present fallback
|
||||
behavior. This belongs to metrics-store qualification, not model prompting.
|
||||
|
||||
|
||||
Resource-scoped Assistant performance reads query retained CPU, memory and
|
||||
disk series using the registry target, then retain the existing 120-point
|
||||
output limit. A backend restart does not define the requested history window.
|
||||
|
||||
@@ -2063,6 +2063,7 @@
|
||||
"pkg/aicontracts/investigation.go",
|
||||
"pkg/aicontracts/orchestrator_deps.go",
|
||||
"pkg/extensions/ai_autofix.go",
|
||||
"pkg/reporting/evidence.go",
|
||||
"scripts/generate-pulse-intelligence-docs.go"
|
||||
],
|
||||
"verification": {
|
||||
@@ -2071,6 +2072,19 @@
|
||||
"exact_files": [],
|
||||
"require_explicit_path_policy_coverage": true,
|
||||
"path_policies": [
|
||||
{
|
||||
"id": "retained-metric-evidence",
|
||||
"label": "retained metric evidence and observation coverage proof",
|
||||
"match_prefixes": [],
|
||||
"match_files": [
|
||||
"pkg/reporting/evidence.go"
|
||||
],
|
||||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"pkg/reporting/evidence_test.go"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "patrol-qualification",
|
||||
"label": "Patrol independent-ground-truth qualification, replay, and publication proof",
|
||||
|
||||
@@ -331,6 +331,39 @@ func (s *Service) loadPatrolModelReadiness() {
|
||||
persisted.Result.PatrolCapable = false
|
||||
persisted.Result.MaxVerifiedMode = ""
|
||||
}
|
||||
// Older subscription transports persisted the CLI's terminal envelope in
|
||||
// the error detail. Recover only its explicit refusal signal, not prose or
|
||||
// incidental rate-limit events, without spending another provider request.
|
||||
if persisted.Result.Provider == config.AIProviderClaudeSubscription && persisted.Result.Cause == PatrolFailureCauseProviderConnection {
|
||||
for i, detail := range persisted.Result.Details {
|
||||
if !strings.HasPrefix(detail, "Multi-turn continuation probe failed: claude subscription agent failed:") {
|
||||
continue
|
||||
}
|
||||
start := strings.Index(detail, `{"type":"result"`)
|
||||
if start < 0 {
|
||||
continue
|
||||
}
|
||||
var terminal struct {
|
||||
StopReason string `json:"stop_reason"`
|
||||
}
|
||||
if json.NewDecoder(strings.NewReader(detail[start:])).Decode(&terminal) != nil || terminal.StopReason != "refusal" {
|
||||
continue
|
||||
}
|
||||
failure := patrolRuntimeFailureFromError(providers.ErrProviderRequestRefused)
|
||||
persisted.Result.Cause = failure.Cause
|
||||
persisted.Result.Summary = failure.Summary
|
||||
persisted.Result.Recommendation = failure.Recommendation
|
||||
persisted.Result.Details[i] = "Multi-turn continuation probe failed: " + failure.Detail
|
||||
persisted.Result.Dimensions.ToolProtocol.Summary = failure.Summary
|
||||
if tool := &persisted.Result.Dimensions.ToolProtocol; tool.Attempts > 0 && tool.Passed == tool.Attempts {
|
||||
tool.Summary = "Initial tool use passed. The provider refused the continuation request."
|
||||
}
|
||||
persisted.Result.Modes.Monitor = PatrolModeSuitability{Status: PatrolModeNotAssessed, Summary: failure.Description}
|
||||
persisted.Result.Success = false
|
||||
persisted.Result.PatrolCapable = false
|
||||
persisted.Result.MaxVerifiedMode = ""
|
||||
}
|
||||
}
|
||||
persisted.Result.CacheKey = persisted.CacheKey
|
||||
s.patrolModelReadinessCache.result = clonePatrolModelReadinessResult(&persisted.Result)
|
||||
s.patrolModelReadinessCache.recordedAt = persisted.RecordedAt
|
||||
@@ -786,6 +819,9 @@ func runPatrolModelReadinessWithProvider(ctx context.Context, cfg *config.AIConf
|
||||
failure := patrolRuntimeFailureFromErrorCtx(ctx, probeErr)
|
||||
probeFailure = &failure
|
||||
toolSummary = failure.Summary
|
||||
if toolPassed == len(scenarios) && failure.Cause == PatrolFailureCauseProviderRefusal {
|
||||
toolSummary = "Initial tool use passed. The provider refused the continuation request."
|
||||
}
|
||||
}
|
||||
// A mid-run cancellation invalidates nothing the model already proved and
|
||||
// proves nothing about what it never attempted: keep completed per-scenario
|
||||
@@ -893,6 +929,9 @@ func runPatrolModelReadinessWithProvider(ctx context.Context, cfg *config.AIConf
|
||||
result.Cause = probeFailure.Cause
|
||||
result.Summary = probeFailure.Summary
|
||||
result.Recommendation = probeFailure.Recommendation
|
||||
if probeFailure.Cause == PatrolFailureCauseProviderRefusal {
|
||||
result.Modes.Monitor.Summary = probeFailure.Description
|
||||
}
|
||||
}
|
||||
if interrupted {
|
||||
result.Status = PatrolModelReadinessNotAssessed
|
||||
|
||||
@@ -474,3 +474,52 @@ func TestPatrolReadinessCacheCorrectsLegacyIncompleteLatency(t *testing.T) {
|
||||
t.Fatalf("recorded evidence changed: %+v", cached)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatrolReadinessProviderRefusalRemainsBlocked(t *testing.T) {
|
||||
cfg := readinessTestConfig()
|
||||
result := runPatrolModelReadinessWithProvider(context.Background(), cfg, config.AIProviderOllama, "test-model", "ollama:test-model", &scriptedReadinessProvider{contextWindow: 32768, continuationErr: providers.ErrProviderRequestRefused})
|
||||
if result.Cause != PatrolFailureCauseProviderRefusal || result.Success || result.PatrolCapable || result.MaxVerifiedMode != "" {
|
||||
t.Fatalf("refusal verdict = %+v", result)
|
||||
}
|
||||
if result.Dimensions.ToolProtocol.Passed != 3 || result.Dimensions.ContextQuality.Passed != 2 || result.Dimensions.Latency.Status != PatrolModelReadinessNotAssessed {
|
||||
t.Fatalf("refusal lost completed probe evidence: %+v", result.Dimensions)
|
||||
}
|
||||
if !strings.Contains(result.Modes.Monitor.Summary, "explicitly refused") || !strings.Contains(result.Recommendation, "permits this workflow") {
|
||||
t.Fatalf("refusal lacks an actionable explanation: %+v", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatrolReadinessCacheRecoversExplicitLegacyRefusal(t *testing.T) {
|
||||
for _, stopReason := range []string{"refusal", "end_turn"} {
|
||||
t.Run(stopReason, func(t *testing.T) {
|
||||
persistence := config.NewConfigPersistence(t.TempDir())
|
||||
cfg := readinessTestConfig()
|
||||
cfg.Model = "claude-subscription:test-model"
|
||||
cfg.PatrolModel = cfg.Model
|
||||
service := NewService(persistence, nil)
|
||||
service.cfg = cfg
|
||||
result := emptyPatrolModelReadinessResult()
|
||||
result.Provider, result.Model = config.AIProviderClaudeSubscription, "test-model"
|
||||
result.Cause = PatrolFailureCauseProviderConnection
|
||||
result.Details = []string{`Multi-turn continuation probe failed: claude subscription agent failed: exit status 1: {"type":"result","stop_reason":"` + stopReason + `","result":"private diagnostic with refusal in prose"}`}
|
||||
result.Dimensions.ToolProtocol = PatrolModelReadinessDimension{Status: PatrolModelReadinessPass, Passed: 3, Attempts: 3}
|
||||
result.Dimensions.ContextQuality = PatrolModelReadinessDimension{Status: PatrolModelReadinessPass, Passed: 2, Attempts: 2}
|
||||
result.CacheKey = service.patrolModelReadinessCacheKey(cfg, result.Provider, result.Model)
|
||||
at := time.Now().Add(-time.Hour)
|
||||
service.recordPatrolModelReadiness(result, at)
|
||||
reloaded := NewService(persistence, nil)
|
||||
reloaded.cfg = cfg
|
||||
cached, recordedAt := reloaded.CachedPatrolModelReadiness()
|
||||
if cached == nil || !recordedAt.Equal(at) || cached.Dimensions.ToolProtocol.Passed != 3 || cached.Dimensions.ContextQuality.Passed != 2 {
|
||||
t.Fatalf("saved evidence changed: %+v at %s", cached, recordedAt)
|
||||
}
|
||||
if stopReason == "refusal" {
|
||||
if cached.Cause != PatrolFailureCauseProviderRefusal || strings.Contains(cached.Details[0], "private diagnostic") || reloaded.PatrolRuntimeReadiness().Ready {
|
||||
t.Fatalf("legacy refusal was lost or authorized: %+v", cached)
|
||||
}
|
||||
} else if cached.Cause != PatrolFailureCauseProviderConnection {
|
||||
t.Fatalf("incidental refusal prose changed the cause: %+v", cached)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,6 +36,7 @@ const (
|
||||
PatrolFailureCauseProviderRateLimited PatrolFailureCause = "provider_rate_limited"
|
||||
PatrolFailureCauseProviderAuth PatrolFailureCause = "provider_auth"
|
||||
PatrolFailureCauseProviderConnection PatrolFailureCause = "provider_connection"
|
||||
PatrolFailureCauseProviderRefusal PatrolFailureCause = "provider_refusal"
|
||||
// PatrolFailureCauseInterrupted marks a run that was cancelled mid-flight
|
||||
// (operator cancel or a dropped client connection). It is deliberately not
|
||||
// a provider fault: an interrupted run carries no evidence about the
|
||||
|
||||
@@ -173,6 +173,11 @@ func ClassifyProviderConnectionFailure(err error) PatrolRuntimeFailureDiagnostic
|
||||
}
|
||||
|
||||
switch failure.Cause {
|
||||
case PatrolFailureCauseProviderRefusal:
|
||||
diagnostic.Title = failure.Summary
|
||||
diagnostic.Summary = failure.Summary
|
||||
diagnostic.Description = failure.Description
|
||||
diagnostic.Recommendation = failure.Recommendation
|
||||
case PatrolFailureCauseInterrupted:
|
||||
diagnostic.Title = "Connection test interrupted"
|
||||
diagnostic.Summary = "Connection test interrupted"
|
||||
@@ -311,6 +316,13 @@ func patrolRuntimeFailureFromErrorCtx(ctx context.Context, err error) patrolRunt
|
||||
failure.Cause = PatrolFailureCauseProviderNotConfigured
|
||||
failure.Description = "Pulse Patrol cannot use the local " + setup.displayName + " subscription because its CLI executable or login is unavailable to the operating-system account running Pulse."
|
||||
failure.Recommendation = setup.recommendation
|
||||
case errors.Is(err, providers.ErrProviderRequestRefused):
|
||||
failure.Title = "Pulse Patrol: Provider refused this request"
|
||||
failure.Summary = "Provider refused this request"
|
||||
failure.Cause = PatrolFailureCauseProviderRefusal
|
||||
failure.Description = "The provider explicitly refused this request under its usage policy. Patrol could not complete the evaluation."
|
||||
failure.Recommendation = "Review the provider's usage policy or contact its support. Use a provider integration that permits this workflow, then verify Patrol again."
|
||||
failure.Detail = failure.Description
|
||||
case patrolMalformedToolHistory(lower):
|
||||
failure.Title = "Pulse Patrol: Malformed tool-call conversation history"
|
||||
failure.Summary = "Malformed tool-call conversation history"
|
||||
|
||||
@@ -42,6 +42,25 @@ func TestPatrolRuntimeFailureFromError_PopulatesImpactForAllCauses(t *testing.T)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatrolRuntimeFailurePreservesProviderRefusal(t *testing.T) {
|
||||
failure := patrolRuntimeFailureFromError(providers.ErrProviderRequestRefused)
|
||||
if failure.Cause != PatrolFailureCauseProviderRefusal || failure.Summary != "Provider refused this request" {
|
||||
t.Fatalf("refusal misclassified: %+v", failure)
|
||||
}
|
||||
connection := ClassifyProviderConnectionFailure(providers.ErrProviderRequestRefused)
|
||||
if connection.Cause != failure.Cause || connection.Recommendation != failure.Recommendation {
|
||||
t.Fatalf("connection diagnostic lost refusal: %+v", connection)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if interrupted := patrolRuntimeFailureFromErrorCtx(ctx, providers.ErrProviderRequestRefused); interrupted.Cause != PatrolFailureCauseInterrupted {
|
||||
t.Fatalf("operator cancellation lost precedence: %+v", interrupted)
|
||||
}
|
||||
if generic := patrolRuntimeFailureFromError(errors.New("connection refused")); generic.Cause != PatrolFailureCauseProviderConnection {
|
||||
t.Fatalf("socket failure was misclassified as refusal: %+v", generic)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatrolRuntimeFailureFromError_ClassifiesNoToolCapableEndpoint(t *testing.T) {
|
||||
// OpenRouter surfaces this when account-level provider/data filters
|
||||
// exclude every tool-capable route for the selected model.
|
||||
|
||||
@@ -22,6 +22,10 @@ import (
|
||||
// continues to own and execute every infrastructure tool call.
|
||||
type SubscriptionAgent string
|
||||
|
||||
// ErrProviderRequestRefused preserves an explicit provider refusal independently
|
||||
// of transport failures. It must not be recovered as a completed tool turn.
|
||||
var ErrProviderRequestRefused = errors.New("provider refused this request under its usage policy")
|
||||
|
||||
type SubscriptionAgentSetupIssue string
|
||||
|
||||
const (
|
||||
@@ -108,6 +112,7 @@ type claudePrintResponse struct {
|
||||
Result string `json:"result"`
|
||||
PermissionDenials []json.RawMessage `json:"permission_denials"`
|
||||
TerminalReason string `json:"terminal_reason,omitempty"`
|
||||
StopReason string `json:"stop_reason,omitempty"`
|
||||
NumTurns int `json:"num_turns,omitempty"`
|
||||
Usage struct {
|
||||
InputTokens int `json:"input_tokens"`
|
||||
@@ -457,6 +462,11 @@ func (c *SubscriptionAgentClient) run(ctx context.Context, name string, args []s
|
||||
if ctx.Err() != nil {
|
||||
return nil, fmt.Errorf("%s subscription agent timed out: %w", name, ctx.Err())
|
||||
}
|
||||
if name == "claude" {
|
||||
if terminal, ok := decodeClaudeTerminalResponse(stdout.buffer.Bytes()); ok && terminal.StopReason == "refusal" {
|
||||
return nil, ErrProviderRequestRefused
|
||||
}
|
||||
}
|
||||
message := strings.TrimSpace(stderr.buffer.String())
|
||||
if message == "" {
|
||||
message = strings.TrimSpace(stdout.buffer.String())
|
||||
@@ -809,6 +819,9 @@ func decodeSubscriptionAgentTurn(agent SubscriptionAgent, raw []byte) (subscript
|
||||
if err := json.Unmarshal(raw, &wrapper); err != nil {
|
||||
return turn, fmt.Errorf("decode Claude subscription response: %w", err)
|
||||
}
|
||||
if wrapper.StopReason == "refusal" {
|
||||
return turn, ErrProviderRequestRefused
|
||||
}
|
||||
if len(wrapper.PermissionDenials) > 0 {
|
||||
return turn, errors.New("Claude subscription agent attempted a denied built-in tool")
|
||||
}
|
||||
@@ -898,6 +911,9 @@ func decodeClaudeSubscriptionAgentResponse(req ChatRequest, raw []byte) (subscri
|
||||
if !terminalFound {
|
||||
return subscriptionAgentTurn{}, errors.New("Claude subscription stream did not contain a terminal result")
|
||||
}
|
||||
if terminal.StopReason == "refusal" {
|
||||
return subscriptionAgentTurn{}, ErrProviderRequestRefused
|
||||
}
|
||||
if len(terminal.PermissionDenials) > 0 {
|
||||
return subscriptionAgentTurn{}, errors.New("Claude subscription agent attempted a denied built-in tool")
|
||||
}
|
||||
@@ -914,6 +930,9 @@ func decodeClaudeSubscriptionAgentResponse(req ChatRequest, raw []byte) (subscri
|
||||
|
||||
func decodeClaudePrintResponse(wrapper claudePrintResponse) (subscriptionAgentTurn, error) {
|
||||
var turn subscriptionAgentTurn
|
||||
if wrapper.StopReason == "refusal" {
|
||||
return turn, ErrProviderRequestRefused
|
||||
}
|
||||
if len(wrapper.PermissionDenials) > 0 {
|
||||
return turn, errors.New("Claude subscription agent attempted a denied built-in tool")
|
||||
}
|
||||
|
||||
@@ -88,6 +88,33 @@ func TestCappedBufferBoundsChildOutput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubscriptionAgentPreservesExplicitRefusalAcrossProcessExit(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("fake subscription CLI uses a POSIX shell script")
|
||||
}
|
||||
for _, exitCode := range []string{"0", "1"} {
|
||||
t.Run("exit_"+exitCode, func(t *testing.T) {
|
||||
binDir := t.TempDir()
|
||||
// Even a preceding declared call or apparently successful envelope
|
||||
// must not be routed once the terminal provider verdict is refusal.
|
||||
writeExecutable(t, filepath.Join(binDir, "claude"), `#!/bin/sh
|
||||
printf '%s\n' '{"type":"assistant","message":{"content":[{"type":"tool_use","id":"read-1","name":"pulse_read","input":{}}]}}'
|
||||
printf '%s\n' '{"type":"result","subtype":"success","is_error":true,"stop_reason":"refusal","terminal_reason":"api_error","result":"private provider detail","structured_output":{"content":"done","stop_reason":"end_turn","tool_calls":[]}}'
|
||||
exit `+exitCode+"\n")
|
||||
t.Setenv("PATH", binDir+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||
client := NewSubscriptionAgentClient(SubscriptionAgentClaude, "test-model", time.Second)
|
||||
var events int
|
||||
err := client.ChatStream(context.Background(), ChatRequest{Tools: []Tool{{Name: "pulse_read"}}}, func(StreamEvent) { events++ })
|
||||
if !errors.Is(err, ErrProviderRequestRefused) || events != 0 {
|
||||
t.Fatalf("refusal = %v, emitted events = %d", err, events)
|
||||
}
|
||||
if strings.Contains(err.Error(), "private provider detail") {
|
||||
t.Fatal("raw CLI envelope escaped the transport")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubscriptionAgentRequestTimeout(t *testing.T) {
|
||||
if got := subscriptionAgentRequestTimeout(30 * time.Second); got != SubscriptionAgentMinimumRequestTimeout {
|
||||
t.Fatalf("short configured timeout = %s, want %s", got, SubscriptionAgentMinimumRequestTimeout)
|
||||
|
||||
@@ -12,28 +12,21 @@ import (
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// registerSummarizeTools registers the pulse_summarize tool which
|
||||
// exposes the reporting synthesis engine to chat sessions as a
|
||||
// retrospective question-answering capability. The tool wraps the
|
||||
// engine's NarrativeFor and FleetNarrativeFor entry points so
|
||||
// operators can ask "what's hot on pve1 this week" or "where should
|
||||
// I look across my fleet" without round-tripping through report
|
||||
// generation. v1 returns heuristic narrative (the same deterministic
|
||||
// observations the report PDF carries when AI is unconfigured); a
|
||||
// follow-up commit will thread the per-tenant AI narrator through
|
||||
// the chat session so this tool can return AI-generated synthesis
|
||||
// in the same shape.
|
||||
// registerSummarizeTools exposes retained evidence for model-owned synthesis.
|
||||
// Report narratives remain available to report consumers through reporting.Engine.
|
||||
func (e *PulseToolExecutor) registerSummarizeTools() {
|
||||
e.registry.registerBuiltin(RegisteredTool{
|
||||
Definition: Tool{
|
||||
Name: agentcapabilities.PulseSummarizeToolName,
|
||||
Description: `Generate a retrospective summary of one resource or a fleet across a time window. Use this when the operator asks questions like "what's been happening with pve1 this week" or "where should I look across my fleet" — answers grounded in metric stats, alerts, storage state, disk health, and Patrol findings within the window.
|
||||
Description: `Read retained metric evidence for one resource or a fleet over 24h, 7d, or 30d. Returns measured statistics, units, actual first/latest observation times, retained point counts and largest gaps. The requested window does not imply complete or fresh coverage. Means are unweighted means of returned retained points, which may already be retention aggregates.
|
||||
|
||||
Two modes via the 'action' parameter:
|
||||
- "resource": summarises a single resource. Required: resource_id (ID or name); resource_type only when the ID is not a known resource.
|
||||
- "fleet": summarises a fleet across multiple resources. resource_ids is optional — omit it and the tool enumerates the known fleet itself (infrastructure first, bounded). Never ask the operator for resource IDs.
|
||||
This tool reads metrics only. Alerts, findings, disk health, backup coverage and topology are not queried. Use the relevant tools for those sources before drawing conclusions about health or causes.
|
||||
|
||||
Time window defaults to the last 7 days; supported ranges: 24h, 7d, 30d.`,
|
||||
Actions:
|
||||
- resource: resource_id is required, resource_type is only needed for IDs unknown to the registry.
|
||||
- fleet: omit resource_ids to enumerate the known fleet, or provide comma-separated IDs or names. resource_type optionally filters enumeration.
|
||||
|
||||
Default window: 7d. Interpret the evidence in the current conversation. No separate model or heuristic diagnosis runs inside this tool.`,
|
||||
InputSchema: InputSchema{
|
||||
Type: "object",
|
||||
Properties: map[string]PropertySchema{
|
||||
@@ -69,8 +62,8 @@ Time window defaults to the last 7 days; supported ranges: 24h, 7d, 30d.`,
|
||||
Governance: ToolGovernance{
|
||||
ActionMode: ToolActionRead,
|
||||
ApprovalPolicy: ToolApprovalScopeOnly,
|
||||
ApprovalSummary: "no approval required; pure read of metrics history and findings store.",
|
||||
Summary: "Returns a retrospective synthesis (observations, recommendations, outliers, period comparison) for one resource or a fleet within a time window.",
|
||||
ApprovalSummary: "no approval required; pure read of retained metrics.",
|
||||
Summary: "Returns retained metric statistics and observation coverage for one resource or a fleet within a time window.",
|
||||
},
|
||||
})
|
||||
}
|
||||
@@ -97,9 +90,9 @@ func summarizeRangeWindow(raw string) time.Duration {
|
||||
}
|
||||
|
||||
func (e *PulseToolExecutor) executeSummarize(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
|
||||
engine := reporting.GetEngine()
|
||||
if engine == nil {
|
||||
return NewErrorResult(fmt.Errorf("reporting engine not initialized")), nil
|
||||
engine, ok := reporting.GetEngine().(reporting.MetricEvidenceProvider)
|
||||
if !ok || engine == nil {
|
||||
return NewErrorResult(fmt.Errorf("retained metric evidence is unavailable from the reporting engine")), nil
|
||||
}
|
||||
|
||||
action, _ := args["action"].(string)
|
||||
@@ -122,24 +115,36 @@ func (e *PulseToolExecutor) executeSummarize(ctx context.Context, args map[strin
|
||||
}
|
||||
}
|
||||
|
||||
// EvidenceScope states what was collected, independently from an empty result.
|
||||
// In particular, an empty metrics map says nothing about alert or disk health.
|
||||
type summarizeEvidenceScope struct {
|
||||
Source string `json:"source"`
|
||||
NotQueried []string `json:"not_queried"`
|
||||
Aggregation string `json:"aggregation"`
|
||||
}
|
||||
|
||||
func retainedSummaryScope() summarizeEvidenceScope {
|
||||
return summarizeEvidenceScope{
|
||||
Source: "retained_metrics",
|
||||
NotQueried: []string{"alerts", "findings", "disk_health", "backups", "topology"},
|
||||
Aggregation: "Mean and latest describe retained point values, which may be bucket averages. Min and max preserve recorded bucket extrema, with their bucket timestamps, so peaks can exceed the plotted averages. The mean is unweighted. First and last timestamps do not prove continuous coverage.",
|
||||
}
|
||||
}
|
||||
|
||||
type summarizeResourceResponse struct {
|
||||
OK bool `json:"ok"`
|
||||
Action string `json:"action"`
|
||||
ResourceType string `json:"resource_type"`
|
||||
ResourceID string `json:"resource_id"`
|
||||
WindowStart time.Time `json:"window_start"`
|
||||
WindowEnd time.Time `json:"window_end"`
|
||||
NarrativeSource string `json:"narrative_source"`
|
||||
HealthStatus string `json:"health_status,omitempty"`
|
||||
HealthMessage string `json:"health_message,omitempty"`
|
||||
Observations []reporting.NarrativeBullet `json:"observations,omitempty"`
|
||||
Recommendations []string `json:"recommendations,omitempty"`
|
||||
Disclaimer string `json:"disclaimer,omitempty"`
|
||||
OK bool `json:"ok"`
|
||||
Action string `json:"action"`
|
||||
ResourceType string `json:"resource_type"`
|
||||
ResourceID string `json:"resource_id"`
|
||||
WindowStart time.Time `json:"window_start"`
|
||||
WindowEnd time.Time `json:"window_end"`
|
||||
Scope summarizeEvidenceScope `json:"scope"`
|
||||
Evidence *reporting.MetricEvidence `json:"evidence"`
|
||||
}
|
||||
|
||||
func (e *PulseToolExecutor) summarizeResource(
|
||||
_ context.Context,
|
||||
engine reporting.Engine,
|
||||
ctx context.Context,
|
||||
engine reporting.MetricEvidenceProvider,
|
||||
args map[string]interface{},
|
||||
start, end time.Time,
|
||||
) (CallToolResult, error) {
|
||||
@@ -182,14 +187,12 @@ func (e *PulseToolExecutor) summarizeResource(
|
||||
}
|
||||
req.Start = start
|
||||
req.End = end
|
||||
req.Narrator = e.reportNarrator
|
||||
req.FindingsProvider = e.reportFindingsProvider
|
||||
narrative, err := engine.NarrativeFor(req)
|
||||
evidence, err := engine.MetricEvidenceFor(ctx, req)
|
||||
if err != nil {
|
||||
return NewErrorResult(fmt.Errorf("narrative generation failed: %w", err)), nil
|
||||
return NewErrorResult(fmt.Errorf("metric evidence query failed: %w", err)), nil
|
||||
}
|
||||
if narrative == nil {
|
||||
return NewErrorResult(fmt.Errorf("narrative generation produced no result")), nil
|
||||
if evidence == nil {
|
||||
return NewErrorResult(fmt.Errorf("metric evidence query produced no result")), nil
|
||||
}
|
||||
|
||||
// Telemetry: structured event line per summarize invocation so
|
||||
@@ -200,45 +203,34 @@ func (e *PulseToolExecutor) summarizeResource(
|
||||
Str("org_id", e.orgID).
|
||||
Str("action", "resource").
|
||||
Str("resource_type", canonicalType).
|
||||
Str("narrative_source", narrative.Source).
|
||||
Bool("ai_configured", e.reportNarrator != nil).
|
||||
Bool("findings_configured", e.reportFindingsProvider != nil).
|
||||
Str("evidence_source", "retained_metrics").
|
||||
Time("window_start", start).
|
||||
Time("window_end", end).
|
||||
Msg("Reporting: pulse_summarize invoked")
|
||||
|
||||
return NewJSONResult(summarizeResourceResponse{
|
||||
OK: true,
|
||||
Action: "resource",
|
||||
ResourceType: canonicalType,
|
||||
ResourceID: resourceID,
|
||||
WindowStart: start,
|
||||
WindowEnd: end,
|
||||
NarrativeSource: narrative.Source,
|
||||
HealthStatus: narrative.HealthStatus,
|
||||
HealthMessage: narrative.HealthMessage,
|
||||
Observations: narrative.Observations,
|
||||
Recommendations: narrative.Recommendations,
|
||||
Disclaimer: narrative.Disclaimer,
|
||||
OK: true,
|
||||
Action: "resource",
|
||||
ResourceType: canonicalType,
|
||||
ResourceID: resourceID,
|
||||
WindowStart: start,
|
||||
WindowEnd: end,
|
||||
Scope: retainedSummaryScope(),
|
||||
Evidence: evidence,
|
||||
}), nil
|
||||
}
|
||||
|
||||
type summarizeFleetResponse struct {
|
||||
OK bool `json:"ok"`
|
||||
Action string `json:"action"`
|
||||
ResourceIDs []string `json:"resource_ids"`
|
||||
Resources []summarizeFleetEntry `json:"resources,omitempty"`
|
||||
Enumerated bool `json:"enumerated,omitempty"`
|
||||
Note string `json:"note,omitempty"`
|
||||
WindowStart time.Time `json:"window_start"`
|
||||
WindowEnd time.Time `json:"window_end"`
|
||||
NarrativeSource string `json:"narrative_source"`
|
||||
HealthStatus string `json:"health_status,omitempty"`
|
||||
HealthMessage string `json:"health_message,omitempty"`
|
||||
Outliers []reporting.FleetOutlier `json:"outliers,omitempty"`
|
||||
Patterns []reporting.NarrativeBullet `json:"patterns,omitempty"`
|
||||
Recommendations []string `json:"recommendations,omitempty"`
|
||||
Disclaimer string `json:"disclaimer,omitempty"`
|
||||
OK bool `json:"ok"`
|
||||
Action string `json:"action"`
|
||||
ResourceIDs []string `json:"resource_ids"`
|
||||
Resources []summarizeFleetEntry `json:"resources"`
|
||||
Enumerated bool `json:"enumerated,omitempty"`
|
||||
Note string `json:"note,omitempty"`
|
||||
WindowStart time.Time `json:"window_start"`
|
||||
WindowEnd time.Time `json:"window_end"`
|
||||
Scope summarizeEvidenceScope `json:"scope"`
|
||||
Evidence []*reporting.MetricEvidence `json:"evidence"`
|
||||
}
|
||||
|
||||
// summarizeFleetEntry names one fleet member in the response so the model can
|
||||
@@ -431,8 +423,8 @@ func (e *PulseToolExecutor) buildSummarizeCandidateIndex() *summarizeCandidateIn
|
||||
const summarizeFleetMaxResources = 50
|
||||
|
||||
func (e *PulseToolExecutor) summarizeFleet(
|
||||
_ context.Context,
|
||||
engine reporting.Engine,
|
||||
ctx context.Context,
|
||||
engine reporting.MetricEvidenceProvider,
|
||||
args map[string]interface{},
|
||||
start, end time.Time,
|
||||
) (CallToolResult, error) {
|
||||
@@ -523,21 +515,17 @@ func (e *PulseToolExecutor) summarizeFleet(
|
||||
entries = append(entries, summarizeFleetEntry{ID: cand.id, Type: cand.reportType, Name: cand.name})
|
||||
}
|
||||
|
||||
req := reporting.MultiReportRequest{
|
||||
Title: "Fleet summary",
|
||||
Start: start,
|
||||
End: end,
|
||||
Resources: resources,
|
||||
FleetNarrator: e.reportFleetNarrator,
|
||||
Narrator: e.reportNarrator,
|
||||
FindingsProvider: e.reportFindingsProvider,
|
||||
}
|
||||
narrative, err := engine.FleetNarrativeFor(req)
|
||||
if err != nil {
|
||||
return NewErrorResult(fmt.Errorf("fleet narrative generation failed: %w", err)), nil
|
||||
}
|
||||
if narrative == nil {
|
||||
return NewErrorResult(fmt.Errorf("fleet narrative generation produced no result")), nil
|
||||
evidence := make([]*reporting.MetricEvidence, 0, len(resources))
|
||||
for _, req := range resources {
|
||||
req.Start, req.End = start, end
|
||||
item, err := engine.MetricEvidenceFor(ctx, req)
|
||||
if err != nil {
|
||||
return NewErrorResult(fmt.Errorf("metric evidence query failed for %s: %w", req.ResourceID, err)), nil
|
||||
}
|
||||
if item == nil {
|
||||
return NewErrorResult(fmt.Errorf("metric evidence query produced no result for %s", req.ResourceID)), nil
|
||||
}
|
||||
evidence = append(evidence, item)
|
||||
}
|
||||
|
||||
log.Info().
|
||||
@@ -547,28 +535,21 @@ func (e *PulseToolExecutor) summarizeFleet(
|
||||
Str("resource_type", canonicalDefault).
|
||||
Bool("enumerated", enumerated).
|
||||
Int("resource_count", len(ids)).
|
||||
Str("narrative_source", narrative.Source).
|
||||
Bool("ai_configured", e.reportFleetNarrator != nil).
|
||||
Bool("findings_configured", e.reportFindingsProvider != nil).
|
||||
Str("evidence_source", "retained_metrics").
|
||||
Time("window_start", start).
|
||||
Time("window_end", end).
|
||||
Msg("Reporting: pulse_summarize invoked")
|
||||
|
||||
return NewJSONResult(summarizeFleetResponse{
|
||||
OK: true,
|
||||
Action: "fleet",
|
||||
ResourceIDs: ids,
|
||||
Resources: entries,
|
||||
Enumerated: enumerated,
|
||||
Note: note,
|
||||
WindowStart: start,
|
||||
WindowEnd: end,
|
||||
NarrativeSource: narrative.Source,
|
||||
HealthStatus: narrative.HealthStatus,
|
||||
HealthMessage: narrative.HealthMessage,
|
||||
Outliers: narrative.Outliers,
|
||||
Patterns: narrative.Patterns,
|
||||
Recommendations: narrative.Recommendations,
|
||||
Disclaimer: narrative.Disclaimer,
|
||||
OK: true,
|
||||
Action: "fleet",
|
||||
ResourceIDs: ids,
|
||||
Resources: entries,
|
||||
Enumerated: enumerated,
|
||||
Note: note,
|
||||
WindowStart: start,
|
||||
WindowEnd: end,
|
||||
Scope: retainedSummaryScope(),
|
||||
Evidence: evidence,
|
||||
}), nil
|
||||
}
|
||||
|
||||
@@ -124,11 +124,7 @@ func TestSummarizeTool_ResourceReturnsHeuristicNarrative(t *testing.T) {
|
||||
}
|
||||
_ = engine
|
||||
|
||||
// Write metrics via the same store. The engine was constructed with
|
||||
// MetricsStore so we need to reach back into the store; instead, rely
|
||||
// on writing via package-level access through engine internals.
|
||||
// Simpler: skip data and accept that the heuristic narrator returns
|
||||
// "insufficient data" — which is itself a valid narrative we can assert.
|
||||
// No retained samples must remain an explicit empty evidence result.
|
||||
res, err := exec.executeSummarize(context.Background(), map[string]interface{}{
|
||||
"action": "resource",
|
||||
"resource_type": "node",
|
||||
@@ -153,15 +149,15 @@ func TestSummarizeTool_ResourceReturnsHeuristicNarrative(t *testing.T) {
|
||||
if parsed.Action != "resource" {
|
||||
t.Errorf("Action = %q, want resource", parsed.Action)
|
||||
}
|
||||
if parsed.NarrativeSource != reporting.NarrativeSourceHeuristic {
|
||||
t.Errorf("NarrativeSource = %q, want heuristic (v1 always heuristic)", parsed.NarrativeSource)
|
||||
if parsed.Scope.Source != "retained_metrics" || parsed.Evidence == nil || len(parsed.Evidence.Metrics) != 0 {
|
||||
t.Fatalf("expected empty retained evidence, got %+v", parsed)
|
||||
}
|
||||
if parsed.HealthStatus == "" {
|
||||
t.Error("expected HealthStatus populated")
|
||||
}
|
||||
if len(parsed.Observations) == 0 {
|
||||
t.Error("expected at least one observation from the heuristic narrator")
|
||||
for _, field := range []string{"health_status", "health_message", "observations", "recommendations"} {
|
||||
if strings.Contains(res.Content[0].Text, `"`+field+`"`) {
|
||||
t.Fatalf("empty evidence invented %s: %s", field, res.Content[0].Text)
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
func TestSummarizeTool_FleetParsesCommaSeparatedIDs(t *testing.T) {
|
||||
@@ -192,8 +188,8 @@ func TestSummarizeTool_FleetParsesCommaSeparatedIDs(t *testing.T) {
|
||||
t.Errorf("ResourceIDs[%d] = %q, want %q", i, parsed.ResourceIDs[i], w)
|
||||
}
|
||||
}
|
||||
if parsed.NarrativeSource != reporting.NarrativeSourceHeuristic {
|
||||
t.Errorf("NarrativeSource = %q, want heuristic", parsed.NarrativeSource)
|
||||
if parsed.Scope.Source != "retained_metrics" || len(parsed.Evidence) != 3 {
|
||||
t.Fatalf("expected retained evidence for all resources: %+v", parsed)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -294,25 +290,23 @@ func TestSummarizeTool_UsesReportNarratorWhenConfigured(t *testing.T) {
|
||||
if res.IsError {
|
||||
t.Fatalf("unexpected error: %+v", res.Content)
|
||||
}
|
||||
if !narrator.called {
|
||||
t.Fatal("expected narrator to be invoked")
|
||||
if narrator.called {
|
||||
t.Fatal("metric evidence must not invoke a second model")
|
||||
}
|
||||
var parsed summarizeResourceResponse
|
||||
if err := json.Unmarshal([]byte(res.Content[0].Text), &parsed); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if parsed.NarrativeSource != reporting.NarrativeSourceAI {
|
||||
t.Errorf("NarrativeSource = %q, want ai", parsed.NarrativeSource)
|
||||
if parsed.Scope.Source != "retained_metrics" || parsed.Evidence == nil {
|
||||
t.Fatalf("expected measured evidence, got %+v", parsed)
|
||||
}
|
||||
if parsed.HealthMessage != "AI says fine" {
|
||||
t.Errorf("HealthMessage = %q, want AI says fine", parsed.HealthMessage)
|
||||
}
|
||||
if len(parsed.Observations) != 1 || parsed.Observations[0].Text != "AI bullet" {
|
||||
t.Errorf("Observations = %#v", parsed.Observations)
|
||||
if strings.Contains(res.Content[0].Text, "AI says fine") || strings.Contains(res.Content[0].Text, "health_status") {
|
||||
t.Fatal("narrator judgment leaked into metric evidence")
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
func TestSummarizeTool_FleetUsesFleetNarratorWhenConfigured(t *testing.T) {
|
||||
func TestSummarizeTool_FleetDoesNotInvokeNestedNarrator(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
store, err := metrics.NewStore(metrics.StoreConfig{
|
||||
DBPath: filepath.Join(dir, "metrics.db"),
|
||||
@@ -359,19 +353,20 @@ func TestSummarizeTool_FleetUsesFleetNarratorWhenConfigured(t *testing.T) {
|
||||
if res.IsError {
|
||||
t.Fatalf("unexpected error: %+v", res.Content)
|
||||
}
|
||||
if !fleet.called {
|
||||
t.Fatal("expected fleet narrator to be invoked")
|
||||
if fleet.called {
|
||||
t.Fatal("fleet evidence must not invoke a second model")
|
||||
}
|
||||
var parsed summarizeFleetResponse
|
||||
if err := json.Unmarshal([]byte(res.Content[0].Text), &parsed); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if parsed.NarrativeSource != reporting.NarrativeSourceAI {
|
||||
t.Errorf("NarrativeSource = %q, want ai", parsed.NarrativeSource)
|
||||
if parsed.Scope.Source != "retained_metrics" || len(parsed.Evidence) != 2 {
|
||||
t.Fatalf("expected both resources as evidence, got %+v", parsed)
|
||||
}
|
||||
if len(parsed.Outliers) != 1 || parsed.Outliers[0].ResourceName != "alpha" {
|
||||
t.Errorf("Outliers = %#v", parsed.Outliers)
|
||||
if strings.Contains(res.Content[0].Text, "Memory creeping up") || strings.Contains(res.Content[0].Text, "outliers") {
|
||||
t.Fatal("fleet judgment leaked into metric evidence")
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// The production unified provider (the monitor adapter) must satisfy the
|
||||
@@ -633,3 +628,37 @@ func TestSummarizeRangeWindow(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSummarizeToolReturnsHighUtilizationAsEvidenceWithoutDiagnosis(t *testing.T) {
|
||||
store, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir()))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
at := time.Now().Add(-time.Hour).Truncate(time.Minute)
|
||||
store.Write("node", "delly-node-id", "memory", 92, at)
|
||||
store.Flush()
|
||||
previous := reporting.GetEngine()
|
||||
reporting.SetEngine(reporting.NewReportEngine(reporting.EngineConfig{MetricsStore: store}))
|
||||
defer reporting.SetEngine(previous)
|
||||
exec := NewPulseToolExecutor(ExecutorConfig{})
|
||||
exec.SetUnifiedResourceProvider(newSummarizeStubProvider())
|
||||
for _, args := range []map[string]interface{}{
|
||||
{"action": "resource", "resource_id": "delly", "range": "24h"},
|
||||
{"action": "fleet", "resource_ids": "delly", "range": "24h"},
|
||||
} {
|
||||
result, err := exec.executeSummarize(context.Background(), args)
|
||||
if err != nil || result.IsError {
|
||||
t.Fatalf("evidence = %+v, %v", result, err)
|
||||
}
|
||||
text := result.Content[0].Text
|
||||
for _, absent := range []string{"health_status", "HEALTHY", "CRITICAL", "pressure", "recommendations"} {
|
||||
if strings.Contains(text, absent) {
|
||||
t.Fatalf("unexpected diagnosis %q in %s", absent, text)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(text, `"mean":92`) || !strings.Contains(text, `"retained_points":1`) || !strings.Contains(text, `"disk_health"`) || !strings.Contains(text, `"alerts"`) {
|
||||
t.Fatalf("reading or unqueried source scope missing: %s", text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
package reporting
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// MetricEvidenceProvider exposes the same retained metrics as reports without
|
||||
// invoking a narrator or assigning health, urgency, or recommended actions.
|
||||
type MetricEvidenceProvider interface {
|
||||
MetricEvidenceFor(context.Context, MetricReportRequest) (*MetricEvidence, error)
|
||||
}
|
||||
|
||||
type MetricEvidence struct {
|
||||
ResourceID string `json:"resource_id"`
|
||||
ResourceType string `json:"resource_type"`
|
||||
GeneratedAt time.Time `json:"generated_at"`
|
||||
WindowStart time.Time `json:"window_start"`
|
||||
WindowEnd time.Time `json:"window_end"`
|
||||
Metrics map[string]RetainedMetricStats `json:"metrics"`
|
||||
}
|
||||
|
||||
// RetainedMetricStats describes returned points, which may already be retention
|
||||
// aggregates. Their count and time span do not prove continuous observation.
|
||||
type RetainedMetricStats struct {
|
||||
Unit string `json:"unit,omitempty"`
|
||||
RetainedPoints int `json:"retained_points"`
|
||||
Min float64 `json:"min"`
|
||||
Max float64 `json:"max"`
|
||||
Mean float64 `json:"mean"`
|
||||
Latest float64 `json:"latest"`
|
||||
MinBucketAt time.Time `json:"min_bucket_at"`
|
||||
MaxBucketAt time.Time `json:"max_bucket_at"`
|
||||
FirstAt time.Time `json:"first_at"`
|
||||
LastAt time.Time `json:"last_at"`
|
||||
MaxGapSeconds float64 `json:"max_gap_seconds"`
|
||||
}
|
||||
|
||||
func (e *ReportEngine) MetricEvidenceFor(ctx context.Context, req MetricReportRequest) (*MetricEvidence, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if e.getMetricsStore() == nil {
|
||||
return nil, fmt.Errorf("metrics store not initialized")
|
||||
}
|
||||
data, err := e.queryMetrics(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
evidence := &MetricEvidence{
|
||||
ResourceID: data.ResourceID, ResourceType: data.ResourceType,
|
||||
GeneratedAt: data.GeneratedAt, WindowStart: data.Start, WindowEnd: data.End,
|
||||
Metrics: make(map[string]RetainedMetricStats, len(data.Metrics)),
|
||||
}
|
||||
for name, points := range data.Metrics {
|
||||
if len(points) == 0 {
|
||||
continue
|
||||
}
|
||||
stats := data.Summary.ByMetric[name]
|
||||
entry := RetainedMetricStats{
|
||||
Unit: GetMetricUnit(name), RetainedPoints: len(points),
|
||||
Min: points[0].Value, Max: points[0].Value, Mean: stats.Avg, Latest: stats.Current,
|
||||
MinBucketAt: points[0].Timestamp, MaxBucketAt: points[0].Timestamp,
|
||||
FirstAt: points[0].Timestamp, LastAt: points[len(points)-1].Timestamp,
|
||||
}
|
||||
for i, point := range points {
|
||||
// Retention buckets preserve extrema separately from their mean.
|
||||
low, high := point.Value, point.Value
|
||||
if point.Min <= point.Value && point.Max >= point.Value {
|
||||
low, high = point.Min, point.Max
|
||||
}
|
||||
if low < entry.Min {
|
||||
entry.Min, entry.MinBucketAt = low, point.Timestamp
|
||||
}
|
||||
if high > entry.Max {
|
||||
entry.Max, entry.MaxBucketAt = high, point.Timestamp
|
||||
}
|
||||
if i > 0 {
|
||||
entry.MaxGapSeconds = max(entry.MaxGapSeconds, point.Timestamp.Sub(points[i-1].Timestamp).Seconds())
|
||||
}
|
||||
}
|
||||
evidence.Metrics[name] = entry
|
||||
}
|
||||
return evidence, nil
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package reporting
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMetricEvidenceRetainsObservedCoverageAcrossRestart(t *testing.T) {
|
||||
cfg := metrics.DefaultConfig(t.TempDir())
|
||||
cfg.RetentionRaw = 8 * 24 * time.Hour
|
||||
store, err := metrics.NewStore(cfg)
|
||||
require.NoError(t, err)
|
||||
start := time.Now().Add(-6 * time.Hour).Truncate(time.Minute)
|
||||
for i, delta := range []time.Duration{0, 2 * time.Minute, time.Hour} {
|
||||
store.Write("node", "native-node", "memory", float64(88+i), start.Add(delta))
|
||||
}
|
||||
require.NoError(t, store.Close())
|
||||
// A retained point can carry a mean and separately recorded extrema.
|
||||
// Preserve both the peak and its timestamp rather than using its mean.
|
||||
db, err := sql.Open("sqlite", cfg.DBPath)
|
||||
require.NoError(t, err)
|
||||
_, err = db.Exec("UPDATE metrics SET min_value = 80, max_value = 100 WHERE timestamp = ?", start.Add(2*time.Minute).Unix())
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, db.Close())
|
||||
store, err = metrics.NewStore(cfg)
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
engine := NewReportEngine(EngineConfig{MetricsStore: store})
|
||||
narrator := &stubNarrator{err: errors.New("must not run")}
|
||||
req := MetricReportRequest{ResourceID: "canonical-agent", MetricsResourceID: "native-node", ResourceType: "node", Start: start.Add(-18 * time.Hour), End: start.Add(6 * time.Hour), Narrator: narrator}
|
||||
evidence, err := engine.MetricEvidenceFor(context.Background(), req)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "canonical-agent", evidence.ResourceID)
|
||||
reading := evidence.Metrics["memory"]
|
||||
require.Equal(t, 3, reading.RetainedPoints)
|
||||
require.Equal(t, "%", reading.Unit)
|
||||
require.Equal(t, 89.0, reading.Mean)
|
||||
require.Equal(t, 90.0, reading.Latest)
|
||||
require.Equal(t, 80.0, reading.Min)
|
||||
require.Equal(t, 100.0, reading.Max)
|
||||
require.True(t, reading.MinBucketAt.Equal(start.Add(2*time.Minute)))
|
||||
require.True(t, reading.MaxBucketAt.Equal(start.Add(2*time.Minute)))
|
||||
require.True(t, reading.FirstAt.Equal(start))
|
||||
require.True(t, reading.LastAt.Equal(start.Add(time.Hour)))
|
||||
require.Equal(t, (58 * time.Minute).Seconds(), reading.MaxGapSeconds)
|
||||
require.Empty(t, narrator.seen.ResourceID, "evidence must not invoke report judgment")
|
||||
|
||||
req.MetricsResourceID = "unobserved-node"
|
||||
empty, err := engine.MetricEvidenceFor(context.Background(), req)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, empty.Metrics)
|
||||
require.Empty(t, empty.Metrics)
|
||||
|
||||
require.NoError(t, store.Close())
|
||||
_, err = engine.MetricEvidenceFor(context.Background(), req)
|
||||
require.Error(t, err, "query errors must not become empty evidence")
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
_, err = engine.MetricEvidenceFor(ctx, req)
|
||||
require.ErrorIs(t, err, context.Canceled)
|
||||
}
|
||||
Reference in New Issue
Block a user