Preserve investigation evidence and Proxmox temperatures

Real homelab evaluation exposed missing node sensor observations and loss of
recent tool evidence before the model reached its conclusion. Project both
canonical sensor sources and retain observations while context fits.

Add source and selection regressions plus a multi-turn retention proof.
Record the live diagnosis result and the unresolved Patrol provider block.
This commit is contained in:
rcourtman
2026-09-05 20:47:38 +01:00
parent 5de3377915
commit 3e8dbb8276
7 changed files with 300 additions and 24 deletions
@@ -95,3 +95,79 @@ blocked access, rejected action, executed action and verified resolution.
The owning governance gap is `patrol-assistant-customer-outcome-qualification`.
This interaction repair does not close that gap or authorize release publication.
## Maintainer homelab evaluation, 2026-09-05
The local development runtime used real existing connections with mock mode
disabled, `claude-subscription:claude-opus-5` through Claude CLI 2.1.211 and the
maintainer's Max login. Assistant was read-only, automatic fixes were disabled,
and the development background-work guard stayed enabled. No injected faults
or infrastructure writes were needed. This is one maintainer environment,
not independent paid-customer qualification.
The initial fleet question completed five provider turns and fourteen read-only
tool calls in 172 seconds. The transport recorded 12 input tokens and 8,406 output
tokens. Those CLI counters exclude cached input and are not complete context
usage or billable cost evidence. The visible answer was not a qualification pass:
- The temperature tool omitted a hot Proxmox node despite canonical telemetry
reporting roughly 94 degrees Celsius. A direct read-only sensor check confirmed
elevated temperature. The shared tool now projects Proxmox and agent readings
with canonical identity and source, retaining sensor timestamps when available.
- The answer treated a warning-filtered count as the total alert count.
- The resource summary reported healthy while its own observations included a
critical memory-pressure observation and omitted the thermal evidence.
- The visible final answer stopped mid-sentence. The shared artifact guard drops
the remainder of a turn when it recognises provider-call text. The original
pre-sanitized final response was not retained, so the precise trigger for this
run is unproven. Completion handling and ordinary tool-reference prose need
a focused reproduction rather than an inferred parser fix.
- Storage-risk wording exceeded the observed evidence about recoverability, and
shared storage was asserted as the cause without a captured topology check.
The simple native subscription transport probe passed. The separate real Patrol
readiness probe passed initial connectivity and tool/context checks but its
continuation returned a provider policy refusal. The UI misleadingly framed the
incomplete probe as a latency problem. This did not qualify Patrol or justify
bypassing the provider failure. Preserve the unresolved classification issue,
provider suitability, response completeness, evidence reconciliation and actual
Patrol outcomes under the existing customer-outcome qualification gap.
The temperature-only retest received canonical per-core, package and disk
readings from the repaired tool. Its final answer still claimed that no thermal
alert existed, although that same conversation contained an active thermal
alert. Runtime tracing identified unconditional compaction after two tool turns:
by the final answer, message context had fallen to roughly 968 estimated tokens,
far below its 128,000-token fallback window. This replaced earlier observations
with abbreviated summaries before they were needed for the conclusion.
The runtime now preserves tool observations while the full request fits the
model context window. Existing overflow handling remains authoritative. A
five-turn executable regression verifies that the final request retains the
original alert, temperature and collection timestamp from all four tool rounds.
This repairs the evidence-loss mechanism without claiming that a particular
model will always reconcile evidence correctly. Canonical lookup type mismatches,
summary health, provider readiness and complete customer outcomes remain separate
qualification gaps.
With both repairs applied, the final live temperature question completed in
181 seconds. The answer retained the per-core/package and NVMe readings, the
source timestamp, and the independently returned active thermal alert with its
threshold. It delivered a complete response and distinguished missing history
and disk-health coverage from observed sensor data. This is a bounded diagnosis
improvement, not a complete pass: hardware-cause inference from a 24-hour CPU
average remained stronger than the evidence warranted, canonical lookup type
mismatches still consumed tool calls, and the roughly three-minute answer latency
remained below the interactive product bar. No remediation was requested or run.
Browser verification exercised `/patrol` at 1440x1000 and 390x1000 with the real
provider, inspected temperature tool details expanded with Enter and collapsed
with a click, scrolled the evidence and complete answer, closed and reopened the
drawer with Escape, and reloaded the persisted session. Provider selection was
also checked at `/settings/pulse-intelligence/provider`. The local receipt binds
the two changed runtime files by SHA-256. No provider responses were mocked.
A final identical readiness retry again received the provider policy refusal.
The normal manual Patrol trigger returned HTTP 409 `patrol_readiness_not_ready`.
No Patrol run or infrastructure action was accepted. Actual Patrol evaluation
with this model remains blocked on a working supported provider path.
+1 -1
View File
@@ -10201,7 +10201,7 @@
},
{
"id": "patrol-assistant-customer-outcome-qualification",
"summary": "The 2026-09-05 product review found that Explain with Assistant opened a blank generic conversation and the attention workbench discarded richer canonical finding context. The explicit explanation dispatcher and canonical handoff repair address that frontend break, with scripted browser proof and regression tests. They do not qualify model reasoning, real infrastructure actions, or repeated customer value. Latest-report, monitoring-active, multi-ping telemetry excluding dev and deployment proof contained 127 paid installs, 71 with Patrol enabled and 23 with Assistant calls. Fourteen reported verified resolutions were concentrated in one install. Paid includes all non-free tiers, and activity cooccurrence is not a linked successful journey. Schema 17 outcome/provider/cost fields had no adoption in that review. Three real jobs remain to qualify across named provider/model/version configurations: unhealthy service diagnosis, backup or capacity risk, and supported VM/LXC plan, approval and verified result. Repeatability must be established across independent paid customer environments without pooling free-tier or ineligible installs.",
"summary": "The 2026-09-05 product review found that Explain with Assistant opened a blank generic conversation and the attention workbench discarded richer canonical finding context. The explicit explanation dispatcher and canonical handoff repair address that frontend break, with scripted browser proof and regression tests. They do not qualify model reasoning, real infrastructure actions, or repeated customer value. Latest-report, monitoring-active, multi-ping telemetry excluding dev and deployment proof contained 127 paid installs, 71 with Patrol enabled and 23 with Assistant calls. Fourteen reported verified resolutions were concentrated in one install. Paid includes all non-free tiers, and activity cooccurrence is not a linked successful journey. Schema 17 outcome/provider/cost fields had no adoption in that review. Three real jobs remain to qualify across named provider/model/version configurations: unhealthy service diagnosis, backup or capacity risk, and supported VM/LXC plan, approval and verified result. Repeatability must be established across independent paid customer environments without pooling free-tier or ineligible installs. The same-day real maintainer homelab evaluation with Claude Opus 5 exposed missing Proxmox temperature-tool evidence, contradictory summary health, incomplete visible prose, alert-count confusion and an inaccurately classified provider refusal during Patrol readiness. The shared temperature projection and premature within-window evidence compaction are repaired, while the documented reasoning, completion, readiness and end-to-end outcome failures remain unqualified.",
"owner": "project-owner",
"status": "planned",
"recorded_at": "2026-09-05",
@@ -19,6 +19,25 @@
## Purpose
Within an active investigation, tool observations are preserved across provider
turns while the full request fits the model context window. Turn age is not a
compaction trigger and does not prove evidence has been understood or retained.
The existing pre-request context-limit boundary owns overflow compaction,
re-estimation and fail-closed rejection. `TestAgenticLoopPreservesEarlierEvidenceWithinContextLimit`
exercises four tool rounds and checks that the final answer request still carries
the original alert state, sensor reading and observation timestamp.
The shared `pulse_metrics` temperature read projects both canonical agent and
Proxmox node observations through typed read state. Proxmox sensors must remain
available without an installed host agent. Results preserve canonical resource
identity and source, with the sensor collection timestamp when supplied, and
keep linked providers distinct rather than choosing between conflicting
readings. Hostname, display name, or canonical identity can select a host, and
`resource_id` filters exactly. Missing observations do not imply that an agent
must be installed. Standby-skipped disks are not current temperature evidence.
`TestExecuteGetTemperaturesCanonicalSources` verifies provider provenance,
selection, sensor coverage, and absence handling.
Incident timeline summaries must distinguish provider conditions from metric
threshold evidence. For `resource-incident` alert events, numeric value and
threshold placeholders must not render as a measured comparison; retain the
+4 -10
View File
@@ -978,11 +978,10 @@ func (a *AgenticLoop) executeWithTools(ctx context.Context, sessionID string, me
controlToolExecutedThisRun := false
advertisedActionGateBlocks := 0
// Track where each turn's messages begin in providerMessages for compaction.
// We keep the last N turns' tool results in full; older ones get compacted.
const compactionKeepTurns = 2 // Keep last 2 turns' tool results in full (KA preserves key facts)
const compactionMinChars = 300 // Only compact results longer than this
currentTurnStartIndex := len(providerMessages) // Initial messages are never compacted
// Preserve collected evidence across provider turns. Age alone is not a
// reason to replace observations with summaries. The pre-request context
// limit check below owns compaction when the request actually needs it.
currentTurnStartIndex := len(providerMessages)
// Generic Assistant/Watch wrap-up nudge. Patrol investigation has a
// separate evidence budget and completion checkpoint below.
@@ -1004,11 +1003,6 @@ agenticLoop:
(patrolOutputLimitRecoveryPending && !patrolOutputLimitRecoveryAttempted) ||
(investigationOutputLimitRecoveryPending && !investigationOutputLimitRecoveryAttempted) ||
(investigationEvidenceStartRepairPending && !investigationEvidenceStartRepairAttempted) {
// === CONTEXT COMPACTION: Compact old tool results to prevent context blowout ===
if turn > 0 {
compactOldToolResults(providerMessages, currentTurnStartIndex, compactionKeepTurns, compactionMinChars, a.knowledgeAccumulator)
}
// Check if aborted
a.mu.Lock()
if a.aborted[sessionID] {
@@ -1,6 +1,11 @@
package chat
import (
"context"
"fmt"
"github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"
"github.com/stretchr/testify/require"
"strings"
"testing"
@@ -122,3 +127,43 @@ func TestToolTokenCaching(t *testing.T) {
t.Fatal("tool token estimate should be positive")
}
}
func TestAgenticLoopPreservesEarlierEvidenceWithinContextLimit(t *testing.T) {
executor := tools.NewPulseToolExecutor(tools.ExecutorConfig{ControlLevel: tools.ControlLevelReadOnly})
evidence := `{"resource_id":"agent-hot","temperature":94,"alert_active":true,"observed_at":"2026-09-05T19:19:11Z","note":"` + strings.Repeat("observed evidence ", 40) + `"}`
executor.RegisterTool(tools.RegisteredTool{
Invocation: tools.StaticInvocation(agentcapabilities.ToolCallKindRead, agentcapabilities.MutationNone),
Definition: tools.Tool{Name: "read_observation", InputSchema: tools.InputSchema{Type: "object", Properties: map[string]tools.PropertySchema{"sample": {Type: "string"}}}},
Handler: func(context.Context, *tools.PulseToolExecutor, map[string]interface{}) (tools.CallToolResult, error) {
return tools.NewTextResult(evidence), nil
},
})
turns := 0
provider := &mockStreamingProvider{chatStreamFunc: func(_ context.Context, req providers.ChatRequest, callback providers.StreamCallback) error {
turns++
require.Less(t, EstimateRequestTokens(req), providers.ContextWindowTokens("anthropic:claude-opus-5"))
if turns == 5 {
found := 0
for _, m := range req.Messages {
if m.ToolResult != nil {
require.False(t, m.ToolResult.IsError)
require.Equal(t, evidence, m.ToolResult.Content, "earlier observations must still be present on the final answer turn")
found++
}
}
require.Equal(t, 4, found)
callback(providers.StreamEvent{Type: "content", Data: providers.ContentEvent{Text: "The current temperature alert is active."}})
callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{StopReason: "end_turn"}})
} else {
require.Less(t, turns, 5)
callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{StopReason: "tool_use", ToolCalls: []providers.ToolCall{{ID: fmt.Sprintf("read-%d", turns), Name: "read_observation", Input: map[string]interface{}{"sample": fmt.Sprint(turns)}}}}})
}
return nil
}}
loop := NewAgenticLoop(provider, executor, "Answer from the supplied observations.")
loop.SetProviderInfo("anthropic", "claude-opus-5")
loop.SetMaxTurns(6)
_, err := loop.ExecuteWithTools(context.Background(), "evidence-retention", []Message{{Role: "user", Content: "Check temperature and alerts."}}, []providers.Tool{{Name: "read_observation"}}, func(StreamEvent) {})
require.NoError(t, err)
require.Equal(t, 5, turns)
}
+78
View File
@@ -2,11 +2,14 @@ package tools
import (
"context"
"encoding/json"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestExecuteGetDiskHealth(t *testing.T) {
@@ -52,3 +55,78 @@ func TestExecuteGetTemperatures(t *testing.T) {
assert.NoError(t, err)
assert.False(t, result.IsError)
}
// The temperature tool must see the same Proxmox-only observations as the
// resource API. A node does not need a second agent to expose its sensors.
func TestExecuteGetTemperaturesCanonicalSources(t *testing.T) {
collected := time.Date(2026, 9, 5, 19, 19, 11, 0, time.UTC)
hot := 94.0
registry := unifiedresources.NewRegistry(nil)
registry.IngestResources([]unifiedresources.Resource{
{ID: "agent-hot", Type: unifiedresources.ResourceTypeAgent, Name: "Hot node", Proxmox: &unifiedresources.ProxmoxData{
NodeName: "hot-node", Temperature: &hot, TemperatureDetails: &models.Temperature{
Available: true, HasCPU: true, CPUPackage: 94, CPUMax: 94, LastUpdate: collected,
Cores: []models.CoreTemp{{Core: 0, Temp: 87}},
NVMe: []models.NVMeTemp{{Device: "nvme0", Temp: 60}},
SMART: []models.DiskTemp{{Device: "sda", Temperature: 38}, {Device: "sdb", Temperature: 30, StandbySkipped: true}},
GPU: []models.GPUTemp{{Device: "gpu0", Edge: 55, Junction: 65}},
},
}, Agent: &unifiedresources.AgentData{Hostname: "hot-node", Sensors: &unifiedresources.HostSensorMeta{TemperatureCelsius: map[string]float64{"cpu_package": 80}, FanRPM: map[string]float64{"fan1": 2000}}}},
{ID: "agent-empty", Type: unifiedresources.ResourceTypeAgent, Name: "empty", Proxmox: &unifiedresources.ProxmoxData{NodeName: "empty"}},
})
executor := NewPulseToolExecutor(ExecutorConfig{ReadState: registry, ControlLevel: ControlLevelReadOnly})
for _, filter := range []map[string]interface{}{
{}, {"host": "hot-node"}, {"host": "Hot node"}, {"host": "agent-hot"}, {"resource_id": "agent-hot"},
} {
result, err := executor.executeGetTemperatures(context.Background(), filter)
require.NoError(t, err)
var rows []struct {
ResourceID string `json:"resource_id"`
Source string `json:"source"`
CPU map[string]float64 `json:"cpu_temps"`
Disks map[string]float64 `json:"disk_temps"`
Fans map[string]float64 `json:"fan_rpm"`
Other map[string]float64 `json:"other_temps"`
Updated string `json:"last_updated"`
}
require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &rows))
require.Len(t, rows, 2)
assert.Equal(t, "agent-hot", rows[0].ResourceID)
assert.Equal(t, "agent-hot", rows[1].ResourceID)
assert.Equal(t, "agent", rows[0].Source)
assert.Equal(t, 80.0, rows[0].CPU["cpu_package"])
assert.Equal(t, 2000.0, rows[0].Fans["fan1"])
assert.Equal(t, "proxmox", rows[1].Source)
assert.Equal(t, 94.0, rows[1].CPU["cpu_max"])
assert.Equal(t, 87.0, rows[1].CPU["cpu_core_0"])
assert.Equal(t, 60.0, rows[1].Disks["nvme0"])
assert.Equal(t, 38.0, rows[1].Disks["sda"])
assert.NotContains(t, rows[1].Disks, "sdb")
assert.Equal(t, 65.0, rows[1].Other["gpu0_junction"])
assert.Equal(t, collected.Format(time.RFC3339), rows[1].Updated)
}
for _, filter := range []map[string]interface{}{{"host": "empty"}, {"resource_id": "missing"}, {"host": "hot-node", "resource_id": "agent-empty"}} {
result, err := executor.executeGetTemperatures(context.Background(), filter)
require.NoError(t, err)
assert.Contains(t, result.Content[0].Text, "No temperature data available")
assert.NotContains(t, result.Content[0].Text, "installed")
}
}
func TestExecuteGetTemperaturesProxmoxWithoutAgent(t *testing.T) {
state := &mockStateProvider{}
state.On("ReadSnapshot").Return(models.StateSnapshot{Nodes: []models.Node{{
ID: "cluster-node", Name: "hot-node", Status: "online",
Temperature: &models.Temperature{Available: true, HasCPU: true, CPUPackage: 94},
}}})
executor := NewPulseToolExecutor(ExecutorConfig{StateProvider: state, ControlLevel: ControlLevelReadOnly})
result, err := executor.ExecuteTool(context.Background(), "pulse_metrics", map[string]interface{}{"type": "temperatures", "host": "hot-node"})
require.NoError(t, err)
require.False(t, result.IsError)
var rows []map[string]interface{}
require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &rows))
require.Len(t, rows, 1)
assert.Equal(t, "proxmox", rows[0]["source"])
assert.NotEmpty(t, rows[0]["resource_id"])
assert.Equal(t, 94.0, rows[0]["cpu_temps"].(map[string]interface{})["cpu_package"])
}
+77 -13
View File
@@ -43,7 +43,7 @@ Examples:
},
"resource_id": {
Type: "string",
Description: "Filter by specific resource ID (for performance, baselines)",
Description: "Filter by specific resource ID (for performance, temperatures, baselines)",
},
"resource_type": {
Type: "string",
@@ -370,6 +370,9 @@ func (e *PulseToolExecutor) executeGetPatterns(_ context.Context, _ map[string]i
func (e *PulseToolExecutor) executeGetTemperatures(_ context.Context, args map[string]interface{}) (CallToolResult, error) {
hostFilter, _ := args["host"].(string)
resourceFilter, _ := args["resource_id"].(string)
hostFilter = strings.TrimSpace(hostFilter)
resourceFilter = strings.TrimSpace(resourceFilter)
rs, err := e.readStateForControl()
if err != nil {
@@ -377,6 +380,8 @@ func (e *PulseToolExecutor) executeGetTemperatures(_ context.Context, args map[s
}
type HostTemps struct {
ResourceID string `json:"resource_id"`
Source string `json:"source"`
Hostname string `json:"hostname"`
Platform string `json:"platform,omitempty"`
CPU map[string]float64 `json:"cpu_temps,omitempty"`
@@ -397,7 +402,8 @@ func (e *PulseToolExecutor) executeGetTemperatures(_ context.Context, args map[s
if hostname == "" {
hostname = strings.TrimSpace(host.Name())
}
if hostFilter != "" && hostname != hostFilter {
if (hostFilter != "" && hostname != hostFilter && host.Name() != hostFilter && host.ID() != hostFilter) ||
(resourceFilter != "" && host.ID() != resourceFilter) {
continue
}
@@ -405,17 +411,19 @@ func (e *PulseToolExecutor) executeGetTemperatures(_ context.Context, args map[s
if sensors == nil {
continue
}
if len(sensors.TemperatureCelsius) == 0 && len(sensors.FanRPM) == 0 {
if len(sensors.TemperatureCelsius) == 0 && len(sensors.FanRPM) == 0 && len(sensors.Additional) == 0 {
continue
}
temps := HostTemps{
Hostname: hostname,
Platform: host.Platform(),
CPU: make(map[string]float64),
Disks: make(map[string]float64),
Fans: make(map[string]float64),
Other: make(map[string]float64),
ResourceID: host.ID(),
Source: "agent",
Hostname: hostname,
Platform: host.Platform(),
CPU: make(map[string]float64),
Disks: make(map[string]float64),
Fans: make(map[string]float64),
Other: make(map[string]float64),
}
// Categorize temperatures
@@ -447,11 +455,67 @@ func (e *PulseToolExecutor) executeGetTemperatures(_ context.Context, args map[s
results = append(results, temps)
}
if len(results) == 0 {
if hostFilter != "" {
return NewTextResult(fmt.Sprintf("No temperature data available for host '%s'. The host may not have a Pulse agent installed or sensors may not be available.", hostFilter)), nil
// Both views come from the canonical resource registry. Keep observations
// from linked providers distinct instead of silently choosing a reading or
// merging sensors collected at different times.
for _, node := range rs.Nodes() {
if node == nil || (hostFilter != "" && node.Name() != hostFilter && node.NodeName() != hostFilter && node.ID() != hostFilter) ||
(resourceFilter != "" && node.ID() != resourceFilter) {
continue
}
return NewTextResult("No temperature data available. Ensure Pulse unified agents are installed on hosts and lm-sensors is available."), nil
details := node.TemperatureDetails()
if !node.HasTemperature() && (details == nil || !details.Available) {
continue
}
temps := HostTemps{
ResourceID: node.ID(), Source: "proxmox", Hostname: node.NodeName(), Platform: "proxmox",
CPU: make(map[string]float64), Disks: make(map[string]float64), Other: make(map[string]float64),
}
if temps.Hostname == "" {
temps.Hostname = node.Name()
}
if node.HasTemperature() {
temps.CPU["cpu_max"] = node.Temperature()
}
if details != nil && details.Available {
if !details.LastUpdate.IsZero() {
temps.LastUpdated = details.LastUpdate.Format(time.RFC3339)
}
if details.HasCPU {
temps.CPU["cpu_package"] = details.CPUPackage
}
for _, core := range details.Cores {
temps.CPU[fmt.Sprintf("cpu_core_%d", core.Core)] = core.Temp
}
for _, disk := range details.NVMe {
temps.Disks[disk.Device] = disk.Temp
}
for _, disk := range details.SMART {
if !disk.StandbySkipped {
temps.Disks[disk.Device] = float64(disk.Temperature)
}
}
for _, gpu := range details.GPU {
for label, value := range map[string]float64{"edge": gpu.Edge, "junction": gpu.Junction, "memory": gpu.Mem} {
if value != 0 {
temps.Other[gpu.Device+"_"+label] = value
}
}
}
}
if len(temps.CPU)+len(temps.Disks)+len(temps.Other) > 0 {
results = append(results, temps)
}
}
if len(results) == 0 {
if resourceFilter != "" {
return NewTextResult(fmt.Sprintf("No temperature data available for resource %q in the current canonical resource observations.", resourceFilter)), nil
}
if hostFilter != "" {
return NewTextResult(fmt.Sprintf("No temperature data available for host '%s' in the current canonical resource observations.", hostFilter)), nil
}
return NewTextResult("No temperature data available in the current canonical resource observations."), nil
}
output, _ := json.MarshalIndent(results, "", " ")