Keep VM command and Discovery evidence truthful

Project verified guest status and decoded output instead of qm transport success. Reject ambiguous terminal outcomes without replay; retain known failed or truncated results as failures without inventing uncertainty. Admit only successful stdout to Discovery and preserve saved results when attempted probes yield no evidence. Keep backup precautions, source bounds and native acceptance dependencies unchanged.

Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-04 19:46:32 +01:00
parent 4188c2a9ba
commit d7ef00f2f2
11 changed files with 232 additions and 61 deletions
@@ -3143,6 +3143,24 @@ traverse an agent.
## Completion Obligations
### VM command results describe the guest, not transport success
A verified terminal `qm guest exec` dictionary must project its guest exit or
signal and decoded stdout/stderr through the existing command-result fields.
Zero CLI exit alone cannot supply success. Nonzero guest exits, signals and
explicit output truncation are failures, but known termination does not require
an uncertainty cooldown. Missing, malformed or conflicting terminal outcomes
remain deferred; postflight lock failure retains the raw CLI diagnostic and
cannot publish usable guest evidence. Preserve single-attempt execution,
approval/identity checks, output bounds and fresh config admission.
`TestCommandClientGuestOutcomeContract` covers status/output projection,
truncation, ambiguity and independent subsequent admission.
`TestGuestExecRealServerAgentDiscoveryAdmission` must carry decoded healthy
evidence, exclude failed guest output, and resume on an explicit healthy scan
over the real registered server/agent WebSocket boundary. Fake provider/config
fixtures do not establish native PVE/HAOS safety or delivered behaviour.
### Docker update readback is not mutation history
The agent's post-update inspect can observe a running replacement whose health
@@ -3553,10 +3571,14 @@ loss imposes a one-minute uncertainty pause; no command is replayed. The next
explicit request must pass fresh config admission. A local process exit or kill
is not proof that guest work stopped: a zero CLI exit with only a guest PID,
nonterminal, malformed or ambiguous JSON stays uncertain. Only one complete
canonical `exited` true/1 dictionary establishes observed guest termination;
this does not alter the existing CLI exit/output payload semantics or prove
thaw. Postflight failure retains that actual exit/output but disqualifies
successful live evidence.
canonical `exited` true/1 dictionary with exactly one valid guest exit code or
signal supplies observed termination. Verified completion projects guest
stdout/stderr and exit status in the existing payload; it cannot promote a
failed guest command from `qm`'s zero exit. A Linux signal uses the conventional
128+signal exit status. Explicit truncated output is a known failed observation,
not an unknown handoff. Missing/ambiguous status or malformed output stays
deferred. Postflight failure retains the actual raw CLI exit/output instead of
qualifying it as live evidence. None of these outcomes establishes thaw.
The guard is not shared with the HTTP poller, separate agent processes or PVE
backup workers, and config checks do not reserve QGA. Agent restart also does
@@ -2117,6 +2117,23 @@ service-history reads plus denial/recovery without fabricated samples.
## Completion Obligations
### Discovery accepts successful command evidence only
Only stdout from a successful result with zero exit status and no error enters
Discovery analysis or saved observations. Stderr, failed/partial output, a nil
result and contradictory success/status cannot identify a service. Independent
healthy output survives optional probe failures. When an attempted scan has
failures and no successful stdout evidence, return a fixed no-evidence error
and truthful terminal progress; manual metadata fallback and background
enhancement must not replace saved results, notes or observation times. An
explicit later healthy scan resumes normally; no failed command is replayed.
`TestDiscoveryFailedCommandEvidenceContract` covers VM/nested-Docker/LXC/host/
Docker output admission, optional partial evidence, saved manual/background
results and explicit recovery. Registered server/agent controls cover actual
guest result propagation, with fake config/provider boundaries. This is source
truthfulness, not native thaw, containing-release or workload acceptance.
1. Update this contract when monitoring truth ownership changes
2. Tighten guardrails when `GetState()`-centric paths are removed
3. Keep discovery-provider, host-agent ingest, guest-memory trust, metrics-history, storage-risk, Docker/Podman container collection, Docker report/model payloads, Proxmox Ceph and ZFS compatibility, Docker Swarm collection, mock runtime fixtures, and container bootstrap proof routes explicit in `registry.json`
@@ -2254,6 +2271,14 @@ service-history reads plus denial/recovery without fabricated samples.
### Discovery observes VM guest execution pauses
Verified VM results describe the guest's terminal status and decoded output,
not successful CLI delivery. Discovery admits only successful stdout as command
evidence for every resource type; stderr and failed/partial output are not
observations. An attempted scan with failures but no successful stdout returns
an error without replacing saved results or invoking analysis. Normal empty
successful probes and independent healthy output alongside optional failures
retain their existing semantics.
Manual, automatic and nested-Docker VM scans use the existing agent execution
boundary; they cannot infer safety from the Web drawer's snapshot. VM routing
requires an exact connected agent ID or a unique owning-node hostname, never a
+2 -1
View File
@@ -528,5 +528,6 @@ func TestCommandClientGuestCompletionContract(t *testing.T) {
}
func TestCommandClientGuestOutcomeContract(t *testing.T) {
testGuestExecProjectsGuestOutcome(t)
t.Run("execution-and-admission", testGuestExecProjectsGuestOutcome)
t.Run("terminal-dictionary", TestGuestExecTerminalDictionary)
}
+16 -3
View File
@@ -1666,9 +1666,10 @@ func (c *CommandClient) executeCommand(ctx context.Context, payload executeComma
}
if guard != nil {
guest, terminal := decodeGuestExecOutcome(result.Stdout)
// Killing the local qm process does not prove a handed-off guest
// command stopped. Never retry it or immediately start the next one.
if err != nil || cmdCtx.Err() != nil || outputIncomplete || !guestExecCompleted(result.Stdout) {
if err != nil || cmdCtx.Err() != nil || outputIncomplete || !terminal {
uncertain = true
result.Success = false
result.Error = agentexec.GuestExecDeferred(agentexec.GuestExecCompletionUnknown).Error()
@@ -1676,9 +1677,21 @@ func (c *CommandClient) executeCommand(ctx context.Context, payload executeComma
uncertain = true
result.Success = false
result.Error = err.Error()
} else {
// For verified completion, the command payload describes the
// guest, not qm's successful delivery of a failing command.
// Wrapper stderr remains diagnostic output, never evidence.
if result.Stderr != "" && guest.stderr != "" {
guest.stderr += "\n" + result.Stderr
} else if result.Stderr != "" {
guest.stderr = result.Stderr
}
result.Stdout, result.Stderr = guest.stdout, guest.stderr
result.ExitCode, result.Error = guest.exitCode, guest.error
result.Success = guest.error == ""
}
// Retain the actual process exit/output, even if postflight safety
// is unknown. Discovery must not treat that output as fresh evidence.
// Unverified completion/postflight retains the actual CLI exit/output.
// Discovery must not treat that raw diagnostic as fresh evidence.
}
return result
+78 -16
View File
@@ -3,50 +3,112 @@ package hostagent
import (
"bytes"
"encoding/json"
"fmt"
"io"
"strconv"
"strings"
"unicode/utf8"
)
// qm can exit zero after handing off a guest PID without observing completion
// (for example when synchronous waiting expires). Only its complete terminal
// dictionary, not a local CLI exit, permits another QGA command. Keep the raw
// output in the ordinary command result; this decoder never logs guest data.
func guestExecCompleted(stdout string) bool {
if len(stdout) == 0 || len(stdout) > maxCommandOutputSize {
return false
type guestExecOutcome struct {
stdout, stderr string
exitCode int
error string
}
// qm's exit is the transport outcome, not the guest's exit. Only a complete,
// unambiguous terminal dictionary supplies guest output and status. Unknown
// completion retains the raw CLI result at the caller and fences admission;
// known failure/truncation is not an unknown handoff. Never log guest data.
func decodeGuestExecOutcome(stdout string) (guestExecOutcome, bool) {
var outcome guestExecOutcome
if len(stdout) == 0 || len(stdout) > maxCommandOutputSize || !utf8.ValidString(stdout) {
return outcome, false
}
d := json.NewDecoder(strings.NewReader(stdout))
start, err := d.Token()
if err != nil || start != json.Delim('{') {
return false
return outcome, false
}
fields := make(map[string]json.RawMessage)
for d.More() {
token, err := d.Token()
name, ok := token.(string)
if err != nil || !ok {
return false
return outcome, false
}
if _, duplicate := fields[name]; duplicate {
return false
return outcome, false
}
if name != "exited" && strings.EqualFold(name, "exited") {
return false
for _, canonical := range []string{"exited", "exitcode", "signal", "out-data", "err-data", "out-truncated", "err-truncated"} {
if name != canonical && strings.EqualFold(name, canonical) {
return outcome, false
}
}
var value json.RawMessage
if d.Decode(&value) != nil {
return false
return outcome, false
}
fields[name] = value
}
end, err := d.Token()
if err != nil || end != json.Delim('}') {
return false
return outcome, false
}
if _, err := d.Token(); err != io.EOF {
return false
return outcome, false
}
// PVE/QGA's numeric true is 1; accept JSON true from compatible codecs.
exited := bytes.TrimSpace(fields["exited"])
return bytes.Equal(exited, []byte("1")) || bytes.Equal(exited, []byte("true"))
if !bytes.Equal(exited, []byte("1")) && !bytes.Equal(exited, []byte("true")) {
return outcome, false
}
exit, hasExit := fields["exitcode"]
signal, hasSignal := fields["signal"]
if hasExit == hasSignal {
return outcome, false
}
status, maximum := exit, 255
if hasSignal {
status, maximum = signal, 64 // Linux QGA execution only.
}
code, err := strconv.Atoi(string(bytes.TrimSpace(status)))
if err != nil || code < 0 || code > maximum || (hasSignal && code == 0) {
return outcome, false
}
outcome.exitCode = code
if hasSignal {
outcome.exitCode += 128
outcome.error = fmt.Sprintf("guest command terminated by signal %d", code)
} else if code != 0 {
outcome.error = fmt.Sprintf("guest command exited with status %d", code)
}
for key, target := range map[string]*string{"out-data": &outcome.stdout, "err-data": &outcome.stderr} {
if data, exists := fields[key]; exists {
if bytes.Equal(bytes.TrimSpace(data), []byte("null")) || json.Unmarshal(data, target) != nil {
return guestExecOutcome{}, false
}
}
}
for _, key := range []string{"out-truncated", "err-truncated"} {
if flag, exists := fields[key]; exists {
switch string(bytes.TrimSpace(flag)) {
case "true", "1":
// Keep a known nonzero/signal outcome even when its output
// is also incomplete; neither qualifies as success.
if outcome.error == "" {
outcome.error = "guest command output was truncated"
}
case "false", "0":
default:
return guestExecOutcome{}, false
}
}
}
return outcome, true
}
func guestExecCompleted(stdout string) bool {
_, terminal := decodeGuestExecOutcome(stdout)
return terminal
}
@@ -2,6 +2,7 @@ package hostagent
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"os/exec"
@@ -45,7 +46,7 @@ func testGuestExecRealServerAgentDiscoveryAdmission(t *testing.T) {
defer ts.Close()
logger := zerolog.Nop()
c := NewCommandClient(Config{PulseURL: ts.URL, APIToken: "fixture", StateDir: t.TempDir(), Logger: &logger}, "node-agent", "node", "linux", "6")
var locked, lockDuringHandoff atomic.Bool
var locked, lockDuringHandoff, failedGuest atomic.Bool
var qgaCalls, configReads atomic.Int32
locked.Store(true)
c.guestExecAdmission = newGuestExecGuard(func(_ context.Context, vmid string) ([]byte, error) {
@@ -69,7 +70,11 @@ func testGuestExecRealServerAgentDiscoveryAdmission(t *testing.T) {
locked.Store(true)
}
}
return exec.CommandContext(ctx, "sh", "-c", `printf '%s' '{"exited":1,"exitcode":0,"out-data":"safe-observation"}'`)
reply := `{"exited":1,"exitcode":0,"out-data":"safe-observation"}`
if name == localGuestExecQM && failedGuest.Load() {
reply = `{"exited":1,"exitcode":7,"out-data":"misleading service identity","err-data":"failed probe"}`
}
return exec.CommandContext(ctx, "sh", "-c", "printf '%s' "+shellQuote(reply))
}
t.Cleanup(func() { execCommandContext = oldExec })
ctx, cancel := context.WithCancel(context.Background())
@@ -109,19 +114,34 @@ func testGuestExecRealServerAgentDiscoveryAdmission(t *testing.T) {
if err != nil || got == nil || int32(len(got.CommandOutputs)) != want || qgaCalls.Load() != want {
t.Fatalf("healthy explicit scan changed: %+v %v calls=%d", got, err, qgaCalls.Load())
}
for _, output := range got.CommandOutputs {
if output != "safe-observation" {
t.Fatalf("CLI wrapper became discovery evidence: %q", output)
}
}
failedGuest.Store(true)
got, err = scanner.ScanVM(context.Background(), "node-agent", "node", "105")
if !errors.Is(err, servicediscovery.ErrNoCommandEvidence) || got == nil || len(got.CommandOutputs) != 0 || qgaCalls.Load() != 2*want {
t.Fatalf("known failed guest became discovery evidence or uncertainty: %+v %v calls=%d", got, err, qgaCalls.Load())
}
failedGuest.Store(false)
got, err = scanner.ScanVM(context.Background(), "node-agent", "node", "105")
if err != nil || got == nil || int32(len(got.CommandOutputs)) != want || qgaCalls.Load() != 3*want {
t.Fatalf("explicit healthy scan could not resume: %+v %v calls=%d", got, err, qgaCalls.Load())
}
lockDuringHandoff.Store(true)
got, err = scanner.ScanVM(context.Background(), "node-agent", "node", "105")
if err == nil || !agentexec.IsGuestExecDeferred(err.Error()) || got == nil || len(got.CommandOutputs) != 0 || qgaCalls.Load() != want+1 {
if err == nil || !agentexec.IsGuestExecDeferred(err.Error()) || got == nil || len(got.CommandOutputs) != 0 || qgaCalls.Load() != 3*want+1 {
t.Fatalf("in-flight deferral was lost/retried: %+v %v calls=%d", got, err, qgaCalls.Load())
}
// Host probes do not use this QGA guard, even while VM admission is paused.
if host, err := scanner.ScanHost(context.Background(), "node-agent", "node"); err != nil || host == nil || len(host.CommandOutputs) == 0 || qgaCalls.Load() != want+1 {
if host, err := scanner.ScanHost(context.Background(), "node-agent", "node"); err != nil || host == nil || len(host.CommandOutputs) == 0 || qgaCalls.Load() != 3*want+1 {
t.Fatalf("host control changed: %+v %v", host, err)
}
if !s.IsAgentConnected("node-agent") {
t.Fatal("deferral dropped session liveness")
}
if configReads.Load() != 2+want*2+2 {
if configReads.Load() != 2+3*want*2+2 {
t.Fatalf("config retries/unexpected reads: %d", configReads.Load())
}
if !strings.Contains(err.Error(), "operation lock") {
+6 -2
View File
@@ -100,7 +100,11 @@ func TestGuestExecCommandPreflightAndPostflight(t *testing.T) {
if tc.wantCalls == 0 && (got.Stdout != "" || got.Stderr != "") {
t.Fatal("preflight published guest output")
}
if tc.wantCalls == 1 && (got.Stdout != `{"exited":1,"exitcode":0,"out-data":"original-result"}` || got.ExitCode != 0) {
wantOutput := `{"exited":1,"exitcode":0,"out-data":"original-result"}`
if tc.wantSuccess {
wantOutput = "original-result"
}
if tc.wantCalls == 1 && (got.Stdout != wantOutput || got.ExitCode != 0) {
t.Fatalf("lost actual process completion: %#v", got)
}
if strings.Contains(got.Error, "private") {
@@ -201,7 +205,7 @@ func TestGuestExecCanceledHandoffNeverRetriesAndRechecksOnExplicitResumption(t *
return exec.CommandContext(ctx, "sh", "-c", `printf '%s' '{"exited":1,"exitcode":0,"out-data":"resumed"}'`)
}
got = c.executeCommand(context.Background(), payload)
if !got.Success || got.Stdout != `{"exited":1,"exitcode":0,"out-data":"resumed"}` || reads.Load() != 3 || calls.Load() != 2 {
if !got.Success || got.Stdout != "resumed" || reads.Load() != 3 || calls.Load() != 2 {
t.Fatalf("explicit resumption failed: %#v", got)
}
}
+42 -20
View File
@@ -69,6 +69,12 @@ const (
defaultDeepScannerTimeout = 30 * time.Second
)
// ErrNoCommandEvidence prevents an attempted but failed scan from replacing
// saved observations through metadata-only fallback. It contains no command
// output or provider error. Individual optional failures can still coexist
// with independent, successful stdout evidence.
var ErrNoCommandEvidence = errors.New("Discovery commands produced no successful output")
// NewDeepScanner creates a new deep scanner.
func NewDeepScanner(executor CommandExecutor) *DeepScanner {
return &DeepScanner{
@@ -241,6 +247,7 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
var mu sync.Mutex
var safetyError error
failedCommands := 0
runCommand := func(cmd DiscoveryCommand) {
cancelBeforeAdmission := func() {
if vmTarget {
@@ -275,6 +282,7 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
if targetType == "container" || targetType == "vm" {
if err := ValidateResourceID(targetID); err != nil {
mu.Lock()
failedCommands++
result.Errors[cmd.Name] = fmt.Sprintf("invalid target ID: %v", err)
mu.Unlock()
return
@@ -319,6 +327,7 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
}
if err != nil {
failedCommands++
if !cmd.Optional {
result.Errors[cmd.Name] = err.Error()
}
@@ -331,23 +340,29 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
return
}
if cmdResult != nil {
output := cmdResult.Stdout
if cmdResult.Stderr != "" && output != "" {
output += "\n--- stderr ---\n" + cmdResult.Stderr
} else if cmdResult.Stderr != "" {
output = cmdResult.Stderr
if cmdResult == nil {
failedCommands++
if !cmd.Optional {
result.Errors[cmd.Name] = "command returned no result"
}
if output != "" {
result.CommandOutputs[cmd.Name] = output
}
if !cmdResult.Success && cmdResult.Error != "" && !cmd.Optional {
result.Errors[cmd.Name] = cmdResult.Error
}
if !cmdResult.Success {
} else {
successful := cmdResult.Success && cmdResult.ExitCode == 0 && cmdResult.Error == ""
if successful {
// stdout from a successful command is observation evidence.
// Failed/partial stdout and any stderr are diagnostics only;
// they must not identify a service or enter saved evidence.
if cmdResult.Stdout != "" {
result.CommandOutputs[cmd.Name] = cmdResult.Stdout
}
} else {
failedCommands++
if !cmd.Optional {
message := cmdResult.Error
if message == "" {
message = fmt.Sprintf("command failed with status %d", cmdResult.ExitCode)
}
result.Errors[cmd.Name] = message
}
event := scanLog.Debug()
if !cmd.Optional {
event = scanLog.Warn()
@@ -391,6 +406,10 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
wg.Wait()
result.CompletedAt = time.Now()
scanError := safetyError
if scanError == nil && failedCommands > 0 && len(result.CommandOutputs) == 0 {
scanError = ErrNoCommandEvidence
}
// Broadcast scan completion
completionProgress := DiscoveryProgress{
@@ -403,9 +422,12 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
ElapsedMs: result.CompletedAt.Sub(startTime).Milliseconds(),
PercentComplete: 100,
}
if safetyError != nil {
completionProgress.CurrentStep = "Guest execution paused"
completionProgress.Error = safetyError.Error()
if scanError != nil {
completionProgress.CurrentStep = "No command evidence"
if safetyError != nil {
completionProgress.CurrentStep = "Guest execution paused"
}
completionProgress.Error = scanError.Error()
completionProgress.CompletedSteps = len(result.CommandOutputs)
completionProgress.PercentComplete = float64(completionProgress.CompletedSteps) / float64(len(commands)) * 100
}
@@ -418,7 +440,7 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
Dur("duration", result.CompletedAt.Sub(result.StartedAt)).
Msg("Deep scan completed")
return result, safetyError
return result, scanError
}
// buildCommand wraps the command appropriately for the resource type.
@@ -356,11 +356,11 @@ func TestDeepScanner_OutputHandling(t *testing.T) {
if err != nil {
t.Fatalf("Scan error: %v", err)
}
if out := result.CommandOutputs["docker_containers"]; !strings.Contains(out, "--- stderr ---") {
t.Fatalf("expected combined stderr output, got %s", out)
if out := result.CommandOutputs["docker_containers"]; out != "out" {
t.Fatalf("expected successful stdout only, got %s", out)
}
if out := result.CommandOutputs["docker_images"]; out != "err-only" {
t.Fatalf("expected stderr-only output, got %s", out)
if out, exists := result.CommandOutputs["docker_images"]; exists {
t.Fatalf("stderr-only diagnostic became evidence: %s", out)
}
}
@@ -374,8 +374,8 @@ func TestDeepScanner_CommandErrorHandling(t *testing.T) {
TargetID: "host1",
Hostname: "host1",
})
if err != nil {
t.Fatalf("Scan error: %v", err)
if !errors.Is(err, ErrNoCommandEvidence) {
t.Fatalf("expected failed scan to retain no-evidence outcome, got %v", err)
}
if _, ok := result.Errors["docker_containers"]; !ok {
t.Fatalf("expected error for non-optional command")
+2 -2
View File
@@ -1884,9 +1884,9 @@ func (s *Service) DiscoverResource(ctx context.Context, req DiscoveryRequest) (*
var scanError error
if s.scanner != nil && s.IsCommandScanningEnabled() {
scanResult, scanError = s.scanner.Scan(ctx, req)
if scanError != nil && agentexec.IsGuestExecDeferred(scanError.Error()) {
if scanError != nil && (agentexec.IsGuestExecDeferred(scanError.Error()) || errors.Is(scanError, ErrNoCommandEvidence)) {
// Keep saved observations/notes and the original in-flight outcome;
// metadata fallback must not turn a paused live scan into success.
// metadata fallback must not turn a paused or failed live scan into success.
inProg.err = scanError
return nil, inProg.err
}
+4 -2
View File
@@ -2766,8 +2766,10 @@ func TestDiscoveryGuestSafetyStopsAndPreservesSavedEvidence(t *testing.T) {
}
func TestDiscoveryFailedCommandEvidenceContract(t *testing.T) {
testDiscoveryRejectsFailedCommandEvidence(t)
testDiscoveryAllFailedCommandsKeepSavedEvidence(t)
t.Run("successful-stdout-only", testDiscoveryRejectsFailedCommandEvidence)
t.Run("saved-evidence-and-recovery", testDiscoveryAllFailedCommandsKeepSavedEvidence)
t.Run("stderr-is-not-evidence", TestDeepScanner_OutputHandling)
t.Run("failed-scan-outcome", TestDeepScanner_CommandErrorHandling)
}
func TestDiscoveryGuestCancellationContract(t *testing.T) {