Guard VM Discovery and command execution at the owning host agent

Require local current-node lock evidence and registered guard support before QGA execution. Serialise VM probes, stop on deferral, retain actual in-flight outcomes and saved Discovery evidence, and pause uncertain completion without replay. Config checks do not reserve QGA or prove native thaw; the existing backup monitoring-outage precaution remains.

Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-04 17:59:09 +01:00
parent def087ed90
commit be319e8204
18 changed files with 1090 additions and 103 deletions
@@ -741,6 +741,11 @@ installer download and the agent's subsequent Pulse TLS connection.
## Canonical Files
- `internal/agentexec/guest_exec_safety.go`
- `internal/hostagent/guest_exec_guard.go`
- `internal/hostagent/guest_exec_config_linux.go`
- `internal/hostagent/guest_exec_config_other.go`
1. `internal/api/agent_install_command_shared.go`
2. `internal/api/configapi/config_setup_handlers.go`
2a. `internal/api/configapi/setup_script_render.go`
@@ -3527,6 +3532,44 @@ Agent` secondary handoff against the live setup wizard instead of relying
## Current State
### VM guest execution admission (backup precaution)
`AgentRegisterPayload.guest_exec_guard_version` is optional; version 1 on a
Linux full-trust command session means the agent implements the local guard,
not that the VM is thawed or safe to overlap a backup. The server binds it to
the authenticated session and refuses VM command/file dispatch to absent,
unknown-version or non-Linux support. Version strings, user approval and the
internal Trusted flag cannot substitute for that capability. Host/LXC commands,
tenant admission, approvals and typed action-runner exclusions are unchanged.
The host agent validates and canonicalises the numeric VMID, requires its
current local-node config, and reads the fixed trusted `qm config --current`
without QGA before and after one absolute-path `qm guest exec`. Missing,
ambiguous, oversized or locked config fails closed. Per-VM process-wide
admission serialises concurrent commands, respects cancellation and bounds
state to 4,096 entries. Unknown handed-off completion or postflight lock/owner
loss imposes a one-minute uncertainty pause; no command is replayed. The next
explicit request must pass fresh config admission. A local process exit or kill
is not proof that guest work stopped; postflight failure retains the actual
exit/output but disqualifies successful live evidence.
The guard is not shared with the HTTP poller, separate agent processes or PVE
backup workers, and config checks do not reserve QGA. Agent restart also does
not retain the in-memory uncertainty pause. Continue the stop-before-affected-
backup / independent covered-filesystem-write restart precaution until native
overlap, thaw, writes, liveness and resumption are verified. Do not unlock,
thaw or automatically restart from this guard. Raw host-shell commands and
typed VM lifecycle operations are not relabelled as guarded guest reads.
`TestCommandClientGuestExecutionAdmissionContract` pins no unverified handoff;
`TestGuestExecRealServerAgentDiscoveryAdmission` connects real registration,
server/agent WebSockets and VM/nested-Docker Discovery with fake config/provider
boundaries. `TestVMGuestExecCapabilityBoundToRegisteredSession` pins old/unknown
agent rejection and retained deferral. Config ambiguity, cancellation,
serialization/capacity and explicit-resumption controls are source proofs, not
native PVE/HAOS-QGA, thaw, installed-agent or release acceptance.
### Credential-safe container diagnostics bootstrap (1 October 2026)
The existing container-runtime migration command now shares the Proxmox
@@ -2252,6 +2252,33 @@ service-history reads plus denial/recovery without fabricated samples.
## Current State
### Discovery observes VM guest execution pauses
Manual, automatic and nested-Docker VM scans use the existing agent execution
boundary; they cannot infer safety from the Web drawer's snapshot. VM routing
requires an exact connected agent ID or a unique owning-node hostname, never a
sole unrelated agent or guest-name hint. The agent independently requires its
current local VM config and checks the operation lock. Discovery runs VM probes
serially and stops on any explicit guest safety deferral, including optional
probes. Deferred in-flight output is not fresh discovery evidence; progress
retains an error and actual partial completion rather than 100% success.
A manual deferral is returned as an error without replacing saved discoveries,
notes or raw observations with metadata fallback. Background enhancement keeps
its existing evidence rather than analysing partial deferred output. Clearing
a pause triggers no scan or retry by itself; an existing normal trigger or
explicit request starts a new, guarded attempt. Healthy catalogue commands,
non-VM parallelism, permissions, feature gates and saved-result access remain.
`TestDiscoveryGuestSafetyStopsAndPreservesSavedEvidence` covers both VM/nested
Docker scans, first/later optional-command deferral, manual error and automatic
saved-evidence preservation. Node ambiguity controls exclude misrouting. These
source-connected controls and the agent protocol guard do not reserve QGA
against PVE backup start, establish native thaw, or satisfy the existing native
overlap/covered-write/liveness/resumption dependency. Preserve the monitoring
and alert outage precaution until that operational outcome is verified.
### Saved quiet-hours policy before queue activation
`Monitor.New` binds `alerts.Manager.QuietHoursNotificationPolicy` to the
+40
View File
@@ -0,0 +1,40 @@
package agentexec
import (
"fmt"
"strings"
)
// Version 1 requires local lock admission, per-VM serialization, and an
// uncertainty cooldown. It does NOT reserve QGA against PVE backup workers.
const GuestExecGuardProtocolVersion = 1
const guestExecDeferredPrefix = "VM guest execution deferred: "
// GuestExecDeferred carries only a fixed safety reason, never guest/config data.
func GuestExecDeferred(reason string) error {
return fmt.Errorf("%s%s", guestExecDeferredPrefix, reason)
}
func IsGuestExecDeferred(message string) bool {
return strings.HasPrefix(message, guestExecDeferredPrefix)
}
const (
GuestExecGuardUnavailable = "update the host agent for backup-safety support"
GuestExecInvalidTarget = "the local VM identity is invalid"
GuestExecLockUnverified = "the local VM backup lock could not be verified"
GuestExecVMLocked = "the VM has an active operation lock"
GuestExecCompletionUnknown = "guest command completion could not be verified"
GuestExecCooldown = "a previous guest command is still uncertain"
GuestExecCapacity = "guest command admission is at capacity"
GuestExecTargetUnverified = "the owning node agent could not be uniquely identified"
)
func requireVMGuestExecGuard(ac *agentConn, targetType string) error {
if strings.EqualFold(strings.TrimSpace(targetType), "vm") &&
(ac == nil || ac.guestExecGuardVersion != GuestExecGuardProtocolVersion || ac.agent.Platform != "linux") {
return GuestExecDeferred(GuestExecGuardUnavailable)
}
return nil
}
@@ -0,0 +1,74 @@
package agentexec
import (
"context"
"encoding/json"
"fmt"
"testing"
"time"
)
func testVMGuestExecCapabilityGate(t *testing.T) {
for _, tc := range []struct {
version int
platform string
admitted bool
}{
{0, "linux", false}, {1, "linux", true}, {2, "linux", false}, {1, "windows", false},
} {
t.Run(fmt.Sprintf("%s/version-%d", tc.platform, tc.version), func(t *testing.T) {
s := NewServer(allowAllTestTokens)
ts := newWSServer(t, s)
defer ts.Close()
conn, _, err := dialAgentExecWebSocket(t, ts.URL)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
wsWriteMessage(t, conn, mustNewMessage(t, MsgTypeAgentRegister, "", AgentRegisterPayload{AgentID: "node-agent", Hostname: "node", Version: "future-version-is-not-capability", Platform: tc.platform, Token: "fixture", GuestExecGuardVersion: tc.version}))
if !wsReadRegisteredPayload(t, conn).Success {
t.Fatal("registration rejected")
}
cmd := ExecuteCommandPayload{RequestID: "scan", Command: "echo inspected", TargetType: "vm", TargetID: "105", Trusted: true, Timeout: 2}
if !tc.admitted {
if result, err := s.ExecuteCommand(context.Background(), "node-agent", cmd); result != nil || err == nil || !IsGuestExecDeferred(err.Error()) {
t.Fatalf("legacy target dispatched: result=%#v err=%v", result, err)
}
if result, err := s.ReadFile(context.Background(), "node-agent", ReadFilePayload{RequestID: "read", Path: "/etc/os-release", TargetType: "vm", TargetID: "105"}); result != nil || err == nil || !IsGuestExecDeferred(err.Error()) {
t.Fatalf("legacy file target dispatched: result=%#v err=%v", result, err)
}
// The first message must be the pong, not an unsafe VM dispatch.
wsWriteMessage(t, conn, mustNewMessage(t, MsgTypeAgentPing, "", nil))
if msg := wsReadRawMessage(t, conn); msg.Type != MsgTypePong {
t.Fatalf("unsafe message queued: %s", msg.Type)
}
return
}
type outcome struct {
result *CommandResultPayload
err error
}
done := make(chan outcome, 1)
go func() {
result, err := s.ExecuteCommand(context.Background(), "node-agent", cmd)
done <- outcome{result, err}
}()
msg, err := wsReadRawMessageWithTimeout(conn, 2*time.Second)
if err != nil || msg.Type != MsgTypeExecuteCmd || msg.Payload == nil {
t.Fatalf("dispatch=%#v err=%v", msg, err)
}
var payload ExecuteCommandPayload
if err := json.Unmarshal(*msg.Payload, &payload); err != nil {
t.Fatal(err)
}
if payload.TargetID != "105" || payload.TargetType != "vm" || payload.Command != cmd.Command || !payload.Trusted {
t.Fatalf("target changed: %#v", payload)
}
wsWriteMessage(t, conn, mustNewMessage(t, MsgTypeCommandResult, payload.RequestID, CommandResultPayload{RequestID: payload.RequestID, Success: false, ExitCode: -1, Error: GuestExecDeferred(GuestExecVMLocked).Error()}))
got := <-done
if got.err != nil || got.result == nil || got.result.Success || !IsGuestExecDeferred(got.result.Error) {
t.Fatalf("deferral lost: %#v %v", got.result, got.err)
}
})
}
}
+22 -14
View File
@@ -130,15 +130,16 @@ type CommandAuthorizationRequest struct {
}
type agentConn struct {
conn *websocket.Conn
agent ConnectedAgent
admission AgentAdmission
sessionKey string
authorityKey string
approvalGrantKey []byte
writeMu sync.Mutex
done chan struct{}
doneOnce sync.Once
conn *websocket.Conn
agent ConnectedAgent
admission AgentAdmission
sessionKey string
authorityKey string
guestExecGuardVersion int
approvalGrantKey []byte
writeMu sync.Mutex
done chan struct{}
doneOnce sync.Once
}
type pendingHostOperation struct {
@@ -1376,11 +1377,12 @@ func (s *Server) HandleWebSocket(w http.ResponseWriter, r *http.Request) {
ActionPreflightVersion: reg.ActionPreflightVersion,
DockerObservationVersion: reg.DockerObservationVersion,
},
admission: admission,
sessionKey: agentSessionKey(admission.OrganizationID, admission.AgentID),
authorityKey: uuid.NewString(),
approvalGrantKey: DeriveApprovalGrantKey(reg.Token),
done: make(chan struct{}),
admission: admission,
sessionKey: agentSessionKey(admission.OrganizationID, admission.AgentID),
authorityKey: uuid.NewString(),
approvalGrantKey: DeriveApprovalGrantKey(reg.Token),
guestExecGuardVersion: reg.GuestExecGuardVersion,
done: make(chan struct{}),
}
// Clear deadline for normal operation - both on the WebSocket and underlying connection
@@ -2087,6 +2089,9 @@ func (s *Server) ExecuteCommand(ctx context.Context, agentID string, cmd Execute
if err := requireLegacyFullTrustConnection(ac, "execute_command"); err != nil {
return nil, err
}
if err := requireVMGuestExecGuard(ac, cmd.TargetType); err != nil {
return nil, err
}
if err := s.authorizeCommandPayload(cmd); err != nil {
return nil, err
}
@@ -2782,6 +2787,9 @@ func (s *Server) ReadFile(ctx context.Context, agentID string, req ReadFilePaylo
if err := requireLegacyFullTrustConnection(ac, "read_file"); err != nil {
return nil, err
}
if err := requireVMGuestExecGuard(ac, req.TargetType); err != nil {
return nil, err
}
readLog := log.With().
Str("agent_id", agentID).
+8 -1
View File
@@ -2169,7 +2169,10 @@ func TestTypedOperation_SocketDropAfterSendUnblocksDispatch(t *testing.T) {
t.Fatal(err)
}
done := make(chan error, 1)
go func() { _, err := s.ExecuteProxmoxGuestLifecycle(context.Background(), "typed-a1", request); done <- err }()
go func() {
_, err := s.ExecuteProxmoxGuestLifecycle(context.Background(), "typed-a1", request)
done <- err
}()
if sent, ok := agent.nextMessage(3 * time.Second); !ok || sent.Type != MsgTypeProxmoxGuestLifecycle {
t.Fatalf("dispatched request = %+v", sent)
}
@@ -2247,3 +2250,7 @@ func TestRegistrationReservesFirstFrameBeforePublishingSession(t *testing.T) {
t.Fatal("published session permits command writes before registration acknowledgement")
}
}
func TestVMGuestExecCapabilityBoundToRegisteredSession(t *testing.T) {
testVMGuestExecCapabilityGate(t)
}
+1
View File
@@ -107,6 +107,7 @@ type AgentRegisterPayload struct {
OperationReceiptVersion int `json:"operation_receipt_version,omitempty"`
ActionPreflightVersion int `json:"action_preflight_version,omitempty"`
DockerObservationVersion int `json:"docker_observation_version,omitempty"`
GuestExecGuardVersion int `json:"guest_exec_guard_version,omitempty"`
}
// RegisteredPayload is sent by server after successful registration
@@ -518,3 +518,7 @@ func TestCommandClient_ReplayedRequestWaitsForInFlightHandlerInsteadOfDropping(t
func TestCommandClientGuestExecutionAdmissionContract(t *testing.T) {
testVMGuestExecNoUnverifiedHandoff(t)
}
func TestGuestExecRealServerAgentDiscoveryAdmission(t *testing.T) {
testGuestExecRealServerAgentDiscoveryAdmission(t)
}
+61 -1
View File
@@ -108,6 +108,7 @@ type CommandClient struct {
caCertPath string
serverFingerprint string
deploySSHUser string
guestExecAdmission *guestExecGuard // nil selects the process-wide local guard
commandPolicy *agentexec.CommandPolicy
packageUpdates *packageUpdateManager
storageCleanup *storageCleanupManager
@@ -274,6 +275,7 @@ type registerPayload struct {
OperationReceiptVersion int `json:"operation_receipt_version,omitempty"`
ActionPreflightVersion int `json:"action_preflight_version,omitempty"`
DockerObservationVersion int `json:"docker_observation_version,omitempty"`
GuestExecGuardVersion int `json:"guest_exec_guard_version,omitempty"`
}
type registeredPayload struct {
@@ -538,6 +540,7 @@ func (c *CommandClient) sendRegistration(conn *websocket.Conn) error {
OperationReceiptVersion: c.operationReceiptVersion(),
ActionPreflightVersion: agentexec.ActionPreflightProtocolVersion,
DockerObservationVersion: agentexec.DockerContainerObservationProtocolVersion,
GuestExecGuardVersion: c.guestExecGuardVersion(),
})
if err != nil {
return fmt.Errorf("marshal registration payload: %w", err)
@@ -552,6 +555,13 @@ func (c *CommandClient) sendRegistration(conn *websocket.Conn) error {
return conn.WriteJSON(msg)
}
func (c *CommandClient) guestExecGuardVersion() int {
if runtime.GOOS == "linux" && !c.actionRunnerOnly {
return agentexec.GuestExecGuardProtocolVersion
}
return 0
}
func (c *CommandClient) operationReceiptVersion() int {
if c != nil && c.operationReceiptErr == nil && c.operationReceipts != nil {
return operationreceipt.ProtocolVersion
@@ -1556,11 +1566,44 @@ func (c *CommandClient) executeCommand(ctx context.Context, payload executeComma
cmdCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
var guard *guestExecGuard
var vmid string
uncertain := false
if strings.EqualFold(strings.TrimSpace(payload.TargetType), "vm") {
var valid bool
vmid, valid = canonicalGuestExecVMID(payload.TargetID)
if !valid {
result.Error = agentexec.GuestExecDeferred(agentexec.GuestExecInvalidTarget).Error()
result.ExitCode = -1
return result
}
guard = c.guestExecAdmission
if guard == nil {
guard = localGuestExecGuard
}
release, err := guard.acquire(cmdCtx, vmid)
if err != nil {
result.Error = err.Error()
result.ExitCode = -1
return result
}
defer func() { release(uncertain) }()
if err := guard.verifyUnlocked(cmdCtx, vmid); err != nil {
result.Error = err.Error()
result.ExitCode = -1
return result
}
}
command := wrapCommand(payload)
// Execute the command
var cmd *exec.Cmd
if runtime.GOOS == "windows" {
if guard != nil {
// No host-shell or PATH interpretation of the owning VM/CLI. The
// guest shell still receives the original, policy-approved command.
cmd = execCommandContext(cmdCtx, localGuestExecQM, "guest", "exec", vmid, "--", "sh", "-c", payload.Command)
} else if runtime.GOOS == "windows" {
cmd = execCommandContext(cmdCtx, "cmd", "/C", command)
} else {
cmd = execCommandContext(cmdCtx, "sh", "-c", command)
@@ -1585,6 +1628,7 @@ func (c *CommandClient) executeCommand(ctx context.Context, payload executeComma
cmd.Stderr = stderr
err := cmd.Run()
outputIncomplete := errors.Is(err, exec.ErrWaitDelay)
result.Stdout = stdout.String()
result.Stderr = stderr.String()
@@ -1618,6 +1662,22 @@ func (c *CommandClient) executeCommand(ctx context.Context, payload executeComma
result.Success = true
}
if guard != nil {
// Killing the local qm process does not prove a handed-off guest
// command stopped. Never retry it or immediately start the next one.
if err != nil || cmdCtx.Err() != nil || outputIncomplete {
uncertain = true
result.Success = false
result.Error = agentexec.GuestExecDeferred(agentexec.GuestExecCompletionUnknown).Error()
} else if err := guard.verifyUnlocked(cmdCtx, vmid); err != nil {
uncertain = true
result.Success = false
result.Error = err.Error()
}
// Retain the actual process exit/output, even if postflight safety
// is unknown. Discovery must not treat that output as fresh evidence.
}
return result
}
@@ -0,0 +1,42 @@
//go:build linux
package hostagent
import (
"context"
"errors"
"os"
"os/exec"
"path/filepath"
"time"
)
const localGuestExecQM = "/usr/sbin/qm"
func readLocalGuestExecConfig(ctx context.Context, vmid string) ([]byte, error) {
// qm can read a remote node's cluster config too. Require the VM to be
// owned by this node before AND after the non-QGA config query.
configPath := filepath.Join("/etc/pve/local/qemu-server", vmid+".conf")
isLocal := func() bool {
info, err := os.Lstat(configPath)
return err == nil && info.Mode().IsRegular()
}
if !isLocal() {
return nil, errors.New("local guest config unavailable")
}
resolved, err := filepath.EvalSymlinks(localGuestExecQM)
if err != nil || validateTrustedExecutable(resolved, 0) != nil {
return nil, errors.New("trusted qm unavailable")
}
readCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
cmd := exec.CommandContext(readCtx, resolved, "config", vmid, "--current")
configureCommandProcessGroup(cmd)
cmd.WaitDelay = time.Second
stdout, stderr := newCappedBuffer(guestExecConfigLimit), newCappedBuffer(1024)
cmd.Stdout, cmd.Stderr = stdout, stderr
if err := cmd.Run(); err != nil || stdout.truncated || readCtx.Err() != nil || !isLocal() {
return nil, errors.New("local guest config unverified")
}
return []byte(stdout.String()), nil
}
@@ -0,0 +1,14 @@
//go:build !linux
package hostagent
import (
"context"
"errors"
)
const localGuestExecQM = "/usr/sbin/qm"
func readLocalGuestExecConfig(context.Context, string) ([]byte, error) {
return nil, errors.New("local PVE guest config unavailable on this platform")
}
@@ -0,0 +1,130 @@
package hostagent
import (
"context"
"net/http"
"net/http/httptest"
"os/exec"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
"github.com/rcourtman/pulse-go-rewrite/internal/servicediscovery"
"github.com/rs/zerolog"
)
type guestSafetyDiscoveryExecutor struct{ server *agentexec.Server }
func (e guestSafetyDiscoveryExecutor) ExecuteCommand(ctx context.Context, id string, p servicediscovery.ExecuteCommandPayload) (*servicediscovery.CommandResultPayload, error) {
r, err := e.server.ExecuteCommand(ctx, id, agentexec.ExecuteCommandPayload{RequestID: p.RequestID, Command: p.Command, TargetType: p.TargetType, TargetID: p.TargetID, Timeout: p.Timeout, Trusted: true})
if err != nil || r == nil {
return nil, err
}
return &servicediscovery.CommandResultPayload{RequestID: r.RequestID, Success: r.Success, Stdout: r.Stdout, Stderr: r.Stderr, ExitCode: r.ExitCode, Error: r.Error}, nil
}
func (e guestSafetyDiscoveryExecutor) GetConnectedAgents() []servicediscovery.ConnectedAgent {
var out []servicediscovery.ConnectedAgent
for _, a := range e.server.GetConnectedAgents() {
out = append(out, servicediscovery.ConnectedAgent{AgentID: a.AgentID, Hostname: a.Hostname})
}
return out
}
func (e guestSafetyDiscoveryExecutor) IsAgentConnected(id string) bool {
return e.server.IsAgentConnected(id)
}
// Real server/session/registration/agent/Discovery paths, with a fake non-QGA
// config reader and fake provider process. This is NOT native PVE acceptance.
func testGuestExecRealServerAgentDiscoveryAdmission(t *testing.T) {
s := agentexec.NewServer(func(token, agent, host string) bool {
return token == "fixture" && agent == "node-agent" && host == "node"
})
ts := httptest.NewServer(http.HandlerFunc(s.HandleWebSocket))
defer ts.Close()
logger := zerolog.Nop()
c := NewCommandClient(Config{PulseURL: ts.URL, APIToken: "fixture", StateDir: t.TempDir(), Logger: &logger}, "node-agent", "node", "linux", "6")
var locked, lockDuringHandoff atomic.Bool
var qgaCalls, configReads atomic.Int32
locked.Store(true)
c.guestExecAdmission = newGuestExecGuard(func(_ context.Context, vmid string) ([]byte, error) {
configReads.Add(1)
if vmid != "105" {
t.Errorf("wrong VM read: %s", vmid)
}
if locked.Load() {
return []byte("name: vm\nlock: backup\n"), nil
}
return []byte("name: vm\n"), nil
})
oldExec := execCommandContext
execCommandContext = func(ctx context.Context, name string, args ...string) *exec.Cmd {
if name == localGuestExecQM {
qgaCalls.Add(1)
if len(args) != 7 || args[2] != "105" {
t.Errorf("wrong VM handoff: %#v", args)
}
if lockDuringHandoff.Load() {
locked.Store(true)
}
}
return exec.CommandContext(ctx, "sh", "-c", "printf safe-observation")
}
t.Cleanup(func() { execCommandContext = oldExec })
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- c.Run(ctx) }()
defer func() {
cancel()
_ = c.Close()
s.Shutdown()
select {
case <-done:
case <-time.After(3 * time.Second):
t.Error("agent did not stop")
}
}()
deadline := time.Now().Add(3 * time.Second)
for !s.IsAgentConnected("node-agent") && time.Now().Before(deadline) {
time.Sleep(5 * time.Millisecond)
}
if !s.IsAgentConnected("node-agent") {
t.Fatal("agent not connected")
}
scanner := servicediscovery.NewDeepScanner(guestSafetyDiscoveryExecutor{s})
for _, rt := range []servicediscovery.ResourceType{servicediscovery.ResourceTypeVM, servicediscovery.ResourceTypeDockerVM} {
id := "105"
if rt == servicediscovery.ResourceTypeDockerVM {
id = "105:app"
}
got, err := scanner.Scan(context.Background(), servicediscovery.DiscoveryRequest{ResourceType: rt, ResourceID: id, TargetID: "node-agent"})
if err == nil || !agentexec.IsGuestExecDeferred(err.Error()) || got == nil || len(got.CommandOutputs) != 0 || qgaCalls.Load() != 0 {
t.Fatalf("locked scan escaped: %+v %v calls=%d", got, err, qgaCalls.Load())
}
}
locked.Store(false)
got, err := scanner.ScanVM(context.Background(), "node-agent", "node", "105")
want := int32(len(servicediscovery.GetCommandsForResource(servicediscovery.ResourceTypeVM)))
if err != nil || got == nil || int32(len(got.CommandOutputs)) != want || qgaCalls.Load() != want {
t.Fatalf("healthy explicit scan changed: %+v %v calls=%d", got, err, qgaCalls.Load())
}
lockDuringHandoff.Store(true)
got, err = scanner.ScanVM(context.Background(), "node-agent", "node", "105")
if err == nil || !agentexec.IsGuestExecDeferred(err.Error()) || got == nil || len(got.CommandOutputs) != 0 || qgaCalls.Load() != want+1 {
t.Fatalf("in-flight deferral was lost/retried: %+v %v calls=%d", got, err, qgaCalls.Load())
}
// Host probes do not use this QGA guard, even while VM admission is paused.
if host, err := scanner.ScanHost(context.Background(), "node-agent", "node"); err != nil || host == nil || len(host.CommandOutputs) == 0 || qgaCalls.Load() != want+1 {
t.Fatalf("host control changed: %+v %v", host, err)
}
if !s.IsAgentConnected("node-agent") {
t.Fatal("deferral dropped session liveness")
}
if configReads.Load() != 2+want*2+2 {
t.Fatalf("config retries/unexpected reads: %d", configReads.Load())
}
if !strings.Contains(err.Error(), "operation lock") {
t.Fatalf("lost lock reason: %v", err)
}
}
+153
View File
@@ -0,0 +1,153 @@
package hostagent
import (
"context"
"strconv"
"strings"
"sync"
"time"
"unicode/utf8"
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
)
const (
guestExecConfigLimit = 64 << 10
guestExecCooldown = time.Minute
maxGuestExecGuardEntries = 4096
)
type guestExecEntry struct {
done chan struct{}
until time.Time
}
type guestExecGuard struct {
mu sync.Mutex
entries map[string]guestExecEntry
readConfig func(context.Context, string) ([]byte, error)
now func() time.Time
}
// Shared across reconnects and CommandClients in this agent process. This is
// not a lock shared with the Pulse HTTP poller, other processes, or PVE/QGA.
var localGuestExecGuard = newGuestExecGuard(readLocalGuestExecConfig)
func newGuestExecGuard(readConfig func(context.Context, string) ([]byte, error)) *guestExecGuard {
return &guestExecGuard{entries: make(map[string]guestExecEntry), readConfig: readConfig, now: time.Now}
}
func canonicalGuestExecVMID(raw string) (string, bool) {
raw = strings.TrimSpace(raw)
if !safeTargetIDPattern.MatchString(raw) {
return "", false
}
id, err := strconv.ParseUint(raw, 10, 32)
if err != nil || id == 0 {
return "", false
}
return strconv.FormatUint(id, 10), true
}
// Wait only for another local command, never poll QGA or retry a dispatched
// command. A waiter rechecks cooldown and the authoritative config on admission.
func (g *guestExecGuard) acquire(ctx context.Context, vmid string) (func(bool), error) {
for {
if err := ctx.Err(); err != nil {
return nil, err
}
g.mu.Lock()
now := g.now()
entry, exists := g.entries[vmid]
if entry.done != nil {
done := entry.done
g.mu.Unlock()
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-done:
continue
}
}
if now.Before(entry.until) {
g.mu.Unlock()
return nil, agentexec.GuestExecDeferred(agentexec.GuestExecCooldown)
}
if !exists && len(g.entries) >= maxGuestExecGuardEntries {
for key, e := range g.entries {
if e.done == nil && !now.Before(e.until) {
delete(g.entries, key)
}
}
if len(g.entries) >= maxGuestExecGuardEntries {
g.mu.Unlock()
return nil, agentexec.GuestExecDeferred(agentexec.GuestExecCapacity)
}
}
done := make(chan struct{})
g.entries[vmid] = guestExecEntry{done: done}
g.mu.Unlock()
return func(uncertain bool) {
g.mu.Lock()
defer g.mu.Unlock()
if uncertain {
g.entries[vmid] = guestExecEntry{until: g.now().Add(guestExecCooldown)}
} else {
delete(g.entries, vmid)
}
close(done)
}, nil
}
}
func (g *guestExecGuard) verifyUnlocked(ctx context.Context, vmid string) error {
if err := ctx.Err(); err != nil {
return err
}
config, err := g.readConfig(ctx, vmid)
if err != nil {
return agentexec.GuestExecDeferred(agentexec.GuestExecLockUnverified)
}
lock, valid := guestExecConfigLock(config)
if !valid {
return agentexec.GuestExecDeferred(agentexec.GuestExecLockUnverified)
}
if lock != "" {
return agentexec.GuestExecDeferred(agentexec.GuestExecVMLocked)
}
return nil
}
// qm config --current emits one key/value dictionary, not pending/snapshot
// sections. Reject empty, malformed, oversized or duplicate/case-conflicting
// fields rather than allowing last-key-wins to erase a lock.
func guestExecConfigLock(config []byte) (string, bool) {
if len(config) == 0 || len(config) > guestExecConfigLimit || !utf8.Valid(config) {
return "", false
}
fields := make(map[string]bool)
lock := ""
for _, line := range strings.Split(string(config), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, value, ok := strings.Cut(line, ":")
if !ok || key == "" || strings.TrimSpace(key) != key || fields[key] {
return "", false
}
for _, c := range key {
if !(c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || c == '_') {
return "", false
}
}
fields[key] = true
if key == "lock" {
lock = strings.TrimSpace(value)
if lock == "" {
return "", false
}
}
}
return lock, len(fields) > 0
}
+206
View File
@@ -0,0 +1,206 @@
package hostagent
import (
"context"
"errors"
"os/exec"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
)
func TestGuestExecConfigLockDictionary(t *testing.T) {
for _, tc := range []struct {
name, config, lock string
valid bool
}{
{"unlocked", "cores: 2\nname: home-assistant\n", "", true},
{"unicode name", "name: café\nmemory: 8192\n", "", true},
{"backup", "name: vm\nlock: backup\n", "backup", true},
{"other operation", "lock: migrate\n", "migrate", true},
{"comment", "# lock: backup\nname: vm\n", "", true},
{"erased duplicate", "lock: backup\nlock:\n", "", false},
{"duplicate unlocked", "cores: 2\ncores: 2\n", "", false},
{"case conflict", "lock: backup\nLOCK:\n", "", false},
{"empty lock", "name: vm\nlock:\n", "", false},
{"snapshot", "name: vm\n[snapshot]\nlock: backup\n", "", false},
{"empty", "", "", false},
{"only comments", "# no config\n", "", false},
{"malformed", "lock backup\n", "", false},
{"command error", "ERROR: VM unavailable\n", "", false},
{"bad key", "name: vm\n lock : backup\n", "", false},
{"non utf8", "name: \xff\n", "", false},
{"overflow", "name: " + strings.Repeat("x", guestExecConfigLimit), "", false},
} {
t.Run(tc.name, func(t *testing.T) {
lock, valid := guestExecConfigLock([]byte(tc.config))
if valid != tc.valid || lock != tc.lock {
t.Fatalf("lock=%q valid=%v", lock, valid)
}
})
}
}
func TestGuestExecCommandPreflightAndPostflight(t *testing.T) {
for _, tc := range []struct {
name string
configs []string
failRead int
wantCalls int
wantSuccess bool
wantReason string
}{
{"healthy", []string{"name: vm\n", "name: vm\n"}, -1, 1, true, ""},
{"backup before", []string{"lock: backup\n"}, -1, 0, false, agentexec.GuestExecVMLocked},
{"migration before", []string{"lock: migrate\n"}, -1, 0, false, agentexec.GuestExecVMLocked},
{"unverified owner", []string{""}, 0, 0, false, agentexec.GuestExecLockUnverified},
{"ambiguous lock", []string{"lock: backup\nlock:\n"}, -1, 0, false, agentexec.GuestExecLockUnverified},
{"backup in flight", []string{"name: vm\n", "lock: backup\n"}, -1, 1, false, agentexec.GuestExecVMLocked},
{"owner lost in flight", []string{"name: vm\n", ""}, 1, 1, false, agentexec.GuestExecLockUnverified},
} {
t.Run(tc.name, func(t *testing.T) {
var reads, calls atomic.Int32
g := newGuestExecGuard(func(_ context.Context, vmid string) ([]byte, error) {
if vmid != "105" {
t.Errorf("config read escaped canonical identity: %s", vmid)
}
i := int(reads.Add(1)) - 1
if i == tc.failRead {
return nil, errors.New("private config/provider text")
}
if i >= len(tc.configs) {
t.Errorf("unexpected config retry %d", i)
return nil, errors.New("retry")
}
return []byte(tc.configs[i]), nil
})
c := &CommandClient{guestExecAdmission: g}
oldExec := execCommandContext
execCommandContext = func(ctx context.Context, name string, args ...string) *exec.Cmd {
calls.Add(1)
if name != localGuestExecQM || !reflect.DeepEqual(args, []string{"guest", "exec", "105", "--", "sh", "-c", "echo original"}) {
t.Errorf("changed command/identity: %s %#v", name, args)
}
return exec.CommandContext(ctx, "sh", "-c", "printf 'original-result'")
}
t.Cleanup(func() { execCommandContext = oldExec })
payload := testApprovedCommandPayload(t, c, executeCommandPayload{RequestID: "scan", Command: "echo original", TargetType: " VM ", TargetID: "000105", Trusted: true})
got := c.executeCommand(context.Background(), payload)
if got.Success != tc.wantSuccess || int(calls.Load()) != tc.wantCalls {
t.Fatalf("result=%#v calls=%d", got, calls.Load())
}
if tc.wantReason != "" && got.Error != agentexec.GuestExecDeferred(tc.wantReason).Error() {
t.Fatalf("error=%q", got.Error)
}
if tc.wantCalls == 0 && (got.Stdout != "" || got.Stderr != "") {
t.Fatal("preflight published guest output")
}
if tc.wantCalls == 1 && (got.Stdout != "original-result" || got.ExitCode != 0) {
t.Fatalf("lost actual process completion: %#v", got)
}
if strings.Contains(got.Error, "private") {
t.Fatal("provider/config data leaked")
}
// A lock clearing after a preflight rejection is not an automatic
// retry. A postflight unknown keeps even a NEW request in cooldown.
if tc.wantCalls == 1 && !got.Success {
before := reads.Load()
next := c.executeCommand(context.Background(), payload)
if next.Success || next.Error != agentexec.GuestExecDeferred(agentexec.GuestExecCooldown).Error() || reads.Load() != before || calls.Load() != 1 {
t.Fatalf("uncertainty caused replay: %#v", next)
}
}
})
}
}
func TestGuestExecSerializationCancellationCapacityAndExplicitResumption(t *testing.T) {
now := time.Now()
g := newGuestExecGuard(func(context.Context, string) ([]byte, error) { return []byte("name: vm\n"), nil })
g.now = func() time.Time { return now }
release, err := g.acquire(context.Background(), "105")
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
if _, err := g.acquire(ctx, "105"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("same-target waiter: %v", err)
}
other, err := g.acquire(context.Background(), "106")
if err != nil {
t.Fatal(err)
}
other(false)
release(true)
if _, err := g.acquire(context.Background(), "105"); err == nil || err.Error() != agentexec.GuestExecDeferred(agentexec.GuestExecCooldown).Error() {
t.Fatalf("no uncertainty fence: %v", err)
}
g.mu.Lock()
now = now.Add(guestExecCooldown)
g.mu.Unlock()
resumed, err := g.acquire(context.Background(), "105")
if err != nil {
t.Fatal(err)
}
resumed(false)
g.mu.Lock()
if len(g.entries) != 0 {
t.Fatal("completed entries retained")
}
for i := 0; i < maxGuestExecGuardEntries; i++ {
g.entries["fixture-"+strings.Repeat("x", i%32)+time.Duration(i).String()] = guestExecEntry{until: now.Add(time.Minute)}
}
g.mu.Unlock()
if _, err := g.acquire(context.Background(), "200"); err == nil || err.Error() != agentexec.GuestExecDeferred(agentexec.GuestExecCapacity).Error() {
t.Fatalf("unbounded admission: %v", err)
}
g.mu.Lock()
now = now.Add(time.Minute)
g.mu.Unlock()
cleaned, err := g.acquire(context.Background(), "200")
if err != nil {
t.Fatal(err)
}
cleaned(false)
}
func TestGuestExecCanceledHandoffNeverRetriesAndRechecksOnExplicitResumption(t *testing.T) {
var reads, calls atomic.Int32
g := newGuestExecGuard(func(context.Context, string) ([]byte, error) { reads.Add(1); return []byte("name: vm\n"), nil })
now := time.Now()
g.now = func() time.Time { return now }
c := &CommandClient{guestExecAdmission: g}
oldExec := execCommandContext
execCommandContext = func(ctx context.Context, _ string, _ ...string) *exec.Cmd {
calls.Add(1)
return exec.CommandContext(ctx, "sh", "-c", "sleep 5")
}
t.Cleanup(func() { execCommandContext = oldExec })
payload := testApprovedCommandPayload(t, c, executeCommandPayload{Command: "echo original", TargetType: "vm", TargetID: "105", Trusted: true})
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
got := c.executeCommand(ctx, payload)
if got.Success || got.Error != agentexec.GuestExecDeferred(agentexec.GuestExecCompletionUnknown).Error() {
t.Fatalf("canceled result=%#v", got)
}
next := c.executeCommand(context.Background(), payload)
if next.Success || calls.Load() != 1 || reads.Load() != 1 {
t.Fatalf("canceled handoff replayed: %#v calls=%d reads=%d", next, calls.Load(), reads.Load())
}
g.mu.Lock()
now = now.Add(guestExecCooldown)
g.mu.Unlock()
execCommandContext = func(ctx context.Context, _ string, _ ...string) *exec.Cmd {
calls.Add(1)
return exec.CommandContext(ctx, "sh", "-c", "printf resumed")
}
got = c.executeCommand(context.Background(), payload)
if !got.Success || got.Stdout != "resumed" || reads.Load() != 3 || calls.Load() != 2 {
t.Fatalf("explicit resumption failed: %#v", got)
}
}
+152 -87
View File
@@ -7,6 +7,7 @@ import (
"time"
"github.com/google/uuid"
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rs/zerolog/log"
)
@@ -191,7 +192,14 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
}
// Find the agent for this host
vmTarget := s.getTargetType(req.ResourceType) == "vm"
agentID := s.findAgentForTarget(requestTargetID, req.Hostname)
if vmTarget {
agentID = s.findAgentForVMTarget(requestTargetID)
if agentID == "" {
return nil, agentexec.GuestExecDeferred(agentexec.GuestExecTargetUnverified)
}
}
if agentID == "" {
scanLog.Warn().
Str("action", "scan_precondition_failed").
@@ -230,109 +238,135 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
var wg sync.WaitGroup
var mu sync.Mutex
for _, cmd := range commands {
wg.Add(1)
go func(cmd DiscoveryCommand) {
defer wg.Done()
var safetyError error
runCommand := func(cmd DiscoveryCommand) {
select {
case semaphore <- struct{}{}:
defer func() { <-semaphore }()
case <-ctx.Done():
select {
case semaphore <- struct{}{}:
defer func() { <-semaphore }()
case <-ctx.Done():
return
}
// Build the actual command to run
actualCmd := s.buildCommand(req.ResourceType, req.ResourceID, cmd.Command)
// Get the target ID for the agent
targetID := s.getTargetID(req.ResourceType, req.ResourceID)
// Only validate TargetID when it will be interpolated into shell commands
// by the agent (container/vm types). Agent/docker types don't use TargetID
// in command wrapping, so they can have any format (including colons for IPv6).
targetType := s.getTargetType(req.ResourceType)
if targetType == "container" || targetType == "vm" {
if err := ValidateResourceID(targetID); err != nil {
mu.Lock()
result.Errors[cmd.Name] = fmt.Sprintf("invalid target ID: %v", err)
mu.Unlock()
return
}
}
// Build the actual command to run
actualCmd := s.buildCommand(req.ResourceType, req.ResourceID, cmd.Command)
// Execute the command
cmdCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
// Get the target ID for the agent
targetID := s.getTargetID(req.ResourceType, req.ResourceID)
cmdResult, err := s.executor.ExecuteCommand(cmdCtx, agentID, ExecuteCommandPayload{
RequestID: uuid.New().String(),
Command: actualCmd,
TargetType: s.getTargetType(req.ResourceType),
TargetID: targetID,
Timeout: cmd.Timeout,
})
// Only validate TargetID when it will be interpolated into shell commands
// by the agent (container/vm types). Agent/docker types don't use TargetID
// in command wrapping, so they can have any format (including colons for IPv6).
targetType := s.getTargetType(req.ResourceType)
if targetType == "container" || targetType == "vm" {
if err := ValidateResourceID(targetID); err != nil {
mu.Lock()
result.Errors[cmd.Name] = fmt.Sprintf("invalid target ID: %v", err)
mu.Unlock()
return
}
mu.Lock()
defer mu.Unlock()
// A safety deferral is never an optional probe failure or usable
// partial evidence. Stop this VM scan, including nested Docker.
if vmTarget {
message := ""
if err != nil {
message = err.Error()
} else if cmdResult != nil {
message = cmdResult.Error
}
if agentexec.IsGuestExecDeferred(message) {
safetyError = fmt.Errorf("%s", message)
result.Errors[cmd.Name] = message
return
}
}
if err != nil {
if !cmd.Optional {
result.Errors[cmd.Name] = err.Error()
}
scanLog.Debug().
Str("action", "command_execute_failed").
Err(err).
Str("command", cmd.Name).
Bool("optional", cmd.Optional).
Msg("Command failed during discovery")
return
}
if cmdResult != nil {
output := cmdResult.Stdout
if cmdResult.Stderr != "" && output != "" {
output += "\n--- stderr ---\n" + cmdResult.Stderr
} else if cmdResult.Stderr != "" {
output = cmdResult.Stderr
}
// Execute the command
cmdCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
if output != "" {
result.CommandOutputs[cmd.Name] = output
}
cmdResult, err := s.executor.ExecuteCommand(cmdCtx, agentID, ExecuteCommandPayload{
RequestID: uuid.New().String(),
Command: actualCmd,
TargetType: s.getTargetType(req.ResourceType),
TargetID: targetID,
Timeout: cmd.Timeout,
})
if !cmdResult.Success && cmdResult.Error != "" && !cmd.Optional {
result.Errors[cmd.Name] = cmdResult.Error
}
mu.Lock()
defer mu.Unlock()
if err != nil {
if !cmdResult.Success {
event := scanLog.Debug()
if !cmd.Optional {
result.Errors[cmd.Name] = err.Error()
event = scanLog.Warn()
}
scanLog.Debug().
Str("action", "command_execute_failed").
Err(err).
event.
Str("action", "command_result_failed").
Str("command", cmd.Name).
Bool("optional", cmd.Optional).
Msg("Command failed during discovery")
return
Int("exit_code", cmdResult.ExitCode).
Str("request_id", cmdResult.RequestID).
Str("command_error", cmdResult.Error).
Msg("Deep scan command reported failure")
}
}
if cmdResult != nil {
output := cmdResult.Stdout
if cmdResult.Stderr != "" && output != "" {
output += "\n--- stderr ---\n" + cmdResult.Stderr
} else if cmdResult.Stderr != "" {
output = cmdResult.Stderr
}
if output != "" {
result.CommandOutputs[cmd.Name] = output
}
if !cmdResult.Success && cmdResult.Error != "" && !cmd.Optional {
result.Errors[cmd.Name] = cmdResult.Error
}
if !cmdResult.Success {
event := scanLog.Debug()
if !cmd.Optional {
event = scanLog.Warn()
}
event.
Str("action", "command_result_failed").
Str("command", cmd.Name).
Bool("optional", cmd.Optional).
Int("exit_code", cmdResult.ExitCode).
Str("request_id", cmdResult.RequestID).
Str("command_error", cmdResult.Error).
Msg("Deep scan command reported failure")
}
// Update progress and broadcast
s.mu.Lock()
if prog, ok := s.progress[resourceID]; ok {
prog.CompletedSteps++
prog.CurrentCommand = cmd.Name
progressCopy := *prog
s.mu.Unlock()
s.notifyProgress(&progressCopy)
} else {
s.mu.Unlock()
}
}
for _, cmd := range commands {
if vmTarget {
// QGA has a serial command channel; parallel Discovery probes are
// not useful and can queue behind a backup or uncertain command.
runCommand(cmd)
if safetyError != nil {
break
}
// Update progress and broadcast
s.mu.Lock()
if prog, ok := s.progress[resourceID]; ok {
prog.CompletedSteps++
prog.CurrentCommand = cmd.Name
progressCopy := *prog
s.mu.Unlock()
s.notifyProgress(&progressCopy)
} else {
s.mu.Unlock()
}
}(cmd)
} else {
wg.Add(1)
go func(cmd DiscoveryCommand) { defer wg.Done(); runCommand(cmd) }(cmd)
}
}
wg.Wait()
@@ -349,6 +383,12 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
ElapsedMs: result.CompletedAt.Sub(startTime).Milliseconds(),
PercentComplete: 100,
}
if safetyError != nil {
completionProgress.CurrentStep = "Guest execution paused"
completionProgress.Error = safetyError.Error()
completionProgress.CompletedSteps = len(result.CommandOutputs)
completionProgress.PercentComplete = float64(completionProgress.CompletedSteps) / float64(len(commands)) * 100
}
s.notifyProgress(&completionProgress)
scanLog.Info().
@@ -358,7 +398,7 @@ func (s *DeepScanner) Scan(ctx context.Context, req DiscoveryRequest) (*ScanResu
Dur("duration", result.CompletedAt.Sub(result.StartedAt)).
Msg("Deep scan completed")
return result, nil
return result, safetyError
}
// buildCommand wraps the command appropriately for the resource type.
@@ -479,6 +519,31 @@ func (s *DeepScanner) findAgentForTarget(targetID, hostname string) string {
return ""
}
// VM execution never falls back to an unrelated sole agent or a guest-name
// hint. A short hostname alias is admitted only when it uniquely names a node.
func (s *DeepScanner) findAgentForVMTarget(targetID string) string {
targetID = strings.TrimSpace(targetID)
if targetID == "" {
return ""
}
agents := s.executor.GetConnectedAgents()
for _, agent := range agents {
if agent.AgentID == targetID {
return agent.AgentID
}
}
matched := ""
for _, agent := range agents {
if unifiedresources.HostnamesEquivalent(agent.Hostname, targetID) {
if matched != "" {
return ""
}
matched = agent.AgentID
}
}
return matched
}
// GetProgress returns a copy of the current progress of a scan.
// Returns nil if no scan is in progress for the resource.
// A copy is returned to avoid data races with the scan goroutine.
@@ -0,0 +1,102 @@
package servicediscovery
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
)
type deferringGuestExecutor struct {
*stubExecutor
calls, deferAt int
}
func (e *deferringGuestExecutor) ExecuteCommand(ctx context.Context, agentID string, p ExecuteCommandPayload) (*CommandResultPayload, error) {
e.calls++
if e.calls == e.deferAt {
return &CommandResultPayload{RequestID: p.RequestID, Success: false, ExitCode: 0, Stdout: "in-flight output is not fresh evidence", Error: agentexec.GuestExecDeferred(agentexec.GuestExecVMLocked).Error()}, nil
}
return &CommandResultPayload{RequestID: p.RequestID, Success: true, Stdout: "earlier observation"}, nil
}
func testDiscoveryGuestSafetyStopsAndPreservesSavedEvidence(t *testing.T) {
for _, rt := range []ResourceType{ResourceTypeVM, ResourceTypeDockerVM} {
for _, deferAt := range []int{1, 2} {
t.Run(string(rt)+"/"+time.Duration(deferAt).String(), func(t *testing.T) {
id := "105"
if rt == ResourceTypeDockerVM {
id = "105:app"
}
e := &deferringGuestExecutor{stubExecutor: &stubExecutor{agents: []ConnectedAgent{{AgentID: "node", Hostname: "node"}}}, deferAt: deferAt}
scanner := NewDeepScanner(e)
var terminal *DiscoveryProgress
scanner.SetProgressCallback(func(p *DiscoveryProgress) { copy := *p; terminal = &copy })
req := DiscoveryRequest{ResourceType: rt, ResourceID: id, TargetID: "node", Force: true}
got, err := scanner.Scan(context.Background(), req)
if err == nil || !agentexec.IsGuestExecDeferred(err.Error()) || got == nil || e.calls != deferAt || len(got.CommandOutputs) != deferAt-1 {
t.Fatalf("scan continued: %+v %v calls=%d", got, err, e.calls)
}
if terminal == nil || terminal.Error == "" || terminal.PercentComplete == 100 || scanner.IsScanning(MakeResourceID(rt, "node", id)) {
t.Fatalf("false completion/progress: %+v", terminal)
}
for _, out := range got.CommandOutputs {
if strings.Contains(out, "in-flight") {
t.Fatal("unverified output became fresh evidence")
}
}
store, err := NewStore(t.TempDir())
if err != nil {
t.Fatal(err)
}
store.crypto = nil
saved := &ResourceDiscovery{ID: MakeResourceID(rt, "node", id), ResourceType: rt, ResourceID: id, TargetID: "node", ServiceName: "saved observation", UserNotes: "keep", RawCommandOutput: map[string]string{"old": "retain"}, UpdatedAt: time.Now().Add(-time.Hour), DiscoveredAt: time.Now().Add(-time.Hour)}
if err := store.Save(saved); err != nil {
t.Fatal(err)
}
before, _ := store.Get(saved.ID)
beforeJSON, _ := json.Marshal(before)
cfg := DefaultConfig()
cfg.CommandScanning = true
service := NewService(store, scanner, cfg)
analyzer := &stubAnalyzer{response: `{"service_type":"nginx","confidence":0.9}`}
service.SetAIAnalyzer(analyzer)
e.calls = 0
if fresh, err := service.DiscoverResource(context.Background(), req); fresh != nil || err == nil || !agentexec.IsGuestExecDeferred(err.Error()) {
t.Fatalf("manual deferral became success: %+v %v", fresh, err)
}
after, _ := store.Get(saved.ID)
afterJSON, _ := json.Marshal(after)
if string(beforeJSON) != string(afterJSON) || analyzer.calls != 0 || e.calls != deferAt {
t.Fatal("manual pause replaced saved evidence, analysed partial output, or retried")
}
e.calls = 0
if got := service.enhanceWithDeepScan(context.Background(), saved, DockerHost{}); got != saved || analyzer.calls != 0 || e.calls != deferAt {
t.Fatal("automatic deferral fabricated fresh evidence")
}
})
}
}
}
func TestVMDiscoveryRequiresUniqueOwningNode(t *testing.T) {
for _, tc := range []struct {
name, target, hint string
agents []ConnectedAgent
}{
{"sole unrelated agent", "node", "other", []ConnectedAgent{{AgentID: "other", Hostname: "other"}}},
{"ambiguous short hostname", "node", "", []ConnectedAgent{{AgentID: "a", Hostname: "node.one"}, {AgentID: "b", Hostname: "node.two"}}},
{"empty target", "", "node", []ConnectedAgent{{AgentID: "a", Hostname: "node"}}},
} {
t.Run(tc.name, func(t *testing.T) {
e := &deferringGuestExecutor{stubExecutor: &stubExecutor{agents: tc.agents}, deferAt: 1}
scanner := NewDeepScanner(e)
if got, err := scanner.Scan(context.Background(), DiscoveryRequest{ResourceType: ResourceTypeVM, ResourceID: "105", TargetID: tc.target, Hostname: tc.hint}); got != nil || err == nil || !agentexec.IsGuestExecDeferred(err.Error()) || e.calls != 0 {
t.Fatalf("unverified node dispatched: %+v %v calls=%d", got, err, e.calls)
}
})
}
}
+7
View File
@@ -15,6 +15,7 @@ import (
"sync"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/jsonresponse"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rs/zerolog/log"
@@ -1883,6 +1884,12 @@ func (s *Service) DiscoverResource(ctx context.Context, req DiscoveryRequest) (*
var scanError error
if s.scanner != nil && s.IsCommandScanningEnabled() {
scanResult, scanError = s.scanner.Scan(ctx, req)
if scanError != nil && agentexec.IsGuestExecDeferred(scanError.Error()) {
// Keep saved observations/notes and the original in-flight outcome;
// metadata fallback must not turn a paused live scan into success.
inProg.err = scanError
return nil, inProg.err
}
if scanError != nil {
log.Warn().
Err(scanError).
@@ -2760,3 +2760,7 @@ func TestService_BackfillPreservesConcurrentManualRepair(t *testing.T) {
}
}
}
func TestDiscoveryGuestSafetyStopsAndPreservesSavedEvidence(t *testing.T) {
testDiscoveryGuestSafetyStopsAndPreservesSavedEvidence(t)
}