Commit Graph

225 Commits

Author SHA1 Message Date
pulse-triage[bot] 14d3dc3b6c Fix MD RAID spare counts without hiding member failures
Retain configured RAID members separately from source-native totals through collection, reports and canonical read views. Correct the healthy legacy mdadm tuple from #2369 without subtracting spares from mdstat requirements or suppressing real deficits, failures and recovery warnings. Preserve observed zero-active counts through fallback, and verify collector, wire, ingestion, health and canonical activation/resolution boundaries.

Change-source: pulse-maintainer
2026-10-01 19:56:47 +01:00
pulse-triage[bot] 2fd114b3ac Keep Proxmox bootstrap credentials out of copied commands
Separate PVE/PBS setup and telemetry credentials from shell source and download URLs. Use silent root/sudo input, private-file handoff, complete downloads and the agent preflight; preserve scope, TLS defaults, single-line paste and coherent rolling-upgrade metadata. Reveal tokens through the existing dialog and discard late issuance after close. Pin executable shell, history, TLS/registration and browser contracts without using real credentials.

Change-source: pulse-maintainer
2026-10-01 05:23:08 +01:00
pulse-triage[bot] 60f455fd2a Read identified ARM CPU temperatures without lm-sensors
Fall back to bounded, type-checked thermal-zone and hwmon sysfs readings when lm-sensors is absent or empty. Classify Armada thermal JSON as a CPU reading, preserve nonempty sensor output, and cover units, invalid values, and the agent mapping.

Change-source: pulse-maintainer
2026-09-28 20:26:28 +01:00
courtmanr@gmail.com 9e2ee3d535 Fail typed dispatches fast when their agent session drops
The agentexec read loop removed a dead session from the agent map but never
closed its done channel, so any dispatch already sent on that session kept
waiting for a result that could never arrive and only returned at its
operation timeout. That's what made the Proxmox cancellation replay test
flaky on the shared CI runner. The restarted action runner dropped its first
session right after registering, the replay had already been sent there,
and the server waited the full 30s while the runner sat reconnected on a
new session.

The read loop now signals the session done on exit, so in-flight typed
dispatches and commands report the disconnect straight away. The
integration test sends the same request again on the runner's next
session when that happens, which is the same durable replay it is there
to prove, still with exactly one mutation. It also keeps the runner's log
and prints it on failure so the cause of any future drop is visible.
2026-09-28 10:31:58 +01:00
Jamin W. Collins 410134450d feat(agent): allow disabling Proxmox peer sensor collection 2026-09-23 08:18:44 -06:00
pulse-triage[bot] 92d6eae77f fix(hostagent): recover Windows host info when MachineGuid is braced
The Windows unified agent exits at startup when HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid is a braced GUID. gopsutil v4.26.6 reads the value through a fixed 74-byte buffer and rejects any length other than 36, so a braced 38-character GUID fails the whole InfoWithContext call and newAgent aborts with 'fetch host info'. Recover the host info on Windows when the host-ID read is the failing field: read MachineGuid directly, strip braces and fill the other fields from gopsutil accessors, while keeping any non-host-ID failure fatal. Canonicalise Windows machine IDs by removing braces so both forms map to one stable agent identity. Includes agent-lifecycle contract section and focused proof in the registry-accepted agent_new_test.go. Windows recovery is cross-compiled with GOOS=windows; runtime execution on Windows is not qualified in this environment.

Change-source: pulse-maintainer
2026-09-20 08:29:15 +01:00
pulse-triage[bot] c8cead7d5a fix(agents): preserve explicit disk includes at ingestion
Carry the operator include override in disk reports so server filtering does not discard selected tmpfs mounts again. Keep automatic filtering for unmarked reports and agent exclusion precedence. Cover both ingest paths, collection, forwarding and wire compatibility for #1875.

Change-source: pulse-maintainer
2026-09-09 11:35:51 +01:00
pulse-triage[bot] 3334cccfd9 fix(unraid): suppress parity warning for explicit pool-only arrays
Issue #1895 reports parity alerts when mdNumDisks=0 on a pool-only Unraid system. Array service state alone does not establish that a parity array exists.

Preserve the optional disk count from collection through canonical runtime conversion and suppress only the no-parity warning for an explicit zero. Missing or malformed counts retain legacy behaviour, and disk failure reasons remain active.

Validated focused Unraid tests in hostagent, storagehealth, monitoring, unifiedresources and alerts, including JSON zero preservation and canonical round trip. The new pool-only regression fails against the previous warning condition. Both agent and server need this change; no release or reporter retest is claimed.

Change-source: pulse-maintainer
2026-09-04 23:41:22 +01:00
pulse-triage[bot] 0f972f42f2 Isolate host and Docker CPU sampling baselines (#1894)
Near-synchronous host and Docker reports consumed a shared CPU baseline, measuring collection bursts rather than each module's reporting interval. Retain a collector per host collector and a separate Docker module collector while preserving the package-level convenience API and disk filters.

Add an interleaved-counter regression covering both collection entry points. It fails when routed through the shared baseline and passes with isolated state. All hostmetrics, hostagent and dockeragent tests pass, as do the focused CPU regression tests under the race detector.

Change-source: pulse-maintainer
2026-09-04 22:29:32 +01:00
pulse-triage[bot] ace72f8442 Ignore Unraid auto filesystem on empty slots
Treat Unraid's fsType=auto value as a placeholder rather than disk assignment evidence at both agent collection and server ingestion boundaries. Preserve real assigned and explicit missing members.

Change-source: pulse-maintainer
Contract-Neutral: Unraid fsType=auto placeholder normalization fixes false missing-slot alerts without changing wire or subsystem contracts
(cherry picked from commit fd843da7dfc51bba29bb5ddcbcdb4480d0e67079)
(cherry picked from commit 7a6456969d)
2026-09-02 05:40:46 +01:00
rcourtman b1044cd8a4 Let replayed request ids wait for the in-flight handler instead of dropping
Since 60d0651a88 every typed request registers a per-connection cancellable
slot that its handler goroutine releases in a deferred cleanup after sending
its result. The server replays a request id when it wants the durable receipt
again, and that replay can reach the reader before the previous handler's
deferred release runs. launchCancellableRequest treated that as a duplicate
and dropped it, so the server waited out the operation's full timeout for a
result the agent already held. The Linux x64 native-verification leg failed
this way on 12 of the last 25 main runs, always on a "replay 1" dispatch of
host update, storage cleanup, or Docker lifecycle.

Give each slot a done channel that closes on release. A replay whose id is
still registered on the same connection now waits for that release and then
runs, answering from the durable receipt. Invalid ids and over-capacity
requests are still dropped. A unit test pins the wait-then-run behaviour and
the agent-lifecycle contract records the replay rule.
2026-09-01 23:22:55 +01:00
rcourtman e1e4c3e700 Wait for the server to observe runner disconnect before replay reconnect
The cancellation replay test stopped the first action runner and started a
second one as soon as the client goroutine exited. The server observes the
socket close on its own reader, so under GOMAXPROCS=1 with the race detector
the second startRunner saw the stale session as still connected and the
replay was dispatched to the dead socket, timing out after 30s on the
sharded release preflight while passing on multi-core hosts. Wait until the
server reports the agent disconnected before reconnecting, as the agentexec
server tests already do.
2026-09-01 22:51:56 +01:00
rcourtman 60d0651a88 Cancel abandoned typed agent operations 2026-09-01 11:21:26 +01:00
rcourtman 4e22a49f02 Add native PVE action qualification harness 2026-09-01 10:43:35 +01:00
rcourtman 01ff388c7e Preserve native Proxmox guest handoff 2026-09-01 09:31:40 +01:00
rcourtman 2ad094c927 Deflake typed action containment tests
The group-writable executable check wrote its fixture through os.WriteFile,
which filters the mode through the process umask, so under the runner's 022
umask the group-write bit never reached disk and validateTrustedExecutable
correctly trusted the file. Chmod the fixture to the asserted mode.

The Proxmox guest executor tests stamped their after observations one second
past the test-start clock, but actionStartedAt is stamped inside
ExecuteAction after handler setup, so a loaded runner overran the margin and
the independent observation was discarded as pre-action. Widen the
observation offset to a minute.

Both failures broke build-and-test on main (run 33454838531). Reproduced the
hostagent failure on Linux under umask 022 and verified both packages green
after the fix.
2026-09-01 06:15:46 +01:00
rcourtman f9ae289bd5 Contain typed action subprocesses with systemd 2026-09-01 01:26:39 +01:00
rcourtman f967857928 Harden secure agent recovery transports 2026-08-31 23:12:00 +01:00
rcourtman cf021bc9ce Surface typed helper container degradation 2026-08-31 15:53:50 +01:00
rcourtman d19398d82a Surface typed helper degradation in Agent Doctor 2026-08-31 13:06:28 +01:00
Richard Courtman 08f7c5f0d5 Harden secure agent runtime boundaries 2026-08-31 00:06:24 +01:00
Richard Courtman 770733fc92 Make action runner rotation activation-safe 2026-08-30 18:00:33 +01:00
Pulse Test c0b190bfa1 Add application response contracts to availability checks 2026-08-30 17:38:59 +01:00
Pulse Test 712f1a3bfe Add availability history and fleet view 2026-08-30 15:38:34 +01:00
pulse-triage[bot] 95a7191ca9 Enforce native private state permissions 2026-08-30 03:31:00 +01:00
pulse-triage[bot] 184b3cbfa6 Restore Windows native agent verification 2026-08-30 02:05:30 +01:00
Pulse Test d06ffc233d Harden secure agent runtime transitions 2026-08-30 01:41:57 +01:00
Pulse Test 5b2e377843 Expose secure agent runtime posture 2026-08-30 00:38:44 +01:00
Pulse Test d607d5cf46 Separate agent remediation runtime 2026-08-29 23:48:28 +01:00
Pulse Test 6d4ee48000 Add typed agent privilege helper 2026-08-29 22:51:58 +01:00
Pulse Test a966264bb1 Contain agent command authority 2026-08-29 22:12:41 +01:00
pulse-triage[bot] 5570ac1e2e fix(agent): silence unchanged remote config refreshes 2026-08-28 21:56:26 +01:00
pulse-triage[bot] 9e820355b6 Fix Unraid sentinel empty slot reporting 2026-08-28 14:20:43 +01:00
pulse-triage[bot] cb27b8eb24 Fix Unraid empty slot health reporting 2026-08-28 01:23:12 +01:00
rcourtman a9076de15a fix(patrol): require Docker health recovery 2026-08-27 20:43:54 +01:00
rcourtman 6163a2a564 feat(patrol): verify Docker actions independently 2026-08-27 20:14:38 +01:00
pulse-triage[bot] 8fde82b8a2 Keep LXC filesystem paths native-proof
Join Proxmox config paths with Linux semantics across native test platforms, add the regression proof, and update the canonical agent lifecycle contract for the new namespace-statfs collector.

Change-source: pulse-maintainer
2026-08-27 06:51:36 +01:00
rcourtman 8318c74687 fix(hostagent): stop pct df from starving LXC filesystem collection
pct df takes the container config lock and costs over a second per
guest, and every per-container query shared one 10-second budget with
pct list, so hosts with more than a handful of running containers only
ever reported filesystem usage for the first few (signal: killed on
the rest).

Resolve per-mount usage with statfs through /proc/<pid>/root instead:
read the config-declared mounts from /etc/pve/lxc, resolve the init
pid via lxc-info, and drop config mounts whose device identity matches
their parent directory rather than inventing the parent's numbers.
pct df remains as a per-container fallback for least-privilege
installs that cannot traverse /proc/<pid>/root, each container now
carries its own deadline inside a bounded collection window, and
exhausting that window logs how many containers were skipped instead
of failing silently at debug level.

Refs #1477
2026-08-27 06:22:10 +01:00
pulse-triage[bot] a4a26fe842 Preserve availability result delivery order
Change-source: pulse-maintainer
2026-08-26 19:32:17 +01:00
rcourtman 4bca38395a Harden custom sensor test fixture permissions 2026-08-26 19:25:56 +01:00
rcourtman 434e1448ff Restore Docker update preflight through unified agent 2026-08-23 08:35:26 +01:00
rcourtman ebb08e0de2 Make native-agent test fixtures path-portable on Windows
The dockeragent credential-store tests keyed their fake filesystem and
path-bearing env seams on POSIX literals, while dockerConfigCredentials
builds candidate config paths with filepath.Join. On Windows every lookup
therefore missed the fixture and the nine credential-expecting
TestDockerConfigCredentials_* cases failed with a nil credential; the
hostagent PULSE_PCT_PATH override test likewise asserted a POSIX path that
is not absolute on Windows. Map fixture keys and env values through
filepath.FromSlash and anchor the override on t.TempDir().

Taken from PR #1755 (ff96aed8c) so the correction lands on main instead of
a parked agent branch. The unified-agent-native Windows x64 job has been
red on every push run since these tests landed on 2026-08-13; the
intervening green runs were pull_request events exercising that PR's own
corrected fixtures.

Contract-Neutral: test fixtures only, no production runtime change.
2026-08-21 22:36:52 +01:00
rcourtman 4c7b1a2434 Fix Docker-in-LXC probe storm against slow Proxmox hosts
The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:

1. Dispatcher re-issued a probe while the previous one was still
   executing. The poll cycle's enrichment context had expired, so
   ExecuteCommand dispatched, returned the context error 50ms later,
   and the next 3s cycle sent the identical command again — unbounded
   concurrency against a host that was slow to begin with. The
   monitoring dispatcher now takes a per-guest in-flight claim before
   dispatching probe or inventory commands (completed probes release
   it; abandoned ones hold it for a 2-minute window), and both dispatch
   paths bail out under a dead context.

2. The host agent never got the July process-leak fix: 45480a5cc
   landed only on pulse/v6-release, so main-line agents killed just the
   direct shell on timeout, orphaning pct exec → lxc-attach children
   and blocking Wait on their inherited pipes (10s timeouts reported as
   300s+ durations). Port it: run each command in its own process
   group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
   treat ErrWaitDelay after a clean exit as success.

3. Server-side abandonment never reached the agent. ExecuteCommand and
   ReadFile now refuse to dispatch under an already-expired context,
   and send a best-effort cancel_command when they stop waiting; the
   agent cancels the in-flight execution (killing its process group)
   and reports "command canceled". Older agents ignore the unknown
   message type.

Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.

Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.

Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
2026-08-20 23:33:36 +01:00
rcourtman 4686efd8c8 Add a supported least-privilege agent install profile
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.

The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
2026-08-20 20:49:44 +01:00
rcourtman 2d801e20f3 test(hostagent): stop the standby tests reading the runner's own disks
TestCollectDeviceSMARTStandby and its observability twin stub smartctl's
execution but not the device-class probe, so collectDeviceSMART fell through
to the *runner's* real /sys/block/sda/queue/rotational.

smartctlArgs only sends the -n standby guard when the disk is not a confirmed
SSD (#1516), and smartctlArgsUseStandbyExitStatus gates the standby reading of
exit status 3 on that guard having been sent. On a Linux host whose own
/dev/sda is non-rotational the guard is dropped, exit 3 stops meaning standby,
and both tests fail deterministically:

    run smartctl for /dev/sda: exit status 3

They pass on macOS only because linuxNonRotationalBlockDevice returns false off
Linux, and on Linux only where /dev/sda is absent or spinning. That makes the
release-qualification suite unrunnable on an ordinary SSD-backed Linux worker.

Pin the probed disk to rotational through the package's existing stubLinuxSysfs
seam so the guard is always in play. Product behaviour is untouched; this only
stops two unit tests depending on the hardware underneath them.
2026-08-19 09:45:44 +01:00
rcourtman ff0c9ea4d8 fix(agent): warn when the server overrides a presented agent ID
Host identity continuity deliberately keeps a known machine on its
enrolled ID, so a custom --agent-id or hand-edited agent-id state file
is silently ignored for an already-known host, and the acknowledged ID
then overwrites the state file within one report cycle. That looked
like corruption from the operator's seat. Warn once per resolved
identity with both IDs and the supported path to a fresh enrollment
(remove the host in Pulse first).

Refs #1739

Contract-Neutral: diagnostic logging only in hostagent report ack path, refs #1739
2026-08-18 06:25:20 +01:00
rcourtman 48fac73ff0 Rework Patrol around outcome-driven autonomous operations 2026-08-14 12:52:51 +01:00
rcourtman e31fc37983 Gate Patrol actions on agent preflight 2026-08-14 01:12:49 +01:00
rcourtman a7646e5f86 Fix Patrol autonomy and refusal reporting 2026-08-13 23:26:08 +01:00
rcourtman a39935182d Parse smartctl 7.5 power_mode object on guarded probes
smartmontools 7.5 emits power_mode as an {ata_value, name} object whenever
the -n guard runs CHECK POWER MODE, which is every rotational-disk probe.
The parser declared the field as a string, so json.Unmarshal failed for the
whole document and healthy spinning disks degraded to the lossy text
fallback, surfacing as no usable SMART data while guard-free SSD probes
kept working. That is the exact rotational-only failure split in the
discussion 1690 debug log (SAS3224 HBA, smartctl 7.5). Decode both shapes
the field has used, key standby on the reported name, and never fail the
document over this field. Also broaden the text fallback standby match to
the EPC names (STANDBY_Y, STANDBY (OS), SLEEP) that the mode-suffixed
match missed.

Refs #1690

Contract-Neutral: smartctl 7.5 power_mode JSON parse fix, internal decoder only, no DiskSMART payload or subsystem contract delta (discussion #1690)
2026-08-13 10:25:28 +01:00