The server-side disk-to-pool matcher learned nvme-eui.<hex> zpool member
references in 4dcc18fbd, but the host agent keeps its own copy of the
matching for SMART pool annotation and never derived a key from the
nvme-eui form, nor stripped the eui. prefix smartctl reports on NVMe
WWNs. systemd's nvme by-id links can also carry a trailing _<n>
namespace suffix (nvme-MODEL_SERIAL_1), which made the serial
derivation return the namespace digit instead of the serial. Both gaps
left a pool member unmatched so the disk fell back to the generic
usage string (issue #1540).
Contract-Neutral: bugfix restoring intended disk-to-pool annotation (#1540); no contract surface changes
Two diagnosability gaps found by exercising the docker-update vertical
live with a real agent, each of which turns 'approved action cannot
execute' into a dead end with no visible cause:
- The durable operation-receipt store failing to open (e.g. unwritable
state dir) silently registered the agent with receipt version 0; the
only symptom was a server-side claim that the agent was too old. The
agent now logs the real cause and state dir loudly at startup.
- The server readiness copy asserted 'older agent version' for any
receipt-version mismatch. It now names both causes (old version or
unavailable state directory) and points at the agent logs.
(The related docker collect-cycle watchdog landed separately in
a0f75b1bb with a hung-daemon reproducer.)
Contract-Neutral: agent-side diagnosability: receipt-store failure warning + honest readiness copy; no public contract delta
v6.1.0-rc.1 retired the legacy update endpoints before a replacement
existed, so the UI's Update button failed with an internal-jargon 410
(issue #1564). This lands the replacement end to end: update_container
is a typed agentexec operation with its own strict codec, durable
receipts, and a request digest bound to the image digest the plan
observed; the unified agent bridges execution to the Docker module's
existing pull/backup/recreate/verify/rollback implementation (which now
reports rollback attempt and outcome); and the container action
executor plans, dispatches, and reconciles the operation with declared
backup/rollback compensation truth. Containers advertise an
admin-approval update capability while an image update with a stated
current digest is detected. The legacy endpoints stay retired but
return actionable copy.
Proven live against a Colima daemon: single-container update, the
issue-1564 shared-network-namespace update, and the full UI journey
(Update button, governed review, approve, run) all completed with the
namespace preserved and the backup retained.
Two root causes behind #1516's remaining reports:
- A node whose Proxmox disks/list query fails (PVE probes SMART per disk
inside that call, so dozens of disks can exceed the API window) now
falls back to the linked host agent's smartctl inventory instead of
leaving the Physical Disks view empty. Each node also gets its own
attempt window so one slow node no longer starves the rest of the
cluster, and a poll that runs out of budget saves partial results.
- The -n standby probe guard is dropped for positively confirmed
non-rotational devices in both the host agent and the node sensor
wrapper. The guard exists to avoid spinning up sleeping HDDs; an SSD
has nothing to spin up, and some SATA SSDs answer CHECK POWER MODE
with a bogus standby state that permanently hid their temperature,
attributes and history.
Refs #1516
The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
Refs #1515
The install command is generated under the Settings infrastructure
installer, not a "Settings > Agents" tab. Correct the agent log, installer
warning, and Machines tooltip to say "the Pulse UI" so the recovery step is
accurate.
Refs #1515
When an upgraded or restored Pulse server no longer recognises an agent's
API token, the report endpoint returns 401. The agent buffered and retried
that report forever with only a generic warning, and the server kept the
node green at its last known agent version because a Proxmox node stays
online via the PVE API poll even after its agent dies.
- Agent: special-case 401 on /api/agents/agent/report. Drop the report
instead of buffering it and log a throttled, actionable error pointing the
operator at the install command to mint a fresh token.
- Installer: verify_agent_server_registration now tells a rejected token
(401/403) apart from "agent has not reported yet" and prints the recovery
steps at install time instead of a vague soft warning.
- Status layer: resourceFromHost flags a stale agent (its host marked
offline by the staleness evaluator) and carries the agent's own last
report time. Coalescing keeps a node online via the PVE source but the
dead agent stays flagged, so its version is no longer presented as current.
The Machines table renders such versions as "(stale)".
Keep Proxmox setup tokens node-scoped so cluster installs do not rotate one shared Pulse API token.
Suppress command-enable config for tokens that cannot register command channels and keep reusable installer tokens out of agent:exec.
Unraid's mdResyncAction field can retain its last value (e.g. "check")
after a parity check is canceled, causing Pulse to report a stale sync
action indefinitely. The mdResync/mdResyncPos field is the authoritative
indicator: it drops to 0 when no resync is running. Gate SyncAction on
a non-zero position so the alert clears once the sync actually stops.
Refs #1485
On the #1483 reporter's Proxmox node a SATA SSD (INTEL SSDSC2BW240A4) never
appeared in Pulse even though smartctl reads it fine, because the agent's
discovery and probing both had single points of failure:
- Discovery trusted smartctl --scan-open exclusively whenever it returned at
least one target. The scan silently omits any device it fails to open at
scan time (the failure is only a #-comment in its output), so one bad open
hid a real disk while its neighbours were listed.
- A scan-provided -d type got exactly one probe attempt on Linux. A type whose
full query (-i -A -H) fails or returns nothing dropped the disk silently.
- A probe yielding no usable SMART data dropped the disk entirely instead of
reporting the identity the kernel can prove.
Fixes, all in the agent collector:
- Union discovery: the kernel block device list (/sys/block, lsblk fallback)
is the ground truth for which disks exist; smartctl --scan-open only
contributes device-type hints. Any physical disk the scan misses gets an
untyped probe target.
- Untyped retry: a typed Linux probe that errors, yields no usable data, or
fails to open (exit bit 2, distinct from -n standby,3's exit 3) retries with
smartctl auto-detection before giving up. Multiplexed controller members
(megaraid, cciss, ...) are exempt since dropping -d would probe the array.
- Identity-only fallback: when every probe fails but the disk exists in
/sys/block with nonzero capacity, report device/model/serial/size with
health UNKNOWN instead of hiding it. No SMART data is fabricated;
multiplexed array paths and zero-capacity media are excluded.
- Exclusion follow-through: --disk-exclude now also matches the canonical
post-refine name (nvme0n1), not just the scan label (nvme0).
Regression tests use smartctl/lsblk fixtures captured from PVE 9.1.9 hosts
and the issue report. Verified live on two PVE nodes: NVMe keyed by
namespace with byte-exact pvesh sizes, SATA disk reported with full identity.
Completes the #1483 fix started in bd20069c6 (namespace devpath, authoritative
sizes, merge downgrade guard). Closes#1483.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Back-port the smartctl regex half of v5 fix 0c2de2938 to v6 (the
prerelease-aware CompareVersions half is already present). smartctl text
fallback now matches 'Current Drive Temperature:' in addition to
'Current Temperature:', so OPNsense/pfSense disk temperatures are read
instead of reported as missing. Adds a regression test.
On a Proxmox node, physical disks collected by the host agent were keyed by
the NVMe controller (e.g. "nvme0 [nvme]") instead of the namespace, reported
sizeBytes 0 (or a stale filesystem-usage value), and flickered as the agent
reading intermittently replaced the authoritative Proxmox disks/list reading.
Root causes:
- smartctl --scan-open reports NVMe disks by their controller char device
(/dev/nvme0), and that scan label became the reported devPath.
- DiskSMART carried no capacity, so the server backfilled size by matching the
SMART device against host filesystem-usage entries, which never match a whole
partitioned/LVM/ZFS disk, leaving size 0.
- The unified-resource merge let the agent's controller label overwrite the
canonical Proxmox /dev/... devPath.
Fixes:
- The agent now reports the canonical block device (an NVMe controller resolves
to its namespace) and the authoritative capacity from /sys/block, with the
smartctl user_capacity / nvme_total_capacity as a cross-platform fallback.
Disks behind multiplexing controllers (megaraid, cciss, areca) keep their
disambiguating label and smartctl-reported size.
- SizeBytes flows through the agent report, host model, and adapter; the
filesystem-usage match is demoted to a legacy fallback.
- The merge keeps a canonical /dev/<device> devPath and never downgrades it to
a scan label, so an un-updated agent can no longer corrupt Proxmox data.
Refs #1483.
Discovery wraps every probe in `docker exec <container> sh -c '...'`.
The agentexec command policy lists `^docker\s+exec\s` as RequireApproval
(a sound default for user-driven docker exec) and Discovery has no path
to mint or supply an ApprovalID. Result: every probe was rejected, the
scanner returned empty CommandOutputs, and the AI fell back to
"Unknown Infrastructure Resource" at confidence 0. The Discovery sub-tab
rendered empty after a "successful" run.
Add a Trusted bool to ExecuteCommandPayload on both the server-facing
agentexec type and the agent's wire struct. When set, the approval gate
is skipped on both ends and the server does not attempt to auto-mint an
approval grant (which would fail with "approval id is required").
PolicyBlock still applies; this is not a way to run arbitrary commands.
Only the discoveryCommandAdapter sets Trusted=true. The flag is never
populated from a deserialised HTTP body or any user-driven path. Patrol
fixes, Assistant remediation, and AI tool calls continue to flow through
the governed approval-record path with a real ApprovalID.
Contracts: amend agent-lifecycle Completion Obligations and Current
State to document the lone exception to the on-agent approval rail, and
amend ai-runtime to fence the Trusted flag to the discovery adapter
only.
Separate first-class platform support from Pulse Agent host profiles and classify Unraid as an agent-backed host profile while preserving it as presentation-only platform vocabulary.
Parse the /proc/mdstat operation keyword for mdadm arrays and propagate it through host reports, models, unified resources, monitoring views, alert metadata, and AI storage summaries.
Treat recovery and reshape as rebuild signals while silencing routine check and resync maintenance, with fallback rebuild detection only when no mdstat operation is available.
Tests cover mdstat operation parsing plus recovery, check, and resync alert behavior.
Fixes#1446
rcourtman