Keep canonical disk risk, source freshness and retained history intact when
Assistant and Patrol gather evidence. Proposal acceptance validates an action
contract and must not rewrite uncertain conclusions as established root cause.
Preserve complete subscription tool batches without exposing routing envelopes
as answers. Keep wide answer tables readable and keyboard-scrollable on mobile.
Optimize retained tier reconciliation without discarding gaps or newer samples.
Record failed real-model diagnoses and outstanding autonomous qualification
separately from passing data-path and interface checks.
A reachable PBS can deny or fail its node-status endpoint while datastore collection succeeds. The resulting zero values previously resolved active CPU and memory alerts without a healthy measurement. Carry internal availability evidence from polling and skip metric evaluation for those samples, preserving independent connectivity and policy suppression. Add failing-before unit coverage and an HTTP polling lifecycle regression for dispatch and recent recovery history.
Change-source: pulse-maintainer
Issue #1895 reports parity alerts when mdNumDisks=0 on a pool-only Unraid system. Array service state alone does not establish that a parity array exists.
Preserve the optional disk count from collection through canonical runtime conversion and suppress only the no-parity warning for an explicit zero. Missing or malformed counts retain legacy behaviour, and disk failure reasons remain active.
Validated focused Unraid tests in hostagent, storagehealth, monitoring, unifiedresources and alerts, including JSON zero preservation and canonical round trip. The new pool-only regression fails against the previous warning condition. Both agent and server need this change; no release or reporter retest is claimed.
Change-source: pulse-maintainer
A PBS-to-PBS sync that fails mid-snapshot leaves a manifest-less copy of an
already-completed backup on the target datastore. That copy has the same
signature as a snapshot being written right now (no size, no
index.json.blob), so the guest showed Backup Running indefinitely until the
sync was repaired.
An in-flight snapshot only reports the guest as backing up when it is newer
than the guest's last completed backup and within the same 24h cutoff that
already ages out stale running vzdump tasks. A genuine in-progress backup is
always newer than the last completed one; a partial sync copy never is.
Refs #1815
Sequentially adding standalone Proxmox estates with the same native node name could lend the first site agent to the second through the short-hostname fallback. Both provider nodes then shared one agent identity and collapsed into one state slot. Require exact endpoint, certificate, or agent-bridged full endpoint evidence before a cross-instance standalone merge; reject and self-heal guessed links that already belong to another node.
Refs #1753
Contract-Neutral: narrows internal Proxmox node/agent identity reconciliation without changing API shapes or extension contracts
Third increment of the resource-payload-static-metadata gap, the
snapshot-side statics. Measurement first: the mock estate ships only 5
distinct non-default policy postures across 723 resources and 56
distinct templated AI-safe summary strings across the same rows
(~150KB of inline duplication per snapshot), and every canonical alias
list carries the row's own wire id verbatim.
Non-default policies and AI-safe summaries now dedupe through the same
content-addressed catalog contract as capabilities: state-level
policyCatalog and aiSafeSummaryCatalog hold the distinct blobs,
resources carry policyRef / aiSafeSummaryRef, and ingestion expands
refs back to the inline shape (default posture synthesis unchanged for
ref-less rows). The broadcast conversion returns one catalogs struct so
the next deduped field is a loop entry, and catalog changes ride the
same delta frame as the refs that need them. Alias lists stop shipping
the row's own wire id — every identity consumer unions resource.id
beside the alias spread — while the primary id stays so aliases remain
the complete live vocabulary a host is reachable by, per the
established slimming invariant.
Verified against a 1857-resource estate: zero inline policies or
summaries on the wire (743 refs resolving into 1KB + 7KB catalogs),
zero alias lists carrying the wire id, catalog-expanded postures and
summaries reaching the store byte-equivalently, and the estate
rendering clean on desktop and the 36-row phone window.
Remaining gap scope: canonicalIdentity alias/superseded history is
genuine per-resource identity data (~600KB per snapshot) with no safe
derivation shortcut, and the proxmox-facet/platformData mirror
duplication (~200KB) is deliberate canonical-merge fabric; both stay
recorded under resource-payload-static-metadata for a future owner.
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).
The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.
Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.
Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.
/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.
Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.
The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
An in-progress PBS snapshot (reproduced on VM 117 at 9% written) was
counted as a completed backup: the guest's LastBackup jumped to "now" the
moment vzdump started, and the workloads badge showed a green healthy
shield for a backup that did not exist yet. If the job aborted, that
shield had never been true.
Detect the in-flight state from what the PBS listing actually says - no
size and no index.json.blob manifest in files - and flag the snapshot
InProgress instead of complete. On the PVE side, correlate storage content
entries with live vzdump tasks: a guest archive whose ctime falls inside a
running task's window is that task's partial output, and a PBS-storage
entry without a size has no manifest yet.
In-flight artifacts now feed a new per-guest BackupInProgress flag rather
than LastBackup, so the badge age always reflects the latest COMPLETED
backup. The workloads badge gets a distinct blue "Running" state (green
would claim success, None would hide that work is underway), with the
last-completed age kept in the tooltip and accessible label. Recovery
points map to OutcomeRunning with no completion time, keeping posture
freshness and the Backups page's latest-restore pointers anchored to
completed backups; the recoverable-artifact list shows a Running chip.
A cached in-flight snapshot forces a per-poll re-fetch of its group so
completion or abort is noticed promptly instead of after the cache TTL.
Connected systems showed a PBS machine twice (API row plus host-agent row)
whenever the PBS connection was configured with an address the agent never
reports, because PBS identity was limited to the configured name and host.
The PBS poll now captures the hostname the node reports about itself
(GET /nodes, the previously unused client GetNodeName) on
models.PBSInstance.NodeName, and /api/connections includes it in the PBS
connection's host aliases, so directPlatformHostAttachment merges the agent
running on that machine into the PBS row the same way PVE composes from
API-reported node identity. Node-name fetch failure stays partial data and
never fails the poll. Reported by Johannes Strasser (rc.5).
Adversarial review of 84dba861b found three ways the new evidence paths
could still attribute a snapshot to the wrong cluster.
The submission-source learner was asymmetric. Clusters only became known
to it through snapshots that were already attributable, so a cluster with
no uniquely-attributable snapshot was invisible - and a source token both
clusters share then mapped to exactly one visible cluster and looked
decisive. The visible cluster got the other's backups while the other
guest stayed at zero. Callers now declare every connection owning a
candidate guest for a PBS instance, and the learner refuses to resolve
anything for that instance until each of them has had a snapshot
attributed to it. Observation is not scoped per PBS instance, so a
cluster seen submitting to its own PBS server still counts as visible -
the reported two-server topology keeps working.
PVE storage confirmations were treated as authorship. A pbs-type storage
listing proves the connection can SEE a snapshot, which a shared token, a
synced datastore, or an offsite copy all arrange without the connection
having made it, and a single confirmer previously outscored everything
else. Confirmations now carry the storage they came from, and only a
storage view that never lists a snapshot some other connection also lists
can attribute a colliding VMID. An overlapping view has demonstrated it
sees other clusters' snapshots, so nothing it lists attributes anything.
Where an exclusive view and the learned source mapping both speak they
must agree, otherwise the snapshot drops as it did before #1639. The
disjoint case - each cluster mounting only its own datastore - is
unchanged.
Confirmations were evicted by partial poll failures. A storage whose
content query failed contributed nothing, and the partial set overwrote
the previous one, flipping attribution between cycles. They now go
through the same per-storage preservation as storage backups.
Contract text calling the PVE listing "the only deterministic
attribution" is reworded to match the weakened semantics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v6.1.0's identity rework (eab73d245) made the VMID-only fallback refuse
to fire whenever a typed VMID exists on more than one PVE location.
Root-namespace snapshots with no matching comment then score zero for
every guest, so on setups with two clusters and overlapping VMIDs most
guests showed no backup at all - while PVE itself listed the backups
fine, because monitoring discards pbs-type storage contents entirely
whenever a direct PBS connection is configured.
Attribution is now evidence-driven instead of dropped:
Storage backup polling keeps a per-connection record of every snapshot
its own pbs-type storage listed (type, VMID, backup time) even though
the raw entries stay out of the PVE backup list. Which cluster listed a
snapshot is deterministic attribution, and it survives fully mirrored
clusters that share one datastore and token. The evidence is
monitoring-internal, cleared on instance retirement or when the storage
poll stops seeing pbs content, and never serialized into state payloads
or snapshots.
Guest backup-time sync and the recovery-point mapper additionally learn
each PBS submission source's cluster (owner token, datastore, PBS
instance - strongest first, scoped to the PBS instance) from the poll's
attributable snapshots, then resolve collision VMIDs whose snapshots
carry no evidence of their own. A source seen from several clusters is
not a discriminator, an unfamiliar component stops resolution rather
than deferring to weaker ones, and a snapshot decisively attributed to
another cluster is kept away from this one. Unattributable snapshots
still drop rather than guess.
Backup-age alert attribution no longer suffix-matches the subject ref's
connection label against guest locations. The label there is a PVE or
PBS instance name, not a PBS namespace, and loose matching could
cross-attribute clusters sharing a VMID; it now requires exact
normalized equality.
Reported in #1639 (two PVE clusters with PBS 4.0/4.1, VM 173 shown 974
days overdue despite valid verified backups).
Fixes#1639
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Host agents key their identity on the machine-derived agent ID, so MSP
template deployments that clone /etc/machine-id fold two physical
machines at different sites into one host row whose reports overwrite
each other (hostname, report IP, and interfaces flapping between
sites), silently poisoning node-agent linking.
Mirror the Docker host identity-collapse doctrine (#1584) for host
report ingest: track hostname and report-IP revisits per resolved
agent identity inside the monitoring-owned flap window, publish an
active conflict as models.Host.IdentityConflict through unified
resources, and warn on the Machines page. The report IP is tracked
alongside the hostname because template fleets often reuse hostnames
across sites (pve01 at two customers), leaving the address as the only
field that betrays the clone. A one-time hostname rename never
revisits and is not flagged; the conflict clears on its own once only
one machine keeps reporting for the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Support evidence from the MSP case showed the clusters mix without ever
sharing a corosync name and without the config-layer consolidation
firing, which rules out the same-name path and points one layer down.
Two mechanisms combine there. A newly added connection whose add-time
cluster detection failed (#437) commits its first polls' nodes with an
empty cluster name, because pollPVEInstance ran membership detection
after the node-state commit - and every aggregation guard deliberately
lets empty cluster names merge freely. And host agents key their
identity on /etc/machine-id, which cloned template deployments reuse
across sites, so two different pve01 machines collapse into one agent
row whose shared LinkedAgentID then folds the unclassified node into
the established cluster's slot, overwriting it - the reported "enacon
appeared renamed to rewo" data loss.
Weak-evidence folds across connection instances - a bare-hostname
endpoint alias or a shared linked-agent identity - now require positive
same-machine proof (matching non-empty cluster identity or matching TLS
fingerprints) whenever cluster identity is in play on either side. Two
views that are both unclassified still dedup freely, and address-based
endpoint aliases keep folding on the contradiction checks alone, so the
designed standalone-into-cluster folds survive. PVE polling now also
runs cluster membership detection before the cycle's node-state commit
and re-reads the refreshed instance config, so nodes carry their
cluster identity from the first state write whenever detection
succeeds instead of transiting aggregation unclassified.
Reported via support by an MSP running clusters enacon and rewo that
reuse pve01/pve02 node names across sites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The config layer already refuses to consolidate two same-named clusters
whose TOFU-captured TLS fingerprints contradict, and node aggregation
keeps any two same-named clusters from different connection instances
apart unconditionally. Two gaps remained one layer down. First, the
endpoint-IP agent match bypassed the contradiction guard entirely, so
two sites reusing RFC1918 addressing (the MSP support case: pve01 on
192.168.1.11 at both sites) still bound the second site's node to the
first site's host agent, attaching the wrong machine's telemetry.
Second, the unconditional split had no way to recognize the legitimate
duplicate - the same cluster added twice through different member
addresses with no config-level endpoint overlap could never fold back
into one node slot.
The aggregation layer now receives the config layer's identity
evidence: each PVE node carries the TLS certificate fingerprint of its
own named endpoint record (standalone nodes carry the instance
fingerprint; a cluster member never inherits the instance-level
fingerprint, which pins whichever member the connection URL reaches).
Same-named clusters from different instances merge only when both views
carry the same fingerprint; contradicting or unknown evidence keeps the
fail-safe split. Agent binding applies the identical doctrine: hostname
and address matches are rejected when the candidate agent's linked
nodes live in a different named cluster or carry a different
fingerprint, closing the previously unguarded endpoint-IP path.
Reported via support by an MSP whose sites reuse cluster names, node
names, and RFC1918 ranges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ceph Quincy and later (including Squid on PVE 9) dropped the monmap
mons array and the mgrmap active_name/standbys arrays from ceph status
output, replacing them with monmap.num_mons and mgrmap.num_standbys,
with quorum membership reported at the top level of the payload. Both
the host agent parser and the Proxmox API path only understood the
legacy arrays, so modern clusters showed 0 monitors and undercounted
managers.
- hostagent: read num_mons/num_standbys and top-level quorum data,
taking the largest available signal, and base the mon/mgr service
rows on the same counts
- pkg/proxmox: decode mgrmap num_standbys and top-level
quorum_names/quorum on CephStatus
- monitoring: fall back to the new fields when counting MON/MGR
daemons, and log Ceph 401/403 failures at warn level with a hint to
grant Sys.Audit on / instead of hiding them at debug
- models: prefer the larger non-zero MON/MGR counts when merging Ceph
cluster records from multiple sources
Fixes#1626, Refs discussion #1290
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A second pass targeting PARTIALLY covered functions rather than untouched
ones, so every case here is an arm the existing suites never reached.
Percentages are per-function coverage, measured before and after.
- pkg/securityutil: the SSRF guards, which is where the uncovered arms
actually matter. isCarrierGradeNATIPv4 and isLocalNetworkIP are pinned at
the first and last address of 100.64.0.0/10 and just outside both ends,
across loopback, link-local, every RFC1918 range and IPv6 unique-local.
joinURLPath, IsLocalNetworkHost, resolveOutboundIPAddrs and
cloneRestrictedTransport to 100 percent, resolvePermittedOutboundIPs to
96.8, with the transport clone asserted independent of its source.
- internal/models: eleven deep-copy helpers from as low as 25 percent to 100.
Every one asserts real independence, mutating each nested slice, map and
pointer field of the clone and checking the original is untouched, which is
the failure mode a deep-copy helper actually has.
- internal/alerts: metricClearThreshold 28.6 to 100, resourceTypeLabel and
alertspecsMetricTriggered 50 to 100, the four canonical spec-id and
tracking-key builders 66.7 to 100, inferCanonicalKindFromLegacyAlert to 100,
and the backup-snapshot and ack-identity predicates.
- internal/servicediscovery: the four fingerprint generators to 100, each
asserted for both stability and sensitivity; the three command builders and
ValidateResourceID on their exact output and each rejection reason.
- internal/storagehealth: zfsScanActive and firstNonEmpty.
cephClusterSourceRank is deliberately left at 75 percent: its default arm is
unreachable because normalizeCephClusterSource can only return the two cases
above it. That is recorded rather than faked.
No source file is modified. Adversarial review returned no rejects and flagged
nine re-hit subtests; all nine were removed and every target function
re-measured at an identical percentage, proving they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Seventeen files closing the zero-coverage functions the current source drop
left behind. Every named target was measured off 0.0 percent by a per-function
coverage delta, re-measured against current main.
- config: the five new durable Proxmox cluster-node identity helpers
(deterministic id, endpoint equality, alias lookup, id existence, lookup by
id) to 100 percent; VMware and agent-profile persistence round-trips under
t.TempDir including that AppendProfileChangeLog appends rather than
replaces; AI chat session save, load, delete, per-user scoping and age
cleanup, with explicit timestamps rather than time.Now-relative fuzz;
PVEInstance.DeepCopy asserted for nested independence.
- truenas: incidentFromPoolStatus over every pool health string,
RecordsFromSnapshot over nil, empty and populated snapshots, both
TransportStatus accessors, and the RPC handshake and auth typed errors
through errors.Is and errors.As.
- unifiedresources: the maintenance-window operator-state lifecycle on
MemoryStore including the not-found and already-cleared arms, plus the
four remaining View accessors asserted on their exact formatted output.
- api: restoreAgentExecMetadata, buildAlertConnectionSnapshotsWithRuntimeSources
and both mock series generators, asserted on shape, ordering and
determinism rather than non-emptiness.
- cmd/pulse-control-plane: the four remaining MSP and mobile proof report
printers, asserted on the concrete strings in captured stdout.
- ai: cost.EmptySummary, approval.emptyExecutionState, demo.IsDemoRuntimeIntended
and tools.findCanonicalAppContainerResourceByReferences across no-match,
first-match, later-match and ambiguous references.
- monitoring, models, alerts: trueNASAppRunning,
supplementalProviderOwnedSourcesForOrg, IOCounterPresence.Effective,
ValidAlertIntentSignal and intentTimePointer.
No source file is modified. Adversarial review returned no rejects across all
seventeen files and flagged four padding cases plus one dead table field; all
were removed and the per-function coverage re-measured as identical, proving
they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Two PVE clusters whose nodes share hostnames (pve01/pve02 on different
subnets) collapsed into one: the agent-link hostname fallback bound the
second cluster's node to the first cluster's agent despite contradicting
IP evidence, and the shared LinkedAgentID then overrode the
cluster-scoped merge key, so the fresher node overwrote the first
cluster's slot and relabeled it. The endpoint-host merge alias had the
same hole when TLS settings degrade node endpoints to bare node names.
Hostname matches now reject candidate agents whose linked nodes live in
a different named cluster or whose reported IPs exclude the node's
endpoint IP, and neither a shared agent identity nor an endpoint alias
can merge two nodes whose named clusters contradict. Empty cluster
names still merge freely so standalone views keep folding into their
cluster node.
Reported via support by an MSP running two clusters plus a standalone
that reuse pve01/pve02 across sites.
Removed-agent records kept no platform identity, so the fleet doctor
emitted an empty platform for removed rows and the Agent Doctor
uninstall handoff could never tighten to a single platform command.
Host and Docker removal now capture the agent's last-known reported
platform onto the removed record, and the fleet diagnostic resolves the
retained value through the same strict normalization as live subjects,
so /api/agents/diagnostics reports a platform for removed agents and
the handoff shipped in 4fc86ec18 scopes to one command automatically.
A retained value that does not normalize to a known platform still
yields an empty diagnostic platform rather than a guess, and removed
Kubernetes clusters retain nothing because the cluster report never
carries a platform. The field is additive and optional, so removed
lists serialized before it existed load unchanged. Monitoring contract
updated; proofs staged in agent_fleet_doctor_test.go and
monitor_host_agents_test.go.
The connections ledger derives agent rows from the unified fabric, which
includes machines whose telemetry comes from platform integrations (vSphere
ESXi hosts, TrueNAS). Agent Doctor rendered every one as a permanent
'Unknown / no structured reason' row, while agents the ledger does not carry
(Docker-only, Kubernetes-only) were silently dropped from the fleet view.
- Expose HostView.IntegrationSource() (source-set based: only SourceAgent
ingest counts, since integration providers fabricate an Agent payload) and
plumb it through models.Host to the connections ledger as the optional
integrationSource field.
- Agent Doctor skips integration-backed connections and appends
diagnostics-only agents, honoring scope, so the doctor covers exactly the
real Pulse Agent fleet.
- Update readiness agent checks no longer count integration-backed machines
as registered agents.
- Humanize doctor copy: plain-language stale message with '10m 2s'-style
durations, offline wording without enum leakage, no 'Supported target:
Unknown' cell when no target is published, host-local command banner only
when a command is actually offered, and a compact non-zero summary strip.
Contracts updated for unified-resources, monitoring, api-contracts,
agent-lifecycle, and dependent storage-recovery; verification via
views_test.go, monitor_host_agents_test.go, state_host_test.go,
contract_test.go, and the frontend connections API test.
rcourtman