A Proxmox disk that only the node's sensors report never raised a
temperature alert: CheckDiskHealth judges health and wearout only, and
unified evaluation covers TrueNAS disks, not PVE ones. Physical Disks
could read the disk Running Hot under the per-type thresholds while
alerts stayed silent, and whether a disk alerted depended on whether a
Pulse agent happened to run on the node.
The PVE disk poller now hands each disk's current reading to
CheckProxmoxDiskTemperature, which judges it by the same disk
temperature policy as agent and TrueNAS disks (DiskTempByType under the
agent Disk Temp default) under the disk's existing PVE alert reference.
A disk a still-reporting linked agent lists in its SMART report is the
agent's: CheckHost alerts on it and the PVE alert closes as moved to the
agent, so no disk alerts twice. A retained reading holds the alert,
an excluded device or a switched-off policy closes it, warnings wait the
same 300s stability window as other disk temperatures, a config save
re-judges the alert per disk type, and resource history owns its rows by
the disk's recorded identity like the health and wearout alerts.
Two Proxmox connections can each have a node with the same name, which
is common for MSPs running one `px1` per site. With one host agent
linked to site A's `px1`, `getHostAgentTemperatureByID("siteB-px1",
"px1")` returned site A's temperature. The hostname fallback took the
first agent with that hostname, even one linked to another node, and
even when the name was ambiguous enough that the agent linker had
refused to link it. The cluster sensor cache leaked the same way: SSH
readings an agent collects for its cluster siblings were keyed by bare
node name. Site B's `px1` then showed, alerted on and recorded history
for another machine's sensors.
## Change
- The poller hands the lookup the polled node
(`getHostAgentTemperatureForNode`,
`carriedTemperatureOutlivesAgentLease`), and the lookup works on that
node's **slot**: its own source ID, plus, when the read state does not
hold that ID, the one same-named node proven to be the same machine.
- The state folds the views of one machine reached through two
connections (a cluster added twice, or a multi-homed host added by each
address) into a single node kept under one view's ID, and an automatic
agent link follows it. A narrowed fallback on its own would leave those
views without a temperature permanently; that was reproduced with real
`UpdateNodesForInstance` folds.
- Same-machine proof comes from the state's own evidence: new
`models.NodeObservationsSameMachine` and
`models.HostAgentBridgesNodeViews`
(`internal/models/node_machine_identity.go`). `UpdateNodesForInstance`'s
agent-corroboration check now shares its rule with them. Two or more
candidates are ambiguous and none is taken, as the state declines an
ambiguous alias.
- An agent linked to the slot is the node's agent. The hostname fallback
only takes an agent with no node, VM or container link, and only when no
node outside the slot and no second unlinked agent has the name.
- Cluster sensor cache entries record their reporting agent. A reading
serves a node only when that agent's linked node shares one of the
slot's connections, or belongs to a connection whose configured view of
the node has a matching TLS fingerprint (the other connection of a
cluster added twice).
`LinkedNodeID` holds the Proxmox source node ID for every link source
(automatic `findLinkedProxmoxEntityWithHints` → `NodeView.SourceID()`,
manual `LinkHostAgentToNode` validating against `models.Node.ID`,
reconciliation, continuity restore), so the link match needs no
namespace normalization. A manual link stays pinned to the ID the
operator chose (the state never transfers intent to a replacement
provider identity), so a node held under the other view's ID has no
linked agent. The physical-disk SMART merge already matches by
`node.LinkedAgentID` within one instance and is unchanged.
## Tests
- `TestNodeTemperatureStaysWithinSameNamedNodesSite` (registered
monitoring proof, `monitor_additional_test.go`), through
`collectNodeTemperatureData` against registry-backed and state-folded
read states: agent linked to the other site's node; ambiguous unlinked
agents; unambiguous unlinked agent (still served); cluster added through
two connections (alternating real state folds); multi-homed host via two
standalone connections; ambiguous address alias; a namesake agent with a
contradicting fingerprint; sibling readings across both connections of
one cluster; sibling reading from the other site. The leaking cases fail
on main, and each folded-view guard was checked with an overlay negative
control.
- `TestApplyHostReportScopesClusterSensorsToReportersConnection`
(`monitor_host_agents_test.go`): real `ApplyHostReport` ingest of
cluster sensors.
- `TestNodeObservationsSameMachineMatchesStateFold` and
`TestHostAgentBridgesNodeViewsMatchesStateFold` pin the predicates
against `UpdateNodesForInstance`'s actual fold.
- Full `internal/monitoring`, `internal/models` and
`internal/unifiedresources` pass on pulse-dev.
A threshold alert stays open after its reading drops below the trigger.
It holds until the reading reaches the clear level, then must stay there
for the recovery delay. While it held, the evaluator never touched the
alert, so every surface kept showing the last breach. On minipc the
drawer said "Node temperature at 80.0°C" next to a green 72°C in the
node table, with nothing explaining why the warning was still there.
#2068 reports the same thing for a disk alert.
**What changes**
- Open threshold alerts carry a volatile `metricStatus`: the value Pulse
is comparing now (the average for rolling-average rules, with the latest
sample beside it), the phase (`breaching`, `latched` or `recovering`),
the trigger and clear levels, and recovery progress from the reducer's
monotonic run. `value`, `message` and `lastSeen` keep meaning the last
breach, and the hold branch still never notifies.
- The status is stripped from the active-state checkpoint, the recovery
mirror and the degraded recovery projection, so holding adds no durable
writes. Resolved records carry none.
- Stale-alert cleanup and the retention TTLs use the latest observation.
Previously an alert held below its trigger for more than a day was
auto-resolved as "resource no longer being monitored".
- Node, guest and Mail Gateway drawer rows lead with the reading now and
say what clears the alert, with the last breach on hover. The Proxmox
TEMP cell keeps its own open temperature alert's tone and explains it on
hover. Storage rows stop saying "Over N%" once usage is back under the
limit, and the Assistant handoff no longer labels the breach value as
current.
**Verified**
- Unit tests drive a node temperature alert through 85 → 78 → 72 → 75 →
77 → 72 → resolved, and fail against the old evaluator.
- Private mock stack with pve2 stepped 88 → 78 → 72°C at desktop and
phone widths; see the browser receipt in this PR.
Refs #2068
Two Proxmox connections can each have a node with the same name, which
is common for MSPs running one px1 per site. The node temperature
lookup fell back to the first host agent whose hostname matched the
node name, even when that agent was linked to the other site's node,
and even when the name was ambiguous enough that the agent linker had
refused to link it. Site B's px1 then showed, alerted on and recorded
history for site A's sensors. The cluster sensor cache leaked the same
way: an agent's SSH readings for its cluster siblings were keyed by
bare node name, so a site A sibling reading fed site B's node.
The poller now hands the lookup the polled node, and the lookup works
on that node's slot. The state folds the views of one machine reached
through two connections (a cluster added twice, or a multi-homed host
added by each address) into one node held under one view's ID, and
the agent link follows it. Those views used to find their agent only
through the loose hostname match, so a polled view the read state does
not hold also takes the one same-named node proven to be the same
machine. models.NodeObservationsSameMachine and
models.HostAgentBridgesNodeViews give that proof from the evidence
UpdateNodesForInstance folds on, and the state's agent corroboration
check now shares its rule with them. Ambiguous candidates are declined,
as the state declines an ambiguous alias.
An agent linked to the slot is the node's agent. The hostname fallback
only takes an agent with no link when no node outside the slot and no
second unlinked agent has the name. Cluster sensor cache entries record
their reporting agent and serve a node only when that agent's linked
node shares one of the slot's connections, or belongs to a connection
whose configured view of the node has a matching TLS fingerprint.
LinkedNodeID holds the Proxmox source node ID for automatic, manual and
continuity-restored links, so the link match needs no normalization.
The physical-disk SMART merge already matches agents by
node.LinkedAgentID within one instance and is unchanged.
Main added Alert.resolution and its AlertResolution type at the end of
the Alert struct in models.go and of the Alert interface in
types/api.ts, exactly where this branch added metricStatus and its
types, so the two edits conflict. Move this branch's field up beside
metadata and its types below ResolvedAlert. Nothing else changes: the
built frontend is byte-identical, and the Go types are only reordered.
Contract-Neutral: declaration order only; no field, type or behaviour change
Normalization may keep a disk temperature it did not collect (a disk in
standby, a host agent past its reporting lease) as long as its collection
state says so. Two consumers still presented that value as current.
The metrics-history handler answered an empty disk range with the disk's
temperature as a "live" point at now, regardless of its state. Since the
history writers stopped recording retained temperatures, a short-range
chart for a silent agent's disk or a sleeping disk showed the retained
value as a fresh reading. Mock mode also seeded synthetic series from it
and padded sparse stored series out to now. All of these now require a
temperature collected now (diskinventory.TemperatureCollected, the
shared form of the rule the history writers apply).
Unraid array-inventory rows carry no per-field provenance, so a
temperature taken from one looked collected after the agent went silent.
The adapter now derives its state from the host lease that
ExpireHostTelemetry already records, and from the disk's spin state:
unavailable once the host is offline or the disk is spun down. A SMART
row that borrows the inventory reading takes that state, including
available. The native row records only the withdrawing states, because
the registry merge picks temperature and state separately and an
available claim from it would mask the SMART row's own state. A
withdrawn inventory temperature no longer counts as heat in disk risk.
A host agent that stops reporting keeps its last SMART rows in state.
Its frozen disk readings then kept being treated as current:
- `mergeHostAgentSMARTIntoDisks` copied the silent agent's temperature
and I/O counters into the linked Proxmox physical disk, along with its
`available` collection state.
- The registry merged the agent's retained SMART rows into the canonical
disk the same way. `MergeStatus` lets any `available` state win, so the
API kept reporting the readings as collected.
- Every full disk poll (default every 5 minutes) wrote the frozen
temperature to SMART history as a new sample.
## Change
- **Lease expiry is the cut-off.** `State.ExpireHostTelemetry` (run by
the offline sweep once `hostAgentHealthWindow` passes) keeps SMART
temperature and I/O as last-known values but marks them `unavailable`
("host agent stopped reporting"). This follows the physical-disk
retained-evidence clause in the monitoring contract. The agent's next
accepted report replaces them.
- **A source can withdraw its own evidence.** New
`diskinventory.MergeReportedStatus` lets a source's later non-available
statement supersede an `available` state that another row still carries
from the same source.
- Used by the monitoring merge, where skipped disk polls re-merge the
previous state.
- Used in both registry ingest orders. This covers a PVE host that is
down entirely: it is never disk-polled again, so its Proxmox disk row
stays frozen.
- **History records only collected temperatures.** All SMART temperature
history writers (`writeSMARTMetrics`, `smartMetricStoreWrites`,
`writeHostSMARTMetrics`, and the unified disk sync via its collection
passthrough) skip a temperature whose current state is not collected.
This also stops a standby disk's retained pre-sleep reading from being
recorded.
- **Pre-6.2 agents keep their history.** These agents send no collection
provenance, so the merge now records their temperature as available from
`host_agent` instead of inheriting Proxmox's `unsupported`.
- **Expiry is compare-and-set** on the `LastSeen` the sweep judged
stale, so a report accepted in between is never expired. The sweep's
alert-side offline handling for that race is unchanged; that race
already existed.
Deliberately unchanged:
- SMART attributes, wearout and health stay last-known. They are durable
health evidence, and dropping them would resolve disk-health and
low-life alerts whenever an agent goes quiet (cf. #2112).
- Disk temperature alerts are evaluated in `CheckHost` from the agent's
own report, only when a report is ingested, so they never read this
merged value.
## Related
- The node-temperature path (agent sensors → `node.Temperature.SMART`,
which the disk merge labels `proxmox_node_smart`) is gated on the
agent's own sighting in #2582. A carried node reading can still stamp a
disk as collected; that path belongs to the node poller.
- Filed separately:
- the chart API's `liveMetricPoints` returns a retained disk temperature
as a single "live" point when a range has no samples;
- Unraid-native disk temperatures carry no collection state;
- the disk table renders non-collected temperatures with live threshold
colouring;
- a merged agent+node row reads `online` after its agent dies, so
`linkedHostForNode` keeps using the silent agent's filesystem summary.
## Tests
- `TestSilentLinkedAgentSMARTIsRetainedButNotPresentedAsCollected`
(registered monitoring proof `physical_disk_roundtrip_test.go`). Uses a
registry-backed monitor with a real disk poll and a metrics store. A
reporting agent's reading is collected and recorded. After lease expiry,
the canonical disk reads as not collected before any disk poll runs, and
again after a skipped poll and a full poll. The value is retained and no
new history sample is written. A resumed agent is collected and recorded
again. Fails on origin/main. Mutation-checked: removing the expiry
marking, the monitoring supersede, the registry supersede, or the
history gate each makes it fail.
- `TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease`
- `TestPhysicalDiskMergeFollowsAgentWithdrawalOfItsOwnReadings` (both
ingest orders)
- `TestMergeReportedStatusLetsTheSourceWithdrawItsOwnAvailability`
- `TestExpireHostTelemetryMarksSMARTReadingsNoLongerCollected`
- `TestExpireHostTelemetryLeavesHostThatReportedSinceJudgement`
- `TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures`
Ran the full `internal/monitoring`, `internal/unifiedresources`,
`internal/models` and `pkg/diskinventory` packages locally; all pass.
A host agent that stops reporting keeps its last SMART rows in state.
The Proxmox physical-disk merge copied that agent's temperature and I/O
counters into the disk together with its "available" collection state,
the registry merged the agent's retained rows into the canonical disk the
same way, and every full disk poll wrote the frozen temperature to SMART
history as a new sample. A dead agent's disk temperatures therefore stayed
"collected" in the API and kept extending the history chart indefinitely.
Expiring the agent's reporting lease is now the point where those
readings stop being current. ExpireHostTelemetry keeps the SMART
temperature and I/O values as last-known evidence but marks them
unavailable ("host agent stopped reporting"), as the physical-disk
contract asks for retained evidence. Because MergeStatus lets any
available state win, a Proxmox disk row still holding the earlier copy
would mask that, both on skipped disk polls and when the PVE host is down
and never disk-polled again. diskinventory.MergeReportedStatus lets a
source's later word on its own field supersede the availability it
supplied earlier; the monitoring merge and both registry ingest orders
use it. Every SMART temperature history writer now records a
temperature only when its current state is collected, so a retained
reading (an agent past its lease, or a disk in standby) is not written as
a new sample.
Agents before 6.2 send no collection provenance, so their merged
temperature is recorded as collected from "host_agent" instead of
inheriting Proxmox's "unsupported", keeping their history intact.
Expiry is now compare-and-set on the report time the evaluation judged
stale, so a report accepted in between is not expired.
SMART attributes, wearout and health stay last-known on purpose: they
are durable health evidence, and dropping them would resolve disk-health
and low-life alerts when an agent goes quiet. Disk temperature alerts
are evaluated from the agent's own report, so they never read this
merged value. The node-temperature path (agent sensors feeding
node.Temperature.SMART, which the disk merge labels proxmox_node_smart)
is gated by the agent's sighting in #2582; a carried node reading can
still stamp a disk as collected and belongs to that path.
A node alert handed to its Pulse agent closes with an AlertResolution
(#2580), and notifications already say it moved. Every in-app reader
still treated the close as a recovery: Patrol's detail wore a green
Resolved badge over a timeline labelled "Recovery evidence", the alert
history listed the row as resolved and its incident timeline said
"Alert resolved: Node memory at 95%", and the assistant, Patrol's seed
prompt and the pulse_alerts tool saw a plain resolve because
GetRecentlyResolved dropped the resolution.
Operational trust gains a non-recovery cause, ownership_transferred,
which may only enter resolved and only with evidence. The resolve
transition uses it whenever the alert carries a moved_to_agent
resolution, so a handover during a snooze also stops counting as
resolved while snoozed.
models.Alert gains Resolution (reason, successor and the engine's
summary), copied by cloneAlert and filled by GetRecentlyResolved, so
the state snapshot, websocket, assistant context, Patrol seed and the
resolved-alerts tool all report where the alert went. The resolved
resource change carries the summary as its reason with alert_resolution
metadata, and incident memory titles the close with it.
In the UI, the history row reads "moved to agent" with the account in
its title and on the phone card, and Patrol's detail shows a Moved
badge with the handover sentence under the summary.
The Machines/Infrastructure drawer and the Docker host drawer built their
"Needs attention" list from the alerts embedded in each resource. No
backend code ever filled that list, so every resource carried
`alerts: []` and both drawers stayed silent while an alert was open: a
mock stack had 0 of 142 resources with embedded alerts while 14 alerts
were active, and a real dev Pulse showed `alerts: []` on a host whose
temperature alert was open.
Both drawers now read the websocket activeAlerts map, the source the
Proxmox node, guest and Mail Gateway drawers already use, through one
matcher, getAlertsForUnifiedResource. It keys a resource on the ids the
backend writes for it (unified id, canonical primary and superseded ids,
Proxmox source id, metrics target) and, for agent and Docker hosts,
agent:<id> and docker:<host source id> plus the disk, RAID, container
and service alerts nested under them. Rows show the affected resource
and metric like the other drawers, most severe first.
The dead embedded path is gone: the frontend ResourceAlert type and its
unused hasAlerts filter, the Proxmox summary alertCount nobody rendered,
and ResourceFrontend.alerts / ResourceAlertInput on the backend, which
also drops "alerts":[] from every resource in each broadcast.
Now that incident alerts reach the infrastructure drawer, a health
reason counts as stated when it is a whole sentence of an alert's text,
so a storage incident alert that joins its summary and impact does not
repeat beside either reason.
A threshold alert stays open after its reading drops below the trigger:
it holds until the reading reaches the clear level, then must stay there
for the recovery delay. While it held, the evaluator returned without
touching the alert, so Value, Message and LastSeen stayed at the last
breach. On 2026-10-06 minipc's row and guests said "Node temperature at
80.0°C" while the node table read a green 72°C, with nothing explaining
why the warning was still there (also #2068: a disk alert that stayed
open after usage dropped, with no sign of why).
The evaluator now records a volatile MetricStatus on every evaluation:
the value it compared (the average for rolling-average rules, with the
latest sample beside it), the phase (breaching, latched or recovering),
the trigger and clear levels, the recovery delay and the reducer's
monotonic recovery progress. Value, Message and LastSeen keep meaning
the last breach, and the hold branch still never notifies. The status is
stripped from the active-state checkpoint, the recovery mirror and the
degraded recovery projection, so a held alert adds no durable writes. It
reaches the websocket state through the active-alert projection.
Stale-alert cleanup and the retention TTLs now use the latest
observation. Before, an alert held below its trigger for over a day had
an old LastSeen and was auto-resolved as "resource no longer being
monitored" while it was still being evaluated.
Surfaces format the status through one helper and never re-derive the
phase. Node, guest and Mail Gateway drawer rows lead with the reading
now and say what clears the alert, with the last breach on hover. The
Proxmox TEMP cell keeps its own open temperature alert's tone and
explains it on hover instead of reading green beside a warning row.
Storage rows stop saying "Over N%" when usage is back under the limit,
and the Assistant handoff no longer labels the breach value as current.
Refs #2068
Freeze the source expectation as serialized bytes before mutating returned records. A broken clone cannot change both the tested state and its reference, so module, update and applied-config alias regressions must be detected.
Change-source: pulse-maintainer
Avoid cloning and ingesting twenty known non-host Kubernetes metadata kinds in temporary per-report matching registries, while preserving all publication and canonical matching. Clone only the selected native record for exact host reads and detach mutable host update, module and config state. Pin reference equivalence, live report publication, ownership and allocation scaling with connected controls.
Change-source: pulse-maintainer
Expose current, last-known and unavailable memory through raw guest and selected unified metrics without renewing QGA evidence during repeated backup deferrals. Preserve independent PVE/linked-agent source times, numeric selection, History and alert policy. Annotate selected metrics as well as raw facets so cross-source fallback cannot inherit another source freshness. Add connected observation, source ownership, clone and wire controls and update the shared subsystem contracts.
Change-source: pulse-maintainer
Retain configured RAID members separately from source-native totals through collection, reports and canonical read views. Correct the healthy legacy mdadm tuple from #2369 without subtracting spares from mdstat requirements or suppressing real deficits, failures and recovery warnings. Preserve observed zero-active counts through fallback, and verify collector, wire, ingestion, health and canonical activation/resolution boundaries.
Change-source: pulse-maintainer
Recognise exact node.VMID.name notes on Root PBS snapshots so a newer completed CT backup can replace an older name-only match. Keep colliding VMIDs fail-closed when node and name do not distinguish connections, and cover completed and in-flight attribution for Pulse #2292.
Contract-Neutral: Preserve backup attribution safety without changing API fields, persisted shapes, or source contracts.
Change-source: pulse-maintainer
A globally observed second cluster can leave a PBS source apparently mapped to the first while both clusters match a shared-node snapshot. Do not promote a positive but tied guest match into the VMID-only inference path. Regression covers both VM and CT completed and in-flight snapshots with the second cluster visible on another PBS instance.
Contract-Neutral: Preserve existing PBS backup attribution safety without changing API fields, persisted shapes, or source contracts.
Change-source: pulse-maintainer
A newer snapshot from another cluster can share the same typed VMID and node namespace. Compare each positive match with competing guests before selecting by recency; a stronger competitor excludes it and a tie needs independent attribution. Cover VM and CT completed and in-flight snapshots while preserving the newer weaker unique match.
Contract-Neutral: Repair existing backup badge attribution and age selection without changing API fields, persisted shapes, or PBS source contracts.
Change-source: pulse-maintainer
A stronger namespace or comment score currently makes an old snapshot beat a newer, still attributable snapshot. Keep the existing typed-VMID and cross-instance eligibility guards, then choose the latest eligible backup for LastBackup and in-flight state. Add VM and CT regressions for unique fallback and colliding IDs.
Contract-Neutral: Repair existing backup-age selection without changing API fields, persisted shapes, or PBS attribution eligibility.
Change-source: pulse-maintainer
Expose confined, identity-bound filesystem observations through the shared
resource pipeline so investigations can distinguish an exhausted container
mount from unrelated host capacity. Keep unavailable measurements explicit.
Isolate alert-history reads from durable writes and reuse one chronological
fold across polling. Catch up through bounded durable event IDs so simultaneous
readers do not replay every retained snapshot. Retain expired actions when
investigation outcomes move back to needs attention, and keep attached
Assistant context focused.
Record live storage diagnosis, healthy and dependency controls, approved and
rejected Docker outcomes, source-bound browser proof and exact test limits.
Missing-access continuity, VM dispatch completion and remaining Assistant
orchestration defects stay open in the redesign plan.
Preserve both monitoring contracts and combine the diagnostic history
correction with current host continuity and delivery evidence changes.
Verify the combined backend, frontend and browser behaviour before
landing the existing diagnostic work.
Missing block I/O and container image sizes could become false evidence
for diagnosis. Preserve per-direction counter presence and measured zero
through collection, resource conversion and browser rendering. Separate
new observed history from ambiguous retained disk series without deleting
old rows or changing public metric names.
Keep partial host rates distinct and persist a newly enabled Disk I/O
column across the first preference reload.
Commit link and unlink journal updates before publishing state, preserve manual selections across report and provider refresh boundaries, and reserve dormant owners across restart. Re-evaluate known automatic associations using provider names while retaining legacy unknown links.
Change-source: pulse-maintainer
Keep canonical disk risk, source freshness and retained history intact when
Assistant and Patrol gather evidence. Proposal acceptance validates an action
contract and must not rewrite uncertain conclusions as established root cause.
Preserve complete subscription tool batches without exposing routing envelopes
as answers. Keep wide answer tables readable and keyboard-scrollable on mobile.
Optimize retained tier reconciliation without discarding gaps or newer samples.
Record failed real-model diagnoses and outstanding autonomous qualification
separately from passing data-path and interface checks.
A reachable PBS can deny or fail its node-status endpoint while datastore collection succeeds. The resulting zero values previously resolved active CPU and memory alerts without a healthy measurement. Carry internal availability evidence from polling and skip metric evaluation for those samples, preserving independent connectivity and policy suppression. Add failing-before unit coverage and an HTTP polling lifecycle regression for dispatch and recent recovery history.
Change-source: pulse-maintainer
Issue #1895 reports parity alerts when mdNumDisks=0 on a pool-only Unraid system. Array service state alone does not establish that a parity array exists.
Preserve the optional disk count from collection through canonical runtime conversion and suppress only the no-parity warning for an explicit zero. Missing or malformed counts retain legacy behaviour, and disk failure reasons remain active.
Validated focused Unraid tests in hostagent, storagehealth, monitoring, unifiedresources and alerts, including JSON zero preservation and canonical round trip. The new pool-only regression fails against the previous warning condition. Both agent and server need this change; no release or reporter retest is claimed.
Change-source: pulse-maintainer
A PBS-to-PBS sync that fails mid-snapshot leaves a manifest-less copy of an
already-completed backup on the target datastore. That copy has the same
signature as a snapshot being written right now (no size, no
index.json.blob), so the guest showed Backup Running indefinitely until the
sync was repaired.
An in-flight snapshot only reports the guest as backing up when it is newer
than the guest's last completed backup and within the same 24h cutoff that
already ages out stale running vzdump tasks. A genuine in-progress backup is
always newer than the last completed one; a partial sync copy never is.
Refs #1815
Sequentially adding standalone Proxmox estates with the same native node name could lend the first site agent to the second through the short-hostname fallback. Both provider nodes then shared one agent identity and collapsed into one state slot. Require exact endpoint, certificate, or agent-bridged full endpoint evidence before a cross-instance standalone merge; reject and self-heal guessed links that already belong to another node.
Refs #1753
Contract-Neutral: narrows internal Proxmox node/agent identity reconciliation without changing API shapes or extension contracts
Third increment of the resource-payload-static-metadata gap, the
snapshot-side statics. Measurement first: the mock estate ships only 5
distinct non-default policy postures across 723 resources and 56
distinct templated AI-safe summary strings across the same rows
(~150KB of inline duplication per snapshot), and every canonical alias
list carries the row's own wire id verbatim.
Non-default policies and AI-safe summaries now dedupe through the same
content-addressed catalog contract as capabilities: state-level
policyCatalog and aiSafeSummaryCatalog hold the distinct blobs,
resources carry policyRef / aiSafeSummaryRef, and ingestion expands
refs back to the inline shape (default posture synthesis unchanged for
ref-less rows). The broadcast conversion returns one catalogs struct so
the next deduped field is a loop entry, and catalog changes ride the
same delta frame as the refs that need them. Alias lists stop shipping
the row's own wire id — every identity consumer unions resource.id
beside the alias spread — while the primary id stays so aliases remain
the complete live vocabulary a host is reachable by, per the
established slimming invariant.
Verified against a 1857-resource estate: zero inline policies or
summaries on the wire (743 refs resolving into 1KB + 7KB catalogs),
zero alias lists carrying the wire id, catalog-expanded postures and
summaries reaching the store byte-equivalently, and the estate
rendering clean on desktop and the 36-row phone window.
Remaining gap scope: canonicalIdentity alias/superseded history is
genuine per-resource identity data (~600KB per snapshot) with no safe
derivation shortcut, and the proxmox-facet/platformData mirror
duplication (~200KB) is deliberate canonical-merge fabric; both stay
recorded under resource-payload-static-metadata for a future owner.
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).
The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.
Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.
Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.
/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.
Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.
The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
An in-progress PBS snapshot (reproduced on VM 117 at 9% written) was
counted as a completed backup: the guest's LastBackup jumped to "now" the
moment vzdump started, and the workloads badge showed a green healthy
shield for a backup that did not exist yet. If the job aborted, that
shield had never been true.
Detect the in-flight state from what the PBS listing actually says - no
size and no index.json.blob manifest in files - and flag the snapshot
InProgress instead of complete. On the PVE side, correlate storage content
entries with live vzdump tasks: a guest archive whose ctime falls inside a
running task's window is that task's partial output, and a PBS-storage
entry without a size has no manifest yet.
In-flight artifacts now feed a new per-guest BackupInProgress flag rather
than LastBackup, so the badge age always reflects the latest COMPLETED
backup. The workloads badge gets a distinct blue "Running" state (green
would claim success, None would hide that work is underway), with the
last-completed age kept in the tooltip and accessible label. Recovery
points map to OutcomeRunning with no completion time, keeping posture
freshness and the Backups page's latest-restore pointers anchored to
completed backups; the recoverable-artifact list shows a Running chip.
A cached in-flight snapshot forces a per-poll re-fetch of its group so
completion or abort is noticed promptly instead of after the cache TTL.
Connected systems showed a PBS machine twice (API row plus host-agent row)
whenever the PBS connection was configured with an address the agent never
reports, because PBS identity was limited to the configured name and host.
The PBS poll now captures the hostname the node reports about itself
(GET /nodes, the previously unused client GetNodeName) on
models.PBSInstance.NodeName, and /api/connections includes it in the PBS
connection's host aliases, so directPlatformHostAttachment merges the agent
running on that machine into the PBS row the same way PVE composes from
API-reported node identity. Node-name fetch failure stays partial data and
never fails the poll. Reported by Johannes Strasser (rc.5).
rcourtman