GET /api/charts/workloads-summary has had no first-party caller since
the v6 import: ChartsAPI.getWorkloadsSummaryCharts arrived with it and
nothing ever called it. GET /api/charts/storage-summary lost its last
reader when storageSummaryTrendCache.ts was deleted on 2026-05-26.
Both still cost something. The workloads route carried a payload cache,
an SLO test in the release latency gate and a benchmark, and mock mode
rebuilt the 24-hour storage aggregate on every sampler tick for a cache
nothing read.
No stable release ever requested either route from the browser: v6.0.0
and later carry no caller, and the storage trend reader shipped only in
v6.0.0-rc.1 to rc.5. Neither route is in docs/API.md (a list of common
endpoints, not a complete one), and pulse-pro, pulse-mobile and
pulse-dev-infra never call them. A monitoring:read token could reach
them, so a private script may notice, but nothing known depends on
them. pulse-enterprise names the SLO test only as fixture data, and its
latency gate passes a test that is absent from the candidate as
removed. So the routes go with their clients:
handlers, response types, the monitor trend helpers, the mock aggregate
and its prewarm, the SLO budget, the benchmark, and the ChartsAPI
methods and types. Requests to either path now get 404.
buildMockWorkloadMetricHistorySeries goes too, because the
workloads-summary handler was its only production caller.
A TypeScript reference graph from src/index.tsx finds no production
reference to the removed methods or types. The production build loses
only the two static methods from the ChartsAPI chunk (326 B raw, 49 B
gzip); every other file differs only by chunk hashes and SRI values.
deadcode ./cmd/pulse and golangci unused report nothing newly
unreachable. On private mock stacks the Proxmox overview in Trends, a
guest History drawer and the storage page make the same chart requests
before and after at 1440 and 390 px, and neither build requested the
removed routes.
Contract prose that still described either route as live now records
the retirement, and the workload chart guard pins the per-type metric
sets the surviving /api/charts/workloads route requests.
## Problem
PR #2648 stopped the poller's previous-evidence lookup from handing a
replaced disk the old disk's serial. The linked host agent's SMART merge
runs before that lookup and had the same flaw. Its last-resort match is
the device path, and a path names a slot, not a disk. When a disk is
swapped before the agent reports again, or after the agent went silent
and kept its last rows, the agent's row for the old disk matched the
replacement. It filled the old serial or, for SAS, promoted it over the
Proxmox SAS address. The registry then keyed the replacement to the old
disk's canonical resource and SMART history, and it also took the old
disk's temperature and FAILED health. For SAS, the registry's same-path
join (`resolveLinkedPhysicalDisk`) did the same at the canonical level
even when monitoring kept the rows apart.
Proxmox's disk inventory (`PVE::Diskmanage`, checked on a PVE 9 node)
writes the literal `unknown` for a serial or WWN udev cannot read. The
poller recorded it as a collected serial, so the collection state hid
the gap, and the merge's `Serial == ""` fill never let the agent's real
serial replace it.
## Change
- `mergeHostAgentSMARTIntoDisks` refuses a device-path row whose
identity contradicts the Proxmox disk's. The new
`diskinventory.HardwareIdentityConflict` defines a contradiction:
nothing agrees across serial and WWN, and a field both producers report
like for like disagrees.
- WWNs count for every disk.
- Serials count only where Proxmox's serial is the drive's own, as
smartctl's is: a disk typed `nvme`, or one presented with SCSI vendor
`ATA` (libata, or a SAS HBA's SCSI-ATA translation) and not typed
`usb`/`sas`. Elsewhere the Proxmox serial may be a SAS address or SCSI
designator (#1595) or a USB bridge's own serial, so it only counts
toward agreement.
- Spelling differences are not disagreement: udev's underscores for
whitespace, a T10 designator for an ATA drive (`ATA`, the model, then
the serial), the agent's NAA field spelling, and udev's 64-bit prefix of
an NAA 6 WWN.
- The refusal runs after the unique-path check, so a second row on the
path keeps the match ambiguous.
- The merge's WWN step compares normalized identity
(`HardwareIdentityMatch`), so the agent's `5-c50-a1b2c3d4` finds the
Proxmox `0x5000c500a1b2c3d4` disk.
- `resolveLinkedPhysicalDisk` refuses the SAS same-path fallback when
the two WWNs name different disks.
- The poller records Proxmox's `unknown` serial/WWN as empty, so the
serial reads `missing` and the agent can fill it. Retained evidence
carries a previous serial only when it is usable, so a pre-upgrade
record cannot restore the placeholder. Identity keys already ignored the
placeholder. When the agent now fills a disk's serial, that disk's
Proxmox-poll SMART samples move from its WWN or path key to the serial
key, where the agent's own samples already go.
## Tests
- `TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity` (sata
and sas) drives the real poller and registry. Proxmox reports `unknown`,
the agent fills the serial, the agent goes silent, and the disk is
swapped. On main, the SATA replacement takes the old serial and
resource; the SAS replacement keeps its own serial in state, but the
registry joins it to the old resource with its temperature and FAILED
health.
- `TestHostAgentSMARTMergeRefusesPathMatchAcrossContradictingIdentity`
has one row per producer rule: SAS/SATA/NVMe replacements refused;
same-disk SAS without an agent WWN, a renamed SAS disk in the agent's
WWN spelling, udev's serial spelling, a USB enclosure (typed `usb` and
typed `ssd`), a T10 designator and an NAA 6 volume merged; and an
ambiguous path.
- `TestRegistrySASPathJoinRefusesContradictingWWN`,
`TestProxmoxUnknownDiskIdentityIsRecordedAsUnreported` and
`TestHardwareIdentityConflict` cover the rest.
- Negative controls via `go test -overlay`: removing each rule fails
only its own case. These cover the guard, the WWN normalization, the
transport/vendor rule, compaction, the designator rule, the NAA 6
prefix, the registry guard, the ingest normalization and the
usable-serial carry. Against main's `monitor.go` and `registry.go`, the
end-to-end, merge and registry tests fail.
- `TestMergeHostAgentSMARTIntoDisks_PreservesExistingProxmoxIdentity`
merged a row with a different WWN by path alone, which the guard now
rightly refuses. Its row is now the same disk in the agent's WWN
spelling, with all assertions unchanged.
- `internal/monitoring`, `internal/unifiedresources` and
`pkg/diskinventory` pass in full.
## Not covered
- A replacement with no comparable identity still matches its path: no
usable serial or WWN at all, or a SAS/SCSI/USB disk with a WWN on one
side only.
- The registry's seeded source mapping, reused after a restart by the
slot-shaped Proxmox source ID, can still hand a swapped disk the old
canonical resource. That's a separate follow-up.
## Problem
Nested Proxmox clusters (each PVE node is itself a QEMU VM) lose
physical disks from inventory. QEMU gives a virtual disk configured
without a serial a default one built from its drive ID or a per-VM
counter, so every VM built the same way reports the same value:
- Proxmox's `/nodes/{node}/disks/list` serial is udev's
`ID_SERIAL_SHORT` (`PVE/Diskmanage.pm`). A virtio-scsi disk on a Proxmox
VM reports `drive-scsi0` with no `ID_WWN`; on a libvirt VM it is
`drive-scsi0-0-0-0`. A `-drive` given no ID falls back to QEMU's
automatic ID, `scsi0-hd0` for `if=scsi` and `none0` for `if=none`.
- An emulated ATA disk's serial is `QM` plus a five-digit counter
(`QM00001`).
`diskinventory.IsUsableHardwareID` accepted all of these, so
`resourceFromPhysicalDisk` set `MachineID = "drive-scsi0"` on every node
and the registry minted one `machine:drive-scsi0` canonical disk and one
SMART series for the whole cluster. Three nodes with two such disks each
produced 2 canonical disks instead of 6.
## Change
- `IsUsableHardwareID` rejects exactly QEMU's default shapes:
`drive-scsi` plus digits with at most three `-digits` groups,
`scsi<n>-hd<n>`, `none<n>`, and `QM` plus five digits. Real serials that
only share a prefix (`DRIVE-ASSET123`, `drive-sata0`, `QM0001`) stay
identity. It also strips `naa.`/`eui.`/`wwn-`/`0x` framing before its
all-zero and all-F test, so a `0x0000000000000000` WWN is a placeholder
too.
- TrueNAS provider: a disk took its canonical identity from any
non-empty serial, so a placeholder merged disks across appliances. It
now uses only a usable serial, and lists superseded IDs for every disk
without one.
- TrueNAS history keys: for a serial that was already rejected, such as
`UNKNOWN`, native TrueNAS history used TrueNAS's own disk identifier,
and the unified SMART writer (`syncUnifiedPhysicalDiskMetrics`)
substituted the metrics target only for an empty serial, so the samples
went to the canonical resource ID. Neither matched the chart reader's
key, and rejecting QEMU serials would have put TrueNAS VMs on that path.
Native history now falls back to the disk's source ID, and the writer
always writes under the resolved metrics target, so writer, native
history and reader agree. History lookup keys also drop placeholder
serials.
## Upgrade effect
A disk reporting one of these values (nested PVE nodes, TrueNAS or agent
hosts running as QEMU VMs with virtual disks, TrueNAS disks reporting
`UNKNOWN`) changes canonical ID and SMART metric key once. Samples
recorded under the old key are not carried over, because that key mixed
every disk that shared the value. Path-shaped alert references (Proxmox
disk health and wearout, host-agent SMART) are unaffected; anything
keyed on the old canonical ID or metric key starts again under the new
one. This is recorded in the monitoring and agent-lifecycle contracts.
## Tests
- `TestIsUsableHardwareIDRejectsQEMUDefaultDiskSerials`: each default
shape is rejected, falls back to the scoped topology ID and never
matches itself; prefix-sharing real serials stay usable.
`TestPreferredIDRejectsPlaceholderHardwareIdentity` gains the framed
all-zero/all-F WWNs.
- `TestNestedProxmoxDefaultQEMUSerialsStayPerNode`: 3 nodes ×
(`drive-scsi0`, `QM00005`, `drive-scsi0-0-0-1`) give 9 canonical disks,
9 slots and 9 metric keys, every SMART writer key is a key readers
resolve, and agent SMART source IDs stay per host.
-
`TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey`:
two appliances with `UNKNOWN` and `drive-scsiN` serials keep one
canonical disk each, and the SMART writer's samples are readable under
each disk's metrics target.
- `TestProviderPlaceholderDiskSerialsKeyNeitherIdentityNorHistory`:
placeholder serials key no identity, list superseded canonical IDs, drop
out of native history lookup keys, and native history lands under the
registry's metrics target.
- Negative controls via `go test -overlay`, one runtime file reverted at
a time: base `identity.go` fails all three round trips (3 canonical
disks instead of 9, 9 TrueNAS disks instead of 16); base `provider.go`
fails the identity and history assertions; base `monitor.go` fails the
SMART-history-under-reader-key assertion; reverting only the native
history key, the superseded-ID gate or the lookup-key filter fails its
own provider assertion.
- `pkg/diskinventory`, `internal/truenas/...`,
`internal/unifiedresources/...` and `internal/monitoring` pass in full;
`internal/api/...` disk, hardware, serial and TrueNAS tests pass.
Stacked on #2648 (`fix/pve-disk-serial-carry`), whose commit this branch
carries until it merges. Not in this change: the registry's merge of
disks across parents on a canonical-ID collision (separate change), and
a pre-existing writer/reader key mismatch for controller-member disks
without a usable serial or WWN (`PhysicalDiskMetaMetricID` appends
topology to a source ID that already carries it), which is tracked
separately.
The linked host agent's SMART merge falls back to matching a Proxmox
disk by device path, and a path names a slot, not a disk. When a disk
is swapped before the agent reports again, or after the agent went
silent and kept its last rows, the row for the old disk matched the
replacement. It filled or, for SAS, promoted the old serial onto it,
which keyed the replacement to the old disk's canonical resource and
metric history, and lent it the old disk's temperature and failed
health. For SAS the registry's same-path join did the same at the
canonical level even when monitoring kept the rows apart.
Both now refuse a row whose identity contradicts the disk's, judged
only on what both producers report like for like. WWNs count for every
disk. Serials count only where Proxmox's serial is the drive's own, as
smartctl's is: NVMe namespaces, and disks presented with SCSI vendor
"ATA", whose serial udev's ata_id reads from IDENTIFY. Elsewhere
Proxmox's serial may be a SAS address, a SCSI designator or a USB
bridge's own serial (#1595). Reporter spellings are not disagreement:
udev writes whitespace in serials as underscores, its T10 designator
for an ATA drive puts "ATA" and the model before the drive serial, and
its ID_WWN keeps only the first 64 bits of an NAA 6 identifier. The WWN match step compares normalized identity too,
so the agent's 5-c50-a1b2c3d4 spelling finds the Proxmox
0x5000c500a1b2c3d4 disk.
Proxmox's disk inventory reports a serial or WWN udev cannot read as
the literal "unknown". The poller recorded it as a collected serial, so
the collection state hid the gap and the agent's serial never filled
it. It is now recorded as empty, and retained evidence carries only a
usable previous serial so an older record cannot restore it. Identity
keys already ignored the placeholder.
## Problem
The Proxmox disk poller keeps evidence it could not collect this poll by
looking up the previous poll's record of the disk: by serial, then WWN,
then the Proxmox source ID (`instance-node-devpath`), then the device
token. The source ID and device token name a slot, not a disk, and a
replacement in the same slot keeps both. So:
- a replacement reporting a different serial or WWN took the old disk's
retained temperature, controller, pool and I/O;
- a replacement record with an empty serial also took the old disk's
serial. The registry keys disks on serial, so the new disk became the
old disk's canonical resource and SMART metric series, and every later
poll carried the borrowed serial forward.
Current Proxmox reports a missing serial as the literal `unknown`, which
the poller records as reported, so the serial copy needs a producer that
sends an empty serial, such as the host-agent fallback rows. The
evidence copy happens with real Proxmox data.
## Change
- `previousPhysicalDiskEvidence` matches by hardware identity through
`diskinventory.HardwareIdentityMatch` (serial first, then WWN, each
against both fields of the earlier record), and refuses a slot match
when both records carry a serial or WWN and none agrees. Missing
identity is still not a conflict: placeholders such as `unknown`, and an
empty serial, keep the slot's evidence as before.
- The registry's merged disk view keeps the host agent's WWN, which the
agent spells as unpadded NAA 5 fields (`5-c50-a1b2c3d4`,
`internal/hostagent/smartctl.go` `formatWWN`) while Proxmox reports
`0x5000c500a1b2c3d4`. `HardwareIdentityMatch` now re-pads that spelling
for WWN fields only, so one disk seen by both reporters is not taken for
a replacement. The registry's linked-disk join uses the same comparison.
## Tests
- `TestPhysicalDiskReplacementInSameSlotDoesNotInheritPreviousSerial`
drives the poller and registry across two polls after a swap; on base
the replacement inherits the old serial and canonical resource. Its
controls keep a serial-omitting same disk on its resource.
-
`TestPreviousPhysicalDiskEvidenceRejectsSlotMatchAcrossConflictingIdentity`
covers renamed devices matched only by identity, PVE vs agent WWN
spellings, SAS address serials with a matching WWN, placeholders, and
conflicting serial/WWN by source ID and by device token.
- Negative controls via `go test -overlay`: on base the replacement,
conflict and renamed-device cases fail; with the guard but without the
WWN re-padding, the same-disk PVE/agent WWN cases fail.
- `pkg/diskinventory`, `internal/unifiedresources` and
`internal/monitoring` pass in full.
Follow-ups, not in this change: the host-agent SMART merge's device-path
fallback can still fill or promote a serial from a stale SMART row; QEMU
default serials (`drive-scsi0`) collapse nested Proxmox nodes' disks
into one resource; and the registry merges disks across parents on a
canonical-ID collision.
QEMU gives a virtual disk configured without a serial one built from
its drive ID or a per-VM counter, so every VM built the same way reports
the same value. Proxmox's disks/list serial is udev's ID_SERIAL_SHORT,
which for a virtio-scsi disk is "drive-scsi0" (libvirt:
"drive-scsi0-0-0-0"), and an ATA disk's serial is "QM00001".
IsUsableHardwareID accepted these as hardware identity, so every node of
a nested Proxmox cluster minted one machine:drive-scsi0 canonical disk
and one SMART series, and all but one node lost the disk from inventory.
IsUsableHardwareID now rejects exactly QEMU's default shapes: drive-scsi
plus digits with at most three -digits groups, QEMU's automatic drive
IDs scsiN-hdN and noneN, and QM plus five digits. Real serials that
share a prefix, such as DRIVE-ASSET123, stay identity. It also strips
WWN framing before the all-zero test, so a 0x0000000000000000 WWN is a
placeholder too.
The TrueNAS provider set a disk's canonical identity from any non-empty
serial, so placeholders merged disks across appliances. It now uses only
a usable serial. For a serial that was already rejected, such as
UNKNOWN, native TrueNAS history and the unified SMART writer disagreed
with the chart reader: native history used TrueNAS's disk identifier,
and the writer substituted the metrics target only for an empty serial,
so a placeholder sent samples to the canonical resource ID. Rejecting
QEMU serials would have put TrueNAS VMs on that path. Both now use the
key the registry's metrics target resolves, the disk's source ID, and
native history lookups no longer route by placeholder serials.
Disks that report one of these values change canonical ID and SMART
metric key once; samples under the old key are not carried over because
that key mixed every disk that shared the value.
The disk poller keeps evidence it could not collect this poll by looking
up the previous record, first by serial and WWN, then by the Proxmox
source ID and device token. Those last two name a slot, not a disk, and
a replacement in the same slot keeps both. So a replacement reporting a
different serial or WWN took the old disk's retained temperature,
controller, pool and I/O. A replacement record with an empty serial
also took the old serial. The registry keys disks on serial, so the
new disk became the old disk's canonical resource and SMART metric
series, and each later poll carried the borrowed serial forward.
Match on hardware identity through diskinventory.HardwareIdentityMatch,
serial first then WWN, and refuse a slot match when both records carry
a serial or WWN that share no value. Missing identity is still not a
conflict, so Proxmox's literal "unknown" and other placeholders keep
the slot's evidence as before.
The registry's merged disk view keeps the host agent's WWN, which the
agent spells as unpadded NAA 5 fields ("5-c50-a1b2c3d4") while Proxmox
reports "0x5000c500a1b2c3d4". HardwareIdentityMatch now re-pads that
spelling for WWN fields, so the same disk seen by both reporters is
not mistaken for a replacement.
The registry merges a physical disk's rows one step at a time, choosing
the temperature by source preference and merging collection states
separately. The pairing rule from #2618 was exact for two rows only: with
three or more, an earlier step's single merged state stood in for its
rows. A silent agent, an offline Unraid inventory and a stale Proxmox
copy could leave a withdrawn 41 labelled collected, and a legacy agent's
stateless reading could pick up another row's "unsupported".
The merged disk now keeps each row's current reading and state
(merge-time only, never serialized), and the shown value takes its state
from the readings of that value: a collected reading first, then the
agent's own report over a copy. A copy of an agent source's state gives
way to the agent's own withdrawal, including a legacy agent's sourceless
one, and a value source preference keeps after its row reported again
keeps its presented state under that row's later withdrawal. Value
selection is unchanged.
Follows #2591. A node alert that moves to its linked Pulse agent closes
with a `moved_to_agent` resolution (#2580). Two readers still presented
that close as a recovery.
**Performance reports.** The node, VM and container enrichers copied
recently resolved alerts into `reporting.AlertInfo` without the
resolution, so a report generated in the five-minute window after a
handover read HEALTHY, "All systems operating normally", counted the
alert as "triggered and resolved", and sent the AI narrator `resolved:
true`.
- Report rows now carry the alert's resource and resolution
(`reportActiveAlertInfo` / `reportResolvedAlertInfo`).
- `pkg/reporting` has one rule (`unresolvedAlerts`): an alert that
closed without recovering stays live unless the report also shows its
successor's reading for that metric, meaning an open alert or a recovery
since the handover. That way the condition is counted once, and fleets
match handovers across all their resources.
- The single-resource PDF card reads WARNING (or CRITICAL at the alert's
level) with "Alert moved to <agent> - check the agent for the current
reading". The active list names the agent, the alerts table shows
"Moved" with the engine's summary as a note, and recommendations say to
check the agent.
- Fleets count the alert as active and name the agent. Their all-clear
pattern and "No fleet-wide action required" no longer appear while any
alert is active.
- The narrator payload carries `resolved: false`, the summary and
`successor_alert_listed`, with prompt rules for both.
**Resource history.** `alert_resolved` changes were labelled by kind
alone ("Alert resolved", green) with that label prefixed to the
headline. `getResourceChangePresentation` now reads the
`alert_resolution` metadata: the change reads "Alert moved" in the
neutral blue of the history badge, and its summary is the whole headline
in the drawer, `ResourceChangeSummary` and Patrol's assessment handoff.
**Verification**
- A private mock stack running this branch, with a local-only overlay
that closes a real pve1 memory alert through `clearAlertWithResolution`.
Within the window, `/api/admin/reports/generate?resourceType=node`
rendered CRITICAL with the handover message and a "Moved" row, while the
ordinary CPU recovery stayed green.
- In the browser, a full drawer (Availability checks, Fleet view) showed
the producer-made rows as "Alert moved" next to "Alert resolved" at 1280
and 375. A Proxmox node's own drawer uses the `table-row` presentation,
which fetches no change history (see below).
- Tests in registered proof files: `pkg/reporting/alert_state_test.go`
(verdict, successor timing, fleet pool, PDF text and width),
`internal/api/contract_test.go`, `internal/ai/report_narrator_test.go`,
`resourceChangePresentation.test.ts`,
`ResourceDetailDrawer.history.test.tsx` and
`patrolInvestigationContextModel.test.ts`. Each failed with the fix
reverted.
- Second opinion: two read-only review rounds. The temporal match,
fleet-wide pool, narrator dedupe signal, fleet all-clear fallbacks and
contract wording came from them.
**Not fixed here (separate changes)**
- Reports requested as `agent` carry no alert rows at all. That is how
the Reports page addresses every Proxmox node in v6, so such a report
reads HEALTHY even with active critical alerts. This change makes the
shared rule correct for when those reports attach the machine's alerts.
- Node and guest alert lifecycle changes are recorded under legacy alert
resource IDs, which canonical history queries never match. Facet kind
counts still group moved closes under "Alert resolved".
The PDF performance report's disk table coloured a temperature amber
from 50C and red from 60C for every disk type, the last fixed disk heat
line after disk heat moved to the alert disk temperature policy. An NVMe
at 63C, well under its 70C alert trigger, printed red. A user's raised
trigger changed nothing.
Each reported disk now carries the tenant's alert thresholds for its
type (the clear value and the trigger, from
alerts.Manager.DiskTemperatureThreshold). The PDF colours the reading
amber from the clear value and red from the trigger, as the Physical
Disks Temp column does, and leaves it plain when disk temperature
alerting is off.
Normalization may keep a disk temperature it did not collect (a disk in
standby, a host agent past its reporting lease) as long as its collection
state says so. Two consumers still presented that value as current.
The metrics-history handler answered an empty disk range with the disk's
temperature as a "live" point at now, regardless of its state. Since the
history writers stopped recording retained temperatures, a short-range
chart for a silent agent's disk or a sleeping disk showed the retained
value as a fresh reading. Mock mode also seeded synthetic series from it
and padded sparse stored series out to now. All of these now require a
temperature collected now (diskinventory.TemperatureCollected, the
shared form of the rule the history writers apply).
Unraid array-inventory rows carry no per-field provenance, so a
temperature taken from one looked collected after the agent went silent.
The adapter now derives its state from the host lease that
ExpireHostTelemetry already records, and from the disk's spin state:
unavailable once the host is offline or the disk is spun down. A SMART
row that borrows the inventory reading takes that state, including
available. The native row records only the withdrawing states, because
the registry merge picks temperature and state separately and an
available claim from it would mask the SMART row's own state. A
withdrawn inventory temperature no longer counts as heat in disk risk.
A node alert that moves to its linked Pulse agent closes with a
moved_to_agent resolution (#2580), and #2591 taught Patrol, alert
history and the assistant to say so. Two readers still showed the
close as a recovery.
Performance reports dropped the resolution when copying recently
resolved alerts into reporting.AlertInfo. A node report generated in
the five-minute window after a handover read HEALTHY, "All systems
operating normally", counted the alert as triggered and resolved, and
told the AI narrator resolved: true, although nothing had seen the
condition clear. Report rows now carry the resolution and the alert's
resource. pkg/reporting treats an alert that closed without recovering
as still live unless the report also shows its successor's reading for
that metric (an open alert, or a recovery since the handover), so a
handover never reads as healthy and one condition is counted once;
fleet reports match handovers across all their resources. The PDF
card, Quick Stats, active list and heuristic narrative say where the
alert went and to check the agent, the alerts table shows "Moved" with
the engine's summary, fleets count it as active and name the agent,
and the narrator gets resolved: false, the summary and whether the
successor's alert is listed, with prompt rules for both.
Resource change readers labelled every alert_resolved change "Alert
resolved" in green and prefixed the headline with it, so a moved close
read "Alert resolved: Alert moved to pve1 (Host Agent). This is not a
recovery...". getResourceChangePresentation now reads the
alert_resolution metadata: such a change is "Alert moved" in the
neutral blue of the history badge, and its summary is the whole
headline in the drawer, ResourceChangeSummary and Patrol's assessment
handoff.
Reports requested as agent, which is how the Reports page addresses
every Proxmox node, still carry no alert rows at all, and node alert
history is stored under legacy node IDs that the drawer never queries;
both are separate fixes.
A host agent that stops reporting keeps its last SMART rows in state.
The Proxmox physical-disk merge copied that agent's temperature and I/O
counters into the disk together with its "available" collection state,
the registry merged the agent's retained rows into the canonical disk the
same way, and every full disk poll wrote the frozen temperature to SMART
history as a new sample. A dead agent's disk temperatures therefore stayed
"collected" in the API and kept extending the history chart indefinitely.
Expiring the agent's reporting lease is now the point where those
readings stop being current. ExpireHostTelemetry keeps the SMART
temperature and I/O values as last-known evidence but marks them
unavailable ("host agent stopped reporting"), as the physical-disk
contract asks for retained evidence. Because MergeStatus lets any
available state win, a Proxmox disk row still holding the earlier copy
would mask that, both on skipped disk polls and when the PVE host is down
and never disk-polled again. diskinventory.MergeReportedStatus lets a
source's later word on its own field supersede the availability it
supplied earlier; the monitoring merge and both registry ingest orders
use it. Every SMART temperature history writer now records a
temperature only when its current state is collected, so a retained
reading (an agent past its lease, or a disk in standby) is not written as
a new sample.
Agents before 6.2 send no collection provenance, so their merged
temperature is recorded as collected from "host_agent" instead of
inheriting Proxmox's "unsupported", keeping their history intact.
Expiry is now compare-and-set on the report time the evaluation judged
stale, so a report accepted in between is not expired.
SMART attributes, wearout and health stay last-known on purpose: they
are durable health evidence, and dropping them would resolve disk-health
and low-life alerts when an agent goes quiet. Disk temperature alerts
are evaluated from the agent's own report, so they never read this
merged value. The node-temperature path (agent sensors feeding
node.Temperature.SMART, which the disk merge labels proxmox_node_smart)
is gated by the agent's sighting in #2582; a carried node reading can
still stamp a disk as collected and belongs to that path.
Guest lock verification used ordinary redirect-following config reads and last-key-wins JSON decoding. Refuse redirected or ambiguous lock evidence before dispatch and after a complete guest response, without changing ordinary config redirects, pooled TLS policy or session recovery. Cover all six readers and connected cache/History continuity and fresh resumption; this source hardening does not establish native backup thaw.
Change-source: pulse-maintainer
Preserve the actual HTTP response status rather than trusting quoted API errors. Apply the VM-wide cooldown after request timeouts and unexplained server or gateway failures, while retaining exact command-bound terminal refusals as ordinary errors. Reject ambiguous error envelopes and preserve cached guest observations alongside fresh live counters.
Change-source: pulse-maintainer
Guest-agent GETs enqueue serial work: ordinary HTTP pooling, stream recovery and redirects can resend a command despite the application single-attempt guard. Use an isolated single-use HTTP/1 transport with the configured TLS, proxy, dial and timeout policy, and defer redirects without contacting their destination. Keep ordinary API recovery and current CPU/I/O counter presence/receipts unchanged. Pin wire-level lost replies, all read/redirect paths, TLS trust and connected deferred cache/History behaviour.
Change-source: pulse-maintainer
Do not admit guest commands or publish their payloads from an unlocked JSON prefix when the authoritative config response is incomplete. Preserve bounded ordinary config reads, live CPU and I/O observations, cache timestamps and healthy resumption.
Change-source: pulse-maintainer
The HTTP error wrapper hid truncated response bodies from guest admission. Preserve the bounded read failure as a private wrapping type so error statuses also cool down the whole guest. Cover both 200 and 500 truncation, and retain complete permission refusals plus subsequent healthy reads.
Contract-Neutral: Completes the existing full-response/incomplete-body guest safety contract; error text and ordinary API errors retain compatibility, with no new interface or surface.
Change-source: pulse-maintainer
The generic password-session client retries once after a 401; that implicit retry can bypass a fresh lock check inside a guest admission. Disable it only for QGA commands, retain the refusal, and prove healthy guest reads plus unchanged ordinary API session recovery.
Contract-Neutral: Completes the documented no-replay guest admission rule without adding a contract or surface; adds password refusal and healthy recovery controls.
Change-source: pulse-maintainer
Check every QGA read against fresh VM config locks, coordinate configured endpoint aliases and VM migration, discard in-flight backup payloads and cool down uncertain completions without retries. Preserve labelled cache evidence and keep carried samples out of History. Includes real-client monitoring, memory and counter provenance controls; native freeze/thaw acceptance remains separate. Refs #2439.
Change-source: pulse-maintainer
Preserve the exact reviewed specialist commit and its documentation and signer controls alongside the current canonical frontier.
Change-source: pulse-maintainer
Distinguish failed verification from a tampering diagnosis, unsigned history, query errors and runtime gates. Remove live-key replacement and regeneration advice, correct default store paths, and preserve matching keys and history for isolated recovery. Bind the shipped guides to regression checks and exercise recovery distinctions with the real signer and encryption manager on synthetic temporary data.
Contract-Neutral: Audit documentation and synthetic regression controls only; no signing, storage, entitlement, API or frontend runtime behaviour changes.
Change-source: pulse-maintainer
Retain configured RAID members separately from source-native totals through collection, reports and canonical read views. Correct the healthy legacy mdadm tuple from #2369 without subtracting spares from mdstat requirements or suppressing real deficits, failures and recovery warnings. Preserve observed zero-active counts through fallback, and verify collector, wire, ingestion, health and canonical activation/resolution boundaries.
Change-source: pulse-maintainer
Adapt PR2351 to current main without losing bounded probe deadlines, retries, explicit binds or confidentiality. Inspect IPv6 wildcard socket mode so unrelated servers sharing the port cannot determine Pulse health. Apply the telemetry callback only after a persisted explicit boolean transition, including stale-disk and null-input boundaries.
Retain real HTTP/HTTPS wildcard and same-port isolation regressions, callback persistence-order tests, and the owning subsystem contracts. No dependency manifests, frontend source or telemetry payload schema change. Exact runtime proof remains dependent on the unavailable root graph; it is not represented as passed.
Change-source: pulse-maintainer
Original-source: https://github.com/rcourtman/Pulse/pull/2351
Original-commit: f50c4d6e3b
Co-authored-by: rcourtman <8825017+rcourtman@users.noreply.github.com>
Try both loopback families for IPv6 wildcard listeners without changing explicit or IPv4-only binds. Reserve each family a share of the API deadline, keep all later checks on the responding address, and retry a failed observation once with a fresh bounded budget in the telemetry background runner.
Keep genuine API, UI and asset failures visible and report only a closed timeout category, never transport details. Cover both real listener families, startup recovery, failure and privacy boundaries; synchronise the disclosure and subsystem contract.
Change-source: pulse-maintainer
Relay left public checkout on 2026-09-29: it only ever connected the
Pulse Mobile app, which retires on 31 March 2027, and existing Relay
subscribers now carry Pro entitlements. The app still sold it. Every
Community install saw a "Get alerts on your phone ... Available with
Relay and Pro plans" upgrade panel on Alerts destinations, the plan
screen offered a Relay card with "Remote web access via Relay", and the
settings section was called Remote Access although Relay never reached
the web UI.
The Alerts push panel now renders only on instances that have the relay
feature, with no upsell. The settings section, nav, header and locale
catalogs say Pulse Mobile and carry the retirement date. Community sees
only the Pro comparison card and Relay-tier licenses see none; gated
mobile features name Pro as their minimum plan. The relay feature is
labelled "Pulse Relay (Mobile Connection)" in the catalog, plan copy no
longer claims remote web access, the backend pairing diagnostics point
at Settings > Pulse Mobile, and the user docs, including PRIVACY.md,
state that Relay does not provide remote web UI access.
A provider-hosted control plane refused to start once its licence was past
expiry and the 7-day grace, because startup validated it with
ValidateLicense. So at the next restart after a 60-day evaluation ran out,
or after a paying provider's subscription lapsed, the portal went down, and
the portal's Plan tab is the only place a provider can buy or renew.
Reproduced on the walkthrough lab.
Startup now accepts an authentic, key-bound licence even when lapsed
(ValidateLicenseAllowingLapse), preferring a current licence and otherwise
the one that expired later, and logs the lapse. It unlocks nothing: client
runtimes still verify the provider licence in every lease with
ValidateLicense and drop MSP capabilities, new clients are refused with
provider_msp_license_lapsed, and the refresher still refuses a lapsed
licence. The Plan panel says the evaluation or plan ended and offers the
plans, and provider-msp status reports license_lapsed without failing so a
lapsed install can still upgrade.
The same run showed the portal printing control plane error codes such as
provider_msp_license_lapsed instead of their message. The portal now
prefers an error's message over its code, which also fixes
already_subscribed and checkout_unavailable in the Plan tab.
Metric identity lived in idx_metrics_lookup, ordered (type, id, metric,
tier, timestamp). Once a series retains history, that order gives every
series its own insertion point, so each poll's commit rewrites one index
leaf page per series through the WAL and again at checkpoint. On a
three-node homelab running v6.4.5-rc.2 that came to 330-607 KB/s of disk
writes, about 10.6 KB per ~60-byte sample and 27-50 GB a day (#1966).
idx_metrics_query_all already holds the same five columns time-major, so
one poll's samples for a resource share pages. It now carries uniqueness
and serves every read, and the metric-major lookup index is dropped. With
an hour of retained history for 480 series, 30 polls write 6,022 WAL
frames instead of 25,310 (4.2x fewer); the empty-table issue-1124 estate
drops from 36,308 frames to 20,223, and its ceiling tightens from 40,000
to 23,000.
The tier-reconciliation overlap probe now names the identity index too;
left to the planner it seeks on (tier, timestamp) alone once the
metric-major tree is gone, which TestRetainedQueryPlansUseIndexes caught.
Metric-filtered reads now scan a resource's other metrics in the window:
a single-metric 500-node query went from 45 to 59 us and the 24-hour
50-disk smart_temp batch from about 0.25 to 0.49 s. All-metric dashboard
reads are unchanged. Continuous write volume is the user-visible harm, so
the contract records that trade explicitly.
The migration reuses the crash-safe swap: databases whose identity is
already enforced by a unique index defer the rebuild to startup
maintenance and swap in one transaction, and a database with no unique
index migrates synchronously. Keeping the name idx_metrics_query_all
means a downgrade rebuilds only the lookup index and leaves no orphan.
Refs #1966
A provider could stand the MSP bundle up and evaluate it without asking
anyone, but paying still meant emailing its public key to Pulse and
copying a licence file onto the host by hand. With the licence server's
self-serve purchase path (pulse-pro license-server/provider_msp_purchase),
the control plane can do the whole thing itself.
The control plane now signs a challenge with its lease signing key to
fetch the licence its subscription entitles, validates it exactly as
startup does, requires it to bind this platform's key, and keeps it in
the data directory, because the host licence is a read-only secret
mount. Startup prefers that renewed licence while it validates, so a
paying provider keeps starting after the self-issued evaluation on the
host expires. A changed licence restarts the control plane through the
existing graceful shutdown, since the plan version is read at load by
every workspace-limit, backup and status path; client workspaces keep
running. Preflight, proof, recovery and backup accept either signed
source.
The portal gains owner and admin routes for plan state, checkout, the
Stripe billing portal and an immediate licence refresh, and the plan list
comes from the licence server so it shows only what can be bought, at
Stripe's price. Adds the msp_solo plan: 3 client workspaces, the first
paid step above the 2-workspace evaluation.
The startup-watchdog race repair only converted the capture in service_health_test.go, but startup_watchdog_test.go still polls a plain bytes.Buffer while the watchdog goroutine writes the same buffer through zerolog. go test -race therefore still reports a read/write data race in TestStartupWatchdogFiresAndNamesLastPhase and the rest-0 backend shard stays red. Capture those logs through the same mutex-guarded syncBuffer so the poll and the watchdog write are serialized.
Change-source: pulse-maintainer
TestStartupWatchdogLogsPhaseAndStack polls a bytes.Buffer while the watchdog goroutine writes the same buffer through zerolog, so 'go test -race' reports a read/write data race at service_health_test.go:134 and fails the rest-0 backend shard. Capture the log through a mutex-guarded buffer so the poll and the watchdog write are serialized.
Change-source: pulse-maintainer
pkg/server/server.go is a security-privacy runtime path, so the startup
watchdog commit needs its contract obligation and an accepted proof in the
same reviewed unit. Record the local-only diagnostic boundary in the
security-privacy contract and place the watchdog proof in the
registry-accepted pkg/server/service_health_test.go.
Change-source: pulse-maintainer
Bind the UI/API listener early so port conflicts fail fast, but srv.Serve
only starts near the end of Run. Every synchronous initialization step in
between therefore runs with a live listening socket whose accept queue fills
while nothing accepts: the reported #2129 shape (bound port, non-zero Recv-Q,
no HTTP response). A stall there is invisible because no log line is emitted
while it runs.
Track the last completed startup phase and arm an independent watchdog that
fires if serving has not begun within two minutes, logging the last phase and
every goroutine stack. This makes a recurrence diagnosable from the field log
without an operator sending SIGQUIT. The watchdog is stopped when serving
begins or Run exits and is safe to stop more than once.
Change-source: pulse-maintainer
Address the remaining verified findings from the pkg/metrics/store.go review
(#2079 finding 5 and the lower-impact nits):
- writeBatch now retries the whole transaction on a retryable lock error.
BEGIN is deferred, so contention is hit at the first INSERT or COMMIT; the
old loop retried only Begin() and skipped rows. While the one-time startup
maintenance (deferred identity-index rebuild and auto-vacuum VACUUM) is
active, an extended retry budget keeps upgrade-time history instead of
dropping it after the steady-state budget.
- migrateLegacyHostResourceType probes with an indexed EXISTS before opening a
write transaction, so an already-migrated store does not dirty the WAL on
every boot.
- QueryAll caps the per-series MetricPoint preallocation so a fine requested
step over a long range cannot reserve hundreds of thousands of slots.
Regressions: TestStoreStartupMaintenanceSignalsWriteRetryBudget,
TestStoreWriteBatchExtendsRetryDuringStartupMaintenance,
TestEstimateQueryAllBatchSeriesCapacityCapsPreallocation. The
performance-and-scalability contract records the invariants.
Change-source: pulse-maintainer
The canonical completion guard requires the metrics-store hot-path proof in
pkg/metrics/store_additional_test.go. Move the rollup-gap, upsert-spread,
busy-retry and auto-vacuum regressions there and point the
performance-and-scalability contract at that file. The runtime change and
contract section are in the preceding fix(metrics) commit.
Change-source: pulse-maintainer
Address the remaining verified findings from the pkg/metrics/store.go review
(#2079 findings 2-4 and 6):
- rollupTier no longer leaps its checkpoint over a source gap. Previously a
trailing or interspersed gap advanced the checkpoint to the rollup cutoff, so
a lower tier that backfilled the gap later had those rows stranded below the
upper checkpoint and purged by retention.
- Retry conditions now classify SQLITE_BUSY/LOCKED by driver result code. The
modernc driver reports "database is locked (5) (SQLITE_BUSY)", which the
previous exact-string comparison never matched, so a busy writer dropped its
batch without retrying.
- The metrics upsert preserves an existing rollup row's min/max spread with
COALESCE instead of overwriting it with NULL.
- migrateAutoVacuum pins the pragma and VACUUM to one connection and verifies
the persisted mode, so the one-time conversion cannot silently repeat a
full-file VACUUM on every restart.
Regressions: TestStoreRollupTierBackfillsInterspersedGap,
TestStoreWriteBatchPreservesRollupSpread, TestIsRetryableWriteError,
TestIsRetryableWriteErrorMatchesDriverBusyCode,
TestStoreAutoVacuumPersistsAcrossRestart. The performance-and-scalability
contract records the invariants.
Change-source: pulse-maintainer
The shared release preflight worker exports GITHUB_ACTIONS=true so isolated single-repository checkouts take their private-sibling skip. That flag also made the load/SLO latency guards hard-fail on a contended shared host, where the tests' own documented local-contention skip is the intended behaviour. Distinguish a real GitHub Actions run by GITHUB_RUN_ID, which only a hosted runner sets, so the worker keeps the cross-repo skip while latency overruns skip instead of failing. Hosted CI enforcement is unchanged.
Change-source: pulse-maintainer
The ingestion worker closed writeCh when it observed stopCh. A concurrent WriteWithTier that passed its stopping check, or a WriteBatchSync/WriteBatchBounded caller (which never checks stopping), could then send on the closed channel and panic the process during shutdown. Reproduced deterministically: enqueueWrite after Close panics with send on closed channel, and 8 concurrent monitoring writers racing Close panic in boundedEnqueueAndWait (store.go:1034).
Never close writeCh. Drain already-queued requests non-blockingly, then process them. Late writes land in the buffered channel and are discarded with the store rather than crashing it. Record the invariant in the performance-and-scalability contract and pin it in the accepted store proof file.
Change-source: pulse-maintainer
History requests can occupy every database connection and make cheap
statistics reads wait for seconds. Reserve one read-only connection for
current committed counts, owned and closed by the metrics store.
Keep the SQL, data model, load workload and latency budgets unchanged.
Add deterministic pool-saturation, committed-data and shutdown coverage.
First-run setup saved the configured administrator but left the router authorizer on its startup identity. This split Settings capability checks: administrator-only panels remained visible while API Access and Pulse Intelligence were denied until restart. Keep the captured authorizer aligned when setup commits the identity, and clear its prior bypass on successful development reset.
Cover the exact split with real file-backed RBAC policy and session setup, preserve outsider denial and identity replacement, and synchronise concurrent policy reads. Extension implementations still require independent compatibility review; this is not installation or release acceptance.
Change-source: pulse-maintainer
Update compress and Prometheus common independently of the grouped database proposal. Common requires x/net 0.58.0; all other selected modules and existing checksums remain unchanged. Extend real exporter wire coverage for wildcard, browser and JSON fallback requests alongside gzip, protobuf and optional zstd compatibility.
Contract-Neutral: Dependency maintenance and expanded compatibility coverage only; no Pulse API, deployment policy, toolchain or enabled exporter format changes.
Change-source: pulse-maintainer
Update the coherent six-module Prometheus dependency set without changing other selected versions. Cover Pulse metric labels, protobuf and text scrape negotiation, gzip and existing identity fallback, plus the optional zstd codec. The baseline also leaves zstd disabled, so no production registration change is needed.
Contract-Neutral: Dependency and compatibility-test maintenance only; no public API, deployment, telemetry, privacy or scrape-format behaviour changes.
Change-source: pulse-maintainer
Carry the operator include override in disk reports so server filtering does not discard selected tmpfs mounts again. Keep automatic filtering for unmarked reports and agent exclusion precedence. Cover both ingest paths, collection, forwarding and wire compatibility for #1875.
Change-source: pulse-maintainer
Hosted organisation E2E failures exposed an ambiguity between runtime identity and entitlement provisioning. Assert that Community preserves a granted multi_tenant capability without manufacturing it when absent, so fixture repairs do not wrongly require a private binary or weaken capability gating.
Change-source: pulse-maintainer
An admin-browser update read does not establish access for the monitoring token. Cover the proxied target path, configured token, denial propagation and healthy endpoint retention. Remove the unsupported Sys.Audit diagnostic hint: endpoint access is determined by Proxmox, not general inventory access.
Change-source: pulse-maintainer
Contract-Neutral: Diagnostic wording and method-comment correction plus synthetic regression coverage only; no collection logic, public API, permission policy, freshness or counter semantics change.
Return persisted planning acceptance or refusal inside the investigation turn.
Keep model judgment separate from action authority and preserve accepted action
identity across provider failures. Enforce actor/request idempotency atomically
and retain complete approval and independent verification context.
Preserve unknown disk evidence, stream whitespace and historical resolution
timestamps. Keep conversation scrolling inside its own panel. Record real-model,
disposable-lab and browser qualification with explicit population limits.
Refs #1782
Expose confined, identity-bound filesystem observations through the shared
resource pipeline so investigations can distinguish an exhausted container
mount from unrelated host capacity. Keep unavailable measurements explicit.
Isolate alert-history reads from durable writes and reuse one chronological
fold across polling. Catch up through bounded durable event IDs so simultaneous
readers do not replay every retained snapshot. Retain expired actions when
investigation outcomes move back to needs attention, and keep attached
Assistant context focused.
Record live storage diagnosis, healthy and dependency controls, approved and
rejected Docker outcomes, source-bound browser proof and exact test limits.
Missing-access continuity, VM dispatch completion and remaining Assistant
orchestration defects stay open in the redesign plan.
Live funded qualification found hidden tool results and misleading action
submission outcomes. Share the result-bearing transcript across stored chat
and product history, render the retained evidence, and distinguish captured
proposals from broker acceptance. Keep review usable while Patrol is paused.
Record Gemini route pricing and exact qualification limits. Integrate current
main and repeat browser proof for the incoming login flow. Approved/rejected
recovery remains unqualified without the development command agent.
Release qualification crashed in historical baseline SQLite ingestion during the workloads-summary seed. Preserve a metrics-only diagnostic for the same 84,000-row batch shape, with row-count and integrity checks before and after reopen, without the HTTP or reflection fixtures. This does not reproduce or fix the unexplained crash.
Validation: ten focused runs on Go 1.26.7 and one on Go 1.26.8 passed; a one-repeat race run passed. The three-repeat race run timed out at 180 seconds during its final integrity check and remains retained evidence. Omitting the final seed batch makes the row-count assertion fail.
Change-source: pulse-maintainer
Missing block I/O and container image sizes could become false evidence
for diagnosis. Preserve per-direction counter presence and measured zero
through collection, resource conversion and browser rendering. Separate
new observed history from ambiguous retained disk series without deleting
old rows or changing public metric names.
Keep partial host rates distinct and persist a newly enabled Disk I/O
column across the first preference reload.
rcourtman