13 Commits

Author	SHA1	Message	Date
rcourtman	a85bdcdc9a	Stop a stale agent SMART row lending a replaced disk its identity (#2658) ... ## Problem PR #2648 stopped the poller's previous-evidence lookup from handing a replaced disk the old disk's serial. The linked host agent's SMART merge runs before that lookup and had the same flaw. Its last-resort match is the device path, and a path names a slot, not a disk. When a disk is swapped before the agent reports again, or after the agent went silent and kept its last rows, the agent's row for the old disk matched the replacement. It filled the old serial or, for SAS, promoted it over the Proxmox SAS address. The registry then keyed the replacement to the old disk's canonical resource and SMART history, and it also took the old disk's temperature and FAILED health. For SAS, the registry's same-path join (`resolveLinkedPhysicalDisk`) did the same at the canonical level even when monitoring kept the rows apart. Proxmox's disk inventory (`PVE::Diskmanage`, checked on a PVE 9 node) writes the literal `unknown` for a serial or WWN udev cannot read. The poller recorded it as a collected serial, so the collection state hid the gap, and the merge's `Serial == ""` fill never let the agent's real serial replace it. ## Change - `mergeHostAgentSMARTIntoDisks` refuses a device-path row whose identity contradicts the Proxmox disk's. The new `diskinventory.HardwareIdentityConflict` defines a contradiction: nothing agrees across serial and WWN, and a field both producers report like for like disagrees. - WWNs count for every disk. - Serials count only where Proxmox's serial is the drive's own, as smartctl's is: a disk typed `nvme`, or one presented with SCSI vendor `ATA` (libata, or a SAS HBA's SCSI-ATA translation) and not typed `usb`/`sas`. Elsewhere the Proxmox serial may be a SAS address or SCSI designator (#1595) or a USB bridge's own serial, so it only counts toward agreement. - Spelling differences are not disagreement: udev's underscores for whitespace, a T10 designator for an ATA drive (`ATA`, the model, then the serial), the agent's NAA field spelling, and udev's 64-bit prefix of an NAA 6 WWN. - The refusal runs after the unique-path check, so a second row on the path keeps the match ambiguous. - The merge's WWN step compares normalized identity (`HardwareIdentityMatch`), so the agent's `5-c50-a1b2c3d4` finds the Proxmox `0x5000c500a1b2c3d4` disk. - `resolveLinkedPhysicalDisk` refuses the SAS same-path fallback when the two WWNs name different disks. - The poller records Proxmox's `unknown` serial/WWN as empty, so the serial reads `missing` and the agent can fill it. Retained evidence carries a previous serial only when it is usable, so a pre-upgrade record cannot restore the placeholder. Identity keys already ignored the placeholder. When the agent now fills a disk's serial, that disk's Proxmox-poll SMART samples move from its WWN or path key to the serial key, where the agent's own samples already go. ## Tests - `TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity` (sata and sas) drives the real poller and registry. Proxmox reports `unknown`, the agent fills the serial, the agent goes silent, and the disk is swapped. On main, the SATA replacement takes the old serial and resource; the SAS replacement keeps its own serial in state, but the registry joins it to the old resource with its temperature and FAILED health. - `TestHostAgentSMARTMergeRefusesPathMatchAcrossContradictingIdentity` has one row per producer rule: SAS/SATA/NVMe replacements refused; same-disk SAS without an agent WWN, a renamed SAS disk in the agent's WWN spelling, udev's serial spelling, a USB enclosure (typed `usb` and typed `ssd`), a T10 designator and an NAA 6 volume merged; and an ambiguous path. - `TestRegistrySASPathJoinRefusesContradictingWWN`, `TestProxmoxUnknownDiskIdentityIsRecordedAsUnreported` and `TestHardwareIdentityConflict` cover the rest. - Negative controls via `go test -overlay`: removing each rule fails only its own case. These cover the guard, the WWN normalization, the transport/vendor rule, compaction, the designator rule, the NAA 6 prefix, the registry guard, the ingest normalization and the usable-serial carry. Against main's `monitor.go` and `registry.go`, the end-to-end, merge and registry tests fail. - `TestMergeHostAgentSMARTIntoDisks_PreservesExistingProxmoxIdentity` merged a row with a different WWN by path alone, which the guard now rightly refuses. Its row is now the same disk in the agent's WWN spelling, with all assertions unchanged. - `internal/monitoring`, `internal/unifiedresources` and `pkg/diskinventory` pass in full. ## Not covered - A replacement with no comparable identity still matches its path: no usable serial or WWN at all, or a SAS/SCSI/USB disk with a WWN on one side only. - The registry's seeded source mapping, reused after a restart by the slot-shaped Proxmox source ID, can still hand a swapped disk the old canonical resource. That's a separate follow-up.	2026-10-07 14:01:19 +01:00
rcourtman	bc15c3500b	Stop QEMU default disk serials merging nested Proxmox disks (#2653) ... ## Problem Nested Proxmox clusters (each PVE node is itself a QEMU VM) lose physical disks from inventory. QEMU gives a virtual disk configured without a serial a default one built from its drive ID or a per-VM counter, so every VM built the same way reports the same value: - Proxmox's `/nodes/{node}/disks/list` serial is udev's `ID_SERIAL_SHORT` (`PVE/Diskmanage.pm`). A virtio-scsi disk on a Proxmox VM reports `drive-scsi0` with no `ID_WWN`; on a libvirt VM it is `drive-scsi0-0-0-0`. A `-drive` given no ID falls back to QEMU's automatic ID, `scsi0-hd0` for `if=scsi` and `none0` for `if=none`. - An emulated ATA disk's serial is `QM` plus a five-digit counter (`QM00001`). `diskinventory.IsUsableHardwareID` accepted all of these, so `resourceFromPhysicalDisk` set `MachineID = "drive-scsi0"` on every node and the registry minted one `machine:drive-scsi0` canonical disk and one SMART series for the whole cluster. Three nodes with two such disks each produced 2 canonical disks instead of 6. ## Change - `IsUsableHardwareID` rejects exactly QEMU's default shapes: `drive-scsi` plus digits with at most three `-digits` groups, `scsi<n>-hd<n>`, `none<n>`, and `QM` plus five digits. Real serials that only share a prefix (`DRIVE-ASSET123`, `drive-sata0`, `QM0001`) stay identity. It also strips `naa.`/`eui.`/`wwn-`/`0x` framing before its all-zero and all-F test, so a `0x0000000000000000` WWN is a placeholder too. - TrueNAS provider: a disk took its canonical identity from any non-empty serial, so a placeholder merged disks across appliances. It now uses only a usable serial, and lists superseded IDs for every disk without one. - TrueNAS history keys: for a serial that was already rejected, such as `UNKNOWN`, native TrueNAS history used TrueNAS's own disk identifier, and the unified SMART writer (`syncUnifiedPhysicalDiskMetrics`) substituted the metrics target only for an empty serial, so the samples went to the canonical resource ID. Neither matched the chart reader's key, and rejecting QEMU serials would have put TrueNAS VMs on that path. Native history now falls back to the disk's source ID, and the writer always writes under the resolved metrics target, so writer, native history and reader agree. History lookup keys also drop placeholder serials. ## Upgrade effect A disk reporting one of these values (nested PVE nodes, TrueNAS or agent hosts running as QEMU VMs with virtual disks, TrueNAS disks reporting `UNKNOWN`) changes canonical ID and SMART metric key once. Samples recorded under the old key are not carried over, because that key mixed every disk that shared the value. Path-shaped alert references (Proxmox disk health and wearout, host-agent SMART) are unaffected; anything keyed on the old canonical ID or metric key starts again under the new one. This is recorded in the monitoring and agent-lifecycle contracts. ## Tests - `TestIsUsableHardwareIDRejectsQEMUDefaultDiskSerials`: each default shape is rejected, falls back to the scoped topology ID and never matches itself; prefix-sharing real serials stay usable. `TestPreferredIDRejectsPlaceholderHardwareIdentity` gains the framed all-zero/all-F WWNs. - `TestNestedProxmoxDefaultQEMUSerialsStayPerNode`: 3 nodes × (`drive-scsi0`, `QM00005`, `drive-scsi0-0-0-1`) give 9 canonical disks, 9 slots and 9 metric keys, every SMART writer key is a key readers resolve, and agent SMART source IDs stay per host. - `TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey`: two appliances with `UNKNOWN` and `drive-scsiN` serials keep one canonical disk each, and the SMART writer's samples are readable under each disk's metrics target. - `TestProviderPlaceholderDiskSerialsKeyNeitherIdentityNorHistory`: placeholder serials key no identity, list superseded canonical IDs, drop out of native history lookup keys, and native history lands under the registry's metrics target. - Negative controls via `go test -overlay`, one runtime file reverted at a time: base `identity.go` fails all three round trips (3 canonical disks instead of 9, 9 TrueNAS disks instead of 16); base `provider.go` fails the identity and history assertions; base `monitor.go` fails the SMART-history-under-reader-key assertion; reverting only the native history key, the superseded-ID gate or the lookup-key filter fails its own provider assertion. - `pkg/diskinventory`, `internal/truenas/...`, `internal/unifiedresources/...` and `internal/monitoring` pass in full; `internal/api/...` disk, hardware, serial and TrueNAS tests pass. Stacked on #2648 (`fix/pve-disk-serial-carry`), whose commit this branch carries until it merges. Not in this change: the registry's merge of disks across parents on a canonical-ID collision (separate change), and a pre-existing writer/reader key mismatch for controller-member disks without a usable serial or WWN (`PhysicalDiskMetaMetricID` appends topology to a source ID that already carries it), which is tracked separately.	2026-10-07 13:22:34 +01:00
rcourtman	8d00d54740	Stop a stale agent SMART row lending a replaced disk its identity ... The linked host agent's SMART merge falls back to matching a Proxmox disk by device path, and a path names a slot, not a disk. When a disk is swapped before the agent reports again, or after the agent went silent and kept its last rows, the row for the old disk matched the replacement. It filled or, for SAS, promoted the old serial onto it, which keyed the replacement to the old disk's canonical resource and metric history, and lent it the old disk's temperature and failed health. For SAS the registry's same-path join did the same at the canonical level even when monitoring kept the rows apart. Both now refuse a row whose identity contradicts the disk's, judged only on what both producers report like for like. WWNs count for every disk. Serials count only where Proxmox's serial is the drive's own, as smartctl's is: NVMe namespaces, and disks presented with SCSI vendor "ATA", whose serial udev's ata_id reads from IDENTIFY. Elsewhere Proxmox's serial may be a SAS address, a SCSI designator or a USB bridge's own serial (#1595). Reporter spellings are not disagreement: udev writes whitespace in serials as underscores, its T10 designator for an ATA drive puts "ATA" and the model before the drive serial, and its ID_WWN keeps only the first 64 bits of an NAA 6 identifier. The WWN match step compares normalized identity too, so the agent's 5-c50-a1b2c3d4 spelling finds the Proxmox 0x5000c500a1b2c3d4 disk. Proxmox's disk inventory reports a serial or WWN udev cannot read as the literal "unknown". The poller recorded it as a collected serial, so the collection state hid the gap and the agent's serial never filled it. It is now recorded as empty, and retained evidence carries only a usable previous serial so an older record cannot restore it. Identity keys already ignored the placeholder.	2026-10-07 12:16:36 +01:00
rcourtman	41cea2ffab	Stop a replaced PVE disk inheriting the old disk's evidence (#2648) ... ## Problem The Proxmox disk poller keeps evidence it could not collect this poll by looking up the previous poll's record of the disk: by serial, then WWN, then the Proxmox source ID (`instance-node-devpath`), then the device token. The source ID and device token name a slot, not a disk, and a replacement in the same slot keeps both. So: - a replacement reporting a different serial or WWN took the old disk's retained temperature, controller, pool and I/O; - a replacement record with an empty serial also took the old disk's serial. The registry keys disks on serial, so the new disk became the old disk's canonical resource and SMART metric series, and every later poll carried the borrowed serial forward. Current Proxmox reports a missing serial as the literal `unknown`, which the poller records as reported, so the serial copy needs a producer that sends an empty serial, such as the host-agent fallback rows. The evidence copy happens with real Proxmox data. ## Change - `previousPhysicalDiskEvidence` matches by hardware identity through `diskinventory.HardwareIdentityMatch` (serial first, then WWN, each against both fields of the earlier record), and refuses a slot match when both records carry a serial or WWN and none agrees. Missing identity is still not a conflict: placeholders such as `unknown`, and an empty serial, keep the slot's evidence as before. - The registry's merged disk view keeps the host agent's WWN, which the agent spells as unpadded NAA 5 fields (`5-c50-a1b2c3d4`, `internal/hostagent/smartctl.go` `formatWWN`) while Proxmox reports `0x5000c500a1b2c3d4`. `HardwareIdentityMatch` now re-pads that spelling for WWN fields only, so one disk seen by both reporters is not taken for a replacement. The registry's linked-disk join uses the same comparison. ## Tests - `TestPhysicalDiskReplacementInSameSlotDoesNotInheritPreviousSerial` drives the poller and registry across two polls after a swap; on base the replacement inherits the old serial and canonical resource. Its controls keep a serial-omitting same disk on its resource. - `TestPreviousPhysicalDiskEvidenceRejectsSlotMatchAcrossConflictingIdentity` covers renamed devices matched only by identity, PVE vs agent WWN spellings, SAS address serials with a matching WWN, placeholders, and conflicting serial/WWN by source ID and by device token. - Negative controls via `go test -overlay`: on base the replacement, conflict and renamed-device cases fail; with the guard but without the WWN re-padding, the same-disk PVE/agent WWN cases fail. - `pkg/diskinventory`, `internal/unifiedresources` and `internal/monitoring` pass in full. Follow-ups, not in this change: the host-agent SMART merge's device-path fallback can still fill or promote a serial from a stale SMART row; QEMU default serials (`drive-scsi0`) collapse nested Proxmox nodes' disks into one resource; and the registry merges disks across parents on a canonical-ID collision.	2026-10-07 12:12:23 +01:00
rcourtman	dd8bfa4db8	Stop QEMU default disk serials merging nested Proxmox disks ... QEMU gives a virtual disk configured without a serial one built from its drive ID or a per-VM counter, so every VM built the same way reports the same value. Proxmox's disks/list serial is udev's ID_SERIAL_SHORT, which for a virtio-scsi disk is "drive-scsi0" (libvirt: "drive-scsi0-0-0-0"), and an ATA disk's serial is "QM00001". IsUsableHardwareID accepted these as hardware identity, so every node of a nested Proxmox cluster minted one machine:drive-scsi0 canonical disk and one SMART series, and all but one node lost the disk from inventory. IsUsableHardwareID now rejects exactly QEMU's default shapes: drive-scsi plus digits with at most three -digits groups, QEMU's automatic drive IDs scsiN-hdN and noneN, and QM plus five digits. Real serials that share a prefix, such as DRIVE-ASSET123, stay identity. It also strips WWN framing before the all-zero test, so a 0x0000000000000000 WWN is a placeholder too. The TrueNAS provider set a disk's canonical identity from any non-empty serial, so placeholders merged disks across appliances. It now uses only a usable serial. For a serial that was already rejected, such as UNKNOWN, native TrueNAS history and the unified SMART writer disagreed with the chart reader: native history used TrueNAS's disk identifier, and the writer substituted the metrics target only for an empty serial, so a placeholder sent samples to the canonical resource ID. Rejecting QEMU serials would have put TrueNAS VMs on that path. Both now use the key the registry's metrics target resolves, the disk's source ID, and native history lookups no longer route by placeholder serials. Disks that report one of these values change canonical ID and SMART metric key once; samples under the old key are not carried over because that key mixed every disk that shared the value.	2026-10-07 11:51:16 +01:00
rcourtman	9b644b3d49	Stop a replaced PVE disk inheriting the old disk's evidence ... The disk poller keeps evidence it could not collect this poll by looking up the previous record, first by serial and WWN, then by the Proxmox source ID and device token. Those last two name a slot, not a disk, and a replacement in the same slot keeps both. So a replacement reporting a different serial or WWN took the old disk's retained temperature, controller, pool and I/O. A replacement record with an empty serial also took the old serial. The registry keys disks on serial, so the new disk became the old disk's canonical resource and SMART metric series, and each later poll carried the borrowed serial forward. Match on hardware identity through diskinventory.HardwareIdentityMatch, serial first then WWN, and refuse a slot match when both records carry a serial or WWN that share no value. Missing identity is still not a conflict, so Proxmox's literal "unknown" and other placeholders keep the slot's evidence as before. The registry's merged disk view keeps the host agent's WWN, which the agent spells as unpadded NAA 5 fields ("5-c50-a1b2c3d4") while Proxmox reports "0x5000c500a1b2c3d4". HardwareIdentityMatch now re-pads that spelling for WWN fields, so the same disk seen by both reporters is not mistaken for a replacement.	2026-10-07 11:23:34 +01:00
Richard Courtman	ff67e38968	Pair a merged disk's temperature with its rows' own states ... The registry merges a physical disk's rows one step at a time, choosing the temperature by source preference and merging collection states separately. The pairing rule from #2618 was exact for two rows only: with three or more, an earlier step's single merged state stood in for its rows. A silent agent, an offline Unraid inventory and a stale Proxmox copy could leave a withdrawn 41 labelled collected, and a legacy agent's stateless reading could pick up another row's "unsupported". The merged disk now keeps each row's current reading and state (merge-time only, never serialized), and the shown value takes its state from the readings of that value: a collected reading first, then the agent's own report over a copy. A copy of an agent source's state gives way to the agent's own withdrawal, including a legacy agent's sourceless one, and a value source preference keeps after its row reported again keeps its presented state under that row's later withdrawal. Value selection is unchanged.	2026-10-07 11:18:10 +01:00
rcourtman	95b29286dd	Stop serving retained disk temperatures as current readings ... Normalization may keep a disk temperature it did not collect (a disk in standby, a host agent past its reporting lease) as long as its collection state says so. Two consumers still presented that value as current. The metrics-history handler answered an empty disk range with the disk's temperature as a "live" point at now, regardless of its state. Since the history writers stopped recording retained temperatures, a short-range chart for a silent agent's disk or a sleeping disk showed the retained value as a fresh reading. Mock mode also seeded synthetic series from it and padded sparse stored series out to now. All of these now require a temperature collected now (diskinventory.TemperatureCollected, the shared form of the rule the history writers apply). Unraid array-inventory rows carry no per-field provenance, so a temperature taken from one looked collected after the agent went silent. The adapter now derives its state from the host lease that ExpireHostTelemetry already records, and from the disk's spin state: unavailable once the host is offline or the disk is spun down. A SMART row that borrows the inventory reading takes that state, including available. The native row records only the withdrawing states, because the registry merge picks temperature and state separately and an available claim from it would mask the SMART row's own state. A withdrawn inventory temperature no longer counts as heat in disk risk.	2026-10-06 20:16:38 +01:00
rcourtman	549161bf6f	Stop presenting a silent agent's disk readings as collected ... A host agent that stops reporting keeps its last SMART rows in state. The Proxmox physical-disk merge copied that agent's temperature and I/O counters into the disk together with its "available" collection state, the registry merged the agent's retained rows into the canonical disk the same way, and every full disk poll wrote the frozen temperature to SMART history as a new sample. A dead agent's disk temperatures therefore stayed "collected" in the API and kept extending the history chart indefinitely. Expiring the agent's reporting lease is now the point where those readings stop being current. ExpireHostTelemetry keeps the SMART temperature and I/O values as last-known evidence but marks them unavailable ("host agent stopped reporting"), as the physical-disk contract asks for retained evidence. Because MergeStatus lets any available state win, a Proxmox disk row still holding the earlier copy would mask that, both on skipped disk polls and when the PVE host is down and never disk-polled again. diskinventory.MergeReportedStatus lets a source's later word on its own field supersede the availability it supplied earlier; the monitoring merge and both registry ingest orders use it. Every SMART temperature history writer now records a temperature only when its current state is collected, so a retained reading (an agent past its lease, or a disk in standby) is not written as a new sample. Agents before 6.2 send no collection provenance, so their merged temperature is recorded as collected from "host_agent" instead of inheriting Proxmox's "unsupported", keeping their history intact. Expiry is now compare-and-set on the report time the evaluation judged stale, so a report accepted in between is not expired. SMART attributes, wearout and health stay last-known on purpose: they are durable health evidence, and dropping them would resolve disk-health and low-life alerts when an agent goes quiet. Disk temperature alerts are evaluated from the agent's own report, so they never read this merged value. The node-temperature path (agent sensors feeding node.Temperature.SMART, which the disk merge labels proxmox_node_smart) is gated by the agent's sighting in #2582; a carried node reading can still stamp a disk as collected and belongs to that path.	2026-10-06 16:32:43 +01:00
Pulse Test	674ca18968	fix(unifiedresources): merge PVE and agent rows for the same RAID array volume ... The PVE disks/list poll and the agent SMART report both observe a RAID controller's exported volume, but PVE surfaces its NAA identifier as a bare-hex serial while smartctl reports the same value as a naa.-prefixed WWN with no serial. The linked-disk join compared serial to serial and WWN to WWN verbatim, so the volume rendered twice (/dev/sda and sda) with independent metric histories. Hardware identity comparison now normalizes reporter framing (naa., eui., wwn-, 0x prefixes, case) and folds serial and WWN together before comparing. Values are never truncated: sibling volumes on one controller share their leading WWN bytes, and a truncated udev ID_WWN must stay unequal to a full identifier. Placeholder serials no longer count as an identity match. Refs #1720	2026-08-28 06:20:10 +01:00
rcourtman	6a527ba9f2	Fix Proxmox physical disk inventory continuity	2026-07-23 22:05:09 +01:00
rcourtman	4f43878ebb	Add focused branch coverage and repair infrastructure source contract ... Contract-Neutral: test-only coverage and source-contract assertions; no runtime or contract behavior changed	2026-07-20 22:41:19 +01:00
rcourtman	b6a74576bc	Integrate trust-gate reliability fixes	2026-07-20 16:03:29 +01:00