mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
Merge branch 'main' into fix/node-drawer-temperature-records
This commit is contained in:
pulse-triage[bot]
@@ -728,6 +728,12 @@ into monitoring's models. Absent evidence has to carry its declared sentinel
|
||||
rather than a zero value that reads as a real measurement: an absent
|
||||
physical-disk view projects `Wearout` as `unifiedresources.WearoutUnreported`,
|
||||
never `0`, which would announce a spent disk the agent never reported.
|
||||
When the host heartbeat lease expires, `State.ExpireHostTelemetry` keeps that
|
||||
agent's SMART temperature and I/O counters as last-known values but marks them
|
||||
`unavailable` ("host agent stopped reporting"); the agent's next accepted
|
||||
report replaces them. Expiry is compare-and-set on the report time the offline
|
||||
sweep judged stale, so a report admitted between that judgement and the expiry
|
||||
is never expired.
|
||||
|
||||
An enabled availability target assigned to a host agent creates an
|
||||
agent-lifecycle lease for that exact target/agent pairing. First assignment
|
||||
@@ -1262,6 +1268,10 @@ update, profile rollout, command reachability, or fleet-control authority.
|
||||
lifecycle presentation evidence. It keeps mock active-alert and history
|
||||
payloads aligned with the canonical alert contract, but it cannot enroll,
|
||||
identify, link, command, remove, or otherwise grant authority to an agent.
|
||||
The optional `models.Alert.Resolution` field is the same kind of
|
||||
evidence: it says a resolved node alert now belongs to a linked agent's
|
||||
own alert and names that agent as display text, but the link itself stays
|
||||
the agent lifecycle's, and the field cannot create, change or authorize it.
|
||||
27. `internal/monitoring/monitor.go` shared with `monitoring`: monitor construction owns both monitoring runtime initialization and fail-closed agent lifecycle journal hydration before report admission.
|
||||
28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.
|
||||
29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.
|
||||
@@ -8206,6 +8216,21 @@ or create agent continuity evidence. The removal lifecycle proof keeps a
|
||||
removed host blocked across a PBS storage sync so metric timestamp maintenance
|
||||
cannot become an accidental re-enrollment transition.
|
||||
|
||||
### Mock fixture agents reuse only the removal alert boundary
|
||||
|
||||
`internal/monitoring/monitor.go` now holds the set of fixture agents the last
|
||||
mock alert pass evaluated. When a runtime mock config change drops an agent
|
||||
from the estate, `evaluateMockHostAgents` routes it through the alert
|
||||
manager's `HandleHostRemoved`, the same alert boundary a deleted live agent
|
||||
crosses. Leaving mock mode routes every fixture agent through the same call,
|
||||
which also releases their hostname deduplication. Neither path writes a
|
||||
removal tombstone, revokes a token, touches continuity evidence or admits a
|
||||
report: fixture agents have no credentials or durable identity, and real
|
||||
reports stay discarded while mock mode is on.
|
||||
`TestMockHostAgentLeavingFixtureUsesRemovalLifecycle` in
|
||||
`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` pins the
|
||||
alert cleanup.
|
||||
|
||||
### Deploy enrollment swaps credentials as one durable transition
|
||||
|
||||
A deploy bootstrap token remains the live credential until Pulse can durably
|
||||
@@ -8782,3 +8807,15 @@ no update status, instead of reporting "no update available" (#2353). Agent
|
||||
registration, enrolment, install, update, removal and report identity are
|
||||
unchanged; the per-image memo lives in the registry checker and is pruned
|
||||
each collection cycle to the images in use.
|
||||
|
||||
### Host snapshots report a linked agent's own heartbeat
|
||||
|
||||
`internal/monitoring/monitor.go` changed only so a host produced from the read
|
||||
state carries the agent source's own last report and reads offline past the
|
||||
agent's reporting lease, even when the agent's row is merged with a Proxmox
|
||||
node that PVE polling keeps fresh. The agent connection on the Connections
|
||||
list and the update-readiness agent-continuity check therefore age a silent
|
||||
agent from its last report instead of from the PVE poll, and mark it stale at
|
||||
their own heartbeat cutoff (`fleethealth.AgentStaleThreshold`, which is
|
||||
separate from the monitoring reporting lease). Agent registration, enrolment,
|
||||
install, update, removal and report identity are unchanged.
|
||||
|
||||
@@ -4638,7 +4638,22 @@ summaries became the alarm name still carry "<entity> <moref> has VMware alarm
|
||||
<name> (<colour>)", so the title strips that wording to the same alarm name.
|
||||
Patrol's seed context lists active and recently resolved alerts by resource:
|
||||
each line names the alert's resource unless the message already does, because
|
||||
provider incidents such as vCenter alarms say only what was flagged. The projection also collapses open/resolved churn:
|
||||
provider incidents such as vCenter alarms say only what was flagged.
|
||||
A recently resolved alert that closed without recovering, such as a node metric
|
||||
handed to its Pulse agent, must not read as a recovery to Patrol, the
|
||||
assistant or the alert tool: Patrol's seed writes `closed <ago>. <summary>` in place of `resolved <ago>`, the
|
||||
assistant's alert context writes `closed <ago> ago) <summary>`, the
|
||||
`pulse_alerts` `resolved` action returns the summary as `resolution` with
|
||||
`successor_resource_id`, and incident memory titles the close with the summary
|
||||
(`alertResolvedEventSummary`, or the `alert_resolution` resource-change
|
||||
metadata when it projects from the canonical timeline).
|
||||
`TestSeedHealthAndAlerts_HandoverIsNotARecovery`,
|
||||
`TestService_buildAlertContext_HandoverIsNotARecovery`,
|
||||
`TestExecuteListResolvedAlertsReportsHandover` and
|
||||
`TestIncidentStore_HandoverCloseIsNotARecovery` pin it. Performance reports
|
||||
(`pkg/reporting`, the report narrator) do not carry the resolution yet: a
|
||||
report generated within the five-minute recently-resolved window after a
|
||||
handover still lists that alert as resolved. The projection also collapses open/resolved churn:
|
||||
when a record's timeline holds at least four open-to-resolved or
|
||||
resolved-to-open transitions inside the last 24 hours, the item carries a
|
||||
`flapping` summary (transition count, window, first and latest transition) and
|
||||
|
||||
@@ -780,6 +780,39 @@ must migrate through `internal/alerts/operational_contract.go` and name their
|
||||
limited provenance honestly rather than inventing confirmed provider evidence.
|
||||
Acknowledgement remains distinct from resolution, and every resolution
|
||||
transition references recovery evidence separate from its trigger evidence.
|
||||
A resolved alert may carry a typed `AlertResolution`
|
||||
(`internal/alerts/resolution.go`) when it closed without its condition being
|
||||
observed to clear. The one reason so far, `moved_to_agent`, describes a metric
|
||||
alert whose resource's linked Pulse agent now owns the metric.
|
||||
`releaseCanonicalMetricAlert` is the disabled-threshold release extracted from
|
||||
`evaluateCanonicalMetricAlert`: a caller releasing a metric for any other
|
||||
reason passes a resolution, and `clearAlertWithResolution` stamps a copy on the
|
||||
removed alert before it is resolved. The resolved callback snapshot, the
|
||||
history row, the resolved event-log entry (the reason code, with `Summary()` as
|
||||
its message) and the Alert JSON then carry it. Resolved consumers still receive
|
||||
the close, because integrations close their incident by alert ID. The
|
||||
synthesized closing evidence carries the resolution reason and summary instead
|
||||
of `legacy_alert_recovery_projection`, and the resolve transition carries the
|
||||
summary as its `reason`. Its cause is `ownership_transferred`, an
|
||||
`operationaltrust.TransitionCause` that may only enter `resolved` and only with
|
||||
evidence (`resolvedTransitionCause` in `operational_contract.go`), so Patrol's
|
||||
timeline never labels a handover "Recovery evidence" and alert-quality
|
||||
telemetry does not count a handover during a snooze as resolved while snoozed.
|
||||
A nil resolution leaves the disabled-threshold path and every recovery on
|
||||
`recovery_evidence`.
|
||||
`TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and
|
||||
`TestResolvedHistoryRowKeepsResolutionWithoutEventLog` in
|
||||
`internal/alerts/operational_contract_test.go` pin this.
|
||||
The in-app readers carry it too. `GetRecentlyResolved` projects it into
|
||||
`models.Alert.Resolution` (reason, successor and the `Summary()` text) for the
|
||||
state snapshot, the websocket and the assistant and Patrol prompts, pinned by
|
||||
`TestRecentlyResolvedCarriesHandoverResolution` in `history_test.go`. The
|
||||
alerts history lists such a row as `moved to agent` instead of `resolved`, even
|
||||
when it was acknowledged, with the account in the badge title and on the phone
|
||||
card (`buildAlertHistoryItems`, `getAlertResolutionDetail`). The resolved
|
||||
lifecycle change written to the resource timeline carries the summary as its
|
||||
reason and the code as `alert_resolution` metadata, so the row's incident
|
||||
timeline says the alert moved instead of "Alert resolved: <breach message>".
|
||||
|
||||
## Extension Points
|
||||
|
||||
@@ -1592,6 +1625,75 @@ per-alert rate limiting; future notification-gating changes should extend that
|
||||
policy owner rather than burying new checks inside metric or resource-specific
|
||||
evaluators.
|
||||
|
||||
### Agent disk temperature alerts clear when their threshold is off
|
||||
|
||||
Host-agent SMART disk temperature alerts (`diskTemperature`, resource
|
||||
`agent:<host>/disk_temp:<device>`) follow one effective threshold: the
|
||||
resolved agent `DiskTemperature`, refined by the disk type's `DiskTempByType`
|
||||
entry unless an explicit host or linked-resource override set it
|
||||
(`hostDiskTemperatureThresholdNoLock` in `internal/alerts/host.go`). When the
|
||||
host's resolved `DiskTemperature` is off, `CheckHost` clears every disk
|
||||
temperature alert on the host. A disabled `DiskTempByType` entry clears only
|
||||
the alerts of disks of that type, through the disabled metric spec. The
|
||||
all-agents switch, a disabled host override, agent removal and confirmed
|
||||
agent offline clear them with the host's other disk alerts. Before this,
|
||||
nothing cleared them once the agent-level threshold was off.
|
||||
|
||||
A config save judges an open disk temperature alert against the same
|
||||
threshold (`resolveHostAlertThresholdsNoLock` refines it by the alert's
|
||||
`diskType` metadata, and `metric_runtime.go` classifies `diskTemperature` as a
|
||||
threshold metric), so turning the host's `DiskTemperature` off resolves the
|
||||
alert immediately. An alert persisted before `CheckHost` recorded `diskType` (May
|
||||
2026) carries no disk type until its next firing evaluation, so a save judges
|
||||
it against the lowest enabled trigger any of the host's disks can use: the
|
||||
agent threshold or a `DiskTempByType` entry
|
||||
(`lowestHostDiskTemperatureThresholdNoLock`). A save therefore cannot resolve
|
||||
an alert its disk type would still fire.
|
||||
|
||||
For a metric alert that reaches the threshold comparison in
|
||||
`reevaluateActiveAlertsLocked` (docker-host alerts return before it), a config
|
||||
save resolves an alert whose reading is
|
||||
below the new trigger. That includes readings inside a valid recovery band,
|
||||
which the evaluator would hold. A clear level at or above the trigger is no
|
||||
recovery band (`buildCanonicalMetricSpec` drops it), and a reading at the
|
||||
trigger stays firing, as it does in the reducer, so a save no longer resolves
|
||||
an alert the next evaluation raises again.
|
||||
`TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff`,
|
||||
`TestHostDiskTemperatureAlertsClearWhenAgentLeaves`,
|
||||
`TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold`,
|
||||
`TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType` and
|
||||
`TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand` in
|
||||
`internal/alerts/host_unraid_lifecycle_test.go` pin these paths.
|
||||
|
||||
### Agent disk temperature alerts clear when their disk leaves the report
|
||||
|
||||
`CheckHost` also clears a disk temperature alert, and drops its pending
|
||||
threshold run, once its device is missing from three consecutive non-empty
|
||||
SMART lists (`cleanupHostDiskTemperatureAlerts`, a seen map keyed by resource
|
||||
ID like `cleanupHostCustomSensorAlerts`). A removed, replaced or renamed disk
|
||||
never sends the reading that would resolve its alert, and
|
||||
`shouldPreserveAlertOutsideNodeCleanup` exempts `agent:` alerts from node
|
||||
cleanup, so before this only the 24-hour stale-alert sweep in
|
||||
`cleanupStaleMaps` resolved it. One omission is not proof the disk left: the
|
||||
Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk whose
|
||||
probe fails, so the count restarts whenever the disk is reported again. Three
|
||||
reports in a row of failed probes on such a disk still clear its alert.
|
||||
|
||||
An empty SMART list skips this cleanup and does not count toward the three,
|
||||
because the agent omits the list when SMART collection fails or is
|
||||
unsupported. A listed disk in standby or without a temperature also holds its
|
||||
alert, since neither shows the disk cooled: standby reports avoid waking the
|
||||
disk, and most failed Linux probes still list the disk with identity only.
|
||||
SMART health alerts hold through standby the same way. A held alert resolves
|
||||
through the normal recovery path, including any configured recovery delay,
|
||||
once the disk reports cool readings again; with no evaluation for 24 hours,
|
||||
the stale-alert sweep resolves it. Turning the threshold off, or removing the
|
||||
agent, drops the host's pending disk temperature runs and absence counts as
|
||||
well as its alerts.
|
||||
`TestCheckHostClearsDiskTemperatureAlertWhenDiskLeavesSMARTReport` in
|
||||
`internal/alerts/host_unraid_lifecycle_test.go` pins the clear, the holds, the
|
||||
restarted count and the pending-run and count cleanup.
|
||||
|
||||
### Configured flapping thresholds remain reachable
|
||||
|
||||
Every accepted positive `FlappingThreshold`, including values above ten, must
|
||||
@@ -1803,11 +1905,21 @@ host-agent deduplication bookkeeping, and instance-scoped node display-name
|
||||
cache updates; future Proxmox node alert behavior should extend that resource
|
||||
checker owner rather than expanding the central Manager file.
|
||||
When a reporting host agent is linked to a node (`LinkedNodeID`, the monitor's
|
||||
automatic or operator-set identity decision), the agent resource owns that
|
||||
machine's CPU, memory and disk usage alerts, and `CheckNode` releases its own
|
||||
copies through the disabled-threshold path every cycle: the pending run is
|
||||
dropped and any node alert still open from before the link is resolved, never
|
||||
left frozen until the agent goes offline. Deduplication keys on that link,
|
||||
automatic or operator-set identity decision), the agent resource owns each
|
||||
CPU, memory or disk usage metric it actually evaluates, and `CheckNode` releases
|
||||
its own copy of those metrics through the disabled-threshold path every cycle:
|
||||
the pending run is dropped and any node alert still open from before the link is
|
||||
resolved, never left frozen until the agent goes offline. `CheckHost` registers
|
||||
the link only after evaluation, recording which metrics the agent is configured
|
||||
to evaluate (a live CPU or memory threshold; for disk, a live threshold on the
|
||||
summary filesystem `models.SummaryDisk` picks, which is also what the linked
|
||||
node's disk metric reports), and removes it while agent alerts are disabled.
|
||||
Ownership follows configuration, not one report's data, so a missing agent
|
||||
reading keeps the agent's alert open instead of handing the metric back to the
|
||||
node for a cycle. With several agents linked to one node, a metric is owned when
|
||||
any of them evaluates it. A metric no agent evaluates stays with the node, so
|
||||
deduplication never leaves a machine unmonitored.
|
||||
Deduplication keys on that link,
|
||||
never on a hostname match, so a same-named node in another instance keeps its
|
||||
alerts, an agent reporting an FQDN still dedups its node, and an operator unlink
|
||||
hands the alerts back. CPU temperature has no host-agent metric, so the node
|
||||
@@ -1826,7 +1938,9 @@ instance name in `Instance`.
|
||||
`TestCheckNodeReleasesOpenMetricAlertWhenHostAgentRegisters`,
|
||||
`TestCheckNodeMissingTemperatureDoesNotResolveOpenAlert`,
|
||||
`TestCheckNodeMissingTemperatureInterruptsTimingRuns` and
|
||||
`TestConfigSaveKeepsNodeTemperatureAlertOverTrigger` in
|
||||
`TestConfigSaveKeepsNodeTemperatureAlertOverTrigger` and
|
||||
`TestCheckNodeKeepsUsageMetricsTheAgentDoesNotEvaluate`,
|
||||
`TestCheckNodeUsageOwnershipFollowsWhatAgentsEvaluate` in
|
||||
`internal/alerts/threshold_resolution_shared_test.go`, and
|
||||
`TestHostAgentDeduplicationFollowsNodeLink` in
|
||||
`internal/alerts/host_dedup_test.go`, pin these rules.
|
||||
@@ -2489,13 +2603,23 @@ reads that ahead of every fallback. The menu's owner copy comes from
|
||||
`frontend-modern/src/utils/resourceMonitoringPolicy.ts`, where an explicit
|
||||
platform outranks the resource-type guess and the agent-removal copy is kept
|
||||
to the agent's own machine: libvirt VMs and Unraid arrays in resource drawers
|
||||
get the neutral source-system copy. Agent disk-usage, SMART disk and Unraid
|
||||
array alerts still carry their host's `agent` metadata, so their menu names
|
||||
the Pulse agent until those producers emit child-resource metadata. Alerts
|
||||
restored from before `platformType` existed gain it at their next full
|
||||
evaluation; a metric alert held between its clear and trigger thresholds keeps
|
||||
the earlier fallback until then. Menu sentences open with the owner
|
||||
label capitalised.
|
||||
get the neutral source-system copy. Host-agent alerts
|
||||
(`internal/alerts/host.go`) stamp `platformType` `agent`, and only the
|
||||
machine's own alerts keep `resourceType` `agent`: filesystem,
|
||||
disk-temperature and SMART alerts carry `agent-disk`, RAID and Unraid array
|
||||
alerts `agent-storage`, and custom-sensor alerts `agent-sensor`, because the
|
||||
menu writes policy to their child resource id, never to the machine. Each
|
||||
child type keeps `agent` among its `CanonicalResourceTypeKeys`, and
|
||||
configuration re-evaluation treats agent policy as final for host-agent
|
||||
alerts (TrueNAS systems and vSphere hosts keep their platform policy), so the
|
||||
storage and guest switches cannot resolve a SMART, RAID or sensor alert that
|
||||
the next agent report would re-raise. A metric alert held between its clear
|
||||
and trigger thresholds still refreshes `resourceType` and `platformType` from
|
||||
its producer while its value, message and last-seen time stay at the last
|
||||
breach, so a restored alert that is still firing or held picks up the current
|
||||
classification at its next evaluation; one that resolves on that evaluation
|
||||
keeps the stored classification in its resolved record. Menu sentences open
|
||||
with the owner label capitalised.
|
||||
The retired dashboard recent-alert panel must not be reintroduced as a
|
||||
parallel alert surface. Alert summary/tone copy belongs to the alert overview
|
||||
presentation owner, and any future compact alert surface must compose the
|
||||
@@ -2709,7 +2833,8 @@ legacy alert timestamp.
|
||||
Suppression is bounded and reasoned, leaves the default active queue, and
|
||||
remains inspectable. Expiry or explicit unsuppression returns the record to its
|
||||
detector-owned state; it never resolves it. Only fresh sufficient recovery
|
||||
evidence may enter resolving, and only detector recovery may resolve.
|
||||
evidence may enter resolving, and only detector recovery, or a handover
|
||||
recorded as `ownership_transferred` with its closing evidence, may resolve.
|
||||
|
||||
Per-alert snooze is the customer-facing bounded suppression contract, not a
|
||||
page-local timer. `SnoozeAlert` writes the same canonical operational record,
|
||||
|
||||
@@ -661,6 +661,13 @@ transitions. Older payloads without these optional fields remain readable;
|
||||
frontend consumers use `frontend-modern/src/types/operationalTrust.ts` as the
|
||||
single TypeScript projection rather than recreating lifecycle or evidence
|
||||
enums locally.
|
||||
`TransitionCause` includes `ownership_transferred` for a close whose condition
|
||||
another record now owns, such as a node metric handed to its Pulse agent; it
|
||||
may only enter `resolved`. Alert payloads (history, state snapshot and
|
||||
websocket) may also carry an optional `resolution` (`reason`,
|
||||
`successorResourceId`, `successorName`, and on state payloads the alert
|
||||
engine's `summary`), typed once as `AlertResolution` in
|
||||
`frontend-modern/src/types/api.ts`. Its absence means an ordinary recovery.
|
||||
|
||||
## Shared Boundaries
|
||||
|
||||
|
||||
@@ -4995,7 +4995,8 @@ name, so it keeps the shared single-line rhythm. Drawers for unified resources
|
||||
websocket `activeAlerts` map through `getAlertsForUnifiedResource`
|
||||
(`frontend-modern/src/utils/alerts.ts`), most severe first, never from a
|
||||
resource-embedded list, and show each alert's resource name and metric beside
|
||||
its text like the other drawers. Their table rows take the platform host rows'
|
||||
its text like the other drawers. Their table rows (Docker hosts, Machines,
|
||||
Kubernetes nodes) take the platform host rows'
|
||||
red (critical) or yellow tint from the same set through
|
||||
`getUnifiedResourceAlertStyles`, only for unacknowledged alerts and never on
|
||||
an expanded row, so a row and its drawer agree. The remaining Overview rows are additive
|
||||
@@ -5203,6 +5204,15 @@ and the platform alert tables pass it as `now` to `formatRelativeTime` /
|
||||
data refreshes, so an age computed only at render froze ("2h ago" three hours
|
||||
later) whenever its timestamp did not change. A relative-age cell must read the
|
||||
shared clock rather than `Date.now()` at render or a table-local interval.
|
||||
That rule covers every relative age, not only `PlatformTableRelativeTimeValue`
|
||||
cells: a value formatted through `formatPlatformTableRelativeTimeValue` or
|
||||
`formatRelativeTime` (compact phone ages, labelled Detail ages, row subtitles
|
||||
and tooltips, an open drawer's summary rows and annotations) passes `now` from
|
||||
the owning component's `useRelativeTimeNow`, and a countdown or freshness band
|
||||
derived from the same time (replication Next sync, backup age bands) reads it
|
||||
too. The clock can trail the wall clock by up to one tick, so a check that
|
||||
treats a future time as invalid (the backup age band) measures from the later
|
||||
of the two.
|
||||
Read-only metadata badges follow the same primitive-owned shell rule.
|
||||
`frontend-modern/src/components/shared/MetadataBadge.tsx` owns filled and
|
||||
outlined appearances, compact sizing, shape, typed tone vocabulary, fit
|
||||
@@ -5693,6 +5703,18 @@ When focused detail content is removed, `InlineDetailTableRow` restores focus
|
||||
to its current `aria-controls` disclosure with `preventScroll`; live refresh,
|
||||
collapse, and row replacement must not move the surrounding application
|
||||
viewport merely to reveal that control.
|
||||
The detail cell spans the summary row's visible cells, and `InlineDetailTableRow`
|
||||
re-measures that span while the row stays open: whenever summary cells are
|
||||
added, removed, or hidden (a column picker toggle, or a data-driven column
|
||||
such as the vCenter column that shows only with several vCenters), and
|
||||
whenever the caller's requested colspan changes. An open drawer must never
|
||||
keep a phantom column or stop short of a newly shown one. Changes inside a
|
||||
cell's content do not re-measure, so live metric updates stay cheap. When
|
||||
the span grows, the row changes the cell's own width for one forced layout
|
||||
and restores it, because Chromium otherwise keeps a fixed-layout table's old
|
||||
column widths for the widened cell. Callers keep passing their requested
|
||||
colspan; they must not re-key or remount the detail row to force a new span,
|
||||
which would discard drawer state.
|
||||
Inline detail section content is registry-backed separately from the row shell.
|
||||
`DetailSectionTable`, `InlineDetailPanel`, and `detailSectionModel.ts` own
|
||||
detail row compaction, section-table rendering, value-tone classes, and the
|
||||
@@ -7091,6 +7113,13 @@ should extend the model, and section rendering should extend those owners
|
||||
rather than rebuilding any of those concerns in the tab shell.
|
||||
The resource-resolution handoff into the resource-incident panel now belongs to
|
||||
the history state rather than the tab shell.
|
||||
The history model also owns the close outcome: an alert that closed with a
|
||||
typed `resolution` is status `moved` (badge `moved to agent`, its account in
|
||||
`HistoryItem.closeDetail`) rather than `resolved` or `acknowledged`. The table
|
||||
row carries the account in the badge title and the phone card renders it under
|
||||
the message, while `getAlertHistoryStatusPresentation` and
|
||||
`getAlertResolutionDetail` in `utils/alertIncidentPresentation.ts` own the
|
||||
wording.
|
||||
`frontend-modern/src/features/alerts/useAlertHistoryState.ts` re-exposes the
|
||||
`getResource` resolver it is already given, and
|
||||
`frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx` reads it
|
||||
@@ -7561,6 +7590,12 @@ inside the fleet surface with explicit copy while current status and resource
|
||||
navigation remain usable. Desktop and phone layouts must keep every tile
|
||||
keyboard-operable, preserve the textual legend, and avoid horizontal clipping
|
||||
at fleet scale.
|
||||
In the table, a secondary `MetadataBadge` beside a cell's primary value shares
|
||||
one flex line with it: the value is `shrink-0` and the badge `min-w-0` with a
|
||||
truncating label, so the value never clips and rows keep the single height
|
||||
`PlatformWindowedRows` measures. A second line under the value would break
|
||||
that windowing. Passing `flex` to the badge alongside its own `inline-flex`
|
||||
left the display to stylesheet order, which clipped the availability result.
|
||||
The Machines page must not pretend its machine list is a generic overview:
|
||||
the default tab is `Machines`, the Machines table is only for Pulse Agent-backed
|
||||
resources with host telemetry, and the full availability-check row list belongs
|
||||
@@ -8369,6 +8404,13 @@ inventory and Storage search below the fold at desktop and phone widths, and
|
||||
from competing with real warnings for attention, while preserving every
|
||||
affected name on demand. The component test pins collapsed, expanded and collapsed-again
|
||||
states; 1440px, 768px and 390px browser checks verify placement and overflow.
|
||||
Each platform page names what the update brings in the reader's words through
|
||||
`missingLabel`: the latest fixes and machine, node, system or VM details.
|
||||
The notice selects hosts by agent version alone, and command execution is a
|
||||
separate setting the update preserves, so the notice must not promise Patrol
|
||||
actions or command support. Internal phrasing such as "agent-contributed
|
||||
detail", "agent-managed platform detail" or "in-guest telemetry" does not
|
||||
belong in the notice.
|
||||
The shared `InlineNotice` action text uses opaque 800-level colors for its four
|
||||
tones. The current Tailwind configuration overrides several 900-level palette
|
||||
tokens with 25%-alpha colors for translucent backgrounds, so using those
|
||||
|
||||
@@ -1043,6 +1043,15 @@ join another container's history. The real alert-manager callback path is
|
||||
covered by `TestDockerAlertTimelineUsesCanonicalHistoryIdentity` in
|
||||
`internal/monitoring/monitor_alert_handling_test.go`.
|
||||
|
||||
Proxmox node, guest and storage alert lifecycle events carry source-native IDs
|
||||
and pass through the same writer, which resolves them to the canonical
|
||||
resource. Resource facets, resource intelligence and Assistant resource context
|
||||
therefore include their alert history, including events emitted after the
|
||||
resource left inventory. Event IDs still hash the alert's source reference, so
|
||||
replay stays idempotent. The real alert-manager callback path is covered by
|
||||
`TestProxmoxAlertTimelineUsesCanonicalHistoryIdentity` in
|
||||
`internal/monitoring/monitor_alert_handling_test.go`.
|
||||
|
||||
TrueNAS native alert projection preserves the trimmed, uppercase provider level in ResourceIncident.NativeSeverity. INFO and NOTICE retain the same canonical monitor risk; consumers must not lose their distinct actionability when projecting provider evidence. Native CRITICAL, ALERT, and EMERGENCY all project to canonical critical severity; EMERGENCY must not be discarded as unknown or make a still-active condition appear recovered. WARNING remains warning, and INFO and NOTICE remain informational at this projection boundary.
|
||||
|
||||
Verification: `TestIncidentProjectionPreservesNativeSeverity` in `internal/truenas/provider_pool_health_contract_test.go` covers all seven native levels and case/whitespace normalization. `TestTrueNASNativeSeverityDispatch` in `internal/alerts/truenas_native_dispatch_test.go` verifies downstream INFO suppression, NOTICE preservation, notification severity, duplicate-poll retention, and confirmed recovery callback identity. The TrueNAS lifecycle tests in `internal/alerts/unified_incidents_test.go` require repeated EMERGENCY evidence to interrupt recovery confirmation. These are fixture-based projection and manager checks, not appliance ingestion or external notification-provider receipt proof.
|
||||
@@ -1596,6 +1605,58 @@ supporting inventory rather than a new batch of independent exits, so the
|
||||
confirmed host incident clears child alerts instead of producing one alert per
|
||||
container.
|
||||
|
||||
Mock alert evaluation covers host agents the same way. Live agents are
|
||||
evaluated by `CheckHost` as each report lands. Mock mode discards reports, and
|
||||
the `evaluateHostAgents` health sweep reads monitor state, which never holds
|
||||
fixture agents, so `checkMockAlerts` evaluates every fixture agent itself
|
||||
through `evaluateMockHostAgents`. Without that pass no agent CPU, memory, disk,
|
||||
temperature or offline alert could open against mock data, while the same
|
||||
reading on a real agent would alert. The pass keeps the live lifecycle
|
||||
boundaries:
|
||||
|
||||
- An offline fixture goes through `HandleHostOfflineWithCorrelation`, as a
|
||||
lapsed live report does, because `CheckHost` treats its input as a fresh
|
||||
report and would mark the host online.
|
||||
- Agents are evaluated before nodes, so a node with an online linked agent
|
||||
hands its CPU, memory and disk alerts to the agent on the first tick, as
|
||||
host-agent hostname deduplication does in production.
|
||||
- A runtime mock config change rebuilds the estate. An agent that leaves it
|
||||
goes through `HandleHostRemoved`, as a deleted live agent does, so its alerts
|
||||
and hostname deduplication do not outlive it.
|
||||
- Leaving mock mode routes the fixture agents through `HandleHostRemoved`.
|
||||
`ClearActiveAlerts` drops their alerts but not their hostname registrations,
|
||||
and a real node named like a fixture agent (`pve1`) would otherwise keep its
|
||||
metric alerts suppressed with no agent to own them. A pass whose snapshot
|
||||
predates the switch finds mock mode off under the same lock and evaluates
|
||||
nothing. Two overlapping passes can still evaluate an older snapshot after
|
||||
a newer one; the next pass reconciles the set again.
|
||||
|
||||
Fixture agents must carry readings a real agent could report. Mock Kubernetes
|
||||
pods share 0.7 single-pod memory footprints per node, which keeps a node's pods
|
||||
at or below 58% of allocatable memory in steady state, and a 60% per-node
|
||||
ceiling clips the transient while rescheduled pods' readings settle. A dense
|
||||
node and its linked agent therefore no longer read a constant 100% and raise a
|
||||
critical alert no real cluster would. Whether a demo agent alerts at default thresholds depends on
|
||||
the readings each boot draws. On 2026-10-06, private stacks with the public
|
||||
demo's estate (8 nodes, 2 standalone agents and 3 Kubernetes node agents)
|
||||
opened none on one boot and one (a standalone agent at 91% memory) on another.
|
||||
With the agent memory trigger lowered to 60%, exactly the agents above it
|
||||
alerted and their linked Proxmox nodes raised none, and shrinking the estate
|
||||
through the mock settings API cleared the departed agents' alerts on the next
|
||||
pass. Those are measurements, not test guarantees. The tests pin the
|
||||
boundaries:
|
||||
`TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes`,
|
||||
`TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures`,
|
||||
`TestLeavingMockModeReleasesFixtureAgentHostnames` and
|
||||
`TestMockHostAgentPassAfterLeavingMockModeRegistersNothing` in
|
||||
`internal/monitoring/monitor_mock_alerts_test.go`,
|
||||
`TestMockHostAgentLeavingFixtureUsesRemovalLifecycle` in
|
||||
`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`, and
|
||||
`TestMockKubernetesDenseNodesStayBelowAllocatableMemory`,
|
||||
`TestMockKubernetesRescheduledPodsDoNotSaturateReceivingNode` and
|
||||
`TestMockKubernetesLoneHeavyPodStaysUnderNodeCeiling` in
|
||||
`internal/mock/generator_test.go`.
|
||||
|
||||
Host and container-runtime disk collection supports an explicit include list
|
||||
for filesystems hidden by Pulse's automatic virtual/container filtering. The
|
||||
include list is bounded to that automatic filter; explicit disk exclusions
|
||||
@@ -2586,6 +2647,25 @@ online/running/healthy vocabulary. The diagnostics subject set is unchanged:
|
||||
snapshot (real agents), which now matches what agent-only surfaces show once
|
||||
integration-backed ledger rows are excluded.
|
||||
|
||||
A host produced by `hostFromReadStateView` describes the agent, so its
|
||||
`Status` and `LastSeen` are the agent's own. A host agent linked to a Proxmox
|
||||
node merges into one row whose `LastSeen` follows its freshest source and
|
||||
whose status stays online while PVE polling continues, so the row's values
|
||||
would present a silent agent as online and current. The projection reads the
|
||||
agent source's sighting from `HostView.SourceStatus(SourceAgent)`: the host
|
||||
carries that report time and reads `offline` once it is past the reporting
|
||||
lease (`hostAgentReportCurrent`, the `hostAgentHealthWindow` that
|
||||
`evaluateHostAgents` applies to the host in state). Rows without an agent
|
||||
sighting, such as vSphere and TrueNAS integration hosts, keep the row's
|
||||
values. `linkedHostForNode` therefore stops handing a silent agent's retained
|
||||
filesystem summary and ZFS datasets to its node, whose disk falls back to the
|
||||
Proxmox rootfs or `/nodes` reading before disk alerts and history read it, and
|
||||
agent connections built from `HostsSnapshot()` (the Connections list and the
|
||||
update-readiness agent-continuity check) age the agent from its last report
|
||||
rather than the PVE poll time, applying their own heartbeat cutoff.
|
||||
`TestSilentLinkedAgentStopsFeedingNodeDisk` drives the disk and projection
|
||||
path through `evaluateHostAgents` and the registry-backed store refresh.
|
||||
|
||||
Unified Agent host reports now make module readiness and updater/config
|
||||
lifecycle evidence monitoring-owned observed state. Monitoring preserves the
|
||||
last successful one-shot update transition across subsequent reports, forwards
|
||||
@@ -3587,6 +3667,15 @@ truth. Monitoring APIs that still serve `StateSnapshot` must project
|
||||
instead of trusting the cached snapshot fields, so externally served alert
|
||||
counts and recently resolved incidents do not lag behind acknowledgement,
|
||||
resolve, or clear operations between explicit sync points.
|
||||
A recently resolved alert that closed without recovering keeps that on the
|
||||
projection: `models.Alert.Resolution` (reason, successor resource ID and name,
|
||||
and the alert engine's summary) is set only for such a close, `cloneAlert`
|
||||
copies it into every snapshot, and `recordAlertTimelineChange` passes it into
|
||||
the `alert_resolved` resource change so the incident timeline says the alert
|
||||
moved. `TestAlertResolutionWireContract` in
|
||||
`internal/models/metrics_types_test.go` and
|
||||
`TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery` in
|
||||
`internal/monitoring/monitor_alert_handling_test.go` pin it.
|
||||
The container entrypoint in `docker-entrypoint.sh` now also lives under this
|
||||
boundary. Hosted or managed tenant bootstrap changes must preserve safe startup
|
||||
when immutable read-only mounts are layered into `/etc/pulse`; the entrypoint
|
||||
@@ -3684,7 +3773,21 @@ transiently `unavailable`, provider/controller `unsupported`, or unexpectedly
|
||||
`missing`. Normalization may retain the last known value when the current
|
||||
observation is not available, but it must preserve the current state and
|
||||
reason so API and UI consumers do not present retained evidence as freshly
|
||||
collected. Unified-resource physical-disk round trips must retain named
|
||||
collected. A host agent's SMART temperature and I/O counters follow its
|
||||
reporting lease: once `State.ExpireHostTelemetry` expires the agent, they stay
|
||||
as last-known values marked `unavailable` ("host agent stopped reporting"),
|
||||
and the linked Proxmox disk merge and the registry let the agent's own later
|
||||
state supersede the availability it supplied earlier
|
||||
(`diskinventory.MergeReportedStatus`), so neither a skipped disk poll nor a
|
||||
host that is never disk-polled again can carry it forward. Expiry is
|
||||
compare-and-set on the report time the evaluation judged stale, so a report
|
||||
accepted in between is never expired. Every SMART temperature history writer
|
||||
records a temperature only when its current collection state is available (or
|
||||
predates collection state), so a retained reading has to keep its non-available
|
||||
state to stay out of history; a path that relabels a carried reading as
|
||||
available, such as a node-temperature carry stamped `proxmox_node_smart`, is
|
||||
not covered by this rule.
|
||||
Unified-resource physical-disk round trips must retain named
|
||||
`StorageGroup` membership rather than degrading it to the generic `Used`
|
||||
filesystem label.
|
||||
That same host-agent temperature boundary must prefer a recent linked host-agent
|
||||
@@ -3695,6 +3798,55 @@ exists or the agent payload has no usable positive reading. Identity-only or
|
||||
zero-temperature SMART rows do not count as usable by themselves, but the
|
||||
runtime must not keep probing legacy SSH solely to augment an otherwise healthy
|
||||
agent temperature payload with SMART data.
|
||||
A node temperature presented as available must describe the node now, because
|
||||
node alert evaluation, node history writes, reporting, and the UI all read it as
|
||||
a live measurement. A host agent linked to a Proxmox node merges into one host
|
||||
row whose `LastSeen` every PVE poll keeps fresh, while the agent's last sensors
|
||||
stay in state after it stops reporting, so the row's `LastSeen` cannot decide
|
||||
whether those sensors are current. `internal/monitoring/host_agent_temps.go`
|
||||
reads the agent's own sighting from the row's `SourceStatus(SourceAgent)`, feeds
|
||||
the agent's sensors into the node only while that sighting is inside the
|
||||
reporting lease that keeps the agent online (`hostAgentReportCurrent`, the same
|
||||
`hostAgentHealthWindow` that `evaluateHostAgents` applies, so a slow-interval
|
||||
agent's readings count exactly while the agent is shown online), and stamps the
|
||||
reading with that report time. Past the lease the lookup falls back to the
|
||||
cluster sensor cache, which keeps its own recency check. When every source
|
||||
returns nothing, `internal/monitoring/monitor_polling_node_helpers.go` may carry
|
||||
a previous reading, with its original `LastUpdate`, only inside the carry window
|
||||
(twice the PVE polling interval, never under five minutes); an older reading is
|
||||
dropped rather than re-presented as current. A carried reading also keeps a
|
||||
lapsed host agent's lease. Agent readings are stamped with the agent's report
|
||||
time, so `carriedTemperatureOutlivesAgentLease` drops a carried reading stamped
|
||||
no later than the last report of the node's matched agent (by link, then
|
||||
hostname, whether or not that report had sensors) once that agent has lapsed.
|
||||
Readings record no source, so the bound is conservative: an SSH, cluster-cache
|
||||
or agent-plus-SSH merged reading (stamped with the agent's time) of that age is
|
||||
dropped too, being already older than the lease, while a reading stamped after
|
||||
that report keeps the carry window. A carried reading (`carriedNodeTemperature`)
|
||||
keeps only the node's CPU and GPU temperatures: its SMART and NVMe rows would
|
||||
also stamp the node's physical disks, whose freshness and standby state a
|
||||
carried copy cannot follow (a disk that spun down would be stamped and recorded
|
||||
with its pre-sleep value), so they are left out, and nothing is carried when
|
||||
only they remain.
|
||||
The poller reads its previous poll's nodes back from the unified read state
|
||||
through `internal/monitoring/monitor_previous_state.go`. `nodeLastOnline` and
|
||||
the temperature carry with its CPU low/record tracking match them by node ID,
|
||||
which is the Proxmox source ID, so the projection must return
|
||||
`NodeView.SourceID()`, never the unified resource ID, and carry the stored
|
||||
`TemperatureDetails`. Nodes preserved through an instance outage
|
||||
(`preserveOrExpireNodes`) or authoritative membership reconciliation are written
|
||||
back into state from that projection, so it also restores the identity inputs
|
||||
the registry derives a node's canonical ID and cross-view merges from: cluster
|
||||
name, provider-scoped flag, native aliases, TLS fingerprint and linked agent ID,
|
||||
the config-derived ones stamped the way `pollPVENode` stamps them. With the
|
||||
unified ID a node re-keys on every failed poll, and without its cluster
|
||||
identity a cluster member changes canonical ID for the length of an outage.
|
||||
A preserved node keeps its
|
||||
temperature only under the same carry rules and loses it when it goes
|
||||
offline. Node memory carry-over after a failed status read uses the poller's own
|
||||
validated `NodeMemorySnapshot`, not the read state, which holds the agent's
|
||||
reading on a node merged with a host agent, and keeps used, cache and free
|
||||
summing to the total.
|
||||
Legacy SSH temperature collection must also use the Pulse sensor-wrapper
|
||||
contract before falling back to raw lm-sensors output. `internal/monitoring/temperature.go`
|
||||
must request `/usr/local/sbin/pulse-sensors` when it exists, parse the wrapper
|
||||
|
||||
@@ -1216,3 +1216,41 @@ The live monitor does not interpret pending-only cancellation counts as
|
||||
recipient acceptance. Per-occurrence/per-destination receipts remain the sole
|
||||
recovery admission proof, including partial delivery and lost RAM markers after
|
||||
restart; an unannounced destination receives no recovery.
|
||||
|
||||
### A close that was not a recovery says what happened
|
||||
|
||||
A resolved alert can carry an `alerts.AlertResolution` when it closed without
|
||||
recovering. The one reason so far, `moved_to_agent`, means the metric now
|
||||
belongs to the resource's linked Pulse agent. Delivery is unchanged: the close
|
||||
is still queued, gated on the firing receipt and sent, and PagerDuty still
|
||||
receives `resolve` for the alert's dedup key. Only the wording changes.
|
||||
`resolvedAlertMessage` gives the resolution's summary with the alert named
|
||||
("Memory alert moved to pve1 (Host Agent). This is not a recovery: check the
|
||||
agent for the current reading.") in place of "<resource> on <node> is now
|
||||
healthy" for every templated webhook `.Message`. Grouped lists append the
|
||||
short outcome ("moved to pve1 (Host Agent)") to the moved entry. ntfy titles a
|
||||
single moved alert `MOVED: <resource>`. Email and Apprise title it `Pulse alert
|
||||
moved: <resource>` and add the summary after the breach message. Recovery
|
||||
visuals follow the same rule: when any close in a resolved delivery was not a
|
||||
recovery, `WebhookPayloadData.NotRecovered` is set and the built-in templates
|
||||
drop them (Discord's green embed becomes the neutral blue, both Teams cards
|
||||
lose recovery green, Mattermost's check emoji becomes `:arrow_right:`), and
|
||||
ntfy swaps its green check tag for `arrow_right`, for a single alert or a batch.
|
||||
Built-in titles still say Resolved, which is accurate for the closed alert;
|
||||
custom templates can read `.NotRecovered`, and one that hard-codes recovery
|
||||
text keeps it. The resolution rides the Alert JSON, so it survives the delivery
|
||||
queue round trip and appears in the default JSON payload's `alerts` array.
|
||||
Issue #1682 is the precedent: a recipient called "is now healthy" a false
|
||||
factual claim when an alert closed without recovering.
|
||||
The resolve transition such a close links to has cause `ownership_transferred`
|
||||
in `internal/operationaltrust/contracts.go`, not `recovery_evidence`. A
|
||||
notification link records the transition ID and lifecycle state, never the
|
||||
cause, so delivery, receipts and recovery admission treat it exactly like a
|
||||
recovery's resolve transition. `TestOwnershipTransferOnlyClosesWithEvidence`
|
||||
pins that the cause may only enter `resolved` and only with evidence.
|
||||
|
||||
Verification: `resolved_handover_test.go` drives the firing, its receipt and
|
||||
the queued close through the normal entry points for ntfy, and renders Discord,
|
||||
both Teams cards, Mattermost, PagerDuty, a grouped Slack list, a grouped ntfy
|
||||
batch and email/Apprise content, plus the unchanged ordinary recovery wording
|
||||
and visuals.
|
||||
|
||||
@@ -1380,6 +1380,14 @@ to traverse the list again. At widths where the detail replaces the queue's
|
||||
reading position, its visible close affordance is an explicit `Back to list`
|
||||
control; wide layouts retain the compact close icon. Both controls clear the
|
||||
deep link and restore focus to the originating queue row.
|
||||
A resolved item whose latest close is an `ownership_transferred` transition did
|
||||
not recover: its condition moved to another record, such as the linked Pulse
|
||||
agent's own alert. The detail's state badge reads `Moved` in the info tone
|
||||
instead of the green `Resolved`, and the transition's reason (where the
|
||||
condition went, and that it is not a recovery) sits directly under the summary
|
||||
rather than only inside Evidence and history (`getAttentionHandoverClose` in
|
||||
`patrolHomePresentation.ts`). The timeline names that cause `Ownership
|
||||
transferred`, never `Recovery evidence`.
|
||||
|
||||
The active Patrol queue now uses compact severity-accented rows for
|
||||
prioritisation and a single focused review panel for the selected issue. The
|
||||
|
||||
@@ -617,6 +617,14 @@ count times the number of shared storages.
|
||||
Demo Docker container names stay unique per host (numbered suffixes on reused
|
||||
profiles), so image "used by" lists and container rows never collapse two
|
||||
distinct containers into what reads as a duplicated row.
|
||||
Mock Kubernetes pods share their node's allocatable memory: each active pod's
|
||||
memory target is scaled by `mockKubernetesPodMemoryShare` over the active pods
|
||||
on its node, which keeps a node's pods at or below 58% of allocatable memory in
|
||||
steady state, and `capMockKubernetesNodePodMemory` clips them to 60% only while
|
||||
rescheduled pods' smoothed readings settle, so a dense node does not pin itself
|
||||
and its linked agent at 100%. The count and the clip are linear passes over the
|
||||
cluster's pods in each usage update, keyed by node name, with no per-pod scan
|
||||
of other pods or nodes.
|
||||
|
||||
The browser applies resource deltas to its connection-scoped raw baseline, but
|
||||
canonicalizes and reconciles only changed resources plus the host-merge groups
|
||||
@@ -1129,6 +1137,11 @@ change may globally weaken the Task 03 lifecycle-state idempotency invariant.
|
||||
hover tooltips from synthesized mouse events; a row tap remains the primary
|
||||
drawer action. Tooltip suppression belongs to the shared hover-capability
|
||||
boundary rather than per-row viewport branches or listeners.
|
||||
The shared `EnhancedCPUBar` hover panel names the CPU count that the bar
|
||||
shows in parentheses beside the percentage ("22% (18)") as "CPUs": logical
|
||||
CPUs on a host, configured vCPUs on a guest, so "Cores" would be wrong on
|
||||
hyperthreaded hosts. It reads the `cores` prop the row already passes and
|
||||
adds no per-row work.
|
||||
Ultra-narrow workload presentation belongs in the existing layout and
|
||||
column model: below 360 pixels it keeps five identity/health columns with a
|
||||
40-percent identity track. It must not add per-row measurement, viewport
|
||||
|
||||
@@ -5973,6 +5973,7 @@
|
||||
"frontend-modern/src/components/shared/__tests__/Dialog.test.tsx",
|
||||
"frontend-modern/src/components/shared/__tests__/HelpIcon.test.tsx",
|
||||
"frontend-modern/src/components/shared/__tests__/HistoryChart.test.tsx",
|
||||
"frontend-modern/src/components/shared/__tests__/InlineDetailTableRow.test.tsx",
|
||||
"frontend-modern/src/components/shared/__tests__/MobileNavBar.test.tsx",
|
||||
"frontend-modern/src/components/shared/__tests__/PulseDataGrid.test.tsx",
|
||||
"frontend-modern/src/components/shared/__tests__/SearchField.test.tsx",
|
||||
|
||||
@@ -923,8 +923,9 @@ coverage, restore capability, recovery-job proof, storage-health verification,
|
||||
or evidence that a specific resource was protected or recovered.
|
||||
Proxmox page stale-agent notices are adjacent frontend and agent-lifecycle
|
||||
plumbing even though `ProxmoxPageSurface` is a storage/recovery canonical file.
|
||||
Those notices may link an operator to scoped agent update commands for
|
||||
agent-contributed node detail and command support, but they must not be
|
||||
Those notices may link an operator to scoped agent update commands for the
|
||||
latest fixes and node details (the notice's own wording; command execution is
|
||||
a separate setting the update does not change), but they must not be
|
||||
interpreted as backup visibility, recovery readiness, restore capability, or a
|
||||
storage/recovery-owned command path. Suppressing those notices when the API has
|
||||
no deployable agent update target is likewise lifecycle/frontend behavior and
|
||||
@@ -983,6 +984,18 @@ Proxmox backup source/state chips are also a frontend-primitives boundary:
|
||||
badge tones, `proxmoxBackupsTableShared.tsx` owns the backup table helper
|
||||
composition, and visible source/state chips must render through `MetadataBadge`
|
||||
instead of restoring local rounded-sm xs badge spans.
|
||||
Backup ages read the frontend-primitives shared relative-time clock:
|
||||
`ProxmoxBackupAgeText` measures the full and compact age text and its
|
||||
current, aging, or stale band from `useRelativeTimeNow`, never earlier than the
|
||||
wall clock, so a backup row that stays mounted for days moves from current to
|
||||
aging without new inventory, and a backup that finished inside the clock's last
|
||||
tick never bands as a future, unknown age. `ProxmoxPageSurface` reads
|
||||
replication jobs through `createNonSuspendingQuery` with a 30-second background
|
||||
poll, because the Replication table's Last sync and Next sync move on the same
|
||||
clock: a failed read reports the error but keeps the last jobs and the tab, and
|
||||
a direct replication link holds until the first read resolves instead of
|
||||
falling back to Overview. The poll re-reads the existing read-only
|
||||
`/api/replication/jobs` snapshot and opens no new storage or recovery path.
|
||||
Proxmox backup-health and recoverable-artifact summary rows follow the same
|
||||
platform-table density contract as every other provider surface. Storage and
|
||||
recovery owns which backup, datastore, workload-type, and VMID facts matter to
|
||||
|
||||
@@ -485,6 +485,18 @@ layer consumes it only as a fallback replacement for Proxmox LXC memory, so
|
||||
platform metric priority for a healthy agent is unchanged.
|
||||
`TestContainerViewLinkedAgentMemory` pins those boundaries.
|
||||
|
||||
**Host-row agent source freshness (6 October 2026)**
|
||||
|
||||
`HostView.SourceStatus(source)` exposes the per-source delivery freshness the
|
||||
registry already records, matching the VM, container, and physical-disk views.
|
||||
A host agent linked to a Proxmox node merges into one row whose `LastSeen`
|
||||
follows the freshest source, so consumers of agent-owned samples (monitoring's
|
||||
linked-agent node temperature lookup) must read the agent source's own sighting
|
||||
here instead of the row's `LastSeen`. The accessor returns a copy of the recorded
|
||||
status and adds no freshness policy of its own.
|
||||
`TestView_HostViewSourceStatusSeparatesAgentFromMergedRow` pins the merged-row
|
||||
split.
|
||||
|
||||
### Bounded incident-history selection
|
||||
|
||||
Canonical history queries filter exact alert identifiers and observation windows
|
||||
@@ -1250,6 +1262,21 @@ platform-details disclosure. The `KubernetesControllersTable` phone projection
|
||||
keeps controller, kind, ready, and issues and demotes Target with
|
||||
`platform-table-phone-hidden`, so kind labels such as `DaemonSet` and
|
||||
`StatefulSet` fit whole instead of clipping in a 15 percent track.
|
||||
Every relative age a unified-resource consumer derives from a timestamp reads
|
||||
the frontend-primitives shared relative-time clock (`useRelativeTimeNow`)
|
||||
rather than `Date.now()` at render, because rows and open drawers stay mounted
|
||||
while a timestamp that has stopped changing must keep aging: the Kubernetes
|
||||
controller Detail ages and the drawer's controller section, the drawer's Docker
|
||||
container Created, Started and Finished rows, its Last seen, and its Docker
|
||||
update-check and Mail Gateway updated ages, the Machines identity subtitle
|
||||
(`seen ...` while the Last seen column is hidden) and stale-agent Last report
|
||||
tooltip, and the Proxmox replication Last sync and Next sync in the row, phone
|
||||
projection and disclosure. A silent agent or a stalled pvesr scheduler is the
|
||||
case whose age matters most, so a replication Next sync turns overdue on the
|
||||
clock without new data. Replication jobs bypass the unified-resource stream,
|
||||
so `ProxmoxPageSurface` re-reads them in the background every 30 seconds;
|
||||
moving ages over a snapshot read once at mount would age a job that keeps
|
||||
syncing and count it overdue.
|
||||
Kubernetes name columns hold the chevron, status dot and name inside one
|
||||
track, so their md widths leave the name room for a typical node or service
|
||||
name at a 768px viewport: Nodes 20 percent (with Capacity at 16 so its
|
||||
@@ -2230,7 +2257,17 @@ may repeat beside it. The Docker hosts (`DockerHostsTable.tsx`) and Machines
|
||||
`getUnifiedResourceAlertStyles`, so a collapsed row never looks healthy while
|
||||
its drawer lists an unacknowledged alert (an expanded row drops the tint while
|
||||
its drawer shows the alerts); acknowledged alerts stay in the drawer without
|
||||
tinting the row.
|
||||
tinting the row. Kubernetes node rows (`KubernetesNodesTable.tsx`) tint the
|
||||
same way: a node that runs a Pulse agent is an agent row whose alerts are keyed
|
||||
`agent:<id>` and carry the hostname, so the earlier id-and-display-name match
|
||||
left it untinted while its drawer listed a critical alert. The Kubernetes
|
||||
overview's "Nodes needing attention" and the nodes table's default order rank
|
||||
a node by the stronger of its readiness state and those unacknowledged alerts
|
||||
(`getKubernetesNodeAttentionRank`), so a Ready node at 100% memory is named
|
||||
beside the NotReady ones. vSphere host and TrueNAS system rows keep
|
||||
`getAlertStyles` with the host name: their VMs', pools' and apps' alerts carry
|
||||
that name as their node, and no producer keys alerts on those rows' `agent:`
|
||||
aliases (a Pulse agent on a TrueNAS box stays a separate resource).
|
||||
Machine and host overview cards that render compact system, hardware, disk,
|
||||
and temperature facts must also compose the frontend-primitives
|
||||
`InfoCardKeyValueRow`. Mobile rows retain their condensed endpoint layout;
|
||||
@@ -2375,6 +2412,9 @@ for the column when evidence later appears. The
|
||||
compact row action trigger chrome stays under the frontend-primitives
|
||||
`ActionIconButton` boundary rather than becoming a unified-resource-local
|
||||
button shell.
|
||||
The Machines outdated-agent notice names what an update brings in plain
|
||||
words, the latest fixes and machine details, and stays maintenance guidance
|
||||
rather than a membership or health signal.
|
||||
Machines list search and online-state narrowing are frontend route state,
|
||||
not new unified-resource membership fields. `StandalonePageSurface.tsx`
|
||||
owns the `STANDALONE_QUERY_PARAMS` query/status projection and one composite
|
||||
@@ -3478,6 +3518,14 @@ with the offline threshold on hover, not an "N/M" fraction.
|
||||
Recent check timing and fuller failure context may stay in tooltip or drawer
|
||||
detail, but the table row must not duplicate the same probe protocol and
|
||||
result text across both identity and metric cells.
|
||||
The probe source chip ("via Edge 01", or "2/2 locations reporting" for a
|
||||
multi-location check, the same wording Settings uses) shares the result's
|
||||
single line: the result keeps its full width at the cell's right edge and the
|
||||
chip takes the room left, truncating with its full text on hover. Before, the
|
||||
chip pushed "failed" out of its column at every width up to 1440px. Rows stay
|
||||
one line because the table windows them at a single measured height. On
|
||||
phones the method and target columns hide so the check name reads, and both
|
||||
stay in the row drawer.
|
||||
That same frontend-owned compatibility boundary must remain intentionally
|
||||
narrow. Shared resource adapters may admit explicit aliases such as `host`,
|
||||
`truenas`, and `ceph`, and VMware detail mappers may project typed metadata
|
||||
@@ -3534,6 +3582,13 @@ the five field statuses for serial, temperature, I/O, controller, and pool.
|
||||
`unavailable`, `unsupported`, or `missing` state may retain a prior value for
|
||||
continuity, but the state and reason must survive so the consumer cannot claim
|
||||
fresh evidence or synthesize controller-level activity for one member.
|
||||
The one exception is a source withdrawing its own evidence: when the agent
|
||||
row reports a field as no longer available, that supersedes an `available`
|
||||
state the Proxmox row still carries from the same source
|
||||
(`diskinventory.MergeReportedStatus`), in either ingest order. A host agent
|
||||
past its reporting lease must not stay "collected" through the Proxmox row's
|
||||
copy of the agent's own state, including when its host is down and no disk
|
||||
poll refreshes that copy.
|
||||
|
||||
Cross-source correlation compares normalized serial and WWN values across
|
||||
fields without truncation, allowing a PVE bare-hex array-volume serial to join
|
||||
@@ -4189,6 +4244,14 @@ such as `alert_fired`, `alert_acknowledged`, `alert_unacknowledged`,
|
||||
AI-local annotations. Snooze and resume projections must preserve the actor
|
||||
and exact suppression expiry when present; they pause delivery and escalation
|
||||
without acknowledging, resolving, or replacing the underlying incident.
|
||||
An `alert_resolved` change for a close that was not a recovery carries the
|
||||
alert engine's summary as its reason and the reason code as `alert_resolution`
|
||||
metadata (`AlertTimelineChange.ResolutionReason` and `ResolutionSummary`), so
|
||||
incident projection titles the close with where the alert went instead of
|
||||
"Alert resolved: <breach message>". The resource drawer still labels the change
|
||||
by kind (`Alert resolved`) and prefixes its headline with that label; only the
|
||||
reason after it says the alert moved. Ordinary recoveries omit the key, and
|
||||
every other lifecycle kind ignores the fields.
|
||||
Alert-scoped
|
||||
incident memory may still project those events for one investigation thread,
|
||||
but the durable source of truth for resource-affecting alert lifecycle and
|
||||
@@ -4349,7 +4412,13 @@ must render from that shared projection instead of rescanning raw job arrays or
|
||||
inventing local PBS status heuristics,
|
||||
`resourceDetailDrawerIdentityModel.ts` owns the pure identity-card,
|
||||
discovery-summary, source-debug, and debug-bundle derivations that feed the
|
||||
overview and debug drawer surfaces,
|
||||
overview and debug drawer surfaces (the `Identity` card shows each identifier
|
||||
once: Discovery and Metrics Target rows appear only when they name something
|
||||
not already on screen, meaning the header's `getPreferredInfrastructureDisplayName`,
|
||||
the rows above, or the Platform ID row, and Aliases omits the same displayed
|
||||
values; candidate names the drawer never shows do not count, a shown Platform
|
||||
ID counts as identity data for the empty state, and a Machines agent drawer
|
||||
used to list one ID four times),
|
||||
`useResourceDetailDrawerDockerActionsState.ts` owns Docker action runtime, and
|
||||
the overview/debug render-heavy surfaces live in dedicated drawer-local owners
|
||||
instead of staying inline in the shell.
|
||||
@@ -5782,6 +5851,29 @@ action requests, approvals, links or exclusions. Separate monitor, API and
|
||||
Assistant store handles must see current persisted aliases. Missing identity
|
||||
storage is an error, not evidence of empty history. Retention removes an alias
|
||||
only after neither identity has retained journal records.
|
||||
Other alert source references resolve at `MonitorAdapter.RecordChange`
|
||||
through `ResourceRegistry.resolveHistoryReference`, which accepts only durable
|
||||
identities: the canonical ID, a retired era, a source ID (Proxmox node, guest
|
||||
and storage IDs), a node-scoped Proxmox guest reference after migration, or
|
||||
`agent:<host ID>`. Names and hostnames never bind history, unlike the wider
|
||||
`ResolveReferenceID` used by alert policy, so a resource named like a system
|
||||
reference cannot capture its events. A reference two resources answer to binds
|
||||
to neither: its event keeps its own reference and creates or changes no
|
||||
binding, though an earlier binding still covers that reference on read. A
|
||||
reference no resource
|
||||
answers to follows its retained binding after inventory removal, or keeps its
|
||||
own history. Docker references never take this path. Alert journal references
|
||||
without a binding (read once per process, plus any written before inventory
|
||||
named their resource) are retried on each published registry generation for
|
||||
`legacyHistoryBindWindow`. Bound rows keep their recorded resource ID, so a
|
||||
change list may show the source reference on rows written before the binding.
|
||||
Only the full-presentation `ResourceDetailDrawer` reads this history in the UI
|
||||
(facets and resource intelligence); the `table-row` drawer used by Proxmox node
|
||||
and Agents rows, and the guest and storage drawers, do not request it.
|
||||
Assistant and Patrol resource contexts read it from the store. Proof:
|
||||
`TestHistoryIdentityMonitorAdapterResolvesProxmoxAlertReferences` and
|
||||
`TestHistoryIdentityBindsLegacyAlertRowsFromRegistryGenerations` in
|
||||
`internal/unifiedresources/history_identity_test.go`.
|
||||
That same shared timeline vocabulary now includes the `activity` change kind
|
||||
for provider-read breadcrumbs such as VMware tasks and events, plus the
|
||||
`vmware_adapter` source-adapter token for canonical provenance drill-down.
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "bffd21f47026ca2408ee1d0b5bb071c02e60c623",
|
||||
"verified_at": "2026-10-06T15:48:24Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx": "db24691402c341a2ceec3135d6425c03c9706b6cb57630f9c0efc2e956179264"
|
||||
},
|
||||
"routes": [
|
||||
"/standalone/machines"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1600,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Combined AgentsMachinesTable.tsx on main after #2579 (row alert tint) and #2578 (shared relative-time clock) merged; private mock stack built from bffd21f470",
|
||||
"A critical memory alert for Apollo-114 (resourceId agent-acdfdee2953587fb) injected into the websocket activeAlerts and /api/alerts/active; Spectrum-709 left without alerts",
|
||||
"1600px light: Apollo-114 row bg-red-50 (rgb 254,242,242), Spectrum-709 untinted; with Last seen hidden the identity title read \"| seen just now\"",
|
||||
"Playwright page.clock fast-forwarded 3h with /ws held and REST replayed: Apollo-114 title \"| seen 3h ago\" and the red tint unchanged, Spectrum-709 still untinted",
|
||||
"390px dark: Apollo-114 row dark:bg-red-950/25, Seen cell \"just now\" -> \"3h ago\" after the same 3h jump with the tint unchanged",
|
||||
"Apollo-114 drawer lists the injected \"Memory usage is 96%\" alert at both widths; no page errors"
|
||||
],
|
||||
"interactions": [
|
||||
"Opened View > Choose which columns to display and unchecked Last seen on Machines (1600px)",
|
||||
"page.clock.fastForward 3h with data held",
|
||||
"Clicked the Apollo-114 row to open its drawer"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "7aa537abe112e5f0b9c043885c62bc255b700134",
|
||||
"verified_at": "2026-10-06T13:58:52Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Infrastructure/ResourceDetailSummary.tsx",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerKubernetesModel.ts",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts",
|
||||
"frontend-modern/src/features/kubernetes/KubernetesControllersTable.tsx",
|
||||
"frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx",
|
||||
"frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx",
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx",
|
||||
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Infrastructure/ResourceDetailSummary.tsx": "7f789e2e38fab255bd080a647b1f9d20185419c336974bc188fa9c2dec3c2e06",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerKubernetesModel.ts": "34de374717a338b6d7417e0264f3ed1f106c965262a88336a52a3ebe1e0e3208",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts": "a0c08531c1b3c9eba9827ddf02ece46e55ca6eece7629b9b005653ac5129b6ef",
|
||||
"frontend-modern/src/features/kubernetes/KubernetesControllersTable.tsx": "262e641559d38ccadacf45a74a37b3ec95cbb58887c5ab620b966d369cd9a638",
|
||||
"frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx": "147ce30feded49cbe9bf833daa78edd1a6fbcff6db419d9303b21e62c9fe00e3",
|
||||
"frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx": "9da12964e02d081b80c848db8741f69d701f8c99e73ed01e9c8958c9d6b51930",
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx": "4d6c1097737138d1dd6b1ecfc28ee33b0815ffa78279120bcb524cea38053d2a",
|
||||
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx": "b9c718460ec18d6ef0a72b2fe9d291e0e813feaecfc6d4cd2f8694cda467fbdf"
|
||||
},
|
||||
"routes": [
|
||||
"/kubernetes/workloads",
|
||||
"/proxmox/backups",
|
||||
"/proxmox/replication",
|
||||
"/standalone/machines"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1600,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack built from this tree; Playwright page.clock installed at load, /ws held after first render (server frames dropped) and REST GETs replayed from their last response, then the clock fast-forwarded with no data change",
|
||||
"Kubernetes workloads 1600px: Job/CronJob Detail \"Completed 48m ago\" / \"Last success 1h ago\" -> \"Completed 3h ago\" / \"Last success 4h ago\" after 3h",
|
||||
"Kubernetes Job row expansion: controller section \"01:59 PM (48m ago)\" -> \"01:59 PM (3h ago)\" with the absolute time unchanged; Runtime context Last seen \"1s ago\" -> \"3 hours ago\"",
|
||||
"Proxmox backups 1600px and 390px: tracked artifact \"1d ago\" / \"1d\" in the current (green) band -> \"8d ago\" / \"8d\" in the aging (amber) band, title \"Aging backup age\", after 7 days",
|
||||
"Proxmox replication 1600px: row and expanded Last sync \"21m ago\" -> \"3h ago\", Next sync \"in 8m\" -> \"2h overdue\" in red after 3h; one background /api/replication/jobs read fired during the jump",
|
||||
"Proxmox replication 390px compact row: Last sync \"21m\" -> \"3h\", Next sync \"8m\" -> \"-2h\"",
|
||||
"Machines 1600px with Last seen column hidden: identity title \"| seen just now\" -> \"| seen 3h ago\"; 390px Seen cells \"just now\" -> \"3h ago\"",
|
||||
"Control on the same stack with the seven runtime files swapped back to the parent: Kubernetes Detail and expansion, replication expanded Last sync / Next sync and phone row, machines subtitle, and phone backup ages stayed frozen; desktop backup text moved but the band stayed green"
|
||||
],
|
||||
"interactions": [
|
||||
"Clicked a Kubernetes Job controller row to open its expansion",
|
||||
"Clicked \"Expand details for replication job 100-0\"",
|
||||
"Opened View > Choose which columns to display and unchecked Last seen on Machines",
|
||||
"page.clock.fastForward 3h (Kubernetes, replication, machines) and 7 days (backups) with data held"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "27051deb279b8dc42a82922f14ebdddc7e8fbfcc",
|
||||
"verified_at": "2026-10-06T14:58:00Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerIdentityModel.ts",
|
||||
"frontend-modern/src/utils/resourceIdentity.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerIdentityModel.ts": "902911e665585a831c072675c85609bc1e6015e71db0deb79877767d9b454892",
|
||||
"frontend-modern/src/utils/resourceIdentity.ts": "c76d4279e7caca8b96b1a52e4c166f791264217730d631f47b8de3cedf411c32"
|
||||
},
|
||||
"routes": [
|
||||
"/standalone/machines",
|
||||
"/vmware/storage"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 1300
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack on main c4db7a3fba plus this change (rebased onto 27051deb27, whose node-alert dedup touches no file here); opened a Machines agent drawer (Apollo-114) and a vSphere datastore drawer (archive-tier)",
|
||||
"Before (same stack with this change stashed): Machines Identity read Hostname apollo-114 | Primary ID agent:host-linux-1 | Discovery agent:host-linux-1 | Metrics Target agent:host-linux-1 | Platform ID host-linux-1 | IP Addresses | Aliases agent:host-linux-1, host-linux-1, apollo-114, agent-acdfdee2953587fb; vSphere repeated the Primary ID as Metrics Target and in Aliases, plus the hostname and Platform ID",
|
||||
"After: Machines reads Hostname apollo-114 | Primary ID agent:host-linux-1 | Platform ID host-linux-1 | IP Addresses | Aliases agent-acdfdee2953587fb; vSphere reads Hostname | Primary ID | Platform ID | Tags | Aliases vc-mock-1:datastore:datastore-202, datastore-202. Every identifier appears once",
|
||||
"Phone (390): both drawers' Identity cards read the same single values, stacked label/value rows, nothing clipped"
|
||||
],
|
||||
"interactions": [
|
||||
"Opened each drawer and read its Identity card, before and after"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "c4db7a3fba35a14ee41675483726213644b31edc",
|
||||
"verified_at": "2026-10-06T14:33:38Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx": "d8bfb5052c0c150ebf1c11e1672f8776e1caa159805a66774ebaaf57bd66a6a2"
|
||||
},
|
||||
"routes": [
|
||||
"/standalone/availability"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 600,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 768,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 1024,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 1920,
|
||||
"height": 1000
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack on main b19c6aaf70 plus this change (rebased onto c4db7a3fba, whose backup-alert revalidation touches no frontend file); seven availability checks, one of them (Frontend Swarm service) observed from two locations and failing",
|
||||
"Before: that row's Result cell read 'failed\u2026' with a '2 locations \u00b7 2/2 reporting' chip spilling past the column (the chip's flex and the badge's inline-flex left it inline); the result clipped at every width up to 1440. On phones the check names were cut to 'Workshop do\u2026', 'Frontend Sw\u2026' (7 rows clipped by up to 77px) while Method and Target took the room",
|
||||
"After: one line per row, 32-33px at 390, 768 and 1440; the result ('failed') is whole at the cell's right edge and the source chip ('2/2 locations reporting', Settings' wording) truncates to its left with the full text on hover. On phones Method and Target hide (still in the drawer) and check names read in full except one by 3px; the Method cell now truncates with an ellipsis where it clipped"
|
||||
],
|
||||
"interactions": [
|
||||
"Measured every visible header and cell for clipping at six widths",
|
||||
"Measured row heights at three widths",
|
||||
"Hovered the source chip for its full text"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "3b290640a51ef0f488f9cd06b4aa644adbdcc6b3",
|
||||
"verified_at": "2026-10-06T15:43:08Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/kubernetes/KubernetesNodesTable.tsx",
|
||||
"frontend-modern/src/features/kubernetes/KubernetesPageSurface.tsx",
|
||||
"frontend-modern/src/features/kubernetes/kubernetesPageModel.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/kubernetes/KubernetesNodesTable.tsx": "8861f6f170dd9ebab4eebe949e3a311af46916aa10d8d2d2768a33e3d64b2d1a",
|
||||
"frontend-modern/src/features/kubernetes/KubernetesPageSurface.tsx": "9a2096f3ff967cec93024466d15c1b5117f08d89985c1955965d336a38d03bc0",
|
||||
"frontend-modern/src/features/kubernetes/kubernetesPageModel.ts": "0c2f0be19aa05f8e07f1167273b26b17b0643c73855df8246fcf5f591d3ae5b8"
|
||||
},
|
||||
"routes": [
|
||||
"/kubernetes/overview",
|
||||
"/kubernetes/nodes"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 1280,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack (backend built from base 3b290640a5 on its own port, Vite serving this tree); mock mode skips CheckHost, so the private binary added that call through a go build overlay of internal/monitoring/monitor_alerts.go that is not part of this commit",
|
||||
"Kubernetes nodes: Prod Euw1 K8s 02 (agent row agent-c8db98b4db98ed5c) has a backend-raised critical alert keyed agent:host-k8s-k8s-production-1-k8s-production-1-node-2-955a86fe (node prod-euw1-k8s-02, Agent memory at 100.0%); before the change its row was untinted while its drawer listed the alert, now the row is red-50 and its drawer lists the same alert",
|
||||
"Kubernetes nodes default order: K8s 02 (Ready, critical alert) first, then K8s 03 (Degraded), then K8s 01 (Ready); K8s 01 and K8s 03 untinted with no drawer alerts",
|
||||
"Kubernetes overview: Nodes needing attention lists K8s 02 (red) then K8s 03 (Degraded); before the change it listed only K8s 03",
|
||||
"After acknowledging the K8s 02 alert through the private stack API: its row is untinted, it drops out of Nodes needing attention, the nodes table falls back to K8s 03, K8s 01, K8s 02, and its drawer still lists the alert as Acknowledged; unacknowledged again afterwards",
|
||||
"Expanded K8s 02 row drops the tint (surface hover background) while its drawer shows the alert",
|
||||
"Light theme at 1440px and 390px and dark theme at 1280px: red-50 / red-950/25 on K8s 02 only, no horizontal overflow at 390px",
|
||||
"TrueNAS and vSphere tables unchanged: truenas-main stays amber for its pool alerts (node truenas-main) and its drawer lists the pool problems; esxi-07 amber for its own alert",
|
||||
"No page errors or console errors"
|
||||
],
|
||||
"interactions": [
|
||||
"Open Kubernetes overview and Nodes, read each node row's tint and order",
|
||||
"Expand and collapse the K8s 02 row drawer and read its Needs attention list",
|
||||
"Acknowledge then unacknowledge the K8s 02 memory alert and re-read the overview list, row tint, order and drawer",
|
||||
"Repeat the overview and nodes checks at the 390px phone viewport (row tap to expand) and in dark mode"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "982a6f6148cf9d39883e4443191b4d40b59cfd7c",
|
||||
"verified_at": "2026-10-06T15:03:00Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx",
|
||||
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx",
|
||||
"frontend-modern/src/features/alerts/alertHistoryModel.ts",
|
||||
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx",
|
||||
"frontend-modern/src/features/patrol/patrolHomePresentation.ts",
|
||||
"frontend-modern/src/types/api.ts",
|
||||
"frontend-modern/src/types/operationalTrust.ts",
|
||||
"frontend-modern/src/utils/alertIncidentPresentation.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "94bb73163dd86c1f045a0b5559b42486198ab824ef2abccfc3c8c1dcc0ff2784",
|
||||
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx": "aacc7232596a47d64c7e6ec3b29dfaf51be6e1691d9be906b30bfce49f4daf6f",
|
||||
"frontend-modern/src/features/alerts/alertHistoryModel.ts": "30548565d29ca16fe1c1192687c881defc064ba1b267fdcd40fc28922a51fed4",
|
||||
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx": "b3c87b3ead9bde22a091b2c080b9eaca18face9ba153085bb1a1dbb9b6ae9f54",
|
||||
"frontend-modern/src/features/patrol/patrolHomePresentation.ts": "04338cdce5b54bc041267b325a6985a9a8f84ef9e5a8dba8040c7d9fc5ab1811",
|
||||
"frontend-modern/src/types/api.ts": "17a948b6ba78780ef2b95c1f397d2d9a04eebd2726c568a8316e25e7c8f9e6b3",
|
||||
"frontend-modern/src/types/operationalTrust.ts": "9a50a869b1f6ef2885f30a354fb70b36b64fbb1a3f00ba4476d592389f9ba954",
|
||||
"frontend-modern/src/utils/alertIncidentPresentation.ts": "f77c42b48f22bddcccb43a3da43b3771bb03d25c6d23b7efa200a34dae1da0e9"
|
||||
},
|
||||
"routes": [
|
||||
"/alerts/history",
|
||||
"/patrol?attention=pve1::metric-threshold:memory"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1280,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 375,
|
||||
"height": 812
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private non-mock stack (backend :7691, Vite :5291) on a data dir whose alert history was written by the real alerts manager: a pve1 node memory alert closed through releaseCanonicalMetricAlert with a moved_to_agent resolution (successor pve1 (Host Agent)) and a pve1 CPU alert closed as an ordinary release; the handover record's resolve transition carries cause ownership_transferred",
|
||||
"Alerts history at 1280: the CPU row's status badge reads resolved, the memory row's reads moved to agent in blue, and its title is 'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.'",
|
||||
"Memory row Timeline expanded: the incident timeline lists 'Node memory at 95.0%' then 'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.' where it said 'Alert resolved' before; the incident header still says resolved (incident status vocabulary, unchanged)",
|
||||
"Alerts history at 375: the memory card shows the moved to agent badge and the account as a muted line under the breach message; the CPU card is unchanged",
|
||||
"Patrol deep link to the moved record at 1280 and 375: the detail badge reads Moved (info tone) instead of green Resolved, the handover sentence sits directly under 'Node memory at 95.0%', and Evidence and history shows 'Open to Resolved, Ownership transferred' with the same reason; no Recovery evidence label anywhere; no horizontal scroll at 375 and no console errors"
|
||||
],
|
||||
"interactions": [
|
||||
"Loaded /alerts/history at 1280x900 and read both rows' status badges and titles",
|
||||
"Clicked the memory row's Timeline button and read the expanded incident timeline",
|
||||
"Reloaded /alerts/history at 375x812 and read both history cards",
|
||||
"Opened /patrol?attention=pve1::metric-threshold:memory at 1280x900 and 375x812, opened Evidence and history, and read the badge, summary, handover line and timeline",
|
||||
"Checked document width against the viewport at 375 and read the console for errors"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "b19c6aaf709637adedd46de9f61307252ab1131d",
|
||||
"verified_at": "2026-10-06T14:18:34Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Workloads/EnhancedCPUBar.tsx",
|
||||
"frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx",
|
||||
"frontend-modern/src/features/standalone/StandalonePageSurface.tsx",
|
||||
"frontend-modern/src/features/truenas/TrueNASPageSurface.tsx",
|
||||
"frontend-modern/src/features/vmware/VmwarePageSurface.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Workloads/EnhancedCPUBar.tsx": "7aad8eb3f5eb326aac32fb698e95b7fb76d3c2683a794dd104e0ed63e9080bf6",
|
||||
"frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx": "3daf35d191a8df3d47c37d1813763730c1849a5097e3e707ecc6b3faf10143a6",
|
||||
"frontend-modern/src/features/standalone/StandalonePageSurface.tsx": "620c1ebcc9fea62047bea73c4505a21c7ed3fc3f28791e50f4d07b3b5203d6d2",
|
||||
"frontend-modern/src/features/truenas/TrueNASPageSurface.tsx": "52aa6b488694cbeb3103e83abe98cb739953afdc734d9f26b5b3d595f47fbbd0",
|
||||
"frontend-modern/src/features/vmware/VmwarePageSurface.tsx": "cd6bd1afc48a59908b7b0940e9c8e33a084f3f55c513815218e53072db534ce1"
|
||||
},
|
||||
"routes": [
|
||||
"/standalone/machines",
|
||||
"/proxmox/overview",
|
||||
"/truenas/overview",
|
||||
"/vmware/overview"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 1000
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack on main b19c6aaf70 plus this change; Machines has three older agents and Proxmox five, TrueNAS and vSphere none (their notice copy is covered by the page contract tests)",
|
||||
"Before: the Machines notice read 'Update them to v4.26.0 for the latest agent command support and agent-managed platform detail', Proxmox 'agent-contributed Proxmox node detail and command support'",
|
||||
"After: 'Update them to v4.26.0 for the latest fixes and machine details' on Machines and '...fixes and node details' on Proxmox; the notice no longer promises Patrol actions or command support, which is a separate setting the update keeps",
|
||||
"Before: the CPU cell's '22% (18)' had nothing explaining the 18; its hover panel listed Usage and Load (1m) only. After: the hover panel's first row reads 'CPUs 18'"
|
||||
],
|
||||
"interactions": [
|
||||
"Hovered a machine's CPU bar and read the panel",
|
||||
"Read the outdated-agent notice on each platform page"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "d4315cef519bb5729a2b9117cae5d28724c17d84",
|
||||
"verified_at": "2026-10-06T12:59:58Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/shared/InlineDetailTableRow.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/shared/InlineDetailTableRow.tsx": "27f6c907846d17ef0742386e766719d6e529b17223b1dcc3b586e0caa48cbb96"
|
||||
},
|
||||
"routes": [
|
||||
"/standalone",
|
||||
"/proxmox",
|
||||
"/vmware/activity"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 1024,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 768,
|
||||
"height": 900
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack (backend + Vite on 7747/5267) built from main 92d946d057 plus this change, compared with the same stack on unchanged main",
|
||||
"Before, /standalone machines table at 1440: hiding Uptime from View > Columns with a machine drawer open left the drawer at colspan 9 against 8 visible headers, so the header and rows shrank to 939px and a blank 436px phantom column filled the table's right side",
|
||||
"Before, /standalone at 1440: showing System with the drawer open left the drawer at colspan 7 against 8 headers, stopping 79px short of the table edge under the actions column",
|
||||
"Before, /proxmox guest table at 1440, 1024 and 768: hiding Type, Uptime or Backup with a guest drawer open left colspan one above the visible header count (the fixed layout collapses the phantom column to about 1px there)",
|
||||
"After, both tables at 1440 and 768: drawer colSpan equals the visible header and summary cell count after every hide and show, and the drawer, last header cell and last summary cell all end at the table's right edge",
|
||||
"After, growing span case: the colspan update alone left Chromium laying the widened cell out at its old 7-column width (1241px of 1320px); with the one-off width nudge the cell measures the full table width while keeping the same row node",
|
||||
"/vmware/activity with the mock's single vCenter: the Activity drawer still spans 6 of 6 header columns at 1440 and 5 of 5 at 768 (the vCenter-count flip itself is covered by the unit test that removes and restores a summary cell)"
|
||||
],
|
||||
"interactions": [
|
||||
"Opened a machine drawer on /standalone, then hid and showed Uptime, System, Agent and Last seen through View > Columns while the drawer stayed open, measuring drawer colSpan, header and summary visible cell counts and right edges after each toggle",
|
||||
"Opened a guest drawer on /proxmox, hid a column first, then showed and hid Type again through View > Columns with the drawer open, measuring the same values",
|
||||
"Confirmed with a node tag that the drawer row stayed mounted across toggles (no remount, so drawer tab state is kept)",
|
||||
"Opened a /vmware/activity row drawer at both widths and compared its span with the visible headers",
|
||||
"Captured clipped screenshots of the /standalone drawer before and after each toggle"
|
||||
]
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
} from '@/components/shared/detailSectionModel';
|
||||
import { TagBadges } from '@/components/shared/TagBadges';
|
||||
import { formatBytes, formatRelativeTime, formatUptime } from '@/utils/format';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
import { getDockerImageRegistryLink } from '@/features/docker/dockerImageReference';
|
||||
import {
|
||||
getDockerContainerHealthCheckPresentation,
|
||||
@@ -92,7 +93,7 @@ const dockerRuntimeRow = (
|
||||
})
|
||||
: null;
|
||||
|
||||
const dockerSection = (docker: NonNullable<Resource['docker']>): DetailSection => {
|
||||
const dockerSection = (docker: NonNullable<Resource['docker']>, now: number): DetailSection => {
|
||||
const labels = docker.labels ?? {};
|
||||
const labelEntries = Object.entries(labels);
|
||||
const createdAt = dockerTimestampMillis(docker.createdAt);
|
||||
@@ -157,17 +158,17 @@ const dockerSection = (docker: NonNullable<Resource['docker']>): DetailSection =
|
||||
})
|
||||
: null,
|
||||
createdAt
|
||||
? makeDetailRow('Created', formatRelativeTime(createdAt), {
|
||||
? makeDetailRow('Created', formatRelativeTime(createdAt, { now }), {
|
||||
title: new Date(createdAt).toLocaleString(),
|
||||
})
|
||||
: null,
|
||||
startedAt
|
||||
? makeDetailRow('Started', formatRelativeTime(startedAt), {
|
||||
? makeDetailRow('Started', formatRelativeTime(startedAt, { now }), {
|
||||
title: new Date(startedAt).toLocaleString(),
|
||||
})
|
||||
: null,
|
||||
finishedAt
|
||||
? makeDetailRow('Finished', formatRelativeTime(finishedAt), {
|
||||
? makeDetailRow('Finished', formatRelativeTime(finishedAt, { now }), {
|
||||
title: new Date(finishedAt).toLocaleString(),
|
||||
})
|
||||
: null,
|
||||
@@ -210,6 +211,10 @@ const dockerSection = (docker: NonNullable<Resource['docker']>): DetailSection =
|
||||
};
|
||||
|
||||
export const InlineResourceSummaryTables: Component<ResourceSummaryPresentationProps> = (props) => {
|
||||
// An open drawer keeps its rows while the resource's timestamps stay the
|
||||
// same, so container and controller ages read the shared clock to keep
|
||||
// moving.
|
||||
const now = useRelativeTimeNow();
|
||||
const sections = (): DetailSection[] => {
|
||||
const docker = dockerContainerMeta(props.resource);
|
||||
const uptimeSeconds = docker
|
||||
@@ -317,8 +322,8 @@ export const InlineResourceSummaryTables: Component<ResourceSummaryPresentationP
|
||||
]),
|
||||
}
|
||||
: null,
|
||||
props.content !== 'overview' && docker ? dockerSection(docker) : null,
|
||||
props.content !== 'overview' ? buildKubernetesControllerSection(props.resource) : null,
|
||||
props.content !== 'overview' && docker ? dockerSection(docker, now()) : null,
|
||||
props.content !== 'overview' ? buildKubernetesControllerSection(props.resource, now()) : null,
|
||||
props.content !== 'overview'
|
||||
? {
|
||||
label: 'Identity',
|
||||
|
||||
+6
-4
@@ -799,7 +799,7 @@ describe('ResourceDetailDrawer runtime and identity cards', () => {
|
||||
expect(getByText('Primary ID').parentElement?.textContent).toContain('node:homelab-delly');
|
||||
});
|
||||
|
||||
it('shows canonical metrics target identity for docker-backed host resources', () => {
|
||||
it('shows the canonical metrics target once, as the Primary ID', () => {
|
||||
const resource = baseResource({
|
||||
id: 'hash-docker-resource',
|
||||
type: 'docker-host',
|
||||
@@ -824,12 +824,14 @@ describe('ResourceDetailDrawer runtime and identity cards', () => {
|
||||
},
|
||||
});
|
||||
|
||||
const { container, getByText, getAllByText } = render(() => (
|
||||
const { container, getByText, getAllByText, queryByText } = render(() => (
|
||||
<ResourceDetailDrawer resource={resource} />
|
||||
));
|
||||
|
||||
expect(getByText('Metrics Target')).toBeInTheDocument();
|
||||
expect(getAllByText('docker-host:docker-host-1').length).toBeGreaterThan(1);
|
||||
// The metrics target is the Primary ID, so neither its row nor an alias
|
||||
// repeats it.
|
||||
expect(queryByText('Metrics Target')).toBeNull();
|
||||
expect(getAllByText('docker-host:docker-host-1')).toHaveLength(1);
|
||||
expect(getByText('Aliases')).toBeInTheDocument();
|
||||
expect(getAllByText('docker-host-1').length).toBeGreaterThan(0);
|
||||
expect(container.querySelector('.text-\\[11px\\].text-muted.truncate')).toBeNull();
|
||||
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
import { cleanup, render, screen, within } from '@solidjs/testing-library';
|
||||
import { createRoot } from 'solid-js';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { formatPlatformTableDateTimeValue } from '@/features/platformPage/sharedPlatformPage';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { InlineResourceSummaryTables } from '../ResourceDetailSummary';
|
||||
import { useResourceDetailDrawerDerivedState } from '../useResourceDetailDrawerDerivedState';
|
||||
import type { UseResourceDetailDrawerStateResult } from '../useResourceDetailDrawerState';
|
||||
|
||||
// The summary only reads identity and coverage accessors from the drawer
|
||||
// state; the ages under test come from the resource itself.
|
||||
const drawerState = {
|
||||
primaryIdentityRows: () => [],
|
||||
identityIpValues: () => [],
|
||||
identityAliasValues: () => [],
|
||||
aliasPreviewValues: () => [],
|
||||
hasAliasOverflow: () => false,
|
||||
identityCardHasRichData: () => true,
|
||||
sourceSummary: () => null,
|
||||
lastSeen: () => '',
|
||||
lastSeenAbsolute: () => '',
|
||||
} as unknown as UseResourceDetailDrawerStateResult;
|
||||
|
||||
const makeResource = ({
|
||||
id,
|
||||
type,
|
||||
...overrides
|
||||
}: Partial<Resource> & Pick<Resource, 'id' | 'type'>): Resource => ({
|
||||
id,
|
||||
name: id,
|
||||
displayName: id,
|
||||
platformId: 'host-1',
|
||||
platformType: 'kubernetes',
|
||||
sourceType: 'agent',
|
||||
status: 'online',
|
||||
type,
|
||||
lastSeen: Date.parse('2026-05-24T13:31:00Z'),
|
||||
...overrides,
|
||||
});
|
||||
|
||||
const absoluteTime = (timestamp: string): string =>
|
||||
formatPlatformTableDateTimeValue(timestamp, { dateTimeFormat: { year: 'numeric' } });
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe('resource drawer summary relative ages', () => {
|
||||
it('keeps an open Kubernetes controller summary age moving without new data', () => {
|
||||
vi.useFakeTimers({ now: new Date('2026-05-24T13:31:00Z') });
|
||||
|
||||
render(() => (
|
||||
<InlineResourceSummaryTables
|
||||
resource={makeResource({
|
||||
id: 'nightly-import',
|
||||
type: 'k8s-job',
|
||||
kubernetes: {
|
||||
clusterName: 'prod',
|
||||
namespace: 'batch',
|
||||
resourceKind: 'Job',
|
||||
completionTime: '2026-05-24T13:00:00Z',
|
||||
},
|
||||
})}
|
||||
drawer={drawerState}
|
||||
showPlatformId={false}
|
||||
/>
|
||||
));
|
||||
|
||||
const section = () => screen.getByTestId('resource-kubernetes-controller-section');
|
||||
expect(
|
||||
within(section()).getByText(`${absoluteTime('2026-05-24T13:00:00Z')} (31m ago)`),
|
||||
).toBeInTheDocument();
|
||||
|
||||
vi.advanceTimersByTime(2 * 60 * 60 * 1000);
|
||||
|
||||
// The absolute time is unchanged; only the age after it moves.
|
||||
expect(
|
||||
within(section()).getByText(`${absoluteTime('2026-05-24T13:00:00Z')} (2h ago)`),
|
||||
).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps an open container summary Created and Started ages moving without new data', () => {
|
||||
vi.useFakeTimers({ now: new Date('2026-05-24T13:31:00Z') });
|
||||
|
||||
render(() => (
|
||||
<InlineResourceSummaryTables
|
||||
resource={makeResource({
|
||||
id: 'web',
|
||||
type: 'app-container',
|
||||
platformType: 'docker',
|
||||
docker: {
|
||||
containerId: 'abc123',
|
||||
containerState: 'running',
|
||||
createdAt: '2026-05-24T11:31:00Z',
|
||||
startedAt: '2026-05-24T13:21:00Z',
|
||||
},
|
||||
})}
|
||||
drawer={drawerState}
|
||||
showPlatformId={false}
|
||||
/>
|
||||
));
|
||||
|
||||
const row = (label: string) =>
|
||||
within(screen.getByTestId('resource-docker-container-section'))
|
||||
.getByText(label)
|
||||
.closest('tr');
|
||||
expect(row('Created')).toHaveTextContent('2 hours ago');
|
||||
expect(row('Started')).toHaveTextContent('10 mins ago');
|
||||
|
||||
vi.advanceTimersByTime(3 * 60 * 60 * 1000);
|
||||
|
||||
expect(row('Created')).toHaveTextContent('5 hours ago');
|
||||
expect(row('Started')).toHaveTextContent('3 hours ago');
|
||||
});
|
||||
|
||||
it('keeps an open drawer Last seen moving for a resource that stops reporting', () => {
|
||||
vi.useFakeTimers({ now: new Date('2026-05-24T13:31:00Z') });
|
||||
|
||||
const { state, dispose } = createRoot((dispose) => ({
|
||||
state: useResourceDetailDrawerDerivedState({
|
||||
resource: makeResource({
|
||||
id: 'node-1',
|
||||
type: 'agent',
|
||||
lastSeen: Date.parse('2026-05-24T13:30:00Z'),
|
||||
}),
|
||||
debugEnabled: () => false,
|
||||
discoveryFeatureEnabled: () => false,
|
||||
resourceIntelligence: () => null,
|
||||
}),
|
||||
dispose,
|
||||
}));
|
||||
|
||||
expect(state.lastSeen()).toBe('1 min ago');
|
||||
|
||||
// The resource stays silent: its lastSeen never changes again.
|
||||
vi.advanceTimersByTime(2 * 60 * 60 * 1000);
|
||||
|
||||
expect(state.lastSeen()).toBe('2 hours ago');
|
||||
dispose();
|
||||
});
|
||||
});
|
||||
+59
@@ -25,6 +25,65 @@ const baseResource = (overrides: Partial<Resource> = {}): Resource => ({
|
||||
});
|
||||
|
||||
describe('resourceDetailDrawerIdentityModel', () => {
|
||||
it('filters aliases against the name the drawer header actually shows', () => {
|
||||
const identityView = buildResourceIdentityView(
|
||||
baseResource({
|
||||
displayName: '',
|
||||
name: 'legacy',
|
||||
platformId: 'legacy',
|
||||
canonicalIdentity: {
|
||||
displayName: 'Canonical label',
|
||||
aliases: ['legacy', 'Canonical label'],
|
||||
},
|
||||
} as Partial<Resource>),
|
||||
);
|
||||
// The header reads "Canonical label"; "legacy" is shown nowhere else.
|
||||
expect(identityView.identityAliasValues).toContain('legacy');
|
||||
expect(identityView.identityAliasValues).not.toContain('Canonical label');
|
||||
});
|
||||
|
||||
it('counts a shown Platform ID as identity data', () => {
|
||||
const identityView = buildResourceIdentityView(
|
||||
baseResource({
|
||||
name: 'vm-101',
|
||||
displayName: 'VM 101',
|
||||
platformId: 'qemu-101',
|
||||
canonicalIdentity: { aliases: ['qemu-101'] },
|
||||
} as Partial<Resource>),
|
||||
);
|
||||
expect(identityView.identityAliasValues).toEqual([]);
|
||||
expect(identityView.identityCardHasRichData).toBe(true);
|
||||
});
|
||||
|
||||
it('lists only aliases the drawer does not already show', () => {
|
||||
const identityView = buildResourceIdentityView(
|
||||
baseResource({
|
||||
id: 'agent-acdfdee2953587fb',
|
||||
type: 'agent',
|
||||
name: 'Apollo-114',
|
||||
displayName: 'Apollo-114',
|
||||
platformId: 'host-linux-1',
|
||||
identity: { hostname: 'apollo-114' },
|
||||
discoveryTarget: {
|
||||
resourceType: 'agent',
|
||||
agentId: 'host-linux-1',
|
||||
resourceId: 'host-linux-1',
|
||||
},
|
||||
metricsTarget: { resourceType: 'agent', resourceId: 'host-linux-1' },
|
||||
canonicalIdentity: {
|
||||
aliases: ['agent:host-linux-1', 'host-linux-1', 'apollo-114', 'agent-acdfdee2953587fb'],
|
||||
},
|
||||
} as Partial<Resource>),
|
||||
);
|
||||
|
||||
// Hostname, Primary ID and the Platform ID row already show these.
|
||||
expect(identityView.primaryIdentityRows.map((row) => row.label)).toEqual([
|
||||
'Hostname',
|
||||
'Primary ID',
|
||||
]);
|
||||
expect(identityView.identityAliasValues).toEqual(['agent-acdfdee2953587fb']);
|
||||
});
|
||||
|
||||
it('builds canonical identity card state including alias collapse policy', () => {
|
||||
const identityView = buildResourceIdentityView(
|
||||
baseResource({
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import type { Resource } from '@/types/resource';
|
||||
import {
|
||||
getPrimaryResourceIdentityRows,
|
||||
getPreferredInfrastructureDisplayName,
|
||||
getResourceIdentityAliases,
|
||||
shouldShowResourcePlatformId,
|
||||
type ResourceIdentityRow,
|
||||
} from '@/utils/resourceIdentity';
|
||||
import { formatIdentifierLabel } from '@/utils/textPresentation';
|
||||
@@ -26,9 +28,24 @@ export type ResourceDetailDrawerIdentityView = {
|
||||
const ALIAS_COLLAPSE_THRESHOLD = 4;
|
||||
|
||||
export const buildResourceIdentityView = (resource: Resource): ResourceDetailDrawerIdentityView => {
|
||||
const identityAliasValues = getResourceIdentityAliases(resource);
|
||||
const identityIpValues = resource.identity?.ips ?? [];
|
||||
const primaryIdentityRows = getPrimaryResourceIdentityRows(resource);
|
||||
// The drawer header already shows the resource's display name, and the rows
|
||||
// above show these IDs and (when it differs from every name) the platform
|
||||
// ID, so Aliases lists only the other names the resource goes by.
|
||||
const platformIdShown = shouldShowResourcePlatformId(resource);
|
||||
const shown = new Set(
|
||||
[
|
||||
getPreferredInfrastructureDisplayName(resource),
|
||||
platformIdShown ? resource.platformId : undefined,
|
||||
...primaryIdentityRows.map((row) => row.value),
|
||||
]
|
||||
.map((value) => value?.trim().toLowerCase())
|
||||
.filter((value): value is string => Boolean(value)),
|
||||
);
|
||||
const identityAliasValues = getResourceIdentityAliases(resource).filter(
|
||||
(alias) => !shown.has(alias.trim().toLowerCase()),
|
||||
);
|
||||
|
||||
return {
|
||||
identityAliasValues,
|
||||
@@ -36,6 +53,7 @@ export const buildResourceIdentityView = (resource: Resource): ResourceDetailDra
|
||||
primaryIdentityRows,
|
||||
identityCardHasRichData:
|
||||
primaryIdentityRows.length > 0 ||
|
||||
platformIdShown ||
|
||||
identityIpValues.length > 0 ||
|
||||
(resource.tags?.length || 0) > 0 ||
|
||||
identityAliasValues.length > 0,
|
||||
|
||||
+19
-10
@@ -147,8 +147,14 @@ const dangerWhenPositive = (count: number): DetailValueTone => (count > 0 ? 'dan
|
||||
// Kubernetes reports controller timestamps as RFC 3339 strings. The table row
|
||||
// only has room for an age, so the expansion carries the absolute time first
|
||||
// and the age after it: "Oct 2, 2026, 09:13 PM (1h ago)". The value wraps
|
||||
// because a phone-width detail cell is too narrow for both on one line.
|
||||
const timestampRow = (label: string, timestamp?: string | null): DetailRow | null => {
|
||||
// because a phone-width detail cell is too narrow for both on one line. The
|
||||
// age is measured from `now`, which an open drawer passes from the shared
|
||||
// relative-time clock so the age keeps moving while the drawer stays open.
|
||||
const timestampRow = (
|
||||
label: string,
|
||||
timestamp: string | null | undefined,
|
||||
now: number | undefined,
|
||||
): DetailRow | null => {
|
||||
const raw = asString(timestamp);
|
||||
if (!raw) return null;
|
||||
const absolute = formatPlatformTableDateTimeValue(raw, {
|
||||
@@ -156,7 +162,7 @@ const timestampRow = (label: string, timestamp?: string | null): DetailRow | nul
|
||||
dateTimeFormat: { year: 'numeric' },
|
||||
});
|
||||
if (!absolute) return makeRow(label, raw, { wrap: true });
|
||||
const relative = formatPlatformTableRelativeTimeValue(raw, { emptyText: '' });
|
||||
const relative = formatPlatformTableRelativeTimeValue(raw, { emptyText: '', now });
|
||||
return makeRow(label, relative ? `${absolute} (${relative})` : absolute, { wrap: true });
|
||||
};
|
||||
|
||||
@@ -173,7 +179,7 @@ type ControllerRows = { label: string; rows: Array<DetailRow | null> };
|
||||
|
||||
// Each kind leads with what its narrow table row drops: the Detail column
|
||||
// (service name, timestamps) below the large layout and Target on a phone.
|
||||
const controllerRows = (resource: Resource): ControllerRows | null => {
|
||||
const controllerRows = (resource: Resource, now: number | undefined): ControllerRows | null => {
|
||||
const k = resource.kubernetes;
|
||||
if (!k) return null;
|
||||
switch (resource.type) {
|
||||
@@ -218,8 +224,8 @@ const controllerRows = (resource: Resource): ControllerRows | null => {
|
||||
return {
|
||||
label: 'Job',
|
||||
rows: [
|
||||
timestampRow('Started', k.startTime),
|
||||
timestampRow('Completed', k.completionTime),
|
||||
timestampRow('Started', k.startTime, now),
|
||||
timestampRow('Completed', k.completionTime, now),
|
||||
makeRow(
|
||||
'Duration',
|
||||
formatPlatformTableDurationValue(elapsedSeconds(k.startTime, k.completionTime), {
|
||||
@@ -237,8 +243,8 @@ const controllerRows = (resource: Resource): ControllerRows | null => {
|
||||
label: 'CronJob',
|
||||
rows: [
|
||||
makeRow('Schedule', k.schedule, { valueClass: 'font-mono' }),
|
||||
timestampRow('Last run', k.lastScheduleTime),
|
||||
timestampRow('Last success', k.lastSuccessfulTime),
|
||||
timestampRow('Last run', k.lastScheduleTime, now),
|
||||
timestampRow('Last success', k.lastSuccessfulTime, now),
|
||||
makeRow('Suspended', k.suspend === true ? 'Yes' : null, { tone: 'warning' }),
|
||||
countRow('Active', k.active),
|
||||
],
|
||||
@@ -253,8 +259,11 @@ const controllerRows = (resource: Resource): ControllerRows | null => {
|
||||
// controllers table hides Scope and Detail below its large layout and Target
|
||||
// on a phone, so without this a Job's completion time or a CronJob's last
|
||||
// success is unreachable from a narrow row.
|
||||
export const buildKubernetesControllerSection = (resource: Resource): DetailSection | null => {
|
||||
const controller = controllerRows(resource);
|
||||
export const buildKubernetesControllerSection = (
|
||||
resource: Resource,
|
||||
now?: number,
|
||||
): DetailSection | null => {
|
||||
const controller = controllerRows(resource, now);
|
||||
if (!controller) return null;
|
||||
const rows = compactRows([
|
||||
...controller.rows,
|
||||
|
||||
+8
-3
@@ -7,6 +7,7 @@ import type { HostNetworkInterface } from '@/types/api';
|
||||
import { requiresGovernedResourceDisplay } from '@/types/resource';
|
||||
import type { ResourceVMwareMeta } from '@/types/resource';
|
||||
import { formatAbsoluteTime, formatRelativeTime } from '@/utils/format';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
import { getAgentStatusIndicator } from '@/utils/status';
|
||||
import {
|
||||
dedupeResourceBadges,
|
||||
@@ -135,7 +136,11 @@ export const useResourceDetailDrawerDerivedState = (
|
||||
resolveResourceLabelInput?.(resourceId)?.trim() || resourceId;
|
||||
const statusIndicator = createMemo(() => getAgentStatusIndicator({ status: resource().status }));
|
||||
const healthIssue = createMemo(() => getResourceHealthIssuePresentation(resource()));
|
||||
const lastSeen = createMemo(() => formatRelativeTime(resource().lastSeen));
|
||||
// An open drawer keeps its rows while a resource's timestamps stay the same,
|
||||
// which is exactly when a silent resource's Last seen must keep aging, so
|
||||
// the drawer's ages read the shared clock.
|
||||
const now = useRelativeTimeNow();
|
||||
const lastSeen = createMemo(() => formatRelativeTime(resource().lastSeen, { now: now() }));
|
||||
const lastSeenAbsolute = createMemo(() => formatAbsoluteTime(resource().lastSeen));
|
||||
|
||||
const platformBadge = createMemo(() => getPlatformBadge(resource().platformType));
|
||||
@@ -210,7 +215,7 @@ export const useResourceDetailDrawerDerivedState = (
|
||||
if (!raw) return '';
|
||||
const parsed = Date.parse(raw);
|
||||
if (!Number.isFinite(parsed)) return '';
|
||||
return formatRelativeTime(parsed);
|
||||
return formatRelativeTime(parsed, { now: now() });
|
||||
});
|
||||
const dockerHostCommand = createMemo(() => dockerHostData()?.command);
|
||||
const dockerHostCommandActive = createMemo(() => {
|
||||
@@ -278,7 +283,7 @@ export const useResourceDetailDrawerDerivedState = (
|
||||
if (!raw) return '';
|
||||
const parsed = Date.parse(raw);
|
||||
if (!Number.isFinite(parsed)) return '';
|
||||
return formatRelativeTime(parsed);
|
||||
return formatRelativeTime(parsed, { now: now() });
|
||||
});
|
||||
const pbsVisibleJobBreakdown = createMemo(() => buildPbsVisibleJobBreakdown(pbsData()));
|
||||
const pbsActiveTasks = createMemo(() => buildPbsActiveTasks(pbsData()));
|
||||
|
||||
@@ -63,6 +63,15 @@ export function EnhancedCPUBar(props: EnhancedCPUBarProps) {
|
||||
<div class="text-[9px] text-slate-400 mb-1.5 truncate max-w-[200px]">{props.model}</div>
|
||||
</Show>
|
||||
|
||||
{/* The bar's "(18)" beside the percentage is this count: logical
|
||||
CPUs on a host, configured vCPUs on a guest. */}
|
||||
<Show when={props.cores}>
|
||||
<div class="flex justify-between gap-3 py-0.5">
|
||||
<span class="text-slate-400">CPUs</span>
|
||||
<span class="font-medium text-base-content">{props.cores}</span>
|
||||
</div>
|
||||
</Show>
|
||||
|
||||
<div class="flex justify-between gap-3 py-0.5">
|
||||
<span class="text-slate-400">Usage</span>
|
||||
<span class={`font-medium ${presentation().tooltipUsageClass}`}>
|
||||
|
||||
@@ -252,6 +252,20 @@ describe('EnhancedCPUBar', () => {
|
||||
expect(screen.getByText('CPU Details')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('names the CPU count shown beside the percentage', async () => {
|
||||
const { container } = render(() => <EnhancedCPUBar usage={22} cores={18} />);
|
||||
expect(container.textContent).toContain('(18)');
|
||||
await fireEvent.mouseEnter(getBarTrigger(container));
|
||||
const label = screen.getByText('CPUs');
|
||||
expect(label.closest('.flex.justify-between')?.textContent).toContain('18');
|
||||
});
|
||||
|
||||
it('omits the CPU row when the count is unknown', async () => {
|
||||
const { container } = render(() => <EnhancedCPUBar usage={22} />);
|
||||
await fireEvent.mouseEnter(getBarTrigger(container));
|
||||
expect(screen.queryByText('CPUs')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('shows usage value in tooltip', async () => {
|
||||
const { container } = render(() => <EnhancedCPUBar usage={72} />);
|
||||
await fireEvent.mouseEnter(getBarTrigger(container));
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { createSignal, onCleanup, onMount, splitProps, type JSX } from 'solid-js';
|
||||
import { createEffect, createSignal, on, onCleanup, onMount, splitProps, type JSX } from 'solid-js';
|
||||
|
||||
import { TableCell, TableRow, type TableRowProps } from './Table';
|
||||
|
||||
@@ -88,12 +88,57 @@ export function InlineDetailTableRow(props: InlineDetailTableRowProps) {
|
||||
: undefined;
|
||||
if (tableShell) observer?.observe(tableShell);
|
||||
|
||||
// A column picker, or a column that depends on the data (one vCenter
|
||||
// versus several), can add, remove or hide summary cells while this row
|
||||
// stays open, without resizing the table shell. Watch the summary row's
|
||||
// own cells so the detail cell never keeps a stale span. Changes deeper
|
||||
// inside a cell are ignored: live metric updates must not re-measure.
|
||||
const summaryRow = detailRow?.previousElementSibling;
|
||||
let summaryCellsObserver: MutationObserver | undefined;
|
||||
if (summaryRow instanceof HTMLTableRowElement && typeof MutationObserver === 'function') {
|
||||
summaryCellsObserver = new MutationObserver((records) => {
|
||||
const summaryCellsChanged = records.some(
|
||||
(record) =>
|
||||
record.target === summaryRow ||
|
||||
(record.type === 'attributes' && record.target.parentNode === summaryRow),
|
||||
);
|
||||
if (summaryCellsChanged) syncColspanToVisibleSummaryCells();
|
||||
});
|
||||
summaryCellsObserver.observe(summaryRow, {
|
||||
childList: true,
|
||||
subtree: true,
|
||||
attributes: true,
|
||||
attributeFilter: ['class', 'style', 'hidden', 'colspan'],
|
||||
});
|
||||
}
|
||||
|
||||
onCleanup(() => {
|
||||
window.removeEventListener('resize', syncColspanToVisibleSummaryCells);
|
||||
observer?.disconnect();
|
||||
summaryCellsObserver?.disconnect();
|
||||
});
|
||||
});
|
||||
|
||||
// Callers that change the span they ask for (and rows rendered without a
|
||||
// summary row, which fall back to that span) re-sync on the change.
|
||||
createEffect(on(requestedColspan, syncColspanToVisibleSummaryCells, { defer: true }));
|
||||
|
||||
// Chromium keeps a fixed-layout table's column widths when a spanning
|
||||
// cell's colspan grows over a column added since the table last laid out,
|
||||
// so the detail cell would stop short of the new column. A change to the
|
||||
// cell's own width makes it recompute them; restore it after one layout.
|
||||
// The effect is not deferred so its first run records the mounted span.
|
||||
createEffect(
|
||||
on(effectiveColspan, (colspan, previousColspan) => {
|
||||
const detailCell = detailRow?.cells[0];
|
||||
if (!detailCell || previousColspan === undefined || colspan <= previousColspan) return;
|
||||
const width = detailCell.style.width;
|
||||
detailCell.style.width = '0px';
|
||||
void detailCell.offsetWidth;
|
||||
detailCell.style.width = width;
|
||||
}),
|
||||
);
|
||||
|
||||
return (
|
||||
<TableRow ref={detailRow} class={local.class} {...rest}>
|
||||
<TableCell
|
||||
|
||||
@@ -86,6 +86,166 @@ describe('InlineDetailTableRow', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('re-spans when a summary column is removed or restored while the row stays open', async () => {
|
||||
const [showSystem, setShowSystem] = createSignal(true);
|
||||
render(() => (
|
||||
<Table>
|
||||
<TableBody>
|
||||
<tr>
|
||||
<td>Machine</td>
|
||||
<Show when={showSystem()}>
|
||||
<td>System</td>
|
||||
</Show>
|
||||
<td>Seen</td>
|
||||
</tr>
|
||||
<InlineDetailTableRow colspan={3}>
|
||||
<div>Column picker detail</div>
|
||||
</InlineDetailTableRow>
|
||||
</TableBody>
|
||||
</Table>
|
||||
));
|
||||
const detailCell = () => screen.getByText('Column picker detail').closest('td');
|
||||
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '3'));
|
||||
setShowSystem(false);
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '2'));
|
||||
setShowSystem(true);
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '3'));
|
||||
// The relayout nudge for a growing span must not leave a width behind.
|
||||
expect(detailCell()?.style.width).toBe('');
|
||||
});
|
||||
|
||||
it('forces one layout of the widened detail cell when its span grows, none when it shrinks', async () => {
|
||||
const [showSystem, setShowSystem] = createSignal(false);
|
||||
render(() => (
|
||||
<Table>
|
||||
<TableBody>
|
||||
<tr>
|
||||
<td>Machine</td>
|
||||
<Show when={showSystem()}>
|
||||
<td>System</td>
|
||||
</Show>
|
||||
<td>Seen</td>
|
||||
</tr>
|
||||
<InlineDetailTableRow colspan={showSystem() ? 3 : 2}>
|
||||
<div>Growing span detail</div>
|
||||
</InlineDetailTableRow>
|
||||
</TableBody>
|
||||
</Table>
|
||||
));
|
||||
const detailCell = screen.getByText('Growing span detail').closest('td')!;
|
||||
await waitFor(() => expect(detailCell).toHaveAttribute('colspan', '2'));
|
||||
|
||||
// jsdom has no layout, so record the forced layout reads Chromium needs to
|
||||
// recompute fixed-layout column widths for the widened cell.
|
||||
const layoutReads: Array<{ width: string; colspan: string | null }> = [];
|
||||
const offsetWidth = vi
|
||||
.spyOn(HTMLElement.prototype, 'offsetWidth', 'get')
|
||||
.mockImplementation(function (this: HTMLElement) {
|
||||
if (this === detailCell) {
|
||||
layoutReads.push({ width: this.style.width, colspan: this.getAttribute('colspan') });
|
||||
}
|
||||
return 0;
|
||||
});
|
||||
try {
|
||||
// The requested span and the summary cells change in the same update.
|
||||
setShowSystem(true);
|
||||
await waitFor(() => expect(detailCell).toHaveAttribute('colspan', '3'));
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
expect(layoutReads).toEqual([{ width: '0px', colspan: '3' }]);
|
||||
expect(detailCell.style.width).toBe('');
|
||||
|
||||
layoutReads.length = 0;
|
||||
setShowSystem(false);
|
||||
await waitFor(() => expect(detailCell).toHaveAttribute('colspan', '2'));
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
expect(layoutReads).toEqual([]);
|
||||
} finally {
|
||||
offsetWidth.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('re-measures for a summary cell span change but not for churn inside a cell', async () => {
|
||||
const [uptimeSpan, setUptimeSpan] = createSignal(1);
|
||||
const [reading, setReading] = createSignal({ value: '24%', tone: 'text-green-600' });
|
||||
const [busy, setBusy] = createSignal(false);
|
||||
render(() => (
|
||||
<Table>
|
||||
<TableBody>
|
||||
<tr>
|
||||
<td>Machine</td>
|
||||
<td colspan={uptimeSpan()}>
|
||||
<span class={reading().tone}>{reading().value}</span>
|
||||
<Show when={busy()}>
|
||||
<em>refreshing</em>
|
||||
</Show>
|
||||
</td>
|
||||
</tr>
|
||||
<InlineDetailTableRow colspan={2}>
|
||||
<div>Live metric detail</div>
|
||||
</InlineDetailTableRow>
|
||||
</TableBody>
|
||||
</Table>
|
||||
));
|
||||
const detailCell = () => screen.getByText('Live metric detail').closest('td');
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '2'));
|
||||
|
||||
const getComputedStyleSpy = vi.spyOn(window, 'getComputedStyle');
|
||||
try {
|
||||
setReading({ value: '91%', tone: 'text-red-600' });
|
||||
setBusy(true);
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
expect(getComputedStyleSpy).not.toHaveBeenCalled();
|
||||
|
||||
setUptimeSpan(2);
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '3'));
|
||||
expect(getComputedStyleSpy).toHaveBeenCalled();
|
||||
} finally {
|
||||
getComputedStyleSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('re-spans when a summary cell is hidden in place while the row stays open', async () => {
|
||||
const [hideUptime, setHideUptime] = createSignal(false);
|
||||
render(() => (
|
||||
<Table>
|
||||
<TableBody>
|
||||
<tr>
|
||||
<td>Machine</td>
|
||||
<td style={{ display: hideUptime() ? 'none' : undefined }}>Uptime</td>
|
||||
<td>Seen</td>
|
||||
</tr>
|
||||
<InlineDetailTableRow colspan={3}>
|
||||
<div>Hidden cell detail</div>
|
||||
</InlineDetailTableRow>
|
||||
</TableBody>
|
||||
</Table>
|
||||
));
|
||||
const detailCell = () => screen.getByText('Hidden cell detail').closest('td');
|
||||
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '3'));
|
||||
setHideUptime(true);
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '2'));
|
||||
});
|
||||
|
||||
it('follows a changed requested span when there is no summary row to measure', async () => {
|
||||
const [colspan, setColspan] = createSignal(5);
|
||||
render(() => (
|
||||
<Table>
|
||||
<TableBody>
|
||||
<InlineDetailTableRow colspan={colspan()}>
|
||||
<div>Requested span detail</div>
|
||||
</InlineDetailTableRow>
|
||||
</TableBody>
|
||||
</Table>
|
||||
));
|
||||
const detailCell = () => screen.getByText('Requested span detail').closest('td');
|
||||
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '5'));
|
||||
setColspan(6);
|
||||
await waitFor(() => expect(detailCell()).toHaveAttribute('colspan', '6'));
|
||||
});
|
||||
|
||||
it('restores focus to the controlling disclosure when focused detail content closes', async () => {
|
||||
const Fixture = () => {
|
||||
const [open, setOpen] = createSignal(true);
|
||||
|
||||
@@ -229,6 +229,9 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
|
||||
</div>
|
||||
|
||||
<p class="mt-2 text-xs leading-relaxed text-base-content">{alert.description}</p>
|
||||
<Show when={alert.closeDetail}>
|
||||
{(detail) => <p class="mt-1 text-xs leading-relaxed text-muted">{detail()}</p>}
|
||||
</Show>
|
||||
|
||||
<dl class="mt-3 grid grid-cols-3 gap-2 text-[10px] text-muted">
|
||||
<div>
|
||||
|
||||
@@ -74,7 +74,7 @@ export function AlertHistoryTableAlertRow(props: AlertHistoryTableAlertRowProps)
|
||||
</TableCell>
|
||||
|
||||
<TableCell class={getPlatformTableCellClassForKind('badge')}>
|
||||
<span class={historyStatusPresentation().className}>
|
||||
<span class={historyStatusPresentation().className} title={props.alert.closeDetail}>
|
||||
{historyStatusPresentation().label}
|
||||
</span>
|
||||
</TableCell>
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import type { Alert } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import {
|
||||
ALERT_HISTORY_MOVED_STATUS,
|
||||
getAlertResolutionDetail,
|
||||
} from '@/utils/alertIncidentPresentation';
|
||||
import { isPulseSystemAlert } from '@/utils/alertScope';
|
||||
|
||||
import { alertTypeDisplayLabel, unifiedTypeToAlertDisplayType } from './helpers';
|
||||
@@ -25,6 +29,8 @@ export interface HistoryItem {
|
||||
title: string;
|
||||
rawAlertType?: string;
|
||||
description?: string;
|
||||
// Set when the alert closed without recovering: what happened instead.
|
||||
closeDetail?: string;
|
||||
acknowledged?: boolean;
|
||||
systemAlert?: boolean;
|
||||
}
|
||||
@@ -206,11 +212,17 @@ export function buildAlertHistoryItems({
|
||||
alertHistory.forEach((alert) => {
|
||||
if (activeAlertIds.has(alert.id)) return;
|
||||
const systemScoped = isPulseSystemAlert(alert);
|
||||
// A close that was not a recovery must not read as "resolved".
|
||||
const closeDetail = getAlertResolutionDetail(alert.resolution);
|
||||
|
||||
items.push({
|
||||
id: alert.id,
|
||||
source: 'alert',
|
||||
status: alert.acknowledged ? 'acknowledged' : 'resolved',
|
||||
status: closeDetail
|
||||
? ALERT_HISTORY_MOVED_STATUS
|
||||
: alert.acknowledged
|
||||
? 'acknowledged'
|
||||
: 'resolved',
|
||||
startTime: alert.startTime,
|
||||
endTime: alert.lastSeen,
|
||||
duration: formatAlertHistoryDuration(alert.startTime, alert.lastSeen, now),
|
||||
@@ -231,6 +243,7 @@ export function buildAlertHistoryItems({
|
||||
title: alertTypeDisplayLabel(alert.type),
|
||||
rawAlertType: alert.type,
|
||||
description: alert.message,
|
||||
...(closeDetail ? { closeDetail } : {}),
|
||||
acknowledged: alert.acknowledged,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -30,6 +30,7 @@ import {
|
||||
getPlatformResourceDetailRowClass,
|
||||
} from '@/features/platformPage/PlatformResourceDetailTableRow';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
import {
|
||||
compareKubernetesControllers,
|
||||
filterKubernetesResources,
|
||||
@@ -137,9 +138,10 @@ const plainDetail = (text: string): ControllerDetail => ({ text, title: text });
|
||||
|
||||
// Kubernetes reports job timestamps as RFC 3339 strings with microseconds,
|
||||
// which never fit the Detail column. The cell says how long ago the event
|
||||
// happened and keeps the absolute time on hover.
|
||||
const timeDetail = (label: string, timestamp: string): ControllerDetail => {
|
||||
const relative = formatPlatformTableRelativeTimeValue(timestamp, { emptyText: '' });
|
||||
// happened, measured from the shared clock so it keeps moving while the row
|
||||
// stays mounted, and keeps the absolute time on hover.
|
||||
const timeDetail = (label: string, timestamp: string, now: number): ControllerDetail => {
|
||||
const relative = formatPlatformTableRelativeTimeValue(timestamp, { emptyText: '', now });
|
||||
if (!relative) return plainDetail(`${label}: ${timestamp}`);
|
||||
const absolute = formatPlatformTableDateTimeValue(timestamp, {
|
||||
dateTimeFormat: { year: 'numeric' },
|
||||
@@ -147,7 +149,7 @@ const timeDetail = (label: string, timestamp: string): ControllerDetail => {
|
||||
return { text: `${label} ${relative}`, title: `${label}: ${absolute}` };
|
||||
};
|
||||
|
||||
const apiDetail = (resource: Resource): ControllerDetail => {
|
||||
const apiDetail = (resource: Resource, now: number): ControllerDetail => {
|
||||
switch (resource.type) {
|
||||
case 'k8s-replicaset': {
|
||||
if (typeof resource.kubernetes?.fullyLabeledReplicas === 'number') {
|
||||
@@ -171,17 +173,17 @@ const apiDetail = (resource: Resource): ControllerDetail => {
|
||||
);
|
||||
case 'k8s-job':
|
||||
if (resource.kubernetes?.completionTime) {
|
||||
return timeDetail('Completed', resource.kubernetes.completionTime);
|
||||
return timeDetail('Completed', resource.kubernetes.completionTime, now);
|
||||
}
|
||||
return resource.kubernetes?.startTime
|
||||
? timeDetail('Started', resource.kubernetes.startTime)
|
||||
? timeDetail('Started', resource.kubernetes.startTime, now)
|
||||
: plainDetail('—');
|
||||
case 'k8s-cronjob':
|
||||
if (resource.kubernetes?.lastSuccessfulTime) {
|
||||
return timeDetail('Last success', resource.kubernetes.lastSuccessfulTime);
|
||||
return timeDetail('Last success', resource.kubernetes.lastSuccessfulTime, now);
|
||||
}
|
||||
return resource.kubernetes?.lastScheduleTime
|
||||
? timeDetail('Last run', resource.kubernetes.lastScheduleTime)
|
||||
? timeDetail('Last run', resource.kubernetes.lastScheduleTime, now)
|
||||
: plainDetail('—');
|
||||
default:
|
||||
return plainDetail('—');
|
||||
@@ -245,6 +247,7 @@ export const KubernetesControllersTable: Component<{
|
||||
externalSearch?: () => string;
|
||||
externalStatus?: () => KubernetesResourceStatusFilter;
|
||||
}> = (props) => {
|
||||
const now = useRelativeTimeNow();
|
||||
const singleCluster = createMemo(() => isSingleKubernetesCluster(props.resources));
|
||||
const tableState = createPlatformTableFilterState({
|
||||
resources: () => props.resources,
|
||||
@@ -402,7 +405,7 @@ export const KubernetesControllersTable: Component<{
|
||||
const kind = () => controllerKind(resource);
|
||||
const target = () => targetValue(resource);
|
||||
const exceptions = () => exceptionSummary(resource);
|
||||
const detail = () => apiDetail(resource);
|
||||
const detail = () => apiDetail(resource, now());
|
||||
const detailRowId = () => drawer.detailRowId(resource);
|
||||
const isExpanded = () => drawer.isExpanded(resource);
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ import { StackedMemoryBar } from '@/components/Workloads/StackedMemoryBar';
|
||||
import { TableCell, TableRow } from '@/components/shared/Table';
|
||||
import { ResourceNameWithWebInterfaceLink } from '@/components/shared/WebInterfaceLink';
|
||||
import { asTrimmedString } from '@/utils/stringUtils';
|
||||
import { getAlertStyles } from '@/utils/alerts';
|
||||
import { getUnifiedResourceAlertStyles } from '@/utils/alerts';
|
||||
import { useWebSocket } from '@/contexts/appRuntime';
|
||||
import { useAlertsActivation } from '@/stores/alertsActivation';
|
||||
import { unifiedPlatformOverrideIdCandidates } from '@/features/alerts/alertOverridesModel';
|
||||
@@ -41,6 +41,7 @@ import type { Resource } from '@/types/resource';
|
||||
import {
|
||||
compareKubernetesNodes,
|
||||
filterKubernetesResources,
|
||||
getKubernetesNodeAttentionRank,
|
||||
kubernetesClusterLabel,
|
||||
mapKubernetesNodeStatus,
|
||||
type KubernetesResourceStatusFilter,
|
||||
@@ -145,19 +146,27 @@ export const KubernetesNodesTable: Component<{
|
||||
const drawer = createPlatformResourceDetailState({ idPrefix: 'kubernetes-node-drawer' });
|
||||
const resolveResourceLabel = createPlatformResourceLabelResolver(() => props.resources);
|
||||
// User-controlled sorting layered over the attention-first default: rows
|
||||
// are pre-sorted by the status compare, so a user sort keeps that order
|
||||
// for ties and the table falls straight back to it when the sort clears.
|
||||
// are pre-sorted by attention (status and open alerts), then the status
|
||||
// compare, so a user sort keeps that order for ties and the table falls
|
||||
// straight back to it when the sort clears.
|
||||
const sort = createPlatformTableSortState({
|
||||
storageKey: props.sortStorageKey ?? 'kubernetesNodes',
|
||||
sortKeys: KUBERNETES_NODE_SORT_KEYS,
|
||||
descendingFirst: ['cpu', 'memory', 'uptime', 'capacity'],
|
||||
});
|
||||
const sortedRows = createMemo(() =>
|
||||
sort.sortRows(
|
||||
[...tableState.filtered()].sort(compareKubernetesNodes),
|
||||
const sortedRows = createMemo(() => {
|
||||
const rows = tableState.filtered();
|
||||
const enabled = alertsEnabled();
|
||||
const ranks = new Map(
|
||||
rows.map((node) => [node.id, getKubernetesNodeAttentionRank(node, activeAlerts, enabled)]),
|
||||
);
|
||||
return sort.sortRows(
|
||||
[...rows].sort(
|
||||
(a, b) => (ranks.get(a.id) ?? 2) - (ranks.get(b.id) ?? 2) || compareKubernetesNodes(a, b),
|
||||
),
|
||||
getKubernetesNodeSortValue,
|
||||
),
|
||||
);
|
||||
);
|
||||
});
|
||||
|
||||
return (
|
||||
<Show
|
||||
@@ -367,8 +376,12 @@ export const KubernetesNodesTable: Component<{
|
||||
const canRenderMetrics = () => indicator().variant !== 'muted';
|
||||
const detailRowId = () => drawer.detailRowId(node);
|
||||
const isExpanded = () => drawer.isExpanded(node);
|
||||
// The same open alerts the node's drawer lists: a node that
|
||||
// runs a Pulse agent has its alerts keyed on the agent and
|
||||
// labelled with its hostname, so matching the row's id or
|
||||
// display name finds none of them.
|
||||
const nodeAlertStyles = createMemo(() =>
|
||||
getAlertStyles(node.id, activeAlerts, alertsEnabled(), name()),
|
||||
getUnifiedResourceAlertStyles(node, activeAlerts, alertsEnabled()),
|
||||
);
|
||||
const nodeAlertBg = () => {
|
||||
const s = nodeAlertStyles();
|
||||
@@ -380,7 +393,9 @@ export const KubernetesNodesTable: Component<{
|
||||
return (
|
||||
<>
|
||||
<TableRow
|
||||
class={`${getPlatformResourceDetailRowClass(isExpanded())} text-[11px] sm:text-xs ${nodeAlertBg()}`}
|
||||
class={`${getPlatformResourceDetailRowClass(isExpanded())} text-[11px] sm:text-xs ${
|
||||
isExpanded() ? '' : nodeAlertBg()
|
||||
}`}
|
||||
data-kubernetes-node-row={node.id}
|
||||
onClick={() => drawer.toggle(node)}
|
||||
>
|
||||
|
||||
@@ -9,8 +9,10 @@ import {
|
||||
collectOutdatedAgentHosts,
|
||||
formatAgentVersionDisplay,
|
||||
} from '@/features/platformPage/agentVersion';
|
||||
import { useWebSocket } from '@/contexts/appRuntime';
|
||||
import { useUnifiedResources, type UnifiedResourceFacets } from '@/hooks/useUnifiedResources';
|
||||
import { buildKubernetesPath, KUBERNETES_QUERY_PARAMS } from '@/routing/resourceLinks';
|
||||
import { useAlertsActivation } from '@/stores/alertsActivation';
|
||||
import { updateStore } from '@/stores/updates';
|
||||
import {
|
||||
PLATFORM_HEALTH_FILTER_OPTIONS,
|
||||
@@ -710,14 +712,22 @@ function KubernetesOverview(props: KubernetesOverviewProps) {
|
||||
return posture.podAttention + posture.deploymentAttention;
|
||||
});
|
||||
|
||||
const nodesNeedingAttention = createMemo(() => getKubernetesNodesNeedingAttention(props.model()));
|
||||
const { activeAlerts } = useWebSocket();
|
||||
const alertsActivation = useAlertsActivation();
|
||||
const nodesNeedingAttention = createMemo(() =>
|
||||
getKubernetesNodesNeedingAttention(
|
||||
props.model(),
|
||||
activeAlerts,
|
||||
alertsActivation.detectionEnabled(),
|
||||
),
|
||||
);
|
||||
|
||||
return (
|
||||
<div class="space-y-4">
|
||||
{/*
|
||||
Problems lead, inventory follows: active health signals, then the
|
||||
nodes that are not Ready, before the cluster summary and workloads.
|
||||
Filtering stays on the Nodes and Events tabs.
|
||||
nodes that are not Ready or have an open alert, before the cluster
|
||||
summary and workloads. Filtering stays on the Nodes and Events tabs.
|
||||
*/}
|
||||
<Show when={props.model().incidents.length > 0}>
|
||||
<div id="kubernetes-health-signals">
|
||||
@@ -738,7 +748,7 @@ function KubernetesOverview(props: KubernetesOverviewProps) {
|
||||
sortStorageKey="kubernetesNodesNeedingAttention"
|
||||
emptyIcon={k8sIcon()}
|
||||
emptyTitle="All nodes are Ready"
|
||||
emptyDescription="Nodes appear here when they are not Ready, cordoned or degraded."
|
||||
emptyDescription="Nodes appear here when they are not Ready, cordoned, degraded or alerting."
|
||||
showToolbar={false}
|
||||
/>
|
||||
</div>
|
||||
|
||||
+94
@@ -0,0 +1,94 @@
|
||||
import { cleanup, render, screen } from '@solidjs/testing-library';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
vi.mock('@/contexts/appRuntime', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import('@/contexts/appRuntime')>()),
|
||||
useWebSocket: () => ({ activeAlerts: {} }),
|
||||
}));
|
||||
|
||||
vi.mock('@/api/resources', () => ({
|
||||
ResourceAPI: {
|
||||
getFacetBundle: vi.fn().mockResolvedValue({
|
||||
capabilities: [],
|
||||
relationships: [],
|
||||
recentChanges: [],
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { KubernetesControllersTable } from '../KubernetesControllersTable';
|
||||
|
||||
const makeResource = ({
|
||||
id,
|
||||
type,
|
||||
...overrides
|
||||
}: Partial<Resource> & Pick<Resource, 'id' | 'type'>): Resource => ({
|
||||
id,
|
||||
name: id,
|
||||
displayName: id,
|
||||
platformId: 'cluster-1',
|
||||
platformType: 'kubernetes',
|
||||
sourceType: 'agent',
|
||||
sources: ['kubernetes'],
|
||||
status: 'online',
|
||||
type,
|
||||
lastSeen: 1_700_000_000_000,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe('KubernetesControllersTable relative ages', () => {
|
||||
it('keeps Job and CronJob detail ages moving while the rows stay mounted', () => {
|
||||
vi.useFakeTimers({ now: new Date('2026-05-24T13:31:00Z') });
|
||||
|
||||
render(() => (
|
||||
<KubernetesControllersTable
|
||||
resources={[
|
||||
makeResource({
|
||||
id: 'nightly-import',
|
||||
type: 'k8s-job',
|
||||
kubernetes: {
|
||||
clusterName: 'prod',
|
||||
namespace: 'batch',
|
||||
resourceKind: 'Job',
|
||||
desiredReplicas: 1,
|
||||
succeeded: 1,
|
||||
completionTime: '2026-05-24T13:00:00Z',
|
||||
},
|
||||
}),
|
||||
makeResource({
|
||||
id: 'billing-rollup',
|
||||
type: 'k8s-cronjob',
|
||||
kubernetes: {
|
||||
clusterName: 'prod',
|
||||
namespace: 'batch',
|
||||
resourceKind: 'CronJob',
|
||||
schedule: '0 * * * *',
|
||||
lastSuccessfulTime: '2026-05-24T12:55:00Z',
|
||||
},
|
||||
}),
|
||||
]}
|
||||
emptyIcon={<span />}
|
||||
emptyTitle="No controllers"
|
||||
emptyDescription="No controllers"
|
||||
showToolbar={false}
|
||||
/>
|
||||
));
|
||||
|
||||
expect(screen.getByText('Completed 31m ago')).toBeInTheDocument();
|
||||
expect(screen.getByText('Last success 36m ago')).toBeInTheDocument();
|
||||
|
||||
// No data changes: the timestamps stay put and only the clock moves. A
|
||||
// detail computed once at render would still read "31m ago" here.
|
||||
vi.advanceTimersByTime(3 * 60 * 60 * 1000);
|
||||
|
||||
expect(screen.getByText('Completed 3h ago')).toBeInTheDocument();
|
||||
expect(screen.getByText('Last success 3h ago')).toBeInTheDocument();
|
||||
expect(screen.queryByText('Completed 31m ago')).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,8 @@
|
||||
import { cleanup, fireEvent, render, screen } from '@solidjs/testing-library';
|
||||
import { createStore, reconcile } from 'solid-js/store';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import type { Alert } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { KubernetesNodesTable } from '../KubernetesNodesTable';
|
||||
|
||||
@@ -12,8 +14,11 @@ vi.mock('@/components/Workloads/StackedMemoryBar', () => ({
|
||||
StackedMemoryBar: () => <div data-testid="stacked-memory-bar" />,
|
||||
}));
|
||||
|
||||
// The websocket's activeAlerts is a Solid store, so rows must follow it live.
|
||||
const activeAlertsRef = vi.hoisted(() => ({ current: {} as Record<string, Alert> }));
|
||||
|
||||
vi.mock('@/contexts/appRuntime', () => ({
|
||||
useWebSocket: () => ({ activeAlerts: [] }),
|
||||
useWebSocket: () => ({ activeAlerts: activeAlertsRef.current }),
|
||||
}));
|
||||
|
||||
vi.mock('@/stores/alertsActivation', () => ({
|
||||
@@ -51,8 +56,12 @@ const makeNodeResource = (overrides: Partial<Resource> = {}): Resource => ({
|
||||
...overrides,
|
||||
});
|
||||
|
||||
const [activeAlerts, setActiveAlerts] = createStore<Record<string, Alert>>({});
|
||||
activeAlertsRef.current = activeAlerts;
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
setActiveAlerts(reconcile({}));
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
@@ -146,4 +155,79 @@ describe('KubernetesNodesTable', () => {
|
||||
'false',
|
||||
);
|
||||
});
|
||||
it('tints a node row for the open alerts its drawer lists and leads with it', () => {
|
||||
const agentAlert = (id: string, resourceId: string, overrides: Partial<Alert> = {}): Alert => ({
|
||||
id,
|
||||
type: 'memory',
|
||||
level: 'critical',
|
||||
resourceId,
|
||||
resourceName: 'Prod Euw1 K8s 02',
|
||||
node: 'prod-euw1-k8s-02',
|
||||
instance: '',
|
||||
message: 'Agent memory at 100.0%',
|
||||
value: 100,
|
||||
threshold: 90,
|
||||
startTime: '2026-10-06T10:00:00Z',
|
||||
acknowledged: false,
|
||||
...overrides,
|
||||
});
|
||||
// A node that runs a Pulse agent is an agent row; its alerts are keyed
|
||||
// "agent:<agentId>" and carry the hostname, never the row id or name.
|
||||
const agentNode = (id: string, name: string, agentId: string, ready = true) =>
|
||||
makeNodeResource({
|
||||
id,
|
||||
name,
|
||||
displayName: name,
|
||||
type: 'agent',
|
||||
agent: { agentId },
|
||||
kubernetes: { ...makeNodeResource().kubernetes, nodeName: id, ready },
|
||||
} as Partial<Resource>);
|
||||
setActiveAlerts({
|
||||
memory: agentAlert('memory', 'agent:host-k8s-node-2'),
|
||||
});
|
||||
|
||||
const { container } = render(() => (
|
||||
<KubernetesNodesTable
|
||||
resources={[
|
||||
agentNode('node-1', 'Prod Euw1 K8s 01', 'host-k8s-node-1'),
|
||||
agentNode('node-2', 'Prod Euw1 K8s 02', 'host-k8s-node-2'),
|
||||
makeNodeResource({
|
||||
id: 'node-3',
|
||||
name: 'Prod Euw1 K8s 03',
|
||||
kubernetes: { ...makeNodeResource().kubernetes, unschedulable: true },
|
||||
}),
|
||||
]}
|
||||
emptyIcon={<span />}
|
||||
emptyTitle="No nodes"
|
||||
emptyDescription="No nodes"
|
||||
showToolbar={false}
|
||||
/>
|
||||
));
|
||||
|
||||
const row = (id: string) =>
|
||||
container.querySelector(`[data-kubernetes-node-row="${id}"]`) as HTMLElement;
|
||||
const order = () =>
|
||||
[...container.querySelectorAll('[data-kubernetes-node-row]')].map((element) =>
|
||||
element.getAttribute('data-kubernetes-node-row'),
|
||||
);
|
||||
expect(row('node-2')).toHaveClass('bg-red-50');
|
||||
expect(row('node-1')).not.toHaveClass('bg-red-50');
|
||||
// A Ready node with a critical alert leads, ahead of a cordoned node.
|
||||
expect(order()).toEqual(['node-2', 'node-3', 'node-1']);
|
||||
|
||||
// The tint and the order follow the live alert store: acknowledging
|
||||
// clears both, and a new warning tints the row amber.
|
||||
setActiveAlerts('memory', 'acknowledged', true);
|
||||
expect(row('node-2')).not.toHaveClass('bg-red-50');
|
||||
expect(order()).toEqual(['node-3', 'node-1', 'node-2']);
|
||||
setActiveAlerts('memory', { acknowledged: false, level: 'warning' });
|
||||
expect(row('node-2')).toHaveClass('bg-yellow-50');
|
||||
// An expanded row drops the tint while its drawer lists the alert.
|
||||
fireEvent.click(row('node-2'));
|
||||
expect(row('node-2')).not.toHaveClass('bg-yellow-50');
|
||||
fireEvent.click(row('node-2'));
|
||||
expect(row('node-2')).toHaveClass('bg-yellow-50');
|
||||
setActiveAlerts(reconcile({}));
|
||||
expect(row('node-2')).not.toHaveClass('bg-yellow-50');
|
||||
});
|
||||
});
|
||||
|
||||
+68
@@ -1,12 +1,15 @@
|
||||
import { Route, Router } from '@solidjs/router';
|
||||
import { cleanup, render, screen } from '@solidjs/testing-library';
|
||||
import { createStore, reconcile } from 'solid-js/store';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { Alert } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { KubernetesPageSurface } from '../KubernetesPageSurface';
|
||||
|
||||
const mockUseUnifiedResources = vi.fn();
|
||||
const mockPathname = vi.hoisted(() => vi.fn(() => '/'));
|
||||
const mockVersionInfo = vi.hoisted(() => vi.fn());
|
||||
const activeAlertsRef = vi.hoisted(() => ({ current: {} as Record<string, Alert> }));
|
||||
|
||||
const makeResource = (resource: Partial<Resource> & Pick<Resource, 'id' | 'type'>): Resource => ({
|
||||
name: resource.id,
|
||||
@@ -40,6 +43,14 @@ vi.mock('@/hooks/useUnifiedResources', () => ({
|
||||
useUnifiedResources: (...args: unknown[]) => mockUseUnifiedResources(...args),
|
||||
}));
|
||||
|
||||
vi.mock('@/contexts/appRuntime', () => ({
|
||||
useWebSocket: () => ({ activeAlerts: activeAlertsRef.current }),
|
||||
}));
|
||||
|
||||
vi.mock('@/stores/alertsActivation', () => ({
|
||||
useAlertsActivation: () => ({ detectionEnabled: () => true }),
|
||||
}));
|
||||
|
||||
vi.mock('@/stores/updates', () => ({
|
||||
updateStore: {
|
||||
versionInfo: mockVersionInfo,
|
||||
@@ -129,6 +140,7 @@ vi.mock('../KubernetesNodesTable', () => ({
|
||||
<div
|
||||
data-testid="nodes-table"
|
||||
data-rows={props.resources.length}
|
||||
data-ids={props.resources.map((resource) => resource.id).join(',')}
|
||||
data-title={props.title}
|
||||
data-sort-key={props.sortStorageKey}
|
||||
/>
|
||||
@@ -159,6 +171,9 @@ vi.mock('../KubernetesStorageTable', () => ({
|
||||
),
|
||||
}));
|
||||
|
||||
const [activeAlerts, setActiveAlerts] = createStore<Record<string, Alert>>({});
|
||||
activeAlertsRef.current = activeAlerts;
|
||||
|
||||
describe('KubernetesPageSurface contract', () => {
|
||||
beforeEach(() => {
|
||||
mockPathname.mockReturnValue('/');
|
||||
@@ -167,6 +182,7 @@ describe('KubernetesPageSurface contract', () => {
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
setActiveAlerts(reconcile({}));
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
@@ -207,6 +223,58 @@ describe('KubernetesPageSurface contract', () => {
|
||||
).toBeTruthy();
|
||||
});
|
||||
|
||||
it('names a Ready node with an open alert among the nodes needing attention', () => {
|
||||
setResources([
|
||||
makeResource({ id: 'cluster-1', type: 'k8s-cluster' }),
|
||||
makeResource({
|
||||
id: 'node-memory',
|
||||
type: 'agent',
|
||||
sources: ['agent', 'kubernetes'],
|
||||
agent: { agentId: 'host-k8s-02' },
|
||||
kubernetes: { ready: true } as Resource['kubernetes'],
|
||||
} as Partial<Resource> & Pick<Resource, 'id' | 'type'>),
|
||||
makeResource({
|
||||
id: 'node-ok',
|
||||
type: 'k8s-node',
|
||||
kubernetes: { ready: true } as Resource['kubernetes'],
|
||||
}),
|
||||
]);
|
||||
renderSurface();
|
||||
// Every node is Ready, so nothing needs attention until an alert opens.
|
||||
expect(screen.queryByTestId('nodes-table')).toBeNull();
|
||||
|
||||
// The node's agent raises it: keyed on the agent, labelled with the hostname.
|
||||
setActiveAlerts({
|
||||
memory: {
|
||||
id: 'memory',
|
||||
type: 'memory',
|
||||
level: 'critical',
|
||||
resourceId: 'agent:host-k8s-02',
|
||||
resourceName: 'Prod Euw1 K8s 02',
|
||||
node: 'prod-euw1-k8s-02',
|
||||
instance: '',
|
||||
message: 'Agent memory at 100.0%',
|
||||
value: 100,
|
||||
threshold: 90,
|
||||
startTime: '2026-10-06T10:00:00Z',
|
||||
acknowledged: false,
|
||||
},
|
||||
});
|
||||
|
||||
const attention = screen.getByTestId('nodes-table');
|
||||
expect(attention).toHaveAttribute('data-title', 'Nodes needing attention');
|
||||
expect(attention).toHaveAttribute('data-ids', 'node-memory');
|
||||
|
||||
// Acknowledged: someone owns it, and the row is no longer tinted either.
|
||||
setActiveAlerts('memory', 'acknowledged', true);
|
||||
expect(screen.queryByTestId('nodes-table')).toBeNull();
|
||||
setActiveAlerts('memory', 'acknowledged', false);
|
||||
expect(screen.getByTestId('nodes-table')).toHaveAttribute('data-ids', 'node-memory');
|
||||
// Resolved: the alert leaves the store and the node leaves the list.
|
||||
setActiveAlerts(reconcile({}));
|
||||
expect(screen.queryByTestId('nodes-table')).toBeNull();
|
||||
});
|
||||
|
||||
it('shows no attention table when every node is Ready', () => {
|
||||
setResources([
|
||||
makeResource({ id: 'cluster-1', type: 'k8s-cluster' }),
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import type { Alert } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import {
|
||||
getKubernetesNodeAttentionRank,
|
||||
getKubernetesNodesNeedingAttention,
|
||||
splitKubernetesNameTail,
|
||||
isSingleKubernetesCluster,
|
||||
@@ -1178,4 +1180,91 @@ describe('getKubernetesNodesNeedingAttention', () => {
|
||||
|
||||
expect(result.map((entry: { id: string }) => entry.id)).toEqual(['a-down', 'c-cordoned']);
|
||||
});
|
||||
|
||||
// A node running a Pulse agent is an agent row whose alerts are keyed
|
||||
// "agent:<agentId>" and labelled with its hostname.
|
||||
const agentNode = (id: string, agentId: string, kubernetes: Record<string, unknown>) =>
|
||||
({
|
||||
id,
|
||||
name: id,
|
||||
type: 'agent',
|
||||
status: 'online',
|
||||
agent: { agentId },
|
||||
kubernetes,
|
||||
}) as never;
|
||||
const alert = (id: string, resourceId: string, overrides: Partial<Alert> = {}): Alert => ({
|
||||
id,
|
||||
type: 'memory',
|
||||
level: 'critical',
|
||||
resourceId,
|
||||
resourceName: id,
|
||||
node: id,
|
||||
instance: '',
|
||||
message: id,
|
||||
value: 100,
|
||||
threshold: 90,
|
||||
startTime: '2026-10-06T10:00:00Z',
|
||||
acknowledged: false,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
it('names Ready nodes with an unacknowledged open alert, critical ones with NotReady', () => {
|
||||
const nodes = [
|
||||
agentNode('a-ready', 'host-a', { ready: true }),
|
||||
agentNode('b-memory', 'host-b', { ready: true }),
|
||||
agentNode('c-cordoned', 'host-c', { ready: true, unschedulable: true }),
|
||||
agentNode('d-down', 'host-d', { ready: false }),
|
||||
agentNode('e-disk', 'host-e', { ready: true }),
|
||||
agentNode('f-acked', 'host-f', { ready: true }),
|
||||
];
|
||||
const activeAlerts = {
|
||||
memory: alert('memory', 'agent:host-b'),
|
||||
// A component alert the drawer lists counts for its node too.
|
||||
disk: alert('disk', 'agent:host-e/disk:root', { type: 'disk', level: 'warning' }),
|
||||
acked: alert('acked', 'agent:host-f', { acknowledged: true }),
|
||||
};
|
||||
|
||||
const result = getKubernetesNodesNeedingAttention({ nodes } as never, activeAlerts, true);
|
||||
|
||||
// Within a rank, node state orders: NotReady ahead of a Ready node.
|
||||
expect(result.map((entry: { id: string }) => entry.id)).toEqual([
|
||||
'd-down',
|
||||
'b-memory',
|
||||
'c-cordoned',
|
||||
'e-disk',
|
||||
]);
|
||||
// With alert detection off the list falls back to node state alone.
|
||||
expect(
|
||||
getKubernetesNodesNeedingAttention({ nodes } as never, activeAlerts, false).map(
|
||||
(entry: { id: string }) => entry.id,
|
||||
),
|
||||
).toEqual(['d-down', 'c-cordoned']);
|
||||
});
|
||||
|
||||
it('counts alerts keyed on a node without an agent by its own id', () => {
|
||||
// Unified Kubernetes metric and incident alerts carry the resource id and
|
||||
// the cluster as their node.
|
||||
const nodes = [node('k8s-node-a', { ready: true }), node('k8s-node-b', { ready: true })];
|
||||
const activeAlerts = {
|
||||
cpu: alert('cpu', 'k8s-node-b', { type: 'cpu', level: 'warning', node: 'prod-west' }),
|
||||
};
|
||||
|
||||
expect(
|
||||
getKubernetesNodesNeedingAttention({ nodes } as never, activeAlerts, true).map(
|
||||
(entry: { id: string }) => entry.id,
|
||||
),
|
||||
).toEqual(['k8s-node-b']);
|
||||
});
|
||||
|
||||
it('ranks a node by the stronger of its state and its open alerts', () => {
|
||||
const ready = agentNode('ready', 'host-ready', { ready: true });
|
||||
const cordoned = agentNode('cordoned', 'host-cordoned', { ready: true, unschedulable: true });
|
||||
const warning = { w: alert('w', 'agent:host-ready', { level: 'warning' }) };
|
||||
const critical = { c: alert('c', 'agent:host-cordoned') };
|
||||
|
||||
expect(getKubernetesNodeAttentionRank(ready, {}, true)).toBe(2);
|
||||
expect(getKubernetesNodeAttentionRank(ready, warning, true)).toBe(1);
|
||||
expect(getKubernetesNodeAttentionRank(cordoned, {}, true)).toBe(1);
|
||||
expect(getKubernetesNodeAttentionRank(cordoned, critical, true)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -4,7 +4,9 @@ import type {
|
||||
ResourceKubernetesPodContainerStatus,
|
||||
ResourceType,
|
||||
} from '@/types/resource';
|
||||
import type { Alert } from '@/types/api';
|
||||
import type { StatusIndicator, StatusIndicatorVariant } from '@/utils/status';
|
||||
import { getUnifiedResourceAlertStyles } from '@/utils/alerts';
|
||||
import { resolveResourcePlatformType } from '@/utils/sourcePlatforms';
|
||||
import { matchesSearchTermSplit, splitSearchExclusions } from '@/utils/searchQuery';
|
||||
|
||||
@@ -801,17 +803,44 @@ const countKubernetesDanger = (
|
||||
mapper: (resource: Resource) => StatusIndicator,
|
||||
): number => resources.filter((resource) => mapper(resource).variant === 'danger').length;
|
||||
|
||||
// Nodes the overview should name before any inventory: NotReady first, then
|
||||
// cordoned or degraded ones. "2/3 nodes" on the cluster row says something is
|
||||
// wrong but not which node; this list answers that.
|
||||
export function getKubernetesNodesNeedingAttention(model: KubernetesPageModel): Resource[] {
|
||||
const rank = (node: Resource) => {
|
||||
const variant = mapKubernetesNodeStatus(node).variant;
|
||||
return variant === 'danger' ? 0 : variant === 'warning' ? 1 : 2;
|
||||
};
|
||||
// Attention order shared by the overview's "Nodes needing attention" list and
|
||||
// the nodes table's default sort: 0 for a NotReady node or one with an
|
||||
// unacknowledged critical alert, 1 for a cordoned or degraded node or one with
|
||||
// another unacknowledged alert, 2 otherwise. The alerts are the ones the
|
||||
// node's row tint and drawer use, so a Ready node at 100% memory ranks with
|
||||
// the NotReady ones instead of below every degraded node.
|
||||
export function getKubernetesNodeAttentionRank(
|
||||
node: Resource,
|
||||
activeAlerts: Record<string, Alert>,
|
||||
alertsEnabled: boolean,
|
||||
): 0 | 1 | 2 {
|
||||
const variant = mapKubernetesNodeStatus(node).variant;
|
||||
const statusRank = variant === 'danger' ? 0 : variant === 'warning' ? 1 : 2;
|
||||
const alertStyles = getUnifiedResourceAlertStyles(node, activeAlerts, alertsEnabled);
|
||||
const alertRank = !alertStyles.hasUnacknowledgedAlert
|
||||
? 2
|
||||
: alertStyles.severity === 'critical'
|
||||
? 0
|
||||
: 1;
|
||||
return Math.min(statusRank, alertRank) as 0 | 1 | 2;
|
||||
}
|
||||
|
||||
// Nodes the overview should name before any inventory, in attention order.
|
||||
// "2/3 nodes" on the cluster row says something is wrong but not which node;
|
||||
// this list answers that.
|
||||
export function getKubernetesNodesNeedingAttention(
|
||||
model: KubernetesPageModel,
|
||||
activeAlerts: Record<string, Alert> = {},
|
||||
alertsEnabled = false,
|
||||
): Resource[] {
|
||||
return model.nodes
|
||||
.filter((node) => rank(node) < 2)
|
||||
.sort((a, b) => rank(a) - rank(b) || a.name.localeCompare(b.name));
|
||||
.map((node) => ({
|
||||
node,
|
||||
rank: getKubernetesNodeAttentionRank(node, activeAlerts, alertsEnabled),
|
||||
}))
|
||||
.filter((entry) => entry.rank < 2)
|
||||
.sort((a, b) => a.rank - b.rank || compareKubernetesNodes(a.node, b.node))
|
||||
.map((entry) => entry.node);
|
||||
}
|
||||
|
||||
export function buildKubernetesOverviewPosture(
|
||||
|
||||
@@ -68,6 +68,7 @@ import { copyToClipboard } from '@/utils/clipboard';
|
||||
import type { PatrolAutonomyLevel } from '@/api/patrol';
|
||||
import {
|
||||
getAttentionFlappingPresentation,
|
||||
getAttentionHandoverClose,
|
||||
getLinkedPatrolFindings,
|
||||
getPatrolRememberedDecisionPresentation,
|
||||
partitionPatrolAttention,
|
||||
@@ -971,7 +972,10 @@ function AttentionDetail(props: {
|
||||
<div class="space-y-5 px-4 py-4 sm:px-5">
|
||||
<section>
|
||||
<div class="flex flex-wrap items-center gap-x-2 gap-y-1 text-xs text-muted">
|
||||
<StateBadge item={loaded().item} />
|
||||
<StateBadge
|
||||
item={loaded().item}
|
||||
movedClose={Boolean(getAttentionHandoverClose(loaded()))}
|
||||
/>
|
||||
<span aria-hidden="true">·</span>
|
||||
<EvidenceLabel item={loaded().item} detail />
|
||||
<span aria-hidden="true">·</span>
|
||||
@@ -984,6 +988,9 @@ function AttentionDetail(props: {
|
||||
<p class="mt-3 text-sm leading-6 text-base-content">
|
||||
{loaded().item.plainLanguageSummary}
|
||||
</p>
|
||||
<Show when={getAttentionHandoverClose(loaded())}>
|
||||
{(handover) => <p class="mt-1 text-xs leading-5 text-muted">{handover()}</p>}
|
||||
</Show>
|
||||
<Show when={getAttentionFlappingPresentation(loaded().item.flapping)}>
|
||||
{(flapping) => (
|
||||
<p class="mt-2 flex items-start gap-2 rounded-md border border-amber-200 bg-amber-50/70 px-3 py-2 text-xs leading-5 text-amber-900 dark:border-amber-900/25 dark:bg-amber-950/30 dark:text-amber-200">
|
||||
@@ -1793,10 +1800,17 @@ function severityTone(item: AttentionItem): MetadataBadgeTone {
|
||||
}
|
||||
}
|
||||
|
||||
function StateBadge(props: { item: AttentionItem }) {
|
||||
// movedClose marks a resolved item whose condition moved to another owner: it
|
||||
// did not recover, so it must not wear the green "Resolved" badge.
|
||||
function StateBadge(props: { item: AttentionItem; movedClose?: boolean }) {
|
||||
return (
|
||||
<MetadataBadge tone={stateTone(props.item)} size="xs" shape="rounded" appearance="outline">
|
||||
{formatLabel(props.item.state)}
|
||||
<MetadataBadge
|
||||
tone={props.movedClose ? 'info' : stateTone(props.item)}
|
||||
size="xs"
|
||||
shape="rounded"
|
||||
appearance="outline"
|
||||
>
|
||||
{props.movedClose ? 'Moved' : formatLabel(props.item.state)}
|
||||
</MetadataBadge>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -988,6 +988,56 @@ describe('PatrolAttentionWorkbench', () => {
|
||||
expect(screen.getByText('Show all 11 transitions')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('shows an alert handed to its Pulse agent as moved, not recovered', async () => {
|
||||
const reason =
|
||||
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
|
||||
const moved = item({
|
||||
id: 'record-moved',
|
||||
operationalRecordId: 'record-moved',
|
||||
subjectResourceId: 'pve1',
|
||||
subjectResourceName: 'pve1',
|
||||
subjectResourceType: 'node',
|
||||
kind: 'memory',
|
||||
title: 'Memory on pve1',
|
||||
plainLanguageSummary: 'Memory usage at 95%',
|
||||
severity: 'warning',
|
||||
state: 'resolved',
|
||||
protectionPosture: undefined,
|
||||
});
|
||||
const movedDetail = detail(moved);
|
||||
movedDetail.timeline = [
|
||||
...movedDetail.timeline,
|
||||
{
|
||||
id: 'transition-moved',
|
||||
operationalRecordId: 'record-moved',
|
||||
from: 'open',
|
||||
to: 'resolved',
|
||||
at: '2026-07-19T07:45:00Z',
|
||||
cause: 'ownership_transferred',
|
||||
causeKey: 'memory:pve1',
|
||||
evidenceIds: ['evidence-handover'],
|
||||
reason,
|
||||
},
|
||||
];
|
||||
window.history.replaceState({}, '', '/patrol?attention=record-moved');
|
||||
apiMocks.getList.mockResolvedValue(listResponse([], summary()));
|
||||
apiMocks.getDetail.mockResolvedValue(movedDetail);
|
||||
renderWorkbench();
|
||||
|
||||
const panel = await screen.findByRole('complementary', { name: 'pve1 · Memory' });
|
||||
expect(await within(panel).findByText('Moved')).toBeInTheDocument();
|
||||
expect(within(panel).queryByText('Resolved')).not.toBeInTheDocument();
|
||||
// The account sits under the summary, outside the collapsed history.
|
||||
const summaryLine = within(panel).getByText('Memory usage at 95%');
|
||||
expect(summaryLine.nextElementSibling).toHaveTextContent(reason);
|
||||
expect(summaryLine.nextElementSibling?.closest('details')).toBeNull();
|
||||
|
||||
fireEvent.click(within(panel).getByText('Evidence and history'));
|
||||
expect(within(panel).getByText('Open to Resolved')).toBeInTheDocument();
|
||||
expect(within(panel).getByText(/Ownership transferred/)).toBeInTheDocument();
|
||||
expect(within(panel).queryByText(/Recovery evidence/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('opens the canonical governed action review from an eligible attention item', async () => {
|
||||
const actionOffer = {
|
||||
targetResourceId: 'docker:host-1/container-1',
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import type { PatrolObjective } from '@/api/patrol';
|
||||
import type { AttentionItem } from '@/api/patrolAttention';
|
||||
import type { AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
|
||||
import {
|
||||
getAttentionHandoverClose,
|
||||
getPatrolAttentionDecisionReason,
|
||||
getPatrolObjectiveProtectionSummary,
|
||||
partitionPatrolAttention,
|
||||
@@ -145,3 +146,62 @@ describe('patrol home presentation', () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('getAttentionHandoverClose', () => {
|
||||
const transition = (
|
||||
overrides: Partial<AttentionItemDetail['timeline'][number]>,
|
||||
): AttentionItemDetail['timeline'][number] => ({
|
||||
id: 'transition-1',
|
||||
operationalRecordId: 'record-1',
|
||||
from: 'open',
|
||||
to: 'resolved',
|
||||
at: '2026-08-14T08:00:00Z',
|
||||
cause: 'recovery_evidence',
|
||||
causeKey: 'memory:pve1',
|
||||
evidenceIds: ['evidence-1'],
|
||||
...overrides,
|
||||
});
|
||||
const reason =
|
||||
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
|
||||
|
||||
it('returns the handover account only for a record closed by an ownership transfer', () => {
|
||||
const resolved = attentionItem({ state: 'resolved' });
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: resolved,
|
||||
timeline: [transition({ cause: 'ownership_transferred', reason })],
|
||||
}),
|
||||
).toBe(reason);
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: resolved,
|
||||
timeline: [transition({ cause: 'ownership_transferred' })],
|
||||
}),
|
||||
).toBe('This alert moved to another owner when it closed. It did not recover.');
|
||||
expect(getAttentionHandoverClose({ item: resolved, timeline: [transition({})] })).toBeNull();
|
||||
});
|
||||
|
||||
it('follows the latest close, so a later recovery or reopen wins', () => {
|
||||
const handover = transition({ cause: 'ownership_transferred', reason });
|
||||
const reopened = transition({
|
||||
id: 'transition-2',
|
||||
from: 'resolved',
|
||||
to: 'open',
|
||||
at: '2026-08-14T09:00:00Z',
|
||||
cause: 'detector_decision',
|
||||
});
|
||||
const recovered = transition({ id: 'transition-3', at: '2026-08-14T10:00:00Z' });
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: attentionItem({ state: 'resolved' }),
|
||||
timeline: [handover, reopened, recovered],
|
||||
}),
|
||||
).toBeNull();
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: attentionItem({ state: 'open' }),
|
||||
timeline: [handover, reopened],
|
||||
}),
|
||||
).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { PatrolAutonomyLevel, PatrolObjective } from '@/api/patrol';
|
||||
import type { AttentionFlapping, AttentionItem } from '@/api/patrolAttention';
|
||||
import type { AttentionFlapping, AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
|
||||
import type { UnifiedFinding } from '@/stores/aiIntelligence';
|
||||
|
||||
export interface PatrolAutonomyExperience {
|
||||
@@ -207,6 +207,26 @@ export function getAttentionFlappingPresentation(
|
||||
return getFlappingPresentation(flapping.transitionCount, flapping.windowHours);
|
||||
}
|
||||
|
||||
/**
|
||||
* A resolved item whose close handed the condition to another record, such as
|
||||
* a node metric its Pulse agent took over, did not recover. Returns the
|
||||
* account to show instead of a healthy "Resolved", or null for a recovery.
|
||||
*/
|
||||
export function getAttentionHandoverClose(
|
||||
detail: Pick<AttentionItemDetail, 'item' | 'timeline'> | null | undefined,
|
||||
): string | null {
|
||||
if (!detail || detail.item.state !== 'resolved') return null;
|
||||
let close: AttentionItemDetail['timeline'][number] | undefined;
|
||||
for (const transition of detail.timeline ?? []) {
|
||||
if (transition.to !== 'resolved') continue;
|
||||
if (!close || Date.parse(transition.at) >= Date.parse(close.at)) close = transition;
|
||||
}
|
||||
if (close?.cause !== 'ownership_transferred') return null;
|
||||
return (
|
||||
close.reason?.trim() || 'This alert moved to another owner when it closed. It did not recover.'
|
||||
);
|
||||
}
|
||||
|
||||
const normalizeResourceKey = (value: string | undefined): string =>
|
||||
(value ?? '').trim().toLowerCase();
|
||||
|
||||
|
||||
@@ -1,12 +1,5 @@
|
||||
import { useLocation } from '@solidjs/router';
|
||||
import {
|
||||
Show,
|
||||
createEffect,
|
||||
createMemo,
|
||||
createResource,
|
||||
createSignal,
|
||||
type Accessor,
|
||||
} from 'solid-js';
|
||||
import { Show, createEffect, createMemo, createSignal, type Accessor } from 'solid-js';
|
||||
import StorageSurface from '@/components/Storage/Storage';
|
||||
import { WorkloadsFilter } from '@/components/Workloads/WorkloadsFilter';
|
||||
import { WorkloadsSurface } from '@/components/Workloads/WorkloadsSurface';
|
||||
@@ -58,7 +51,9 @@ import { ProxmoxCephTable } from './ProxmoxCephTable';
|
||||
import { ProxmoxMailGatewayTable } from './ProxmoxMailGatewayTable';
|
||||
import { ProxmoxNodesTable } from './ProxmoxNodesTable';
|
||||
import { ProxmoxReplicationTable, fetchReplicationJobs } from './ProxmoxReplicationTable';
|
||||
import { createNonSuspendingQuery } from '@/hooks/createNonSuspendingQuery';
|
||||
import { useUnifiedResources } from '@/hooks/useUnifiedResources';
|
||||
import type { ReplicationJob } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { updateStore } from '@/stores/updates';
|
||||
import {
|
||||
@@ -85,6 +80,8 @@ const PROXMOX_PLATFORM_FILTER = 'proxmox-all';
|
||||
const PROXMOX_WORKLOAD_STATUS_STORAGE_SCOPE = 'proxmox';
|
||||
const PROXMOX_WORKLOAD_EXCLUDED_TYPES = ['app-container'] as const;
|
||||
const PHONE_MOUNTED_TAB_LIMIT = 2;
|
||||
const REPLICATION_JOBS_POLL_MS = 30_000;
|
||||
const NO_REPLICATION_JOBS: ReplicationJob[] = [];
|
||||
const VALID_TABS = new Set<ProxmoxPageTabId>(PROXMOX_TAB_SPECS.map((tab) => tab.id));
|
||||
const PROXMOX_WORKLOAD_STATUS_OPTIONS: readonly WorkloadsStatusOption[] = [
|
||||
{ value: 'all', label: 'All' },
|
||||
@@ -123,13 +120,18 @@ export function ProxmoxPageSurface() {
|
||||
});
|
||||
// Replication jobs come straight from /api/replication/jobs (they bypass
|
||||
// the unified-resource pipeline), so the surface owns the fetch: the job
|
||||
// count gates the Replication tab and the same data feeds the table.
|
||||
// Reading an errored resource throws, hence the `.error` guards.
|
||||
const [replicationJobs, { refetch: refetchReplicationJobs }] =
|
||||
createResource(fetchReplicationJobs);
|
||||
const replicationJobCount = createMemo(() =>
|
||||
replicationJobs.error ? 0 : (replicationJobs() ?? []).length,
|
||||
);
|
||||
// count gates the Replication tab and the same data feeds the table. The
|
||||
// read polls in the background because the table's Last sync and Next sync
|
||||
// keep moving on the shared clock: a snapshot read once at mount would age
|
||||
// a job that keeps syncing and count it overdue. A failed poll keeps the
|
||||
// last jobs, so the tab does not vanish under the user.
|
||||
const replicationJobs = createNonSuspendingQuery({
|
||||
source: () => 'proxmox-pve',
|
||||
fetcher: () => fetchReplicationJobs(),
|
||||
initialValue: NO_REPLICATION_JOBS,
|
||||
pollMs: REPLICATION_JOBS_POLL_MS,
|
||||
});
|
||||
const replicationJobCount = createMemo(() => replicationJobs.value().length);
|
||||
const visibleTabs = createMemo(() => {
|
||||
// An unknown snapshot is not evidence that every optional integration is
|
||||
// present. Never use estate-wide aggregations here: unrelated VMware VMs
|
||||
@@ -146,7 +148,9 @@ export function ProxmoxPageSurface() {
|
||||
const requested = requestedTab();
|
||||
// Do not discard a direct link while counts are still unknown: its own
|
||||
// resource query must be allowed to hydrate before deciding it is absent.
|
||||
return !tabEvidence.facets?.() || visibleTabIds().has(requested) ? requested : 'overview';
|
||||
const countsUnknown =
|
||||
!tabEvidence.facets?.() || (requested === 'replication' && !replicationJobs.resolvedOnce());
|
||||
return countsUnknown || visibleTabIds().has(requested) ? requested : 'overview';
|
||||
});
|
||||
const shouldHydrateTab = (tab: ProxmoxPageTabId) => activeTab() === tab;
|
||||
const overviewResources = useUnifiedResources({
|
||||
@@ -350,7 +354,7 @@ export function ProxmoxPageSurface() {
|
||||
<PlatformOutdatedAgentNotice
|
||||
hosts={outdatedAgentHosts()}
|
||||
targetVersion={serverVersionDisplay()}
|
||||
missingLabel="agent-contributed Proxmox node detail and command support"
|
||||
missingLabel="fixes and node details"
|
||||
copyVariant="latest-detail"
|
||||
actionHref={outdatedAgentUpdatePath()}
|
||||
actionLabel="Open agent upgrade commands"
|
||||
@@ -403,9 +407,9 @@ export function ProxmoxPageSurface() {
|
||||
</Show>
|
||||
<Show when={activeTab() === 'replication'}>
|
||||
<ProxmoxReplicationTable
|
||||
jobs={replicationJobs.error ? undefined : replicationJobs()}
|
||||
error={replicationJobs.error}
|
||||
onRetry={() => void refetchReplicationJobs()}
|
||||
jobs={replicationJobs.resolvedOnce() ? replicationJobs.value() : undefined}
|
||||
error={replicationJobs.error() ?? undefined}
|
||||
onRetry={() => void replicationJobs.refetch()}
|
||||
emptyIcon={<ProxmoxIcon class="h-6 w-6 text-slate-400" />}
|
||||
emptyTitle="No replication jobs"
|
||||
emptyDescription="Replication jobs appear here once PVE is configured to replicate guests between nodes."
|
||||
|
||||
@@ -6,6 +6,7 @@ import { StatusDot } from '@/components/shared/StatusDot';
|
||||
import type { StatusIndicatorVariant } from '@/utils/status';
|
||||
import { TableCell, TableHead, TableRow } from '@/components/shared/Table';
|
||||
import { apiFetch } from '@/utils/apiClient';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
import {
|
||||
PlatformTableToolbar,
|
||||
PlatformErrorState,
|
||||
@@ -234,8 +235,10 @@ const NEXT_SYNC_TONE_CLASS: Record<NextSyncTone, string> = {
|
||||
// An overdue next-sync is the one signal that catches a stalled pvesr
|
||||
// scheduler even while the last sync still reports ok, so it gets its own
|
||||
// column instead of folding into the status pill (which mirrors PVE's own
|
||||
// job state).
|
||||
function nextSyncFor(job: ReplicationJob): { text: string; tone: NextSyncTone } {
|
||||
// job state). The countdown is measured from the shared relative-time clock:
|
||||
// a stalled scheduler leaves nextSync unchanged, so a countdown read once at
|
||||
// render would sit at "in 3m" and never turn overdue.
|
||||
function nextSyncFor(job: ReplicationJob, now: number): { text: string; tone: NextSyncTone } {
|
||||
if (!job.enabled) return { text: '—', tone: 'muted' };
|
||||
let target = 0;
|
||||
if (job.nextSyncUnix && job.nextSyncUnix > 0) {
|
||||
@@ -246,7 +249,7 @@ function nextSyncFor(job: ReplicationJob): { text: string; tone: NextSyncTone }
|
||||
if (Number.isFinite(parsed)) target = parsed;
|
||||
}
|
||||
if (!target) return { text: '—', tone: 'muted' };
|
||||
const minutes = Math.floor((target - Date.now()) / 60_000);
|
||||
const minutes = Math.floor((target - now) / 60_000);
|
||||
if (minutes < 0) {
|
||||
const overdue = Math.abs(minutes);
|
||||
const text = overdue < 60 ? `${overdue}m overdue` : `${Math.floor(overdue / 60)}h overdue`;
|
||||
@@ -283,6 +286,9 @@ export const ProxmoxReplicationTable: Component<{
|
||||
const [search, setSearch] = createSignal('');
|
||||
const [status, setStatus] = createSignal<ReplicationStatusFilter>('all');
|
||||
const [expandedJobKey, setExpandedJobKey] = createSignal<string | null>(null);
|
||||
// Rows stay mounted between job refreshes, so Last sync and Next sync read
|
||||
// the shared clock to keep moving while a job's timestamps do not change.
|
||||
const now = useRelativeTimeNow();
|
||||
|
||||
const filterByStatus = (want: ReplicationStatusFilter) => {
|
||||
const split = splitSearchExclusions(search());
|
||||
@@ -502,7 +508,7 @@ export const ProxmoxReplicationTable: Component<{
|
||||
{(job, index) => {
|
||||
const classification = classifyJob(job);
|
||||
const ind = indicatorFor(classification);
|
||||
const next = nextSyncFor(job);
|
||||
const next = () => nextSyncFor(job, now());
|
||||
const sourceNode = (job.sourceNode ?? '').trim() || '—';
|
||||
const targetNode = (job.targetNode ?? '').trim() || '—';
|
||||
const guestLabel = formatGuestLabel(job);
|
||||
@@ -607,9 +613,9 @@ export const ProxmoxReplicationTable: Component<{
|
||||
}
|
||||
>
|
||||
<span class="tabular-nums text-[10px]">
|
||||
{formatPlatformTableRelativeTimeValue(
|
||||
syncTimeValue(job),
|
||||
).replace(/ ago$/, '')}
|
||||
{formatPlatformTableRelativeTimeValue(syncTimeValue(job), {
|
||||
now: now(),
|
||||
}).replace(/ ago$/, '')}
|
||||
</span>
|
||||
</Show>
|
||||
</TableCell>
|
||||
@@ -618,11 +624,11 @@ export const ProxmoxReplicationTable: Component<{
|
||||
class={`${getPlatformTableCellClassForKind('numeric-value')} text-base-content ${mobilePaddingClass()}`}
|
||||
>
|
||||
<span
|
||||
class={`${NEXT_SYNC_TONE_CLASS[next.tone]} ${isCompact() ? 'text-[10px]' : ''}`}
|
||||
class={`${NEXT_SYNC_TONE_CLASS[next().tone]} ${isCompact() ? 'text-[10px]' : ''}`}
|
||||
>
|
||||
{isCompact()
|
||||
? compactReplicationNextSyncText(next.text, next.tone)
|
||||
: next.text}
|
||||
? compactReplicationNextSyncText(next().text, next().tone)
|
||||
: next().text}
|
||||
</span>
|
||||
</TableCell>
|
||||
</Show>
|
||||
@@ -701,13 +707,15 @@ export const ProxmoxReplicationTable: Component<{
|
||||
<dd class="font-mono text-base-content">{job.schedule || '—'}</dd>
|
||||
<dt class="font-semibold text-muted">Last sync</dt>
|
||||
<dd class="text-base-content">
|
||||
{formatPlatformTableRelativeTimeValue(syncTimeValue(job))}
|
||||
{formatPlatformTableRelativeTimeValue(syncTimeValue(job), {
|
||||
now: now(),
|
||||
})}
|
||||
</dd>
|
||||
<dt class="font-semibold text-muted">Next sync</dt>
|
||||
<dd
|
||||
class={NEXT_SYNC_TONE_CLASS[next.tone] || 'text-base-content'}
|
||||
class={NEXT_SYNC_TONE_CLASS[next().tone] || 'text-base-content'}
|
||||
>
|
||||
{next.text}
|
||||
{next().text}
|
||||
</dd>
|
||||
<dt class="font-semibold text-muted">Duration</dt>
|
||||
<dd class="text-base-content">
|
||||
|
||||
+73
-5
@@ -21,6 +21,7 @@ const mockWorkloadSearch = vi.hoisted(() => vi.fn(() => ''));
|
||||
const mockSelectedNode = vi.hoisted(() => vi.fn<() => string | null>(() => null));
|
||||
const mockHandleNodeSelect = vi.hoisted(() => vi.fn());
|
||||
const mockWorkloadsOptions = vi.hoisted(() => vi.fn());
|
||||
const mockFetchReplicationJobs = vi.hoisted(() => vi.fn(() => Promise.resolve([] as unknown[])));
|
||||
|
||||
const makeResource = (resource: Partial<Resource> & Pick<Resource, 'id' | 'type'>): Resource =>
|
||||
({
|
||||
@@ -152,8 +153,14 @@ vi.mock('../ProxmoxNodesTable', () => ({
|
||||
}));
|
||||
|
||||
vi.mock('../ProxmoxReplicationTable', () => ({
|
||||
ProxmoxReplicationTable: () => <div data-testid="replication-table" />,
|
||||
fetchReplicationJobs: () => Promise.resolve([]),
|
||||
ProxmoxReplicationTable: (props: { jobs?: unknown[]; error?: unknown }) => (
|
||||
<div
|
||||
data-testid="replication-table"
|
||||
data-jobs={props.jobs === undefined ? 'loading' : props.jobs.length}
|
||||
data-error={props.error ? 'yes' : 'no'}
|
||||
/>
|
||||
),
|
||||
fetchReplicationJobs: () => mockFetchReplicationJobs(),
|
||||
}));
|
||||
|
||||
const renderSurface = () =>
|
||||
@@ -247,9 +254,7 @@ describe('ProxmoxPageSurface contract', () => {
|
||||
expect(screen.getByTestId('nodes-table')).toHaveAttribute('data-rows', '1');
|
||||
const notice = screen.getByTestId('platform-outdated-agent-notice');
|
||||
expect(notice).toHaveTextContent('delly runs an older Pulse agent (v5.1.34).');
|
||||
expect(notice).toHaveTextContent(
|
||||
'latest agent-contributed Proxmox node detail and command support',
|
||||
);
|
||||
expect(notice).toHaveTextContent('latest fixes and node details');
|
||||
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
|
||||
'href',
|
||||
'/settings/infrastructure/agent-doctor?agents=agent%3Aagent-delly',
|
||||
@@ -630,6 +635,69 @@ describe('ProxmoxPageSurface contract', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('re-reads replication jobs in the background and keeps the tab through a failed read', async () => {
|
||||
vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] });
|
||||
mockPathname.mockReturnValue('/proxmox/replication');
|
||||
const node = makeResource({ id: 'node:pve1', type: 'agent', name: 'pve1' });
|
||||
mockUseUnifiedResources.mockReturnValue({
|
||||
resources: () => [node],
|
||||
facets: () => ({ incidentCount: 0, byType: {} }),
|
||||
loading: () => false,
|
||||
error: () => null,
|
||||
refetch: vi.fn(async () => []),
|
||||
});
|
||||
let resolveFirstRead: (jobs: unknown[]) => void = () => undefined;
|
||||
mockFetchReplicationJobs.mockReset();
|
||||
mockFetchReplicationJobs.mockReturnValueOnce(
|
||||
new Promise<unknown[]>((resolve) => {
|
||||
resolveFirstRead = resolve;
|
||||
}),
|
||||
);
|
||||
|
||||
try {
|
||||
renderSurface();
|
||||
const tabs = screen.getByTestId('platform-section-tabs');
|
||||
// Counts are known but the first read is pending: a direct link stays put.
|
||||
expect(tabs).toHaveAttribute('data-active', 'replication');
|
||||
expect(screen.getByTestId('replication-table')).toHaveAttribute('data-jobs', 'loading');
|
||||
|
||||
resolveFirstRead([{ id: 'job-1' }]);
|
||||
await waitFor(() =>
|
||||
expect(screen.getByTestId('replication-table')).toHaveAttribute('data-jobs', '1'),
|
||||
);
|
||||
expect(tabs).toHaveAttribute('data-tabs', 'overview,replication');
|
||||
|
||||
// The table's ages move on the shared clock, so its jobs are re-read in
|
||||
// the background rather than aging a snapshot taken at mount.
|
||||
mockFetchReplicationJobs.mockResolvedValueOnce([{ id: 'job-1' }, { id: 'job-2' }]);
|
||||
vi.advanceTimersByTime(30_000);
|
||||
await waitFor(() =>
|
||||
expect(screen.getByTestId('replication-table')).toHaveAttribute('data-jobs', '2'),
|
||||
);
|
||||
|
||||
// A failed read reports the error but keeps the tab and the user on it.
|
||||
mockFetchReplicationJobs.mockRejectedValueOnce(new Error('replication read failed'));
|
||||
vi.advanceTimersByTime(30_000);
|
||||
await waitFor(() =>
|
||||
expect(screen.getByTestId('replication-table')).toHaveAttribute('data-error', 'yes'),
|
||||
);
|
||||
expect(tabs).toHaveAttribute('data-tabs', 'overview,replication');
|
||||
expect(tabs).toHaveAttribute('data-active', 'replication');
|
||||
|
||||
mockFetchReplicationJobs.mockResolvedValueOnce([{ id: 'job-1' }]);
|
||||
vi.advanceTimersByTime(30_000);
|
||||
await waitFor(() =>
|
||||
expect(screen.getByTestId('replication-table')).toHaveAttribute('data-error', 'no'),
|
||||
);
|
||||
expect(screen.getByTestId('replication-table')).toHaveAttribute('data-jobs', '1');
|
||||
expect(mockFetchReplicationJobs).toHaveBeenCalledTimes(4);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
mockFetchReplicationJobs.mockReset();
|
||||
mockFetchReplicationJobs.mockImplementation(() => Promise.resolve([]));
|
||||
}
|
||||
});
|
||||
|
||||
it('does not surface stale-agent notices for development builds without an agent target', () => {
|
||||
mockVersionInfo.mockReturnValue({
|
||||
version: '6.0.0-rc.6+git.172.g2c360f779.dirty',
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { cleanup, fireEvent, render, screen } from '@solidjs/testing-library';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { ReplicationJob } from '@/types/api';
|
||||
import { RELATIVE_TIME_TICK_MS } from '@/utils/relativeTimeClock';
|
||||
import {
|
||||
ProxmoxReplicationTable,
|
||||
REPLICATION_MOBILE_COLUMNS,
|
||||
@@ -132,3 +133,67 @@ describe('ProxmoxReplicationTable', () => {
|
||||
expect(replicationTableSource).toMatch(/job\.lastSyncStatus,\s*job\.error,\s*job\.comment,/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('ProxmoxReplicationTable relative times', () => {
|
||||
const start = Date.parse('2026-10-04T12:00:00Z');
|
||||
// Last synced 5 minutes before the table opens, next sync due 10 minutes after.
|
||||
const job = () =>
|
||||
replicationJob({
|
||||
lastSyncUnix: (start - 5 * 60_000) / 1000,
|
||||
nextSyncUnix: (start + 10 * 60_000) / 1000,
|
||||
});
|
||||
const renderTable = () =>
|
||||
render(() => (
|
||||
<ProxmoxReplicationTable
|
||||
jobs={[job()]}
|
||||
error={undefined}
|
||||
onRetry={() => undefined}
|
||||
emptyIcon={<span />}
|
||||
emptyTitle="No replication jobs"
|
||||
emptyDescription="Replication jobs appear here."
|
||||
/>
|
||||
));
|
||||
// Two hours pass with no new job data; the next clock tick re-reads them.
|
||||
const twoHoursLater = () => {
|
||||
vi.setSystemTime(start + 2 * 60 * 60_000);
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
};
|
||||
|
||||
afterEach(() => vi.useRealTimers());
|
||||
|
||||
it('keeps the compact Last sync and Next sync moving while the job does not change', () => {
|
||||
vi.useFakeTimers({ now: start });
|
||||
vi.spyOn(HTMLElement.prototype, 'clientWidth', 'get').mockReturnValue(400);
|
||||
renderTable();
|
||||
|
||||
expect(screen.getByText('5m')).toBeInTheDocument();
|
||||
expect(screen.getByText('10m')).toBeInTheDocument();
|
||||
|
||||
twoHoursLater();
|
||||
|
||||
// A stalled scheduler leaves nextSync unchanged, so the countdown must
|
||||
// turn overdue on its own.
|
||||
expect(screen.getByText('2h')).toBeInTheDocument();
|
||||
expect(screen.getByText('-1h')).toHaveClass('text-red-600');
|
||||
expect(screen.queryByText('10m')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps the expanded Last sync and Next sync moving while the job does not change', async () => {
|
||||
vi.useFakeTimers({ now: start });
|
||||
renderTable();
|
||||
await fireEvent.click(
|
||||
screen.getByRole('button', { name: 'Expand details for replication job 100-0' }),
|
||||
);
|
||||
const detail = (label: string) =>
|
||||
screen.getByText(label, { selector: 'dt' }).nextElementSibling as HTMLElement;
|
||||
|
||||
expect(detail('Last sync')).toHaveTextContent('5m ago');
|
||||
expect(detail('Next sync')).toHaveTextContent('in 10m');
|
||||
|
||||
twoHoursLater();
|
||||
|
||||
expect(detail('Last sync')).toHaveTextContent('2h ago');
|
||||
expect(detail('Next sync')).toHaveTextContent('1h overdue');
|
||||
expect(detail('Next sync')).toHaveClass('text-red-600');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { cleanup, render, screen } from '@solidjs/testing-library';
|
||||
import { cleanup, render, screen, within } from '@solidjs/testing-library';
|
||||
import { WorkloadTypeBadge } from '@/components/shared/WorkloadTypeBadge';
|
||||
import proxmoxBackupServersTableSource from '../ProxmoxBackupServersTable.tsx?raw';
|
||||
import proxmoxCoverageTableSource from '../ProxmoxCoverageTable.tsx?raw';
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
SortableHead,
|
||||
formatCompactBackupAge,
|
||||
} from '../proxmoxBackupsTableShared';
|
||||
import { RELATIVE_TIME_TICK_MS } from '@/utils/relativeTimeClock';
|
||||
import type { RecoverableArtifact } from '../proxmoxBackupRecoveryModel';
|
||||
|
||||
afterEach(cleanup);
|
||||
@@ -205,3 +206,61 @@ describe('backup-date-evidence age cell', () => {
|
||||
expect(value.title).toContain(createdAt);
|
||||
});
|
||||
});
|
||||
|
||||
describe('backup age cell on the shared clock', () => {
|
||||
afterEach(() => vi.useRealTimers());
|
||||
|
||||
it('keeps the age and its freshness band moving while the artifact does not change', () => {
|
||||
vi.useFakeTimers({ now: new Date('2026-10-04T12:00:00Z') });
|
||||
const createdAt = '2026-10-04T06:00:00Z';
|
||||
const backup = artifact({ createdAt, createdMs: Date.parse(createdAt) });
|
||||
|
||||
render(() => (
|
||||
<>
|
||||
<span data-testid="full">
|
||||
<ProxmoxBackupAgeText artifact={backup} />
|
||||
</span>
|
||||
<span data-testid="compact">
|
||||
<ProxmoxBackupAgeText artifact={backup} compact />
|
||||
</span>
|
||||
</>
|
||||
));
|
||||
|
||||
expect(screen.getByTestId('full')).toHaveTextContent('6h ago');
|
||||
expect(screen.getByText('6h')).toHaveClass('text-emerald-600');
|
||||
|
||||
// Eight days pass with no new backup and no data refresh. The next clock
|
||||
// tick moves both ages and drops the backup out of the current band.
|
||||
vi.setSystemTime(new Date('2026-10-12T12:00:00Z'));
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
|
||||
expect(screen.getByTestId('full')).toHaveTextContent('8d ago');
|
||||
const compact = screen.getByText('8d');
|
||||
expect(compact).toHaveClass('text-amber-600');
|
||||
expect(compact.closest('[title]')?.getAttribute('title')).toBe(
|
||||
`Aging backup age · ${createdAt}`,
|
||||
);
|
||||
});
|
||||
|
||||
it('does not band a backup that finished after the last clock tick as unknown', () => {
|
||||
vi.useFakeTimers({ now: new Date('2026-10-04T12:00:00Z') });
|
||||
// Another mounted cell starts the shared clock at 12:00:00.
|
||||
render(() => <ProxmoxBackupAgeText artifact={artifact()} />);
|
||||
|
||||
// A backup that finished 20s later arrives before the next tick.
|
||||
vi.setSystemTime(new Date('2026-10-04T12:00:25Z'));
|
||||
const createdAt = '2026-10-04T12:00:20Z';
|
||||
render(() => (
|
||||
<span data-testid="fresh">
|
||||
<ProxmoxBackupAgeText
|
||||
artifact={artifact({ createdAt, createdMs: Date.parse(createdAt) })}
|
||||
compact
|
||||
/>
|
||||
</span>
|
||||
));
|
||||
|
||||
const fresh = within(screen.getByTestId('fresh')).getByText('now');
|
||||
expect(fresh).toHaveClass('text-emerald-600');
|
||||
expect(screen.queryByText('Unknown')).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
PlatformTableRelativeTimeValue,
|
||||
formatPlatformTableRelativeTimeValue,
|
||||
} from '@/features/platformPage/sharedPlatformPage';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
|
||||
import {
|
||||
getRecoveryAgeBand,
|
||||
@@ -224,14 +225,20 @@ export function RowMetricBar(props: {
|
||||
|
||||
// Phone columns are too narrow for "18h ago", so the compact form drops the
|
||||
// suffix ("18h", "now") and keeps the full age band and timestamp on hover.
|
||||
export function formatCompactBackupAge(createdAt: string): string {
|
||||
const relative = formatPlatformTableRelativeTimeValue(createdAt);
|
||||
export function formatCompactBackupAge(createdAt: string, now?: number): string {
|
||||
const relative = formatPlatformTableRelativeTimeValue(createdAt, { now });
|
||||
if (relative === 'just now') return 'now';
|
||||
return relative.replace(/ ago$/, '');
|
||||
}
|
||||
|
||||
export function ProxmoxBackupAgeText(props: { artifact: RecoverableArtifact; compact?: boolean }) {
|
||||
const band = () => getRecoveryAgeBand(props.artifact.createdMs);
|
||||
// Backup rows stay mounted while an artifact's creation time never changes,
|
||||
// so the age and its freshness band both read the shared clock. The clock
|
||||
// can trail the wall clock by up to one tick, and a backup that finished
|
||||
// inside that window must not band as a future (unknown) age.
|
||||
const tickNow = useRelativeTimeNow();
|
||||
const now = () => Math.max(tickNow(), Date.now());
|
||||
const band = () => getRecoveryAgeBand(props.artifact.createdMs, now());
|
||||
const title = () => {
|
||||
if (band() === 'unknown') return recoveryAgeTitleByBand.unknown;
|
||||
const parts = [recoveryAgeTitleByBand[band()], props.artifact.createdAt].filter(Boolean);
|
||||
@@ -250,7 +257,7 @@ export function ProxmoxBackupAgeText(props: { artifact: RecoverableArtifact; com
|
||||
when={props.compact}
|
||||
fallback={<PlatformTableRelativeTimeValue value={props.artifact.createdAt} />}
|
||||
>
|
||||
{formatCompactBackupAge(props.artifact.createdAt)}
|
||||
{formatCompactBackupAge(props.artifact.createdAt, now())}
|
||||
</Show>
|
||||
</Show>
|
||||
</span>
|
||||
|
||||
@@ -75,6 +75,7 @@ import {
|
||||
} from '@/utils/raidPresentation';
|
||||
import { asTrimmedString } from '@/utils/stringUtils';
|
||||
import { getPreferredResourceIP } from '@/utils/resourceIdentity';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
import {
|
||||
RESOURCE_METADATA_CHANGED_EVENT,
|
||||
type ResourceMetadataChangedDetail,
|
||||
@@ -1176,6 +1177,9 @@ export const AgentsMachinesTable: Component<{
|
||||
onExternalStatusChange?: (value: PlatformResourceStatusFilter) => void;
|
||||
onResetFilters?: () => void;
|
||||
}> = (props) => {
|
||||
// A machine that stops reporting keeps the same last-seen time, which is
|
||||
// exactly when its age must keep moving, so row ages read the shared clock.
|
||||
const now = useRelativeTimeNow();
|
||||
const [locallyRemovedResourceIds, setLocallyRemovedResourceIds] = createSignal<
|
||||
Record<string, boolean>
|
||||
>({});
|
||||
@@ -1485,7 +1489,7 @@ export const AgentsMachinesTable: Component<{
|
||||
const agentStaleTitle = () => {
|
||||
const last = asTrimmedString(machine.agent?.lastReportAt);
|
||||
const when = last
|
||||
? ` Last report ${formatPlatformTableRelativeTimeValue(last)}.`
|
||||
? ` Last report ${formatPlatformTableRelativeTimeValue(last, { now: now() })}.`
|
||||
: '';
|
||||
return `Agent has stopped reporting.${when} Re-run the install command from the Pulse UI to refresh its token.`;
|
||||
};
|
||||
@@ -1537,7 +1541,7 @@ export const AgentsMachinesTable: Component<{
|
||||
? availabilityFor(machine)?.lastChecked
|
||||
: machine.lastSeen;
|
||||
const lastSeenLabel = () =>
|
||||
formatPlatformTableRelativeTimeValue(lastSeenValue());
|
||||
formatPlatformTableRelativeTimeValue(lastSeenValue(), { now: now() });
|
||||
const machineSubtitle = () =>
|
||||
machineRowSubtitleFor(
|
||||
name(),
|
||||
|
||||
@@ -275,12 +275,12 @@ export const AvailabilityChecksTable: Component<{
|
||||
Check
|
||||
</TableHead>
|
||||
<TableHead
|
||||
class={`${getPlatformTableHeadClassForKind('text')} platform-table-mobile-w-15 md:w-[12%]`}
|
||||
class={`${getPlatformTableHeadClassForKind('text')} platform-table-phone-hidden platform-table-mobile-w-15 md:w-[12%]`}
|
||||
>
|
||||
Method
|
||||
</TableHead>
|
||||
<TableHead
|
||||
class={`${getPlatformTableHeadClassForKind('text')} platform-table-mobile-w-25 md:w-[22%]`}
|
||||
class={`${getPlatformTableHeadClassForKind('text')} platform-table-phone-hidden platform-table-mobile-w-25 md:w-[22%]`}
|
||||
>
|
||||
Target
|
||||
</TableHead>
|
||||
@@ -329,7 +329,7 @@ export const AvailabilityChecksTable: Component<{
|
||||
const target = () => formatTarget(check);
|
||||
const probeSource = () =>
|
||||
(availability()?.locations?.length ?? 0) > 1
|
||||
? `${availability()?.locations?.length} locations · ${availability()?.reportingLocations ?? 0}/${availability()?.expectedLocations ?? availability()?.locations?.length ?? 0} reporting`
|
||||
? `${availability()?.reportingLocations ?? 0}/${availability()?.expectedLocations ?? availability()?.locations?.length ?? 0} locations reporting`
|
||||
: getProbeSourceChipLabel(
|
||||
props.probeAgentOptions ?? [],
|
||||
availability()?.probeAgentId,
|
||||
@@ -366,13 +366,17 @@ export const AvailabilityChecksTable: Component<{
|
||||
</span>
|
||||
</div>
|
||||
</TableCell>
|
||||
{/* On a phone the check name needs the room; the method
|
||||
and target stay in the row drawer. */}
|
||||
<TableCell
|
||||
class={`${getPlatformTableCellClassForKind('text')} text-base-content`}
|
||||
class={`${getPlatformTableCellClassForKind('text')} platform-table-phone-hidden text-base-content`}
|
||||
>
|
||||
{method()}
|
||||
<span class="block truncate" title={method()}>
|
||||
{method()}
|
||||
</span>
|
||||
</TableCell>
|
||||
<TableCell
|
||||
class={`${getPlatformTableCellClassForKind('text')} text-base-content`}
|
||||
class={`${getPlatformTableCellClassForKind('text')} platform-table-phone-hidden text-base-content`}
|
||||
>
|
||||
<span class="block truncate" title={target()}>
|
||||
{target()}
|
||||
@@ -381,22 +385,32 @@ export const AvailabilityChecksTable: Component<{
|
||||
<TableCell
|
||||
class={`${getPlatformTableCellClassForKind('numeric-value')} text-base-content`}
|
||||
>
|
||||
<span class={probe()?.toneClassName ?? ''} title={probe()?.detailLabel}>
|
||||
{result()}
|
||||
</span>
|
||||
<Show when={probeSource()}>
|
||||
{(sourceLabel) => (
|
||||
<MetadataBadge
|
||||
tone="muted"
|
||||
size="xs"
|
||||
appearance="outline"
|
||||
class="mt-0.5 flex"
|
||||
data-availability-probe-source={availability()?.probeAgentId}
|
||||
>
|
||||
{sourceLabel()}
|
||||
</MetadataBadge>
|
||||
)}
|
||||
</Show>
|
||||
{/* One line, as the table windows rows at a single
|
||||
height: the result stays whole at the right edge
|
||||
and the probe source takes what room is left,
|
||||
truncating with its full text on hover. */}
|
||||
<div class="flex min-w-0 items-center justify-end gap-1">
|
||||
<Show when={probeSource()}>
|
||||
{(sourceLabel) => (
|
||||
<MetadataBadge
|
||||
tone="muted"
|
||||
size="xs"
|
||||
appearance="outline"
|
||||
class="min-w-0"
|
||||
title={sourceLabel()}
|
||||
data-availability-probe-source={availability()?.probeAgentId}
|
||||
>
|
||||
<span class="truncate">{sourceLabel()}</span>
|
||||
</MetadataBadge>
|
||||
)}
|
||||
</Show>
|
||||
<span
|
||||
class={`shrink-0 ${probe()?.toneClassName ?? ''}`}
|
||||
title={probe()?.detailLabel}
|
||||
>
|
||||
{result()}
|
||||
</span>
|
||||
</div>
|
||||
</TableCell>
|
||||
<TableCell
|
||||
class={`${getPlatformTableCellClassForKind('numeric-value')} hidden tabular-nums lg:table-cell`}
|
||||
|
||||
@@ -331,7 +331,7 @@ export function StandalonePageSurface() {
|
||||
<PlatformOutdatedAgentNotice
|
||||
hosts={outdatedAgentHosts()}
|
||||
targetVersion={serverVersionDisplay()}
|
||||
missingLabel="agent command support and agent-managed platform detail"
|
||||
missingLabel="fixes and machine details"
|
||||
copyVariant="latest-detail"
|
||||
actionHref={outdatedAgentUpdatePath()}
|
||||
actionLabel="Open agent upgrade commands"
|
||||
|
||||
@@ -6,6 +6,7 @@ import { MonitoringAPI } from '@/api/monitoring';
|
||||
import type { Alert } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { STORAGE_KEYS } from '@/utils/localStorage';
|
||||
import { RELATIVE_TIME_TICK_MS } from '@/utils/relativeTimeClock';
|
||||
import { RESOURCE_METADATA_CHANGED_EVENT } from '@/utils/resourceMetadataEvents';
|
||||
import { AgentsMachinesTable } from '../AgentsMachinesTable';
|
||||
|
||||
@@ -1193,4 +1194,50 @@ describe('AgentsMachinesTable', () => {
|
||||
'Thermal pressure nominal via pmset',
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps a silent machine's seen and last-report ages moving while its data does not change", async () => {
|
||||
const start = Date.parse('2026-07-08T09:05:00Z');
|
||||
vi.useFakeTimers({ toFake: ['Date', 'setInterval', 'clearInterval'], now: start });
|
||||
|
||||
try {
|
||||
render(() => (
|
||||
<AgentsMachinesTable
|
||||
resources={[
|
||||
resource({
|
||||
id: 'omv',
|
||||
name: 'omv',
|
||||
lastSeen: Date.parse('2026-07-08T09:00:00Z'),
|
||||
identity: { ips: ['192.168.0.21'] },
|
||||
agent: {
|
||||
agentVersion: '6.0.2',
|
||||
stale: true,
|
||||
lastReportAt: '2026-07-08T09:00:00Z',
|
||||
},
|
||||
}),
|
||||
]}
|
||||
emptyIcon={emptyIcon}
|
||||
emptyTitle="No machines"
|
||||
emptyDescription="Install Pulse Agent."
|
||||
/>
|
||||
));
|
||||
await openMachineColumnPicker();
|
||||
await fireEvent.click(screen.getByLabelText('Last seen'));
|
||||
|
||||
expect(screen.getByTitle('omv · 192.168.0.21 | seen 5m ago')).toBeInTheDocument();
|
||||
expect(
|
||||
screen.getByTitle(/Agent has stopped reporting\. Last report 5m ago\./),
|
||||
).toBeInTheDocument();
|
||||
|
||||
// The agent stays silent: no new data arrives and only the clock moves.
|
||||
vi.setSystemTime(start + 3 * 60 * 60 * 1000);
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
|
||||
expect(screen.getByTitle('omv · 192.168.0.21 | seen 3h ago')).toBeInTheDocument();
|
||||
expect(
|
||||
screen.getByTitle(/Agent has stopped reporting\. Last report 3h ago\./),
|
||||
).toBeInTheDocument();
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -206,6 +206,50 @@ describe('AvailabilityChecksTable', () => {
|
||||
expect(screen.getByText('via host-gone')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps the result whole on one line beside a multi-location source', () => {
|
||||
renderTable([
|
||||
availabilityResource({
|
||||
availability: {
|
||||
targetId: 'mock-availability-frontend',
|
||||
protocol: 'https',
|
||||
address: 'frontend.demo.pulse.local',
|
||||
enabled: true,
|
||||
available: false,
|
||||
lastChecked: 1_700_000_300_000,
|
||||
reportingLocations: 2,
|
||||
expectedLocations: 2,
|
||||
locations: [
|
||||
{ locationId: 'pulse', kind: 'pulse' },
|
||||
{ locationId: 'agent:host-edge-01', kind: 'agent', probeAgentId: 'host-edge-01' },
|
||||
],
|
||||
} as unknown as Resource['availability'],
|
||||
}),
|
||||
]);
|
||||
|
||||
// Settings words the same fact the same way.
|
||||
const source = screen.getByText('2/2 locations reporting');
|
||||
// One line (rows are windowed at a single height): the source truncates
|
||||
// while the result keeps its full width at the cell's right edge.
|
||||
expect(source).toHaveClass('truncate');
|
||||
const line = source.closest('div');
|
||||
expect(line?.className).toContain('flex');
|
||||
const result = line?.lastElementChild as HTMLElement;
|
||||
expect(result).toHaveClass('shrink-0');
|
||||
expect(result.textContent).not.toBe('');
|
||||
});
|
||||
|
||||
it('gives the check name the room on phones by hiding method and target', () => {
|
||||
renderTable([availabilityResource()]);
|
||||
const headers = screen.getAllByRole('columnheader');
|
||||
const method = headers.find((th) => th.textContent?.trim() === 'Method');
|
||||
const target = headers.find((th) => th.textContent?.trim() === 'Target');
|
||||
expect(method).toHaveClass('platform-table-phone-hidden');
|
||||
expect(target).toHaveClass('platform-table-phone-hidden');
|
||||
const row = screen.getByText('MQTT power meter').closest('tr')!;
|
||||
expect(row.children[1]).toHaveClass('platform-table-phone-hidden');
|
||||
expect(row.children[2]).toHaveClass('platform-table-phone-hidden');
|
||||
});
|
||||
|
||||
it('shows no source chip for locally executed checks', () => {
|
||||
renderTable([availabilityResource()]);
|
||||
|
||||
|
||||
@@ -253,9 +253,7 @@ describe('StandalonePageSurface', () => {
|
||||
|
||||
const notice = screen.getByTestId('platform-outdated-agent-notice');
|
||||
expect(notice).toHaveTextContent('tower runs an older Pulse agent (v5.1.34).');
|
||||
expect(notice).toHaveTextContent(
|
||||
'latest agent command support and agent-managed platform detail',
|
||||
);
|
||||
expect(notice).toHaveTextContent('latest fixes and machine details');
|
||||
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
|
||||
'href',
|
||||
'/settings/infrastructure/agent-doctor?agents=agent%3Aagent-tower',
|
||||
|
||||
@@ -189,7 +189,7 @@ export function TrueNASPageSurface() {
|
||||
<PlatformOutdatedAgentNotice
|
||||
hosts={outdatedAgentHosts()}
|
||||
targetVersion={serverVersionDisplay()}
|
||||
missingLabel="agent-contributed TrueNAS system detail and command support"
|
||||
missingLabel="fixes and system details"
|
||||
copyVariant="latest-detail"
|
||||
actionHref={outdatedAgentUpdatePath()}
|
||||
actionLabel="Open agent upgrade commands"
|
||||
|
||||
@@ -180,9 +180,7 @@ describe('TrueNASPageSurface contract', () => {
|
||||
expect(screen.getByTestId('systems-table')).toHaveAttribute('data-rows', '1');
|
||||
const notice = screen.getByTestId('platform-outdated-agent-notice');
|
||||
expect(notice).toHaveTextContent('truenas-scale runs an older Pulse agent (v5.1.34).');
|
||||
expect(notice).toHaveTextContent(
|
||||
'latest agent-contributed TrueNAS system detail and command support',
|
||||
);
|
||||
expect(notice).toHaveTextContent('latest fixes and system details');
|
||||
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
|
||||
'href',
|
||||
'/settings/infrastructure/agent-doctor?agents=agent%3Aagent-truenas-scale',
|
||||
|
||||
@@ -9,8 +9,10 @@ vi.mock('@/components/shared/responsive', () => ({
|
||||
ResponsiveMetricCell: () => <div data-testid="responsive-metric-cell" />,
|
||||
}));
|
||||
|
||||
const activeAlertsRef = vi.hoisted(() => ({ current: {} as Record<string, unknown> }));
|
||||
|
||||
vi.mock('@/contexts/appRuntime', () => ({
|
||||
useWebSocket: () => ({ activeAlerts: {} as Record<string, never> }),
|
||||
useWebSocket: () => ({ activeAlerts: activeAlertsRef.current }),
|
||||
}));
|
||||
vi.mock('@/stores/alertsActivation', () => ({
|
||||
useAlertsActivation: () => ({
|
||||
@@ -43,6 +45,7 @@ const makeSystem = (overrides: Partial<Resource> & Pick<Resource, 'id'>): Resour
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
activeAlertsRef.current = {};
|
||||
});
|
||||
|
||||
describe('TrueNASSystemsTable', () => {
|
||||
@@ -107,4 +110,42 @@ describe('TrueNASSystemsTable', () => {
|
||||
expect(container.querySelector('[data-truenas-system-row="truenas-healthy"]')).toBeNull();
|
||||
expect(screen.getByText('1 of 2 systems')).toBeInTheDocument();
|
||||
});
|
||||
it("tints a system row for its pools' alerts, which carry the system hostname", () => {
|
||||
// TrueNAS pool, dataset, disk and app alerts are keyed on the child
|
||||
// resource and name the system in "node". The system's drawer states the
|
||||
// same pool problem through its health issue, so the row matches by
|
||||
// hostname as well as id; no producer keys a TrueNAS system's alerts on
|
||||
// its "agent:" alias.
|
||||
activeAlertsRef.current = {
|
||||
pool: {
|
||||
id: 'pool',
|
||||
type: 'truenas-pool-health',
|
||||
level: 'warning',
|
||||
resourceId: 'storage-archive',
|
||||
resourceName: 'archive',
|
||||
node: 'truenas-main',
|
||||
instance: 'TrueNAS',
|
||||
message: 'Pool archive is DEGRADED',
|
||||
value: 0,
|
||||
threshold: 0,
|
||||
startTime: '2026-10-06T10:00:00Z',
|
||||
acknowledged: false,
|
||||
},
|
||||
};
|
||||
|
||||
const { container } = render(() => (
|
||||
<TrueNASSystemsTable
|
||||
systems={[makeSystem({ id: 'truenas-main' }), makeSystem({ id: 'truenas-backup' })]}
|
||||
scope={[makeSystem({ id: 'truenas-main' }), makeSystem({ id: 'truenas-backup' })]}
|
||||
emptyIcon={<span />}
|
||||
emptyTitle="No systems"
|
||||
emptyDescription="No systems"
|
||||
showToolbar={false}
|
||||
/>
|
||||
));
|
||||
|
||||
const row = (id: string) => container.querySelector(`[data-truenas-system-row="${id}"]`);
|
||||
expect(row('truenas-main')).toHaveClass('bg-yellow-50');
|
||||
expect(row('truenas-backup')).not.toHaveClass('bg-yellow-50');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -352,7 +352,7 @@ export function VmwarePageSurface() {
|
||||
<PlatformOutdatedAgentNotice
|
||||
hosts={outdatedAgentVMs()}
|
||||
targetVersion={serverVersionDisplay()}
|
||||
missingLabel="in-guest telemetry and command support"
|
||||
missingLabel="fixes and VM details"
|
||||
copyVariant="latest-detail"
|
||||
actionHref={outdatedAgentUpdatePath()}
|
||||
actionLabel="Open agent upgrade commands"
|
||||
|
||||
@@ -179,7 +179,7 @@ describe('VmwarePageSurface contract', () => {
|
||||
expect(screen.getByTestId('hosts-table')).toHaveAttribute('data-rows', '1');
|
||||
const notice = screen.getByTestId('platform-outdated-agent-notice');
|
||||
expect(notice).toHaveTextContent('app-01 runs an older Pulse agent (v5.1.34).');
|
||||
expect(notice).toHaveTextContent('latest in-guest telemetry and command support.');
|
||||
expect(notice).toHaveTextContent('latest fixes and VM details.');
|
||||
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
|
||||
'href',
|
||||
'/settings/infrastructure/agent-doctor?agents=agent%3Aagent-app-01',
|
||||
|
||||
@@ -100,6 +100,7 @@ import {
|
||||
buildProjectedOverrides,
|
||||
normalizeRawOverridesConfig,
|
||||
} from '@/features/alerts/alertOverridesModel';
|
||||
import { buildAlertHistoryItems } from '@/features/alerts/alertHistoryModel';
|
||||
import {
|
||||
getAlertIncidentAcknowledgedBadgeClass,
|
||||
getAlertIncidentEventFilterActionButtonClass,
|
||||
@@ -1320,3 +1321,59 @@ describe('Unified selector parity', () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('alert history close outcome', () => {
|
||||
it('lists a node alert handed to its Pulse agent as moved, not resolved', () => {
|
||||
const base = {
|
||||
type: 'memory',
|
||||
level: 'warning' as const,
|
||||
resourceName: 'pve1',
|
||||
node: 'pve1',
|
||||
instance: 'pve1',
|
||||
value: 95,
|
||||
threshold: 85,
|
||||
startTime: '2026-10-06T09:00:00Z',
|
||||
lastSeen: '2026-10-06T11:00:00Z',
|
||||
acknowledged: true,
|
||||
};
|
||||
const items = buildAlertHistoryItems({
|
||||
activeAlerts: {},
|
||||
alertHistory: [
|
||||
{
|
||||
...base,
|
||||
id: 'pve1-memory',
|
||||
resourceId: 'pve1',
|
||||
message: 'Memory usage at 95%',
|
||||
resolution: {
|
||||
reason: 'moved_to_agent',
|
||||
successorResourceId: 'agent-pve1',
|
||||
successorName: 'pve1 (Host Agent)',
|
||||
},
|
||||
},
|
||||
{
|
||||
...base,
|
||||
id: 'pve1-cpu',
|
||||
resourceId: 'pve1',
|
||||
message: 'CPU usage at 90%',
|
||||
acknowledged: false,
|
||||
},
|
||||
],
|
||||
getResource: () => undefined,
|
||||
allResources: [],
|
||||
now: Date.parse('2026-10-06T12:00:00Z'),
|
||||
});
|
||||
|
||||
const moved = items.find((item) => item.id === 'pve1-memory');
|
||||
// Acknowledgement does not hide the outcome: the condition went to the agent.
|
||||
expect(moved?.status).toBe('moved');
|
||||
expect(moved?.description).toBe('Memory usage at 95%');
|
||||
expect(moved?.closeDetail).toBe(
|
||||
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
|
||||
);
|
||||
const recovered = items.find((item) => item.id === 'pve1-cpu');
|
||||
expect(recovered?.status).toBe('resolved');
|
||||
expect(recovered).not.toHaveProperty('closeDetail');
|
||||
|
||||
expect(alertHistoryTableAlertRowSource).toContain('title={props.alert.closeDetail}');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1380,6 +1380,18 @@ export interface Alert {
|
||||
latestTransition?: LifecycleTransition;
|
||||
transitions?: LifecycleTransition[];
|
||||
evidence?: EvidenceEnvelope[];
|
||||
// Set only on a closed alert whose condition did not clear, such as a node
|
||||
// metric handed to its Pulse agent. Absent for an ordinary recovery.
|
||||
resolution?: AlertResolution;
|
||||
}
|
||||
|
||||
// Why an alert closed without recovering (alerts.AlertResolution).
|
||||
export interface AlertResolution {
|
||||
reason: 'moved_to_agent' | (string & {});
|
||||
successorResourceId?: string;
|
||||
successorName?: string;
|
||||
// The alert engine's one-line account; carried on state snapshots only.
|
||||
summary?: string;
|
||||
}
|
||||
|
||||
export interface ResolvedAlert extends Alert {
|
||||
|
||||
@@ -97,6 +97,7 @@ export type TransitionCause =
|
||||
| 'suppression'
|
||||
| 'suppression_expired'
|
||||
| 'recovery_evidence'
|
||||
| 'ownership_transferred'
|
||||
| 'collection_stale'
|
||||
| 'collection_unknown';
|
||||
|
||||
|
||||
@@ -10,8 +10,10 @@ import {
|
||||
ALERT_RESOURCE_INCIDENT_REFRESH_LABEL,
|
||||
ALERT_RESOURCE_INCIDENT_SAVE_NOTE_LABEL,
|
||||
ALERT_RESOURCE_INCIDENT_SAVING_NOTE_LABEL,
|
||||
ALERT_HISTORY_MOVED_STATUS,
|
||||
getAlertHistoryStatusPresentation,
|
||||
getAlertIncidentLevelBadgeClass,
|
||||
getAlertResolutionDetail,
|
||||
getAlertIncidentTimelineDialogTitle,
|
||||
getAlertHistoryInvestigationDialogTitle,
|
||||
getAlertResourceIncidentAcknowledgedByLabel,
|
||||
@@ -103,6 +105,29 @@ describe('alertIncidentPresentation', () => {
|
||||
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
|
||||
rowClassName: '',
|
||||
});
|
||||
|
||||
expect(getAlertHistoryStatusPresentation(ALERT_HISTORY_MOVED_STATUS)).toEqual({
|
||||
label: 'moved to agent',
|
||||
className:
|
||||
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
|
||||
rowClassName: '',
|
||||
});
|
||||
});
|
||||
|
||||
it('says a handed-over alert did not recover, and leaves recoveries alone', () => {
|
||||
expect(getAlertResolutionDetail(undefined)).toBeUndefined();
|
||||
expect(
|
||||
getAlertResolutionDetail({ reason: 'moved_to_agent', successorName: 'pve1 (Host Agent)' }),
|
||||
).toBe(
|
||||
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
|
||||
);
|
||||
expect(getAlertResolutionDetail({ reason: 'moved_to_agent' })).toBe(
|
||||
'Moved to its Pulse agent. This is not a recovery: check the agent for the current reading.',
|
||||
);
|
||||
// A future reason the UI has no copy for still shows the engine's account.
|
||||
expect(
|
||||
getAlertResolutionDetail({ reason: 'replaced', summary: 'Alert replaced by a new check.' }),
|
||||
).toBe('Alert replaced by a new check.');
|
||||
});
|
||||
|
||||
it('returns canonical resource incident loading and empty-state copy', () => {
|
||||
|
||||
@@ -187,7 +187,32 @@ describe('resourceIdentity', () => {
|
||||
{ label: 'Cluster', value: 'cluster-a' },
|
||||
{ label: 'Parent', value: 'parent-1' },
|
||||
{ label: 'Discovery', value: 'agent:agent-1' },
|
||||
{ label: 'Metrics Target', value: 'docker-host:docker-host-1' },
|
||||
// The metrics target is the Primary ID shown above, so it is not repeated.
|
||||
]);
|
||||
});
|
||||
|
||||
it('keeps a discovery target that matches only a name the drawer never shows', () => {
|
||||
const rows = getPrimaryResourceIdentityRows(
|
||||
makeResource({
|
||||
displayName: 'Friendly host',
|
||||
canonicalIdentity: { displayName: 'agent:other' },
|
||||
discoveryTarget: { resourceType: 'agent', agentId: 'other', resourceId: 'other' },
|
||||
metricsTarget: { resourceType: 'node', resourceId: 'self' },
|
||||
} as Parameters<typeof makeResource>[0]),
|
||||
);
|
||||
expect(rows.find((row) => row.label === 'Discovery')?.value).toBe('agent:other');
|
||||
});
|
||||
|
||||
it('shows an identity once when discovery and metrics targets name the resource itself', () => {
|
||||
const rows = getPrimaryResourceIdentityRows(
|
||||
makeResource({
|
||||
identity: { hostname: 'tower.local' },
|
||||
discoveryTarget: { resourceType: 'agent', agentId: 'agent-1', resourceId: 'agent-1' },
|
||||
metricsTarget: { resourceType: 'agent', resourceId: 'agent-1' },
|
||||
}),
|
||||
);
|
||||
expect(rows.filter((row) => row.value === 'agent:agent-1').map((row) => row.label)).toEqual([
|
||||
'Primary ID',
|
||||
]);
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { AlertResolution } from '@/types/api';
|
||||
import { getAlertSeverityBadgeClass } from '@/utils/alertSeverityPresentation';
|
||||
|
||||
export type AlertIncidentLevel = 'warning' | 'critical' | string | null | undefined;
|
||||
@@ -76,6 +77,24 @@ export function getAlertIncidentLevelBadgeClass(level: AlertIncidentLevel): stri
|
||||
return getAlertSeverityBadgeClass(String(level ?? 'warning'));
|
||||
}
|
||||
|
||||
// History status for an alert that closed because its Pulse agent took the
|
||||
// metric over. The condition did not clear; the agent's own alert owns it.
|
||||
export const ALERT_HISTORY_MOVED_STATUS = 'moved';
|
||||
|
||||
/**
|
||||
* The account of a close that was not a recovery, for history rows and their
|
||||
* tooltips. Returns undefined for an ordinary recovery so callers keep their
|
||||
* "resolved" wording.
|
||||
*/
|
||||
export function getAlertResolutionDetail(resolution?: AlertResolution | null): string | undefined {
|
||||
if (!resolution) return undefined;
|
||||
if (resolution.reason === 'moved_to_agent') {
|
||||
const successor = resolution.successorName?.trim() || 'its Pulse agent';
|
||||
return `Moved to ${successor}. This is not a recovery: check the agent for the current reading.`;
|
||||
}
|
||||
return resolution.summary?.trim() || undefined;
|
||||
}
|
||||
|
||||
export function getAlertHistoryStatusPresentation(
|
||||
status?: string | null,
|
||||
): AlertHistoryStatusPresentation {
|
||||
@@ -99,6 +118,17 @@ export function getAlertHistoryStatusPresentation(
|
||||
};
|
||||
}
|
||||
|
||||
// Closed without recovering: neither a live problem on this row nor a
|
||||
// healthy outcome, so it gets its own neutral-blue badge.
|
||||
if (normalized === ALERT_HISTORY_MOVED_STATUS) {
|
||||
return {
|
||||
label: 'moved to agent',
|
||||
className:
|
||||
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
|
||||
rowClassName: '',
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
label: normalized || 'resolved',
|
||||
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
|
||||
|
||||
@@ -289,11 +289,21 @@ export const getPrimaryResourceIdentityRows = (resource: Resource): ResourceIden
|
||||
rows.push({ label: 'Parent', value: resource.parentId });
|
||||
}
|
||||
|
||||
// Discovery and metrics targets usually point at the resource itself, so
|
||||
// they show only when they name something not already on screen: the
|
||||
// drawer header's name, the rows above, or the Platform ID row.
|
||||
const shownIdentity = (value: string): boolean =>
|
||||
[
|
||||
getPreferredInfrastructureDisplayName(resource),
|
||||
shouldShowResourcePlatformId(resource) ? asTrimmedString(resource.platformId) : undefined,
|
||||
...rows.map((row) => row.value),
|
||||
].some((shown) => Boolean(shown) && shown!.toLowerCase() === value.toLowerCase());
|
||||
|
||||
const discoveryIdentity = formatIdentityTarget(
|
||||
resource.discoveryTarget?.resourceType,
|
||||
resource.discoveryTarget?.resourceId,
|
||||
);
|
||||
if (discoveryIdentity) {
|
||||
if (discoveryIdentity && !shownIdentity(discoveryIdentity)) {
|
||||
rows.push({ label: 'Discovery', value: discoveryIdentity });
|
||||
}
|
||||
|
||||
@@ -301,7 +311,7 @@ export const getPrimaryResourceIdentityRows = (resource: Resource): ResourceIden
|
||||
resource.metricsTarget?.resourceType,
|
||||
resource.metricsTarget?.resourceId,
|
||||
);
|
||||
if (metricsIdentity) {
|
||||
if (metricsIdentity && !shownIdentity(metricsIdentity)) {
|
||||
rows.push({ label: 'Metrics Target', value: metricsIdentity });
|
||||
}
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package ai
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
@@ -56,6 +57,7 @@ func (a *AlertManagerAdapter) GetRecentlyResolved(minutes int) []ResolvedAlertIn
|
||||
AlertInfo: convertAlertFromModels(&resolved.Alert),
|
||||
ResolvedTime: resolved.ResolvedTime,
|
||||
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
|
||||
Resolution: resolvedAlertResolution(resolved.Resolution),
|
||||
}
|
||||
result = append(result, info)
|
||||
}
|
||||
@@ -98,6 +100,7 @@ func (a *AlertManagerAdapter) GetAlertHistory(resourceID string, limit int) []Re
|
||||
AlertInfo: convertAlertFromModels(&resolved.Alert),
|
||||
ResolvedTime: resolved.ResolvedTime,
|
||||
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
|
||||
Resolution: resolvedAlertResolution(resolved.Resolution),
|
||||
}
|
||||
result = append(result, info)
|
||||
if len(result) >= limit {
|
||||
@@ -245,3 +248,12 @@ func formatDuration(d time.Duration) string {
|
||||
}
|
||||
return fmt.Sprintf("%d days", days)
|
||||
}
|
||||
|
||||
// resolvedAlertResolution is the one-line account of a close that was not a
|
||||
// recovery, empty for an ordinary resolve.
|
||||
func resolvedAlertResolution(resolution *models.AlertResolution) string {
|
||||
if resolution == nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(resolution.Summary)
|
||||
}
|
||||
|
||||
@@ -31,6 +31,9 @@ type ResolvedAlertInfo struct {
|
||||
AlertInfo
|
||||
ResolvedTime time.Time `json:"resolved_time"`
|
||||
Duration string `json:"total_duration"` // How long the alert lasted
|
||||
// Resolution is set only when the alert closed without recovering, such
|
||||
// as a node metric handed to its Pulse agent.
|
||||
Resolution string `json:"resolution,omitempty"`
|
||||
}
|
||||
|
||||
// AlertProvider provides access to the current alert state
|
||||
@@ -131,6 +134,12 @@ func (s *Service) buildAlertContext() string {
|
||||
}
|
||||
for i := 0; i < limit; i++ {
|
||||
a := recentlyResolved[i]
|
||||
if resolution := strings.TrimSpace(a.Resolution); resolution != "" {
|
||||
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, closed %s ago) %s",
|
||||
a.Type, a.ResourceName, a.Message, a.Duration,
|
||||
formatTimeAgo(a.ResolvedTime), resolution))
|
||||
continue
|
||||
}
|
||||
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, resolved %s ago)",
|
||||
a.Type, a.ResourceName, a.Message, a.Duration,
|
||||
formatTimeAgo(a.ResolvedTime)))
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
)
|
||||
|
||||
type stubAlertProvider struct {
|
||||
@@ -151,6 +153,61 @@ func TestService_buildAlertContext_NoActiveWithResolved(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The assistant reads recently resolved alerts through the adapter, so a node
|
||||
// alert handed to its Pulse agent must reach the prompt as a handover and not
|
||||
// as "resolved", which the model would read as the node being healthy.
|
||||
func TestService_buildAlertContext_HandoverIsNotARecovery(t *testing.T) {
|
||||
now := time.Now()
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
adapter := NewAlertManagerAdapter(&stubAlertManager{resolved: []models.ResolvedAlert{
|
||||
{
|
||||
Alert: models.Alert{
|
||||
ID: "pve1-memory",
|
||||
Type: "memory",
|
||||
Level: "warning",
|
||||
ResourceID: "pve1",
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
StartTime: now.Add(-time.Hour),
|
||||
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
|
||||
},
|
||||
ResolvedTime: now.Add(-time.Minute),
|
||||
},
|
||||
{
|
||||
Alert: models.Alert{
|
||||
ID: "pve2-cpu",
|
||||
Type: "cpu",
|
||||
Level: "warning",
|
||||
ResourceID: "pve2",
|
||||
ResourceName: "pve2",
|
||||
Message: "CPU usage at 90%",
|
||||
StartTime: now.Add(-time.Hour),
|
||||
},
|
||||
ResolvedTime: now.Add(-time.Minute),
|
||||
},
|
||||
}})
|
||||
|
||||
resolved := adapter.GetRecentlyResolved(30)
|
||||
if len(resolved) != 2 || resolved[0].Resolution != summary || resolved[1].Resolution != "" {
|
||||
t.Fatalf("GetRecentlyResolved = %+v, want only the handover to carry a resolution", resolved)
|
||||
}
|
||||
if history := adapter.GetAlertHistory("pve1", 5); len(history) != 1 || history[0].Resolution != summary {
|
||||
t.Fatalf("GetAlertHistory = %+v, want the handover resolution", history)
|
||||
}
|
||||
|
||||
s := &Service{}
|
||||
s.SetAlertProvider(adapter)
|
||||
ctx := s.buildAlertContext()
|
||||
if !strings.Contains(ctx, "- **memory** on pve1: Memory usage at 95% (lasted ") ||
|
||||
!strings.Contains(ctx, ", closed 1 minute ago) "+summary) {
|
||||
t.Fatalf("expected the handover line, got: %s", ctx)
|
||||
}
|
||||
if !strings.Contains(ctx, "- **cpu** on pve2: CPU usage at 90% (lasted ") ||
|
||||
!strings.Contains(ctx, ", resolved 1 minute ago)") {
|
||||
t.Fatalf("expected the ordinary recovery line, got: %s", ctx)
|
||||
}
|
||||
}
|
||||
|
||||
func TestService_buildTargetAlertContext(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetAlertProvider(&stubAlertProvider{
|
||||
|
||||
@@ -474,6 +474,17 @@ func (s *IncidentStore) RecordAlertUnacknowledged(alert *alerts.Alert, user stri
|
||||
s.saveAsync()
|
||||
}
|
||||
|
||||
// alertResolvedEventSummary is the timeline line for an alert's close: the
|
||||
// handover account when it closed without recovering, else "Alert resolved".
|
||||
func alertResolvedEventSummary(alert *alerts.Alert) string {
|
||||
if alert != nil {
|
||||
if summary := alert.Resolution.Summary(); summary != "" {
|
||||
return summary
|
||||
}
|
||||
}
|
||||
return "Alert resolved"
|
||||
}
|
||||
|
||||
// RecordAlertResolved records a resolved event and closes the incident.
|
||||
func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time.Time) {
|
||||
if alert == nil {
|
||||
@@ -507,7 +518,7 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time
|
||||
shell.OccurrenceClosedAt = cloneTime(resolvedAt)
|
||||
|
||||
if !s.projectsFromCanonicalLocked() {
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, "Alert resolved", map[string]interface{}{
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
|
||||
"resolved_at": resolvedAt.Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
@@ -563,7 +574,7 @@ func (s *IncidentStore) EnsureAlertOccurrence(alert *alerts.Alert, resolvedAt *t
|
||||
if resolvedAt != nil && !resolvedAt.IsZero() && (shell.OccurrenceRefiredAt == nil || !resolvedAt.Before(*shell.OccurrenceRefiredAt)) {
|
||||
shell.OccurrenceClosedAt = cloneTime(*resolvedAt)
|
||||
if !hasIncidentEventType(shell.Events, IncidentEventAlertResolved) {
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, "Alert resolved", map[string]interface{}{
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
|
||||
"resolved_at": resolvedAt.Format(time.RFC3339),
|
||||
"projection_source": incidentSnapshotSource,
|
||||
})
|
||||
@@ -1113,6 +1124,13 @@ func incidentEventSummaryFromChange(change unifiedresources.ResourceChange, even
|
||||
case IncidentEventAlertUnsnoozed:
|
||||
return "Alert notifications resumed"
|
||||
case IncidentEventAlertResolved:
|
||||
// A handover closes the alert without recovering; its change reason
|
||||
// says so and where the condition went.
|
||||
if _, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertResolution); ok {
|
||||
if reason := strings.TrimSpace(change.Reason); reason != "" {
|
||||
return reason
|
||||
}
|
||||
}
|
||||
return "Alert resolved"
|
||||
default:
|
||||
if summary := strings.TrimSpace(change.Reason); summary != "" {
|
||||
|
||||
@@ -239,6 +239,39 @@ func TestIncidentStore_EqualTimestampLifecycleKeepsResolvedState(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Without a canonical resource timeline the store writes its own close event,
|
||||
// which must say the alert moved rather than resolved.
|
||||
func TestIncidentStore_HandoverCloseIsNotARecovery(t *testing.T) {
|
||||
store := NewIncidentStore(IncidentStoreConfig{MaxIncidents: 10, MaxEventsPerIncident: 10, MaxAgeDays: 30})
|
||||
startedAt := time.Now().UTC().Truncate(time.Second).Add(-time.Hour)
|
||||
movedAt := startedAt.Add(30 * time.Minute)
|
||||
alert := &alerts.Alert{
|
||||
ID: "pve1-memory",
|
||||
Type: "memory",
|
||||
ResourceID: "pve1",
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
StartTime: startedAt,
|
||||
LastSeen: startedAt,
|
||||
Resolution: &alerts.AlertResolution{Reason: alerts.AlertResolutionMovedToAgent, SuccessorName: "pve1 (Host Agent)"},
|
||||
}
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
|
||||
store.RecordAlertFired(alert)
|
||||
store.RecordAlertResolved(alert, movedAt)
|
||||
timeline := store.GetTimelineByAlertAt(alert.ID, startedAt)
|
||||
if timeline == nil || len(timeline.Events) != 2 || timeline.Events[1].Summary != summary {
|
||||
t.Fatalf("recorded timeline = %#v, want the handover close", timeline)
|
||||
}
|
||||
|
||||
snapshot := alert.Clone()
|
||||
snapshot.ID = "pve1-memory-imported"
|
||||
repaired := store.EnsureAlertOccurrence(snapshot, &movedAt)
|
||||
if repaired == nil || len(repaired.Events) != 2 || repaired.Events[1].Summary != summary {
|
||||
t.Fatalf("repaired timeline = %#v, want the handover close", repaired)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIncidentStore_ProjectsCanonicalTimelineWhenAttached(t *testing.T) {
|
||||
store := NewIncidentStore(IncidentStoreConfig{
|
||||
MaxIncidents: 10,
|
||||
|
||||
@@ -3499,6 +3499,12 @@ func (p *PatrolService) seedHealthAndAlertsState(snap patrolRuntimeState, scoped
|
||||
sb.WriteString("# Recently Resolved Alerts\n")
|
||||
for _, r := range alerts {
|
||||
ago := seedFormatTimeAgo(now, r.ResolvedTime)
|
||||
// A handover closes the alert without the condition clearing, so
|
||||
// it must not read as the resource having recovered.
|
||||
if r.Resolution != nil && strings.TrimSpace(r.Resolution.Summary) != "" {
|
||||
sb.WriteString(fmt.Sprintf("- %s — closed %s. %s\n", seedAlertSubject(r.Alert), ago, strings.TrimSpace(r.Resolution.Summary)))
|
||||
continue
|
||||
}
|
||||
sb.WriteString(fmt.Sprintf("- %s — resolved %s\n", seedAlertSubject(r.Alert), ago))
|
||||
}
|
||||
sb.WriteString("\n")
|
||||
|
||||
@@ -922,6 +922,42 @@ func TestSeedHealthAndAlerts_NoIssues(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Patrol's seed must not tell the model a node recovered when its alert only
|
||||
// moved to the node's Pulse agent.
|
||||
func TestSeedHealthAndAlerts_HandoverIsNotARecovery(t *testing.T) {
|
||||
ps := NewPatrolService(nil, nil)
|
||||
cfg := DefaultPatrolConfig()
|
||||
now := time.Now()
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
state := models.StateSnapshot{
|
||||
RecentlyResolved: []models.ResolvedAlert{
|
||||
{
|
||||
Alert: models.Alert{
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
|
||||
},
|
||||
ResolvedTime: now.Add(-3 * time.Minute),
|
||||
},
|
||||
{
|
||||
Alert: models.Alert{ResourceName: "pve2", Message: "CPU usage at 90%"},
|
||||
ResolvedTime: now.Add(-5 * time.Minute),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
out := ps.seedHealthAndAlertsState(patrolRuntimeStateForTest(ps, state), nil, cfg, now)
|
||||
if !strings.Contains(out, "- pve1: Memory usage at 95% — closed 3m ago. "+summary+"\n") {
|
||||
t.Fatalf("expected the handover line, got: %s", out)
|
||||
}
|
||||
if strings.Contains(out, "pve1: Memory usage at 95% — resolved") {
|
||||
t.Fatalf("handover reported as a recovery: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "- pve2: CPU usage at 90% — resolved 5m ago\n") {
|
||||
t.Fatalf("expected the ordinary recovery line, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedHealthAndAlerts_UnknownDiskHealthDoesNotClaimSMARTPassed(t *testing.T) {
|
||||
ps := NewPatrolService(nil, nil)
|
||||
cfg := DefaultPatrolConfig()
|
||||
|
||||
@@ -2364,4 +2364,9 @@ type ResolvedAlertSummary struct {
|
||||
Threshold float64 `json:"threshold,omitempty"`
|
||||
StartTime time.Time `json:"start_time"`
|
||||
ResolvedTime time.Time `json:"resolved_time"`
|
||||
// Resolution is set only when the alert closed without recovering, such
|
||||
// as a node metric handed to its Pulse agent; the successor resource
|
||||
// owns the condition from then on.
|
||||
Resolution string `json:"resolution,omitempty"`
|
||||
SuccessorResourceID string `json:"successor_resource_id,omitempty"`
|
||||
}
|
||||
|
||||
@@ -19,7 +19,7 @@ func (e *PulseToolExecutor) registerAlertsTools() {
|
||||
Actions:
|
||||
- list: List active threshold alerts (CPU > 80%, disk full, etc.)
|
||||
- findings: List AI patrol findings (detected issues)
|
||||
- resolved: List recently resolved alerts
|
||||
- resolved: List recently resolved alerts; one with a resolution closed without recovering
|
||||
- resolve: Mark a finding as resolved
|
||||
- dismiss: Dismiss a finding as not an issue
|
||||
|
||||
@@ -417,7 +417,7 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
|
||||
break
|
||||
}
|
||||
|
||||
alerts = append(alerts, ResolvedAlertSummary{
|
||||
summary := ResolvedAlertSummary{
|
||||
ID: alert.ID,
|
||||
Type: alert.Type,
|
||||
Level: alert.Level,
|
||||
@@ -430,7 +430,12 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
|
||||
Threshold: alert.Threshold,
|
||||
StartTime: alert.StartTime,
|
||||
ResolvedTime: alert.ResolvedTime,
|
||||
})
|
||||
}
|
||||
if resolution := alert.Resolution; resolution != nil {
|
||||
summary.Resolution = resolution.Summary
|
||||
summary.SuccessorResourceID = resolution.SuccessorResourceID
|
||||
}
|
||||
alerts = append(alerts, summary)
|
||||
}
|
||||
|
||||
if alerts == nil {
|
||||
|
||||
@@ -215,6 +215,49 @@ func TestExecuteListResolvedAlerts(t *testing.T) {
|
||||
if len(resp.Alerts) != 1 || resp.Alerts[0].ID != "a1" {
|
||||
t.Fatalf("unexpected resolved alerts: %+v", resp)
|
||||
}
|
||||
if resp.Alerts[0].Resolution != "" || resp.Alerts[0].SuccessorResourceID != "" {
|
||||
t.Fatalf("a recovery must not carry a resolution: %+v", resp.Alerts[0])
|
||||
}
|
||||
}
|
||||
|
||||
// A node alert handed to its Pulse agent closes without recovering, so the
|
||||
// assistant must see that, and where the condition went, instead of a plain
|
||||
// resolve it would read as the node being healthy again.
|
||||
func TestExecuteListResolvedAlertsReportsHandover(t *testing.T) {
|
||||
alertProv := &mockAlertProvider{}
|
||||
executor := NewPulseToolExecutor(ExecutorConfig{})
|
||||
executor.alertProvider = alertProv
|
||||
now := time.Now()
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
alertProv.On("GetRecentlyResolved", 24*60).Return([]models.ResolvedAlert{{
|
||||
Alert: models.Alert{
|
||||
ID: "pve1-memory",
|
||||
Type: "memory",
|
||||
Level: "warning",
|
||||
ResourceID: "pve1",
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
StartTime: now.Add(-time.Hour),
|
||||
Resolution: &models.AlertResolution{
|
||||
Reason: "moved_to_agent",
|
||||
SuccessorResourceID: "agent-pve1",
|
||||
SuccessorName: "pve1 (Host Agent)",
|
||||
Summary: summary,
|
||||
},
|
||||
},
|
||||
ResolvedTime: now,
|
||||
}})
|
||||
|
||||
result, _ := executor.executeListResolvedAlerts(context.Background(), map[string]interface{}{})
|
||||
var resp ResolvedAlertsResponse
|
||||
if err := json.Unmarshal([]byte(result.Content[0].Text), &resp); err != nil {
|
||||
t.Fatalf("decode resolved alerts: %v", err)
|
||||
}
|
||||
if len(resp.Alerts) != 1 ||
|
||||
resp.Alerts[0].Resolution != summary ||
|
||||
resp.Alerts[0].SuccessorResourceID != "agent-pve1" {
|
||||
t.Fatalf("resolved alerts = %+v, want the handover and its successor", resp.Alerts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteListAlertsAndFindings(t *testing.T) {
|
||||
|
||||
@@ -98,6 +98,12 @@ func (m *Manager) addRecentlyResolvedWithPrimaryLock(resolved *ResolvedAlert) {
|
||||
|
||||
// clearAlert removes an alert if it exists.
|
||||
func (m *Manager) clearAlert(alertID string) {
|
||||
m.clearAlertWithResolution(alertID, nil)
|
||||
}
|
||||
|
||||
// clearAlertWithResolution removes an alert if it exists. A non-nil
|
||||
// resolution marks the close as something other than a recovery.
|
||||
func (m *Manager) clearAlertWithResolution(alertID string, resolution *AlertResolution) {
|
||||
m.mu.Lock()
|
||||
alert, exists := m.getActiveAlertNoLock(alertID)
|
||||
if exists {
|
||||
@@ -111,6 +117,7 @@ func (m *Manager) clearAlert(alertID string) {
|
||||
}
|
||||
|
||||
publicID := effectiveAlertID(alert, alertID)
|
||||
alert.Resolution = resolution.Clone()
|
||||
resolvedAlert := m.newResolvedAlert(alert, time.Now(), nil)
|
||||
|
||||
m.addRecentlyResolvedUnlocked(resolvedAlert)
|
||||
|
||||
@@ -430,7 +430,13 @@ func (m *Manager) safeCallResolvedAlertCallback(resolved *ResolvedAlert, fallbac
|
||||
}
|
||||
// Record the resolution before the callback guard: this funnel is the
|
||||
// resolve seam for every lifecycle path, with or without subscribers.
|
||||
m.recordAlertEvent(eventlog.TypeResolved, alert, fallbackID, "", "Alert resolved.", nil)
|
||||
eventReason, eventMessage := "", "Alert resolved."
|
||||
if alert != nil {
|
||||
if summary := alert.Resolution.Summary(); summary != "" {
|
||||
eventReason, eventMessage = string(alert.Resolution.Reason), summary
|
||||
}
|
||||
}
|
||||
m.recordAlertEvent(eventlog.TypeResolved, alert, fallbackID, eventReason, eventMessage, nil)
|
||||
|
||||
callbacks := m.getResolvedCallbacks()
|
||||
resolvedCallback := m.callbacks.resolvedAlertCallback()
|
||||
|
||||
@@ -76,6 +76,30 @@ func formatMetricValue(value float64, suffix string) string {
|
||||
return fmt.Sprintf("%.1f%s", value, suffix)
|
||||
}
|
||||
|
||||
// releaseCanonicalMetricAlert stops evaluating a metric spec the way a
|
||||
// disabled threshold does: any pending run is dropped and any open alert is
|
||||
// resolved. A non-nil resolution says why the close was not a recovery, and
|
||||
// travels with the resolved alert to every consumer.
|
||||
func (m *Manager) releaseCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec, resourceName, node, instance, resourceType string, value float64, resolution *AlertResolution) {
|
||||
if spec.MetricThreshold == nil {
|
||||
return
|
||||
}
|
||||
storageKey := canonicalTrackingKeyForSpec(spec, spec.ID)
|
||||
m.mu.Lock()
|
||||
m.core.ApplyMetric(reducer.MetricSignal{
|
||||
ResourceID: spec.ResourceID,
|
||||
Key: spec.ID,
|
||||
Metric: spec.MetricThreshold.Metric,
|
||||
Value: value,
|
||||
ObservedAt: time.Now(),
|
||||
}, reducer.MetricRule{})
|
||||
m.mu.Unlock()
|
||||
// A guest that moved nodes may hold this alert under its old node-scoped
|
||||
// identity; re-home it first so the clear below can resolve it.
|
||||
m.rehomeStrandedGuestAlert(storageKey, spec.ID, string(spec.Kind), spec.ResourceID, resourceName, node, instance, resourceType)
|
||||
m.clearAlertWithResolution(storageKey, resolution)
|
||||
}
|
||||
|
||||
func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec, resourceName, node, instance, resourceType string, value float64, threshold *HysteresisThreshold, opts *metricOptions) {
|
||||
if spec.MetricThreshold == nil {
|
||||
return
|
||||
@@ -86,19 +110,7 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
|
||||
trackingKey := storageKey
|
||||
metricType := spec.MetricThreshold.Metric
|
||||
if spec.Disabled || spec.MetricThreshold.Trigger <= 0 {
|
||||
m.mu.Lock()
|
||||
m.core.ApplyMetric(reducer.MetricSignal{
|
||||
ResourceID: spec.ResourceID,
|
||||
Key: spec.ID,
|
||||
Metric: metricType,
|
||||
Value: value,
|
||||
ObservedAt: time.Now(),
|
||||
}, reducer.MetricRule{})
|
||||
m.mu.Unlock()
|
||||
// A guest that moved nodes may hold this alert under its old node-scoped
|
||||
// identity; re-home it first so the clear below can resolve it.
|
||||
m.rehomeStrandedGuestAlert(storageKey, spec.ID, string(spec.Kind), spec.ResourceID, resourceName, node, instance, resourceType)
|
||||
m.clearAlert(storageKey)
|
||||
m.releaseCanonicalMetricAlert(spec, resourceName, node, instance, resourceType, value, nil)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -330,6 +342,15 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
|
||||
return
|
||||
}
|
||||
|
||||
// A hold keeps the last breach's value and message, but what the
|
||||
// alert is about is current: one restored from an older checkpoint,
|
||||
// or raised before its producer named a child type or platform,
|
||||
// must not link to the wrong page until it re-fires or clears.
|
||||
if refreshMetricAlertClassification(existingAlert, alertMetadata) {
|
||||
m.setActiveRecoveryAlert(existingAlert, storageKey)
|
||||
m.saveActiveAlertsAsync("canonical metric classification")
|
||||
}
|
||||
|
||||
if !triggered && primary == "" {
|
||||
// Hysteresis latch: below trigger but above recovery — hold
|
||||
// without refreshing, as the pre-cutover engine did.
|
||||
@@ -424,3 +445,31 @@ func alertspecsMetricTriggered(spec *alertspecs.MetricThresholdSpec, observed fl
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// metricAlertClassificationKeys name what a metric alert is about and which
|
||||
// platform owns it. They describe the resource, not the reading.
|
||||
var metricAlertClassificationKeys = []string{"resourceType", alertPlatformTypeKey}
|
||||
|
||||
// refreshMetricAlertClassification copies the producer's current
|
||||
// classification onto an open alert and reports whether anything changed.
|
||||
func refreshMetricAlertClassification(alert *Alert, metadata map[string]interface{}) bool {
|
||||
if alert == nil {
|
||||
return false
|
||||
}
|
||||
changed := false
|
||||
for _, key := range metricAlertClassificationKeys {
|
||||
value, _ := metadata[key].(string)
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
if current, _ := alert.Metadata[key].(string); current == value {
|
||||
continue
|
||||
}
|
||||
if alert.Metadata == nil {
|
||||
alert.Metadata = make(map[string]interface{}, len(metricAlertClassificationKeys))
|
||||
}
|
||||
alert.Metadata[key] = value
|
||||
changed = true
|
||||
}
|
||||
return changed
|
||||
}
|
||||
|
||||
@@ -57,6 +57,13 @@ func CanonicalResourceTypeKeys(resourceType string) []string {
|
||||
keys = addUnique(keys, "agent-disk")
|
||||
keys = addUnique(keys, "agent")
|
||||
keys = addUnique(keys, "storage")
|
||||
case "agent-storage":
|
||||
keys = addUnique(keys, "agent-storage")
|
||||
keys = addUnique(keys, "agent")
|
||||
keys = addUnique(keys, "storage")
|
||||
case "agent-sensor":
|
||||
keys = addUnique(keys, "agent-sensor")
|
||||
keys = addUnique(keys, "agent")
|
||||
case "pbs":
|
||||
keys = addUnique(keys, "pbs")
|
||||
keys = addUnique(keys, "node")
|
||||
|
||||
@@ -431,6 +431,16 @@ func (m *Manager) reevaluateActiveAlertsLocked() {
|
||||
continue
|
||||
}
|
||||
threshold = getThresholdForMetric(thresholds, metricType)
|
||||
// Agent policy is the only policy for agent alerts. Falling
|
||||
// through would let the storage or guest switches resolve a
|
||||
// SMART, RAID or sensor alert that CheckHost re-raises on the
|
||||
// next report.
|
||||
if threshold == nil {
|
||||
if isMetricThresholdAlertType(metricType) {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if alert.Type == "docker-host-offline" ||
|
||||
@@ -579,12 +589,14 @@ func (m *Manager) reevaluateActiveAlertsLocked() {
|
||||
continue
|
||||
}
|
||||
|
||||
// buildCanonicalMetricSpec keeps a clear level only below the trigger,
|
||||
// and without one the evaluator keeps firing at the trigger itself.
|
||||
// A clear level at or above the trigger must not resolve a reading
|
||||
// the next evaluation raises again.
|
||||
clearThreshold := threshold.Clear
|
||||
if clearThreshold <= 0 {
|
||||
clearThreshold = threshold.Trigger
|
||||
}
|
||||
hasRecoveryBand := clearThreshold > 0 && clearThreshold < threshold.Trigger
|
||||
|
||||
if alert.Value <= clearThreshold {
|
||||
if hasRecoveryBand && alert.Value <= clearThreshold {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
log.Info().
|
||||
Str("alertID", alertID).
|
||||
|
||||
@@ -394,6 +394,7 @@ func (hm *HistoryManager) UpdateAlertOperationalContractForAlert(alert *Alert) {
|
||||
hm.history[i].Alert.LatestTransition = snapshot.LatestTransition
|
||||
hm.history[i].Alert.Transitions = snapshot.Transitions
|
||||
hm.history[i].Alert.Evidence = snapshot.Evidence
|
||||
hm.history[i].Alert.Resolution = snapshot.Resolution
|
||||
// The live alert is authoritative for acknowledgement; a row
|
||||
// that keeps its fire-time ack state shows an acknowledged
|
||||
// alert as unacknowledged for its whole active life.
|
||||
|
||||
@@ -7,7 +7,9 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
// newTestHistoryManager creates a HistoryManager using a temp directory
|
||||
@@ -1058,3 +1060,62 @@ func TestSaveHistoryWithRetry_RestoresBackupOnFailure(t *testing.T) {
|
||||
|
||||
_ = hm
|
||||
}
|
||||
|
||||
// The state snapshot, the websocket and the assistant read recently resolved
|
||||
// alerts through GetRecentlyResolved, so a handover must reach them as one
|
||||
// and not as a recovery.
|
||||
func TestRecentlyResolvedCarriesHandoverResolution(t *testing.T) {
|
||||
manager := newUnifiedEvalParityManager(t)
|
||||
manager.UpdateConfig(AlertConfig{
|
||||
Enabled: true,
|
||||
NodeDefaults: ThresholdConfig{
|
||||
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
|
||||
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
|
||||
},
|
||||
})
|
||||
disableTestTimeThresholds(manager)
|
||||
node := models.Node{
|
||||
ID: "pve-1",
|
||||
Name: "pve-1",
|
||||
Instance: "test",
|
||||
Status: "online",
|
||||
CPU: 0.95,
|
||||
Memory: models.Memory{Total: 8 << 30, Used: 7 << 30, Usage: 95},
|
||||
}
|
||||
manager.CheckNode(node)
|
||||
release := func(metric string, resolution *AlertResolution) {
|
||||
t.Helper()
|
||||
spec, err := buildCanonicalMetricSpec(node.ID, node.Name, unifiedresources.ResourceType("node"), metric, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("build %s spec: %v", metric, err)
|
||||
}
|
||||
manager.releaseCanonicalMetricAlert(spec, node.Name, node.Name, node.Instance, "node", 0, resolution)
|
||||
}
|
||||
release("memory", &AlertResolution{
|
||||
Reason: AlertResolutionMovedToAgent,
|
||||
SuccessorResourceID: "agent-pve-1",
|
||||
SuccessorName: "pve-1 (Host Agent)",
|
||||
})
|
||||
release("cpu", nil)
|
||||
|
||||
byType := map[string]models.ResolvedAlert{}
|
||||
for _, resolved := range manager.GetRecentlyResolved() {
|
||||
byType[resolved.Type] = resolved
|
||||
}
|
||||
moved, ok := byType["memory"]
|
||||
if !ok {
|
||||
t.Fatalf("recently resolved = %+v, want the moved memory alert", byType)
|
||||
}
|
||||
want := models.AlertResolution{
|
||||
Reason: "moved_to_agent",
|
||||
SuccessorResourceID: "agent-pve-1",
|
||||
SuccessorName: "pve-1 (Host Agent)",
|
||||
Summary: "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading.",
|
||||
}
|
||||
if moved.Resolution == nil || *moved.Resolution != want {
|
||||
t.Fatalf("moved resolution = %+v, want %+v", moved.Resolution, want)
|
||||
}
|
||||
if recovered, ok := byType["cpu"]; !ok || recovered.Resolution != nil {
|
||||
t.Fatalf("recovered cpu alert = %+v, want an ordinary resolve with no resolution", recovered)
|
||||
}
|
||||
}
|
||||
|
||||
+250
-54
@@ -28,6 +28,27 @@ func hostResourceID(hostID string) string {
|
||||
return fmt.Sprintf("agent:%s", trimmed)
|
||||
}
|
||||
|
||||
// Every alert CheckHost raises about something other than the machine itself
|
||||
// (a filesystem, disk, array or custom sensor) targets a child resource ID,
|
||||
// and the alert card's monitoring policy writes to that ID. These alerts
|
||||
// name their own type so the card does not describe retiring the machine.
|
||||
// Each type keeps "agent" among its alert policy keys
|
||||
// (config.CanonicalResourceTypeKeys), so agent thresholds and switches still
|
||||
// govern them.
|
||||
const (
|
||||
hostDiskAlertResourceType = "agent-disk" // filesystems, disk temperatures, SMART disks
|
||||
hostStorageAlertResourceType = "agent-storage" // RAID and Unraid arrays
|
||||
hostSensorAlertResourceType = "agent-sensor" // custom sensors
|
||||
)
|
||||
|
||||
// hostChildAlertMetadata clones a host's base alert metadata for an alert
|
||||
// about something the agent reports rather than the machine itself.
|
||||
func hostChildAlertMetadata(base map[string]interface{}, resourceType string) map[string]interface{} {
|
||||
metadata := cloneMetadata(base)
|
||||
metadata["resourceType"] = resourceType
|
||||
return metadata
|
||||
}
|
||||
|
||||
func stripHostResourcePrefix(resourceID string) string {
|
||||
trimmed := strings.TrimSpace(resourceID)
|
||||
trimmed = strings.TrimPrefix(trimmed, "agent:")
|
||||
@@ -136,7 +157,22 @@ func (m *Manager) resolveHostAlertThresholdsNoLock(alert *Alert, resourceID stri
|
||||
linkedContainerID = metadataStringValue(alert.Metadata, "linkedContainerId")
|
||||
}
|
||||
|
||||
return m.resolveHostThresholdsNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)
|
||||
thresholds := m.resolveHostThresholdsNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)
|
||||
// A disk temperature alert is judged against the per-type threshold
|
||||
// CheckHost evaluates for that disk, so a config save does not resolve an
|
||||
// alert the next report would raise again.
|
||||
if alert != nil && alert.Type == "diskTemperature" {
|
||||
override, exists := m.hostThresholdOverrideNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)
|
||||
overridden := exists && override.DiskTemperature != nil
|
||||
if diskType, known := alert.Metadata["diskType"].(string); known || overridden {
|
||||
thresholds.DiskTemperature = m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, overridden, diskType)
|
||||
} else {
|
||||
// Alerts persisted before CheckHost recorded diskType carry no
|
||||
// disk type until their next firing evaluation.
|
||||
thresholds.DiskTemperature = m.lowestHostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature)
|
||||
}
|
||||
}
|
||||
return thresholds
|
||||
}
|
||||
|
||||
func sanitizeHostComponent(value string) string {
|
||||
@@ -220,10 +256,6 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
return
|
||||
}
|
||||
|
||||
// Record the Proxmox node this agent is linked to, so the node releases the
|
||||
// usage alerts the agent now owns instead of alerting twice for one machine.
|
||||
m.registerHostAgentNodeLink(host)
|
||||
|
||||
// Cache display name so host alerts show the user-configured name.
|
||||
m.UpdateNodeDisplayName("", host.Hostname, host.DisplayName)
|
||||
|
||||
@@ -242,11 +274,15 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
}
|
||||
m.mu.RUnlock()
|
||||
|
||||
// While this agent evaluates nothing, its linked node keeps its own usage
|
||||
// alerts; the link is registered below once the evaluated metrics are known.
|
||||
if !alertsEnabled {
|
||||
m.unregisterHostAgentNodeLink(host.ID)
|
||||
return
|
||||
}
|
||||
|
||||
if disableAllAgents {
|
||||
m.unregisterHostAgentNodeLink(host.ID)
|
||||
// Clear any existing host alerts when all host alerts are disabled
|
||||
m.clearHostMetricAlerts(host.ID)
|
||||
m.clearHostDiskAlerts(host.ID)
|
||||
@@ -257,6 +293,7 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
}
|
||||
|
||||
if thresholds.Disabled {
|
||||
m.unregisterHostAgentNodeLink(host.ID)
|
||||
m.clearHostMetricAlerts(host.ID)
|
||||
m.clearHostDiskAlerts(host.ID)
|
||||
m.clearHostRAIDAlerts(host.ID)
|
||||
@@ -271,15 +308,16 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
instanceName := hostInstanceName(host)
|
||||
|
||||
baseMetadata := map[string]interface{}{
|
||||
"resourceType": "agent",
|
||||
"hostId": host.ID,
|
||||
"hostname": host.Hostname,
|
||||
"displayName": host.DisplayName,
|
||||
"platform": host.Platform,
|
||||
"osName": host.OSName,
|
||||
"osVersion": host.OSVersion,
|
||||
"agentVersion": host.AgentVersion,
|
||||
"architecture": host.Architecture,
|
||||
"resourceType": "agent",
|
||||
alertPlatformTypeKey: string(unifiedresources.SourceAgent),
|
||||
"hostId": host.ID,
|
||||
"hostname": host.Hostname,
|
||||
"displayName": host.DisplayName,
|
||||
"platform": host.Platform,
|
||||
"osName": host.OSName,
|
||||
"osVersion": host.OSVersion,
|
||||
"agentVersion": host.AgentVersion,
|
||||
"architecture": host.Architecture,
|
||||
}
|
||||
if linkedNodeID := strings.TrimSpace(host.LinkedNodeID); linkedNodeID != "" {
|
||||
baseMetadata["linkedNodeId"] = linkedNodeID
|
||||
@@ -340,24 +378,24 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
}
|
||||
|
||||
if thresholds.DiskTemperature != nil && thresholds.DiskTemperature.Trigger > 0 {
|
||||
// An empty SMART list means collection failed or is unsupported for
|
||||
// this report, not that every disk left, so existing alerts are held.
|
||||
if len(host.Sensors.SMART) > 0 {
|
||||
seenDiskTemps := make(map[string]struct{}, len(host.Sensors.SMART))
|
||||
for _, disk := range host.Sensors.SMART {
|
||||
// A listed disk in standby, or without a temperature after a
|
||||
// failed probe, is still present. Its alert holds until a fresh
|
||||
// reading resolves it instead of clearing and re-raising.
|
||||
tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)
|
||||
seenDiskTemps[tempResourceID] = struct{}{}
|
||||
if disk.Temperature > 0 && !disk.Standby {
|
||||
effectiveTempThreshold := thresholds.DiskTemperature
|
||||
if diskType := strings.ToLower(strings.TrimSpace(disk.Type)); diskType != "" && !diskTempOverridden {
|
||||
m.mu.RLock()
|
||||
if th, ok := m.config.DiskTempByType[diskType]; ok {
|
||||
t := th
|
||||
effectiveTempThreshold = &t
|
||||
}
|
||||
m.mu.RUnlock()
|
||||
}
|
||||
m.mu.RLock()
|
||||
effectiveTempThreshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.Type)
|
||||
m.mu.RUnlock()
|
||||
|
||||
// Use specific resource ID for the disk: hostID/disk-temp:device
|
||||
tempResourceID := fmt.Sprintf("%s/disk_temp:%s", hostResourceID(host.ID), sanitizeHostComponent(disk.Device))
|
||||
tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)
|
||||
|
||||
diskTempMetadata := cloneMetadata(baseMetadata)
|
||||
diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)
|
||||
diskTempMetadata["metric"] = "diskTemperature"
|
||||
diskTempMetadata["device"] = disk.Device
|
||||
diskTempMetadata["temperature"] = disk.Temperature
|
||||
@@ -377,15 +415,24 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})
|
||||
}
|
||||
}
|
||||
// A disk missing from consecutive non-empty reports was removed,
|
||||
// replaced or renamed, so no later reading will resolve its alert.
|
||||
m.cleanupHostDiskTemperatureAlerts(host.ID, seenDiskTemps)
|
||||
}
|
||||
} else {
|
||||
// We can't easily clear all disk temp alerts without tracking them,
|
||||
// but checkMetric logic handles auto-resolution if value drops.
|
||||
// If feature is disabled, ideally we should clear existing alerts.
|
||||
// For now simple implementation.
|
||||
// Disk temperature alerting is off for this host, so no later reading
|
||||
// will resolve an alert it raised while it was on.
|
||||
m.clearHostDiskTemperatureAlerts(host.ID)
|
||||
}
|
||||
|
||||
seenDisks := make(map[string]struct{}, len(host.Disks))
|
||||
// A linked node's disk metric is this agent's summary filesystem (root when
|
||||
// reported), so the agent owns it only while it evaluates that filesystem.
|
||||
summaryDiskResourceID := ""
|
||||
if summary, ok := models.SummaryDisk(host.Disks); ok {
|
||||
summaryDiskResourceID, _ = hostDiskResourceID(host, summary)
|
||||
}
|
||||
evaluatesSummaryDisk := false
|
||||
if len(host.Sensors.SMART) > 0 {
|
||||
for _, disk := range host.Sensors.SMART {
|
||||
diskResourceID, diskName := hostSMARTDiskResourceID(host, disk)
|
||||
@@ -443,8 +490,9 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
if effectiveDiskThreshold == nil {
|
||||
continue
|
||||
}
|
||||
evaluatesSummaryDisk = evaluatesSummaryDisk || (diskResourceID == summaryDiskResourceID && effectiveDiskThreshold.Trigger > 0)
|
||||
|
||||
diskMetadata := cloneMetadata(baseMetadata)
|
||||
diskMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)
|
||||
diskMetadata["metric"] = "disk"
|
||||
diskMetadata["mountpoint"] = disk.Mountpoint
|
||||
diskMetadata["device"] = disk.Device
|
||||
@@ -490,6 +538,20 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
|
||||
m.cleanupHostDiskAlerts(host, seenDisks)
|
||||
|
||||
// The linked node releases exactly the usage metrics this agent is set up
|
||||
// to evaluate and keeps the rest, so deduplication never leaves the machine
|
||||
// unmonitored. Ownership follows configuration, not one report's data: a
|
||||
// missing memory reading keeps the agent's alert open rather than handing
|
||||
// memory back to the node for a cycle.
|
||||
m.registerHostAgentNodeLink(hostAgentNodeLink{
|
||||
agentID: host.ID,
|
||||
agentName: resourceName,
|
||||
nodeID: host.LinkedNodeID,
|
||||
cpu: thresholds.CPU != nil && thresholds.CPU.Trigger > 0,
|
||||
memory: thresholds.Memory != nil && thresholds.Memory.Trigger > 0,
|
||||
disk: evaluatesSummaryDisk,
|
||||
})
|
||||
|
||||
if host.Unraid != nil {
|
||||
m.syncHostUnraidStorageAlert(host, nodeName, instanceName, resourceName, baseMetadata)
|
||||
} else {
|
||||
@@ -515,7 +577,7 @@ func (m *Manager) CheckHost(host models.Host) {
|
||||
raidName := fmt.Sprintf("%s - %s (%s)", resourceName, array.Device, array.Level)
|
||||
raidSpecResourceID := fmt.Sprintf("%s/raid:%s", hostResourceID(host.ID), sanitizeRAIDDevice(array.Device))
|
||||
|
||||
raidMetadata := cloneMetadata(baseMetadata)
|
||||
raidMetadata := hostChildAlertMetadata(baseMetadata, hostStorageAlertResourceType)
|
||||
raidMetadata["metric"] = "raid"
|
||||
raidMetadata["raidDevice"] = array.Device
|
||||
raidMetadata["raidLevel"] = array.Level
|
||||
@@ -626,6 +688,9 @@ func (m *Manager) HandleHostRemoved(host models.Host) {
|
||||
m.clearHostRAIDAlerts(host.ID)
|
||||
m.clearHostUnraidAlerts(host.ID)
|
||||
m.clearHostCustomSensorAlerts(host.ID)
|
||||
// No later report will close the host's pending disk temperature runs or
|
||||
// reset its disk absence counts.
|
||||
m.clearHostDiskTemperatureAlerts(host.ID)
|
||||
}
|
||||
|
||||
// HandleHostTelemetryExpired re-evaluates transient storage-operation evidence
|
||||
@@ -667,15 +732,16 @@ func (m *Manager) HandleHostTelemetryExpired(host models.Host) {
|
||||
m.clearHostUnraidAlerts(host.ID)
|
||||
} else {
|
||||
baseMetadata := map[string]interface{}{
|
||||
"resourceType": "agent",
|
||||
"hostId": host.ID,
|
||||
"hostname": host.Hostname,
|
||||
"displayName": host.DisplayName,
|
||||
"platform": host.Platform,
|
||||
"osName": host.OSName,
|
||||
"osVersion": host.OSVersion,
|
||||
"agentVersion": host.AgentVersion,
|
||||
"architecture": host.Architecture,
|
||||
"resourceType": "agent",
|
||||
alertPlatformTypeKey: string(unifiedresources.SourceAgent),
|
||||
"hostId": host.ID,
|
||||
"hostname": host.Hostname,
|
||||
"displayName": host.DisplayName,
|
||||
"platform": host.Platform,
|
||||
"osName": host.OSName,
|
||||
"osVersion": host.OSVersion,
|
||||
"agentVersion": host.AgentVersion,
|
||||
"architecture": host.Architecture,
|
||||
}
|
||||
if linkedNodeID := strings.TrimSpace(host.LinkedNodeID); linkedNodeID != "" {
|
||||
baseMetadata["linkedNodeId"] = linkedNodeID
|
||||
@@ -787,16 +853,17 @@ func (m *Manager) HandleHostOfflineWithCorrelation(host models.Host, correlation
|
||||
Message: fmt.Sprintf("Host '%s' is offline", resourceName),
|
||||
Correlation: correlation,
|
||||
Metadata: map[string]interface{}{
|
||||
"resourceType": "agent",
|
||||
"hostId": host.ID,
|
||||
"hostname": host.Hostname,
|
||||
"displayName": host.DisplayName,
|
||||
"platform": host.Platform,
|
||||
"osName": host.OSName,
|
||||
"osVersion": host.OSVersion,
|
||||
"linkedNodeId": strings.TrimSpace(host.LinkedNodeID),
|
||||
"linkedVmId": strings.TrimSpace(host.LinkedVMID),
|
||||
"linkedContainerId": strings.TrimSpace(host.LinkedContainerID),
|
||||
"resourceType": "agent",
|
||||
alertPlatformTypeKey: string(unifiedresources.SourceAgent),
|
||||
"hostId": host.ID,
|
||||
"hostname": host.Hostname,
|
||||
"displayName": host.DisplayName,
|
||||
"platform": host.Platform,
|
||||
"osName": host.OSName,
|
||||
"osVersion": host.OSVersion,
|
||||
"linkedNodeId": strings.TrimSpace(host.LinkedNodeID),
|
||||
"linkedVmId": strings.TrimSpace(host.LinkedVMID),
|
||||
"linkedContainerId": strings.TrimSpace(host.LinkedContainerID),
|
||||
},
|
||||
AddToRecent: true,
|
||||
AddToHistory: true,
|
||||
@@ -828,6 +895,7 @@ func (m *Manager) HandleHostOfflineWithCorrelation(host models.Host, correlation
|
||||
|
||||
diskResourcePrefixes := []string{
|
||||
fmt.Sprintf("%s/disk:", resourceKey),
|
||||
hostDiskTemperatureResourcePrefix(host.ID),
|
||||
}
|
||||
raidAlertPrefix := fmt.Sprintf("host-%s-raid-", host.ID)
|
||||
var alertsToClear []string
|
||||
@@ -884,6 +952,7 @@ func (m *Manager) clearHostDiskAlerts(hostID string) {
|
||||
|
||||
prefixes := []string{
|
||||
fmt.Sprintf("%s/disk:", hostResourceID(hostID)),
|
||||
hostDiskTemperatureResourcePrefix(hostID),
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
@@ -908,6 +977,133 @@ func (m *Manager) clearHostDiskAlerts(hostID string) {
|
||||
}
|
||||
}
|
||||
|
||||
// hostDiskTemperatureResourcePrefix is the resource ID prefix of every SMART
|
||||
// disk temperature alert CheckHost raises for a host.
|
||||
func hostDiskTemperatureResourcePrefix(hostID string) string {
|
||||
return hostResourceID(hostID) + "/disk_temp:"
|
||||
}
|
||||
|
||||
// hostDiskTemperatureResourceID is the resource ID of the SMART disk
|
||||
// temperature alert for one device on a host.
|
||||
func hostDiskTemperatureResourceID(hostID, device string) string {
|
||||
return hostDiskTemperatureResourcePrefix(hostID) + sanitizeHostComponent(device)
|
||||
}
|
||||
|
||||
// hostDiskTemperatureThresholdNoLock returns the threshold for one SMART disk
|
||||
// given the host's resolved disk temperature threshold. An enabled threshold
|
||||
// gives way to the disk type's DiskTempByType entry unless an explicit host or
|
||||
// linked-resource override set it. Callers must hold m.mu.
|
||||
func (m *Manager) hostDiskTemperatureThresholdNoLock(hostThreshold *HysteresisThreshold, overridden bool, diskType string) *HysteresisThreshold {
|
||||
if hostThreshold == nil || hostThreshold.Trigger <= 0 || overridden {
|
||||
return hostThreshold
|
||||
}
|
||||
if diskType = strings.ToLower(strings.TrimSpace(diskType)); diskType != "" {
|
||||
if th, ok := m.config.DiskTempByType[diskType]; ok {
|
||||
return &th
|
||||
}
|
||||
}
|
||||
return hostThreshold
|
||||
}
|
||||
|
||||
// lowestHostDiskTemperatureThresholdNoLock returns the lowest enabled
|
||||
// threshold any SMART disk of the host can be evaluated against. An alert
|
||||
// whose disk type is unknown is judged against it, so a config save never
|
||||
// resolves an alert its disk type would still fire. Callers must hold m.mu.
|
||||
func (m *Manager) lowestHostDiskTemperatureThresholdNoLock(hostThreshold *HysteresisThreshold) *HysteresisThreshold {
|
||||
if hostThreshold == nil || hostThreshold.Trigger <= 0 {
|
||||
return hostThreshold
|
||||
}
|
||||
lowest := hostThreshold
|
||||
for _, th := range m.config.DiskTempByType {
|
||||
if th.Trigger > 0 && th.Trigger < lowest.Trigger {
|
||||
t := th
|
||||
lowest = &t
|
||||
}
|
||||
}
|
||||
return lowest
|
||||
}
|
||||
|
||||
// clearHostDiskTemperatureAlerts resolves every SMART disk temperature alert
|
||||
// for a host.
|
||||
func (m *Manager) clearHostDiskTemperatureAlerts(hostID string) {
|
||||
m.cleanupHostDiskTemperatureAlerts(hostID, nil)
|
||||
}
|
||||
|
||||
// hostDiskTemperatureAbsenceConfirmations is how many consecutive non-empty
|
||||
// SMART reports must omit a disk before its temperature alert clears. The
|
||||
// Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk
|
||||
// whose probe fails, so one omission is not evidence the disk left.
|
||||
const hostDiskTemperatureAbsenceConfirmations = 3
|
||||
|
||||
// cleanupHostDiskTemperatureAlerts resolves a host's SMART disk temperature
|
||||
// alerts, and drops their pending threshold runs, once their resource ID has
|
||||
// been missing from seen for hostDiskTemperatureAbsenceConfirmations calls in
|
||||
// a row. A nil seen resolves all of them at once.
|
||||
func (m *Manager) cleanupHostDiskTemperatureAlerts(hostID string, seen map[string]struct{}) {
|
||||
if hostID == "" {
|
||||
return
|
||||
}
|
||||
|
||||
resourcePrefix := hostDiskTemperatureResourcePrefix(hostID)
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
// Absent resources map to the storage keys of their active alerts; a
|
||||
// resource with only a pending run maps to none.
|
||||
absent := make(map[string][]string)
|
||||
for storageKey, alert := range m.activeAlerts {
|
||||
// Only CheckHost raises alerts under this prefix. Matching on it
|
||||
// alone also catches alerts raised before the type became
|
||||
// diskTemperature ("disk_temperature", January 2026).
|
||||
if alert == nil || !strings.HasPrefix(alert.ResourceID, resourcePrefix) {
|
||||
continue
|
||||
}
|
||||
if _, exists := seen[alert.ResourceID]; exists {
|
||||
continue
|
||||
}
|
||||
absent[alert.ResourceID] = append(absent[alert.ResourceID], storageKey)
|
||||
}
|
||||
for _, resourceID := range m.core.PendingResourceIDs() {
|
||||
if !strings.HasPrefix(resourceID, resourcePrefix) {
|
||||
continue
|
||||
}
|
||||
if _, exists := seen[resourceID]; exists {
|
||||
continue
|
||||
}
|
||||
if _, exists := absent[resourceID]; !exists {
|
||||
absent[resourceID] = nil
|
||||
}
|
||||
}
|
||||
|
||||
if m.hostDiskTempAbsences == nil {
|
||||
m.hostDiskTempAbsences = make(map[string]int)
|
||||
}
|
||||
// A disk seen again, or whose state is gone, restarts its count.
|
||||
for resourceID := range m.hostDiskTempAbsences {
|
||||
if !strings.HasPrefix(resourceID, resourcePrefix) {
|
||||
continue
|
||||
}
|
||||
if _, exists := absent[resourceID]; !exists {
|
||||
delete(m.hostDiskTempAbsences, resourceID)
|
||||
}
|
||||
}
|
||||
for resourceID, storageKeys := range absent {
|
||||
if seen != nil {
|
||||
m.hostDiskTempAbsences[resourceID]++
|
||||
if m.hostDiskTempAbsences[resourceID] < hostDiskTemperatureAbsenceConfirmations {
|
||||
continue
|
||||
}
|
||||
}
|
||||
delete(m.hostDiskTempAbsences, resourceID)
|
||||
// A departed disk never sends the reading that would close its run.
|
||||
m.core.DropPendingForResource(resourceID)
|
||||
for _, storageKey := range storageKeys {
|
||||
m.clearAlertNoLock(storageKey)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var customSensorAssessmentCodes = []string{
|
||||
"custom_sensor_warning",
|
||||
"custom_sensor_critical",
|
||||
@@ -959,7 +1155,7 @@ func (m *Manager) syncHostCustomSensorAlerts(host models.Host, nodeName, instanc
|
||||
}
|
||||
}
|
||||
|
||||
metadata := cloneMetadata(baseMetadata)
|
||||
metadata := hostChildAlertMetadata(baseMetadata, hostSensorAlertResourceType)
|
||||
metadata["metric"] = "customSensor"
|
||||
metadata["customSensorId"] = metric.ID
|
||||
metadata["customSensorName"] = metric.Name
|
||||
@@ -1387,7 +1583,7 @@ func (m *Manager) syncHostSMARTDiskAlert(host models.Host, disk models.HostDiskS
|
||||
reasonCodes := storageHealthReasonCodes(reasons)
|
||||
reasonSummaries := storageHealthReasonSummaries(reasons)
|
||||
|
||||
metadata := cloneMetadata(baseMetadata)
|
||||
metadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)
|
||||
metadata["metric"] = alertType
|
||||
metadata["device"] = disk.Device
|
||||
metadata["model"] = disk.Model
|
||||
@@ -1468,7 +1664,7 @@ func (m *Manager) syncHostUnraidStorageAlert(host models.Host, nodeName, instanc
|
||||
reasonCodes := storageHealthReasonCodes(reasons)
|
||||
reasonSummaries := storageHealthReasonSummaries(reasons)
|
||||
|
||||
metadata := cloneMetadata(baseMetadata)
|
||||
metadata := hostChildAlertMetadata(baseMetadata, hostStorageAlertResourceType)
|
||||
metadata["metric"] = "storageTopology"
|
||||
metadata["storagePlatform"] = "unraid"
|
||||
metadata["storageTopology"] = "array"
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
)
|
||||
|
||||
func TestHostAgentDeduplicatesNodeAlerts(t *testing.T) {
|
||||
agent := models.Host{ID: "host-pi", Hostname: "pi", LinkedNodeID: "node/pi"}
|
||||
agent := hostAgentNodeLink{agentID: "host-pi", nodeID: "node/pi", cpu: true, memory: true, disk: true}
|
||||
|
||||
// Test 1: Without a linked host agent, node metrics ARE checked
|
||||
t.Run("node_metrics_checked_without_agent", func(t *testing.T) {
|
||||
@@ -81,7 +81,7 @@ func TestHostAgentDeduplicatesNodeAlerts(t *testing.T) {
|
||||
|
||||
// Register and then unregister
|
||||
m.registerHostAgentNodeLink(agent)
|
||||
m.unregisterHostAgentNodeLink(agent.ID)
|
||||
m.unregisterHostAgentNodeLink(agent.agentID)
|
||||
|
||||
// Verify host agent is NOT linked
|
||||
if m.hasHostAgentForNode("node/pi") {
|
||||
@@ -112,8 +112,9 @@ func TestHostAgentDeduplicatesNodeAlerts(t *testing.T) {
|
||||
// instances can both have a node named "pve", and an agent can report an FQDN.
|
||||
func TestHostAgentDeduplicationFollowsNodeLink(t *testing.T) {
|
||||
m := NewManager()
|
||||
m.config.Enabled = true
|
||||
|
||||
m.registerHostAgentNodeLink(models.Host{ID: "host-a", Hostname: "pve.example.com", LinkedNodeID: "site-a-pve"})
|
||||
m.CheckHost(models.Host{ID: "host-a", Hostname: "pve.example.com", LinkedNodeID: "site-a-pve", CPUUsage: 10})
|
||||
if !m.hasHostAgentForNode("site-a-pve") {
|
||||
t.Error("Expected the linked node to dedup even though the agent hostname is an FQDN")
|
||||
}
|
||||
@@ -121,13 +122,13 @@ func TestHostAgentDeduplicationFollowsNodeLink(t *testing.T) {
|
||||
t.Error("Expected a same-named node in another instance to keep its own alerts")
|
||||
}
|
||||
|
||||
m.registerHostAgentNodeLink(models.Host{ID: "host-b", Hostname: "pve"})
|
||||
m.CheckHost(models.Host{ID: "host-b", Hostname: "pve", CPUUsage: 10})
|
||||
if m.hasHostAgentForNode("site-b-pve") {
|
||||
t.Error("Expected an unlinked agent with a matching hostname not to suppress node alerts")
|
||||
}
|
||||
|
||||
// An operator unlinking the agent hands the node's alerts back.
|
||||
m.registerHostAgentNodeLink(models.Host{ID: "host-a", Hostname: "pve.example.com"})
|
||||
m.CheckHost(models.Host{ID: "host-a", Hostname: "pve.example.com", CPUUsage: 10})
|
||||
if m.hasHostAgentForNode("site-a-pve") {
|
||||
t.Error("Expected an unlinked agent to stop deduplicating its former node")
|
||||
}
|
||||
@@ -168,7 +169,7 @@ func TestHandleHostOfflineUnregistersNodeLink(t *testing.T) {
|
||||
}
|
||||
|
||||
// Register the node link
|
||||
m.registerHostAgentNodeLink(host)
|
||||
m.registerHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: host.LinkedNodeID, cpu: true})
|
||||
|
||||
if !m.hasHostAgentForNode("pve-offlinehost") {
|
||||
t.Error("Expected host agent registered")
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -199,6 +200,352 @@ func TestHandleHostTelemetryExpiredPreservesStaticStorageRisk(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
func TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
turnOff func(m *Manager, hostID string)
|
||||
}{
|
||||
{
|
||||
name: "agent default disabled",
|
||||
turnOff: func(m *Manager, hostID string) {
|
||||
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "host override disables disk temperature",
|
||||
turnOff: func(m *Manager, hostID string) {
|
||||
m.config.Overrides[hostID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0, Clear: 0}}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "host override disables the agent",
|
||||
turnOff: func(m *Manager, hostID string) {
|
||||
m.config.Overrides[hostID] = ThresholdConfig{Disabled: true}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "all agent alerts disabled",
|
||||
turnOff: func(m *Manager, hostID string) {
|
||||
m.config.DisableAllAgents = true
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
host := hostWithSMARTDiskTemp("host-temp-off", "sata", 60)
|
||||
alertID := hostDiskTempAlertID(host)
|
||||
|
||||
m.CheckHost(host)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
t.Fatalf("expected disk temperature alert at 60C (sata trigger 55), active: %v", alertKeys(m))
|
||||
}
|
||||
if alert.Type != "diskTemperature" || !strings.HasPrefix(alert.ResourceID, hostDiskTemperatureResourcePrefix(host.ID)) {
|
||||
t.Fatalf("disk temperature alert type/resource = %q/%q", alert.Type, alert.ResourceID)
|
||||
}
|
||||
|
||||
// Change config directly so only CheckHost can resolve the alert.
|
||||
m.mu.Lock()
|
||||
tc.turnOff(m, host.ID)
|
||||
m.mu.Unlock()
|
||||
|
||||
m.CheckHost(host)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); exists {
|
||||
t.Fatalf("disk temperature alert stayed active after its threshold was turned off, active: %v", alertKeys(m))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHostDiskTemperatureAlertsClearWhenAgentLeaves(t *testing.T) {
|
||||
t.Run("agent removed", func(t *testing.T) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
host := hostWithSMARTDiskTemp("host-temp-removed", "sata", 60)
|
||||
m.CheckHost(host)
|
||||
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); !exists {
|
||||
t.Fatalf("expected disk temperature alert, active: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
m.HandleHostRemoved(host)
|
||||
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); exists {
|
||||
t.Fatalf("disk temperature alert outlived its removed agent, active: %v", alertKeys(m))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("agent confirmed offline", func(t *testing.T) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
host := hostWithSMARTDiskTemp("host-temp-offline", "sata", 60)
|
||||
m.CheckHost(host)
|
||||
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); !exists {
|
||||
t.Fatalf("expected disk temperature alert, active: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
m.HandleHostOffline(host)
|
||||
}
|
||||
if _, exists := testLookupActiveAlert(t, m, canonicalConnectivityStateID(hostResourceID(host.ID))); !exists {
|
||||
t.Fatalf("expected confirmed offline alert, active: %v", alertKeys(m))
|
||||
}
|
||||
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); exists {
|
||||
t.Fatalf("disk temperature alert stayed beside the offline alert, active: %v", alertKeys(m))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestCheckHostClearsDiskTemperatureAlertWhenDiskLeavesSMARTReport(t *testing.T) {
|
||||
const hostID = "host-temp-absent"
|
||||
smartDisk := func(device string, temperature int) models.HostDiskSMART {
|
||||
return models.HostDiskSMART{Device: device, Model: "test-disk", Type: "sata", Temperature: temperature}
|
||||
}
|
||||
alertIDFor := func(device string) string {
|
||||
return canonicalMetricStateID(hostDiskTemperatureResourceID(hostID, device), "diskTemperature")
|
||||
}
|
||||
sdaActive := func(m *Manager) bool {
|
||||
_, exists := testLookupActiveAlert(t, m, alertIDFor("/dev/sda"))
|
||||
return exists
|
||||
}
|
||||
// sda runs hot (sata trigger 55, clear 50); sdb stays cool.
|
||||
hotHost := func(t *testing.T) (*Manager, models.Host) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
host := hostWithSMARTDiskTemp(hostID, "sata", 60)
|
||||
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sda", 60), smartDisk("/dev/sdb", 40)}
|
||||
m.CheckHost(host)
|
||||
if !sdaActive(m) {
|
||||
t.Fatalf("expected sda disk temperature alert at 60C, active: %v", alertKeys(m))
|
||||
}
|
||||
return m, host
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
next []models.HostDiskSMART
|
||||
wantHeld bool
|
||||
wantRaised string
|
||||
}{
|
||||
{name: "disk removed", next: []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}},
|
||||
{name: "device renamed", next: []models.HostDiskSMART{smartDisk("/dev/sdc", 60), smartDisk("/dev/sdb", 40)}, wantRaised: "/dev/sdc"},
|
||||
// No fresh reading says the disk cooled or left, so the alert holds.
|
||||
{name: "SMART collection empty", next: nil, wantHeld: true},
|
||||
{name: "disk in standby", next: []models.HostDiskSMART{{Device: "/dev/sda", Type: "sata", Temperature: 45, Standby: true}, smartDisk("/dev/sdb", 40)}, wantHeld: true},
|
||||
{name: "probe returned no temperature", next: []models.HostDiskSMART{{Device: "/dev/sda", Type: "sata"}, smartDisk("/dev/sdb", 40)}, wantHeld: true},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m, host := hotHost(t)
|
||||
|
||||
host.Sensors.SMART = tc.next
|
||||
for report := 1; report <= hostDiskTemperatureAbsenceConfirmations; report++ {
|
||||
m.CheckHost(host)
|
||||
wantActive := tc.wantHeld || report < hostDiskTemperatureAbsenceConfirmations
|
||||
if active := sdaActive(m); active != wantActive {
|
||||
t.Fatalf("report %d: sda disk temperature alert active = %v, want %v; active: %v", report, active, wantActive, alertKeys(m))
|
||||
}
|
||||
}
|
||||
if tc.wantRaised != "" {
|
||||
if _, exists := testLookupActiveAlert(t, m, alertIDFor(tc.wantRaised)); !exists {
|
||||
t.Fatalf("expected %s disk temperature alert, active: %v", tc.wantRaised, alertKeys(m))
|
||||
}
|
||||
}
|
||||
if !tc.wantHeld {
|
||||
return
|
||||
}
|
||||
|
||||
// A held alert still resolves on the disk's next cool reading.
|
||||
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sda", 45), smartDisk("/dev/sdb", 40)}
|
||||
m.CheckHost(host)
|
||||
if sdaActive(m) {
|
||||
t.Fatalf("held sda disk temperature alert outlived a 45C reading, active: %v", alertKeys(m))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("intermittently dropped disk keeps its alert", func(t *testing.T) {
|
||||
m, host := hotHost(t)
|
||||
present := host.Sensors.SMART
|
||||
dropped := []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}
|
||||
|
||||
// The run of omissions restarts whenever the disk is reported again.
|
||||
for _, smart := range [][]models.HostDiskSMART{dropped, dropped, present, dropped, dropped} {
|
||||
host.Sensors.SMART = smart
|
||||
m.CheckHost(host)
|
||||
if !sdaActive(m) {
|
||||
t.Fatalf("sda disk temperature alert cleared by an omission run shorter than %d reports, active: %v", hostDiskTemperatureAbsenceConfirmations, alertKeys(m))
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// pendingHost leaves a hot sda inside a 300s alert delay, so sda holds a
|
||||
// pending run and no alert.
|
||||
pendingHost := func(t *testing.T) (*Manager, models.Host, func() bool) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
m.mu.Lock()
|
||||
m.config.TimeThresholds = map[string]int{"agent": 300}
|
||||
m.mu.Unlock()
|
||||
sdaResourceID := hostDiskTemperatureResourceID(hostID, "/dev/sda")
|
||||
sdaPending := func() bool {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
for _, resourceID := range m.core.PendingResourceIDs() {
|
||||
if resourceID == sdaResourceID {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
host := hostWithSMARTDiskTemp(hostID, "sata", 60)
|
||||
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sda", 60), smartDisk("/dev/sdb", 40)}
|
||||
m.CheckHost(host)
|
||||
if sdaActive(m) || !sdaPending() {
|
||||
t.Fatalf("expected a pending sda run inside the 300s delay, active: %v", alertKeys(m))
|
||||
}
|
||||
return m, host, sdaPending
|
||||
}
|
||||
|
||||
t.Run("departed disk drops its pending run", func(t *testing.T) {
|
||||
m, host, sdaPending := pendingHost(t)
|
||||
|
||||
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}
|
||||
for report := 1; report <= hostDiskTemperatureAbsenceConfirmations; report++ {
|
||||
m.CheckHost(host)
|
||||
wantPending := report < hostDiskTemperatureAbsenceConfirmations
|
||||
if pending := sdaPending(); pending != wantPending {
|
||||
t.Fatalf("report %d: pending sda run = %v, want %v", report, pending, wantPending)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("threshold off drops pending runs", func(t *testing.T) {
|
||||
m, host, sdaPending := pendingHost(t)
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
m.mu.Unlock()
|
||||
m.CheckHost(host)
|
||||
if sdaPending() {
|
||||
t.Fatal("pending sda run survived its threshold being turned off; re-enabling would inherit its old start time")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("removed host leaves no pending run or absence count", func(t *testing.T) {
|
||||
m, host, sdaPending := pendingHost(t)
|
||||
|
||||
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}
|
||||
m.CheckHost(host)
|
||||
m.HandleHostRemoved(host)
|
||||
if sdaPending() {
|
||||
t.Fatal("pending sda run outlived its removed host")
|
||||
}
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
if len(m.hostDiskTempAbsences) != 0 {
|
||||
t.Fatalf("absence counts outlived their removed host: %v", m.hostDiskTempAbsences)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold(t *testing.T) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
m.mu.Unlock()
|
||||
|
||||
// 72C is above the nvme trigger (70) but below the agent default (75).
|
||||
host := hostWithSMARTDiskTemp("host-temp-save", "nvme", 72)
|
||||
alertID := hostDiskTempAlertID(host)
|
||||
m.CheckHost(host)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
|
||||
t.Fatalf("expected nvme disk temperature alert at 72C, active: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
// An unrelated save keeps the alert CheckHost would raise again.
|
||||
cfg := m.GetConfig()
|
||||
cfg.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 85, Clear: 80}
|
||||
m.UpdateConfig(cfg)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
|
||||
t.Fatalf("unrelated config save resolved a disk temperature alert above its nvme trigger, active: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
// Turning disk temperature off resolves it on save, before any report.
|
||||
cfg = m.GetConfig()
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
m.UpdateConfig(cfg)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); exists {
|
||||
t.Fatalf("disk temperature alert stayed active after the threshold was turned off, active: %v", alertKeys(m))
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType(t *testing.T) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
m.mu.Unlock()
|
||||
|
||||
host := hostWithSMARTDiskTemp("host-temp-legacy", "nvme", 72)
|
||||
alertID := hostDiskTempAlertID(host)
|
||||
m.CheckHost(host)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
t.Fatalf("expected nvme disk temperature alert at 72C, active: %v", alertKeys(m))
|
||||
}
|
||||
// Restored from a release that did not record the disk type.
|
||||
m.mu.Lock()
|
||||
delete(alert.Metadata, "diskType")
|
||||
m.mu.Unlock()
|
||||
|
||||
// 72C is above the nvme trigger (70) but below the agent default (75).
|
||||
cfg := m.GetConfig()
|
||||
cfg.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 85, Clear: 80}
|
||||
m.UpdateConfig(cfg)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
|
||||
t.Fatalf("config save resolved a disk temperature alert of unknown disk type that its nvme trigger still fires, active: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
cfg = m.GetConfig()
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
m.UpdateConfig(cfg)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); exists {
|
||||
t.Fatalf("disk temperature alert of unknown disk type stayed active after the threshold was turned off, active: %v", alertKeys(m))
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand(t *testing.T) {
|
||||
// Normalization keeps a 70/75 entry, and the evaluator treats a clear
|
||||
// level at or above the trigger as no recovery band, firing from 70C.
|
||||
for _, reading := range []int{72, 70} {
|
||||
t.Run(fmt.Sprintf("%dC", reading), func(t *testing.T) {
|
||||
m := configureDiskTempTypeHostManager(t)
|
||||
m.mu.Lock()
|
||||
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 70, Clear: 75}
|
||||
m.mu.Unlock()
|
||||
|
||||
host := hostWithSMARTDiskTemp("host-temp-no-band", "nvme", reading)
|
||||
alertID := hostDiskTempAlertID(host)
|
||||
m.CheckHost(host)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
|
||||
t.Fatalf("expected nvme disk temperature alert at %dC, active: %v", reading, alertKeys(m))
|
||||
}
|
||||
|
||||
cfg := m.GetConfig()
|
||||
cfg.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 85, Clear: 80}
|
||||
m.UpdateConfig(cfg)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
|
||||
t.Fatalf("config save resolved an alert the evaluator keeps firing at %dC, active: %v", reading, alertKeys(m))
|
||||
}
|
||||
|
||||
m.CheckHost(host)
|
||||
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
|
||||
t.Fatalf("expected the next report to keep the alert, active: %v", alertKeys(m))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func hasAlertType(alerts []Alert, alertType string) bool {
|
||||
for _, alert := range alerts {
|
||||
if alert.Type == alertType {
|
||||
@@ -305,3 +652,242 @@ func TestHostCustomSensorEscalationDelivery(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// An agent reports things besides its own machine: filesystems, disks,
|
||||
// arrays and custom sensors. Their alerts target a child resource, and the
|
||||
// alert card's monitoring policy writes to that child, so they must not
|
||||
// carry the machine's "agent" type: the card would offer the machine's
|
||||
// retirement copy ("Agent removal remains available from Machines") for a
|
||||
// policy that never touches the machine. platformType keeps every one of
|
||||
// them linked to the agent's page.
|
||||
func TestHostChildAlertsNameTheirOwnResourceType(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
m.mu.Lock()
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 70}
|
||||
m.config.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 80, Clear: 70}
|
||||
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 50, Clear: 45}
|
||||
m.mu.Unlock()
|
||||
|
||||
pending := int64(2)
|
||||
sensorValue := 23.0
|
||||
linuxHost := models.Host{
|
||||
ID: "child-linux",
|
||||
Hostname: "storage-host",
|
||||
DisplayName: "Storage Host",
|
||||
Platform: "linux",
|
||||
CPUUsage: 95,
|
||||
Disks: []models.Disk{{
|
||||
Mountpoint: "/srv",
|
||||
Device: "/dev/sdc1",
|
||||
Usage: 92,
|
||||
Total: 100,
|
||||
Used: 92,
|
||||
Free: 8,
|
||||
}},
|
||||
Sensors: models.HostSensorSummary{
|
||||
SMART: []models.HostDiskSMART{{
|
||||
Device: "/dev/sda",
|
||||
Model: "IronWolf",
|
||||
Serial: "SERIAL-CHILD-1",
|
||||
Health: "PASSED",
|
||||
Temperature: 55,
|
||||
Attributes: &models.SMARTAttributes{PendingSectors: &pending},
|
||||
}},
|
||||
Custom: []models.HostCustomSensorMetric{{
|
||||
ID: "queue_depth",
|
||||
Name: "Queue depth",
|
||||
Value: &sensorValue,
|
||||
Status: "critical",
|
||||
ObservedAt: time.Now().UTC(),
|
||||
}},
|
||||
},
|
||||
RAID: []models.HostRAIDArray{{
|
||||
Device: "/dev/md0",
|
||||
Level: "raid1",
|
||||
State: "degraded",
|
||||
TotalDevices: 2,
|
||||
ActiveDevices: 1,
|
||||
WorkingDevices: 1,
|
||||
FailedDevices: 1,
|
||||
}},
|
||||
}
|
||||
unraidHost := models.Host{
|
||||
ID: "child-unraid",
|
||||
Hostname: "tower",
|
||||
DisplayName: "Tower",
|
||||
Platform: "unraid",
|
||||
Unraid: &models.HostUnraidStorage{
|
||||
ArrayStarted: true,
|
||||
ArrayState: "STARTED",
|
||||
Disks: []models.HostUnraidDisk{
|
||||
{Name: "disk1", Role: "data", Status: "online", Device: "/dev/sdb"},
|
||||
},
|
||||
},
|
||||
}
|
||||
m.CheckHost(linuxHost)
|
||||
m.CheckHost(unraidHost)
|
||||
|
||||
want := map[string]string{
|
||||
"cpu": "agent",
|
||||
"disk": "agent-disk",
|
||||
"diskTemperature": "agent-disk",
|
||||
"disk-health": "agent-disk",
|
||||
"raid": "agent-storage",
|
||||
"storage-topology": "agent-storage",
|
||||
"custom-sensor": "agent-sensor",
|
||||
}
|
||||
seen := make(map[string]bool, len(want))
|
||||
for _, alert := range m.GetActiveAlerts() {
|
||||
wantType, ok := want[alert.Type]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
seen[alert.Type] = true
|
||||
if got := alert.Metadata["resourceType"]; got != wantType {
|
||||
t.Errorf("%s alert resourceType = %v, want %s", alert.Type, got, wantType)
|
||||
}
|
||||
if got := alert.Metadata["platformType"]; got != "agent" {
|
||||
t.Errorf("%s alert platformType = %v, want agent", alert.Type, got)
|
||||
}
|
||||
hostID, _ := alert.Metadata["hostId"].(string)
|
||||
if wantType != "agent" && alert.ResourceID == hostResourceID(hostID) {
|
||||
t.Errorf("%s alert targets the machine itself (%s), so it is not a child alert", alert.Type, alert.ResourceID)
|
||||
}
|
||||
}
|
||||
for alertType := range want {
|
||||
if !seen[alertType] {
|
||||
t.Errorf("expected an active %s alert", alertType)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Child alerts keep the agent policy path. A configuration save that turns
|
||||
// off storage or guest alerts must leave them alone: CheckHost only consults
|
||||
// the agent switches, so resolving them here would re-raise and re-notify
|
||||
// on the next report.
|
||||
func TestHostChildAlertsIgnoreOtherPlatformSwitchesOnConfigSave(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
m.mu.Lock()
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.mu.Unlock()
|
||||
|
||||
pending := int64(2)
|
||||
sensorValue := 23.0
|
||||
host := models.Host{
|
||||
ID: "child-policy",
|
||||
Hostname: "storage-host",
|
||||
Platform: "linux",
|
||||
Sensors: models.HostSensorSummary{
|
||||
SMART: []models.HostDiskSMART{{
|
||||
Device: "/dev/sda",
|
||||
Serial: "SERIAL-CHILD-2",
|
||||
Health: "PASSED",
|
||||
Attributes: &models.SMARTAttributes{PendingSectors: &pending},
|
||||
}},
|
||||
Custom: []models.HostCustomSensorMetric{{
|
||||
ID: "queue_depth",
|
||||
Name: "Queue depth",
|
||||
Value: &sensorValue,
|
||||
Status: "critical",
|
||||
ObservedAt: time.Now().UTC(),
|
||||
}},
|
||||
},
|
||||
RAID: []models.HostRAIDArray{{
|
||||
Device: "/dev/md0",
|
||||
Level: "raid1",
|
||||
State: "degraded",
|
||||
TotalDevices: 2,
|
||||
ActiveDevices: 1,
|
||||
WorkingDevices: 1,
|
||||
FailedDevices: 1,
|
||||
}},
|
||||
}
|
||||
m.CheckHost(host)
|
||||
childTypes := []string{"disk-health", "raid", "custom-sensor"}
|
||||
for _, alertType := range childTypes {
|
||||
if !hasAlertType(m.GetActiveAlerts(), alertType) {
|
||||
t.Fatalf("expected an active %s alert before the config save", alertType)
|
||||
}
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.DisableAllStorage = true
|
||||
m.config.DisableAllGuests = true
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
|
||||
for _, alertType := range childTypes {
|
||||
if !hasAlertType(m.GetActiveAlerts(), alertType) {
|
||||
t.Errorf("%s alert was resolved by the storage or guest switch", alertType)
|
||||
}
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.DisableAllAgents = true
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
|
||||
for _, alertType := range childTypes {
|
||||
if hasAlertType(m.GetActiveAlerts(), alertType) {
|
||||
t.Errorf("%s alert survived disabling agent alerts", alertType)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The Proxmox node sweep removes alerts whose node is not a Proxmox node
|
||||
// unless it recognises them as agent alerts, by an agent: resource id or the
|
||||
// agent type. Child alerts no longer carry the agent type, so they must keep
|
||||
// the agent: id their canonical spec gives them.
|
||||
func TestHostChildAlertsSurviveProxmoxNodeCleanup(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
pending := int64(2)
|
||||
sensorValue := 23.0
|
||||
host := models.Host{
|
||||
ID: "child-sweep",
|
||||
Hostname: "storage-host",
|
||||
Platform: "linux",
|
||||
Sensors: models.HostSensorSummary{
|
||||
SMART: []models.HostDiskSMART{{
|
||||
Device: "/dev/sda",
|
||||
Serial: "SERIAL-CHILD-3",
|
||||
Health: "PASSED",
|
||||
Attributes: &models.SMARTAttributes{PendingSectors: &pending},
|
||||
}},
|
||||
Custom: []models.HostCustomSensorMetric{{
|
||||
ID: "queue_depth",
|
||||
Name: "Queue depth",
|
||||
Value: &sensorValue,
|
||||
Status: "critical",
|
||||
ObservedAt: time.Now().UTC(),
|
||||
}},
|
||||
},
|
||||
RAID: []models.HostRAIDArray{{
|
||||
Device: "/dev/md0",
|
||||
Level: "raid1",
|
||||
State: "degraded",
|
||||
TotalDevices: 2,
|
||||
ActiveDevices: 1,
|
||||
WorkingDevices: 1,
|
||||
FailedDevices: 1,
|
||||
}},
|
||||
}
|
||||
m.CheckHost(host)
|
||||
childTypes := []string{"disk-health", "raid", "custom-sensor"}
|
||||
for _, alertType := range childTypes {
|
||||
if !hasAlertType(m.GetActiveAlerts(), alertType) {
|
||||
t.Fatalf("expected an active %s alert", alertType)
|
||||
}
|
||||
}
|
||||
|
||||
m.CleanupAlertsForNodes(map[string]bool{"pve1": true})
|
||||
|
||||
for _, alertType := range childTypes {
|
||||
if !hasAlertType(m.GetActiveAlerts(), alertType) {
|
||||
t.Errorf("Proxmox node cleanup removed an agent %s alert", alertType)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,6 +78,7 @@ type Manager struct {
|
||||
unifiedIncidentConfirmations map[string]int // Track consecutive provider-incident observations before activation
|
||||
unifiedIncidentFirstSeen map[string]time.Time // Preserve the first confirmed observation as lifecycle start
|
||||
unifiedIncidentRecoveries map[string]int // Track consecutive healthy observations before provider-incident recovery
|
||||
hostDiskTempAbsences map[string]int // Track consecutive SMART reports missing a disk that holds temperature alert state
|
||||
dockerRestartTracking map[string]*dockerRestartRecord // Track restart counts and times for restart loop detection
|
||||
dockerUpdateFirstSeen map[string]time.Time // Track when image updates were first detected for alert delay
|
||||
// Stable identity tracking prevents update-delay resets when host IDs churn.
|
||||
@@ -102,9 +103,9 @@ type Manager struct {
|
||||
// Cleanup control
|
||||
cleanupStop chan struct{} // Signal to stop cleanup goroutine
|
||||
// Host agent deduplication: the Proxmox node each reporting host agent is
|
||||
// linked to. While a link holds, the agent resource owns that machine's
|
||||
// CPU, memory and disk usage alerts and the node releases its own copies.
|
||||
hostAgentNodeLinks map[string]string // Host agent ID -> linked node ID
|
||||
// linked to and the usage metrics the agent evaluates for it. The node
|
||||
// releases only those metrics and keeps evaluating the rest itself.
|
||||
hostAgentNodeLinks map[string]hostAgentNodeLink // Host agent ID -> node link
|
||||
// Node display name caches. Proxmox nodes can share the same raw node name
|
||||
// across multiple configured instances, so keep instance-scoped entries in
|
||||
// addition to the legacy raw-name cache used by instance-less resources.
|
||||
@@ -233,6 +234,7 @@ func NewManagerWithDataDir(dataDir string, options ...ManagerOption) *Manager {
|
||||
unifiedIncidentConfirmations: make(map[string]int),
|
||||
unifiedIncidentFirstSeen: make(map[string]time.Time),
|
||||
unifiedIncidentRecoveries: make(map[string]int),
|
||||
hostDiskTempAbsences: make(map[string]int),
|
||||
dockerRestartTracking: make(map[string]*dockerRestartRecord),
|
||||
dockerUpdateFirstSeen: make(map[string]time.Time),
|
||||
dockerUpdateFirstSeenByIdentity: make(map[string]time.Time),
|
||||
@@ -245,7 +247,7 @@ func NewManagerWithDataDir(dataDir string, options ...ManagerOption) *Manager {
|
||||
flappingHistory: make(map[string][]time.Time),
|
||||
flappingActive: make(map[string]bool),
|
||||
cleanupStop: make(chan struct{}),
|
||||
hostAgentNodeLinks: make(map[string]string),
|
||||
hostAgentNodeLinks: make(map[string]hostAgentNodeLink),
|
||||
nodeDisplayNames: make(map[string]string),
|
||||
instanceNodeDisplayNames: make(map[string]string),
|
||||
now: time.Now,
|
||||
|
||||
@@ -51,7 +51,7 @@ func metricStabilityFor(metricType string, delaySeconds int) metricStabilityPoli
|
||||
|
||||
func isMetricThresholdAlertType(metricType string) bool {
|
||||
switch metricType {
|
||||
case "cpu", "memory", "disk", "diskRead", "diskWrite", "networkIn", "networkOut", "temperature", "usage":
|
||||
case "cpu", "memory", "disk", "diskRead", "diskWrite", "networkIn", "networkOut", "temperature", "diskTemperature", "usage":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
@@ -77,6 +77,8 @@ func getThresholdForMetric(config ThresholdConfig, metricType string) *Hysteresi
|
||||
return config.NetworkOut
|
||||
case "temperature":
|
||||
return config.Temperature
|
||||
case "diskTemperature":
|
||||
return config.DiskTemperature
|
||||
case "usage":
|
||||
return config.Usage
|
||||
default:
|
||||
|
||||
@@ -969,3 +969,93 @@ func TestAlertCharacterizationResolvedCallbackUsesCanonicalIDForCanonicalAliasCl
|
||||
t.Fatalf("expected resolved callback for %q", alertID)
|
||||
}
|
||||
}
|
||||
|
||||
// A held threshold alert keeps the value and message of its last breach, but
|
||||
// what the alert is about can be newer than the alert: one restored from a
|
||||
// checkpoint written before its producer named a child type or stamped
|
||||
// platformType would otherwise keep linking to the wrong page and naming the
|
||||
// wrong policy owner until it re-fires or clears.
|
||||
func TestAlertCharacterizationHeldMetricAlertRefreshesItsClassification(t *testing.T) {
|
||||
type staleAlert struct {
|
||||
resourceType string
|
||||
platformType string
|
||||
}
|
||||
tests := []struct {
|
||||
name string
|
||||
alertID string
|
||||
fire func(*Manager)
|
||||
hold func(*Manager)
|
||||
stale staleAlert
|
||||
wantResourceType string
|
||||
wantPlatform string
|
||||
wantValue float64
|
||||
}{
|
||||
{
|
||||
name: "agent filesystem",
|
||||
alertID: canonicalMetricStateID("agent:held-host/disk:srv", "disk"),
|
||||
fire: func(m *Manager) {
|
||||
m.CheckHost(models.Host{ID: "held-host", Hostname: "held", Disks: []models.Disk{{Mountpoint: "/srv", Device: "/dev/sdc1", Usage: 92}}})
|
||||
},
|
||||
hold: func(m *Manager) {
|
||||
m.CheckHost(models.Host{ID: "held-host", Hostname: "held", Disks: []models.Disk{{Mountpoint: "/srv", Device: "/dev/sdc1", Usage: 75}}})
|
||||
},
|
||||
stale: staleAlert{resourceType: "agent"},
|
||||
wantResourceType: "agent-disk",
|
||||
wantPlatform: "agent",
|
||||
wantValue: 92,
|
||||
},
|
||||
{
|
||||
name: "vSphere VM",
|
||||
alertID: canonicalMetricStateID("vmware:vc-1:vm:vm-201", "disk"),
|
||||
fire: func(m *Manager) {
|
||||
m.CheckUnifiedResource(&UnifiedResourceInput{ID: "vmware:vc-1:vm:vm-201", Type: "vmware-vm", Name: "app-01", Node: "esxi-01", Instance: "Lab vCenter", Disk: &UnifiedResourceMetric{Percent: 94}})
|
||||
},
|
||||
hold: func(m *Manager) {
|
||||
m.CheckUnifiedResource(&UnifiedResourceInput{ID: "vmware:vc-1:vm:vm-201", Type: "vmware-vm", Name: "app-01", Node: "esxi-01", Instance: "Lab vCenter", Disk: &UnifiedResourceMetric{Percent: 87}})
|
||||
},
|
||||
stale: staleAlert{resourceType: "vSphere VM"},
|
||||
wantResourceType: "vSphere VM",
|
||||
wantPlatform: "vmware-vsphere",
|
||||
wantValue: 94,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
m.mu.Lock()
|
||||
m.config.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 80, Clear: 70}
|
||||
m.mu.Unlock()
|
||||
|
||||
tt.fire(m)
|
||||
m.mu.Lock()
|
||||
alert, ok := m.getActiveAlertNoLock(tt.alertID)
|
||||
if !ok {
|
||||
m.mu.Unlock()
|
||||
t.Fatalf("expected active alert %q", tt.alertID)
|
||||
}
|
||||
alert.Metadata["resourceType"] = tt.stale.resourceType
|
||||
delete(alert.Metadata, alertPlatformTypeKey)
|
||||
m.mu.Unlock()
|
||||
|
||||
tt.hold(m)
|
||||
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
held, ok := m.getActiveAlertNoLock(tt.alertID)
|
||||
if !ok {
|
||||
t.Fatal("held alert resolved; the reading should sit between clear and trigger")
|
||||
}
|
||||
if held.Value != tt.wantValue {
|
||||
t.Fatalf("held alert value = %v, want the last breach %v", held.Value, tt.wantValue)
|
||||
}
|
||||
if got := held.Metadata["resourceType"]; got != tt.wantResourceType {
|
||||
t.Fatalf("resourceType = %v, want %s", got, tt.wantResourceType)
|
||||
}
|
||||
if got := held.Metadata[alertPlatformTypeKey]; got != tt.wantPlatform {
|
||||
t.Fatalf("platformType = %v, want %s", got, tt.wantPlatform)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -136,6 +136,7 @@ type Alert struct {
|
||||
AckTime *time.Time `json:"ackTime,omitempty"`
|
||||
AckUser string `json:"ackUser,omitempty"`
|
||||
Correlation *AlertCorrelation `json:"correlation,omitempty"`
|
||||
Resolution *AlertResolution `json:"resolution,omitempty"` // set only when a close was not a recovery
|
||||
Metadata map[string]interface{} `json:"metadata,omitempty"`
|
||||
LastNotified *time.Time `json:"lastNotified,omitempty"`
|
||||
LastEscalation int `json:"lastEscalation,omitempty"`
|
||||
@@ -168,6 +169,8 @@ func (a *Alert) Clone() *Alert {
|
||||
clone.Correlation = cloneAlertCorrelation(a.Correlation)
|
||||
}
|
||||
|
||||
clone.Resolution = a.Resolution.Clone()
|
||||
|
||||
if len(a.EscalationTimes) > 0 {
|
||||
clone.EscalationTimes = append([]time.Time(nil), a.EscalationTimes...)
|
||||
}
|
||||
|
||||
+95
-50
@@ -93,27 +93,37 @@ func (m *Manager) CheckNode(node models.Node) {
|
||||
m.clearNodeOfflineAlert(node)
|
||||
|
||||
// Check each metric (only if node is online and reachable)
|
||||
// Check for host agent deduplication: if a host agent is linked to this node,
|
||||
// CPU, memory and disk usage alerts belong to the agent resource (CheckHost
|
||||
// evaluates them), so the node releases its own copies instead of alerting twice.
|
||||
if m.hasHostAgentForNode(node.ID) {
|
||||
m.releaseNodeMetricAlerts(node, "cpu", "memory", "disk")
|
||||
} else {
|
||||
var memoryMetric *UnifiedResourceMetric
|
||||
if node.Memory.HasKnownUsage() {
|
||||
memoryMetric = &UnifiedResourceMetric{Percent: node.Memory.Usage}
|
||||
}
|
||||
m.evaluateUnifiedMetrics(&UnifiedResourceInput{
|
||||
ID: node.ID,
|
||||
Type: "node",
|
||||
Name: node.Name,
|
||||
Node: node.Name,
|
||||
Instance: node.Instance,
|
||||
CPU: &UnifiedResourceMetric{Percent: node.CPU * 100},
|
||||
Memory: memoryMetric,
|
||||
Disk: &UnifiedResourceMetric{Percent: node.Disk.Usage},
|
||||
}, thresholds, nil)
|
||||
// Check for host agent deduplication: a usage metric that a linked host
|
||||
// agent evaluates belongs to the agent resource (CheckHost raises it), so
|
||||
// the node releases its own copy instead of alerting twice. A metric the
|
||||
// agent does not evaluate (agent alerts disabled, threshold off, usage
|
||||
// unknown) stays with the node, so the machine is never left unmonitored.
|
||||
link, _ := m.linkedHostAgentForNode(node.ID)
|
||||
input := &UnifiedResourceInput{
|
||||
ID: node.ID,
|
||||
Type: "node",
|
||||
Name: node.Name,
|
||||
Node: node.Name,
|
||||
Instance: node.Instance,
|
||||
}
|
||||
released := make([]string, 0, 3)
|
||||
if link.cpu {
|
||||
released = append(released, "cpu")
|
||||
} else {
|
||||
input.CPU = &UnifiedResourceMetric{Percent: node.CPU * 100}
|
||||
}
|
||||
if link.memory {
|
||||
released = append(released, "memory")
|
||||
} else if node.Memory.HasKnownUsage() {
|
||||
input.Memory = &UnifiedResourceMetric{Percent: node.Memory.Usage}
|
||||
}
|
||||
if link.disk {
|
||||
released = append(released, "disk")
|
||||
} else {
|
||||
input.Disk = &UnifiedResourceMetric{Percent: node.Disk.Usage}
|
||||
}
|
||||
m.releaseNodeMetricAlerts(node, released...)
|
||||
m.evaluateUnifiedMetrics(input, thresholds, nil)
|
||||
|
||||
// CPU temperature stays with the node even when a host agent runs on it:
|
||||
// CheckHost has no CPU temperature metric, and the node poll already
|
||||
@@ -189,32 +199,46 @@ func (m *Manager) releaseNodeMetricAlerts(node models.Node, metrics ...string) {
|
||||
}
|
||||
}
|
||||
|
||||
// registerHostAgentNodeLink records which Proxmox node a reporting host agent
|
||||
// is linked to. While the link holds, the agent resource owns that machine's
|
||||
// CPU, memory and disk usage alerts. The link is the monitor's identity decision
|
||||
// (automatic or operator-set), so an agent whose hostname merely matches a node
|
||||
// name in another instance, or one an operator unlinked, never makes a node
|
||||
// release its alerts, and an agent reporting an FQDN still dedups its node.
|
||||
func (m *Manager) registerHostAgentNodeLink(host models.Host) {
|
||||
hostID := strings.TrimSpace(host.ID)
|
||||
if hostID == "" {
|
||||
// hostAgentNodeLink is a reporting host agent's link to a Proxmox node and the
|
||||
// usage metrics the agent evaluates for that machine. The link is the monitor's
|
||||
// identity decision (automatic or operator-set), so an agent whose hostname
|
||||
// merely matches a node name in another instance, or one an operator unlinked,
|
||||
// never makes a node release its alerts, and an agent reporting an FQDN still
|
||||
// dedups its node.
|
||||
type hostAgentNodeLink struct {
|
||||
agentID string
|
||||
agentName string
|
||||
nodeID string
|
||||
cpu bool
|
||||
memory bool
|
||||
disk bool
|
||||
}
|
||||
|
||||
// registerHostAgentNodeLink records a host agent's node link after CheckHost
|
||||
// has decided which usage metrics it evaluates. A link without a node or
|
||||
// without any evaluated metric is removed, so the node keeps all its alerts.
|
||||
func (m *Manager) registerHostAgentNodeLink(link hostAgentNodeLink) {
|
||||
link.agentID = strings.TrimSpace(link.agentID)
|
||||
link.nodeID = strings.TrimSpace(link.nodeID)
|
||||
if link.agentID == "" {
|
||||
return
|
||||
}
|
||||
linkedNodeID := strings.TrimSpace(host.LinkedNodeID)
|
||||
m.mu.Lock()
|
||||
previous, existed := m.hostAgentNodeLinks[hostID]
|
||||
if linkedNodeID == "" {
|
||||
delete(m.hostAgentNodeLinks, hostID)
|
||||
} else {
|
||||
m.hostAgentNodeLinks[hostID] = linkedNodeID
|
||||
if link.nodeID == "" || !(link.cpu || link.memory || link.disk) {
|
||||
m.unregisterHostAgentNodeLink(link.agentID)
|
||||
return
|
||||
}
|
||||
m.mu.Lock()
|
||||
previous, existed := m.hostAgentNodeLinks[link.agentID]
|
||||
m.hostAgentNodeLinks[link.agentID] = link
|
||||
m.mu.Unlock()
|
||||
|
||||
if !existed || previous != linkedNodeID {
|
||||
if !existed || previous != link {
|
||||
log.Debug().
|
||||
Str("hostID", hostID).
|
||||
Str("hostname", host.Hostname).
|
||||
Str("linkedNodeID", linkedNodeID).
|
||||
Str("hostID", link.agentID).
|
||||
Str("linkedNodeID", link.nodeID).
|
||||
Bool("cpu", link.cpu).
|
||||
Bool("memory", link.memory).
|
||||
Bool("disk", link.disk).
|
||||
Msg("Updated host agent node link for deduplication")
|
||||
}
|
||||
}
|
||||
@@ -227,33 +251,54 @@ func (m *Manager) unregisterHostAgentNodeLink(hostID string) {
|
||||
return
|
||||
}
|
||||
m.mu.Lock()
|
||||
linkedNodeID, existed := m.hostAgentNodeLinks[hostID]
|
||||
link, existed := m.hostAgentNodeLinks[hostID]
|
||||
delete(m.hostAgentNodeLinks, hostID)
|
||||
m.mu.Unlock()
|
||||
|
||||
if existed {
|
||||
log.Debug().
|
||||
Str("hostID", hostID).
|
||||
Str("linkedNodeID", linkedNodeID).
|
||||
Str("linkedNodeID", link.nodeID).
|
||||
Msg("Removed host agent node link from deduplication")
|
||||
}
|
||||
}
|
||||
|
||||
// hasHostAgentForNode reports whether a reporting host agent is linked to the
|
||||
// given Proxmox node, in which case the node's usage alerts belong to the agent.
|
||||
func (m *Manager) hasHostAgentForNode(nodeID string) bool {
|
||||
// linkedHostAgentForNode returns the host agent coverage for the given Proxmox
|
||||
// node. A usage metric counts as owned when any agent linked to the node
|
||||
// evaluates it, so a second linked agent covering a metric the first does not
|
||||
// still keeps the node from alerting twice. The agent identity reported is the
|
||||
// lowest agent ID among the linked agents, so the choice is stable.
|
||||
func (m *Manager) linkedHostAgentForNode(nodeID string) (hostAgentNodeLink, bool) {
|
||||
nodeID = strings.TrimSpace(nodeID)
|
||||
if nodeID == "" {
|
||||
return false
|
||||
return hostAgentNodeLink{}, false
|
||||
}
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
for _, linkedNodeID := range m.hostAgentNodeLinks {
|
||||
if linkedNodeID == nodeID {
|
||||
return true
|
||||
var found hostAgentNodeLink
|
||||
ok := false
|
||||
for _, link := range m.hostAgentNodeLinks {
|
||||
if link.nodeID != nodeID {
|
||||
continue
|
||||
}
|
||||
if !ok || link.agentID < found.agentID {
|
||||
found.agentID = link.agentID
|
||||
found.agentName = link.agentName
|
||||
}
|
||||
found.nodeID = nodeID
|
||||
found.cpu = found.cpu || link.cpu
|
||||
found.memory = found.memory || link.memory
|
||||
found.disk = found.disk || link.disk
|
||||
ok = true
|
||||
}
|
||||
return false
|
||||
return found, ok
|
||||
}
|
||||
|
||||
// hasHostAgentForNode reports whether a reporting host agent is linked to the
|
||||
// given Proxmox node and owns at least one of its usage metrics.
|
||||
func (m *Manager) hasHostAgentForNode(nodeID string) bool {
|
||||
_, ok := m.linkedHostAgentForNode(nodeID)
|
||||
return ok
|
||||
}
|
||||
|
||||
// UpdateNodeDisplayName caches the display name for a node/host so alerts
|
||||
|
||||
@@ -285,6 +285,7 @@ func markOperationalResolved(
|
||||
}
|
||||
alert.Evidence = appendOperationalEvidence(alert.Evidence, recoveryEvidence.Clone())
|
||||
recoveryEvidenceIDs := []string{recoveryEvidence.ID}
|
||||
cause := resolvedTransitionCause(alert.Resolution)
|
||||
from := record.State
|
||||
record.State = operationaltrust.OperationalResolved
|
||||
record.StateChangedAt = resolvedAt
|
||||
@@ -295,7 +296,7 @@ func markOperationalResolved(
|
||||
from,
|
||||
operationaltrust.OperationalResolved,
|
||||
resolvedAt,
|
||||
operationaltrust.TransitionRecoveryEvidence,
|
||||
cause,
|
||||
record.CauseKey,
|
||||
recoveryEvidenceIDs,
|
||||
)
|
||||
@@ -308,9 +309,10 @@ func markOperationalResolved(
|
||||
From: from,
|
||||
To: operationaltrust.OperationalResolved,
|
||||
At: resolvedAt,
|
||||
Cause: operationaltrust.TransitionRecoveryEvidence,
|
||||
Cause: cause,
|
||||
CauseKey: record.CauseKey,
|
||||
EvidenceIDs: recoveryEvidenceIDs,
|
||||
Reason: alert.Resolution.Summary(),
|
||||
}
|
||||
alert.Transitions = appendOperationalTransition(
|
||||
alert.Transitions,
|
||||
@@ -318,6 +320,16 @@ func markOperationalResolved(
|
||||
)
|
||||
}
|
||||
|
||||
// resolvedTransitionCause names why a record entered resolved. Only a close
|
||||
// with no typed resolution is a recovery; a handover keeps its own cause so
|
||||
// timelines and alert-quality counts never report it as one.
|
||||
func resolvedTransitionCause(resolution *AlertResolution) operationaltrust.TransitionCause {
|
||||
if resolution != nil && resolution.Reason == AlertResolutionMovedToAgent {
|
||||
return operationaltrust.TransitionOwnershipTransferred
|
||||
}
|
||||
return operationaltrust.TransitionRecoveryEvidence
|
||||
}
|
||||
|
||||
func (m *Manager) newResolvedAlert(
|
||||
alert *Alert,
|
||||
resolvedAt time.Time,
|
||||
@@ -497,6 +509,16 @@ func legacyAlertRecoveryEvidenceEnvelope(
|
||||
if err != nil {
|
||||
return operationaltrust.EvidenceEnvelope{}, false
|
||||
}
|
||||
reason := &operationaltrust.EvidenceReason{
|
||||
Code: legacyAlertRecoveryEvidenceReason,
|
||||
}
|
||||
// A close that was not a recovery must not be recorded as one.
|
||||
if summary := alert.Resolution.Summary(); summary != "" {
|
||||
reason = &operationaltrust.EvidenceReason{
|
||||
Code: string(alert.Resolution.Reason),
|
||||
Message: summary,
|
||||
}
|
||||
}
|
||||
envelope := operationaltrust.EvidenceEnvelope{
|
||||
ID: id,
|
||||
Source: source,
|
||||
@@ -506,9 +528,7 @@ func legacyAlertRecoveryEvidenceEnvelope(
|
||||
Completeness: operationaltrust.EvidencePartial,
|
||||
Confidence: operationaltrust.EvidenceUnknown,
|
||||
Permissions: operationaltrust.EvidencePermissionsUnknown,
|
||||
Reason: &operationaltrust.EvidenceReason{
|
||||
Code: legacyAlertRecoveryEvidenceReason,
|
||||
},
|
||||
Reason: reason,
|
||||
}
|
||||
return envelope, envelope.Validate() == nil
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
|
||||
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
|
||||
@@ -412,6 +413,175 @@ func TestResolvedOperationalContractUpdatesHistorySnapshot(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery(t *testing.T) {
|
||||
manager := newUnifiedEvalParityManager(t)
|
||||
store, err := eventlog.OpenInMemory()
|
||||
if err != nil {
|
||||
t.Fatalf("open in-memory event log: %v", err)
|
||||
}
|
||||
manager.SetEventLog(store)
|
||||
manager.UpdateConfig(AlertConfig{
|
||||
Enabled: true,
|
||||
NodeDefaults: ThresholdConfig{
|
||||
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
|
||||
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
|
||||
},
|
||||
})
|
||||
disableTestTimeThresholds(manager)
|
||||
node := models.Node{
|
||||
ID: "pve-1",
|
||||
Name: "pve-1",
|
||||
Instance: "test",
|
||||
Status: "online",
|
||||
CPU: 0.95,
|
||||
Memory: models.Memory{Total: 8 << 30, Used: 7 << 30, Usage: 95},
|
||||
}
|
||||
manager.CheckNode(node)
|
||||
if active := manager.GetActiveAlerts(); len(active) != 2 {
|
||||
t.Fatalf("active alerts = %#v, want cpu and memory", active)
|
||||
}
|
||||
|
||||
resolvedByType := map[string]*ResolvedAlert{}
|
||||
manager.SetResolvedAlertCallback(func(resolved *ResolvedAlert) {
|
||||
resolvedByType[resolved.Alert.Type] = resolved
|
||||
})
|
||||
release := func(metric string, resolution *AlertResolution) {
|
||||
t.Helper()
|
||||
spec, err := buildCanonicalMetricSpec(node.ID, node.Name, unifiedresources.ResourceType("node"), metric, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("build %s spec: %v", metric, err)
|
||||
}
|
||||
manager.releaseCanonicalMetricAlert(spec, node.Name, node.Name, node.Instance, "node", 0, resolution)
|
||||
}
|
||||
|
||||
handover := &AlertResolution{
|
||||
Reason: AlertResolutionMovedToAgent,
|
||||
SuccessorResourceID: "agent-pve-1",
|
||||
SuccessorName: "pve-1 (Host Agent)",
|
||||
}
|
||||
release("memory", handover)
|
||||
// The close is recorded as a copy: the caller's value cannot rewrite it.
|
||||
handover.SuccessorName = "changed"
|
||||
release("cpu", nil)
|
||||
|
||||
if active := manager.GetActiveAlerts(); len(active) != 0 {
|
||||
t.Fatalf("active alerts after release = %#v, want none: the close must still happen", active)
|
||||
}
|
||||
|
||||
moved := resolvedByType["memory"]
|
||||
if moved == nil {
|
||||
t.Fatal("memory release did not reach the resolved consumer")
|
||||
}
|
||||
resolution := moved.Alert.Resolution
|
||||
if resolution == nil ||
|
||||
resolution.Reason != AlertResolutionMovedToAgent ||
|
||||
resolution.SuccessorResourceID != "agent-pve-1" ||
|
||||
resolution.SuccessorName != "pve-1 (Host Agent)" {
|
||||
t.Fatalf("resolution = %+v, want moved to pve-1 (Host Agent)", resolution)
|
||||
}
|
||||
wantSummary := "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
if got := resolution.Summary(); got != wantSummary {
|
||||
t.Fatalf("Summary() = %q, want %q", got, wantSummary)
|
||||
}
|
||||
if got := resolution.Outcome(); got != "moved to pve-1 (Host Agent)" {
|
||||
t.Fatalf("Outcome() = %q", got)
|
||||
}
|
||||
if got := resolution.Describe("Memory alert"); got != "Memory alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading." {
|
||||
t.Fatalf("Describe() = %q", got)
|
||||
}
|
||||
|
||||
record := moved.Alert.OperationalRecord
|
||||
if record == nil || record.State != operationaltrust.OperationalResolved {
|
||||
t.Fatalf("operational record = %+v, want resolved", record)
|
||||
}
|
||||
if err := record.Validate(); err != nil {
|
||||
t.Fatalf("operational record Validate() error = %v", err)
|
||||
}
|
||||
transition := moved.Alert.LatestTransition
|
||||
if transition == nil || transition.Reason != wantSummary {
|
||||
t.Fatalf("resolve transition = %+v, want reason %q", transition, wantSummary)
|
||||
}
|
||||
if err := transition.Validate(); err != nil {
|
||||
t.Fatalf("resolve transition Validate() error = %v", err)
|
||||
}
|
||||
// Patrol's timeline prints the cause above the reason, so a handover
|
||||
// must not be labelled as recovery evidence.
|
||||
if transition.Cause != operationaltrust.TransitionOwnershipTransferred {
|
||||
t.Fatalf("resolve transition cause = %q, want %q", transition.Cause, operationaltrust.TransitionOwnershipTransferred)
|
||||
}
|
||||
if recorded := moved.Alert.Transitions[len(moved.Alert.Transitions)-1]; recorded.ID != transition.ID ||
|
||||
recorded.Cause != operationaltrust.TransitionOwnershipTransferred {
|
||||
t.Fatalf("recorded timeline transition = %+v, want the ownership transfer", recorded)
|
||||
}
|
||||
closing := moved.Alert.Evidence[len(moved.Alert.Evidence)-1]
|
||||
if closing.ID != transition.EvidenceIDs[0] ||
|
||||
closing.Reason == nil ||
|
||||
closing.Reason.Code != string(AlertResolutionMovedToAgent) ||
|
||||
closing.Reason.Message != wantSummary {
|
||||
t.Fatalf("closing evidence = %+v, want the handover, not a recovery projection", closing)
|
||||
}
|
||||
|
||||
recovered := resolvedByType["cpu"]
|
||||
if recovered == nil || recovered.Alert.Resolution != nil {
|
||||
t.Fatalf("plain disabled release = %+v, want an ordinary resolve with no resolution", recovered)
|
||||
}
|
||||
if last := recovered.Alert.Evidence[len(recovered.Alert.Evidence)-1]; last.Reason == nil ||
|
||||
last.Reason.Code != legacyAlertRecoveryEvidenceReason {
|
||||
t.Fatalf("plain release evidence = %+v, want legacy recovery projection", last.Reason)
|
||||
}
|
||||
if recovered.Alert.LatestTransition == nil ||
|
||||
recovered.Alert.LatestTransition.Cause != operationaltrust.TransitionRecoveryEvidence {
|
||||
t.Fatalf("plain release transition = %+v, want recovery_evidence", recovered.Alert.LatestTransition)
|
||||
}
|
||||
|
||||
var historyResolution *AlertResolution
|
||||
for _, entry := range manager.GetAlertHistory(10) {
|
||||
if entry.Type == "memory" {
|
||||
historyResolution = entry.Resolution
|
||||
}
|
||||
}
|
||||
if historyResolution == nil || historyResolution.Reason != AlertResolutionMovedToAgent {
|
||||
t.Fatalf("history resolution = %+v, want moved_to_agent", historyResolution)
|
||||
}
|
||||
|
||||
events, err := manager.AlertEvents(eventlog.Filter{AlertID: moved.Alert.ID, Types: []string{eventlog.TypeResolved}})
|
||||
if err != nil {
|
||||
t.Fatalf("AlertEvents: %v", err)
|
||||
}
|
||||
if len(events) != 1 ||
|
||||
events[0].Reason != string(AlertResolutionMovedToAgent) ||
|
||||
events[0].Message != wantSummary {
|
||||
t.Fatalf("resolved events = %+v, want one handover event", events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvedHistoryRowKeepsResolutionWithoutEventLog(t *testing.T) {
|
||||
firstObservedAt := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
|
||||
alert := &Alert{
|
||||
ID: "pve-1-memory",
|
||||
Type: "memory",
|
||||
Level: AlertLevelWarning,
|
||||
ResourceID: "pve-1",
|
||||
CanonicalSpecID: "metric-threshold:memory",
|
||||
CanonicalState: "pve-1::metric-threshold:memory",
|
||||
StartTime: firstObservedAt,
|
||||
LastSeen: firstObservedAt,
|
||||
}
|
||||
ensureOperationalContract(alert, firstObservedAt)
|
||||
history := newTestHistoryManager(t)
|
||||
history.AddAlert(*alert)
|
||||
manager := &Manager{historyManager: history}
|
||||
|
||||
alert.Resolution = &AlertResolution{Reason: AlertResolutionMovedToAgent, SuccessorName: "pve-1 (Host Agent)"}
|
||||
manager.newResolvedAlert(alert, firstObservedAt.Add(time.Minute), nil)
|
||||
|
||||
entries := history.GetAllHistory(1)
|
||||
if len(entries) != 1 || entries[0].Resolution == nil ||
|
||||
entries[0].Resolution.Reason != AlertResolutionMovedToAgent {
|
||||
t.Fatalf("history entries = %+v, want the row to keep the moved_to_agent resolution", entries)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeOperationalRecurrenceReopensStableCause(t *testing.T) {
|
||||
firstObservedAt := time.Date(2026, 7, 18, 20, 0, 0, 0, time.UTC)
|
||||
resolvedAt := firstObservedAt.Add(5 * time.Minute)
|
||||
|
||||
@@ -301,7 +301,8 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
|
||||
StartTime: exported.StartTime,
|
||||
Acknowledged: exported.Acknowledged,
|
||||
// exported is a deep clone, so the map is already private.
|
||||
Metadata: exported.Metadata,
|
||||
Metadata: exported.Metadata,
|
||||
Resolution: modelAlertResolution(exported.Resolution),
|
||||
},
|
||||
ResolvedTime: alert.ResolvedTime,
|
||||
})
|
||||
@@ -309,6 +310,20 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
|
||||
return resolved
|
||||
}
|
||||
|
||||
// modelAlertResolution projects a non-recovery close into the state model, so
|
||||
// snapshot and assistant readers can tell a handover from a recovery.
|
||||
func modelAlertResolution(resolution *AlertResolution) *models.AlertResolution {
|
||||
if resolution == nil {
|
||||
return nil
|
||||
}
|
||||
return &models.AlertResolution{
|
||||
Reason: string(resolution.Reason),
|
||||
SuccessorResourceID: resolution.SuccessorResourceID,
|
||||
SuccessorName: resolution.SuccessorName,
|
||||
Summary: resolution.Summary(),
|
||||
}
|
||||
}
|
||||
|
||||
// GetResolvedAlert returns a copy of a recently resolved alert by ID.
|
||||
func (m *Manager) GetResolvedAlert(alertID string) *ResolvedAlert {
|
||||
// Write lock: getResolvedAlertNoLock backfills the resolvedAlias map on a
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// AlertResolutionReason says why an alert closed when the close was not a
|
||||
// recovery. The close still reaches every resolved consumer, so an incident
|
||||
// opened from the alert does not stay open on the receiving side, but no
|
||||
// surface may report the resource as healthy.
|
||||
type AlertResolutionReason string
|
||||
|
||||
const (
|
||||
// AlertResolutionMovedToAgent means a Pulse agent linked to the alert's
|
||||
// resource now owns the same metric, so the agent's own alert settings
|
||||
// decide whether the condition is still alerted on.
|
||||
AlertResolutionMovedToAgent AlertResolutionReason = "moved_to_agent"
|
||||
)
|
||||
|
||||
// AlertResolution records why an alert closed without its condition being
|
||||
// observed to clear. A nil resolution is an ordinary recovery.
|
||||
type AlertResolution struct {
|
||||
Reason AlertResolutionReason `json:"reason"`
|
||||
// SuccessorResourceID and SuccessorName identify the resource that owns
|
||||
// the alert from now on, named the way that resource's own alerts are.
|
||||
SuccessorResourceID string `json:"successorResourceId,omitempty"`
|
||||
SuccessorName string `json:"successorName,omitempty"`
|
||||
}
|
||||
|
||||
// Clone returns a copy of the resolution.
|
||||
func (r *AlertResolution) Clone() *AlertResolution {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
clone := *r
|
||||
return &clone
|
||||
}
|
||||
|
||||
// Outcome is the short form of a close that was not a recovery, for lists
|
||||
// such as "moved to pve1 (Host Agent)". It is empty for an ordinary recovery.
|
||||
func (r *AlertResolution) Outcome() string {
|
||||
if r == nil {
|
||||
return ""
|
||||
}
|
||||
switch r.Reason {
|
||||
case AlertResolutionMovedToAgent:
|
||||
successor := strings.TrimSpace(r.SuccessorName)
|
||||
if successor == "" {
|
||||
successor = "its Pulse agent"
|
||||
}
|
||||
return "moved to " + successor
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// Summary is the one-line account of a close that was not a recovery. It is
|
||||
// empty for an ordinary recovery.
|
||||
func (r *AlertResolution) Summary() string {
|
||||
return r.Describe("Alert")
|
||||
}
|
||||
|
||||
// Describe is Summary with the alert named by the caller, such as
|
||||
// "Memory alert", so a message about one of several closes says which one.
|
||||
func (r *AlertResolution) Describe(subject string) string {
|
||||
if r == nil {
|
||||
return ""
|
||||
}
|
||||
if subject = strings.TrimSpace(subject); subject == "" {
|
||||
subject = "Alert"
|
||||
}
|
||||
switch r.Reason {
|
||||
case AlertResolutionMovedToAgent:
|
||||
return fmt.Sprintf("%s %s. This is not a recovery: check the agent for the current reading.", subject, r.Outcome())
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
@@ -359,6 +359,158 @@ func TestCheckNodeMissingTemperatureDoesNotResolveOpenAlert(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Deduplication only hands the agent the usage metrics it evaluates. With agent
|
||||
// alerts switched off, or one agent threshold off, the node keeps those alerts
|
||||
// so the machine is never left unmonitored.
|
||||
func TestCheckNodeKeepsUsageMetricsTheAgentDoesNotEvaluate(t *testing.T) {
|
||||
setup := func(t *testing.T) (*Manager, models.Node, models.Host) {
|
||||
m := newTestManager(t)
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
|
||||
m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
|
||||
m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
|
||||
m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
|
||||
m.mu.Unlock()
|
||||
node, host := testNodeWithHostAgent()
|
||||
node.CPU = 0.95
|
||||
node.Memory = models.Memory{Total: 100, Used: 95, Free: 5, Usage: 95}
|
||||
host.CPUUsage = 95
|
||||
host.Memory = node.Memory
|
||||
return m, node, host
|
||||
}
|
||||
nodeCPU := func(node models.Node) string { return canonicalMetricStateID(node.ID, "cpu") }
|
||||
nodeMemory := func(node models.Node) string { return canonicalMetricStateID(node.ID, "memory") }
|
||||
|
||||
t.Run("agent_alerts_disabled", func(t *testing.T) {
|
||||
m, node, host := setup(t)
|
||||
m.CheckHost(host)
|
||||
m.CheckNode(node)
|
||||
if testHasActiveAlert(t, m, nodeCPU(node)) {
|
||||
t.Fatalf("expected the agent to own CPU while its alerts are enabled")
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.DisableAllAgents = true
|
||||
m.mu.Unlock()
|
||||
m.CheckHost(host)
|
||||
m.CheckNode(node)
|
||||
if !testHasActiveAlert(t, m, nodeCPU(node)) || !testHasActiveAlert(t, m, nodeMemory(node)) {
|
||||
t.Fatalf("expected the node to alert on CPU and memory while agent alerts are disabled")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("agent_cpu_threshold_off", func(t *testing.T) {
|
||||
m, node, host := setup(t)
|
||||
m.mu.Lock()
|
||||
m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
m.mu.Unlock()
|
||||
m.CheckHost(host)
|
||||
m.CheckNode(node)
|
||||
if !testHasActiveAlert(t, m, nodeCPU(node)) {
|
||||
t.Fatalf("expected the node to keep CPU when the agent CPU threshold is off")
|
||||
}
|
||||
if testHasActiveAlert(t, m, nodeMemory(node)) {
|
||||
t.Fatalf("expected the agent to keep owning memory")
|
||||
}
|
||||
if got := testActiveAlertIDsOfType(m, "memory"); len(got) != 1 {
|
||||
t.Fatalf("expected exactly one memory alert for the machine, got %v", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Ownership follows what the linked agents are configured to evaluate, not one
|
||||
// report's data: a missing agent memory reading must not hand memory back to
|
||||
// the node, a second linked agent's metrics count, and the node's disk metric
|
||||
// belongs to the agent only when the agent evaluates that same filesystem.
|
||||
func TestCheckNodeUsageOwnershipFollowsWhatAgentsEvaluate(t *testing.T) {
|
||||
setup := func(t *testing.T) (*Manager, models.Node, models.Host) {
|
||||
m := newTestManager(t)
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
|
||||
m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
|
||||
m.config.NodeDefaults.Disk = &HysteresisThreshold{Trigger: 90, Clear: 85}
|
||||
m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
|
||||
m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
|
||||
m.config.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 90, Clear: 85}
|
||||
m.mu.Unlock()
|
||||
node, host := testNodeWithHostAgent()
|
||||
return m, node, host
|
||||
}
|
||||
|
||||
t.Run("missing_agent_memory_reading", func(t *testing.T) {
|
||||
m, node, host := setup(t)
|
||||
node.Memory = models.Memory{Total: 100, Used: 95, Free: 5, Usage: 95}
|
||||
host.Memory = node.Memory
|
||||
m.CheckHost(host)
|
||||
m.CheckNode(node)
|
||||
|
||||
host.Memory = models.Memory{Total: 100, UsageUnavailable: true}
|
||||
m.CheckHost(host)
|
||||
m.CheckNode(node)
|
||||
agentAlertID := canonicalMetricStateID(hostResourceID(host.ID), "memory")
|
||||
if got := testActiveAlertIDsOfType(m, "memory"); len(got) != 1 || got[0] != agentAlertID {
|
||||
t.Fatalf("expected only the agent memory alert while its reading is missing, got %v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("second_linked_agent", func(t *testing.T) {
|
||||
m, node, host := setup(t)
|
||||
node.CPU = 0.95
|
||||
m.mu.Lock()
|
||||
m.config.Overrides = map[string]ThresholdConfig{
|
||||
"agent-a": {CPU: &HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
}
|
||||
m.mu.Unlock()
|
||||
agentA := host
|
||||
agentA.ID = "agent-a"
|
||||
agentB := host
|
||||
agentB.ID = "agent-b"
|
||||
agentB.CPUUsage = 95
|
||||
m.CheckHost(agentA)
|
||||
m.CheckHost(agentB)
|
||||
m.CheckNode(node)
|
||||
agentBAlertID := canonicalMetricStateID(hostResourceID(agentB.ID), "cpu")
|
||||
if got := testActiveAlertIDsOfType(m, "cpu"); len(got) != 1 || got[0] != agentBAlertID {
|
||||
t.Fatalf("expected only agent-b's CPU alert when it covers CPU for the node, got %v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("summary_disk", func(t *testing.T) {
|
||||
m, node, host := setup(t)
|
||||
root := models.Disk{Mountpoint: "/", Device: "/dev/sda1", Total: 100, Used: 95, Free: 5, Usage: 95}
|
||||
data := models.Disk{Mountpoint: "/data", Device: "/dev/sdb1", Total: 100, Used: 10, Free: 90, Usage: 10}
|
||||
host.Disks = []models.Disk{root, data}
|
||||
node.Disk = models.Disk{Total: 100, Used: 95, Free: 5, Usage: 95}
|
||||
rootResourceID, _ := hostDiskResourceID(host, root)
|
||||
nodeDiskAlertID := canonicalMetricStateID(node.ID, "disk")
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.Overrides = map[string]ThresholdConfig{rootResourceID: {Disabled: true}}
|
||||
m.mu.Unlock()
|
||||
m.CheckHost(host)
|
||||
m.CheckNode(node)
|
||||
if !testHasActiveAlert(t, m, nodeDiskAlertID) {
|
||||
t.Fatalf("expected the node to keep its root disk alert while the agent skips that filesystem")
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.Overrides = nil
|
||||
m.mu.Unlock()
|
||||
m.CheckHost(host)
|
||||
m.CheckNode(node)
|
||||
if testHasActiveAlert(t, m, nodeDiskAlertID) {
|
||||
t.Fatalf("expected the node to release its disk alert once the agent evaluates the same filesystem")
|
||||
}
|
||||
if !testHasActiveAlert(t, m, canonicalMetricStateID(rootResourceID, "disk")) {
|
||||
t.Fatalf("expected the agent's root filesystem alert")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A missing reading keeps the incident but must not let a sustained-for or
|
||||
// recovery delay complete across the gap in evidence.
|
||||
func TestCheckNodeMissingTemperatureInterruptsTimingRuns(t *testing.T) {
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user