mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 12:42:49 +00:00
a21e0ef8f8
Follows #2591. A node alert that moves to its linked Pulse agent closes with a `moved_to_agent` resolution (#2580). Two readers still presented that close as a recovery. **Performance reports.** The node, VM and container enrichers copied recently resolved alerts into `reporting.AlertInfo` without the resolution, so a report generated in the five-minute window after a handover read HEALTHY, "All systems operating normally", counted the alert as "triggered and resolved", and sent the AI narrator `resolved: true`. - Report rows now carry the alert's resource and resolution (`reportActiveAlertInfo` / `reportResolvedAlertInfo`). - `pkg/reporting` has one rule (`unresolvedAlerts`): an alert that closed without recovering stays live unless the report also shows its successor's reading for that metric, meaning an open alert or a recovery since the handover. That way the condition is counted once, and fleets match handovers across all their resources. - The single-resource PDF card reads WARNING (or CRITICAL at the alert's level) with "Alert moved to <agent> - check the agent for the current reading". The active list names the agent, the alerts table shows "Moved" with the engine's summary as a note, and recommendations say to check the agent. - Fleets count the alert as active and name the agent. Their all-clear pattern and "No fleet-wide action required" no longer appear while any alert is active. - The narrator payload carries `resolved: false`, the summary and `successor_alert_listed`, with prompt rules for both. **Resource history.** `alert_resolved` changes were labelled by kind alone ("Alert resolved", green) with that label prefixed to the headline. `getResourceChangePresentation` now reads the `alert_resolution` metadata: the change reads "Alert moved" in the neutral blue of the history badge, and its summary is the whole headline in the drawer, `ResourceChangeSummary` and Patrol's assessment handoff. **Verification** - A private mock stack running this branch, with a local-only overlay that closes a real pve1 memory alert through `clearAlertWithResolution`. Within the window, `/api/admin/reports/generate?resourceType=node` rendered CRITICAL with the handover message and a "Moved" row, while the ordinary CPU recovery stayed green. - In the browser, a full drawer (Availability checks, Fleet view) showed the producer-made rows as "Alert moved" next to "Alert resolved" at 1280 and 375. A Proxmox node's own drawer uses the `table-row` presentation, which fetches no change history (see below). - Tests in registered proof files: `pkg/reporting/alert_state_test.go` (verdict, successor timing, fleet pool, PDF text and width), `internal/api/contract_test.go`, `internal/ai/report_narrator_test.go`, `resourceChangePresentation.test.ts`, `ResourceDetailDrawer.history.test.tsx` and `patrolInvestigationContextModel.test.ts`. Each failed with the fix reverted. - Second opinion: two read-only review rounds. The temporal match, fleet-wide pool, narrator dedupe signal, fleet all-clear fallbacks and contract wording came from them. **Not fixed here (separate changes)** - Reports requested as `agent` carry no alert rows at all. That is how the Reports page addresses every Proxmox node in v6, so such a report reads HEALTHY even with active critical alerts. This change makes the shared rule correct for when those reports attach the machine's alerts. - Node and guest alert lifecycle changes are recorded under legacy alert resource IDs, which canonical history queries never match. Facet kind counts still group moved closes under "Alert resolved".
250 lines
8.7 KiB
Go
250 lines
8.7 KiB
Go
package reporting
|
|
|
|
import (
|
|
"time"
|
|
)
|
|
|
|
// ReportFormat represents the output format of a report
|
|
type ReportFormat string
|
|
|
|
const (
|
|
FormatCSV ReportFormat = "csv"
|
|
FormatPDF ReportFormat = "pdf"
|
|
)
|
|
|
|
// MetricReportRequest defines the parameters for generating a report
|
|
type MetricReportRequest struct {
|
|
ResourceType string
|
|
ResourceID string
|
|
MetricType string // Optional, if empty all metrics for the resource are included
|
|
Start time.Time
|
|
End time.Time
|
|
Format ReportFormat
|
|
Title string
|
|
Branding ReportBranding
|
|
|
|
// MetricsResourceID overrides ResourceID for metrics-store queries.
|
|
// The v6 API addresses resources by canonical unified ID while the
|
|
// metrics store is keyed by each platform's native source ID (the
|
|
// resource's metricsTarget); recovery points and Patrol findings stay
|
|
// keyed by the unified ID. Handlers set this from the resource's
|
|
// metricsTarget so the store query can match without changing the
|
|
// identity the rest of the report pipeline sees.
|
|
MetricsResourceID string
|
|
|
|
// Optional enrichment data (populated by handler from monitor state)
|
|
Resource *ResourceInfo // Details about the resource being reported on
|
|
Alerts []AlertInfo // Active and recently resolved alerts for this resource
|
|
Backups []BackupInfo // Backup information for VMs/containers
|
|
Storage []StorageInfo // Storage pools (for nodes)
|
|
Disks []DiskInfo // Physical disk health (for nodes)
|
|
Availability *AvailabilityInfo // Observed availability over the window (from the state timeline)
|
|
|
|
// Optional narrative interpretation. When Narrator is non-nil the
|
|
// engine builds a NarrativeInput from the queried report data and asks
|
|
// it to produce the executive summary; on error or nil it falls back to
|
|
// the heuristic narrator. Findings are passed through to NarrativeInput
|
|
// so a narrator can reference Patrol activity in the period.
|
|
Narrator Narrator
|
|
FindingsProvider FindingsProvider
|
|
}
|
|
|
|
// AvailabilityInfo summarizes a resource's observed availability over the
|
|
// report window, derived from the recorded resource state timeline.
|
|
//
|
|
// Time the resource was absent from the registry or in an unknown state
|
|
// (for example while the monitor itself was restarting) is treated as
|
|
// unobserved: it is excluded from the uptime calculation entirely rather
|
|
// than counted as downtime, and disclosed through ObservedPercent. A
|
|
// monitoring gap is not an outage, and a client-facing stability report
|
|
// must not present one as such.
|
|
type AvailabilityInfo struct {
|
|
// UptimePercent is up / (up + down) over the observed portion of the
|
|
// window. Online and warning states count as up; offline counts as
|
|
// down. Zero when the resource was never observed in the window.
|
|
UptimePercent float64
|
|
// ObservedPercent is the share of the report window during which the
|
|
// resource state was actually being recorded.
|
|
ObservedPercent float64
|
|
// TotalDowntime is the cumulative time spent in a down state.
|
|
TotalDowntime time.Duration
|
|
// LongestOutage is the longest contiguous stretch of down time.
|
|
LongestOutage time.Duration
|
|
// DownIncidents counts distinct transitions into a down state.
|
|
DownIncidents int
|
|
}
|
|
|
|
// Observed reports whether the resource was observed at all in the window.
|
|
func (a *AvailabilityInfo) Observed() bool {
|
|
return a != nil && a.ObservedPercent > 0
|
|
}
|
|
|
|
// ResourceInfo contains details about the resource being reported on
|
|
type ResourceInfo struct {
|
|
Name string
|
|
DisplayName string
|
|
Status string
|
|
Host string // URL for nodes
|
|
Node string // Parent node for VMs/containers
|
|
Instance string // Proxmox instance name
|
|
Uptime int64
|
|
KernelVersion string
|
|
PVEVersion string
|
|
OSName string
|
|
OSVersion string
|
|
IPAddresses []string
|
|
CPUModel string
|
|
CPUCores int
|
|
CPUSockets int
|
|
MemoryTotal int64
|
|
DiskTotal int64
|
|
LoadAverage []float64
|
|
Temperature *float64 // CPU temp if available
|
|
Tags []string
|
|
ClusterName string
|
|
IsCluster bool
|
|
}
|
|
|
|
// AlertInfo contains alert information for the report
|
|
type AlertInfo struct {
|
|
Type string
|
|
Level string // warning, critical
|
|
Message string
|
|
Value float64
|
|
Threshold float64
|
|
StartTime time.Time
|
|
ResolvedTime *time.Time // nil if still active
|
|
Acknowledged bool
|
|
// ResourceID is the resource the alert was raised on, so an alert that
|
|
// moved can find the alert its successor raised for the same metric.
|
|
ResourceID string
|
|
// Resolution is set only on a resolved alert whose close was not a
|
|
// recovery, such as a node alert handed to its Pulse agent. A report
|
|
// never counts such an alert as resolved.
|
|
Resolution *AlertResolution
|
|
}
|
|
|
|
// AlertResolution says why an alert closed without its condition clearing.
|
|
// It mirrors models.AlertResolution so this package stays free of internal
|
|
// dependencies.
|
|
type AlertResolution struct {
|
|
// Reason is the alert engine's reason code, such as
|
|
// AlertResolutionMovedToAgent.
|
|
Reason string
|
|
SuccessorResourceID string
|
|
SuccessorName string
|
|
// Summary is the alert engine's one-line account, such as "Alert moved
|
|
// to pve1 (Host Agent). This is not a recovery: check the agent for the
|
|
// current reading."
|
|
Summary string
|
|
}
|
|
|
|
// BackupInfo contains backup information for VMs/containers
|
|
type BackupInfo struct {
|
|
Type string // vzdump, pbs
|
|
Storage string
|
|
Timestamp time.Time
|
|
Size int64
|
|
Verified bool
|
|
Protected bool
|
|
VolID string
|
|
NextBackup *time.Time
|
|
}
|
|
|
|
// StorageInfo contains storage pool information
|
|
type StorageInfo struct {
|
|
Name string
|
|
Type string // lvm, zfs, dir, nfs, etc.
|
|
Status string
|
|
Total int64
|
|
Used int64
|
|
Available int64
|
|
UsagePerc float64
|
|
Content string // images, rootdir, backup, etc.
|
|
ZFSHealth string // For ZFS pools
|
|
ZFSErrors int // Checksum/read/write errors
|
|
}
|
|
|
|
// DiskInfo contains physical disk health information
|
|
type DiskInfo struct {
|
|
Device string
|
|
Model string
|
|
Serial string
|
|
Type string // nvme, ssd, hdd
|
|
Size int64
|
|
Health string // PASSED, FAILED, UNKNOWN
|
|
Temperature int // Celsius
|
|
WearLevel int // 0-100, percentage of life REMAINING (100 = healthy, 0 = end of life, -1 = unknown)
|
|
// TemperatureWarning and TemperatureCritical are the disk's alert disk
|
|
// temperature thresholds in Celsius (the clear value and the trigger). The
|
|
// reading is amber from Warning and red from Critical, as in the Physical
|
|
// Disks Temp column. A zero Critical leaves the reading uncoloured.
|
|
TemperatureWarning float64
|
|
TemperatureCritical float64
|
|
}
|
|
|
|
// MultiReportRequest defines the parameters for generating a multi-resource report.
|
|
type MultiReportRequest struct {
|
|
Resources []MetricReportRequest // One per resource, each with enrichment
|
|
Format ReportFormat
|
|
Start time.Time
|
|
End time.Time
|
|
Title string
|
|
MetricType string
|
|
Branding ReportBranding
|
|
|
|
// Optional fleet-level narrative interpretation. When FleetNarrator is
|
|
// non-nil the engine builds a FleetNarrativeInput from the queried
|
|
// per-resource report data and asks it to produce the cross-resource
|
|
// summary; on error or nil it falls back to the heuristic fleet
|
|
// narrator. FindingsProvider, when set, is consulted per-resource so
|
|
// patrol findings can flow into per-resource narratives.
|
|
FleetNarrator FleetNarrator
|
|
Narrator Narrator
|
|
FindingsProvider FindingsProvider
|
|
}
|
|
|
|
// MultiReportData holds the data for multi-resource report generation.
|
|
type MultiReportData struct {
|
|
Title string
|
|
Start time.Time
|
|
End time.Time
|
|
GeneratedAt time.Time
|
|
Resources []*ReportData // Reuse existing ReportData per resource
|
|
TotalPoints int
|
|
Brand *ReportBrand
|
|
|
|
// Fleet-level narrative interpretation, populated by the engine when
|
|
// the request supplies a FleetNarrator (or always populated with the
|
|
// heuristic fallback). The renderer prefers this over recomputing
|
|
// observations inline.
|
|
FleetNarrative *FleetNarrative
|
|
}
|
|
|
|
// Engine defines the interface for report generation.
|
|
// This allows the enterprise version to provide PDF/CSV generation.
|
|
//
|
|
// NarrativeFor and FleetNarrativeFor return the structured narrative
|
|
// without rendering, for callers that want the synthesis layer in a
|
|
// non-PDF form (Pulse Assistant tool calls, programmatic consumers).
|
|
type Engine interface {
|
|
Generate(req MetricReportRequest) (data []byte, contentType string, err error)
|
|
GenerateMulti(req MultiReportRequest) (data []byte, contentType string, err error)
|
|
NarrativeFor(req MetricReportRequest) (*Narrative, error)
|
|
FleetNarrativeFor(req MultiReportRequest) (*FleetNarrative, error)
|
|
}
|
|
|
|
var (
|
|
globalEngine Engine
|
|
)
|
|
|
|
// SetEngine sets the global report engine.
|
|
func SetEngine(e Engine) {
|
|
globalEngine = e
|
|
}
|
|
|
|
// GetEngine returns the current global report engine.
|
|
func GetEngine() Engine {
|
|
return globalEngine
|
|
}
|