mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-23 20:25:56 +00:00
Preserve alert lifecycle truth and safe pinned updates
Rejected active-alert writes still reached history, firing events and notification dispatch. Make admission explicit across detector and restore paths so repeated suppressed observations cannot create those side effects. Reconcile restored alerts when persisted resource policy attaches at startup, reconcile again after native aliases become canonical, and refresh shared state before those alerts reach clients. Preserve occurrence identity when retained alerts re-fire, including durable history reconstruction and delayed lifecycle replay. Keep retained table and card components mounted when scrolling shifts the visible window, so policy editors do not lose unsaved state. Show actual future maintenance times and the paused-attention effect. Keep synthetic host machine identities stable across restarts so saved operator policy remains attached during verification. Qualify server update and rollback instructions by helper ownership, and stop signed installer recipes before execution if a download or signature check fails. Refs #2237
This commit is contained in:
rcourtman
+21
-3
@@ -67,7 +67,12 @@ If you run Proxmox VE, the easiest and most “Pulse-native” deployment is the
|
||||
Replace `vX.Y.Z` with the exact release tag you want, then run this on your Proxmox host:
|
||||
|
||||
```bash
|
||||
(
|
||||
set -e
|
||||
export PULSE_VERSION=vX.Y.Z
|
||||
pulse_installer_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "$pulse_installer_dir"' EXIT
|
||||
cd "$pulse_installer_dir"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh.sshsig"
|
||||
ssh-keygen -Y verify \
|
||||
@@ -76,7 +81,7 @@ ssh-keygen -Y verify \
|
||||
-n pulse-install \
|
||||
-s install.sh.sshsig < install.sh
|
||||
bash install.sh --version "${PULSE_VERSION}"
|
||||
rm -f install.sh install.sh.sshsig
|
||||
)
|
||||
```
|
||||
|
||||
> **Note**: The GitHub `install.sh` is the **server** installer. The agent installer is served from your Pulse server at `/install.sh` (see **Settings → Infrastructure → Install on a host**). Do not use the GitHub server installer to install or update `pulse-agent`.
|
||||
@@ -144,7 +149,12 @@ See [KUBERNETES.md](KUBERNETES.md) for ingress and persistence configuration.
|
||||
For Linux servers (VM or bare metal), use the official installer:
|
||||
|
||||
```bash
|
||||
(
|
||||
set -e
|
||||
export PULSE_VERSION=vX.Y.Z
|
||||
pulse_installer_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "$pulse_installer_dir"' EXIT
|
||||
cd "$pulse_installer_dir"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh.sshsig"
|
||||
ssh-keygen -Y verify \
|
||||
@@ -153,7 +163,7 @@ ssh-keygen -Y verify \
|
||||
-n pulse-install \
|
||||
-s install.sh.sshsig < install.sh
|
||||
sudo bash install.sh --version "${PULSE_VERSION}"
|
||||
rm -f install.sh install.sh.sshsig
|
||||
)
|
||||
```
|
||||
|
||||
> **Note**: This installs the Pulse server. Use the `/install.sh` endpoint from **Settings → Infrastructure → Install on a host** for installing or upgrading `pulse-agent` on monitored hosts.
|
||||
@@ -250,7 +260,15 @@ Pulse can update the server runtime to the latest stable version.
|
||||
|----------|---------|
|
||||
| **Docker** | `docker compose pull && docker compose up -d` |
|
||||
| **Kubernetes** | `helm repo update && helm upgrade pulse pulse/pulse -n pulse` |
|
||||
| **Systemd / Proxmox LXC** | `sudo /bin/update` |
|
||||
| **Systemd / Proxmox LXC with the Pulse-owned helper** | `sudo /bin/update` |
|
||||
|
||||
Use `/bin/update --version vX.Y.Z` for an exact target only when the helper was
|
||||
installed by the Pulse server installer. On Proxmox community-scripts
|
||||
containers, `/bin/update` can belong to a different updater that ignores
|
||||
`--version`. If the helper is absent or its owner is unknown, use the
|
||||
[signed server-installer flow](#2-bare-metal--systemd) with `PULSE_VERSION` set
|
||||
to the exact target tag. The same ownership check applies to rollback. After
|
||||
the service restarts, verify the installed version with `GET /api/version`.
|
||||
|
||||
Docker without Compose: `docker restart` keeps the old image running. Run `docker pull rcourtman/pulse:vX.Y.Z`, then `docker stop pulse && docker rm pulse` and re-run your original `docker run` command.
|
||||
|
||||
|
||||
+22
-4
@@ -148,13 +148,23 @@ Preferred path:
|
||||
|
||||
- **Settings → System → Updates**
|
||||
|
||||
If you prefer CLI, use the installed update helper for the target version:
|
||||
If you prefer CLI, use this command only when `/bin/update` was installed by
|
||||
the Pulse server installer:
|
||||
|
||||
```bash
|
||||
sudo /bin/update --version vX.Y.Z
|
||||
```
|
||||
|
||||
`/bin/update` is installed by the supported systemd and Proxmox LXC server installer. If your host does not have it yet, follow the signed server-installer flow in [INSTALL.md](INSTALL.md). Agent updates and v5-to-v6 agent upgrades still use the `/install.sh` command generated in **Settings → Infrastructure → Install on a host**; that screen is for both first installs and in-place agent upgrades.
|
||||
On Proxmox community-scripts containers, `/bin/update` can belong to the
|
||||
community-scripts updater, which can ignore `--version`. If that is your
|
||||
installation, the helper is absent, or you cannot confirm its owner, follow the
|
||||
[signed server-installer flow](INSTALL.md#2-bare-metal--systemd) and set
|
||||
`PULSE_VERSION` to the exact target tag. This also applies to rollback. Verify
|
||||
the installed version with `GET /api/version` after the service restarts.
|
||||
|
||||
Agent updates and v5-to-v6 agent upgrades use the `/install.sh` command generated
|
||||
in **Settings → Infrastructure → Install on a host**. That screen is for both
|
||||
first installs and in-place agent upgrades.
|
||||
|
||||
Operator note for builds after `v6.0.0-rc.2`: the historical Pulse update
|
||||
signer was not recovered. Hosts pinned to the `rc.2` trust root should not
|
||||
@@ -273,12 +283,16 @@ older v5 agents do not understand observer configuration.
|
||||
### Can I keep Pulse v5 stable while I test Pulse v6?
|
||||
|
||||
Yes. Keep a rollback path available while you evaluate v6. The final release
|
||||
on the v5 line is 5.1.36, so the stable rollback command is:
|
||||
on the v5 line is 5.1.36. On installations with the Pulse-owned update helper,
|
||||
the rollback command is:
|
||||
|
||||
```bash
|
||||
sudo /bin/update --version v5.1.36
|
||||
```
|
||||
|
||||
The [installer ownership requirements](#systemd-and-proxmox-lxc-installs) above
|
||||
also apply to this command.
|
||||
|
||||
### Why did my v5 install upgrade itself to v6?
|
||||
|
||||
Pulse 5.1.29 and later pin the built-in updater to the 5.1.x line and never
|
||||
@@ -286,12 +300,16 @@ offer v6, so upgrading from those versions is always a manual step. Pulse
|
||||
5.1.28 and older have no such pin: installs with auto-update enabled follow
|
||||
the newest stable GitHub release, which is now v6. If that happened to you,
|
||||
your data and configuration carry over; run through the Post-Upgrade
|
||||
Checklist above to confirm everything still works. To return to v5, run:
|
||||
Checklist above to confirm everything still works. To return to v5 with the
|
||||
Pulse-owned update helper, run:
|
||||
|
||||
```bash
|
||||
sudo /bin/update --version v5.1.36
|
||||
```
|
||||
|
||||
For community-scripts installations or an unknown helper owner, use the
|
||||
[installer ownership guidance](#systemd-and-proxmox-lxc-installs) above.
|
||||
|
||||
## Migration Notes (v6)
|
||||
|
||||
### Unified Navigation (Bookmarks and Deep Links)
|
||||
|
||||
+66
@@ -85,3 +85,69 @@ resolution is load-bearing client-side while
|
||||
history behind a detail endpoint needs the alert-override/threshold matching
|
||||
reworked onto an on-demand lookup. Platform source payloads (~1.4MB) are
|
||||
live data, not static metadata, and are out of scope for this gap.
|
||||
|
||||
## 2026-09-24 reliability audit measurement
|
||||
|
||||
The gap remains open after the September telemetry coalescing improvements.
|
||||
An isolated production frontend build on `pulse-dev`, based on
|
||||
`0abe518219f75512fc3f9ca0fdbd4ce549e30e90` plus the alert-admission and installer
|
||||
guidance repair, used the 50-node synthetic estate. Random metrics changed
|
||||
every two seconds. The measured `/api/state` response was 4,015,466 bytes.
|
||||
This is a REST response measurement, not a measurement of every websocket
|
||||
frame or proof of the contribution of an individual field.
|
||||
|
||||
The existing browser performance recipe ran on 2026-09-24 from 13:27:23 to
|
||||
13:29:56 UTC. At 1440 x 900 without CPU throttling, the 30-second idle window
|
||||
contained seven long tasks totalling 3,531 ms, with a maximum of 1,037 ms.
|
||||
At 390 x 844 with 4x CPU throttling, seven long tasks totalled 10,104 ms, with
|
||||
a maximum of 2,748 ms. Host load was 0.76/1.55/2.22 before the run and
|
||||
1.47/1.55/2.12 afterwards. The recipe exited successfully, but its desktop
|
||||
top-level navigation locator no longer matched the UI. Those missing steps
|
||||
are not navigation proof. The idle measurements do not establish causality
|
||||
between static metadata and all observed stalls.
|
||||
|
||||
A separate settled-server 30.02-second CPU profile contained 3.54 seconds
|
||||
of CPU samples, or 11.79% of one core. Canonical identity matching accounted
|
||||
for 880 ms cumulative in that profile. Three 1,000-resource component runs
|
||||
measured registry rebuilds at 14.7–15.3 ms, full snapshot construction at
|
||||
9.4–9.5 ms and delta construction at 0.107–0.109 ms. These results neither
|
||||
reproduce a reported production CPU incident nor qualify browser
|
||||
responsiveness. The remaining browser stalls require their own trace-led
|
||||
investigation and cross-client contract proof before this gap can close.
|
||||
|
||||
Reproduction uses the performance rig recipe in
|
||||
`pulse-dev-infra/docs/perf-rig.md`, with its existing `perf-audit.mjs` pointed
|
||||
at the isolated synthetic server. Script SHA-256:
|
||||
`b068bb3f96009418e21c63d249458447c3041212152536925ef65f694b79cde6`.
|
||||
CPU profile SHA-256:
|
||||
`49a75bb9ad6323238e32dfdffa5342cbc77b77282f0ddd4687df012b14488e30`.
|
||||
Raw logs and profiles are retained in the local reliability audit evidence
|
||||
bundle under `pulse-audit-implementation-20260924/verification/performance-baseline/`.
|
||||
|
||||
### Final reliability-source observation
|
||||
|
||||
The same recipe was repeated after the alert lifecycle, startup reconciliation,
|
||||
retained-row rendering and stable synthetic identity repairs, using final
|
||||
backend SHA-256
|
||||
`77d17ff9e21ccdef3f5fff9f7b91b841abe3570bf8e04e1eda501c554e3a5889`
|
||||
and frontend index SHA-256
|
||||
`b9f1288bd188c8e549ccd4cdfa7aa75b56d49df3fc0f59de9e11dac90d449172`.
|
||||
The functional browser was closed before measurement. The run lasted from
|
||||
16:03:53 to 16:07:21 UTC. Although the preceding one-minute load sample was
|
||||
1.88, the actual start sample rose to 2.49/2.35/3.93 and the end sample was
|
||||
4.08/3.42/4.05. It therefore does not qualify as a controlled comparison or
|
||||
performance improvement proof.
|
||||
|
||||
Desktop idle recorded seven long tasks totalling 5,949 ms, maximum 2,705 ms.
|
||||
The phone viewport with 4x CPU throttling recorded eight totalling 12,971 ms,
|
||||
maximum 3,732 ms. The authenticated REST snapshots were 4,003,769 and
|
||||
4,003,907 bytes respectively, both HTTP 200. The unchanged desktop navigation
|
||||
locators still missed several routes. The harness exit code was zero, but
|
||||
those steps do not count as navigation acceptance. Functional route and policy
|
||||
acceptance was performed separately in the final browser interaction matrix.
|
||||
|
||||
These observations leave this gap open. They do not attribute the change in
|
||||
long-task cost to the source repairs or prove a particular payload field is
|
||||
responsible. Stable fixture machine IDs are created at graph construction,
|
||||
not during each metric refresh. The raw final run and host-load receipts are
|
||||
retained under `pulse-audit-implementation-20260924/verification/performance-final/`.
|
||||
|
||||
@@ -9786,7 +9786,7 @@
|
||||
},
|
||||
{
|
||||
"id": "resource-payload-static-metadata",
|
||||
"summary": "The client-facing resource stream ships ~3KB of mostly-static metadata per resource (duplicated capabilities blobs with only 10 distinct values across 1,508 resources, canonicalIdentity alias/superseded-id history, aiSafeSummary prose, and policy routing) on every snapshot, REST recovery, and reconnect. At 50-node scale the snapshot is a single 4.73MB websocket frame and browser main-thread cost per delta reaches ~1s even on desktop hardware. Slimming the stream is a wire-format change with consumers beyond frontend-modern (pulse-mobile OTA compatibility, Pro surfaces, AI runtime), so it needs an owned slice with contract updates and cross-client verification.",
|
||||
"summary": "Capabilities catalog, default-policy omission and alias deduplication reduced the resource payload in August, but canonical identity history remains inline. The September 24 reliability audit measured a 4,015,466-byte /api/state response and 1,037 ms maximum desktop idle long task on the 50-node synthetic estate. A final-source repeat still observed multi-second long tasks, but elevated host load prevents a controlled comparison. This remains an open cross-client payload and browser responsiveness gap requiring trace-led attribution, contract updates and mobile/Pro/runtime verification before closure. See the evidence record for measurements and limits.",
|
||||
"owner": "project-owner",
|
||||
"status": "triaged",
|
||||
"recorded_at": "2026-08-24",
|
||||
|
||||
@@ -45,6 +45,17 @@ covers unchanged active evaluations, repeated fired/resolved transitions,
|
||||
JSON checkpoint reload, canonical-backed shells and delayed historical events.
|
||||
`TestIncidentStore_LifecycleRapidRecurrence` covers distinct subsecond starts.
|
||||
|
||||
An explicit newer refire reopens the same retained occurrence. The checkpoint
|
||||
persists its refire timestamp so an older resolution, historical read repair,
|
||||
or repeated refire cannot override the latest transition. Canonical resource
|
||||
history carries `alert_started_at` separately from transition time, allowing
|
||||
`QueryIncidents` to reconstruct one occurrence after fire, resolution and refire
|
||||
even without a saved incident shell. Legacy history without this metadata keeps
|
||||
its timestamp-based fallback. `internal/ai/memory/incidents_refire_test.go` pins
|
||||
checkpoint and stale-replay behaviour, while
|
||||
`TestMonitorLifecycleRefireReopensRetainedOccurrence` exercises both local and
|
||||
canonical projections, including reconstruction from canonical history.
|
||||
|
||||
This does not migrate existing duplicate records, change bounded retention,
|
||||
make acknowledgement event replay idempotent, or prove aggregate write-byte
|
||||
reductions or recipient delivery. An evicted occurrence can still be recreated
|
||||
|
||||
@@ -15,6 +15,22 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Retained occurrence re-fires preserve incident truth
|
||||
|
||||
A reducer re-fire inside the retention window keeps its original occurrence
|
||||
identity and reopens that incident. It is distinct from replaying the initial
|
||||
firing. Canonical resource changes carry `alert_started_at` separately from the
|
||||
transition timestamp, so history reconstruction also retains one occurrence.
|
||||
Legacy records without that key retain their historical timestamp projection.
|
||||
Checkpointed re-fire timestamps reject earlier resolution replay and historical
|
||||
read repair. A later resolution still closes the occurrence, and replaying the
|
||||
older re-fire cannot reopen it. This does not change alert retention or delivery.
|
||||
|
||||
`TestMonitorLifecycleRefireReopensRetainedOccurrence` covers both history paths,
|
||||
canonical reconstruction without an incident checkpoint, repeated replay and
|
||||
historical read repair. `TestIncidentRefireCheckpointPreservesTransitionOrder`
|
||||
covers checkpoint reload on either side of the final resolution.
|
||||
|
||||
### Automatic acknowledgement lifecycle
|
||||
|
||||
Cleanup-triggered acknowledgement uses the same canonical retention records and
|
||||
@@ -2543,6 +2559,14 @@ hidden by expected-offline.
|
||||
|
||||
Every active-alert writer passes through the same operator-state gate, and
|
||||
notification delivery consults the same decision before quiet-hours policy.
|
||||
The active-store writer reports admission explicitly. Detectors must not add
|
||||
history, recent-alert state, fired events, rate-limit consumption, or delivery
|
||||
intent after a rejected write. Lifecycle evaluators report suppression without
|
||||
an activation transition. Restore and guest-identity migration must not revive
|
||||
acknowledgement or tracking state for a rejected alert. Repeated observations,
|
||||
policy removal or expiry, and post-mutation reconciliation are exercised across
|
||||
provider incidents, metric and canonical lifecycle writers in
|
||||
`internal/alerts/alert_admission_test.go`.
|
||||
Operator suppression is never converted into a quiet-hours replay, including
|
||||
for recovery notifications carrying stale replay metadata.
|
||||
After a persisted policy mutation, `ReconcileResourceOperatorState` resolves
|
||||
|
||||
@@ -556,7 +556,18 @@ as complete.
|
||||
|
||||
Published exact-version install and rollback guidance must preserve the server
|
||||
and Unified Agent installer boundary. Supported systemd and Proxmox LXC
|
||||
deployments use the signed `/bin/update --version vX.Y.Z` server helper;
|
||||
deployments may use `/bin/update --version vX.Y.Z` only when the installed
|
||||
helper belongs to the Pulse server installer. Community-scripts Proxmox
|
||||
containers may own a different `/bin/update` that ignores the version argument.
|
||||
Generated install and rollback sections must each state this ownership boundary
|
||||
and link the signed server-installer flow from the candidate's versioned docs,
|
||||
with the exact installation or rollback target, for absent or unverified helpers.
|
||||
Operators verify the resulting version after restart.
|
||||
Signed installer recipes run in a fresh temporary directory and stop on any
|
||||
download or signature-verification failure before executing the installer.
|
||||
`scripts/tests/test_signed_installer_docs.py` executes both documented recipes
|
||||
with controlled download and verification failures and checks exact-version
|
||||
forwarding on success.
|
||||
`/opt/pulse/scripts/install.sh` and release archives' `scripts/install.sh` are
|
||||
Unified Agent installers and must never be presented as server rollback
|
||||
commands. Docker guidance instead pins the target image and recreates the
|
||||
|
||||
@@ -623,6 +623,27 @@ that manual-channel consequence must stay visible at desktop and narrow widths.
|
||||
|
||||
## Shared Boundaries
|
||||
|
||||
### Maintenance schedule copy
|
||||
|
||||
Operator maintenance banners show local absolute start and end timestamps,
|
||||
including future dates. Past-time relative formatting must not turn a future
|
||||
expiry into "just now". The banner describes paused attention rather than
|
||||
promising that rejected alert observations are acknowledged. Active and
|
||||
future-window mounted regressions pin these user-visible claims.
|
||||
|
||||
### Retained state in bounded platform windows
|
||||
|
||||
`PlatformWindowedRows` and `PlatformWindowedList` keep one keyed renderer owner
|
||||
while the visible window moves. Items present in both windows retain component
|
||||
identity, active detail tabs and unsaved edits. Passing a new window must update
|
||||
the existing renderer rather than instantiate another renderer around it.
|
||||
Removed items still unmount, and row budgets and spacer behavior are unchanged.
|
||||
The mounted scroll regression in `PlatformWindowedRows.test.tsx` exercises both
|
||||
renderers with an edited input retained across overlapping windows. The browser
|
||||
journey is a narrow Proxmox node Manage form, scrolling to lifecycle Save while
|
||||
the synthetic estate continues updating, plus the alert timeline and note form.
|
||||
|
||||
|
||||
### PBS host history correlation
|
||||
|
||||
The Backups surface passes its complete deduplicated route inventory to the PBS table. Only PBS servers render as rows; other resources supply correlation inputs. Preserve PBS drawer identity and use the correlated canonical history target. Missing disk utilisation does not gate CPU/memory history.
|
||||
|
||||
@@ -17,6 +17,17 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Synthetic machine identity survives restart
|
||||
|
||||
Linked mock hosts and Docker hosts derive their machine ID from their stable
|
||||
fixture host ID. Docker agent IDs are stable as well.
|
||||
Changing metric samples or rebuilding the graph must not give the same fixture
|
||||
machine a new canonical agent ID, orphaning saved operator policy. The existing
|
||||
`TestFixtureIdentityStableAcrossBoots` compares machine IDs per host and the
|
||||
unified canonical agent IDs, in addition to source IDs and names. This is a
|
||||
fixture guarantee. Production identity matching must still reject a conflicting
|
||||
nonempty machine ID rather than transfer policy to a different real machine.
|
||||
|
||||
### Canonical registry publication during accepted ingest — issue #2199
|
||||
|
||||
The monitor adapter builds a replacement resource generation away from readers.
|
||||
@@ -3918,6 +3929,18 @@ invent lifecycle state; Alerts owns signal suppression and unified resources
|
||||
owns persistence. `internal/monitoring/monitor_alert_intent_test.go` and the
|
||||
alerts intent-policy proof pin this adapter boundary.
|
||||
|
||||
Alert restore precedes resource-store attachment during startup. Once the
|
||||
adapter attaches the persisted operator-policy resolver, monitoring asks Alerts
|
||||
to reconcile the restored set and refreshes shared alert state if it changed.
|
||||
Resource publication repeats reconciliation before evaluation and shared-state
|
||||
export, because native aliases may only become resolvable after the first
|
||||
observation populates the canonical registry. Persisted agent policy applies
|
||||
to those restored native-ID alerts even when the initial registry was empty.
|
||||
Muted or retired resources cannot retain active alerts merely because policy
|
||||
was unavailable during restore. The resolution survives a subsequent restart
|
||||
and preserves unaffected alerts. This ordering is pinned by
|
||||
`internal/monitoring/monitor_alert_restore_test.go`.
|
||||
|
||||
`internal/maintenancesentinel/` is monitoring-owned post-maintenance assurance.
|
||||
Its bounded sweep derives every concrete one-shot or recurring occurrence that
|
||||
ended in the seven-day lookback, de-duplicates on canonical resource plus exact
|
||||
|
||||
@@ -1421,6 +1421,22 @@ without contention or cross-delivery.
|
||||
|
||||
## Current State
|
||||
|
||||
### Retained component state in bounded platform windows
|
||||
|
||||
`PlatformWindowedRows` and `PlatformWindowedList` retain one keyed renderer
|
||||
owner as overlapping visible windows change. Components that remain visible
|
||||
keep their local state and unsaved edits. Removed items still unmount, and the
|
||||
existing row budgets and spacers remain bounded. The mounted regressions in
|
||||
`PlatformWindowedRows.test.tsx` exercise both renderers. The final browser
|
||||
receipt exercises a narrow Proxmox Manage form through scrolling, saving and
|
||||
viewport changes while the synthetic estate updates.
|
||||
|
||||
This correctness proof does not close the large-estate performance gap. The
|
||||
final measurements and host-load limitation remain recorded in
|
||||
`records/resource-payload-static-metadata-2026-08-24.md`. They do not establish
|
||||
a controlled performance improvement or satisfy the open SLO qualification.
|
||||
|
||||
|
||||
### Large API responses negotiate gzip without corrupting edge cases
|
||||
|
||||
The main listener compresses eligible JSON and frontend text assets for
|
||||
@@ -3139,6 +3155,14 @@ resource results and defensive-copy isolation, while
|
||||
`internal/api/alerting/alerts_test.go` proves that mock transport does not need
|
||||
the production incident store.
|
||||
|
||||
Synthetic linked-host machine identities and Docker machine/agent identities
|
||||
are derived from stable fixture host identity when constructing the graph.
|
||||
Rebuilding the same estate therefore preserves canonical agent IDs and the
|
||||
resource policy attached to them. `internal/mock/generator_test.go` verifies
|
||||
nonempty machine and agent IDs and canonical identity across fixture rebuilds.
|
||||
This changes fixture generation only, without relaxing production identity
|
||||
resolution or adding work to the recurring update path.
|
||||
|
||||
### Update evidence reuses the bounded node observation
|
||||
|
||||
Package evidence adds only one bounded status, one bounded reason, and the
|
||||
|
||||
@@ -2519,6 +2519,7 @@
|
||||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/alerts/alert_admission_test.go",
|
||||
"internal/alerts/canonical_stateful_test.go",
|
||||
"internal/alerts/incident_synthesis_test.go",
|
||||
"internal/alerts/intent_policy_test.go",
|
||||
@@ -2754,6 +2755,7 @@
|
||||
"allow_same_subsystem_tests": false,
|
||||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/alerts/alert_admission_test.go",
|
||||
"internal/alerts/alerts_test.go",
|
||||
"internal/alerts/callback_config_coverage_test.go",
|
||||
"internal/alerts/canonical_override_migration_test.go",
|
||||
@@ -7188,6 +7190,7 @@
|
||||
"frontend-modern/src/stores/__tests__/websocket-unified.test.ts",
|
||||
"frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts",
|
||||
"internal/mock/alert_incidents_test.go",
|
||||
"internal/mock/generator_test.go",
|
||||
"internal/mock/integration_coverage_test.go",
|
||||
"internal/mock/platform_fixtures_test.go"
|
||||
]
|
||||
@@ -8723,6 +8726,7 @@
|
||||
"exact_files": [
|
||||
"internal/ai/tools/filesystem_evidence_test.go",
|
||||
"internal/monitoring/docker_filesystem_evidence_test.go",
|
||||
"internal/monitoring/monitor_alert_handling_test.go",
|
||||
"internal/unifiedresources/action_request_identity_test.go",
|
||||
"internal/unifiedresources/action_result_v2_test.go",
|
||||
"internal/unifiedresources/availability_link_test.go",
|
||||
|
||||
@@ -2544,6 +2544,17 @@ persists recurrence JSON and scope in additive columns while legacy one-shot
|
||||
rows retain their existing meaning. Scheduling, updating, changing scope, or
|
||||
clearing either form remains an atomic operator-state plus resource-timeline
|
||||
lifecycle write.
|
||||
|
||||
The operator-state banner projects these authoritative occurrence boundaries
|
||||
as local absolute start and end timestamps, including scheduled future dates.
|
||||
An active maintenance window describes paused attention rather than claiming
|
||||
that rejected observations have been acknowledged. Persisted policy remains
|
||||
keyed by canonical resource identity. Monitoring reconciles restored native
|
||||
alert aliases after registry publication, before evaluation or shared-state
|
||||
publication. Mounted operator-state regressions cover active and scheduled
|
||||
copy, while the monitoring restore regression covers SQLite policy and native
|
||||
aliases before the first observation.
|
||||
|
||||
`NormalizeResourceOperatorState` trims whitespace, de-duplicates capability
|
||||
names, guarantees a non-nil empty capability list, and lower-cases the
|
||||
criticality value before persistence. SQLite reads must apply that
|
||||
@@ -4411,6 +4422,15 @@ projection for notes, analyses, command breadcrumbs, runbooks, and other
|
||||
operator-facing incident memory. Agents must not model the same durable backend
|
||||
fact in both places as competing primary histories.
|
||||
|
||||
Alert lifecycle resource changes record a nonzero occurrence start in
|
||||
`MetadataAlertStartedAt` (`alert_started_at`), separately from the transition
|
||||
timestamp. A refire retains this start, so the canonical incident query can
|
||||
reconstruct the original occurrence without an incident-memory checkpoint.
|
||||
Legacy changes without the metadata retain timestamp-based projection.
|
||||
`TestMonitorLifecycleRefireReopensRetainedOccurrence` in
|
||||
`internal/monitoring/monitor_alert_handling_test.go` proves the metadata bridge,
|
||||
ordered lifecycle history and canonical-only reconstruction together.
|
||||
|
||||
The unified resource core is strong and canonical, but monitoring and some
|
||||
frontend/API consumers are still being tightened around it.
|
||||
|
||||
|
||||
@@ -1,37 +1,55 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "3b1b43293e21d06d1286124f96f78b1247443dea",
|
||||
"verified_at": "2026-09-24T09:02:03Z",
|
||||
"base_sha": "0abe518219f75512fc3f9ca0fdbd4ce549e30e90",
|
||||
"verified_at": "2026-09-24T16:05:19Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Settings/BrandingSettingsCard.tsx"
|
||||
"frontend-modern/src/components/Infrastructure/ResourceOperatorStateSection.tsx",
|
||||
"frontend-modern/src/features/platformPage/PlatformWindowedList.tsx",
|
||||
"frontend-modern/src/features/platformPage/PlatformWindowedRows.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Settings/BrandingSettingsCard.tsx": "6516a05aff71b1d96660f81189b6df8bceccf7e94d15d6c3213556e794f2ae40"
|
||||
"frontend-modern/src/components/Infrastructure/ResourceOperatorStateSection.tsx": "5c47a4d5d5ff12b0f6cf0e495bd02ebdadccd81135ad1ff91f9f13e90b5a81af",
|
||||
"frontend-modern/src/features/platformPage/PlatformWindowedList.tsx": "1842c634054e46e24d965a063719065ca1156b3f061d594c878a8866d5b33a3e",
|
||||
"frontend-modern/src/features/platformPage/PlatformWindowedRows.tsx": "6f5dbc611dfbb00bc2d770fcf06b8dbf329ebd76dc022bb63191480a221cb457"
|
||||
},
|
||||
"routes": [
|
||||
"/settings/system-general (Application branding card, Header preview)"
|
||||
"/proxmox/overview",
|
||||
"/alerts",
|
||||
"/docs/INSTALL",
|
||||
"/docs/UPGRADE_v6"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1280,
|
||||
"width": 1440,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 900,
|
||||
"height": 900
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"No custom logo and no custom name: the preview falls back to the default Pulse lockup",
|
||||
"Light colour scheme at 1280x900 and 390x844",
|
||||
"Dark colour scheme at 1280x900"
|
||||
"Expanded Disaster Recovery D node with Manage selected, unsaved and saved lifecycle selection, and disabled Saving controls.",
|
||||
"Future maintenance scheduled for 25 September with actual local start/end dates, edited expiry, cancelled window, and active maintenance with an actual future deadline.",
|
||||
"Mute, retired and active policy states, persisted maintenance and mute after a graceful same-data restart, and stable canonical resource identity.",
|
||||
"Active and absent node connectivity alert, open retained incident with ordered firing, resolution and re-fire events, expanded Evidence details, and empty disabled Save Note.",
|
||||
"Empty filtered History, persisted search after reload, cleared search results and intermediate-width filter controls.",
|
||||
"Signed installer code and helper-ownership guidance, with internal code scrolling and narrow document layout."
|
||||
],
|
||||
"interactions": [
|
||||
"Open Settings > General with the white_label capability present; assert the Header preview renders exactly one svg.pulse-brand-logo (blue disc, white ring, white centre dot) and no \"\u25cf\" text glyph",
|
||||
"Compare the preview mark with the page header lockup: both render PulseBrandMark at 20px",
|
||||
"Check the page does not scroll horizontally at 390px wide"
|
||||
"Show all 50 nodes, expand Disaster Recovery D, select Manage, change lifecycle, scroll to Save, save, return Active and save. Manage and unsaved selection survive the retained-row window shift and viewport changes.",
|
||||
"Create resource-only future maintenance, edit its end, cancel it, then create a one-hour active window. Inspect actual banner pixels at phone and desktop widths.",
|
||||
"Save Mute all attention with active maintenance, reload, gracefully restart the same final binary with unchanged data, and reload. Confirm identical canonical agent identity and policy, with the node connectivity alert absent.",
|
||||
"Cancel maintenance while keeping mute, verify the alert stays absent, then restore normal monitoring. Mute and resume the active occurrence once more and verify one open incident with its original start and ordered resolve/refire events.",
|
||||
"Open Monitoring menu, close with Escape and verify focus returns, reopen and dismiss with outside click. Inspect menu placement and clipping at phone width.",
|
||||
"Open Timeline and Evidence details, inspect occurrence and observation timestamps, verify empty Save Note is disabled without submitting a note.",
|
||||
"Filter History to no results, reload to verify persisted search, then clear it. Follow signed-installer and ownership links to their headings and inspect code and narrow layout."
|
||||
],
|
||||
"command": "node verify.mjs (Playwright against a private mock-mode backend on :7681 and Vite on :5199)",
|
||||
"notes": "Chromium via Playwright 1.56.1 from repos/pulse/node_modules. Real app and backend in PULSE_MOCK_MODE with ALLOW_ADMIN_BYPASS; the only stub is /api/license/runtime-capabilities with white_label added, because the mock install has no white-label entitlement and the card is gated behind it. Before the fix the same preview drew a blue circle containing a \"\u25cf\" text character instead of the Pulse brand mark, as seen in a provider-hosted client workspace on the walkthrough lab. Console errors were only the mock environment's failing update check."
|
||||
"command": "Playwright browser inspection against an isolated synthetic server on pulse-dev:7701 through a loopback SSH tunnel.",
|
||||
"notes": "Final backend SHA-256 77d17ff9e21ccdef3f5fff9f7b91b841abe3570bf8e04e1eda501c554e3a5889. Frontend index SHA-256 b9f1288bd188c8e549ccd4cdfa7aa75b56d49df3fc0f59de9e11dac90d449172. Actual pixels and DOM state were inspected. Repeated suppressed observations, expiry, delayed lifecycle replay and native-alias startup ordering are covered by failing-baseline regression tests and final race checks. The synthetic provider does not qualify production notification destinations or installed upgrade/rollback behavior."
|
||||
}
|
||||
|
||||
@@ -67,7 +67,12 @@ If you run Proxmox VE, the easiest and most “Pulse-native” deployment is the
|
||||
Replace `vX.Y.Z` with the exact release tag you want, then run this on your Proxmox host:
|
||||
|
||||
```bash
|
||||
(
|
||||
set -e
|
||||
export PULSE_VERSION=vX.Y.Z
|
||||
pulse_installer_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "$pulse_installer_dir"' EXIT
|
||||
cd "$pulse_installer_dir"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh.sshsig"
|
||||
ssh-keygen -Y verify \
|
||||
@@ -76,7 +81,7 @@ ssh-keygen -Y verify \
|
||||
-n pulse-install \
|
||||
-s install.sh.sshsig < install.sh
|
||||
bash install.sh --version "${PULSE_VERSION}"
|
||||
rm -f install.sh install.sh.sshsig
|
||||
)
|
||||
```
|
||||
|
||||
> **Note**: The GitHub `install.sh` is the **server** installer. The agent installer is served from your Pulse server at `/install.sh` (see **Settings → Infrastructure → Install on a host**). Do not use the GitHub server installer to install or update `pulse-agent`.
|
||||
@@ -144,7 +149,12 @@ See [KUBERNETES.md](KUBERNETES.md) for ingress and persistence configuration.
|
||||
For Linux servers (VM or bare metal), use the official installer:
|
||||
|
||||
```bash
|
||||
(
|
||||
set -e
|
||||
export PULSE_VERSION=vX.Y.Z
|
||||
pulse_installer_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "$pulse_installer_dir"' EXIT
|
||||
cd "$pulse_installer_dir"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh"
|
||||
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh.sshsig"
|
||||
ssh-keygen -Y verify \
|
||||
@@ -153,7 +163,7 @@ ssh-keygen -Y verify \
|
||||
-n pulse-install \
|
||||
-s install.sh.sshsig < install.sh
|
||||
sudo bash install.sh --version "${PULSE_VERSION}"
|
||||
rm -f install.sh install.sh.sshsig
|
||||
)
|
||||
```
|
||||
|
||||
> **Note**: This installs the Pulse server. Use the `/install.sh` endpoint from **Settings → Infrastructure → Install on a host** for installing or upgrading `pulse-agent` on monitored hosts.
|
||||
@@ -250,7 +260,15 @@ Pulse can update the server runtime to the latest stable version.
|
||||
|----------|---------|
|
||||
| **Docker** | `docker compose pull && docker compose up -d` |
|
||||
| **Kubernetes** | `helm repo update && helm upgrade pulse pulse/pulse -n pulse` |
|
||||
| **Systemd / Proxmox LXC** | `sudo /bin/update` |
|
||||
| **Systemd / Proxmox LXC with the Pulse-owned helper** | `sudo /bin/update` |
|
||||
|
||||
Use `/bin/update --version vX.Y.Z` for an exact target only when the helper was
|
||||
installed by the Pulse server installer. On Proxmox community-scripts
|
||||
containers, `/bin/update` can belong to a different updater that ignores
|
||||
`--version`. If the helper is absent or its owner is unknown, use the
|
||||
[signed server-installer flow](#2-bare-metal--systemd) with `PULSE_VERSION` set
|
||||
to the exact target tag. The same ownership check applies to rollback. After
|
||||
the service restarts, verify the installed version with `GET /api/version`.
|
||||
|
||||
Docker without Compose: `docker restart` keeps the old image running. Run `docker pull rcourtman/pulse:vX.Y.Z`, then `docker stop pulse && docker rm pulse` and re-run your original `docker run` command.
|
||||
|
||||
|
||||
@@ -148,13 +148,23 @@ Preferred path:
|
||||
|
||||
- **Settings → System → Updates**
|
||||
|
||||
If you prefer CLI, use the installed update helper for the target version:
|
||||
If you prefer CLI, use this command only when `/bin/update` was installed by
|
||||
the Pulse server installer:
|
||||
|
||||
```bash
|
||||
sudo /bin/update --version vX.Y.Z
|
||||
```
|
||||
|
||||
`/bin/update` is installed by the supported systemd and Proxmox LXC server installer. If your host does not have it yet, follow the signed server-installer flow in [INSTALL.md](INSTALL.md). Agent updates and v5-to-v6 agent upgrades still use the `/install.sh` command generated in **Settings → Infrastructure → Install on a host**; that screen is for both first installs and in-place agent upgrades.
|
||||
On Proxmox community-scripts containers, `/bin/update` can belong to the
|
||||
community-scripts updater, which can ignore `--version`. If that is your
|
||||
installation, the helper is absent, or you cannot confirm its owner, follow the
|
||||
[signed server-installer flow](INSTALL.md#2-bare-metal--systemd) and set
|
||||
`PULSE_VERSION` to the exact target tag. This also applies to rollback. Verify
|
||||
the installed version with `GET /api/version` after the service restarts.
|
||||
|
||||
Agent updates and v5-to-v6 agent upgrades use the `/install.sh` command generated
|
||||
in **Settings → Infrastructure → Install on a host**. That screen is for both
|
||||
first installs and in-place agent upgrades.
|
||||
|
||||
Operator note for builds after `v6.0.0-rc.2`: the historical Pulse update
|
||||
signer was not recovered. Hosts pinned to the `rc.2` trust root should not
|
||||
@@ -273,12 +283,16 @@ older v5 agents do not understand observer configuration.
|
||||
### Can I keep Pulse v5 stable while I test Pulse v6?
|
||||
|
||||
Yes. Keep a rollback path available while you evaluate v6. The final release
|
||||
on the v5 line is 5.1.36, so the stable rollback command is:
|
||||
on the v5 line is 5.1.36. On installations with the Pulse-owned update helper,
|
||||
the rollback command is:
|
||||
|
||||
```bash
|
||||
sudo /bin/update --version v5.1.36
|
||||
```
|
||||
|
||||
The [installer ownership requirements](#systemd-and-proxmox-lxc-installs) above
|
||||
also apply to this command.
|
||||
|
||||
### Why did my v5 install upgrade itself to v6?
|
||||
|
||||
Pulse 5.1.29 and later pin the built-in updater to the 5.1.x line and never
|
||||
@@ -286,12 +300,16 @@ offer v6, so upgrading from those versions is always a manual step. Pulse
|
||||
5.1.28 and older have no such pin: installs with auto-update enabled follow
|
||||
the newest stable GitHub release, which is now v6. If that happened to you,
|
||||
your data and configuration carry over; run through the Post-Upgrade
|
||||
Checklist above to confirm everything still works. To return to v5, run:
|
||||
Checklist above to confirm everything still works. To return to v5 with the
|
||||
Pulse-owned update helper, run:
|
||||
|
||||
```bash
|
||||
sudo /bin/update --version v5.1.36
|
||||
```
|
||||
|
||||
For community-scripts installations or an unknown helper owner, use the
|
||||
[installer ownership guidance](#systemd-and-proxmox-lxc-installs) above.
|
||||
|
||||
## Migration Notes (v6)
|
||||
|
||||
### Unified Navigation (Bookmarks and Deep Links)
|
||||
|
||||
@@ -27,7 +27,7 @@ import { describeResourceInventoryOwnership } from '@/utils/resourceMonitoringPo
|
||||
* - Lock the resource against automated remediation (action broker
|
||||
* refuses dispatch with resource_remediation_locked:)
|
||||
* - Schedule a maintenance window during which all findings on the
|
||||
* resource get auto-acknowledged with cause=maintenance_window
|
||||
* resource have their alert and Patrol attention paused
|
||||
*
|
||||
* The section stays compact and out of the way until the operator has
|
||||
* something to say about the resource — fresh-install resources see a
|
||||
@@ -576,18 +576,16 @@ export const ResourceOperatorStateSection: Component<ResourceOperatorStateSectio
|
||||
|
||||
<Show when={activeMaintenanceWindow()}>
|
||||
<div class="rounded border border-amber-200 bg-amber-50 px-3 py-2 text-xs text-amber-800 dark:border-amber-800 dark:bg-amber-900 dark:text-amber-200">
|
||||
<span class="font-semibold">Maintenance window active.</span> Findings raised on this
|
||||
resource
|
||||
<span class="font-semibold">Maintenance window active.</span> Attention on this resource
|
||||
<Show when={activeMaintenanceWindow()!.maintenanceScope === 'resource_and_descendants'}>
|
||||
{' '}
|
||||
and its descendants
|
||||
</Show>{' '}
|
||||
are auto-acknowledged until{' '}
|
||||
{formatRelativeTime(
|
||||
is paused until{' '}
|
||||
{new Date(
|
||||
activeMaintenanceWindow()!.maintenanceActiveEndAt ??
|
||||
activeMaintenanceWindow()!.maintenanceEndAt!,
|
||||
{ compact: true },
|
||||
)}
|
||||
).toLocaleString()}
|
||||
.
|
||||
<Show when={activeMaintenanceWindow()!.maintenanceReason}>
|
||||
<span class="block mt-0.5">Reason: {activeMaintenanceWindow()!.maintenanceReason}</span>
|
||||
@@ -597,11 +595,9 @@ export const ResourceOperatorStateSection: Component<ResourceOperatorStateSectio
|
||||
|
||||
<Show when={scheduledMaintenanceWindow() && !activeMaintenanceWindow()}>
|
||||
<div class="rounded border border-blue-200 bg-blue-50 px-3 py-2 text-xs text-blue-800 dark:border-blue-800 dark:bg-blue-900 dark:text-blue-200">
|
||||
<span class="font-semibold">Maintenance window scheduled.</span> Auto-acknowledgement will
|
||||
start{' '}
|
||||
{formatRelativeTime(scheduledMaintenanceWindow()!.maintenanceStartAt!, { compact: true })}{' '}
|
||||
and end{' '}
|
||||
{formatRelativeTime(scheduledMaintenanceWindow()!.maintenanceEndAt!, { compact: true })}.
|
||||
<span class="font-semibold">Maintenance window scheduled.</span> Attention will be paused
|
||||
from {new Date(scheduledMaintenanceWindow()!.maintenanceStartAt!).toLocaleString()} until{' '}
|
||||
{new Date(scheduledMaintenanceWindow()!.maintenanceEndAt!).toLocaleString()}.
|
||||
<Show when={scheduledMaintenanceWindow()!.maintenanceReason}>
|
||||
<span class="block mt-0.5">
|
||||
Reason: {scheduledMaintenanceWindow()!.maintenanceReason}
|
||||
|
||||
+24
@@ -92,3 +92,27 @@ describe('ResourceOperatorStateSection render with capabilityNames: null', () =>
|
||||
expect(screen.queryByText('Allowed actions')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('maintenance schedule timestamps', () => {
|
||||
it.each(['active', 'scheduled'] as const)(
|
||||
'shows absolute future times for a %s window',
|
||||
async (kind) => {
|
||||
const start = new Date(Date.now() + (kind === 'active' ? -60_000 : 3_600_000)).toISOString();
|
||||
const end = new Date(Date.now() + 7_200_000).toISOString();
|
||||
apiClientMock.apiFetchJSON.mockResolvedValue({
|
||||
canonicalId: `maintenance-${kind}`,
|
||||
maintenanceStartAt: start,
|
||||
maintenanceEndAt: end,
|
||||
});
|
||||
render(() => <ResourceOperatorStateSection resourceId={`maintenance-${kind}`} />);
|
||||
const badge = await screen.findByText(
|
||||
kind === 'active' ? 'Maintenance window active.' : 'Maintenance window scheduled.',
|
||||
);
|
||||
expect(badge.parentElement).toHaveTextContent(new Date(end).toLocaleString());
|
||||
if (kind === 'scheduled')
|
||||
expect(badge.parentElement).toHaveTextContent(new Date(start).toLocaleString());
|
||||
expect(badge.parentElement).not.toHaveTextContent('just now');
|
||||
expect(badge.parentElement).toHaveTextContent(/attention.*paused/i);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
-2
@@ -157,8 +157,6 @@ describe('ResourceOperatorStateSection', () => {
|
||||
// nothing once it ends.
|
||||
expect(sectionSource).toContain('scheduledMaintenanceWindow');
|
||||
expect(sectionSource).toContain('Maintenance window scheduled.');
|
||||
expect(sectionSource).toContain('Auto-acknowledgement will');
|
||||
expect(sectionSource).toContain("start{' '}");
|
||||
});
|
||||
|
||||
it('exposes Edit window and Cancel window controls when a window exists', () => {
|
||||
|
||||
@@ -18,19 +18,23 @@ export function PlatformWindowedList<Item>(props: PlatformWindowedListProps<Item
|
||||
enableThreshold: props.enableThreshold,
|
||||
windowSize: props.windowSize,
|
||||
});
|
||||
const renderItems = (items: readonly Item[], globalOffset: number) => (
|
||||
<For each={items}>{(item, index) => props.children(item, () => globalOffset + index())}</For>
|
||||
// Pass reactive inputs through to one For owner so overlapping windows
|
||||
// preserve the mounted item and any local interaction state.
|
||||
const renderItems = (items: Accessor<readonly Item[]>, globalOffset: Accessor<number>) => (
|
||||
<For each={items()}>
|
||||
{(item, index) => props.children(item, () => globalOffset() + index())}
|
||||
</For>
|
||||
);
|
||||
|
||||
return (
|
||||
<Show when={windowing.isWindowed()} fallback={renderItems(props.items(), 0)}>
|
||||
<Show when={windowing.isWindowed()} fallback={renderItems(props.items, () => 0)}>
|
||||
<div
|
||||
ref={windowing.setAnchorRef}
|
||||
aria-hidden="true"
|
||||
data-platform-window-spacer="top"
|
||||
style={{ height: `${windowing.topSpacerHeight()}px` }}
|
||||
/>
|
||||
{renderItems(windowing.visibleItems(), windowing.startIndex())}
|
||||
{renderItems(windowing.visibleItems, windowing.startIndex)}
|
||||
<div
|
||||
aria-hidden="true"
|
||||
data-platform-window-spacer="bottom"
|
||||
|
||||
@@ -132,16 +132,21 @@ export function PlatformWindowedRows<Row>(props: PlatformWindowedRowsProps<Row>)
|
||||
windowSize: props.windowSize,
|
||||
});
|
||||
|
||||
const renderRows = (items: readonly (Row | StablePlatformRow<Row>)[], globalOffset: number) => (
|
||||
<For each={items}>
|
||||
// Keep the For owner mounted when its window changes. Recreating it would
|
||||
// discard editors and expanded detail state in rows retained by both windows.
|
||||
const renderRows = (
|
||||
items: Accessor<readonly (Row | StablePlatformRow<Row>)[]>,
|
||||
globalOffset: Accessor<number>,
|
||||
) => (
|
||||
<For each={items()}>
|
||||
{(item, index) =>
|
||||
props.children(isStableRow(item) ? item.value() : item, () => globalOffset + index())
|
||||
props.children(isStableRow(item) ? item.value() : item, () => globalOffset() + index())
|
||||
}
|
||||
</For>
|
||||
);
|
||||
|
||||
return (
|
||||
<Show when={windowing.isWindowed()} fallback={renderRows(renderItems(), 0)}>
|
||||
<Show when={windowing.isWindowed()} fallback={renderRows(renderItems, () => 0)}>
|
||||
<tr
|
||||
ref={windowing.setAnchorRef}
|
||||
aria-hidden="true"
|
||||
@@ -154,7 +159,7 @@ export function PlatformWindowedRows<Row>(props: PlatformWindowedRowsProps<Row>)
|
||||
style={{ height: `${windowing.topSpacerHeight()}px` }}
|
||||
/>
|
||||
</tr>
|
||||
{renderRows(windowing.visibleItems(), windowing.startIndex())}
|
||||
{renderRows(windowing.visibleItems, windowing.startIndex)}
|
||||
<tr
|
||||
aria-hidden="true"
|
||||
data-platform-window-spacer="bottom"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { cleanup, render, screen } from '@solidjs/testing-library';
|
||||
import { cleanup, fireEvent, render, screen } from '@solidjs/testing-library';
|
||||
import { createSignal, onCleanup } from 'solid-js';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { PlatformWindowedRows } from '../PlatformWindowedRows';
|
||||
import { PlatformWindowedList } from '../PlatformWindowedList';
|
||||
@@ -89,6 +89,81 @@ describe('PlatformWindowedRows', () => {
|
||||
expect(input).toHaveValue('still editing');
|
||||
});
|
||||
|
||||
it.each(['table', 'list'] as const)(
|
||||
'preserves editing state in retained %s items when the scroll window moves',
|
||||
async (kind) => {
|
||||
let scrollTop = 0;
|
||||
const rect = vi
|
||||
.spyOn(Element.prototype, 'getBoundingClientRect')
|
||||
.mockImplementation(function (this: Element) {
|
||||
const top = this.getAttribute('data-platform-window-spacer') === 'top' ? -scrollTop : 0;
|
||||
return {
|
||||
x: 0,
|
||||
y: top,
|
||||
top,
|
||||
left: 0,
|
||||
right: 200,
|
||||
bottom: top + 40,
|
||||
width: 200,
|
||||
height: 40,
|
||||
toJSON: () => ({}),
|
||||
} as DOMRect;
|
||||
});
|
||||
const height = Object.getOwnPropertyDescriptor(window, 'innerHeight');
|
||||
Object.defineProperty(window, 'innerHeight', { configurable: true, value: 120 });
|
||||
const visibility = (Element.prototype as Element & { checkVisibility?: () => boolean })
|
||||
.checkVisibility;
|
||||
(Element.prototype as Element & { checkVisibility?: () => boolean }).checkVisibility = () =>
|
||||
true;
|
||||
const items = Array.from({ length: 30 }, (_, id) => ({ id }));
|
||||
try {
|
||||
const content = (item: { id: number }) => <input aria-label={`Draft ${item.id}`} />;
|
||||
const { container } = render(() =>
|
||||
kind === 'table' ? (
|
||||
<table>
|
||||
<tbody>
|
||||
<PlatformWindowedRows items={() => items} windowSize={8} enableThreshold={8}>
|
||||
{(item, index) => (
|
||||
<tr data-row={item.id} data-position={index()}>
|
||||
<td>{content(item)}</td>
|
||||
</tr>
|
||||
)}
|
||||
</PlatformWindowedRows>
|
||||
</tbody>
|
||||
</table>
|
||||
) : (
|
||||
<PlatformWindowedList items={() => items} windowSize={8} enableThreshold={8}>
|
||||
{(item, index) => (
|
||||
<div data-row={item.id} data-position={index()}>
|
||||
{content(item)}
|
||||
</div>
|
||||
)}
|
||||
</PlatformWindowedList>
|
||||
),
|
||||
);
|
||||
const draft = screen.getByRole('textbox', { name: 'Draft 4' }) as HTMLInputElement;
|
||||
await fireEvent.input(draft, { target: { value: 'keep this edit' } });
|
||||
scrollTop = 160;
|
||||
await fireEvent.scroll(window);
|
||||
expect(container.querySelector('[data-platform-window-spacer="top"]')).not.toHaveStyle({
|
||||
height: '0px',
|
||||
});
|
||||
expect(screen.getByRole('textbox', { name: 'Draft 4' })).toBe(draft);
|
||||
expect(draft).toHaveValue('keep this edit');
|
||||
expect(container.querySelector('[data-row="4"]')).toHaveAttribute('data-position', '4');
|
||||
expect(container.querySelectorAll('[data-row]')).toHaveLength(8);
|
||||
} finally {
|
||||
cleanup();
|
||||
rect.mockRestore();
|
||||
if (height) Object.defineProperty(window, 'innerHeight', height);
|
||||
if (visibility)
|
||||
(Element.prototype as Element & { checkVisibility?: () => boolean }).checkVisibility =
|
||||
visibility;
|
||||
else Reflect.deleteProperty(Element.prototype, 'checkVisibility');
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it('renders keyed rows once in their latest order', async () => {
|
||||
const [items, setItems] = createSignal([
|
||||
{ key: 'alpha', label: 'Alpha' },
|
||||
|
||||
@@ -205,6 +205,9 @@ func (s *IncidentStore) QueryIncidents(query IncidentQuery) (IncidentPage, error
|
||||
continue
|
||||
}
|
||||
started := incidentEventTimestamp(change)
|
||||
if explicit := incidentChangeStartedAt(change); !explicit.IsZero() {
|
||||
started = explicit
|
||||
}
|
||||
var occurrence *Incident
|
||||
for _, candidate := range byAlert[identifier] {
|
||||
if canonicalBoundary[candidate] {
|
||||
@@ -241,7 +244,16 @@ func (s *IncidentStore) QueryIncidents(query IncidentQuery) (IncidentPage, error
|
||||
continue
|
||||
}
|
||||
occurrence := firedOwners[change.ID]
|
||||
if occurrence == nil {
|
||||
explicitStart := incidentChangeStartedAt(change)
|
||||
if occurrence == nil && !explicitStart.IsZero() {
|
||||
for _, candidate := range byAlert[identifier] {
|
||||
if sameResource(candidate.ResourceID, change.ResourceID) && boundaries[candidate].Equal(explicitStart) {
|
||||
occurrence = candidate
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if occurrence == nil && explicitStart.IsZero() {
|
||||
for _, candidate := range byAlert[identifier] {
|
||||
if change.Kind != unifiedresources.ChangeCommandExecuted && change.Kind != unifiedresources.ChangeRunbookExecuted && !sameResource(candidate.ResourceID, change.ResourceID) {
|
||||
continue
|
||||
@@ -386,3 +398,18 @@ func incidentStartDelta(a, b time.Time) time.Duration {
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// Modern lifecycle events carry occurrence identity separately from transition
|
||||
// time. Refires keep the same occurrence even though they happen after recovery.
|
||||
// Legacy history without that identity retains its timestamp-based projection.
|
||||
func incidentChangeStartedAt(change unifiedresources.ResourceChange) time.Time {
|
||||
value, ok := change.Metadata[unifiedresources.MetadataAlertStartedAt].(string)
|
||||
if !ok {
|
||||
return time.Time{}
|
||||
}
|
||||
started, err := time.Parse(time.RFC3339Nano, value)
|
||||
if err != nil || started.After(incidentEventTimestamp(change)) {
|
||||
return time.Time{}
|
||||
}
|
||||
return started
|
||||
}
|
||||
|
||||
@@ -78,19 +78,20 @@ type Incident struct {
|
||||
}
|
||||
|
||||
type incidentShell struct {
|
||||
ID string
|
||||
AlertIdentifier string
|
||||
AlertType string
|
||||
Level string
|
||||
ResourceID string
|
||||
ResourceName string
|
||||
ResourceType string
|
||||
Node string
|
||||
Instance string
|
||||
Message string
|
||||
OpenedAt time.Time
|
||||
OccurrenceClosedAt *time.Time
|
||||
Events []IncidentEvent
|
||||
ID string
|
||||
AlertIdentifier string
|
||||
AlertType string
|
||||
Level string
|
||||
ResourceID string
|
||||
ResourceName string
|
||||
ResourceType string
|
||||
Node string
|
||||
Instance string
|
||||
Message string
|
||||
OpenedAt time.Time
|
||||
OccurrenceClosedAt *time.Time
|
||||
OccurrenceRefiredAt *time.Time
|
||||
Events []IncidentEvent
|
||||
}
|
||||
|
||||
type incidentJSON struct {
|
||||
@@ -115,24 +116,25 @@ type incidentJSON struct {
|
||||
}
|
||||
|
||||
type incidentShellJSON struct {
|
||||
ID string `json:"id"`
|
||||
AlertIdentifier string `json:"alertIdentifier"`
|
||||
AlertType string `json:"alertType"`
|
||||
Level string `json:"level"`
|
||||
ResourceID string `json:"resourceId"`
|
||||
ResourceName string `json:"resourceName"`
|
||||
ResourceType string `json:"resourceType,omitempty"`
|
||||
Node string `json:"node,omitempty"`
|
||||
Instance string `json:"instance,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
OpenedAt time.Time `json:"openedAt"`
|
||||
OccurrenceClosedAt *time.Time `json:"occurrenceClosedAt,omitempty"`
|
||||
Events []IncidentEvent `json:"events,omitempty"`
|
||||
Status IncidentStatus `json:"status,omitempty"`
|
||||
ClosedAt *time.Time `json:"closedAt,omitempty"`
|
||||
Acknowledged bool `json:"acknowledged,omitempty"`
|
||||
AckUser string `json:"ackUser,omitempty"`
|
||||
AckTime *time.Time `json:"ackTime,omitempty"`
|
||||
ID string `json:"id"`
|
||||
AlertIdentifier string `json:"alertIdentifier"`
|
||||
AlertType string `json:"alertType"`
|
||||
Level string `json:"level"`
|
||||
ResourceID string `json:"resourceId"`
|
||||
ResourceName string `json:"resourceName"`
|
||||
ResourceType string `json:"resourceType,omitempty"`
|
||||
Node string `json:"node,omitempty"`
|
||||
Instance string `json:"instance,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
OpenedAt time.Time `json:"openedAt"`
|
||||
OccurrenceClosedAt *time.Time `json:"occurrenceClosedAt,omitempty"`
|
||||
OccurrenceRefiredAt *time.Time `json:"occurrenceRefiredAt,omitempty"`
|
||||
Events []IncidentEvent `json:"events,omitempty"`
|
||||
Status IncidentStatus `json:"status,omitempty"`
|
||||
ClosedAt *time.Time `json:"closedAt,omitempty"`
|
||||
Acknowledged bool `json:"acknowledged,omitempty"`
|
||||
AckUser string `json:"ackUser,omitempty"`
|
||||
AckTime *time.Time `json:"ackTime,omitempty"`
|
||||
}
|
||||
|
||||
func (i Incident) MarshalJSON() ([]byte, error) {
|
||||
@@ -192,19 +194,20 @@ func (i *Incident) UnmarshalJSON(data []byte) error {
|
||||
|
||||
func (s incidentShell) MarshalJSON() ([]byte, error) {
|
||||
return json.Marshal(incidentShellJSON{
|
||||
ID: s.ID,
|
||||
AlertIdentifier: strings.TrimSpace(s.AlertIdentifier),
|
||||
AlertType: s.AlertType,
|
||||
Level: s.Level,
|
||||
ResourceID: s.ResourceID,
|
||||
ResourceName: s.ResourceName,
|
||||
ResourceType: s.ResourceType,
|
||||
Node: s.Node,
|
||||
Instance: s.Instance,
|
||||
Message: s.Message,
|
||||
OpenedAt: s.OpenedAt,
|
||||
OccurrenceClosedAt: s.OccurrenceClosedAt,
|
||||
Events: s.Events,
|
||||
ID: s.ID,
|
||||
AlertIdentifier: strings.TrimSpace(s.AlertIdentifier),
|
||||
AlertType: s.AlertType,
|
||||
Level: s.Level,
|
||||
ResourceID: s.ResourceID,
|
||||
ResourceName: s.ResourceName,
|
||||
ResourceType: s.ResourceType,
|
||||
Node: s.Node,
|
||||
Instance: s.Instance,
|
||||
Message: s.Message,
|
||||
OpenedAt: s.OpenedAt,
|
||||
OccurrenceClosedAt: s.OccurrenceClosedAt,
|
||||
OccurrenceRefiredAt: s.OccurrenceRefiredAt,
|
||||
Events: s.Events,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -217,19 +220,20 @@ func (s *incidentShell) UnmarshalJSON(data []byte) error {
|
||||
return err
|
||||
}
|
||||
*s = incidentShell{
|
||||
ID: payload.ID,
|
||||
AlertIdentifier: strings.TrimSpace(payload.AlertIdentifier),
|
||||
AlertType: payload.AlertType,
|
||||
Level: payload.Level,
|
||||
ResourceID: payload.ResourceID,
|
||||
ResourceName: payload.ResourceName,
|
||||
ResourceType: payload.ResourceType,
|
||||
Node: payload.Node,
|
||||
Instance: payload.Instance,
|
||||
Message: payload.Message,
|
||||
OpenedAt: payload.OpenedAt,
|
||||
OccurrenceClosedAt: payload.OccurrenceClosedAt,
|
||||
Events: cloneIncidentEvents(payload.Events),
|
||||
ID: payload.ID,
|
||||
AlertIdentifier: strings.TrimSpace(payload.AlertIdentifier),
|
||||
AlertType: payload.AlertType,
|
||||
Level: payload.Level,
|
||||
ResourceID: payload.ResourceID,
|
||||
ResourceName: payload.ResourceName,
|
||||
ResourceType: payload.ResourceType,
|
||||
Node: payload.Node,
|
||||
Instance: payload.Instance,
|
||||
Message: payload.Message,
|
||||
OpenedAt: payload.OpenedAt,
|
||||
OccurrenceClosedAt: payload.OccurrenceClosedAt,
|
||||
OccurrenceRefiredAt: payload.OccurrenceRefiredAt,
|
||||
Events: cloneIncidentEvents(payload.Events),
|
||||
}
|
||||
if s.OccurrenceClosedAt == nil && payload.ClosedAt != nil {
|
||||
s.OccurrenceClosedAt = cloneTime(*payload.ClosedAt)
|
||||
@@ -396,6 +400,36 @@ func (s *IncidentStore) RecordAlertFired(alert *alerts.Alert) {
|
||||
s.saveAsync()
|
||||
}
|
||||
|
||||
// RecordAlertRefired reopens a retained occurrence only for a newer explicit
|
||||
// refire. Its timestamp survives checkpoints so delayed lifecycle replay cannot
|
||||
// close it with the previous resolution or reopen it after a later resolution.
|
||||
func (s *IncidentStore) RecordAlertRefired(alert *alerts.Alert, occurredAt time.Time) {
|
||||
if alert == nil || occurredAt.IsZero() {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
shell := s.findLifecycleOccurrenceLocked(alert)
|
||||
if shell == nil {
|
||||
shell = newIncidentShellFromAlert(alert)
|
||||
s.incidents = append(s.incidents, shell)
|
||||
}
|
||||
if shell.OccurrenceRefiredAt != nil && !occurredAt.After(*shell.OccurrenceRefiredAt) {
|
||||
return
|
||||
}
|
||||
if shell.OccurrenceClosedAt != nil && !occurredAt.After(*shell.OccurrenceClosedAt) {
|
||||
return
|
||||
}
|
||||
shell.OccurrenceRefiredAt = cloneTime(occurredAt)
|
||||
shell.OccurrenceClosedAt = nil
|
||||
updateIncidentShellFromAlert(shell, alert)
|
||||
if !s.projectsFromCanonicalLocked() {
|
||||
s.addEventAtLocked(shell, IncidentEventAlertFired, occurredAt, formatAlertSummary(alert), nil)
|
||||
}
|
||||
s.trimLocked()
|
||||
s.saveAsync()
|
||||
}
|
||||
|
||||
// RecordAlertAcknowledged records an acknowledgement event for an alert.
|
||||
func (s *IncidentStore) RecordAlertAcknowledged(alert *alerts.Alert, user string) {
|
||||
if alert == nil {
|
||||
@@ -467,6 +501,9 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time
|
||||
now := time.Now()
|
||||
resolvedAt = now
|
||||
}
|
||||
if shell.OccurrenceRefiredAt != nil && resolvedAt.Before(*shell.OccurrenceRefiredAt) {
|
||||
return
|
||||
}
|
||||
shell.OccurrenceClosedAt = cloneTime(resolvedAt)
|
||||
|
||||
if !s.projectsFromCanonicalLocked() {
|
||||
@@ -523,7 +560,7 @@ func (s *IncidentStore) EnsureAlertOccurrence(alert *alerts.Alert, resolvedAt *t
|
||||
changed = true
|
||||
}
|
||||
|
||||
if resolvedAt != nil && !resolvedAt.IsZero() {
|
||||
if resolvedAt != nil && !resolvedAt.IsZero() && (shell.OccurrenceRefiredAt == nil || !resolvedAt.Before(*shell.OccurrenceRefiredAt)) {
|
||||
shell.OccurrenceClosedAt = cloneTime(*resolvedAt)
|
||||
if !hasIncidentEventType(shell.Events, IncidentEventAlertResolved) {
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, "Alert resolved", map[string]interface{}{
|
||||
@@ -1628,6 +1665,9 @@ func cloneIncidentShell(src *incidentShell) *incidentShell {
|
||||
t := *src.OccurrenceClosedAt
|
||||
clone.OccurrenceClosedAt = &t
|
||||
}
|
||||
if src.OccurrenceRefiredAt != nil {
|
||||
clone.OccurrenceRefiredAt = cloneTime(*src.OccurrenceRefiredAt)
|
||||
}
|
||||
clone.Events = cloneIncidentEvents(src.Events)
|
||||
return &clone
|
||||
}
|
||||
@@ -1657,7 +1697,7 @@ func normalizeIncidentShellState(shell *incidentShell) {
|
||||
sortIncidentEvents(shell.Events)
|
||||
if shell.OccurrenceClosedAt == nil {
|
||||
for i := len(shell.Events) - 1; i >= 0; i-- {
|
||||
if shell.Events[i].Type == IncidentEventAlertResolved {
|
||||
if shell.Events[i].Type == IncidentEventAlertResolved && (shell.OccurrenceRefiredAt == nil || !shell.Events[i].Timestamp.Before(*shell.OccurrenceRefiredAt)) {
|
||||
shell.OccurrenceClosedAt = cloneTime(shell.Events[i].Timestamp)
|
||||
break
|
||||
}
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
package memory
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestIncidentRefireCheckpointPreservesTransitionOrder(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
config := IncidentStoreConfig{DataDir: dir}
|
||||
store := NewIncidentStore(config)
|
||||
start := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
|
||||
alert := &alerts.Alert{ID: "retained-node", ResourceID: "node", StartTime: start}
|
||||
resolved := start.Add(time.Minute)
|
||||
refired := start.Add(2 * time.Minute)
|
||||
store.RecordAlertFired(alert)
|
||||
store.RecordAlertResolved(alert, resolved)
|
||||
store.RecordAlertRefired(alert, refired)
|
||||
store.flush()
|
||||
store = NewIncidentStore(config)
|
||||
t.Cleanup(store.flush)
|
||||
for i := 0; i < 10; i++ {
|
||||
store.RecordAlertFired(alert)
|
||||
store.RecordAlertResolved(alert, resolved)
|
||||
store.RecordAlertRefired(alert, refired)
|
||||
}
|
||||
store.EnsureAlertOccurrence(alert, &resolved)
|
||||
current := store.GetTimelineByAlertAt(alert.ID, start)
|
||||
require.NotNil(t, current)
|
||||
require.Equal(t, IncidentStatusOpen, current.Status)
|
||||
require.Nil(t, current.ClosedAt)
|
||||
require.Len(t, current.Events, 3)
|
||||
finalResolution := start.Add(3 * time.Minute)
|
||||
store.RecordAlertResolved(alert, finalResolution)
|
||||
store.flush()
|
||||
store = NewIncidentStore(config)
|
||||
t.Cleanup(store.flush)
|
||||
store.RecordAlertRefired(alert, refired)
|
||||
current = store.GetTimelineByAlertAt(alert.ID, start)
|
||||
require.Equal(t, IncidentStatusResolved, current.Status)
|
||||
require.Equal(t, &finalResolution, current.ClosedAt)
|
||||
require.Len(t, current.Events, 4)
|
||||
}
|
||||
@@ -463,7 +463,9 @@ func (m *Manager) restoreActiveAlertSnapshots(alerts []*Alert, source string, re
|
||||
}
|
||||
seen[alert.ID] = true
|
||||
|
||||
m.setActiveAlertNoLock(alert.ID, alert)
|
||||
if !m.setActiveAlertNoLock(alert.ID, alert) {
|
||||
continue
|
||||
}
|
||||
if legacyID != alert.ID {
|
||||
if m.activeAlertAlias == nil {
|
||||
m.activeAlertAlias = make(map[string]string)
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
|
||||
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/storagehealth"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
// Exercise the detector entrypoints, not just the active map: rejecting an
|
||||
// alert must also reject its history, firing event and notification intent.
|
||||
func TestOperatorSuppressionPreventsFiringSideEffects(t *testing.T) {
|
||||
const resourceID = "storage:tank"
|
||||
evaluators := map[string]func(*testing.T, *Manager){
|
||||
"provider-incident": func(t *testing.T, m *Manager) {
|
||||
m.SyncUnifiedResourceIncidents([]unifiedresources.Resource{admissionTestResource()})
|
||||
},
|
||||
"metric": func(t *testing.T, m *Manager) {
|
||||
m.checkMetric(resourceID, "tank", "nas", "nas", "storage", "cpu", 95,
|
||||
&HysteresisThreshold{Trigger: 80, Clear: 70}, nil)
|
||||
},
|
||||
"canonical-metric": func(t *testing.T, m *Manager) {
|
||||
spec := alertspecs.ResourceAlertSpec{
|
||||
ID: "metric-threshold:cpu", ResourceID: resourceID,
|
||||
ResourceType: unifiedresources.ResourceTypeStorage,
|
||||
Kind: alertspecs.AlertSpecKindMetricThreshold, Severity: alertspecs.AlertSeverityWarning,
|
||||
MetricThreshold: &alertspecs.MetricThresholdSpec{Metric: "cpu", Trigger: 80, Direction: alertspecs.ThresholdDirectionAbove},
|
||||
}
|
||||
m.evaluateCanonicalMetricAlert(spec, "tank", "nas", "nas", "storage", 95,
|
||||
&HysteresisThreshold{Trigger: 80, Clear: 70}, nil)
|
||||
},
|
||||
"lifecycle": func(t *testing.T, m *Manager) {
|
||||
spec, err := buildCanonicalDiscreteStateSpec(resourceID, "Pool state", unifiedresources.ResourceTypeStorage,
|
||||
AlertLevelWarning, 1, false, "health", []string{"degraded"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.evaluateCanonicalLifecycleAlert(canonicalLifecycleAlertParams{
|
||||
Spec: spec, Evidence: alertspecs.AlertEvidence{ObservedAt: time.Now(),
|
||||
DiscreteState: &alertspecs.DiscreteStateEvidence{StateKey: "health", Observed: "degraded"}},
|
||||
AlertID: spec.ID, AlertType: "zfs-pool-state", ResourceID: resourceID,
|
||||
ResourceName: "tank", AddToRecent: true, AddToHistory: true,
|
||||
})
|
||||
},
|
||||
"stateful": func(t *testing.T, m *Manager) {
|
||||
m.syncCanonicalHealthAssessmentAlert(canonicalHealthAssessmentAlertParams{
|
||||
SpecID: "pool-health", Signal: "zfs_pool", Codes: zfsPoolAssessmentCodes,
|
||||
Reasons: []storagehealth.Reason{{Code: "zfs_pool_state", Severity: storagehealth.RiskCritical, Summary: "Pool is degraded"}},
|
||||
AlertID: "pool-health", AlertType: "zfs-pool-state", SpecResourceID: resourceID,
|
||||
ResourceID: resourceID, ResourceName: "tank", ResourceType: unifiedresources.ResourceTypeStorage,
|
||||
})
|
||||
},
|
||||
}
|
||||
start, end := time.Now().Add(-time.Hour), time.Now().Add(time.Hour)
|
||||
for name, policy := range map[string]OperatorIntentContext{
|
||||
"muted": {MonitoringMode: "muted"},
|
||||
"retired": {LifecycleState: "retired"},
|
||||
"maintenance": {MaintenanceStartAt: &start, MaintenanceEndAt: &end},
|
||||
} {
|
||||
for family, evaluate := range evaluators {
|
||||
t.Run(name+"/"+family, func(t *testing.T) {
|
||||
m := newEventLogManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
m.SetAlertCallback(func(*Alert) {})
|
||||
setPolicy := func(current OperatorIntentContext) {
|
||||
m.SetOperatorIntentContextResolver(func(string, time.Time) (OperatorIntentContext, bool) {
|
||||
return current, true
|
||||
})
|
||||
}
|
||||
setPolicy(policy)
|
||||
for range 20 {
|
||||
evaluate(t, m)
|
||||
}
|
||||
if got := len(m.GetActiveAlerts()); got != 0 {
|
||||
t.Fatalf("suppressed active alerts = %d", got)
|
||||
}
|
||||
if got := len(m.historyManager.GetAllHistory(100)); got != 0 {
|
||||
t.Fatalf("suppressed history entries = %d", got)
|
||||
}
|
||||
if events := queryAlertEvents(t, m, eventlog.Filter{}); len(events) != 0 {
|
||||
t.Fatalf("suppressed detector produced lifecycle/delivery events: %+v", events)
|
||||
}
|
||||
m.mu.RLock()
|
||||
recent := len(m.recentAlerts)
|
||||
m.mu.RUnlock()
|
||||
if recent != 0 {
|
||||
t.Fatalf("suppressed recent alerts = %d", recent)
|
||||
}
|
||||
|
||||
// Expiry or removal must admit the still-present condition once.
|
||||
if name == "maintenance" {
|
||||
expired := time.Now().Add(-time.Minute)
|
||||
setPolicy(OperatorIntentContext{MaintenanceStartAt: &start, MaintenanceEndAt: &expired})
|
||||
} else {
|
||||
setPolicy(OperatorIntentContext{})
|
||||
}
|
||||
for range 20 {
|
||||
evaluate(t, m)
|
||||
}
|
||||
if got := len(m.GetActiveAlerts()); got != 1 {
|
||||
t.Fatalf("active alerts after unsuppression = %d, want 1", got)
|
||||
}
|
||||
events := queryAlertEvents(t, m, eventlog.Filter{Types: []string{eventlog.TypeFired, eventlog.TypeRefired}})
|
||||
if len(events) != 1 {
|
||||
t.Fatalf("firing events after unsuppression = %d, want 1", len(events))
|
||||
}
|
||||
|
||||
setPolicy(OperatorIntentContext{MonitoringMode: "muted"})
|
||||
if got := m.ReconcileResourceOperatorState(resourceID); got != 1 {
|
||||
t.Fatalf("newly muted active alerts cleared = %d, want 1", got)
|
||||
}
|
||||
before := len(queryAlertEvents(t, m, eventlog.Filter{}))
|
||||
for range 20 {
|
||||
evaluate(t, m)
|
||||
}
|
||||
if got := len(m.GetActiveAlerts()); got != 0 {
|
||||
t.Fatalf("muted alert reappeared after reconciliation: %d", got)
|
||||
}
|
||||
if got := len(queryAlertEvents(t, m, eventlog.Filter{})); got != before {
|
||||
t.Fatalf("suppressed re-evaluation added events after reconciliation: %d -> %d", before, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSuppressedAlertRestoreCannotReviveAcknowledgementOrEscalation(t *testing.T) {
|
||||
m := newEventLogManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
m.SetAlertCallback(func(*Alert) {})
|
||||
resource := admissionTestResource()
|
||||
for range 3 {
|
||||
m.SyncUnifiedResourceIncidents([]unifiedresources.Resource{resource})
|
||||
}
|
||||
active := m.GetActiveAlerts()
|
||||
if len(active) != 1 {
|
||||
t.Fatalf("unsuppressed control alerts = %d, want 1", len(active))
|
||||
}
|
||||
if err := m.AcknowledgeAlert(active[0].ID, "operator"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
snapshot := m.GetActiveAlerts()
|
||||
|
||||
restored := newEventLogManager(t)
|
||||
configureUnifiedEvalManager(t, restored, unifiedEvalBaseConfig())
|
||||
restored.SetAlertCallback(func(*Alert) {})
|
||||
restored.SetOperatorIntentContextResolver(func(string, time.Time) (OperatorIntentContext, bool) {
|
||||
return OperatorIntentContext{LifecycleState: "retired"}, true
|
||||
})
|
||||
if err := restored.restoreActiveAlertSnapshots([]*Alert{&snapshot[0]}, "test restart", true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resource.Incidents[0].Severity = storagehealth.RiskCritical
|
||||
for range 20 {
|
||||
restored.SyncUnifiedResourceIncidents([]unifiedresources.Resource{resource})
|
||||
}
|
||||
if len(restored.GetActiveAlerts()) != 0 || len(queryAlertEvents(t, restored, eventlog.Filter{})) != 0 {
|
||||
t.Fatal("suppressed restore or escalation produced an active alert or event")
|
||||
}
|
||||
restored.SetOperatorIntentContextResolver(nil)
|
||||
for range 20 {
|
||||
restored.SyncUnifiedResourceIncidents([]unifiedresources.Resource{resource})
|
||||
}
|
||||
active = restored.GetActiveAlerts()
|
||||
if len(active) != 1 || active[0].Acknowledged || active[0].Level != AlertLevelCritical {
|
||||
t.Fatalf("unsuppressed condition did not create a fresh critical alert: %+v", active)
|
||||
}
|
||||
if events := queryAlertEvents(t, restored, eventlog.Filter{Types: []string{eventlog.TypeFired}}); len(events) != 1 {
|
||||
t.Fatalf("firing events after restore and unsuppression = %d, want 1", len(events))
|
||||
}
|
||||
}
|
||||
|
||||
func admissionTestResource() unifiedresources.Resource {
|
||||
return unifiedresources.Resource{
|
||||
ID: "storage:tank", Type: unifiedresources.ResourceTypeStorage, Name: "tank",
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceTrueNAS},
|
||||
Storage: &unifiedresources.StorageMeta{Platform: "truenas", Topology: "pool", Protection: "zfs", IsZFS: true},
|
||||
Incidents: []unifiedresources.ResourceIncident{{Provider: "truenas", NativeID: "native-1",
|
||||
Code: "truenas_volume_status", Severity: storagehealth.RiskWarning, Summary: "Pool is degraded"}},
|
||||
}
|
||||
}
|
||||
@@ -100,13 +100,15 @@ func (m *Manager) hasActiveAlertNoLock(id string) bool {
|
||||
return ok
|
||||
}
|
||||
|
||||
func (m *Manager) setActiveAlertNoLock(storageKey string, alert *Alert) {
|
||||
// setActiveAlertNoLock reports whether the alert was admitted. Detector callers
|
||||
// must check admission before recording a firing lifecycle or dispatching it.
|
||||
func (m *Manager) setActiveAlertNoLock(storageKey string, alert *Alert) bool {
|
||||
if storageKey == "" || alert == nil {
|
||||
return
|
||||
return false
|
||||
}
|
||||
backfillCanonicalIdentity(alert)
|
||||
if suppressed, _ := m.operatorSuppressionForAlertNoLock(alert, time.Now().UTC()); suppressed {
|
||||
return
|
||||
return false
|
||||
}
|
||||
ensureOperationalContract(alert, time.Now())
|
||||
requestedKey := storageKey
|
||||
@@ -127,6 +129,7 @@ func (m *Manager) setActiveAlertNoLock(storageKey string, alert *Alert) {
|
||||
if m.historyManager != nil {
|
||||
m.historyManager.UpdateAlertOperationalContractForAlert(alert)
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (m *Manager) unregisterActiveAlertAliasNoLock(storageKey string, alert *Alert) {
|
||||
|
||||
@@ -495,7 +495,11 @@ func (m *Manager) evaluateCanonicalLifecycleAlert(params canonicalLifecycleAlert
|
||||
alert.AckTime = nil
|
||||
alert.AckUser = ""
|
||||
}
|
||||
m.setActiveAlertNoLock(storageKey, alert)
|
||||
if !m.setActiveAlertNoLock(storageKey, alert) {
|
||||
result.State.State = alertspecs.AlertStateSuppressed
|
||||
result.State.Reason = "operator-suppressed"
|
||||
return result, true
|
||||
}
|
||||
if params.AddToRecent {
|
||||
m.recentAlerts[trackingKey] = alert
|
||||
}
|
||||
@@ -761,7 +765,13 @@ func (m *Manager) evaluateCanonicalStatefulAlert(params canonicalStatefulAlertPa
|
||||
applyCanonicalIdentity(alert, params.Spec.ID, string(params.Spec.Kind))
|
||||
applyCanonicalOperationalEvidence(alert, params.Spec, params.Evidence, time.Now())
|
||||
m.preserveAlertState(storageKey, alert)
|
||||
m.setActiveAlertNoLock(storageKey, alert)
|
||||
if !m.setActiveAlertNoLock(storageKey, alert) {
|
||||
result.State.State = alertspecs.AlertStateSuppressed
|
||||
result.State.Reason = "operator-suppressed"
|
||||
result.State.ActiveSince = time.Time{}
|
||||
result.Transition = nil
|
||||
return result, true
|
||||
}
|
||||
if params.AddToRecent {
|
||||
m.recentAlerts[trackingKey] = alert
|
||||
}
|
||||
|
||||
@@ -295,7 +295,9 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
|
||||
alert.AckTime = nil
|
||||
alert.AckUser = ""
|
||||
}
|
||||
m.setActiveAlertNoLock(storageKey, alert)
|
||||
if !m.setActiveAlertNoLock(storageKey, alert) {
|
||||
return
|
||||
}
|
||||
m.recentAlerts[trackingKey] = alert
|
||||
m.historyManager.AddAlert(*alert)
|
||||
m.recordAlertEvent(eventlog.TypeFired, alert, storageKey, "metric-threshold", message, nil)
|
||||
@@ -360,6 +362,9 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
|
||||
applyCanonicalIdentity(existingAlert, spec.ID, string(spec.Kind))
|
||||
applyCanonicalOperationalEvidence(existingAlert, spec, evidence, time.Now())
|
||||
|
||||
if !m.setActiveAlertNoLock(storageKey, existingAlert) {
|
||||
return
|
||||
}
|
||||
shouldRenotify := false
|
||||
if existingAlert.Acknowledged {
|
||||
} else if m.shouldNotifyAfterCooldown(existingAlert) {
|
||||
@@ -375,7 +380,6 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
|
||||
existingAlert.LastNotified = nil
|
||||
}
|
||||
}
|
||||
m.setActiveAlertNoLock(storageKey, existingAlert)
|
||||
default:
|
||||
if !exists || existingAlert == nil {
|
||||
return
|
||||
|
||||
@@ -192,7 +192,9 @@ func (m *Manager) migrateGuestAlertNoLock(storageKey, specID, kind, resourceID,
|
||||
}
|
||||
applyCanonicalIdentity(matchedAlert, specID, kind)
|
||||
|
||||
m.setActiveAlertNoLock(storageKey, matchedAlert)
|
||||
if !m.setActiveAlertNoLock(storageKey, matchedAlert) {
|
||||
return nil
|
||||
}
|
||||
m.moveAlertTrackingStateNoLock(oldTrackingKey, storageKey, matchedAlert)
|
||||
|
||||
if matchCount > 1 {
|
||||
|
||||
@@ -411,7 +411,9 @@ func (m *Manager) checkMetric(resourceID, resourceName, node, instance, resource
|
||||
if exists && existingAlert != nil {
|
||||
// Update existing alert
|
||||
applyCanonicalIdentity(existingAlert, canonicalSpecID, string(alertspecs.AlertSpecKindMetricThreshold))
|
||||
m.setActiveAlertNoLock(canonicalStateID, existingAlert)
|
||||
if !m.setActiveAlertNoLock(canonicalStateID, existingAlert) {
|
||||
return
|
||||
}
|
||||
existingAlert.LastSeen = time.Now()
|
||||
existingAlert.Value = value
|
||||
if dn := m.resolveNodeDisplayName(existingAlert.Instance, existingAlert.Node); dn != "" {
|
||||
@@ -537,7 +539,9 @@ func (m *Manager) checkMetric(resourceID, resourceName, node, instance, resource
|
||||
alert.AckUser = ""
|
||||
}
|
||||
trackingKey = canonicalTrackingKeyOrFallback(alert, canonicalStateID)
|
||||
m.setActiveAlertNoLock(canonicalStateID, alert)
|
||||
if !m.setActiveAlertNoLock(canonicalStateID, alert) {
|
||||
return
|
||||
}
|
||||
m.recentAlerts[trackingKey] = alert
|
||||
m.historyManager.AddAlert(*alert)
|
||||
m.recordAlertEvent(eventlog.TypeFired, alert, canonicalStateID, "metric-threshold", message, nil)
|
||||
|
||||
@@ -294,7 +294,9 @@ func (m *Manager) SyncUnifiedResourceIncidents(resources []unifiedresources.Reso
|
||||
existing.Evidence = appendOperationalEvidence(existing.Evidence, envelope.Clone())
|
||||
}
|
||||
applyCanonicalIdentity(existing, alert.CanonicalSpecID, alert.CanonicalKind)
|
||||
m.setActiveAlertNoLock(storageKey, existing)
|
||||
if !m.setActiveAlertNoLock(storageKey, existing) {
|
||||
continue
|
||||
}
|
||||
// Like metric alerts, an existing provider incident becoming critical
|
||||
// must notify again. Keep its lifecycle and all delivery policy gates;
|
||||
// unchanged severity and downgrades must not create notification noise.
|
||||
@@ -307,7 +309,9 @@ func (m *Manager) SyncUnifiedResourceIncidents(resources []unifiedresources.Reso
|
||||
}
|
||||
|
||||
m.preserveAlertState(storageKey, alert)
|
||||
m.setActiveAlertNoLock(storageKey, alert)
|
||||
if !m.setActiveAlertNoLock(storageKey, alert) {
|
||||
continue
|
||||
}
|
||||
m.recentAlerts[canonicalTrackingKeyForAlert(alert)] = alert
|
||||
m.historyManager.AddAlert(*alert)
|
||||
m.recordAlertEvent(eventlog.TypeFired, alert, storageKey, "unified-incident", alert.Message, nil)
|
||||
|
||||
@@ -2647,10 +2647,10 @@ func generateDockerHosts(config MockConfig) []models.DockerHost {
|
||||
|
||||
host := models.DockerHost{
|
||||
ID: hostID,
|
||||
AgentID: fmt.Sprintf("agent-%s", randomHexString(6)),
|
||||
AgentID: fmt.Sprintf("agent-%s", mockStableHexString(16, "docker-agent", hostID)),
|
||||
Hostname: hostname,
|
||||
DisplayName: humanizeHostDisplayName(hostname),
|
||||
MachineID: randomHexString(32),
|
||||
MachineID: mockStableHexString(32, "docker-host-machine", hostID),
|
||||
OS: dockerOperatingSystems[rand.Intn(len(dockerOperatingSystems))],
|
||||
KernelVersion: dockerKernelVersions[rand.Intn(len(dockerKernelVersions))],
|
||||
Architecture: dockerArchitectures[rand.Intn(len(dockerArchitectures))],
|
||||
@@ -3913,7 +3913,7 @@ func buildMockLinkedHostFromKubernetesNode(
|
||||
IntervalSeconds: 30,
|
||||
LastSeen: lastSeen,
|
||||
AgentVersion: hostAgentVersions[rand.Intn(len(hostAgentVersions))],
|
||||
MachineID: randomHexString(32),
|
||||
MachineID: mockStableHexString(32, "linked-host-machine", hostID),
|
||||
Tags: tags,
|
||||
}
|
||||
host.NetInRate = SampleMetric("agent", host.ID, "netin", now)
|
||||
@@ -4191,7 +4191,7 @@ func buildMockLinkedHostFromNode(node models.Node, hostID string, hostIndex int,
|
||||
IntervalSeconds: 30,
|
||||
LastSeen: lastSeen,
|
||||
AgentVersion: hostAgentVersions[rand.Intn(len(hostAgentVersions))],
|
||||
MachineID: randomHexString(32),
|
||||
MachineID: mockStableHexString(32, "linked-host-machine", hostID),
|
||||
Tags: []string{"mock", "proxmox-node"},
|
||||
LinkedNodeID: node.ID,
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
func TestBuildFixtureStateIncludesDockerHosts(t *testing.T) {
|
||||
@@ -730,9 +731,12 @@ func collectFixtureIdentities(graph FixtureGraph) map[string][]string {
|
||||
for _, host := range state.Hosts {
|
||||
add("host", host.ID)
|
||||
add("host-hostname", host.Hostname)
|
||||
add("host-machine-id", host.ID+"="+host.MachineID)
|
||||
}
|
||||
for _, dockerHost := range state.DockerHosts {
|
||||
add("docker-host", dockerHost.ID)
|
||||
add("docker-host-machine-id", dockerHost.ID+"="+dockerHost.MachineID)
|
||||
add("docker-agent-id", dockerHost.ID+"="+dockerHost.AgentID)
|
||||
for _, c := range dockerHost.Containers {
|
||||
add("docker-container", c.ID)
|
||||
}
|
||||
@@ -763,6 +767,13 @@ func collectFixtureIdentities(graph FixtureGraph) map[string][]string {
|
||||
add("physical-disk", disk.ID)
|
||||
}
|
||||
|
||||
resources, _ := graph.UnifiedResourceSnapshot()
|
||||
for _, resource := range resources {
|
||||
if resource.Type == unifiedresources.ResourceTypeAgent {
|
||||
add("canonical-agent", resource.ID)
|
||||
}
|
||||
}
|
||||
|
||||
for category := range ids {
|
||||
sort.Strings(ids[category])
|
||||
}
|
||||
@@ -778,6 +789,9 @@ func TestFixtureIdentityStableAcrossBoots(t *testing.T) {
|
||||
|
||||
first := collectFixtureIdentities(buildFixtureGraph(DefaultConfig, now))
|
||||
second := collectFixtureIdentities(buildFixtureGraph(DefaultConfig, now))
|
||||
if len(first["canonical-agent"]) == 0 || len(second["canonical-agent"]) == 0 {
|
||||
t.Fatal("fixture graph must project canonical agents before comparing their identities")
|
||||
}
|
||||
|
||||
categories := map[string]struct{}{}
|
||||
for category := range first {
|
||||
|
||||
@@ -1139,3 +1139,69 @@ func TestMonitorLifecycleReplayPreservesOccurrenceTimelines(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitorLifecycleRefireReopensRetainedOccurrence(t *testing.T) {
|
||||
for _, canonical := range []bool{false, true} {
|
||||
name := "local"
|
||||
if canonical {
|
||||
name = "canonical"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
store := unifiedresources.NewMemoryStore()
|
||||
incidents := memory.NewIncidentStore(memory.IncidentStoreConfig{})
|
||||
m := &Monitor{incidentStore: incidents, resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store))}
|
||||
if canonical {
|
||||
incidents.SetResourceTimelineStore(m.resourceStore.(memory.IncidentTimelineStore))
|
||||
}
|
||||
start := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
|
||||
alert := &alerts.Alert{ID: "node-connectivity", ResourceID: "node", StartTime: start}
|
||||
fire := alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: start, Alert: alert}
|
||||
resolve := alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: start.Add(time.Minute), Alert: alert}
|
||||
refire := alerts.LifecycleEvent{Type: eventlog.TypeRefired, OccurredAt: start.Add(2 * time.Minute), Alert: alert}
|
||||
m.handleAlertLifecycleEvent(fire)
|
||||
m.handleAlertLifecycleEvent(resolve)
|
||||
original := incidents.GetTimelineByAlertAt(alert.ID, start)
|
||||
require.Equal(t, memory.IncidentStatusResolved, original.Status)
|
||||
m.handleAlertLifecycleEvent(refire)
|
||||
assertOpen := func() {
|
||||
t.Helper()
|
||||
current := incidents.GetTimelineByAlertAt(alert.ID, start)
|
||||
require.Equal(t, original.ID, current.ID)
|
||||
require.Equal(t, memory.IncidentStatusOpen, current.Status)
|
||||
require.Nil(t, current.ClosedAt)
|
||||
require.Len(t, current.Events, 3)
|
||||
require.Len(t, incidents.ListIncidentsByResource(alert.ResourceID, 0), 1)
|
||||
}
|
||||
assertOpen()
|
||||
if canonical {
|
||||
// Canonical history must recover the retained occurrence even when
|
||||
// no incident checkpoint survives.
|
||||
recovered := memory.NewIncidentStore(memory.IncidentStoreConfig{})
|
||||
recovered.SetResourceTimelineStore(m.resourceStore.(memory.IncidentTimelineStore))
|
||||
page, err := recovered.QueryIncidents(memory.IncidentQuery{ResourceID: alert.ResourceID})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, page.Incidents, 1)
|
||||
require.Equal(t, memory.IncidentStatusOpen, page.Incidents[0].Status)
|
||||
require.Equal(t, start, page.Incidents[0].OpenedAt)
|
||||
require.Len(t, page.Incidents[0].Events, 3)
|
||||
}
|
||||
for i := 0; i < 10; i++ {
|
||||
m.handleAlertLifecycleEvent(fire)
|
||||
m.handleAlertLifecycleEvent(resolve)
|
||||
m.handleAlertLifecycleEvent(refire)
|
||||
}
|
||||
assertOpen()
|
||||
// Historical read repair must not close an occurrence that re-fired.
|
||||
incidents.EnsureAlertOccurrence(alert, &resolve.OccurredAt)
|
||||
assertOpen()
|
||||
finalResolve := resolve
|
||||
finalResolve.OccurredAt = start.Add(3 * time.Minute)
|
||||
m.handleAlertLifecycleEvent(finalResolve)
|
||||
m.handleAlertLifecycleEvent(refire)
|
||||
current := incidents.GetTimelineByAlertAt(alert.ID, start)
|
||||
require.Equal(t, memory.IncidentStatusResolved, current.Status)
|
||||
require.Equal(t, &finalResolve.OccurredAt, current.ClosedAt)
|
||||
require.Len(t, current.Events, 4)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -95,6 +95,11 @@ func (m *Monitor) installOperatorIntentResolver(store ResourceStoreInterface) {
|
||||
}
|
||||
return context, found || context.MaintenanceEndAt != nil
|
||||
})
|
||||
// Alerts restore before the resource store is attached during startup.
|
||||
// Reconcile that restored set as soon as persisted policy is available.
|
||||
if m.alertManager.ReconcileOperatorIntentState() > 0 && m.state != nil {
|
||||
m.SyncAlertState()
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Monitor) resolveBackupIntentContext(_ string, instance, node string, vmid int, now time.Time) (alerts.BackupIntentContext, bool) {
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
package monitoring
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
func TestInstallOperatorIntentResolverReconcilesRestoredAlerts(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
start, end := now.Add(-time.Hour), now.Add(time.Hour)
|
||||
for name, policy := range map[string]unifiedresources.ResourceOperatorState{
|
||||
"muted": {MonitoringMode: unifiedresources.MonitoringModeMuted},
|
||||
"retired": {LifecycleState: unifiedresources.LifecycleStateRetired},
|
||||
"maintenance": {MaintenanceStartAt: &start, MaintenanceEndAt: &end},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
fixture := []alerts.Alert{}
|
||||
store := unifiedresources.NewMemoryStore()
|
||||
registry := unifiedresources.NewRegistry(store)
|
||||
for _, id := range []string{"vm:101", "vm:202"} {
|
||||
registry.IngestResources([]unifiedresources.Resource{{ID: id, Type: unifiedresources.ResourceTypeVM}})
|
||||
fixture = append(fixture, alerts.Alert{
|
||||
ID: id + "::metric-threshold:cpu", ResourceID: id, ResourceName: id,
|
||||
Type: "cpu", Level: alerts.AlertLevelWarning, StartTime: start, LastSeen: now,
|
||||
})
|
||||
}
|
||||
policy.CanonicalID = "vm:101"
|
||||
if err := store.SetResourceOperatorState(policy); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
payload, err := json.Marshal(fixture)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Join(dir, "alerts"), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "alerts", "active-alerts.json"), payload, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Runtime startup restores durable alerts before the resource store
|
||||
// and its persisted operator policies are attached to the monitor.
|
||||
manager := alerts.NewManagerWithDataDir(dir, alerts.WithDurableAlertStore())
|
||||
t.Cleanup(manager.Stop)
|
||||
if got := len(manager.GetActiveAlerts()); got != 2 {
|
||||
t.Fatalf("restored alerts before policy attachment = %d, want 2", got)
|
||||
}
|
||||
monitor := &Monitor{alertManager: manager, state: models.NewState()}
|
||||
monitor.state.UpdateActiveAlerts(monitor.activeAlertsSnapshot())
|
||||
monitor.installOperatorIntentResolver(unifiedresources.NewMonitorAdapter(registry))
|
||||
active := manager.GetActiveAlerts()
|
||||
if len(active) != 1 || active[0].ResourceID != "vm:202" {
|
||||
t.Fatalf("policy attachment retained suppressed alerts or cleared the control: %+v", active)
|
||||
}
|
||||
if got := monitor.state.GetSnapshot().ActiveAlerts; len(got) != 1 || got[0].ResourceID != "vm:202" {
|
||||
t.Fatalf("shared state retained a suppressed restored alert: %+v", got)
|
||||
}
|
||||
manager.Stop()
|
||||
restarted := alerts.NewManagerWithDataDir(dir, alerts.WithDurableAlertStore())
|
||||
t.Cleanup(restarted.Stop)
|
||||
active = restarted.GetActiveAlerts()
|
||||
if len(active) != 1 || active[0].ResourceID != "vm:202" {
|
||||
t.Fatalf("suppressed alert returned after another restart: %+v", active)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePublicationReconcilesRestoredNativeAlertAliases(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
store, err := unifiedresources.NewSQLiteResourceStore(dir, "default")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
snapshot := models.StateSnapshot{
|
||||
Nodes: []models.Node{{ID: "pve-node", Name: "pve1", Instance: "pve", Status: "online", LinkedAgentID: "host-1"}},
|
||||
Hosts: []models.Host{{ID: "host-1", Hostname: "pve1", MachineID: "machine-1", LinkedNodeID: "pve-node", Status: "online", LastSeen: time.Now()}},
|
||||
}
|
||||
beforeRestart := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store))
|
||||
beforeRestart.PopulateFromSnapshot(snapshot)
|
||||
canonicalID, found := beforeRestart.ResolveCanonicalResourceID("pve-node")
|
||||
if !found || canonicalID == "pve-node" {
|
||||
t.Fatalf("expected a distinct merged canonical agent ID, got %q, found=%v", canonicalID, found)
|
||||
}
|
||||
if err := store.SetResourceOperatorState(unifiedresources.ResourceOperatorState{
|
||||
CanonicalID: canonicalID, MonitoringMode: unifiedresources.MonitoringModeMuted,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := store.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store, err = unifiedresources.NewSQLiteResourceStore(dir, "default")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = store.Close() })
|
||||
|
||||
now := time.Now().UTC()
|
||||
var restored []alerts.Alert
|
||||
for _, id := range []string{"pve-node", "agent:host-1", "unaffected"} {
|
||||
restored = append(restored, alerts.Alert{
|
||||
ID: id + "::connectivity", ResourceID: id, ResourceName: id,
|
||||
Type: "connectivity", Level: alerts.AlertLevelWarning, StartTime: now.Add(-time.Hour), LastSeen: now,
|
||||
})
|
||||
}
|
||||
payload, err := json.Marshal(restored)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Join(dir, "alerts"), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "alerts", "active-alerts.json"), payload, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
manager := alerts.NewManagerWithDataDir(dir, alerts.WithDurableAlertStore())
|
||||
t.Cleanup(manager.Stop)
|
||||
if got := len(manager.GetActiveAlerts()); got != 3 {
|
||||
t.Fatalf("restored alerts = %d, want 3", got)
|
||||
}
|
||||
monitor := &Monitor{alertManager: manager, state: models.NewState()}
|
||||
// Startup can attach the policy store before any resource observation.
|
||||
// The persisted canonical policy cannot yet resolve these native aliases.
|
||||
monitor.SetResourceStore(unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)))
|
||||
if got := len(manager.GetActiveAlerts()); got != 3 {
|
||||
t.Fatalf("alerts before resource publication = %d, want 3", got)
|
||||
}
|
||||
monitor.updateResourceStore(snapshot)
|
||||
active := manager.GetActiveAlerts()
|
||||
if len(active) != 1 || active[0].ResourceID != "unaffected" {
|
||||
t.Fatalf("resource publication did not reconcile restored aliases: %+v", active)
|
||||
}
|
||||
if got := monitor.state.GetSnapshot().ActiveAlerts; len(got) != 1 || got[0].ResourceID != "unaffected" {
|
||||
t.Fatalf("shared state retained suppressed native aliases: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -97,6 +97,10 @@ func (m *Monitor) syncUnifiedResourceAlertsToState(resources []unifiedresources.
|
||||
return
|
||||
}
|
||||
|
||||
// Resource publication can make persisted canonical policy resolvable for
|
||||
// restored native-ID alerts that preceded the first observation at startup.
|
||||
m.alertManager.ReconcileOperatorIntentState()
|
||||
|
||||
config := m.alertManager.GetConfig()
|
||||
migrationPlan := alerts.PlanAlertIdentityMigration(config, resources)
|
||||
if migrationPlan.UnsupportedVersion {
|
||||
|
||||
@@ -335,7 +335,11 @@ func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {
|
||||
switch event.Type {
|
||||
case eventlog.TypeFired, eventlog.TypeRefired:
|
||||
if m.incidentStore != nil {
|
||||
m.incidentStore.RecordAlertFired(timelineAlert)
|
||||
if event.Type == eventlog.TypeRefired {
|
||||
m.incidentStore.RecordAlertRefired(timelineAlert, event.OccurredAt)
|
||||
} else {
|
||||
m.incidentStore.RecordAlertFired(timelineAlert)
|
||||
}
|
||||
}
|
||||
occurredAt := event.OccurredAt
|
||||
if event.Type == eventlog.TypeFired && !alert.StartTime.IsZero() {
|
||||
@@ -566,6 +570,7 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
|
||||
|
||||
change := unifiedresources.BuildAlertTimelineChange(alert.ResourceID, kind, occurredAt, actor, unifiedresources.AlertTimelineChange{
|
||||
AlertIdentifier: alert.ID,
|
||||
AlertStartedAt: alert.StartTime,
|
||||
AlertType: alert.Type,
|
||||
AlertLevel: string(alert.Level),
|
||||
AlertMessage: alert.Message,
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
|
||||
const (
|
||||
MetadataAlertIdentifier = "alert_identifier"
|
||||
MetadataAlertStartedAt = "alert_started_at"
|
||||
MetadataAlertType = "alert_type"
|
||||
MetadataAlertLevel = "alert_level"
|
||||
MetadataAlertMessage = "alert_message"
|
||||
@@ -30,6 +31,7 @@ const resourceChangeOutputExcerptLimit = 500
|
||||
// durable in the canonical resource history rather than only incident memory.
|
||||
type AlertTimelineChange struct {
|
||||
AlertIdentifier string
|
||||
AlertStartedAt time.Time
|
||||
AlertType string
|
||||
AlertLevel string
|
||||
AlertMessage string
|
||||
@@ -69,6 +71,9 @@ func BuildAlertTimelineChange(resourceID string, kind ChangeKind, occurredAt tim
|
||||
MetadataAlertThreshold: alert.AlertThreshold,
|
||||
},
|
||||
}
|
||||
if !alert.AlertStartedAt.IsZero() {
|
||||
change.Metadata[MetadataAlertStartedAt] = alert.AlertStartedAt.UTC().Format(time.RFC3339Nano)
|
||||
}
|
||||
for key, value := range cloneChangeMetadata(alert.AlertMetadata) {
|
||||
if _, exists := change.Metadata[key]; exists {
|
||||
continue
|
||||
|
||||
@@ -481,17 +481,29 @@ def sanitize_release_notes(raw_text: str, version: str) -> str:
|
||||
return _collapse_blank_lines(text)
|
||||
|
||||
|
||||
def server_installer_fallback(target: str, docs_version: str) -> str:
|
||||
return (
|
||||
"On Proxmox community-scripts containers, or if you cannot confirm who installed "
|
||||
"`/bin/update`, follow the "
|
||||
f"[signed server-installer instructions](https://github.com/rcourtman/Pulse/blob/v{docs_version}/docs/INSTALL.md#2-bare-metal--systemd) "
|
||||
f"with `PULSE_VERSION={target}`. The community-scripts updater can ignore `--version`."
|
||||
)
|
||||
|
||||
|
||||
def build_installation_section(version: str) -> str:
|
||||
return "\n".join(
|
||||
[
|
||||
"## Install",
|
||||
"",
|
||||
"For systemd and Proxmox LXC installs, use **Settings → System → Updates** or:",
|
||||
"For systemd and Proxmox LXC server updates, use **Settings → System → Updates**. "
|
||||
"Use this CLI command only when `/bin/update` was installed by the Pulse server installer:",
|
||||
"",
|
||||
"```bash",
|
||||
f"sudo /bin/update --version v{version}",
|
||||
"```",
|
||||
"",
|
||||
server_installer_fallback(f"v{version}", version),
|
||||
"",
|
||||
"For Docker:",
|
||||
"",
|
||||
"```bash",
|
||||
@@ -514,10 +526,15 @@ def build_rollback_section(args: argparse.Namespace) -> str:
|
||||
"",
|
||||
f"The rollback target is `{args.rollback_target}`:",
|
||||
"",
|
||||
"For systemd and Proxmox LXC servers, use this command only when "
|
||||
"`/bin/update` was installed by the Pulse server installer:",
|
||||
"",
|
||||
"```bash",
|
||||
args.rollback_command,
|
||||
"```",
|
||||
"",
|
||||
server_installer_fallback(args.rollback_target, args.version),
|
||||
"",
|
||||
"For Docker Compose, set the Pulse image to the rollback target and recreate the container.",
|
||||
]
|
||||
)
|
||||
|
||||
@@ -548,6 +548,18 @@ Old metadata section.
|
||||
self.assertIn("https://pulserelay.pro/download.html", body)
|
||||
self.assertIn("The rollback target is `v5.1.28`", body)
|
||||
self.assertIn("sudo /bin/update --version v5.1.28", body)
|
||||
for section, target in (
|
||||
(render_release_body.build_installation_section(namespace.version), "v6.0.0-rc.2"),
|
||||
(render_release_body.build_rollback_section(namespace), "v5.1.28"),
|
||||
):
|
||||
self.assertLess(section.index("only when"), section.index("sudo /bin/update"))
|
||||
self.assertIn("installed by the Pulse server installer", section)
|
||||
self.assertIn("community-scripts updater can ignore `--version`", section)
|
||||
self.assertIn(f"`PULSE_VERSION={target}`", section)
|
||||
self.assertIn(
|
||||
"/blob/v6.0.0-rc.2/docs/INSTALL.md#2-bare-metal--systemd", section,
|
||||
)
|
||||
self.assertNotIn("/blob/main/", section)
|
||||
self.assertIn(
|
||||
"For Docker Compose, set the Pulse image to the rollback target",
|
||||
body,
|
||||
@@ -577,6 +589,7 @@ Controls remain readable without horizontal scrolling.
|
||||
"Args",
|
||||
(),
|
||||
{
|
||||
"version": "6.4.0",
|
||||
"rollback_target": "v6.3.2",
|
||||
"rollback_command": "sudo /bin/update --version v6.3.2",
|
||||
},
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Execute the published installer recipes with controlled transport failures."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
class SignedInstallerDocsTest(unittest.TestCase):
|
||||
def test_download_and_signature_failures_never_execute_installer(self):
|
||||
blocks = [
|
||||
block for block in re.findall(r"```bash\n(.*?)\n```", (ROOT / "docs/INSTALL.md").read_text(), re.S)
|
||||
if "ssh-keygen -Y verify" in block
|
||||
]
|
||||
self.assertEqual(len(blocks), 2, "Exercise both LXC and systemd recipes")
|
||||
for recipe, block in enumerate(blocks):
|
||||
for failure in ("installer-download", "signature-download", "signature", "none"):
|
||||
with self.subTest(recipe=recipe, failure=failure), tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
commands = root / "bin"
|
||||
commands.mkdir()
|
||||
scripts = {
|
||||
"curl": '''#!/bin/bash
|
||||
set -e
|
||||
url="${!#}"
|
||||
case "$url" in
|
||||
*/install.sh)
|
||||
[[ "$FAILURE" != installer-download ]] || exit 22
|
||||
printf '%s\\n' '#!/bin/bash' 'printf "%s\\n" "$@" > "$INSTALL_RECEIPT"' > install.sh
|
||||
;;
|
||||
*/install.sh.sshsig)
|
||||
[[ "$FAILURE" != signature-download ]] || exit 22
|
||||
printf '%s\\n' fixture-signature > install.sh.sshsig
|
||||
;;
|
||||
*) exit 23 ;;
|
||||
esac
|
||||
''',
|
||||
"ssh-keygen": '#!/bin/bash\n[[ "$FAILURE" != signature ]]\n',
|
||||
"sudo": '#!/bin/bash\nexec "$@"\n',
|
||||
}
|
||||
for name, source in scripts.items():
|
||||
executable = commands / name
|
||||
executable.write_text(source)
|
||||
executable.chmod(0o755)
|
||||
receipt = root / "executed"
|
||||
env = {**os.environ, "PATH": f"{commands}:{os.environ['PATH']}",
|
||||
"FAILURE": failure, "INSTALL_RECEIPT": str(receipt)}
|
||||
result = subprocess.run(["bash", "-c", block.replace("vX.Y.Z", "v6.4.5-rc.2")],
|
||||
cwd=root, env=env, capture_output=True, text=True)
|
||||
if failure == "none":
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(receipt.read_text(), "--version\nv6.4.5-rc.2\n")
|
||||
else:
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertFalse(receipt.exists(), "Unverified installer was executed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user