Use unique provider-observed node interface addresses to restore automatic
host-agent linking when standalone sites share a short hostname but use
distinct FQDN endpoints. Keep reused addresses ambiguous and fail closed.
Refs #1753
Contract-Neutral: narrows internal agent-to-node identity reconciliation without changing API or extension shapes
Serialize monitoring-owned host, Docker, and Kubernetes token cleanup with API-token persistence. Restore the complete live inventory when a reduced inventory cannot commit so restart never silently reverses an apparent revocation, and govern the boundary with forced-failure proof.
Change-source: pulse-maintainer
Keep in-memory observations authoritative over duplicate persisted history, add direct provider and cache reset coverage, and register the rolling-window proof files with canonical governance.\n\nChange-source: pulse-maintainer
Contract-Neutral: monitor construction only moves alerts-owned durable-store initialization before alert workers; monitoring and host-agent behavior are unchanged
Sort backup-server and datastore rows deterministically so provider refresh order cannot reset expanded detail state. Exclude optical devices from PVE, legacy agent, and canonical physical-disk projections.
Fixes#1785Fixes#1786
Contract-Neutral: Corrects refresh ordering and non-disk filtering without changing public APIs or subsystem contracts.
Change-source: pulse-maintainer
Allow one retiring-agent health window after a trusted replacement install token is minted, while preserving clone-safe forks for ambiguous, conflicting, or longer-lived identities. Retire superseded token bindings and document the remaining migration gap for identities already forked.
Change-source: pulse-maintainer
Phase 1 capstone of docs/ALERT_ENGINE_EVOLUTION.md. The deterministic
reducer now runs continuously inside the live manager against the same
production observations: the canonical lifecycle path (connectivity,
powered-state, discrete-state kinds) via a deferred hook that replays
each evaluation — including the resolved intent context — through the
reducer; the poll-driven recovery paths (PBS/PMG/storage, node,
connection-degraded); and manual acknowledge/unacknowledge/clear. The
feed seeds from active canonical alerts at enable so restarts do not
read as mass divergence.
Every state disagreement is counted (Manager.ShadowDivergences) and
recorded in the alert event log as a shadow_divergence event with both
engines' states, rate-limited to one report per key per ten minutes.
After each divergence the reducer resyncs to the manager, so one
divergence yields one event — including divergences caused by manager
mutations the feed does not observe. Appends never block evaluation and
a disabled feed is a nil-check no-op.
This converts the parity harnesses' test-time guarantee into an
always-on invariant; the production divergence rate becomes the
go/no-go evidence for each Phase 2 family cutover. The full
activation/ack/recovery/re-fire cycle runs divergence-free in tests.
Bind monitor-side event-log bootstrap to its canonical boundary and exercise the new event infrastructure through the registry-recognized alerts, API, monitoring, and agent-lifecycle proof files.
Change-source: pulse-maintainer
Adds internal/alerts/eventlog: a SQLite-backed, additive event log that
records lifecycle transitions (resolved, acknowledged, unacknowledged,
escalated, flapping detected) and notification decisions (dispatched,
deferred by quiet hours, suppressed — with the mechanism that held
them). Appends never block alert evaluation: a full buffer drops the
event and counts the drop; a store that fails to open degrades to
recording nothing. 90-day retention, hourly prune.
The manager emits at the existing funnels only — dispatchAlert and the
safe-call resolve/ack/escalate seams — so no lifecycle behavior
changes. Lifecycle "fired" is deliberately not recorded yet: the
active-alert store funnel also runs on persisted restore, so firing
waits for the explicit activation seam in a later phase. The monitoring
bootstrap enables the log per manager; ephemeral managers and tests
record nothing unless they opt in.
GET /api/alerts/events (monitoring:read) reads the log with
alertIdentifier/type/since/limit filters, newest first.
Phase 0 of docs/ALERT_ENGINE_EVOLUTION.md (coverage gap
alert-engine-suppression-observability).
Contract-Neutral: Freezes mock sampler cadence only inside the existing cache regression and updates release-preflight assertions to recognize current conditional runner routing and shard-degradation behavior; runtime and public contracts are unchanged.
Change-source: pulse-maintainer
Third increment of the resource-payload-static-metadata gap, the
snapshot-side statics. Measurement first: the mock estate ships only 5
distinct non-default policy postures across 723 resources and 56
distinct templated AI-safe summary strings across the same rows
(~150KB of inline duplication per snapshot), and every canonical alias
list carries the row's own wire id verbatim.
Non-default policies and AI-safe summaries now dedupe through the same
content-addressed catalog contract as capabilities: state-level
policyCatalog and aiSafeSummaryCatalog hold the distinct blobs,
resources carry policyRef / aiSafeSummaryRef, and ingestion expands
refs back to the inline shape (default posture synthesis unchanged for
ref-less rows). The broadcast conversion returns one catalogs struct so
the next deduped field is a loop entry, and catalog changes ride the
same delta frame as the refs that need them. Alias lists stop shipping
the row's own wire id — every identity consumer unions resource.id
beside the alias spread — while the primary id stays so aliases remain
the complete live vocabulary a host is reachable by, per the
established slimming invariant.
Verified against a 1857-resource estate: zero inline policies or
summaries on the wire (743 refs resolving into 1KB + 7KB catalogs),
zero alias lists carrying the wire id, catalog-expanded postures and
summaries reaching the store byte-equivalently, and the estate
rendering clean on desktop and the 36-row phone window.
Remaining gap scope: canonicalIdentity alias/superseded history is
genuine per-resource identity data (~600KB per snapshot) with no safe
derivation shortcut, and the proxmox-facet/platformData mirror
duplication (~200KB) is deliberate canonical-merge fabric; both stay
recorded under resource-payload-static-metadata for a future owner.
The retention and max-points trims re-sliced the series front
(metrics[i:]), which keeps the entire backing array reachable while
hiding the trimmed prefix from cap(). After the dense mock trends seed
(~2,800 points/series) aged out of its 6h window, every series held a
few hundred live points while pinning its seed-sized array - roughly
700MB across the demo estate's ~8,000 series. HeapSys drifted over the
droplet's memory.high, kernel reclaim throttling starved the accept
loop, and demo.pulserelay.pro served 502s (2026-08-25 outage; verified
live via pprof: appendMetric-attributed live heap grew +117MB in six
minutes after restart while goroutines stayed at 80).
Trims now compact in place so cap() reflects the true array size, and
releaseTrimmedCapacity copies the window into a right-sized array when
it occupies under a quarter of the backing array. A healthy sliding
window sits between 1x and 2x capacity and never pays the copy.
Regression tests pin both paths: a live append after an aged-out seed
backfill, and Cleanup on a mostly-expired series, including an aliasing
assertion that catches the cap()-blind front re-slice.
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).
The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.
Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.
Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.
/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.
Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
allowHostAgentReenrollLocked trusted only the durable continuity store
when one was attached, so a removal block living only in persisted
monitor state (written by releases predating the store) could never be
cleared: every fresh re-enroll read as already-consumed and the host
was rejected with 400 on every report indefinitely, regardless of new
tokens or reinstalls.
Refs #1772
With the adaptive scheduler disabled (the default), every planning pass
on the main poll tick rebuilt each instance task with NextRun=now and
Upsert overwrote the pending slot, so availability targets polled at
the tick cadence regardless of their configured interval. Reproduced
at exactly 10s polling on a 60s ICMP target; c70431caa only covered
the adaptive-scheduler path. Planning now preserves a pending future
slot, tightening it only when a freshly shortened interval justifies
an earlier run.
Refs #1745
The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.
The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
Discovering Docker inside Proxmox LXC guests previously required an
undocumented environment variable plus a hand-edited systemd unit,
which is how the surface kept shipping invisible. The opt-in is now a
persisted system setting with an admin-only toggle in Settings →
System → General, presented as its own "Docker in Proxmox LXCs" card
under the existing Docker/Podman updates card, with the requirements
(node agent with command execution) and data-collection bounds stated
inline.
The setting rides the admin-gated system settings endpoint
(RequireAdmin + settings:write, matching every other server-wide
toggle). Flipping it persists first, then applies to the runtime
config and fires a reconfigure hook so the router rebuilds the
monitor's Docker checker and inventory collector immediately — no
restart. PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY remains
authoritative: when set it locks the toggle (EnvironmentLockBadge in
the UI, HTTP 400 from the API), and an unrelated settings save can
never clobber the runtime value. The opt-in guardrail test now pins
the settings path, the env lock, and the persisted field.
Installer and node setup-guide callouts now point at the toggle first
with the environment variable as the locking override, and the docs,
their public mirrors, and the agent-lifecycle subsystem contract say
the same. New i18n strings shipped for en/de/es.
Verified live in mock mode: toggle off/on from the UI drives the
collector teardown/setup log lines, the value survives a server
restart, and all three touched surfaces render at desktop and mobile
widths (receipt in frontend-modern/browser-verification.json).
An in-progress PBS snapshot (reproduced on VM 117 at 9% written) was
counted as a completed backup: the guest's LastBackup jumped to "now" the
moment vzdump started, and the workloads badge showed a green healthy
shield for a backup that did not exist yet. If the job aborted, that
shield had never been true.
Detect the in-flight state from what the PBS listing actually says - no
size and no index.json.blob manifest in files - and flag the snapshot
InProgress instead of complete. On the PVE side, correlate storage content
entries with live vzdump tasks: a guest archive whose ctime falls inside a
running task's window is that task's partial output, and a PBS-storage
entry without a size has no manifest yet.
In-flight artifacts now feed a new per-guest BackupInProgress flag rather
than LastBackup, so the badge age always reflects the latest COMPLETED
backup. The workloads badge gets a distinct blue "Running" state (green
would claim success, None would hide that work is underway), with the
last-completed age kept in the tooltip and accessible label. Recovery
points map to OutcomeRunning with no completion time, keeping posture
freshness and the Backups page's latest-restore pointers anchored to
completed backups; the recoverable-artifact list shows a Running chip.
A cached in-flight snapshot forces a per-poll re-fetch of its group so
completion or abort is noticed promptly instead of after the cache TTL.
The pct exec socket probe that discovers Docker inside LXC guests
failed invisibly: errors logged at debug only, and a failed probe never
set DockerCheckedAt, so the guest stayed in first-check state and was
re-probed on every poll forever. On a node where lxc-attach could not
enter unprivileged guests that meant ~49 pct execs per minute of
steady-state churn, with nothing at default log level to say why the
Proxmox page showed no Docker.
Track consecutive probe failures per guest in the monitor. The first
failure of a streak, and the point where the backoff reaches its cap,
log at warn with the guest, node, and error; repeats stay at debug.
Failed probes retry with exponential backoff from one minute to a
thirty-minute ceiling, a success clears the streak with an info log,
and reconfiguring the checker resets all streaks so command execution
enabled from the UI at runtime still retries immediately. Entries not
refreshed for a day belong to deleted guests and are pruned.
PULSE_PROXMOX_GUEST_DOCKER_INVENTORY_VMIDS now gates the socket probe
as well as inventory collection. It previously only limited inventory,
so opting into specific guests still probed every running LXC on every
cycle; now guests outside the allowlist are never pct exec'd at all.
Subsystem contract and configuration docs updated to match.
An agent whose token authenticated seconds ago cannot be in a
credential outage, yet the fleet doctor flagged it critical with
"missing or revoked" whenever the token id missed the server's token
registry view, and the offered repair-authentication loop cannot fix a
stale server-side registry view. When the subject's own row shows the
judged token authenticating within the freshness window, emit a
warning-severity registry-stale verdict naming the id and the last
authentication instead. Genuinely stale rows keep the critical verdict.
Refs #1730
The Proxmox node sweep removes any non-preserved alert whose Node is
empty, and system alerts have no node, so every sweep silently deleted
the notification-delivery alert. Its five-minute evaluation then
re-raised it as a brand-new alert, firing a fresh notification each
cycle with no recovery in between, which reads as an alert appearing,
vanishing without a recovery, and paging again minutes later. System
alerts are now preserved outside node cleanup.
Raises also carried the delivery counts inside the message, and a
message change re-notifies, so each newly retained failure re-paged
even while the condition was unchanged. System alerts now take an
optional fingerprint: while level and fingerprint hold, a re-raise
refreshes the message and metadata silently. The delivery alert
fingerprints on status and failure classes, keeping its counter text
current without paging on drift.
Refs #1721
Contract-Neutral: behavioral fix: delivery-health system alert survived node sweep and stops re-paging on count drift, no public contract delta (Refs #1721)
Pulse could tell that its own delivery had stopped and had no way to say so.
Queue health was computed for one API endpoint and rendered on the alerts
surfaces, and that was the whole escalation path: a destination that stops
delivering cannot announce itself through a notification, so an operator who
does not open the alerts page learns nothing.
Telemetry: 78 installs retry every notification to the maxAttempts ceiling and
deliver almost nothing, 155,967 attempts against 221 deliveries in a week. 50
have delivered nothing in 30 days and 28 more delivered normally at some point
and silently stopped. The dominant terminal failure classes are authentication
and configuration, so most are fixable in minutes once someone knows.
Move the queue-health rule into internal/notifications as ClassifyQueueHealth
and DeliveryHealth, next to the queue that produces the counts, and delegate
the API handler to it so the endpoint and the new evaluator cannot drift into
two rules. A queue that cannot be read reports unavailable rather than healthy,
because silence is the failure mode being guarded against.
Monitoring evaluates delivery health on the poll ticker and raises or clears
the notification-delivery system alert, which puts it in the alert list and the
navigation badge. Reading queue health costs a SQLite query and the poll
cadence can be seconds, so the evaluation is throttled to five minutes.
RaiseSystemAlert is idempotent for an unchanged condition, so the timer neither
re-notifies nor accumulates alerts.
The alert is a warning rather than critical. It is real, but a new
critical-by-default alert appearing on upgrade across the affected installs is
a bigger change than this should make on its own.
The new Monitor field is deliberately named to fit inside the existing struct
alignment column. A longer name makes gofmt re-pad the whole block, which
breaks the canonical guardrail tests that pin those field declarations
verbatim.
Proof: delivery_health_test.go pins that in-flight work and retries stay
healthy while retained terminal failures do not, and that an unreadable queue
is never reported healthy. system_alerts_test.go pins the message for each
outcome including singular and plural, that it points at the destinations
surface, and that the throttle honours its interval. internal/monitoring,
internal/notifications and internal/alerts suites all pass.
Contract-Neutral: Staged contract deltas cover every boundary this change actually moves: notifications.md gains delivery_health.go plus the ClassifyQueueHealth rule, and monitoring.md gains system_alerts.go plus the system-alert evaluation extension point. The residual demands are inapplicable. agent-lifecycle is demanded only because monitor.go is one of its canonical files, and its verification artifact is host-agent deletion and re-enrollment proof, which this change does not touch: the diff to monitor.go is one struct field and one call in the poll loop. api-contracts and storage-recovery are demanded only because internal/api/notifications.go falls under a broad internal/api reference, and that file's change is a pure delegation of an internal helper to notifications.ClassifyQueueHealth with a byte-identical HTTP response and no route or payload delta.
The thresholds UI stores per-resource overrides keyed by unified
registry resource IDs, while the engine check paths look up legacy
monitor IDs, so a per-node override on an API-only PVE node saved,
displayed as Custom, and never applied: the node kept alerting at the
global defaults. Reproduced live against mock: memory at 81% with a 42%
override stored under the registry ID raised nothing.
Route the node, PMG, PBS, and host/linked-node override lookups through
the canonical registry identity resolver the intent-policy path already
uses, with the exact legacy key still winning so overrides stored by
older UI versions keep resolving unchanged.
Proof: TestNodeThresholdOverrideStoredUnderRegistryIDApplies drives the
real populate path for an API-only node, stores a memory override under
the registry resource ID the UI uses, and asserts CheckNode fires at the
override threshold while controls without the override or without the
resolver stay quiet.
Refs #1738
Contract-Neutral: behavioral fix#1738: threshold override lookup resolves registry identity, no public contract delta
A Critical "credential missing or revoked" verdict never said which
credential it judged, so a host row whose last-seen token id went stale
after re-enrollment reads as a live outage even while the agent is
reporting fine on a fresh token, and the operator has nothing to map
against the token list. Emit the judged token record UUID (not secret
material) alongside the missing and expired verdicts.
Refs #1730
Contract-Neutral: diagnostic evidence detail only in fleet doctor credential verdicts, refs #1730
Pulse Test