Record durable persistence as the token-revocation commit boundary across API, security, agent-lifecycle, and storage contracts. Route the exact-removal and rollback proofs through the canonical token lifecycle suite.
Change-source: pulse-maintainer
Roll back the in-memory token inventory and return an error when durable persistence fails. Cover exact deletion from a multi-token inventory and the persistence-failure rollback path.
Change-source: pulse-maintainer
Bind monitor-side event-log bootstrap to its canonical boundary and exercise the new event infrastructure through the registry-recognized alerts, API, monitoring, and agent-lifecycle proof files.
Change-source: pulse-maintainer
Bind the delivery diagnosis, active-card presentation, and additive event log to their canonical contracts and recognized API/frontend tests. Sync the shipped API reference, record current browser evidence, and restore sorted truthful control-plane status evidence.
Change-source: pulse-maintainer
Adds internal/alerts/eventlog: a SQLite-backed, additive event log that
records lifecycle transitions (resolved, acknowledged, unacknowledged,
escalated, flapping detected) and notification decisions (dispatched,
deferred by quiet hours, suppressed — with the mechanism that held
them). Appends never block alert evaluation: a full buffer drops the
event and counts the drop; a store that fails to open degrades to
recording nothing. 90-day retention, hourly prune.
The manager emits at the existing funnels only — dispatchAlert and the
safe-call resolve/ack/escalate seams — so no lifecycle behavior
changes. Lifecycle "fired" is deliberately not recorded yet: the
active-alert store funnel also runs on persisted restore, so firing
waits for the explicit activation seam in a later phase. The monitoring
bootstrap enables the log per manager; ephemeral managers and tests
record nothing unless they opt in.
GET /api/alerts/events (monitoring:read) reads the log with
alertIdentifier/type/since/limit filters, newest first.
Phase 0 of docs/ALERT_ENGINE_EVOLUTION.md (coverage gap
alert-engine-suppression-observability).
GET /api/alerts/delivery-diagnosis without alertIdentifier now returns
the diagnosis array for every active alert in one manager pass, so list
surfaces do not need a request per alert. Extracts the per-alert
diagnosis into a locked helper shared by both paths; single-alert
behavior is unchanged.
First slice of coverage gap alert-engine-suppression-observability
(docs/ALERT_ENGINE_EVOLUTION.md Phase 0).
Contract-Neutral: Storage history behavior is documented in the staged API and storage contracts and covered by staged focused regressions; shared router references do not change agent-lifecycle, performance, or security contracts, and the proof allowlists do not yet name the focused test files.
TestWebSocketSendsInitialState asserted initialState arrives as the
second message, but the hub interleaves 30s heartbeat pings with the
connect sequence, so a slow state build on a loaded runner put a ping
there first and reddened main twice in three runs. Wait for each
expected type the way TestWebsocketPayloadContractShape already does.
The delivery and payload assertions are unchanged.
Follow-up to 6686cdce2: the 100-default/500-max list page bounds existed
as separate untyped constants in internal/api and internal/recovery/store,
relying on a contract clause to keep them aligned. Export
DefaultListPageLimit / MaxListPageLimit from internal/recovery/model
(re-exported through internal/recovery) and alias both consumers to them,
so drift is impossible by construction. store_test.go and
recovery_handlers_test.go pin both consumers to the shared constants;
the three pagination-meta contract clauses now name the single source.
Telemetry could see only saved connections, so an install that tried to
reach a node and could not was indistinguishable from one that never
opened the add-node dialog. Both report zero configured connections and
stall at the same activation stage. Fleet data shows that population is
real and concentrated three to one in container deployments, and nothing
recorded whether those installs attempted a connection at all.
Record node connection test attempts and failures in a bounded,
day-bucketed tally in the config directory, pruned to a 31-day retention
window, and report both over the install-ID rotation window as
node_test_attempts_30d and node_test_failures_30d.
Recording starts only once a request carries a target and credentials, so
an incomplete form is never counted as a node that could not be reached.
A host string that turns out to be unusable does count, because the
attempt was made and it failed. Only the add-node dialog endpoint is
instrumented: instrumenting the unused test-config endpoint as well would
double-count a single operator action.
The tally holds counts alone. Hosts, credentials, and error text never
enter it, which is why it is plain JSON rather than encrypted history.
/api/recovery/points and /api/recovery/rollups clamp the requested page
size to [100 default, 500 max] in both the mock paginators and the store
paths, but the meta block was computed from the raw query value. A client
requesting limit=1000 with 1200 rollups was told totalPages=2 while the
server served 3 pages of 500, so iterating totalPages silently dropped
rollups; limit<=0 reported totalPages=1 at an effective limit of 100.
Normalize page and limit once at parse time, compute meta from the
normalized values, and echo the effective limit. Contract clause 34 in
api-contracts.md pins the obligation; storage-recovery and agent-lifecycle
record the boundary alignment and adjacency; recovery_handlers_test.go
pins above-max and non-positive limit meta.
The larger demo estate overflowed the rollups mock test's single
500-row page and pushed internal/api past the 25m -race budget, and
the alert history tests raced TempDir cleanup by never joining the
periodic save worker.
Walk every rollups page in the integration test, raise the shard
budget to 50m, and stop history managers through Stop() so the save
worker is joined before cleanup.
Contract-Neutral: test-only CI fix: paginate rollups mock test, join history save worker in test teardown, raise shard go test timeout for the 50-node demo estate; no runtime or contract delta
The hosted-tenant mint site (GenerateHostedTenantAgentInstallCommand)
issued binding-eligible install tokens without the command_policy_intent
metadata that 87ee3cd44 added to the self-hosted mint sites, so a hosted
tenant reinstalling an agent never got the stale-disabled-policy
reconciliation on first report (#1728). Hosted installs have no
enableCommands toggle, so the intent follows the install type the same
way the self-hosted flow does: pve enabled, pbs disabled.
Project freshly minted installer command intent onto the stable host before returning remote config, and consume that intent once so later admin policy changes remain authoritative.
GET /api/notifications/apprise and the PUT echo returned the saved API key
verbatim, so any settings:read token could exfiltrate a write-capable
secret while the sibling email handler blanks the SMTP password under the
same scope. Responses now drop the key and report hasApiKey instead, the
update and test-send paths fall back to the saved key when the request
leaves it blank (matching the email password contract), and the settings
form tells the user a saved key is kept when the field stays empty.
A refused action plan, decision, or execution returned its 409 to the
client and left no trace in the server journal, so every remote report
of "Docker / Podman command agent is not connected" stalled on greps
that could never match anything. Refusals now log one warn line with
the resource, capability, and reason code, and the Docker command-agent
resolver reports which lookup missed (stale enrollment token binding vs
agent-id/hostname session), carried as an optional diagnostic detail on
the readiness contract and in the refusal envelope.
Refs #1728
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
The delivery health verdict said when something was wrong; nothing showed
what actually fired and where it went. Expose the queue's retained
per-attempt audit rows as GET /api/notifications/delivery-log (newest
first, retention-labeled, webhook secrets redacted from error text) and
render them as a Recent delivery activity card on the alert destinations
tab, with outcome badges, destination names, and failure classes. Audit
rows now persist the normalized destination identity in a destination_id
column; older rows fall back to their operational links.
Test sends bypass both the queue and the activation gate, which is exactly
how installs came to believe delivery worked while every real alert was
suppressed (483 installs in the 08-18 telemetry read). Successful test
responses now carry deliveryPaused: true whenever the manager is gated
off, and the destinations UI warns instead of celebrating.
A LoadSystemSettings error in configureMonitorDependencies silently
skipped the tenant-monitor inherit block, leaving the monitor on
deny-all-private webhook CIDRs with nothing in the logs to explain why
private webhook targets fail SSRF validation. reloadSystemSettings
fails closed on the same error (embedding off) deliberately, but was
equally silent, so a persistent read failure looked identical to
embedding being switched off on purpose.
Both paths now log a warning with the error, mirroring the pattern from
cf5b86e62 in internal/config. A missing system.json (fresh install)
still stays quiet. Tests cover the warning on read failure, the quiet
fresh-install path, and that the reload still fails closed.
Discovering Docker inside Proxmox LXC guests previously required an
undocumented environment variable plus a hand-edited systemd unit,
which is how the surface kept shipping invisible. The opt-in is now a
persisted system setting with an admin-only toggle in Settings →
System → General, presented as its own "Docker in Proxmox LXCs" card
under the existing Docker/Podman updates card, with the requirements
(node agent with command execution) and data-collection bounds stated
inline.
The setting rides the admin-gated system settings endpoint
(RequireAdmin + settings:write, matching every other server-wide
toggle). Flipping it persists first, then applies to the runtime
config and fires a reconfigure hook so the router rebuilds the
monitor's Docker checker and inventory collector immediately — no
restart. PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY remains
authoritative: when set it locks the toggle (EnvironmentLockBadge in
the UI, HTTP 400 from the API), and an unrelated settings save can
never clobber the runtime value. The opt-in guardrail test now pins
the settings path, the env lock, and the persisted field.
Installer and node setup-guide callouts now point at the toggle first
with the environment variable as the locking override, and the docs,
their public mirrors, and the agent-lifecycle subsystem contract say
the same. New i18n strings shipped for en/de/es.
Verified live in mock mode: toggle off/on from the UI drives the
collector teardown/setup log lines, the value survives a server
restart, and all three touched surfaces render at desktop and mobile
widths (receipt in frontend-modern/browser-verification.json).
The pct exec socket probe that discovers Docker inside LXC guests
failed invisibly: errors logged at debug only, and a failed probe never
set DockerCheckedAt, so the guest stayed in first-check state and was
re-probed on every poll forever. On a node where lxc-attach could not
enter unprivileged guests that meant ~49 pct execs per minute of
steady-state churn, with nothing at default log level to say why the
Proxmox page showed no Docker.
Track consecutive probe failures per guest in the monitor. The first
failure of a streak, and the point where the backoff reaches its cap,
log at warn with the guest, node, and error; repeats stay at debug.
Failed probes retry with exponential backoff from one minute to a
thirty-minute ceiling, a success clears the streak with an info log,
and reconfiguring the checker resets all streaks so command execution
enabled from the UI at runtime still retries immediately. Entries not
refreshed for a day belong to deleted guests and are pruned.
PULSE_PROXMOX_GUEST_DOCKER_INVENTORY_VMIDS now gates the socket probe
as well as inventory collection. It previously only limited inventory,
so opting into specific guests still probed every running LXC on every
cycle; now guests outside the allowlist are never pct exec'd at all.
Subsystem contract and configuration docs updated to match.
TestRecoveryPointsEndpointReturnsMockData sampled the unfiltered first page
of /api/recovery/points and expected to find a kubernetes point in it. That
held only while the mock estate was small.
Points come back newest-completed-first and a page is hard-capped at 500. The
rescaled Proxmox demo estate now generates 2498 points, 939 of them newer than
the newest kubernetes point, so kubernetes starts at index 939 and never
reaches page one. The 36 kubernetes points are still generated and still
served, and the endpoint, the store ordering and the fixture shape are all
correct: a platform that snapshots every few days legitimately falls behind a
platform that backs up hourly. TrueNAS was already at index 254 and would have
broken next.
Ask the endpoint per platform instead, which is how the API is designed to
answer the question and how the UI asks it. The assertion no longer tracks
estate size, and it additionally covers the platform filter.
Contract-Neutral: test-only change, no behavior or contract delta.
Two counters could not answer the question they exist for.
pulse_intelligence_patrol_runs_30d was counted from the operator-facing
patrol run history, which is capped at MaxPatrolRunHistory (100). On any
install patrolling on a normal schedule that cap is reached within hours
of a thirty-day window opening, so the field reported the cap rather than
the run count, and the calls-per-run ratio derived from it was inflated by
a censored denominator. Patrol run history now carries an uncapped daily
tally beside the capped list, advanced from a persisted high-water mark so
repeated full-list saves cannot double count, pruned to 31 days, and read
in preference to the list. An install with no tally yet falls back to the
list, so the counter never regresses on upgrade.
Schema v8 split agent-side pre-mutation refusals into target-change,
prerequisite and contract categories so they would stop hiding in "other".
In production all three are zero fleet-wide and "other" still absorbs every
refusal, because the dominant reason code is the legacy preflight_refused
aggregate recorded when an agent sends no machine reason code at all.
Agents older than the typed refusal contract report every refusal that way,
so the split is starved rather than broken, and folding the two together
made those two states indistinguishable. Schema v9 counts uncoded refusals
separately.
Pulse could tell that its own delivery had stopped and had no way to say so.
Queue health was computed for one API endpoint and rendered on the alerts
surfaces, and that was the whole escalation path: a destination that stops
delivering cannot announce itself through a notification, so an operator who
does not open the alerts page learns nothing.
Telemetry: 78 installs retry every notification to the maxAttempts ceiling and
deliver almost nothing, 155,967 attempts against 221 deliveries in a week. 50
have delivered nothing in 30 days and 28 more delivered normally at some point
and silently stopped. The dominant terminal failure classes are authentication
and configuration, so most are fixable in minutes once someone knows.
Move the queue-health rule into internal/notifications as ClassifyQueueHealth
and DeliveryHealth, next to the queue that produces the counts, and delegate
the API handler to it so the endpoint and the new evaluator cannot drift into
two rules. A queue that cannot be read reports unavailable rather than healthy,
because silence is the failure mode being guarded against.
Monitoring evaluates delivery health on the poll ticker and raises or clears
the notification-delivery system alert, which puts it in the alert list and the
navigation badge. Reading queue health costs a SQLite query and the poll
cadence can be seconds, so the evaluation is throttled to five minutes.
RaiseSystemAlert is idempotent for an unchanged condition, so the timer neither
re-notifies nor accumulates alerts.
The alert is a warning rather than critical. It is real, but a new
critical-by-default alert appearing on upgrade across the affected installs is
a bigger change than this should make on its own.
The new Monitor field is deliberately named to fit inside the existing struct
alignment column. A longer name makes gofmt re-pad the whole block, which
breaks the canonical guardrail tests that pin those field declarations
verbatim.
Proof: delivery_health_test.go pins that in-flight work and retries stay
healthy while retained terminal failures do not, and that an unreadable queue
is never reported healthy. system_alerts_test.go pins the message for each
outcome including singular and plural, that it points at the destinations
surface, and that the throttle honours its interval. internal/monitoring,
internal/notifications and internal/alerts suites all pass.
Contract-Neutral: Staged contract deltas cover every boundary this change actually moves: notifications.md gains delivery_health.go plus the ClassifyQueueHealth rule, and monitoring.md gains system_alerts.go plus the system-alert evaluation extension point. The residual demands are inapplicable. agent-lifecycle is demanded only because monitor.go is one of its canonical files, and its verification artifact is host-agent deletion and re-enrollment proof, which this change does not touch: the diff to monitor.go is one struct field and one call in the poll loop. api-contracts and storage-recovery are demanded only because internal/api/notifications.go falls under a broad internal/api reference, and that file's change is a pure delegation of an internal helper to notifications.ClassifyQueueHealth with a byte-identical HTTP response and no route or payload delta.
Adding a PVE/PBS source through the Connect via API path discarded the
name typed in the add dialog: the setup bootstrap only carried type and
host, and auto-registration named the new connection after the node's
self-reported hostname. A user asking for a cluster named enacon got a
connection called pve01 and could only rename it after adoption.
Carry the typed name on the one-time setup token
(SetupTokenRecord.DesiredName) via an optional name field on
/api/setup-script-url, and let the setup-token-authenticated
auto-register completion name a newly created connection from it,
falling back to the hostname-derived default when absent. Dedup and
cluster-member adoption identity stay hostname/candidate based, so the
carried name cannot fork an existing registration.
The app shell decides which primary platform pages exist by classifying
every resource in the legacy full-state payload, which is why it has to
download that payload before it can render navigation. This publishes the
same answer as a `platformAdmission` facet on the canonical resource
aggregations, so admission has one definition instead of two that can
drift.
Counts cannot answer it, which is the whole reason this is a facet rather
than a client-side tally over `bySource`. A TrueNAS or Proxmox host
reports through the agent source and carries the "agent" platform scope,
so a count-based derivation admits the standalone page for an estate that
has no Pulse agent in it at all. Ownership is per-resource evidence, so it
is evaluated per resource here.
Verified against the live client classifier over real estates rather than
by inspection: the facet and the classifier agree on all six pages across
ten estates each for a 652-resource and a 216-resource estate, including
the cases that break a count-based derivation (Proxmox-only, TrueNAS-only,
vSphere-only and provider-owned-agents-only estates all correctly withhold
the standalone page, while a genuine Pulse agent admits it).
The tenant-fallback contract pin gains the new field. Its invariant, that
an empty tenant returns no resources rather than seeding from the raw
snapshot, is unchanged and still pinned; an empty estate admits nothing.
No caller reads the facet yet. Moving the shell onto it is a separate
change, gated by the same parity comparison.
Contracts: unified-resources and api-contracts gain the facet and its
derivation rule; agent-lifecycle and storage-recovery gain the ownership
clause, since an agent-typed host owned by a provider must not admit the
standalone page.
Contract-Neutral: Patrol autonomy behavior is documented in its primary ai-runtime and shared api-contracts contracts plus the relevant agent-lifecycle boundary; storage-recovery only has a broad internal/api extension reference and does not own this wire contract
The gate that mirrors command-channel admission
(commandConfigAllowedForToken) runs on every agent report, before the
agent ever attempts channel registration. When it refuses, the server
silently serves commandsEnabled=false, the agent never tries to
register, so none of the channel rejection warnings fire, and an
operator who enabled lifecycle management and granted agent:exec sees
"Remote commands: Not enabled" with nothing in any log. Emit the
refusal and the binding metadata it was judged against so the mismatch
is diagnosable from journalctl.
Refs #1728
Contract-Neutral: diagnostic logging only in agent command config gate, no behavioral or public contract change
pulse-triage[bot]