Restore the complete prior API-token inventory when an agent install credential cannot be persisted, so failed issuance neither admits an undisclosed secret nor evicts an older valid token. Prove the package and API failure boundaries and govern the shared lifecycle contract.
Change-source: pulse-maintainer
Restore the complete pre-creation token inventory when persistence fails so sorting cannot evict an older valid token or leave an unreturned credential active. Govern and test the durable creation boundary across API, security, agent lifecycle, and storage.
Change-source: pulse-maintainer
Record durable persistence as the token-revocation commit boundary across API, security, agent-lifecycle, and storage contracts. Route the exact-removal and rollback proofs through the canonical token lifecycle suite.
Change-source: pulse-maintainer
Roll back the in-memory token inventory and return an error when durable persistence fails. Cover exact deletion from a multi-token inventory and the persistence-failure rollback path.
Change-source: pulse-maintainer
Bind monitor-side event-log bootstrap to its canonical boundary and exercise the new event infrastructure through the registry-recognized alerts, API, monitoring, and agent-lifecycle proof files.
Change-source: pulse-maintainer
Bind the delivery diagnosis, active-card presentation, and additive event log to their canonical contracts and recognized API/frontend tests. Sync the shipped API reference, record current browser evidence, and restore sorted truthful control-plane status evidence.
Change-source: pulse-maintainer
Adds internal/alerts/eventlog: a SQLite-backed, additive event log that
records lifecycle transitions (resolved, acknowledged, unacknowledged,
escalated, flapping detected) and notification decisions (dispatched,
deferred by quiet hours, suppressed — with the mechanism that held
them). Appends never block alert evaluation: a full buffer drops the
event and counts the drop; a store that fails to open degrades to
recording nothing. 90-day retention, hourly prune.
The manager emits at the existing funnels only — dispatchAlert and the
safe-call resolve/ack/escalate seams — so no lifecycle behavior
changes. Lifecycle "fired" is deliberately not recorded yet: the
active-alert store funnel also runs on persisted restore, so firing
waits for the explicit activation seam in a later phase. The monitoring
bootstrap enables the log per manager; ephemeral managers and tests
record nothing unless they opt in.
GET /api/alerts/events (monitoring:read) reads the log with
alertIdentifier/type/since/limit filters, newest first.
Phase 0 of docs/ALERT_ENGINE_EVOLUTION.md (coverage gap
alert-engine-suppression-observability).
GET /api/alerts/delivery-diagnosis without alertIdentifier now returns
the diagnosis array for every active alert in one manager pass, so list
surfaces do not need a request per alert. Extracts the per-alert
diagnosis into a locked helper shared by both paths; single-alert
behavior is unchanged.
First slice of coverage gap alert-engine-suppression-observability
(docs/ALERT_ENGINE_EVOLUTION.md Phase 0).
Contract-Neutral: Storage history behavior is documented in the staged API and storage contracts and covered by staged focused regressions; shared router references do not change agent-lifecycle, performance, or security contracts, and the proof allowlists do not yet name the focused test files.
TestWebSocketSendsInitialState asserted initialState arrives as the
second message, but the hub interleaves 30s heartbeat pings with the
connect sequence, so a slow state build on a loaded runner put a ping
there first and reddened main twice in three runs. Wait for each
expected type the way TestWebsocketPayloadContractShape already does.
The delivery and payload assertions are unchanged.
Follow-up to 6686cdce2: the 100-default/500-max list page bounds existed
as separate untyped constants in internal/api and internal/recovery/store,
relying on a contract clause to keep them aligned. Export
DefaultListPageLimit / MaxListPageLimit from internal/recovery/model
(re-exported through internal/recovery) and alias both consumers to them,
so drift is impossible by construction. store_test.go and
recovery_handlers_test.go pin both consumers to the shared constants;
the three pagination-meta contract clauses now name the single source.
Telemetry could see only saved connections, so an install that tried to
reach a node and could not was indistinguishable from one that never
opened the add-node dialog. Both report zero configured connections and
stall at the same activation stage. Fleet data shows that population is
real and concentrated three to one in container deployments, and nothing
recorded whether those installs attempted a connection at all.
Record node connection test attempts and failures in a bounded,
day-bucketed tally in the config directory, pruned to a 31-day retention
window, and report both over the install-ID rotation window as
node_test_attempts_30d and node_test_failures_30d.
Recording starts only once a request carries a target and credentials, so
an incomplete form is never counted as a node that could not be reached.
A host string that turns out to be unusable does count, because the
attempt was made and it failed. Only the add-node dialog endpoint is
instrumented: instrumenting the unused test-config endpoint as well would
double-count a single operator action.
The tally holds counts alone. Hosts, credentials, and error text never
enter it, which is why it is plain JSON rather than encrypted history.
/api/recovery/points and /api/recovery/rollups clamp the requested page
size to [100 default, 500 max] in both the mock paginators and the store
paths, but the meta block was computed from the raw query value. A client
requesting limit=1000 with 1200 rollups was told totalPages=2 while the
server served 3 pages of 500, so iterating totalPages silently dropped
rollups; limit<=0 reported totalPages=1 at an effective limit of 100.
Normalize page and limit once at parse time, compute meta from the
normalized values, and echo the effective limit. Contract clause 34 in
api-contracts.md pins the obligation; storage-recovery and agent-lifecycle
record the boundary alignment and adjacency; recovery_handlers_test.go
pins above-max and non-positive limit meta.
The larger demo estate overflowed the rollups mock test's single
500-row page and pushed internal/api past the 25m -race budget, and
the alert history tests raced TempDir cleanup by never joining the
periodic save worker.
Walk every rollups page in the integration test, raise the shard
budget to 50m, and stop history managers through Stop() so the save
worker is joined before cleanup.
Contract-Neutral: test-only CI fix: paginate rollups mock test, join history save worker in test teardown, raise shard go test timeout for the 50-node demo estate; no runtime or contract delta
The hosted-tenant mint site (GenerateHostedTenantAgentInstallCommand)
issued binding-eligible install tokens without the command_policy_intent
metadata that 87ee3cd44 added to the self-hosted mint sites, so a hosted
tenant reinstalling an agent never got the stale-disabled-policy
reconciliation on first report (#1728). Hosted installs have no
enableCommands toggle, so the intent follows the install type the same
way the self-hosted flow does: pve enabled, pbs disabled.
Project freshly minted installer command intent onto the stable host before returning remote config, and consume that intent once so later admin policy changes remain authoritative.
GET /api/notifications/apprise and the PUT echo returned the saved API key
verbatim, so any settings:read token could exfiltrate a write-capable
secret while the sibling email handler blanks the SMTP password under the
same scope. Responses now drop the key and report hasApiKey instead, the
update and test-send paths fall back to the saved key when the request
leaves it blank (matching the email password contract), and the settings
form tells the user a saved key is kept when the field stays empty.
A refused action plan, decision, or execution returned its 409 to the
client and left no trace in the server journal, so every remote report
of "Docker / Podman command agent is not connected" stalled on greps
that could never match anything. Refusals now log one warn line with
the resource, capability, and reason code, and the Docker command-agent
resolver reports which lookup missed (stale enrollment token binding vs
agent-id/hostname session), carried as an optional diagnostic detail on
the readiness contract and in the refusal envelope.
Refs #1728
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
The delivery health verdict said when something was wrong; nothing showed
what actually fired and where it went. Expose the queue's retained
per-attempt audit rows as GET /api/notifications/delivery-log (newest
first, retention-labeled, webhook secrets redacted from error text) and
render them as a Recent delivery activity card on the alert destinations
tab, with outcome badges, destination names, and failure classes. Audit
rows now persist the normalized destination identity in a destination_id
column; older rows fall back to their operational links.
Test sends bypass both the queue and the activation gate, which is exactly
how installs came to believe delivery worked while every real alert was
suppressed (483 installs in the 08-18 telemetry read). Successful test
responses now carry deliveryPaused: true whenever the manager is gated
off, and the destinations UI warns instead of celebrating.
A LoadSystemSettings error in configureMonitorDependencies silently
skipped the tenant-monitor inherit block, leaving the monitor on
deny-all-private webhook CIDRs with nothing in the logs to explain why
private webhook targets fail SSRF validation. reloadSystemSettings
fails closed on the same error (embedding off) deliberately, but was
equally silent, so a persistent read failure looked identical to
embedding being switched off on purpose.
Both paths now log a warning with the error, mirroring the pattern from
cf5b86e62 in internal/config. A missing system.json (fresh install)
still stays quiet. Tests cover the warning on read failure, the quiet
fresh-install path, and that the reload still fails closed.
Pulse Test