Contract-Neutral: Storage history behavior is documented in the staged API and storage contracts and covered by staged focused regressions; shared router references do not change agent-lifecycle, performance, or security contracts, and the proof allowlists do not yet name the focused test files.
Add a language-neutral condition key and explicit resource context to generic alert payloads so receivers can translate notifications without parsing English text.
Refs #1779
Change-source: pulse-maintainer
Contract-Neutral: The storage wearout sentinel contract is unchanged; mock fixtures now encode unreported endurance as the existing canonical -1 value and carry canonical mock proof.
Contract-Neutral: The release-note line documents an alert evaluator fix and does not change release promotion or artifact dispatch.
Change-source: pulse-maintainer
The retention and max-points trims re-sliced the series front
(metrics[i:]), which keeps the entire backing array reachable while
hiding the trimmed prefix from cap(). After the dense mock trends seed
(~2,800 points/series) aged out of its 6h window, every series held a
few hundred live points while pinning its seed-sized array - roughly
700MB across the demo estate's ~8,000 series. HeapSys drifted over the
droplet's memory.high, kernel reclaim throttling starved the accept
loop, and demo.pulserelay.pro served 502s (2026-08-25 outage; verified
live via pprof: appendMetric-attributed live heap grew +117MB in six
minutes after restart while goroutines stayed at 80).
Trims now compact in place so cap() reflects the true array size, and
releaseTrimmedCapacity copies the window into a right-sized array when
it occupies under a quarter of the backing array. A healthy sliding
window sits between 1x and 2x capacity and never pays the copy.
Regression tests pin both paths: a live append after an aged-out seed
backfill, and Cleanup on a mostly-expired series, including an aliasing
assertion that catches the cap()-blind front re-slice.
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).
The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.
Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.
Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.
/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.
Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
TestWebSocketSendsInitialState asserted initialState arrives as the
second message, but the hub interleaves 30s heartbeat pings with the
connect sequence, so a slow state build on a loaded runner put a ping
there first and reddened main twice in three runs. Wait for each
expected type the way TestWebsocketPayloadContractShape already does.
The delivery and payload assertions are unchanged.
allowHostAgentReenrollLocked trusted only the durable continuity store
when one was attached, so a removal block living only in persisted
monitor state (written by releases predating the store) could never be
cleared: every fresh re-enroll read as already-consumed and the host
was rejected with 400 on every report indefinitely, regardless of new
tokens or reinstalls.
Refs #1772
With the adaptive scheduler disabled (the default), every planning pass
on the main poll tick rebuilt each instance task with NextRun=now and
Upsert overwrote the pending slot, so availability targets polled at
the tick cadence regardless of their configured interval. Reproduced
at exactly 10s polling on a 60s ICMP target; c70431caa only covered
the adaptive-scheduler path. Planning now preserves a pending future
slot, tightening it only when a freshly shortened interval justifies
an earlier run.
Refs #1745
Host identities fork onto "<base>-<hex>" when an agent re-enrolls under
a new token (re-running an install with --proxmox is enough), and
discovery then stores records under the forked host ID while the PVE
node keeps reporting the base agent UUID as its linked agent.
Store.ListByTarget matches TargetID byte-for-byte and
ListDiscoveriesByTarget did no alias resolution, so the resource
drawer's list call returned an empty 200 and the Discovery tab reported
"no saved discovery run" for hosts that had been analyzed.
GetDiscoveryByResource's normalization missed too: a base UUID matches
neither the forked host ID nor its hostname.
Expand the requested target into every spelling that provably names the
same target (equivalentDiscoveryTargetIDs): a matched node's linked
agent ID and name, a matched host's own ID/hostname pairing, and hex
fork suffixes of any of those, including the twice-forked
truncated-base shape. Filter the list path against that set and seed
the same spellings into the single-record alias path.
Trimmed from PR #1715 as submitted: the transitive hostname closure
(node name -> hostnames -> other hosts' IDs) is dropped. Multi-estate
deployments legitimately reuse node names and hostnames, and the hop
would have surfaced another estate's records in the drawer, discovery
readiness, and AI context. A regression test pins the non-bridging
behavior.
Follow-up to 6686cdce2: the 100-default/500-max list page bounds existed
as separate untyped constants in internal/api and internal/recovery/store,
relying on a contract clause to keep them aligned. Export
DefaultListPageLimit / MaxListPageLimit from internal/recovery/model
(re-exported through internal/recovery) and alias both consumers to them,
so drift is impossible by construction. store_test.go and
recovery_handlers_test.go pin both consumers to the shared constants;
the three pagination-meta contract clauses now name the single source.
Telemetry could see only saved connections, so an install that tried to
reach a node and could not was indistinguishable from one that never
opened the add-node dialog. Both report zero configured connections and
stall at the same activation stage. Fleet data shows that population is
real and concentrated three to one in container deployments, and nothing
recorded whether those installs attempted a connection at all.
Record node connection test attempts and failures in a bounded,
day-bucketed tally in the config directory, pruned to a 31-day retention
window, and report both over the install-ID rotation window as
node_test_attempts_30d and node_test_failures_30d.
Recording starts only once a request carries a target and credentials, so
an incomplete form is never counted as a node that could not be reached.
A host string that turns out to be unusable does count, because the
attempt was made and it failed. Only the add-node dialog endpoint is
instrumented: instrumenting the unused test-config endpoint as well would
double-count a single operator action.
The tally holds counts alone. Hosts, credentials, and error text never
enter it, which is why it is plain JSON rather than encrypted history.
/api/recovery/points and /api/recovery/rollups clamp the requested page
size to [100 default, 500 max] in both the mock paginators and the store
paths, but the meta block was computed from the raw query value. A client
requesting limit=1000 with 1200 rollups was told totalPages=2 while the
server served 3 pages of 500, so iterating totalPages silently dropped
rollups; limit<=0 reported totalPages=1 at an effective limit of 100.
Normalize page and limit once at parse time, compute meta from the
normalized values, and echo the effective limit. Contract clause 34 in
api-contracts.md pins the obligation; storage-recovery and agent-lifecycle
record the boundary alignment and adjacency; recovery_handlers_test.go
pins above-max and non-positive limit meta.
The larger demo estate overflowed the rollups mock test's single
500-row page and pushed internal/api past the 25m -race budget, and
the alert history tests raced TempDir cleanup by never joining the
periodic save worker.
Walk every rollups page in the integration test, raise the shard
budget to 50m, and stop history managers through Stop() so the save
worker is joined before cleanup.
Contract-Neutral: test-only CI fix: paginate rollups mock test, join history save worker in test teardown, raise shard go test timeout for the 50-node demo estate; no runtime or contract delta
The hosted-tenant mint site (GenerateHostedTenantAgentInstallCommand)
issued binding-eligible install tokens without the command_policy_intent
metadata that 87ee3cd44 added to the self-hosted mint sites, so a hosted
tenant reinstalling an agent never got the stale-disabled-policy
reconciliation on first report (#1728). Hosted installs have no
enableCommands toggle, so the intent follows the install type the same
way the self-hosted flow does: pve enabled, pbs disabled.
Project freshly minted installer command intent onto the stable host before returning remote config, and consume that intent once so later admin policy changes remain authoritative.
GET /api/notifications/apprise and the PUT echo returned the saved API key
verbatim, so any settings:read token could exfiltrate a write-capable
secret while the sibling email handler blanks the SMTP password under the
same scope. Responses now drop the key and report hasApiKey instead, the
update and test-send paths fall back to the saved key when the request
leaves it blank (matching the email password contract), and the settings
form tells the user a saved key is kept when the field stays empty.
A refused action plan, decision, or execution returned its 409 to the
client and left no trace in the server journal, so every remote report
of "Docker / Podman command agent is not connected" stalled on greps
that could never match anything. Refusals now log one warn line with
the resource, capability, and reason code, and the Docker command-agent
resolver reports which lookup missed (stale enrollment token binding vs
agent-id/hostname session), carried as an optional diagnostic detail on
the readiness contract and in the refusal envelope.
Refs #1728
Resolve PBS alert overrides once per observation and reuse that snapshot during offline lifecycle evaluation. Add canonical-ID coverage for disabled eligibility and dispatch, enabled alerts and recovery, unrelated metrics, and concurrent configuration updates.
Contract-Neutral: Narrows internal PBS offline alert eligibility without changing external APIs or release artifacts.
Co-authored-by: rcourtman <rcourtman@users.noreply.github.com>
Presentation host coalescing buckets agent rows by short hostname and
PVE node names are short, so a pve01 in one estate merged with a pve01
in another, re-parenting both estates workloads onto one row and
cross-wiring reachability. Veto the merge when the rows carry proof of
distinct machines, a disjoint machine ID set, differing DMI UUIDs, or
differing Proxmox cluster names, while equal cluster names still merge
so the same cluster added under two connections keeps its single row.
Apply the same cluster discrimination to agent-node link inference,
whose short endpoint alias keys and short-hostname corroboration
fallback could hand one estate agent to the other estate node.
Refs #1753
Contract-Neutral: behavioral bugfix in unified resources presentation coalescing and node link inference; no API shape or subsystem contract change
applyThresholdOverride only ever copied DisableConnectivity=true from an
override, so once the global default disabled connectivity alerts no
override could turn them back on. A guest row set to Warning or Critical
saved its severity, showed as Custom, and never fired, while the
frontend already encodes re-enablement by pairing an explicit severity
with the row offline control (off stores DisableConnectivity=true).
Treat an override that carries an explicit powered-off severity as that
row offline control set to warning or critical and clear the inherited
disable, matching the Docker container state path which already works
this way.
Refs #1738
Contract-Neutral: Refs #1738: behavioral bugfix in alerts override merge, no API or contract shape change
The dockeragent credential-store tests keyed their fake filesystem and
path-bearing env seams on POSIX literals, while dockerConfigCredentials
builds candidate config paths with filepath.Join. On Windows every lookup
therefore missed the fixture and the nine credential-expecting
TestDockerConfigCredentials_* cases failed with a nil credential; the
hostagent PULSE_PCT_PATH override test likewise asserted a POSIX path that
is not absolute on Windows. Map fixture keys and env values through
filepath.FromSlash and anchor the override on t.TempDir().
Taken from PR #1755 (ff96aed8c) so the correction lands on main instead of
a parked agent branch. The unified-agent-native Windows x64 job has been
red on every push run since these tests landed on 2026-08-13; the
intervening green runs were pull_request events exercising that PR's own
corrected fixtures.
Contract-Neutral: test fixtures only, no production runtime change.
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
Field telemetry showed installs with Patrol enabled recording weeks of
empty error runs (runs_30d=122, ai_calls=0, findings=0): provider
initialization failed once at boot (model resolution can need the
provider's live catalog, so Pulse racing a booting Ollama server loses)
and was never retried, while the run loop kept recording "Patrol
provider not configured" errors that told operators who had configured
a provider to configure one.
- Retry provider initialization on every scheduled run, so a boot-time
race strands Patrol for at most one interval instead of until the
next settings save. LoadConfig records the redacted init failure.
- Name the real failure in the blocked reason and run record when the
configured provider failed to initialize, instead of claiming no
provider is configured.
- Raise the deduped Patrol runtime finding when scheduled runs are
skipped by a persistent readiness blocker or missing provider, so the
state reaches the findings surfaces and alert notification channels
once, instead of living only on the Patrol page banner. Transient
circuit-breaker blocks stay finding-free; the attempts that opened
the breaker already raised their own.
- Resolve the runtime finding when Patrol is turned off; opting out is
a resolution, not a state to keep nagging about.
- Record the extended runtime-failure surface in the ai-runtime
subsystem contract.
A hard-mounted network filesystem with an unreachable server blocks
statfs in an uninterruptible kernel wait. The collector issued that
syscall inline for every mount, so one dead NFS mount silently froze
the whole reporting cycle and stretched agent shutdown into the kernel
retry window, and it paid that price for mounts the fstype filter was
going to discard anyway.
Decide type- and mountpoint-based skips before the usage syscall, so
network filesystems are never probed unless explicitly included, and
bound every remaining usage call with a timeout that leaves at most one
in-flight probe per mountpoint. A mount whose call never returned is
skipped on later cycles and re-included when the stalled call answers.
Refs discussion #1747
Contract-Neutral: Discussion #1747: behavioral bugfix in host disk collection; no payload or API shape change, filtered mounts were never reported
Since the operational-trust records shipped in v6.2.0, every new firing
of a previously resolved alert folded into that alert's first history
row: setActiveAlertNoLock stamped the new occurrence's open record onto
the previous occurrence's resolved row, and the history dedup then
treated any two open records with the same identity as one incident
regardless of how far apart they were. Alert history therefore froze at
the upgrade date while notifications kept flowing, which is exactly how
users reported it.
Resolved rows now keep their final record unless the update belongs to
the same occurrence, and two open records no longer merge on identity
alone. The observation-gap window still coalesces genuine flapping, and
the five-minute refire continuity path is unchanged.
Refs #1497
Contract-Neutral: Refs #1497: behavioral bugfix in alert history occurrence dedup; no API shape change, history row schema unchanged, no subsystem contract names sameHistoryIncident
pulse-triage[bot]