Third increment of the resource-payload-static-metadata gap, the
snapshot-side statics. Measurement first: the mock estate ships only 5
distinct non-default policy postures across 723 resources and 56
distinct templated AI-safe summary strings across the same rows
(~150KB of inline duplication per snapshot), and every canonical alias
list carries the row's own wire id verbatim.
Non-default policies and AI-safe summaries now dedupe through the same
content-addressed catalog contract as capabilities: state-level
policyCatalog and aiSafeSummaryCatalog hold the distinct blobs,
resources carry policyRef / aiSafeSummaryRef, and ingestion expands
refs back to the inline shape (default posture synthesis unchanged for
ref-less rows). The broadcast conversion returns one catalogs struct so
the next deduped field is a loop entry, and catalog changes ride the
same delta frame as the refs that need them. Alias lists stop shipping
the row's own wire id — every identity consumer unions resource.id
beside the alias spread — while the primary id stays so aliases remain
the complete live vocabulary a host is reachable by, per the
established slimming invariant.
Verified against a 1857-resource estate: zero inline policies or
summaries on the wire (743 refs resolving into 1KB + 7KB catalogs),
zero alias lists carrying the wire id, catalog-expanded postures and
summaries reaching the store byte-equivalently, and the estate
rendering clean on desktop and the 36-row phone window.
Remaining gap scope: canonicalIdentity alias/superseded history is
genuine per-resource identity data (~600KB per snapshot) with no safe
derivation shortcut, and the proxmox-facet/platformData mirror
duplication (~200KB) is deliberate canonical-merge fabric; both stay
recorded under resource-payload-static-metadata for a future owner.
Second increment of the resource-payload-static-metadata gap: the full
activeAlerts array (~37-75KB at mock-estate churn) re-shipped whenever
any single alert changed. Alerts now ride the generalized keyed delta
transport: the per-field keying and transition handling in the state
delta engine is a declarative keyedDeltaFields list shared with the
connected-infrastructure projection, and activeAlertsDelta carries
id-keyed merge patches, removed ids, and order. Measured on the mock
estate the per-tick alert payload drops to 184-1981 bytes.
One boundary held deliberately fixed: alert application timing does not
change. The client applies alert deltas to an isolated baseline and
feeds the exact same applyActiveAlerts path immediately on arrival,
input-active or not — alert lifecycle truth never waits for the
operator-input idle gate (alerts subsystem boundary, documented in
6ecdc81d1), and the pending-acknowledgment safeguards see byte-
equivalent records. Alerts handed to the store are cloned so it never
adopts baseline-owned objects. Deltas without a baseline are ignored
and heal on the next full payload; unkeyable entries fall back to
whole-field diffing.
Remaining gap scope unchanged: snapshot-side static metadata
(canonicalIdentity alias history ~609KB per snapshot, and the partially
regressed aiSafeSummary/policy omissions) stays open under
resource-payload-static-metadata.
First increment of the resource-payload-static-metadata gap: the
connected-infrastructure reporting projection was re-shipped whole
(~55-60KB at 50-node scale) on every realtime broadcast even though
measurement showed the only field that ever changes tick-to-tick is
lastSeen, and the client deep-reconciled all 69 rich items per tick —
the dominant reactive cost inside every websocket message after the
resource merge itself.
The projection now rides the same per-client keyed delta engine as
resources. buildClientStateSnapshot keys it by id beside the resource
map, and buildClientStateDelta emits connectedInfrastructureDelta with
id-keyed JSON merge patches, removed ids, and order through one shared
keyed-array builder. Measured on the mock estate the per-tick payload
drops from ~55-60KB to 0.7-4KB of timestamp patches. Entries that
cannot key by id fall the whole field back to plain whole-payload
diffing, and keyed/unkeyed baseline transitions re-ship the full array
once, so the keyed path can never corrupt the projection.
The client mirrors the resource baseline pattern: full payloads
establish an isolated raw baseline, deltas apply as merge patches
through the now-generic keyed apply helper, and the store sync hands
reconcile reference-stable objects for untouched items so a tick
reconciles only what changed instead of deep-walking the projection.
Deltas without a baseline are ignored and heal on the next full
payload; the operator-input deferral gate carries the pending changed
ids across a gesture and flushes them in the same coalesced batch.
The remaining gap scope (snapshot-side static metadata such as
canonicalIdentity alias history, ~609KB per snapshot) stays open under
resource-payload-static-metadata.
Contract-Neutral: Storage history behavior is documented in the staged API and storage contracts and covered by staged focused regressions; shared router references do not change agent-lifecycle, performance, or security contracts, and the proof allowlists do not yet name the focused test files.
Add a language-neutral condition key and explicit resource context to generic alert payloads so receivers can translate notifications without parsing English text.
Refs #1779
Change-source: pulse-maintainer
Contract-Neutral: The storage wearout sentinel contract is unchanged; mock fixtures now encode unreported endurance as the existing canonical -1 value and carry canonical mock proof.
Contract-Neutral: The release-note line documents an alert evaluator fix and does not change release promotion or artifact dispatch.
Change-source: pulse-maintainer
The retention and max-points trims re-sliced the series front
(metrics[i:]), which keeps the entire backing array reachable while
hiding the trimmed prefix from cap(). After the dense mock trends seed
(~2,800 points/series) aged out of its 6h window, every series held a
few hundred live points while pinning its seed-sized array - roughly
700MB across the demo estate's ~8,000 series. HeapSys drifted over the
droplet's memory.high, kernel reclaim throttling starved the accept
loop, and demo.pulserelay.pro served 502s (2026-08-25 outage; verified
live via pprof: appendMetric-attributed live heap grew +117MB in six
minutes after restart while goroutines stayed at 80).
Trims now compact in place so cap() reflects the true array size, and
releaseTrimmedCapacity copies the window into a right-sized array when
it occupies under a quarter of the backing array. A healthy sliding
window sits between 1x and 2x capacity and never pays the copy.
Regression tests pin both paths: a live append after an aged-out seed
backfill, and Cleanup on a mostly-expired series, including an aliasing
assertion that catches the cap()-blind front re-slice.
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).
The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.
Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.
Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.
/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.
Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
TestWebSocketSendsInitialState asserted initialState arrives as the
second message, but the hub interleaves 30s heartbeat pings with the
connect sequence, so a slow state build on a loaded runner put a ping
there first and reddened main twice in three runs. Wait for each
expected type the way TestWebsocketPayloadContractShape already does.
The delivery and payload assertions are unchanged.
allowHostAgentReenrollLocked trusted only the durable continuity store
when one was attached, so a removal block living only in persisted
monitor state (written by releases predating the store) could never be
cleared: every fresh re-enroll read as already-consumed and the host
was rejected with 400 on every report indefinitely, regardless of new
tokens or reinstalls.
Refs #1772
With the adaptive scheduler disabled (the default), every planning pass
on the main poll tick rebuilt each instance task with NextRun=now and
Upsert overwrote the pending slot, so availability targets polled at
the tick cadence regardless of their configured interval. Reproduced
at exactly 10s polling on a 60s ICMP target; c70431caa only covered
the adaptive-scheduler path. Planning now preserves a pending future
slot, tightening it only when a freshly shortened interval justifies
an earlier run.
Refs #1745
Host identities fork onto "<base>-<hex>" when an agent re-enrolls under
a new token (re-running an install with --proxmox is enough), and
discovery then stores records under the forked host ID while the PVE
node keeps reporting the base agent UUID as its linked agent.
Store.ListByTarget matches TargetID byte-for-byte and
ListDiscoveriesByTarget did no alias resolution, so the resource
drawer's list call returned an empty 200 and the Discovery tab reported
"no saved discovery run" for hosts that had been analyzed.
GetDiscoveryByResource's normalization missed too: a base UUID matches
neither the forked host ID nor its hostname.
Expand the requested target into every spelling that provably names the
same target (equivalentDiscoveryTargetIDs): a matched node's linked
agent ID and name, a matched host's own ID/hostname pairing, and hex
fork suffixes of any of those, including the twice-forked
truncated-base shape. Filter the list path against that set and seed
the same spellings into the single-record alias path.
Trimmed from PR #1715 as submitted: the transitive hostname closure
(node name -> hostnames -> other hosts' IDs) is dropped. Multi-estate
deployments legitimately reuse node names and hostnames, and the hop
would have surfaced another estate's records in the drawer, discovery
readiness, and AI context. A regression test pins the non-bridging
behavior.
Follow-up to 6686cdce2: the 100-default/500-max list page bounds existed
as separate untyped constants in internal/api and internal/recovery/store,
relying on a contract clause to keep them aligned. Export
DefaultListPageLimit / MaxListPageLimit from internal/recovery/model
(re-exported through internal/recovery) and alias both consumers to them,
so drift is impossible by construction. store_test.go and
recovery_handlers_test.go pin both consumers to the shared constants;
the three pagination-meta contract clauses now name the single source.
Telemetry could see only saved connections, so an install that tried to
reach a node and could not was indistinguishable from one that never
opened the add-node dialog. Both report zero configured connections and
stall at the same activation stage. Fleet data shows that population is
real and concentrated three to one in container deployments, and nothing
recorded whether those installs attempted a connection at all.
Record node connection test attempts and failures in a bounded,
day-bucketed tally in the config directory, pruned to a 31-day retention
window, and report both over the install-ID rotation window as
node_test_attempts_30d and node_test_failures_30d.
Recording starts only once a request carries a target and credentials, so
an incomplete form is never counted as a node that could not be reached.
A host string that turns out to be unusable does count, because the
attempt was made and it failed. Only the add-node dialog endpoint is
instrumented: instrumenting the unused test-config endpoint as well would
double-count a single operator action.
The tally holds counts alone. Hosts, credentials, and error text never
enter it, which is why it is plain JSON rather than encrypted history.
/api/recovery/points and /api/recovery/rollups clamp the requested page
size to [100 default, 500 max] in both the mock paginators and the store
paths, but the meta block was computed from the raw query value. A client
requesting limit=1000 with 1200 rollups was told totalPages=2 while the
server served 3 pages of 500, so iterating totalPages silently dropped
rollups; limit<=0 reported totalPages=1 at an effective limit of 100.
Normalize page and limit once at parse time, compute meta from the
normalized values, and echo the effective limit. Contract clause 34 in
api-contracts.md pins the obligation; storage-recovery and agent-lifecycle
record the boundary alignment and adjacency; recovery_handlers_test.go
pins above-max and non-positive limit meta.
The larger demo estate overflowed the rollups mock test's single
500-row page and pushed internal/api past the 25m -race budget, and
the alert history tests raced TempDir cleanup by never joining the
periodic save worker.
Walk every rollups page in the integration test, raise the shard
budget to 50m, and stop history managers through Stop() so the save
worker is joined before cleanup.
Contract-Neutral: test-only CI fix: paginate rollups mock test, join history save worker in test teardown, raise shard go test timeout for the 50-node demo estate; no runtime or contract delta
The hosted-tenant mint site (GenerateHostedTenantAgentInstallCommand)
issued binding-eligible install tokens without the command_policy_intent
metadata that 87ee3cd44 added to the self-hosted mint sites, so a hosted
tenant reinstalling an agent never got the stale-disabled-policy
reconciliation on first report (#1728). Hosted installs have no
enableCommands toggle, so the intent follows the install type the same
way the self-hosted flow does: pve enabled, pbs disabled.
Project freshly minted installer command intent onto the stable host before returning remote config, and consume that intent once so later admin policy changes remain authoritative.
GET /api/notifications/apprise and the PUT echo returned the saved API key
verbatim, so any settings:read token could exfiltrate a write-capable
secret while the sibling email handler blanks the SMTP password under the
same scope. Responses now drop the key and report hasApiKey instead, the
update and test-send paths fall back to the saved key when the request
leaves it blank (matching the email password contract), and the settings
form tells the user a saved key is kept when the field stays empty.
A refused action plan, decision, or execution returned its 409 to the
client and left no trace in the server journal, so every remote report
of "Docker / Podman command agent is not connected" stalled on greps
that could never match anything. Refusals now log one warn line with
the resource, capability, and reason code, and the Docker command-agent
resolver reports which lookup missed (stale enrollment token binding vs
agent-id/hostname session), carried as an optional diagnostic detail on
the readiness contract and in the refusal envelope.
Refs #1728
Resolve PBS alert overrides once per observation and reuse that snapshot during offline lifecycle evaluation. Add canonical-ID coverage for disabled eligibility and dispatch, enabled alerts and recovery, unrelated metrics, and concurrent configuration updates.
Contract-Neutral: Narrows internal PBS offline alert eligibility without changing external APIs or release artifacts.
Co-authored-by: rcourtman <rcourtman@users.noreply.github.com>
Presentation host coalescing buckets agent rows by short hostname and
PVE node names are short, so a pve01 in one estate merged with a pve01
in another, re-parenting both estates workloads onto one row and
cross-wiring reachability. Veto the merge when the rows carry proof of
distinct machines, a disjoint machine ID set, differing DMI UUIDs, or
differing Proxmox cluster names, while equal cluster names still merge
so the same cluster added under two connections keeps its single row.
Apply the same cluster discrimination to agent-node link inference,
whose short endpoint alias keys and short-hostname corroboration
fallback could hand one estate agent to the other estate node.
Refs #1753
Contract-Neutral: behavioral bugfix in unified resources presentation coalescing and node link inference; no API shape or subsystem contract change
applyThresholdOverride only ever copied DisableConnectivity=true from an
override, so once the global default disabled connectivity alerts no
override could turn them back on. A guest row set to Warning or Critical
saved its severity, showed as Custom, and never fired, while the
frontend already encodes re-enablement by pairing an explicit severity
with the row offline control (off stores DisableConnectivity=true).
Treat an override that carries an explicit powered-off severity as that
row offline control set to warning or critical and clear the inherited
disable, matching the Docker container state path which already works
this way.
Refs #1738
Contract-Neutral: Refs #1738: behavioral bugfix in alerts override merge, no API or contract shape change
The dockeragent credential-store tests keyed their fake filesystem and
path-bearing env seams on POSIX literals, while dockerConfigCredentials
builds candidate config paths with filepath.Join. On Windows every lookup
therefore missed the fixture and the nine credential-expecting
TestDockerConfigCredentials_* cases failed with a nil credential; the
hostagent PULSE_PCT_PATH override test likewise asserted a POSIX path that
is not absolute on Windows. Map fixture keys and env values through
filepath.FromSlash and anchor the override on t.TempDir().
Taken from PR #1755 (ff96aed8c) so the correction lands on main instead of
a parked agent branch. The unified-agent-native Windows x64 job has been
red on every push run since these tests landed on 2026-08-13; the
intervening green runs were pull_request events exercising that PR's own
corrected fixtures.
Contract-Neutral: test fixtures only, no production runtime change.
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
Field telemetry showed installs with Patrol enabled recording weeks of
empty error runs (runs_30d=122, ai_calls=0, findings=0): provider
initialization failed once at boot (model resolution can need the
provider's live catalog, so Pulse racing a booting Ollama server loses)
and was never retried, while the run loop kept recording "Patrol
provider not configured" errors that told operators who had configured
a provider to configure one.
- Retry provider initialization on every scheduled run, so a boot-time
race strands Patrol for at most one interval instead of until the
next settings save. LoadConfig records the redacted init failure.
- Name the real failure in the blocked reason and run record when the
configured provider failed to initialize, instead of claiming no
provider is configured.
- Raise the deduped Patrol runtime finding when scheduled runs are
skipped by a persistent readiness blocker or missing provider, so the
state reaches the findings surfaces and alert notification channels
once, instead of living only on the Patrol page banner. Transient
circuit-breaker blocks stay finding-free; the attempts that opened
the breaker already raised their own.
- Resolve the runtime finding when Patrol is turned off; opting out is
a resolution, not a state to keep nagging about.
- Record the extended runtime-failure surface in the ai-runtime
subsystem contract.
rcourtman