The retention and max-points trims re-sliced the series front
(metrics[i:]), which keeps the entire backing array reachable while
hiding the trimmed prefix from cap(). After the dense mock trends seed
(~2,800 points/series) aged out of its 6h window, every series held a
few hundred live points while pinning its seed-sized array - roughly
700MB across the demo estate's ~8,000 series. HeapSys drifted over the
droplet's memory.high, kernel reclaim throttling starved the accept
loop, and demo.pulserelay.pro served 502s (2026-08-25 outage; verified
live via pprof: appendMetric-attributed live heap grew +117MB in six
minutes after restart while goroutines stayed at 80).
Trims now compact in place so cap() reflects the true array size, and
releaseTrimmedCapacity copies the window into a right-sized array when
it occupies under a quarter of the backing array. A healthy sliding
window sits between 1x and 2x capacity and never pays the copy.
Regression tests pin both paths: a live append after an aged-out seed
backfill, and Cleanup on a mostly-expired series, including an aliasing
assertion that catches the cap()-blind front re-slice.
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).
The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.
Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.
Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.
/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.
Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
allowHostAgentReenrollLocked trusted only the durable continuity store
when one was attached, so a removal block living only in persisted
monitor state (written by releases predating the store) could never be
cleared: every fresh re-enroll read as already-consumed and the host
was rejected with 400 on every report indefinitely, regardless of new
tokens or reinstalls.
Refs #1772
With the adaptive scheduler disabled (the default), every planning pass
on the main poll tick rebuilt each instance task with NextRun=now and
Upsert overwrote the pending slot, so availability targets polled at
the tick cadence regardless of their configured interval. Reproduced
at exactly 10s polling on a 60s ICMP target; c70431caa only covered
the adaptive-scheduler path. Planning now preserves a pending future
slot, tightening it only when a freshly shortened interval justifies
an earlier run.
Refs #1745
The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.
The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
Discovering Docker inside Proxmox LXC guests previously required an
undocumented environment variable plus a hand-edited systemd unit,
which is how the surface kept shipping invisible. The opt-in is now a
persisted system setting with an admin-only toggle in Settings →
System → General, presented as its own "Docker in Proxmox LXCs" card
under the existing Docker/Podman updates card, with the requirements
(node agent with command execution) and data-collection bounds stated
inline.
The setting rides the admin-gated system settings endpoint
(RequireAdmin + settings:write, matching every other server-wide
toggle). Flipping it persists first, then applies to the runtime
config and fires a reconfigure hook so the router rebuilds the
monitor's Docker checker and inventory collector immediately — no
restart. PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY remains
authoritative: when set it locks the toggle (EnvironmentLockBadge in
the UI, HTTP 400 from the API), and an unrelated settings save can
never clobber the runtime value. The opt-in guardrail test now pins
the settings path, the env lock, and the persisted field.
Installer and node setup-guide callouts now point at the toggle first
with the environment variable as the locking override, and the docs,
their public mirrors, and the agent-lifecycle subsystem contract say
the same. New i18n strings shipped for en/de/es.
Verified live in mock mode: toggle off/on from the UI drives the
collector teardown/setup log lines, the value survives a server
restart, and all three touched surfaces render at desktop and mobile
widths (receipt in frontend-modern/browser-verification.json).
An in-progress PBS snapshot (reproduced on VM 117 at 9% written) was
counted as a completed backup: the guest's LastBackup jumped to "now" the
moment vzdump started, and the workloads badge showed a green healthy
shield for a backup that did not exist yet. If the job aborted, that
shield had never been true.
Detect the in-flight state from what the PBS listing actually says - no
size and no index.json.blob manifest in files - and flag the snapshot
InProgress instead of complete. On the PVE side, correlate storage content
entries with live vzdump tasks: a guest archive whose ctime falls inside a
running task's window is that task's partial output, and a PBS-storage
entry without a size has no manifest yet.
In-flight artifacts now feed a new per-guest BackupInProgress flag rather
than LastBackup, so the badge age always reflects the latest COMPLETED
backup. The workloads badge gets a distinct blue "Running" state (green
would claim success, None would hide that work is underway), with the
last-completed age kept in the tooltip and accessible label. Recovery
points map to OutcomeRunning with no completion time, keeping posture
freshness and the Backups page's latest-restore pointers anchored to
completed backups; the recoverable-artifact list shows a Running chip.
A cached in-flight snapshot forces a per-poll re-fetch of its group so
completion or abort is noticed promptly instead of after the cache TTL.
The pct exec socket probe that discovers Docker inside LXC guests
failed invisibly: errors logged at debug only, and a failed probe never
set DockerCheckedAt, so the guest stayed in first-check state and was
re-probed on every poll forever. On a node where lxc-attach could not
enter unprivileged guests that meant ~49 pct execs per minute of
steady-state churn, with nothing at default log level to say why the
Proxmox page showed no Docker.
Track consecutive probe failures per guest in the monitor. The first
failure of a streak, and the point where the backoff reaches its cap,
log at warn with the guest, node, and error; repeats stay at debug.
Failed probes retry with exponential backoff from one minute to a
thirty-minute ceiling, a success clears the streak with an info log,
and reconfiguring the checker resets all streaks so command execution
enabled from the UI at runtime still retries immediately. Entries not
refreshed for a day belong to deleted guests and are pruned.
PULSE_PROXMOX_GUEST_DOCKER_INVENTORY_VMIDS now gates the socket probe
as well as inventory collection. It previously only limited inventory,
so opting into specific guests still probed every running LXC on every
cycle; now guests outside the allowlist are never pct exec'd at all.
Subsystem contract and configuration docs updated to match.
An agent whose token authenticated seconds ago cannot be in a
credential outage, yet the fleet doctor flagged it critical with
"missing or revoked" whenever the token id missed the server's token
registry view, and the offered repair-authentication loop cannot fix a
stale server-side registry view. When the subject's own row shows the
judged token authenticating within the freshness window, emit a
warning-severity registry-stale verdict naming the id and the last
authentication instead. Genuinely stale rows keep the critical verdict.
Refs #1730
The Proxmox node sweep removes any non-preserved alert whose Node is
empty, and system alerts have no node, so every sweep silently deleted
the notification-delivery alert. Its five-minute evaluation then
re-raised it as a brand-new alert, firing a fresh notification each
cycle with no recovery in between, which reads as an alert appearing,
vanishing without a recovery, and paging again minutes later. System
alerts are now preserved outside node cleanup.
Raises also carried the delivery counts inside the message, and a
message change re-notifies, so each newly retained failure re-paged
even while the condition was unchanged. System alerts now take an
optional fingerprint: while level and fingerprint hold, a re-raise
refreshes the message and metadata silently. The delivery alert
fingerprints on status and failure classes, keeping its counter text
current without paging on drift.
Refs #1721
Contract-Neutral: behavioral fix: delivery-health system alert survived node sweep and stops re-paging on count drift, no public contract delta (Refs #1721)
Pulse could tell that its own delivery had stopped and had no way to say so.
Queue health was computed for one API endpoint and rendered on the alerts
surfaces, and that was the whole escalation path: a destination that stops
delivering cannot announce itself through a notification, so an operator who
does not open the alerts page learns nothing.
Telemetry: 78 installs retry every notification to the maxAttempts ceiling and
deliver almost nothing, 155,967 attempts against 221 deliveries in a week. 50
have delivered nothing in 30 days and 28 more delivered normally at some point
and silently stopped. The dominant terminal failure classes are authentication
and configuration, so most are fixable in minutes once someone knows.
Move the queue-health rule into internal/notifications as ClassifyQueueHealth
and DeliveryHealth, next to the queue that produces the counts, and delegate
the API handler to it so the endpoint and the new evaluator cannot drift into
two rules. A queue that cannot be read reports unavailable rather than healthy,
because silence is the failure mode being guarded against.
Monitoring evaluates delivery health on the poll ticker and raises or clears
the notification-delivery system alert, which puts it in the alert list and the
navigation badge. Reading queue health costs a SQLite query and the poll
cadence can be seconds, so the evaluation is throttled to five minutes.
RaiseSystemAlert is idempotent for an unchanged condition, so the timer neither
re-notifies nor accumulates alerts.
The alert is a warning rather than critical. It is real, but a new
critical-by-default alert appearing on upgrade across the affected installs is
a bigger change than this should make on its own.
The new Monitor field is deliberately named to fit inside the existing struct
alignment column. A longer name makes gofmt re-pad the whole block, which
breaks the canonical guardrail tests that pin those field declarations
verbatim.
Proof: delivery_health_test.go pins that in-flight work and retries stay
healthy while retained terminal failures do not, and that an unreadable queue
is never reported healthy. system_alerts_test.go pins the message for each
outcome including singular and plural, that it points at the destinations
surface, and that the throttle honours its interval. internal/monitoring,
internal/notifications and internal/alerts suites all pass.
Contract-Neutral: Staged contract deltas cover every boundary this change actually moves: notifications.md gains delivery_health.go plus the ClassifyQueueHealth rule, and monitoring.md gains system_alerts.go plus the system-alert evaluation extension point. The residual demands are inapplicable. agent-lifecycle is demanded only because monitor.go is one of its canonical files, and its verification artifact is host-agent deletion and re-enrollment proof, which this change does not touch: the diff to monitor.go is one struct field and one call in the poll loop. api-contracts and storage-recovery are demanded only because internal/api/notifications.go falls under a broad internal/api reference, and that file's change is a pure delegation of an internal helper to notifications.ClassifyQueueHealth with a byte-identical HTTP response and no route or payload delta.
The thresholds UI stores per-resource overrides keyed by unified
registry resource IDs, while the engine check paths look up legacy
monitor IDs, so a per-node override on an API-only PVE node saved,
displayed as Custom, and never applied: the node kept alerting at the
global defaults. Reproduced live against mock: memory at 81% with a 42%
override stored under the registry ID raised nothing.
Route the node, PMG, PBS, and host/linked-node override lookups through
the canonical registry identity resolver the intent-policy path already
uses, with the exact legacy key still winning so overrides stored by
older UI versions keep resolving unchanged.
Proof: TestNodeThresholdOverrideStoredUnderRegistryIDApplies drives the
real populate path for an API-only node, stores a memory override under
the registry resource ID the UI uses, and asserts CheckNode fires at the
override threshold while controls without the override or without the
resolver stay quiet.
Refs #1738
Contract-Neutral: behavioral fix#1738: threshold override lookup resolves registry identity, no public contract delta
A Critical "credential missing or revoked" verdict never said which
credential it judged, so a host row whose last-seen token id went stale
after re-enrollment reads as a live outage even while the agent is
reporting fine on a fresh token, and the operator has nothing to map
against the token list. Emit the judged token record UUID (not secret
material) alongside the missing and expired verdicts.
Refs #1730
Contract-Neutral: diagnostic evidence detail only in fleet doctor credential verdicts, refs #1730
Read paths refresh canonical identity on every clone, so views always
showed the derived aliases, but the stored registry entries never carried
them: snapshot ingest built resources without the canonical block and only
the IngestResources seed path stored refreshed clones. GetByReference's
canonical-alias fallback reads the stored entries directly, so prefixed
refs like "agent:{hostID}" never resolved on a production rebuild and
per-resource alert grace overrides on merged PVE-node + pulse-agent hosts
silently fell back to factory policy. The earlier alias fix (dd008f1c2)
only proved the IngestResources-seeded path. Recompute identities in each
ingest finalize pass, after links and merges settle.
Proof: TestMergedProxmoxHostAgentGraceOverrideHoldsCPUAlert drives the
real populate path (node plus linked agent merging into one resource),
saves a 600s metric.cpu override under the registry resource ID the UI
uses, and asserts CheckHost holds the alert while a control manager
without the override fires immediately.
Refs #1497
Contract-Neutral: behavioral fix#1497: populate derived canonical identity on stored registry entries, clones already exposed identical data, no public contract delta
The broadcast ticker logged seven fields at Info level every tick and
built a full state snapshot to do it, before checking whether any
WebSocket subscriber existed. An idle server paid a snapshot copy plus
roughly 8600 log lines a day for nothing. The log is now Debug and both
the log and the snapshot build sit behind the subscriber check.
The poll task worker pool was also fixed at ten workers no matter how
many instances an estate monitors. POLL_TASK_WORKERS now overrides the
count and the cap, bounded at 128, following the existing env knob
pattern. Behavior without the variable is unchanged.
Contract-Neutral: operational log demotion and an opt-in worker pool env knob, no monitoring or agent-lifecycle contract semantics change
matchZFSPoolForStorage's sole-pool fallback attached the node's only ZFS
pool to every storage the per-node endpoint returned, including NFS, CIFS,
PBS, and RBD entries that can never be backed by a local pool. One failing
device then raised a duplicate ZFS device alert per shared storage. Gate
the matcher on isInherentlySharedStorageType so remote-backed storages are
never matched, while dir-type storages keep the single-pool fallback.
Refs #1731
An explicit connection address override on a cluster member was silently
discarded when VerifySSL was enabled and the member had no per-endpoint
fingerprint: the hostname-for-TLS preference displaced the operator's
address, so overriding an undialable discovered hostname changed nothing.
The override now wins in every TLS mode, and the hostname preference
applies only to auto-discovered addresses.
Failed endpoint recovery attempts also logged their cause at debug level
only, leaving the recurring 'No endpoints recovered' warning without a
reason. The warning now carries per-endpoint failure reasons and the
sanitized error is stored so the UI health status shows it too.
Refs #1665
Contract-Neutral: behavioral bug fix refs #1665: cluster member IPOverride honored in all TLS modes, recovery failure reasons surfaced; no public contract delta
The host agent's node-local pct df inventory was only applied by the
per-node container poll fallback. Cluster instances are served by the
efficient cluster/resources path, which never called the enrichment, so
installs on that path showed rootfs-only container filesystems (and,
since the config-mount restoration, config-listed mounts with unknown
usage) no matter how healthy the linked agent was. Reported on #1477
after the reporter installed agents specifically to get per-mount usage.
Reproduced end to end against a live PVE cluster: the agent shipped the
full inventory, the server linked the agent and populated the
filesystem cache every report, and the poll path never read it. With
the enrichment applied after metadata enrichment, mirroring the
per-node path's ordering, the same rig surfaces real per-mount usage.
Refs #1477
Contract-Neutral: behavioral fix: cluster/resources poll path now applies agent pct df enrichment like the per-node path (#1477), no public contract delta
Every poll tick rebuilds the adaptive plan, and BuildPlan anchored the
next run on the previously planned NextRun by unconditionally adding the
selected interval. While a planned slot has not elapsed, each planning
pass therefore pushed it another full interval into the future. With the
default 10s tick that compounds NextRun ahead of wall-clock as soon as
the adaptive interval stretches beyond the tick cadence, so an instance
whose data was fresh at startup was never due again: API polling starved
permanently, the connection dropped to Agent and stale exactly when the
stretch began, and no error was ever logged. Matches the fourth #1437
reproduction (standalone degrades at ~2 minutes, cluster at ~4, stays
degraded), whose bundle shows a clean log with no polls after startup.
A pending future slot is now kept as planned, and it tightens to
now+interval when a staleness-driven interval shrink justifies an
earlier run. An elapsed slot still advances exactly one interval, so
ordinary cadence is unchanged.
Refs #1437
Contract-Neutral: behavioral fix: pending planned poll slot no longer compounds NextRun past wall-clock (#1437), no public contract delta
Integration test for the exact production path: stock PVE answers the
LXC status query with an empty diskinfo map, enrichContainerMetadata
discovers the mpX mount from the container config alone, and the unified
resource projection serializes it with capacity, the -1 unknown-usage
sentinel, and an omitted used field while the live rootfs row survives.
Related to #1477
Stock Proxmox reports no per-mount LXC usage through the status API, so
v6's API-polled containers listed only rootfs. The v5.1.32 fallback that
synthesized mount rows from the container config never crossed to the v6
line, and the v6.2.0 pct-df agent path only covers nodes running the
unified agent. Restore the fallback and improve it: parse size= so
config-only rows carry capacity, mark live usage unknown with the -1
sentinel, and merge without displacing the aggregate-seeded rootfs row.
Frontend consumers stop fabricating percents for sentinel rows: the
workloads row bar and summary math exclude them (tooltip lists them with
capacity), the drawer Filesystems block renders ?/<size> with no percent,
disk normalization preserves the sentinel, and per-machine max-disk
derivations skip them. Mock mode seeds one running container in this
exact shape so the surfaces stay exercised.
Related to #1477
2bc4ed725 moved the unified metrics sync sites to WriteBatchBounded but
left the three source guardrails pinning the literal WriteBatchSync
call, so they failed. The guardrails' intent is that pipeline writes go
through the canonical batched ingestion path, which WriteBatchBounded
is; pin the new name.
Contract-Neutral: test-only: guardrail pins follow the WriteBatchBounded pipeline rename from 2bc4ed725
TestSeedMockMetricsHistory_SeedsVMwareMetricsStore and the TrueNAS
variant compared seeded storage values against the package-level
mock.SampleMetric, which resolves resource roles from the global
registry that only other tests populate. The assertions therefore
passed or failed depending on which tests ran earlier in the process:
isolated runs failed deterministically, and today's CI reshard flipped
the rest-0 shard red for commits that never touched the mock layer.
Both tests now build the same graph-aware sampler the seeder uses, so
the expectation is self-contained and order-independent.
Contract-Neutral: test-only: seed assertions sample via the seeder's graph sampler, removes cross-test registry dependence
6b79aa997 bounded WriteBatchSync itself, which broke its read-your-writes
contract on slow disks: CI's metrics write-amplification and 500-node
load tests count committed rows after writing, and mock seeding reads
store coverage straight back, so the 2-second early return failed both
(runs 31475700902, 31494553977). Fast local disks masked it.
WriteBatchSync returns to a full commit wait. The monitoring pipeline's
four sync sites move to WriteBatchBounded, which carries the bounded
enqueue-plus-wait semantics, so the #1437 slow-disk stall fix stays
exactly where the hazard is. Both paths share prepareWriteBatch
validation, and a new regression test pins WriteBatchSync waiting past
the bounded budget.
Refs #1437
Contract-Neutral: behavioral fix: split bounded pipeline writes from synchronous batch writes, restores read-your-writes (#1437 follow-up), no public contract delta
Adaptive polling deliberately stretches an instance's cadence toward its
max interval (5 minutes by default) while data is fresh, but the
connections aggregator judged staleness against the configured cadence
with a 2-minute floor. Any adaptive-enabled install therefore cycled
healthy PVE/PBS/PMG connections into stale for the back half of every
stretched poll gap: the Infrastructure page dropped the source badge
from API + Agent to Agent and connection-degraded alerts fired against
a schedule the poller was honoring.
The aggregator now scales the active-to-stale cutoff by the scheduler's
currently planned interval when that exceeds the configured cadence,
via Monitor.PlannedPollInterval and per-instance planned intervals in
the aggregator inputs. A plan tighter than the configured cadence never
tightens the cutoff, so genuine poll outages still trip the floor on
time. Connection-degraded alerts and the runtime inventory gate consume
the same derived state and inherit the fix.
Refs #1437
Contract-Neutral: behavioral fix: stale cutoff follows adaptive planned interval (#1437), no public contract delta
The 2026-08-07 telemetry read showed installs at or above 5 PVE nodes, 10
Docker hosts, or 3 VMware hosts convert to paid at ~8x the rate of smaller
estates. The ping now carries that classification as a server-derived
boolean so receiver-side cohort queries keep a stable column even if the
thresholds move later; it is derived in the pkg/server snapshot closure
from the same AggregateInstallSnapshotCounts values the payload already
sends, so no new information leaves the install.
The thresholds move to internal/monitoring/business_estate.go as the
single definition; the session-capability surface behind the in-product
business-estate card delegates to it, and dropping the now-unneeded direct
pkg/licensing import there restores TestPkgLicensingImportBoundary, which
f0e2243b4 had left red. All three payload surfaces (Ping struct, private
receiver, TelemetryPingPreview) move together per
check_telemetry_schema_parity.py, and both PRIVACY.md copies document the
field.
Verified live on an isolated worktree backend with 6 mock PVE nodes: the
Settings telemetry preview renders schema_version 8 with business_estate
true, and /api/security/status still reports
sessionCapabilities.businessEstate true through the delegated thresholds.
The #1601 follow-on: per-container alert overrides were keyed by Docker
container ID (docker:{host}/{containerID}), which changes on every
recreate, so each image update silently re-armed alerts the user had
switched off and left a dead entry behind in alerts.json — the unbounded
growth that pushed the reporter's config past the old 64KB body cap
(raised in 38434a513). The v6 thresholds UI additionally wrote keys from
the unified hash id (docker:{host}/app-container-{16hex}), which the
evaluator never read at all.
Overrides now key on stable identity, docker:{host}/{containerName}:
- The evaluator resolves the name key first and falls back to the legacy
container-ID key so pre-migration entries keep working
(evaluateDockerContainer, checkDockerContainerState, the
container-update resolver, and reevaluateActiveAlertsLocked).
- MigrateDockerContainerOverrideKeys runs in the monitor sync next to
MigrateCanonicalOverrideKeys, driven by the unified resource snapshot:
it re-homes live legacy-ID and unified-hash keys onto the name key and
prunes orphaned ID-shaped entries, ending the per-update orphan
accumulation. Name-keyed entries for absent containers are kept so a
recreate under the same name still honours them.
- The UI candidate chain (single implementation in alertOverridesModel)
now leads with docker:{host}/{name} and trails the container-ID, short
ID, unified-hash and slash-tail forms, so rows bind pre-existing
overrides of every historical shape and the next save re-homes them.
Rows carry overrideStorageId/overrideIdCandidates so toggle,
connectivity, offline-state, edit and remove all write the stable key.
- The ignored-containers card copy now documents the wildcard forms
(runner-*, *-dev, *staging*) shipped in b5fa6a9af, under the title
"Ignored container patterns".
Contract deltas: alerts, frontend-primitives, monitoring, and
unified-resources now pin the name-keyed override identity, the single
frontend candidate-chain owner, the sync-cadence migration, and the
resource-facet-backed table identity respectively.
Verified live against a mock instance: a UI toggle persists
docker:{host}/loki and binds back after reload, and seeded
legacy/hash/orphan keys converge to name keys on disk within two sync
ticks. go test ./internal/alerts/... ./internal/monitoring/... green;
recreate survival pinned in
TestDockerContainerOverrideSurvivesContainerRecreate.
Refs #1601
Mock mode suspends pull-based collection outright, but push-based agent
reports were never given the same treatment, so a real machine still landed
in monitor state while the unified read path substituted the mock snapshot
over the top. The hosts were hidden, everything downstream was not: a real
Unraid box raised a live storage-topology alert next to fixture data, and
its identity persisted through host continuity.
Three vectors, each closed at its source.
Agent ingest now drops real reports while mock mode is on. ApplyHostReport,
ApplyDockerReport and ApplyKubernetesReport acknowledge the report with the
reporting agent's own identity and touch no state, so nothing raises alerts,
persists continuity, records metrics or feeds the online/offline sweep. The
acknowledgement stays a success so a real agent does not read a demo server
as an outage and retry-storm it.
recentStandaloneHostContinuityEntries returns nothing in mock mode. Those
entries are written to disk from real reports and outlive the toggle, and
every consumer injects them after the read path has already substituted the
mock snapshot, so a machine that reported before mock mode was enabled came
back by its real hostname. There is no real-polling exception here: agent
ingest is not gated on PULSE_MOCK_KEEP_REAL_POLLING and the read state is
mock either way.
Active-alert restore is now opt-out, and mock mode opts out. SetMockMode
already clears active alerts when the toggle flips, but a process booting
with mock mode already enabled never ran that path and restored real alerts
from active-alerts.json.
TestHostedTenantAgentInstallTokenCannotReportToOtherTenant used mock mode as
scaffolding. Under the ingest guard both tenants would be empty and its
isolation assertion would pass without exercising the boundary, so it now
runs in real mode. Every new test pairs the mock assertion with a real-mode
one for the same reason.
rcourtman