The 2026-08-07 telemetry read showed installs at or above 5 PVE nodes, 10
Docker hosts, or 3 VMware hosts convert to paid at ~8x the rate of smaller
estates. The ping now carries that classification as a server-derived
boolean so receiver-side cohort queries keep a stable column even if the
thresholds move later; it is derived in the pkg/server snapshot closure
from the same AggregateInstallSnapshotCounts values the payload already
sends, so no new information leaves the install.
The thresholds move to internal/monitoring/business_estate.go as the
single definition; the session-capability surface behind the in-product
business-estate card delegates to it, and dropping the now-unneeded direct
pkg/licensing import there restores TestPkgLicensingImportBoundary, which
f0e2243b4 had left red. All three payload surfaces (Ping struct, private
receiver, TelemetryPingPreview) move together per
check_telemetry_schema_parity.py, and both PRIVACY.md copies document the
field.
Verified live on an isolated worktree backend with 6 mock PVE nodes: the
Settings telemetry preview renders schema_version 8 with business_estate
true, and /api/security/status still reports
sessionCapabilities.businessEstate true through the delegated thresholds.
The #1601 follow-on: per-container alert overrides were keyed by Docker
container ID (docker:{host}/{containerID}), which changes on every
recreate, so each image update silently re-armed alerts the user had
switched off and left a dead entry behind in alerts.json — the unbounded
growth that pushed the reporter's config past the old 64KB body cap
(raised in 38434a513). The v6 thresholds UI additionally wrote keys from
the unified hash id (docker:{host}/app-container-{16hex}), which the
evaluator never read at all.
Overrides now key on stable identity, docker:{host}/{containerName}:
- The evaluator resolves the name key first and falls back to the legacy
container-ID key so pre-migration entries keep working
(evaluateDockerContainer, checkDockerContainerState, the
container-update resolver, and reevaluateActiveAlertsLocked).
- MigrateDockerContainerOverrideKeys runs in the monitor sync next to
MigrateCanonicalOverrideKeys, driven by the unified resource snapshot:
it re-homes live legacy-ID and unified-hash keys onto the name key and
prunes orphaned ID-shaped entries, ending the per-update orphan
accumulation. Name-keyed entries for absent containers are kept so a
recreate under the same name still honours them.
- The UI candidate chain (single implementation in alertOverridesModel)
now leads with docker:{host}/{name} and trails the container-ID, short
ID, unified-hash and slash-tail forms, so rows bind pre-existing
overrides of every historical shape and the next save re-homes them.
Rows carry overrideStorageId/overrideIdCandidates so toggle,
connectivity, offline-state, edit and remove all write the stable key.
- The ignored-containers card copy now documents the wildcard forms
(runner-*, *-dev, *staging*) shipped in b5fa6a9af, under the title
"Ignored container patterns".
Contract deltas: alerts, frontend-primitives, monitoring, and
unified-resources now pin the name-keyed override identity, the single
frontend candidate-chain owner, the sync-cadence migration, and the
resource-facet-backed table identity respectively.
Verified live against a mock instance: a UI toggle persists
docker:{host}/loki and binds back after reload, and seeded
legacy/hash/orphan keys converge to name keys on disk within two sync
ticks. go test ./internal/alerts/... ./internal/monitoring/... green;
recreate survival pinned in
TestDockerContainerOverrideSurvivesContainerRecreate.
Refs #1601
Mock mode suspends pull-based collection outright, but push-based agent
reports were never given the same treatment, so a real machine still landed
in monitor state while the unified read path substituted the mock snapshot
over the top. The hosts were hidden, everything downstream was not: a real
Unraid box raised a live storage-topology alert next to fixture data, and
its identity persisted through host continuity.
Three vectors, each closed at its source.
Agent ingest now drops real reports while mock mode is on. ApplyHostReport,
ApplyDockerReport and ApplyKubernetesReport acknowledge the report with the
reporting agent's own identity and touch no state, so nothing raises alerts,
persists continuity, records metrics or feeds the online/offline sweep. The
acknowledgement stays a success so a real agent does not read a demo server
as an outage and retry-storm it.
recentStandaloneHostContinuityEntries returns nothing in mock mode. Those
entries are written to disk from real reports and outlive the toggle, and
every consumer injects them after the read path has already substituted the
mock snapshot, so a machine that reported before mock mode was enabled came
back by its real hostname. There is no real-polling exception here: agent
ingest is not gated on PULSE_MOCK_KEEP_REAL_POLLING and the read state is
mock either way.
Active-alert restore is now opt-out, and mock mode opts out. SetMockMode
already clears active alerts when the toggle flips, but a process booting
with mock mode already enabled never ran that path and restored real alerts
from active-alerts.json.
TestHostedTenantAgentInstallTokenCannotReportToOtherTenant used mock mode as
scaffolding. Under the ingest guard both tenants would be empty and its
isolation assertion would pass without exercising the boundary, so it now
runs in real mode. Every new test pairs the mock assertion with a real-mode
one for the same reason.
Seeded mock history recorded docker hosts with cpu, memory and disk only,
while the synthetic generator used past the seed window emits the full guest
metric set. The result was an inversion across chart ranges: a docker host had
no diskread, diskwrite, netin or netout history at 5m through 24h and full
history at 7d and 30d. Real docker hosts report both through the agent, so the
seed now covers the same series the generator does and every range agrees.
The guardrail test asserts seeded coverage matches the synthetic metric set
rather than a hand-listed set, so a future series added to one path cannot
quietly skip the other.
Mock history is seeded for 48h, so chart windows longer than that fall through
to the synthetic generator in mock_chart_history.go. That generator produced
cpu, memory, disk and the I/O pairs but never memoryused, so a 7d workloads
read returned 64 points for every other series and zero for memoryused. The
memory column in host-capacity mode had no series to draw at all, which reads
as a broken column rather than missing mock data. Real installs are unaffected:
the live PVE tick writes memoryused to the metrics store and the store rollup
groups by metric_type without an allowlist.
The synthetic generator now derives memoryused from the sampled memory
percentage and the fixture memory capacity, the same derivation live mock ticks
and the seeder already use, so the series stays continuous across the seed
boundary. Capacity comes from a new fixture registry beside the existing metric
role registry rather than a per-call fixture graph clone. Docker containers and
pods stay out of it, matching the Proxmox-only memoryused contract.
Update source-shape guards for the shared per-generation resource list and cached mock unified view. Demote the onboarding E2E spec after its rate-limit/shared-state retry flake on main so it continues running without reddening the advisory verdict.
Contract-Neutral: advisory CI guardrails and probation tier bookkeeping only; no runtime or public contract delta
persistGuestIdentity spawned a detached goroutine per changed guest to write
guest_metadata.json, with a comment noting it avoided blocking the monitor.
Nothing tracked those goroutines, so neither Monitor.Stop nor
MultiTenantMonitor.Stop could wait for them and a queued write could land after
shutdown. In hosted mode that means a write into a tenant directory that
offboarding is already removing, and a stray guest_metadata.json.tmp left
behind when the atomic write is interrupted.
The store now owns the goroutine. SetAsync tracks the write on a WaitGroup and
WaitForPendingWrites drains it under a bounded timeout matching
tenantMonitorShutdownTimeout, so a wedged store cannot hold up tenant teardown.
Monitor.Stop drains before closing the metrics store.
This is what made TestHostedTenantAgentInstallTokenCannotReportToOtherTenant
flaky: t.TempDir cleanup raced a queued write into orgs/client-b and failed
with "directory not empty". The test itself is unchanged, because it was never
a test bug. A goroutine dump at cleanup time showed the writers still live,
created by persistGuestIdentity, blocked on the store mutex.
Verified causally rather than by observation alone: the target test fails 0/4
with the drain removed and passes 8/8 with it, against 2/3 failures on the
unmodified baseline. The regression tests fail if SetAsync stops tracking its
goroutine.
Note for a future pass, deliberately not changed here: each changed guest still
triggers a full-file save, so one poll cycle over N changed guests does N
marshals and N atomic writes that serialize on the store mutex anyway. Fixing
that means coalescing at the call site and is a behavioural change beyond this
defect.
Fixes#1681Fixes#1682Fixes#1683
Contract-Neutral: Notification grouping initialization and alert-config propagation do not alter the broadly referenced agent-lifecycle or storage-recovery contracts; primary alerts, notifications, API, and monitoring contracts and regression proofs are updated.
In mock mode every unified read-state access built two throwaway
registries: mock.UnifiedResourceSnapshot constructed one to derive the
resource list, and the monitor's currentUnifiedStateView ingested that
list into another, deep-cloning all resources both ways. Chart requests,
broadcasts, alert evaluation, and API reads each repaid that full cost —
the dominant share of the demo's 76TB/9.5d allocation churn, since every
one of those reads runs against a world that only changes on the 2-second
mock tick.
Introduce fixtureDataVersion, a token that advances on every observable
mock-graph change (metric ticks and the structural changes that bump
fixtureRevision, which stays structural-only so seeded trend history
remains reusable). Memoize the package-level UnifiedResourceSnapshot and
the monitor's mock-branch state view against it, so consumers between
ticks share one immutable build. Sharing mirrors the semantics the
persistent-store ReadState path has always had in real mode: all
consumers were audited — they ingest (which clones), copy before
top-level writes, or build fresh outputs. Real-mode paths are untouched.
Contract-Neutral: mock snapshot memoization: identical data served from cache, no contract delta
Mock mode seeded 90 days of synthetic history for every resource at
startup. Measured on the demo droplet with identical entity counts, that
put the process near 1GB of Go heap before the first tick; a 6h seed
runs at ~200MB. Every local dev instance and CI run in mock mode paid
that gigabyte silently, and on 1GB hosts it OOMed the box.
48h keeps day-scale charts fully populated. Long-range chart work opts
in with PULSE_MOCK_TRENDS_SEED_DURATION, which is how the public demo
already pins its own seed (6h).
Contract-Neutral: mock seed default resize: dev-only mock path, no contract delta
A full monitor reload (triggered by every node settings save) starts
from empty host state, and the physical disk poll runs on the first
cycle. Evaluating disk alerts before a node's agent has re-reported
treats the agent's --disk-exclude patterns as absent, so excluded disks
fired wearout/health alerts that the next cycle resolved. Defer disk
alert evaluation for unlinked nodes during a two minute settle window
after monitor start.
Refs #1674
Contract-Neutral: behavioral fix: defer disk alert evaluation until host-agent linkage settles after reload, no contract delta (#1674)
golangci-lint had accumulated 12 findings since 5abb2d8f4. All fixed with
real dedup (no nolint suppressions) and the repo's existing errcheck idioms:
- dupl internal/monitoring: docker/host identity-conflict trackers were
structural clones; extracted a shared identityFlapTracker core with a
domain-neutral identityConflict result. Per-domain files now hold only
the window const and the model translation. Tracker-behavior tests
consolidated into identity_flap_tracker_test.go; Monitor-level
translation and Apply*Report integration tests remain per domain.
- dupl internal/api/router.go: VM/container workload chart loops shared a
16-line live-fallback block; extracted guestChartSeriesWithLiveFallback
over a guestLiveMetricsView interface both views satisfy.
- dupl internal/storagehealth/risk.go: SMART attribute copying extracted
into applySMARTAttributes shared by both assessors (same
*models.SMARTAttributes type on both inputs).
- errcheck pkg/audit/sqlite_logger.go: three defer tx.Rollback() sites
now use the repo-wide defer func() { _ = tx.Rollback() }() idiom.
- errcheck telemetry/notifications tests: send() errors now fail the
test; queue.Stop() uses the package's _ = idiom.
Full test suites pass for all six touched packages.
Contract-Neutral: lint-hygiene restoration: dupl dedup (identical logic extracted to shared helpers) and errcheck idiom fixes; no public-contract or behavioral delta
A guest's unified canonical ID hashed its node-scoped source ID
(instance:node:vmid), so a live migration to another cluster node
re-minted the resource and orphaned every operator-owned row keyed by
the old ID: explicit availability check links (fail-closed by design,
the reported symptom in #1669), alert overrides, operator state, action
audits, manual links, and recovery subjects. VMIDs are unique within a
cluster, so guests now derive their canonical ID from instance+VMID
("proxmox-guest:<instance>:<vmid>") and keep it across migrations. The
guest-metadata half of #1669 was fixed separately at the metadata-store
layer.
Existing installs converge through record-declared succession: ingest
declares the retired node-scoped IDs superseded for every node the
instance currently knows (current names plus native aliases), so rows
orphaned by pre-upgrade migrations also re-key. Successions are now
recorded durably in a canonical_id_successions table, which memoizes
the re-key (steady-state rebuilds re-declare the same eras every tick
without touching SQL) and lets change-journal reads merge retired guest
eras the way pin EraIDs do for hosts. The succession re-key also covers
manual link/exclusion rows.
Availability links resolve retired canonical IDs and old-node source
triples through a registry superseded index plus guest-triple parsing
(persistence keys only, ambiguity fails closed), and the stored
LinkedResourceID re-homes to the current canonical ID on the
alert-migration cadence. Recovery subjects converge on the same
derivation via CanonicalSubjectResourceID, the mapper's registry-miss
fallback, node-independent external guest keys, and the store's startup
backfill, which also sweeps posture rows stranded under retired subject
keys. Metrics history and frontend row identity key off the node-scoped
source ID and are deliberately unchanged.
A hostname change on the same machine and token forked the host module
onto a suffixed identity to stay safe against cloned VMs sharing a
machine ID (#1584), but the fork never healed, so one machine stayed
split into two agent identities forever. In unified installs the
workload modules keep reporting under the base agent ID, so Agent
Doctor flagged both halves with no way to merge them (#1667, Home
Assistant add-on with agent_hostname set after first boot).
Once the pre-rename record has stopped reporting for three health
windows the collision is proven to be a rename, not a clone: the stale
record and the forked identity are superseded and the binding returns
to the base identity. A rename discovered while the old record is
already stale now adopts the base identity without forking at all.
Live clones keep reporting under the old hostname, so they never
match the heal condition and stay forked.
Contract-Neutral: host agent identity rename heal; behavioral bug fix, no wire contract or payload change
Every /api/state request and websocket hydrate ran a full unified
registry rebuild while holding the adapter's mutation lock, including
synchronous SQLite writes for change records and identity pins plus
re-reads of overrides and pins on registry construction. On slow
volumes a single rebuild holds the lock for the duration of those
transactions, and every state read queues behind it. #1665 hit exactly
this with the data dir on NFS, where the same instance's metrics logs
show single commits taking 30+ seconds, and the UI sat on /api/state
for minutes.
Read paths now refresh through TryReplaceRegistryForRead. It skips
while the current generation is younger than two seconds, collapsing
same-cycle rebuild storms, and it never queues behind an in-flight
ingest rebuild since that rebuild is already publishing a generation
at least as fresh. Consume-once supplemental payloads are only drained
once a rebuild commits. Cold start still blocks and builds the first
generation so a fresh session cannot render empty. Ingest boundaries
keep rebuilding eagerly through the unchanged populate methods.
Contract-Neutral: internal registry rebuild scheduling on the state read path; no wire contract or payload change
Connected systems showed a PBS machine twice (API row plus host-agent row)
whenever the PBS connection was configured with an address the agent never
reports, because PBS identity was limited to the configured name and host.
The PBS poll now captures the hostname the node reports about itself
(GET /nodes, the previously unused client GetNodeName) on
models.PBSInstance.NodeName, and /api/connections includes it in the PBS
connection's host aliases, so directPlatformHostAttachment merges the agent
running on that machine into the PBS row the same way PVE composes from
API-reported node identity. Node-name fetch failure stays partial data and
never fails the poll. Reported by Johannes Strasser (rc.5).
Do not let a short per-request connection timeout shrink the whole inventory cycle below its 90-second default. The seven-node support bundle for #1437 showed hundreds of successful guest reads followed by exact 30-second cancellation and dead-lettering, preventing a coherent generation from publishing. Larger request timeouts can still expand the cycle up to MAX_POLL_TIMEOUT.\n\nFixes #1437
Proxmox's per-node storage endpoint GET /nodes/{node}/storage does not
return what the node can actually use. It returns every storage in the
datacenter config and reports the ones the node is excluded from with
enabled:0/active:0 instead of leaving them out. pollStorageWithNodes
ingested every row it got back, so a storage restricted to one node
appeared on all the others as a disabled entry, which the UI renders as
Offline.
The poller already fetches the datacenter config once via GetAllStorage
and keeps it keyed by storage name, so the restriction is in hand at the
point each per-node row is turned into a models.Storage. Skip the row
when the config carries a non-empty nodes restriction that does not
include the node being polled. Node names are compared case-insensitively
to match how node identity is compared elsewhere in the poller.
Only the restriction drops a row. A storage with no restriction still
shows up on every node it is reported from, including when it is disabled
everywhere, so a datacenter-wide disabled storage stays visible as
disabled rather than vanishing. Because the shared-storage aggregation
builds Nodes/NodeIDs/NodeCount from the surviving per-node rows, this also
stops a restricted shared storage from claiming cluster members that
cannot mount it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reset added in b45bd66b9 only fired when the temperature SSH key
file on disk changed (mtime/size). An operator who repairs SSH access
any other way — fixing authorized_keys on the host, repairing
known_hosts, restoring network reachability — still waited out a
backoff window that may have compounded toward fifteen minutes.
A system-settings save is the natural operator touchpoint after such a
repair, so the settings handler now fans ResetSSHFailureBackoff out to
every live tenant monitor after a successful save, clearing the
per-host temperature SSH backoff and the knownhosts keyscan backoff.
The reset touches in-memory retry timing only; nothing is persisted
and no request field controls it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The discovery-policy check resolved endpoint hostnames with a bare
net.LookupIP while the actual dials went through pkg/tlsutil's process-global
cached resolver, so the policy and the connection reasoned about two different
DNS views. That split is why 108aa4e20 had to skip resolution entirely for the
default policy, which left the injected 169.254.0.0/16 blocklist enforced only
against literal IPs: a hostname endpoint pointed at the metadata range walked
straight through.
Resolve through tlsutil.LookupHostCached instead. The shared resolver caches
answers and lookup failures alike until its next refresh, so repeat poll cycles
cost a cache hit rather than a query and the per-poll DNS volume that opened
#1638 stays gone. With that in place the default-policy skip is removed and the
blocklist applies to resolved addresses again, and the five-minute decision
cache is dropped rather than kept: it bought nothing on top of the resolver
cache, made the verdict trail the configuration, and memoized the fail-open
"resolution failed, allow" outcome for minutes even with an explicit allowlist
configured. Its claim to match a DNS refresh interval that operators configure
through DNS_CACHE_TIMEOUT goes with it.
The SSH backoffs now only escalate for work that ran. A knownhosts manager
suppressing a call inside its own window reports ErrKeyscanSuppressed, and the
temperature layer neither records a failure nor pays for the RPi fallback in
that case. An expired collection deadline is our own budget rather than
evidence about the host, so it holds the window at the floor. Both backoffs
decay once a retry deadline is more than one window past, and replacing the
temperature SSH key on disk clears both maps so a repaired key is tried on the
next cycle instead of after fifteen minutes.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rcourtman