Seeded mock history recorded docker hosts with cpu, memory and disk only,
while the synthetic generator used past the seed window emits the full guest
metric set. The result was an inversion across chart ranges: a docker host had
no diskread, diskwrite, netin or netout history at 5m through 24h and full
history at 7d and 30d. Real docker hosts report both through the agent, so the
seed now covers the same series the generator does and every range agrees.
The guardrail test asserts seeded coverage matches the synthetic metric set
rather than a hand-listed set, so a future series added to one path cannot
quietly skip the other.
Mock history is seeded for 48h, so chart windows longer than that fall through
to the synthetic generator in mock_chart_history.go. That generator produced
cpu, memory, disk and the I/O pairs but never memoryused, so a 7d workloads
read returned 64 points for every other series and zero for memoryused. The
memory column in host-capacity mode had no series to draw at all, which reads
as a broken column rather than missing mock data. Real installs are unaffected:
the live PVE tick writes memoryused to the metrics store and the store rollup
groups by metric_type without an allowlist.
The synthetic generator now derives memoryused from the sampled memory
percentage and the fixture memory capacity, the same derivation live mock ticks
and the seeder already use, so the series stays continuous across the seed
boundary. Capacity comes from a new fixture registry beside the existing metric
role registry rather than a per-call fixture graph clone. Docker containers and
pods stay out of it, matching the Proxmox-only memoryused contract.
The workloads and Proxmox nodes trend cells drew every percent series on a
fixed 0-100 window inside a 16px cell, so any series living in the bottom of
that range rendered on top of the axis rule and read as missing data. A guest's
share of host memory is single digits by construction: at 2% the path sat at
y=15.7 against a baseline at y=16, which is what "memory column trends not
working" looks like. Idle guests hit the same wall on CPU.
Percent series now scale to their own peak like the I/O series already do,
zero-floored, with a 5% floor ceiling so idle noise stays flat rather than
amplified, and still capped at 100. Bars mode remains the level view and every
cell keeps its current-value label, so nothing loses the absolute reading.
Advance the governed install, Docker, and Helm metadata together; add the RC9 release packet for the post-RC8 notification, lifecycle, and resource-read fixes; and record the existing mobile-candidate compatibility decision.
ResourceHandlers.getStore opens a SQLite handle per org and caches it for the
process lifetime, and nothing ever closed them. CleanupTenant already released
patrol, AI, RBAC, license and monitor-adapter state for a deleted org but left
the resource store open, so an offboarded tenant kept its file descriptors and
its unified_resources.db-wal/-shm files alive, and its directory could not be
fully removed.
CloseTenantStore releases and evicts one org's store and is now called from
CleanupTenant alongside the other per-tenant teardown. CloseStores releases all
of them, exposed as Router.ShutdownResourceStores next to the existing
Shutdown*/Stop* helpers.
Found while investigating the flaky hosted-tenant test. It is not what made that
test flaky, which was a detached guest-metadata write fixed separately, but the
leak is real on its own: verified by observing that -wal and -shm sidecars, which
exist only while a connection is open, survived the test before this change and
do not after it.
persistGuestIdentity spawned a detached goroutine per changed guest to write
guest_metadata.json, with a comment noting it avoided blocking the monitor.
Nothing tracked those goroutines, so neither Monitor.Stop nor
MultiTenantMonitor.Stop could wait for them and a queued write could land after
shutdown. In hosted mode that means a write into a tenant directory that
offboarding is already removing, and a stray guest_metadata.json.tmp left
behind when the atomic write is interrupted.
The store now owns the goroutine. SetAsync tracks the write on a WaitGroup and
WaitForPendingWrites drains it under a bounded timeout matching
tenantMonitorShutdownTimeout, so a wedged store cannot hold up tenant teardown.
Monitor.Stop drains before closing the metrics store.
This is what made TestHostedTenantAgentInstallTokenCannotReportToOtherTenant
flaky: t.TempDir cleanup raced a queued write into orgs/client-b and failed
with "directory not empty". The test itself is unchanged, because it was never
a test bug. A goroutine dump at cleanup time showed the writers still live,
created by persistGuestIdentity, blocked on the store mutex.
Verified causally rather than by observation alone: the target test fails 0/4
with the drain removed and passes 8/8 with it, against 2/3 failures on the
unmodified baseline. The regression tests fail if SetAsync stops tracking its
goroutine.
Note for a future pass, deliberately not changed here: each changed guest still
triggers a full-file save, so one poll cycle over N changed guests does N
marshals and N atomic writes that serialize on the store mutex anyway. Fixing
that means coalescing at the call site and is a behavioural change beyond this
defect.
Fixes#1681Fixes#1682Fixes#1683
Contract-Neutral: Notification grouping initialization and alert-config propagation do not alter the broadly referenced agent-lifecycle or storage-recovery contracts; primary alerts, notifications, API, and monitoring contracts and regression proofs are updated.
Schema v6 shipped audit_logging_persistent and audit_events_30d as Pro adoption
signals. Neither discriminated. pkg/server installs the SQLite audit logger on
every install for defense in depth and gates only the read/export endpoints, so
the boolean was true on all 8 installs that had taken rc.8 and 0 rows in the
retained table have ever had it false. The event count measured that background
write volume: three of those eight unlicensed community installs were pegged at
the receiver's 100000 clamp ceiling, with the rest between 4863 and 67509.
Schema v7 replaces both with audit_reads_30d, a count of requests that cleared
the license gate on an audit read or export surface. A read requires a human
action, so unlike store presence or write volume it cannot settle into a
constant. The recorder is wrapped INSIDE RequireLicenseFeature so unentitled
requests never count, and the persisted marker carries a timestamp and a coarse
activity class from a fixed allowlist. Query filters, actors, ranges, and every
audit row read stay on the install.
The retired columns are left in the live database. They hold real rc.8 rows and
migrations only add, so dropping them would be a pointless risk; nothing writes
them once the receiver struct loses the fields.
Adds the guard this class needed. LicensedFeatureAdoptionFields registers every
field that exists to measure licensed-feature adoption, and
TestLicensedFeatureAdoptionFieldsDiscriminate builds an unused install through
the real production snapshot paths, installs a real SQLite audit logger exactly
as pkg/server does, records a baseline audit event, and fails if any registered
field is non-zero. Pinning a console logger there would have made the guard pass
while the payload lied, so it deliberately does not. The guard was verified by
reintroducing the v6 sourcing and confirming it fails with the field named.
A companion test pins the three retired fields so they cannot return under
their old names.
This is the third instance of one bug class. v6 removed
pulse_intelligence_patrol_autofixes_30d, hardcoded to zero with no increment
site, and then introduced two fields that were constant in the other direction.
Three occurrences is a guard, not a habit.
Verified end to end on a running unlicensed install: the payload that reported
audit_logging_persistent true under v6 now reports audit_reads_30d 0, and
seeding two in-window reads, one outside the window, and one with an invalid
activity class yields 2.
Six of the eight Pro-exclusive features had no telemetry field at all, so
there was no way to answer whether RBAC, audit logging, scheduled reporting,
agent profiles, alert-triggered AI, or Kubernetes AI were being used by the
installs paying for them. Schema v6 adds nine content-free adoption signals:
alert_ai_enabled AIConfig.IsAlertTriggeredAnalysisEnabled()
rbac_custom_roles non-built-in roles, per org
rbac_user_assignments user-to-role assignments, per org
audit_logging_persistent a persistent audit store is active, not console
audit_events_30d audit events retained inside the window
report_schedules configured scheduled reports
report_schedules_enabled scheduled reports switched on
report_schedules_run_30d schedules whose last run falls inside the window
agent_profiles configured agent profiles
Counts only. Role names, permissions, usernames, schedule names, delivery
recipients, report scope, profile names, and every audit event field stay on
the install. kubernetes_ai needs no field of its own: it is derivable at read
time from alert_ai_enabled combined with the existing kubernetes_clusters
count, and a dedicated field would be redundant.
Config-sourced signals are read through applyLicensedFeatureConfigSnapshot;
RBAC and audit live behind the router and are read through
Router.ApplyLicensedFeatureTelemetrySnapshot. The RBAC read goes through a new
TenantRBACProvider.PeekManager so a background telemetry read can never
provision an RBAC store for an org that has never used RBAC.
Also removes pulse_intelligence_patrol_autofixes_30d and the AutoFixCount
field behind it. patrol_run.go hardcoded AutoFixCount to 0 and no increment
site existed anywhere in the tree, so the counter was zero in all 233,364
retained production pings. That was a wiring bug, not evidence that nobody
uses Patrol fixes; governed fixes are delivered through the approved-action
pipeline, which is already instrumented. The field was plumbed through run
records, history persistence, the Assistant handoff, and telemetry while being
structurally incapable of holding a non-zero value.
Verified end to end against a running install rather than only in unit tests,
which is precisely the check the autofix counter never had: seeding three
report schedules (two enabled, one last run inside the window) and two agent
profiles produced report_schedules 3, report_schedules_enabled 2,
report_schedules_run_30d 1, agent_profiles 2 in the Settings telemetry
preview, and signing in moved audit_events_30d to 1.
The private receiver landed first in pulse-pro 78ff7dd so the new fields are
accepted on arrival.
Three defects on 2026-08-05 shared one root. A settings capability and the
route it describes each answered "is this caller an admin" from their own copy
of the rule, and the copies had drifted, so the frontend gated tabs on
promises the routes did not keep. 28fd2d1c1 had four capabilities reporting
true against a 403. 7d066f1d5 had three guards refusing callers the settings
routes admitted. 563a3aa06 had the platform admin route disagreeing with the
billingAdmin capability it publishes.
Obligation 35 states the rule the code now follows. A session admin decision
is sessionUserCarriesAdminPrivileges reached through ensureAdminSession or the
status snapshot rather than re-derived, an org-scoped caller is never an
instance administrator, and authorizer output alone cannot establish a
capability while DefaultAuthorizer allows everything.
It also records why a source read is not sufficient evidence here. The gate is
frequently a wrapper two calls above the handler, which is how a capability
looks ungated when it is not, and cost a retracted claim before it cost
anything else.
Appended rather than renumbered so existing pins stay valid.
TrueNAS SCALE catalog apps ship one-shot init containers from ixSystems'
own base images (permissions, postgres_upgrade, pgvecto_upgrade). They run
to completion and stay exited for the life of the app, so essentially every
installed app produced a standing CRITICAL that could never clear.
TrueNAS already separates a completed workload from a failed one before
Pulse sees it. Its app state machine reports a container that exits with a
normal exit code as EXITED and one that exits abnormally as CRASHED, then
rolls any CRASHED container up into an app-level CRASHED. Pulse was
treating EXITED as a failure, which inverted the platform's own semantics
and fired precisely on the apps TrueNAS reported RUNNING.
EXITED now raises nothing. CRASHED still raises a per-container incident so
the app-level CRASHED names the failing service, which is the case that
was actually worth alerting on.
The same false assumption sat in the app's rendered container state, which
read app.Containers[0] and so let whichever init container app.query
happened to return first decide how a healthy app displayed. That now
collapses the workloads using the precedence TrueNAS itself uses.
Refs #1677
The mock:on and mock:off scripts used GNU 'sed -i' syntax, which fails on
the default macOS BSD sed. The '||' fallback then appended a fresh
PULSE_MOCK_MODE line on every invocation, so a local .env accumulated three
contradictory entries.
They also targeted the repo-root .env, which hot-dev.sh does not consult
when choosing the data directory. hot-dev.sh reads the canonical flag from
tmp/dev-config/.env, written authoritatively by toggle-mock.sh. The npm
wrappers were therefore a no-op that corrupted .env as a side effect, while
leaving the operator believing mock mode had switched.
Point them at toggle-mock.sh, which handles BSD sed, writes the canonical
file, syncs the runtime env and restarts the managed runtime. Add
mock:status and mock:edit; the hot-dev startup banner already advertised
'npm run mock:edit', which did not exist.
Extend the deployment-installability contract to cover the mock wrappers
alongside the existing repo-root dev entry rules, and add a regression test
asserting the wrappers delegate rather than rewriting the flag inline, and
that every mock command named in the hot-dev banner exists.
The second thing a user read in the documentation portal was the v6
Execution Canonical Source section, telling them to consult
release-control internal files, run status_audit.py, registry_audit.py and
contract_audit.py, and observe pre-commit partial-staging rules for
.husky/pre-commit and canonical-governance.yml. That is agent and
maintainer execution guidance, and it means nothing to somebody who has
just installed Pulse. The docs are shipped and served raw under /docs, so
it reached self-hosted operators as written.
Move it into SOURCE_OF_TRUTH.md as an Execution Entry Point subsection,
which is where the section already pointed and which is excluded from the
shipped docs set. Nothing is lost: the three facts that lived only in the
index, contract_audit.py, the hook-sensitive partial-staging list, and the
substantive-contract-section rule, are all carried across, and every other
reference already existed under Canonical Control Files.
The operator index now runs from its welcome line straight into Getting
Started.
The evaluation mode added earlier today was hollow. An unlicensed control
plane starts, but release-build client runtimes only trust entitlement
leases chained to a Pulse-signed licence, so its client workspaces ran
without the capabilities the provider was evaluating. Standing the stack
up proved the portal and the isolation boundary and nothing else.
setup.sh now requests a capped evaluation licence from the licence server
when no licence path is set, sending only the public half of the key it
generated locally. It degrades rather than blocks: a missing key, an
unreachable server, or a licence-free response leaves the install
unlicensed with an explicit warning, an existing licence on disk is
reused, and PULSE_PROVIDER_MSP_SKIP_EVAL_LICENSE skips it for air-gapped
hosts. Guarded with an if-test rather than a trailing true inside the
command substitution, because the derive helper calls die and exit in a
subshell is not a status that can be caught, so setup.sh aborted under
set -e.
Second fix. The lease capability ceiling was selected by licence
presence, so an unlicensed provider control plane fell through to the
Pulse-hosted branch and minted leases claiming relay, mobile and push,
which a provider deployment cannot serve and which previously caused
repeating relay registration failures in client runtimes. The ceiling now
follows hosting via SetProviderHosted. providerChained keeps its narrower
meaning of having a licence available to embed.
Third. Corrects a sentence I wrote into cloud-paid.md this afternoon
claiming msp_eval carries the same MSP capabilities. A plan version
selects the workspace cap; it does not entitle a workspace.
The regression test was negative-tested by reverting the ceiling selector
and confirming it catches relay. All three setup.sh degradation paths
were exercised directly. Licence server side is pulse-pro 7f6a319 and is
not live until the next deploy-license-server run.
Two mandatory round-trips stood between an interested MSP and their first
screen, and neither was technical.
setup.sh required four image digests shipped as literal <pin>
placeholders, so the only way to get them was to ask. All four images are
publicly readable, so there was never anything to hand out. setup.sh now
resolves each blank pin to an immutable digest from its published tag via
buildx imagetools and writes it back to .env; hand-set values are left
alone.
setup.sh then died outright without a licence file, so nobody could start
the stack, create a workspace, or see the portal until a human minted a
licence for them. The control plane already ran unlicensed via
ProviderMSPPlanSourceEnvFallback; only the installer refused. A licence
path that is set but missing is still a hard failure, since that is a
misconfiguration rather than a choice.
Unlicensed now means evaluation rather than the cheapest paid tier. The
env fallback defaulted to msp_starter, handing every unlicensed
deployment the full 5-client Starter allowance and leaving no boundary
between evaluating and buying. Adds msp_eval at 2 workspaces: same
capabilities, smaller cap, not purchasable, not on the public ladder.
An isolation guarantee is the one claim a provider cannot evaluate from a
screenshot, and both MSP leads this year went quiet at exactly this step.
Contracts: cloud-paid records the unlicensed plan rule and the
strictly-below-paid invariant; deployment-installability records
credential-free, correspondence-free installability.
Verification: TestMSPEvalCapStaysBelowCheapestPaidTier,
TestCanonicalizePlanVersion_MSPEval,
TestProviderMSPSetupScriptSupportsUnlicensedEvaluation. The last was
negative-tested by reintroducing a <pin> placeholder and confirming it
fails. ensure_image_pins exercised against the live registries.
licensing, cloudcp, control-plane and installtests all green.
rcourtman