Contract-Neutral: Restores keyboard focus after existing API token dialogs close; token authority, scopes, endpoints, and security contracts are unchanged.
The workloads and Proxmox nodes trend cells drew every percent series on a
fixed 0-100 window inside a 16px cell, so any series living in the bottom of
that range rendered on top of the axis rule and read as missing data. A guest's
share of host memory is single digits by construction: at 2% the path sat at
y=15.7 against a baseline at y=16, which is what "memory column trends not
working" looks like. Idle guests hit the same wall on CPU.
Percent series now scale to their own peak like the I/O series already do,
zero-floored, with a 5% floor ceiling so idle noise stays flat rather than
amplified, and still capped at 100. Bars mode remains the level view and every
cell keeps its current-value label, so nothing loses the absolute reading.
Advance the governed install, Docker, and Helm metadata together; add the RC9 release packet for the post-RC8 notification, lifecycle, and resource-read fixes; and record the existing mobile-candidate compatibility decision.
Update source-shape guards for the shared per-generation resource list and cached mock unified view. Demote the onboarding E2E spec after its rate-limit/shared-state retry flake on main so it continues running without reddening the advisory verdict.
Contract-Neutral: advisory CI guardrails and probation tier bookkeeping only; no runtime or public contract delta
ResourceHandlers.getStore opens a SQLite handle per org and caches it for the
process lifetime, and nothing ever closed them. CleanupTenant already released
patrol, AI, RBAC, license and monitor-adapter state for a deleted org but left
the resource store open, so an offboarded tenant kept its file descriptors and
its unified_resources.db-wal/-shm files alive, and its directory could not be
fully removed.
CloseTenantStore releases and evicts one org's store and is now called from
CleanupTenant alongside the other per-tenant teardown. CloseStores releases all
of them, exposed as Router.ShutdownResourceStores next to the existing
Shutdown*/Stop* helpers.
Found while investigating the flaky hosted-tenant test. It is not what made that
test flaky, which was a detached guest-metadata write fixed separately, but the
leak is real on its own: verified by observing that -wal and -shm sidecars, which
exist only while a connection is open, survived the test before this change and
do not after it.
persistGuestIdentity spawned a detached goroutine per changed guest to write
guest_metadata.json, with a comment noting it avoided blocking the monitor.
Nothing tracked those goroutines, so neither Monitor.Stop nor
MultiTenantMonitor.Stop could wait for them and a queued write could land after
shutdown. In hosted mode that means a write into a tenant directory that
offboarding is already removing, and a stray guest_metadata.json.tmp left
behind when the atomic write is interrupted.
The store now owns the goroutine. SetAsync tracks the write on a WaitGroup and
WaitForPendingWrites drains it under a bounded timeout matching
tenantMonitorShutdownTimeout, so a wedged store cannot hold up tenant teardown.
Monitor.Stop drains before closing the metrics store.
This is what made TestHostedTenantAgentInstallTokenCannotReportToOtherTenant
flaky: t.TempDir cleanup raced a queued write into orgs/client-b and failed
with "directory not empty". The test itself is unchanged, because it was never
a test bug. A goroutine dump at cleanup time showed the writers still live,
created by persistGuestIdentity, blocked on the store mutex.
Verified causally rather than by observation alone: the target test fails 0/4
with the drain removed and passes 8/8 with it, against 2/3 failures on the
unmodified baseline. The regression tests fail if SetAsync stops tracking its
goroutine.
Note for a future pass, deliberately not changed here: each changed guest still
triggers a full-file save, so one poll cycle over N changed guests does N
marshals and N atomic writes that serialize on the store mutex anyway. Fixing
that means coalescing at the call site and is a behavioural change beyond this
defect.
Fixes#1681Fixes#1682Fixes#1683
Contract-Neutral: Notification grouping initialization and alert-config propagation do not alter the broadly referenced agent-lifecycle or storage-recovery contracts; primary alerts, notifications, API, and monitoring contracts and regression proofs are updated.
Every list-shaped resources request deep-cloned the whole registry:
HandleListResources via ListForPresentation, the storage summary and
incidents handlers and the k8s namespaces handler via List/ListByType,
and the stats handler once more. With the frontend polling three pages
plus summaries every few seconds, the same unchanged world was cloned
dozens of times between snapshot updates.
Cache the raw and presentation lists on the existing per-generation
registry cache entry (same invalidation: entries rebuild when the seed
lastUpdate moves) and hand requests a flat top-level copy instead. Every
decorator in the request pipeline was audited to write only top-level
fields on request-owned elements: action availability, discovery
targets and readiness, metrics targets, canonical metadata refresh, and
contract types all assign freshly built values. The one nested writer —
the PMG list prune, which cleared relay domains through the shared
pointer — now clones the PMG struct before clearing, with a regression
test pinning both the clone-on-write and the cache's immunity to
request decoration. Read-only consumers (stats aggregation, storage
filtering, namespace counting) use the shared list with no copy at all.
Single-resource lookups (presentationResourceByReference and its
callers) still deep-clone per lookup; they are cold paths and stay
as-is.
Contract-Neutral: per-generation shared resource lists: response bytes pinned by contract tests, no payload delta
Schema v6 shipped audit_logging_persistent and audit_events_30d as Pro adoption
signals. Neither discriminated. pkg/server installs the SQLite audit logger on
every install for defense in depth and gates only the read/export endpoints, so
the boolean was true on all 8 installs that had taken rc.8 and 0 rows in the
retained table have ever had it false. The event count measured that background
write volume: three of those eight unlicensed community installs were pegged at
the receiver's 100000 clamp ceiling, with the rest between 4863 and 67509.
Schema v7 replaces both with audit_reads_30d, a count of requests that cleared
the license gate on an audit read or export surface. A read requires a human
action, so unlike store presence or write volume it cannot settle into a
constant. The recorder is wrapped INSIDE RequireLicenseFeature so unentitled
requests never count, and the persisted marker carries a timestamp and a coarse
activity class from a fixed allowlist. Query filters, actors, ranges, and every
audit row read stay on the install.
The retired columns are left in the live database. They hold real rc.8 rows and
migrations only add, so dropping them would be a pointless risk; nothing writes
them once the receiver struct loses the fields.
Adds the guard this class needed. LicensedFeatureAdoptionFields registers every
field that exists to measure licensed-feature adoption, and
TestLicensedFeatureAdoptionFieldsDiscriminate builds an unused install through
the real production snapshot paths, installs a real SQLite audit logger exactly
as pkg/server does, records a baseline audit event, and fails if any registered
field is non-zero. Pinning a console logger there would have made the guard pass
while the payload lied, so it deliberately does not. The guard was verified by
reintroducing the v6 sourcing and confirming it fails with the field named.
A companion test pins the three retired fields so they cannot return under
their old names.
This is the third instance of one bug class. v6 removed
pulse_intelligence_patrol_autofixes_30d, hardcoded to zero with no increment
site, and then introduced two fields that were constant in the other direction.
Three occurrences is a guard, not a habit.
Verified end to end on a running unlicensed install: the payload that reported
audit_logging_persistent true under v6 now reports audit_reads_30d 0, and
seeding two in-window reads, one outside the window, and one with an invalid
activity class yields 2.
In mock mode every unified read-state access built two throwaway
registries: mock.UnifiedResourceSnapshot constructed one to derive the
resource list, and the monitor's currentUnifiedStateView ingested that
list into another, deep-cloning all resources both ways. Chart requests,
broadcasts, alert evaluation, and API reads each repaid that full cost —
the dominant share of the demo's 76TB/9.5d allocation churn, since every
one of those reads runs against a world that only changes on the 2-second
mock tick.
Introduce fixtureDataVersion, a token that advances on every observable
mock-graph change (metric ticks and the structural changes that bump
fixtureRevision, which stays structural-only so seeded trend history
remains reusable). Memoize the package-level UnifiedResourceSnapshot and
the monitor's mock-branch state view against it, so consumers between
ticks share one immutable build. Sharing mirrors the semantics the
persistent-store ReadState path has always had in real mode: all
consumers were audited — they ingest (which clones), copy before
top-level writes, or build fresh outputs. Real-mode paths are untouched.
Contract-Neutral: mock snapshot memoization: identical data served from cache, no contract delta
Contract-Neutral: Demotes one flaky E2E spec under the documented stable-to-probation tier policy; product behavior and public contracts are unchanged.
Six of the eight Pro-exclusive features had no telemetry field at all, so
there was no way to answer whether RBAC, audit logging, scheduled reporting,
agent profiles, alert-triggered AI, or Kubernetes AI were being used by the
installs paying for them. Schema v6 adds nine content-free adoption signals:
alert_ai_enabled AIConfig.IsAlertTriggeredAnalysisEnabled()
rbac_custom_roles non-built-in roles, per org
rbac_user_assignments user-to-role assignments, per org
audit_logging_persistent a persistent audit store is active, not console
audit_events_30d audit events retained inside the window
report_schedules configured scheduled reports
report_schedules_enabled scheduled reports switched on
report_schedules_run_30d schedules whose last run falls inside the window
agent_profiles configured agent profiles
Counts only. Role names, permissions, usernames, schedule names, delivery
recipients, report scope, profile names, and every audit event field stay on
the install. kubernetes_ai needs no field of its own: it is derivable at read
time from alert_ai_enabled combined with the existing kubernetes_clusters
count, and a dedicated field would be redundant.
Config-sourced signals are read through applyLicensedFeatureConfigSnapshot;
RBAC and audit live behind the router and are read through
Router.ApplyLicensedFeatureTelemetrySnapshot. The RBAC read goes through a new
TenantRBACProvider.PeekManager so a background telemetry read can never
provision an RBAC store for an org that has never used RBAC.
Also removes pulse_intelligence_patrol_autofixes_30d and the AutoFixCount
field behind it. patrol_run.go hardcoded AutoFixCount to 0 and no increment
site existed anywhere in the tree, so the counter was zero in all 233,364
retained production pings. That was a wiring bug, not evidence that nobody
uses Patrol fixes; governed fixes are delivered through the approved-action
pipeline, which is already instrumented. The field was plumbed through run
records, history persistence, the Assistant handoff, and telemetry while being
structurally incapable of holding a non-zero value.
Verified end to end against a running install rather than only in unit tests,
which is precisely the check the autofix counter never had: seeding three
report schedules (two enabled, one last run inside the window) and two agent
profiles produced report_schedules 3, report_schedules_enabled 2,
report_schedules_run_30d 1, agent_profiles 2 in the Settings telemetry
preview, and signing in moved audit_events_30d to 1.
The private receiver landed first in pulse-pro 78ff7dd so the new fields are
accepted on arrival.
CollapsibleSection capped its expanded state at max-h-[5000px] with
overflow-hidden, so any section taller than that was silently truncated. The
component wraps fourteen sections across the threshold tabs, so this was never
limited to the one screen it was reported against. On this instance the guests
section measures 10958px at 1440 wide, which the old cap cut roughly in half.
The cap existed because CSS cannot transition to height:auto, so the animation
needed a concrete bound. Raising it to a larger number only moves the cliff, so
the collapse now switches the grid row track between 0fr and 1fr and the track
carries the natural height with no ceiling.
The transition is scoped to opacity rather than transition-all, and that part
is load bearing. Under transition-all the browser holds grid-template-rows at
whatever pixel value it had when the class flipped and never resolves the new
track, which leaves a section stuck open or stuck shut permanently rather than
merely un-animated. The first version of this change did exactly that, and the
unit tests all passed because they assert class names rather than computed
layout. Only the browser run caught it.
Verified at 1440x1000 and 600x900: expanded reports the full row track with
inner scrollHeight equal to clientHeight, collapsed reports 0px with the
content clipped, and three toggle cycles alternate correctly at both widths.
The height tween is gone, which is honest, since it never worked past 5000px
and this engine does not interpolate grid-template-rows anyway. Opacity still
fades.
Refs #1680
Contract-Neutral: CSS-only collapse mechanism in one presentational component; no alert threshold behaviour, payload, or public contract changes, and the browser verification receipt is staged in the same commit
Mock mode seeded 90 days of synthetic history for every resource at
startup. Measured on the demo droplet with identical entity counts, that
put the process near 1GB of Go heap before the first tick; a 6h seed
runs at ~200MB. Every local dev instance and CI run in mock mode paid
that gigabyte silently, and on 1GB hosts it OOMed the box.
48h keeps day-scale charts fully populated. Long-range chart work opts
in with PULSE_MOCK_TRENDS_SEED_DURATION, which is how the public demo
already pins its own seed (6h).
Contract-Neutral: mock seed default resize: dev-only mock path, no contract delta
Three defects on 2026-08-05 shared one root. A settings capability and the
route it describes each answered "is this caller an admin" from their own copy
of the rule, and the copies had drifted, so the frontend gated tabs on
promises the routes did not keep. 28fd2d1c1 had four capabilities reporting
true against a 403. 7d066f1d5 had three guards refusing callers the settings
routes admitted. 563a3aa06 had the platform admin route disagreeing with the
billingAdmin capability it publishes.
Obligation 35 states the rule the code now follows. A session admin decision
is sessionUserCarriesAdminPrivileges reached through ensureAdminSession or the
status snapshot rather than re-derived, an org-scoped caller is never an
instance administrator, and authorizer output alone cannot establish a
capability while DefaultAuthorizer allows everything.
It also records why a source read is not sufficient evidence here. The gate is
frequently a wrapper two calls above the handler, which is how a capability
looks ungated when it is not, and cost a retracted claim before it cost
anything else.
Appended rather than renumbered so existing pins stay valid.
A live heap profile of the demo (9.5 days uptime) showed 76TB of
cumulative allocations, with inferMetricRole accounting for 12% of the
total: the classifier table and its keyword slices were rebuilt on every
call, and normalizeMetricRoleTokens constructed a fresh strings.Replacer
per token, paying the lazy trie build each time. Both run per resource
per 2-second mock tick, so the demo spent a measurable share of its
single vCPU feeding the garbage collector.
Both structures are static; make them package-level.
Contract-Neutral: mock allocation hoists: behavior-identical, no contract delta
canAccessPlatformAdminSurface publishes billingAdmin for any instance
administrator. RequirePlatformAdmin compared the session user against
cfg.AuthUser alone, so on an instance whose only administrators are SSO
principals the UI offered the surface and the route refused it. Same
capability against enforcement split as 28fd2d1c1, on the hosted routes.
The session branch now uses sessionUserCarriesAdminPrivileges, which is what
the capability already resolves to.
A straight swap would have been worse than the bug. That helper treats any SSO
principal as an administrator when no local admin is configured, and a hosted
control plane authenticates its tenants by SSO, so on a control plane with no
local admin every tenant would have become a platform admin. The session
branch is therefore also gated on the request not being org-scoped, matching
what ensureAdminSession and the security status snapshot already do. Removing
that gate lets an org-scoped tenant session reach the surface with a 200,
which the parity test pins.
The org-scope test itself was written inline in two places and is now one
helper, sessionIsOrgScoped, so the instance-versus-tenant boundary has a
single definition rather than a copy per caller.
RequireOrgOwnerOrPlatformAdmin is untouched. It has no session branch in its
platform-admin switch by design and requires org ownership instead.
Contract-Neutral: behavioral fix on existing routes, no request or response shape change; platform admin route aligned with the billingAdmin capability it already publishes
Pulse under a systemd MemoryMax, docker --memory, or Kubernetes limit ran
with a GC that was blind to the cap: GOGC=100 lets total heap float to
roughly twice the live set, so a capped process grows until the kernel
OOM-kills it instead of collecting harder near the boundary. The demo
droplet's kill history (three OOM kills in July under an 800M cap) is
this mechanism, and any containerized install with a memory limit is
exposed the same way.
At startup, when GOMEMLIMIT is not set by the operator, resolve the
process cgroup (v2 ancestor walk taking the smallest memory.max on the
path, v1 limit_in_bytes fallback) and set the runtime soft limit to 90%
of it, leaving headroom for stacks, mmap, and CGO. Best effort: no
detectable limit leaves GC defaults untouched.
Contract-Neutral: runtime GC limit alignment: no payload or contract delta
On the OIDC-only pattern there is no local admin, so SSO principals are the
only administrators the instance has. ensureAdminSession already knows that:
sessionUserCarriesAdminPrivileges admits the configured admin, anyone holding
an RBAC admin grant, and any SSO principal when no local admin is configured.
Three guards did not use it. They compared the session username against
cfg.AuthUser directly, which on those instances is empty, so they could admit
nobody at all. The same operator was admitted by the settings routes and
refused by discovery, by public URL capture, and by config export and import,
which is three answers to one question.
Verified against an unlicensed OIDC-only router before and after. Before,
sessionUserCarriesAdminPrivileges returned true for the SSO owner while
canCapturePublicURL and discovery isAdminRequest returned false and
/api/config/export returned 403. After, all four agree.
This does not widen anything. On an instance that does configure a local
admin, an unrelated SSO principal is still not an administrator and every one
of these guards still refuses them, which the parity test pins in both
directions.
RequirePlatformAdmin is deliberately left alone. Its stricter session rule is
documented as intentional for the hosted control plane rather than an
oversight, and loosening it is a different decision from this one.
Contract-Neutral: behavioral fix on existing routes, no request or response shape change; three guards switched to the canonical session-admin helper they should already have used
handleChangePassword refuses a non-admin proxy user outright. The session
branch of that rule was never written, so any authenticated session reached
the current-password comparison.
Knowing the current password is the real gate on the change, so this was not
an escalation for someone who did not already know it. What it did give an
authenticated non-admin was an oracle: submit a guess at the local admin
password and read the answer off the 401, under the login rate limit and
lockout rather than outside them. With the password in hand the same caller
could then change it and lock the administrator out.
ensureAdminSession is a no-op for requests carrying no session cookie, so the
Basic Auth path this handler was largely written around is unaffected, and it
admits SSO principals on instances with no local admin.
Found by sweeping every proxy-admin check for a missing session counterpart.
The other 43 call sites were either identity extraction rather than
authorization, already carried a session branch, or were compensated
downstream by ensureSettingsScope.
Contract-Neutral: behavioral hardening on an existing route, no request or response shape change; session callers now held to the same admin rule the proxy branch already applied
The read deadline was refreshed only by protocol pong frames, so a client
whose ping/pong control frames were delayed more than 6 seconds or eaten
by a middlebox (Cloudflare edge, AV proxies) was killed every 60 seconds
even while its 30-second JSON heartbeats were arriving. Demo logs show
the signature clearly: sessions dying at exactly 60s/114s/168s, i.e.
zero, one, or two pongs before execution, while the user watched the
badge cycle through Reconnecting on an otherwise healthy connection.
Refresh the read deadline on every successful read, widen it to 90s so a
background tab throttled to one heartbeat per minute still survives, and
ping every 30s so pong-only clients tolerate two lost round trips.
Contract-Neutral: websocket keepalive hardening: read-deadline/ping cadence only, no payload or contract delta
A settings capability is a promise the routes have to keep. Without an RBAC
licence the authorizer allows every action, so capabilities derived from it
alone reported true while the matching route stayed gated by
ensureSettingsScope and in turn ensureAdminSession.
canAccessPermissionSurface already refused to trust the authorizer for a
non-admin proxy caller. The session half of that rule was never written, so a
non-admin session on an unlicensed instance was told apiAccessRead,
apiAccessWrite, singleSignOnRead and singleSignOnWrite were all available. The
nav gates on exactly those flags, so the API Access and Single Sign-On tabs
rendered, their first request came back 403, and the user got an error toast
on a tab they were never able to use. Everything routed through
canAccessAdminSurface was already correct, which is why authenticationRead and
the audit surfaces behaved and these two did not.
The fallback uses snapshot.sessionIsAdmin, which derives from the same
sessionUserCarriesAdminPrivileges the routes enforce, so the capability now
matches the answer the route will give. That keeps the OIDC-only pattern
working, where SSO principals are the instance's only administrators.
Nothing was reachable that should not have been. This is a capability
reporting fix, not an access control one.
Refs #1672
Contract-Neutral: settingsCapabilities JSON shape is unchanged (same 14 fields, same types); this corrects a wrong value returned to non-admin sessions when no RBAC authorizer is registered, no public-contract delta
TrueNAS SCALE catalog apps ship one-shot init containers from ixSystems'
own base images (permissions, postgres_upgrade, pgvecto_upgrade). They run
to completion and stay exited for the life of the app, so essentially every
installed app produced a standing CRITICAL that could never clear.
TrueNAS already separates a completed workload from a failed one before
Pulse sees it. Its app state machine reports a container that exits with a
normal exit code as EXITED and one that exits abnormally as CRASHED, then
rolls any CRASHED container up into an app-level CRASHED. Pulse was
treating EXITED as a failure, which inverted the platform's own semantics
and fired precisely on the apps TrueNAS reported RUNNING.
EXITED now raises nothing. CRASHED still raises a per-container incident so
the app-level CRASHED names the failing service, which is the case that
was actually worth alerting on.
The same false assumption sat in the app's rendered container state, which
read app.Containers[0] and so let whichever init container app.query
happened to return first decide how a healthy app displayed. That now
collapses the workloads using the precedence TrueNAS itself uses.
Refs #1677
Contract-Neutral: Responsive API-token presentation and a fingerprint-scoped scanner false-positive exception; no security contract, permission, or API behavior changes.
rcourtman