Stream ungrouped retained reads in the existing time-major index order,
avoiding temporary metric sorts without changing tier reconciliation or
chart samples. Extend query-plan and filtered-result coverage.
Add an explicit strict mode for the existing 90 ms API budget and an
optional fresh-worker CI proof with fixed executable identities and raw
capacity evidence. Keep ordinary hosted budgets and required gates intact.
Contract-Neutral: Chart payloads, retained-tier reconciliation, sampling, index layout and source/release gates are unchanged; the implementation removes an unnecessary sort and adds controlled verification of existing latency budgets.
Keep monitoring readings, trusted connections and recovery guidance
reliable when observations are stale, guest responses are incomplete or
agent recovery is interrupted.
Qualified source: `b6e56c5baa2ad995264faa80877543300bc10273`. All nine
[required source
checks](https://github.com/rcourtman/Pulse/pull/2888/checks), the
[native
matrix](https://github.com/rcourtman/Pulse/actions/runs/39148879516) and
[Core E2E](https://github.com/rcourtman/Pulse/actions/runs/39148879470)
passed. Intel accounted for all 545 installer roots, with 532 passes, 13
explicit platform skips and no failures. Its remaining installer phase
passed in 18m48s, followed by native agent build and execution. The
matrix covers Linux x64/ARM64, macOS Intel/ARM64 and Windows x64
execution plus the FreeBSD cross-build contract.
- Preserve reading provenance, measured zero, stored History and VM/LXC
memory interpretation. Selected-agent staleness cannot renew a retained
observation or invalidate independent provider readings.
- Reject incomplete or foreign guest responses and retain saved
certificate trust, scoped access, private installer input, native NAS
teardown and bounded recovery guidance.
- Keep initial Proxmox login within its caller and total client timeout
while accepted-session renewal remains shared. Preserve the PBS owner's
context outcome when cancellation races a response, prevent a cancelled
fallback and reject late session publication.
- Assert troubleshooting workflows and destinations semantically. Run
every discovered Python safety control and native Go root independently
with existing fixture, operation, root and job deadlines. Script-smoke
and native verification share one control suite, including real
compiled-Go failure, exit-zero, timeout and exact-selector controls.
- Integrate current main while retaining selected-agent freshness, guest
memory interpretation, native GNU prerequisites, API concurrency
controls and both parents' subsystem contracts.
Worker proof passed 113 focused frontend cases, API and guest-transport
race checks, all 59 Python fixture controls, production frontend build
and full frontend lint/types. The exhaustive worker installer inventory
accounted for 551 roots: 535 passed and 16 explicitly skipped for
platform reasons. The eight consolidated runner controls passed through
the script-smoke harness and native Go wrapper. Authentication proof
passed all 340 Proxmox tests, 108 PBS tests and the affected monitoring
cancellation controls under the race detector. The new PBS test first
failed all ten JSON/form cases on unchanged runtime, then passed with
the correction. Status, ownership and normal commit checks passed.
Browser proof uses production components and CSS with synthetic
observations and endpoints. Drawer History passed at 1280×900, 1024×768
and 390×844. Private installation passed at 1280×900 and 390×900 in both
themes. It covers stale/current/unknown readings, provider caveats,
stored History and keyboard inspection, wrapped signing keys, token
copy/dismissal and Last seen advancing without another lookup. Final
screenshots were inspected, the source-bound receipt covers the combined
runtime files, and owned browsers and servers were closed.
Retain the [previous Frontend
assertions](https://github.com/rcourtman/Pulse/actions/runs/38660106123/job/116179237804),
[Intel
timeouts](https://github.com/rcourtman/Pulse/actions/runs/39138980077/job/117709536519),
[script-smoke integration
failure](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117728549560)
and [PBS cancellation
race](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117729442632)
with their diagnosed causes and corrected proof. The final matrix above
qualifies the combined corrections.
The [earlier advisory benchmark
comparison](https://github.com/rcourtman/Pulse/actions/runs/39138980114/job/117709718583)
reported three legacy history-filter regressions. A worker repeat
reproduced one. Longer controls showed wide timing variation even
between byte-identical binaries. The [45a82ea03c
comparison](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117728549510)
passed against a newer baseline. The [final
comparison](https://github.com/rcourtman/Pulse/actions/runs/39148879492/job/117738375776)
completed execution but failed on test-only empty/legacy filtering,
2.945 to 3.263ns, +10.82% (p<.001, ten samples). Production empty
filtering improved 16.70%, interleaved filtering rose 2.96%, and history
allocation counts were unchanged. The filter and benchmark sources are
byte-identical across the final comparison. This does not establish a
cause or performance closure. Artifact 12145720151 retains full paired
samples, comparison and metadata. The overall Build and Test workflow is
red from this advisory job while all nine protected required checks
pass. The existing L1 reliability follow-up owns repeatability and
attribution. No benchmark threshold, protected check or release gate
changed.
Main's benchmark/SLO test and documentation changes in PR3053 compose
without conflict. The candidate benchmark artifact records synthetic
merge c21333c676ac004935c9683027490203d8f869ea with tree
fa059ae37b, combining this head and main
52b40fd611. A bounded worker composition
proof also executed all three chart benchmark cases under the race
detector with source hashes unchanged. Its SLO test explicitly skipped
under the race detector, and one-iteration execution supplies no
performance comparison.
This proposal qualifies source delivery. Release inclusion, installed
upgrade/recovery, guest thaw, fresh writes on every backed-up filesystem
and workload liveness remain separate outcomes. Reliability keeps its
existing priority. Recovery/change feature expansion stays deferred.
Related to #1099, #2439, #2619, #2822, #2897, #2757, #2511, #2513,
#1788, #2755 and #2140.
A cancelled login can consume an empty response before the transport reports
cancellation. Preserve that context outcome for the owner and joined readers,
reject late session publication and prevent a cancelled refusal from retrying
another encoding. Keep live response validation and later recovery intact.
Script smoke still asserted the removed grouped runner while native
verification used a separate suite. Share the canonical execution controls
so both entrypoints reject lost checks, false passes and changed deadlines.
The chart benchmark warmed its response cache and then timed a fresh
payload-retaining recorder. That hid store-backed work behind test buffer
allocation and made the regression signal ambiguous.
Measure cached handler work, fresh-service computation and real loopback
HTTP transfer separately. Wait for store startup maintenance, preserve the
original failed comparison and record the paired diagnostic limits. Make
the adjacent SLO exercise uncached chart generation at its existing budget.
Refs #3052
Preserve the landed L6 source, response-verdict controls and ownership evidence
alongside the maintenance source repairs. Final protected checks and native
qualification must cover this combined source before landing.
Compile and discover one exact test binary per native workflow phase, then
execute every discovered root check independently. This removes inventory
growth from another check's ten-minute Go alarm while preserving fixture,
installer and overall CI job deadlines.
Retain exhaustive phase coverage and reject missing completion, failed checks,
unexpected exit zero and timeouts using real compiled Go execution controls.
Register the runner with deployment installability and retain the failed Intel
and advisory benchmark evidence in the existing reliability record.
Initial password authentication has one owner and must end with its
constructor. Keep its complete JSON/form exchange within the caller and
client timeout. Retained-session renewal still serves concurrent readers
independently of any one caller's cancellation.
Retain generation-based coalescing and the shared parser, exercise both
initial encodings under the client timeout, and document the ownership
boundary alongside existing startup and cluster cancellation controls.
Retain the failed CI and reproduced baseline in the existing reliability
record while final-source qualification remains open.
Preserve selected memory freshness and the VM/LXC interpretation.
Retain both native tool contracts and current API concurrency controls.
Qualify this source separately from release and installed recovery.
Contract-Neutral: Integrate already-governed parents without a new contract change. Incoming verification files already match main. Fresh integration tests and source-bound browser proof cover the combined memory and installer surfaces.
Connection closes and invalid local jobs entered the unknown failure bucket,
and disconnects during SMTP authentication could stop transient retries.
Use structural network evidence and declared configuration verdicts while
preserving unconfirmed SMTP acceptance as unknown.
Propagate interrupted HTTP Apprise responses through the shared bounded
response reader so they cannot create successful queue receipts. Share ntfy
template rendering and withhold private template and payload diagnostics.
Describe retained terminal failures as retries stopped, since a permanent
verdict can stop the first attempt before the budget is exhausted.
Literal LOGIN prompt matching rejected valid credentials and substring AUTH
selection could choose an unadvertised mechanism. Use exchange order and
complete mechanism names, preserve server reply classes, and classify
unsupported mechanism lists as configuration failures.
Share the implicit TLS decision between connection tests and delivery so a
port-465 test does not attempt STARTTLS inside an established TLS session.
Add real SMTP controls and persisted queue verdict verification.
Replace stale unshipped-fix claims with stable ancestry and reporter
results. Link remaining delivery and recovery outcomes to the existing
manual reliability target without changing priority or release gates.
Serial pinned-address dialing could spend the request deadline on a stalled
first address while a validated fallback was healthy. Share one bounded
two-worker dialer with restricted HTTP clients and retain cancellation in
webhook DNS admission and dialing.
Add real stalled-TCP and DNS cancellation controls, persistent queue verdict
readback, shared transport ownership and explicit proof routing.
Link the ordered audit into the planned product target and existing L15
and L19 residuals so future work has an explicit pickup path. Preserve
the active reliability target, lane states and release qualification.
Supported SMTP providers could pass validation with their default
username while delivery skipped authentication and failed with SMTP 530.
Preparing overlapping sends could also overwrite another delivery's
sender and recipients. Resolve provider credentials when creating the
SMTP manager and keep each delivery's configuration independent while
preserving the shared rate limit. SMTP attempt logs also report the
resolved authentication state for provider defaults.
Real SMTP regression controls cover all four provider defaults, accepted
envelopes, concurrent sends, shared rate limits and persisted queue
outcomes after restart. Invalid credentials still produce an
authentication dead letter. The notification contract and investigation
record retain the broader reliability gap because aggregate telemetry
cannot attribute fleet failures to these two defects.
Validation on pulse-dev with Go 1.26.9:
- Full notification package passed.
- Focused SMTP and queue controls passed normally and under the race
detector.
- Release-control status, registry and contract audits passed.
Provider-default usernames could leave hasAuth=false in SMTP diagnostics even
though the delivery manager required authentication. Report the resolved
manager state at the SMTP attempt and remove the premature admission-level
credential inference.
Refresh synthetic SMTP, persistent queue and race evidence for the final source.
Contract-Neutral: Diagnostic-only follow-up: report resolved SMTP hasAuth state. Existing real SMTP, persisted queue and race controls were rerun and retained.
Use current source, browser evidence and existing demand to sequence the
next product work after release reliability. Preserve the active target
and the existing lane completion states.
Refs #1812
Supported provider defaults could pass validation without enabling SMTP
authentication, and preparing concurrent sends could overwrite another
delivery's sender and recipients. Resolve credentials when the SMTP manager
is created and keep each delivery's configuration independent while sharing
the existing rate-limit state.
Verify real SMTP acceptance, invalid-credential dead letters, persisted
queue outcomes after restart, concurrent envelopes and shared rate limits.
Configured report schedules and recent attempts do not show successful
delivery or repeat use. Add durable aggregate reporting counters to the
existing opt-out-controlled heartbeat, covering resources, VM inventory,
backup health and Patrol digests.
Count successful generation, saved files, SMTP acceptance, definite
delivery failures and uncertain acceptance separately. Track distinct
successful saved-schedule periods and active days, including manual
downloads. Deduplicate retries and same-period reruns for repeat-period
counts. The adoption report distinguishes unmeasured senders and partial
observation from measured zero use. Generation across multiple days is
reported separately from successful repeat periods because it can
include retries.
No report contents, recipients, raw schedule or tenant identifiers,
scopes, errors or date ranges enter these new telemetry fields. Mock
activity is suppressed. Pricing, billing, customer entitlements and the
existing outbound opt-out remain unchanged. Update the schema-v20
disclosure and privacy document.
The shared telemetry observer has an exact-path adapter declaration.
Boundary regression fixtures reject sibling reporting implementation and
private licensing imports.
Validation: reporting persistence, API delivery/retry/restart and
successful-write tests, server snapshot tests, the full telemetry
package, 45 adoption-report tests, frontend type-check and build, and
governance audits passed. The three telemetry browser tests and a
separate pixel interaction check passed against the final source at
1280, 390 and 768 pixels, including saved opt-out states and the privacy
table. Source-bound browser receipt included.
Companion receiver: rcourtman/pulse-pro#472. Receiver deployment and a
client release are separate activation steps. This change alone does not
start production collection.
A scheduled retry across UTC days can generate another artifact without
completing a new successful reporting period. Keep multi-day generation as
an activity observation rather than inferred repeat use, while retaining
successful distinct-period counts as their own signal.
Exercise both cross-day generation without success and multiple successful
periods on one day so the adoption summary cannot conflate these metrics.
The usage observer records aggregate outcomes through the existing settings
store without implementing a paid renderer or entitlement decision. Declare
its exact shared-adapter path so the repository audit can distinguish it
from new private reporting implementation.
Exercise the real audit against sibling paths and private licensing imports
to retain those restrictions for the observer.
Record successful response writes, scheduled generation, file saves and SMTP
outcomes in a durable content-free tally. Distinct active days include manual
exports, while saved-schedule periods deduplicate retries and monthly reuse.
Publish bounded scalars through the existing opt-out-controlled heartbeat and
report missing coverage explicitly. Preserve pricing and customer entitlements.
Release candidate cut from main at 0d5b877425. It carries every change merged since stable v6.5.0.
Contract-Neutral: Release-only v6.6.0-rc.4 version, release notes and distribution metadata; no public install or runtime contract changes.
Change-source: pulse-maintainer
Client backup reports can now be scheduled weekly or monthly as PDF or
CSV, scoped to current VMs and containers by resource or client tag.
Reports show recorded outcomes, daily failure rates, last retained
success, failure details and missing evidence. Weekly reports cover
seven completed local calendar days. Monthly reports cover the previous
calendar month.
Backup executions and observation coverage are retained in each
organization's recovery store. Shrinking provider task inventories and
restarts preserve recorded failures. Unavailable history, permission
limits, collection gaps and incomplete multi-guest attribution remain
explicit. A successful task does not certify that a backup is retained
or restorable. This reports Proxmox VE backup jobs, including jobs
writing to PBS.
The existing scheduler, branding, delivery checks, bounded retries and
deliberate uncertain-delivery recovery are reused. The public adapter
supplies scoped evidence to a private PDF/CSV renderer. Pricing, feature
entitlements and retention remain unchanged.
Validation:
- Frontend: 136 focused tests, full type check, lint, production build
and bundle budgets.
- Race checks cover recovery persistence, collector quality, reporting,
organization isolation and runtime registration. Private rendering and
runtime build checks cover the composed Pro implementation.
- Live runtime checks on `/settings/support/reporting` and
`/alerts/notifications` at 1440x1000, 820x844 and 390x844 in light and
dark themes, with actual pixel review. The 61-workload selection remains
bounded and keyboard usable. All five generated PDF pages were
inspected.
- Real client-scoped CSV and MIME attachment contents, retained failure
after an empty provider inventory and restart, retry after restart,
permanent rejection, uncertain acknowledgement and deliberate recovery.
An unmatched client scope produces missing evidence with blank counts.
Retain backup execution outcomes and collection coverage in each
organization's recovery store so provider inventory changes and restarts
cannot erase report evidence. Keep missing and incomplete history distinct
from an observed period with no attempts.
Reuse existing report scopes, calendar schedules, branding and delivery
recovery for client PDF and CSV reports. Register the paid renderer through
the existing private runtime boundary. Pricing and grants are unchanged.
The organization-owner fix judged browser sessions only. API tokens on
instance routes were still checked for scope alone, and the tenant
middleware resolves an org-bound token to its own organization without
the caller naming one. A token bound to a tenant organization and
holding settings:write therefore reached system settings, SSO
providers, token management, relay, log level, updates and the other
instance surfaces.
ensureAdminSession, behind RequireAdmin, and ensureSettingsScope now
refuse a token unless it acts for the instance: not bound to any
organization other than the default one, with OrgIDs taking precedence
over OrgID. The route's scope still applies. The org variants
(RequireOrgAdmin, ensureOrgAdminSession, ensureOrgSettingsWriteScope)
keep judging tokens by scope alone. tokenHasInstanceWideAuthority is
now the same binding rule plus full-access scope.
The test notification broadcasts only to the request organization, so
it moves to RequireOrgAdmin. Its tenant delivery test already relied on
an org-bound token sending it, and that organization's managers may now
send it from a session too.
No path an organization manager can reach mints a settings scope: relay
mobile, install, deploy and action-runner tokens carry relay, agent and
report scopes only, and token management is instance administration.
The API reference and the api-contracts, security-privacy,
agent-lifecycle and storage-recovery contracts record the token rule.
VM inventory was available only as an on-demand download. It can now be
scheduled weekly or monthly as CSV, emailed or saved on the Pulse
server. Each attempt exports all current VMs in the organization,
including newly discovered VMs, through the existing exporter.
Historical performance-report limits and windows remain separate.
The schedule uses existing delivery configuration checks, persisted
success/failure/uncertain states, bounded retries and deliberate
recovery controls. Pricing, feature gates, retention entitlements and
existing customer grants are unchanged.
Live checks also reproduced two unattended inventory defects: reporting
retained a stopped monitor after configuration reload, and canonical
snapshot reads relied on dashboard reads to publish polled VM inventory.
Reporting now receives reload replacements through an atomic manager
reference. Canonical snapshot reads reuse the existing publication
boundary, including its freshness fast path, provider/mode fences and
nonblocking warm reads.
Validation:
- 105 focused frontend tests; full type check, lint, production build
and bundle budgets.
- Race checks for reporting, inventory, reload, publication and
organization isolation; clean private runtime build.
- Current runtime browser checks on `/settings/support/reporting` and
`/alerts/notifications` at 1440×1000, 820×844 and 390×844, with actual
pixel review. The first inventory read exports 61 normally polled VMs;
real MIME CSV matches the download; a newly discovered 62nd VM enters
the next export.
- Local SMTP acceptance, 451 retry across a real restart, permanent
rejection, uncertain acknowledgement and deliberate recovery; disk
delivery, edit/reload, disable, details and delete.
- Staged status, registry, contract, completion and exact-content
browser guards pass. Regression proof is registered against the specific
runtime paths it covers.
## Why
Report generation read every report's metrics from the default
organization's store, whatever organization the request came from. The
engine was wired in `pkg/server/server.go` to the default monitor's
store, and `ReportEngine.queryMetrics` queried it with the caller's
resource ID. A tenant request for a resource outside its own inventory
therefore read the default organization's history, and even a tenant's
own resources were reported from the wrong store. Pulse Assistant's
`pulse_summarize` used the same engine, so it read the same store. Other
tenants' stores were not reachable.
#3033 closed reporting to organization owners until this was fixed.
Providers build reports inside client workspaces (4d47504331), so
reporting reopens to them here.
## What changes
- **Engine** (`pkg/reporting`): `MetricReportRequest` and
`MultiReportRequest` carry `OrgID`. With
`EngineConfig.MetricsStoreForOrg` set, the engine resolves that
organization's own store and refuses a request without an organization
(`ErrReportOrgRequired`). A multi-resource report applies its own
`OrgID` to every resource.
- **Server wiring** (`pkg/server/reporting_engine.go`): the default
engine resolves stores per request through
`MultiTenantMonitor.GetMonitor(orgID)`, so it follows monitor reloads
and never falls back to the default organization.
- **Handlers**: single and multi-resource reports must name a resource
in the request organization's inventory: a unified resource ID, or a
legacy node/VM/container ID. Anything else returns 404
`resource_not_found`, or 503 `inventory_unavailable` when the inventory
can't be read.
- **Schedules**: a run resolves explicit resources and tags against its
owning organization's inventory, which the per-org schedule store and
scheduler context already supply. Resources the organization no longer
holds are skipped and logged; a scope with none left fails the run.
- **Pulse Assistant**: `pulse_summarize` passes its executor's
organization.
- **Access**: every `/api/admin/reports/*` route (catalog, generate,
generate-multi, VM inventory export, schedules) uses
`ensureOrgAdminSession`, and `reportingRead` is offered to organization
managers again (it still requires `read:nodes`).
- **Contracts**: rule 35 in api-contracts, the superseding note in
storage-recovery, plus ai-runtime, security-privacy and agent-lifecycle.
Behaviour change: a report can no longer be generated for a resource
that has left the organization's inventory, even if its retained history
is still in the store.
## Proof
- `internal/api/reporting_org_scope_test.go`: org B gets 404 for org A's
resource, by legacy and unified ID, through GET, POST and
generate-multi, and receives only org B's values even though org A's
store holds points under the same ID.
- `TestReportingScheduledRunReadsOnlyItsOwningOrganizationsMetrics`: an
org B schedule naming org A's resources reports only org B's resource
from org B's store; a scope of only foreign resources fails.
- `pkg/reporting/engine_org_scope_test.go` and
`TestReportEngineReadsOnlyTheRequestOrganizationsMetrics`: per-org
resolution and refusal without an org, across Generate, GenerateMulti,
NarrativeFor, FleetNarrativeFor and MetricEvidenceFor.
- `pkg/server/reporting_engine_test.go`: the server's engine resolves
the request org's monitor store, and refuses an empty, unprovisioned or
unavailable org.
- `TestSummarizeToolReadsOnlyItsOrganizationsMetrics`;
`TestSecurityStatusOrgBoundSettingsFollowOrgManagementRule` now holds
`reportingRead` in the org-bound set.
- On pulse-dev: `internal/api` (`-timeout 50m`), `pkg/reporting`,
`pkg/server`, `internal/ai/tools` and `internal/repoctl` pass. The
reporting and schedule tests also pass under `-race`.
## Governance
The canonical completion guard maps `pkg/server/server.go` to
security-privacy's "privacy docs and telemetry" verification policy,
which accepted only telemetry tests as proof. The `server.go` change
here is the report engine's per-organization store wiring, a
tenant-isolation boundary. `registry.json` now gives
`pkg/server/server.go` its own security-privacy path policy,
`server-runtime-wiring`: it keeps every proof the telemetry policy
accepted and also accepts `pkg/server/reporting_engine_test.go`. The
other telemetry files keep their existing policy, so they can't cite the
wiring test. That test proves the server's engine resolves the request
organization's monitor store and refuses an empty, unprovisioned or
unavailable organization. No telemetry test was edited to satisfy the
guard. The commit passed the full pre-commit hook.
## Coordination note
`vm-inventory-schedules-20261017` currently holds workspace claims on
`internal/api/report_schedules.go`,
`internal/api/reporting_runtime_snapshot_test.go`, `api-contracts.md`,
`agent-lifecycle.md` and `storage-recovery.md`, with uncommitted work in
its own checkout. This PR's hunks in those files
(`resolveReportScheduleResources`, a new helper appended to the snapshot
test, rule 35 and the `reportingRead` mount line in api-contracts, one
paragraph each in agent-lifecycle and storage-recovery) don't overlap
that work. This commit was built in a separate checkout, not in that
agent's working tree.
Reuse the canonical inventory exporter and report delivery workflow so
operators can email or save a current organization-wide inventory weekly
or monthly. Keep delivery checks, bounded retries and recovery states
shared, with existing pricing and customer entitlements unchanged.
Refresh the shared inventory for unattended reads and rebind reporting
after monitor reload so schedules use the current infrastructure state.
The report engine read every report's metrics from the default
organization's store, whatever organization the request came from. A
tenant request for a resource ID outside its own inventory fell through
to that store with the caller's ID, so one organization could read the
default organization's history, and even a tenant's own resources were
reported from the wrong store. Reporting was closed to org owners until
this was fixed.
The engine now resolves the metrics store of each request's OrgID
through the multi-tenant monitor and refuses a request that names none,
so no report path can fall back to the default store. Report handlers
reject resources outside the requesting organization's inventory,
scheduled runs resolve their scope against their owning organization's
inventory and skip resources it no longer holds, and Pulse Assistant's
pulse_summarize reads its executor's organization.
With every report read scoped, the reporting routes move to
ensureOrgAdminSession and reportingRead is offered to org managers
again, so providers can build reports inside client workspaces.
The report engine wiring lives in pkg/server/server.go, which the
completion guard only let telemetry tests prove. server.go now has its
own security-privacy path policy that keeps those proofs and also
accepts pkg/server/reporting_engine_test.go; the other telemetry files
keep their existing policy unchanged.
`PULSE_METRICS_DB_PATH` sets `Config.MetricsDBPath`.
`MultiTenantMonitor.GetMonitor` builds each non-default tenant from
`baseConfig.DeepCopy()`, which kept that path, so `monitoring.New`
opened the same `metrics.db` for every organization. With the override
set, per-org charts, reports and Pulse Assistant summaries all read from
one shared store.
Changes:
- `GetMonitor` clears `MetricsDBPath` on non-default tenant configs
after setting their `DataPath`. Each org then opens `metrics.db` in its
own data directory, and `DeleteOrganization` removes it along with the
rest of the org's data.
- The default org still uses the override, which covers the
single-install tmpfs/RAM-disk case it exists for. Metrics for
non-default orgs stay on the data volume.
- New `multi_tenant_monitor_metrics_path_test.go` sets the override,
provisions two orgs, and checks that each opens its own `metrics.db` and
cannot read series written by the default org or the other org. It fails
on current `main`.
- `TestMetricsStoreRuntimeOverridesStayOnMonitorBoundary` now also
requires the clear to come between the tenant `DataPath` assignment and
`New`.
- The monitoring contract and `CONFIGURATION.md` (plus its public
mirror) now describe the default-org-only behaviour.
Verification: `go test ./internal/monitoring/ ./internal/config/` passes
on Linux. `canonical_completion_guard.py` passes for this commit.
The agent-lifecycle and storage-recovery contracts now state that
registration, unregistration, node change and discovery events stay
with the organization that owns them. The tenant delivery tests move
into the proof files the canonical completion guard names for those
paths, and a route-level test covers the test-notification route's
authorization, method, payload and tenant delivery.
PULSE_METRICS_DB_PATH sets Config.MetricsDBPath, and GetMonitor builds
each non-default tenant from a deep copy of the base config, which kept
that path. monitoring.New then opened the same metrics.db for every
organization, so per-org charts, reports and Assistant summaries read
from one shared store.
Clear MetricsDBPath on non-default tenant configs so each org opens
metrics.db under its own data directory. Org deletion already removes
that directory, so its metrics go with it. The override still moves the
default org's database, which is the single-install case it exists for.
The previous commit moved pulse_read's read-only forms into an
argument-aware classifier; the ai-runtime contract now states what
pulse_read admits and names the proving tests.
Scheduled reports could appear successful when email was unconfigured,
and a temporary delivery failure had no durable retry or clear recovery
path. Reports now checks email configuration, records
running/saved/accepted/failed/unconfirmed outcomes, and exposes
actionable errors and Run/Retry controls at desktop and mobile widths.
Temporary mail failures receive at most three attempts with one- and
five-minute delays. Retries survive restart and preserve resource-report
windows. Lost completed-message acknowledgements and interrupted email
runs remain unconfirmed until a deliberate retry, preventing automatic
duplicate sends. Edits, disablement and deletion cancel pending retries.
Every attempt rechecks the existing entitlement.
This implements the scheduled-report delivery slice of L6. Pricing,
feature gates, retention limits and existing customer entitlements are
unchanged. SMTP acceptance is distinguished from inbox receipt.
The schedule editor loads when creating or editing a schedule, with a
visible loading state and initial keyboard focus. The reporting screen
stays within its existing bundle budget without raising limits.
Validation:
- Full configuration and notifications Go suites, reporting API race
tests.
- 301 targeted frontend and architecture tests, TypeScript, full
frontend lint and production frontend/private-runtime builds.
- Existing bundle-size, total JS/CSS and preload checks passed.
ReportingPanel is 10.94 kB gzip against its 11.63 kB limit.
- Playwright on `/settings/support/reporting` and
`/alerts/notifications` at 1440, 820 and 390 pixels: configuration
failure, disabled draft, real CSV attachment, disk PDF, persisted retry
across a process restart, permanent rejection, uncertain
acknowledgement, deliberate retry and delete/disable recovery.
- Controlled-response browser matrix for running, loading/error,
polling, keyboard focus, details and responsive layouts. Final pixels
reviewed separately from delivery assertions.
- Release-control status, ownership, contract, completion and browser
guards.
Browser delivery uses synthetic infrastructure, a disposable signed Pro
grant and local SMTP only. This source change does not publish a release
or alter customer installations.
Keep reporting within its existing bundle budget by deferring edit-only form
code until a schedule is created or edited. Show loading beside the schedule
controls and focus the name field when the editor is ready. Reuse the same
reporting field primitive and retain behavioral recovery-action coverage.
ensureAdminSession, the session check behind RequireAdmin and the
settings scope helpers, admitted anyone who could manage the
organization named in the request. That was meant for the org-bound
settings pages, but it applied to every admin route. On a multi-tenant
install an organization owner could therefore rotate the instance
administrator's API token and receive its secret, widen any token's
scopes, add or replace SSO providers, change system settings, relay
and log level, apply or roll back updates, restart the service, and
read instance logs and diagnostics.
ensureAdminSession now admits only instance administrators: the
configured admin or an instance RBAC admin grant, whichever org the
request names. Routes whose handlers act only on the request
organization opt in explicitly through RequireOrgAdmin,
ensureOrgAdminSession or ensureOrgSettingsWriteScope, which also admit
that organization's owners and admins. Those routes cover agents,
nodes, connections, availability, TrueNAS and VMware, agent profiles,
install commands, deploy status, configuration transfer and the
org-bound relay mobile token. Anything that does not opt in fails
closed.
Reporting stays instance administration, and the security status
snapshot stops offering it to org managers, because report generation
still reads the instance's default metrics store. Notifications,
LAN discovery and the action-capability fallback also stay instance
administration. The two credential routes no longer need their own
check, so ensureInstanceAdminSession is folded into this rule.
Persist delivery attempts and outcomes so temporary mail failures survive
restart without losing their retry or reporting window. Stop automatic resend
when SMTP acceptance is uncertain, and show actionable delivery checks and
manual recovery in Reports. Keep current pricing, gates and customer grants.
The route inventory treated RequireOrgAdmin routes as unauthenticated
"bare" routes, and the agent diagnostics contract and the API
reference anchors still named RequireAdmin for routes that now opt
into organization authority. The security-privacy contract still said
org managers receive reportingRead through ensureAdminSession; it now
names the org-bound routes' check and records that instance
administration refuses organization owners. The agent-lifecycle
contract notes that agent routes declare that authority explicitly.
The license server broker pins one Pulse Pro release, and that release
can be a prerelease. The Pro update check already skips a prerelease pin
on the stable channel, and apply already refuses it. The runtime switch
check from #2949 skipped that rule, so:
- an in-place stable install saw a switch offer that always failed at
apply with "stable channel cannot install prerelease builds"
- a Docker stable install got digest-pinned commands that moved it onto
the preview Pro image
Changes:
- `stableChannelRefusesProPin` in `internal/updates/pro_update.go` is
now the single rule for the Pro update check, the runtime switch check
and apply.
- `CheckProRuntimeSwitch` returns `available: false` with the new reason
`prerelease_only` and the pinned `version`. This happens before the
in-place and Docker branches, so neither path offers the pin. Installs
on the rc channel are still offered it.
- Regression tests cover: the stable channel refusing a pin flagged as a
prerelease, a pin whose version is a prerelease without the flag, and a
same-version flagged pin; the configured default channel; the rc channel
still offered the pin; and apply refusing before any download with the
message the API maps to 409. A separate API test keeps that refusal a
409 with the message passed through, not the generic 500.
Follow-ups outside this PR:
- The Updates card (#2983, not yet on main) does not know
`prerelease_only` and shows nothing for it. Copy for it is in a PR
stacked on that branch.
Contracts: `api-contracts.md` adds `prerelease_only` to the closed
`reason` set for `GET /api/updates/pro-runtime`, and
`deployment-installability.md` records that the switch check shares the
stable-channel guard. The commit no longer carries a Contract-Neutral
trailer.
Replaces #3007, rebuilt on current main with those contract notes.
Since #2991 stopped guessing a PMG backup node from the connection name,
a failed `/config/cluster/status` read leaves no node to ask for
backups, and the poller still published that empty result.
`UpdatePMGBackups` replaces the instance's whole list, so one transient
cluster-status failure, or every node's backup read failing, wiped every
known PMG backup and the Backups view flapped empty until the next good
poll.
Only a node whose backup list was read is now authoritative for that
cycle. Nodes that were not read (every node after a failed cluster read,
a node whose backup read failed, or one a cancelled poll never reached)
keep their previous backups. This is the same per-scope carry the PVE
storage backup poller applies to failed storages
(`preserveFailedStorageBackups`). A node that answered with no backups,
or that a successful cluster read no longer lists, still clears its
entries. The failed read is still recorded as a poll failure. #2991's
rule against guessing node names is unchanged.
Tests in `monitor_pmg_test.go`: the three retention tests fail on main
and pass with this change. A complete enumeration that returns no
backups still clears the list.
The monitoring subsystem contract now records the carry (new section
"PMG backups keep last-known entries for unread nodes"), and the #2960
note no longer says a failed cluster read publishes an empty backup
list.
Replaces #3005, rebuilt on current main with the contract note.
ensureAdminSession, the session check behind RequireAdmin and the
settings scope helpers, admitted anyone who could manage the
organization named in the request. That was meant for the org-bound
settings pages, but it applied to every admin route. On a multi-tenant
install an organization owner could therefore rotate the instance
administrator's API token and receive its secret, widen any token's
scopes, add or replace SSO providers, change system settings, relay
and log level, apply or roll back updates, restart the service, and
read instance logs and diagnostics.
ensureAdminSession now admits only instance administrators: the
configured admin or an instance RBAC admin grant, whichever org the
request names. Routes whose handlers act only on the request
organization opt in explicitly through RequireOrgAdmin,
ensureOrgAdminSession or ensureOrgSettingsWriteScope, which also admit
that organization's owners and admins. Those routes cover agents,
nodes, connections, availability, TrueNAS and VMware, agent profiles,
install commands, deploy status, configuration transfer and the
org-bound relay mobile token. Anything that does not opt in fails
closed.
Reporting stays instance administration, and the security status
snapshot stops offering it to org managers, because report generation
still reads the instance's default metrics store. Notifications,
LAN discovery and the action-capability fallback also stay instance
administration. The two credential routes no longer need their own
check, so ensureInstanceAdminSession is folded into this rule.
The hub kept tenant-scoped delivery for state and alerts, but every
other event went through Broadcast/BroadcastMessage, which sent to all
connected clients. In a multi-tenant install a member of one
organization therefore received another organization's discovery
progress, discovered servers (addresses, ports, versions, hostnames),
node auto-registration and node-change events.
Discovery now carries the org of the monitor that owns it, and setup
and node handlers send through BroadcastMessageToTenant with the
request's org (the setup token's org for auto-registration). The
global BroadcastMessage accepts only instance-wide types (ping,
settingsUpdate, update:progress) and refuses everything else, so a new
caller cannot leak by default. Legacy clients without an org are
registered under the default tenant, so single-org installs receive
these events as before. The unused unscoped alert wrappers are removed.
The license server broker pins a single Pulse Pro release, which can be a
prerelease. The update check skips such a pin on the stable channel and
apply refuses it, but the runtime switch check never applied that rule.
A stable install was offered an in-place switch that always failed with
"stable channel cannot install prerelease builds", and Docker installs
were handed digest-pinned commands for the preview Pro image.
The check, the switch check and apply now share one helper,
stableChannelRefusesProPin, so a check can no longer offer what apply
refuses. The switch check reports the new prerelease_only reason with the
pinned version so the Updates card can explain the wait; rc-channel
installs are still offered the prerelease pin.
The API contract adds prerelease_only to the closed reason set for
GET /api/updates/pro-runtime, and the deployment contract records that
the switch check shares the stable-channel guard.
Proxmox status/current sets agent=1 whenever the VM config enables the
QEMU guest agent (qemu-server vm_status: "QEMU Guest Agent is enabled in
config"), whether or not the in-guest service is running. Since
eb2424037d every successful status read renewed the guest-agent
fallback evidence on that flag, so a VM whose agent never answered kept
ten minutes of fallback eligibility after status/current later failed,
and Pulse sent filesystem and metadata guest commands that could only
time out.
Drop the status-flag renewal. Evidence is still renewed by a usable
filesystem reading and by useful guest metadata, and the original
observation age, canonical carry and every safety fence from
eb2424037d are unchanged. The new contract subtest polls with the real
integer agent flag: an enabled-but-silent agent admits no fallback
commands once status fails, while an agent that answered keeps its
fallback.
Since PMG stopped guessing a backup node from the connection name, a
failed cluster-status read leaves no node to ask, and the poller still
published the empty result. UpdatePMGBackups replaces the instance's
list, so one transient cluster failure (or every node's backup read
failing) wiped all known PMG backups and the Backups view flapped empty
until the next good poll.
Only a node whose backup list was read is now authoritative for that
cycle. Nodes that were not read, including every node after a failed
cluster read, keep their previous backups, the same per-scope carry
the PVE storage poller applies to failed storages. A complete read that
returns no backups, or a cluster listing that drops a node, still
clears them. The failed read stays recorded as a poll failure.
Proxmox's `/cluster/resources` `mem` for a container is its cgroup usage
with page cache already subtracted (`PVE::CGroup::get_memory_stat`:
`memory.current - memory.stat file` on cgroup v2, `usage_in_bytes -
total_cache` on v1). #3011 makes the backend read it as known pressure
for containers. The frontend still listed `cluster-resources` as
cache-inclusive for every guest, so the container drawer said the
reading "may include cached memory the container can reuse" and appended
agent setup advice.
- `memoryObservation.ts` classifies `cluster-resources` by guest kind:
cache-excluded for a container, cache-inclusive for a QEMU guest or a
guest of unknown kind. The other Proxmox fallbacks are unchanged.
- The canonical resource drawer
(`resourceDetailDrawerMetricsHistoryModel.ts`) now passes the guest
kind, so it qualifies the same reading the same way. A retained VM
reading there also gets the VM wording the Workloads drawer already
uses.
- The frontend-primitives contract clause that said a container's
Proxmox reading always carries the caveat is corrected.
Tests: the new container cases in `memoryObservation.test.ts`,
`GuestDrawerOverview.memory-cache.test.tsx`, `GuestRow.test.tsx` and
`resourceDetailDrawerMetricsHistoryModel.branchcov0712.test.ts` fail
against main's source (6 failures) and pass with the fix. Existing
container-copy tests now use `status-mem`, which is still
cache-inclusive.
Browser pass: private mock backend built from main. Before the fix, LXC
`auth-service-01` (`cluster-resources`) showed `Current · Proxmox (may
include cache)` and an `About this reading` note. After the fix it shows
`Current · Proxmox` with no note. A QEMU drawer keeps its caveat.
Checked at 1440 and 390 px wide.
Refs #2762
rcourtman