Stream ungrouped retained reads in the existing time-major index order,
avoiding temporary metric sorts without changing tier reconciliation or
chart samples. Extend query-plan and filtered-result coverage.
Add an explicit strict mode for the existing 90 ms API budget and an
optional fresh-worker CI proof with fixed executable identities and raw
capacity evidence. Keep ordinary hosted budgets and required gates intact.
Contract-Neutral: Chart payloads, retained-tier reconciliation, sampling, index layout and source/release gates are unchanged; the implementation removes an unnecessary sort and adds controlled verification of existing latency budgets.
Keep monitoring readings, trusted connections and recovery guidance
reliable when observations are stale, guest responses are incomplete or
agent recovery is interrupted.
Qualified source: `b6e56c5baa2ad995264faa80877543300bc10273`. All nine
[required source
checks](https://github.com/rcourtman/Pulse/pull/2888/checks), the
[native
matrix](https://github.com/rcourtman/Pulse/actions/runs/39148879516) and
[Core E2E](https://github.com/rcourtman/Pulse/actions/runs/39148879470)
passed. Intel accounted for all 545 installer roots, with 532 passes, 13
explicit platform skips and no failures. Its remaining installer phase
passed in 18m48s, followed by native agent build and execution. The
matrix covers Linux x64/ARM64, macOS Intel/ARM64 and Windows x64
execution plus the FreeBSD cross-build contract.
- Preserve reading provenance, measured zero, stored History and VM/LXC
memory interpretation. Selected-agent staleness cannot renew a retained
observation or invalidate independent provider readings.
- Reject incomplete or foreign guest responses and retain saved
certificate trust, scoped access, private installer input, native NAS
teardown and bounded recovery guidance.
- Keep initial Proxmox login within its caller and total client timeout
while accepted-session renewal remains shared. Preserve the PBS owner's
context outcome when cancellation races a response, prevent a cancelled
fallback and reject late session publication.
- Assert troubleshooting workflows and destinations semantically. Run
every discovered Python safety control and native Go root independently
with existing fixture, operation, root and job deadlines. Script-smoke
and native verification share one control suite, including real
compiled-Go failure, exit-zero, timeout and exact-selector controls.
- Integrate current main while retaining selected-agent freshness, guest
memory interpretation, native GNU prerequisites, API concurrency
controls and both parents' subsystem contracts.
Worker proof passed 113 focused frontend cases, API and guest-transport
race checks, all 59 Python fixture controls, production frontend build
and full frontend lint/types. The exhaustive worker installer inventory
accounted for 551 roots: 535 passed and 16 explicitly skipped for
platform reasons. The eight consolidated runner controls passed through
the script-smoke harness and native Go wrapper. Authentication proof
passed all 340 Proxmox tests, 108 PBS tests and the affected monitoring
cancellation controls under the race detector. The new PBS test first
failed all ten JSON/form cases on unchanged runtime, then passed with
the correction. Status, ownership and normal commit checks passed.
Browser proof uses production components and CSS with synthetic
observations and endpoints. Drawer History passed at 1280×900, 1024×768
and 390×844. Private installation passed at 1280×900 and 390×900 in both
themes. It covers stale/current/unknown readings, provider caveats,
stored History and keyboard inspection, wrapped signing keys, token
copy/dismissal and Last seen advancing without another lookup. Final
screenshots were inspected, the source-bound receipt covers the combined
runtime files, and owned browsers and servers were closed.
Retain the [previous Frontend
assertions](https://github.com/rcourtman/Pulse/actions/runs/38660106123/job/116179237804),
[Intel
timeouts](https://github.com/rcourtman/Pulse/actions/runs/39138980077/job/117709536519),
[script-smoke integration
failure](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117728549560)
and [PBS cancellation
race](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117729442632)
with their diagnosed causes and corrected proof. The final matrix above
qualifies the combined corrections.
The [earlier advisory benchmark
comparison](https://github.com/rcourtman/Pulse/actions/runs/39138980114/job/117709718583)
reported three legacy history-filter regressions. A worker repeat
reproduced one. Longer controls showed wide timing variation even
between byte-identical binaries. The [45a82ea03c
comparison](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117728549510)
passed against a newer baseline. The [final
comparison](https://github.com/rcourtman/Pulse/actions/runs/39148879492/job/117738375776)
completed execution but failed on test-only empty/legacy filtering,
2.945 to 3.263ns, +10.82% (p<.001, ten samples). Production empty
filtering improved 16.70%, interleaved filtering rose 2.96%, and history
allocation counts were unchanged. The filter and benchmark sources are
byte-identical across the final comparison. This does not establish a
cause or performance closure. Artifact 12145720151 retains full paired
samples, comparison and metadata. The overall Build and Test workflow is
red from this advisory job while all nine protected required checks
pass. The existing L1 reliability follow-up owns repeatability and
attribution. No benchmark threshold, protected check or release gate
changed.
Main's benchmark/SLO test and documentation changes in PR3053 compose
without conflict. The candidate benchmark artifact records synthetic
merge c21333c676ac004935c9683027490203d8f869ea with tree
fa059ae37b, combining this head and main
52b40fd611. A bounded worker composition
proof also executed all three chart benchmark cases under the race
detector with source hashes unchanged. Its SLO test explicitly skipped
under the race detector, and one-iteration execution supplies no
performance comparison.
This proposal qualifies source delivery. Release inclusion, installed
upgrade/recovery, guest thaw, fresh writes on every backed-up filesystem
and workload liveness remain separate outcomes. Reliability keeps its
existing priority. Recovery/change feature expansion stays deferred.
Related to #1099, #2439, #2619, #2822, #2897, #2757, #2511, #2513,
#1788, #2755 and #2140.
A cancelled login can consume an empty response before the transport reports
cancellation. Preserve that context outcome for the owner and joined readers,
reject late session publication and prevent a cancelled refusal from retrying
another encoding. Keep live response validation and later recovery intact.
The workload chart benchmark warmed its response cache, then mostly
timed a fresh recorder allocating a roughly 1.6 MiB response buffer. It
did not reliably measure the store-backed computation it claimed to
protect.
Split it into cached handler work, fresh-service computation and fully
drained loopback HTTP transfer. The store fixture waits for startup
maintenance. The adjacent SLO now exercises uncached chart generation at
its existing budgets. Retain the original #3052 failure and document the
new baseline boundary. Runtime chart behavior is unchanged.
Validation: 20 alternating samples on both recorded runtime trees with
the same revised harness and Go 1.26.9 found no significant change in
any of the three cases. The race smoke passed all three cases with
source hashes verified before and after execution. The focused SLO
measured 216 ms p95 and skipped its 90 ms budget under the shared-worker
contention policy. Non-race release qualification retains the 550 ms
hosted budget. Source CI uses race tests and skips latency SLOs. The
investigation record retains the exact scope and evidence.
The original +12.72% remains unresolved historical evidence. This change
makes no product speedup claim.
Script smoke still asserted the removed grouped runner while native
verification used a separate suite. Share the canonical execution controls
so both entrypoints reject lost checks, false passes and changed deadlines.
The chart benchmark warmed its response cache and then timed a fresh
payload-retaining recorder. That hid store-backed work behind test buffer
allocation and made the regression signal ambiguous.
Measure cached handler work, fresh-service computation and real loopback
HTTP transfer separately. Wait for store startup maintenance, preserve the
original failed comparison and record the paired diagnostic limits. Make
the adjacent SLO exercise uncached chart generation at its existing budget.
Refs #3052
Preserve the landed L6 source, response-verdict controls and ownership evidence
alongside the maintenance source repairs. Final protected checks and native
qualification must cover this combined source before landing.
Compile and discover one exact test binary per native workflow phase, then
execute every discovered root check independently. This removes inventory
growth from another check's ten-minute Go alarm while preserving fixture,
installer and overall CI job deadlines.
Retain exhaustive phase coverage and reject missing completion, failed checks,
unexpected exit zero and timeouts using real compiled Go execution controls.
Register the runner with deployment installability and retain the failed Intel
and advisory benchmark evidence in the existing reliability record.
Initial password authentication has one owner and must end with its
constructor. Keep its complete JSON/form exchange within the caller and
client timeout. Retained-session renewal still serves concurrent readers
independently of any one caller's cancellation.
Retain generation-based coalescing and the shared parser, exercise both
initial encodings under the client timeout, and document the ownership
boundary alongside existing startup and cluster cancellation controls.
Retain the failed CI and reproduced baseline in the existing reliability
record while final-source qualification remains open.
The NAS suite exhausts its aggregate alarm after fifteen successful
controls on macOS Intel. Run each discovered NAS and updater control
under its existing bound, retaining the native phase's ten-minute alarm
and all installer operation deadlines.
Keep required controls mandatory and reject skipped, absent or failed
execution. Exercise discovery and false-pass rejection with real Python
controls, including a newly added method.
Plain connection closes and opaque network-operation errors could be
recorded as unknown, while a disconnect during SMTP authentication could
stop retries as a credential failure. Structural network evidence now
takes precedence over diagnostic prose. Explicit unconfirmed SMTP
acceptance after DATA keeps its existing unknown verdict and transport
retry stop.
HTTP Apprise discarded response-read errors and could mark interrupted
2xx responses sent with successful receipts. It now shares the bounded
webhook response verdict and safe diagnostics. Invalid payload
templates, malformed retained configuration and unsupported retained
types become configuration failures on their first attempt. Enhanced
ntfy uses the canonical renderer, and errors/logs withhold private
template and rendered payload text.
The shared delivery activity label and retention description say retries
stopped. Dead-letter status alone cannot establish budget exhaustion
when a permanent verdict stops the first attempt.
Regression controls use real HTTP and SMTP receivers plus reopened
queue, audit, receipt and telemetry records. They cover transient SMTP
recovery, deterministic failures without outgoing requests, interrupted
and complete responses, authoritative HTTP/SMTP verdicts, genuine
unknowns and confidentiality. Evidence and remaining destination-level
scope are recorded in
docs/release-control/v6/internal/records/notification-unclassified-reliability-2026-10-18.md
under the existing notification-delivery-terminal-failure-diagnosis gap.
Aggregate telemetry cannot attribute fleet impact or establish recovery.
Validation passed: the full notifications package, targeted notification
race controls, 51 frontend tests, and the production Notifications
browser fixture at 1440x1000 and 390x1100. Browser checks cover labels,
safe details, refresh, cancelled retry with focus return and reload.
Staged status, registry, contract, completion and browser-receipt audits
pass.
Terminal failures can stop early and do not prove retry-budget exhaustion.
Update the existing operator qualification expectation to match the shared
retention description already exercised in the browser.
Preserve selected memory freshness and the VM/LXC interpretation.
Retain both native tool contracts and current API concurrency controls.
Qualify this source separately from release and installed recovery.
Contract-Neutral: Integrate already-governed parents without a new contract change. Incoming verification files already match main. Fresh integration tests and source-bound browser proof cover the combined memory and installer surfaces.
Connection closes and invalid local jobs entered the unknown failure bucket,
and disconnects during SMTP authentication could stop transient retries.
Use structural network evidence and declared configuration verdicts while
preserving unconfirmed SMTP acceptance as unknown.
Propagate interrupted HTTP Apprise responses through the shared bounded
response reader so they cannot create successful queue receipts. Share ntfy
template rendering and withhold private template and payload diagnostics.
Describe retained terminal failures as retries stopped, since a permanent
verdict can stop the first attempt before the budget is exhausted.
Require the existing troubleshooting jobs and their destinations without
making additional help entries break unrelated safety checks. Keep the
opening backup warning scoped to its actual entry point.
Run closed service and middleware models through the containing fixture's
selected Python interpreter without site startup. Retain every safety
scenario and deadline while qualifying the pending maintenance source.
Refs #2888
Contract-Neutral: Test-only navigation assertions and confined model interpreter selection. Production behavior and safety deadlines are unchanged.
SMTP LOGIN rejected correct credentials when a receiver used legacy,
empty or localized challenge text. Authentication now follows
username/password exchange order and refuses incomplete or excessive
challenges without copying receiver-controlled prompts into errors. AUTH
selection matches complete mechanism names and reports unsupported lists
as configuration failures before attempting an unadvertised mechanism.
Connection tests and delivery share the implicit TLS rule for port 465,
avoiding a second STARTTLS upgrade inside an established TLS session.
The existing PLAIN preference, legacy probe, TLS trust policy and
structured server verdicts remain covered.
Validation exercises real plain/TLS/STARTTLS receivers, connection
tests, ordinary/threaded/attachment sends, bad credentials, temporary
replies, certificate refusal, and persisted queue/audit/telemetry
verdicts after restart. The full notification package, targeted race
controls, and status, registry and contract audits pass. Evidence is
recorded in
docs/release-control/v6/internal/records/notification-authentication-reliability-2026-10-18.md
under the existing notification-delivery-terminal-failure-diagnosis gap.
Aggregate telemetry cannot identify which fleet failures these defects
explain or establish fleet recovery.
Literal LOGIN prompt matching rejected valid credentials and substring AUTH
selection could choose an unadvertised mechanism. Use exchange order and
complete mechanism names, preserve server reply classes, and classify
unsupported mechanism lists as configuration failures.
Share the implicit TLS decision between connection tests and delivery so a
port-465 test does not attempt STARTTLS inside an established TLS session.
Add real SMTP controls and persisted queue verdict verification.
The reliability roadmap still described a shipped source-binding repair
as unlanded and the backup-running correction as preview-only. Reconcile
both against stable v6.5.0 and link the remaining work and manual exit
evidence through the existing L1 follow-up and active target.
The linked record distinguishes completed repairs from final-source
maintenance qualification, installed customer recovery, each candidate's
own soak, stable delivery, and normal operational recovery. The
recovery/change plan remains deferred. Active priority, lane states,
release gates and the planned feature sequence are preserved.
Validation: governance audits, exact before/after state and sequence
checks, ancestry and reporter evidence, and normal commit and push
checks on the validation worker.
Replace stale unshipped-fix claims with stable ancestry and reporter
results. Link remaining delivery and recovery outcomes to the existing
manual reliability target without changing priority or release gates.
Pinned HTTP dialing could spend a request's timeout on a stalled first
address even when another approved address was healthy. Both
notification and restricted HTTP clients now use one bounded fallback
dialer. It limits overlapping connections to two, shares the connection
budget, preserves DNS pinning and closes unused successful connections.
Webhook DNS admission and dialing now honor cancellation with a
five-second lookup ceiling. Redirect validation retains the request
context and empty DNS results fail closed. Shared transport ownership
and proof routing are explicit.
Validation covers a real stalled TCP listener beside a healthy HTTP
receiver, cancellation through the production resolver, one queued
delivery and its successful persisted verdict after restart, full
affected package tests and targeted race checks. Aggregate fleet
telemetry identifies connectivity as the largest failure class but
cannot attribute those failures to these defects or establish fleet
recovery.
Serial pinned-address dialing could spend the request deadline on a stalled
first address while a validated fallback was healthy. Share one bounded
two-worker dialer with restricted HTTP clients and retain cancellation in
webhook DNS admission and dialing.
Add real stalled-TCP and DNS cancellation controls, persistent queue verdict
readback, shared transport ownership and explicit proof routing.
The recovery and change investigation audit was saved separately from
the roadmap records used to select future work. Link it from the planned
product target and the existing L15/L19 follow-ups, with an explicit
handoff after reliability yields priority.
The recorded order is backup task outcomes, focused change
investigation, workload protection assessment, safe configuration diffs,
then demand-dependent recovery-test evidence. The active reliability
target, lane states, release gates and demand thresholds stay unchanged.
Validation: control-plane, status, registry and contract audits,
explicit before/after governance invariants, and normal commit and push
checks on the validation worker.
Link the ordered audit into the planned product target and existing L15
and L19 residuals so future work has an explicit pickup path. Preserve
the active reliability target, lane states and release qualification.
Supported SMTP providers could pass validation with their default
username while delivery skipped authentication and failed with SMTP 530.
Preparing overlapping sends could also overwrite another delivery's
sender and recipients. Resolve provider credentials when creating the
SMTP manager and keep each delivery's configuration independent while
preserving the shared rate limit. SMTP attempt logs also report the
resolved authentication state for provider defaults.
Real SMTP regression controls cover all four provider defaults, accepted
envelopes, concurrent sends, shared rate limits and persisted queue
outcomes after restart. Invalid credentials still produce an
authentication dead letter. The notification contract and investigation
record retain the broader reliability gap because aggregate telemetry
cannot attribute fleet failures to these two defects.
Validation on pulse-dev with Go 1.26.9:
- Full notification package passed.
- Focused SMTP and queue controls passed normally and under the race
detector.
- Release-control status, registry and contract audits passed.
Provider-default usernames could leave hasAuth=false in SMTP diagnostics even
though the delivery manager required authentication. Report the resolved
manager state at the SMTP attempt and remove the premature admission-level
credential inference.
Refresh synthetic SMTP, persistent queue and race evidence for the final source.
Contract-Neutral: Diagnostic-only follow-up: report resolved SMTP hasAuth state. Existing real SMTP, persisted queue and race controls were rerun and retained.
Audit Pulse's recovery and change investigation flows against PegaProx
and ProxCenter, then prioritise the gaps worth tackling after
reliability. The plan records existing capabilities, customer demand,
source ownership, dependencies and acceptance criteria. It preserves the
current reliability target and lane states.
Validation includes a current-source build on the development worker,
desktop and narrow browser inspection with mock data, and injected
posture and inventory failures. Real provider recovery and competitor
reliability were outside this audit.
Use current source, browser evidence and existing demand to sequence the
next product work after release reliability. Preserve the active target
and the existing lane completion states.
Refs #1812
Supported provider defaults could pass validation without enabling SMTP
authentication, and preparing concurrent sends could overwrite another
delivery's sender and recipients. Resolve credentials when the SMTP manager
is created and keep each delivery's configuration independent while sharing
the existing rate-limit state.
Verify real SMTP acceptance, invalid-credential dead letters, persisted
queue outcomes after restart, concurrent envelopes and shared rate limits.
Configured report schedules and recent attempts do not show successful
delivery or repeat use. Add durable aggregate reporting counters to the
existing opt-out-controlled heartbeat, covering resources, VM inventory,
backup health and Patrol digests.
Count successful generation, saved files, SMTP acceptance, definite
delivery failures and uncertain acceptance separately. Track distinct
successful saved-schedule periods and active days, including manual
downloads. Deduplicate retries and same-period reruns for repeat-period
counts. The adoption report distinguishes unmeasured senders and partial
observation from measured zero use. Generation across multiple days is
reported separately from successful repeat periods because it can
include retries.
No report contents, recipients, raw schedule or tenant identifiers,
scopes, errors or date ranges enter these new telemetry fields. Mock
activity is suppressed. Pricing, billing, customer entitlements and the
existing outbound opt-out remain unchanged. Update the schema-v20
disclosure and privacy document.
The shared telemetry observer has an exact-path adapter declaration.
Boundary regression fixtures reject sibling reporting implementation and
private licensing imports.
Validation: reporting persistence, API delivery/retry/restart and
successful-write tests, server snapshot tests, the full telemetry
package, 45 adoption-report tests, frontend type-check and build, and
governance audits passed. The three telemetry browser tests and a
separate pixel interaction check passed against the final source at
1280, 390 and 768 pixels, including saved opt-out states and the privacy
table. Source-bound browser receipt included.
Companion receiver: rcourtman/pulse-pro#472. Receiver deployment and a
client release are separate activation steps. This change alone does not
start production collection.
A scheduled retry across UTC days can generate another artifact without
completing a new successful reporting period. Keep multi-day generation as
an activity observation rather than inferred repeat use, while retaining
successful distinct-period counts as their own signal.
Exercise both cross-day generation without success and multiple successful
periods on one day so the adoption summary cannot conflate these metrics.
The usage observer records aggregate outcomes through the existing settings
store without implementing a paid renderer or entitlement decision. Declare
its exact shared-adapter path so the repository audit can distinguish it
from new private reporting implementation.
Exercise the real audit against sibling paths and private licensing imports
to retain those restrictions for the observer.
Record successful response writes, scheduled generation, file saves and SMTP
outcomes in a durable content-free tally. Distinct active days include manual
exports, while saved-schedule periods deduplicate retries and monthly reuse.
Publish bounded scalars through the existing opt-out-controlled heartbeat and
report missing coverage explicitly. Preserve pricing and customer entitlements.
Release candidate cut from main at 0d5b877425. It carries every change merged since stable v6.5.0.
Contract-Neutral: Release-only v6.6.0-rc.4 version, release notes and distribution metadata; no public install or runtime contract changes.
Change-source: pulse-maintainer
Client backup reports can now be scheduled weekly or monthly as PDF or
CSV, scoped to current VMs and containers by resource or client tag.
Reports show recorded outcomes, daily failure rates, last retained
success, failure details and missing evidence. Weekly reports cover
seven completed local calendar days. Monthly reports cover the previous
calendar month.
Backup executions and observation coverage are retained in each
organization's recovery store. Shrinking provider task inventories and
restarts preserve recorded failures. Unavailable history, permission
limits, collection gaps and incomplete multi-guest attribution remain
explicit. A successful task does not certify that a backup is retained
or restorable. This reports Proxmox VE backup jobs, including jobs
writing to PBS.
The existing scheduler, branding, delivery checks, bounded retries and
deliberate uncertain-delivery recovery are reused. The public adapter
supplies scoped evidence to a private PDF/CSV renderer. Pricing, feature
entitlements and retention remain unchanged.
Validation:
- Frontend: 136 focused tests, full type check, lint, production build
and bundle budgets.
- Race checks cover recovery persistence, collector quality, reporting,
organization isolation and runtime registration. Private rendering and
runtime build checks cover the composed Pro implementation.
- Live runtime checks on `/settings/support/reporting` and
`/alerts/notifications` at 1440x1000, 820x844 and 390x844 in light and
dark themes, with actual pixel review. The 61-workload selection remains
bounded and keyboard usable. All five generated PDF pages were
inspected.
- Real client-scoped CSV and MIME attachment contents, retained failure
after an empty provider inventory and restart, retry after restart,
permanent rejection, uncertain acknowledgement and deliberate recovery.
An unmatched client scope produces missing evidence with blank counts.
Keep the full catalog assertion aligned with the intentional default for
servers that omit the optional backup report capability. The normalized
catalog must disable backup health while retaining all existing fields.
Retain backup execution outcomes and collection coverage in each
organization's recovery store so provider inventory changes and restarts
cannot erase report evidence. Keep missing and incomplete history distinct
from an observed period with no attempts.
Reuse existing report scopes, calendar schedules, branding and delivery
recovery for client PDF and CSV reports. Register the paid renderer through
the existing private runtime boundary. Pricing and grants are unchanged.
## Why
#3033 stopped organization owners' browser sessions from reaching
instance administration, but API tokens on instance routes were still
judged by scope alone. The tenant middleware resolves an org-bound token
to its own organization without the caller naming one. So a token bound
to a tenant organization and holding `settings:write` passed the tenant
check, then `ensureSettingsScope` / `RequireAdmin`.
Before this change, such a token with no org header:
- got `200 {"success":true}` from `POST /api/system/settings/update`;
- reached the SSO provider create handler;
- listed tokens;
- read relay config, log level and update status.
`TestOrgBoundSettingsTokenRouterMatrix` fails on `main` with exactly
those responses.
## What changes
- **Instance routes** (`ensureAdminSession`, which `RequireAdmin` uses,
and `ensureSettingsScope`): a token must act for the instance.
`tokenActsForInstance` admits a token only if it isn't bound to any
organization other than the default one; `OrgIDs` takes precedence over
`OrgID`. The route's own scope still applies.
- A tenant-bound token gets `403` whatever its scopes, `*` included,
just as that organization's owner does.
- The audit, RBAC admin, audit-webhook and change-password routes call
`ensureAdminSession` directly, so they now refuse org-bound tokens too,
matching their session rule.
- **Org routes** (`RequireOrgAdmin`, `ensureOrgAdminSession`,
`ensureOrgSettingsWriteScope`): unchanged for tokens. These are judged
by scope alone.
- `tokenHasInstanceWideAuthority` (quick setup, token regeneration) is
now `*` plus the same binding rule.
- **`/api/test-notification` moves to `RequireOrgAdmin`.** It only
broadcasts to the request organization, and #3031's
`TestRegisteredTestNotificationStaysWithRequestOrganization` relies on
an org-bound token sending it. Side effect: that organization's managers
can now send it from a session.
- **Minting paths an org manager can reach** were checked; none mints
`settings:*`.
- Relay mobile mints `relay:mobile:access`.
- Docker prepare-token mints `docker:report` plus agent report and
config read.
- Install tokens mint agent and report scopes, plus `agent:exec` only
when commands are enabled.
- Deploy mints `agent:enroll`, then the runtime agent scopes.
- The action runner mints `agent:exec`.
- Token create, edit and rotate are instance administration.
- **Docs and contracts:**
- `docs/API.md` and its shipped mirror now state the token rule.
- api-contracts rule 35 records the token split.
- security-privacy records it in the 17 October section.
- agent-lifecycle and storage-recovery carry short dependent notes.
## Proof
- `internal/api/org_bound_token_authority_test.go`:
- A unit matrix of 7 token bindings × 3 scope sets across the instance
and org helpers.
- `RequireAdmin` vs `RequireOrgAdmin`.
- A full-router matrix: an instance-wide token and an org-bound token on
11 instance routes, plus an org-bound token with and without settings
scope on 2 org routes.
- `TestSettingsWriteScopeSplitsInstanceAndOrgBoundToken`,
`TestContract_OrgBoundTokenCannotReachInstanceAdministration`,
`TestRegisteredTestNotificationAdmitsRequestOrganizationManagers`, and a
`docsLinks.test.ts` case.
- Full `go test ./internal/api/... -timeout 50m` passes on pulse-dev.
The new route-inventory and docsLinks cases, which were added after that
run, also pass on their own.
- `scripts/tests/test_api_reference_routes.py` and
`scripts/check_docs_mirror.py` pass.
- The commit passed the full `.husky` pre-commit hook.
## Coordination
- The four contract files are under a workspace claim held by
`client-backup-reports-20261017`. While this was being prepared, that
agent had uncommitted edits to them in `repos/pulse`, and those hunks
don't overlap this one:
- api-contracts: theirs was around line 12164; this one edits rule 35,
around line 5060.
- security-privacy: theirs appended at the end of the file; this one
stays inside the 17 October section.
- storage-recovery: theirs was around line 7021; this one edits the
test-notification paragraph, around line 6987.
- agent-lifecycle: this one adds to the 17 October agent-routes note.
- At push time `repos/pulse` no longer showed uncommitted edits to these
files.
- This commit was built and committed in an isolated pulse-dev checkout,
outside `repos/pulse` and the shared working trees.
The organization-owner fix judged browser sessions only. API tokens on
instance routes were still checked for scope alone, and the tenant
middleware resolves an org-bound token to its own organization without
the caller naming one. A token bound to a tenant organization and
holding settings:write therefore reached system settings, SSO
providers, token management, relay, log level, updates and the other
instance surfaces.
ensureAdminSession, behind RequireAdmin, and ensureSettingsScope now
refuse a token unless it acts for the instance: not bound to any
organization other than the default one, with OrgIDs taking precedence
over OrgID. The route's scope still applies. The org variants
(RequireOrgAdmin, ensureOrgAdminSession, ensureOrgSettingsWriteScope)
keep judging tokens by scope alone. tokenHasInstanceWideAuthority is
now the same binding rule plus full-access scope.
The test notification broadcasts only to the request organization, so
it moves to RequireOrgAdmin. Its tenant delivery test already relied on
an org-bound token sending it, and that organization's managers may now
send it from a session too.
No path an organization manager can reach mints a settings scope: relay
mobile, install, deploy and action-runner tokens carry relay, agent and
report scopes only, and token management is instance administration.
The API reference and the api-contracts, security-privacy,
agent-lifecycle and storage-recovery contracts record the token rule.
VM inventory was available only as an on-demand download. It can now be
scheduled weekly or monthly as CSV, emailed or saved on the Pulse
server. Each attempt exports all current VMs in the organization,
including newly discovered VMs, through the existing exporter.
Historical performance-report limits and windows remain separate.
The schedule uses existing delivery configuration checks, persisted
success/failure/uncertain states, bounded retries and deliberate
recovery controls. Pricing, feature gates, retention entitlements and
existing customer grants are unchanged.
Live checks also reproduced two unattended inventory defects: reporting
retained a stopped monitor after configuration reload, and canonical
snapshot reads relied on dashboard reads to publish polled VM inventory.
Reporting now receives reload replacements through an atomic manager
reference. Canonical snapshot reads reuse the existing publication
boundary, including its freshness fast path, provider/mode fences and
nonblocking warm reads.
Validation:
- 105 focused frontend tests; full type check, lint, production build
and bundle budgets.
- Race checks for reporting, inventory, reload, publication and
organization isolation; clean private runtime build.
- Current runtime browser checks on `/settings/support/reporting` and
`/alerts/notifications` at 1440×1000, 820×844 and 390×844, with actual
pixel review. The first inventory read exports 61 normally polled VMs;
real MIME CSV matches the download; a newly discovered 62nd VM enters
the next export.
- Local SMTP acceptance, 451 retry across a real restart, permanent
rejection, uncertain acknowledgement and deliberate recovery; disk
delivery, edit/reload, disable, details and delete.
- Staged status, registry, contract, completion and exact-content
browser guards pass. Regression proof is registered against the specific
runtime paths it covers.
## Why
Report generation read every report's metrics from the default
organization's store, whatever organization the request came from. The
engine was wired in `pkg/server/server.go` to the default monitor's
store, and `ReportEngine.queryMetrics` queried it with the caller's
resource ID. A tenant request for a resource outside its own inventory
therefore read the default organization's history, and even a tenant's
own resources were reported from the wrong store. Pulse Assistant's
`pulse_summarize` used the same engine, so it read the same store. Other
tenants' stores were not reachable.
#3033 closed reporting to organization owners until this was fixed.
Providers build reports inside client workspaces (4d47504331), so
reporting reopens to them here.
## What changes
- **Engine** (`pkg/reporting`): `MetricReportRequest` and
`MultiReportRequest` carry `OrgID`. With
`EngineConfig.MetricsStoreForOrg` set, the engine resolves that
organization's own store and refuses a request without an organization
(`ErrReportOrgRequired`). A multi-resource report applies its own
`OrgID` to every resource.
- **Server wiring** (`pkg/server/reporting_engine.go`): the default
engine resolves stores per request through
`MultiTenantMonitor.GetMonitor(orgID)`, so it follows monitor reloads
and never falls back to the default organization.
- **Handlers**: single and multi-resource reports must name a resource
in the request organization's inventory: a unified resource ID, or a
legacy node/VM/container ID. Anything else returns 404
`resource_not_found`, or 503 `inventory_unavailable` when the inventory
can't be read.
- **Schedules**: a run resolves explicit resources and tags against its
owning organization's inventory, which the per-org schedule store and
scheduler context already supply. Resources the organization no longer
holds are skipped and logged; a scope with none left fails the run.
- **Pulse Assistant**: `pulse_summarize` passes its executor's
organization.
- **Access**: every `/api/admin/reports/*` route (catalog, generate,
generate-multi, VM inventory export, schedules) uses
`ensureOrgAdminSession`, and `reportingRead` is offered to organization
managers again (it still requires `read:nodes`).
- **Contracts**: rule 35 in api-contracts, the superseding note in
storage-recovery, plus ai-runtime, security-privacy and agent-lifecycle.
Behaviour change: a report can no longer be generated for a resource
that has left the organization's inventory, even if its retained history
is still in the store.
## Proof
- `internal/api/reporting_org_scope_test.go`: org B gets 404 for org A's
resource, by legacy and unified ID, through GET, POST and
generate-multi, and receives only org B's values even though org A's
store holds points under the same ID.
- `TestReportingScheduledRunReadsOnlyItsOwningOrganizationsMetrics`: an
org B schedule naming org A's resources reports only org B's resource
from org B's store; a scope of only foreign resources fails.
- `pkg/reporting/engine_org_scope_test.go` and
`TestReportEngineReadsOnlyTheRequestOrganizationsMetrics`: per-org
resolution and refusal without an org, across Generate, GenerateMulti,
NarrativeFor, FleetNarrativeFor and MetricEvidenceFor.
- `pkg/server/reporting_engine_test.go`: the server's engine resolves
the request org's monitor store, and refuses an empty, unprovisioned or
unavailable org.
- `TestSummarizeToolReadsOnlyItsOrganizationsMetrics`;
`TestSecurityStatusOrgBoundSettingsFollowOrgManagementRule` now holds
`reportingRead` in the org-bound set.
- On pulse-dev: `internal/api` (`-timeout 50m`), `pkg/reporting`,
`pkg/server`, `internal/ai/tools` and `internal/repoctl` pass. The
reporting and schedule tests also pass under `-race`.
## Governance
The canonical completion guard maps `pkg/server/server.go` to
security-privacy's "privacy docs and telemetry" verification policy,
which accepted only telemetry tests as proof. The `server.go` change
here is the report engine's per-organization store wiring, a
tenant-isolation boundary. `registry.json` now gives
`pkg/server/server.go` its own security-privacy path policy,
`server-runtime-wiring`: it keeps every proof the telemetry policy
accepted and also accepts `pkg/server/reporting_engine_test.go`. The
other telemetry files keep their existing policy, so they can't cite the
wiring test. That test proves the server's engine resolves the request
organization's monitor store and refuses an empty, unprovisioned or
unavailable organization. No telemetry test was edited to satisfy the
guard. The commit passed the full pre-commit hook.
## Coordination note
`vm-inventory-schedules-20261017` currently holds workspace claims on
`internal/api/report_schedules.go`,
`internal/api/reporting_runtime_snapshot_test.go`, `api-contracts.md`,
`agent-lifecycle.md` and `storage-recovery.md`, with uncommitted work in
its own checkout. This PR's hunks in those files
(`resolveReportScheduleResources`, a new helper appended to the snapshot
test, rule 35 and the `reportingRead` mount line in api-contracts, one
paragraph each in agent-lifecycle and storage-recovery) don't overlap
that work. This commit was built in a separate checkout, not in that
agent's working tree.
Reuse the canonical inventory exporter and report delivery workflow so
operators can email or save a current organization-wide inventory weekly
or monthly. Keep delivery checks, bounded retries and recovery states
shared, with existing pricing and customer entitlements unchanged.
Refresh the shared inventory for unattended reads and rebind reporting
after monitor reload so schedules use the current infrastructure state.
The hub kept tenant-scoped delivery for state and alerts, but every
other event went through Broadcast/BroadcastMessage, which sent to all
connected clients. In a multi-tenant install a member of one
organization therefore received another organization's discovery
progress, discovered servers (addresses, ports, versions, hostnames),
node auto-registration and node-change events.
Discovery now carries the org of the monitor that owns it, and setup
and node handlers send through BroadcastMessageToTenant with the
request's org (the setup token's org for auto-registration). The
global BroadcastMessage accepts only instance-wide types (ping,
settingsUpdate, update:progress) and refuses everything else, so a new
caller cannot leak by default. Legacy clients without an org are
registered under the default tenant, so single-org installs receive
these events as before. The unused unscoped alert wrappers are removed.
The report engine read every report's metrics from the default
organization's store, whatever organization the request came from. A
tenant request for a resource ID outside its own inventory fell through
to that store with the caller's ID, so one organization could read the
default organization's history, and even a tenant's own resources were
reported from the wrong store. Reporting was closed to org owners until
this was fixed.
The engine now resolves the metrics store of each request's OrgID
through the multi-tenant monitor and refuses a request that names none,
so no report path can fall back to the default store. Report handlers
reject resources outside the requesting organization's inventory,
scheduled runs resolve their scope against their owning organization's
inventory and skip resources it no longer holds, and Pulse Assistant's
pulse_summarize reads its executor's organization.
With every report read scoped, the reporting routes move to
ensureOrgAdminSession and reportingRead is offered to org managers
again, so providers can build reports inside client workspaces.
The report engine wiring lives in pkg/server/server.go, which the
completion guard only let telemetry tests prove. server.go now has its
own security-privacy path policy that keeps those proofs and also
accepts pkg/server/reporting_engine_test.go; the other telemetry files
keep their existing policy unchanged.
`PULSE_METRICS_DB_PATH` sets `Config.MetricsDBPath`.
`MultiTenantMonitor.GetMonitor` builds each non-default tenant from
`baseConfig.DeepCopy()`, which kept that path, so `monitoring.New`
opened the same `metrics.db` for every organization. With the override
set, per-org charts, reports and Pulse Assistant summaries all read from
one shared store.
Changes:
- `GetMonitor` clears `MetricsDBPath` on non-default tenant configs
after setting their `DataPath`. Each org then opens `metrics.db` in its
own data directory, and `DeleteOrganization` removes it along with the
rest of the org's data.
- The default org still uses the override, which covers the
single-install tmpfs/RAM-disk case it exists for. Metrics for
non-default orgs stay on the data volume.
- New `multi_tenant_monitor_metrics_path_test.go` sets the override,
provisions two orgs, and checks that each opens its own `metrics.db` and
cannot read series written by the default org or the other org. It fails
on current `main`.
- `TestMetricsStoreRuntimeOverridesStayOnMonitorBoundary` now also
requires the clear to come between the tenant `DataPath` assignment and
`New`.
- The monitoring contract and `CONFIGURATION.md` (plus its public
mirror) now describe the default-org-only behaviour.
Verification: `go test ./internal/monitoring/ ./internal/config/` passes
on Linux. `canonical_completion_guard.py` passes for this commit.
pulse_read runs commands on monitored hosts, usually as root, without
approval at every control level, as long as the classifier calls them
read-only. The classifier approved a command from its first word and
checked arguments only for find, curl and timeout. So date -s, sort -o,
journalctl --vacuum-time/--rotate, fuser -k, dmesg -C, sar -o, ss -K,
hostname NAME, ifconfig/arp/ip setters, smartctl SCT setters and many
other state-changing forms passed as known read-only, and route add was
blocked only because "add " matched the dd substring rule.
Every allowlisted program with mutating flags or subcommands now has an
explicit safe-forms rule, and unknown options fail closed. Commands
are parsed as literal shell words. Expansions, redirects other than the
fixed fd ones, command chaining and globs on validated programs are
refused, and every pipeline stage must itself be read-only. route is
judged on purpose. Ordinary diagnostics (journalctl -u ... --since,
dmesg -T, ss -tlnp, ip addr, sar 1 1, docker ps/logs, smartctl -a,
systemctl status) stay read-only. Plain blkid now needs -p, because
without it blkid updates its cache.
The canonical completion guard checks each commit on its own; the
delivery commit relies on the following commit for its contracts and
proof files, so record that exact pair.
rcourtman