An open drawer could retain a current-only reply after its agent entered
the first-report wait. Withdraw live snapshots immediately and refresh on
reporting transitions while preserving real stored observations.
Integrate published main while retaining the webhook acknowledgement and
chart optimizations. Keep guest metric admission under provider ownership.
Refs #2980
An expired saved enrollment still has no current telemetry. Keep summary
charts from turning that deadline into zero-valued live samples, while
preserving real stored history. Cover both sides of the reporting lease
and align the remaining pending-label assertions with the shared wording.
Refs #2980
Keep retained enrollments unknown while their accepted-report lease is
current, then show offline if no fresh report arrives. Preserve historical
timestamps, removals and report ordering without manufacturing telemetry.
Use the shared first-report state in Machines, resource details and Settings.
Refs #2980
Return the explicit success boolean the existing editor requires after durable and live removal, while retaining the legacy status field. Pin the response contract so a successful deletion is not displayed as failed and does not encourage blind retries.
Change-source: pulse-maintainer
Share the existing time-nearest selection by original indices, so chart conversion materialises only selected observations. Preserve exact timestamps, values, tie breaks, degenerate spans and oldest-source metadata. Check the frozen former sampler across 1,440 cases and the exported fallback contract.
Change-source: pulse-maintainer
Contract-Neutral: Exact chart observations, wire schema, sampling policy and guest/storage lifecycle contracts are unchanged; only index selection and allocation order change.
Resolve adjacent contract append conflicts by keeping bounded hosted-agent classification in its chart section and preserving all newer SSO, RBAC and uninstall guarantees. Runtime, workflow and synthetic chart controls merge without conflict; no latency budget or customer journey changes.
Change-source: pulse-maintainer
**Reject competing audit webhook destination fields**
Parse one bounded update object without discarding repeated urls
members, including escaped and case-folded aliases. Preserve deliberate
clears, singleton aliases and unknown fields; reject ambiguous or
incomplete selections before validation and mutation.
**Bind audit guidance smoke checks to the unambiguous decoder**
The complete-selection admission moved out of the inline struct. Retain
the route/access and organisation assertions, and require decoding
before URL validation instead of the obsolete struct spelling.
Contract-Neutral: Align a static documentation check with the reviewed
decoder; runtime and published contracts are unchanged.
**Reject competing public SSO and recovery request fields**
Validate complete bounded admin objects before decoding or applying
them. Preserve singleton aliases, unknown fields, case-sensitive role
map keys, explicit access clears, saved secrets and existing tenant and
one-use token boundaries. Add public producer, persistence and API
contract controls.
**Preserve published audit smoke bindings in the SSO candidate**
Resolve the overlapping audit-doc test correction against published main
52457777c4 by retaining its stronger
bounded decoder and mutation-order assertions. No runtime or frontend
source changes; the reviewed SSO and recovery proof remains applicable.
Contract-Neutral: Integration-only static test reconciliation; API,
runtime, frontend and documented contracts are unchanged.
---
Fleet candidate `20261024T084021Z-core-runtime` from the `core-runtime`
lane. Reviewed tip `7a50adc5ac6924829c1e871c638db9b461b40011` on `main`.
[How Pulse handles
triage](https://github.com/rcourtman/Pulse/blob/main/docs/AI_TRANSPARENCY.md)
Retained hosted samples include both an over-budget failure and an under-budget pass with one provisioning agent occupying about one logical CPU. Attribute only that bounded Actions overhead; retain raw CPU and absolute 90 ms latency, and keep other or unattributed CPU, guest, steal and swap invalidating. Exercise the real CLI and workflow shells with adverse controls; no timed workload or release acceptance is claimed.
Change-source: pulse-maintainer
**Reject competing audit webhook destination fields**
Parse one bounded update object without discarding repeated urls
members, including escaped and case-folded aliases. Preserve deliberate
clears, singleton aliases and unknown fields; reject ambiguous or
incomplete selections before validation and mutation.
**Bind audit help checks to the bounded selection decoder**
Repair the containing CI failure from PR 3186 without restoring
last-member-wins parsing. Keep the explicit array, request bound, URL
validation before mutation and organisation-scoped replacement checks;
runtime selection semantics remain covered by the existing handler
tests.
Contract-Neutral: Static guidance/source-binding test repair only; the
composed audit runtime and documented API, access, storage and delivery
contracts are unchanged.
---
Fleet candidate `20261024T082008Z-delivery-trust` from the
`delivery-trust` lane. Reviewed tip
`58e46e21d7c9bd265e88485d26bfed38e03fb31e` on `main`.
[How Pulse handles
triage](https://github.com/rcourtman/Pulse/blob/main/docs/AI_TRANSPARENCY.md)
The Patrol inbox passed each alert's own resource reference through as the
item subject: docker:<host>/<container> for a Docker container,
<instance>:<node>:<vmid> for a guest, agent:<host>/disk:<label> for a
filesystem. Patrol findings, protection postures and resource actions carry
the unified resource ID instead, so "Browse Patrol findings for this
resource" opened an empty list, postures never attached, and the alert
detail said Patrol had raised no finding while it had one on that exact
container.
The attention projection now resolves each record's subject through the
org registry's alert reference resolver and names the unified resource. The
operational record keeps the alert's reference, and a reference inventory
cannot bind stays as before. The resolver gains Docker container references
with a complete ID, the key RecordChange already joins their history to, so
container alerts resolve like every other kind and alert-triggered Patrol
scopes for them keep the triggering alert. It also reads source targets
from the registry's view cache rather than scanning every mapping per
reference, which a page of alerts would otherwise multiply.
The alert-mirror reconcile compared the same mismatched strings and never
folded a finding under its alert; it now matches either identity. A
filesystem, sensor or ZFS pool alert resolves only to the owner its siblings
share, so the resolver marks it as a component and the reconcile does not
match on that owner. The inbox links a finding to the exact alert it
restates by the stamped alert ID, not by resource and kind, which several
alerts on one host share. The restart offer checks evidence against the
record's own reference, since the item subject is now the unified container
its registry lookup needs.
When no finding restates the alert but Patrol has open findings on the same
resource, the detail names them instead of saying Patrol raised nothing,
and the browse button appears only when there is something to browse.
Parse one bounded update object without discarding repeated urls members, including escaped and case-folded aliases. Preserve deliberate clears, singleton aliases and unknown fields; reject ambiguous or incomplete selections before validation and mutation.
Change-source: pulse-maintainer
The second tenant control must preserve the first tenant deliberate clear, not require that the earlier update never happened. Capture both update counters before the rejected request and require them to stay unchanged. No production behaviour changes.
Change-source: pulse-maintainer
Contract-Neutral: Correct the synthetic tenant control baseline; production audit handlers and contracts are unchanged.
Compose retained 8478eac3 additively on current main. Preserve both independently appended contract sections at each of the three documentation conflicts; workflow, registered executable controls and classifier stay unchanged.
Change-source: pulse-maintainer
Isolate the retained PR3095 completion (7f4ec192 and 93d87118) without importing unrelated history. Keep the exact proposed Python classifier and maintained chart shells, real CLI/workflow controls and dependent contracts. Inconclusive capacity never establishes the 90 ms budget; child, cleanup and malformed-proof failures remain errors.
Source-PR: https://github.com/rcourtman/Pulse/pull/3095
Source-Commit: b31fabd778
Co-authored-by: rcourtman <8825017+rcourtman@users.noreply.github.com>
Change-source: pulse-maintainer
Three gaps remained in how Pulse projects a Patrol investigation's
action and closes its runs.
Projections of one investigation could interleave: the transition
callback, an investigation read and a run's close each read the action
audit and applied it, so one that read an older state could land after
one that read a newer state, and an approval could land between a
run's reconciliation and its finalization. All of them now read and
apply under the investigation's lock, and a close finalizes before it
releases it.
The projection replaced the whole session through Update, so it could
undo run progress the orchestrator saved in between. It now writes
only the changed action and outcome when the store offers SetAction
(aicontracts.InvestigationActionStore), and the reference it writes
carries the broker's canonical result and verification status and the
model's causal attribution, which the audit does not hold.
A run that hung past its limit without closing was recovered by the
stuck-run sweep but never finalized. The sweep now finalizes it with a
copy of its session marked failed, leaving the orchestrator's session
as it is. A later real close is finalized instead, and a refresh no
longer reopens a record that finalization closed.
A Patrol run that planned a fix and then failed, normally on its final
model call, was never finalized. Core's run goroutine returned on the
error before it rebuilt the investigation record, published the
unified finding and sent the approval push, so the record kept showing
the run as running and nobody was asked to approve the queued fix.
Manual re-investigations never reached that goroutine, even when they
succeeded, and a run a restart interrupted was only marked closed.
Core now finalizes a run when its close lands on the finding or when
the stuck-run sweep recovers it with a closed session. It first
reconciles the run's action from the action audit, because the
orchestrator writes the action back as planned and an approval or
expiry during the final model call may have overtaken it. For the same
reason a close no longer moves a finding back to fix_queued from an
outcome the action reached during the run. Finalization skips a run
the finding no longer names and asks for approval once per action. An
action's outcome keeps its single push from transition reconciliation,
so a close no longer sends a second fix-completed notification.
The action inbox and decision-queue expiry sweeps also publish the
plans they expire, as a single action's expiry already was, so a
Patrol finding's queued fix moves to needs_attention when its plan
expires. Reconciliation follows refresh lineage to ignore a transition
for a plan that the investigation's current action replaced, so the
expiry of a refreshed plan cannot take the investigation back.
With the web chat window and its backend routes retired (#3126,
#3137), the pulse_assistant surface only serves Pulse Mobile's chat.
Its manifest description still called it the explanation and handoff
surface for Patrol findings, and that text is public: the docs/AI.md
overview, the MCP README surface block, and the External agents
settings panel all show it. It now names the Pulse Mobile role and says
the web chat is retired.
The browser's fallback affordances for an Assistant surface with none
declared still offered interactive questions. They now match the
backend default of tools only, so no client path advertises a question
tool nobody can answer.
## Why
The in-app Assistant chat window was retired in #3126. Several backend
routes existed only for that window, and Pulse Mobile, the remaining
chat client, never called them. This removes them and the code that only
they reached.
## What changes
- **Routes removed:**
- `GET /api/ai/assistant/surface-tools`
- `POST /api/ai/workflow-prompts/render`
- `POST /api/ai/question/{id}/answer`
- the `/summarize`, `/fork`, `/undo`, `/redo`, `/steer`, `/diff`,
`/revert` and `/unrevert` sub-resources under `/api/ai/sessions/{id}`.
These now return `404` after the existing `ai:chat` scope check.
- **Code removed behind them:**
- the handlers and their `AIService` methods;
- the chat service's fork, undo, redo, steer, summarize, answer-question
and surface-tool methods;
- the session store's fork and turn-undo operations and the persisted
redo stack (`can_redo` leaves session summaries);
- session compaction, the agent loop's steering inbox and
question-answer wait, the `steered` message flag, and the `question` and
`steer_applied` stream events.
- **The question tool is no longer offered.** Interactive chat still
offered `pulse_question`, and Pulse Mobile has no way to answer it, so a
Mobile turn that called it waited until the request ended. Now:
- no profile offers the tool;
- the agent loop answers any call with a refusal telling the model to
ask in its reply text, the same path Patrol runs already took;
- the chat system prompt no longer mentions a clarification tool;
- the agent capabilities manifest's Pulse Assistant surface no longer
declares the `interactiveQuestions` affordance, so the prompt's
affordance text, the fallback governance names, and the generated
`docs/AI.md` and `cmd/pulse-mcp/README.md` surface blocks drop it. The
External agents settings panel stops showing an "Interactive questions"
affordance for Pulse Assistant. The shared `pulse_question` declaration
stays.
- **Docs and contracts:**
- `docs/API.md` lists the remaining routes under "Chat Sessions (Pulse
Mobile)", including the rename route Mobile already used.
- The ai-runtime and api-contracts rules for the removed workflows are
rewritten, and the dependent contracts carry dated notes.
## Unchanged
- `/api/ai/chat` and the session list, create, rename, delete, messages
and abort routes, which are the Pulse Mobile relay route list.
- `/api/ai/status`, which the dev health check uses.
- `/api/ai/workflow-prompts/activity`, which Patrol uses.
- Patrol investigations, provider routing, model-boundary sanitizing and
approvals.
- Old session files: they load as before and drop the redo stack and
`steered` flags on their next write.
## Not in this PR
- The Pulse Assistant surface description in the manifest, which still
calls it a handoff surface for Patrol findings. It now describes Pulse
Mobile chat and can be reworded with the surface contract when Mobile
retires.
- The browser's legacy fallback in
`frontend-modern/src/api/agentCapabilities.ts`. It still gives Pulse
Assistant `interactiveQuestions` when a manifest declares no affordances
at all. A same-version server always declares them, so that path is
unused; it changes with the surface description.
- The dead `highlight.js` chunk rule in `vite.config.ts`.
## Release note
> Removed API routes the retired in-app chat window used: Assistant
surface tools, workflow-prompt rendering, question answers, and the
session summarize, fork, undo, redo, steer, diff, revert and unrevert
routes. Pulse Mobile's chat is unaffected.
The in-app Assistant chat window was retired in #3126. Several backend
routes existed only for it, and Pulse Mobile, the remaining chat client,
never called them: assistant surface-tools, workflow-prompt render,
question answers, and the session summarize, fork, undo, redo, steer,
diff, revert and unrevert sub-resources. This removes them along with
the handlers, service and session-store methods, the persisted redo
stack, session compaction, mid-turn steering and the question-answer
wait behind them.
The question tool needed more than a route removal. Interactive chat
still offered pulse_question, and Pulse Mobile has no way to answer it,
so a Mobile turn that called it waited until the request ended. No
profile offers it now. The agentic loop answers any call with a refusal
that tells the model to ask in its reply text, the path Patrol runs
already took, and the chat prompt drops its clarification-tool advice.
The capabilities manifest's Assistant surface stops declaring
interactive questions, so the generated docs and fallback governance
names follow it.
/api/ai/chat and the session list, create, rename, delete, messages and
abort routes Pulse Mobile uses are unchanged, as is /api/ai/status. Old
session files load as before and drop the redo stack and steered flag
on their next write. docs/API.md lists the remaining chat routes, and
the subsystem contracts record the removal.
## Why
The Patrol weekly digest's **Investigated** line, on the Patrol page and
in the weekly email, was unreliable:
- It counted `investigation_outcome` lifecycle rows and read their
outcome from `metadata["outcome"]`. The only writer of those rows
(`updateInvestigationOutcomeLocked`) writes
`prev_outcome`/`next_outcome`, so every counted row was labelled
`unknown`.
- Those rows are later steps of an earlier run's fix (approved, run,
verified), not investigations. Meanwhile the `investigation_updated`
rows every run writes were ignored, apart from one `LastInvestigatedAt`
fallback count per finding.
- So two runs on one finding counted once, a fix approved after last
week's run counted as this week's investigation, and the email could
print "Investigated: 3" followed by "No issues needed a closer look."
## What changes
- `digestInvestigationRuns` counts one investigation per run that
finished in the window, from the rows the findings store already writes
for every run. That includes a run whose own fix was verified before it
closed, and a run whose status a regression cleared mid-run. A fix that
moves on afterwards relabels its run instead of counting again.
- A run that failed with no outcome is labelled `failed`, the bucket
name telemetry already uses. It sits after `timed_out` in
`PatrolDigestOutcomeOrder`.
- The summary-field fallback is removed. Every run since the v6 release
writes lifecycle rows, and the fallback could count one run in two
adjacent weeks.
- Email: `failed` reads "ended with an error". When the count is above
zero but no outcome in the breakdown has wording, the line shows the
count alone instead of "No issues needed a closer look." or the
watch-only sentence.
- `internal/ai/findings.go` is unchanged; the digest reads the rows the
store writes today.
Known limits: lifecycle rows carry no session ID, so a late end written
by an older run is read as the end of whichever run is open. That's the
store-side stale-write problem, which is being closed separately with
session checks. A run whose rows the 50-row lifecycle has already
dropped is not counted.
## Proof
- `internal/ai/patrol_digest_test.go` drives the store's own writers:
the orchestrator's start and end updates, `UpdateInvestigationOutcome`,
`ReconcileInvestigationOutcome`, `RecoverStuckInvestigation`, and
`Resolve` + `Add` for a mid-run regression. The scenario covers ten
finished runs plus one still running. The old code reports 8
investigations, 3 of them `unknown`.
- `internal/api/contract_test.go` follows the scheduled email's path:
store writers → `AISettingsHandler.BuildPatrolDigest` →
`renderPatrolDigestEmail`. The old code reports two runs as `1 unknown`.
- `internal/api/patrol_digest_email_test.go` checks the `failed` wording
and the non-zero-count guard.
- On pulse-dev, `go vet` passes for `./internal/ai/` and
`./internal/api/`, and the full `./internal/ai/` package and every
`PatrolDigest` test in `./internal/api/` pass. The new tests fail
against the previous digest and email (negative controls via
`-overlay`).
## Related
The Patrol page's This week card gets the same `failed` wording and the
same non-zero-count guard in #3132. Until that lands, a week with only
failed runs reads "No issues needed a closer look." on the card. That's
what those runs already showed as `unknown`, so it isn't a regression.
The weekly digest counted investigation_outcome lifecycle rows and read
their outcome from a metadata["outcome"] key the findings store never
writes, so every one of them showed as "unknown". Those rows are later
steps of an earlier run's fix (approved, run or verified), not new
investigations, while the investigation_updated rows every run writes
were ignored apart from one fallback count per finding. Two runs on one
finding counted once, a fix approved after last week's run counted as
this week's investigation, and the email could print "No issues needed
a closer look." under a non-zero count.
The digest now counts one investigation per run that finished in the
window, from the rows the store already writes for every run, including
a run whose own fix was verified before it closed. A fix that moves on
afterwards relabels its run instead of counting again. A run that failed
with no outcome is "failed", the bucket telemetry already uses; the
email words it "ended with an error" and no longer denies a non-zero
count. The summary-field fallback is removed: every run since the v6
release writes lifecycle rows, and the fallback could count one run in
two adjacent weeks.
The tests drive the findings store's own writers rather than hand-built
rows, which is how the wrong metadata key went unnoticed.
The private investigation orchestrator writes a finding's investigation
fields through orchestratorFindingsAdapter, and every write was applied
unconditionally. A run still in flight when its finding was resolved
and detected again, or when the stuck-run sweep gave it up and a retry
started another run, overwrote the newer state with its own session,
status, outcome and attempts when it finished. A stale "resolved" made
the regressed occurrence terminal, so Patrol stopped investigating it.
Closing writes now go through FindingsStore.CompleteInvestigation,
which records them only while the finding still names the run: the
session must be the finding's current one and the run must not predate
the finding's latest regression. Admission goes through
StartInvestigation, which refuses a run whose session started before
that regression, so a delayed admission cannot install an earlier
occurrence's attempt count or replace a newer run.
Both compare a run's start only with an occurrence start this process
observed: a regression, or the finding's removal by ClearAll or
Cleanup, kept unpersisted with the clock's monotonic reading for a day.
A regression an earlier process recorded predates every run of this
one, so a clock set back across a restart cannot make every new run
look stale. Records and action outcomes keep comparing with
LastRegressionAt, since they also judge sessions of earlier processes.
The adapter takes each run's start from the orchestrator's session
store, so the aicontracts boundary and the enterprise orchestrator are
unchanged.
Main gained the Assistant retirement (#3126), which also appended notes
to the agent-lifecycle, ai-runtime, api-contracts and storage-recovery
contracts. Keep both: main's notes, then this change's.
Patrol investigations inferred their outcome from prose markers, so a
bold or bulleted RESOLVED line became the terminal needs_attention and
the finding was never investigated again. Timeouts clustered on
subscription agents. Patrol's 60s stream idle timeout bounds a whole
buffered CLI turn there, three investigations queued on one serial CLI
slot, and the final summary call was capped at 20s.
- Add patrol_conclude_investigation, an investigation-only,
side-effect-free capture of resolved, cannot_fix or needs_attention.
It is carried on InvestigationRunResult and
OrchestratorInvestigationResult.
- After a conclusion, the next turn is the summary. The final turn
offers only the tools that end a run. A deadline reserve makes the
current turn final before the run times out, and the summary call
may use the time left.
- Give investigation turns on subscription agents a 4-minute turn
budget, a 25-minute default timeout and one investigation at a time.
These limits are republished on config reload without holding the
service lock across the Patrol lock.
- Derive Patrol's outer deadline from the effective timeout. Sessions
now carry their own deadline, and the stuck-run sweep measures a live
run against it, so neither a lowered timeout nor the previous run's
time resets a running investigation.
- Record a run orphaned by a restart as interrupted, not timed out. A
first investigation cut short that way used to stay running for
good, which also blocked a manual re-run. Recovery applies only while
the finding still shows that run, takes a result the run's session
already recorded, and keeps an action outcome such as fix_queued.
- Add unclassified and interrupted outcomes, with lifecycle, telemetry
bucket, digest and record handling.
The Pulse Intelligence overview contract still required docs/AI.md to say
the Assistant is the in-app Pro surface beside pulse-mcp. That paragraph
went with the chat window, so the test now pins what the doc says
instead: Pulse has no in-app chat window, MCP clients connect through
pulse-mcp, and capabilities land in the canonical manifest first.
Pulse Assistant was a chat drawer beside the monitoring UI. Operators who
want to talk to their infrastructure already use Claude Code, Codex, or
another MCP client, and Pulse MCP gives those agents the same governed
tools. Keeping a second, weaker chat in the web app cost a 66 kB chunk,
a launcher on every page, hand-off buttons on alerts, findings, approvals
and drawers, and thousands of lines of OpenCode-parity drawer code, for a
feature the maintainer judged ineffective.
This removes the browser chat and everything that only fed it:
- The drawer, its launcher (desktop and phone), the command-palette
Assistant commands, the Escape and blocking-dialog shell plumbing, and
the chat store, client, dev stream fixture and generated SSE types.
- Every "Open in Assistant", "Explain with Assistant", "Discuss" and
"Ask Assistant" hand-off on alerts, incident timelines, Patrol findings,
approvals, run history, the attention workbench and resource drawers,
with their context models and the chat-only helpers they used.
- The Assistant settings page and its session maintenance section. The
page becomes "External agents" at /settings/pulse-intelligence/agents;
the old /assistant path, ?tab=assistant and system-ai-assistant redirect
there with the hash kept.
Alert Investigate now starts a targeted Patrol run (resource, alert ID and
type) and says where the findings land, and it ignores repeat clicks while
the request is in flight. The run route needs an admin with ai:execute, so
security status gains settingsCapabilities.patrolRun, derived from the same
snapshot as the other admin capabilities but following the route for
org-scoped admins, organization-bound tokens and a cookie beside a token,
and Investigate is offered only where it is true. The Patrol
tool transcript block moves to components/patrol. A small aiAvailability
store holds the session assistantEnabled fact that gates Patrol entry
points and Investigate; it now clears and refreshes on organization switch,
refreshes when organization loading settles on another org and after
provider removal, and ignores answers that arrive after a newer one.
The backend chat service stays: Patrol investigations run on it and
Pulse Mobile chat uses it until 31 March 2027, so /api/ai/chat, the
session endpoints and the stream contract are unchanged. A new contract
snapshot pins the steer_applied event that only the deleted TypeScript
projection used to cover.
highlight.js, used only by the chat's code highlighter, leaves the
dependencies, as does the DOMPurify carve-out that kept its fence hints.
Contracts, registry and readiness proofs drop the retired surfaces; the
ai-runtime contract loses about 1,600 lines of drawer rules while keeping
every backend session, stream and provider rule. The production build
ships about 100 kB less gzipped JavaScript.
A free self-hosted install had no way to buy from inside Pulse. Plans &
Billing was unlisted, the plan page showed only the current plan unless an
upgrade prompt had sent the operator there, and those prompts are hidden on
free installs. Plan-locked lines in Settings said "Higher license plan
required" without saying where the plan lives.
Plans & Billing is now listed for every install that does not hide
commercial surfaces, and its plan page shows a free install the Pro
comparison and its purchase action. Plan-locked lines in Settings link to
it. Proactive surfaces stay quiet: no badges, banners, trial prompts or
upgrade calls to action on monitoring pages, and demo and white-label
runtimes still hide everything commercial.
Purchase start also failed for plain-HTTP installs on a LAN address, the
default homelab setup, because there is no secure address to return the
signed purchase token to. Those installs now get a handoff that finishes
on the website, which shows and emails the activation key; no return token
is minted or sent over plain HTTP.
Every Buy, Upgrade and Renew action opened the Pulse Account portal on cloud.pulserelay.pro, which has returned a Cloudflare 522 since hosted Cloud was hibernated on 2026-06-01, so no one could buy from inside Pulse. The licence server now serves the same handoff-to-checkout page at https://license.pulserelay.pro/buy, using the existing portal handoff and checkout session APIs, so payment still returns to the instance and activates it.
Release preparation #3065 failed on two fixture boundaries: the PTY
could signal Bash before its read setup finished, and a guest-agent HTTP
case could inherit a previous fixture’s cooldown after ephemeral port
reuse. Wait for the actual blocked reader and isolate the registry per
serial HTTP scenario, retaining shared admission across clients and
aliases within each scenario.
Controlled regressions fail before the fixes and pass afterward,
covering premature signal delivery and inherited cooldown state. This
changes test fixtures only and preserves the production installer,
guest-agent safeguards and release qualification controls.
Validation:
- Full config API and Proxmox packages passed on Linux.
- Real BusyBox input and blocked-reader regression passed five
repetitions.
- Guest-agent tests passed with the race detector.
- Status and subsystem registry audits passed.
Refs #3065.
The two CI attempts for release preparation #3065 failed in fixture setup:
the PTY signalled a running Bash reader before input setup completed, and
an HTTP fixture inherited an earlier origin's uncertainty cooldown when
an ephemeral port was reused.
Wait for noncanonical, silent terminal input and a blocked Linux reader
before signalling. Isolate the shared guest-agent registry per serial
HTTP scenario while retaining admission sharing across its clients and
aliases. Move the existing isolation helper to the shared fixture file
and update its callers. Production guard and installer code are unchanged.
Both new regressions reproduce the old failures before the fixture fixes.
Retain those failures and the earlier unsuccessful compile proof. Source
test stabilization is separate from release qualification and publication.
Refs #3065
Keep absent bounds nullable across the SQLite row boundary and apply their
existing value fallback during decoding. Preserve explicit bounds and the
same fallback before SQL and streamed display aggregation.
The full metrics suite passes. Ten alternating diagnostic samples reduce
allocations by 24.59% and bytes by 4.24%; their latency difference remains
inconclusive. Preserve the strict candidate failure and competing-capacity
receipt without claiming a qualified 90 ms result.
Contract-Neutral: Retained chart values, extrema, tier reconciliation, sampling and cache boundaries remain unchanged; only decoding allocation cost and capacity-error reporting change.
Execute the fixed candidate directly at the existing 90 ms budget. Keep
relative comparisons in the paired benchmark job so an invalid baseline
cannot suppress the candidate check.
Retain per-process CPU and measurement I/O counters. Require quiet I/O
before admission while counting the test's own reads in its latency.
Preserve the rejected hosted measurement and inconclusive tier-pruning
experiment as adverse evidence.
Contract-Neutral: Only verification harness and evidence change; chart payloads, retained reconciliation, latency budgets and required source/release gates are unchanged.
Stream ungrouped retained reads in the existing time-major index order,
avoiding temporary metric sorts without changing tier reconciliation or
chart samples. Extend query-plan and filtered-result coverage.
Add an explicit strict mode for the existing 90 ms API budget and an
optional fresh-worker CI proof with fixed executable identities and raw
capacity evidence. Keep ordinary hosted budgets and required gates intact.
Contract-Neutral: Chart payloads, retained-tier reconciliation, sampling, index layout and source/release gates are unchanged; the implementation removes an unnecessary sort and adds controlled verification of existing latency budgets.
Keep monitoring readings, trusted connections and recovery guidance
reliable when observations are stale, guest responses are incomplete or
agent recovery is interrupted.
Qualified source: `b6e56c5baa2ad995264faa80877543300bc10273`. All nine
[required source
checks](https://github.com/rcourtman/Pulse/pull/2888/checks), the
[native
matrix](https://github.com/rcourtman/Pulse/actions/runs/39148879516) and
[Core E2E](https://github.com/rcourtman/Pulse/actions/runs/39148879470)
passed. Intel accounted for all 545 installer roots, with 532 passes, 13
explicit platform skips and no failures. Its remaining installer phase
passed in 18m48s, followed by native agent build and execution. The
matrix covers Linux x64/ARM64, macOS Intel/ARM64 and Windows x64
execution plus the FreeBSD cross-build contract.
- Preserve reading provenance, measured zero, stored History and VM/LXC
memory interpretation. Selected-agent staleness cannot renew a retained
observation or invalidate independent provider readings.
- Reject incomplete or foreign guest responses and retain saved
certificate trust, scoped access, private installer input, native NAS
teardown and bounded recovery guidance.
- Keep initial Proxmox login within its caller and total client timeout
while accepted-session renewal remains shared. Preserve the PBS owner's
context outcome when cancellation races a response, prevent a cancelled
fallback and reject late session publication.
- Assert troubleshooting workflows and destinations semantically. Run
every discovered Python safety control and native Go root independently
with existing fixture, operation, root and job deadlines. Script-smoke
and native verification share one control suite, including real
compiled-Go failure, exit-zero, timeout and exact-selector controls.
- Integrate current main while retaining selected-agent freshness, guest
memory interpretation, native GNU prerequisites, API concurrency
controls and both parents' subsystem contracts.
Worker proof passed 113 focused frontend cases, API and guest-transport
race checks, all 59 Python fixture controls, production frontend build
and full frontend lint/types. The exhaustive worker installer inventory
accounted for 551 roots: 535 passed and 16 explicitly skipped for
platform reasons. The eight consolidated runner controls passed through
the script-smoke harness and native Go wrapper. Authentication proof
passed all 340 Proxmox tests, 108 PBS tests and the affected monitoring
cancellation controls under the race detector. The new PBS test first
failed all ten JSON/form cases on unchanged runtime, then passed with
the correction. Status, ownership and normal commit checks passed.
Browser proof uses production components and CSS with synthetic
observations and endpoints. Drawer History passed at 1280×900, 1024×768
and 390×844. Private installation passed at 1280×900 and 390×900 in both
themes. It covers stale/current/unknown readings, provider caveats,
stored History and keyboard inspection, wrapped signing keys, token
copy/dismissal and Last seen advancing without another lookup. Final
screenshots were inspected, the source-bound receipt covers the combined
runtime files, and owned browsers and servers were closed.
Retain the [previous Frontend
assertions](https://github.com/rcourtman/Pulse/actions/runs/38660106123/job/116179237804),
[Intel
timeouts](https://github.com/rcourtman/Pulse/actions/runs/39138980077/job/117709536519),
[script-smoke integration
failure](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117728549560)
and [PBS cancellation
race](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117729442632)
with their diagnosed causes and corrected proof. The final matrix above
qualifies the combined corrections.
The [earlier advisory benchmark
comparison](https://github.com/rcourtman/Pulse/actions/runs/39138980114/job/117709718583)
reported three legacy history-filter regressions. A worker repeat
reproduced one. Longer controls showed wide timing variation even
between byte-identical binaries. The [45a82ea03c
comparison](https://github.com/rcourtman/Pulse/actions/runs/39145479921/job/117728549510)
passed against a newer baseline. The [final
comparison](https://github.com/rcourtman/Pulse/actions/runs/39148879492/job/117738375776)
completed execution but failed on test-only empty/legacy filtering,
2.945 to 3.263ns, +10.82% (p<.001, ten samples). Production empty
filtering improved 16.70%, interleaved filtering rose 2.96%, and history
allocation counts were unchanged. The filter and benchmark sources are
byte-identical across the final comparison. This does not establish a
cause or performance closure. Artifact 12145720151 retains full paired
samples, comparison and metadata. The overall Build and Test workflow is
red from this advisory job while all nine protected required checks
pass. The existing L1 reliability follow-up owns repeatability and
attribution. No benchmark threshold, protected check or release gate
changed.
Main's benchmark/SLO test and documentation changes in PR3053 compose
without conflict. The candidate benchmark artifact records synthetic
merge c21333c676ac004935c9683027490203d8f869ea with tree
fa059ae37b, combining this head and main
52b40fd611. A bounded worker composition
proof also executed all three chart benchmark cases under the race
detector with source hashes unchanged. Its SLO test explicitly skipped
under the race detector, and one-iteration execution supplies no
performance comparison.
This proposal qualifies source delivery. Release inclusion, installed
upgrade/recovery, guest thaw, fresh writes on every backed-up filesystem
and workload liveness remain separate outcomes. Reliability keeps its
existing priority. Recovery/change feature expansion stays deferred.
Related to #1099, #2439, #2619, #2822, #2897, #2757, #2511, #2513,
#1788, #2755 and #2140.
The chart benchmark warmed its response cache and then timed a fresh
payload-retaining recorder. That hid store-backed work behind test buffer
allocation and made the regression signal ambiguous.
Measure cached handler work, fresh-service computation and real loopback
HTTP transfer separately. Wait for store startup maintenance, preserve the
original failed comparison and record the paired diagnostic limits. Make
the adjacent SLO exercise uncached chart generation at its existing budget.
Refs #3052
Preserve selected memory freshness and the VM/LXC interpretation.
Retain both native tool contracts and current API concurrency controls.
Qualify this source separately from release and installed recovery.
Contract-Neutral: Integrate already-governed parents without a new contract change. Incoming verification files already match main. Fresh integration tests and source-bound browser proof cover the combined memory and installer surfaces.
The usage observer records aggregate outcomes through the existing settings
store without implementing a paid renderer or entitlement decision. Declare
its exact shared-adapter path so the repository audit can distinguish it
from new private reporting implementation.
Exercise the real audit against sibling paths and private licensing imports
to retain those restrictions for the observer.
Record successful response writes, scheduled generation, file saves and SMTP
outcomes in a durable content-free tally. Distinct active days include manual
exports, while saved-schedule periods deduplicate retries and monthly reuse.
Publish bounded scalars through the existing opt-out-controlled heartbeat and
report missing coverage explicitly. Preserve pricing and customer entitlements.
Client backup reports can now be scheduled weekly or monthly as PDF or
CSV, scoped to current VMs and containers by resource or client tag.
Reports show recorded outcomes, daily failure rates, last retained
success, failure details and missing evidence. Weekly reports cover
seven completed local calendar days. Monthly reports cover the previous
calendar month.
Backup executions and observation coverage are retained in each
organization's recovery store. Shrinking provider task inventories and
restarts preserve recorded failures. Unavailable history, permission
limits, collection gaps and incomplete multi-guest attribution remain
explicit. A successful task does not certify that a backup is retained
or restorable. This reports Proxmox VE backup jobs, including jobs
writing to PBS.
The existing scheduler, branding, delivery checks, bounded retries and
deliberate uncertain-delivery recovery are reused. The public adapter
supplies scoped evidence to a private PDF/CSV renderer. Pricing, feature
entitlements and retention remain unchanged.
Validation:
- Frontend: 136 focused tests, full type check, lint, production build
and bundle budgets.
- Race checks cover recovery persistence, collector quality, reporting,
organization isolation and runtime registration. Private rendering and
runtime build checks cover the composed Pro implementation.
- Live runtime checks on `/settings/support/reporting` and
`/alerts/notifications` at 1440x1000, 820x844 and 390x844 in light and
dark themes, with actual pixel review. The 61-workload selection remains
bounded and keyboard usable. All five generated PDF pages were
inspected.
- Real client-scoped CSV and MIME attachment contents, retained failure
after an empty provider inventory and restart, retry after restart,
permanent rejection, uncertain acknowledgement and deliberate recovery.
An unmatched client scope produces missing evidence with blank counts.
Retain backup execution outcomes and collection coverage in each
organization's recovery store so provider inventory changes and restarts
cannot erase report evidence. Keep missing and incomplete history distinct
from an observed period with no attempts.
Reuse existing report scopes, calendar schedules, branding and delivery
recovery for client PDF and CSV reports. Register the paid renderer through
the existing private runtime boundary. Pricing and grants are unchanged.
The organization-owner fix judged browser sessions only. API tokens on
instance routes were still checked for scope alone, and the tenant
middleware resolves an org-bound token to its own organization without
the caller naming one. A token bound to a tenant organization and
holding settings:write therefore reached system settings, SSO
providers, token management, relay, log level, updates and the other
instance surfaces.
ensureAdminSession, behind RequireAdmin, and ensureSettingsScope now
refuse a token unless it acts for the instance: not bound to any
organization other than the default one, with OrgIDs taking precedence
over OrgID. The route's scope still applies. The org variants
(RequireOrgAdmin, ensureOrgAdminSession, ensureOrgSettingsWriteScope)
keep judging tokens by scope alone. tokenHasInstanceWideAuthority is
now the same binding rule plus full-access scope.
The test notification broadcasts only to the request organization, so
it moves to RequireOrgAdmin. Its tenant delivery test already relied on
an org-bound token sending it, and that organization's managers may now
send it from a session too.
No path an organization manager can reach mints a settings scope: relay
mobile, install, deploy and action-runner tokens carry relay, agent and
report scopes only, and token management is instance administration.
The API reference and the api-contracts, security-privacy,
agent-lifecycle and storage-recovery contracts record the token rule.
VM inventory was available only as an on-demand download. It can now be
scheduled weekly or monthly as CSV, emailed or saved on the Pulse
server. Each attempt exports all current VMs in the organization,
including newly discovered VMs, through the existing exporter.
Historical performance-report limits and windows remain separate.
The schedule uses existing delivery configuration checks, persisted
success/failure/uncertain states, bounded retries and deliberate
recovery controls. Pricing, feature gates, retention entitlements and
existing customer grants are unchanged.
Live checks also reproduced two unattended inventory defects: reporting
retained a stopped monitor after configuration reload, and canonical
snapshot reads relied on dashboard reads to publish polled VM inventory.
Reporting now receives reload replacements through an atomic manager
reference. Canonical snapshot reads reuse the existing publication
boundary, including its freshness fast path, provider/mode fences and
nonblocking warm reads.
Validation:
- 105 focused frontend tests; full type check, lint, production build
and bundle budgets.
- Race checks for reporting, inventory, reload, publication and
organization isolation; clean private runtime build.
- Current runtime browser checks on `/settings/support/reporting` and
`/alerts/notifications` at 1440×1000, 820×844 and 390×844, with actual
pixel review. The first inventory read exports 61 normally polled VMs;
real MIME CSV matches the download; a newly discovered 62nd VM enters
the next export.
- Local SMTP acceptance, 451 retry across a real restart, permanent
rejection, uncertain acknowledgement and deliberate recovery; disk
delivery, edit/reload, disable, details and delete.
- Staged status, registry, contract, completion and exact-content
browser guards pass. Regression proof is registered against the specific
runtime paths it covers.
## Why
Report generation read every report's metrics from the default
organization's store, whatever organization the request came from. The
engine was wired in `pkg/server/server.go` to the default monitor's
store, and `ReportEngine.queryMetrics` queried it with the caller's
resource ID. A tenant request for a resource outside its own inventory
therefore read the default organization's history, and even a tenant's
own resources were reported from the wrong store. Pulse Assistant's
`pulse_summarize` used the same engine, so it read the same store. Other
tenants' stores were not reachable.
#3033 closed reporting to organization owners until this was fixed.
Providers build reports inside client workspaces (4d47504331), so
reporting reopens to them here.
## What changes
- **Engine** (`pkg/reporting`): `MetricReportRequest` and
`MultiReportRequest` carry `OrgID`. With
`EngineConfig.MetricsStoreForOrg` set, the engine resolves that
organization's own store and refuses a request without an organization
(`ErrReportOrgRequired`). A multi-resource report applies its own
`OrgID` to every resource.
- **Server wiring** (`pkg/server/reporting_engine.go`): the default
engine resolves stores per request through
`MultiTenantMonitor.GetMonitor(orgID)`, so it follows monitor reloads
and never falls back to the default organization.
- **Handlers**: single and multi-resource reports must name a resource
in the request organization's inventory: a unified resource ID, or a
legacy node/VM/container ID. Anything else returns 404
`resource_not_found`, or 503 `inventory_unavailable` when the inventory
can't be read.
- **Schedules**: a run resolves explicit resources and tags against its
owning organization's inventory, which the per-org schedule store and
scheduler context already supply. Resources the organization no longer
holds are skipped and logged; a scope with none left fails the run.
- **Pulse Assistant**: `pulse_summarize` passes its executor's
organization.
- **Access**: every `/api/admin/reports/*` route (catalog, generate,
generate-multi, VM inventory export, schedules) uses
`ensureOrgAdminSession`, and `reportingRead` is offered to organization
managers again (it still requires `read:nodes`).
- **Contracts**: rule 35 in api-contracts, the superseding note in
storage-recovery, plus ai-runtime, security-privacy and agent-lifecycle.
Behaviour change: a report can no longer be generated for a resource
that has left the organization's inventory, even if its retained history
is still in the store.
## Proof
- `internal/api/reporting_org_scope_test.go`: org B gets 404 for org A's
resource, by legacy and unified ID, through GET, POST and
generate-multi, and receives only org B's values even though org A's
store holds points under the same ID.
- `TestReportingScheduledRunReadsOnlyItsOwningOrganizationsMetrics`: an
org B schedule naming org A's resources reports only org B's resource
from org B's store; a scope of only foreign resources fails.
- `pkg/reporting/engine_org_scope_test.go` and
`TestReportEngineReadsOnlyTheRequestOrganizationsMetrics`: per-org
resolution and refusal without an org, across Generate, GenerateMulti,
NarrativeFor, FleetNarrativeFor and MetricEvidenceFor.
- `pkg/server/reporting_engine_test.go`: the server's engine resolves
the request org's monitor store, and refuses an empty, unprovisioned or
unavailable org.
- `TestSummarizeToolReadsOnlyItsOrganizationsMetrics`;
`TestSecurityStatusOrgBoundSettingsFollowOrgManagementRule` now holds
`reportingRead` in the org-bound set.
- On pulse-dev: `internal/api` (`-timeout 50m`), `pkg/reporting`,
`pkg/server`, `internal/ai/tools` and `internal/repoctl` pass. The
reporting and schedule tests also pass under `-race`.
## Governance
The canonical completion guard maps `pkg/server/server.go` to
security-privacy's "privacy docs and telemetry" verification policy,
which accepted only telemetry tests as proof. The `server.go` change
here is the report engine's per-organization store wiring, a
tenant-isolation boundary. `registry.json` now gives
`pkg/server/server.go` its own security-privacy path policy,
`server-runtime-wiring`: it keeps every proof the telemetry policy
accepted and also accepts `pkg/server/reporting_engine_test.go`. The
other telemetry files keep their existing policy, so they can't cite the
wiring test. That test proves the server's engine resolves the request
organization's monitor store and refuses an empty, unprovisioned or
unavailable organization. No telemetry test was edited to satisfy the
guard. The commit passed the full pre-commit hook.
## Coordination note
`vm-inventory-schedules-20261017` currently holds workspace claims on
`internal/api/report_schedules.go`,
`internal/api/reporting_runtime_snapshot_test.go`, `api-contracts.md`,
`agent-lifecycle.md` and `storage-recovery.md`, with uncommitted work in
its own checkout. This PR's hunks in those files
(`resolveReportScheduleResources`, a new helper appended to the snapshot
test, rule 35 and the `reportingRead` mount line in api-contracts, one
paragraph each in agent-lifecycle and storage-recovery) don't overlap
that work. This commit was built in a separate checkout, not in that
agent's working tree.
Reuse the canonical inventory exporter and report delivery workflow so
operators can email or save a current organization-wide inventory weekly
or monthly. Keep delivery checks, bounded retries and recovery states
shared, with existing pricing and customer entitlements unchanged.
Refresh the shared inventory for unattended reads and rebind reporting
after monitor reload so schedules use the current infrastructure state.
The report engine read every report's metrics from the default
organization's store, whatever organization the request came from. A
tenant request for a resource ID outside its own inventory fell through
to that store with the caller's ID, so one organization could read the
default organization's history, and even a tenant's own resources were
reported from the wrong store. Reporting was closed to org owners until
this was fixed.
The engine now resolves the metrics store of each request's OrgID
through the multi-tenant monitor and refuses a request that names none,
so no report path can fall back to the default store. Report handlers
reject resources outside the requesting organization's inventory,
scheduled runs resolve their scope against their owning organization's
inventory and skip resources it no longer holds, and Pulse Assistant's
pulse_summarize reads its executor's organization.
With every report read scoped, the reporting routes move to
ensureOrgAdminSession and reportingRead is offered to org managers
again, so providers can build reports inside client workspaces.
The report engine wiring lives in pkg/server/server.go, which the
completion guard only let telemetry tests prove. server.go now has its
own security-privacy path policy that keeps those proofs and also
accepts pkg/server/reporting_engine_test.go; the other telemetry files
keep their existing policy unchanged.
The agent-lifecycle and storage-recovery contracts now state that
registration, unregistration, node change and discovery events stay
with the organization that owns them. The tenant delivery tests move
into the proof files the canonical completion guard names for those
paths, and a route-level test covers the test-notification route's
authorization, method, payload and tenant delivery.
Scheduled reports could appear successful when email was unconfigured,
and a temporary delivery failure had no durable retry or clear recovery
path. Reports now checks email configuration, records
running/saved/accepted/failed/unconfirmed outcomes, and exposes
actionable errors and Run/Retry controls at desktop and mobile widths.
Temporary mail failures receive at most three attempts with one- and
five-minute delays. Retries survive restart and preserve resource-report
windows. Lost completed-message acknowledgements and interrupted email
runs remain unconfirmed until a deliberate retry, preventing automatic
duplicate sends. Edits, disablement and deletion cancel pending retries.
Every attempt rechecks the existing entitlement.
This implements the scheduled-report delivery slice of L6. Pricing,
feature gates, retention limits and existing customer entitlements are
unchanged. SMTP acceptance is distinguished from inbox receipt.
The schedule editor loads when creating or editing a schedule, with a
visible loading state and initial keyboard focus. The reporting screen
stays within its existing bundle budget without raising limits.
Validation:
- Full configuration and notifications Go suites, reporting API race
tests.
- 301 targeted frontend and architecture tests, TypeScript, full
frontend lint and production frontend/private-runtime builds.
- Existing bundle-size, total JS/CSS and preload checks passed.
ReportingPanel is 10.94 kB gzip against its 11.63 kB limit.
- Playwright on `/settings/support/reporting` and
`/alerts/notifications` at 1440, 820 and 390 pixels: configuration
failure, disabled draft, real CSV attachment, disk PDF, persisted retry
across a process restart, permanent rejection, uncertain
acknowledgement, deliberate retry and delete/disable recovery.
- Controlled-response browser matrix for running, loading/error,
polling, keyboard focus, details and responsive layouts. Final pixels
reviewed separately from delivery assertions.
- Release-control status, ownership, contract, completion and browser
guards.
Browser delivery uses synthetic infrastructure, a disposable signed Pro
grant and local SMTP only. This source change does not publish a release
or alter customer installations.
rcourtman