Free self-hosted sessions now see reactive commercial surfaces: paid-feature
nav items stay visible with panel-owned inline gates (Relay precedent extended
to reporting, roles, users, audit log, audit webhooks), gate CTAs render, and
Plans & Billing is discoverable and carries an MSP/provider section with the
free 60-day evaluation. Adds a one-shot business-estate card for authenticated
free sessions whose monitored estate crosses business-scale thresholds
(>=5 PVE nodes, >=10 Docker hosts, or >=3 VMware hosts); the flag rides
sessionCapabilities only, never the pre-auth presentation policy, so estate
size cannot leak to anonymous visitors. Demo mode and white-label runtimes
(including MSP tenant containers) keep full commercial suppression; the
multi_tenant organization nav items deliberately stay hidden.
Decision record supersedes the 2026-04-25 opt-in record:
docs/release-control/v6/internal/records/self-hosted-commercial-surfaces-revision-2026-08-07.md
RA5 summary, contract pins, nav integration pins, and the RA5 browser spec
are updated to the revised invariant in the same commit.
The server now echoes its version on unified-agent report acks, and the
agent nudges its auto-updater the moment an ack carries a newer version.
After a server upgrade, agents converge within one report interval
instead of waiting out the hourly update check, so the "older Pulse
agent" notice self-resolves in seconds once upgraded agents report in.
The hourly loop stays as the retry and backstop path. Nudges dedupe per
server version, refuse downgrades, skip disabled and development-mode
updaters, and never fire from observer destination acks — only the
authoritative server may steer an agent's updater, and a nudged check
re-validates against the server and runs the existing checksum and
self-test pipeline before swapping binaries. Agents deployed before
this change still converge on their old hourly cadence once; every
upgrade after that lands within a report cycle.
Contract deltas recorded in agent-lifecycle and api-contracts, with
boundary notes in security-privacy (no update authority in the echo),
performance-and-scalability (no steady-state work), and
storage-recovery (nothing persisted).
The PVE and PBS branches of handleCanonicalAutoRegister each carried a
verbatim copy of the identity-matching ladder — exact host, then TLS
fingerprint conflict rejection, resolved host identity, then DHCP
continuity by node name plus token. golangci-lint's dupl flagged the pair.
Extract one matcher, findCanonicalAutoRegisterMatch, over a
canonicalAutoRegisterCandidate identity view that both instance types
project onto. Behavior is unchanged: the same ladder in the same order,
the same log messages and fields, and the same (index, preserveHost)
result, with the previous "break with preserveHost still false" paths
becoming explicit returns.
golangci-lint run ./... green; full go test ./internal/api/ green.
Contract-Neutral: dupl-only refactor of handleCanonicalAutoRegister: PVE/PBS identity-matching ladder hoisted into one helper, zero public-contract or behavioral delta
Alert config grows one Overrides entry per toggled resource, so the
64KB cap on PUT /api/alerts/config rejected saves from instances with
a few hundred disabled containers with 'http: request body too large'.
The 32KB cap on bulk acknowledge/clear failed ack-all during large
alert floods, the exact situation it exists for. Intent policies carry
per-resource rules with the same scaling shape.
All four now share a 1MB bound, which still caps memory per request
but no longer rejects legitimate fleet-sized payloads. Endpoint tests
pin a >64KB config save and a >32KB bulk ack at 200.
Refs #1601
Contract-Neutral: raise alert config and bulk ack request body caps; no payload shape or field changes
/download/pulse-agent served whatever agent artifact sat on disk with no
relation to the running build. Local agent binaries are build outputs that
nothing refreshes on their own, so they go stale silently: a dev backend was
found serving a four-week-old v6.0.5 agent while reporting 6.2.0-rc.8.
Staleness is not cosmetic. The installer renders its service wrapper from the
server's current template, so an agent predating a flag that template now
passes exits immediately with "flag provided but not defined" and crash-loops
under its watchdog. That is how a real host lost its agent: the version
mismatch was reported only as an installer warning, after download, easy to
read as noise.
Validation already scanned the binary for its report-contract endpoints, so
the version check joins that same single pass and rejects a binary that does
not carry this server's agent version. Refusal is loud where the old warning
was quiet: a dev server answers 404 naming the stale path and the build
command, and a published release falls through to the existing release-asset
proxy and fetches the matching version, which makes production self-healing
rather than silently downgraded.
The expected version resolves through updates.GetCurrentVersion rather than
the compiled-in serverVersion. The first cut of this guard used serverVersion
and was inert on exactly the builds that need it: the enterprise binary
compiles in "dev-pro", no version parser accepts it, and the check disabled
itself. It passed its unit tests and still served the stale binary; only
replaying the real v6.0.5 artifact through the running server exposed it.
"dev-pro" is now pinned in the version table with that reasoning attached.
The shared download-test fixture built a binary carrying the report endpoint
but no version string, which a real agent always has, so it now stamps the
expected version. Verified non-vacuous in both directions: those tests fail
with the guard active and the thin fixture, and pass with a faithful one.
Contract-Neutral: storage-recovery is pulled in only by the broad internal/api/ Extension Points prefix and this change does not move that boundary: it constrains which agent binary /download/pulse-agent serves, touching no storage provider, backup target, recovery repository or protected-workload evidence. The agent-lifecycle, deployment-installability and api-contracts deltas staged here cover every boundary the change actually moves.
Mock mode suspends pull-based collection outright, but push-based agent
reports were never given the same treatment, so a real machine still landed
in monitor state while the unified read path substituted the mock snapshot
over the top. The hosts were hidden, everything downstream was not: a real
Unraid box raised a live storage-topology alert next to fixture data, and
its identity persisted through host continuity.
Three vectors, each closed at its source.
Agent ingest now drops real reports while mock mode is on. ApplyHostReport,
ApplyDockerReport and ApplyKubernetesReport acknowledge the report with the
reporting agent's own identity and touch no state, so nothing raises alerts,
persists continuity, records metrics or feeds the online/offline sweep. The
acknowledgement stays a success so a real agent does not read a demo server
as an outage and retry-storm it.
recentStandaloneHostContinuityEntries returns nothing in mock mode. Those
entries are written to disk from real reports and outlive the toggle, and
every consumer injects them after the read path has already substituted the
mock snapshot, so a machine that reported before mock mode was enabled came
back by its real hostname. There is no real-polling exception here: agent
ingest is not gated on PULSE_MOCK_KEEP_REAL_POLLING and the read state is
mock either way.
Active-alert restore is now opt-out, and mock mode opts out. SetMockMode
already clears active alerts when the toggle flips, but a process booting
with mock mode already enabled never ran that path and restored real alerts
from active-alerts.json.
TestHostedTenantAgentInstallTokenCannotReportToOtherTenant used mock mode as
scaffolding. Under the ingest guard both tenants would be empty and its
isolation assertion would pass without exercising the boundary, so it now
runs in real mode. Every new test pairs the mock assertion with a real-mode
one for the same reason.
Mock mode never initialises real PVE/PBS/PMG clients and does not run the
platform pollers, so every configured real source sat in the connections
ledger at "awaiting first poll" forever. That published real connection
names and addresses through /api/connections while the rest of the payload
was authored fixtures, and surfaced them on monitoring copy: the Proxmox
workloads empty state rendered "Collection pending: minipc" next to three
mock nodes.
/api/config/nodes already substitutes mock entries wholesale in mock mode
and rejects node mutations outright, so the ledger was the one surface that
had not been brought in line. Move the mock-mode input shaping into
applyMockLedgerInputs in platform_mock_connections.go, which already owns
the mock vSphere, TrueNAS, and availability ledger fixtures, and drop the
config and persistence derived sources there. PULSE_MOCK_KEEP_REAL_POLLING
keeps the previous behaviour, since those sources do collect under it.
Proof is at the payload level rather than the aggregator inputs: the new
handler test asserts the served /api/connections body contains no real
source by name or address, and was verified red before the fix.
Contract-Neutral: agent-lifecycle is named only by the broad internal/api/ Extension Points prefix and this change does not move that boundary: agent rows come from the monitor hosts snapshot, which is untouched. The api-contracts and storage-recovery deltas cover the boundaries actually moved. No payload field was added or changed, so the backend-API-payload proof list does not apply; the handler-level payload proof is TestConnectionsHandleListDropsRealSourcesInMockMode in internal/api/connections_handlers_mock_test.go, verified red without the fix.
ResourceHandlers.getStore opens a SQLite handle per org and caches it for the
process lifetime, and nothing ever closed them. CleanupTenant already released
patrol, AI, RBAC, license and monitor-adapter state for a deleted org but left
the resource store open, so an offboarded tenant kept its file descriptors and
its unified_resources.db-wal/-shm files alive, and its directory could not be
fully removed.
CloseTenantStore releases and evicts one org's store and is now called from
CleanupTenant alongside the other per-tenant teardown. CloseStores releases all
of them, exposed as Router.ShutdownResourceStores next to the existing
Shutdown*/Stop* helpers.
Found while investigating the flaky hosted-tenant test. It is not what made that
test flaky, which was a detached guest-metadata write fixed separately, but the
leak is real on its own: verified by observing that -wal and -shm sidecars, which
exist only while a connection is open, survived the test before this change and
do not after it.
Fixes#1681Fixes#1682Fixes#1683
Contract-Neutral: Notification grouping initialization and alert-config propagation do not alter the broadly referenced agent-lifecycle or storage-recovery contracts; primary alerts, notifications, API, and monitoring contracts and regression proofs are updated.
Every list-shaped resources request deep-cloned the whole registry:
HandleListResources via ListForPresentation, the storage summary and
incidents handlers and the k8s namespaces handler via List/ListByType,
and the stats handler once more. With the frontend polling three pages
plus summaries every few seconds, the same unchanged world was cloned
dozens of times between snapshot updates.
Cache the raw and presentation lists on the existing per-generation
registry cache entry (same invalidation: entries rebuild when the seed
lastUpdate moves) and hand requests a flat top-level copy instead. Every
decorator in the request pipeline was audited to write only top-level
fields on request-owned elements: action availability, discovery
targets and readiness, metrics targets, canonical metadata refresh, and
contract types all assign freshly built values. The one nested writer —
the PMG list prune, which cleared relay domains through the shared
pointer — now clones the PMG struct before clearing, with a regression
test pinning both the clone-on-write and the cache's immunity to
request decoration. Read-only consumers (stats aggregation, storage
filtering, namespace counting) use the shared list with no copy at all.
Single-resource lookups (presentationResourceByReference and its
callers) still deep-clone per lookup; they are cold paths and stay
as-is.
Contract-Neutral: per-generation shared resource lists: response bytes pinned by contract tests, no payload delta
Schema v6 shipped audit_logging_persistent and audit_events_30d as Pro adoption
signals. Neither discriminated. pkg/server installs the SQLite audit logger on
every install for defense in depth and gates only the read/export endpoints, so
the boolean was true on all 8 installs that had taken rc.8 and 0 rows in the
retained table have ever had it false. The event count measured that background
write volume: three of those eight unlicensed community installs were pegged at
the receiver's 100000 clamp ceiling, with the rest between 4863 and 67509.
Schema v7 replaces both with audit_reads_30d, a count of requests that cleared
the license gate on an audit read or export surface. A read requires a human
action, so unlike store presence or write volume it cannot settle into a
constant. The recorder is wrapped INSIDE RequireLicenseFeature so unentitled
requests never count, and the persisted marker carries a timestamp and a coarse
activity class from a fixed allowlist. Query filters, actors, ranges, and every
audit row read stay on the install.
The retired columns are left in the live database. They hold real rc.8 rows and
migrations only add, so dropping them would be a pointless risk; nothing writes
them once the receiver struct loses the fields.
Adds the guard this class needed. LicensedFeatureAdoptionFields registers every
field that exists to measure licensed-feature adoption, and
TestLicensedFeatureAdoptionFieldsDiscriminate builds an unused install through
the real production snapshot paths, installs a real SQLite audit logger exactly
as pkg/server does, records a baseline audit event, and fails if any registered
field is non-zero. Pinning a console logger there would have made the guard pass
while the payload lied, so it deliberately does not. The guard was verified by
reintroducing the v6 sourcing and confirming it fails with the field named.
A companion test pins the three retired fields so they cannot return under
their old names.
This is the third instance of one bug class. v6 removed
pulse_intelligence_patrol_autofixes_30d, hardcoded to zero with no increment
site, and then introduced two fields that were constant in the other direction.
Three occurrences is a guard, not a habit.
Verified end to end on a running unlicensed install: the payload that reported
audit_logging_persistent true under v6 now reports audit_reads_30d 0, and
seeding two in-window reads, one outside the window, and one with an invalid
activity class yields 2.
Six of the eight Pro-exclusive features had no telemetry field at all, so
there was no way to answer whether RBAC, audit logging, scheduled reporting,
agent profiles, alert-triggered AI, or Kubernetes AI were being used by the
installs paying for them. Schema v6 adds nine content-free adoption signals:
alert_ai_enabled AIConfig.IsAlertTriggeredAnalysisEnabled()
rbac_custom_roles non-built-in roles, per org
rbac_user_assignments user-to-role assignments, per org
audit_logging_persistent a persistent audit store is active, not console
audit_events_30d audit events retained inside the window
report_schedules configured scheduled reports
report_schedules_enabled scheduled reports switched on
report_schedules_run_30d schedules whose last run falls inside the window
agent_profiles configured agent profiles
Counts only. Role names, permissions, usernames, schedule names, delivery
recipients, report scope, profile names, and every audit event field stay on
the install. kubernetes_ai needs no field of its own: it is derivable at read
time from alert_ai_enabled combined with the existing kubernetes_clusters
count, and a dedicated field would be redundant.
Config-sourced signals are read through applyLicensedFeatureConfigSnapshot;
RBAC and audit live behind the router and are read through
Router.ApplyLicensedFeatureTelemetrySnapshot. The RBAC read goes through a new
TenantRBACProvider.PeekManager so a background telemetry read can never
provision an RBAC store for an org that has never used RBAC.
Also removes pulse_intelligence_patrol_autofixes_30d and the AutoFixCount
field behind it. patrol_run.go hardcoded AutoFixCount to 0 and no increment
site existed anywhere in the tree, so the counter was zero in all 233,364
retained production pings. That was a wiring bug, not evidence that nobody
uses Patrol fixes; governed fixes are delivered through the approved-action
pipeline, which is already instrumented. The field was plumbed through run
records, history persistence, the Assistant handoff, and telemetry while being
structurally incapable of holding a non-zero value.
Verified end to end against a running install rather than only in unit tests,
which is precisely the check the autofix counter never had: seeding three
report schedules (two enabled, one last run inside the window) and two agent
profiles produced report_schedules 3, report_schedules_enabled 2,
report_schedules_run_30d 1, agent_profiles 2 in the Settings telemetry
preview, and signing in moved audit_events_30d to 1.
The private receiver landed first in pulse-pro 78ff7dd so the new fields are
accepted on arrival.
canAccessPlatformAdminSurface publishes billingAdmin for any instance
administrator. RequirePlatformAdmin compared the session user against
cfg.AuthUser alone, so on an instance whose only administrators are SSO
principals the UI offered the surface and the route refused it. Same
capability against enforcement split as 28fd2d1c1, on the hosted routes.
The session branch now uses sessionUserCarriesAdminPrivileges, which is what
the capability already resolves to.
A straight swap would have been worse than the bug. That helper treats any SSO
principal as an administrator when no local admin is configured, and a hosted
control plane authenticates its tenants by SSO, so on a control plane with no
local admin every tenant would have become a platform admin. The session
branch is therefore also gated on the request not being org-scoped, matching
what ensureAdminSession and the security status snapshot already do. Removing
that gate lets an org-scoped tenant session reach the surface with a 200,
which the parity test pins.
The org-scope test itself was written inline in two places and is now one
helper, sessionIsOrgScoped, so the instance-versus-tenant boundary has a
single definition rather than a copy per caller.
RequireOrgOwnerOrPlatformAdmin is untouched. It has no session branch in its
platform-admin switch by design and requires org ownership instead.
Contract-Neutral: behavioral fix on existing routes, no request or response shape change; platform admin route aligned with the billingAdmin capability it already publishes
On the OIDC-only pattern there is no local admin, so SSO principals are the
only administrators the instance has. ensureAdminSession already knows that:
sessionUserCarriesAdminPrivileges admits the configured admin, anyone holding
an RBAC admin grant, and any SSO principal when no local admin is configured.
Three guards did not use it. They compared the session username against
cfg.AuthUser directly, which on those instances is empty, so they could admit
nobody at all. The same operator was admitted by the settings routes and
refused by discovery, by public URL capture, and by config export and import,
which is three answers to one question.
Verified against an unlicensed OIDC-only router before and after. Before,
sessionUserCarriesAdminPrivileges returned true for the SSO owner while
canCapturePublicURL and discovery isAdminRequest returned false and
/api/config/export returned 403. After, all four agree.
This does not widen anything. On an instance that does configure a local
admin, an unrelated SSO principal is still not an administrator and every one
of these guards still refuses them, which the parity test pins in both
directions.
RequirePlatformAdmin is deliberately left alone. Its stricter session rule is
documented as intentional for the hosted control plane rather than an
oversight, and loosening it is a different decision from this one.
Contract-Neutral: behavioral fix on existing routes, no request or response shape change; three guards switched to the canonical session-admin helper they should already have used
handleChangePassword refuses a non-admin proxy user outright. The session
branch of that rule was never written, so any authenticated session reached
the current-password comparison.
Knowing the current password is the real gate on the change, so this was not
an escalation for someone who did not already know it. What it did give an
authenticated non-admin was an oracle: submit a guess at the local admin
password and read the answer off the 401, under the login rate limit and
lockout rather than outside them. With the password in hand the same caller
could then change it and lock the administrator out.
ensureAdminSession is a no-op for requests carrying no session cookie, so the
Basic Auth path this handler was largely written around is unaffected, and it
admits SSO principals on instances with no local admin.
Found by sweeping every proxy-admin check for a missing session counterpart.
The other 43 call sites were either identity extraction rather than
authorization, already carried a session branch, or were compensated
downstream by ensureSettingsScope.
Contract-Neutral: behavioral hardening on an existing route, no request or response shape change; session callers now held to the same admin rule the proxy branch already applied
A settings capability is a promise the routes have to keep. Without an RBAC
licence the authorizer allows every action, so capabilities derived from it
alone reported true while the matching route stayed gated by
ensureSettingsScope and in turn ensureAdminSession.
canAccessPermissionSurface already refused to trust the authorizer for a
non-admin proxy caller. The session half of that rule was never written, so a
non-admin session on an unlicensed instance was told apiAccessRead,
apiAccessWrite, singleSignOnRead and singleSignOnWrite were all available. The
nav gates on exactly those flags, so the API Access and Single Sign-On tabs
rendered, their first request came back 403, and the user got an error toast
on a tab they were never able to use. Everything routed through
canAccessAdminSurface was already correct, which is why authenticationRead and
the audit surfaces behaved and these two did not.
The fallback uses snapshot.sessionIsAdmin, which derives from the same
sessionUserCarriesAdminPrivileges the routes enforce, so the capability now
matches the answer the route will give. That keeps the OIDC-only pattern
working, where SSO principals are the instance's only administrators.
Nothing was reachable that should not have been. This is a capability
reporting fix, not an access control one.
Refs #1672
Contract-Neutral: settingsCapabilities JSON shape is unchanged (same 14 fields, same types); this corrects a wrong value returned to non-admin sessions when no RBAC authorizer is registered, no public-contract delta
golangci-lint had accumulated 12 findings since 5abb2d8f4. All fixed with
real dedup (no nolint suppressions) and the repo's existing errcheck idioms:
- dupl internal/monitoring: docker/host identity-conflict trackers were
structural clones; extracted a shared identityFlapTracker core with a
domain-neutral identityConflict result. Per-domain files now hold only
the window const and the model translation. Tracker-behavior tests
consolidated into identity_flap_tracker_test.go; Monitor-level
translation and Apply*Report integration tests remain per domain.
- dupl internal/api/router.go: VM/container workload chart loops shared a
16-line live-fallback block; extracted guestChartSeriesWithLiveFallback
over a guestLiveMetricsView interface both views satisfy.
- dupl internal/storagehealth/risk.go: SMART attribute copying extracted
into applySMARTAttributes shared by both assessors (same
*models.SMARTAttributes type on both inputs).
- errcheck pkg/audit/sqlite_logger.go: three defer tx.Rollback() sites
now use the repo-wide defer func() { _ = tx.Rollback() }() idiom.
- errcheck telemetry/notifications tests: send() errors now fail the
test; queue.Stop() uses the package's _ = idiom.
Full test suites pass for all six touched packages.
Contract-Neutral: lint-hygiene restoration: dupl dedup (identical logic extracted to shared helpers) and errcheck idiom fixes; no public-contract or behavioral delta
The findings runtime keys resources by whatever ID the producer used:
unified-derived findings carry the hashed canonical resource ID, but
Patrol guest inventory rows carry the node-scoped Proxmox source ID
(instance:node:vmid). The operator-state provider closure looked the
reference up in the unified store directly, which keys by canonical ID
only, so maintenance windows, intentionally-offline intent, and
criticality silently never reached guest findings - Patrol could
auto-propose fixes for a guest the operator had flagged as in
maintenance, and the suppression auto-ack never fired for guests.
A reference that misses the store now resolves through the registry
(GetByReference, which after the node-independent guest identity change
also covers retired canonical-ID eras and node-scoped guest source
references) and retries under the canonical ID. Direct canonical hits
stay as cheap as before; the resolution hop only runs on a miss.
Contract-Neutral: operator-state provider ref-resolution fix: router.go closure bug fix with api-contracts delta staged; dependent subsystem contracts (agent-lifecycle, performance-and-scalability, security-privacy, storage-recovery) have no behavioral delta
A node joining a PVE cluster after setup could never be trusted when the
primary was fingerprint-pinned (#1664). Two independent gaps: cluster
discovery validation only relaxed TLS when client construction failed,
but a pinned-fingerprint mismatch surfaces from the first API call, so
the member was judged not-a-Proxmox-node and its captured fingerprint
discarded; and the cluster client's TOFU refresh only ran for endpoints
that already had a per-endpoint fingerprint, handling rotation but never
first trust. Validation now retries with the member's own captured
fingerprint before rejecting it, TOFU capture runs on first use, and
discovery failures now distinguish DNS, refused, timeout, and TLS
causes in the endpoint error instead of one generic message.
Contract-Neutral: cluster discovery TLS validation bug fix; no wire contract or payload change
Transport-level failures from httpClient.Do (connection refused/reset,
DNS, TLS, timeouts) were wrapped as plain errors, so they fell through
userFriendlyActivationError's LicenseServerError branch to the generic
contact-support fallback instead of the retryable temporarily-unavailable
message. Users behind flaky networks got a dead-end message, and the
2026-08-01 rc.6 paid-runtime gate failure was misdiagnosed the same way.
Classify Do failures at the client layer as a retryable LicenseServerError
(code license_server_unreachable, StatusCode 0 so revocation/suspension/
migration status gates never match) with the cause reachable via Unwrap so
the status poller's context.Canceled check keeps working. The cloud-paid
transport boundary contract now names this classification. Tests pin the
classification, the cancellation chain, and the end-to-end user-facing
retryable message.
Connected systems showed a PBS machine twice (API row plus host-agent row)
whenever the PBS connection was configured with an address the agent never
reports, because PBS identity was limited to the configured name and host.
The PBS poll now captures the hostname the node reports about itself
(GET /nodes, the previously unused client GetNodeName) on
models.PBSInstance.NodeName, and /api/connections includes it in the PBS
connection's host aliases, so directPlatformHostAttachment merges the agent
running on that machine into the PBS row the same way PVE composes from
API-reported node identity. Node-name fetch failure stays partial data and
never fails the poll. Reported by Johannes Strasser (rc.5).
A PBS instance whose host also runs a Pulse agent showed twice under
Settings > Infrastructure > Connected systems: once as the API
connection and once as an agent host, while the equivalent PVE setup
collapses into one API + Agent row. Two defects in the connections
grouping caused this.
The PBS and PMG pollers write pbs-<name>/pmg-<name> unified instance
IDs while connection IDs carry the bare configured name, so
primaryConnectionIDForResource built connection IDs that never matched
and the unified-resource attachment pass could not see either platform.
Beyond that, agent-to-PBS pairing only had the literal host-string
fallback, which fails when the connection is configured with an FQDN or
DNS alias the agent never reports. Attachment now also consumes the
shared top-level-system resolver, whose same-machine contract (machine
identity, short-form hostname equivalence, unique IPs) is what the
Infrastructure surfaces already use to count such a machine as one
system. Groups naming more than one platform connection fail closed,
and Proxmox stays with the LinkedAgentID passes and their
cluster-membership semantics.
Reported by Johannes Strasser against v6.2.0-rc.5 (pbs01/pbs02 each
listed twice).
An action that reached executing had exactly one way out: a terminal
operation receipt from the agent. Expiry skips executing rows on purpose,
and reconciliation bailed for every non-terminal query answer, so a Docker
update whose agent restarted mid-pull sat in executing forever with no
operator route out of it.
The agent-side receipt store rewrites accepted and started receipts to
interrupted on every Open(), and nothing can move an interrupted or
tombstoned receipt back to terminal. An identity-correlated answer of
either kind is therefore proof the operation will never report, and the
three executors now settle the action immediately as inconclusive with a
message telling the operator Pulse cannot confirm the effect and the
resource needs checking by hand.
A not_found answer, or a receipt still merely accepted or started, may
still be completed, so those keep waiting and only settle once the
dispatch attempt is older than one hour, the same threshold the
pulse-intelligence telemetry already uses to call an executing action
stuck. Every typed operation timeout is far shorter than that, so an
in-flight mutation is never cut short. A transport error answering the
query is still not evidence and preserves receipt_pending unchanged.
RecoverExecutingActions drives all of this on the existing two-minute
recovery loop, so rows already wedged before the upgrade heal themselves
without anyone touching them.
For the residue that reconciliation cannot reach, an agent that was
reinstalled or a legacy executing row with no dispatch attempt, POST
/api/actions/{id}/force-fail writes the same inconclusive terminal truth
under an operator attribution. It never touches the transport, refuses
anything already terminal, and is gated on admin plus settings:write on
top of the execute capability check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A host running both PVE and PBS is a deployment the docs call officially
supported, and the agent's RunAll registers each product in turn from the
one install token. The bootstrap grant recorded consumption per TOKEN, so
the PVE leg spent it, the PBS leg came back canRegister=false, the agent
wrote a proxmox-pbs-registration-blocked marker, and install.sh printed an
ERROR banner over a PVE source that had registered perfectly well.
- server: consumption is now recorded per canonical type. One PVE create
and one PBS create per token, each still one-shot — a second create of
the same type takes the same 403. The bounds that are not per type stay
singular: the 24h mint-age clock and the first-use bound_hostname are
shared, so whichever type registers first pins the hostname for both and
the second type cannot be aimed at another machine. The per-type ledger
lives in proxmox_registration_consumed_types; a record carrying only
proxmox_registration_completed=true predates it and still reads as every
type consumed, so upgrading cannot revive a token already spent in the
field. The unsuffixed completion block keeps tracking the most recent
completion, which also leaves an older binary reading the same store
failing closed.
- rollback: the consume-before-persist undo is scoped to the keys one
consumption wrote, so a failed PBS source save restores the PBS grant
without resurrecting the PVE grant that already produced a source.
- agent: RunAll no longer lets one product's failure speak for the host. It
attempts and returns the remaining products, errors only when every
detected product failed, and publishes the detected products in a
proxmox-detected-types state marker.
- installer: report_proxmox_registration_outcome reads that marker, waits
for an outcome from each detected product, and prints a success or denial
line per product instead of one verdict. Agents predating the marker keep
the old first-outcome-wins timing so a single-product host does not wait
out the window. The blocked-marker path now only fires on genuine refusals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Patrol tools" readiness check read the cached model-readiness
snapshot's tool-protocol dimension on its own. Since the interrupted-run
handling landed (8d0d74e35, b78330405), a run cancelled after every tool
scenario already passed keeps ToolProtocol at pass while the overall
status reports not_assessed, so the check reported "Patrol ready" from an
evaluation that never completed.
The check now requires the snapshot's own overall verdict (Success)
before reporting ready. A snapshot carrying no verdict at all — overall
status not_assessed, or the interrupted or internal_error cause — is not
turned into a failure either: it falls back to the base-config classifier
exactly as an absent snapshot does, capped at a warning. That cap matters
because not_ready is a blocking status in this payload: it clears
readiness.ready, which disables the Patrol run control in
usePatrolIntelligenceState and drops the page into the setup-only view.
#1640 promises a severed or cancelled check never blames the model and
never blocks Patrol from running in Watch mode, and the runtime gate on
POST /api/ai/patrol/run (PatrolRuntimeReadiness) already treats an
unassessed mode as a warning, so a blocking tools check would have
contradicted the route that actually runs Patrol.
A completed run whose tool protocol passed while the overall verdict fell
short now warns instead of claiming ready. It must not block either: the
dimension that actually failed carries the verdict on its own check
(context quality blocks, latency warns), so blocking here would have
turned today's latency warning into a hard stop.
Regression tests: internal/api/issue1640_readiness_gate_test.go covers
the gate across interrupted, internal-error, completed-pass,
completed-fail, and short-of-pass snapshots, asserting the resulting
runnability of the readiness payload;
internal/ai/issue1640_readiness_gate_test.go drives a real evaluation
that is cancelled at the continuation probe to produce the
ToolProtocol=pass / status=not_assessed snapshot end to end and pins that
PatrolRuntimeReadiness keeps Patrol runnable. The new API test file is
registered in the subsystem verification registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SSO settings panel presents the OIDC Callback / Redirect URL and the
SAML SP metadata and ACS URLs as the values to register with an Identity
Provider. When PULSE_PUBLIC_URL was unset, providerToResponse built them
on a hardcoded http://localhost:7655, so an admin copied a localhost URL
into their IdP and got an opaque failure there with nothing pointing back
at Pulse as the cause.
The base URL is now resolved from the configured public URL when set —
still authoritative — and otherwise from the inbound request, which by
construction arrived over an address that reaches Pulse. This follows the
pattern buildSSOOIDCCallbackURL already used for the live OIDC login
flow, and the frontend already used for the SAML SP metadata preview
(window.location.origin). The scheme/host derivation is factored out of
that builder into requestForwardedScheme, requestForwardedHost and
requestOriginBaseURL on router.go, so forwarded headers stay behind the
same trusted-proxy gate; buildSSOOIDCCallbackURL's output is unchanged.
When neither source resolves a host the fields are now omitted rather
than carrying a wrong absolute URL. The panel renders guidance pointing
at the public URL setting instead of a copy button, for both the OIDC and
the SAML blocks. The add-provider modal also no longer claims the URL
will be "shown here" after save — the modal closes on save, so it now
says to copy it from the provider card.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reset added in b45bd66b9 only fired when the temperature SSH key
file on disk changed (mtime/size). An operator who repairs SSH access
any other way — fixing authorized_keys on the host, repairing
known_hosts, restoring network reachability — still waited out a
backoff window that may have compounded toward fifteen minutes.
A system-settings save is the natural operator touchpoint after such a
repair, so the settings handler now fans ResetSSHFailureBackoff out to
every live tenant monitor after a successful save, clearing the
per-host temperature SSH backoff and the knownhosts keyscan backoff.
The reset touches in-memory retry timing only; nothing is persisted
and no request field controls it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to 8d0d74e35. The keepalive mechanism was right, the edges
were not.
1. The evaluation ran on a bare goroutine with no recover, so a panic in
provider streaming or validation took the whole Pulse process down.
Before that commit the same panic was on the request goroutine and the
recovery middleware turned it into a logged 500. The goroutine now
recovers, logs the panic with its stack, and answers with an ordinary
readiness result carrying the new internal_error cause and every
dimension reported as not assessed. A Pulse defect is not a model
verdict.
2. Headers were only Set, never committed, despite the comment, the
commit message, and api-contracts.md all claiming otherwise. The
status line went out with the first keepalive at +10s, so a proxy
with a sub-10s time-to-first-byte budget still severed the request.
The transport now writes and flushes WriteHeader(200) before the
ticker starts, matching the pattern the file already uses for SSE.
3. The flusher was resolved with a discarded ok, so a writer that is not
an http.Flusher silently buffered the keepalives and degraded back to
the original bug. It is now checked and logged; the response still
completes, so a warning is the right level here rather than the hard
failure the SSE handlers use.
4. TestIssue1640HandlerUsesKeepaliveTransport grepped the handler source
for substrings, which proves nothing about behaviour. Replaced with a
real httptest.NewServer test that runs a 300ms evaluation and asserts
the client sees the 200 and a body byte before the evaluation
completes, and that the padded body still parses as the expected JSON.
Added coverage for the panic path and the non-flushable writer, and
fixed the eager body[:1] that would panic when a transport regression
left the body empty.
5. The settings readiness banner had no not_assessed branch, so an
interrupted run still rendered the red "Patrol model not verified"
headline: the exact blame-the-model presentation the backend fix
removed. Tone and headline are now exported pure functions with a
neutral treatment for not_assessed and interrupted results, and an
interrupted run cannot claim verification from a max_verified_mode
recorded before the cancellation.
6. createAPIErrorFromResponse let a short plain-text body override an
explicit caller fallbackMessage. A caller passing a fallback knows
which operation it was performing; an intermediary writing the body
does not. Precedence is now canonical JSON, then caller fallback,
then body, with the HTML and oversize suppression unchanged.
7. patrolRunCancelled classified on the raw "context canceled" substring
as its first switch case. Ollama embeds that phrase in its own error
body when it aborts an upstream request, so a genuine provider
failure on a healthy run was classified interrupted and finish()
persisted it as not_assessed. Cancellation is now established from
the run itself (errors.Is(err, context.Canceled), or a cancelled run
context), never from error wording, and the readiness paths classify
through a context-aware entry point. context.DeadlineExceeded keeps
its provider-path timeout classification.
The readiness gate in HandlePatrolModelReadiness keying off ToolProtocol
alone is untouched, as agreed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial follow-ups to ac43506e6, which let a host-typed install token
bootstrap-create a Proxmox source. The one-shot machinery held up; these
are the four holes around it.
- test integrity: TestIssue1644HostInstallTokenGrantStaysHostnameBound
completed a registration first, so the second request died at the
completion gate and the bound_hostname comparison was never reached —
the test passed with the binding deleted. It now binds without
consuming (checkRegistration), rejects a different serverName while
the grant is still live, and then completes on the bound hostname to
show the grant was never the reason for the rejection.
- grant TTL: install tokens are minted with no expiry, so every host
install token on a Proxmox box carried a live create-a-source
capability forever. The grant now expires 24h after mint on its own
clock (install_issued_at stamped at mint, falling back to the record's
CreatedAt, failing closed with neither). Expired grants take the same
403 path with a distinct warn.
- replay window: SaveNodesConfig ran before the grant was consumed, so a
persistently failing token store left a source on disk next to an
unconsumed grant — a repeatable create-N-sources primitive. The grant
is now consumed and persisted first, and a failed source save rolls
the consumption back, so either both stores advanced or neither did.
- exec binding: auto-register writes bound_hostname with no
bound_agent_id and no binding version, which is exactly the shape
canBindAgentInstallExecToken refuses, so host-token command enrollment
was being admitted by the legacy pre-v6.1.1 migration branch. That
record shape is now handled explicitly as a clean first use (hostname
equivalence required), and a bound_hostname written by registration is
no longer overwritten by an equivalent spelling the agent reports,
because the still-unconsumed grant compares against it.
Single consumption across types is unchanged: a combined PVE+PBS host
still gets exactly one grant.
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus TTL and exec-first-bind contract pins in internal/api/contract_test.go.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Settings > Infrastructure installer mints generic host install
tokens, but install.sh auto-detects Proxmox and the agent presents type
pve/pbs at /api/auto-register. The bootstrap grant required an exact
install_type match, so every generic install on a Proxmox node was
denied source creation and the denial was a single buried journal warn.
Four-part fix (#1644):
- server: extend the one-shot bootstrap grant to host-issued install
tokens presenting a canonical Proxmox type. Typed tokens stay pinned,
the grant keeps its settings-write mint requirement, first-hostname
binding, serialized completion, and single consumption across types.
- agent: a canRegister=false denial now logs at error level, returns a
setup error, and records the operator-facing reason in a
proxmox-<type>-registration-blocked state marker.
- installer: report the Proxmox registration outcome in install output
by reading the registered/blocked markers, and poll the server lookup
for a bounded retry window before warning that registration was not
confirmed (readyz flips before the first report cycle).
- setup script: the auto-register transport now captures the HTTP
status alongside the body (no -f), making the invalid-setup-token
branch reachable via 401/403 instead of a dead server-string grep,
and operator guidance names Settings -> Infrastructure instead of the
retired Nodes page (also updated in docs/PBS.md and the pinned
assertions in contract, setup-script, and repoctl docs tests).
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus new install.sh proofs for the retry window and blocked-marker
surfacing, and the updated hostagent blocked-registration test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes#1640. Three defects around POST /api/ai/patrol/readiness on slow
local hardware behind a reverse proxy:
1. The handler ran up to four sequential provider calls (~45s and more on
slow Ollama boxes) while writing nothing to the response, so any
intermediary with a ~30s read timeout severed the request mid-run. The
handler now commits headers up front and streams flushed newline
keepalives every 10s while the evaluation runs, then appends the normal
JSON payload. Leading newlines are insignificant JSON whitespace, so
existing clients parse the response unchanged.
2. A severed connection cancels the request context, and
patrolRuntimeFailureFromError classified the resulting context.Canceled
as a generic "Provider analysis error", blaming the provider and model
for an infrastructure event. Mid-run cancellation is now classified as
the new "interrupted" cause: the overall status and every unfinished
dimension and autonomy mode report not assessed, per-scenario evidence
completed before the interruption is preserved in the returned result,
and the readiness cache keeps the last completed evaluation.
context.DeadlineExceeded keeps its provider-path timeout classification.
3. createAPIErrorFromResponse pre-seeded the error message with the raw
response body, making its non-JSON guard dead code, so full HTML proxy
error pages became Error.message and were rendered into the readiness
result boxes. Non-JSON bodies now surface only when they are short
plain text; anything with markup or excessive length collapses to a
generic status-derived message.
Regression tests: internal/ai/issue1640_readiness_cancellation_test.go,
internal/api/issue1640_readiness_transport_test.go, and
frontend-modern/src/utils/__tests__/apiClient.issue1640.test.ts, all
registered in the subsystem verification registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while triaging #1643 and #1637: both fields were stored via
system.json and accepted, validated, and echoed by the settings API,
but nothing ever consumed them. The unattended update schedule is owned
entirely by the systemd timer install.sh renders, and no UI control ever
set the fields, so the API persisted a schedule preference that could
never take effect.
Remove the fields from config.Config, SystemSettings, the settings
handlers, and the frontend config type, along with the interval
validation and the .env AUTO_UPDATE_CHECK_INTERVAL rewrite (a legacy
line is now preserved verbatim). Legacy clients that still send the
keys get them silently ignored instead of validated, and a system.json
written before the removal still loads cleanly - both behaviors are
pinned by new tests. User docs no longer describe the phantom schedule
settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v6.1.2 (c41edb65a) left the agent config gate and the command-channel
admission evaluating exec-token bindings under different policies: the
config gate admitted on bound-hostname OR bound-agent-ID while channel
admission required both to match, compared hostnames with plain case
folding instead of the system-wide short-vs-FQDN equivalence rule, and
had no recovery path for hosts whose immutable agent ID still matched
but whose hostname had drifted since binding. Affected agents kept
reporting CommandsEnabled=true while every channel registration was
rejected, so fleets showed a permanent "Remote control blocked" chip
with reinstall as the only recourse (reported by a customer with a
large Docker fleet after upgrading to v6.1.2).
- Single-source the binding decision in evaluateAgentExecBinding; both
admitAgentExecToken and commandConfigAllowedForToken now consume it,
so the config payload can never advertise command execution that
admission would reject.
- Treat the immutable machine-derived agent ID as the primary binding
identity: an exact ID match re-binds a drifted (renamed) hostname in
place instead of stranding the host; hostname match alone still fails
closed for version-2 bindings.
- Compare hostnames with unifiedresources.HostnamesEquivalent (plus
case-insensitive exact match for IP literals) across admission,
session validation, and legacy migration, so docker01 vs docker01.lan
no longer splits the decision.
- Stop treating a miss on the token-scoped connectivity lookup as
authoritative in the connections ledger: host.TokenID is sticky
across token rotation/revocation, and a shared token fronting more
than one live session fails closed in the token lookup, so fall
through to the agent-ID and hostname lookups before reporting an
enabled host as blocked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Availability targets gain an optional probe agent assignment. Setting
it requires the external_probe entitlement, enforced only at the
moment of assignment - local targets never consult the license path.
Assigned targets are delivered to their agent through the signed
agent-config channel, skipped by the local poller, and resume local
execution automatically if the entitlement lapses. Probe-reported
results are accepted only from the currently assigned agent, share
the local failure-threshold accounting, carry source attribution,
and derive to indeterminate at read time when reports go stale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add external_probe to the licensing feature enumeration, Pro tier
membership, self-hosted feature catalog, feature map, upgrade matrix
and pricing handoff, with the entitlement contract goldens updated.
The capability is served by the community binary - the entitlement
alone gates it, like relay. No existing free functionality changes:
local availability checks stay ungated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rcourtman