An action that reached executing had exactly one way out: a terminal
operation receipt from the agent. Expiry skips executing rows on purpose,
and reconciliation bailed for every non-terminal query answer, so a Docker
update whose agent restarted mid-pull sat in executing forever with no
operator route out of it.
The agent-side receipt store rewrites accepted and started receipts to
interrupted on every Open(), and nothing can move an interrupted or
tombstoned receipt back to terminal. An identity-correlated answer of
either kind is therefore proof the operation will never report, and the
three executors now settle the action immediately as inconclusive with a
message telling the operator Pulse cannot confirm the effect and the
resource needs checking by hand.
A not_found answer, or a receipt still merely accepted or started, may
still be completed, so those keep waiting and only settle once the
dispatch attempt is older than one hour, the same threshold the
pulse-intelligence telemetry already uses to call an executing action
stuck. Every typed operation timeout is far shorter than that, so an
in-flight mutation is never cut short. A transport error answering the
query is still not evidence and preserves receipt_pending unchanged.
RecoverExecutingActions drives all of this on the existing two-minute
recovery loop, so rows already wedged before the upgrade heal themselves
without anyone touching them.
For the residue that reconciliation cannot reach, an agent that was
reinstalled or a legacy executing row with no dispatch attempt, POST
/api/actions/{id}/force-fail writes the same inconclusive terminal truth
under an operator attribution. It never touches the transport, refuses
anything already terminal, and is gated on admin plus settings:write on
top of the execute capability check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A host running both PVE and PBS is a deployment the docs call officially
supported, and the agent's RunAll registers each product in turn from the
one install token. The bootstrap grant recorded consumption per TOKEN, so
the PVE leg spent it, the PBS leg came back canRegister=false, the agent
wrote a proxmox-pbs-registration-blocked marker, and install.sh printed an
ERROR banner over a PVE source that had registered perfectly well.
- server: consumption is now recorded per canonical type. One PVE create
and one PBS create per token, each still one-shot — a second create of
the same type takes the same 403. The bounds that are not per type stay
singular: the 24h mint-age clock and the first-use bound_hostname are
shared, so whichever type registers first pins the hostname for both and
the second type cannot be aimed at another machine. The per-type ledger
lives in proxmox_registration_consumed_types; a record carrying only
proxmox_registration_completed=true predates it and still reads as every
type consumed, so upgrading cannot revive a token already spent in the
field. The unsuffixed completion block keeps tracking the most recent
completion, which also leaves an older binary reading the same store
failing closed.
- rollback: the consume-before-persist undo is scoped to the keys one
consumption wrote, so a failed PBS source save restores the PBS grant
without resurrecting the PVE grant that already produced a source.
- agent: RunAll no longer lets one product's failure speak for the host. It
attempts and returns the remaining products, errors only when every
detected product failed, and publishes the detected products in a
proxmox-detected-types state marker.
- installer: report_proxmox_registration_outcome reads that marker, waits
for an outcome from each detected product, and prints a success or denial
line per product instead of one verdict. Agents predating the marker keep
the old first-outcome-wins timing so a single-product host does not wait
out the window. The blocked-marker path now only fires on genuine refusals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Patrol tools" readiness check read the cached model-readiness
snapshot's tool-protocol dimension on its own. Since the interrupted-run
handling landed (8d0d74e35, b78330405), a run cancelled after every tool
scenario already passed keeps ToolProtocol at pass while the overall
status reports not_assessed, so the check reported "Patrol ready" from an
evaluation that never completed.
The check now requires the snapshot's own overall verdict (Success)
before reporting ready. A snapshot carrying no verdict at all — overall
status not_assessed, or the interrupted or internal_error cause — is not
turned into a failure either: it falls back to the base-config classifier
exactly as an absent snapshot does, capped at a warning. That cap matters
because not_ready is a blocking status in this payload: it clears
readiness.ready, which disables the Patrol run control in
usePatrolIntelligenceState and drops the page into the setup-only view.
#1640 promises a severed or cancelled check never blames the model and
never blocks Patrol from running in Watch mode, and the runtime gate on
POST /api/ai/patrol/run (PatrolRuntimeReadiness) already treats an
unassessed mode as a warning, so a blocking tools check would have
contradicted the route that actually runs Patrol.
A completed run whose tool protocol passed while the overall verdict fell
short now warns instead of claiming ready. It must not block either: the
dimension that actually failed carries the verdict on its own check
(context quality blocks, latency warns), so blocking here would have
turned today's latency warning into a hard stop.
Regression tests: internal/api/issue1640_readiness_gate_test.go covers
the gate across interrupted, internal-error, completed-pass,
completed-fail, and short-of-pass snapshots, asserting the resulting
runnability of the readiness payload;
internal/ai/issue1640_readiness_gate_test.go drives a real evaluation
that is cancelled at the continuation probe to produce the
ToolProtocol=pass / status=not_assessed snapshot end to end and pins that
PatrolRuntimeReadiness keeps Patrol runnable. The new API test file is
registered in the subsystem verification registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SSO settings panel presents the OIDC Callback / Redirect URL and the
SAML SP metadata and ACS URLs as the values to register with an Identity
Provider. When PULSE_PUBLIC_URL was unset, providerToResponse built them
on a hardcoded http://localhost:7655, so an admin copied a localhost URL
into their IdP and got an opaque failure there with nothing pointing back
at Pulse as the cause.
The base URL is now resolved from the configured public URL when set —
still authoritative — and otherwise from the inbound request, which by
construction arrived over an address that reaches Pulse. This follows the
pattern buildSSOOIDCCallbackURL already used for the live OIDC login
flow, and the frontend already used for the SAML SP metadata preview
(window.location.origin). The scheme/host derivation is factored out of
that builder into requestForwardedScheme, requestForwardedHost and
requestOriginBaseURL on router.go, so forwarded headers stay behind the
same trusted-proxy gate; buildSSOOIDCCallbackURL's output is unchanged.
When neither source resolves a host the fields are now omitted rather
than carrying a wrong absolute URL. The panel renders guidance pointing
at the public URL setting instead of a copy button, for both the OIDC and
the SAML blocks. The add-provider modal also no longer claims the URL
will be "shown here" after save — the modal closes on save, so it now
says to copy it from the provider card.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reset added in b45bd66b9 only fired when the temperature SSH key
file on disk changed (mtime/size). An operator who repairs SSH access
any other way — fixing authorized_keys on the host, repairing
known_hosts, restoring network reachability — still waited out a
backoff window that may have compounded toward fifteen minutes.
A system-settings save is the natural operator touchpoint after such a
repair, so the settings handler now fans ResetSSHFailureBackoff out to
every live tenant monitor after a successful save, clearing the
per-host temperature SSH backoff and the knownhosts keyscan backoff.
The reset touches in-memory retry timing only; nothing is persisted
and no request field controls it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to 8d0d74e35. The keepalive mechanism was right, the edges
were not.
1. The evaluation ran on a bare goroutine with no recover, so a panic in
provider streaming or validation took the whole Pulse process down.
Before that commit the same panic was on the request goroutine and the
recovery middleware turned it into a logged 500. The goroutine now
recovers, logs the panic with its stack, and answers with an ordinary
readiness result carrying the new internal_error cause and every
dimension reported as not assessed. A Pulse defect is not a model
verdict.
2. Headers were only Set, never committed, despite the comment, the
commit message, and api-contracts.md all claiming otherwise. The
status line went out with the first keepalive at +10s, so a proxy
with a sub-10s time-to-first-byte budget still severed the request.
The transport now writes and flushes WriteHeader(200) before the
ticker starts, matching the pattern the file already uses for SSE.
3. The flusher was resolved with a discarded ok, so a writer that is not
an http.Flusher silently buffered the keepalives and degraded back to
the original bug. It is now checked and logged; the response still
completes, so a warning is the right level here rather than the hard
failure the SSE handlers use.
4. TestIssue1640HandlerUsesKeepaliveTransport grepped the handler source
for substrings, which proves nothing about behaviour. Replaced with a
real httptest.NewServer test that runs a 300ms evaluation and asserts
the client sees the 200 and a body byte before the evaluation
completes, and that the padded body still parses as the expected JSON.
Added coverage for the panic path and the non-flushable writer, and
fixed the eager body[:1] that would panic when a transport regression
left the body empty.
5. The settings readiness banner had no not_assessed branch, so an
interrupted run still rendered the red "Patrol model not verified"
headline: the exact blame-the-model presentation the backend fix
removed. Tone and headline are now exported pure functions with a
neutral treatment for not_assessed and interrupted results, and an
interrupted run cannot claim verification from a max_verified_mode
recorded before the cancellation.
6. createAPIErrorFromResponse let a short plain-text body override an
explicit caller fallbackMessage. A caller passing a fallback knows
which operation it was performing; an intermediary writing the body
does not. Precedence is now canonical JSON, then caller fallback,
then body, with the HTML and oversize suppression unchanged.
7. patrolRunCancelled classified on the raw "context canceled" substring
as its first switch case. Ollama embeds that phrase in its own error
body when it aborts an upstream request, so a genuine provider
failure on a healthy run was classified interrupted and finish()
persisted it as not_assessed. Cancellation is now established from
the run itself (errors.Is(err, context.Canceled), or a cancelled run
context), never from error wording, and the readiness paths classify
through a context-aware entry point. context.DeadlineExceeded keeps
its provider-path timeout classification.
The readiness gate in HandlePatrolModelReadiness keying off ToolProtocol
alone is untouched, as agreed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial follow-ups to ac43506e6, which let a host-typed install token
bootstrap-create a Proxmox source. The one-shot machinery held up; these
are the four holes around it.
- test integrity: TestIssue1644HostInstallTokenGrantStaysHostnameBound
completed a registration first, so the second request died at the
completion gate and the bound_hostname comparison was never reached —
the test passed with the binding deleted. It now binds without
consuming (checkRegistration), rejects a different serverName while
the grant is still live, and then completes on the bound hostname to
show the grant was never the reason for the rejection.
- grant TTL: install tokens are minted with no expiry, so every host
install token on a Proxmox box carried a live create-a-source
capability forever. The grant now expires 24h after mint on its own
clock (install_issued_at stamped at mint, falling back to the record's
CreatedAt, failing closed with neither). Expired grants take the same
403 path with a distinct warn.
- replay window: SaveNodesConfig ran before the grant was consumed, so a
persistently failing token store left a source on disk next to an
unconsumed grant — a repeatable create-N-sources primitive. The grant
is now consumed and persisted first, and a failed source save rolls
the consumption back, so either both stores advanced or neither did.
- exec binding: auto-register writes bound_hostname with no
bound_agent_id and no binding version, which is exactly the shape
canBindAgentInstallExecToken refuses, so host-token command enrollment
was being admitted by the legacy pre-v6.1.1 migration branch. That
record shape is now handled explicitly as a clean first use (hostname
equivalence required), and a bound_hostname written by registration is
no longer overwritten by an equivalent spelling the agent reports,
because the still-unconsumed grant compares against it.
Single consumption across types is unchanged: a combined PVE+PBS host
still gets exactly one grant.
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus TTL and exec-first-bind contract pins in internal/api/contract_test.go.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Settings > Infrastructure installer mints generic host install
tokens, but install.sh auto-detects Proxmox and the agent presents type
pve/pbs at /api/auto-register. The bootstrap grant required an exact
install_type match, so every generic install on a Proxmox node was
denied source creation and the denial was a single buried journal warn.
Four-part fix (#1644):
- server: extend the one-shot bootstrap grant to host-issued install
tokens presenting a canonical Proxmox type. Typed tokens stay pinned,
the grant keeps its settings-write mint requirement, first-hostname
binding, serialized completion, and single consumption across types.
- agent: a canRegister=false denial now logs at error level, returns a
setup error, and records the operator-facing reason in a
proxmox-<type>-registration-blocked state marker.
- installer: report the Proxmox registration outcome in install output
by reading the registered/blocked markers, and poll the server lookup
for a bounded retry window before warning that registration was not
confirmed (readyz flips before the first report cycle).
- setup script: the auto-register transport now captures the HTTP
status alongside the body (no -f), making the invalid-setup-token
branch reachable via 401/403 instead of a dead server-string grep,
and operator guidance names Settings -> Infrastructure instead of the
retired Nodes page (also updated in docs/PBS.md and the pinned
assertions in contract, setup-script, and repoctl docs tests).
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus new install.sh proofs for the retry window and blocked-marker
surfacing, and the updated hostagent blocked-registration test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes#1640. Three defects around POST /api/ai/patrol/readiness on slow
local hardware behind a reverse proxy:
1. The handler ran up to four sequential provider calls (~45s and more on
slow Ollama boxes) while writing nothing to the response, so any
intermediary with a ~30s read timeout severed the request mid-run. The
handler now commits headers up front and streams flushed newline
keepalives every 10s while the evaluation runs, then appends the normal
JSON payload. Leading newlines are insignificant JSON whitespace, so
existing clients parse the response unchanged.
2. A severed connection cancels the request context, and
patrolRuntimeFailureFromError classified the resulting context.Canceled
as a generic "Provider analysis error", blaming the provider and model
for an infrastructure event. Mid-run cancellation is now classified as
the new "interrupted" cause: the overall status and every unfinished
dimension and autonomy mode report not assessed, per-scenario evidence
completed before the interruption is preserved in the returned result,
and the readiness cache keeps the last completed evaluation.
context.DeadlineExceeded keeps its provider-path timeout classification.
3. createAPIErrorFromResponse pre-seeded the error message with the raw
response body, making its non-JSON guard dead code, so full HTML proxy
error pages became Error.message and were rendered into the readiness
result boxes. Non-JSON bodies now surface only when they are short
plain text; anything with markup or excessive length collapses to a
generic status-derived message.
Regression tests: internal/ai/issue1640_readiness_cancellation_test.go,
internal/api/issue1640_readiness_transport_test.go, and
frontend-modern/src/utils/__tests__/apiClient.issue1640.test.ts, all
registered in the subsystem verification registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while triaging #1643 and #1637: both fields were stored via
system.json and accepted, validated, and echoed by the settings API,
but nothing ever consumed them. The unattended update schedule is owned
entirely by the systemd timer install.sh renders, and no UI control ever
set the fields, so the API persisted a schedule preference that could
never take effect.
Remove the fields from config.Config, SystemSettings, the settings
handlers, and the frontend config type, along with the interval
validation and the .env AUTO_UPDATE_CHECK_INTERVAL rewrite (a legacy
line is now preserved verbatim). Legacy clients that still send the
keys get them silently ignored instead of validated, and a system.json
written before the removal still loads cleanly - both behaviors are
pinned by new tests. User docs no longer describe the phantom schedule
settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v6.1.2 (c41edb65a) left the agent config gate and the command-channel
admission evaluating exec-token bindings under different policies: the
config gate admitted on bound-hostname OR bound-agent-ID while channel
admission required both to match, compared hostnames with plain case
folding instead of the system-wide short-vs-FQDN equivalence rule, and
had no recovery path for hosts whose immutable agent ID still matched
but whose hostname had drifted since binding. Affected agents kept
reporting CommandsEnabled=true while every channel registration was
rejected, so fleets showed a permanent "Remote control blocked" chip
with reinstall as the only recourse (reported by a customer with a
large Docker fleet after upgrading to v6.1.2).
- Single-source the binding decision in evaluateAgentExecBinding; both
admitAgentExecToken and commandConfigAllowedForToken now consume it,
so the config payload can never advertise command execution that
admission would reject.
- Treat the immutable machine-derived agent ID as the primary binding
identity: an exact ID match re-binds a drifted (renamed) hostname in
place instead of stranding the host; hostname match alone still fails
closed for version-2 bindings.
- Compare hostnames with unifiedresources.HostnamesEquivalent (plus
case-insensitive exact match for IP literals) across admission,
session validation, and legacy migration, so docker01 vs docker01.lan
no longer splits the decision.
- Stop treating a miss on the token-scoped connectivity lookup as
authoritative in the connections ledger: host.TokenID is sticky
across token rotation/revocation, and a shared token fronting more
than one live session fails closed in the token lookup, so fall
through to the agent-ID and hostname lookups before reporting an
enabled host as blocked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Availability targets gain an optional probe agent assignment. Setting
it requires the external_probe entitlement, enforced only at the
moment of assignment - local targets never consult the license path.
Assigned targets are delivered to their agent through the signed
agent-config channel, skipped by the local poller, and resume local
execution automatically if the entitlement lapses. Probe-reported
results are accepted only from the currently assigned agent, share
the local failure-threshold accounting, carry source attribution,
and derive to indeterminate at read time when reports go stale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add external_probe to the licensing feature enumeration, Pro tier
membership, self-hosted feature catalog, feature map, upgrade matrix
and pricing handoff, with the entitlement contract goldens updated.
The capability is served by the community binary - the entitlement
alone gates it, like relay. No existing free functionality changes:
local availability checks stay ungated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live-setter fix for #1619 pushed backup and PMG cadences into
running monitors but left pbsPollingInterval out entirely: it was
neither applied to the handler's base config nor pushed to live
monitors, so a saved PBS interval still waited for an unrelated
restart. Add the matching override, setter and scheduler wiring, apply
the value to the base config, and fan it out from the settings handler
like the other cadence settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Saving backupPollingInterval, pmgPollingInterval or backupPollingEnabled
only mutated the API handler's base config. Monitors poll against a
detached DeepCopy of that config, and only pvePollingInterval set the
reload flag, so the saved values never reached shouldRunBackupPoll or
the PMG scheduler until an unrelated monitor reload happened.
Add mutex-guarded runtime overrides on Monitor with live setters, fan
them out to every tenant monitor from the settings handler (mirroring
forEachNotificationManager), and clear the per-instance last-poll
timestamps when lowering the backup interval or re-enabling backup
polling so the next cycle runs an immediate catch-up poll.
Fixes#1619
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The connections ledger marked any PVE/PBS/PMG/VMware/TrueNAS connection
stale once its last successful poll was more than a fixed 2 minutes old,
so polling cadences above 120s read as permanently stale and latched the
connection-degraded alert with no path back to active (#1620).
Derive the active->stale cutoff as max(3 x polling interval, 2 minutes),
mirroring how availability probes already scale. PVE/PBS/PMG cadences
come from the server config; TrueNAS uses the per-instance interval and
VMware the poller summary interval. The 3x multiplier absorbs the
evaluation-order skew where connection alerts are checked concurrently
with the poll on the same scheduler tick, leaving LastSuccess roughly
one interval old even when healthy.
Fixes#1620
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seventeen files closing the zero-coverage functions the current source drop
left behind. Every named target was measured off 0.0 percent by a per-function
coverage delta, re-measured against current main.
- config: the five new durable Proxmox cluster-node identity helpers
(deterministic id, endpoint equality, alias lookup, id existence, lookup by
id) to 100 percent; VMware and agent-profile persistence round-trips under
t.TempDir including that AppendProfileChangeLog appends rather than
replaces; AI chat session save, load, delete, per-user scoping and age
cleanup, with explicit timestamps rather than time.Now-relative fuzz;
PVEInstance.DeepCopy asserted for nested independence.
- truenas: incidentFromPoolStatus over every pool health string,
RecordsFromSnapshot over nil, empty and populated snapshots, both
TransportStatus accessors, and the RPC handshake and auth typed errors
through errors.Is and errors.As.
- unifiedresources: the maintenance-window operator-state lifecycle on
MemoryStore including the not-found and already-cleared arms, plus the
four remaining View accessors asserted on their exact formatted output.
- api: restoreAgentExecMetadata, buildAlertConnectionSnapshotsWithRuntimeSources
and both mock series generators, asserted on shape, ordering and
determinism rather than non-emptiness.
- cmd/pulse-control-plane: the four remaining MSP and mobile proof report
printers, asserted on the concrete strings in captured stdout.
- ai: cost.EmptySummary, approval.emptyExecutionState, demo.IsDemoRuntimeIntended
and tools.findCanonicalAppContainerResourceByReferences across no-match,
first-match, later-match and ambiguous references.
- monitoring, models, alerts: trueNASAppRunning,
supplementalProviderOwnedSourcesForOrg, IOCounterPresence.Effective,
ValidAlertIntentSignal and intentTimePointer.
No source file is modified. Adversarial review returned no rejects across all
seventeen files and flagged four padding cases plus one dead table field; all
were removed and the per-function coverage re-measured as identical, proving
they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
d235aab3c put MigrateFromFiles on a production path for the default org and
failed NewSQLiteManager whenever the import was rejected. That propagated to
GetManager, so every RBAC route returned 503 including ResetAdminRole, which
is the operator's only way back. The realistic trigger is a v5 install that
recreated a role the v6 store already holds under the same ID, so an
ordinary upgrade could leave an operator with no route to repair it short of
hand-editing the legacy JSON.
Rejecting the import is right and stays: importLegacyRBAC is transactional
and the legacy files are left in place, so a failure leaves the database
un-migrated rather than half-migrated. Missing roles deny access rather than
granting it, which is why the store is safe to keep serving.
The store now stays live and records the failure on MigrationError. The
deliberate fail-closed behaviour of the management surface is preserved
rather than removed: the handler manager accessors surface the migration
failure as the same 503 rbac_store_unavailable as before. Recovery reaches
the provider directly, so it is exempt by construction.
The existing 503 contract test is what caught the first attempt at this,
which simply let the surface serve un-migrated data. It now additionally
asserts recovery is reachable, so the two halves are pinned together.
Preserve tenant-scoped metadata through partial URL updates and project stable URLs across runtime identities. Use a safe adjacent launch control across overview tables with desktop and mobile regression coverage.
courtmanr@gmail.com