An action that reached executing had exactly one way out: a terminal
operation receipt from the agent. Expiry skips executing rows on purpose,
and reconciliation bailed for every non-terminal query answer, so a Docker
update whose agent restarted mid-pull sat in executing forever with no
operator route out of it.
The agent-side receipt store rewrites accepted and started receipts to
interrupted on every Open(), and nothing can move an interrupted or
tombstoned receipt back to terminal. An identity-correlated answer of
either kind is therefore proof the operation will never report, and the
three executors now settle the action immediately as inconclusive with a
message telling the operator Pulse cannot confirm the effect and the
resource needs checking by hand.
A not_found answer, or a receipt still merely accepted or started, may
still be completed, so those keep waiting and only settle once the
dispatch attempt is older than one hour, the same threshold the
pulse-intelligence telemetry already uses to call an executing action
stuck. Every typed operation timeout is far shorter than that, so an
in-flight mutation is never cut short. A transport error answering the
query is still not evidence and preserves receipt_pending unchanged.
RecoverExecutingActions drives all of this on the existing two-minute
recovery loop, so rows already wedged before the upgrade heal themselves
without anyone touching them.
For the residue that reconciliation cannot reach, an agent that was
reinstalled or a legacy executing row with no dispatch attempt, POST
/api/actions/{id}/force-fail writes the same inconclusive terminal truth
under an operator attribution. It never touches the transport, refuses
anything already terminal, and is gated on admin plus settings:write on
top of the execute capability check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SSO settings panel presents the OIDC Callback / Redirect URL and the
SAML SP metadata and ACS URLs as the values to register with an Identity
Provider. When PULSE_PUBLIC_URL was unset, providerToResponse built them
on a hardcoded http://localhost:7655, so an admin copied a localhost URL
into their IdP and got an opaque failure there with nothing pointing back
at Pulse as the cause.
The base URL is now resolved from the configured public URL when set —
still authoritative — and otherwise from the inbound request, which by
construction arrived over an address that reaches Pulse. This follows the
pattern buildSSOOIDCCallbackURL already used for the live OIDC login
flow, and the frontend already used for the SAML SP metadata preview
(window.location.origin). The scheme/host derivation is factored out of
that builder into requestForwardedScheme, requestForwardedHost and
requestOriginBaseURL on router.go, so forwarded headers stay behind the
same trusted-proxy gate; buildSSOOIDCCallbackURL's output is unchanged.
When neither source resolves a host the fields are now omitted rather
than carrying a wrong absolute URL. The panel renders guidance pointing
at the public URL setting instead of a copy button, for both the OIDC and
the SAML blocks. The add-provider modal also no longer claims the URL
will be "shown here" after save — the modal closes on save, so it now
says to copy it from the provider card.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial follow-ups to ac43506e6, which let a host-typed install token
bootstrap-create a Proxmox source. The one-shot machinery held up; these
are the four holes around it.
- test integrity: TestIssue1644HostInstallTokenGrantStaysHostnameBound
completed a registration first, so the second request died at the
completion gate and the bound_hostname comparison was never reached —
the test passed with the binding deleted. It now binds without
consuming (checkRegistration), rejects a different serverName while
the grant is still live, and then completes on the bound hostname to
show the grant was never the reason for the rejection.
- grant TTL: install tokens are minted with no expiry, so every host
install token on a Proxmox box carried a live create-a-source
capability forever. The grant now expires 24h after mint on its own
clock (install_issued_at stamped at mint, falling back to the record's
CreatedAt, failing closed with neither). Expired grants take the same
403 path with a distinct warn.
- replay window: SaveNodesConfig ran before the grant was consumed, so a
persistently failing token store left a source on disk next to an
unconsumed grant — a repeatable create-N-sources primitive. The grant
is now consumed and persisted first, and a failed source save rolls
the consumption back, so either both stores advanced or neither did.
- exec binding: auto-register writes bound_hostname with no
bound_agent_id and no binding version, which is exactly the shape
canBindAgentInstallExecToken refuses, so host-token command enrollment
was being admitted by the legacy pre-v6.1.1 migration branch. That
record shape is now handled explicitly as a clean first use (hostname
equivalence required), and a bound_hostname written by registration is
no longer overwritten by an equivalent spelling the agent reports,
because the still-unconsumed grant compares against it.
Single consumption across types is unchanged: a combined PVE+PBS host
still gets exactly one grant.
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus TTL and exec-first-bind contract pins in internal/api/contract_test.go.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Settings > Infrastructure installer mints generic host install
tokens, but install.sh auto-detects Proxmox and the agent presents type
pve/pbs at /api/auto-register. The bootstrap grant required an exact
install_type match, so every generic install on a Proxmox node was
denied source creation and the denial was a single buried journal warn.
Four-part fix (#1644):
- server: extend the one-shot bootstrap grant to host-issued install
tokens presenting a canonical Proxmox type. Typed tokens stay pinned,
the grant keeps its settings-write mint requirement, first-hostname
binding, serialized completion, and single consumption across types.
- agent: a canRegister=false denial now logs at error level, returns a
setup error, and records the operator-facing reason in a
proxmox-<type>-registration-blocked state marker.
- installer: report the Proxmox registration outcome in install output
by reading the registered/blocked markers, and poll the server lookup
for a bounded retry window before warning that registration was not
confirmed (readyz flips before the first report cycle).
- setup script: the auto-register transport now captures the HTTP
status alongside the body (no -f), making the invalid-setup-token
branch reachable via 401/403 instead of a dead server-string grep,
and operator guidance names Settings -> Infrastructure instead of the
retired Nodes page (also updated in docs/PBS.md and the pinned
assertions in contract, setup-script, and repoctl docs tests).
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus new install.sh proofs for the retry window and blocked-marker
surfacing, and the updated hostagent blocked-registration test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while triaging #1643 and #1637: both fields were stored via
system.json and accepted, validated, and echoed by the settings API,
but nothing ever consumed them. The unattended update schedule is owned
entirely by the systemd timer install.sh renders, and no UI control ever
set the fields, so the API persisted a schedule preference that could
never take effect.
Remove the fields from config.Config, SystemSettings, the settings
handlers, and the frontend config type, along with the interval
validation and the .env AUTO_UPDATE_CHECK_INTERVAL rewrite (a legacy
line is now preserved verbatim). Legacy clients that still send the
keys get them silently ignored instead of validated, and a system.json
written before the removal still loads cleanly - both behaviors are
pinned by new tests. User docs no longer describe the phantom schedule
settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v6.1.2 (c41edb65a) left the agent config gate and the command-channel
admission evaluating exec-token bindings under different policies: the
config gate admitted on bound-hostname OR bound-agent-ID while channel
admission required both to match, compared hostnames with plain case
folding instead of the system-wide short-vs-FQDN equivalence rule, and
had no recovery path for hosts whose immutable agent ID still matched
but whose hostname had drifted since binding. Affected agents kept
reporting CommandsEnabled=true while every channel registration was
rejected, so fleets showed a permanent "Remote control blocked" chip
with reinstall as the only recourse (reported by a customer with a
large Docker fleet after upgrading to v6.1.2).
- Single-source the binding decision in evaluateAgentExecBinding; both
admitAgentExecToken and commandConfigAllowedForToken now consume it,
so the config payload can never advertise command execution that
admission would reject.
- Treat the immutable machine-derived agent ID as the primary binding
identity: an exact ID match re-binds a drifted (renamed) hostname in
place instead of stranding the host; hostname match alone still fails
closed for version-2 bindings.
- Compare hostnames with unifiedresources.HostnamesEquivalent (plus
case-insensitive exact match for IP literals) across admission,
session validation, and legacy migration, so docker01 vs docker01.lan
no longer splits the decision.
- Stop treating a miss on the token-scoped connectivity lookup as
authoritative in the connections ledger: host.TokenID is sticky
across token rotation/revocation, and a shared token fronting more
than one live session fails closed in the token lookup, so fall
through to the agent-ID and hostname lookups before reporting an
enabled host as blocked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add external_probe to the licensing feature enumeration, Pro tier
membership, self-hosted feature catalog, feature map, upgrade matrix
and pricing handoff, with the entitlement contract goldens updated.
The capability is served by the community binary - the entitlement
alone gates it, like relay. No existing free functionality changes:
local availability checks stay ungated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d235aab3c put MigrateFromFiles on a production path for the default org and
failed NewSQLiteManager whenever the import was rejected. That propagated to
GetManager, so every RBAC route returned 503 including ResetAdminRole, which
is the operator's only way back. The realistic trigger is a v5 install that
recreated a role the v6 store already holds under the same ID, so an
ordinary upgrade could leave an operator with no route to repair it short of
hand-editing the legacy JSON.
Rejecting the import is right and stays: importLegacyRBAC is transactional
and the legacy files are left in place, so a failure leaves the database
un-migrated rather than half-migrated. Missing roles deny access rather than
granting it, which is why the store is safe to keep serving.
The store now stays live and records the failure on MigrationError. The
deliberate fail-closed behaviour of the management surface is preserved
rather than removed: the handler manager accessors surface the migration
failure as the same 503 rbac_store_unavailable as before. Recovery reaches
the provider directly, so it is exempt by construction.
The existing 503 contract test is what caught the first attempt at this,
which simply let the surface serve un-migrated data. It now additionally
asserts recovery is reachable, so the two halves are pinned together.
v6.1.0-rc.4 shipped the attention workbench gated on monitoring:read
only, so mobile relay tokens (which carry just relay:mobile-access) got
403 on alert sync, finding detail, and acknowledge the moment a server
upgraded to rc.4. The attention routes supersede the legacy patrol
findings routes, which accepted the mobile capability, so they join the
governed relay mobile runtime route inventory: reads accept
relay:mobile-access alongside monitoring:read (legacy ai:execute),
mutations alongside monitoring:write.
One vCenter connection spans many ESXi hosts, so the Infrastructure source
row now lists them the way Proxmox cluster rows list their nodes: an
expandable member list with per-host state, aliases, and last-seen. Members
are API-side composition only — no primary marker, no agent connection, and
the member subtitle reads 'vSphere host' instead of cluster-node wording.
TrueNAS keeps no member composition because that connection monitors exactly
one machine.
Mock mode now feeds its vCenter and TrueNAS fixtures into the connections
aggregator (only when no real instances are configured), so the mock ledger
shows the same platform source rows a real deployment would instead of
omitting vSphere and TrueNAS entirely.
vSphere ESXi hosts and TrueNAS boxes rendered as standalone rows under
'Pulse Agent hosts' in Infrastructure settings even though no Pulse Agent
runs on them. The rows were unmanageable duplicates of their owning platform
connection: they carry no credentials, pause, or remove semantics, and they
can never attach to their vSphere/TrueNAS system because attachment requires
a shared host while the machine hostname differs from the vCenter address.
They also inflated connected-system counts.
buildConnections now skips hosts with a non-empty IntegrationSource, so the
ledger and grouped systems represent those machines solely through their
owning platform connection. Per-machine visibility is unchanged on the
vSphere/TrueNAS platform pages and Machines, which read the unified fabric
directly. Connection.integrationSource stays declared as defense-in-depth
for agent-only client workflows. No alert impact: agent-type rows were
already dropped from alert snapshots.
The connections ledger derives agent rows from the unified fabric, which
includes machines whose telemetry comes from platform integrations (vSphere
ESXi hosts, TrueNAS). Agent Doctor rendered every one as a permanent
'Unknown / no structured reason' row, while agents the ledger does not carry
(Docker-only, Kubernetes-only) were silently dropped from the fleet view.
- Expose HostView.IntegrationSource() (source-set based: only SourceAgent
ingest counts, since integration providers fabricate an Agent payload) and
plumb it through models.Host to the connections ledger as the optional
integrationSource field.
- Agent Doctor skips integration-backed connections and appends
diagnostics-only agents, honoring scope, so the doctor covers exactly the
real Pulse Agent fleet.
- Update readiness agent checks no longer count integration-backed machines
as registered agents.
- Humanize doctor copy: plain-language stale message with '10m 2s'-style
durations, offline wording without enum leakage, no 'Supported target:
Unknown' cell when no target is published, host-local command banner only
when a command is actually offered, and a compact non-zero summary strip.
Contracts updated for unified-resources, monitoring, api-contracts,
agent-lifecycle, and dependent storage-recovery; verification via
views_test.go, monitor_host_agents_test.go, state_host_test.go,
contract_test.go, and the frontend connections API test.
Field report (Johannes Strasser, 6.1.0-rc.3 mobile chat). The assistant
could find an LXC container but told him it had no tools to restart it.
Relay-mobile tokens carry only the relay access scope, which does not
satisfy the execute-authority check, so mobile chat sessions were offered
read tools only. The gap was one-sided: mobile could already approve and
execute planned actions, it just could not propose them.
Relay-mobile callers now hold execute authority, and the chat entrypoint
pins their sessions to approval-required mode, so every mobile-originated
action lands on the approval queue regardless of the configured control
level. Web sessions are unchanged.
The Patrol finding-notification work (23695681a, e0e454abf) added
patrol_finding_notifications_enabled and
patrol_finding_notify_min_severity to the AI settings payload but did
not update the six JSON snapshot pins, leaving Build and Test red on
every push since. The fields are intended API surface; the snapshots
now include them.
Contract-Neutral: snapshot refresh for the intended Patrol finding-notification fields added in 23695681a/e0e454abf; no behavior change
Telemetry shows thousands of installs configure an AI provider but almost
none ever use the interactive Assistant. A live first-session exercise
(fresh install, Ollama qwen3:8b quickstart) found why: after enabling,
nothing changes on screen — the launcher and handoff buttons are gated on
sessionCapabilities.assistantEnabled, which was only read at page load;
the empty transcript was blank; and the blessed Ollama+qwen3:8b path
reported Patrol degraded while telling the user to pull the model they
had just selected.
- Setup-modal success now opens the Assistant drawer, and the AI settings
save paths refresh the assistantEnabled capability in place
(aiChatStore.refreshEnabledFromServer) so entry points appear without a
reload; toasts point at the Assistant instead of back at settings.
- The empty transcript owns a plain-language welcome and three suggested
prompts that dispatch as real turns (ASSISTANT_SUGGESTED_PROMPTS).
- Patrol static readiness: the blessed Ollama Patrol model is Ready;
other Ollama models keep the warning, now naming the selected model.
- pulse_summarize fleet argument errors instruct the model to enumerate
resources itself instead of interrogating the operator (observed live:
'how is my machine doing?' ended in a resource-ID elicitation).
Contracts: ai-runtime and frontend-primitives Current State updated.
Tests: full internal/ai + internal/api suites green; vitest ChatMessages,
AISettings, aiChat store, and settingsArchitecture suites green; flow
verified live end-to-end.
make lint (dupl) flagged two clone pairs: shouldRestartAIChat vs
aiSettingsUpdateTouchesPatrolReadiness shared the forty-field provider
config block, and the approved/rejected action decision-ID collectors in
Pulse Intelligence telemetry differed only by lifecycle state and audit
predicate.
Extract aiSettingsUpdateTouchesProviderConfig for the shared field
check and parameterize pulseIntelligenceActionDecisionIDs by state,
label, and audit predicate, keeping the two named wrappers. The
decision-telemetry contract tests now pin the wrapper call sites, which
still fix the lifecycle state and audit predicate per decision kind.
Contract-Neutral: behavior-preserving dedup of internal helpers to clear dupl lint findings; no public-contract delta
098ba4eaa changed plan resource versions to hash relationship identity
instead of observation stamps, and updated the planner unit tests, but
missed the pinned API contract snapshot: its fixture carries a vm-to-node
relationship, so the resourceVersion (and the actionId, decisionId, and
planHash derived from it) legitimately changed. TestContract_ActionPlanJSONSnapshot
has been red on main since that commit; this re-pins the snapshot to the
deterministic values the new hashing produces.
v6.1.0-rc.1 retired the legacy update endpoints before a replacement
existed, so the UI's Update button failed with an internal-jargon 410
(issue #1564). This lands the replacement end to end: update_container
is a typed agentexec operation with its own strict codec, durable
receipts, and a request digest bound to the image digest the plan
observed; the unified agent bridges execution to the Docker module's
existing pull/backup/recreate/verify/rollback implementation (which now
reports rollback attempt and outcome); and the container action
executor plans, dispatches, and reconciles the operation with declared
backup/rollback compensation truth. Containers advertise an
admin-approval update capability while an image update with a stated
current digest is detected. The legacy endpoints stay retired but
return actionable copy.
Proven live against a Colima daemon: single-container update, the
issue-1564 shared-network-namespace update, and the full UI journey
(Update button, governed review, approve, run) all completed with the
namespace preserved and the backup retained.
Richard Courtman