Commit Graph

3825 Commits

Author SHA1 Message Date
courtmanr@gmail.com 0ff0a37f8d Use server time for probe freshness 2026-07-29 21:28:26 +01:00
courtmanr@gmail.com a53d45e2d3 Harden external probe outage alerting 2026-07-29 20:09:23 +01:00
courtmanr@gmail.com f4dd8096aa Prepare v6.2.0-rc.4 release
Fourth candidate on the v6.2.0 line, superseding v6.2.0-rc.3 and still following stable v6.1.2. This hardening cut includes action recovery, kiosk containment, alert and notification accounting, node-restricted Proxmox storage, container-runtime selection, and host CPU sampling fixes that landed after rc.3. Version pins move to 6.2.0-rc.4 across the repo root, Docker bootstrap defaults, and Helm metadata; stable install pointers remain on v6.1.2 until governed promotion.
2026-07-29 14:49:51 +01:00
courtmanr@gmail.com 848e166f5d Fix alert and notification telemetry signals 2026-07-29 14:17:19 +01:00
rcourtman 000986f125 Keep kiosk sessions off settings surfaces and silence the truncated-status security banner (#1650)
Kiosk mode is frontend-only and a kiosk API token carries monitoring:read
alone, but the shell treated kiosk as a display preference rather than an
authority fact. Pressing Escape on a kiosk display revealed the full header,
the navigation, and a security-warning banner offering links into Settings.

Three separate bugs produced that:

Escape called toggleKioskMode() unconditionally. setKioskMode(false) writes
'false' to sessionStorage, so the exit was sticky for the rest of the browser
session and the auto-enable effect never re-armed. Escape now exits kiosk only
for a session that can reach settings; a scope-limited session gets the
temporary header reveal the hover and touch affordances already provide.

The redirect that keeps sessions off /settings, /patrol and the
alert-configuration tabs ran only while kiosk was on, so once Escape had turned
kiosk off a kiosk-token session could navigate into Settings. It now also keys
off whether the session's token can read settings, so it holds with kiosk off.
The panels themselves already 403 server-side; this is UI containment, not an
auth change.

/api/security/status truncates its payload by authority and names the tier in
detailLevel. A token without settings:read gets an 'authenticated' payload with
no posture fields at all, which the banner read as disabled controls and turned
into a false security score with an "Enable Security" link. The visibility
helper now refuses to assert anything the payload does not carry: an
authenticated payload raises nothing, a public payload is authoritative for
hasAuthentication only so a genuinely unauthenticated instance is still warned
about, and a privileged payload keeps the full assessment. The remaining global
banners deep-link into settings too, so that block is gated on the same
settings-access predicate.

Scope membership is now decided once by sessionHasSettingsAccess in
AppLayout.tsx and shared with App.tsx instead of being re-derived inline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 14:06:50 +01:00
courtmanr@gmail.com 8aac88588c Classify the action force-fail route in the mutation registry (#1649)
The Canonical Governance mutation registry audit scans the real router
sources and fails closed on any infrastructure route it cannot resolve to
a registry disposition, so the new POST /api/actions/{id}/force-fail
route broke the audit on main. Register it as the lifecycle entry
action.api.force-fail, executed by internal/actionlifecycle.Service.ForceFail
under the same execute_action capability, admin approval floor, and
committed-lifecycle delivery as execute, and classify the route in the
runtime surface catalogue.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 12:33:49 +01:00
Richard Courtman ccea5ba761 Fail actions stranded without agent completion evidence and add force-fail (#1649)
An action that reached executing had exactly one way out: a terminal
operation receipt from the agent. Expiry skips executing rows on purpose,
and reconciliation bailed for every non-terminal query answer, so a Docker
update whose agent restarted mid-pull sat in executing forever with no
operator route out of it.

The agent-side receipt store rewrites accepted and started receipts to
interrupted on every Open(), and nothing can move an interrupted or
tombstoned receipt back to terminal. An identity-correlated answer of
either kind is therefore proof the operation will never report, and the
three executors now settle the action immediately as inconclusive with a
message telling the operator Pulse cannot confirm the effect and the
resource needs checking by hand.

A not_found answer, or a receipt still merely accepted or started, may
still be completed, so those keep waiting and only settle once the
dispatch attempt is older than one hour, the same threshold the
pulse-intelligence telemetry already uses to call an executing action
stuck. Every typed operation timeout is far shorter than that, so an
in-flight mutation is never cut short. A transport error answering the
query is still not evidence and preserves receipt_pending unchanged.

RecoverExecutingActions drives all of this on the existing two-minute
recovery loop, so rows already wedged before the upgrade heal themselves
without anyone touching them.

For the residue that reconciliation cannot reach, an agent that was
reinstalled or a legacy executing row with no dispatch attempt, POST
/api/actions/{id}/force-fail writes the same inconclusive terminal truth
under an operator attribution. It never touches the transport, refuses
anything already terminal, and is gated on admin plus settings:write on
top of the execute capability check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 12:14:28 +01:00
Richard Courtman 97c5449bef Fix storage listed on nodes excluded by its node restriction (#1645)
Proxmox's per-node storage endpoint GET /nodes/{node}/storage does not
return what the node can actually use. It returns every storage in the
datacenter config and reports the ones the node is excluded from with
enabled:0/active:0 instead of leaving them out. pollStorageWithNodes
ingested every row it got back, so a storage restricted to one node
appeared on all the others as a disabled entry, which the UI renders as
Offline.

The poller already fetches the datacenter config once via GetAllStorage
and keeps it keyed by storage name, so the restriction is in hand at the
point each per-node row is turned into a models.Storage. Skip the row
when the config carries a non-empty nodes restriction that does not
include the node being polled. Node names are compared case-insensitively
to match how node identity is compared elsewhere in the poller.

Only the restriction drops a row. A storage with no restriction still
shows up on every node it is reported from, including when it is disabled
everywhere, so a datacenter-wide disabled storage stays visible as
disabled rather than vanishing. Because the shared-storage aggregation
builds Nodes/NodeIDs/NodeCount from the surviving per-node rows, this also
stops a restricted shared storage from claiming cluster members that
cannot mount it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 11:45:56 +01:00
courtmanr@gmail.com c7d7c3abea Add stranded pulse-intelligence gate section to the high-risk matrix
The pulse-intelligence-rg-01-rg-12 release gate was registered in
status.json but its matrix section only existed in a never-pushed
v6.1.3 packet staged in a local checkout, leaving
documentation_currentness_test failing on every branch since. Salvaged
verbatim from that staged edit; the gate's latest exercised record and
block conditions are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 23:45:24 +01:00
courtmanr@gmail.com e734bd0097 Fix transient rootless Podman socket outranking rootful Docker (#1647)
The installer's discover_rootless_container_runtime only globbed
/run/user/* sockets and never consulted the system Docker daemon, so a
socket-activated rootless Podman API socket (alive only for root's login
session on Debian/OMV) won discovery over a healthy rootful Docker and
pinned PULSE_DOCKER_RUNTIME=podman plus CONTAINER_HOST/PODMAN_HOST/
XDG_RUNTIME_DIR into the agent unit. The env-application block also ran
for explicit --enable-docker installs. Rootless discovery now defers to
system_docker_runtime_is_active (docker info with DOCKER_HOST stripped,
or a live /var/run/docker.sock probe) before touching rootless sockets.

On the agent, detectRuntime short-circuited to podman whenever the
preference said podman, mislabeling connections that actually fell
through to the Docker socket and disabling Swarm collection. The
preference is now an ordering hint: a podman-preferred connection landing
on a docker endpoint reports docker, while unlabeled endpoints with no
runtime signals still honor the pin. When the bound socket disappears
mid-run the agent re-runs runtime discovery after three consecutive
daemon-unavailable collects, swapping the connection behind a
swappableDockerClient so concurrent goroutines keep a stable handle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 22:49:48 +01:00
courtmanr@gmail.com ad97ae890d Backfill agent-lifecycle governance for the marshal-hook race fix
68e557e9f moved the Docker agent's JSON-marshal hook from a package global
to a per-Agent field, which touches internal/dockeragent/agent.go and
internal/dockeragent/container_update.go. It meant to take the
Contract-Neutral bypass, but a blank line separated the trailer from
Co-Authored-By, so git's trailer parser dropped it, CI ran the completion
guard without a reason, and Canonical Governance went red on three counts:
missing contract docs/release-control/v6/internal/subsystems/agent-lifecycle.md,
missing verification artifact for "agent runtime transport trust proof", and
missing verification artifact for "Docker container recreate, rollback,
durable result, and live network-mode proof". The sibling timer-hook commit
8e5ef365d has the same defect, so this backfill covers both seams.

Document the seams in the agent-lifecycle contract's Current State: both
newTimerFn and jsonMarshalFn are unexported Agent fields reached through the
newTimer and jsonMarshal methods, nil falls back to time.NewTimer and
json.Marshal, no package-level hook global remains in internal/dockeragent,
and injection happens at construction only so the fields need no mutex. The
seams stay internal — no enrollment, transport trust, command admission,
acknowledgement, or recreate/rollback semantics move, and they must not be
promoted into NewAgent options or any server-facing surface.

Pin that shape in the two registered verification artifacts.
agent_internal_test.go asserts the field names, types, and unexported-ness,
parses every non-test source in the package to fail if either hook returns as
a package-level var, proves the nil defaults run the standard library, drives
two Agents with different hooks concurrently so the isolation is checked under
-race, and proves sendCommandAck marshals through the receiver's own seam.
container_update_test.go proves decodeUpdateContainerPayload routes through
a.jsonMarshal, that an injected failure never reaches a sibling Agent, and
pins the method form at compile time.

Guard dry-run over the staged set passes with no Contract-Neutral bypass, and
replaying 68e557e9f's file list plus these three files is green, so the
original commit would have passed had it carried them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 22:33:55 +01:00
courtmanr@gmail.com d1ee6e928b Fix multi-organization Proxmox connection identity 2026-07-28 17:15:54 +01:00
rcourtman 16ff5544c1 Prepare v6.2.0-rc.2 release
Second candidate on the v6.2.0 line, superseding v6.2.0-rc.1 and still
following stable v6.1.2. This is a hardening and bugfix candidate:
auto-update reliability including in-place migration of already-deployed
update units, PBS backup attribution across multiple Proxmox clusters,
Proxmox installer registration including per-canonical-type bootstrap
grants on combined PVE+PBS hosts, Patrol readiness streaming transport
plus a completed-verdict readiness gate, discovery-policy DNS and SSH
retry churn, explicit per-metric threshold off toggles, request-derived
SSO callback URLs, and the new Entra ID SSO guide. Version pins move to
6.2.0-rc.2 across the repo root, Docker bootstrap defaults, and Helm
metadata per the deployment-installability contract; stable install
pointers remain on v6.1.2 until governed promotion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 16:05:56 +01:00
courtmanr@gmail.com 72599bd1ec Allow one Proxmox bootstrap per canonical type on combined hosts (#1644)
A host running both PVE and PBS is a deployment the docs call officially
supported, and the agent's RunAll registers each product in turn from the
one install token. The bootstrap grant recorded consumption per TOKEN, so
the PVE leg spent it, the PBS leg came back canRegister=false, the agent
wrote a proxmox-pbs-registration-blocked marker, and install.sh printed an
ERROR banner over a PVE source that had registered perfectly well.

- server: consumption is now recorded per canonical type. One PVE create
  and one PBS create per token, each still one-shot — a second create of
  the same type takes the same 403. The bounds that are not per type stay
  singular: the 24h mint-age clock and the first-use bound_hostname are
  shared, so whichever type registers first pins the hostname for both and
  the second type cannot be aimed at another machine. The per-type ledger
  lives in proxmox_registration_consumed_types; a record carrying only
  proxmox_registration_completed=true predates it and still reads as every
  type consumed, so upgrading cannot revive a token already spent in the
  field. The unsuffixed completion block keeps tracking the most recent
  completion, which also leaves an older binary reading the same store
  failing closed.

- rollback: the consume-before-persist undo is scoped to the keys one
  consumption wrote, so a failed PBS source save restores the PBS grant
  without resurrecting the PVE grant that already produced a source.

- agent: RunAll no longer lets one product's failure speak for the host. It
  attempts and returns the remaining products, errors only when every
  detected product failed, and publishes the detected products in a
  proxmox-detected-types state marker.

- installer: report_proxmox_registration_outcome reads that marker, waits
  for an outcome from each detected product, and prints a success or denial
  line per product instead of one verdict. Agents predating the marker keep
  the old first-outcome-wins timing so a single-product host does not wait
  out the window. The blocked-marker path now only fires on genuine refusals.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 14:03:55 +01:00
courtmanr@gmail.com 4d972a68ee Gate Patrol readiness on the completed overall verdict (#1640)
The "Patrol tools" readiness check read the cached model-readiness
snapshot's tool-protocol dimension on its own. Since the interrupted-run
handling landed (8d0d74e35, b78330405), a run cancelled after every tool
scenario already passed keeps ToolProtocol at pass while the overall
status reports not_assessed, so the check reported "Patrol ready" from an
evaluation that never completed.

The check now requires the snapshot's own overall verdict (Success)
before reporting ready. A snapshot carrying no verdict at all — overall
status not_assessed, or the interrupted or internal_error cause — is not
turned into a failure either: it falls back to the base-config classifier
exactly as an absent snapshot does, capped at a warning. That cap matters
because not_ready is a blocking status in this payload: it clears
readiness.ready, which disables the Patrol run control in
usePatrolIntelligenceState and drops the page into the setup-only view.
#1640 promises a severed or cancelled check never blames the model and
never blocks Patrol from running in Watch mode, and the runtime gate on
POST /api/ai/patrol/run (PatrolRuntimeReadiness) already treats an
unassessed mode as a warning, so a blocking tools check would have
contradicted the route that actually runs Patrol.

A completed run whose tool protocol passed while the overall verdict fell
short now warns instead of claiming ready. It must not block either: the
dimension that actually failed carries the verdict on its own check
(context quality blocks, latency warns), so blocking here would have
turned today's latency warning into a hard stop.

Regression tests: internal/api/issue1640_readiness_gate_test.go covers
the gate across interrupted, internal-error, completed-pass,
completed-fail, and short-of-pass snapshots, asserting the resulting
runnability of the readiness payload;
internal/ai/issue1640_readiness_gate_test.go drives a real evaluation
that is cancelled at the continuation probe to produce the
ToolProtocol=pass / status=not_assessed snapshot end to end and pins that
PatrolRuntimeReadiness keeps Patrol runnable. The new API test file is
registered in the subsystem verification registry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 13:57:48 +01:00
courtmanr@gmail.com a92825b9db Derive SSO callback URLs from the request when no public URL is set
The SSO settings panel presents the OIDC Callback / Redirect URL and the
SAML SP metadata and ACS URLs as the values to register with an Identity
Provider. When PULSE_PUBLIC_URL was unset, providerToResponse built them
on a hardcoded http://localhost:7655, so an admin copied a localhost URL
into their IdP and got an opaque failure there with nothing pointing back
at Pulse as the cause.

The base URL is now resolved from the configured public URL when set —
still authoritative — and otherwise from the inbound request, which by
construction arrived over an address that reaches Pulse. This follows the
pattern buildSSOOIDCCallbackURL already used for the live OIDC login
flow, and the frontend already used for the SAML SP metadata preview
(window.location.origin). The scheme/host derivation is factored out of
that builder into requestForwardedScheme, requestForwardedHost and
requestOriginBaseURL on router.go, so forwarded headers stay behind the
same trusted-proxy gate; buildSSOOIDCCallbackURL's output is unchanged.

When neither source resolves a host the fields are now omitted rather
than carrying a wrong absolute URL. The panel renders guidance pointing
at the public URL setting instead of a copy button, for both the OIDC and
the SAML blocks. The add-provider modal also no longer claims the URL
will be "shown here" after save — the modal closes on save, so it now
says to copy it from the provider card.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 13:52:39 +01:00
courtmanr@gmail.com 893aa0b2cd Clear temperature SSH failure backoff on system-settings save (#1638)
The reset added in b45bd66b9 only fired when the temperature SSH key
file on disk changed (mtime/size). An operator who repairs SSH access
any other way — fixing authorized_keys on the host, repairing
known_hosts, restoring network reachability — still waited out a
backoff window that may have compounded toward fifteen minutes.

A system-settings save is the natural operator touchpoint after such a
repair, so the settings handler now fans ResetSSHFailureBackoff out to
every live tenant monitor after a successful save, clearing the
per-host temperature SSH backoff and the knownhosts keyscan backoff.
The reset touches in-memory retry timing only; nothing is persisted
and no request field controls it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 13:28:55 +01:00
courtmanr@gmail.com 71d53a37aa Fix CI fallout from the readiness hardening
Two failures on c3fb35c8f. The subsystem lookup unit tests hold their own
hardcoded copies of the settings-shell-and-framing exact_files list, a
third snapshot surface beyond the guard test's, and registering the new
banner proof file left them stale. Synced all eight copies.

The apiClient message-precedence change broke a pre-existing pin that a
short plain-text body outranks the caller fallback on retryable errors
(useReportingPanelState). Body-over-fallback was the long-standing
behavior; what #1640 actually required was dropping markup and oversized
bodies, which stays. Restored body-wins for sane plain text, flipped the
two precedence tests introduced alongside the change, and corrected the
cloud-paid contract paragraph to describe the real order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:53:58 +01:00
courtmanr@gmail.com c3fb35c8f8 Harden Patrol readiness streaming transport (#1640)
Follow-up to 8d0d74e35. The keepalive mechanism was right, the edges
were not.

1. The evaluation ran on a bare goroutine with no recover, so a panic in
   provider streaming or validation took the whole Pulse process down.
   Before that commit the same panic was on the request goroutine and the
   recovery middleware turned it into a logged 500. The goroutine now
   recovers, logs the panic with its stack, and answers with an ordinary
   readiness result carrying the new internal_error cause and every
   dimension reported as not assessed. A Pulse defect is not a model
   verdict.

2. Headers were only Set, never committed, despite the comment, the
   commit message, and api-contracts.md all claiming otherwise. The
   status line went out with the first keepalive at +10s, so a proxy
   with a sub-10s time-to-first-byte budget still severed the request.
   The transport now writes and flushes WriteHeader(200) before the
   ticker starts, matching the pattern the file already uses for SSE.

3. The flusher was resolved with a discarded ok, so a writer that is not
   an http.Flusher silently buffered the keepalives and degraded back to
   the original bug. It is now checked and logged; the response still
   completes, so a warning is the right level here rather than the hard
   failure the SSE handlers use.

4. TestIssue1640HandlerUsesKeepaliveTransport grepped the handler source
   for substrings, which proves nothing about behaviour. Replaced with a
   real httptest.NewServer test that runs a 300ms evaluation and asserts
   the client sees the 200 and a body byte before the evaluation
   completes, and that the padded body still parses as the expected JSON.
   Added coverage for the panic path and the non-flushable writer, and
   fixed the eager body[:1] that would panic when a transport regression
   left the body empty.

5. The settings readiness banner had no not_assessed branch, so an
   interrupted run still rendered the red "Patrol model not verified"
   headline: the exact blame-the-model presentation the backend fix
   removed. Tone and headline are now exported pure functions with a
   neutral treatment for not_assessed and interrupted results, and an
   interrupted run cannot claim verification from a max_verified_mode
   recorded before the cancellation.

6. createAPIErrorFromResponse let a short plain-text body override an
   explicit caller fallbackMessage. A caller passing a fallback knows
   which operation it was performing; an intermediary writing the body
   does not. Precedence is now canonical JSON, then caller fallback,
   then body, with the HTML and oversize suppression unchanged.

7. patrolRunCancelled classified on the raw "context canceled" substring
   as its first switch case. Ollama embeds that phrase in its own error
   body when it aborts an upstream request, so a genuine provider
   failure on a healthy run was classified interrupted and finish()
   persisted it as not_assessed. Cancellation is now established from
   the run itself (errors.Is(err, context.Canceled), or a cancelled run
   context), never from error wording, and the readiness paths classify
   through a context-aware entry point. context.DeadlineExceeded keeps
   its provider-path timeout classification.

The readiness gate in HandlePatrolModelReadiness keying off ToolProtocol
alone is untouched, as agreed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:17:25 +01:00
courtmanr@gmail.com b45bd66b94 Route discovery policy DNS through the cached resolver (#1638)
The discovery-policy check resolved endpoint hostnames with a bare
net.LookupIP while the actual dials went through pkg/tlsutil's process-global
cached resolver, so the policy and the connection reasoned about two different
DNS views. That split is why 108aa4e20 had to skip resolution entirely for the
default policy, which left the injected 169.254.0.0/16 blocklist enforced only
against literal IPs: a hostname endpoint pointed at the metadata range walked
straight through.

Resolve through tlsutil.LookupHostCached instead. The shared resolver caches
answers and lookup failures alike until its next refresh, so repeat poll cycles
cost a cache hit rather than a query and the per-poll DNS volume that opened
#1638 stays gone. With that in place the default-policy skip is removed and the
blocklist applies to resolved addresses again, and the five-minute decision
cache is dropped rather than kept: it bought nothing on top of the resolver
cache, made the verdict trail the configuration, and memoized the fail-open
"resolution failed, allow" outcome for minutes even with an explicit allowlist
configured. Its claim to match a DNS refresh interval that operators configure
through DNS_CACHE_TIMEOUT goes with it.

The SSH backoffs now only escalate for work that ran. A knownhosts manager
suppressing a call inside its own window reports ErrKeyscanSuppressed, and the
temperature layer neither records a failure nor pays for the RPi fallback in
that case. An expired collection deadline is our own budget rather than
evidence about the host, so it holds the window at the floor. Both backoffs
decay once a retry deadline is more than one window past, and replacing the
temperature SSH key on disk clears both maps so a repaired key is tried on the
next cycle instead of after fifteen minutes.

Refs discussion #1638.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:08:10 +01:00
courtmanr@gmail.com 7e4a4464a1 Tighten host-token Proxmox bootstrap grant (#1644)
Adversarial follow-ups to ac43506e6, which let a host-typed install token
bootstrap-create a Proxmox source. The one-shot machinery held up; these
are the four holes around it.

- test integrity: TestIssue1644HostInstallTokenGrantStaysHostnameBound
  completed a registration first, so the second request died at the
  completion gate and the bound_hostname comparison was never reached —
  the test passed with the binding deleted. It now binds without
  consuming (checkRegistration), rejects a different serverName while
  the grant is still live, and then completes on the bound hostname to
  show the grant was never the reason for the rejection.

- grant TTL: install tokens are minted with no expiry, so every host
  install token on a Proxmox box carried a live create-a-source
  capability forever. The grant now expires 24h after mint on its own
  clock (install_issued_at stamped at mint, falling back to the record's
  CreatedAt, failing closed with neither). Expired grants take the same
  403 path with a distinct warn.

- replay window: SaveNodesConfig ran before the grant was consumed, so a
  persistently failing token store left a source on disk next to an
  unconsumed grant — a repeatable create-N-sources primitive. The grant
  is now consumed and persisted first, and a failed source save rolls
  the consumption back, so either both stores advanced or neither did.

- exec binding: auto-register writes bound_hostname with no
  bound_agent_id and no binding version, which is exactly the shape
  canBindAgentInstallExecToken refuses, so host-token command enrollment
  was being admitted by the legacy pre-v6.1.1 migration branch. That
  record shape is now handled explicitly as a clean first use (hostname
  equivalence required), and a bound_hostname written by registration is
  no longer overwritten by an equivalent spelling the agent reports,
  because the still-unconsumed grant compares against it.

Single consumption across types is unchanged: a combined PVE+PBS host
still gets exactly one grant.

Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus TTL and exec-first-bind contract pins in internal/api/contract_test.go.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:07:11 +01:00
courtmanr@gmail.com 55048fb181 Harden PBS backup attribution against shared sources (#1639)
Adversarial review of 84dba861b found three ways the new evidence paths
could still attribute a snapshot to the wrong cluster.

The submission-source learner was asymmetric. Clusters only became known
to it through snapshots that were already attributable, so a cluster with
no uniquely-attributable snapshot was invisible - and a source token both
clusters share then mapped to exactly one visible cluster and looked
decisive. The visible cluster got the other's backups while the other
guest stayed at zero. Callers now declare every connection owning a
candidate guest for a PBS instance, and the learner refuses to resolve
anything for that instance until each of them has had a snapshot
attributed to it. Observation is not scoped per PBS instance, so a
cluster seen submitting to its own PBS server still counts as visible -
the reported two-server topology keeps working.

PVE storage confirmations were treated as authorship. A pbs-type storage
listing proves the connection can SEE a snapshot, which a shared token, a
synced datastore, or an offsite copy all arrange without the connection
having made it, and a single confirmer previously outscored everything
else. Confirmations now carry the storage they came from, and only a
storage view that never lists a snapshot some other connection also lists
can attribute a colliding VMID. An overlapping view has demonstrated it
sees other clusters' snapshots, so nothing it lists attributes anything.
Where an exclusive view and the learned source mapping both speak they
must agree, otherwise the snapshot drops as it did before #1639. The
disjoint case - each cluster mounting only its own datastore - is
unchanged.

Confirmations were evicted by partial poll failures. A storage whose
content query failed contributed nothing, and the partial set overwrote
the previous one, flipping attribution between cycles. They now go
through the same per-storage preservation as storage backups.

Contract text calling the PVE listing "the only deterministic
attribution" is reworded to match the weakened semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:02:46 +01:00
courtmanr@gmail.com 7018e799de Fix off-toggle edge cases in threshold editors (#1642)
The per-metric Off toggles added in 085208a82 disagreed with the alert
engine in five ways. The engine disables a metric whenever its trigger is
<= 0, so a stored 0 (which an older help banner told operators to type) is
off, and an unset global default is off too. Edit-mode detection tested for
-1 exactly, so those rows rendered On while nothing was alerting, and the
global-defaults rows coerced an emptied input to 0, which disabled the
metric in the engine while the row still showed On. All of it now reads the
same <= 0 rule through isAlertResourceMetricOff in the shared resource
table model, and a cleared input is treated as mid-edit rather than as a
disable request.

Re-enabling was worse than cosmetic. The row editor seeds its state from
the defaults, so switching a metric back on wrote undefined, the save path
skipped it as unchanged, and the resource kept inheriting a globally
disabled default. Bulk edit had the same hole, where undefined means
unchanged, so re-enabling a metric across a selection was impossible. Both
now stage an explicit enabled threshold when there is nothing enabled to
inherit, and the row editor still clears the override when the inherited
default is already on.

The row editor's input saved and closed on blur, so tabbing to the Off
badge dismissed the editor before the badge could be reached. Blur now
ignores focus moving inside the same metric editor, which covers the badge
without changing mouse behavior.

Documents the off sentinel rule in the alerts contract, since these editors
are named there and the previous commit bypassed it as contract-neutral.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:58:57 +01:00
courtmanr@gmail.com 5c1c89a363 Guard auto-update asset staging and migrate stale update units
Adversarial review of 9db25ba60 found four residual defects in the
auto-update asset install, plus a doc line it left contradicting itself.

- install_auto_update_assets copied the bundled helper into the staged
  mktemp file with an unchecked cp, and both call sites invoke the
  function under `if !`, which suppresses errexit for its whole body. A
  failed copy (ENOSPC, EIO) fell through to
  configure_auto_update_script_repo, whose awk emits a lone GITHUB_REPO=
  line for empty input, so a shebang-less one-line stub replaced the
  working helper with a "script" that only ever exits 0 - silently
  disabling unattended updates. Check the cp, and refuse the swap unless
  the staged helper is non-empty and starts with #!.
- Both units were rendered with a bare truncating `cat > "$unit"` whose
  status was never checked, and the function's last statement is
  safe_systemctl daemon-reload, which returns 0 by design. A failing
  write truncated a working unit and still reported success. Render each
  unit to ${path}.tmp and commit it with a checked rename, so a failure
  leaves the installed unit byte-identical.
- The widened ReadWritePaths could not reach deployed boxes: the unit
  that grants the write access is itself the file that has to be
  rewritten, and on an existing install the sandbox running the
  installer excludes /etc/systemd/system and /usr/local/bin (EROFS). The
  Go update pipeline cannot carry it either - pulse.service runs as
  User=pulse with its own ProtectSystem=strict over the install and
  config dirs only. So probe each destination directory up front and,
  when one is blocked, re-exec this already-signature-verified installer
  through systemd-run with a new internal --repair-auto-update-units
  entry point: PID 1 forks the transient unit, so it starts in the host
  mount namespace instead of inheriting the sandbox. The installer is
  copied into the install dir first because the calling unit's
  PrivateTmp=yes hides its /tmp copy from PID 1. The escape needs root
  and systemd-run, and never recurses.
- Keep the ReadWritePaths entries as directory grants: every write now
  commits with a rename from a sibling staging file, and rename needs
  write access on the containing directory, so the file-level entries
  systemd would otherwise accept cannot work. Document the tradeoff in
  the unit and the subsystem contract instead.

The deployment-installability contract still claimed the update sandbox
leaves "only the install dir, config dir and /tmp" writable, which the
paragraph the same file gained in 9db25ba60 contradicts; the same stale
rationale had been copied into two test comments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:56:52 +01:00
courtmanr@gmail.com 84dba861b5 Fix PBS backup attribution for VMIDs shared across clusters
v6.1.0's identity rework (eab73d245) made the VMID-only fallback refuse
to fire whenever a typed VMID exists on more than one PVE location.
Root-namespace snapshots with no matching comment then score zero for
every guest, so on setups with two clusters and overlapping VMIDs most
guests showed no backup at all - while PVE itself listed the backups
fine, because monitoring discards pbs-type storage contents entirely
whenever a direct PBS connection is configured.

Attribution is now evidence-driven instead of dropped:

Storage backup polling keeps a per-connection record of every snapshot
its own pbs-type storage listed (type, VMID, backup time) even though
the raw entries stay out of the PVE backup list. Which cluster listed a
snapshot is deterministic attribution, and it survives fully mirrored
clusters that share one datastore and token. The evidence is
monitoring-internal, cleared on instance retirement or when the storage
poll stops seeing pbs content, and never serialized into state payloads
or snapshots.

Guest backup-time sync and the recovery-point mapper additionally learn
each PBS submission source's cluster (owner token, datastore, PBS
instance - strongest first, scoped to the PBS instance) from the poll's
attributable snapshots, then resolve collision VMIDs whose snapshots
carry no evidence of their own. A source seen from several clusters is
not a discriminator, an unfamiliar component stops resolution rather
than deferring to weaker ones, and a snapshot decisively attributed to
another cluster is kept away from this one. Unattributable snapshots
still drop rather than guess.

Backup-age alert attribution no longer suffix-matches the subject ref's
connection label against guest locations. The label there is a PVE or
PBS instance name, not a PBS namespace, and loose matching could
cross-attribute clusters sharing a VMID; it now requires exact
normalized equality.

Reported in #1639 (two PVE clusters with PBS 4.0/4.1, VM 173 shown 974
days overdue despite valid verified backups).

Fixes #1639

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:24:01 +01:00
courtmanr@gmail.com ac43506e6e Fix Proxmox registration for host-token installs
The Settings > Infrastructure installer mints generic host install
tokens, but install.sh auto-detects Proxmox and the agent presents type
pve/pbs at /api/auto-register. The bootstrap grant required an exact
install_type match, so every generic install on a Proxmox node was
denied source creation and the denial was a single buried journal warn.

Four-part fix (#1644):
- server: extend the one-shot bootstrap grant to host-issued install
  tokens presenting a canonical Proxmox type. Typed tokens stay pinned,
  the grant keeps its settings-write mint requirement, first-hostname
  binding, serialized completion, and single consumption across types.
- agent: a canRegister=false denial now logs at error level, returns a
  setup error, and records the operator-facing reason in a
  proxmox-<type>-registration-blocked state marker.
- installer: report the Proxmox registration outcome in install output
  by reading the registered/blocked markers, and poll the server lookup
  for a bounded retry window before warning that registration was not
  confirmed (readyz flips before the first report cycle).
- setup script: the auto-register transport now captures the HTTP
  status alongside the body (no -f), making the invalid-setup-token
  branch reachable via 401/403 instead of a dead server-string grep,
  and operator guidance names Settings -> Infrastructure instead of the
  retired Nodes page (also updated in docs/PBS.md and the pinned
  assertions in contract, setup-script, and repoctl docs tests).

Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus new install.sh proofs for the retry window and blocked-marker
surfacing, and the updated hostagent blocked-registration test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:21:14 +01:00
courtmanr@gmail.com 8d0d74e35c Keep the Patrol readiness check alive through proxies and classify cancellation honestly
Fixes #1640. Three defects around POST /api/ai/patrol/readiness on slow
local hardware behind a reverse proxy:

1. The handler ran up to four sequential provider calls (~45s and more on
   slow Ollama boxes) while writing nothing to the response, so any
   intermediary with a ~30s read timeout severed the request mid-run. The
   handler now commits headers up front and streams flushed newline
   keepalives every 10s while the evaluation runs, then appends the normal
   JSON payload. Leading newlines are insignificant JSON whitespace, so
   existing clients parse the response unchanged.

2. A severed connection cancels the request context, and
   patrolRuntimeFailureFromError classified the resulting context.Canceled
   as a generic "Provider analysis error", blaming the provider and model
   for an infrastructure event. Mid-run cancellation is now classified as
   the new "interrupted" cause: the overall status and every unfinished
   dimension and autonomy mode report not assessed, per-scenario evidence
   completed before the interruption is preserved in the returned result,
   and the readiness cache keeps the last completed evaluation.
   context.DeadlineExceeded keeps its provider-path timeout classification.

3. createAPIErrorFromResponse pre-seeded the error message with the raw
   response body, making its non-JSON guard dead code, so full HTML proxy
   error pages became Error.message and were rendered into the readiness
   result boxes. Non-JSON bodies now surface only when they are short
   plain text; anything with markup or excessive length collapses to a
   generic status-derived message.

Regression tests: internal/ai/issue1640_readiness_cancellation_test.go,
internal/api/issue1640_readiness_transport_test.go, and
frontend-modern/src/utils/__tests__/apiClient.issue1640.test.ts, all
registered in the subsystem verification registry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:15:46 +01:00
courtmanr@gmail.com 193c96afc3 Remove dead autoUpdateCheckInterval and autoUpdateTime settings
Found while triaging #1643 and #1637: both fields were stored via
system.json and accepted, validated, and echoed by the settings API,
but nothing ever consumed them. The unattended update schedule is owned
entirely by the systemd timer install.sh renders, and no UI control ever
set the fields, so the API persisted a schedule preference that could
never take effect.

Remove the fields from config.Config, SystemSettings, the settings
handlers, and the frontend config type, along with the interval
validation and the .env AUTO_UPDATE_CHECK_INTERVAL rewrite (a legacy
line is now preserved verbatim). Legacy clients that still send the
keys get them silently ignored instead of validated, and a system.json
written before the removal still loads cleanly - both behaviors are
pinned by new tests. User docs no longer describe the phantom schedule
settings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:13:37 +01:00
courtmanr@gmail.com 9db25ba60e Fix doubled auto-update schedule and unfreeze sandboxed refreshes
Residual auto-update defects found while triaging #1643 and #1637
(the primary regression was fixed in 806cbe83d):

- The generated pulse-update.timer carried both OnCalendar=daily and
  OnCalendar=02:00, so with RandomizedDelaySec=4h every box attempted
  two updates per day (00:00-04:00 and 02:00-06:00 windows). Keep the
  single documented 02:00 schedule.
- The generated pulse-update.service sandbox (ProtectSystem=strict)
  excluded the helper and unit directories from ReadWritePaths, so the
  unattended path could never refresh /usr/local/bin/pulse-auto-update.sh
  or rewrite the units - updater fixes only reached boxes via manual
  installs. Grant the sandbox write access to both directories on
  purpose.
- Because the unattended path replaces the helper bash is currently
  executing, stage the new helper next to its destination and swap it
  in with an atomic rename only after repo configuration succeeds. A
  failed download or configure now leaves the previously working helper
  in place instead of rm -f'ing it out from under the enabled timer's
  ExecStart.
- Delete scripts/systemd/pulse-update.{service,timer}: orphaned
  reference copies that had drifted from the units install.sh actually
  generates and were referenced by nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:13:37 +01:00
courtmanr@gmail.com 108aa4e201 Stop discovery-policy DNS lookups and SSH re-execs on every poll cycle
The cluster-endpoint discovery-policy check ran a raw net.LookupIP per node
per poll cycle since c5f5af7ab, bypassing the process-global cached resolver,
and the injected default subnet blocklist (169.254.0.0/16) made the
zero-policy fast path unreachable so even unconfigured installs generated
that DNS volume. Evaluate the default link-local-only policy against literal
endpoint IPs without resolution, and memoize custom-policy verdicts per
endpoint for the shared 5-minute DNS-cache TTL so repeat polls stay off the
resolver. Also cache ssh-keyscan failures with doubling backoff in the
knownhosts manager and back off temperature SSH collection per host after
failures instead of re-executing ssh twice per node every 10s cycle.

Refs discussion #1638.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:07:19 +01:00
courtmanr@gmail.com 085208a824 Add explicit per-metric off toggles to alert threshold editors
Disabling one metric for one resource required knowing to type -1 into
the override editor (the alert engine treats trigger <= 0 as disabled),
and the global-defaults row could only re-enable a disabled metric by
click, not disable one. Add an On/Off badge next to the threshold input
in the override row editor (desktop and mobile), in the global-defaults
row (desktop and mobile), and in the bulk edit dialog, mirroring the
existing Backup/Snapshot StatusBadge pattern in the same rows. The
toggle writes -1; toggling back on restores the inherit-default state
in override editors and the metric's enabled default in global
defaults. In the desktop row editor the badge swallows mousedown so it
doesn't blur the input, which would save and close the editor before
the click lands.

Also fix the off-value inconsistency found while triaging #1642: the
thresholds help banner told users to type 0, which the edit-state off
detection (=== -1) never recognises. Standardize the advertised
disable value on -1, matching docs/FAQ.md and the engine's actual off
sentinel; read-mode cells keep treating <= 0 as Off so overrides saved
as 0 under the old banner still display correctly.

Closes #1642.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contract-Neutral: UI-only off toggle over existing -1 threshold semantics; no alert truth or identity change
2026-07-28 11:06:08 +01:00
courtmanr@gmail.com 48c269f58d docs: add Entra ID SSO and group mapping guide (#1635)
Expands the Azure AD provider example into a full Microsoft Entra ID walkthrough
contributed by @drgimpfen in #1635: app registration (single tenant), Web
platform redirect URI, Token configuration groups claim as Group ID, Enterprise
Application "Assignment required = Yes", and mapping roles from the group Object
ID so a rename in Entra doesn't break admin access. Corrects the contributed
draft to use the provider-scoped v6 callback path
(/api/oidc/<provider-id>/callback, not the legacy v5 /api/oidc/callback) and
adds the issuer URL and client ID/secret steps the draft omitted. Keeps the
draft's AADSTS650053 finding: Entra has no `groups` scope, so groups must come
from the ID token claim rather than the scope list.

Adds a warning the draft did not cover: on group overage Entra drops the groups
claim for a _claim_names/_claim_sources marker that Pulse does not follow, and
since a configured mapping is authoritative that login clears the user's roles.

Three pre-existing doc defects fixed alongside:

- OIDC.md claimed there is no environment-variable override for group role
  mappings, contradicting CONFIGURATION.md and RBAC.md.
  OIDC_GROUP_ROLE_MAPPINGS does exist (internal/config/oidc.go), it just only
  feeds the legacy env-configured provider.
- RBAC.md claimed logins with zero matching groups do not clear existing role
  assignments. applySSORoleAssignments runs with mappingAuthoritative=true
  whenever the provider has any mappings, so it calls UpdateUserRoles with the
  empty set and does clear them.
- Two RBAC.md links pointed at OIDC.md#group-to-role-mapping-pro, but the
  heading renders as #group-to-role-mapping-pro-and-above.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 10:43:27 +01:00
courtmanr@gmail.com 1dc19bfec0 Remove dead cache-aware RRD fields from the guest RRD path
Recorded PVE 8 and PVE 9 guest rrddata responses (fixtures under
pkg/proxmox/testdata/rrd/) prove guest RRD never carries the cache-aware
memused/memavailable columns — they exist only in node RRD — so every
consumer branch reading them was dead code that #1634's listing fallback
(7d7d2b6a3) had already routed around.

Drop the two fields from GuestRRDPoint (now time/maxmem only, matching
the recordings), delete the dead VM RRD memory fallback and its
getVMRRDMetrics/getVMRRDMemory helpers plus the vmRRDMemCache they fed,
remove the pointless per-poll guest RRD fetch from the LXC memory path,
and retire the guest RRD lookups from PVEClientInterface. VMMemoryRaw
loses its never-populated RRD diagnostic fields, and guest reliability
scoring no longer treats the node-only rrd-* sources as trusted guest
evidence. The knownDeadGuestRRDFields allowlist in the fixture
alignment test is gone; a new reflection guard in
code_standards_test.go keeps GuestRRDPoint pinned to recorded columns,
and cleanupRRDCache pruning of the guest-agent meminfo cache gains
direct coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 18:36:50 +01:00
courtmanr@gmail.com bb73ec6cbd Align agent command config gate with channel admission
v6.1.2 (c41edb65a) left the agent config gate and the command-channel
admission evaluating exec-token bindings under different policies: the
config gate admitted on bound-hostname OR bound-agent-ID while channel
admission required both to match, compared hostnames with plain case
folding instead of the system-wide short-vs-FQDN equivalence rule, and
had no recovery path for hosts whose immutable agent ID still matched
but whose hostname had drifted since binding. Affected agents kept
reporting CommandsEnabled=true while every channel registration was
rejected, so fleets showed a permanent "Remote control blocked" chip
with reinstall as the only recourse (reported by a customer with a
large Docker fleet after upgrading to v6.1.2).

- Single-source the binding decision in evaluateAgentExecBinding; both
  admitAgentExecToken and commandConfigAllowedForToken now consume it,
  so the config payload can never advertise command execution that
  admission would reject.
- Treat the immutable machine-derived agent ID as the primary binding
  identity: an exact ID match re-binds a drifted (renamed) hostname in
  place instead of stranding the host; hostname match alone still fails
  closed for version-2 bindings.
- Compare hostnames with unifiedresources.HostnamesEquivalent (plus
  case-insensitive exact match for IP literals) across admission,
  session validation, and legacy migration, so docker01 vs docker01.lan
  no longer splits the decision.
- Stop treating a miss on the token-scoped connectivity lookup as
  authoritative in the connections ledger: host.TokenID is sticky
  across token rotation/revocation, and a shared token fronting more
  than one live session fails closed in the token lookup, so fall
  through to the agent-ID and hostname lookups before reporting an
  enabled host as blocked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 17:45:34 +01:00
courtmanr@gmail.com 4ea99183fd Color platform table metric bars from configured alert thresholds
The Docker host, Docker container, Proxmox node, Kubernetes cluster and
node, TrueNAS system and app, and vSphere host tables rendered their
CPU/memory/disk bars without a thresholds prop, so coloring fell through
to the hardcoded METRIC_THRESHOLDS display defaults (memory 75/85). A
Docker host at 84% memory showed a red bar even when the operator had
raised its alert thresholds — the alert runtime resolved the override
correctly and stayed quiet while the row shouted.

Each table now resolves display thresholds through the alerts activation
store with its runtime scope (agent, docker, node, kubernetes, truenas,
vmware) and the resource's override identity candidates, mirroring the
AgentsMachinesTable pattern. The display resolver gains the kubernetes /
truenas / vmware scopes backed by kubernetesDefaults / truenasDefaults /
vmwareDefaults with factory fallbacks, and the unified platform
candidate chain is exported from alertOverridesModel so the tables bind
overrides to the same rows buildProjectedOverrides indexes.

The alertOverridesModel chunk folded into sharedPlatformPage now that
platform tables import it; bundle baseline regenerated within budget.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 17:45:18 +01:00
courtmanr@gmail.com 7d7d2b6a3b Restore LXC memory fallback to the cluster-resources listing
Real PVE guest RRD responses carry only the cache-inclusive mem/maxmem
columns; the cache-aware memused/memavailable columns exist only in node
RRD, so the LXC RRD branches can never match a live response. Removing
the cluster-resources fallback in bf67ba920 therefore left every running
LXC reporting unavailable memory, rendered as 0% (#1634).

Running containers now fall back to the listing value under the
low-trust cluster-resources source when cache-aware RRD evidence is
absent; unavailable remains reserved for running containers with no
listing evidence at all.

Fixes #1634

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 17:12:24 +01:00
courtmanr@gmail.com f45c08902e Prepare v6.2.0-rc.1 release
First candidate on the v6.2.0 line, following stable v6.1.2. Carries
the External Probes Pro feature, the three-part multi-site Proxmox
identity isolation work, cloned machine-id collapse detection,
unattended-update service recovery, the TrueNAS same-host redirect
follow, and verified telemetry outcomes. Version pins move to
6.2.0-rc.1 across the repo root, Docker bootstrap defaults, and Helm
metadata per the deployment-installability contract; stable install
pointers remain on v6.1.2 until governed promotion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 14:08:29 +01:00
courtmanr@gmail.com b200955ab8 Backfill monitoring contract for the TrueNAS handshake-redirect upgrade
bf24a9a9c changed truenas runtime transport behavior without the
canonical paperwork, so its governance run failed the completion guard.
Document the redirect rule in the monitoring subsystem contract — a
plaintext handshake redirect never wakes the REST bridge; a same-host
https redirect upgrades once to wss and the upgrade is scheme-only and
monotonic — and move the Issue1631 regression tests into
transport_test.go, which is a registered verification artifact for the
truenas runtime path policy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 13:16:52 +01:00
courtmanr@gmail.com 13371a6b6c Surface host-agent identity collapse from cloned machine-ids
Host agents key their identity on the machine-derived agent ID, so MSP
template deployments that clone /etc/machine-id fold two physical
machines at different sites into one host row whose reports overwrite
each other (hostname, report IP, and interfaces flapping between
sites), silently poisoning node-agent linking.

Mirror the Docker host identity-collapse doctrine (#1584) for host
report ingest: track hostname and report-IP revisits per resolved
agent identity inside the monitoring-owned flap window, publish an
active conflict as models.Host.IdentityConflict through unified
resources, and warn on the Machines page. The report IP is tracked
alongside the hostname because template fleets often reuse hostnames
across sites (pve01 at two customers), leaving the address as the only
field that betrays the clone. A one-time hostname rename never
revisits and is not flagged; the conflict clears on its own once only
one machine keeps reporting for the window.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 12:26:18 +01:00
courtmanr@gmail.com 17117c32ef Close the unclassified-node window in cross-instance node aggregation
Support evidence from the MSP case showed the clusters mix without ever
sharing a corosync name and without the config-layer consolidation
firing, which rules out the same-name path and points one layer down.
Two mechanisms combine there. A newly added connection whose add-time
cluster detection failed (#437) commits its first polls' nodes with an
empty cluster name, because pollPVEInstance ran membership detection
after the node-state commit - and every aggregation guard deliberately
lets empty cluster names merge freely. And host agents key their
identity on /etc/machine-id, which cloned template deployments reuse
across sites, so two different pve01 machines collapse into one agent
row whose shared LinkedAgentID then folds the unclassified node into
the established cluster's slot, overwriting it - the reported "enacon
appeared renamed to rewo" data loss.

Weak-evidence folds across connection instances - a bare-hostname
endpoint alias or a shared linked-agent identity - now require positive
same-machine proof (matching non-empty cluster identity or matching TLS
fingerprints) whenever cluster identity is in play on either side. Two
views that are both unclassified still dedup freely, and address-based
endpoint aliases keep folding on the contradiction checks alone, so the
designed standalone-into-cluster folds survive. PVE polling now also
runs cluster membership detection before the cycle's node-state commit
and re-reads the refreshed instance config, so nodes carry their
cluster identity from the first state write whenever detection
succeeds instead of transiting aggregation unclassified.

Reported via support by an MSP running clusters enacon and rewo that
reuse pve01/pve02 node names across sites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 12:07:44 +01:00
courtmanr@gmail.com 3efca89525 Document external probes and count adoption in telemetry
Add availability_probe_targets and availability_probe_agents to the
telemetry ping - counts only, no agent names or addresses - with the
disclosure table updated in both privacy doc copies. Document the
feature in the availability-checks configuration guide and the
unified agent guide, including the ICMP capability caveat for
containerized probes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 11:13:04 +01:00
courtmanr@gmail.com a5b3816fd3 Add Pro-gated probe assignment to availability targets
Availability targets gain an optional probe agent assignment. Setting
it requires the external_probe entitlement, enforced only at the
moment of assignment - local targets never consult the license path.
Assigned targets are delivered to their agent through the signed
agent-config channel, skipped by the local poller, and resume local
execution automatically if the entitlement lapses. Probe-reported
results are accepted only from the currently assigned agent, share
the local failure-threshold accounting, carry source attribution,
and derive to indeterminate at read time when reports go stale.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 11:13:04 +01:00
courtmanr@gmail.com 0966ae9594 Measure verified telemetry outcomes 2026-07-27 10:15:48 +01:00
courtmanr@gmail.com 07d2f98455 Gate same-name cross-instance node aggregation on TLS identity
The config layer already refuses to consolidate two same-named clusters
whose TOFU-captured TLS fingerprints contradict, and node aggregation
keeps any two same-named clusters from different connection instances
apart unconditionally. Two gaps remained one layer down. First, the
endpoint-IP agent match bypassed the contradiction guard entirely, so
two sites reusing RFC1918 addressing (the MSP support case: pve01 on
192.168.1.11 at both sites) still bound the second site's node to the
first site's host agent, attaching the wrong machine's telemetry.
Second, the unconditional split had no way to recognize the legitimate
duplicate - the same cluster added twice through different member
addresses with no config-level endpoint overlap could never fold back
into one node slot.

The aggregation layer now receives the config layer's identity
evidence: each PVE node carries the TLS certificate fingerprint of its
own named endpoint record (standalone nodes carry the instance
fingerprint; a cluster member never inherits the instance-level
fingerprint, which pins whichever member the connection URL reaches).
Same-named clusters from different instances merge only when both views
carry the same fingerprint; contradicting or unknown evidence keeps the
fail-safe split. Agent binding applies the identical doctrine: hostname
and address matches are rejected when the candidate agent's linked
nodes live in a different named cluster or carry a different
fingerprint, closing the previously unguarded endpoint-IP path.

Reported via support by an MSP whose sites reuse cluster names, node
names, and RFC1918 ranges.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 10:00:33 +01:00
courtmanr@gmail.com f21d05e3cb Veto PVE cluster consolidation on contradicting TLS fingerprints
Two clusters at different sites that reuse the same corosync cluster
name and the same RFC1918 addressing presented colliding member IPs,
which ConsolidatePVEInstances took as strong endpoint overlap and folded
the second site's connection into the first. The standalone-into-cluster
merge had the same hole for a standalone whose address collides with a
cluster endpoint at another site. The v6.1.1 node-aggregation guard
never fires in this case because it keys on cluster-name conflict, and
same-name clusters do not conflict.

Address coincidence is weak evidence across sites, but the TOFU-captured
TLS certificate fingerprints already stored on instances and cluster
endpoints are strong evidence: contradicting fingerprints for the same
authority, node name, or endpoint address mean different machines.
Consolidation now refuses to merge in that case. The fail-safe direction
is deliberate - a certificate rotation may leave a genuinely duplicated
cluster as two views, but two distinct clusters are never silently
folded into one.

Reported via support by an MSP whose two customer clusters kept merging
after the v6.1.1 aggregation fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:43:43 +01:00
courtmanr@gmail.com 806cbe83dc Fix auto-update leaving service down after installer failure
Two bugs from #1630 that combined to take installs down silently:

1. perform_update()'s install-failed rollback branch restored the backup
   but never restarted pulse.service. Since the generated
   pulse-update.service gates on ExecCondition=systemctl is-active,
   every later timer run was then skipped and the install stayed down
   until manual intervention. Restart is now guaranteed by a
   service_was_active-guarded restart in that branch plus an
   ensure_service_restarted RETURN trap so no exit path can miss it
   (re-fix of #1323, originally c0b3a0e66, lost in 778a2577b and only
   partially restored in 672e81985).

2. install.sh aborted under errexit when writing the /bin/update helper
   on a read-only filesystem - after the new binary was installed and
   the service stopped, landing in bug 1's no-restart branch. The stock
   pulse-update.service uses ProtectSystem=strict, so /bin and
   /usr/local/bin are read-only on stock unattended updates; transient
   read-only remounts hit the same path. The helper write, PATH
   appends, and the /usr/local/bin/pulse symlink are now idempotent and
   non-fatal with a warning (install_binary_symlink).

Contract: deployment-installability now pins fail-closed service
availability for unattended updates and non-fatal writes outside the
hardened unit's writable set, with proofs in pulse_auto_update_test.go
and root_install_sh_test.go plus shell regression coverage in
scripts/tests/test-pulse-auto-update.sh (installer-exits-nonzero path)
and scripts/tests/test-install-update-resilience.sh (read-only helper
and symlink paths, verified under set -e, root-safe via ENOTDIR).

Fixes #1630

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:38:37 +01:00
courtmanr@gmail.com fe33d23210 Document websocket lifecycle synchronization 2026-07-26 22:02:22 +01:00
courtmanr@gmail.com 81974c7f62 Authorize v6.1.2 unsigned Windows release exception 2026-07-26 21:07:47 +01:00
courtmanr@gmail.com a9d215401e Reset ephemeral Windows state before native lifecycle proof 2026-07-26 20:51:21 +01:00
courtmanr@gmail.com 9bb5a8bee1 Isolate Windows native lifecycle self-test state 2026-07-26 20:45:20 +01:00