Same class of race fixed for newTimerFn in 8e5ef365d: tests swapped the
package-level jsonMarshalFn hook while async goroutines leaked from
earlier tests (sendCommandAck ack retries via runAsync) could still be
reading it, tripping the race detector. Replace the global with a
per-Agent jsonMarshalFn seam (nil defaults to json.Marshal), make the
decode payload helpers Agent methods so they use it, and inject the
failing marshaller into the tests that previously swapped the global.
Verified with go test ./internal/dockeragent/ -race -count=20.
Contract-Neutral: test seam refactor to fix data race, no public contract delta
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tests swapped the package-level newTimerFn hook while async goroutines
leaked from earlier tests (backup-cleanup and stop-command paths) were
still reading it in waitForAsyncDelay, tripping the race detector on CI.
Replace the global with a per-Agent newTimerFn seam (nil defaults to
time.NewTimer), make waitForContextDelay an Agent method, and inject the
immediate timer into the tests that previously swapped the global.
Verified with go test ./internal/dockeragent/ -race -count=20.
Contract-Neutral: test seam refactor to fix data race, no public contract delta
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A host running both PVE and PBS is a deployment the docs call officially
supported, and the agent's RunAll registers each product in turn from the
one install token. The bootstrap grant recorded consumption per TOKEN, so
the PVE leg spent it, the PBS leg came back canRegister=false, the agent
wrote a proxmox-pbs-registration-blocked marker, and install.sh printed an
ERROR banner over a PVE source that had registered perfectly well.
- server: consumption is now recorded per canonical type. One PVE create
and one PBS create per token, each still one-shot — a second create of
the same type takes the same 403. The bounds that are not per type stay
singular: the 24h mint-age clock and the first-use bound_hostname are
shared, so whichever type registers first pins the hostname for both and
the second type cannot be aimed at another machine. The per-type ledger
lives in proxmox_registration_consumed_types; a record carrying only
proxmox_registration_completed=true predates it and still reads as every
type consumed, so upgrading cannot revive a token already spent in the
field. The unsuffixed completion block keeps tracking the most recent
completion, which also leaves an older binary reading the same store
failing closed.
- rollback: the consume-before-persist undo is scoped to the keys one
consumption wrote, so a failed PBS source save restores the PBS grant
without resurrecting the PVE grant that already produced a source.
- agent: RunAll no longer lets one product's failure speak for the host. It
attempts and returns the remaining products, errors only when every
detected product failed, and publishes the detected products in a
proxmox-detected-types state marker.
- installer: report_proxmox_registration_outcome reads that marker, waits
for an outcome from each detected product, and prints a success or denial
line per product instead of one verdict. Agents predating the marker keep
the old first-outcome-wins timing so a single-product host does not wait
out the window. The blocked-marker path now only fires on genuine refusals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Patrol tools" readiness check read the cached model-readiness
snapshot's tool-protocol dimension on its own. Since the interrupted-run
handling landed (8d0d74e35, b78330405), a run cancelled after every tool
scenario already passed keeps ToolProtocol at pass while the overall
status reports not_assessed, so the check reported "Patrol ready" from an
evaluation that never completed.
The check now requires the snapshot's own overall verdict (Success)
before reporting ready. A snapshot carrying no verdict at all — overall
status not_assessed, or the interrupted or internal_error cause — is not
turned into a failure either: it falls back to the base-config classifier
exactly as an absent snapshot does, capped at a warning. That cap matters
because not_ready is a blocking status in this payload: it clears
readiness.ready, which disables the Patrol run control in
usePatrolIntelligenceState and drops the page into the setup-only view.
#1640 promises a severed or cancelled check never blames the model and
never blocks Patrol from running in Watch mode, and the runtime gate on
POST /api/ai/patrol/run (PatrolRuntimeReadiness) already treats an
unassessed mode as a warning, so a blocking tools check would have
contradicted the route that actually runs Patrol.
A completed run whose tool protocol passed while the overall verdict fell
short now warns instead of claiming ready. It must not block either: the
dimension that actually failed carries the verdict on its own check
(context quality blocks, latency warns), so blocking here would have
turned today's latency warning into a hard stop.
Regression tests: internal/api/issue1640_readiness_gate_test.go covers
the gate across interrupted, internal-error, completed-pass,
completed-fail, and short-of-pass snapshots, asserting the resulting
runnability of the readiness payload;
internal/ai/issue1640_readiness_gate_test.go drives a real evaluation
that is cancelled at the continuation probe to produce the
ToolProtocol=pass / status=not_assessed snapshot end to end and pins that
PatrolRuntimeReadiness keeps Patrol runnable. The new API test file is
registered in the subsystem verification registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SSO settings panel presents the OIDC Callback / Redirect URL and the
SAML SP metadata and ACS URLs as the values to register with an Identity
Provider. When PULSE_PUBLIC_URL was unset, providerToResponse built them
on a hardcoded http://localhost:7655, so an admin copied a localhost URL
into their IdP and got an opaque failure there with nothing pointing back
at Pulse as the cause.
The base URL is now resolved from the configured public URL when set —
still authoritative — and otherwise from the inbound request, which by
construction arrived over an address that reaches Pulse. This follows the
pattern buildSSOOIDCCallbackURL already used for the live OIDC login
flow, and the frontend already used for the SAML SP metadata preview
(window.location.origin). The scheme/host derivation is factored out of
that builder into requestForwardedScheme, requestForwardedHost and
requestOriginBaseURL on router.go, so forwarded headers stay behind the
same trusted-proxy gate; buildSSOOIDCCallbackURL's output is unchanged.
When neither source resolves a host the fields are now omitted rather
than carrying a wrong absolute URL. The panel renders guidance pointing
at the public URL setting instead of a copy button, for both the OIDC and
the SAML blocks. The add-provider modal also no longer claims the URL
will be "shown here" after save — the modal closes on save, so it now
says to copy it from the provider card.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reset added in b45bd66b9 only fired when the temperature SSH key
file on disk changed (mtime/size). An operator who repairs SSH access
any other way — fixing authorized_keys on the host, repairing
known_hosts, restoring network reachability — still waited out a
backoff window that may have compounded toward fifteen minutes.
A system-settings save is the natural operator touchpoint after such a
repair, so the settings handler now fans ResetSSHFailureBackoff out to
every live tenant monitor after a successful save, clearing the
per-host temperature SSH backoff and the knownhosts keyscan backoff.
The reset touches in-memory retry timing only; nothing is persisted
and no request field controls it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to 8d0d74e35. The keepalive mechanism was right, the edges
were not.
1. The evaluation ran on a bare goroutine with no recover, so a panic in
provider streaming or validation took the whole Pulse process down.
Before that commit the same panic was on the request goroutine and the
recovery middleware turned it into a logged 500. The goroutine now
recovers, logs the panic with its stack, and answers with an ordinary
readiness result carrying the new internal_error cause and every
dimension reported as not assessed. A Pulse defect is not a model
verdict.
2. Headers were only Set, never committed, despite the comment, the
commit message, and api-contracts.md all claiming otherwise. The
status line went out with the first keepalive at +10s, so a proxy
with a sub-10s time-to-first-byte budget still severed the request.
The transport now writes and flushes WriteHeader(200) before the
ticker starts, matching the pattern the file already uses for SSE.
3. The flusher was resolved with a discarded ok, so a writer that is not
an http.Flusher silently buffered the keepalives and degraded back to
the original bug. It is now checked and logged; the response still
completes, so a warning is the right level here rather than the hard
failure the SSE handlers use.
4. TestIssue1640HandlerUsesKeepaliveTransport grepped the handler source
for substrings, which proves nothing about behaviour. Replaced with a
real httptest.NewServer test that runs a 300ms evaluation and asserts
the client sees the 200 and a body byte before the evaluation
completes, and that the padded body still parses as the expected JSON.
Added coverage for the panic path and the non-flushable writer, and
fixed the eager body[:1] that would panic when a transport regression
left the body empty.
5. The settings readiness banner had no not_assessed branch, so an
interrupted run still rendered the red "Patrol model not verified"
headline: the exact blame-the-model presentation the backend fix
removed. Tone and headline are now exported pure functions with a
neutral treatment for not_assessed and interrupted results, and an
interrupted run cannot claim verification from a max_verified_mode
recorded before the cancellation.
6. createAPIErrorFromResponse let a short plain-text body override an
explicit caller fallbackMessage. A caller passing a fallback knows
which operation it was performing; an intermediary writing the body
does not. Precedence is now canonical JSON, then caller fallback,
then body, with the HTML and oversize suppression unchanged.
7. patrolRunCancelled classified on the raw "context canceled" substring
as its first switch case. Ollama embeds that phrase in its own error
body when it aborts an upstream request, so a genuine provider
failure on a healthy run was classified interrupted and finish()
persisted it as not_assessed. Cancellation is now established from
the run itself (errors.Is(err, context.Canceled), or a cancelled run
context), never from error wording, and the readiness paths classify
through a context-aware entry point. context.DeadlineExceeded keeps
its provider-path timeout classification.
The readiness gate in HandlePatrolModelReadiness keying off ToolProtocol
alone is untouched, as agreed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The discovery-policy check resolved endpoint hostnames with a bare
net.LookupIP while the actual dials went through pkg/tlsutil's process-global
cached resolver, so the policy and the connection reasoned about two different
DNS views. That split is why 108aa4e20 had to skip resolution entirely for the
default policy, which left the injected 169.254.0.0/16 blocklist enforced only
against literal IPs: a hostname endpoint pointed at the metadata range walked
straight through.
Resolve through tlsutil.LookupHostCached instead. The shared resolver caches
answers and lookup failures alike until its next refresh, so repeat poll cycles
cost a cache hit rather than a query and the per-poll DNS volume that opened
#1638 stays gone. With that in place the default-policy skip is removed and the
blocklist applies to resolved addresses again, and the five-minute decision
cache is dropped rather than kept: it bought nothing on top of the resolver
cache, made the verdict trail the configuration, and memoized the fail-open
"resolution failed, allow" outcome for minutes even with an explicit allowlist
configured. Its claim to match a DNS refresh interval that operators configure
through DNS_CACHE_TIMEOUT goes with it.
The SSH backoffs now only escalate for work that ran. A knownhosts manager
suppressing a call inside its own window reports ErrKeyscanSuppressed, and the
temperature layer neither records a failure nor pays for the RPi fallback in
that case. An expired collection deadline is our own budget rather than
evidence about the host, so it holds the window at the floor. Both backoffs
decay once a retry deadline is more than one window past, and replacing the
temperature SSH key on disk clears both maps so a repaired key is tried on the
next cycle instead of after fifteen minutes.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial follow-ups to ac43506e6, which let a host-typed install token
bootstrap-create a Proxmox source. The one-shot machinery held up; these
are the four holes around it.
- test integrity: TestIssue1644HostInstallTokenGrantStaysHostnameBound
completed a registration first, so the second request died at the
completion gate and the bound_hostname comparison was never reached —
the test passed with the binding deleted. It now binds without
consuming (checkRegistration), rejects a different serverName while
the grant is still live, and then completes on the bound hostname to
show the grant was never the reason for the rejection.
- grant TTL: install tokens are minted with no expiry, so every host
install token on a Proxmox box carried a live create-a-source
capability forever. The grant now expires 24h after mint on its own
clock (install_issued_at stamped at mint, falling back to the record's
CreatedAt, failing closed with neither). Expired grants take the same
403 path with a distinct warn.
- replay window: SaveNodesConfig ran before the grant was consumed, so a
persistently failing token store left a source on disk next to an
unconsumed grant — a repeatable create-N-sources primitive. The grant
is now consumed and persisted first, and a failed source save rolls
the consumption back, so either both stores advanced or neither did.
- exec binding: auto-register writes bound_hostname with no
bound_agent_id and no binding version, which is exactly the shape
canBindAgentInstallExecToken refuses, so host-token command enrollment
was being admitted by the legacy pre-v6.1.1 migration branch. That
record shape is now handled explicitly as a clean first use (hostname
equivalence required), and a bound_hostname written by registration is
no longer overwritten by an equivalent spelling the agent reports,
because the still-unconsumed grant compares against it.
Single consumption across types is unchanged: a combined PVE+PBS host
still gets exactly one grant.
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus TTL and exec-first-bind contract pins in internal/api/contract_test.go.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial review of 84dba861b found three ways the new evidence paths
could still attribute a snapshot to the wrong cluster.
The submission-source learner was asymmetric. Clusters only became known
to it through snapshots that were already attributable, so a cluster with
no uniquely-attributable snapshot was invisible - and a source token both
clusters share then mapped to exactly one visible cluster and looked
decisive. The visible cluster got the other's backups while the other
guest stayed at zero. Callers now declare every connection owning a
candidate guest for a PBS instance, and the learner refuses to resolve
anything for that instance until each of them has had a snapshot
attributed to it. Observation is not scoped per PBS instance, so a
cluster seen submitting to its own PBS server still counts as visible -
the reported two-server topology keeps working.
PVE storage confirmations were treated as authorship. A pbs-type storage
listing proves the connection can SEE a snapshot, which a shared token, a
synced datastore, or an offsite copy all arrange without the connection
having made it, and a single confirmer previously outscored everything
else. Confirmations now carry the storage they came from, and only a
storage view that never lists a snapshot some other connection also lists
can attribute a colliding VMID. An overlapping view has demonstrated it
sees other clusters' snapshots, so nothing it lists attributes anything.
Where an exclusive view and the learned source mapping both speak they
must agree, otherwise the snapshot drops as it did before #1639. The
disjoint case - each cluster mounting only its own datastore - is
unchanged.
Confirmations were evicted by partial poll failures. A storage whose
content query failed contributed nothing, and the partial set overwrote
the previous one, flipping attribution between cycles. They now go
through the same per-storage preservation as storage backups.
Contract text calling the PVE listing "the only deterministic
attribution" is reworded to match the weakened semantics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v6.1.0's identity rework (eab73d245) made the VMID-only fallback refuse
to fire whenever a typed VMID exists on more than one PVE location.
Root-namespace snapshots with no matching comment then score zero for
every guest, so on setups with two clusters and overlapping VMIDs most
guests showed no backup at all - while PVE itself listed the backups
fine, because monitoring discards pbs-type storage contents entirely
whenever a direct PBS connection is configured.
Attribution is now evidence-driven instead of dropped:
Storage backup polling keeps a per-connection record of every snapshot
its own pbs-type storage listed (type, VMID, backup time) even though
the raw entries stay out of the PVE backup list. Which cluster listed a
snapshot is deterministic attribution, and it survives fully mirrored
clusters that share one datastore and token. The evidence is
monitoring-internal, cleared on instance retirement or when the storage
poll stops seeing pbs content, and never serialized into state payloads
or snapshots.
Guest backup-time sync and the recovery-point mapper additionally learn
each PBS submission source's cluster (owner token, datastore, PBS
instance - strongest first, scoped to the PBS instance) from the poll's
attributable snapshots, then resolve collision VMIDs whose snapshots
carry no evidence of their own. A source seen from several clusters is
not a discriminator, an unfamiliar component stops resolution rather
than deferring to weaker ones, and a snapshot decisively attributed to
another cluster is kept away from this one. Unattributable snapshots
still drop rather than guess.
Backup-age alert attribution no longer suffix-matches the subject ref's
connection label against guest locations. The label there is a PVE or
PBS instance name, not a PBS namespace, and loose matching could
cross-attribute clusters sharing a VMID; it now requires exact
normalized equality.
Reported in #1639 (two PVE clusters with PBS 4.0/4.1, VM 173 shown 974
days overdue despite valid verified backups).
Fixes#1639
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Settings > Infrastructure installer mints generic host install
tokens, but install.sh auto-detects Proxmox and the agent presents type
pve/pbs at /api/auto-register. The bootstrap grant required an exact
install_type match, so every generic install on a Proxmox node was
denied source creation and the denial was a single buried journal warn.
Four-part fix (#1644):
- server: extend the one-shot bootstrap grant to host-issued install
tokens presenting a canonical Proxmox type. Typed tokens stay pinned,
the grant keeps its settings-write mint requirement, first-hostname
binding, serialized completion, and single consumption across types.
- agent: a canRegister=false denial now logs at error level, returns a
setup error, and records the operator-facing reason in a
proxmox-<type>-registration-blocked state marker.
- installer: report the Proxmox registration outcome in install output
by reading the registered/blocked markers, and poll the server lookup
for a bounded retry window before warning that registration was not
confirmed (readyz flips before the first report cycle).
- setup script: the auto-register transport now captures the HTTP
status alongside the body (no -f), making the invalid-setup-token
branch reachable via 401/403 instead of a dead server-string grep,
and operator guidance names Settings -> Infrastructure instead of the
retired Nodes page (also updated in docs/PBS.md and the pinned
assertions in contract, setup-script, and repoctl docs tests).
Regression proof: internal/api/issue1644_host_install_token_proxmox_test.go
plus new install.sh proofs for the retry window and blocked-marker
surfacing, and the updated hostagent blocked-registration test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes#1640. Three defects around POST /api/ai/patrol/readiness on slow
local hardware behind a reverse proxy:
1. The handler ran up to four sequential provider calls (~45s and more on
slow Ollama boxes) while writing nothing to the response, so any
intermediary with a ~30s read timeout severed the request mid-run. The
handler now commits headers up front and streams flushed newline
keepalives every 10s while the evaluation runs, then appends the normal
JSON payload. Leading newlines are insignificant JSON whitespace, so
existing clients parse the response unchanged.
2. A severed connection cancels the request context, and
patrolRuntimeFailureFromError classified the resulting context.Canceled
as a generic "Provider analysis error", blaming the provider and model
for an infrastructure event. Mid-run cancellation is now classified as
the new "interrupted" cause: the overall status and every unfinished
dimension and autonomy mode report not assessed, per-scenario evidence
completed before the interruption is preserved in the returned result,
and the readiness cache keeps the last completed evaluation.
context.DeadlineExceeded keeps its provider-path timeout classification.
3. createAPIErrorFromResponse pre-seeded the error message with the raw
response body, making its non-JSON guard dead code, so full HTML proxy
error pages became Error.message and were rendered into the readiness
result boxes. Non-JSON bodies now surface only when they are short
plain text; anything with markup or excessive length collapses to a
generic status-derived message.
Regression tests: internal/ai/issue1640_readiness_cancellation_test.go,
internal/api/issue1640_readiness_transport_test.go, and
frontend-modern/src/utils/__tests__/apiClient.issue1640.test.ts, all
registered in the subsystem verification registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while triaging #1643 and #1637: both fields were stored via
system.json and accepted, validated, and echoed by the settings API,
but nothing ever consumed them. The unattended update schedule is owned
entirely by the systemd timer install.sh renders, and no UI control ever
set the fields, so the API persisted a schedule preference that could
never take effect.
Remove the fields from config.Config, SystemSettings, the settings
handlers, and the frontend config type, along with the interval
validation and the .env AUTO_UPDATE_CHECK_INTERVAL rewrite (a legacy
line is now preserved verbatim). Legacy clients that still send the
keys get them silently ignored instead of validated, and a system.json
written before the removal still loads cleanly - both behaviors are
pinned by new tests. User docs no longer describe the phantom schedule
settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cluster-endpoint discovery-policy check ran a raw net.LookupIP per node
per poll cycle since c5f5af7ab, bypassing the process-global cached resolver,
and the injected default subnet blocklist (169.254.0.0/16) made the
zero-policy fast path unreachable so even unconfigured installs generated
that DNS volume. Evaluate the default link-local-only policy against literal
endpoint IPs without resolution, and memoize custom-policy verdicts per
endpoint for the shared 5-minute DNS-cache TTL so repeat polls stay off the
resolver. Also cache ssh-keyscan failures with doubling backoff in the
knownhosts manager and back off temperature SSH collection per host after
failures instead of re-executing ssh twice per node every 10s cycle.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Recorded PVE 8 and PVE 9 guest rrddata responses (fixtures under
pkg/proxmox/testdata/rrd/) prove guest RRD never carries the cache-aware
memused/memavailable columns — they exist only in node RRD — so every
consumer branch reading them was dead code that #1634's listing fallback
(7d7d2b6a3) had already routed around.
Drop the two fields from GuestRRDPoint (now time/maxmem only, matching
the recordings), delete the dead VM RRD memory fallback and its
getVMRRDMetrics/getVMRRDMemory helpers plus the vmRRDMemCache they fed,
remove the pointless per-poll guest RRD fetch from the LXC memory path,
and retire the guest RRD lookups from PVEClientInterface. VMMemoryRaw
loses its never-populated RRD diagnostic fields, and guest reliability
scoring no longer treats the node-only rrd-* sources as trusted guest
evidence. The knownDeadGuestRRDFields allowlist in the fixture
alignment test is gone; a new reflection guard in
code_standards_test.go keeps GuestRRDPoint pinned to recorded columns,
and cleanupRRDCache pruning of the guest-agent meminfo cache gains
direct coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v6.1.2 (c41edb65a) left the agent config gate and the command-channel
admission evaluating exec-token bindings under different policies: the
config gate admitted on bound-hostname OR bound-agent-ID while channel
admission required both to match, compared hostnames with plain case
folding instead of the system-wide short-vs-FQDN equivalence rule, and
had no recovery path for hosts whose immutable agent ID still matched
but whose hostname had drifted since binding. Affected agents kept
reporting CommandsEnabled=true while every channel registration was
rejected, so fleets showed a permanent "Remote control blocked" chip
with reinstall as the only recourse (reported by a customer with a
large Docker fleet after upgrading to v6.1.2).
- Single-source the binding decision in evaluateAgentExecBinding; both
admitAgentExecToken and commandConfigAllowedForToken now consume it,
so the config payload can never advertise command execution that
admission would reject.
- Treat the immutable machine-derived agent ID as the primary binding
identity: an exact ID match re-binds a drifted (renamed) hostname in
place instead of stranding the host; hostname match alone still fails
closed for version-2 bindings.
- Compare hostnames with unifiedresources.HostnamesEquivalent (plus
case-insensitive exact match for IP literals) across admission,
session validation, and legacy migration, so docker01 vs docker01.lan
no longer splits the decision.
- Stop treating a miss on the token-scoped connectivity lookup as
authoritative in the connections ledger: host.TokenID is sticky
across token rotation/revocation, and a shared token fronting more
than one live session fails closed in the token lookup, so fall
through to the agent-ID and hostname lookups before reporting an
enabled host as blocked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real PVE guest RRD responses carry only the cache-inclusive mem/maxmem
columns; the cache-aware memused/memavailable columns exist only in node
RRD, so the LXC RRD branches can never match a live response. Removing
the cluster-resources fallback in bf67ba920 therefore left every running
LXC reporting unavailable memory, rendered as 0% (#1634).
Running containers now fall back to the listing value under the
low-trust cluster-resources source when cache-aware RRD evidence is
absent; unavailable remains reserved for running containers with no
listing evidence at all.
Fixes#1634
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bf24a9a9c changed truenas runtime transport behavior without the
canonical paperwork, so its governance run failed the completion guard.
Document the redirect rule in the monitoring subsystem contract — a
plaintext handshake redirect never wakes the REST bridge; a same-host
https redirect upgrades once to wss and the upgrade is scheme-only and
monotonic — and move the Issue1631 regression tests into
transport_test.go, which is a registered verification artifact for the
truenas runtime path policy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Host agents key their identity on the machine-derived agent ID, so MSP
template deployments that clone /etc/machine-id fold two physical
machines at different sites into one host row whose reports overwrite
each other (hostname, report IP, and interfaces flapping between
sites), silently poisoning node-agent linking.
Mirror the Docker host identity-collapse doctrine (#1584) for host
report ingest: track hostname and report-IP revisits per resolved
agent identity inside the monitoring-owned flap window, publish an
active conflict as models.Host.IdentityConflict through unified
resources, and warn on the Machines page. The report IP is tracked
alongside the hostname because template fleets often reuse hostnames
across sites (pve01 at two customers), leaving the address as the only
field that betrays the clone. A one-time hostname rename never
revisits and is not flagged; the conflict clears on its own once only
one machine keeps reporting for the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Support evidence from the MSP case showed the clusters mix without ever
sharing a corosync name and without the config-layer consolidation
firing, which rules out the same-name path and points one layer down.
Two mechanisms combine there. A newly added connection whose add-time
cluster detection failed (#437) commits its first polls' nodes with an
empty cluster name, because pollPVEInstance ran membership detection
after the node-state commit - and every aggregation guard deliberately
lets empty cluster names merge freely. And host agents key their
identity on /etc/machine-id, which cloned template deployments reuse
across sites, so two different pve01 machines collapse into one agent
row whose shared LinkedAgentID then folds the unclassified node into
the established cluster's slot, overwriting it - the reported "enacon
appeared renamed to rewo" data loss.
Weak-evidence folds across connection instances - a bare-hostname
endpoint alias or a shared linked-agent identity - now require positive
same-machine proof (matching non-empty cluster identity or matching TLS
fingerprints) whenever cluster identity is in play on either side. Two
views that are both unclassified still dedup freely, and address-based
endpoint aliases keep folding on the contradiction checks alone, so the
designed standalone-into-cluster folds survive. PVE polling now also
runs cluster membership detection before the cycle's node-state commit
and re-reads the refreshed instance config, so nodes carry their
cluster identity from the first state write whenever detection
succeeds instead of transiting aggregation unclassified.
Reported via support by an MSP running clusters enacon and rewo that
reuse pve01/pve02 node names across sites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add availability_probe_targets and availability_probe_agents to the
telemetry ping - counts only, no agent names or addresses - with the
disclosure table updated in both privacy doc copies. Document the
feature in the availability-checks configuration guide and the
unified agent guide, including the ICMP capability caveat for
containerized probes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unified agent gains an availability module: probe assignments
arrive through the signed remote-config channel (missing key clears
the schedule), each enabled target runs on its own clamped interval
through the shared probe core, and results queue in a bounded
drop-oldest buffer. A result is offered to the primary server until
one delivery succeeds and never again after - buffered offline
reports are stripped of availability results so the disk buffer
cannot replay observations the queue still holds. ApplyHostReport
feeds accepted reports into the probe ingestion path, where the
ownership check and failure accounting live, and the probe agent id
is projected onto unified availability resources for source
attribution in the UI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Availability targets gain an optional probe agent assignment. Setting
it requires the external_probe entitlement, enforced only at the
moment of assignment - local targets never consult the license path.
Assigned targets are delivered to their agent through the signed
agent-config channel, skipped by the local poller, and resume local
execution automatically if the entitlement lapses. Probe-reported
results are accepted only from the currently assigned agent, share
the local failure-threshold accounting, carry source attribution,
and derive to indeterminate at read time when reports go stale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add external_probe to the licensing feature enumeration, Pro tier
membership, self-hosted feature catalog, feature map, upgrade matrix
and pricing handoff, with the entitlement contract goldens updated.
The capability is served by the community binary - the entitlement
alone gates it, like relay. No existing free functionality changes:
local availability checks stay ungated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the ICMP, TCP, HTTP and UDP probe core from the monitoring
availability poller into internal/availabilityprobe so the host agent
can execute the same checks without importing the monitoring package.
Monitoring keeps type aliases and thin wrappers, so callers and the
canonical guardrail contracts are unchanged. Pure refactor for the
upcoming external probe feature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The config layer already refuses to consolidate two same-named clusters
whose TOFU-captured TLS fingerprints contradict, and node aggregation
keeps any two same-named clusters from different connection instances
apart unconditionally. Two gaps remained one layer down. First, the
endpoint-IP agent match bypassed the contradiction guard entirely, so
two sites reusing RFC1918 addressing (the MSP support case: pve01 on
192.168.1.11 at both sites) still bound the second site's node to the
first site's host agent, attaching the wrong machine's telemetry.
Second, the unconditional split had no way to recognize the legitimate
duplicate - the same cluster added twice through different member
addresses with no config-level endpoint overlap could never fold back
into one node slot.
The aggregation layer now receives the config layer's identity
evidence: each PVE node carries the TLS certificate fingerprint of its
own named endpoint record (standalone nodes carry the instance
fingerprint; a cluster member never inherits the instance-level
fingerprint, which pins whichever member the connection URL reaches).
Same-named clusters from different instances merge only when both views
carry the same fingerprint; contradicting or unknown evidence keeps the
fail-safe split. Agent binding applies the identical doctrine: hostname
and address matches are rejected when the candidate agent's linked
nodes live in a different named cluster or carry a different
fingerprint, closing the previously unguarded endpoint-IP path.
Reported via support by an MSP whose sites reuse cluster names, node
names, and RFC1918 ranges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves the two open Dependabot alerts for GHSA-r28c-9q8g-f849
(source-map auto-loading path traversal, patched in 8.5.18). postcss
is a build-time dependency, so no shipped runtime code was affected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two clusters at different sites that reuse the same corosync cluster
name and the same RFC1918 addressing presented colliding member IPs,
which ConsolidatePVEInstances took as strong endpoint overlap and folded
the second site's connection into the first. The standalone-into-cluster
merge had the same hole for a standalone whose address collides with a
cluster endpoint at another site. The v6.1.1 node-aggregation guard
never fires in this case because it keys on cluster-name conflict, and
same-name clusters do not conflict.
Address coincidence is weak evidence across sites, but the TOFU-captured
TLS certificate fingerprints already stored on instances and cluster
endpoints are strong evidence: contradicting fingerprints for the same
authority, node name, or endpoint address mean different machines.
Consolidation now refuses to merge in that case. The fail-safe direction
is deliberate - a certificate rotation may leave a genuinely duplicated
cluster as two views, but two distinct clusters are never silently
folded into one.
Reported via support by an MSP whose two customer clusters kept merging
after the v6.1.1 aggregation fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v6.1.2 JSON-RPC migration (b81ba7dd0) broke http-configured
appliances sitting behind TrueNAS's HTTP -> HTTPS redirect: the REST
transport followed the redirect transparently, but a websocket
handshake cannot, so every poll failed with status=302 bad handshake.
When the plaintext handshake answers with a 3xx whose Location is an
https URL on the same host, retry once over TLS (honouring the
configured skip-verify/fingerprint settings) and keep the upgraded
wss endpoint for the client's lifetime. Cross-host and downgrade
redirects are refused with an actionable error naming the target, as
is a redirect whose TLS retry fails.
Fixes#1631
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Issue #1621: saving only "intentionally offline" left the SQLite
auto_remediation_policy_json column NULL, and the read path only
normalized the policy inside the non-NULL branch, so the API served
"capabilityNames": null. The section's dirty-state memo spread that
value during render and crashed the expanded row into the route error
boundary. Normalize the policy unconditionally after the scan so the
wire always carries [], type capabilityNames as nullable in the TS API
surface, and guard the spread in the component.
Issue #1622: the inline table-row presentation gated the entire
operator-overrides section on the resource exposing an auto-authorizable
capability, so stopped Docker containers (which only expose start,
never auto-authorized) lost intentionally-offline and
never-auto-remediate exactly when they matter most. Render the section
whenever the resource has an id; the automatic-actions block already
self-gates on eligible capabilities. Also add min-w-0 to the section's
flex text columns and overflow-x-clip to the inline row content box so
the long never-auto-remediate copy wraps instead of escaping the row
border below the lg breakpoint.
Regression coverage: issue-named Go test for the NULL-column read path
and a render test feeding the section the real pre-fix wire payload
with capabilityNames: null.
Fixes#1621Fixes#1622
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scheduled multi-guest vzdump jobs run under a single UPID whose VMID slot
is empty, so pollBackupTasks stored them with VMID 0 and the guest-centric
backups coverage view dropped them entirely: only individually backed-up
guests ever showed task status. (Regressed with the v6.0.0 guest-centric
redesign, which removed the flat task table that used to render job runs.)
pollBackupTasks now fetches the job task's log and parses the per-guest
markers ("Starting Backup of VM", "Finished Backup of VM (duration)",
"Backup of VM failed - reason") into synthetic per-guest BackupTask
entries. Their IDs embed the parent UPID, keeping them stable across polls
and distinct from individually-run backups; per-guest times are
reconstructed from the job start plus the printed durations. Finished
jobs' logs are immutable, so results are cached per instance|UPID and each
finished run is fetched at most once, with a per-cycle fetch cap so a
historical backlog trickles in without stalling the backup poll budget.
The task listing now uses source=all + typefilter=vzdump, so running jobs
are visible too: guests covered by an in-progress job get a "running"
synthetic task, which also feeds resolveBackupIntentContext and
suppresses offline/backup alerts for guests the job is actively backing
up. The frontend needs no changes - synthetic tasks carry real VMIDs and
flow through the existing coverage model, recovery mapper, and alert
intent paths.
Contract: monitoring.md completion obligation 13 records the per-guest
synthesis boundary; proofs land in monitor_backup_job_tasks_test.go,
monitor_alert_intent_test.go, and cluster_client_api_test.go.
Reported by Johannes Strasser (support thread "PBS Bug").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ceph Quincy and later (including Squid on PVE 9) dropped the monmap
mons array and the mgrmap active_name/standbys arrays from ceph status
output, replacing them with monmap.num_mons and mgrmap.num_standbys,
with quorum membership reported at the top level of the payload. Both
the host agent parser and the Proxmox API path only understood the
legacy arrays, so modern clusters showed 0 monitors and undercounted
managers.
- hostagent: read num_mons/num_standbys and top-level quorum data,
taking the largest available signal, and base the mon/mgr service
rows on the same counts
- pkg/proxmox: decode mgrmap num_standbys and top-level
quorum_names/quorum on CephStatus
- monitoring: fall back to the new fields when counting MON/MGR
daemons, and log Ceph 401/403 failures at warn level with a hint to
grant Sys.Audit on / instead of hiding them at debug
- models: prefer the larger non-zero MON/MGR counts when merging Ceph
cluster records from multiple sources
Fixes#1626, Refs discussion #1290
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The readiness advisor failed capable local models for adapter and probe
defects rather than model incapability (#1624 Ollama, #1614 llama.cpp):
- Send num_ctx sized to the haystack fixtures (clamped to the model's
trained window) so Ollama no longer truncates ~25KB prompts at its
4096-token server default; the trained-window guard alone passed while
the runtime request was being truncated.
- Forward an explicitly pinned temperature 0 instead of dropping it to
Ollama's 0.8 default against a nonce-exact validator (ChatRequest gains
TemperatureSet; Ollama options temperature is now a pointer).
- Raise the probe generation cap from 256 to 2048 tokens so qwen3-style
<think> reasoning cannot exhaust the budget before the tool call, and
surface the provider done_reason when validation fails.
- Synthesise tool-call IDs in the OpenAI-compatible adapter (streaming
finalizer and buffered path) when the server omits them, as llama.cpp
commonly does, mirroring the Ollama adapter instead of failing tool
protocol 0/3 on transport shape.
- Probe with the Patrol loop's 60s stream stall allowance instead of the
12s chat default (chat.PatrolProviderStreamIdleTimeout is now exported).
- Stop discarding probe and validator errors: log them, carry them in a
new PatrolModelReadinessResult.Details field surfaced through the API
snapshot and Settings UI, and keep a transport-level probe failure's
specific diagnosis instead of overwriting it with the generic
capability wording.
Builds on 4a2335ce7, which already reclassifies protocol failure as
"provider connected; Patrol capability not verified".
Fixes#1624Fixes#1614
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adding the long-named override fields to the Monitor struct widened
gofmt's alignment for the whole field block, which rewrote ~60
unrelated lines and broke four canonical-guardrail tests that pin
struct fields verbatim (TestAvailabilityProviderStaysOnCanonical
MonitoringPath and friends). Fold the overrides into a nested
runtimePollingOverrides struct with short field names so the block's
alignment — set by guestMetadataRefreshJitter — is untouched, leaving
monitor.go a six-line functional diff against main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live-setter fix for #1619 pushed backup and PMG cadences into
running monitors but left pbsPollingInterval out entirely: it was
neither applied to the handler's base config nor pushed to live
monitors, so a saved PBS interval still waited for an unrelated
restart. Add the matching override, setter and scheduler wiring, apply
the value to the base config, and fan it out from the settings handler
like the other cadence settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Saving backupPollingInterval, pmgPollingInterval or backupPollingEnabled
only mutated the API handler's base config. Monitors poll against a
detached DeepCopy of that config, and only pvePollingInterval set the
reload flag, so the saved values never reached shouldRunBackupPoll or
the PMG scheduler until an unrelated monitor reload happened.
Add mutex-guarded runtime overrides on Monitor with live setters, fan
them out to every tenant monitor from the settings handler (mirroring
forEachNotificationManager), and clear the per-instance last-poll
timestamps when lowering the backup interval or re-enabling backup
polling so the next cycle runs an immediate catch-up poll.
Fixes#1619
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The connections ledger marked any PVE/PBS/PMG/VMware/TrueNAS connection
stale once its last successful poll was more than a fixed 2 minutes old,
so polling cadences above 120s read as permanently stale and latched the
connection-degraded alert with no path back to active (#1620).
Derive the active->stale cutoff as max(3 x polling interval, 2 minutes),
mirroring how availability probes already scale. PVE/PBS/PMG cadences
come from the server config; TrueNAS uses the per-instance interval and
VMware the poller summary interval. The 3x multiplier absorbs the
evaluation-order skew where connection alerts are checked concurrently
with the poll on the same scheduler tick, leaving LastSuccess roughly
one interval old even when healthy.
Fixes#1620
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TestUnifiedProviderIncidentRecoveryConfirmationSurvivesRestart hardcoded
observedAt as 2026-07-24, but the restore path drops persisted alerts
older than 24h, so the test began failing once the calendar moved past
the fixed date. Use a recent relative timestamp instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Branch-coverage tests for twenty partially-covered pure helpers that the
existing suites only reached through higher-level paths, so their arms were
never pinned directly.
internal/alerts read model and filter evaluation: metadataBoolValue 25.0 to
100, metadataIntValue 27.8 to 100 and numericConditionValue 22.2 to 100, each
covering every accepted underlying type plus the wrong-type and missing-key
arms; the three sort-rank helpers now assert an exact rank for every input
class including the default, which is what pins the ordering.
internal/alerts guest and backup identity: snapshotAlertStillTriggered 15.4 to
100 with the threshold pinned at, below and above; parseStableGuestOverrideKey
30.0 to 100; extractGuestSnapshot 35.7 to 100 across its type switch; and
hasActiveAlertTrackingKeyNoLock 40.0 to 100 including the canonical-scan and
nil-skip arms.
internal/alerts/specs: both Validate methods 57.1 and 58.8 to 100 with every
rejection reason asserted on its specific error, and metricTriggered and
metricStillLatched 50.0 to 100 with the metric pinned at, below and above the
threshold in both latch states.
internal/actionplanner: validateParamType 29.4 to 100 over every parameter type
plus the unknown default, validateParamValue 38.9 to 100 per rejection reason,
and both normalize helpers 38.5 and 40.0 to 100 over nil, empty, duplicate and
well-formed input. enumString also moved 50.0 to 100 as a consequence.
Package statement coverage moves to 87.1 percent for internal/alerts, 81.8 for
internal/alerts/specs and 88.5 for internal/actionplanner. Tests only, no
source change.
A third pass on partially covered functions, led by the alert evaluation
predicates where a wrong arm means a missed or spurious alert.
- internal/alerts/specs: matches 47.9 to 100, and all six matches helpers
(severity threshold, change threshold, baseline anomaly, health assessment,
posture threshold, and the severity latch) from 50 to 75 percent up to 100.
Each threshold is pinned at, just below and just above, and the latch arm is
exercised both latched and unlatched with concrete verdicts.
- pkg/audit: exportCSV 76 to 88 with commas, quotes and newlines in the detail
field asserted through a parsed round-trip; NewSigner error arms, both
IsPersistent predicates and VerifySignature against a tampered payload and a
wrong key.
- internal/agentupdate: retryBackoffDelay, sleepWithContext, Snapshot and
writeSelfTestTokenFile to 100, the token file exercised under t.TempDir
including the unwritable-directory arm.
- internal/notifications: writeMultipartBodyPart and alertNodeDisplay to 100,
attachment handling to 69, all asserted on the produced MIME text. No test
opens a network or SMTP connection.
- internal/unifiedresources: the three pure action-dispatch helpers to 100.
- internal/alerts/config: CanonicalResourceTypeKeys 34.3 to 78.4.
Five targets deliberately did not move and are recorded rather than faked:
the error arms of writeEmailThreadingHeaders, buildMultipartEmailMessage and
copyWebhookConfig are unreachable because those functions write only into a
local bytes.Buffer, which never errors; exportJSON's only gap is a
json.MarshalIndent failure that its event struct cannot produce; and
verifyBinaryMagic's remaining gap is a deferred close-error handler.
No source file is modified. Adversarial review returned no rejects and flagged
seven near-duplicate subtests; all were removed and every target function
re-measured at an identical percentage.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
A second pass targeting PARTIALLY covered functions rather than untouched
ones, so every case here is an arm the existing suites never reached.
Percentages are per-function coverage, measured before and after.
- pkg/securityutil: the SSRF guards, which is where the uncovered arms
actually matter. isCarrierGradeNATIPv4 and isLocalNetworkIP are pinned at
the first and last address of 100.64.0.0/10 and just outside both ends,
across loopback, link-local, every RFC1918 range and IPv6 unique-local.
joinURLPath, IsLocalNetworkHost, resolveOutboundIPAddrs and
cloneRestrictedTransport to 100 percent, resolvePermittedOutboundIPs to
96.8, with the transport clone asserted independent of its source.
- internal/models: eleven deep-copy helpers from as low as 25 percent to 100.
Every one asserts real independence, mutating each nested slice, map and
pointer field of the clone and checking the original is untouched, which is
the failure mode a deep-copy helper actually has.
- internal/alerts: metricClearThreshold 28.6 to 100, resourceTypeLabel and
alertspecsMetricTriggered 50 to 100, the four canonical spec-id and
tracking-key builders 66.7 to 100, inferCanonicalKindFromLegacyAlert to 100,
and the backup-snapshot and ack-identity predicates.
- internal/servicediscovery: the four fingerprint generators to 100, each
asserted for both stability and sensitivity; the three command builders and
ValidateResourceID on their exact output and each rejection reason.
- internal/storagehealth: zfsScanActive and firstNonEmpty.
cephClusterSourceRank is deliberately left at 75 percent: its default arm is
unreachable because normalizeCephClusterSource can only return the two cases
above it. That is recorded rather than faked.
No source file is modified. Adversarial review returned no rejects and flagged
nine re-hit subtests; all nine were removed and every target function
re-measured at an identical percentage, proving they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Seventeen files closing the zero-coverage functions the current source drop
left behind. Every named target was measured off 0.0 percent by a per-function
coverage delta, re-measured against current main.
- config: the five new durable Proxmox cluster-node identity helpers
(deterministic id, endpoint equality, alias lookup, id existence, lookup by
id) to 100 percent; VMware and agent-profile persistence round-trips under
t.TempDir including that AppendProfileChangeLog appends rather than
replaces; AI chat session save, load, delete, per-user scoping and age
cleanup, with explicit timestamps rather than time.Now-relative fuzz;
PVEInstance.DeepCopy asserted for nested independence.
- truenas: incidentFromPoolStatus over every pool health string,
RecordsFromSnapshot over nil, empty and populated snapshots, both
TransportStatus accessors, and the RPC handshake and auth typed errors
through errors.Is and errors.As.
- unifiedresources: the maintenance-window operator-state lifecycle on
MemoryStore including the not-found and already-cleared arms, plus the
four remaining View accessors asserted on their exact formatted output.
- api: restoreAgentExecMetadata, buildAlertConnectionSnapshotsWithRuntimeSources
and both mock series generators, asserted on shape, ordering and
determinism rather than non-emptiness.
- cmd/pulse-control-plane: the four remaining MSP and mobile proof report
printers, asserted on the concrete strings in captured stdout.
- ai: cost.EmptySummary, approval.emptyExecutionState, demo.IsDemoRuntimeIntended
and tools.findCanonicalAppContainerResourceByReferences across no-match,
first-match, later-match and ambiguous references.
- monitoring, models, alerts: trueNASAppRunning,
supplementalProviderOwnedSourcesForOrg, IOCounterPresence.Effective,
ValidAlertIntentSignal and intentTimePointer.
No source file is modified. Adversarial review returned no rejects across all
seventeen files and flagged four padding cases plus one dead table field; all
were removed and the per-function coverage re-measured as identical, proving
they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
8d23529c0 made a configured availability check a first-class, source-owned
resource. It no longer collapses into the resource it matches: the
network-endpoint row survives and owns probe status, incidents, history and
the outgoing checks relationship, while the matched resource carries an
additive facet. The unified-resources contract states that explicitly.
TestFixtureGraphAttachesServiceAvailabilityFixturesToServiceResources still
asserted the old collapsing model. It failed any service target that remained
a network endpoint, selected the matched service by target ID alone even
though the check row now carries the same ID, and looked for the checks edge
on the matched resource rather than on the check that owns it.
The test now pins the documented behaviour. Both the Docker and Kubernetes
checks must keep their source-owned endpoint row, the matched service is
selected by resource type, and the outgoing checks edge is asserted on the
check row for both targets rather than only for Docker.
This failure was invisible in CI. Build and Test runs the frontend suite
before the Go suites, and the frontend has been red since 2026-07-23, so no
Go package ran on main for over a day.
Verified: internal/mock, internal/dockeragent and internal/websocket, the
three packages the first completed post-frontend-fix run reported, plus
gofmt, the canonical completion guard, the status, control-plane, registry
and contract audits, and all thirteen release-control unit test modules.
courtmanr@gmail.com