Commit Graph

3198 Commits

Author SHA1 Message Date
rcourtman ab01e56db0 Make session listing O(changed files), not O(all transcript bytes)
GET /api/ai/sessions fully re-parsed every session transcript on every
call, roughly three times over (handoff refresh List, per-session action
loads, final List). On a long-lived install the endpoint took tens of
seconds, pegged the CPU, and the dev watchdog kill-looped the backend.

- SessionStore.List() now serves summaries from a per-file cache
  validated by (modTime, size); only changed files are re-read, with a
  raw-skip decode that never deep-parses message payloads.
- The cache persists as .sessions_index.json next to the session files,
  so the first list after a restart is index-speed too (18ms on a 662MB
  legacy directory vs 23s before).
- writeSession/Delete keep cache and index coherent; external edits
  self-heal via the (modTime, size) check.
- ListSessions refreshes handoff action status only for the 20 newest
  sessions that carry actions, matching what the recent-sessions UI can
  surface; the send path already refreshes the active session on load.
- ai-runtime contract updated: listing-cost clause and bounded-refresh
  clause replace the unbounded refresh promise.
2026-06-11 14:11:17 +01:00
rcourtman 201819f8f8 test(go): skip perf-budget overruns on contended local hosts, enforce in CI
A full local go test run on 2026-06-11 failed pkg/metrics while vite
builds were saturating the machine. Reproduced under synthetic 8-core
load: only the TestSLO_* wall-clock p95 assertions failed, never the
functional tests. A latency budget measured on a shared dev machine
cannot distinguish host contention from a code regression; load storms
inflated medians up to ~4x with no code change, so no absolute
threshold separates the two.

Route the latency SLO assertions (pkg/metrics, internal/api,
internal/monitoring) and the load/stress perf assertions (internal/api)
through helpers that keep strict enforcement on GitHub Actions runners
(controlled environment, existing hosted-runner envelopes unchanged)
and skip locally on overrun, printing the full p50/p95/p99
distribution in the skip message. A local pass still means the budget
was genuinely met. Error-response and correctness checks remain hard
failures. CI -race runs are unaffected: these tests already skip under
the race detector.

Verified: pkg/metrics, internal/api, and internal/monitoring all pass
with 8 CPU burners saturating the host, the scenario that previously
turned TestSLO_RollupTierBatchedFleet and four other budget tests red.
2026-06-11 13:12:59 +01:00
rcourtman 38b2007134 test(api): make log stream SSE test deterministic via history replay
TestLogHandlers_HandleStreamLogs_SendsEvents raced the handler
goroutine against a broadcaster write and a 10ms sleep. The global
broadcaster is shared with every concurrently running test and drops
messages to full subscriber channels, so under CI load the test's
line could be dropped and the body stayed empty; it never failed
locally because the flood only exists during a full package run on a
slow runner (first CI execution of the Go steps in ~100 runs exposed
it).

Write the line into broadcaster history first and run the handler
synchronously with a pre-cancelled context: Subscribe replays history
before the live select loop, so subscription, SSE formatting, and
context cancellation are all exercised with no goroutine, no sleep,
and no dependence on concurrent traffic. Live-channel delivery stays
covered by the broadcaster's own package tests.
2026-06-11 13:10:16 +01:00
rcourtman a405bfc7e8 test(go): stamp parity storage fixture LastSeen; raise CI go test timeout
Two failures surfaced once CI reached the Go steps for the first time
in ~100 runs:

- TestParityStorageFields predates models.Storage.LastSeen (e2a036ce2,
  zero = never seen, no fabricated stamps), so the fixture ingested as
  never-seen and LastSeen() was honestly zero. Stamp the fixture like
  the real pollers do and assert exact round-trip, matching the docker
  host parity test.
- internal/api alone takes ~10m under -race on a fast machine, so the
  10m per-binary budget set when the package was smaller times out on
  CI runners. Raise to 25m. The package has no sleep-debt (all test
  sleeps are milliseconds); it is legitimately heavy.
2026-06-11 12:02:29 +01:00
rcourtman 03db72f557 fix(mock): stamp PVE node, guest, and storage sightings in fixture refresh
Mock fixtures never set LastSeen on PVE nodes, VMs, LXC containers, or
storage. The registry used to paper over that by replacing zero sightings
with ingest time; since 53faa4e46 preserves zero ("never seen") and stamps
those sources "unknown", mock mode rendered its whole PVE estate with dash
last-seen and unknown source freshness.

Stamp sightings in updateFixtureStateMetricsAt, which runs at generation
and on every refresh tick, before the RandomMetrics gate so static-metrics
fixtures stay fresh too. Online nodes and everything they host get the
refresh time (a poll delivers its full inventory, stopped guests
included); anything on an offline node keeps its old stamp, with zero
backdated ten minutes so the UI shows a stale sighting rather than
"never".
2026-06-11 11:12:07 +01:00
rcourtman b349366dae fix(api): count warning-status workloads as running in workloads summary
workloadSummaryStatusIsRunning treated every status except StatusOnline as
not running, so a guest at StatusWarning landed in the Stopped bucket.
Warning is an attention state on a running workload: statusFromGuest maps
degraded guest states to it (power-off maps to StatusOffline), and since
53faa4e46 the registry stale pass degrades online resources with stale
source sightings to it as well. That stale pass surfaced the bug as
provider-backed (VMware) VMs counting Total:1 Running:0 Stopped:1 in
TestContract_WorkloadChartsUseCanonicalWorkloadIDsForProviderBackedVMs and
TestHandleWorkloadsSummaryCharts_UsesCanonicalWorkloadIDsForVMwareVMs,
which seeded sightings older than the VMware stale threshold. Both pass
again with warning counted as running.
2026-06-11 11:10:56 +01:00
rcourtman e2a036ce2e fix(unifiedresources): carry real poll timestamps for storage and docker container sightings
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.

- models.Storage gains LastSeen (omitzero), stamped where entries are
  built: the PVE storage poll (poll start time, including synthesized
  cluster-shared entries; preserved entries for unpolled nodes keep their
  old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
  pool projection (cluster LastUpdated), and the mock generator (offline
  mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
  ("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
  wholesale with each host report, so the host report timestamp is the
  container sighting. This matches every other docker sub-resource adapter
  (services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
  type pbs) to SourcePBS, parented to the PBS instance. Keying them
  SourceProxmox would judge their freshness against the 60s Proxmox stale
  threshold while PBS polls every 60s by default, flapping healthy
  datastores stale between polls; SourcePBS carries the cadence-matched
  120s threshold. PVE-reported pbs-typed storage.cfg backends stay
  SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
  PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
  reports the honest sighting; mock refresh re-stamps available storage on
  each simulated poll.

The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
2026-06-11 10:57:22 +01:00
rcourtman 53faa4e46a fix(unifiedresources): stop fabricating sightings and online stamps for never-seen resources
Ingest stamped every resource's SourceStatus "online" and replaced a zero
LastSeen with time.Now(), so a source that never delivered a resource
(synthesized offline PVE placeholders, never-polled instances) reported a
fresh sighting and a healthy source forever: markStaleLocked skips zero
LastSeen entries, so the stamp was permanently exempt from stale-marking.
That fabricated freshness fed monitor previous-state reads (the
resurrection loop mitigated in 8372a22c5), the frontend lastSeen display,
the monitored-systems ledger, and AI/agent context.

- ingest/mergeInto/IngestResources/presentation coalesce: stamp per-source
  delivery status from the actual sighting ("unknown" when the source has
  never delivered, "online" otherwise) and preserve zero LastSeen instead
  of substituting ingest time.
- replaceRegistry: run MarkStale over the fully assembled registry. The
  IngestSnapshot stale pass ran before record sources (TrueNAS, VMware)
  were ingested, leaving them stamped "online" at any age.
- monitorLastSeenUnix: report 0 for never-seen instead of time.Now(), so
  the frontend renders a dash rather than "just now".
- connected infrastructure: clamp zero LastSeen to 0 ms instead of the
  negative UnixMilli of the zero time.

preserveOrExpireNodes keeps requiring a real online sighting for grace;
the registry no longer manufactures the timestamps that made that guard
necessary, so the 8372a22c5 failure mode is impossible by construction.
2026-06-11 10:23:49 +01:00
rcourtman 8372a22c51 fix(monitoring): show offline nodes for PVE instances unreachable since startup
A configured PVE instance that was already down when Pulse started had
no node state, so the unreachable-poll path returned without writing
anything and the instance was invisible on the Proxmox overview until
its first successful poll. Synthesize offline node entries from the
instance config in that case: cluster configs get one row per
discovered member endpoint with the same node ID a live poll would
assign, standalone instances get a single row named after the
configured instance. The next successful poll replaces them wholesale.

Also stop the offline grace policy from resurrecting such placeholders.
Previous node state round-trips through the unified registry, which
stamps zero LastSeen values with the ingest time, so a never-seen
placeholder came back "recently seen" every cycle and flipped to a
permanent online/degraded state. Grace now requires a real online
sighting: a nodeLastOnline entry from this process, or a previous
snapshot that itself reports the node online with a recent poll
timestamp.

Verified live on the dev stack: a synthetic instance pointing at an
unreachable TEST-NET address appears as an offline node row on the
Proxmox overview immediately after a fresh-state backend start and
stays offline past the grace and stale windows.

Fixes #1433
2026-06-11 09:47:42 +01:00
rcourtman f9b10316b8 fix(licensing): surface unreadable persisted v5 licenses and retry failed startup exchanges
Two silent-downgrade paths in the v5→v6 migration are now visible and
self-healing:

License load/decrypt failure (was: one log line, no UI state): when
license.enc exists but cannot be read, getTenantComponents now persists
a terminal commercial_migration state with reason
persisted_license_unreadable, so the licence panel and global banner
tell the customer to re-enter their v5 key instead of leaving them to
discover missing Pro features.

Failed startup exchange (was: once per process, Community until manual
restart): exchange failures classified as pending — license-server
blips, DNS failures, rate limits — now schedule a background retry loop
with backoff (30s → 30m cap) that re-attempts the exchange until it
succeeds, hits a terminal classification, or the org's service activates
through another path. The loop stops cleanly on manual activation and
StopAllBackgroundLoops.

Registers the new files in the subsystem registry (cloud-paid
commercial-migration verification policy + shared ownership) and pins
both behaviors in the cloud-paid and api-contracts contracts.
2026-06-11 08:44:35 +01:00
rcourtman e8455e84b7 fix(resources): pin canonical host IDs to durable identity so they survive restarts
Canonical IDs for merged-source hosts (PVE node + pulse-agent) were minted
from whichever identity keys the creating record happened to carry: the
agent record knows the machine ID, the Proxmox node record only knows
cluster+hostname. The registry rebuilds from scratch every tick, so a boot
window where the agent had not checked in yet minted a cluster-keyed ID
(agent-7a62... for delly) while steady state minted a machine-keyed one
(agent-bdd4...). Every restart re-ran the race, fragmenting the
resource_changes journal into per-boot eras (9.4k vs 6.1k rows for the
same host) and silently truncating report availability and UI timelines.

Fix, in the layer that owns identity:
- Persist identity pins (canonical_id <-> machine_id/dmi/cluster/hostname)
  in the previously schema-only resource_identities table, written by the
  store-backed registry after monitor-adapter rebuilds, diff-aware so
  steady-state ticks cost no writes.
- Complete weak incoming identities from the pins before matching and ID
  derivation, so a node-only boot window derives the same machine-keyed
  canonical ID as steady state. Derivation itself is unchanged; ephemeral
  nil-store registries behave exactly as before.
- Expand change-journal reads (Get/Count families, SQLite and memory) to
  the full era set recomputed from the pinned identity keys, healing
  historical journals at query time with no row migration. Reads keyed by
  a stale era ID resolve to the same merged timeline.

Regression tests cover both ingest orders, restart simulation via the
monitor adapter, era ID derivation, and era-merged journal reads on both
store implementations. Contracts updated: unified-resources obligation 25
(durable identity pins), monitoring obligation 10 (adapter rebuild
persistence).
2026-06-11 08:33:00 +01:00
rcourtman 43bc15c12f fix(config): restore legacy PORT fallback for FRONTEND_PORT with deprecation warning
v5 documented PORT as a legacy alias for FRONTEND_PORT. v6 dropped the
fallback silently, so an install configured with PORT=8080 upgrades to
v6 and comes up on 7655 with no explanation, unreachable behind its
existing port mapping or reverse proxy.

Honor PORT again at lowest precedence (FRONTEND_PORT wins when both are
set) and log a deprecation warning at startup, mirroring the existing
legacy DISABLE_AUTH handling. Document PORT as deprecated in
CONFIGURATION.md and note the compatibility behavior in the v6
changelog.
2026-06-10 22:32:28 +01:00
rcourtman 9bcf767b1b fix(reporting): resolve report subjects through the canonical unified view and paginate the metric card grid
Two regressions surfaced by generating a real-mode agent report after a
backend restart:

- Report subject lookups used Monitor.GetUnifiedResources (the raw
  resource store), but the raw store's canonical IDs depend on per-boot
  ingest order for merged-source hosts: after a restart the same host
  resolved to a different agent-<hash> than the one the UI and
  /api/state advertise, so reports lost the resource name, availability,
  and metrics translation entirely. Subject enrichment now reads
  Monitor.UnifiedResourceSnapshot and MetricsTargetForResource resolves
  through GetUnifiedReadStateOrSnapshot first (raw store as fallback) -
  the same re-ingested registry every other read surface uses.
- The performance summary card grid positions cards absolutely and never
  paginated: an agent host reporting 8+ metric families walked off the
  page bottom, fought fpdf's auto page break, and scattered one orphan
  element per page (a 7-day delly report rendered 18 pages, ten of them
  near-blank). The grid now starts a new page before a row that will not
  fit; the same report renders 9 pages with intact cards.

Verified live: real-mode delly report shows name, 288 data points,
availability, charts, and correctly paginated cards. The underlying
canonical-ID instability (raw store vs re-ingested view, and the
resource_changes journal fragmenting across boot eras) is a separate
root issue tracked for its own fix.
2026-06-10 22:02:48 +01:00
rcourtman 47392b5f68 fix(mock): derive all fixture identities from stable hashes so IDs survive restarts
Mock fixture IDs churned across backend restarts, breaking resource
identity continuity in mock mode: k8s pod and deployment names picked
their namespace and prefix with rand.Intn per boot, the ceph FSID was
pure rand.Int63n, and generic agent hosts randomized both platform
profile (which feeds the host ID) and hostname. Every restart re-keyed
those unified resources and re-seeded ~31k orphan metric rows that
lingered for the full 90d mock retention.

Derive all of them from mockStableChoice/mockStableDecimalString over
stable inputs (cluster ID, item ordinal, instance name) instead,
matching the generator's existing stable-ID idiom and its documented
contract. Add a regression test that builds the fixture graph twice
and requires every identity set to match.

Verified live: two mock-mode boots now share all 185 distinct metric
resource IDs and the second boot's backfill seeds 0 rows (was ~31k).
2026-06-10 21:49:15 +01:00
rcourtman f136b1ef59 fix(mock): backfill the metrics store in mock mode so reports get real history
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.

Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.

Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).

Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
2026-06-10 21:25:02 +01:00
rcourtman 8ce834cc59 fix(resources): resolve agent metrics targets to the host agent ID for merged hosts
Real-mode agent host metrics are written to the metrics store keyed by
host.ID (monitor_agents.go writes "agent"/host.ID, pinned by a canonical
guardrail), and ingestHost registers that same ID as the SourceAgent
mapping. But BuildMetricsTarget preferred the Proxmox/VMware source ID
for agent resources, so a host that is also a Proxmox node (delly,
minipc, pi) advertised an "agent"-typed metrics target (e.g.
"homelab-delly") that nothing ever writes. Every reader that trusts the
registry target queried zero rows: performance reports resolved via
MetricsResourceID rendered "Data Points: 0", and the resource drawer
history endpoint fell back to in-memory node history (losing
temperature and disk I/O series the agent records).

Flip the priority: the agent source wins for agent resources, platform
sources (Proxmox, VMware, TrueNAS, Docker) remain fallbacks for hosts
without a reporting agent. This matches the write path exactly - the
bulk charts feed (hostAgentChartRequest) already preferred the agent ID,
and pure-agent hosts already resolved to it, so merged hosts simply
become consistent. No history is orphaned: "agent" store rows were
always keyed by host.ID, and node-poller rows ("node"/<node.ID>) were
never reachable through the "agent"-typed target.

Verified live: all five real agents now resolve metricsTarget to their
agent UUID, /api/metrics-store/history returns store rows for the new
target, and the delly performance report renders 312 data points where
it previously rendered zero.

Regression tests: BuildMetricsTarget prefers the agent source for
merged-source resources, and a registry ingest test asserts the merged
node+agent resource resolves its target to host.ID - the agent store
write key.
2026-06-10 18:13:54 +01:00
rcourtman 686c2e8716 feat(reporting): availability section computed from the resource state timeline
Performance reports answered 'what were the averages' but never 'was my
infrastructure up' - the question a managed-service client reads a
monthly report for. Reports now carry an Availability summary derived
from the recorded resource change timeline (state_transition entries
keyed by the canonical unified ID):

- uptime percent over the observed portion of the window, outage count,
  total downtime, and longest outage, rendered in the executive summary
  with an explicit semantics note; fleet summaries gain a per-resource
  Uptime column and CSV exports gain availability header lines
- absent/unknown spans are unobserved time: excluded from the uptime
  math entirely and disclosed as coverage, never counted as downtime.
  The journal records a registry absence for every monitor restart, so
  treating gaps as outages would invent fleet-wide downtime every time
  the operator restarts Pulse
- warning states count as up (the resource is reachable and serving);
  the uptime label clamps rounding so any real downtime can never
  display as a clean 100%
- resources with no timeline render no availability section at all
  rather than a fabricated number

Verified live against a real 7-day window: uptime/outage/downtime
figures reconcile with the raw resource_changes journal.
2026-06-10 17:48:11 +01:00
rcourtman e6c0c4d385 fix(reporting): resolve unified resource IDs to metrics targets and render honest report subjects
Performance reports were structurally disconnected from the v6 ID
space: the UI (and any API caller working from /api/state) addresses
resources by canonical unified ID, while the metrics store is keyed by
each platform's native source ID (the resource's metricsTarget). The
engine queried the store with the unified ID verbatim, so every report
rendered 'Data Points: 0' regardless of how much history existed, and
covers showed raw hash IDs a report reader cannot map to a machine.

- MetricReportRequest gains MetricsResourceID: handlers resolve the
  unified ID through the tenant monitor's resource store (new
  Monitor.MetricsTargetForResource accessor; the registry computes
  targets on demand, they are not persisted on snapshot structs) and
  the engine uses it for store queries only. Recovery points and
  Patrol findings stay keyed by the unified ID.
- Legacy snapshot models and their alerts are keyed by the metrics
  target ID, so enrichment now matches either ID space and resource
  names/status resolve again on covers, headers, and fleet rows.
- Fleet summaries mirror the single-report guard: zero data points
  across the fleet renders a muted NO DATA card instead of a green
  HEALTHY 'All systems operating normally' - false reassurance is the
  worst failure mode for a client-facing stability report.
- Em dashes in PDF-bound literals become hyphens; fpdf core fonts are
  cp1252 and rendered them as mojibake.
2026-06-10 17:10:30 +01:00
rcourtman 82fdc6cfe2 fix(monitoring): refresh discovered cluster endpoints when the cluster re-IPs
Cluster endpoints were discovered once at add time and never re-read:
detectClusterMembership returned immediately for instances already marked
IsCluster, so when a cluster moved subnets Pulse kept dialing the dead
per-node addresses for failover and kept displaying them on Settings ->
Infrastructure, even though monitoring still worked through the main host
fallback. The manual /refresh-cluster endpoint exists but is not wired to
any UI control, so nothing ever corrected the stored addresses.

The 5-minute cluster re-check now also covers cluster instances: it
re-reads /cluster/status through the configured main host, and when a
node's address or the node set actually changed (volatile fields like
Online/LastSeen are ignored) it updates the stored endpoints, persists the
config, and rebuilds the failover client so polling moves to the new
addresses. User-managed fields (IP overrides, guest URLs, fingerprints)
are preserved by the existing discovery helpers, and fields the cluster
status API omits are inherited from the stored endpoint rather than
erased.

Refs #1493
2026-06-10 16:20:35 +01:00
rcourtman 6eaea2fc3a fix(monitoring): clear poll lastError on success so healthy connections drop stale error banners
recordTaskResult reset the failure counters on a successful poll but left
LastErrorAt/LastErrorMessage/LastErrorCategory in place forever. The
connections aggregator surfaces that field as a live error banner on
Settings -> Infrastructure and derives the Unauthorized state from it, so
one transient cluster outage (or startup blip) pinned a red "no healthy
nodes available" banner onto a connection that polls fine, and a single
past 401 could pin a healthy connection to Unauthorized.

LastError now means "current outstanding failure since the last success":
recorded on failure, cleared on success. Connections that are genuinely
failing (no successful poll) keep their banner.

Refs #1493
2026-06-10 16:19:24 +01:00
rcourtman 02955619f5 fix(monitoring): mark PVE nodes offline when the whole instance is unreachable
pollPVEInstance returned early on a GetNodes error, so a host that
stopped answering entirely (shutdown, network loss) kept its last
successful snapshot in state forever: node row frozen at online with
stale uptime. The existing 60s offline grace policy only ran for
cluster-reported peer-down and empty-poll cycles, never for a whole
instance going dark.

Route the poll error path through the same grace policy: nodes of an
unreachable instance stay online (connection degraded) within the
grace window and flip to offline with cleared uptime and CPU once it
lapses, matching the cluster-reported node-down behavior.

Verified live: registered a synthetic PVE instance, killed it, and the
node flips to offline one poll cycle after the grace period.

Fixes #1441
2026-06-10 16:08:12 +01:00
rcourtman c511c935b3 fix(disks): ground SMART disk discovery in /sys/block so no physical disk vanishes
On the #1483 reporter's Proxmox node a SATA SSD (INTEL SSDSC2BW240A4) never
appeared in Pulse even though smartctl reads it fine, because the agent's
discovery and probing both had single points of failure:

- Discovery trusted smartctl --scan-open exclusively whenever it returned at
  least one target. The scan silently omits any device it fails to open at
  scan time (the failure is only a #-comment in its output), so one bad open
  hid a real disk while its neighbours were listed.
- A scan-provided -d type got exactly one probe attempt on Linux. A type whose
  full query (-i -A -H) fails or returns nothing dropped the disk silently.
- A probe yielding no usable SMART data dropped the disk entirely instead of
  reporting the identity the kernel can prove.

Fixes, all in the agent collector:

- Union discovery: the kernel block device list (/sys/block, lsblk fallback)
  is the ground truth for which disks exist; smartctl --scan-open only
  contributes device-type hints. Any physical disk the scan misses gets an
  untyped probe target.
- Untyped retry: a typed Linux probe that errors, yields no usable data, or
  fails to open (exit bit 2, distinct from -n standby,3's exit 3) retries with
  smartctl auto-detection before giving up. Multiplexed controller members
  (megaraid, cciss, ...) are exempt since dropping -d would probe the array.
- Identity-only fallback: when every probe fails but the disk exists in
  /sys/block with nonzero capacity, report device/model/serial/size with
  health UNKNOWN instead of hiding it. No SMART data is fabricated;
  multiplexed array paths and zero-capacity media are excluded.
- Exclusion follow-through: --disk-exclude now also matches the canonical
  post-refine name (nvme0n1), not just the scan label (nvme0).

Regression tests use smartctl/lsblk fixtures captured from PVE 9.1.9 hosts
and the issue report. Verified live on two PVE nodes: NVMe keyed by
namespace with byte-exact pvesh sizes, SATA disk reported with full identity.

Completes the #1483 fix started in bd20069c6 (namespace devpath, authoritative
sizes, merge downgrade guard). Closes #1483.
2026-06-10 15:14:34 +01:00
rcourtman a17882458a Rename CP_TRIAL_ACTIVATION_PRIVATE_KEY to CP_ENTITLEMENT_SIGNING_PRIVATE_KEY in the provider MSP bundle
The variable signs hosted entitlement leases; the trial-activation name is
left over from the retired trial era and reads as trial machinery to an
operator generating their licensing root key. The provider MSP bundle has
no installed base yet, so the canonical rename is free today and frozen
the moment the first design partner installs.

- Control plane reads CP_ENTITLEMENT_SIGNING_PRIVATE_KEY first and falls
  back to CP_TRIAL_ACTIVATION_PRIVATE_KEY, so existing Pulse-hosted cloud
  deployments (deploy/cloud, hibernated snapshot) keep working unchanged.
- deploy/provider-msp (.env.example, compose, setup.sh), MSP.md, and the
  install-test pins use the canonical name; error messages name it too.
- deploy/cloud intentionally keeps the legacy name: that stack historically
  signed hosted trial activations, and its snapshot predates the rename.
2026-06-10 14:08:52 +01:00
rcourtman d25b99cbcc Make provider-MSP client runtimes verifiably licensed via chained entitlement leases
Provider-hosted MSP client workspaces previously sat at Community tier
forever: the runtime refreshed leases against the built-in Pulse Cloud URL
(hibernated, 522) instead of the provider control plane, and release-build
images verify leases only against the embedded Pulse key, which an
operator-generated CP_TRIAL_ACTIVATION_PRIVATE_KEY can never satisfy.

- Inject PULSE_PRO_TRIAL_SIGNUP_URL=CP_BASE_URL into client containers so
  lease refresh targets the provider control plane.
- Chain trust through the Pulse-signed provider MSP license: the license
  binds the provider's lease signing public key
  (entitlement_signing_public_key claim); the control plane embeds the
  license in every lease (provider_license claim); release-build runtimes
  verify embedded Pulse root -> provider license -> lease signature.
- Cap chain-verified leases at ProviderChainedLeaseCapabilities: MSP tier
  plus white_label (branded per-client reports), minus Pulse-service-backed
  relay/mobile_app/push_notifications, which otherwise loop doomed
  registrations against Pulse's relay.
- Fail fast at control-plane startup when the license does not bind the
  configured signing key, instead of provisioning silently unlicensed
  client workspaces.

Verified live on a Colima harness: release-tagged tenant image with test
embedded root, Traefik TLS, full provider-msp proof, tenant reports
valid=true plan_version=msp_growth with white_label and zero relay
failures.
2026-06-10 13:58:52 +01:00
rcourtman 96001d134e Harden MSP tenant isolation: scope org-bound tokens away from default org, propagate webhook allowlist to all tenants
Two gaps found by exercising the MSP pilot path live on a throwaway
multi-tenant instance:

1. CheckAccess granted any authenticated principal access to the default
   org, so a token bound to a client org could read the provider's own
   default-org estate if it leaked from a client site. Org-bound tokens
   now fall through to the explicit binding check for the default org;
   authenticated users and legacy unbound tokens keep default-org access,
   and binding "default" explicitly still grants it.

2. The webhook private-target allowlist (instance-wide system setting)
   only ever reached the default org's notification manager on
   startup/reload, and only the request-context org on settings update.
   Tenant orgs' webhooks to private targets (per-client Gotify over VPN,
   the canonical MSP alert route) failed SSRF validation with no org-side
   remedy, and any allowlist died with a restart. Settings updates and
   reloads now fan out to every live tenant manager via the new
   MultiTenantMonitor.ForEachMonitor, and tenant monitors inherit the
   persisted allowlist and public URL at creation.

Both fixes verified live: org-bound token vs default org returns 403;
client-org webhooks to a private target succeed after restart and for
orgs created after the allowlist was saved. MSP.md validation checklist
gains the default-org probe and the allowlist guidance; MULTI_TENANT.md
documents the binding semantics. Contracts updated for api-contracts,
security-privacy, and monitoring with adjacency notes for
agent-lifecycle, storage-recovery, and performance-and-scalability.
2026-06-10 11:37:39 +01:00
rcourtman cbd0311055 Inject workspace display name into hosted tenant runtime env
Hosted tenant containers received PULSE_TENANT_ID but not
PULSE_TENANT_NAME, so alert webhook payloads from provider-hosted client
runtimes fell back to the raw tenant ID instead of a human-readable
workspace label. Resolve the display name from the tenant registry at
container-create time via a ManagerConfig resolver and stamp it
alongside the tenant ID. Display-name changes after creation apply on
the next runtime rollout, which recreates the container with freshly
resolved env.
2026-06-10 11:13:37 +01:00
rcourtman 1cf92e6c10 Pin metadata GET zero-record payload contract
Adds TestContract_MetadataGetPayloadsUseZeroRecordsInsteadOf404: empty
guest/docker metadata maps must serialize as {} (never null) and a
missing resource must return a 200 zero record echoing the requested ID
(never a 404). This is the proof companion to the
metadata_handlers_shared.go consolidation in the previous commit — it
was authored with that change but lost to a shared-index race at commit
time.
2026-06-10 10:53:57 +01:00
rcourtman 6340cd36f2 Dedupe internal/api handler families behind shared flows and generics
Clears the sixteen dupl pair groups in internal/api plus the pkg/pulsecli
pair:

- router.go: privileged settings endpoints share the
  serveSetupTokenOrSettingsWrite gate; patrol findings convert via one
  unifiedFindingFromAI; the five infrastructure-summary chart loops share
  collectGuestChartData / fillChartSeriesFromBatch; the VM/LXC workloads
  summary loops share appendGuestWorkloadSummaries.
- deploy_handlers.go: preflight and job status/SSE handlers share
  handleDeployJobStatus / handleDeployJobEvents.
- recovery_handlers.go: points series/facets share
  parseRecoveryListPointsOptions.
- truenas_handlers.go / vmware_handlers.go / router_routes_registration.go:
  the connection update flow (locate, decode-with-fallback, normalize,
  preserve masked secrets, validate, save, redact) moves to the new
  platform_connection_shared.go (updatePlatformConnection +
  decodeOptionalInstanceRequest + the admin-gated item-route builder);
  per-platform wrappers carry nolint'd declarative wiring only.
- docker_metadata.go / guest_metadata.go: GET/PUT payload semantics move
  to metadata_handlers_shared.go; the zero-record-instead-of-404 contract
  is pinned by TestContract_MetadataGetPayloadsUseZeroRecordsInsteadOf404.
- kubernetes_agents.go / docker_agents.go: lifecycle PUTs share
  per-handler action helpers.
- config_node_handlers.go: PBS/PMG probes share
  testProxmoxPlatformConnection.
- cloud_handoff_handlers.go / purchase_return_redemptions.go: secrets
  sqlite stores open through openHardenedSecretsDB so permission
  hardening stays single-sourced.
- ai_handlers.go / chat_service_adapter.go: the GetMessages adapters are
  deliberate contract mirrors — suppressed with nolint and enforced by
  TestOrchestratorAndChatAdaptersMapTheSameMessageFields.
- pkg/pulsecli/actions.go: action subcommands seed env defaults via
  actionAPIDefaults (tested); the audit/events cobra registration pair is
  nolint'd parallel wiring.
- .golangci.yml: exclude gitignored tmp/ from ./... typechecking.
- subsystem_lookup_test.py: refresh the pinned api-contracts.md line
  numbers shifted by the contract additions.

golangci-lint run ./... is now fully green. Full internal/api and
pkg/pulsecli test suites pass.
2026-06-10 10:52:05 +01:00
rcourtman a223b02dd5 Stamp tenant identity into alert webhook payloads
Alert webhook payloads now carry the tenant that fired them, so MSP/PSA
receivers (ConnectWise and similar) can route tickets by client without
inferring the tenant from which webhook endpoint fired.

- WebhookPayloadData gains TenantID/TenantName, exposed to custom
  templates as {{.TenantID}}/{{.TenantName}}.
- Defaults come from PULSE_TENANT_ID/PULSE_TENANT_NAME (already injected
  into provider-hosted client runtimes; name falls back to ID).
- Shared-process multi-tenant orgs override via a lazy org-backed
  resolver wired in MultiTenantMonitor.GetMonitor, so display-name
  renames are picked up without restart.
- Generic service template emits a tenant block when identity is set,
  omits it otherwise; single-tenant payloads are unchanged.
- Notifications and monitoring subsystem contracts updated with the
  tenant-identity ownership boundary; guardrail test pins the org
  wiring.
2026-06-10 10:03:39 +01:00
rcourtman 9f722a442a Dedupe alerts PMG queue checks, vmware clones, licensing and proxmox client clones
Clears the dupl pairs outside internal/api:

- internal/alerts/pmg.go: the total/deferred/hold per-node queue checks
  share evaluatePMGNodeQueueAlert; the historical short-circuit (a
  below-threshold clear or invalid spec skips the node's remaining
  checks) is preserved via the helper's skip-node return.
- internal/vmware/provider.go: the cloneInventory* family rides generic
  cloneSliceWith / cloneShallowSlice helpers instead of fourteen copies
  of the same nil/make/loop scaffold.
- internal/vmware/client.go + client_signals.go: the byte-identical
  Automation and VI/JSON fetchers delegate to one getSessionScopedJSON.
- internal/vmware/fixtures.go: added to the dupl exclude list in
  .golangci.yml — literal mock fixture catalogs are the same category as
  the existing internal/mock/ exclusion.
- pkg/licensing/license_server_client.go: Activate and
  ExchangeLegacyLicense share postActivationRequest (idempotent POST +
  shared activation response decode).
- pkg/proxmox/client.go: LXC/VM RRD fetches share getGuestRRDData.
- pkg/pulsecli/actions.go is intentionally left for the api-contracts
  slice.

Full test suites pass for internal/alerts, internal/vmware,
pkg/licensing, pkg/proxmox.
2026-06-10 09:49:03 +01:00
rcourtman 8439ce6e6b Dedupe AI runtime guest-family, tool-pipeline, and provider clones
Clears the ten dupl pairs across internal/ai:

- patrol_intelligence.go, tools_query.go, tools_storage.go: VM and LXC
  system-container paths collapse into generics over read-state view
  method subsets (gatherGuestIntelligenceFromViews, canonicalGuestGetResult
  + guestViewGetResult, addCanonicalGuestSearchMatches +
  addGuestViewSearchMatches, appendGuestDiskSummaries).
- tools_file.go: append/write share executeFileMutation driven by
  fileMutationSpec (approval-command text, shell redirect, verification
  strategy stay per-action and verbatim).
- tools_kubernetes.go: deployment restart / pod delete share
  executeKubernetesResourceAction driven by kubernetesResourceAction.
- providers/anthropic.go + anthropic_oauth.go: message conversion shared
  via convertMessagesToAnthropic (the OAuth copy was annotated 'same as
  regular client').
- memory/changes.go + memory/remediation.go: history loading shared via
  the generic loadMemoryHistory in memory/paths.go (10 MiB cap, sort,
  missing-file semantics preserved via a found flag).
- findings.go and unified/alerts.go: Finding/findingJSON and
  UnifiedFinding/unifiedFindingJSON are deliberate marshal-mirror twins
  (AlertIdentifier json:"-" vs alert_identifier round-trip); merging
  would break every public literal. Suppressed with nolint:dupl and
  enforced instead by new reflect-based mirror-sync tests.

Contract Extension Points name the shared helpers and the mirror
invariant. Full ./internal/ai/... test tree passes.
2026-06-10 09:38:52 +01:00
rcourtman b855b21c7f Dedupe monitoring pollers behind shared provider, lifecycle, and seeding helpers
Clears the six dupl pairs in internal/monitoring:

- poll_providers.go: PVE/PBS/PMG listInstances / describeInstances /
  connectionStatus closures collapse into generic sortedClientNames,
  describeProviderInstances, and providerConnectionStatuses helpers; the
  PBS/PMG adapters are now built by newPrefixedPollProvider from a
  prefixedPollProviderSpec (prefix drives status keys, health keys, and
  the fallback instance name).
- truenas_poller.go / vmware_poller.go: the Start scheduling loop
  (double-start guard, stopped-channel handshake, sync+poll cadence)
  moves to startPollerLoop, and the active-connection config policy
  (enabled instances keyed by trimmed connection ID, defaults applied)
  moves to loadActiveInstanceConfigs, both in the new
  platform_poller_shared.go.
- monitor_polling_vm.go / monitor_polling_containers.go: traditional
  polling now records guest series via the existing canonical
  recordGuestMetric helper (io/network sentinels preserve the historical
  cpu/memory/disk-only behavior on this path); the source-shape
  guardrails in canonical_guardrails_test.go and
  memory_source_catalog_test.go pin the new delegation.
- mock_metrics_history.go: native and TrueNAS disk seeding share one
  seedDiskTelemetry closure.

Proof: new TestTrueNASPollerDoubleStartKeepsSingleLoop pins the shared
lifecycle loop; new TestSeedMockMetricsHistory_DiskTelemetryParityAcrossNativeAndTrueNAS
pins four-series parity for both disk sources. Contract Extension Points
name the shared wiring layer and poller scaffold.

Full internal/monitoring test suite passes.
2026-06-10 09:19:51 +01:00
rcourtman 74417de150 Share one Resource scaffold across namespaced Kubernetes adapters
The k8s workload/config adapters in unifiedresources each hand-rolled the
same Resource literal (Technology, LastSeen, UpdatedAt, Kubernetes facet,
label tags) plus namespacedKubernetesIdentity return; dupl flagged the
StatefulSet/Job and ReplicaSet/DaemonSet clones. Fold the scaffold into
namespacedKubernetesResource and delegate all 19 namespaced adapters
(Service included) to it; cluster-scoped kinds (PV, StorageClass,
Namespace) keep bespoke identity construction.

Contract: unified-resources Extension Points now name the shared scaffold
as the way new namespaced kinds assemble resources. Proof: new
TestNamespacedKubernetesResourceScaffold pins the scaffold fields and
namespaced hostname identity.

Full internal/unifiedresources test suite passes.
2026-06-10 09:16:14 +01:00
rcourtman dad1152fc4 Surface finding-key collisions as lifecycle events instead of silent identity hijack
Patrol finding identity is the LLM-assigned key (resource+category+key
hash -> ID), and recordFinding adds by plain ID merge: when the LLM
reuses a key for a genuinely different issue on the same resource, the
new report silently overwrites the existing finding's title,
description, and evidence while inheriting its lifecycle — and on a
resolved finding, the reactivation counts as a regression of the OLD
issue, inflating the regression counter with a fiction.

Forking the key on dissimilar content would be worse: LLM titles vary
run to run, and splitting one real issue into duplicate findings is a
bigger trust hit than a conflated history. So the merge semantics stand,
and the identity shift is recorded honestly: when a same-ID
re-detection's title shares essentially no keywords with the existing
title (keywordOverlap <= findingIdentityShiftMaxTitleOverlap, 0.2 —
resource/category/key are equal by construction so text is the only
discriminating signal), FindingsStore.Add appends a content_replaced
lifecycle event preserving both titles in metadata and logs the
collision for frequency observability. Rephrasings and identical
re-detections stay event-free per the heartbeat rule. The UI renders
the event through the existing identifier-formatter fallback ("Content
replaced"); a dedicated label is patrol-intelligence UI work and lands
with that subsystem's own ceremony.

Teeth in findings_lifecycle_test.go: distinct-issue collision records
exactly one event with both titles; rephrased and identical
re-detections record none; the resolved-finding collision shows BOTH
content_replaced and regressed so the regression can be read for what
it is. Contract: Current State entry in ai-runtime.md. Full internal/ai
tree green.
2026-06-09 22:03:20 +01:00
rcourtman 41e41be851 Extract saveHistoryData helper to dedupe AI usage / patrol run history saves
SaveAIUsageHistory and SavePatrolRunHistory were byte-for-byte clones
(lock, EnsureConfigDir, marshal, optional encrypt, write, debug log)
flagged by dupl. Fold the shared shape into saveHistoryData, the save-
side counterpart of the existing loadHistoryData generic, preserving
the exact error-wrap strings and log messages.

Full internal/config test suite passes.
2026-06-09 21:57:03 +01:00
rcourtman ab9552716c Auto-clear event/persistent Patrol findings on affirmative deterministic verification
The finding lifecycle was asymmetric around the deterministic-resolve
gate: the gate correctly blocks LLM resolves of event/persistent
findings (backup, reliability, security, general) when the verifier
still detects the signal, but nothing ever cleared such a finding when
the underlying issue WAS fixed — absence-based stale auto-resolve only
covers performance/capacity, so a fixed backup stayed an active finding
indefinitely unless the LLM happened to call patrol_resolve_finding.

reconcileStaleFindings now runs the deterministic verifier for seeded,
unreported event/persistent findings whose key has one, and resolves
ONLY on an affirmative "signal gone" verification. Still-present or
inconclusive results fail closed (same standard as the resolve gate);
verifications are capped per run (3) with deferred candidates logged
and retried next run. Test seam: PatrolService.verifyFixResolvedFn.

Two enabling repairs found during verification:
- hasDeterministicVerifierForKey listed 2 of the 7 keys the
  verifyFixDeterministically dispatch handles, so the LLM-resolve gate
  silently skipped existing verifiers for backup-stale and
  guest-unreachable findings. Now aligned; documented as the single
  source of truth for both consumers.
- The finding-key vocabulary was forked: the patrol_report_finding tool
  suggested keys (high-cpu, ...) the verifier switch never matched, so
  verification rarely engaged for new findings. normalizeFindingKey now
  aliases unambiguous directional synonyms onto the canonical verifier
  vocabulary (high-cpu -> cpu-high etc.; pbs-job-failed and node-offline
  deliberately NOT aliased — different resource models), and the tool
  description teaches the canonical keys.

NOT changed: the synthetic ai-patrol-error finding — verification showed
the reported "accumulation" is a non-problem (deterministic ID, store
merges repeats as heartbeats, successful runs auto-resolve it).

Teeth: 8 new tests in patrol_reconcile_test.go including the
idempotence invariant (repeat reconcile over unchanged state produces
zero resolves and zero lifecycle growth) and the cap-defers case.
Contract: verified stale-resolve clause added beside the
deterministic-resolve-gate in ai-runtime.md. Full internal/ai tree green.
2026-06-09 21:47:14 +01:00
rcourtman b707512e38 Clear all errcheck and gofmt violations so make lint gates on real findings
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:

- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
  friends) now use the canonical setMockModeForTest/testutil.SetMockMode
  helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
  mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
  fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
  rollbacks in defers use the explicit-discard idiom, migration renames and
  rollup commits log failures, the hosted reaper goroutine logs an error
  exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
  discard best-effort calls (restore-chmods, handler-closure unmarshals)
  per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
  three findings test files.

Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.

Full test suites pass for every touched package.
2026-06-09 21:42:21 +01:00
rcourtman cb1ced8b8c Add the interaction-quality scenario corpus (chat-feel regression teeth)
Consolidates the OpenCode-restraint arc's user-visible promises into one
runnable corpus, mirroring the Discovery corpus pattern
(internal/servicediscovery/scenario_corpus_test.go). Each scenario
drives a full ExecuteStream turn against a scripted provider and pins
the browser-facing event stream: ordered/forbidden event types,
answer-text teeth, and payload teeth.

Six scenarios: clean plain-answer turn (done stamped with the model
route, no tool noise), greeting answered directly with tools offered
but unused, tool turn rendered as compact tool events with real tool
names and no provider call ids in the answer, clean no-narrative
fallback sentence (no raw JSON / call_ ids), invisible pre-event
provider retry (no error event), and exactly one clear error event on
terminal provider failure.

Teeth proven by mutation probes: a wrong ordered type and a wrong
answer assertion both fail with the promise named in the message.
Corpus documented in ai-runtime.md Current State as the canonical home
for interaction-quality regressions, so Patrol-phase churn on the
shared agentic loop cannot silently regress the chat feel.
2026-06-09 21:19:25 +01:00
rcourtman 945ed2198d Make the Assistant resolve a missing target before asking, not deflect
The system prompt framed ask-first as the only safe behavior for a
missing target ("Missing target information is not a safe default...
ask for the missing target"), so any "run X" / "check Y" request
without an explicit host deflected back to the operator — including on
single-host deployments with no real ambiguity. OpenCode-parity gap:
a competent operator looks first.

New policy: resolve-before-asking. Use read-only query/topology tools
to identify plausible targets; if exactly one plausible target matches,
run read-only diagnostics against it and name it in the answer; ask
only when several plausible targets remain or the action changes state.
Unchanged safety: placeholder targets (current_resource outside an
attached-resource turn) stay forbidden in all modes, never guess an
unresolved target, and write actions still need an explicit target.

Contract: resolve-before-asking entries in Extension Points and Current
State supersede the ask-first framing. Proof:
TestBuildSystemPrompt_CurrentResourceRequiresResourceHandoff pins the
new boundary strings. Live "just run X" behavior check is owed to the
interaction-quality corpus (next unit); dev-instance agent exec is
currently broken (known command-token issue), so it cannot be observed
end-to-end from here today.
2026-06-09 21:12:45 +01:00
rcourtman 36b9b2a53d Make Assistant tool selection model-owned; remove the prompt-keyword tool-scope router
assistantToolScopeForPrompt classified prompt wording into text-only /
query-only / full tool scopes — the prompt-keyword router the ai-runtime
contract forbids. It withheld every tool from greetings and exact-reply
turns, zeroed the manifest on false-positive phrases ("before using any
tools, tell me your plan"), filtered inventory prompts to pulse_query
only, and its query-only path injected a prefetched topology payload
into the user message and then withheld tools entirely.

Now every interactive turn that reaches the selected model carries the
full governed manifest from toolsForExecutionMode; the model decides
whether to answer, ask, or call tools. Removed with the router: the
query-only topology prefetch + text-only downgrade, the inventory
sanitizer allow-list span, and the summary-only pulse_query input
rewrite (preferSummaryOnlyQueries) that mutated model-chosen tool
inputs. The two contract-sanctioned behaviors survive unchanged: the
context-only resource handoff manifest boundary, and the deterministic
count-only local answer (pulse:local-inventory), now gated directly by
assistantPromptQualifiesForLocalInventoryCount whose false positives
fail safe — to the model, never away from it.

Contract: superseded the query-only/direct-text scoping paragraphs in
ai-runtime.md with the model-owned manifest rule. Proofs:
TestService_ExecuteStream_ToolManifestIsModelOwned,
TestService_ExecuteStream_InventoryBreakdownIsModelOwned,
TestAssistantPromptQualifiesForLocalInventoryCount. Full internal/ai +
internal/api trees green; chat package lints clean.
2026-06-09 21:07:28 +01:00
rcourtman 61c8e8ca00 Make agent command-exec token rejection actionable, not a silent 'Invalid token'
An agent enrolled for metrics but whose token the server doesn't recognise (or
that lacks the agent:exec scope, or is bound to a different agent) was rejected
on the command-exec WebSocket with a bare 'Invalid token' and — for the
token-not-found case — no server log at all. The agent then retried forever,
logging only 'Invalid token', so the operator had no signal that discovery
deep-scan was failing or why. (Confirmed live: delly/minipc agents pointed at a
backend that didn't recognise their token retried thousands of times; discovery
abstained for every guest as a result.)

- agentexec/server.go: the registration-rejection message the agent logs
  verbatim now says 'agent token not authorized for command execution — re-run
  the agent installer to enroll an agent:exec-scoped token'.
- api/agent_exec_token_binding.go: the previously-silent token-not-recognised
  branch now logs the specific reason with the agent hostname.

Contract-neutral: same rejection behaviour, just legible. Regression test:
TestHandleWebSocket_RejectionMessageIsActionable. Verified live end-to-end.
2026-06-09 17:55:09 +01:00
rcourtman 2b521431be Make the discovery abstention message name the real cause, not a wrong fix
The metadata-only abstention is only reached when command scanning is already
enabled but the host agent returned no command output. The old reason text told
the user to 'Enable Pulse Commands' — a toggle that is provably already on when
this message appears — so the guidance was self-contradictory. The reason now
states commands are enabled and points at the actual gap: confirm the host
agent is connected and its API token has the agent:exec scope.
2026-06-09 16:35:00 +01:00
rcourtman 5321c8b7a2 Canonicalize Proxmox-guest discovery records to the node-name key
Discovery records for Proxmox guests (VM / system-container) are canonically
keyed by node name + VMID — the form the background fingerprint loop and the
Assistant prefetch already use. But the browser action path addresses guests by
the linked agent UUID (discoveryTarget.agentId, the action-authorization target),
and normalizeDiscoveryRequest was a no-op for guest types — so a UUID-targeted
trigger/lookup stored and read records under a second, divergent key. Result: the
resource drawer reported "Not discovered" for a guest the background loop had
already discovered, and duplicate records accumulated (node-keyed + UUID-keyed).

normalizeDiscoveryRequest now canonicalizes guest targets to the node name:
resolve a linked-agent-UUID target back to its hosting node (node->linked-agent
map, falling back to the agent host's hostname when it is a known node). Both
creation (DiscoverResource) and lookup (GetDiscoveryByResource) funnel through
this chokepoint, so every path converges on the node-name key; the caller already
aliases the original target, so pre-existing UUID-keyed records are still found
and consolidated onto the canonical key on the next run.

The agent UUID is unchanged as the action-authorization target — this governs
only the record key, so the discoveryTarget security contract is untouched.

- Test: TestService_ProxmoxGuestDiscoveryCanonicalizesToNodeKey.
- Live-verified on the dev instance: the Home Assistant LXC drawer readiness went
  missing -> fresh (discoveryId system-container:delly:101).
2026-06-09 14:33:12 +01:00
rcourtman d29d9684db Assert discovery freshness reaches the model end-to-end
Extend the cloud-context e2e test to stamp the discovery with an UpdatedAt and
assert "Last discovered: 2 days ago" reaches the model on both the handoff and
@-mention paths — proving freshness survives the full ExecuteStream path (cloud
policy, sanitizer allow-list, prompt injection), not just the formatter unit.
Test-only.
2026-06-09 11:02:39 +01:00
rcourtman e2f82f8b17 Add discovery freshness to the local/full context formatter
formatSingleDiscovery (the local/full path, FormatForAIContext) reported a
resource's service, access, paths, and ports but not how old the discovery was —
so Ollama/local users lacked the recency signal cloud users just got. Emit a
"Last Discovered: <age>" header fact when the timestamp is known, completing
freshness parity across the cloud-safe and full paths. Omitted when unknown.

Test: TestFormatSingleDiscovery_IncludesFreshness (present when set, absent when
zero).
2026-06-09 10:58:38 +01:00
rcourtman 55dfdcb1d3 Instruct the Assistant to attribute facts and caveat staleness
The model now receives provenance — per-fact source/confidence and discovery
freshness — but nothing told it to use it, so answers stated facts the user
couldn't trace or weigh. Add a GROUNDING & PROVENANCE section to the base system
prompt: briefly attribute facts to their source ("Debian 12, per
/etc/os-release"), note recency for time-sensitive claims, do not present stale
context as current, and keep attribution concise rather than citing every line.

This is the prose-attribution layer of the provenance work (the visible part
chosen as the first, lowest-risk step over a heavier citations UI). Whether the
model attributes well in practice is a live-behavior question to verify by
exercising the Assistant — the prompt only guarantees the instruction reaches it.

- Test: TestBuildSystemPrompt_IncludesProvenanceGuidance.
- Contract: ai-runtime base prompt must instruct provenance attribution.
2026-06-09 10:53:05 +01:00
rcourtman 9ec52406ad Carry discovery freshness into the cloud-safe context
The pushed cloud-safe operational context told the model a resource's access
pattern, paths, and ports but not how OLD the discovery was — so the Assistant
could present a 2-week-old cached scan as current. For a monitoring assistant,
recency is the most important provenance signal.

FormatCloudSafeContext now appends "Last discovered: <age>" (via the existing
FormatDiscoveryAge helper, previously unused) when the timestamp is known, and
the push-path conversion (cloudSafeOperationalContext) carries UpdatedAt through.
A timestamp is non-identifying, so it adds no PII. Omitted when unknown.

- Tests: FormatCloudSafeContext freshness present/absent; cloudSafeOperationalContext
  carries UpdatedAt end of the push conversion (and still emits no PII).
- Contract: ai-runtime cloud-safe context must carry discovery age when known.

Follow-up: same freshness line on the local/full path (formatSingleDiscovery).
2026-06-09 10:46:10 +01:00
rcourtman a24e944f1b Carry per-fact discovery provenance through to the model
buildDiscoveryToolResponse rebuilt each fact as {category,key,value} and dropped
the Source (the command/origin that produced it) and Confidence the DiscoveryFact
already carries — so the model could state facts it couldn't attribute or weight.
Serialize both per fact (omitted when empty) so the Assistant can report "Debian
12, per /etc/os-release" instead of a bare, untraceable claim.

This is the first piece of the provenance/trust groundwork: make the model
*receive* the source metadata it needs to cite. Surfacing it to the user (UI) and
push-path discovery freshness are follow-ups.

- Test: TestBuildDiscoveryToolResponse_IncludesFactProvenance (source+confidence
  present when set, omitted when empty).
- Contract: ai-runtime section 8 documents the pulse_discovery fact provenance.
2026-06-09 10:35:29 +01:00
rcourtman ec91dd43ea Deliver cloud-safe operational context to the model on the resource handoff path
The drawer "Ask Assistant" handoff anchored a resource but never delivered its
operational context (access command, config/data/log paths, ports) to the model
— that only reached the model via @-mentions. Route handoff resources through
the same prefetch path so the proactive path matches the @-mention path on cloud
turns; PII (hostname/IP/alias) stays redacted at the model boundary.

- Reconcile the handoff Data Boundary directive: the model may use the PII-free
  operational context to answer and guide the user, while still never revealing
  raw hostnames/IPs/aliases/secrets. Action Boundary unchanged (read-only; any
  mutation goes through the governed approval flow).
- Fix a pre-existing clobber: the plain-text resource resolver overwrote the
  prefetch summary on the @-mention path, dropping the operational context
  before it reached the model. Run it only as a fallback when no structured
  mention resolved.
- Add the first end-to-end test asserting the operational context reaches a
  cloud model on BOTH the handoff and @-mention paths while PII is redacted.
- Update the ai-runtime contract to reflect the reconciled handoff data boundary,
  handoff operational-context parity, and the plain-text fallback ordering.
2026-06-09 10:18:52 +01:00
rcourtman bdb212744c Remove the cloud_context_privacy dial; fix cloud context to a lean posture
Per maintainer decision: the cloud-context-privacy feature was bloat. The real
fix for the "useless Assistant on cloud" problem was the earlier sensitivity
recalibration (ordinary workloads = Internal, not redacted); the dial layered a
configurable knob on top of an already-solved problem, guarding mostly-non-secret
data on a destination the operator opted into, and demanded every model-bound path
stay dial-aware (a standing leak surface). The privacy control users actually
understand is the choice of model — cloud provider vs. local Ollama.

Removed entirely:
- AIConfig.CloudContextPrivacy dial + constants + GetCloudContextPrivacy /
  NormalizeCloudContextPrivacy, AND the now-dead legacy
  ShareOperationalContextWithCloud boolean + ShouldShareOperationalContextWithCloud
  (internal/config/ai.go); the config-load migration (persistence.go).
- Both fields from the /api/settings/ai request/response, validation, and sync
  (ai_handlers.go) + the JSON contract snapshots.
- The "Cloud model privacy" 3-option UI control, form field, presentation copy,
  and CloudContextPrivacy type (frontend), plus their tests.
- The dial branching in the seam: chat/service.go cloudPrivacyLevel,
  CloudContextPolicy.Level + local_only suppression + the localOnly directive
  (context_prefetch.go), the inventory resourceLabel dial logic (resource_context*),
  and the modelboundary RedactLocalOnlyResourcesOnly option.

Fixed lean posture (no setting): a cloud-routed model receives real infrastructure
context, with two always-on invariants enforced by the model-boundary sanitizer —
credentials are always stripped, and local-only/Restricted resources (the floor)
never leave the local trust boundary. Local (Ollama) always full. The sanitizer's
default is now the local-only floor; it remains the universal backstop installed on
EVERY model-bound path (chat, session compaction, discovery/report/analysis via the
shared helper). Kept the two standalone fixes from this effort: compaction now
routes through the sanitizer, and directives no longer inject the "redacted by
policy" placeholder.

Governance: ai-runtime contract rewritten to a fixed-posture rule; api-contracts /
frontend-primitives / agent-lifecycle / storage-recovery dial references removed.
Tests updated to the floor-only behavior (local-only redacted, Sensitive flows,
secrets stripped). Full internal/ai/..., config, api suites green; frontend
type-check + tests + lint green.
2026-06-09 09:43:21 +01:00