Commit Graph

4152 Commits

Author SHA1 Message Date
rcourtman f18f15bf7b Release seed-pinned metrics history backing arrays on trim
The retention and max-points trims re-sliced the series front
(metrics[i:]), which keeps the entire backing array reachable while
hiding the trimmed prefix from cap(). After the dense mock trends seed
(~2,800 points/series) aged out of its 6h window, every series held a
few hundred live points while pinning its seed-sized array - roughly
700MB across the demo estate's ~8,000 series. HeapSys drifted over the
droplet's memory.high, kernel reclaim throttling starved the accept
loop, and demo.pulserelay.pro served 502s (2026-08-25 outage; verified
live via pprof: appendMetric-attributed live heap grew +117MB in six
minutes after restart while goroutines stayed at 80).

Trims now compact in place so cap() reflects the true array size, and
releaseTrimmedCapacity copies the window into a right-sized array when
it occupies under a quarter of the backing array. A healthy sliding
window sits between 1x and 2x capacity and never pays the copy.

Regression tests pin both paths: a live append after an aged-out seed
backfill, and Cleanup on a mostly-expired series, including an aliasing
assertion that catches the cap()-blind front re-slice.
2026-08-25 16:55:20 +01:00
rcourtman 6e5571997e fix: show RAID controller member disks in Proxmox storage 2026-08-25 15:19:28 +01:00
rcourtman d5440ff43e Slim the resource stream and catch up realtime gaps incrementally
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).

The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.

Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.

Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.

/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.

Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
2026-08-25 12:24:32 +01:00
rcourtman d3bfb3bb7b Tolerate heartbeat pings in websocket initial state test
TestWebSocketSendsInitialState asserted initialState arrives as the
second message, but the hub interleaves 30s heartbeat pings with the
connect sequence, so a slow state build on a loaded runner put a ping
there first and reddened main twice in three runs. Wait for each
expected type the way TestWebsocketPayloadContractShape already does.
The delivery and payload assertions are unchanged.
2026-08-25 07:19:24 +01:00
rcourtman 2ca84cddd7 Prevent connection alerts from bypassing offline policy 2026-08-24 22:45:59 +01:00
rcourtman b67ac28f25 fix(monitoring): clear host removal blocks from every store on re-enroll
allowHostAgentReenrollLocked trusted only the durable continuity store
when one was attached, so a removal block living only in persisted
monitor state (written by releases predating the store) could never be
cleared: every fresh re-enroll read as already-consumed and the host
was rejected with 400 on every report indefinitely, regardless of new
tokens or reinstalls.

Refs #1772
2026-08-24 16:36:22 +01:00
rcourtman 8d9d567f62 fix(monitoring): honor fixed poll intervals when adaptive scheduling is off
With the adaptive scheduler disabled (the default), every planning pass
on the main poll tick rebuilt each instance task with NextRun=now and
Upsert overwrote the pending slot, so availability targets polled at
the tick cadence regardless of their configured interval. Reproduced
at exactly 10s polling on a 60s ICMP target; c70431caa only covered
the adaptive-scheduler path. Planning now preserves a pending future
slot, tightening it only when a freshly shortened interval justifies
an earlier run.

Refs #1745
2026-08-24 16:36:22 +01:00
Martin Minkus 0e8f63cdd5 Resolve discovery lookups across forked host identities
Host identities fork onto "<base>-<hex>" when an agent re-enrolls under
a new token (re-running an install with --proxmox is enough), and
discovery then stores records under the forked host ID while the PVE
node keeps reporting the base agent UUID as its linked agent.
Store.ListByTarget matches TargetID byte-for-byte and
ListDiscoveriesByTarget did no alias resolution, so the resource
drawer's list call returned an empty 200 and the Discovery tab reported
"no saved discovery run" for hosts that had been analyzed.
GetDiscoveryByResource's normalization missed too: a base UUID matches
neither the forked host ID nor its hostname.

Expand the requested target into every spelling that provably names the
same target (equivalentDiscoveryTargetIDs): a matched node's linked
agent ID and name, a matched host's own ID/hostname pairing, and hex
fork suffixes of any of those, including the twice-forked
truncated-base shape. Filter the list path against that set and seed
the same spellings into the single-record alias path.

Trimmed from PR #1715 as submitted: the transitive hostname closure
(node name -> hostnames -> other hosts' IDs) is dropped. Multi-estate
deployments legitimately reuse node names and hostnames, and the hop
would have surfaced another estate's records in the drawer, discovery
readiness, and AI context. A regression test pins the non-bridging
behavior.
2026-08-24 15:18:30 +01:00
rcourtman dec1c94c27 Define recovery list pagination bounds once in the recovery model
Follow-up to 6686cdce2: the 100-default/500-max list page bounds existed
as separate untyped constants in internal/api and internal/recovery/store,
relying on a contract clause to keep them aligned. Export
DefaultListPageLimit / MaxListPageLimit from internal/recovery/model
(re-exported through internal/recovery) and alias both consumers to them,
so drift is impossible by construction. store_test.go and
recovery_handlers_test.go pin both consumers to the shared constants;
the three pagination-meta contract clauses now name the single source.
2026-08-24 10:21:10 +01:00
rcourtman 9e37d629ac Measure node connection test outcomes
Telemetry could see only saved connections, so an install that tried to
reach a node and could not was indistinguishable from one that never
opened the add-node dialog. Both report zero configured connections and
stall at the same activation stage. Fleet data shows that population is
real and concentrated three to one in container deployments, and nothing
recorded whether those installs attempted a connection at all.

Record node connection test attempts and failures in a bounded,
day-bucketed tally in the config directory, pruned to a 31-day retention
window, and report both over the install-ID rotation window as
node_test_attempts_30d and node_test_failures_30d.

Recording starts only once a request carries a target and credentials, so
an incomplete form is never counted as a node that could not be reached.
A host string that turns out to be unusable does count, because the
attempt was made and it failed. Only the add-node dialog endpoint is
instrumented: instrumenting the unused test-config endpoint as well would
double-count a single operator action.

The tally holds counts alone. Hosts, credentials, and error text never
enter it, which is why it is plain JSON rather than encrypted history.
2026-08-24 10:17:17 +01:00
rcourtman 6686cdce2c Report recovery pagination meta from the normalized limit
/api/recovery/points and /api/recovery/rollups clamp the requested page
size to [100 default, 500 max] in both the mock paginators and the store
paths, but the meta block was computed from the raw query value. A client
requesting limit=1000 with 1200 rollups was told totalPages=2 while the
server served 3 pages of 500, so iterating totalPages silently dropped
rollups; limit<=0 reported totalPages=1 at an effective limit of 100.

Normalize page and limit once at parse time, compute meta from the
normalized values, and echo the effective limit. Contract clause 34 in
api-contracts.md pins the obligation; storage-recovery and agent-lifecycle
record the boundary alignment and adjacency; recovery_handlers_test.go
pins above-max and non-positive limit meta.
2026-08-24 09:18:36 +01:00
rcourtman 814f700883 Fix main CI fallout from the 50-node demo estate
The larger demo estate overflowed the rollups mock test's single
500-row page and pushed internal/api past the 25m -race budget, and
the alert history tests raced TempDir cleanup by never joining the
periodic save worker.

Walk every rollups page in the integration test, raise the shard
budget to 50m, and stop history managers through Stop() so the save
worker is joined before cleanup.

Contract-Neutral: test-only CI fix: paginate rollups mock test, join history save worker in test teardown, raise shard go test timeout for the 50-node demo estate; no runtime or contract delta
2026-08-24 08:56:07 +01:00
rcourtman b0d6c95a93 Stamp command policy intent on hosted tenant install tokens
The hosted-tenant mint site (GenerateHostedTenantAgentInstallCommand)
issued binding-eligible install tokens without the command_policy_intent
metadata that 87ee3cd44 added to the self-hosted mint sites, so a hosted
tenant reinstalling an agent never got the stale-disabled-policy
reconciliation on first report (#1728). Hosted installs have no
enableCommands toggle, so the intent follows the install type the same
way the self-hosted flow does: pve enabled, pbs disabled.
2026-08-24 08:49:49 +01:00
rcourtman 87ee3cd449 Fix command policy after agent reinstall
Project freshly minted installer command intent onto the stable host before returning remote config, and consume that intent once so later admin policy changes remain authoritative.
2026-08-23 23:39:20 +01:00
rcourtman 10116cc17e Scale large-estate realtime monitoring 2026-08-23 19:58:40 +01:00
rcourtman 3bc613c915 Scale large-estate workload and Proxmox demo performance 2026-08-23 15:09:32 +01:00
rcourtman a7f87471f4 Stop returning the stored Apprise API key from the notifications API
GET /api/notifications/apprise and the PUT echo returned the saved API key
verbatim, so any settings:read token could exfiltrate a write-capable
secret while the sibling email handler blanks the SMTP password under the
same scope. Responses now drop the key and report hasApiKey instead, the
update and test-send paths fall back to the saved key when the request
leaves it blank (matching the email password contract), and the settings
form tells the user a saved key is kept when the field stays empty.
2026-08-23 14:39:07 +01:00
rcourtman ddf081a55a Bound Docker storage inventory collection 2026-08-23 09:34:40 +01:00
rcourtman 66eb537522 Fix local subscription CLI service setup 2026-08-23 09:28:44 +01:00
rcourtman 3a9dffa850 Fix Docker command recovery after token rotation 2026-08-23 09:18:44 +01:00
rcourtman bce5564025 fix notification terminal failure recovery 2026-08-23 09:05:54 +01:00
rcourtman 434e1448ff Restore Docker update preflight through unified agent 2026-08-23 08:35:26 +01:00
rcourtman ea5c105ff2 Log refused actions and name the missed command-agent lookup
A refused action plan, decision, or execution returned its 409 to the
client and left no trace in the server journal, so every remote report
of "Docker / Podman command agent is not connected" stalled on greps
that could never match anything. Refusals now log one warn line with
the resource, capability, and reason code, and the Docker command-agent
resolver reports which lookup missed (stale enrollment token binding vs
agent-id/hostname session), carried as an optional diagnostic detail on
the readiness contract and in the refusal envelope.

Refs #1728
2026-08-23 06:33:30 +01:00
rcourtman c313a80daa Prevent disabled PBS offline alert dispatch (#1759)
Resolve PBS alert overrides once per observation and reuse that snapshot during offline lifecycle evaluation. Add canonical-ID coverage for disabled eligibility and dispatch, enabled alerts and recovery, unrelated metrics, and concurrent configuration updates.

Contract-Neutral: Narrows internal PBS offline alert eligibility without changing external APIs or release artifacts.

Co-authored-by: rcourtman <rcourtman@users.noreply.github.com>
2026-08-22 15:52:27 +01:00
rcourtman 53ba9786c5 Stop same-short-name hosts collapsing across estates
Presentation host coalescing buckets agent rows by short hostname and
PVE node names are short, so a pve01 in one estate merged with a pve01
in another, re-parenting both estates workloads onto one row and
cross-wiring reachability. Veto the merge when the rows carry proof of
distinct machines, a disjoint machine ID set, differing DMI UUIDs, or
differing Proxmox cluster names, while equal cluster names still merge
so the same cluster added under two connections keeps its single row.
Apply the same cluster discrimination to agent-node link inference,
whose short endpoint alias keys and short-hostname corroboration
fallback could hand one estate agent to the other estate node.

Refs #1753

Contract-Neutral: behavioral bugfix in unified resources presentation coalescing and node link inference; no API shape or subsystem contract change
2026-08-22 07:58:43 +01:00
rcourtman 022db18ef7 Let a per-resource severity override re-enable disabled offline alerts
applyThresholdOverride only ever copied DisableConnectivity=true from an
override, so once the global default disabled connectivity alerts no
override could turn them back on. A guest row set to Warning or Critical
saved its severity, showed as Custom, and never fired, while the
frontend already encodes re-enablement by pairing an explicit severity
with the row offline control (off stores DisableConnectivity=true).
Treat an override that carries an explicit powered-off severity as that
row offline control set to warning or critical and clear the inherited
disable, matching the Docker container state path which already works
this way.

Refs #1738

Contract-Neutral: Refs #1738: behavioral bugfix in alerts override merge, no API or contract shape change
2026-08-22 06:24:04 +01:00
rcourtman ebb08e0de2 Make native-agent test fixtures path-portable on Windows
The dockeragent credential-store tests keyed their fake filesystem and
path-bearing env seams on POSIX literals, while dockerConfigCredentials
builds candidate config paths with filepath.Join. On Windows every lookup
therefore missed the fixture and the nine credential-expecting
TestDockerConfigCredentials_* cases failed with a nil credential; the
hostagent PULSE_PCT_PATH override test likewise asserted a POSIX path that
is not absolute on Windows. Map fixture keys and env values through
filepath.FromSlash and anchor the override on t.TempDir().

Taken from PR #1755 (ff96aed8c) so the correction lands on main instead of
a parked agent branch. The unified-agent-native Windows x64 job has been
red on every push run since these tests landed on 2026-08-13; the
intervening green runs were pull_request events exercising that PR's own
corrected fixtures.

Contract-Neutral: test fixtures only, no production runtime change.
2026-08-21 22:36:52 +01:00
rcourtman f38e95eb72 feat(telemetry): export the Patrol blocked cause at schema v10
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.

Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
2026-08-21 21:43:54 +01:00
rcourtman 7c37a85cb2 Surface and self-heal Patrol's provider-unavailable state
Field telemetry showed installs with Patrol enabled recording weeks of
empty error runs (runs_30d=122, ai_calls=0, findings=0): provider
initialization failed once at boot (model resolution can need the
provider's live catalog, so Pulse racing a booting Ollama server loses)
and was never retried, while the run loop kept recording "Patrol
provider not configured" errors that told operators who had configured
a provider to configure one.

- Retry provider initialization on every scheduled run, so a boot-time
  race strands Patrol for at most one interval instead of until the
  next settings save. LoadConfig records the redacted init failure.
- Name the real failure in the blocked reason and run record when the
  configured provider failed to initialize, instead of claiming no
  provider is configured.
- Raise the deduped Patrol runtime finding when scheduled runs are
  skipped by a persistent readiness blocker or missing provider, so the
  state reaches the findings surfaces and alert notification channels
  once, instead of living only on the Patrol page banner. Transient
  circuit-breaker blocks stay finding-free; the attempts that opened
  the breaker already raised their own.
- Resolve the runtime finding when Patrol is turned off; opting out is
  a resolution, not a state to keep nagging about.
- Record the extended runtime-failure surface in the ai-runtime
  subsystem contract.
2026-08-21 21:05:54 +01:00
rcourtman 08827bb887 Extract chart and resource query services 2026-08-21 18:36:21 +01:00
rcourtman 1327dddad5 Compress release backend test selectors 2026-08-21 17:04:18 +01:00
rcourtman 5bfcd3005a Isolate chat startup failure test data 2026-08-21 16:03:18 +01:00
rcourtman e61715462a Canonicalize install security helpers 2026-08-21 14:56:11 +01:00
rcourtman c68d5dd3d8 Extract configuration API runtime package 2026-08-21 14:56:07 +01:00
rcourtman 58bf77c1ae Extract alert delivery API package 2026-08-21 14:56:04 +01:00
rcourtman 9dac68fd62 Parallelize release qualification on PVE runners 2026-08-21 12:24:47 +01:00
rcourtman af6d482515 Keep an unresponsive mount from freezing host disk collection
A hard-mounted network filesystem with an unreachable server blocks
statfs in an uninterruptible kernel wait. The collector issued that
syscall inline for every mount, so one dead NFS mount silently froze
the whole reporting cycle and stretched agent shutdown into the kernel
retry window, and it paid that price for mounts the fstype filter was
going to discard anyway.

Decide type- and mountpoint-based skips before the usage syscall, so
network filesystems are never probed unless explicitly included, and
bound every remaining usage call with a timeout that leaves at most one
in-flight probe per mountpoint. A mount whose call never returned is
skipped on later cycles and re-included when the stalled call answers.

Refs discussion #1747

Contract-Neutral: Discussion #1747: behavioral bugfix in host disk collection; no payload or API shape change, filtered mounts were never reported
2026-08-21 06:43:31 +01:00
rcourtman b549e232a5 Keep each alert recurrence as its own history row
Since the operational-trust records shipped in v6.2.0, every new firing
of a previously resolved alert folded into that alert's first history
row: setActiveAlertNoLock stamped the new occurrence's open record onto
the previous occurrence's resolved row, and the history dedup then
treated any two open records with the same identity as one incident
regardless of how far apart they were. Alert history therefore froze at
the upgrade date while notifications kept flowing, which is exactly how
users reported it.

Resolved rows now keep their final record unless the update belongs to
the same occurrence, and two open records no longer merge on identity
alone. The observation-gap window still coalesces genuine flapping, and
the five-minute refire continuity path is unchanged.

Refs #1497

Contract-Neutral: Refs #1497: behavioral bugfix in alert history occurrence dedup; no API shape change, history row schema unchanged, no subsystem contract names sameHistoryIncident
2026-08-21 06:30:16 +01:00
rcourtman 6bad0bd884 Give notification retries a horizon that survives a destination restart
The default attempt budget was three, and with the 1s/2s/4s backoff a
destination that was unreachable for about ten seconds had its
notifications dead-lettered permanently. A webhook receiver rebooting
alongside the infrastructure it monitors is routine, not terminal.
Eight attempts under the same doubling schedule span roughly three
minutes before dead-lettering. Dead-letter semantics are unchanged.

Refs #1721

Contract-Neutral: Refs #1721: raises the default delivery attempt budget only; dead-letter semantics, retention, and outcome vocabulary unchanged, notifications.md pins no attempt count
2026-08-21 06:21:33 +01:00
rcourtman b54b4c81c7 Stop stamping powered-off severity onto overrides that never set it
normalizeOverrides ran every override through NormalizePoweredOffSeverity,
which maps an unset severity to an explicit warning. Any guest with any
per-guest override (a disk tweak, a note) therefore had its powered-off
alerts silently downgraded from a global critical default to warning, and
the stamped value also round-tripped back to the UI as if the user had
chosen it. Leave unset severities empty so the merge keeps following the
global default, and normalize only values the user actually set.

Refs #1738

Contract-Neutral: Refs #1738: behavioral bugfix in alerts override normalization; poweredOffSeverity stays an optional field, no subsystem contract names it, no API shape change
2026-08-21 06:19:12 +01:00
rcourtman 7da385cf28 Prepare v6.3.0-rc.4 release 2026-08-21 00:00:37 +01:00
rcourtman 4c7b1a2434 Fix Docker-in-LXC probe storm against slow Proxmox hosts
The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:

1. Dispatcher re-issued a probe while the previous one was still
   executing. The poll cycle's enrichment context had expired, so
   ExecuteCommand dispatched, returned the context error 50ms later,
   and the next 3s cycle sent the identical command again — unbounded
   concurrency against a host that was slow to begin with. The
   monitoring dispatcher now takes a per-guest in-flight claim before
   dispatching probe or inventory commands (completed probes release
   it; abandoned ones hold it for a 2-minute window), and both dispatch
   paths bail out under a dead context.

2. The host agent never got the July process-leak fix: 45480a5cc
   landed only on pulse/v6-release, so main-line agents killed just the
   direct shell on timeout, orphaning pct exec → lxc-attach children
   and blocking Wait on their inherited pipes (10s timeouts reported as
   300s+ durations). Port it: run each command in its own process
   group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
   treat ErrWaitDelay after a clean exit as success.

3. Server-side abandonment never reached the agent. ExecuteCommand and
   ReadFile now refuse to dispatch under an already-expired context,
   and send a best-effort cancel_command when they stop waiting; the
   agent cancels the in-flight execution (killing its process group)
   and reports "command canceled". Older agents ignore the unknown
   message type.

Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.

Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.

Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
2026-08-20 23:33:36 +01:00
rcourtman 4686efd8c8 Add a supported least-privilege agent install profile
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.

The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
2026-08-20 20:49:44 +01:00
rcourtman 74873e2b55 Add a user-facing alert delivery log and honest test-send results
The delivery health verdict said when something was wrong; nothing showed
what actually fired and where it went. Expose the queue's retained
per-attempt audit rows as GET /api/notifications/delivery-log (newest
first, retention-labeled, webhook secrets redacted from error text) and
render them as a Recent delivery activity card on the alert destinations
tab, with outcome badges, destination names, and failure classes. Audit
rows now persist the normalized destination identity in a destination_id
column; older rows fall back to their operational links.

Test sends bypass both the queue and the activation gate, which is exactly
how installs came to believe delivery worked while every real alert was
suppressed (483 installs in the 08-18 telemetry read). Successful test
responses now carry deliveryPaused: true whenever the manager is gated
off, and the destinations UI warns instead of celebrating.
2026-08-20 17:35:13 +01:00
rcourtman 23304bc017 Preserve corrupt AI findings and chat files instead of clobbering them
LoadAIFindings and LoadAIChatSessions swallow parse failures by returning
empty data with a nil error, and an undecryptable file lands in the same
branch because a failed Decrypt falls back to parsing the ciphertext as
JSON. Every read-modify-write saver (SaveAIFindingsWithSuppression,
SaveAIChatSession, DeleteAIChatSession, CleanupOldAIChatSessions) then
sees "empty, no error" and rewrites the file, permanently destroying
whatever it still held — user-authored suppression rules and chat
conversations. 9038afc68 fixed the read-error variant of this clobber;
this is the parse/decrypt variant that bypassed that abort because the
loader hid the failure.

Policy is preserve-for-recovery: the loader moves the unreadable file
aside to <file>.corrupt-<timestamp> and logs a warning before returning
empty data, keeping the self-healing behavior without the silent data
destruction. If the move itself fails, the load now fails, so the savers
abort instead of rewriting. recordActivityHistoryLocked keeps its
reset-on-parse-error behavior — those files are content-free telemetry
markers.

Tests cover corrupt and wrong-key-encrypted findings/session files being
preserved across the previously clobbering saves, and the move-failure
path aborting every saver while leaving the corrupt file intact.
2026-08-20 15:37:41 +01:00
rcourtman 9038afc687 Stop failed reads from silently clobbering preserved save data
SaveAIFindingsWithSuppression and SavePatrolRunHistory read the existing
file to carry data forward (suppression rules, the daily run tally) and
treated a failed read as nothing to preserve, rewriting the file without
it. Same clobber mechanic as the system.json reset fixed in cf5b86e62.
Both loaders return empty data with no error for a missing file, so an
error at these sites is a real read failure.

Suppression rules are user-authored config, so that save now aborts and
returns the read error. The run tally is telemetry, so that save
proceeds but logs a warning that the tally restarts. Tests cover the
abort leaving the file intact, explicit rules saving despite a failing
read, and the tally warning plus restart.
2026-08-20 14:49:59 +01:00
rcourtman ffe47fa4b7 Warn when a system settings read fails in router settings paths
A LoadSystemSettings error in configureMonitorDependencies silently
skipped the tenant-monitor inherit block, leaving the monitor on
deny-all-private webhook CIDRs with nothing in the logs to explain why
private webhook targets fail SSRF validation. reloadSystemSettings
fails closed on the same error (embedding off) deliberately, but was
equally silent, so a persistent read failure looked identical to
embedding being switched off on purpose.

Both paths now log a warning with the error, mirroring the pattern from
cf5b86e62 in internal/config. A missing system.json (fresh install)
still stays quiet. Tests cover the warning on read failure, the quiet
fresh-install path, and that the reload still fails closed.
2026-08-20 14:49:36 +01:00
rcourtman cf5b86e624 Stop a failed system settings read from resetting settings silently
A boot-time LoadSystemSettings error fell into the missing-file branch:
no log line, every persisted system setting reverted to defaults for
the run, and a default system.json was written over the file on disk.
Observed once in practice (2026-08-20): a restart came up with a
persisted toggle unset while system.json still held the correct value.

Distinguish the error from the genuinely-missing file: retry the read
once (transient failures at boot were the observed shape), then warn
with the error and file path and run on defaults without touching
system.json. Tests cover the warning, the untouched file, and the
retry-once behavior.
2026-08-20 14:34:05 +01:00
rcourtman a4d37447a0 Give the Docker-in-LXC opt-in an admin Settings toggle
Discovering Docker inside Proxmox LXC guests previously required an
undocumented environment variable plus a hand-edited systemd unit,
which is how the surface kept shipping invisible. The opt-in is now a
persisted system setting with an admin-only toggle in Settings →
System → General, presented as its own "Docker in Proxmox LXCs" card
under the existing Docker/Podman updates card, with the requirements
(node agent with command execution) and data-collection bounds stated
inline.

The setting rides the admin-gated system settings endpoint
(RequireAdmin + settings:write, matching every other server-wide
toggle). Flipping it persists first, then applies to the runtime
config and fires a reconfigure hook so the router rebuilds the
monitor's Docker checker and inventory collector immediately — no
restart. PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY remains
authoritative: when set it locks the toggle (EnvironmentLockBadge in
the UI, HTTP 400 from the API), and an unrelated settings save can
never clobber the runtime value. The opt-in guardrail test now pins
the settings path, the env lock, and the persisted field.

Installer and node setup-guide callouts now point at the toggle first
with the environment variable as the locking override, and the docs,
their public mirrors, and the agent-lifecycle subsystem contract say
the same. New i18n strings shipped for en/de/es.

Verified live in mock mode: toggle off/on from the UI drives the
collector teardown/setup log lines, the value survives a server
restart, and all three touched surfaces render at desktop and mobile
widths (receipt in frontend-modern/browser-verification.json).
2026-08-20 14:27:12 +01:00
rcourtman a64dd2f005 Give mock mode running-backup coverage
Mock mode never exercised the backup-running states from "Stop showing
an in-flight backup as a completed one": no guest carried
BackupInProgress, no PBS listing showed an in-flight snapshot, and no
vzdump task was ever running, so the blue Running badge and the Running
artifact chips had no local reproduction.

A post-scenario fixture now flags roughly one guest in twenty-five per
last-backup age bucket - never, fresh, stale, overdue - so the Running
badge appears alongside every badge story, and gives each flagged guest
the evidence a live run actually produces: an in-flight PBS snapshot
shaped the way the PBS API lists one still being written (no size, no
verification, only the guest config blob in files) and a running vzdump
task with no end time. Running after applyDemoBackupScenario keeps the
per-tick reapplication idempotent, because the scenario's curated sweep
drops the previous pass's artifacts before the fixture re-adds them.

The first-cycle showcase guests are excluded: their exact rows are
pinned by the demo posture tests (a failed run three hours ago, a guest
with no recovery evidence at all), and a backup running right now would
mask exactly the states those rows exist to demonstrate. Later profile
cycles repeat the same stories under suffixed names, so the fixture
draws its running guests from them.
2026-08-20 14:22:10 +01:00