Scope TrueNAS child source IDs under the system source key so duplicate pool, dataset, app, VM, share, and disk names from different appliances do not merge.
Use the resulting hierarchy for per-system TrueNAS overview counts and scoped mock metric targets.
Every ~5 minutes the Pulse agent on a node already covered by an
existing cluster endpoint re-announces itself via POST
/api/auto-register. The handler ran the full add-the-standalone +
consolidate path, which produced an identical persisted-config
state. But it still wrote nodes.enc and called reloadFunc, which
stops the multi-tenant monitor, sleeps 1 s, and rebuilds it from
scratch. During the rebuild the WebSocket-visible resource store
goes empty (nodes:0, vms:0, containers:0, dockerHosts:0), so every
connected client sees every row flash blank and reappear. The
Docker page made the flicker obvious because the rows are dense
and stacked, but the regression hit every page in the UI.
Fingerprint the persisted-config payload before any mutation and
again after consolidation; when the fingerprints match, the
auto-register is a no-op and SaveNodesConfig + reloadFunc are both
skipped. Real changes (a new node, a rotated token, a captured
fingerprint) still flow through unchanged.
Add two regression tests: one drives the warm-up + steady-state
heartbeat pair and asserts the second call does not call
reloadFunc; the other rotates a token and asserts reloadFunc still
fires.
Mail Gateway rows now expand inline to show the detail the slim
ResourcePMGMeta projection couldn't carry: every cluster node with
status / role / uptime / loadavg / postfix queue (active/deferred/
hold/incoming counts + oldest message age), quarantine breakdown
(spam / virus / attachment / blacklisted), spam-score distribution,
top 8 domains by mail volume, and the mail-flow detail strip
(bounces in/out, greylist, junk, RBL + pregreet rejects, bytes
in/out, average process time).
Drawer fetches /api/pmg/instances?id=<id> on first open so the row
keeps its slim payload; the new endpoint projects
Monitor.GetState().PMGInstances directly under ScopeMonitoringRead
with optional id / name filters.
Same inline expand-row pattern as the ceph and workloads drawers
(TableRow inserted below, colspan all columns).
Both sub-tabs were mounting the same RecoverySurface with a forced
platform filter, so /proxmox/replication and /proxmox/backups looked
identical and neither was bespoke to its title.
Replication is a PVE-only concept (zfs send/receive between nodes).
Surface it through a dedicated /api/replication/jobs endpoint that
projects Monitor.ReplicationJobsSnapshot, and render a bespoke
ProxmoxReplicationTable with status, job ID, guest, source→target
nodes, schedule, last sync, duration, fail count and error columns.
PVE backups split into three meaningfully different artifacts:
qm/pct snapshots, vzdump archives on storage, and recent backup-job
tasks. Expose all three through /api/backups/pve (projecting
Monitor.PVEBackupsSnapshot) and render a bespoke ProxmoxBackupsTable
with three sub-tabs, each with columns tuned to that artifact class.
Both endpoints are read-only under ScopeMonitoringRead and accept
optional instance/node/vmid/platform query filters.
Show workload-capable source failures on Workloads and keep matching Proxmox host agents attached to their API source when inventory collection is blocked.
Keep desired config fingerprints as response metadata derived from the signed command and settings payload.
Use merged agent profile settings when building remote config fingerprints.
Ensure dry-run-only actions fail with action_dry_run_only before executor availability checks, and bridge action completion verification projection through router payload mapping.
Wire the alerts manager's new flapping-detected callback in the AI
intelligence initialization path. Two things happen on each first
transition into the flapping cooldown window for a tracking key:
1. A reliability-category finding is written directly to the findings
store via emitFlappingPostmortemFinding. Path B from the lane brief:
the finding is durable without depending on patrol synthesis, so
the operator sees the diagnosis the moment Pulse decides to
suppress. The finding ID is derived from the canonical tracking
key ("alert-flapping:<trackingKey>") so re-detection inside the
cooldown window folds into the existing record via the same-ID
branch of FindingsStore.Add -- one finding per flapping condition,
not one per dispatch.
2. A scoped FlappingPostmortemPatrolScope is enqueued on the trigger
manager so an actual patrol run can enrich the finding with deeper
context once it lands.
The finding body names the flapping threshold, window, and cooldown
the manager is currently configured with, plus an action hint
(widen threshold, raise cooldown, or stabilise the resource). That
turns the suppressed alert from silence into a closable item on the
FindingsPanel.
FindingCategoryReliability is reused; no new category, no parent/
child finding structure -- those are deferred per the lane brief.
ActionAuditRecord gains a VerificationOutcome{status, evidenceSummary}
field with a closed enum (unknown/verified/unverified/failed). Existing
records read back as unknown by default via the normalizer and a new
SQLite column verification_outcome_json. The redaction pass scrubs the
evidence summary alongside other operator-authored text.
A new agentexec/verifier_postconditions.go registers postconditions for
qm.start, pct.start, docker.restart, systemctl.restart, and
kubectl.rollout, each parsed by verifier_postconditions_test.go.
Three pre-existing action JSON snapshot tests
(TestContract_ActionDecisionJSONSnapshot,
TestContract_ActionExecutionJSONSnapshot,
TestContract_UnifiedActionAuditsJSONSnapshot) now include the new
verificationOutcome field. The two flagged failing contract tests on
this branch
(TestContract_ActionDryRunOnlyExecutionErrorJSONSnapshot,
TestContract_RouterBridgesVerificationOntoActionCompleted) are
unrelated to this change and were left alone per lane D-002 scope.
When a maintenance window ends on a resource, the sentinel runs
deterministic checks (active alerts, Patrol findings, failed actions
since window start, basic post-window metric recovery) and writes a
durable LoopReport. Operators can list reports per resource, mark them
reviewed, or rerun verification immediately. UI surfaces the section in
the resource detail drawer; scoped Patrol runs and Assistant deep-link
are deferred until those entry points stabilise.
A Proxmox host wedged on a ZFS deadlock yesterday took the cluster API poll
with it (context deadline exceeded). The unified connections aggregator
flipped the Connection from active to stale to unreachable, and the
Settings / Infrastructure page rendered the right badges, but no top-nav
alert ever fired because nothing was actively notifying off that derived
state. Patrol's deterministic triage flagged it every minute, but its LLM
investigation stage has been broken since 2026-02-26 so flags never
escalated into user-visible findings. Result: a 3 hour outage I only
noticed because I happened to open Settings.
This wires an active notification off the same connection state the
Settings badges already use:
- internal/alerts/connection.go: new CheckConnection +
clearConnectionDegradedAlert that fire connection-degraded after three
consecutive stale or unreachable observations. Severity scales: stale
warning, unreachable / unauthorized critical. Clear runs through the
same recovery-confirmation gate as clearNodeOfflineAlert so a single
flap back to active doesn't silently resolve a real outage. Paused,
disabled, and non-platform connections are no-ops.
- internal/api/connections_alerts.go: snapshot translator that turns
api.Connection into the narrow alerts.ConnectionSnapshot view. Keeping
the snapshot type inside the alerts package preserves the existing
api -> monitoring import direction; the monitor would have cycled if
it called back into api directly.
- internal/monitoring: new SetConnectionsSnapshotLister hook + a
per-tick checkConnectionAlerts call in the main poll loop, alongside
the existing evaluate*Agents passes.
- internal/api/router.go: register the lister closure on r.monitor so
the alerts loop sees the same Connection rows the HTTP handler does.
- internal/alerts/specs/types.go: add "connection" to the migration
bridge list of accepted ResourceTypes, alongside node / docker-host /
proxmox-disk / etc. The connection concept doesn't have a canonical
unified resource type yet; this matches the existing pattern for
alert-keyed resources that aren't first-class canonical.
Test coverage in internal/alerts/connection_test.go covers active never
fires, three stale observations escalate from pending to warning,
unreachable escalates warning to critical, unauthorized fires critical
cold, paused / disabled / agent never fire, recovery confirmation gate,
and a stale flap during recovery resets the gate.
TestResourceAlertSpecValidateAllowsConnectionMigrationBridgeType mirrors
the existing migration-bridge proof tests for the new type.
rcourtman