The PVE backup poll classified storage as PBS-backed by the pbs- name
prefix as well as the storage type. A dir storage merely named pbs-*
(content backup, is_mountpoint) lost all of its vzdump backups from the
Backups view whenever a direct PBS connection was also configured,
because the poll skipped its content as 'PBS is authoritative'. Classify
by storage type only.
Fixes#1592
Contract-Neutral: behavior fix with regression test; monitoring contract docs carry another agent's WIP on the shared tree, delta deferred
The recovery store was upsert-only: backups and snapshots deleted at
the source lingered as recovery_points rows until the 90-day retention
prune. ListRollups kept returning a rollup with a frozen LastSuccessAt,
so the backup-age alert for a deleted guest re-raised every poll cycle
and acknowledging or clearing it could never stick (#1580).
Each backup poll already publishes a complete per-instance enumeration
(partial failures early-return or carry previous entries forward), so
attach a reconcile scope to that ingest batch. After the upsert, points
in the scope (provider + id class + instance) that were not part of the
enumeration are deleted, which lets the existing per-cycle alert sweep
resolve the alert. An empty enumeration is meaningful and clears the
scope, covering the delete-all-backups case from the report.
Also make the async ingest queue batches instead of overwriting the
single pending slot, which silently dropped a full poll cycle whenever
two sources coalesced behind an active batch.
The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
The RC3 memory bound for PBS backup polling summarized any group with
more than 8 snapshots into a single synthesized entry built from group
metadata. A synthesized entry has no verification, size, file, or
per-snapshot time data, so most real deployments saw every backup as
Unverified with no size, PBS files not listed, and a backup timeline
collapsed onto the latest backup day.
Keep the issue #1524 memory bounds but derive them from real data:
always fetch snapshots for stale groups, retain the newest bounded set
per group (limit raised from 8 to 100 to cover real keep policies), and
keep the newest-first global live-state cap. Remove the synthesized
group placeholder path entirely and update the monitoring subsystem
contract and tests to pin real-snapshot bounding.
Fixes#1541
Refs #1524
Refs #1510
Refs #1501
Refs #1507
Refs #1442
- persist scoped workloads status filters across platform navigation
- derive host memory pressure from available memory
- reapply system settings after every monitor reload path
- bound PBS backup snapshot polling workers during large backup scans
Back-port v5 fix 0dca8a037 to v6. Extract pollPVEBackupsAndSnapshots:
the two backup-inventory scans share a bounded backupCtx, which is then
cancelled, and pollGuestSnapshots runs on the parent context. Because
the parent has no deadline, pollGuestSnapshots establishes its own
60s-4min budget instead of inheriting an already-exhausted backup
deadline and skipping entirely (the v6 early-return at
monitor_backups.go made this strictly worse). Adds a regression test
proving snapshots still poll after the storage scan exhausts its budget.
rcourtman