Preserve exact reviewed tip b5b1fa9777 and upstream a85bdcdc9a additively. Retain both alert threshold test sets. Combine upstream mock-mode epoch fencing with reviewed structural-revision checks and Docker departure ownership; add a regression that a current revision cannot revive an ended epoch. Resolve the monitoring contract to describe both boundaries, without rewriting accepted history.
Publication remains deferred for genuine final-content browser receipts and independent CodeQL disposition. Composition checks do not establish native recovery, deployment, exact-head hosted acceptance or release readiness.
Change-source: pulse-maintainer
SetMockMode(false) switched the mock package off and then cleared active
alerts, but every path that evaluates mode-dependent data checked the
mode once, up front. A mock alert pass (started from the ticker with go,
so passes overlap), a frontend or ingest refresh evaluating a GetState
snapshot, a backup-age check over fixture recovery rollups, or a
connection check over the mock ledger could already hold fixture data
when the switch happened. Its evaluations then landed after the clear,
in live mode, where nothing evaluates or resolves fixture resources, so
the alerts stayed until the 24-hour stale sweep.
A pass-level lock around checkMockAlerts was tried and reverted: it left
the other paths open and made the mock-mode API wait behind a whole pass
of synchronous event-log writes.
The monitor now owns a mock-mode epoch fence. The mock pass, backup-age
and snapshot checks, connection checks, Docker pruning and the
resource-store refresh take a scope before reading anything
mode-dependent and send each alert-manager call through it. SetMockMode,
now serialized, flips the mode, ends the epoch, waits only for calls
already admitted, and then clears. Unified metrics are evaluated one
resource per call, so the wait covers single calls, not passes.
Registry rebuilds are tracked rather than waited on. A refresh evaluates
the registry it read back only if no rebuild from an ended epoch
overlapped it, and the readers that open or remove alerts from the
inventory use current state until a clean rebuild in the new epoch
replaces the previous mode's registry. The mock supplemental fixture
sources now serve nothing outside mock mode: the router swaps them for
the real pollers only after the switch, refreshing between sources, so
a single switch to live mode used to raise fixture vSphere incidents.
The new tests park a real evaluation at a fixed point (a held lock or a
blocking supplier), switch modes and release it. Against the unfenced
code the mock pass left about 40 fixture alerts, the frontend refresh 5
vSphere incidents and the backup check 137 backup-age alerts in live
mode, and a mock-mode connection observation counted toward a live
connection's confirmation. The drain, republish and pruning tests fail
against a fence without its wait, overlap check or registry tracking.
Switching mock mode with several tenant monitors running remains
uncovered and is noted in the monitoring contract.
Use observed HTTP status to distinguish authentication rejection, denial and unknown cause. Point to the actual saved connection and attempted endpoint without prescribing a role, mutating ACLs or exposing token identities or provider text. Preserve inventory and setup semantics and privilege separation; cover custom/manual identities and real-client 401/403/proxy producer paths.
Contract-Neutral: Correct existing diagnostic guidance and expose only internal HTTP-status evidence; no API shape, collection, inventory, setup permission or guest-safety contract changes.
Change-source: pulse-maintainer
Stage successful fetch times until raw state publication, preserving expired cache on every enumeration abort. Check cancellation after running-task correlation too, so an interrupted final read cannot publish incomplete artifacts or renew their cache. The next ordinary poll then refetches lost verification changes rather than reusing an old verified row.
Contract-Neutral: Repairs existing PBS cancellation, cache-publication and collection-quality semantics; no wire, schema, policy, freshness window, route or guest-probe contract change.
Change-source: pulse-maintainer
Carry snapshot and partial-namespace failures into provider history evidence, do not renew failed cache reads, and reconcile absent points only after complete enumeration. Preserve ingest ordering across partial polls and prove ordinary root-namespace identity, withdrawal, failure and recovery.
Contract-Neutral: Repairs existing PBS completeness, permission and authoritative-reconciliation semantics; no wire, policy, schema, route or guest-probe contract changes.
Change-source: pulse-maintainer
Admit detached metadata in order, merge identity fields without restoring stale
operator edits, and persist complete snapshots with one owned writer. Seal late
writes and return shutdown failures before organisation data or offboarding is
removed. Preserve existing drain budgets, file format, tenant boundaries and
atomic persistence. Add controlled burst, ordering, write-error, blocked-close
and real deletion controls.
Contract-Neutral: Behavioural repair of existing JSON persistence and organisation deletion; no route, field, permission, identity format, schema or provider authority change. Shutdown failure uses the existing HTTP error envelope. Internal ownership methods and explanatory contract text change, not the public contract shape.
Change-source: pulse-maintainer
For #2465, select latest provider evidence with the existing scope index
rather than ranking every retained payload. Preserve history and evidence
validation. Share complete, failure-preserving backup alert evaluation
across PVE/PBS/mock polls and expose unavailable reads through the existing
deduplicated system warning; keep existing age alerts until a complete read.
Contract-Neutral: Application query and existing system-alert behaviour only; no API, schema, resource identity, agent lifecycle, entitlement or notification contract shape changes. The new warning uses the existing open system-alert type, identity and metadata contract. Semantic contracts and connected verification are included.
Change-source: pulse-maintainer
Cached incomplete snapshots could retain an earlier no-writer observation when both the snapshot refresh and current task query failed. Reset task evidence before each live read so degraded visibility remains unknown and conservative.
Change-source: pulse-maintainer
Contract-Neutral: poll-scoped PBS task evidence cache correction; behavioral bug fix, no wire contract change
An in-progress PBS snapshot (reproduced on VM 117 at 9% written) was
counted as a completed backup: the guest's LastBackup jumped to "now" the
moment vzdump started, and the workloads badge showed a green healthy
shield for a backup that did not exist yet. If the job aborted, that
shield had never been true.
Detect the in-flight state from what the PBS listing actually says - no
size and no index.json.blob manifest in files - and flag the snapshot
InProgress instead of complete. On the PVE side, correlate storage content
entries with live vzdump tasks: a guest archive whose ctime falls inside a
running task's window is that task's partial output, and a PBS-storage
entry without a size has no manifest yet.
In-flight artifacts now feed a new per-guest BackupInProgress flag rather
than LastBackup, so the badge age always reflects the latest COMPLETED
backup. The workloads badge gets a distinct blue "Running" state (green
would claim success, None would hide that work is underway), with the
last-completed age kept in the tooltip and accessible label. Recovery
points map to OutcomeRunning with no completion time, keeping posture
freshness and the Backups page's latest-restore pointers anchored to
completed backups; the recoverable-artifact list shows a Running chip.
A cached in-flight snapshot forces a per-poll re-fetch of its group so
completion or abort is noticed promptly instead of after the cache TTL.
persistGuestIdentity spawned a detached goroutine per changed guest to write
guest_metadata.json, with a comment noting it avoided blocking the monitor.
Nothing tracked those goroutines, so neither Monitor.Stop nor
MultiTenantMonitor.Stop could wait for them and a queued write could land after
shutdown. In hosted mode that means a write into a tenant directory that
offboarding is already removing, and a stray guest_metadata.json.tmp left
behind when the atomic write is interrupted.
The store now owns the goroutine. SetAsync tracks the write on a WaitGroup and
WaitForPendingWrites drains it under a bounded timeout matching
tenantMonitorShutdownTimeout, so a wedged store cannot hold up tenant teardown.
Monitor.Stop drains before closing the metrics store.
This is what made TestHostedTenantAgentInstallTokenCannotReportToOtherTenant
flaky: t.TempDir cleanup raced a queued write into orgs/client-b and failed
with "directory not empty". The test itself is unchanged, because it was never
a test bug. A goroutine dump at cleanup time showed the writers still live,
created by persistGuestIdentity, blocked on the store mutex.
Verified causally rather than by observation alone: the target test fails 0/4
with the drain removed and passes 8/8 with it, against 2/3 failures on the
unmodified baseline. The regression tests fail if SetAsync stops tracking its
goroutine.
Note for a future pass, deliberately not changed here: each changed guest still
triggers a full-file save, so one poll cycle over N changed guests does N
marshals and N atomic writes that serialize on the store mutex anyway. Fixing
that means coalescing at the call site and is a behavioural change beyond this
defect.
Adversarial review of 84dba861b found three ways the new evidence paths
could still attribute a snapshot to the wrong cluster.
The submission-source learner was asymmetric. Clusters only became known
to it through snapshots that were already attributable, so a cluster with
no uniquely-attributable snapshot was invisible - and a source token both
clusters share then mapped to exactly one visible cluster and looked
decisive. The visible cluster got the other's backups while the other
guest stayed at zero. Callers now declare every connection owning a
candidate guest for a PBS instance, and the learner refuses to resolve
anything for that instance until each of them has had a snapshot
attributed to it. Observation is not scoped per PBS instance, so a
cluster seen submitting to its own PBS server still counts as visible -
the reported two-server topology keeps working.
PVE storage confirmations were treated as authorship. A pbs-type storage
listing proves the connection can SEE a snapshot, which a shared token, a
synced datastore, or an offsite copy all arrange without the connection
having made it, and a single confirmer previously outscored everything
else. Confirmations now carry the storage they came from, and only a
storage view that never lists a snapshot some other connection also lists
can attribute a colliding VMID. An overlapping view has demonstrated it
sees other clusters' snapshots, so nothing it lists attributes anything.
Where an exclusive view and the learned source mapping both speak they
must agree, otherwise the snapshot drops as it did before #1639. The
disjoint case - each cluster mounting only its own datastore - is
unchanged.
Confirmations were evicted by partial poll failures. A storage whose
content query failed contributed nothing, and the partial set overwrote
the previous one, flipping attribution between cycles. They now go
through the same per-storage preservation as storage backups.
Contract text calling the PVE listing "the only deterministic
attribution" is reworded to match the weakened semantics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v6.1.0's identity rework (eab73d245) made the VMID-only fallback refuse
to fire whenever a typed VMID exists on more than one PVE location.
Root-namespace snapshots with no matching comment then score zero for
every guest, so on setups with two clusters and overlapping VMIDs most
guests showed no backup at all - while PVE itself listed the backups
fine, because monitoring discards pbs-type storage contents entirely
whenever a direct PBS connection is configured.
Attribution is now evidence-driven instead of dropped:
Storage backup polling keeps a per-connection record of every snapshot
its own pbs-type storage listed (type, VMID, backup time) even though
the raw entries stay out of the PVE backup list. Which cluster listed a
snapshot is deterministic attribution, and it survives fully mirrored
clusters that share one datastore and token. The evidence is
monitoring-internal, cleared on instance retirement or when the storage
poll stops seeing pbs content, and never serialized into state payloads
or snapshots.
Guest backup-time sync and the recovery-point mapper additionally learn
each PBS submission source's cluster (owner token, datastore, PBS
instance - strongest first, scoped to the PBS instance) from the poll's
attributable snapshots, then resolve collision VMIDs whose snapshots
carry no evidence of their own. A source seen from several clusters is
not a discriminator, an unfamiliar component stops resolution rather
than deferring to weaker ones, and a snapshot decisively attributed to
another cluster is kept away from this one. Unattributable snapshots
still drop rather than guess.
Backup-age alert attribution no longer suffix-matches the subject ref's
connection label against guest locations. The label there is a PVE or
PBS instance name, not a PBS namespace, and loose matching could
cross-attribute clusters sharing a VMID; it now requires exact
normalized equality.
Reported in #1639 (two PVE clusters with PBS 4.0/4.1, VM 173 shown 974
days overdue despite valid verified backups).
Fixes#1639
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scheduled multi-guest vzdump jobs run under a single UPID whose VMID slot
is empty, so pollBackupTasks stored them with VMID 0 and the guest-centric
backups coverage view dropped them entirely: only individually backed-up
guests ever showed task status. (Regressed with the v6.0.0 guest-centric
redesign, which removed the flat task table that used to render job runs.)
pollBackupTasks now fetches the job task's log and parses the per-guest
markers ("Starting Backup of VM", "Finished Backup of VM (duration)",
"Backup of VM failed - reason") into synthetic per-guest BackupTask
entries. Their IDs embed the parent UPID, keeping them stable across polls
and distinct from individually-run backups; per-guest times are
reconstructed from the job start plus the printed durations. Finished
jobs' logs are immutable, so results are cached per instance|UPID and each
finished run is fetched at most once, with a per-cycle fetch cap so a
historical backlog trickles in without stalling the backup poll budget.
The task listing now uses source=all + typefilter=vzdump, so running jobs
are visible too: guests covered by an in-progress job get a "running"
synthetic task, which also feeds resolveBackupIntentContext and
suppresses offline/backup alerts for guests the job is actively backing
up. The frontend needs no changes - synthetic tasks carry real VMIDs and
flow through the existing coverage model, recovery mapper, and alert
intent paths.
Contract: monitoring.md completion obligation 13 records the per-guest
synthesis boundary; proofs land in monitor_backup_job_tasks_test.go,
monitor_alert_intent_test.go, and cluster_client_api_test.go.
Reported by Johannes Strasser (support thread "PBS Bug").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Saving backupPollingInterval, pmgPollingInterval or backupPollingEnabled
only mutated the API handler's base config. Monitors poll against a
detached DeepCopy of that config, and only pvePollingInterval set the
reload flag, so the saved values never reached shouldRunBackupPoll or
the PMG scheduler until an unrelated monitor reload happened.
Add mutex-guarded runtime overrides on Monitor with live setters, fan
them out to every tenant monitor from the settings handler (mirroring
forEachNotificationManager), and clear the per-instance last-poll
timestamps when lowering the backup interval or re-enabling backup
polling so the next cycle runs an immediate catch-up poll.
Fixes#1619
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PVE backup poll classified storage as PBS-backed by the pbs- name
prefix as well as the storage type. A dir storage merely named pbs-*
(content backup, is_mountpoint) lost all of its vzdump backups from the
Backups view whenever a direct PBS connection was also configured,
because the poll skipped its content as 'PBS is authoritative'. Classify
by storage type only.
Fixes#1592
Contract-Neutral: behavior fix with regression test; monitoring contract docs carry another agent's WIP on the shared tree, delta deferred
The recovery store was upsert-only: backups and snapshots deleted at
the source lingered as recovery_points rows until the 90-day retention
prune. ListRollups kept returning a rollup with a frozen LastSuccessAt,
so the backup-age alert for a deleted guest re-raised every poll cycle
and acknowledging or clearing it could never stick (#1580).
Each backup poll already publishes a complete per-instance enumeration
(partial failures early-return or carry previous entries forward), so
attach a reconcile scope to that ingest batch. After the upsert, points
in the scope (provider + id class + instance) that were not part of the
enumeration are deleted, which lets the existing per-cycle alert sweep
resolve the alert. An empty enumeration is meaningful and clears the
scope, covering the delete-all-backups case from the report.
Also make the async ingest queue batches instead of overwriting the
single pending slot, which silently dropped a full poll cycle whenever
two sources coalesced behind an active batch.
The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
The RC3 memory bound for PBS backup polling summarized any group with
more than 8 snapshots into a single synthesized entry built from group
metadata. A synthesized entry has no verification, size, file, or
per-snapshot time data, so most real deployments saw every backup as
Unverified with no size, PBS files not listed, and a backup timeline
collapsed onto the latest backup day.
Keep the issue #1524 memory bounds but derive them from real data:
always fetch snapshots for stale groups, retain the newest bounded set
per group (limit raised from 8 to 100 to cover real keep policies), and
keep the newest-first global live-state cap. Remove the synthesized
group placeholder path entirely and update the monitoring subsystem
contract and tests to pin real-snapshot bounding.
Fixes#1541
Refs #1524
Refs #1510
Refs #1501
Refs #1507
Refs #1442
- persist scoped workloads status filters across platform navigation
- derive host memory pressure from available memory
- reapply system settings after every monitor reload path
- bound PBS backup snapshot polling workers during large backup scans
Back-port v5 fix 0dca8a037 to v6. Extract pollPVEBackupsAndSnapshots:
the two backup-inventory scans share a bounded backupCtx, which is then
cancelled, and pollGuestSnapshots runs on the parent context. Because
the parent has no deadline, pollGuestSnapshots establishes its own
60s-4min budget instead of inheriting an already-exhausted backup
deadline and skipping entirely (the v6 early-return at
monitor_backups.go made this strictly worse). Adds a regression test
proving snapshots still poll after the storage scan exhausts its budget.
pulse-triage[bot]