Cover previously-unexecuted branches in modules that gained behavior
since the last wave. All six are newly added *_w0716_coverage_test.go
files that modify no source and no existing test.
- internal/ai/chat investigation-budget injection plus the agentic
tool-event and string helpers
- internal/ai/qualification manifest Validate error arms and the Patrol
docker-predicate and finding-prerequisite validators
- internal/unifiedresources adapters transforms and ContractResourceType
- internal/config AIConfig nil-receiver default branches
- pkg/aicontracts orchestrator error methods and investigation config
Cloned VMs that keep the same /etc/machine-id collapse into a single
Docker host in Pulse, with each clone's report silently overwriting the
other (#1584). The server now tracks per-identity hostname and machine
ID observations and flags a conflict when a value flaps back to one
already seen inside a 15-minute window, a signature a one-time rename
never produces. The conflict rides the DockerHost model through unified
resources, and the Docker page shows a warning naming the flapping
hostnames with the machine-id remedy. The warning self-clears once one
clone stops reporting for the window.
v6.1.0-rc.1 retired the legacy update endpoints before a replacement
existed, so the UI's Update button failed with an internal-jargon 410
(issue #1564). This lands the replacement end to end: update_container
is a typed agentexec operation with its own strict codec, durable
receipts, and a request digest bound to the image digest the plan
observed; the unified agent bridges execution to the Docker module's
existing pull/backup/recreate/verify/rollback implementation (which now
reports rollback attempt and outcome); and the container action
executor plans, dispatches, and reconciles the operation with declared
backup/rollback compensation truth. Containers advertise an
admin-approval update capability while an image update with a stated
current digest is detected. The legacy endpoints stay retired but
return actionable copy.
Proven live against a Colima daemon: single-container update, the
issue-1564 shared-network-namespace update, and the full UI journey
(Update button, governed review, approve, run) all completed with the
namespace preserved and the backup retained.
Two TrueNAS systems that report the same hostname collapsed into one
flapping resource (#1573, #1575): systemSourceID keyed the system by the
snapshot-reported hostname, every child pool/dataset/app/VM/share/disk
was scoped under it, and the client minted the system's machine key from
the DMI serial with a hostname fallback, so serial-less systems sharing
a hostname (and DR clones sharing a serial) also fully merged in the
identity matcher.
The system source ID now scopes to the connection ID the poller passes
through NewLiveProviderForConnection; the hostname arm survives only for
fixture snapshots, which carry no connection. The ingest identity drops
the machine key entirely (DR clones share DMI serials, and vendor
placeholder serials collide across unrelated machines), the client no
longer falls back to the hostname for MachineID, and ingest skips
identity-pin completion for SourceTrueNAS so a stale pre-fix pin or a
same-named agent host's pin cannot lend the system a machine key and
re-merge what connection scoping keeps apart. Agent.AgentID and the
native metric history keys follow the source ID minus its system:
prefix, so BuildMetricsTarget keeps resolving one series.
Rows minted under the retired hostname-keyed derivation re-key once via
record-declared succession: records name their old canonical IDs in
IngestRecord.SupersededCanonicalIDs and IngestRecords applies the
existing ApplyCanonicalIDSuccessions semantics (operator state and
action audits re-key, the superseded pin drops, never while the old ID
still belongs to a live resource, journal rows are never rewritten).
Alert identities and persisted metric series under old child IDs are
not re-keyed: active alerts re-arm under the new IDs and TrueNAS host
charts are backed by native read-through history.
dockerUpdateStatusChanged compared the whole DockerUpdateStatusMeta with
reflect.DeepEqual, so the LastChecked stamp refreshed by every periodic
update check emitted a resource_changes row per container per cycle and
flooded unified_resources.db (discussion #1577, same class as #1496).
Only availability, digests, and error text count as change now.
Cover the uncovered branches of migrateActionAuditCanonicalization (store.go):
idempotent no-op skip + len(updates)==0 early return, terminal-state exclusion
(expired not in the WHERE clause), mixed-batch skip-vs-update partition, and the
un-normalizable strict-error WARN branch (with an in-test honesty guard that
fails if the fixture stops reaching the strict path). The shipped repro test
covers only the happy path. New _test.go only; no source or existing tests
touched. Adds 4 tests; go test/vet/gofmt green.
Lifecycle transitions compare stored request/plan/origin JSON byte-for-byte
as their concurrent-writer guard. Rows written by v6.0.x re-marshal
differently after read-time normalization (backfilled plan expiry,
legacy-unknown policy decision, approval requirement), so every transition
on them matched zero rows and was swallowed as already-final. The visible
symptom: pre-upgrade pending approvals sit in the Actions inbox forever
with no way to clear them - the review dialog blocks approve/reject by
design for records without policy provenance, and the expiry sweep
silently failed to retire them.
Rewrite non-terminal audit rows to the current canonical JSON shape once
at store open; the existing expiry sweep then moves stale pre-upgrade
approvals to History on the next inbox load. Rows that cannot be expressed
in the current shape are left as stored and logged.
Reported by Johannes Strasser on v6.1.0-rc.1.
Sibling of #1559: canonicalIDFromIdentity hashed the short hostname in
its cluster arm, so two machine-keyless Docker Swarm members with FQDN
hostnames sharing a first label (cloud.a, cloud.b) derived the same
cluster:<swarm>:cloud canonical ID and fully merged in the registry.
PVE is unaffected (single-label node names hash identically).
Both hostname-derived arms now hash NormalizeFullHostname. Derivation
stays a pure function of identity so store-less registries keep deriving
the same IDs as the durable one. Compatibility for hosts minted under
the short era: journal reads already merge both eras via
ResourceIdentityPin.EraIDs, and new canonical-ID succession re-keys
resource_operator_state and action_audits rows to the successor ID at
pin persist (never across a contradicting machine key, never while the
old ID is still live, never rewriting journal rows) so operator intent
like never-auto-remediate survives the era change. Already-merged
keyless pairs cannot be retroactively split; the merged rows succeed to
whichever member persists first.
The code fix (preserve full dotted hostnames in identity pins and the
presentation host coalescer, with short/FQDN equivalence kept for
matching) rode commit 1d3b8e194 through a shared-index race. This
commit carries the unified-resources subsystem contract update and
moves the end-to-end regression test into registry_test.go.
Proposals now require their full correlation identity before anything
persists: a Submit without a finding ID or investigation ID is refused,
so a planned action can never lose the deterministic link back to its
Patrol finding. Pinned in the plan-only broker contract test.
The persisted-state transition callback becomes org-scoped
(OnActionTransition func(orgID, record)) and is wired through
ResourceHandlers.SetActionTransitionPublisher into the shared lifecycle
service, so a multi-tenant Patrol reconciler can key per-tenant stores
and can never apply a transition to the wrong tenant. Publication still
strictly follows persistence; code-standards pins guard the org-scoped
signature and wiring.
pkg/aicontracts/action_broker.go gains machine ownership: a shared
ai-runtime/api-contracts registry boundary (owned_files plus a sorted
shared_ownerships entry) with path policies proving through
pkg/aicontracts/contracts_test.go, and Shared Boundaries entries
inserted in canonical sorted order in both contracts.
Slice 2b-1 of the typed-lifecycle ratchet; the Patrol reconciler itself
lands with the orchestrator wiring now that ai_handlers.go is free.
pkg/aicontracts gains the plan-only OrchestratorActionBroker contract:
ActionProposal (typed capability reference, no command, host, risk, or
approval fields), a read-only ActionCapabilityCatalog with parameter
sensitivity, ActionDisposition over the existing safe ActionPlanInfo
projection, and an additive Action *ActionReference on
InvestigationSession and InvestigationRecord. ProposedFix/ApprovalID are
documented as migration-only; OrchestratorDeps gains the ActionBroker
seam while CmdExecutor/ApprovalStore are marked legacy pending removal.
internal/api/patrol_action_broker.go implements the seam tenant-bound
over ResourceHandlers.ActionLifecycle(): fixed pulse_patrol actor,
broker-owned ActionOrigin stamped through the service's internal
PlanWithOptions (the public plan endpoint cannot claim an origin),
plan-only submission even for ApprovalNone capabilities, and refusal of
proposals that populate IsSensitive parameters before any persistence.
The lifecycle service adds Capabilities (same registry resolution and
typed errors as planning) and an OnActionTransition persisted-state
callback covering plan, decision, and terminal execution transitions,
published only after the store write succeeds, so Patrol can reconcile
decisions and outcomes deterministically. ActionAuditRecord carries the
new broker-owned Origin, persisted in action_audits.origin_json with a
schema migration and round-trip normalization.
Contracts updated across api-contracts, ai-runtime, unified-resources,
agent-lifecycle, and storage-recovery; proofs added in
pkg/aicontracts/contracts_test.go (propose-only method set, command-free
wire shape, additive reference), internal/api/contract_test.go
(plan-only broker pins), broker behavior tests, lifecycle origin and
transition tests, and a SQLite origin round-trip test.
Slice 2a of the typed-lifecycle enforcement ratchet: additive core
fabric only; enterprise migration and side-door deletion follow.
Planning, approval decisions, and execution for typed resource actions
move out of the HTTP handlers in internal/api/actions.go into a new
internal/actionlifecycle.Service owned by api-contracts. The REST
handlers become thin decode/actor/error-mapping adapters over the one
shared service, and ResourceHandlers.ActionLifecycle() exposes the same
service for in-process consumers, so a future Patrol action broker
inherits identical resource lookup, availability checks, plan hashing,
audit persistence, remediation locks, plan-drift revalidation,
execution, and terminal publication instead of loopback HTTP or a
parallel lifecycle.
Behavior is preserved: same status codes, error codes, and audit/
lifecycle persistence ordering, backed by the existing api contract
tests plus new fail-closed proofs for the service itself (unknown
resource/capability, availability refusal, unapproved execution,
remediation lock, plan drift, missing executor, missing store).
Contract text in api-contracts, agent-lifecycle, and storage-recovery
now names the service alongside actions.go and planner.go; the
subsystem registry owns internal/actionlifecycle/ under api-contracts
with a dedicated path policy; the code-standards and contract source
pins follow the moved invariants; and the subsystem_lookup line-number
pin shifts with the api-contracts canonical-files list insertion.
This is the first slice of making the typed action lifecycle the only
autonomous execution route for Patrol, Assistant, and MCP.
In a multi-manager swarm every manager reports the same cluster-scoped
objects (services, nodes, secrets, configs), and the registry dedupe
picked the candidate whose host had the freshest LastSeen. That ordering
flips between polls, so the winner alternated: swarm node resources
re-parented to whichever manager won (ParentID tracks the reporting
host), and services flipped name/status/updateStatus whenever the
managers' views differed slightly. Every rebuild then wrote phantom
relationship_change / state_transition / config_update rows into
resource_changes, the same class of churn as the registry-rebuild spam
fixed in 74131e56e. Mock mode had the same symptom and was fixed by
pinning a single reporting leader, but real deployments still hit this.
Select winners deterministically instead: richer candidate fields first
(unchanged intent), then managers with an available control plane, and
freshness only when the gap exceeds the docker stale threshold, i.e.
the losing manager has genuinely gone quiet; otherwise the lowest host
ID wins as a stable tiebreak. Equal candidates can no longer alternate.
Regression test rebuilds the registry across polls with two managers,
jittered LastSeen ordering, flipped snapshot host order, and slightly
divergent service views, asserting no change emission; it fails against
the old LastSeen-ordering rule with the exact swarm-node re-parenting
seen in production.
Refs #1515
When an upgraded or restored Pulse server no longer recognises an agent's
API token, the report endpoint returns 401. The agent buffered and retried
that report forever with only a generic warning, and the server kept the
node green at its last known agent version because a Proxmox node stays
online via the PVE API poll even after its agent dies.
- Agent: special-case 401 on /api/agents/agent/report. Drop the report
instead of buffering it and log a throttled, actionable error pointing the
operator at the install command to mint a fresh token.
- Installer: verify_agent_server_registration now tells a rejected token
(401/403) apart from "agent has not reported yet" and prints the recovery
steps at install time instead of a vague soft warning.
- Status layer: resourceFromHost flags a stale agent (its host marked
offline by the staleness evaluator) and carries the agent's own last
report time. Coalescing keeps a node online via the PVE source but the
dead agent stays flagged, so its version is no longer presented as current.
The Machines table renders such versions as "(stale)".
Every state update rebuilds the resource registry from scratch, which
reconstructs all relationship edges with fresh ObservedAt/LastSeenAt
stamps and fresh metadata maps. recordRegistryChanges compared the old
and new slices with reflect.DeepEqual, so every relationship-bearing
resource emitted a relationship_change row on every rebuild cycle even
when nothing changed. On the public demo this wrote roughly 450k rows
per day (1.2M of 1.6M rows were literal from==to no-ops), grew
unified_resources.db to 1.6GB in three days, starved the store's single
connection until retention pruning failed with SQLITE_BUSY, and drove
the droplet into the swap-thrash outage on 2026-07-08. Same mechanism
as issue #1496.
Compare relationship sets by edge identity instead: canonical source,
canonical target, type, and active state, order-insensitive. Volatile
provenance fields no longer count as change.
Also cap resource_changes at 200k rows during retention pruning so a
pathological writer can never grow the table unbounded inside the
30-day retention window, and record both invariants in the
unified-resources subsystem contract.
When a Proxmox node has an agent installed, SMART/temperature metrics are
written by the host agent under the agent's disk source ID. But
BuildMetricsTarget preferred the Proxmox source first, and Proxmox API
does not expose detailed SMART data. The mismatch meant the frontend
queried a metrics key that nothing writes, so disk metrics appeared empty.
Move the Agent source check ahead of Proxmox/TrueNAS for
ResourceTypePhysicalDisk, mirroring the existing pattern for
ResourceTypeAgent where agent source already wins over platform sources.
When a disk has a serial number, it is used as the canonical metric ID
regardless of source (PreferredPhysicalDiskMetricID), so the reorder
only affects disks without serials — exactly the case where source
priority determines the key.
Refs #1487
The unified_resources.db grew without bound (2GB reported) because:
1. No VACUUM: DELETE freed rows internally but never shrank the file.
Added auto_vacuum(INCREMENTAL) to the DSN for new databases, plus a
one-time migrateAutoVacuum() that converts existing databases.
reclaimFreePages() now runs after each prune cycle to return freed
pages to the OS via PRAGMA incremental_vacuum.
2. Missing retention: action_lifecycle_events, export_audits, and
loop_reports had no retention at all. Added 90-day retention for
lifecycle/export audits and 30-day for loop_reports, matching the
existing action_audits/resource_changes cadence.
3. Slow cleanup cadence: the retention loop ran every 6h and never on
startup. Reduced to hourly and added an initial prune 30s after
startup so a restart with a bloated DB starts recovering immediately.
Mirrors the proven pattern from metrics.db (auto_vacuum INCREMENTAL +
incremental_vacuum + WAL checkpoint).
Refs #1496
When the SQLite resource database is corrupted (malformed disk image),
NewSQLiteResourceStore now backs up the corrupted file to
*.corrupted.<timestamp> and recreates a fresh database. Without this,
every /api/resources request returned 500 with no recovery path
until the admin manually deleted the file.
Resource data is derived from monitor state and repopulated on the
next poll cycle; user-authored metadata (links, notes) in the corrupted
file is preserved in the backup.
The resource_changes and action_audits tables grew without bound
because no retention mechanism existed. In production deployments with
frequent monitoring cycles, the database could reach multiple GB within
weeks.
Add a background goroutine that runs every 6 hours and deletes:
- resource_changes older than 30 days
- action_audits older than 90 days
The goroutine starts in NewSQLiteResourceStore and is stopped via the
retentionStop channel in Close(). The first prune runs on the first
ticker interval (6h after startup), not immediately, so test fixtures
with historical timestamps are not affected.
Fixes#1496.
rcourtman