Agent install and update commands advertised http://<LAN-IP>:7655 on
installs where the operator browses Pulse over an https FQDN, because
boot-time detection pre-fills config.PublicURL with an IP guess and
resolvePublicURL treated that guess as configuration. Track
auto-detected values (boot probe or first-request capture) separately:
explicit config from the env var or the settings UI still wins, but an
auto-detected guess now yields to the origin of the admin request that
is asking, so copied commands match how the instance is actually
reached.
Also stop boot detection from clobbering a settings-persisted public
URL on restart, which silently reverted the operator's saved value
whenever PULSE_PUBLIC_URL was unset.
Refs #1692
Contract-Neutral: behavioral fix to advertised-URL derivation: no API shape, field, or endpoint changes; PublicURLAutoDetected is internal (json:-)
persistGuestIdentity spawned a detached goroutine per changed guest to write
guest_metadata.json, with a comment noting it avoided blocking the monitor.
Nothing tracked those goroutines, so neither Monitor.Stop nor
MultiTenantMonitor.Stop could wait for them and a queued write could land after
shutdown. In hosted mode that means a write into a tenant directory that
offboarding is already removing, and a stray guest_metadata.json.tmp left
behind when the atomic write is interrupted.
The store now owns the goroutine. SetAsync tracks the write on a WaitGroup and
WaitForPendingWrites drains it under a bounded timeout matching
tenantMonitorShutdownTimeout, so a wedged store cannot hold up tenant teardown.
Monitor.Stop drains before closing the metrics store.
This is what made TestHostedTenantAgentInstallTokenCannotReportToOtherTenant
flaky: t.TempDir cleanup raced a queued write into orgs/client-b and failed
with "directory not empty". The test itself is unchanged, because it was never
a test bug. A goroutine dump at cleanup time showed the writers still live,
created by persistGuestIdentity, blocked on the store mutex.
Verified causally rather than by observation alone: the target test fails 0/4
with the drain removed and passes 8/8 with it, against 2/3 failures on the
unmodified baseline. The regression tests fail if SetAsync stops tracking its
goroutine.
Note for a future pass, deliberately not changed here: each changed guest still
triggers a full-file save, so one poll cycle over N changed guests does N
marshals and N atomic writes that serialize on the store mutex anyway. Fixing
that means coalescing at the call site and is a behavioural change beyond this
defect.
Schema v6 shipped audit_logging_persistent and audit_events_30d as Pro adoption
signals. Neither discriminated. pkg/server installs the SQLite audit logger on
every install for defense in depth and gates only the read/export endpoints, so
the boolean was true on all 8 installs that had taken rc.8 and 0 rows in the
retained table have ever had it false. The event count measured that background
write volume: three of those eight unlicensed community installs were pegged at
the receiver's 100000 clamp ceiling, with the rest between 4863 and 67509.
Schema v7 replaces both with audit_reads_30d, a count of requests that cleared
the license gate on an audit read or export surface. A read requires a human
action, so unlike store presence or write volume it cannot settle into a
constant. The recorder is wrapped INSIDE RequireLicenseFeature so unentitled
requests never count, and the persisted marker carries a timestamp and a coarse
activity class from a fixed allowlist. Query filters, actors, ranges, and every
audit row read stay on the install.
The retired columns are left in the live database. They hold real rc.8 rows and
migrations only add, so dropping them would be a pointless risk; nothing writes
them once the receiver struct loses the fields.
Adds the guard this class needed. LicensedFeatureAdoptionFields registers every
field that exists to measure licensed-feature adoption, and
TestLicensedFeatureAdoptionFieldsDiscriminate builds an unused install through
the real production snapshot paths, installs a real SQLite audit logger exactly
as pkg/server does, records a baseline audit event, and fails if any registered
field is non-zero. Pinning a console logger there would have made the guard pass
while the payload lied, so it deliberately does not. The guard was verified by
reintroducing the v6 sourcing and confirming it fails with the field named.
A companion test pins the three retired fields so they cannot return under
their old names.
This is the third instance of one bug class. v6 removed
pulse_intelligence_patrol_autofixes_30d, hardcoded to zero with no increment
site, and then introduced two fields that were constant in the other direction.
Three occurrences is a guard, not a habit.
Verified end to end on a running unlicensed install: the payload that reported
audit_logging_persistent true under v6 now reports audit_reads_30d 0, and
seeding two in-window reads, one outside the window, and one with an invalid
activity class yields 2.
Six of the eight Pro-exclusive features had no telemetry field at all, so
there was no way to answer whether RBAC, audit logging, scheduled reporting,
agent profiles, alert-triggered AI, or Kubernetes AI were being used by the
installs paying for them. Schema v6 adds nine content-free adoption signals:
alert_ai_enabled AIConfig.IsAlertTriggeredAnalysisEnabled()
rbac_custom_roles non-built-in roles, per org
rbac_user_assignments user-to-role assignments, per org
audit_logging_persistent a persistent audit store is active, not console
audit_events_30d audit events retained inside the window
report_schedules configured scheduled reports
report_schedules_enabled scheduled reports switched on
report_schedules_run_30d schedules whose last run falls inside the window
agent_profiles configured agent profiles
Counts only. Role names, permissions, usernames, schedule names, delivery
recipients, report scope, profile names, and every audit event field stay on
the install. kubernetes_ai needs no field of its own: it is derivable at read
time from alert_ai_enabled combined with the existing kubernetes_clusters
count, and a dedicated field would be redundant.
Config-sourced signals are read through applyLicensedFeatureConfigSnapshot;
RBAC and audit live behind the router and are read through
Router.ApplyLicensedFeatureTelemetrySnapshot. The RBAC read goes through a new
TenantRBACProvider.PeekManager so a background telemetry read can never
provision an RBAC store for an org that has never used RBAC.
Also removes pulse_intelligence_patrol_autofixes_30d and the AutoFixCount
field behind it. patrol_run.go hardcoded AutoFixCount to 0 and no increment
site existed anywhere in the tree, so the counter was zero in all 233,364
retained production pings. That was a wiring bug, not evidence that nobody
uses Patrol fixes; governed fixes are delivered through the approved-action
pipeline, which is already instrumented. The field was plumbed through run
records, history persistence, the Assistant handoff, and telemetry while being
structurally incapable of holding a non-zero value.
Verified end to end against a running install rather than only in unit tests,
which is precisely the check the autofix counter never had: seeding three
report schedules (two enabled, one last run inside the window) and two agent
profiles produced report_schedules 3, report_schedules_enabled 2,
report_schedules_run_30d 1, agent_profiles 2 in the Settings telemetry
preview, and signing in moved audit_events_30d to 1.
The private receiver landed first in pulse-pro 78ff7dd so the new fields are
accepted on arrival.
Guest metadata (web interface URL, tags, notes) is keyed by the
node-scoped guest ID, so a VM or container migrating to another node in
the same cluster orphaned everything attached to it (#1669). VMIDs are
unique within a Proxmox cluster, so when the exact ID and the legacy
formats all miss, an entry for the same instance and VMID under a
different node is the same guest and migrates to the new node-scoped ID.
The explicit availability check link stores the hashed canonical
resource ID, which also embeds the node; that half needs a
node-independent canonical guest identity and is tracked separately.
Contract-Neutral: guest metadata follows node migration; behavioral bug fix, no wire contract or payload change
Found while triaging #1643 and #1637: both fields were stored via
system.json and accepted, validated, and echoed by the settings API,
but nothing ever consumed them. The unattended update schedule is owned
entirely by the systemd timer install.sh renders, and no UI control ever
set the fields, so the API persisted a schedule preference that could
never take effect.
Remove the fields from config.Config, SystemSettings, the settings
handlers, and the frontend config type, along with the interval
validation and the .env AUTO_UPDATE_CHECK_INTERVAL rewrite (a legacy
line is now preserved verbatim). Legacy clients that still send the
keys get them silently ignored instead of validated, and a system.json
written before the removal still loads cleanly - both behaviors are
pinned by new tests. User docs no longer describe the phantom schedule
settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Availability targets gain an optional probe agent assignment. Setting
it requires the external_probe entitlement, enforced only at the
moment of assignment - local targets never consult the license path.
Assigned targets are delivered to their agent through the signed
agent-config channel, skipped by the local poller, and resume local
execution automatically if the entitlement lapses. Probe-reported
results are accepted only from the currently assigned agent, share
the local failure-threshold accounting, carry source attribution,
and derive to indeterminate at read time when reports go stale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two clusters at different sites that reuse the same corosync cluster
name and the same RFC1918 addressing presented colliding member IPs,
which ConsolidatePVEInstances took as strong endpoint overlap and folded
the second site's connection into the first. The standalone-into-cluster
merge had the same hole for a standalone whose address collides with a
cluster endpoint at another site. The v6.1.1 node-aggregation guard
never fires in this case because it keys on cluster-name conflict, and
same-name clusters do not conflict.
Address coincidence is weak evidence across sites, but the TOFU-captured
TLS certificate fingerprints already stored on instances and cluster
endpoints are strong evidence: contradicting fingerprints for the same
authority, node name, or endpoint address mean different machines.
Consolidation now refuses to merge in that case. The fail-safe direction
is deliberate - a certificate rotation may leave a genuinely duplicated
cluster as two views, but two distinct clusters are never silently
folded into one.
Reported via support by an MSP whose two customer clusters kept merging
after the v6.1.1 aggregation fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seventeen files closing the zero-coverage functions the current source drop
left behind. Every named target was measured off 0.0 percent by a per-function
coverage delta, re-measured against current main.
- config: the five new durable Proxmox cluster-node identity helpers
(deterministic id, endpoint equality, alias lookup, id existence, lookup by
id) to 100 percent; VMware and agent-profile persistence round-trips under
t.TempDir including that AppendProfileChangeLog appends rather than
replaces; AI chat session save, load, delete, per-user scoping and age
cleanup, with explicit timestamps rather than time.Now-relative fuzz;
PVEInstance.DeepCopy asserted for nested independence.
- truenas: incidentFromPoolStatus over every pool health string,
RecordsFromSnapshot over nil, empty and populated snapshots, both
TransportStatus accessors, and the RPC handshake and auth typed errors
through errors.Is and errors.As.
- unifiedresources: the maintenance-window operator-state lifecycle on
MemoryStore including the not-found and already-cleared arms, plus the
four remaining View accessors asserted on their exact formatted output.
- api: restoreAgentExecMetadata, buildAlertConnectionSnapshotsWithRuntimeSources
and both mock series generators, asserted on shape, ordering and
determinism rather than non-emptiness.
- cmd/pulse-control-plane: the four remaining MSP and mobile proof report
printers, asserted on the concrete strings in captured stdout.
- ai: cost.EmptySummary, approval.emptyExecutionState, demo.IsDemoRuntimeIntended
and tools.findCanonicalAppContainerResourceByReferences across no-match,
first-match, later-match and ambiguous references.
- monitoring, models, alerts: trueNASAppRunning,
supplementalProviderOwnedSourcesForOrg, IOCounterPresence.Effective,
ValidAlertIntentSignal and intentTimePointer.
No source file is modified. Adversarial review returned no rejects across all
seventeen files and flagged four padding cases plus one dead table field; all
were removed and the per-function coverage re-measured as identical, proving
they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Preserve tenant-scoped metadata through partial URL updates and project stable URLs across runtime identities. Use a safe adjacent launch control across overview tables with desktop and mobile regression coverage.
Adds branch-coverage tests for eight packages whose target functions were
measured at 0% before this change. Every named target was verified to move
by running each package's coverage with and without the new file.
- internal/ai/eval: all 36 Scenario constructors and the four PatrolScenario
constructors 0% -> 100%. These are catalog-invariant tests, not literal
echoes: unique names, populated required fields, runnable assertions, tool
references checked against the agentcapabilities registry, and exact
assertion-count deltas across the env-gated conditional appends. A parity
test scans scenarios.go itself, so adding a constructor without registering
it in the table now fails rather than silently going untested.
- cmd/pulse-control-plane: nine MSP and tenant-runtime print helpers
0% -> 100%, covering the nil, empty-slice and optional-field arms.
- internal/ai/memory: RemediationLog GetByID, MarkRolledBack and
GetRollbackable 0% -> 100%, pinning the overwrite-vs-preserve contract on
RollbackInfo and each falsy arm of the rollbackable predicate.
- internal/alerts/config: AlertConfig.UnmarshalJSON 0% -> 90% and
NormalizeAlertConfigAliases 52.9% -> 94.1%.
- internal/config: RunMigrationIfNeeded 0% -> 100%, copyFile 0% -> 88.9%.
- internal/mock: AvailabilityFixtures, FixtureGraph.SupplementalChanges and
generateMockHostRate 0% -> 100%.
- internal/api: testProxmoxPlatformConnection 0% -> 100% through its injected
connect func, so no network is involved.
- internal/servicediscovery: needsDeepScan 0% -> 100% across every return arm
including the confidence boundary.
No source file is modified. Adversarial review found no rejects; two findings
were acted on before committing, replacing a circular catalog-count assertion
with the real source-parity scan and reducing an AllPatrolScenarios test that
compared the function against the same constructors it calls to the ordering
and completeness signal that is actually independent.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Contract-Neutral: test-only branch coverage, no source or contract change
Eight new branch-coverage tests taking thirty-one previously unreached
functions from zero to covered, with no source or existing test touched.
internal/kubernetesagent: twenty-one pure report helpers, including the pointer
converters proved non-aliasing in both directions, the ingress host and address
collectors across their trim, dedupe and insertion-order arms, the endpoint
slice readiness count where a nil Ready field counts as ready, and the target
role predicate.
internal/agentexec: the sudo long-option value gate over the real option list
including the inline equals form, and the approval grant verification error
unwrapped through errors.Is.
internal/alerts: the alert config alias normalization across the nil config
guard, the empty threshold early return, the blank type-key continue arm and
the legacy-delete versus supported-keep split, asserting both maps stay
independent.
internal/alerts/specs: the resource incident rollup evidence validation, each
failure arm asserted on its concrete error and the check order pinned when
several fields are invalid at once.
internal/cloudcp/docker: the not-found predicate through a wrapped error, the
route host label precedence, and the Traefik host rule parser across quoting
styles, combined matchers, multiple host clauses and malformed input.
internal/cloudcp/portal: the anonymous bootstrap builder, asserting no tenant
or user identity field is ever populated on the anonymous result.
internal/config: the legacy OIDC environment provider, including the arm where
an already-configured provider is present and the redirect derivation from a
public URL with a trailing slash.
internal/dockeragent: the update-all payload decode across wrong-typed and
missing fields, and the docker filter conversion.
Contract-Neutral: test-only branch coverage, no contract surface touched
router_routes_auth_security: all seven SAML and OIDC conversion helpers, each
driven through both its nil-input and fully populated arms with field-by-field
assertions, from 0.0% to 100.0%.
truenas_app_providers: the four TrueNAS app-to-tool mappers for ports,
networks, volumes and containers, plus the log output formatter, each over
nil, empty, populated and partially populated input, from 0.0% to 100.0%.
config report_schedules and multi_tenant: the report schedule store path,
load and save, and the organization load, strict load, save, list and delete
operations, all over a temp directory. The pair contrast that matters is
covered explicitly: LoadOrganization defaults for a missing org where
LoadOrganizationStrict errors. Targets move from 0.0% to between 87.5% and
100.0%, with incidental lift on the surrounding normalize and parse helpers.
All four files are new; no source or existing test was touched.
New *_branchcov0718_test.go files raise coverage of previously-uncovered pure
functions across ten packages. Covered areas include securityutil SSRF and URL
validation, truenas path and telemetry parse helpers, storagehealth SMART and
physical-disk risk assessment, vmware inventory sort keys and error classifiers,
servicediscovery token filtering and readiness, telemetry evidence-from-history,
models ToFrontend converters and frontend NormalizeCollections normalizers,
actionplanner type predicates and canonical resource-id sort, config API-token
accessors, and licensing state accessors.
Test-only, with no runtime or subsystem-contract change. Verified in a clean
worktree at HEAD with go vet and package tests green, gofmt clean, and every
named target function moved from 0 percent to covered.
Patrol findings previously reached operators only through the Relay
mobile push path. Anyone relying on the email, webhook, or Apprise
destinations they already configured for alerts got no proactive signal
when Patrol detected a problem and had to open /patrol to learn about
it. This is the delivery half of #1369.
Each newly stored warning or critical finding now also flows through a
FindingNotifyCallback wired in the router, which projects the finding
into the alert shape the notification manager delivers. The callback
fires only on the genuinely-new path in recordFindingWithInvestigation,
so a finding notifies at most once per lifetime regardless of how many
later runs re-detect it, and SendAlert's own per-ID cooldown backstops
that. Demo mode never notifies.
Gating lives in AIConfig via patrol_finding_notifications_enabled
(default on, matching the long-standing default for mobile push) and
patrol_finding_notify_min_severity (warning or critical, default
warning). The enabled flag persists without omitempty so an explicit
opt-out survives reload while pre-existing configs inherit the default.
The settings surface for these fields follows in a separate commit once
the AI settings handler is free.
Cover previously-unexecuted branches in modules that gained behavior
since the last wave. All six are newly added *_w0716_coverage_test.go
files that modify no source and no existing test.
- internal/ai/chat investigation-budget injection plus the agentic
tool-event and string helpers
- internal/ai/qualification manifest Validate error arms and the Patrol
docker-predicate and finding-prerequisite validators
- internal/unifiedresources adapters transforms and ContractResourceType
- internal/config AIConfig nil-receiver default branches
- pkg/aicontracts orchestrator error methods and investigation config
Adds table-driven branch-coverage unit tests for previously uncovered pure
functions across internal/models, internal/config, pkg/securityutil and
pkg/aicontracts. New test files only, with no source changes.
Covers model collection normalizers, reflect-based metric deep-copy,
removed-host agent bookkeeping, report-schedule and entitlement-billing
normalization, effective AI control and Patrol autonomy resolution,
websocket-origin host normalization, and command-safety classifiers. 61
TestBranchCov functions in 8 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously untested pure
helper functions across internal/alerts/config, internal/config,
internal/ai/safety, internal/ai/modelresolution, internal/operationreceipt,
internal/models, internal/securityutil and pkg/securityutil. New test files
only, with no source changes.
Covers alert-config normalization and validation, sensitive-path and
redaction classifiers, URL normalizers, provider model resolution,
operation-receipt decoding, credential masking, and account-to-org role
mapping. 57 TestBranchCov functions in 12 files, all vet and gofmt clean.
Discovery is high-fan-out background work (one model call per container or
service), but GetDiscoveryModel fell back straight to the shared default
model. An operator who picked a cheap Patrol model to keep background AI
inexpensive still had scheduled service-context refreshes burning the
expensive shared model, dozens of calls per run, with no chat activity
(reported by a 6.0.5 user via support). Discovery now follows the same
fallback chain as auto-fix: DiscoveryModel, then PatrolModel, then the
shared default. Settings copy and the ai-runtime contract state the chain.
Ollama is the zero-cost AI path but the setup row offered only a Server
URL, and Patrol then failed on models that cannot emit tool_calls
(#1463, #847, #1425, #1152, #880). Bless qwen3:8b, the model family
Ollama's own tool-calling docs are written against, verified locally
against Patrol's real preflight: qwen3:8b emitted the tool call on
every run; qwen3:4b never did (0/4), so no low-RAM tag is suggested.
- Registry: SuggestedModel/Note/Equivalents on AIProviderDefinition,
projected on /api/settings/ai providers; Ollama default model goes
llama3.2 -> qwen3:8b.
- Provider row: copyable 'ollama pull qwen3:8b' block with hardware
note, and a next-step hint when a successful test resolves a model
outside the blessing set.
- Model resolution: exact-ID blessed preference, so pulling qwen3:8b
makes it the auto-resolved Patrol model with no manual selection.
- Readiness copy names the blessed model (its contract pins landed
with 94ccc7a0c's staging; this commit restores green).
- manual_ollama_preflight_test.go is the env-gated re-blessing
harness; contracts updated for ai-runtime, api-contracts,
frontend-primitives, and the agent-lifecycle/storage-recovery
dependent boundaries; subsystem_lookup_test line pins follow the
api-contracts.md insertion.
The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
All TCP and HTTP availability probes were failing with 'no route to host'
because the Pulse process's Go networking stack cannot reach local subnet
IPs when a Tailscale system extension is active. The ICMP probe already
worked because it uses exec.Command('ping'). Apply the same pattern:
- probeTCP: try Go net.Dialer first, fall back to exec.Command('nc')
- probeHTTP: try Go http.Client first, fall back to exec.Command('curl')
- Both fallbacks use OS-appropriate timeout flags (macOS: -G, Linux: -w)
- HTTP probes now skip TLS verification (availability check, not identity)
- Add AvailabilityProbeHTTPS protocol constant and wire it through
Validate, HTTPURL, Normalize, and the probe switch
- Replace bulky TooltipPortal tooltip on AvailabilityProbeCell with a
clean native title attribute ('14 ms - checked 3s ago')
Before: 0/20 probes green (all 'no route to host')
After: 14/20 probes green (6 legitimately down hosts)
Agentless availability checks (ICMP/TCP/HTTP) were always minting
standalone network-endpoint resources, leaving them disconnected from
the known Proxmox/Docker guest they actually monitor. This made
availability evidence invisible on the platform resource row where the
user expects it, per the performance-and-scalability bounded-row
contract.
Backend (unified-resources ingest):
- Add LinkedResourceID field to AvailabilityData and AvailabilityTarget
- resolveAvailabilityLink: explicit link first, then exact-IP unambiguous
correlation; skip hostname-only (lossy); guard against overwriting a
different target's facet
- Unlinked/unmatched probes still mint network-endpoint (fallback)
Frontend:
- Relax getAvailabilityProbePresentation for any resource with availability
- Add compact protocol badge to UnifiedResourceHostTableCard name cell
- Add optional 'Link to resource' field to availability target form
- Add linkedResourceId to frontend types
Contracts: api-contracts, unified-resources, monitoring,
performance-and-scalability, storage-recovery.
Governance: coverage_gap + candidate_lane in status.json.
Z.ai keys on the coding subscription get 429 'Insufficient balance' on the standard /api/paas/v4 endpoint. Add a per-provider BaseURLField to the Z.ai registry entry, a ZaiBaseURL override on AIConfig (returned by GetBaseURLForProvider when set, else the standard default), the handler request/response/apply plumbing, and a 'Custom Base URL' field on the Z.ai provider card so operators can point at /api/coding/paas/v4. Mirrors the existing OpenAI custom-base-URL override; the standard endpoint remains the default for pay-as-you-go users. Adds config/handler/frontend proofs and updates the ai-runtime, api-contracts, frontend-primitives, agent-lifecycle, and storage-recovery contracts.
Add a registry-backed provider model for runtime metadata and chat-compatible transports.
Support Z.ai, Groq, Mistral, Cerebras, Together, and Fireworks through the shared chat-compatible client path while keeping native providers on their existing adapters. Expose provider metadata through settings and render the expanded provider set in the frontend.
rcourtman