Performance reports answered 'what were the averages' but never 'was my
infrastructure up' - the question a managed-service client reads a
monthly report for. Reports now carry an Availability summary derived
from the recorded resource change timeline (state_transition entries
keyed by the canonical unified ID):
- uptime percent over the observed portion of the window, outage count,
total downtime, and longest outage, rendered in the executive summary
with an explicit semantics note; fleet summaries gain a per-resource
Uptime column and CSV exports gain availability header lines
- absent/unknown spans are unobserved time: excluded from the uptime
math entirely and disclosed as coverage, never counted as downtime.
The journal records a registry absence for every monitor restart, so
treating gaps as outages would invent fleet-wide downtime every time
the operator restarts Pulse
- warning states count as up (the resource is reachable and serving);
the uptime label clamps rounding so any real downtime can never
display as a clean 100%
- resources with no timeline render no availability section at all
rather than a fabricated number
Verified live against a real 7-day window: uptime/outage/downtime
figures reconcile with the raw resource_changes journal.
Performance reports were structurally disconnected from the v6 ID
space: the UI (and any API caller working from /api/state) addresses
resources by canonical unified ID, while the metrics store is keyed by
each platform's native source ID (the resource's metricsTarget). The
engine queried the store with the unified ID verbatim, so every report
rendered 'Data Points: 0' regardless of how much history existed, and
covers showed raw hash IDs a report reader cannot map to a machine.
- MetricReportRequest gains MetricsResourceID: handlers resolve the
unified ID through the tenant monitor's resource store (new
Monitor.MetricsTargetForResource accessor; the registry computes
targets on demand, they are not persisted on snapshot structs) and
the engine uses it for store queries only. Recovery points and
Patrol findings stay keyed by the unified ID.
- Legacy snapshot models and their alerts are keyed by the metrics
target ID, so enrichment now matches either ID space and resource
names/status resolve again on covers, headers, and fleet rows.
- Fleet summaries mirror the single-report guard: zero data points
across the fleet renders a muted NO DATA card instead of a green
HEALTHY 'All systems operating normally' - false reassurance is the
worst failure mode for a client-facing stability report.
- Em dashes in PDF-bound literals become hyphens; fpdf core fonts are
cp1252 and rendered them as mojibake.
Two gaps found by exercising the MSP pilot path live on a throwaway
multi-tenant instance:
1. CheckAccess granted any authenticated principal access to the default
org, so a token bound to a client org could read the provider's own
default-org estate if it leaked from a client site. Org-bound tokens
now fall through to the explicit binding check for the default org;
authenticated users and legacy unbound tokens keep default-org access,
and binding "default" explicitly still grants it.
2. The webhook private-target allowlist (instance-wide system setting)
only ever reached the default org's notification manager on
startup/reload, and only the request-context org on settings update.
Tenant orgs' webhooks to private targets (per-client Gotify over VPN,
the canonical MSP alert route) failed SSRF validation with no org-side
remedy, and any allowlist died with a restart. Settings updates and
reloads now fan out to every live tenant manager via the new
MultiTenantMonitor.ForEachMonitor, and tenant monitors inherit the
persisted allowlist and public URL at creation.
Both fixes verified live: org-bound token vs default org returns 403;
client-org webhooks to a private target succeed after restart and for
orgs created after the allowlist was saved. MSP.md validation checklist
gains the default-org probe and the allowlist guidance; MULTI_TENANT.md
documents the binding semantics. Contracts updated for api-contracts,
security-privacy, and monitoring with adjacency notes for
agent-lifecycle, storage-recovery, and performance-and-scalability.
Adds TestContract_MetadataGetPayloadsUseZeroRecordsInsteadOf404: empty
guest/docker metadata maps must serialize as {} (never null) and a
missing resource must return a 200 zero record echoing the requested ID
(never a 404). This is the proof companion to the
metadata_handlers_shared.go consolidation in the previous commit — it
was authored with that change but lost to a shared-index race at commit
time.
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
An agent enrolled for metrics but whose token the server doesn't recognise (or
that lacks the agent:exec scope, or is bound to a different agent) was rejected
on the command-exec WebSocket with a bare 'Invalid token' and — for the
token-not-found case — no server log at all. The agent then retried forever,
logging only 'Invalid token', so the operator had no signal that discovery
deep-scan was failing or why. (Confirmed live: delly/minipc agents pointed at a
backend that didn't recognise their token retried thousands of times; discovery
abstained for every guest as a result.)
- agentexec/server.go: the registration-rejection message the agent logs
verbatim now says 'agent token not authorized for command execution — re-run
the agent installer to enroll an agent:exec-scoped token'.
- api/agent_exec_token_binding.go: the previously-silent token-not-recognised
branch now logs the specific reason with the agent hostname.
Contract-neutral: same rejection behaviour, just legible. Regression test:
TestHandleWebSocket_RejectionMessageIsActionable. Verified live end-to-end.
Per maintainer decision: the cloud-context-privacy feature was bloat. The real
fix for the "useless Assistant on cloud" problem was the earlier sensitivity
recalibration (ordinary workloads = Internal, not redacted); the dial layered a
configurable knob on top of an already-solved problem, guarding mostly-non-secret
data on a destination the operator opted into, and demanded every model-bound path
stay dial-aware (a standing leak surface). The privacy control users actually
understand is the choice of model — cloud provider vs. local Ollama.
Removed entirely:
- AIConfig.CloudContextPrivacy dial + constants + GetCloudContextPrivacy /
NormalizeCloudContextPrivacy, AND the now-dead legacy
ShareOperationalContextWithCloud boolean + ShouldShareOperationalContextWithCloud
(internal/config/ai.go); the config-load migration (persistence.go).
- Both fields from the /api/settings/ai request/response, validation, and sync
(ai_handlers.go) + the JSON contract snapshots.
- The "Cloud model privacy" 3-option UI control, form field, presentation copy,
and CloudContextPrivacy type (frontend), plus their tests.
- The dial branching in the seam: chat/service.go cloudPrivacyLevel,
CloudContextPolicy.Level + local_only suppression + the localOnly directive
(context_prefetch.go), the inventory resourceLabel dial logic (resource_context*),
and the modelboundary RedactLocalOnlyResourcesOnly option.
Fixed lean posture (no setting): a cloud-routed model receives real infrastructure
context, with two always-on invariants enforced by the model-boundary sanitizer —
credentials are always stripped, and local-only/Restricted resources (the floor)
never leave the local trust boundary. Local (Ollama) always full. The sanitizer's
default is now the local-only floor; it remains the universal backstop installed on
EVERY model-bound path (chat, session compaction, discovery/report/analysis via the
shared helper). Kept the two standalone fixes from this effort: compaction now
routes through the sanitizer, and directives no longer inject the "redacted by
policy" placeholder.
Governance: ai-runtime contract rewritten to a fixed-posture rule; api-contracts /
frontend-primitives / agent-lifecycle / storage-recovery dial references removed.
Tests updated to the floor-only behavior (local-only redacted, Sensitive flows,
secrets stripped). Full internal/ai/..., config, api suites green; frontend
type-check + tests + lint green.
Introduce the single privacy dial that will govern what infrastructure context
cloud models may see, replacing the binary share_operational_context_with_cloud
toggle as the canonical operator control. This increment adds and surfaces the
setting; it does not change the redaction seam (that is increment 2).
Config (internal/config/ai.go): add AIConfig.CloudContextPrivacy with the
full|redacted|local_only levels, default "full", plus NormalizeCloudContextPrivacy
and the nil-safe GetCloudContextPrivacy getter. NewDefaultAIConfig defaults a fresh
self-hosted install to "full" so the Assistant answers with real resource detail
out of the box. The legacy ShareOperationalContextWithCloud boolean is retained as
the field the redaction seam still reads until it is wired into the dial directly.
Migration (internal/config/persistence.go): LoadAIConfig derives the dial from the
legacy toggle for pre-dial configs (legacy on -> full, off/absent -> redacted) and
persists it, leaving the legacy boolean untouched so existing installs keep their
current cloud behavior byte-for-byte. Fresh installs (no config file) default to full.
API (internal/api/ai_handlers.go): round-trip cloud_context_privacy through
/api/settings/ai field-by-field like discovery_enabled. The response always
serializes GetCloudContextPrivacy() (no omitempty) so the UI binds a 3-option
control to the concrete value; the update request carries an optional *string
validated against NormalizeCloudContextPrivacy (unknown values -> 400). When the
dial is provided it supersedes and re-syncs the legacy boolean (full -> true,
redacted/local_only -> false) so the existing seam honors the dial's full/redacted
axis without new redaction code paths.
Frontend: replace the binary "Share operational context with cloud models" toggle
with a "Cloud model privacy" 3-option FormSelect in AIRuntimeControlsSection.tsx,
bound to state.form.cloudContextPrivacy and the cloud_context_privacy payload via
useAISettingsState. CloudContextPrivacy type + payload fields in types/ai.ts;
label/help/option/summary copy in aiSettingsPresentation.ts.
Governance (ai-runtime + frontend-primitives substantive deltas; dependent
api-contracts, agent-lifecycle, storage-recovery notes): the contracts now name the
dial as canonical with the legacy boolean as the synced/migrated seam field.
Proofs: ai_config_test.go (getter/normalize/default), persistence_ai_test.go
(migration cases), ai_handlers_test.go (round-trip + legacy sync + 400),
contract_test.go JSON snapshots, settingsArchitecture + aiSettingsPresentation tests.
Live-verified in the preview drawer: dial renders with all three levels, the
migrated value (redacted) is selected, the summary updates reactively, and an
end-to-end UI save round-trips full (legacy sync true) then restores redacted.
Wire AIConfig.ShareOperationalContextWithCloud through /api/settings/ai so the
existing chat-path opt-in (commit 32d597267) is operator-reachable, not
config-file-only.
Backend (internal/api/ai_handlers.go): add share_operational_context_with_cloud
to the AI settings response (always serialized so a toggle can bind to the
concrete value) and to the update request as an optional *bool, applied
field-by-field exactly like discovery_enabled (omitted = persisted opt-in
unchanged).
Frontend: add a 'Share operational context with cloud models' toggle to the
Assistant runtime controls, bound to the canonical useAISettingsState form and
the api/ai.ts AISettings/AISettingsUpdateRequest payload. Help/summary copy
(PII-free scope, hostnames/IPs/aliases stay redacted, default off, local Ollama
always gets full context) lives in aiSettingsPresentation.ts.
Governance: substantive ai-runtime + frontend-primitives deltas plus
dependent-contract notes (api-contracts, agent-lifecycle, storage-recovery);
path-policy proofs in ai_handlers_test.go (round-trip), settingsArchitecture
and aiSettingsPresentation tests. JSON snapshot contracts updated for the new
always-serialized field.
Use neutral assistant wait status while keeping selected model route metadata visible. Make transient retry copy explicit that Pulse retries the selected route rather than switching providers.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Two follow-ups caught by independent re-review:
- #29 (8855b78c0): GetConnectionStatuses keys a name-less instance by host,
but the diagnostics lookup used 'pve-'/'pbs-'+Name only, so the
monitor-state merge was a silent no-op for unnamed instances. Fall back
to host to match.
- #22 (d9da9b18f, #1254): add a full-router test asserting an UNBOUND
agent:report token gets 404 (not 200) on config fetch — continuously
verifying the token-binding security boundary that the scope change
relies on.
#23 (a6b773a85) made Restart load the AI config once and pass it to
startWithConfig, so the start path no longer re-loads it. The test still
expected LoadAIConfig().Twice(); updated to .Once() to match the fixed
single-load behavior. (Caught by running the full internal/api suite,
which the per-fix targeted runs had missed.)
Back-port the core of v5 fix 1de1392c9 to v6 (minimal, not the full
dedup refactor). The model list API now sets ModelInfo.Provider (stamped
in prefixProviderModels and passed through HandleListModels), and the
frontend groupModelsByProvider prefers the server-supplied provider over
splitting the model id, so models with opaque ids (e.g. Ollama-hosted
llama3-8b) group under the correct provider. Adds a grouping regression
test.
Back-port v5 fix 716340dcd to v6. Extracts startWithConfig from Start so
Restart passes the config it already loaded down to the start path, instead
of Start re-reading loadAIConfig a second time. Avoids starting the AI
service with a stale/inconsistent config if it changed between Restart's
load and the start. Both Restart start paths now use startWithConfig.
Back-port v5 fix d310c257a to v6, adapted to v6's connection-status key
format. computeDiagnostics now merges a failed PVE/PBS diagnostics probe
with the monitor's live connection state: if the long-running poller still
reports the instance connected, a transient probe failure (network blip,
TLS re-check) no longer flips it to 'disconnected' in the UI. Uses v6's
'pve-<name>'/'pbs-<name>' status keys (v5 used a bare node name, which
would not match in v6). Adds a merge-logic regression test.
Back-port v5 fixes 5997fd81f and 0a7b93a84 to v6:
- GetWebhooks list response now includes the configured mention so the UI
shows it after reload instead of blanking it.
- sendResolvedWebhook now assigns data.Mention (v6 set it for grouped/
firing webhooks but dropped it on resolved), and the Discord/Slack/Teams/
Mattermost ResolvedPayloadTemplate strings gained {{if .Mention}} guards.
Without these, a configured @everyone/@channel was silently omitted from
resolved/cleared notifications. Adds list-API and per-service resolved
mention regression tests.
Back-port v5 fix 04a828264 to v6. canReadConfig now accepts
ScopeAgentReport (legacy host-agent:report) in addition to the config/
manage/settings scopes; resolveConfigAgent already restricts a
report-only token to the host it is bound to, so it can only read its
own config. Without this, a v5 agent upgraded to a v6 server while still
holding a report-only token gets HTTP 403 on config fetch and breaks.
Retargets TestHandleConfigMissingConfigScope to monitoring:read and adds
TestHandleConfigAllowsBoundAgentReportScope as a #1254 regression guard.
rcourtman