The delivery health verdict said when something was wrong; nothing showed
what actually fired and where it went. Expose the queue's retained
per-attempt audit rows as GET /api/notifications/delivery-log (newest
first, retention-labeled, webhook secrets redacted from error text) and
render them as a Recent delivery activity card on the alert destinations
tab, with outcome badges, destination names, and failure classes. Audit
rows now persist the normalized destination identity in a destination_id
column; older rows fall back to their operational links.
Test sends bypass both the queue and the activation gate, which is exactly
how installs came to believe delivery worked while every real alert was
suppressed (483 installs in the 08-18 telemetry read). Successful test
responses now carry deliveryPaused: true whenever the manager is gated
off, and the destinations UI warns instead of celebrating.
A LoadSystemSettings error in configureMonitorDependencies silently
skipped the tenant-monitor inherit block, leaving the monitor on
deny-all-private webhook CIDRs with nothing in the logs to explain why
private webhook targets fail SSRF validation. reloadSystemSettings
fails closed on the same error (embedding off) deliberately, but was
equally silent, so a persistent read failure looked identical to
embedding being switched off on purpose.
Both paths now log a warning with the error, mirroring the pattern from
cf5b86e62 in internal/config. A missing system.json (fresh install)
still stays quiet. Tests cover the warning on read failure, the quiet
fresh-install path, and that the reload still fails closed.
Discovering Docker inside Proxmox LXC guests previously required an
undocumented environment variable plus a hand-edited systemd unit,
which is how the surface kept shipping invisible. The opt-in is now a
persisted system setting with an admin-only toggle in Settings →
System → General, presented as its own "Docker in Proxmox LXCs" card
under the existing Docker/Podman updates card, with the requirements
(node agent with command execution) and data-collection bounds stated
inline.
The setting rides the admin-gated system settings endpoint
(RequireAdmin + settings:write, matching every other server-wide
toggle). Flipping it persists first, then applies to the runtime
config and fires a reconfigure hook so the router rebuilds the
monitor's Docker checker and inventory collector immediately — no
restart. PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY remains
authoritative: when set it locks the toggle (EnvironmentLockBadge in
the UI, HTTP 400 from the API), and an unrelated settings save can
never clobber the runtime value. The opt-in guardrail test now pins
the settings path, the env lock, and the persisted field.
Installer and node setup-guide callouts now point at the toggle first
with the environment variable as the locking override, and the docs,
their public mirrors, and the agent-lifecycle subsystem contract say
the same. New i18n strings shipped for en/de/es.
Verified live in mock mode: toggle off/on from the UI drives the
collector teardown/setup log lines, the value survives a server
restart, and all three touched surfaces render at desktop and mobile
widths (receipt in frontend-modern/browser-verification.json).
The pct exec socket probe that discovers Docker inside LXC guests
failed invisibly: errors logged at debug only, and a failed probe never
set DockerCheckedAt, so the guest stayed in first-check state and was
re-probed on every poll forever. On a node where lxc-attach could not
enter unprivileged guests that meant ~49 pct execs per minute of
steady-state churn, with nothing at default log level to say why the
Proxmox page showed no Docker.
Track consecutive probe failures per guest in the monitor. The first
failure of a streak, and the point where the backoff reaches its cap,
log at warn with the guest, node, and error; repeats stay at debug.
Failed probes retry with exponential backoff from one minute to a
thirty-minute ceiling, a success clears the streak with an info log,
and reconfiguring the checker resets all streaks so command execution
enabled from the UI at runtime still retries immediately. Entries not
refreshed for a day belong to deleted guests and are pruned.
PULSE_PROXMOX_GUEST_DOCKER_INVENTORY_VMIDS now gates the socket probe
as well as inventory collection. It previously only limited inventory,
so opting into specific guests still probed every running LXC on every
cycle; now guests outside the allowlist are never pct exec'd at all.
Subsystem contract and configuration docs updated to match.
TestRecoveryPointsEndpointReturnsMockData sampled the unfiltered first page
of /api/recovery/points and expected to find a kubernetes point in it. That
held only while the mock estate was small.
Points come back newest-completed-first and a page is hard-capped at 500. The
rescaled Proxmox demo estate now generates 2498 points, 939 of them newer than
the newest kubernetes point, so kubernetes starts at index 939 and never
reaches page one. The 36 kubernetes points are still generated and still
served, and the endpoint, the store ordering and the fixture shape are all
correct: a platform that snapshots every few days legitimately falls behind a
platform that backs up hourly. TrueNAS was already at index 254 and would have
broken next.
Ask the endpoint per platform instead, which is how the API is designed to
answer the question and how the UI asks it. The assertion no longer tracks
estate size, and it additionally covers the platform filter.
Contract-Neutral: test-only change, no behavior or contract delta.
Two counters could not answer the question they exist for.
pulse_intelligence_patrol_runs_30d was counted from the operator-facing
patrol run history, which is capped at MaxPatrolRunHistory (100). On any
install patrolling on a normal schedule that cap is reached within hours
of a thirty-day window opening, so the field reported the cap rather than
the run count, and the calls-per-run ratio derived from it was inflated by
a censored denominator. Patrol run history now carries an uncapped daily
tally beside the capped list, advanced from a persisted high-water mark so
repeated full-list saves cannot double count, pruned to 31 days, and read
in preference to the list. An install with no tally yet falls back to the
list, so the counter never regresses on upgrade.
Schema v8 split agent-side pre-mutation refusals into target-change,
prerequisite and contract categories so they would stop hiding in "other".
In production all three are zero fleet-wide and "other" still absorbs every
refusal, because the dominant reason code is the legacy preflight_refused
aggregate recorded when an agent sends no machine reason code at all.
Agents older than the typed refusal contract report every refusal that way,
so the split is starved rather than broken, and folding the two together
made those two states indistinguishable. Schema v9 counts uncoded refusals
separately.
Pulse could tell that its own delivery had stopped and had no way to say so.
Queue health was computed for one API endpoint and rendered on the alerts
surfaces, and that was the whole escalation path: a destination that stops
delivering cannot announce itself through a notification, so an operator who
does not open the alerts page learns nothing.
Telemetry: 78 installs retry every notification to the maxAttempts ceiling and
deliver almost nothing, 155,967 attempts against 221 deliveries in a week. 50
have delivered nothing in 30 days and 28 more delivered normally at some point
and silently stopped. The dominant terminal failure classes are authentication
and configuration, so most are fixable in minutes once someone knows.
Move the queue-health rule into internal/notifications as ClassifyQueueHealth
and DeliveryHealth, next to the queue that produces the counts, and delegate
the API handler to it so the endpoint and the new evaluator cannot drift into
two rules. A queue that cannot be read reports unavailable rather than healthy,
because silence is the failure mode being guarded against.
Monitoring evaluates delivery health on the poll ticker and raises or clears
the notification-delivery system alert, which puts it in the alert list and the
navigation badge. Reading queue health costs a SQLite query and the poll
cadence can be seconds, so the evaluation is throttled to five minutes.
RaiseSystemAlert is idempotent for an unchanged condition, so the timer neither
re-notifies nor accumulates alerts.
The alert is a warning rather than critical. It is real, but a new
critical-by-default alert appearing on upgrade across the affected installs is
a bigger change than this should make on its own.
The new Monitor field is deliberately named to fit inside the existing struct
alignment column. A longer name makes gofmt re-pad the whole block, which
breaks the canonical guardrail tests that pin those field declarations
verbatim.
Proof: delivery_health_test.go pins that in-flight work and retries stay
healthy while retained terminal failures do not, and that an unreadable queue
is never reported healthy. system_alerts_test.go pins the message for each
outcome including singular and plural, that it points at the destinations
surface, and that the throttle honours its interval. internal/monitoring,
internal/notifications and internal/alerts suites all pass.
Contract-Neutral: Staged contract deltas cover every boundary this change actually moves: notifications.md gains delivery_health.go plus the ClassifyQueueHealth rule, and monitoring.md gains system_alerts.go plus the system-alert evaluation extension point. The residual demands are inapplicable. agent-lifecycle is demanded only because monitor.go is one of its canonical files, and its verification artifact is host-agent deletion and re-enrollment proof, which this change does not touch: the diff to monitor.go is one struct field and one call in the poll loop. api-contracts and storage-recovery are demanded only because internal/api/notifications.go falls under a broad internal/api reference, and that file's change is a pure delegation of an internal helper to notifications.ClassifyQueueHealth with a byte-identical HTTP response and no route or payload delta.
Adding a PVE/PBS source through the Connect via API path discarded the
name typed in the add dialog: the setup bootstrap only carried type and
host, and auto-registration named the new connection after the node's
self-reported hostname. A user asking for a cluster named enacon got a
connection called pve01 and could only rename it after adoption.
Carry the typed name on the one-time setup token
(SetupTokenRecord.DesiredName) via an optional name field on
/api/setup-script-url, and let the setup-token-authenticated
auto-register completion name a newly created connection from it,
falling back to the hostname-derived default when absent. Dedup and
cluster-member adoption identity stay hostname/candidate based, so the
carried name cannot fork an existing registration.
The app shell decides which primary platform pages exist by classifying
every resource in the legacy full-state payload, which is why it has to
download that payload before it can render navigation. This publishes the
same answer as a `platformAdmission` facet on the canonical resource
aggregations, so admission has one definition instead of two that can
drift.
Counts cannot answer it, which is the whole reason this is a facet rather
than a client-side tally over `bySource`. A TrueNAS or Proxmox host
reports through the agent source and carries the "agent" platform scope,
so a count-based derivation admits the standalone page for an estate that
has no Pulse agent in it at all. Ownership is per-resource evidence, so it
is evaluated per resource here.
Verified against the live client classifier over real estates rather than
by inspection: the facet and the classifier agree on all six pages across
ten estates each for a 652-resource and a 216-resource estate, including
the cases that break a count-based derivation (Proxmox-only, TrueNAS-only,
vSphere-only and provider-owned-agents-only estates all correctly withhold
the standalone page, while a genuine Pulse agent admits it).
The tenant-fallback contract pin gains the new field. Its invariant, that
an empty tenant returns no resources rather than seeding from the raw
snapshot, is unchanged and still pinned; an empty estate admits nothing.
No caller reads the facet yet. Moving the shell onto it is a separate
change, gated by the same parity comparison.
Contracts: unified-resources and api-contracts gain the facet and its
derivation rule; agent-lifecycle and storage-recovery gain the ownership
clause, since an agent-typed host owned by a provider must not admit the
standalone page.
Contract-Neutral: Patrol autonomy behavior is documented in its primary ai-runtime and shared api-contracts contracts plus the relevant agent-lifecycle boundary; storage-recovery only has a broad internal/api extension reference and does not own this wire contract
The gate that mirrors command-channel admission
(commandConfigAllowedForToken) runs on every agent report, before the
agent ever attempts channel registration. When it refuses, the server
silently serves commandsEnabled=false, the agent never tries to
register, so none of the channel rejection warnings fire, and an
operator who enabled lifecycle management and granted agent:exec sees
"Remote commands: Not enabled" with nothing in any log. Emit the
refusal and the binding metadata it was judged against so the mismatch
is diagnosable from journalctl.
Refs #1728
Contract-Neutral: diagnostic logging only in agent command config gate, no behavioral or public contract change
A reverse-proxy deployment that set PROXY_AUTH_ROLE_HEADER without also
setting PROXY_AUTH_ADMIN_ROLE granted every proxy-authenticated user full
administrator access. CheckProxyAuth only evaluated roles when both values
were non-empty, so the half-configuration skipped role gating entirely and
returned isAdmin=true. docs/PROXY_AUTH.md has always documented an `admin`
default for that variable, but the Config struct's envconfig `default` tags
are legacy and never applied (config.go), so nothing ever populated it.
CheckProxyAuth is the single admin verdict all 20+ proxy-auth gates consume,
so the fail-open reached every one of them. Verified on a scratch instance
with PROXY_AUTH_ROLE_HEADER set and no admin role: a request carrying only
`X-Proxy-Roles: user` received HTTP 200 and the full admin payload from
GET /api/system/settings, HTTP 200 from POST /api/system/settings/update,
and proxyAuthIsAdmin=true from /api/security/status. All three now return
403 / false, while `X-Proxy-Roles: admin` still passes.
Resolve the documented default in both layers that can produce the verdict:
config load populates ProxyAuthAdminRole when proxy auth is configured, and
CheckProxyAuth now keys role gating on the role header alone, resolving an
empty admin role through config.DefaultProxyAuthAdminRole. Configuring a
role header is the operator's signal that admin access is role-gated;
leaving the admin role unset must not switch that off.
Deployments that intentionally treat every proxied user as an admin are
unaffected: that is still expressed by leaving the role header unset.
Non-admin sessions cannot read GET /api/system/settings, so the Settings
General Monitoring Cadence card fell back to the Realtime (10s) preset
regardless of the configured interval; an issue #1601 reporter read that
as the server polling faster for non-admins. Publish the effective
pvePollingInterval on the authenticated runtime-display projection
(runtime config first, persisted value only as fallback, matching the
admin route's precedence), consume it in the viewer fallback of the
settings state, and run that initialization for sessions without
infrastructureRead too, whose ungated General panel previously never
initialized presentation state at all.
Adaptive polling deliberately stretches an instance's cadence toward its
max interval (5 minutes by default) while data is fresh, but the
connections aggregator judged staleness against the configured cadence
with a 2-minute floor. Any adaptive-enabled install therefore cycled
healthy PVE/PBS/PMG connections into stale for the back half of every
stretched poll gap: the Infrastructure page dropped the source badge
from API + Agent to Agent and connection-degraded alerts fired against
a schedule the poller was honoring.
The aggregator now scales the active-to-stale cutoff by the scheduler's
currently planned interval when that exceeds the configured cadence,
via Monitor.PlannedPollInterval and per-instance planned intervals in
the aggregator inputs. A plan tighter than the configured cadence never
tightens the cutoff, so genuine poll outages still trip the floor on
time. Connection-degraded alerts and the runtime inventory gate consume
the same derived state and inherit the fix.
Refs #1437
Contract-Neutral: behavioral fix: stale cutoff follows adaptive planned interval (#1437), no public contract delta
The Plans & Billing navigation entry is suppressed for sessions without
commercial context, which hid the only activation-form entry point from
exactly the customers who need it: a fresh Pro install has no license yet,
so hideUpgrade stayed true until the key the customer could not find a
place to paste was pasted. The compiled Pro binary is only distributed
through the paid broker flow, so a Pro-edition session is commercial
context by construction — edition.IsPro() now sets commercialContext in
securityStatusPresentationPolicy.
Ordinary free self-hosted sessions run the community binary and keep the
opt-in default (the edition marker keys off the compiled binary, never
license state). Demo mode and white-label keep full suppression via
hideCommercial, which wins over edition-derived context. Complements
1b9bc9482 (setup-completion activation pointer); together a fresh Pro
install gets both the first-run pointer and a permanently visible
Plans & Billing entry. api-contracts, storage-recovery, and
agent-lifecycle contracts record the policy delta.
Clears the red main streak that began at 63a0adf9a:
- newTestConfigHandlers never stopped its alerts.Manager, leaking a
periodicSaveAlerts goroutine that logs through the global zerolog
logger and races with the authorization-refusal contract test's
logger swap under -race
- InfrastructureAgentDoctorPage used a raw select, tripping the
settings native-select guardrail; converted to the shared labelled
FormSelect primitive
- the metric-title residual pin still expected the removed -1 sentinel
copy; aligned with the Off-control wording from 16179dd0a
Contract-Neutral: test-only CI red fix: stops alerts.Manager goroutine leak in api test helper, converts agent doctor raw select to the canonical FormSelect primitive the frontend-primitives contract already mandates, updates a stale test pin; no public contract delta
rcourtman