An explicit connection address override on a cluster member was silently
discarded when VerifySSL was enabled and the member had no per-endpoint
fingerprint: the hostname-for-TLS preference displaced the operator's
address, so overriding an undialable discovered hostname changed nothing.
The override now wins in every TLS mode, and the hostname preference
applies only to auto-discovered addresses.
Failed endpoint recovery attempts also logged their cause at debug level
only, leaving the recurring 'No endpoints recovered' warning without a
reason. The warning now carries per-endpoint failure reasons and the
sanitized error is stored so the UI health status shows it too.
Refs #1665
Contract-Neutral: behavioral bug fix refs #1665: cluster member IPOverride honored in all TLS modes, recovery failure reasons surfaced; no public contract delta
* Honor the configured request timeout for discovery AI analysis
Both discovery services accepted an AIAnalysisTimeout config field, but
neither production construction site ever set it, so every AI analysis
call ran under the hardcoded 45s fallback. RequestTimeoutSeconds reached
the provider HTTP client but never the per-analysis context deadline, so
raising it could not help a slow local model: the outer 45s deadline
always fired first and surfaced "AI analysis timed out after 45s".
Add GetDiscoveryAIAnalysisTimeout, which returns the configured request
timeout when one is set and keeps the tighter 45s discovery default
otherwise, and wire it into both service constructors. Both services also
gain SetAIAnalysisTimeout so a settings change applies live instead of
requiring a restart; the timeout is now read through a mutex-guarded
accessor since it can be mutated while scans are running.
* Inherit the configured request timeout for discovery analysis
Returning 45 seconds when RequestTimeoutSeconds is zero left the
discovery deadline disagreeing with what operators are shown. Zero is
omitted from the settings response, the frontend displays 300 via
request_timeout_seconds ?? 300 and states that value applies to
discovery, and it only submits the field when the form value differs
from the displayed current value. An installation showing 300 seconds
therefore still timed out discovery after 45, and saving the visible
default did not correct it.
GetDiscoveryAIAnalysisTimeout now delegates to GetRequestTimeout so the
persisted default canonicalizes to 300 seconds. The discovery services
keep their own 45-second fallback for construction without an AI
configuration.
smartmontools 7.5 emits power_mode as an {ata_value, name} object whenever
the -n guard runs CHECK POWER MODE, which is every rotational-disk probe.
The parser declared the field as a string, so json.Unmarshal failed for the
whole document and healthy spinning disks degraded to the lossy text
fallback, surfacing as no usable SMART data while guard-free SSD probes
kept working. That is the exact rotational-only failure split in the
discussion 1690 debug log (SAS3224 HBA, smartctl 7.5). Decode both shapes
the field has used, key standby on the reported name, and never fail the
document over this field. Also broaden the text fallback standby match to
the EPC names (STANDBY_Y, STANDBY (OS), SLEEP) that the mode-suffixed
match missed.
Refs #1690
Contract-Neutral: smartctl 7.5 power_mode JSON parse fix, internal decoder only, no DiskSMART payload or subsystem contract delta (discussion #1690)
Container update detection only ever negotiated anonymous pull tokens, so
containers from registries that reject anonymous digest HEADs pinned a
permanent "authentication required" badge (#1706). The agent already runs
on the Docker host, so the checker now resolves the same credential store
docker pull uses - config.json auths entries, credsStore/credHelpers
credential helpers (docker-credential-<name> get), and Podman's auth.json -
and presents the stored login: Basic auth on Bearer token negotiation and
on the hardcoded Docker Hub / ghcr.io token endpoints, direct answers to
Basic challenges, and the refresh-token grant for identity-token logins
such as Azure ACR.
Credentials never leave the host: they are only presented to the registry
or its token endpoint, helper output stays out of reported check errors,
and lookups are cached in memory for five minutes. Helper names are
validated before exec, and a stale login falls back to the anonymous path
so checks that used to work keep working. Set
PULSE_DISABLE_REGISTRY_CREDENTIALS=true (--disable-registry-credentials)
to keep detection anonymous-only. The agent-lifecycle and security-privacy
subsystem contracts pin the host-local credential boundary.
The host agent's node-local pct df inventory was only applied by the
per-node container poll fallback. Cluster instances are served by the
efficient cluster/resources path, which never called the enrichment, so
installs on that path showed rootfs-only container filesystems (and,
since the config-mount restoration, config-listed mounts with unknown
usage) no matter how healthy the linked agent was. Reported on #1477
after the reporter installed agents specifically to get per-mount usage.
Reproduced end to end against a live PVE cluster: the agent shipped the
full inventory, the server linked the agent and populated the
filesystem cache every report, and the poll path never read it. With
the enrichment applied after metadata enrichment, mirroring the
per-node path's ordering, the same rig surfaces real per-mount usage.
Refs #1477
Contract-Neutral: behavioral fix: cluster/resources poll path now applies agent pct df enrichment like the per-node path (#1477), no public contract delta
Every poll tick rebuilds the adaptive plan, and BuildPlan anchored the
next run on the previously planned NextRun by unconditionally adding the
selected interval. While a planned slot has not elapsed, each planning
pass therefore pushed it another full interval into the future. With the
default 10s tick that compounds NextRun ahead of wall-clock as soon as
the adaptive interval stretches beyond the tick cadence, so an instance
whose data was fresh at startup was never due again: API polling starved
permanently, the connection dropped to Agent and stale exactly when the
stretch began, and no error was ever logged. Matches the fourth #1437
reproduction (standalone degrades at ~2 minutes, cluster at ~4, stays
degraded), whose bundle shows a clean log with no polls after startup.
A pending future slot is now kept as planned, and it tightens to
now+interval when a staleness-driven interval shrink justifies an
earlier run. An elapsed slot still advances exactly one interval, so
ordinary cadence is unchanged.
Refs #1437
Contract-Neutral: behavioral fix: pending planned poll slot no longer compounds NextRun past wall-clock (#1437), no public contract delta
Configuring PROXY_AUTH_ROLE_HEADER without PROXY_AUTH_ADMIN_ROLE now gates
admin access on an exact-case match against the "admin" default. That closes
the fail-open, but it also means an IdP sending "Admins", "Admin", or
"authentik Admins" grants nobody admin — and a proxy-auth-only deployment has
no local credential to fall back on. The change is recoverable by setting
PROXY_AUTH_ADMIN_ROLE and restarting, but only if the operator can tell that
is what happened.
Log a startup warning naming the header, the effective admin role, the
case-sensitivity, and both ways out. The warning is scoped to deployments
that actually configured a role header; defaulting the value with no role
header is inert and stays at info. The troubleshooting entry in
docs/PROXY_AUTH.md now leads with the exact-match rule, since that is where
a locked-out operator looks first.
A reverse-proxy deployment that set PROXY_AUTH_ROLE_HEADER without also
setting PROXY_AUTH_ADMIN_ROLE granted every proxy-authenticated user full
administrator access. CheckProxyAuth only evaluated roles when both values
were non-empty, so the half-configuration skipped role gating entirely and
returned isAdmin=true. docs/PROXY_AUTH.md has always documented an `admin`
default for that variable, but the Config struct's envconfig `default` tags
are legacy and never applied (config.go), so nothing ever populated it.
CheckProxyAuth is the single admin verdict all 20+ proxy-auth gates consume,
so the fail-open reached every one of them. Verified on a scratch instance
with PROXY_AUTH_ROLE_HEADER set and no admin role: a request carrying only
`X-Proxy-Roles: user` received HTTP 200 and the full admin payload from
GET /api/system/settings, HTTP 200 from POST /api/system/settings/update,
and proxyAuthIsAdmin=true from /api/security/status. All three now return
403 / false, while `X-Proxy-Roles: admin` still passes.
Resolve the documented default in both layers that can produce the verdict:
config load populates ProxyAuthAdminRole when proxy auth is configured, and
CheckProxyAuth now keys role gating on the role header alone, resolving an
empty admin role through config.DefaultProxyAuthAdminRole. Configuring a
role header is the operator's signal that admin access is role-gated;
leaving the admin role unset must not switch that off.
Deployments that intentionally treat every proxied user as an admin are
unaffected: that is still expressed by leaving the role header unset.
Non-admin sessions cannot read GET /api/system/settings, so the Settings
General Monitoring Cadence card fell back to the Realtime (10s) preset
regardless of the configured interval; an issue #1601 reporter read that
as the server polling faster for non-admins. Publish the effective
pvePollingInterval on the authenticated runtime-display projection
(runtime config first, persisted value only as fallback, matching the
admin route's precedence), consume it in the viewer fallback of the
settings state, and run that initialization for sessions without
infrastructureRead too, whose ungated General panel previously never
initialized presentation state at all.
The per-guest Backup/Snapshot toggle persisted a full copy of the current
global defaults just to flip enabled, freezing the threshold values into
the override. Later global edits (a 32-day backup warning) then silently
never applied to toggled guests, which kept firing at the frozen factory
7-day warning. Reported twice in discussion #1126.
- Guest overrides now resolve against the globals at evaluation time:
zero-valued fields inherit the global value, explicit values still win,
and hand-written sparse overrides stop decoding as accidental zeros.
- The toggles write enabled-only overrides instead of freezing a copy.
- Normalization rewrites stored overrides whose thresholds exactly match
the current globals into sparse form, which is behavior-preserving at
migration time and un-freezes existing installs.
- The Backups/Snapshots global editors reconcile warning/critical pairs
by adjusting the untouched field, so typing a 32-day warning no longer
silently snaps back to the 14-day critical default.
Refs #1126
Integration test for the exact production path: stock PVE answers the
LXC status query with an empty diskinfo map, enrichContainerMetadata
discovers the mpX mount from the container config alone, and the unified
resource projection serializes it with capacity, the -1 unknown-usage
sentinel, and an omitted used field while the live rootfs row survives.
Related to #1477
Stock Proxmox reports no per-mount LXC usage through the status API, so
v6's API-polled containers listed only rootfs. The v5.1.32 fallback that
synthesized mount rows from the container config never crossed to the v6
line, and the v6.2.0 pct-df agent path only covers nodes running the
unified agent. Restore the fallback and improve it: parse size= so
config-only rows carry capacity, mark live usage unknown with the -1
sentinel, and merge without displacing the aggregate-seeded rootfs row.
Frontend consumers stop fabricating percents for sentinel rows: the
workloads row bar and summary math exclude them (tooltip lists them with
capacity), the drawer Filesystems block renders ?/<size> with no percent,
disk normalization preserves the sentinel, and per-machine max-disk
derivations skip them. Mock mode seeds one running container in this
exact shape so the surfaces stay exercised.
Related to #1477
2bc4ed725 moved the unified metrics sync sites to WriteBatchBounded but
left the three source guardrails pinning the literal WriteBatchSync
call, so they failed. The guardrails' intent is that pipeline writes go
through the canonical batched ingestion path, which WriteBatchBounded
is; pin the new name.
Contract-Neutral: test-only: guardrail pins follow the WriteBatchBounded pipeline rename from 2bc4ed725
TestSeedMockMetricsHistory_SeedsVMwareMetricsStore and the TrueNAS
variant compared seeded storage values against the package-level
mock.SampleMetric, which resolves resource roles from the global
registry that only other tests populate. The assertions therefore
passed or failed depending on which tests ran earlier in the process:
isolated runs failed deterministically, and today's CI reshard flipped
the rest-0 shard red for commits that never touched the mock layer.
Both tests now build the same graph-aware sampler the seeder uses, so
the expectation is self-contained and order-independent.
Contract-Neutral: test-only: seed assertions sample via the seeder's graph sampler, removes cross-test registry dependence
6b79aa997 bounded WriteBatchSync itself, which broke its read-your-writes
contract on slow disks: CI's metrics write-amplification and 500-node
load tests count committed rows after writing, and mock seeding reads
store coverage straight back, so the 2-second early return failed both
(runs 31475700902, 31494553977). Fast local disks masked it.
WriteBatchSync returns to a full commit wait. The monitoring pipeline's
four sync sites move to WriteBatchBounded, which carries the bounded
enqueue-plus-wait semantics, so the #1437 slow-disk stall fix stays
exactly where the hazard is. Both paths share prepareWriteBatch
validation, and a new regression test pins WriteBatchSync waiting past
the bounded budget.
Refs #1437
Contract-Neutral: behavioral fix: split bounded pipeline writes from synchronous batch writes, restores read-your-writes (#1437 follow-up), no public contract delta
Adaptive polling deliberately stretches an instance's cadence toward its
max interval (5 minutes by default) while data is fresh, but the
connections aggregator judged staleness against the configured cadence
with a 2-minute floor. Any adaptive-enabled install therefore cycled
healthy PVE/PBS/PMG connections into stale for the back half of every
stretched poll gap: the Infrastructure page dropped the source badge
from API + Agent to Agent and connection-degraded alerts fired against
a schedule the poller was honoring.
The aggregator now scales the active-to-stale cutoff by the scheduler's
currently planned interval when that exceeds the configured cadence,
via Monitor.PlannedPollInterval and per-instance planned intervals in
the aggregator inputs. A plan tighter than the configured cadence never
tightens the cutoff, so genuine poll outages still trip the floor on
time. Connection-degraded alerts and the runtime inventory gate consume
the same derived state and inherit the fix.
Refs #1437
Contract-Neutral: behavioral fix: stale cutoff follows adaptive planned interval (#1437), no public contract delta
Every attention row on every install (and all 38 rows on the public demo)
carried a warning 'Evidence incomplete; timing unavailable' / 'Evidence
timing unavailable' metadata line. Unknown freshness just means the evidence
source publishes no validity window - only availability-poller evidence does
- so it is not an operator-facing warning, and the 'timing unavailable' claim
was false with per-envelope observation times rendered beside it.
- Presentation: the scan row now surfaces an evidence phrase only for states
worth a glance (unavailable, incomplete, stale, or window-backed current);
complete evidence with unknown freshness stays quiet on the row and shows
a muted 'Evidence recorded' badge in the detail, mirroring the existing
unavailable-protection row rule.
- Alerts bridge: provider incidents without a backing availability check now
carry a first-class complete/confirmed envelope derived from the observed
incident payload instead of falling to the legacy partial shim, and each
sync merges the cycle's freshly observed envelopes into the active alert so
raise-time evidence does not age out while the condition is still observed.
Verified live on a scratch mock instance at 1280px and 744px: all 44+ mock
attention rows now project complete evidence and render without the noise
line; detail shows 'Evidence recorded' with per-observation timestamps.
The Plans & Billing navigation entry is suppressed for sessions without
commercial context, which hid the only activation-form entry point from
exactly the customers who need it: a fresh Pro install has no license yet,
so hideUpgrade stayed true until the key the customer could not find a
place to paste was pasted. The compiled Pro binary is only distributed
through the paid broker flow, so a Pro-edition session is commercial
context by construction — edition.IsPro() now sets commercialContext in
securityStatusPresentationPolicy.
Ordinary free self-hosted sessions run the community binary and keep the
opt-in default (the edition marker keys off the compiled binary, never
license state). Demo mode and white-label keep full suppression via
hideCommercial, which wins over edition-derived context. Complements
1b9bc9482 (setup-completion activation pointer); together a fresh Pro
install gets both the first-run pointer and a permanently visible
Plans & Billing entry. api-contracts, storage-recovery, and
agent-lifecycle contracts record the policy delta.
Clears the red main streak that began at 63a0adf9a:
- newTestConfigHandlers never stopped its alerts.Manager, leaking a
periodicSaveAlerts goroutine that logs through the global zerolog
logger and races with the authorization-refusal contract test's
logger swap under -race
- InfrastructureAgentDoctorPage used a raw select, tripping the
settings native-select guardrail; converted to the shared labelled
FormSelect primitive
- the metric-title residual pin still expected the removed -1 sentinel
copy; aligned with the Off-control wording from 16179dd0a
Contract-Neutral: test-only CI red fix: stops alerts.Manager goroutine leak in api test helper, converts agent doctor raw select to the canonical FormSelect primitive the frontend-primitives contract already mandates, updates a stale test pin; no public contract delta
rcourtman