A LoadSystemSettings error in configureMonitorDependencies silently
skipped the tenant-monitor inherit block, leaving the monitor on
deny-all-private webhook CIDRs with nothing in the logs to explain why
private webhook targets fail SSRF validation. reloadSystemSettings
fails closed on the same error (embedding off) deliberately, but was
equally silent, so a persistent read failure looked identical to
embedding being switched off on purpose.
Both paths now log a warning with the error, mirroring the pattern from
cf5b86e62 in internal/config. A missing system.json (fresh install)
still stays quiet. Tests cover the warning on read failure, the quiet
fresh-install path, and that the reload still fails closed.
A boot-time LoadSystemSettings error fell into the missing-file branch:
no log line, every persisted system setting reverted to defaults for
the run, and a default system.json was written over the file on disk.
Observed once in practice (2026-08-20): a restart came up with a
persisted toggle unset while system.json still held the correct value.
Distinguish the error from the genuinely-missing file: retry the read
once (transient failures at boot were the observed shape), then warn
with the error and file path and run on defaults without touching
system.json. Tests cover the warning, the untouched file, and the
retry-once behavior.
Discovering Docker inside Proxmox LXC guests previously required an
undocumented environment variable plus a hand-edited systemd unit,
which is how the surface kept shipping invisible. The opt-in is now a
persisted system setting with an admin-only toggle in Settings →
System → General, presented as its own "Docker in Proxmox LXCs" card
under the existing Docker/Podman updates card, with the requirements
(node agent with command execution) and data-collection bounds stated
inline.
The setting rides the admin-gated system settings endpoint
(RequireAdmin + settings:write, matching every other server-wide
toggle). Flipping it persists first, then applies to the runtime
config and fires a reconfigure hook so the router rebuilds the
monitor's Docker checker and inventory collector immediately — no
restart. PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY remains
authoritative: when set it locks the toggle (EnvironmentLockBadge in
the UI, HTTP 400 from the API), and an unrelated settings save can
never clobber the runtime value. The opt-in guardrail test now pins
the settings path, the env lock, and the persisted field.
Installer and node setup-guide callouts now point at the toggle first
with the environment variable as the locking override, and the docs,
their public mirrors, and the agent-lifecycle subsystem contract say
the same. New i18n strings shipped for en/de/es.
Verified live in mock mode: toggle off/on from the UI drives the
collector teardown/setup log lines, the value survives a server
restart, and all three touched surfaces render at desktop and mobile
widths (receipt in frontend-modern/browser-verification.json).
Mock mode never exercised the backup-running states from "Stop showing
an in-flight backup as a completed one": no guest carried
BackupInProgress, no PBS listing showed an in-flight snapshot, and no
vzdump task was ever running, so the blue Running badge and the Running
artifact chips had no local reproduction.
A post-scenario fixture now flags roughly one guest in twenty-five per
last-backup age bucket - never, fresh, stale, overdue - so the Running
badge appears alongside every badge story, and gives each flagged guest
the evidence a live run actually produces: an in-flight PBS snapshot
shaped the way the PBS API lists one still being written (no size, no
verification, only the guest config blob in files) and a running vzdump
task with no end time. Running after applyDemoBackupScenario keeps the
per-tick reapplication idempotent, because the scenario's curated sweep
drops the previous pass's artifacts before the fixture re-adds them.
The first-cycle showcase guests are excluded: their exact rows are
pinned by the demo posture tests (a failed run three hours ago, a guest
with no recovery evidence at all), and a backup running right now would
mask exactly the states those rows exist to demonstrate. Later profile
cycles repeat the same stories under suffixed names, so the fixture
draws its running guests from them.
An in-progress PBS snapshot (reproduced on VM 117 at 9% written) was
counted as a completed backup: the guest's LastBackup jumped to "now" the
moment vzdump started, and the workloads badge showed a green healthy
shield for a backup that did not exist yet. If the job aborted, that
shield had never been true.
Detect the in-flight state from what the PBS listing actually says - no
size and no index.json.blob manifest in files - and flag the snapshot
InProgress instead of complete. On the PVE side, correlate storage content
entries with live vzdump tasks: a guest archive whose ctime falls inside a
running task's window is that task's partial output, and a PBS-storage
entry without a size has no manifest yet.
In-flight artifacts now feed a new per-guest BackupInProgress flag rather
than LastBackup, so the badge age always reflects the latest COMPLETED
backup. The workloads badge gets a distinct blue "Running" state (green
would claim success, None would hide that work is underway), with the
last-completed age kept in the tooltip and accessible label. Recovery
points map to OutcomeRunning with no completion time, keeping posture
freshness and the Backups page's latest-restore pointers anchored to
completed backups; the recoverable-artifact list shows a Running chip.
A cached in-flight snapshot forces a per-poll re-fetch of its group so
completion or abort is noticed promptly instead of after the cache TTL.
The first cut of the fixture ran inside buildFixtureState, but every
fixture graph build and metric tick then applies the demo scenario
layer, which renames guests, aliases cluster instances, forces guest
states, and reprofiles every Docker host. The nested hosts got swept
into the Docker fleet profile cycle: renamed away from their parent
guests, handed native engine inventory the production pct exec path
never reports, and one could be forced offline as the demo's
disconnected host.
The fixture now runs from applyDemoScenarioGraph after the workload
and Docker scenarios, so nested hosts bind to the guests' final names,
instances, and states, and it is idempotent across ticks: an existing
nested host re-syncs from its guest instead of duplicating, one whose
guest stops running goes offline like a real probe target, and the
sync strips every native-agent surface each pass. The Docker scenario
now skips proxmox-lxc-docker: hosts entirely, with the forced-offline
index counted over native hosts only.
The Proxmox page's nested-container row cue, the drawer's nested Docker
card, and the proxmox-pve platform scoping of nested workloads had no
local reproduction at all: mock mode never generated a Docker host with
the proxmox-lxc-docker: identity prefix, so the surface could only be
seen against a live estate with node agents, exec-scoped tokens, and
the server-side inventory opt-in. That is how it shipped invisible.
The fixture now marks the first two running LXC guests as probed
Docker hosts and appends a nested Docker host for each, following the
production proxmoxGuestDockerAgentID convention of
proxmox-lxc-docker:<instance>:<node>:<vmid>. One guest nests a single
container and one nests several, so the row cue renders both its
singular and plural counts. The nested hosts mirror the shape of the
real pct exec inventory: guest-derived sizing and usage, one rootfs
disk, and no native engine inventory, host I/O rates, temperature,
machine ID, or Swarm membership.
The pct exec socket probe that discovers Docker inside LXC guests
failed invisibly: errors logged at debug only, and a failed probe never
set DockerCheckedAt, so the guest stayed in first-check state and was
re-probed on every poll forever. On a node where lxc-attach could not
enter unprivileged guests that meant ~49 pct execs per minute of
steady-state churn, with nothing at default log level to say why the
Proxmox page showed no Docker.
Track consecutive probe failures per guest in the monitor. The first
failure of a streak, and the point where the backoff reaches its cap,
log at warn with the guest, node, and error; repeats stay at debug.
Failed probes retry with exponential backoff from one minute to a
thirty-minute ceiling, a success clears the streak with an info log,
and reconfiguring the checker resets all streaks so command execution
enabled from the UI at runtime still retries immediately. Entries not
refreshed for a day belong to deleted guests and are pruned.
PULSE_PROXMOX_GUEST_DOCKER_INVENTORY_VMIDS now gates the socket probe
as well as inventory collection. It previously only limited inventory,
so opting into specific guests still probed every running LXC on every
cycle; now guests outside the allowlist are never pct exec'd at all.
Subsystem contract and configuration docs updated to match.
lxc-attach into an unprivileged guest writes /proc/<pid>/uid_map, which
needs CAP_SETUID in the parent user namespace. NoNewPrivileges drops
CAP_SETUID from the effective set and also stops lxc-attach falling back
to the setuid newuidmap/newgidmap helpers, so the socket probe dies with
"write_id_mapping: 61 Operation not permitted".
install.sh already relaxed NoNewPrivileges for this, but only when the
agent was installed with --enable-commands. Command execution is also
togglable from the server afterwards: applyRemoteConfig starts the
command client without rewriting the unit. An agent installed without
the flag and switched on later therefore ends up able to run commands
and unable to attach to unprivileged guests, so Docker inside every
unprivileged LXC disappears from the Proxmox page. The probe failure is
logged at debug level and retried on every poll, so the surface looks
empty rather than broken while the agent re-probes the whole guest list.
Grant CAP_SETUID/CAP_SETGID to any PVE agent rather than gating on the
install-time flag, so the later toggle lands on a unit that can attach.
Ambient capabilities restore exactly the privilege lxc-attach needs and
leave the rest of the sandbox intact; the existing install-time
relaxation is unchanged.
Verified on a live PVE node. With the hardened unit the probe succeeded
only on the three privileged guests and failed on every unprivileged
one. After the ambient grant CapEff regained CAP_SETUID and both
unprivileged Docker guests were discovered, taking that node from one
Docker LXC to three.
The budget check had not run since 08-18 because the vitest step was
failing first. With the tests fixed it now flags a week of intentional
feature growth: the estate-overview and mobile table work grew every
major chunk (index, sharedPlatformPage, Alerts, Proxmox, AIIntelligence)
by 3-10 kB gzip, and removing the saved-views lazy boundary folded the
FormTextarea chunk into its importers. Rebaseline from a lockfile-clean
build; totals stay within the overall budget.
An agent whose token authenticated seconds ago cannot be in a
credential outage, yet the fleet doctor flagged it critical with
"missing or revoked" whenever the token id missed the server's token
registry view, and the offered repair-authentication loop cannot fix a
stale server-side registry view. When the subject's own row shows the
judged token authenticating within the freshness window, emit a
warning-severity registry-stale verdict naming the id and the last
authentication instead. Genuinely stale rows keep the critical verdict.
Refs #1730
The estate-overview redesign folded attention banners and guest totals
into the canonical tables and the shared View menu, updating its own
co-located tests but leaving three pins in other files on the old
surfaces. Repin them: the Docker single-host View menu now legitimately
carries the inventory-totals toggle but still no layout choice, node
availability reads as the warning-toned ready/total ratio on the
Kubernetes cluster row, and Proxmox guest totals flow through the
toolbar inventory counts instead of a page-local memo.
Saved views stored a named URL query string in localStorage under
`pulse:filterbar:saved-views:<key>` and offered it back from a "Saved"
control on eight surfaces. A saved view was only ever the page's URL:
`saveCurrent` snapshotted `window.location.search` and `applyView`
navigated back to it.
The browser already does that, better. A bookmark syncs across devices,
survives clearing site data, is searchable, can be foldered, and can be
shared with someone else. The localStorage copy did none of those, so a
"saved" view silently vanished on a new browser or a cleared cache.
There is also no evidence anyone used it. It shipped 2026-05-01 in
v6.0.5 and no Pulse issue, comment, or discussion has mentioned it since.
#1510 is the telling case: filed two months after it shipped, by a user
who wanted their Proxmox status filter to survive a tab switch, which is
exactly the job this feature claimed. They did not reach for it, and the
fix that satisfied them was sticky route state (d20c556db).
Removed the hook, the menu, the `savedViewsKey` prop, and the per-surface
keys. URL-backed filters and sticky route state are untouched, so a
filtered page is still a shareable, bookmarkable link. On mobile, a
surface with no other auxiliary control no longer renders an action row
that existed only to hold the Saved button.
Subsystem contracts keep the URL-ownership rule this feature depended on
and now state it as the primary contract, with a note not to reintroduce
an in-app view library. A guardrail asserts the FilterBar barrel does not
re-export SavedViews.
User-visible removal: needs a release-note line at the next cut.
The helper added in 54f623cff omitted onPrimaryClick and onUtilityClick, so
tsc rejected the MobileNavBar element while vitest, which does not type-check,
still passed. Left main failing type-check.
The Assistant overlay and its backdrop reserved a hardcoded
calc(5rem + env(safe-area-inset-bottom)) for a bar that measures 45px. At
375x812 the backdrop ended at y=732 while the bar starts at 767, leaving a
35px band that was neither dimmed nor click-blocked: page content showed
through and stayed interactive outside an open modal.
The bar's height is content-driven and already includes the safe-area inset
via pb-safe, so no caller can hardcode it correctly. Five sites had tried:
the Assistant panel, its backdrop, the star banner, and both halves of the
.filter-bottom-nav-aware-panel rule, which also carried a derived 6rem.
MobileNavBar now publishes its measured height as --pulse-mobile-nav-height
and those five sites read it. Publishing happens in onMount rather than the
ref, because Solid runs refs before the node is in the document where
offsetHeight is 0; reading after mount forces layout and gets the real value
on first paint. A ResizeObserver covers content-driven changes and a resize
listener covers the xl breakpoint, where the bar swaps between its height and
0 by display alone.
The declared :root value is only a pre-measurement fallback and deliberately
under-estimates. Over-reserving is what produces the un-dimmed band; reserving
slightly less is harmless because the opaque bar paints over it.
Verified at 375x812 and 1280x800: backdrop and panel now both end at 767
against a bar top of 767, gap 0; the point 5px above the bar hit-tests to the
Assistant panel rather than page content; the filter panel resolves to 45px on
mobile; and the star banner's md:bottom-4 still wins at 16px.
Adds the regression proofs the shape guard asks for, including an
architecture test that fails if any runtime source reintroduces a literal
bar height, and records the ownership boundary in the frontend-primitives
and ai-runtime contracts.
Running any release-control script leaves __pycache__ directories in the
tree. scripts/release_control/* already ignores the top-level one as a
side effect of its blanket wildcard, but scripts/release_control/internal/
is deliberately re-included further down so its 31 tracked scripts stay
visible, and that re-inclusion left internal/__pycache__ uncovered. It
showed as untracked in every agent's git status and was one stray
git add -A away from committing bytecode.
Cover it repo-wide rather than patching the one path, so future Python
directories stay clean too. Verified that the rule shadows no tracked
file, the set of tracked-but-ignored paths is unchanged at 27 either
side of the change.
ec8b88fe2 registered the alert-form docker identities as Patrol scope
aliases for hosts and containers, but Swarm service alerts publish a
third form, docker:<host>/service/<serviceID>, which matches neither
the host alias docker:<host> nor the container alias
docker:<host>/<containerID>. Have Patrol investigate on a docker
service alert still returned patrol_scope_unresolved, and the
automatic alert-fired patrol path sent the same identity, so scoped
runs it triggered for service alerts failed resolution the same way.
Mirror the service resource IDs onto the owning docker host record.
Patrol has no service-level analysis, and a service's tasks run as
containers on the host, so the host is the smallest unit Patrol can
investigate for a service alert. Export the alerts helper for the same
reason ec8b88fe2 exported DockerResourceID, so the identity format
cannot drift between the two subsystems.
Hosts without an ID are skipped. The host-less fallback form
docker-service:<name> is not host qualified and would alias unrelated
records together across hosts, which is the same anti-aliasing rule
dockerAlertScopeAlias already applies.
Covered by tests asserting the alias equals what the alerts subsystem
emits for the same input, that the host-less guard registers nothing,
and that a service without an ID follows the canonical name-derived
form.
Refs discussion #1699
Contract-Neutral: behavioral bug fix refs discussion 1699, patrol scope resolution could not resolve docker service alert resource IDs, no public contract delta
TestRecoveryPointsEndpointReturnsMockData sampled the unfiltered first page
of /api/recovery/points and expected to find a kubernetes point in it. That
held only while the mock estate was small.
Points come back newest-completed-first and a page is hard-capped at 500. The
rescaled Proxmox demo estate now generates 2498 points, 939 of them newer than
the newest kubernetes point, so kubernetes starts at index 939 and never
reaches page one. The 36 kubernetes points are still generated and still
served, and the endpoint, the store ordering and the fixture shape are all
correct: a platform that snapshots every few days legitimately falls behind a
platform that backs up hourly. TrueNAS was already at index 254 and would have
broken next.
Ask the endpoint per platform instead, which is how the API is designed to
answer the question and how the UI asks it. The assertion no longer tracks
estate size, and it additionally covers the platform filter.
Contract-Neutral: test-only change, no behavior or contract delta.
Two counters could not answer the question they exist for.
pulse_intelligence_patrol_runs_30d was counted from the operator-facing
patrol run history, which is capped at MaxPatrolRunHistory (100). On any
install patrolling on a normal schedule that cap is reached within hours
of a thirty-day window opening, so the field reported the cap rather than
the run count, and the calls-per-run ratio derived from it was inflated by
a censored denominator. Patrol run history now carries an uncapped daily
tally beside the capped list, advanced from a persisted high-water mark so
repeated full-list saves cannot double count, pruned to 31 days, and read
in preference to the list. An install with no tally yet falls back to the
list, so the counter never regresses on upgrade.
Schema v8 split agent-side pre-mutation refusals into target-change,
prerequisite and contract categories so they would stop hiding in "other".
In production all three are zero fleet-wide and "other" still absorbs every
refusal, because the dominant reason code is the legacy preflight_refused
aggregate recorded when an agent sends no machine reason code at all.
Agents older than the typed refusal contract report every refusal that way,
so the split is starved rather than broken, and folding the two together
made those two states indistinguishable. Schema v9 counts uncoded refusals
separately.
The Proxmox node sweep removes any non-preserved alert whose Node is
empty, and system alerts have no node, so every sweep silently deleted
the notification-delivery alert. Its five-minute evaluation then
re-raised it as a brand-new alert, firing a fresh notification each
cycle with no recovery in between, which reads as an alert appearing,
vanishing without a recovery, and paging again minutes later. System
alerts are now preserved outside node cleanup.
Raises also carried the delivery counts inside the message, and a
message change re-notifies, so each newly retained failure re-paged
even while the condition was unchanged. System alerts now take an
optional fingerprint: while level and fingerprint hold, a re-raise
refreshes the message and metadata silently. The delivery alert
fingerprints on status and failure classes, keeping its counter text
current without paging on drift.
Refs #1721
Contract-Neutral: behavioral fix: delivery-health system alert survived node sweep and stops re-paging on count drift, no public contract delta (Refs #1721)
TestCollectDeviceSMARTStandby and its observability twin stub smartctl's
execution but not the device-class probe, so collectDeviceSMART fell through
to the *runner's* real /sys/block/sda/queue/rotational.
smartctlArgs only sends the -n standby guard when the disk is not a confirmed
SSD (#1516), and smartctlArgsUseStandbyExitStatus gates the standby reading of
exit status 3 on that guard having been sent. On a Linux host whose own
/dev/sda is non-rotational the guard is dropped, exit 3 stops meaning standby,
and both tests fail deterministically:
run smartctl for /dev/sda: exit status 3
They pass on macOS only because linuxNonRotationalBlockDevice returns false off
Linux, and on Linux only where /dev/sda is absent or spinning. That makes the
release-qualification suite unrunnable on an ordinary SSD-backed Linux worker.
Pin the probed disk to rotational through the package's existing stubLinuxSysfs
seam so the guard is always in play. Product behaviour is untouched; this only
stops two unit tests depending on the hardware underneath them.
TestSubscriptionAgentClientsUseStructuredSingleTurnProcess failed three times
during the v6.3.0-rc.3 release qualification, always at exactly its 5s
deadline, with "codex subscription agent timed out". It was green 5/5 in
isolation: 3/3 as a single test under -race and 2/2 for the whole package.
It only failed inside the full `make test` run, where -race instrumentation
and roughly 150 packages compete at once.
The test is hermetic. It writes a fake codex shell script onto PATH, so there
is no CLI, network, or credential involved and nothing about the deadline is
asserting product latency. It exists to stop a hang. Under race instrumentation
on a loaded machine, spawning that script legitimately takes longer than the
budget, so the deadline was reporting how busy the host was rather than whether
the code works.
Scale the deadlines when the race detector is enabled, using the raceEnabled
build-tag constant that internal/api, internal/monitoring, and pkg/metrics
already use. Scaling rather than skipping is deliberate: the surrounding
package convention is to skip latency SLO tests under -race, but this one is a
functional proof of the structured single-turn process, tool calls, and
credential non-leakage. Skipping it would remove exactly the path we most want
the race detector to see.
The assertion in TestSubscriptionAgentRequestTimeout is untouched, since that
one is testing the timeout policy itself rather than waiting on anything.
Proof: package passes 2/2 under -race and once without, and both build variants
vet clean.
Contract-Neutral: Test-only change scaling hermetic subscription-agent deadlines under -race via the existing raceEnabled build-tag pattern. No runtime, contract or payload delta. Proof is the package itself, which passes 2/2 under -race and once without.
The desktop thresholds table carries an Offline Alerts column with the
global default control and per-row toggles, but the card layout used at
narrow container widths (including zoomed desktop browsers) omitted it
entirely, so connectivity alert state was invisible and untogglable
there. Mirror both controls into the cards: an Offline alerts tile on
the Global Defaults card and an Offline row on each resource card, with
the same tri-state cycle for guests and docker containers and the same
On/Off badge for nodes, agents, PBS and docker hosts.
Refs #1738
Contract-Neutral: frontend-only: card threshold layout renders the existing offline alert controls, no API or contract delta (Refs #1738)
Contract-Neutral: Release-note copy correction for the v6.3.0-rc.3 packet. Restores the 'Read-only observers' Highlights bullet that build_release_assets_test.go pins for the 6.3 packet and that an earlier trim dropped, and lists the alert-delivery repairs under Fixed where the changelog already carried them. Both governed gates verified: render_release_body validates, and scripts/installtests passes. No contract, runtime or version-pin change.
Contract-Neutral: Release-note copy fix for the v6.3.0-rc.3 packet already prepared in 2f6f2396b. Trims Highlights to the three bullets the canonical render_release_body validator permits and keeps each under the 140-character cap. No contract, runtime or version-pin change; deployment-installability.md and the version pins are unchanged from the prep commit.
Contract-Neutral: Routine release-preparation cut for v6.3.0-rc.3. deployment-installability.md IS staged with the substantive active-prerelease and mobile-decision updates for this cut. The residual demand is a verification artifact, and the registry-listed proof scripts/installtests/build_release_assets_test.go plus install_docker_sh_test.go already cover this change and pass against it; they validate the packet dynamically from the repo-root VERSION rather than pinning a literal, so they have no diff to stage. Both were run green immediately before this commit and caught two real errors in the mobile-decision clauses, which are fixed here.
Contract-Neutral: Completes the already-governed mobile table readability contract on remaining Docker overview and standalone availability rows without changing the contract.
The alert card assumes an alert has a resource behind it. It reads
resourceName unguarded and guards node and resourceId with Show, so a
system alert renders only because those guards happen to be there. Nothing
stopped a later change from adding an unguarded resource field and breaking
the notification-delivery alert, which is the one alert whose whole purpose
is to be visible when nothing else can reach the operator.
Drive OverviewTab with the exact payload internal/alerts marshals for a
raised system alert, captured from the Go side rather than hand-written:
empty resourceId, empty node, resourceName "Pulse". Assert the card names
Pulse rather than inventing a resource, title-cases the hyphenated type,
carries the message and its pointer to the destinations surface, and renders
no empty node line or undefined text.
Pulse could tell that its own delivery had stopped and had no way to say so.
Queue health was computed for one API endpoint and rendered on the alerts
surfaces, and that was the whole escalation path: a destination that stops
delivering cannot announce itself through a notification, so an operator who
does not open the alerts page learns nothing.
Telemetry: 78 installs retry every notification to the maxAttempts ceiling and
deliver almost nothing, 155,967 attempts against 221 deliveries in a week. 50
have delivered nothing in 30 days and 28 more delivered normally at some point
and silently stopped. The dominant terminal failure classes are authentication
and configuration, so most are fixable in minutes once someone knows.
Move the queue-health rule into internal/notifications as ClassifyQueueHealth
and DeliveryHealth, next to the queue that produces the counts, and delegate
the API handler to it so the endpoint and the new evaluator cannot drift into
two rules. A queue that cannot be read reports unavailable rather than healthy,
because silence is the failure mode being guarded against.
Monitoring evaluates delivery health on the poll ticker and raises or clears
the notification-delivery system alert, which puts it in the alert list and the
navigation badge. Reading queue health costs a SQLite query and the poll
cadence can be seconds, so the evaluation is throttled to five minutes.
RaiseSystemAlert is idempotent for an unchanged condition, so the timer neither
re-notifies nor accumulates alerts.
The alert is a warning rather than critical. It is real, but a new
critical-by-default alert appearing on upgrade across the affected installs is
a bigger change than this should make on its own.
The new Monitor field is deliberately named to fit inside the existing struct
alignment column. A longer name makes gofmt re-pad the whole block, which
breaks the canonical guardrail tests that pin those field declarations
verbatim.
Proof: delivery_health_test.go pins that in-flight work and retries stay
healthy while retained terminal failures do not, and that an unreadable queue
is never reported healthy. system_alerts_test.go pins the message for each
outcome including singular and plural, that it points at the destinations
surface, and that the throttle honours its interval. internal/monitoring,
internal/notifications and internal/alerts suites all pass.
Contract-Neutral: Staged contract deltas cover every boundary this change actually moves: notifications.md gains delivery_health.go plus the ClassifyQueueHealth rule, and monitoring.md gains system_alerts.go plus the system-alert evaluation extension point. The residual demands are inapplicable. agent-lifecycle is demanded only because monitor.go is one of its canonical files, and its verification artifact is host-agent deletion and re-enrollment proof, which this change does not touch: the diff to monitor.go is one struct field and one call in the poll loop. api-contracts and storage-recovery are demanded only because internal/api/notifications.go falls under a broad internal/api reference, and that file's change is a pure delegation of an internal helper to notifications.ClassifyQueueHealth with a byte-identical HTTP response and no route or payload delta.
Every alert type is scoped to a monitored resource: a node, a disk, a
container, a host. That leaves no way to report a condition whose subject is
Pulse, and the condition that needs it most is broken notification delivery,
where the channel that would carry the warning is the thing that failed.
Telemetry shows what that costs. 78 installs retry every notification to the
maxAttempts ceiling and deliver almost nothing, 155,967 attempts against 221
deliveries in a week; 50 have delivered nothing in 30 days and 28 more
delivered normally at some point and silently stopped. The dominant terminal
failure classes are authentication and configuration, so most are fixable in
minutes by an operator who has no way to learn there is anything to fix. Until
now the only evidence lived on a config tab, because a delivery failure cannot
announce itself through delivery.
Add RaiseSystemAlert and ClearSystemAlert. A system alert carries the stable
pulse-system- identity prefix so repeated raises update one alert instead of
accumulating, sets no ResourceID so surfaces skip resource-linked affordances
rather than offering a link to nothing, and stamps systemAlert metadata for
surfaces that want to tell the two apart. Raising is idempotent for an
unchanged condition and only re-notifies on a change of level or message, so an
evaluator on a timer cannot turn into the notification storm this is meant to
warn about. Routing through the ordinary pipeline is the point: the alert
reaches the alert list and the navigation badge, which is the escalation path
that does not depend on delivery working.
This commit adds the capability and its contract entry. Nothing raises a
notification-delivery alert yet: the evaluator needs a package holding both the
notification queue and the alert manager, and internal/monitoring plus
pkg/server are held by another agent's path claim. The evaluator and its UI
verification follow.
Proof: four cases in alerts_test.go pin the stable identity and absent
ResourceID, the idempotence that keeps a repeated raise from re-notifying, the
level change that does re-notify while still updating a single alert, the clear
path including clearing something absent, and the rejection of a blank type.
Full internal/alerts suite passes under -race.
checkFlappingLocked recorded a cooldown deadline in suppressedUntil and then
never read it. Suppression was decided purely by whether the trailing
FlappingWindowSeconds still held FlappingThreshold state changes, so it lifted
the moment the sliding window drained instead of holding for
FlappingCooldownMinutes. Only the metric paths read that map, and they read it
for the separate SuppressionWindow feature.
Two consequences. A resource oscillating just under the threshold rate, four
changes per five minutes against a threshold of five, was never damped at all
and dispatched every single state change. And flappingActive was only ever set
to true, never cleared except by age-based cleanup, so even after the first
episode no later episode could arm a fresh cooldown.
Pulse states the guarantee it was not providing. The flapping postmortem
finding tells the operator that notifications were "suppressed for" the
configured cooldown and recommends raising it, which changed nothing.
Honour an active cooldown before the window check, and release the latch once
the cooldown has been served so a later episode can arm a new one. Releasing
also clears the stale window history, so a resource that has gone quiet starts
from a clean slate rather than re-tripping on expired state changes.
This damps notification volume. It does not reduce alert churn: flapping is
evaluated in dispatchAlert, so the alert has already been raised and recorded
in history by then. Telemetry shows 923 installs firing 3.64M alerts in 30 days
with only 20,910 standing, and that churn needs per-alert-type root causes such
as #1721.
Proof: three cases in alerts_test.go drive checkFlappingLocked directly and pin
that the cooldown keeps suppressing after the window drains, that it releases
and can re-arm for a later episode, and that disabled flapping detection
ignores a stored cooldown. All three fail against the previous implementation.
Contract-Neutral: behavioral fix: the flapping cooldown now gates the dispatch path that already recorded it. No alerts subsystem contract delta (no Canonical Files, Shared Boundaries or Extension Points change, no config/API/payload change) so alerts.md has nothing substantive to record. The registry-listed verification artifact demand IS satisfied: proof lands in internal/alerts/alerts_test.go.
Notification delivery health was rendered by exactly one component on the
destinations config tab. A destination that stops delivering is invisible by
nature, because the failure is the channel that would have reported it, so
confining the only evidence to a tab people open once during setup means the
state persists indefinitely.
Production telemetry shows it persisting. 78 installs retry every notification
to the maxAttempts ceiling and deliver almost nothing: 155,967 attempts and 221
deliveries across a week. 50 of them have not delivered a single notification
in 30 days, and another 28 delivered normally at some point and have since
stopped, which is the worse case because nothing about the estate looks
different. The dominant terminal failure classes are authentication and
configuration, so most of these are fixable in minutes by the operator who
cannot currently tell there is anything to fix.
Extract the delivery-health load into useNotificationDeliveryHealth and render
the existing warning on the alerts overview as well as the destinations tab.
The shared hook also holds the trigger rule, so both surfaces agree on what
counts as needing attention and neither can warn before the first load
resolves.
This does not make Pulse monitor its own delivery. Every alert type is
resource-scoped and there is no system-scoped alert concept, so raising this
like any other monitored condition remains open.
Proof: useNotificationDeliveryHealth.test.tsx pins the trigger rule across
healthy, degraded, server-reported-unavailable and unreachable-endpoint cases,
plus the silent-until-loaded guard. Verified live at 1280x800 and 375x812: a
healthy queue leaves the overview clean, and a degraded payload renders the
warning above the stat cards with the dominant failure class named.
Contract-Neutral: frontend-only reuse of the existing /api/notifications/health payload on a second surface; no subsystem Canonical Files/Shared Boundaries/Extension Points delta and no API change. Proof ships as useNotificationDeliveryHealth.test.tsx; the alerts verification-artifact list enumerates existing filenames, so a new hook's own test cannot appear on it.
Alert notification delivery is gated on the alert config's activation
state: monitor.go calls SetEnabled(enabled && ActivationState ==
ActivationActive), and a new install defaults to ActivationPending. In
that state alerts keep firing and every sendAlert returns at the
disabled check with only a debug log, so nothing reaches the queue.
The destinations surface never said so. A user could configure SMTP or a
webhook, enable it, and send a test that passes, because SendTestNotification
checks only the destination's own Enabled flag and bypasses the delivery
gate entirely. The overview tab showed "Notifications paused" but the
Notifications tab, where that belief is actually formed, showed nothing.
Production telemetry shows the resulting cohort: 510 installs with an
enabled destination, alerts standing right now, and zero delivery attempts
in 7 days; 483 of them have made no attempt in 30 days. On 6.2.1 that is
357 of 1219 notification-enabled installs.
Surface the pause on the destinations tab with the consequence spelled out,
including that a passing test proves nothing while delivery is off, and
offer the activation action inline. The card stays quiet until the alert
config resolves so it cannot flash a false warning on tab open.
Proof: AlertDeliveryPausedCard.test.tsx pins the consequence copy, the
test-send caveat, the activate callback and the busy state;
alertDestinationsPresentation.test.ts pins the copy for all three paused
reasons. Verified live against a backend in pending_review at 1280x800 and
375x812: card renders above the email section, activating clears it, and
deactivating restores it.
Contract-Neutral: frontend-only disclosure of existing alert delivery activation state; no subsystem Canonical Files/Shared Boundaries/Extension Points delta, no API or payload change. Proof ships as AlertDeliveryPausedCard.test.tsx plus alertDestinationsPresentation.test.ts; the alerts verification-artifact list enumerates existing filenames, so a new component's own test cannot appear on it.
Adding a PVE/PBS source through the Connect via API path discarded the
name typed in the add dialog: the setup bootstrap only carried type and
host, and auto-registration named the new connection after the node's
self-reported hostname. A user asking for a cluster named enacon got a
connection called pve01 and could only rename it after adoption.
Carry the typed name on the one-time setup token
(SetupTokenRecord.DesiredName) via an optional name field on
/api/setup-script-url, and let the setup-token-authenticated
auto-register completion name a newly created connection from it,
falling back to the hostname-derived default when absent. Dedup and
cluster-member adoption identity stay hostname/candidate based, so the
carried name cannot fork an existing registration.
rcourtman