Presentation host coalescing buckets agent rows by short hostname and
PVE node names are short, so a pve01 in one estate merged with a pve01
in another, re-parenting both estates workloads onto one row and
cross-wiring reachability. Veto the merge when the rows carry proof of
distinct machines, a disjoint machine ID set, differing DMI UUIDs, or
differing Proxmox cluster names, while equal cluster names still merge
so the same cluster added under two connections keeps its single row.
Apply the same cluster discrimination to agent-node link inference,
whose short endpoint alias keys and short-hostname corroboration
fallback could hand one estate agent to the other estate node.
Refs #1753
Contract-Neutral: behavioral bugfix in unified resources presentation coalescing and node link inference; no API shape or subsystem contract change
applyThresholdOverride only ever copied DisableConnectivity=true from an
override, so once the global default disabled connectivity alerts no
override could turn them back on. A guest row set to Warning or Critical
saved its severity, showed as Custom, and never fired, while the
frontend already encodes re-enablement by pairing an explicit severity
with the row offline control (off stores DisableConnectivity=true).
Treat an override that carries an explicit powered-off severity as that
row offline control set to warning or critical and clear the inherited
disable, matching the Docker container state path which already works
this way.
Refs #1738
Contract-Neutral: Refs #1738: behavioral bugfix in alerts override merge, no API or contract shape change
The dockeragent credential-store tests keyed their fake filesystem and
path-bearing env seams on POSIX literals, while dockerConfigCredentials
builds candidate config paths with filepath.Join. On Windows every lookup
therefore missed the fixture and the nine credential-expecting
TestDockerConfigCredentials_* cases failed with a nil credential; the
hostagent PULSE_PCT_PATH override test likewise asserted a POSIX path that
is not absolute on Windows. Map fixture keys and env values through
filepath.FromSlash and anchor the override on t.TempDir().
Taken from PR #1755 (ff96aed8c) so the correction lands on main instead of
a parked agent branch. The unified-agent-native Windows x64 job has been
red on every push run since these tests landed on 2026-08-13; the
intervening green runs were pull_request events exercising that PR's own
corrected fixtures.
Contract-Neutral: test fixtures only, no production runtime change.
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
Field telemetry showed installs with Patrol enabled recording weeks of
empty error runs (runs_30d=122, ai_calls=0, findings=0): provider
initialization failed once at boot (model resolution can need the
provider's live catalog, so Pulse racing a booting Ollama server loses)
and was never retried, while the run loop kept recording "Patrol
provider not configured" errors that told operators who had configured
a provider to configure one.
- Retry provider initialization on every scheduled run, so a boot-time
race strands Patrol for at most one interval instead of until the
next settings save. LoadConfig records the redacted init failure.
- Name the real failure in the blocked reason and run record when the
configured provider failed to initialize, instead of claiming no
provider is configured.
- Raise the deduped Patrol runtime finding when scheduled runs are
skipped by a persistent readiness blocker or missing provider, so the
state reaches the findings surfaces and alert notification channels
once, instead of living only on the Patrol page banner. Transient
circuit-breaker blocks stay finding-free; the attempts that opened
the breaker already raised their own.
- Resolve the runtime finding when Patrol is turned off; opting out is
a resolution, not a state to keep nagging about.
- Record the extended runtime-failure surface in the ai-runtime
subsystem contract.
A hard-mounted network filesystem with an unreachable server blocks
statfs in an uninterruptible kernel wait. The collector issued that
syscall inline for every mount, so one dead NFS mount silently froze
the whole reporting cycle and stretched agent shutdown into the kernel
retry window, and it paid that price for mounts the fstype filter was
going to discard anyway.
Decide type- and mountpoint-based skips before the usage syscall, so
network filesystems are never probed unless explicitly included, and
bound every remaining usage call with a timeout that leaves at most one
in-flight probe per mountpoint. A mount whose call never returned is
skipped on later cycles and re-included when the stalled call answers.
Refs discussion #1747
Contract-Neutral: Discussion #1747: behavioral bugfix in host disk collection; no payload or API shape change, filtered mounts were never reported
Since the operational-trust records shipped in v6.2.0, every new firing
of a previously resolved alert folded into that alert's first history
row: setActiveAlertNoLock stamped the new occurrence's open record onto
the previous occurrence's resolved row, and the history dedup then
treated any two open records with the same identity as one incident
regardless of how far apart they were. Alert history therefore froze at
the upgrade date while notifications kept flowing, which is exactly how
users reported it.
Resolved rows now keep their final record unless the update belongs to
the same occurrence, and two open records no longer merge on identity
alone. The observation-gap window still coalesces genuine flapping, and
the five-minute refire continuity path is unchanged.
Refs #1497
Contract-Neutral: Refs #1497: behavioral bugfix in alert history occurrence dedup; no API shape change, history row schema unchanged, no subsystem contract names sameHistoryIncident
The default attempt budget was three, and with the 1s/2s/4s backoff a
destination that was unreachable for about ten seconds had its
notifications dead-lettered permanently. A webhook receiver rebooting
alongside the infrastructure it monitors is routine, not terminal.
Eight attempts under the same doubling schedule span roughly three
minutes before dead-lettering. Dead-letter semantics are unchanged.
Refs #1721
Contract-Neutral: Refs #1721: raises the default delivery attempt budget only; dead-letter semantics, retention, and outcome vocabulary unchanged, notifications.md pins no attempt count
normalizeOverrides ran every override through NormalizePoweredOffSeverity,
which maps an unset severity to an explicit warning. Any guest with any
per-guest override (a disk tweak, a note) therefore had its powered-off
alerts silently downgraded from a global critical default to warning, and
the stamped value also round-tripped back to the UI as if the user had
chosen it. Leave unset severities empty so the merge keeps following the
global default, and normalize only values the user actually set.
Refs #1738
Contract-Neutral: Refs #1738: behavioral bugfix in alerts override normalization; poweredOffSeverity stays an optional field, no subsystem contract names it, no API shape change
The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.
The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
The delivery health verdict said when something was wrong; nothing showed
what actually fired and where it went. Expose the queue's retained
per-attempt audit rows as GET /api/notifications/delivery-log (newest
first, retention-labeled, webhook secrets redacted from error text) and
render them as a Recent delivery activity card on the alert destinations
tab, with outcome badges, destination names, and failure classes. Audit
rows now persist the normalized destination identity in a destination_id
column; older rows fall back to their operational links.
Test sends bypass both the queue and the activation gate, which is exactly
how installs came to believe delivery worked while every real alert was
suppressed (483 installs in the 08-18 telemetry read). Successful test
responses now carry deliveryPaused: true whenever the manager is gated
off, and the destinations UI warns instead of celebrating.
LoadAIFindings and LoadAIChatSessions swallow parse failures by returning
empty data with a nil error, and an undecryptable file lands in the same
branch because a failed Decrypt falls back to parsing the ciphertext as
JSON. Every read-modify-write saver (SaveAIFindingsWithSuppression,
SaveAIChatSession, DeleteAIChatSession, CleanupOldAIChatSessions) then
sees "empty, no error" and rewrites the file, permanently destroying
whatever it still held — user-authored suppression rules and chat
conversations. 9038afc68 fixed the read-error variant of this clobber;
this is the parse/decrypt variant that bypassed that abort because the
loader hid the failure.
Policy is preserve-for-recovery: the loader moves the unreadable file
aside to <file>.corrupt-<timestamp> and logs a warning before returning
empty data, keeping the self-healing behavior without the silent data
destruction. If the move itself fails, the load now fails, so the savers
abort instead of rewriting. recordActivityHistoryLocked keeps its
reset-on-parse-error behavior — those files are content-free telemetry
markers.
Tests cover corrupt and wrong-key-encrypted findings/session files being
preserved across the previously clobbering saves, and the move-failure
path aborting every saver while leaving the corrupt file intact.
SaveAIFindingsWithSuppression and SavePatrolRunHistory read the existing
file to carry data forward (suppression rules, the daily run tally) and
treated a failed read as nothing to preserve, rewriting the file without
it. Same clobber mechanic as the system.json reset fixed in cf5b86e62.
Both loaders return empty data with no error for a missing file, so an
error at these sites is a real read failure.
Suppression rules are user-authored config, so that save now aborts and
returns the read error. The run tally is telemetry, so that save
proceeds but logs a warning that the tally restarts. Tests cover the
abort leaving the file intact, explicit rules saving despite a failing
read, and the tally warning plus restart.
A LoadSystemSettings error in configureMonitorDependencies silently
skipped the tenant-monitor inherit block, leaving the monitor on
deny-all-private webhook CIDRs with nothing in the logs to explain why
private webhook targets fail SSRF validation. reloadSystemSettings
fails closed on the same error (embedding off) deliberately, but was
equally silent, so a persistent read failure looked identical to
embedding being switched off on purpose.
Both paths now log a warning with the error, mirroring the pattern from
cf5b86e62 in internal/config. A missing system.json (fresh install)
still stays quiet. Tests cover the warning on read failure, the quiet
fresh-install path, and that the reload still fails closed.
A boot-time LoadSystemSettings error fell into the missing-file branch:
no log line, every persisted system setting reverted to defaults for
the run, and a default system.json was written over the file on disk.
Observed once in practice (2026-08-20): a restart came up with a
persisted toggle unset while system.json still held the correct value.
Distinguish the error from the genuinely-missing file: retry the read
once (transient failures at boot were the observed shape), then warn
with the error and file path and run on defaults without touching
system.json. Tests cover the warning, the untouched file, and the
retry-once behavior.
Discovering Docker inside Proxmox LXC guests previously required an
undocumented environment variable plus a hand-edited systemd unit,
which is how the surface kept shipping invisible. The opt-in is now a
persisted system setting with an admin-only toggle in Settings →
System → General, presented as its own "Docker in Proxmox LXCs" card
under the existing Docker/Podman updates card, with the requirements
(node agent with command execution) and data-collection bounds stated
inline.
The setting rides the admin-gated system settings endpoint
(RequireAdmin + settings:write, matching every other server-wide
toggle). Flipping it persists first, then applies to the runtime
config and fires a reconfigure hook so the router rebuilds the
monitor's Docker checker and inventory collector immediately — no
restart. PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY remains
authoritative: when set it locks the toggle (EnvironmentLockBadge in
the UI, HTTP 400 from the API), and an unrelated settings save can
never clobber the runtime value. The opt-in guardrail test now pins
the settings path, the env lock, and the persisted field.
Installer and node setup-guide callouts now point at the toggle first
with the environment variable as the locking override, and the docs,
their public mirrors, and the agent-lifecycle subsystem contract say
the same. New i18n strings shipped for en/de/es.
Verified live in mock mode: toggle off/on from the UI drives the
collector teardown/setup log lines, the value survives a server
restart, and all three touched surfaces render at desktop and mobile
widths (receipt in frontend-modern/browser-verification.json).
Mock mode never exercised the backup-running states from "Stop showing
an in-flight backup as a completed one": no guest carried
BackupInProgress, no PBS listing showed an in-flight snapshot, and no
vzdump task was ever running, so the blue Running badge and the Running
artifact chips had no local reproduction.
A post-scenario fixture now flags roughly one guest in twenty-five per
last-backup age bucket - never, fresh, stale, overdue - so the Running
badge appears alongside every badge story, and gives each flagged guest
the evidence a live run actually produces: an in-flight PBS snapshot
shaped the way the PBS API lists one still being written (no size, no
verification, only the guest config blob in files) and a running vzdump
task with no end time. Running after applyDemoBackupScenario keeps the
per-tick reapplication idempotent, because the scenario's curated sweep
drops the previous pass's artifacts before the fixture re-adds them.
The first-cycle showcase guests are excluded: their exact rows are
pinned by the demo posture tests (a failed run three hours ago, a guest
with no recovery evidence at all), and a backup running right now would
mask exactly the states those rows exist to demonstrate. Later profile
cycles repeat the same stories under suffixed names, so the fixture
draws its running guests from them.
An in-progress PBS snapshot (reproduced on VM 117 at 9% written) was
counted as a completed backup: the guest's LastBackup jumped to "now" the
moment vzdump started, and the workloads badge showed a green healthy
shield for a backup that did not exist yet. If the job aborted, that
shield had never been true.
Detect the in-flight state from what the PBS listing actually says - no
size and no index.json.blob manifest in files - and flag the snapshot
InProgress instead of complete. On the PVE side, correlate storage content
entries with live vzdump tasks: a guest archive whose ctime falls inside a
running task's window is that task's partial output, and a PBS-storage
entry without a size has no manifest yet.
In-flight artifacts now feed a new per-guest BackupInProgress flag rather
than LastBackup, so the badge age always reflects the latest COMPLETED
backup. The workloads badge gets a distinct blue "Running" state (green
would claim success, None would hide that work is underway), with the
last-completed age kept in the tooltip and accessible label. Recovery
points map to OutcomeRunning with no completion time, keeping posture
freshness and the Backups page's latest-restore pointers anchored to
completed backups; the recoverable-artifact list shows a Running chip.
A cached in-flight snapshot forces a per-poll re-fetch of its group so
completion or abort is noticed promptly instead of after the cache TTL.
The first cut of the fixture ran inside buildFixtureState, but every
fixture graph build and metric tick then applies the demo scenario
layer, which renames guests, aliases cluster instances, forces guest
states, and reprofiles every Docker host. The nested hosts got swept
into the Docker fleet profile cycle: renamed away from their parent
guests, handed native engine inventory the production pct exec path
never reports, and one could be forced offline as the demo's
disconnected host.
The fixture now runs from applyDemoScenarioGraph after the workload
and Docker scenarios, so nested hosts bind to the guests' final names,
instances, and states, and it is idempotent across ticks: an existing
nested host re-syncs from its guest instead of duplicating, one whose
guest stops running goes offline like a real probe target, and the
sync strips every native-agent surface each pass. The Docker scenario
now skips proxmox-lxc-docker: hosts entirely, with the forced-offline
index counted over native hosts only.
The Proxmox page's nested-container row cue, the drawer's nested Docker
card, and the proxmox-pve platform scoping of nested workloads had no
local reproduction at all: mock mode never generated a Docker host with
the proxmox-lxc-docker: identity prefix, so the surface could only be
seen against a live estate with node agents, exec-scoped tokens, and
the server-side inventory opt-in. That is how it shipped invisible.
The fixture now marks the first two running LXC guests as probed
Docker hosts and appends a nested Docker host for each, following the
production proxmoxGuestDockerAgentID convention of
proxmox-lxc-docker:<instance>:<node>:<vmid>. One guest nests a single
container and one nests several, so the row cue renders both its
singular and plural counts. The nested hosts mirror the shape of the
real pct exec inventory: guest-derived sizing and usage, one rootfs
disk, and no native engine inventory, host I/O rates, temperature,
machine ID, or Swarm membership.
The pct exec socket probe that discovers Docker inside LXC guests
failed invisibly: errors logged at debug only, and a failed probe never
set DockerCheckedAt, so the guest stayed in first-check state and was
re-probed on every poll forever. On a node where lxc-attach could not
enter unprivileged guests that meant ~49 pct execs per minute of
steady-state churn, with nothing at default log level to say why the
Proxmox page showed no Docker.
Track consecutive probe failures per guest in the monitor. The first
failure of a streak, and the point where the backoff reaches its cap,
log at warn with the guest, node, and error; repeats stay at debug.
Failed probes retry with exponential backoff from one minute to a
thirty-minute ceiling, a success clears the streak with an info log,
and reconfiguring the checker resets all streaks so command execution
enabled from the UI at runtime still retries immediately. Entries not
refreshed for a day belong to deleted guests and are pruned.
PULSE_PROXMOX_GUEST_DOCKER_INVENTORY_VMIDS now gates the socket probe
as well as inventory collection. It previously only limited inventory,
so opting into specific guests still probed every running LXC on every
cycle; now guests outside the allowlist are never pct exec'd at all.
Subsystem contract and configuration docs updated to match.
An agent whose token authenticated seconds ago cannot be in a
credential outage, yet the fleet doctor flagged it critical with
"missing or revoked" whenever the token id missed the server's token
registry view, and the offered repair-authentication loop cannot fix a
stale server-side registry view. When the subject's own row shows the
judged token authenticating within the freshness window, emit a
warning-severity registry-stale verdict naming the id and the last
authentication instead. Genuinely stale rows keep the critical verdict.
Refs #1730
ec8b88fe2 registered the alert-form docker identities as Patrol scope
aliases for hosts and containers, but Swarm service alerts publish a
third form, docker:<host>/service/<serviceID>, which matches neither
the host alias docker:<host> nor the container alias
docker:<host>/<containerID>. Have Patrol investigate on a docker
service alert still returned patrol_scope_unresolved, and the
automatic alert-fired patrol path sent the same identity, so scoped
runs it triggered for service alerts failed resolution the same way.
Mirror the service resource IDs onto the owning docker host record.
Patrol has no service-level analysis, and a service's tasks run as
containers on the host, so the host is the smallest unit Patrol can
investigate for a service alert. Export the alerts helper for the same
reason ec8b88fe2 exported DockerResourceID, so the identity format
cannot drift between the two subsystems.
Hosts without an ID are skipped. The host-less fallback form
docker-service:<name> is not host qualified and would alias unrelated
records together across hosts, which is the same anti-aliasing rule
dockerAlertScopeAlias already applies.
Covered by tests asserting the alias equals what the alerts subsystem
emits for the same input, that the host-less guard registers nothing,
and that a service without an ID follows the canonical name-derived
form.
Refs discussion #1699
Contract-Neutral: behavioral bug fix refs discussion 1699, patrol scope resolution could not resolve docker service alert resource IDs, no public contract delta
TestRecoveryPointsEndpointReturnsMockData sampled the unfiltered first page
of /api/recovery/points and expected to find a kubernetes point in it. That
held only while the mock estate was small.
Points come back newest-completed-first and a page is hard-capped at 500. The
rescaled Proxmox demo estate now generates 2498 points, 939 of them newer than
the newest kubernetes point, so kubernetes starts at index 939 and never
reaches page one. The 36 kubernetes points are still generated and still
served, and the endpoint, the store ordering and the fixture shape are all
correct: a platform that snapshots every few days legitimately falls behind a
platform that backs up hourly. TrueNAS was already at index 254 and would have
broken next.
Ask the endpoint per platform instead, which is how the API is designed to
answer the question and how the UI asks it. The assertion no longer tracks
estate size, and it additionally covers the platform filter.
Contract-Neutral: test-only change, no behavior or contract delta.
Two counters could not answer the question they exist for.
pulse_intelligence_patrol_runs_30d was counted from the operator-facing
patrol run history, which is capped at MaxPatrolRunHistory (100). On any
install patrolling on a normal schedule that cap is reached within hours
of a thirty-day window opening, so the field reported the cap rather than
the run count, and the calls-per-run ratio derived from it was inflated by
a censored denominator. Patrol run history now carries an uncapped daily
tally beside the capped list, advanced from a persisted high-water mark so
repeated full-list saves cannot double count, pruned to 31 days, and read
in preference to the list. An install with no tally yet falls back to the
list, so the counter never regresses on upgrade.
Schema v8 split agent-side pre-mutation refusals into target-change,
prerequisite and contract categories so they would stop hiding in "other".
In production all three are zero fleet-wide and "other" still absorbs every
refusal, because the dominant reason code is the legacy preflight_refused
aggregate recorded when an agent sends no machine reason code at all.
Agents older than the typed refusal contract report every refusal that way,
so the split is starved rather than broken, and folding the two together
made those two states indistinguishable. Schema v9 counts uncoded refusals
separately.
The Proxmox node sweep removes any non-preserved alert whose Node is
empty, and system alerts have no node, so every sweep silently deleted
the notification-delivery alert. Its five-minute evaluation then
re-raised it as a brand-new alert, firing a fresh notification each
cycle with no recovery in between, which reads as an alert appearing,
vanishing without a recovery, and paging again minutes later. System
alerts are now preserved outside node cleanup.
Raises also carried the delivery counts inside the message, and a
message change re-notifies, so each newly retained failure re-paged
even while the condition was unchanged. System alerts now take an
optional fingerprint: while level and fingerprint hold, a re-raise
refreshes the message and metadata silently. The delivery alert
fingerprints on status and failure classes, keeping its counter text
current without paging on drift.
Refs #1721
Contract-Neutral: behavioral fix: delivery-health system alert survived node sweep and stops re-paging on count drift, no public contract delta (Refs #1721)
TestCollectDeviceSMARTStandby and its observability twin stub smartctl's
execution but not the device-class probe, so collectDeviceSMART fell through
to the *runner's* real /sys/block/sda/queue/rotational.
smartctlArgs only sends the -n standby guard when the disk is not a confirmed
SSD (#1516), and smartctlArgsUseStandbyExitStatus gates the standby reading of
exit status 3 on that guard having been sent. On a Linux host whose own
/dev/sda is non-rotational the guard is dropped, exit 3 stops meaning standby,
and both tests fail deterministically:
run smartctl for /dev/sda: exit status 3
They pass on macOS only because linuxNonRotationalBlockDevice returns false off
Linux, and on Linux only where /dev/sda is absent or spinning. That makes the
release-qualification suite unrunnable on an ordinary SSD-backed Linux worker.
Pin the probed disk to rotational through the package's existing stubLinuxSysfs
seam so the guard is always in play. Product behaviour is untouched; this only
stops two unit tests depending on the hardware underneath them.
TestSubscriptionAgentClientsUseStructuredSingleTurnProcess failed three times
during the v6.3.0-rc.3 release qualification, always at exactly its 5s
deadline, with "codex subscription agent timed out". It was green 5/5 in
isolation: 3/3 as a single test under -race and 2/2 for the whole package.
It only failed inside the full `make test` run, where -race instrumentation
and roughly 150 packages compete at once.
The test is hermetic. It writes a fake codex shell script onto PATH, so there
is no CLI, network, or credential involved and nothing about the deadline is
asserting product latency. It exists to stop a hang. Under race instrumentation
on a loaded machine, spawning that script legitimately takes longer than the
budget, so the deadline was reporting how busy the host was rather than whether
the code works.
Scale the deadlines when the race detector is enabled, using the raceEnabled
build-tag constant that internal/api, internal/monitoring, and pkg/metrics
already use. Scaling rather than skipping is deliberate: the surrounding
package convention is to skip latency SLO tests under -race, but this one is a
functional proof of the structured single-turn process, tool calls, and
credential non-leakage. Skipping it would remove exactly the path we most want
the race detector to see.
The assertion in TestSubscriptionAgentRequestTimeout is untouched, since that
one is testing the timeout policy itself rather than waiting on anything.
Proof: package passes 2/2 under -race and once without, and both build variants
vet clean.
Contract-Neutral: Test-only change scaling hermetic subscription-agent deadlines under -race via the existing raceEnabled build-tag pattern. No runtime, contract or payload delta. Proof is the package itself, which passes 2/2 under -race and once without.
Pulse could tell that its own delivery had stopped and had no way to say so.
Queue health was computed for one API endpoint and rendered on the alerts
surfaces, and that was the whole escalation path: a destination that stops
delivering cannot announce itself through a notification, so an operator who
does not open the alerts page learns nothing.
Telemetry: 78 installs retry every notification to the maxAttempts ceiling and
deliver almost nothing, 155,967 attempts against 221 deliveries in a week. 50
have delivered nothing in 30 days and 28 more delivered normally at some point
and silently stopped. The dominant terminal failure classes are authentication
and configuration, so most are fixable in minutes once someone knows.
Move the queue-health rule into internal/notifications as ClassifyQueueHealth
and DeliveryHealth, next to the queue that produces the counts, and delegate
the API handler to it so the endpoint and the new evaluator cannot drift into
two rules. A queue that cannot be read reports unavailable rather than healthy,
because silence is the failure mode being guarded against.
Monitoring evaluates delivery health on the poll ticker and raises or clears
the notification-delivery system alert, which puts it in the alert list and the
navigation badge. Reading queue health costs a SQLite query and the poll
cadence can be seconds, so the evaluation is throttled to five minutes.
RaiseSystemAlert is idempotent for an unchanged condition, so the timer neither
re-notifies nor accumulates alerts.
The alert is a warning rather than critical. It is real, but a new
critical-by-default alert appearing on upgrade across the affected installs is
a bigger change than this should make on its own.
The new Monitor field is deliberately named to fit inside the existing struct
alignment column. A longer name makes gofmt re-pad the whole block, which
breaks the canonical guardrail tests that pin those field declarations
verbatim.
Proof: delivery_health_test.go pins that in-flight work and retries stay
healthy while retained terminal failures do not, and that an unreadable queue
is never reported healthy. system_alerts_test.go pins the message for each
outcome including singular and plural, that it points at the destinations
surface, and that the throttle honours its interval. internal/monitoring,
internal/notifications and internal/alerts suites all pass.
Contract-Neutral: Staged contract deltas cover every boundary this change actually moves: notifications.md gains delivery_health.go plus the ClassifyQueueHealth rule, and monitoring.md gains system_alerts.go plus the system-alert evaluation extension point. The residual demands are inapplicable. agent-lifecycle is demanded only because monitor.go is one of its canonical files, and its verification artifact is host-agent deletion and re-enrollment proof, which this change does not touch: the diff to monitor.go is one struct field and one call in the poll loop. api-contracts and storage-recovery are demanded only because internal/api/notifications.go falls under a broad internal/api reference, and that file's change is a pure delegation of an internal helper to notifications.ClassifyQueueHealth with a byte-identical HTTP response and no route or payload delta.
Every alert type is scoped to a monitored resource: a node, a disk, a
container, a host. That leaves no way to report a condition whose subject is
Pulse, and the condition that needs it most is broken notification delivery,
where the channel that would carry the warning is the thing that failed.
Telemetry shows what that costs. 78 installs retry every notification to the
maxAttempts ceiling and deliver almost nothing, 155,967 attempts against 221
deliveries in a week; 50 have delivered nothing in 30 days and 28 more
delivered normally at some point and silently stopped. The dominant terminal
failure classes are authentication and configuration, so most are fixable in
minutes by an operator who has no way to learn there is anything to fix. Until
now the only evidence lived on a config tab, because a delivery failure cannot
announce itself through delivery.
Add RaiseSystemAlert and ClearSystemAlert. A system alert carries the stable
pulse-system- identity prefix so repeated raises update one alert instead of
accumulating, sets no ResourceID so surfaces skip resource-linked affordances
rather than offering a link to nothing, and stamps systemAlert metadata for
surfaces that want to tell the two apart. Raising is idempotent for an
unchanged condition and only re-notifies on a change of level or message, so an
evaluator on a timer cannot turn into the notification storm this is meant to
warn about. Routing through the ordinary pipeline is the point: the alert
reaches the alert list and the navigation badge, which is the escalation path
that does not depend on delivery working.
This commit adds the capability and its contract entry. Nothing raises a
notification-delivery alert yet: the evaluator needs a package holding both the
notification queue and the alert manager, and internal/monitoring plus
pkg/server are held by another agent's path claim. The evaluator and its UI
verification follow.
Proof: four cases in alerts_test.go pin the stable identity and absent
ResourceID, the idempotence that keeps a repeated raise from re-notifying, the
level change that does re-notify while still updating a single alert, the clear
path including clearing something absent, and the rejection of a blank type.
Full internal/alerts suite passes under -race.
checkFlappingLocked recorded a cooldown deadline in suppressedUntil and then
never read it. Suppression was decided purely by whether the trailing
FlappingWindowSeconds still held FlappingThreshold state changes, so it lifted
the moment the sliding window drained instead of holding for
FlappingCooldownMinutes. Only the metric paths read that map, and they read it
for the separate SuppressionWindow feature.
Two consequences. A resource oscillating just under the threshold rate, four
changes per five minutes against a threshold of five, was never damped at all
and dispatched every single state change. And flappingActive was only ever set
to true, never cleared except by age-based cleanup, so even after the first
episode no later episode could arm a fresh cooldown.
Pulse states the guarantee it was not providing. The flapping postmortem
finding tells the operator that notifications were "suppressed for" the
configured cooldown and recommends raising it, which changed nothing.
Honour an active cooldown before the window check, and release the latch once
the cooldown has been served so a later episode can arm a new one. Releasing
also clears the stale window history, so a resource that has gone quiet starts
from a clean slate rather than re-tripping on expired state changes.
This damps notification volume. It does not reduce alert churn: flapping is
evaluated in dispatchAlert, so the alert has already been raised and recorded
in history by then. Telemetry shows 923 installs firing 3.64M alerts in 30 days
with only 20,910 standing, and that churn needs per-alert-type root causes such
as #1721.
Proof: three cases in alerts_test.go drive checkFlappingLocked directly and pin
that the cooldown keeps suppressing after the window drains, that it releases
and can re-arm for a later episode, and that disabled flapping detection
ignores a stored cooldown. All three fail against the previous implementation.
Contract-Neutral: behavioral fix: the flapping cooldown now gates the dispatch path that already recorded it. No alerts subsystem contract delta (no Canonical Files, Shared Boundaries or Extension Points change, no config/API/payload change) so alerts.md has nothing substantive to record. The registry-listed verification artifact demand IS satisfied: proof lands in internal/alerts/alerts_test.go.
Adding a PVE/PBS source through the Connect via API path discarded the
name typed in the add dialog: the setup bootstrap only carried type and
host, and auto-registration named the new connection after the node's
self-reported hostname. A user asking for a cluster named enacon got a
connection called pve01 and could only rename it after adoption.
Carry the typed name on the one-time setup token
(SetupTokenRecord.DesiredName) via an optional name field on
/api/setup-script-url, and let the setup-token-authenticated
auto-register completion name a newly created connection from it,
falling back to the hostname-derived default when absent. Dedup and
cluster-member adoption identity stay hostname/candidate based, so the
carried name cannot fork an existing registration.
The app shell decides which primary platform pages exist by classifying
every resource in the legacy full-state payload, which is why it has to
download that payload before it can render navigation. This publishes the
same answer as a `platformAdmission` facet on the canonical resource
aggregations, so admission has one definition instead of two that can
drift.
Counts cannot answer it, which is the whole reason this is a facet rather
than a client-side tally over `bySource`. A TrueNAS or Proxmox host
reports through the agent source and carries the "agent" platform scope,
so a count-based derivation admits the standalone page for an estate that
has no Pulse agent in it at all. Ownership is per-resource evidence, so it
is evaluated per resource here.
Verified against the live client classifier over real estates rather than
by inspection: the facet and the classifier agree on all six pages across
ten estates each for a 652-resource and a 216-resource estate, including
the cases that break a count-based derivation (Proxmox-only, TrueNAS-only,
vSphere-only and provider-owned-agents-only estates all correctly withhold
the standalone page, while a genuine Pulse agent admits it).
The tenant-fallback contract pin gains the new field. Its invariant, that
an empty tenant returns no resources rather than seeding from the raw
snapshot, is unchanged and still pinned; an empty estate admits nothing.
No caller reads the facet yet. Moving the shell onto it is a separate
change, gated by the same parity comparison.
Contracts: unified-resources and api-contracts gain the facet and its
derivation rule; agent-lifecycle and storage-recovery gain the ownership
clause, since an agent-typed host owned by a provider must not admit the
standalone page.
The thresholds UI stores per-resource overrides keyed by unified
registry resource IDs, while the engine check paths look up legacy
monitor IDs, so a per-node override on an API-only PVE node saved,
displayed as Custom, and never applied: the node kept alerting at the
global defaults. Reproduced live against mock: memory at 81% with a 42%
override stored under the registry ID raised nothing.
Route the node, PMG, PBS, and host/linked-node override lookups through
the canonical registry identity resolver the intent-policy path already
uses, with the exact legacy key still winning so overrides stored by
older UI versions keep resolving unchanged.
Proof: TestNodeThresholdOverrideStoredUnderRegistryIDApplies drives the
real populate path for an API-only node, stores a memory override under
the registry resource ID the UI uses, and asserts CheckNode fires at the
override threshold while controls without the override or without the
resolver stay quiet.
Refs #1738
Contract-Neutral: behavioral fix#1738: threshold override lookup resolves registry identity, no public contract delta
A Critical "credential missing or revoked" verdict never said which
credential it judged, so a host row whose last-seen token id went stale
after re-enrollment reads as a live outage even while the agent is
reporting fine on a fresh token, and the operator has nothing to map
against the token list. Emit the judged token record UUID (not secret
material) alongside the missing and expired verdicts.
Refs #1730
Contract-Neutral: diagnostic evidence detail only in fleet doctor credential verdicts, refs #1730
Host identity continuity deliberately keeps a known machine on its
enrolled ID, so a custom --agent-id or hand-edited agent-id state file
is silently ignored for an already-known host, and the acknowledged ID
then overwrites the state file within one report cycle. That looked
like corruption from the operator's seat. Warn once per resolved
identity with both IDs and the supported path to a fresh enrollment
(remove the host in Pulse first).
Refs #1739
Contract-Neutral: diagnostic logging only in hostagent report ack path, refs #1739
rcourtman