The three-shard backend gate landed with a 16 GiB admission requirement
that exceeded the 8-vCPU PVE worker's own idle availability (measured
16.1-16.7 GiB, and 14.1-14.9 GiB beside the sibling release compilers),
so the next release would have hard-failed at admission before running a
single test. Direct probes on the worker measured the complete gate at a
~6-7.5 GiB footprint (8.9 GiB MemAvailable floor from a 16.4 GiB start,
zero swap), so admission now requires 10 GiB for three shards and 8 GiB
for two, and degrades the shard count instead of failing the release
when the bounded wait expires without headroom.
Shard CPU is now weighted by planned test volume from the plan manifest.
Top-level tests execute serially per test-binary process, so width mainly
buys runtime, GC, and race-detector headroom for the 3595-test prefix
shard; the ~15-test wait-bound tails cannot use it. Probes measured the
prefix shard at 569s with 2 procs versus 484s with 4, with total
allocation still equal to the worker's vCPU count.
Validated by three direct runs of the canonical script on the worker:
the full gate passed in 8m10s wall (auto-admitted three shards, procs
4/2/2, shard walls 484s/331s/224s) against the 12m09s two-shard release
job measured in run 32514803052. One rig run also proved the gate fails
closed: a stale contract pin failed the non-API graph and the harness
terminated every descendant shard process.
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
Field telemetry showed installs with Patrol enabled recording weeks of
empty error runs (runs_30d=122, ai_calls=0, findings=0): provider
initialization failed once at boot (model resolution can need the
provider's live catalog, so Pulse racing a booting Ollama server loses)
and was never retried, while the run loop kept recording "Patrol
provider not configured" errors that told operators who had configured
a provider to configure one.
- Retry provider initialization on every scheduled run, so a boot-time
race strands Patrol for at most one interval instead of until the
next settings save. LoadConfig records the redacted init failure.
- Name the real failure in the blocked reason and run record when the
configured provider failed to initialize, instead of claiming no
provider is configured.
- Raise the deduped Patrol runtime finding when scheduled runs are
skipped by a persistent readiness blocker or missing provider, so the
state reaches the findings surfaces and alert notification channels
once, instead of living only on the Patrol page banner. Transient
circuit-breaker blocks stay finding-free; the attempts that opened
the breaker already raised their own.
- Resolve the runtime finding when Patrol is turned off; opting out is
a resolution, not a state to keep nagging about.
- Record the extended runtime-failure surface in the ai-runtime
subsystem contract.
The publish-body condensation removed exact operator-facing statements pinned by the prerelease packet test. Restore those statements within the three-highlight limit, record the packet contract, and strengthen the proof to require the complete publish-safe sentences.
The internal/api race suite now routinely exceeds the old 20-minute package timeout on hosted runners while passing. Set a governed 30-minute package timeout and 40-minute release job ceiling, pin the relationship with contract tests, and refresh the rc.4 packet with the fixes landed since preparation.
The installer staged the download in /tmp and installed the runtime
binary to /usr/local/bin, both on the small RAM-backed QTS/QuTS hero
root, and the boot wrapper copied 34MiB back onto that root at every
boot. Roots without ~50MiB of headroom could not install at all, and
setting TMPDIR only moved the staging half of the requirement.
QNAP's own QPKG packages execute from the data volume, so do the same:
relocate the install dir to the data volume's state dir before the
preflight and download, default TMPDIR there too, skip the boot-time
self-copy when the stored and runtime binaries are one file, and remove
a pre-relocation runtime copy from /usr/local/bin to give that space
back. Split layouts with an operator-supplied state dir keep the copy
semantics. The rendered wrapper is exercised in both layouts by the
installer tests.
Refs #1617
Contract-Neutral: Refs #1617: QNAP installer layout fix with its deployment-installability contract clause staged in this commit; residual proof policies for unrelated boundaries do not apply to this shell-only change
The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
Live proof on a real systemd host (Debian 12 container, full
install/report/update/uninstall cycle) caught what no unit test did:
NoNewPrivileges=true blocks sudo outright, so the --grant-smart and
--grant-pct helpers failed inside the service and SMART/pct silently
disappeared while the install reported success. A unit with an active
grant now sets NoNewPrivileges=false, keeping the rest of the hardening;
a grantless least-privilege install keeps NNP enabled. Docs and contracts
state the trade-off: each grant is a scoped, auditable widening.
Also proven in the same live cycle: the flag-conflict refusals, user and
sudoers provisioning, wrapper execution as the service user, the healthy
privilege report ({runningAsRoot:false, serviceUser:pulse-agent,
smartctlHelper:true}), --update preserving the profile without repeated
flags, and uninstall removing the sudoers file and helpers.
rcourtman