With the adaptive scheduler disabled (the default), every planning pass
on the main poll tick rebuilt each instance task with NextRun=now and
Upsert overwrote the pending slot, so availability targets polled at
the tick cadence regardless of their configured interval. Reproduced
at exactly 10s polling on a 60s ICMP target; c70431caa only covered
the adaptive-scheduler path. Planning now preserves a pending future
slot, tightening it only when a freshly shortened interval justifies
an earlier run.
Refs #1745
A refused action plan, decision, or execution returned its 409 to the
client and left no trace in the server journal, so every remote report
of "Docker / Podman command agent is not connected" stalled on greps
that could never match anything. Refusals now log one warn line with
the resource, capability, and reason code, and the Docker command-agent
resolver reports which lookup missed (stale enrollment token binding vs
agent-id/hostname session), carried as an optional diagnostic detail on
the readiness contract and in the refusal envelope.
Refs #1728
The three-shard backend gate landed with a 16 GiB admission requirement
that exceeded the 8-vCPU PVE worker's own idle availability (measured
16.1-16.7 GiB, and 14.1-14.9 GiB beside the sibling release compilers),
so the next release would have hard-failed at admission before running a
single test. Direct probes on the worker measured the complete gate at a
~6-7.5 GiB footprint (8.9 GiB MemAvailable floor from a 16.4 GiB start,
zero swap), so admission now requires 10 GiB for three shards and 8 GiB
for two, and degrades the shard count instead of failing the release
when the bounded wait expires without headroom.
Shard CPU is now weighted by planned test volume from the plan manifest.
Top-level tests execute serially per test-binary process, so width mainly
buys runtime, GC, and race-detector headroom for the 3595-test prefix
shard; the ~15-test wait-bound tails cannot use it. Probes measured the
prefix shard at 569s with 2 procs versus 484s with 4, with total
allocation still equal to the worker's vCPU count.
Validated by three direct runs of the canonical script on the worker:
the full gate passed in 8m10s wall (auto-admitted three shards, procs
4/2/2, shard walls 484s/331s/224s) against the 12m09s two-shard release
job measured in run 32514803052. One rig run also proved the gate fails
closed: a stale contract pin failed the non-API graph and the harness
terminated every descendant shard process.
An enabled Patrol that can never run and one that runs and finds nothing
were indistinguishable in the fleet: both presented as high run counts
with zero AI calls and zero findings. The install that motivated this
sat blocked for over a month because provider initialisation failed once
at boot and was never retried; the self-heal landed separately, but
telemetry still cannot see which blocked cause dominates in the field.
Schema v10 exports the fixed machine cause code (for example
provider_not_configured) only while Patrol is in the blocked runtime
state. The cause rides the router-owned Pulse Intelligence snapshot into
the outbound ping. Blocked-reason text, provider endpoints, model names,
and configuration stay on the install; an untyped blocked reason exports
nothing rather than free text, and a disabled, active, or mid-run Patrol
exports an empty value even when a stale cause is still recorded.
Field telemetry showed installs with Patrol enabled recording weeks of
empty error runs (runs_30d=122, ai_calls=0, findings=0): provider
initialization failed once at boot (model resolution can need the
provider's live catalog, so Pulse racing a booting Ollama server loses)
and was never retried, while the run loop kept recording "Patrol
provider not configured" errors that told operators who had configured
a provider to configure one.
- Retry provider initialization on every scheduled run, so a boot-time
race strands Patrol for at most one interval instead of until the
next settings save. LoadConfig records the redacted init failure.
- Name the real failure in the blocked reason and run record when the
configured provider failed to initialize, instead of claiming no
provider is configured.
- Raise the deduped Patrol runtime finding when scheduled runs are
skipped by a persistent readiness blocker or missing provider, so the
state reaches the findings surfaces and alert notification channels
once, instead of living only on the Patrol page banner. Transient
circuit-breaker blocks stay finding-free; the attempts that opened
the breaker already raised their own.
- Resolve the runtime finding when Patrol is turned off; opting out is
a resolution, not a state to keep nagging about.
- Record the extended runtime-failure surface in the ai-runtime
subsystem contract.
rcourtman