Websocket-first routes delayed their REST fetch by 1.2s to give the
realtime snapshot a chance to paint first and save a request. That
trade only exists for `prefer-ws` routes. Every current platform
surface is `prefer-ws-then-rest`, which always follows its websocket
paint with a REST revalidation, so the wait never saved the request.
It only postponed the canonical snapshot, and left the route blank
whenever the websocket snapshot was slower than the wait.
That failure mode scales the wrong way: the larger the estate, the
bigger the initial websocket frame, the more likely the wait is lost
outright, and the more the blank screen costs. On the public demo the
websocket snapshot is 1.9MB and lands ~2.5s after mount, so the wait
was pure dead time on every cold load.
Routes that revalidate over REST now race the two transports and paint
from whichever arrives first. `prefer-ws` routes keep the wait, which
is where it earns its keep.
Measured on a production bundle against the demo estate with the
realtime transport unavailable, so the canonical fetch is the only path
to paint: bootstrap-to-fetch gap 1720-1882ms to 519-613ms, first paint
5083-6384ms to 3349-4860ms.
Covered by a regression test asserting the fetch is issued without the
hydration timer advancing, plus one pinning that `prefer-ws` still
waits.
Contract-Neutral: Scheduling-only change to initial hydration: identical endpoints, params and payloads; unified-resources contract documents no initial-fetch timing clause
fetchUnifiedResources walked its pages nose-to-tail, awaiting each
response before requesting the next. Only the first page is actually
needed to learn how many there are; the rest are independent, so the
walk spent a full network round trip per page before the first
canonical snapshot could land and the page could paint.
On the public demo (209 resources, 3 pages at limit=100) the serial
walk measured 1672-2317ms of the roughly 6s cold-load blank screen.
Fanning the remaining pages out collapses that to one round trip plus
the slowest straggler, measured at 1217-1246ms against the same
backend, with pages 2 and 3 leaving in the same tick.
The outer loop still re-reads totalPages from the fetched pages, so an
estate that grows a page mid-flight is picked up exactly as before.
Covered by a regression test that holds pages 2 and 3 unresolved and
asserts both were requested while the first was still outstanding; it
fails against the serial implementation.
Contract-Neutral: Concurrency-only change to unified resource pagination: identical endpoints, params, payloads and assembled result; unified-resources contract documents no page-walk scheduling clause
Contract-Neutral: Patrol autonomy behavior is documented in its primary ai-runtime and shared api-contracts contracts plus the relevant agent-lifecycle boundary; storage-recovery only has a broad internal/api extension reference and does not own this wire contract
The gate that mirrors command-channel admission
(commandConfigAllowedForToken) runs on every agent report, before the
agent ever attempts channel registration. When it refuses, the server
silently serves commandsEnabled=false, the agent never tries to
register, so none of the channel rejection warnings fire, and an
operator who enabled lifecycle management and granted agent:exec sees
"Remote commands: Not enabled" with nothing in any log. Emit the
refusal and the binding metadata it was judged against so the mismatch
is diagnosable from journalctl.
Refs #1728
Contract-Neutral: diagnostic logging only in agent command config gate, no behavioral or public contract change
Read paths refresh canonical identity on every clone, so views always
showed the derived aliases, but the stored registry entries never carried
them: snapshot ingest built resources without the canonical block and only
the IngestResources seed path stored refreshed clones. GetByReference's
canonical-alias fallback reads the stored entries directly, so prefixed
refs like "agent:{hostID}" never resolved on a production rebuild and
per-resource alert grace overrides on merged PVE-node + pulse-agent hosts
silently fell back to factory policy. The earlier alias fix (dd008f1c2)
only proved the IngestResources-seeded path. Recompute identities in each
ingest finalize pass, after links and merges settle.
Proof: TestMergedProxmoxHostAgentGraceOverrideHoldsCPUAlert drives the
real populate path (node plus linked agent merging into one resource),
saves a 600s metric.cpu override under the registry resource ID the UI
uses, and asserts CheckHost holds the alert while a control manager
without the override fires immediately.
Refs #1497
Contract-Neutral: behavioral fix#1497: populate derived canonical identity on stored registry entries, clones already exposed identical data, no public contract delta
A second adversarial review round surfaced two robustness gaps. The
incident recorder treated any ID absent from a metrics batch as an
empty successful sample, which silently rewrites the semantics of a
provider whose per-ID method errors or synthesizes for unknown IDs;
batch misses now fall through to the per-ID call, keeping the batch
for the hot path only. And one bySource mutation site, the PBS host
disk association helper, predated the source-target cache invalidation
invariant; it now invalidates like every other mutation path. The gap
was unobservable today because the adjacent ingest already invalidated,
but the invariant is only useful if it is total.
Contract-Neutral: robustness fixes inside existing cache and sampling semantics, no resolved values change
The cached inverse source-target index assumed a clean viewsDirty flag
meant current mappings, but batch ingests release the registry lock
between records and only mark views dirty in their epilogue. A reader
interleaving mid-batch was served pre-batch metrics targets from the
cache, where the pre-index code always scanned live state. Every
bySource mutation path now drops the cache inside its critical
section, sending mid-batch readers to the live scan until the next
view rebuild, which restores the exact pre-index freshness. A test
pins the invalidation to the mutation itself rather than the epilogue
flag.
Contract-Neutral: staleness fix inside the cache lifecycle, resolved values and selection semantics unchanged
The mock unified snapshot resolved a metrics target per resource on a
freshly ingested registry whose view caches were still dirty, so every
resolution fell back to a scan over all source mappings, turning the
loop quadratic at demo scale. One view read builds the caches first,
making each resolution an index lookup.
Contract-Neutral: performance-only cache warm before target resolution, resolved targets unchanged
The derived-parent recompute in buildChildCounts may hit the agent
fallback scan once per guest, and that scan walked every resource with
per-entry string checks, an O(n^2) pass per registry ingest. The pass
now buckets agent resources by node name up front and the fallback
walks only the matching bucket. Selection is order-independent (best
score with smallest-ID tie-break) and scores read live through shared
pointers, so results are unchanged; callers outside the pass keep the
full scan.
Contract-Neutral: performance-only scan bucketing in the parent recompute pass, selection semantics unchanged
The incident recorder asks for current metrics for every monitored
resource every five seconds, and the adapter answered each request by
scanning all views with a formatted VMID comparison per candidate. At
thousands of resources that is millions of comparisons and allocations
per tick, forever. The adapter now exposes a batch method that builds
the full lookup in one pass over the views, keyed by every ID form the
per-ID method matches with the same first-match precedence, and the
recorder uses it when the provider offers it. Per-ID behavior is pinned
to the batch by an equivalence test, including VMID collisions.
Contract-Neutral: performance-only batch lookup addition, per-ID metrics resolution is pinned unchanged by equivalence tests
The fallback attachment pass compared every group against every other
group with pairwise hostname normalization, allocating sorted root and
host sets per call. Resolving top-level systems runs inside the state
request path, so large estates paid a quadratic normalization storm per
request. Hostname equivalence requires the two names to share either a
comparable form or a short form, so an index of groups by those forms
prunes each group's candidates to the provable superset and the
original pairwise matcher still decides every surviving pair. Attachment
decisions are unchanged.
Contract-Neutral: performance-only candidate pruning, the original matcher still decides every attachment
Rebuilding the registry view caches resolved each resource's metrics
target by scanning every bySource mapping, an O(resources x mappings)
pass holding the write lock. At about five thousand resources that is
tens of millions of map iterations per rebuild, and rebuilds run on
every poll cycle. The rebuild now inverts the mappings once and the
cached index also serves single-resource lookups while the views are
clean, with the legacy scan kept for dirty-state callers. An
equivalence test pins index results to the legacy scan.
The agent-parent fallback scan also ran its per-entry string
canonicalization before the type check, so every guest in the estate
paid allocation cost in a scan that only ever matches agents. The
cheap checks now run first.
Contract-Neutral: performance-only rework of metrics-target resolution and scan ordering, resolved values are pinned unchanged by equivalence tests
A handler that flushes before its first write commits the response
headers with an implicit 200. The gzip wrapper deferred its
compress-or-not decision to the first write, so such a handler would
send headers without Content-Encoding and then a compressed body,
which clients cannot decode. Every current flusher is an SSE handler
whose content type never compresses, so this was latent, but any
future streaming JSON endpoint would have corrupted silently. Flush
now runs the decision first, matching net/http header-commit
semantics.
The broadcast ticker logged seven fields at Info level every tick and
built a full state snapshot to do it, before checking whether any
WebSocket subscriber existed. An idle server paid a snapshot copy plus
roughly 8600 log lines a day for nothing. The log is now Debug and both
the log and the snapshot build sit behind the subscriber check.
The poll task worker pool was also fixed at ten workers no matter how
many instances an estate monitors. POLL_TASK_WORKERS now overrides the
count and the cap, bounded at 128, following the existing env knob
pattern. Behavior without the variable is unchanged.
Contract-Neutral: operational log demotion and an opt-in worker pool env knob, no monitoring or agent-lifecycle contract semantics change
The browser advertises max_message_bytes on the upgrade request and the
server withholds any state frame larger than that, falling back to REST
recovery. The advertised value was still the historical 8 MiB, which
pushed estates past roughly 3100 resources off the cheap socket delta
path and onto a 30 second full-state REST poll loop. 32 MiB keeps
estates about four times that size on socket deltas. The server honors
whatever the client advertises, so no backend change is needed.
Contract-Neutral: transport frame-limit constant, no alert lifecycle semantics change
Large estates that exceed the WebSocket frame ceiling recover over an
uncompressed /api/state poll loop, which made the biggest deployments
pay the heaviest transfer cost. Wrap the main HTTP handler in a gzip
middleware that compresses JSON and other text responses roughly an
order of magnitude, while leaving WebSocket upgrades, SSE streams,
Range requests, small declared bodies, and binary content untouched.
Contract-Neutral: transport-only gzip response middleware, no RBAC or persistence semantics change
rcourtman