LogParserCollector and EventLogCollector called no guard at all, while
File/Json/Xml did — and rules/gather-rules.md already promised customers
that log-parser paths were allowlisted. A tenant admin could author a
logparser rule against C:\Users via POST /api/rules/gather and have up to
1000 lines uploaded as telemetry, bypassing the hard C:\Users block the
same page guarantees; eventlog could read any channel, Security included.
The backend validates no collectorType, target, or parameters, so the
agent is the only place this can be enforced.
Both now guard. logparser cannot check rule.Target directly (GetFullPath
throws on a wildcard), so it checks the directory for wildcard targets and
re-checks each resolved path. Event log channels get an allowlist matched
on a "/" boundary. Hard-blocked channels live in a C# constant rather than
guardrails.json: a parse error drops the allowlists to empty, which is
fail-closed, but would have lifted a block sourced from the same file.
Also fixes an inert timeout in CommandCollector: ReadToEnd() blocks until
the pipe closes, so WaitForExit(30000) was only reached once the process
had already exited and a hung command pinned its worker indefinitely.
Adds the first real CI gate. No workflow ran tests on a pull request — the
deploy workflows build without testing and build-agent.yml covered only
DecisionCore.Tests on manual dispatch — while trust/security-faq.md
already told customers automated tests run in CI. ci.yml now runs all six
suites on every PR and merge, plus a job that fails on generated-guardrail
drift, alongside CodeQL and Dependabot. OfficeBinaryWatcherTests is marked
SerialThreading; its bounded re-probe races the shared ThreadPool and
failed once under full-suite load.
Verified: 3101 backend, 1911 agent V2, 474 decision core, 44 dialog, 572
web, 447 MCP. The timeout fix is verified by runtime, not assertion — a
Start-Sleep 120 command now returns after ~30s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A German question returns zero results from search_docs, not poor ones: both the
documentation and the rules are English, and all-MiniLM-L6-v2 is an English-only
model, so the query vector lands nowhere near any chunk. For a German-speaking
admin driving an MCP client in their own language, the tool is effectively blind
and gives no hint why.
The caller is itself a language model, so the cheapest correct fix is to say so:
both tool descriptions now instruct it to translate the question to English before
searching and to answer the user in their own language. No infrastructure, no
model change, no ranking impact.
A multilingual embedding model would handle it natively and was considered, but it
is substantially larger — which matters on a scale-to-zero container — and it would
change English retrieval as well, so it would have to be measured against
scripts/eval-docs-search.ts rather than swapped in on the assumption that more
languages cannot hurt.
Verified: 447 tests pass, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The form offered trigger, phase scope and emit mode unconditionally, which
produced combinations that cannot do anything plus one silently wrong state:
- A scope on a Phase Start / Phase End rule that names a concrete phase can only
ever suppress the single firing — the phase IS the moment. Same for emit mode:
a one-shot has no second result to dedup against.
- Selecting "From a phase onwards" (or "Only during specific phases") and not
filling the detail dropdown serialised to null, i.e. the rule ran everywhere
while the form still read "From a phase onwards".
supportsPhaseScope / supportsEmitMode now drive visibility; buildScopeFields
nulls hidden-control state so a value from an earlier trigger choice cannot leak
into the payload; validateScopeSelection rejects an unfilled scope mode on both
create and save. Startup is the deliberate exception — a scope there defers the
one-shot instead of suppressing it, so the scope control stays and only emit mode
is hidden. One-shot triggers now state in-form when they collect.
Sub-dropdown hints no longer claim "No selection = all phases" (they turn amber
and name the required action instead).
Tests: app/gather-rules/__tests__/gatherRuleFormLogic.test.ts (15) pins the
applicability matrix, the payload nulling incl. the trigger-switch leak, the
unfilled-scope rejection, and the JSON-mode scopeMode derivation.
vitest 572/572; tsc silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live test found a third recall gap. The fallback only topped up when semantic
search returned FEWER than topK results — but the more common failure is a full
list of unconvincing ones. Querying "WOW6432Node" returned three chunks scoring
~0.29, none containing the word, while the single chunk that does was never
considered at all.
The fix is a ranking change, so it was measured rather than reasoned about. Adds
scripts/eval-docs-search.ts, which scores strategies over 2465 queries in three
deliberately different classes: every chunk heading (easy by construction, so a
regression detector), rare identifiers (the class this fallback serves), and
hand-written questions (the reality anchor — not derived from the corpus, so it
shares little vocabulary with it). Embeddings are computed once and each strategy
is a pure function over the same candidates, so sweeping a threshold costs no
extra inference.
known-item literal handwritten MRR fixed broken
append-only (before) 71.9% 66.2% 1.000 - -
weak < 0.30 71.9% 72.6% 1.000 142 0
weak < 0.35 73.7% 81.1% 1.000 335 0
weak < 0.40 76.3% 88.2% 0.944 502 3
weak < 0.45 77.2% 93.0% 0.889 613 5
0.35 is strictly dominant — better in every class, worse in none — so that is
what ships. Higher thresholds keep repairing identifier queries but start
demoting hand-written questions, the class that most resembles real use, so the
gain stops being free. When the head is weak, exact matches move to the front
while at least one semantic result is kept, so a weak-but-correct hit is never
pushed out entirely.
The measurement also resized the problem: identifier queries were succeeding only
66.2% of the time, so the reported case was one of roughly 750, not an outlier.
That is the part a single anecdote could not have shown, and the reason the
harness exists before the fix rather than after it.
Verified: 447 tests pass, including new cases for the weak path and for leaving a
confident ranking untouched. On the real corpus "WOW6432Node" now returns the
chunk that contains it at rank 1, while "how do I deploy the agent with Intune"
(0.819) and "what firewall URLs do I need to allow" (0.374) are unchanged and the
negative controls stay empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two measured recall gaps in search_docs, both silent.
1. Truncation. MiniLM's context is 256 tokens; 87 of 324 chunks exceeded it and
13% of all tokens never reached the ranking — the tokenizer drops the tail
without complaint. Concretely, in built-in-rules.md :: Apps (734 tokens) the
strings ANALYZE-APP-013, ANALYZE-OFFICE-001, ANALYZE-CORR-003 and Click-to-Run
all sit past the cutoff, so "proxy configuration blocking app downloads" did not
retrieve that section at all.
Shrinking chunks cannot fix this: the corpus ranges 2.43 to 5.81 characters per
token, so no character budget bounds tokens, and a budget safe for the worst case
(~620 chars) would shred prose and tables alike. So the chunk stays whole and the
embedding is split instead — index() cuts long text into 250-token windows with
40 overlap, a document carries embeddings[][], and search() scores it by its BEST
window. Max rather than mean: averaging would blur a long section's several
topics into one mediocre direction, the same dilution that makes large context
windows worse than chunking.
The tokenizer is a build-time dependency only (vocabulary, not ONNX weights,
~93 ms); production boots from precomputed vectors and never loads it.
2. No general fallback. Proper nouns embed poorly — "RealmJoin" occurs in 3
chunks yet scores under the threshold. Fuzzy matching was built, measured and
rejected: at a threshold low enough to surface RealmJoin (0.28), Fuse also
returns 5 hits for "SCEPman", a word this corpus contains zero times. Literal
containment has neither problem, and every provider already exposes it. So a
short result set is now topped up via lexicalMatch on distinctive query terms,
requiring two matching terms as corroboration — without that rule "how do I bake
sourdough bread" matched a page about LLM providers, because "bake" is a
substring of "baked into the image". Keyword hits are appended and flagged, never
sorted together with cosine scores.
Also sharpens the tool description with the measured reason to prefer this over
fetching the docs site (~1k tokens for topK=3 against ~9k raw / ~600k rendered,
version-matched, and matching on meaning), and replaces the growing tail of
positional docs parameters with a DocsSearchBundle.
Verified: token coverage 87% -> 100%; "Office Click-to-Run install never
finished" 0.318 -> 0.442; "proxy configuration blocking app downloads" absent ->
0.559 at rank 1; the ten reference queries keep byte-identical scores and rank 1
(short chunks are one window, so the same vector as before — no long-chunk bias).
The fallback is not a dead path: across 904 rare corpus tokens queried alone it
contributes hits for 431, while negative controls stay empty. Precompute output
is accepted by the boot validator and a mutated corpus still rejected; index
2.63 -> 3.12 MB for 324 chunks / 424 vectors. The rules corpus gains the same
treatment unplanned: 128 rules now carry 172 vectors. 445 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP server could answer telemetry questions ("why did this session fail")
but not product ones ("how do I deploy the agent", "where is my data stored").
Those answers already exist in the autopilotmonitor-docs bundle; search_docs
makes them semantically searchable.
Deliberately a separate index from search_knowledge: rules and prose answer
different questions, and sharing one topK would dilute both. Registered for
every role — published documentation carries no tenant data — and skipped
entirely when no corpus was baked in, so a doc-less build advertises no
broken tool.
Corpus handling:
- Chunked at h2/h3 and <details>/<summary>, each chunk prefixed with a
"Page > Section > Subsection" breadcrumb so a mid-page hit is self-describing.
- Newlines are normalized before any parsing. This is a correctness
precondition, not cosmetics: the bundle ships CRLF and JS `.` does not match
\r, so /^(#{2,3})\s+(.*)$/ matched zero headings — silently, degrading every
page to unlabelled paragraph splits.
- Two size exceptions: tables are never split (a split table leaves `| --- |`
rows with no header), and unbroken changelog lists split at line level (left
whole, their tails fall outside the model window and become unsearchable).
- Short sections are kept — a one-line answer is often the best hit, and the
breadcrumb supplies the context a short body lacks.
Build and delivery:
- Docs are embedded at image build time into an optional third
PrecomputedSection. Optional so a checkout without the docs repo does not
invalidate the rules and event-type vectors; the reverse (server has docs,
file does not) is staleness and is rejected.
- The deploy workflow checks the public docs repo into the build context and
passes its SHA as DOCS_COMMIT, which /health reports alongside the chunk
count — editing the docs does not update a running server, so staleness is
made checkable rather than memorable.
- Embeddings are rounded to 6 decimals on serialize: 55% smaller index (2.63
vs 5.88 MB) for a ~1e-5 norm error against a 1e-3 tolerance. indexPrecomputed
now asserts unit norm, since the hydrate path never checked it.
Infra: the Bicep template was two changes behind the live deployment (EU
cutover and ACR hardening, both applied directly in Azure). Reconciled against
the running resources — managed-identity image pull replaces the admin
credentials and the acr-password secret — and infra/ is no longer gitignored so
it stops drifting. Container raised to 0.5 vCPU / 1.0 GiB for the cold-start
query-embedder load. The template does not model the mcp.autopilotmonitor.com
custom domain, so a full deployment would remove it; the header warns about it.
Verified: 428 tests pass; 45 pages produce 324 chunks; sample queries hit the
right page top-1/2; all 8 derived URLs return 200 live; the precompute output
is accepted by the boot validator and a mutated corpus is rejected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to gather-rule phase scoping. One-shot collection at a phase START
already existed (phase_change dedups per (rule, phase)) and at enrollment END
(on_event + enrollment_complete); the END of a specific phase had no trigger.
New trigger value "phase_exit": fires once when TriggerPhase is LEFT (empty =
every phase transition). Together with phase_change it forms the two one-shot
bookends of a phase.
Agent (GatherRuleExecutor.OnPhaseChanged):
- Exit rules are evaluated BEFORE _currentPhase advances, so both the
TriggerPhase match and the phase-scope gate see the phase being LEFT. A rule
scoped to DeviceSetup therefore fires on its own exit even though the phase
being entered is out of scope.
- A transition into Failed fires the exit rules of the failing phase — the
snapshot at a failure boundary is the point of the trigger.
- Exit of Unknown never fires; a repeated identical phase signal is not a
transition.
- Dedup key "exit:{ruleId}|{phase}" shares _phaseRulesExecuted with the
phase_change keys but stays in a disjoint key space, so an enter-rule and an
exit-rule on the same phase cannot consume each other's slot.
Portal: trigger labels are now "Phase Start" / "Phase End"; the canonical phase
dropdown and the create/save payloads cover both via PHASE_TRIGGERS; the card
renders "entering X" / "leaving X".
No ConfigVersion bump: no new field is transported. Old agents do not know the
value and simply never fire the rule (silent no-op).
Tests: GatherRuleExecutorPhaseExitTests (11) — enter-vs-exit, no refire on
re-entry, empty-triggerPhase, exit-of-Unknown, same-phase-signal, into-Failed,
Complete, scope-against-left-phase, out-of-scope slot preservation, key-space
disjointness, on_change interplay. Build 0 Errors; V2.Core.Tests 1884/1884;
Functions.Tests 3099/3099; vitest 557/557; tsc silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rework after the first real enrollment (session b2e890c1) exposed three
defects in a654c730:
- Shell-Core "RebootCoalescing" match was a false positive by construction:
the token appears in the ROUTINE SubcategoryProcessing_Started bootstrap
marker ("Starting subcategory DeviceSetup.RebootCoalescing...") on EVERY
enrollment. esp_reboot_coalescing removed entirely (branch, constant,
catalog, tests).
- 2800 reboot-required flags fire at every policy sync; only DeviceSetup-time
flags cause the coalesced reboot. The session's flags fired during
AccountSetup - no reboot - yet our Warning events claimed "device will
restart". Raw event is now a neutral Info observation; the reboot CLAIM
moved to ANALYZE-ESP-005 with a REQUIRED system_reboot_detected >= 1
condition (no observed reboot -> advisory silent).
- One event per 2800 record was spam: records are now buffered and debounced
(10 s) into ONE aggregated event per burst (rebootUris[]/uriCount/
firstRebootUri/recordCount/unparsedCount). Watermark is persisted at FLUSH,
not claim - a pre-emit reboot kill loses nothing, the post-restart backfill
re-emits the burst at its historical time.
Verified 2800 description shape recorded in code, tests, and concept doc:
"The following URI has triggered a reboot: (./Device/...)."; extraction
covered by a real-text regression test. Lesson added to tasks/lessons.md
(observation-level events vs outcome-gated claims; substring tokens appear
in framework bookkeeping records).
Tests: Agent.V2.Core 1867/1867, Functions 3099/3099, MCP 400 passed,
solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).
- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
remediation = reassign profiles device -> user groups; +20 confidence on
coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
message texts are unverified until E2E VM validation (tolerant matching).
Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2026-07-18 storage migration reset every row's system Timestamp,
exposing that AuditLogs, OpsEvents and Events displayed, filtered and
retention-swept on it (custom "Timestamp" writes were silent no-ops —
reserved system property). All migrated rows read as 2026-07-18.
- New Helpers/BusinessTimestamp: RowKey decoders (audit !revtick_guid,
ops bare revtick, event yyyyMMddHHmmssfff_seq), RowKey-range OData
clause builders (tick-exact, with the mandatory RowKey lt '"' guard
so no lower bound can ever match legacy bare-GUID audit rows), and
dual DateTimeOffset/DateTime column reads.
- Writes in all three tables now persist OccurredUtc (custom column,
copied verbatim by future migrations); no-op ["Timestamp"] removed.
- Read mappers resolve OccurredUtc -> RowKey decode -> system Timestamp;
GetEarliestSessionEventTimestampAsync uses the same resolution so
StartedAt/Duration reconciliation can no longer absorb the migration
date on migrated sessions.
- Date-window filters and retention moved off the system Timestamp onto
RowKey ranges; ops retention loses its full-table scan and the
post-migration mass-delete cliff; audit retention runs a temporary
second pass for legacy GUID rows (TODO: remove after 2027-03-01).
- New GlobalAdmin backfill endpoint POST /api/maintenance/
backfill-occurred-utc (dry-run by default, batched, idempotent MERGE
of only OccurredUtc, table=audit|ops) — Events rows are deliberately
never mutated; their read path decodes the RowKey instead.
- Tests: decoders, clause boundary proofs incl. GUID-safety pins,
mapper precedence, backfill row decisions (Functions.Tests 3078).
- Docs: docs/backend/business-timestamps.md (system Timestamp is never
authoritative; migration checklist rule).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New rules/ section in the knowledge bundle: semantics table, from-phase latch
behavior (never via Failed, sticky through it), on_change x emitOnlyIfExists
composition, restart re-emit, no-phase-signal caveat, old-agent degradation,
and the prerequisite toggle-wipe fix. Index + log updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-home agents to a new API base URL through the agent/config response
(backend move or per-tenant EU->US region move) - the lesson from the
2026-07-18 WEU->GWC cutover, where the compiled-in API hostname hard-cut
every in-flight agent (no DNS indirection possible: mTLS breaks behind a
proxy, Flex custom-domain TLS is broken).
Backend (ConfigVersion 33):
- AdminConfiguration.AgentMigrateApiBaseUrl (global) +
AgentMigrateTenantOverridesJson (tenantId->URL; empty value pins a
tenant against the global target; override wins)
- GetAgentConfigFunction.ResolveMigrateTarget serves the validated target;
AgentMigrateServed / AgentMigrateRejected warnings as delivery evidence
- TableConfigRepository Store+Map; admin portal Global Settings round-trip
Agent (kill-switch-mirrored hardening):
- Shared AgentEndpointMigrationRules: https-only, *.azurewebsites.net on a
label boundary, no path/port/userinfo - enforced on BOTH sides
- Live fetch only: RemoteConfigService cache strips the field on write AND
read; EndpointMigration.ResolveTarget re-checks the fetch outcome
- AgentRuntimeConfig.Resolve returns before merge/integrity/bootstrap
cleanup on a pending migration (no spurious IntegrityCheckFailed against
a stale migration stub), Program.RunAgent rebuilds Phase 3+4 against the
new URL - single hop, kill signal wins before and after the hop
Tests: AgentEndpointMigrationTests (backend rules + resolution),
EndpointMigrationTests + cache-hardening strip (agent). Full suites green
(Functions 2992, Agent Core 1822, vitest 557, tsc silent).
Doc: docs/agent/endpoint-migration.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build numbers now come from a public counter blob (buildcounter-v2.txt,
reserve-before-build with ETag-CAS) shared by local build.ps1 and CI, so the
two paths can never mint the same number. The CI step that committed
buildcounter.txt bumps to main is gone; the repo file is now a gitignored
offline fallback for -SkipUpload smoke builds only.
CI parity fixes vs the local build:
- ship SummaryDialog.exe.config in the ZIP (WPF AppContextSwitchOverrides;
without it the dialog dies with 0x8007045A when spawned from SYSTEM)
- verify EXE version == manifest == reserved build number before zipping
- update the AdminConfiguration hash-oracle only on publish_as_stable
CI-built release ZIPs now carry a GitHub Artifact Attestation (Sigstore
keyless provenance, actions/attest-build-provenance@v4).
Doc: docs/agent/build-counter-blob.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A flip-flopping user-scoped WHfB CSP read "disabled" once, the arm-C
completion synthesized HelloOutcome="Skipped" and the session completed
5s later while the user sat inside the Hello wizard that had started
230ms earlier (Shell-Core 62404, previously InformationalEvent-only).
Layer 1 — engine (DecisionCore):
- New DecisionSignalKind.HelloWizardStarted + set-once fact
DecisionState.HelloWizardStartedUtc (additive-nullable, census +
snapshot wired).
- Prevention: centralized HelloSatisfiedForCompletion /
HelloPolicyDisabledWithoutWizard predicates veto the policy-disabled
stand-in at all five Skipped-synthesis sites and in the
completion_waiting prerequisites; the AdvisoryCompletion handler gets
a HelloWizardObservedPromote branch (its fallthrough is
re-arm-or-fail).
- Cure: HandleHelloWizardStartedV1 retracts an engine-synthesized
"Skipped" (exact-case discriminator vs the lowercase tracker
vocabulary; first fact-nulling reducer, stage-agnostic because the
RealmJoin gate-defer parks synthetic skips pre-Finalizing), cancels
finalizing_grace, returns to AwaitingHello and arms HelloSafety.
- Agent path: ShellCoreTracker raises HelloWizardStarted before
FinalizingSetupPhaseTriggered; 62404 included in the startup backfill
(fire-once); coordinator timestamp mirror; adapter emit (fire-once).
Layer 2 — HelloTracker confirmation read:
- ApplyPolicyRead state machine: "disabled" commits only after a second
confirming read (~10s poll); "enabled" commits immediately;
disabled→enabled flags flipFlopDetected=true; pending keeps the
policy-unknown wait cadence. Audit fields confirmedReads /
flipFlopDetected on hello_policy_detected.
Fixtures: signal-kinds/hello-wizard-started-v1.json +
enrollment-sessions/userdriven-hello-wizard-unskip-v1.jsonl (772fe502
ordering, ends Completed with HelloOutcome=completed).
Tests: DecisionCore 462/462, V2.Core 1775/1775.
Docs: OKF concept doc docs/agent/hello-wizard-unskip.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopt the Open Knowledge Format (GoogleCloudPlatform/knowledge-catalog)
for contributor/AI-facing technical docs: YAML frontmatter on the three
existing concept documents, new index.md (progressive disclosure, root
carries okf_version) and log.md, plus the maintenance convention in
.claude/CLAUDE.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oliver Kieselbach