- New "phase" stage: OnPhaseChanged writes a transition marker line
(phase change: Old -> New) so rule-level lines read against the current
phase; only on real changes, written outside _scopeLock.
- LogParser traces each regex match: line number, matched text (<=200 chars)
and capture groups (<=100 chars, same selection as event data), capped at
10 lines per file per run; the per-file summary keeps the full count.
- Invalid-regex error line now includes the offending pattern string.
Tests: 4 new (transition dedup, match trace, cap, pattern in error);
agent suite 2137/2137 green. OKF doc + docs/log.md updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer feedback: backfilled modern_deployment_error events (pre-agent
Autopilot event-log errors replayed at agent start) rendered as plain
Error rows with one identical emission timestamp — indistinguishable
from a live failure.
Web (EventTimeline.tsx, effective for already-stored events):
- "⟲ Backfilled" badge (slate, explanatory tooltip) on data.backfilled
- row time now shows the original event-log timeCreated with a
"(reported …)" suffix; metadata block gains a "Recorded" line;
raw mode gets a compact ⟲ marker
- severity deliberately NOT downgraded — unresolved pre-agent errors
must stay visible in the Error filter
Agent (ModernDeploymentTracker):
- backfill scan buffers the capped record set, emits an Info summary
event first (modern_deployment_log, backfillSummary: true +
replayedCount/droppedOverCap/lookbackMinutes) announcing the block,
then replays oldest-first so timeline sequence follows event-log
chronology (cap semantics unchanged: newest 25 win)
No new EventType, no ConfigVersion bump. Tests: summary emission x2,
tracker subset 12/12; tsc + vitest 651/651 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customers cannot tell why a gather rule produces nothing in the timeline
(e.g. a logparser rule with emit_mode on_change that never emits). New
per-tenant boolean EnableGatherRuleDebugLog (EnableImeMatchLog-style
plumbing end to end): when enabled, the agent writes
%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log with one
pipe-delimited line per evaluation decision across all previously silent
"produced nothing" exits — disabled rules, interval without
intervalSeconds, first-tick-after-one-full-interval, phase-scope skips
incl. currentPhase=Unknown, per-(rule, phase) dedup, empty collector
results (emitOnlyIfExists), on_change hash suppression (hash prefix +
streak), guard blocks, logparser per-file outcomes (position skips,
format=text hint when every line fails CMTrace parsing, maxLines and
wildcard caps), and full exception stacks the agent log drops.
- Backend: ConfigVersion 35 -> 36; TenantConfiguration/AgentConfigResponse
field + TableConfigRepository Store+Map
- Agent: GatherRuleDebugLog writer (10 MB cap, one .old generation,
failures swallowed, optional console echo), GatherRuleContext.DebugLog
(null-safe no-op), trace points in executor + collectors; never writes
under _scopeLock (reasons captured inside, written after release);
merger bool->path projection with --gather-debug-log CLI override
tolerance; deliberately NOT in the offline-cache strip lists;
--run-gather-rules always traces; zero-rules delivery leaves a one-line
file instead of nothing
- Side fix: GatherRuleExecutorHost now passes the logger into
InformationalEventPost so gather-event emissions hit the DEBUG log
- Web: tenant settings toggle (Settings -> Agent) + admin tenant-config
report rows
- Tests: merger on/off/override, writer format+rotation, executor debug
lines, logparser debug lines (agent 2131, backend 3561, tsc green)
- Docs: docs/agent/gather-rule-debug-log.md + index/log entries
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.
- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
Sidecar registers later (legitimate startup ordering); DevicePreparation-only
snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
classified healthy); probe/doc headers and OKF doc corrected, web event-type
description updated
Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.
Agent tests 2043/2043, web tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.
New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):
- EspPolicyProviderProbe: fail-soft registry read of the
EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
continuous-incompleteness dwell (resets on completion/disappearance;
absent key = normal early enrollment), one-shot per provider,
StartupEventGate fingerprint over the sorted stalled key set for
cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
purely observational, no decision-engine coupling, no config toggle,
no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
StallProbe idle gate (idle clock resets on session activity) and the
idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
web gather-rule autocomplete (stall category).
Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.
Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).
Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.
Both the detector and the embedded allowlist take effect with the next agent
release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).
Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).
Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).
Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).
- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
(dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
hard-timeout deadline, release the gate with outcome
FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
complete the first deployment) nor by a first observation of 210
(session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list
Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.
The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.
The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.
Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.
Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rework after the first real enrollment (session b2e890c1) exposed three
defects in a654c730:
- Shell-Core "RebootCoalescing" match was a false positive by construction:
the token appears in the ROUTINE SubcategoryProcessing_Started bootstrap
marker ("Starting subcategory DeviceSetup.RebootCoalescing...") on EVERY
enrollment. esp_reboot_coalescing removed entirely (branch, constant,
catalog, tests).
- 2800 reboot-required flags fire at every policy sync; only DeviceSetup-time
flags cause the coalesced reboot. The session's flags fired during
AccountSetup - no reboot - yet our Warning events claimed "device will
restart". Raw event is now a neutral Info observation; the reboot CLAIM
moved to ANALYZE-ESP-005 with a REQUIRED system_reboot_detected >= 1
condition (no observed reboot -> advisory silent).
- One event per 2800 record was spam: records are now buffered and debounced
(10 s) into ONE aggregated event per burst (rebootUris[]/uriCount/
firstRebootUri/recordCount/unparsedCount). Watermark is persisted at FLUSH,
not claim - a pre-emit reboot kill loses nothing, the post-restart backfill
re-emits the burst at its historical time.
Verified 2800 description shape recorded in code, tests, and concept doc:
"The following URI has triggered a reboot: (./Device/...)."; extraction
covered by a real-text regression test. Lesson added to tasks/lessons.md
(observation-level events vs outcome-gated claims; substring tokens appear
in framework bookkeeping records).
Tests: Agent.V2.Core 1867/1867, Functions 3099/3099, MCP 400 passed,
solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).
- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
remediation = reassign profiles device -> user groups; +20 confidence on
coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
message texts are unverified until E2E VM validation (tolerant matching).
Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-home agents to a new API base URL through the agent/config response
(backend move or per-tenant EU->US region move) - the lesson from the
2026-07-18 WEU->GWC cutover, where the compiled-in API hostname hard-cut
every in-flight agent (no DNS indirection possible: mTLS breaks behind a
proxy, Flex custom-domain TLS is broken).
Backend (ConfigVersion 33):
- AdminConfiguration.AgentMigrateApiBaseUrl (global) +
AgentMigrateTenantOverridesJson (tenantId->URL; empty value pins a
tenant against the global target; override wins)
- GetAgentConfigFunction.ResolveMigrateTarget serves the validated target;
AgentMigrateServed / AgentMigrateRejected warnings as delivery evidence
- TableConfigRepository Store+Map; admin portal Global Settings round-trip
Agent (kill-switch-mirrored hardening):
- Shared AgentEndpointMigrationRules: https-only, *.azurewebsites.net on a
label boundary, no path/port/userinfo - enforced on BOTH sides
- Live fetch only: RemoteConfigService cache strips the field on write AND
read; EndpointMigration.ResolveTarget re-checks the fetch outcome
- AgentRuntimeConfig.Resolve returns before merge/integrity/bootstrap
cleanup on a pending migration (no spurious IntegrityCheckFailed against
a stale migration stub), Program.RunAgent rebuilds Phase 3+4 against the
new URL - single hop, kill signal wins before and after the hop
Tests: AgentEndpointMigrationTests (backend rules + resolution),
EndpointMigrationTests + cache-hardening strip (agent). Full suites green
(Functions 2992, Agent Core 1822, vitest 557, tsc silent).
Doc: docs/agent/endpoint-migration.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gh attestation verify only passes with the repo's current name
(okieselbach/AutopilotMonitor); the signed workflow identity carries it,
and the old dashed name fails identity matching even though git-remote
redirects still work. Verified against the first attested dry-run ZIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build numbers now come from a public counter blob (buildcounter-v2.txt,
reserve-before-build with ETag-CAS) shared by local build.ps1 and CI, so the
two paths can never mint the same number. The CI step that committed
buildcounter.txt bumps to main is gone; the repo file is now a gitignored
offline fallback for -SkipUpload smoke builds only.
CI parity fixes vs the local build:
- ship SummaryDialog.exe.config in the ZIP (WPF AppContextSwitchOverrides;
without it the dialog dies with 0x8007045A when spawned from SYSTEM)
- verify EXE version == manifest == reserved build number before zipping
- update the AdminConfiguration hash-oracle only on publish_as_stable
CI-built release ZIPs now carry a GitHub Artifact Attestation (Sigstore
keyless provenance, actions/attest-build-provenance@v4).
Doc: docs/agent/build-counter-blob.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probe enrollment e7dbac7a (24H2) showed Windows writes IsAutoPilotDisabled
(capital P) while the troubleshooting FAQ documents IsAutopilotDisabled —
the missing-event flat evidence lookup was case-sensitive and would have
missed it. Registry value names are case-insensitive by contract, so the
dump dictionary now uses OrdinalIgnoreCase; original casing is preserved
in the emitted payload. Doc notes the casing caveat.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In the normal IME deployment model the agent installs only after MDM
enrollment, so TPM attestation and ZTD errors from earlier (retried)
attempts sat invisible in the event log. The tracker now replays
Autopilot-channel Level 1-2 records from the last 4h at startup through
the regular emission pipeline.
Spam containment (answering "spam or backfilled event?" — individual
events, but marked and bounded):
- errors only (Warning-level 100 heartbeat excluded by XPath)
- harmless-ID downgrade + burst rollup still applies to backfilled records
- cap of 25 records per scan, newest first (tail logged, not emitted)
- RecordId watermark persisted to autopilot-error-backfill.json, advanced
by both backfill and live watcher -> restarts never re-emit
- events carry backfilled=true + original timeCreated; the event's own
timestamp is emission time, so backend StartedAt alignment is not
dragged backwards by pre-enrollment records
Docs: Observability matrix updated (TPM row now visible-via-backfill) +
new "Startup error backfill" section. Tests: 5 new (XPath, watermark
roundtrip/monotonicity, backfilled/live markers, rollup-on-backfill);
full V2 suite 1804 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agent is normally installed by an IME platform script AFTER MDM
enrollment succeeded, so pre-enrollment failures (enrollment blockers,
TPM attestation) are invisible in the normal flow — no enrollment means
no agent and no authenticable session. Await-enrollment mode does not
help either: the binary only polls for the MDM certificate and cannot
authenticate without it. Only bootstrap-token deployments (one pilot
customer) can report pre-MDM.
- New Observability section in docs/agent/autopilot-ztd-diagnostics.md
with a per-rule visibility matrix (normal / bootstrap / await) and the
follow-up option of backfilling Autopilot-channel error records
- ANALYZE-ENRL-003/004 descriptions state the limitation explicitly so
portal users understand why these rules rarely fire
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A flip-flopping user-scoped WHfB CSP read "disabled" once, the arm-C
completion synthesized HelloOutcome="Skipped" and the session completed
5s later while the user sat inside the Hello wizard that had started
230ms earlier (Shell-Core 62404, previously InformationalEvent-only).
Layer 1 — engine (DecisionCore):
- New DecisionSignalKind.HelloWizardStarted + set-once fact
DecisionState.HelloWizardStartedUtc (additive-nullable, census +
snapshot wired).
- Prevention: centralized HelloSatisfiedForCompletion /
HelloPolicyDisabledWithoutWizard predicates veto the policy-disabled
stand-in at all five Skipped-synthesis sites and in the
completion_waiting prerequisites; the AdvisoryCompletion handler gets
a HelloWizardObservedPromote branch (its fallthrough is
re-arm-or-fail).
- Cure: HandleHelloWizardStartedV1 retracts an engine-synthesized
"Skipped" (exact-case discriminator vs the lowercase tracker
vocabulary; first fact-nulling reducer, stage-agnostic because the
RealmJoin gate-defer parks synthetic skips pre-Finalizing), cancels
finalizing_grace, returns to AwaitingHello and arms HelloSafety.
- Agent path: ShellCoreTracker raises HelloWizardStarted before
FinalizingSetupPhaseTriggered; 62404 included in the startup backfill
(fire-once); coordinator timestamp mirror; adapter emit (fire-once).
Layer 2 — HelloTracker confirmation read:
- ApplyPolicyRead state machine: "disabled" commits only after a second
confirming read (~10s poll); "enabled" commits immediately;
disabled→enabled flags flipFlopDetected=true; pending keeps the
policy-unknown wait cadence. Audit fields confirmedReads /
flipFlopDetected on hello_policy_detected.
Fixtures: signal-kinds/hello-wizard-started-v1.json +
enrollment-sessions/userdriven-hello-wizard-unskip-v1.jsonl (772fe502
ordering, ends Completed with HelloOutcome=completed).
Tests: DecisionCore 462/462, V2.Core 1775/1775.
Docs: OKF concept doc docs/agent/hello-wizard-unskip.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub resolves /-prefixed links against the repo root (missing /docs),
so cross-doc navigation broke. Switched all agent docs to standard
relative links and updated the bundle convention in index.md + CLAUDE.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oliver Kieselbach