The 2026-09-01 WG-seal soak on post-SignalIngress-fix builds (memory:
project_wg_seal_postfix_soak_residual) found sessions whose DecisionEngine
never sealed Part 1 while the agent kept running for hours - and telemetry
could not distinguish a frozen ingress worker from a deliberate Weak
sealing-classifier verdict, because the metrics snapshots carried no
pipeline counters at all.
Every agent_metrics_snapshot now carries the block. The fields are MONOTONIC
totals plus a monotonic queue peak - deliberately no momentary queue length:
a snapshot samples the integral, so any two snapshots prove whether the
worker ran in between (a frozen worker keeps signal_pending_count growing
while processed/applied stand still; a momentary gauge would alias between
ticks). Same field pattern the snapshot already uses for the spool
(spool_total_enqueued_count / spool_peak_pending_item_count).
- signal_last_ordinal / signal_processed_total (new ProcessedSignalCount) /
signal_pending_count / signal_reentrant_total: SignalIngress worker
progress; signal_queue_peak is a new lock-free Interlocked-max recorded in
Post (fast + slow path; worker-local follow-up items deliberately excluded)
- decision_stage + decision_last_applied_ordinal: from the immutable
DecisionState - answers the WG question directly ("minutes after the
tracker's whiteglove_complete and still not WhiteGloveSealed")
- wg_sealing_level / wg_sealing_score: only after the sealing classifier
produced at least one verdict (key ABSENCE, not Unknown/0 noise, for every
non-WG session) - separates "classifier said Weak" from "never ran"
Wiring is a read-only probe chain following the imeTrackerHealthProbe
pattern: EnrollmentOrchestrator passes () => _processor?.CurrentState into
IComponentFactory.CreateCollectorHosts (new optional decisionStateProbe;
test fakes updated), PeriodicCollectorLifecycleHost combines it with its
existing concrete-SignalIngress reference in BuildSignalPipelineProbe, and
AgentSelfMetricsCollector writes the fields fail-soft (a throwing probe
degrades to a snapshot without the block - pinned by test). The snapshot
event travels the collector/event path, which is independent of the signal
pipeline it measures - field-proven by the freeze cases, whose perf/metrics
events kept flowing while the engine stood still.
No backend/web/MCP change needed: no new event type, DataJson is free-form.
Tests: 5 new collector facts (full block, absence semantics x2, fail-soft,
no-probe compatibility) + 2 new SignalIngress facts (processed total stable
across drain; queue peak survives the queue draining - the property a
momentary gauge would not have). Agent suite 2812 green, solution builds
with 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three CmTraceTimeSkewRegression fires on 2026-09-01 (one tenant, sessions
e797117b / c06d639d / d7c8032b, agent 2.0.1445, each exactly -60.0 min,
residual 0.0, 85-88 % grid-conformant) all came from the same stale leg: those
sessions are pre-provisioning, their Part 1 ran on 2026-08-20 under agent
2.0.1409 (before per-line anchoring) and left 23-26 IME samples at -60.00,
while the Part 2 leg that completed that morning is clean at -0.16. The
terminal reconcile scans the whole event partition, so the old leg supplied the
majority of the IME samples and fired against a build the device had
self-updated to minutes earlier. Real skew, 12 days old, already fixed - and
>=821 pending pre-provisioning sessions with agent < 1412 are queued to repeat
it.
Event rows carry no agent version, so the split is drawn on ReceivedAt:
ResolveEraStartUtc walks the batch stamps of BOTH sides back from the newest
one and ends the era at the first gap > EraGapHours (2.0) - above any in-leg
upload gap (self-metrics every few minutes, session_stalled at 60 min idle,
maintenance sweep at 2 h), below every pre-provisioning handover. Median, grid
residual, conformity AND the distinct-batch guard now all run on the same
era-filtered sample set; the batch counters used to come from the full scan.
SessionSkewScan carries ReceivedAt per sample and its 20k cap keeps the NEWEST
samples (chunked trim) - keeping the head would have blinded the detector on
exactly the sessions large enough to reach it. The ops event reports
eraStartUtc plus the excluded per-side counts, so an inherited leg stays
visible without being alarmed on.
Tests: 3 new era facts, mutation-proven (EraGapHours = 100000 turns exactly
those three red). The handover fact also asserts that the old leg judged on its
own still fires, so it is the window that silences it, not a moved threshold.
Backend 5334 green, Agent core 2805 green, solution builds with 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shutdown run allowed every logged-in user (explorer.exe owner) dynamically, so a backdoor account created after the startup scan and signed in before enrollment completed was scored 0. The WMI inventory is now read first and a logged-in name that is a local account is rejected for the account and the profile check (reported as logged_in_local_accounts); only Entra/domain identities — the only ones seen in real sessions — remain dynamically allowed for their profile folder.
Built-ins are recognised by RID 500/501/503/504 on the machine SID in addition to their English names: since disabled accounts are inventoried (88dfd331), localized Gast/Invité/Administrateur would otherwise flag every non-English device. kioskUser0 (Assigned Access autologon) joins the built-in list; the profile check skips junctions and $<32 hex> folders (cannot be an account name, pure payload noise). ANALYZE-ID-002 1.3.0 explanation mentions the new field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The path guards, the enumeration's reparse-point skip and the per-file attribute check all judged path strings, while the FileStream feeding the ZIP opened that string later (after earlier files were compressed). Under any recursive source folder a local user can write to, a subdirectory turned into a junction in that window made SYSTEM copy the junction target into the tenant-uploaded package.
PinnedSourceFolder holds each section folder open for the whole section (final path must equal the validated path, 8.3-expanded; FILE_LIST_DIRECTORY without delete share, so NTFS refuses renaming it or any ancestor) and opens each candidate with FILE_FLAG_OPEN_REPARSE_POINT, requiring the handle's final path to be exactly <canonical folder>\<relative path>; the archive is fed from that same handle. Manifest: FOLDER REJECTED (...) / SKIPPED (resolved outside validated folder) / FAILED (open). The race is pinned by a test that swaps the directory from the BeforeSourceFileOpen seam; on the previous code the same test copies the junction target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EffectRunner posts ClassifierVerdictIssued from inside DecisionStepProcessor.ApplyStep,
i.e. on the SignalIngress worker thread. With the bounded channel (256) full, Post's
back-pressure slow path blocked the channel's only consumer as a producer: nothing could
ever drain, every collector Post blocked behind it, and the whole decision/monitoring
pipeline froze silently until the max-lifetime watchdog.
Post now detects a call from the worker thread itself and routes the item to a
worker-local follow-up queue that WorkerLoop drains (iteratively, ahead of the channel)
right after the current item. Same processing path — ordinals, SignalLog, emit,
reduce, ApplyStep — just never blocking. Worker posts are accepted during the Stop
drain too, since the verdict belongs to the step being finished. ReentrantPostCount
exposed for observability/tests.
Tests: full channel + worker post releases a blocked foreign producer and processes the
verdict before the next channel item (fails against the previous code by hanging);
follow-up chain drained in order without recursion; verdict posted during Stop drain is
not lost. Docs: agent/decision-engine.md, agent/overview.md, log.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CmTraceTimeSkewRegression fired 7x in its first soak week; every checked case
was a relaunched agent re-tailing the IME log and uploading one burst holding
most of the session's IME samples with a continuum of ages whose median hit the
15-minute grid by chance. The tripwire now also requires >= 80 % of the IME
deltas (relative to the other side's median) to individually sit within 2 min of
some grid multiple; era-mixing regressions still fire. Fraction is carried in the
ops event message and details.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The timeline showed WiFi segments with an SSID and no signal, and an "Avg WiFi
signal" stat that simply vanished — on a location-gated device that reads as a
product defect rather than the Windows 24H2 gate that caused it.
No agent change was needed: buildSegments already merges every wifi_signal_info
into the WiFi segment covering its timestamp, so wifiDataLimitedReason rides in
with the sample that was already being read.
- NetSegment.dataLimitedReason, filled from the sample.
- Segments that never got a sample of their own (a mid-session reconnect emits
no wifi_signal_info) inherit it: the gate is a device-level setting and cannot
flip during an enrollment. A segment WITH a real reading is never overwritten
— pinned by a test.
- Tooltip: "Signal: unavailable — Location services off on this device" where
the percentage would be.
- Stats row: "Avg WiFi signal: n/a (Location services off)" instead of dropping
the row.
network_state_change is deliberately left alone. That event has never carried
signalPercent or radioType — NetworkChangeDetector collects them into its
snapshot for change detection, but only before_/after_wifiSsid reaches the
payload — so there is no absent field there for a reason to explain, and the
SSID it does carry is unaffected by the gate. Adding it would ship a field
nothing reads.
Verified: web tsc + eslint (0 warnings) + vitest 893/893.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A missing WiFi signal reading is indistinguishable from a bug unless the payload
says why, and the device-level location setting is not something an admin would
think to check. Since 058248d9 the SSID survives the gate via the WinRT tier —
but signal, radio and channel silently vanish with no explanation anywhere.
When the WinRT tier wins AND the native tier reported rc=5, wifi_signal_info now
carries wifiDataLimitedReason = "location_services_off" and the event message
gains "— signal unavailable, Location services off", so the reason reads straight
off the timeline. The portal's WiFi card matches that literal and names the
Intune setting that restores the full payload.
The claim is evidence-only: any other native failure (no WLAN service, no
connected interface) leaves the field null rather than blaming a privacy setting
we did not observe. IsLocationGateDenied is the single predicate, pinned at its
boundaries (rc=5 yes; rc=50, rc=5023, rc=1062 no), and the literal is pinned as
the published agent/portal/MCP contract.
New wifi_signal_info group in the MCP DEVICE_PROPERTIES_CATALOG, so a fleet can
be sized before anyone changes a policy:
search_sessions(deviceProperties: {
"wifi_signal_info.wifiDataLimitedReason": "location_services_off" })
Backend is untouched — the event payload is opaque DataJson with no projection.
network_state_change and the network timeline deliberately do not carry the
reason yet; that is a second payload plus web change for no extra insight.
Verified: build 0 warnings, agent suite 2553/2553, web tsc + eslint (0 warnings)
+ vitest 889/889, MCP tsc.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wifi_signal_info was missing on most Windows 11 24H2+ WiFi enrollments
(build 22631 3/3, 26100 37/178, 26200 43/95 over n=276 sessions since 08-01).
Not a parser or language problem: every event that exists carries a full SSID,
so the failure is binary at the reader level.
Since 24H2 Windows gates WlanQueryInterface with
wlan_intf_opcode_current_connection — the only opcode we use — behind
precise-location consent and returns ERROR_ACCESS_DENIED. The consent prompt
only fires for a process in the user's context outside System32, so the agent
(SYSTEM, session 0, before the OOBE privacy page) is denied silently, and it
resolves consent from the HKLM ConsentStore because the S-1-5-18 hive has none
— while the interactive user on the same machine reads the SSID fine via the
per-user NonPackaged value. That is why it tracks the image's location setting,
not hardware, driver or language.
netsh wlan hangs off the same gate, so the 2026-07 netsh fallback and its
multi-language labels never covered the case they were built for; more labels
cannot fix it.
- New WinRtWifiSsidReader: Microsoft's documented replacement
WlanConnectionProfileDetails.GetConnectedSsid (ungated, SSID only), bound via
the CLR WinRT projection — no winmd, no NuGet, no csproj change — the same
mechanism OobeStateReader already uses from the SYSTEM context.
- New WifiInfoProvider owns the tier order (native -> WinRT -> netsh) that both
collectors had duplicated.
- The step diagnostics added in 2026-07 were logged at Debug while the default
level is Info, so they never reached a field log. The all-tiers-failed case is
now a Warning that decodes rc=5 into the location-gate explanation.
Signal, PHY and channel have no ungated source and stay null on tier 2;
deriving a percentage from GetSignalBars would fabricate precision.
Verified: build 0 warnings, agent suite 2542/2542, interop struct sizes measured
live (532/604), and the full WinRT chain resolves and returns the SSID under
net48.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IME never writes its "Completed user session" line when the user has no
assigned Win32 apps - its zero-app branch returns before that statement
(ApplicationPoller.cs:713-721, identical in 1.80/1.83/1.97/1.104). So
ImeUserSessionCompletedUtc stayed null, every arm of
ShouldTransitionToAwaitingHello was unsatisfiable, and the 30-min
advisory_completion backstop resolved healthy enrollments as
esp_exit_without_completion_evidence. One tenant: 10 of 30 sessions in a week,
all false positives - DAD-validated desktop, Hello provisioned, RealmJoin still
installing. Every Succeeded session there carries ime_user_session_completed;
not one of the failed ones does.
IME-USER-SESSION-ZERO-APPS reads the line IME does write there and routes it
through the same OnUserSessionCompleted callback. That is IME's own verdict
rather than a weakened gate: the zero-app branch locks the ESP in with an empty
application list, and the sibling emit site returns ProvisioningComplete right
after logging it. Guarded to the AccountSetup phase so it cannot burn the
adapter fire-once flag on a pre-sign-in timestamp, and replayed on the phase
transition when the observation predates the phase marker. AreUserEspAppsSettled
accepts an empty app list only with that explicit observation, never on
emptiness alone.
Second, independent defect: advisory_completion (30 min) overran the RealmJoin
completion gate, which deliberately holds the session and carries its own
bounded resolution (60 min from detection, activity-extended, 4 h cap). One
session installed 25 RealmJoin packages inside the window, the last 2 min before
the failure, while completion_waiting still read "waiting on:
realmjoin_resolution". The handler now re-arms while that gate is closed and its
own deadline is still armed; convergence comes from the gate itself, which
always ends with an Outcome set.
ime_user_session_completed gains an evidence field (zero_user_apps_enumerated /
user_session_completed) - no new event type, so the Constants ->
resource-catalog -> shared-manifests triple stays untouched. The pattern ships
via the config API and reaches existing agents, where an unknown action is a
single Debug line; the behaviour needs the agent release.
Build 0 warnings. DecisionCore 558/558, V2.Core 2530/2530, Backend 4634/4634,
MCP event-types-drift 6/6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The diagnostics package's built-in collection is now ONE catalog in Shared
(DiagnosticsBuiltInSections.All): the agent iterates it when building the ZIP and
the backend serves the same list to the portal (GET /api/diagnostics/paths,
MemberRead), so what administrators see is exactly what the agent collects.
Agent
- New conditional sections: RealmJoin client/package/Chocolatey/per-user logs
behind the tenant's RealmJoin Watcher toggle (mirrored onto
AgentConfiguration.EnableRealmJoinWatcher by RemoteConfigMerger) and the Device
Preparation bootstrapper event log behind the deterministic WDP marker (probe
once per build, fail toward Classic). ZIP layout mirrors the disk
(RealmJoinLogs/Windows|Choco|User). Every skip is a package-manifest line.
- Built-ins bypass DiagnosticsPathGuards by design (reviewed code); caps,
reparse-point skip and evtx export still apply. No ConfigVersion bump.
Backend
- GET diagnostics/paths (MemberRead, JWT-scoped) with the catalog + global paths.
- MergeDiagnosticsLogPaths: global ∪ tenant dedup (trimmed, case-insensitive) —
duplicates used to produce duplicate ZIP entry names.
Web
- Collapsed read-only built-in list (RealmJoin on/off in the tenant header,
neutral in the Global-Admin view), one-line DiagnosticsPathRow for every entry,
add-row above the lists; tenant context no longer fetches /api/global/config.
Tests: agent RealmJoin on/off, no user session, bootstrapper evtx, probe failure,
merger mirror; backend catalog invariants, payload shape, dedup, policy; web
builtInSectionDisplay. Docs: docs/agent/diagnostics-package.md (+index/log).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two enrollment blind spots become timeline ground truth from the System event
log via one EventLogWatcher (WindowsUpdateTracker pattern, 24h backfill with
RecordId watermark recovering pre-agent records):
- system_clock_changed (Kernel-General 1): oldTime/newTime/signed timeDeltaMs,
reason and the setting process. 2s floor suppresses w32time 1ms micro-slews;
steps >= 5min are Warning + immediate upload. Payload instants are the
authoritative source (event timestamps can be clamped server-side).
- system_sleep_episode (Power-Troubleshooter 1 classic sleep/hibernate,
Kernel-Power 507 Modern Standby; 506 deliberately unconsumed): kind,
enteredAt/exitedAt, durationSeconds, wake source / AC state. 60s real-sleep
floor filters Modern Standby screen-off scenarios.
ConfigVersion 39 (Collectors.EnableSystemTimelineWatcher +
SystemEventBackfillLookbackMinutes). Both types are non-activity on agent and
backend (idle/stall clocks unaffected; a sleeping device stays Stalled).
Consumers: TimeJumpBadge gains the ground-truth cause "clock-set"
(classifyTimeJump matches backward display steps against recorded clock sets);
EventTimeline badge blocks for both events plus the previously bare
power_state_change; SessionInfoCard keeps the wall-clock duration and adds a
standby note; TimeAttributionCalculator builds SleepSpans (RebootSpan clone,
AttributionVersion 3) persisted as SleepSeconds/SleepSpansJson and rendered in
TimeAttributionLane; ANALYZE-DEV-012 (info, single) fires once per session for
episodes >= 5min - deliberately the only new rule, clock findings stay with
DEV-008 to avoid duplicate-finding flood.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On Windows 11 24H2/25H2 (observed on builds 26200 and 26220, session
161b838c) the MS_Power_Management_Event_Provider is still registered in the
WMI repository but activating the extrinsic push query fails with
WBEM_E_NOT_FOUND - the legacy provider no longer loads on exactly the OS
builds new devices enroll with, so the power watcher degraded fleet-wide.
Arm chain: try the push event first (still alive on older builds), then
fall back to the intrinsic __InstanceModificationEvent WITHIN 30 on
Win32_Battery. The WITHIN sampling runs inside WinMgmt, not in the agent;
the handler path is unchanged (payload ignored, re-probe + diff), worst-case
detection latency ~35s incl. debounce. collector_degraded now fires only
when BOTH attempts fail; the arm log line names the active mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Event-driven AC/battery observation during enrollment (no polling): a WMI
Win32_PowerManagementEvent (EventCode 10) push subscription re-probes
GetSystemPowerStatus and diffs snapshots through a pure tracker into
power_state_change events - transition ac_to_battery (Warning, immediate) /
battery_to_ac (Info) / threshold_crossed at 50/30/15% (per-level lifetime
latch, lowest-crossed-only on jumps, arm-time baseline covers sessions
starting low, 5s trailing-edge debounce, emission cap 20). Never armed on
devices without a battery; arm failure surfaces as collector_degraded.
Kill-switch Collectors.EnablePowerStateWatcher (ConfigVersion 38).
power_state_change is classified non-activity on both sides
(SignalActivityClassifier + IsPeriodicOrStallEvent) so battery drain can
neither keep idle collectors alive nor heal a Stalled session.
New analyze rules, both with on_event:power_state_change interim triggers
and deliberately single-required-condition (event_data conditions scan
events independently - no same-instance join):
- ANALYZE-DEV-009 (high): thresholdPercent equals 15
- ANALYZE-DEV-010 (warning): transition equals ac_to_battery
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 60-min RealmJoinTimeout deadline arms at RJ detection (the RJ agent MSI
install during DeviceSetup) and used to cut off monitoring mid-install: report
55e6afd61c9d (session 46a96078) lost the tail of a large package catalog whose
last observed package completed 3 seconds before the deadline, with the first
deployment only starting 16 minutes into the window.
When the deadline now fires with LastDeploymentPhase 100/101 and deployment
activity (phase change or package observation, persisted restart-safe in
RealmJoinFacts.LastActivityUtc; the detection-time phase capture deliberately
does not count) inside the last 60 min, the handler re-arms to
lastActivity + 60 min instead of timing out — Info event
realmjoin_timeout_extended — clamped at an absolute cap of detection + 4 h.
realmjoin_timeout carries a reason payload (hard_timeout / inactivity /
absolute_cap); a fire of a deadline incarnation superseded by a re-arm
dead-ends via the OccurredAtUtc == DueAtUtc scheduler contract. The idle case
(no activity ever) still times out at 60 min, and phase-200/210 activity never
extends. Packages are only registry-visible at completion, so the inactivity
window keeps the full original 60 min.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operator side: CmTraceSkewTripwire piggybacks on the terminal counter
reconcile's partition scan (zero extra reads) and compares median
ReceivedAt-OccurredUtc deltas of IME-derived vs other events. A clean
15-min-grid divergence (residual < 2 min, >=20 samples / >=3 distinct
batches per side) emits the CmTraceTimeSkewRegression ops event
(Agent/Warning, goal: never fires). Bias-dominated sessions are
suppressed; measuredWriterOffsetMinutes is never consulted (sticky
after era flip-backs). Kill switch: CmTraceSkewTripwireDisabled.
Customer side: new clock_skew analyze condition source (skewMetric
clock_jump | sustained_offset, threshold seconds, gt/gte magnitude)
behind built-in ANALYZE-DEV-008 (300 s = the Kerberos/Entra 5-min skew
boundary). Batch-median frames with spool-spread filter, plateau step
detection (a gradual backlog-drain ramp cannot fake a jump) and
end-state persistence; IME-derived events are excluded so a CMTrace
anchoring regression can never fire the customer rule. Evidence carries
field=skewSummary/value for {{skewSummary}} interpolation.
Full mirroring chain: schema enum + skewMetric property, shared
manifest + combine.js regeneration, MCP authoring guide + semantic
lint, web SOURCES/form/card, ops-event dual-register, docs. P5 is
subsumed by the tripwire. PercentileMath is now the single nearest-rank
implementation (Apps/PlatformMetrics delegate).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why the reader-zone assumption and the per-file offset application both
failed, the per-line self-anchoring rule with its strict freshness
contract (LastCheckedUtc on every poll look, 30 s window, no freshness
across restarts), the provenance vocabulary that keeps every event
re-derivable, and the documented limits: the bounded grid-replay edge,
snapshot readers staying on the marked fallback, and the still-open
e9753578 label/anchor crossing with its tripwire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The five stuck sits-d Cloud PCs carried SkipUserStatusPage=True — verified
across three agent processes (10+ registry reads, True every time) and six
hours of live Shell-Core watching with zero 62407: Windows never rendered a
user ESP page, so no exit and no AccountSetup provisioning registry can ever
exist on this flow. Arm B of ShouldTransitionToAwaitingHello was open from
the bootstrap EspConfigDetected onward, but every evaluation site hangs off
a signal the flow cannot produce, and the AdvisoryCompletion backstop arms
exclusively from an EspExiting signal — an open gate nobody knocks on, six
hours to the max-lifetime watchdog. The earlier lost-exit-through-reboot
diagnosis is retired by the same evidence.
HandleImeUserSessionCompletedV1's completion attempt now accepts the
observed skip in place of IsPostAccountSetupFinalExit (anchor + genuine IME
user-session completion + real-user desktop stay mandatory; Device
Preparation excluded). The knock deliberately lives on the IME edge, not on
desktop arrival: Fix 10's AccountSetup bounce-back would undo a desktop-side
promotion two signals later, while the IME phase line always precedes the
user-session-complete line in log order, so a bounced promotion is
re-knocked by the very next re-emission — restart re-parse included.
Docs and comments no longer claim an ESP-exit recovery that does not exist
(stray duplicate summary in ShellCoreTracker, ImeLogHost restore-callback
comment); the OKF doc is reframed on the evidence and names the residual
SkipUser=false exit-in-downtime gap honestly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Round-four review, confirmed — and the cause sits a layer below where I had been
fixing. The IsBackfill return added last round only guarded the coordinator's own
completion logic. Upstream of it, ShellCoreTracker still raised
FinalizingSetupPhaseTriggered, the coordinator still forwarded EspExited to the
adapter before that return, and the adapter discards the args entirely
(OnEspExited(...) => EmitEspExiting()). So a replayed exit still arrived at the
reducer as an ordinary, unmarked EspExiting.
There it is decisive: HandleEspExitingV1 passes espFinalExitInFlight: true for
every arriving exit, and IsPostAccountSetupFinalExit orders exits by ingest
ordinal rather than timestamp — so a historic exit replayed today reads as
post-AccountSetup by construction. With state restored from the snapshot
(AccountSetup entered, a genuine IME user session, desktop arrived) arm C of
ShouldTransitionToAwaitingHello opens on the Device→Account handoff. The reducer
is itself a completion gate and cannot tell the two apart, so the fix has to be
upstream.
The replay is therefore narrowed to what is provably safe: the Hello-wizard start
(62404). That fact is conservative — it vetoes a premature "Hello is disabled"
skip and can never by itself complete a session, so replaying it can only make the
agent wait longer. It is also the observation session 772fe502 wrote the replay
for. A 62407 is no longer raised at all, neither exit nor failure: the exit
because Windows writes the identical description for the intermediate and the
final one and nothing downstream can order it, the failure because re-injecting it
as fresh can fail a session that recovered on retry (ANALYZE-ESP-006). Skipped
records are reported once as an agent_trace with counts and bounds — silently
dropping evidence is how this class of bug survives.
The cross-layer proof is now a pair, as the review asked:
ClassicEspExitingOnRestoredStateTests pins that the reducer opens on any arriving
exit (including one with an hour-old source timestamp), and
ShellCoreTrackerReplayScopeTests pins that the replay never produces one.
Dead machinery removed with it: EspExitedEventArgs.IsBackfill, allowEspExit, the
newest-exit selection, the IsBackfill return. The method is now called
BackfillRecentHelloWizardStart.
One pre-existing integration test drove EspExited through HandleBackfillRecord
because the replay is easier to call than the event-log watcher, and its class doc
claimed that was the production path — it never was, the replay had no caller when
that test was written. Switched to the live path and the doc corrected.
Agent V2.Core 2361, DecisionCore 538, Backend 4287, all green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round-three review, confirmed. The previous round stopped a replayed exit from
arming the DEFERRED re-check but left the immediate synthesis attempt right below
it untouched. My justification for that — "pre-existing behaviour" — was simply
wrong: the immediate attempt on a replayed exit never existed, because the
backfill had no caller. I introduced it, and by moving the replay behind the IME
state restore I made the dangerous state reachable by design: if the newest 62407
in the window is only the intermediate DeviceSetup→AccountSetup exit while the
restored app states are already terminal, that attempt opens the strong
AccountSetup gate before the final exit has even happened. A test was pinning the
faulty assumption as expected behaviour.
A replayed exit now returns before touching completion at all — no deferred edge,
no immediate attempt. Nothing available to it can establish its position in time:
Shell-Core writes the identical description for both exits, and every probe that
could order it reads state as it is now rather than at the event's time. The
replay keeps the job it was written for in session 772fe502 — feeding EspExiting,
FinalizingSetup and the Hello wizard to the reducer for the window in which no
agent process was observing. Completion is the exclusive business of exits this
agent watched live.
With the synthesis attempt gone, the replay no longer needs restored app states,
so it moves back into EspAndHelloHost.Start(); that also keeps this host's startup
independent of the IME host's. ImeLogHost.onStateRestored goes back to doing just
the re-check, which is still needed for a live exit observed before the restore
finishes.
The test asserting the faulty assumption is inverted, plus coverage that the
replay is not neutered (EspExiting still forwards) and that a live exit following
a replayed one completes normally.
Agent V2.Core 2364, DecisionCore 533, Backend 4287, all green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round-two review, both findings confirmed against the code.
A replayed exit was being confirmed as post-AccountSetup by reading the registry
at replay time. That read reflects NOW, not the event's time, so an agent that was
down across the DeviceSetup→AccountSetup transition would confirm the stale
intermediate exit and, once the apps settled, synthesise a completion while the
AccountSetup page was still up. Windows offers no way to tell the two apart from
the event itself — Shell-Core writes the identical description
CommercialOOBE_ESPProgress_Page_Exiting for both. So the deferred edge is now
armed by LIVE exits only (EspExitedEventArgs.IsBackfill): only for those is the
continuous-observation argument valid. A replayed exit keeps the single immediate
synthesis attempt it always had.
For that attempt to be worth anything it must see real data, so the replay moved
out of EspAndHelloHost.Start() into ReplayEspExitBackfill(), invoked from
ImeLogHost.onStateRestored — after LoadState() has restored the package states.
That is what makes "the agent came back long after the ESP finished" recoverable.
The five-minute window for non-reboot restarts was unsound: the scheduled task
carries a BootTrigger only, with no restart-on-failure, so after an
exception_crash the agent does not return until the next boot — hours later, long
after Windows wrote the final exit. The window is now measured rather than
guessed: back to the last moment the previous run is known to have been alive,
taking the wider of the snapshot mtime (rewritten every decision step, available
for every exit type) and LastBootUtc. The five-minute default is the floor,
ClampLookbackMinutes the ceiling, and a future timestamp from clock skew is
ignored instead of producing a negative window.
Also switched the backfill tests to the real Shell-Core wording — the invented
string matched the regex but hid the fact that both exits look identical.
9 new tests. Agent V2.Core 2362, DecisionCore 533, Backend 4287, all green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the review hardening: the replay ran on EVERY agent start with a
5-minute window, first run included. On a first run there is no gap to recover —
no earlier agent process existed — and a 62407 already sitting in the Shell-Core
log belongs to a phase transition this agent was never meant to observe. Replaying
it pushes an EspExiting into the reducer that the pre-fix agent never saw, i.e. a
happy-path behaviour change for no gain.
ResolveEspExitBackfillLookbackMinutes now keys the window to the previous exit
type: first_run (or unknown) disables the replay entirely, clean/hard_kill/
exception_crash keep the 5-minute default because the process is back within
seconds, and reboot_kill widens to "since LastBootUtc" because that gap spans the
forced restart until the post-reboot logon. The result is still clamped to
BackfillLookbackMaxMinutes, so policy and backstop together cannot build an
unbounded event-log query.
With this the happy path is byte-identical to pre-fix behaviour: on a first run
nothing replays, no edge is remembered, and the deferred re-check returns on a
volatile bool read.
No other backfill is affected — the Hello, MDM-reboot, Windows-Update and
ModernDeployment trackers each own a private backfill invoked from their own
Start(), none of which this change touches.
13 policy tests including clock skew across the reboot, a missing boot timestamp
and the clamp backstop. Agent V2.Core 2353, DecisionCore 533, Backend 4287.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three real regressions, all confirmed against the code before touching it — two of
them introduced by the recovery itself.
The remembered exit edge was unqualified. Shell-Core raises 62407 at every ESP
phase transition, so OnEspExited also latched the DeviceSetup→AccountSetup one;
the deferred re-check would then open the strong AccountSetup gate as soon as the
last user app settled, while the AccountSetup page was still up and its other
subcategories still running. IsConfirmedPostAccountSetupExit() now demands
positive evidence (AccountSetup activity observed, or SkipUser == true) and is
deliberately stricter than the existing IsIntermediateDeviceEspExit() forward
guard: an unknown SkipUser counts as not confirmed. Erring strict costs the
pre-fix stall; erring loose costs a premature Succeeded. The immediate on-edge
synthesis keeps its previous behaviour verbatim.
The replay picked the oldest exit, not the final one. The reader walks
oldest-first and the exit branch is fire-once, so the widened downtime window —
which routinely holds both the intermediate and the final exit — handed over the
intermediate one and the fire-once guard swallowed the exit that matters.
ReplayBackfillRecords buffers the batch and lets only the newest exit-matching
record through; ESP failures and the Hello-wizard rail keep replaying
chronologically.
The replay ran before the IME state restore. ImeLogTracker.Start() restores the
persisted package states via LoadState() without raising OnAppStateChanged, and
ImeLogHost starts after EspAndHelloHost, so the replay evaluated an empty app
list — leaving the sharpest case unfixed: apps already terminal before the
reboot, only the final exit lost to the downtime. ImeLogHost gained an
onStateRestored callback wired to the re-check, and the host order is now pinned
by DefaultComponentFactoryOrderingTests so a reshuffle fails loudly.
11 new tests. Agent V2.Core 2340, DecisionCore 533, Backend 4287, all green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven sits-d Cloud PCs enrolled the same day with identical configuration
(~138 required apps at the user-ESP gate, DeviceGuard/DmaGuard device-assigned).
The two that finished AccountSetup before the coalesced reboot landed succeeded;
the five the reboot killed in Stage=EspAccountSetup never derived a completion
again and ended Incomplete on healthy devices. Two defects behind that:
ShellCoreTracker.BackfillRecentEspExitEvents() has existed since session
772fe502 but no caller ever invoked it, so the Shell-Core watcher only saw
records written after Start(). Every agent restart silently dropped an ESP exit
(62407) or Hello-wizard start (62404) from its downtime. EspAndHelloHost.Start()
now calls it, and the fixed 5-minute window is sized to the actual downtime when
the previous run died to a reboot (LastBootUtc, clamped to 360 min).
The user-apps-settled synthesis ran only on the EspExited edge. When the ESP page
exits with apps still in flight the single attempt misses and nothing re-checks —
8110e262 and a89aac2d both reached 138/138 apps, 0 failed, and still never
completed. The exit edge is now recorded and the synthesis re-evaluated from
ImeLogTracker.OnAppStateChanged. No new completion path: the gate conditions are
unchanged, this only gives the existing conservative check a second look. The
fire-once claim moves to Interlocked (edge and re-check run on different threads)
and the re-check does not re-emit app_install_starved.
Rules: ANALYZE-ESP-005 gains on_event:session_stalled so a hung session produces
a finding at the 60-minute stall instead of staying finding-less for hours
(verified against 8110e262 — fired, confidence 95). New ANALYZE-APP-016 turns the
already-emitted app_install_starved events into a customer finding and, unlike
ANALYZE-APP-015, fires even on a succeeded session: 3b9291aa was recorded
Succeeded with 99 required apps that never started installing.
ANALYZE-APP-015 and ANALYZE-ESP-001 were considered for the stall trigger and
deliberately left alone — APP-015 is the documented worked example of the
terminal-precondition trap, and ESP-001's phase_duration gate on a still-open
phase is unverified mid-run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
afee7ae0 audit: WDP sessions ran the full Classic/ESP apparatus. Gates key on
the deterministic BootstrapperAgent ExecutionContext marker only (CloudAssigned*
fallbacks keep Classic behavior).
Paket A (event hygiene): no autopilot_profile/-missing + ZTD query/probe on WDP;
esp_config_detected null-gate (no more unknown/unknown); starved sweep skipped
on WDP (data stays in app_tracking_summary/final-status).
Paket B (resources): EspPolicyProviderStallHost + ProvisioningStatusTracker not
started on WDP; ModernDeployment noise IDs 100/417/418/1005 fully suppressed
(Critical always passes) + Autopilot error backfill skipped; StallProbe skips
ESP category scan and treats noise IDs as harmless.
Paket C (DecisionCore): enrollmentTypeDeterministic payload key seeds
DevicePreparation at High confidence (fallbacks stay Medium); completion_waiting
drops the unsatisfiable account_setup_provisioning_complete gate on WDP; arm D
promotes on real-user desktop; new device_prep_completion 30-min backstop closes
the park-until-watchdog hole; parked tripwire gets a WDP branch.
17 new tests; suites green (532/2307/4287). OKF doc agent/wdp-scenario-gates.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agent log directory mixed three conventions (agent_20260817.log,
bootstrap_agent.log, msi-bootstrap.log) while everything else in the
diagnostics ZIP is already kebab-case. All log files now follow
<component>-<content>.log:
agent.log (was agent_YYYYMMDD.log; segments agent-NNN.log)
bootstrap-script.log (was bootstrap_agent.log; script v2.4)
bootstrap-msi.log (was msi-bootstrap.log)
crash-<utc>.log (was crash_<utc>.log)
gather-rules-debug.log (was gather_rules_debug.log)
ime-pattern-matches.log (was ime_pattern_matches.log)
The date is dropped from the main log: the agent only lives for one
enrollment, and date rotation split the chronology at local midnight -
which can fall mid-enrollment while the clock is still on the OOBE
default timezone. The existing 50MB size rotation stays as the growth
bound. Legacy dated files are ignored by the rotation probe; crash
detection still picks up legacy crash_*.log once so an in-place
self-update cannot lose a pending crash classification.
Diag collectors glob *.log and need no change. MCP zip-map and docs
note the legacy names for archives from pre-rename agents. Settings UI
hint paths updated. MSI rebuild + bootstrap script publish (v2.4) +
MCP redeploy required to ship.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
P2 of the 2026-08-17 IME code audit. IME log wording is an unstable
contract (audit found 9 silently dead patterns, reworded lines, quote
drift, [DO TEL] removal); the registry is IME's internal product
contract - the ESP page and Intune reporting consume it.
ImeRegistryAppStateObserver + ImeRegistryAppStateHost (always-on
observability, no DecisionEngine coupling):
- one recursive RegistryWatcher on HKLM\...\IntuneManagementExtension
covers Win32Apps (EnforcementStateMessage JSON, ExitCode, Intent),
EspTrackingWin32Apps and SideCarPolicies\StatusServiceReports;
2 s debounced snapshot-and-diff + 60 s periodic fallback tick
(poll-only degraded mode if the watcher fails to arm)
- silent startup baseline: Win32Apps keys survive re-enrollments,
pre-existing state never replays as fresh events
- registry_app_state (Info, state-change-only, 200/session cap with
one-shot cap notice)
- app_state_reconciliation (Warning, once per app, immediate upload):
a changed entry terminal for >= 90 s is compared against the
log-derived AppPackageState - divergence is the built-in
log-pattern drift alarm (registry = truth, logs = narrative)
Event types dual-registered (Constants + MCP resource-catalog + shared
manifests regenerated). OKF concept doc agent/registry-app-state-pillar.md
incl. WDP Paket-3 reuse notes. Tests: pure parse/diff/divergence logic +
tick flow via snapshot override (baseline silence, emission, dwell,
fire-once, cap). All suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On Windows 365 Cloud PCs a missing Autopilot profile (ProfileAvailable=0)
is the expected state - Cloud PCs are provisioned by the Windows 365
service, not by Autopilot - so the warning shown for physical devices is
misleading there.
- Agent: emit autopilot_profile_missing as Info with a Cloud-PC-specific
message when CloudPcDetector matches; payload now carries isCloudPc.
Evidence collection (ZTD verdict, registry, endpoint probe) unchanged.
- Web: DeviceDetailsCard renders a neutral info note instead of the amber
warning for Cloud PC sessions (session.isCloudPc, sticky, covers
existing sessions; or the new event marker).
- MCP: resource-catalog documents the new isCloudPc data field.
- Docs: Cloud PC exception noted in agent/autopilot-ztd-diagnostics.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Classic UD fleets held at the Device-ESP timeout wall by one slow blocking
app showed ~100% Failed (tenant c9787ba2: 948/1000 esp_terminal_failure)
although the profile allows "Continue anyway" and users demonstrably reach
the desktop — the classic advisory defang requires AccountSetupEnteredUtc
and never applied there.
New operator-set tenant opt-in EnableEspContinueAnywayObservation
(ConfigVersion 36 -> 37, default off):
- DecisionCore: a Device-phase EspTerminalFailure with ContinueAnyway
allowed + opt-in observed (EspConfigDetected payload fact — kernel stays
config-free) + Classic/Unknown non-pre-prov scenario defangs into the
advisory rail with a 60-min AdvisoryCompletion window. Real end via the
DAD-validated real-user desktop + Hello gate (IME user-session gate is
structurally unavailable without the AccountSetup anchor): eager
completion on DesktopArrived, category recovery via the 4910a5a5 hook,
expiry un-defangs to the original esp_terminal_failure — every observed
session still gets a terminal verdict.
- Honest outcome: enrollment_complete under an unresolved advisory carries
espSoftFailure/completionSource (also on the classic post-AccountSetup
advisory path); backend stamps EspSoftFailure/CompletionSource on the
Sessions row (+ index mirror, terminal-tuple preservation); web renders
an amber "with issues" badge plus a session-detail banner. No new
terminal status — Succeeded + marker.
- Agent: remote-config field merged into AgentConfiguration; stamped into
both esp_config_detected emitters (orchestrator bootstrap via live
Func<bool> seam, DeviceInfoCollector via host/factory pass-through).
- Admin UI: GA-only toggle in the tenant management modal (next to
Bootstrap Token) + stored/runtime rows in the tenant config report;
GA-gated on both write paths (PUT silently reverts for tenant admins,
field patch rejects with 400).
Tests: 13 new ContinueAnywayObservationTests; DecisionCore 513, Agent V2
2162, Functions 3948, SummaryDialog 44, vitest 632, tsc clean.
Docs: docs/agent/continue-anyway-observation.md (OKF) + index/log.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New "phase" stage: OnPhaseChanged writes a transition marker line
(phase change: Old -> New) so rule-level lines read against the current
phase; only on real changes, written outside _scopeLock.
- LogParser traces each regex match: line number, matched text (<=200 chars)
and capture groups (<=100 chars, same selection as event data), capped at
10 lines per file per run; the per-file summary keeps the full count.
- Invalid-regex error line now includes the offending pattern string.
Tests: 4 new (transition dedup, match trace, cap, pattern in error);
agent suite 2137/2137 green. OKF doc + docs/log.md updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer feedback: backfilled modern_deployment_error events (pre-agent
Autopilot event-log errors replayed at agent start) rendered as plain
Error rows with one identical emission timestamp — indistinguishable
from a live failure.
Web (EventTimeline.tsx, effective for already-stored events):
- "⟲ Backfilled" badge (slate, explanatory tooltip) on data.backfilled
- row time now shows the original event-log timeCreated with a
"(reported …)" suffix; metadata block gains a "Recorded" line;
raw mode gets a compact ⟲ marker
- severity deliberately NOT downgraded — unresolved pre-agent errors
must stay visible in the Error filter
Agent (ModernDeploymentTracker):
- backfill scan buffers the capped record set, emits an Info summary
event first (modern_deployment_log, backfillSummary: true +
replayedCount/droppedOverCap/lookbackMinutes) announcing the block,
then replays oldest-first so timeline sequence follows event-log
chronology (cap semantics unchanged: newest 25 win)
No new EventType, no ConfigVersion bump. Tests: summary emission x2,
tracker subset 12/12; tsc + vitest 651/651 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customers cannot tell why a gather rule produces nothing in the timeline
(e.g. a logparser rule with emit_mode on_change that never emits). New
per-tenant boolean EnableGatherRuleDebugLog (EnableImeMatchLog-style
plumbing end to end): when enabled, the agent writes
%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log with one
pipe-delimited line per evaluation decision across all previously silent
"produced nothing" exits — disabled rules, interval without
intervalSeconds, first-tick-after-one-full-interval, phase-scope skips
incl. currentPhase=Unknown, per-(rule, phase) dedup, empty collector
results (emitOnlyIfExists), on_change hash suppression (hash prefix +
streak), guard blocks, logparser per-file outcomes (position skips,
format=text hint when every line fails CMTrace parsing, maxLines and
wildcard caps), and full exception stacks the agent log drops.
- Backend: ConfigVersion 35 -> 36; TenantConfiguration/AgentConfigResponse
field + TableConfigRepository Store+Map
- Agent: GatherRuleDebugLog writer (10 MB cap, one .old generation,
failures swallowed, optional console echo), GatherRuleContext.DebugLog
(null-safe no-op), trace points in executor + collectors; never writes
under _scopeLock (reasons captured inside, written after release);
merger bool->path projection with --gather-debug-log CLI override
tolerance; deliberately NOT in the offline-cache strip lists;
--run-gather-rules always traces; zero-rules delivery leaves a one-line
file instead of nothing
- Side fix: GatherRuleExecutorHost now passes the logger into
InformationalEventPost so gather-event emissions hit the DEBUG log
- Web: tenant settings toggle (Settings -> Agent) + admin tenant-config
report rows
- Tests: merger on/off/override, writer format+rotation, executor debug
lines, logparser debug lines (agent 2131, backend 3561, tsc green)
- Docs: docs/agent/gather-rule-debug-log.md + index/log entries
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.
- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
Sidecar registers later (legitimate startup ordering); DevicePreparation-only
snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
classified healthy); probe/doc headers and OKF doc corrected, web event-type
description updated
Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.
Agent tests 2043/2043, web tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.
New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):
- EspPolicyProviderProbe: fail-soft registry read of the
EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
continuous-incompleteness dwell (resets on completion/disappearance;
absent key = normal early enrollment), one-shot per provider,
StartupEventGate fingerprint over the sorted stalled key set for
cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
purely observational, no decision-engine coupling, no config toggle,
no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
StallProbe idle gate (idle clock resets on session activity) and the
idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
web gather-rule autocomplete (stall category).
Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.
Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).
Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.
Both the detector and the embedded allowlist take effect with the next agent
release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).
Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).
Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).
Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).
- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
(dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
hard-timeout deadline, release the gate with outcome
FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
complete the first deployment) nor by a first observation of 210
(session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list
Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.
The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.
The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.
Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.
Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rework after the first real enrollment (session b2e890c1) exposed three
defects in a654c730:
- Shell-Core "RebootCoalescing" match was a false positive by construction:
the token appears in the ROUTINE SubcategoryProcessing_Started bootstrap
marker ("Starting subcategory DeviceSetup.RebootCoalescing...") on EVERY
enrollment. esp_reboot_coalescing removed entirely (branch, constant,
catalog, tests).
- 2800 reboot-required flags fire at every policy sync; only DeviceSetup-time
flags cause the coalesced reboot. The session's flags fired during
AccountSetup - no reboot - yet our Warning events claimed "device will
restart". Raw event is now a neutral Info observation; the reboot CLAIM
moved to ANALYZE-ESP-005 with a REQUIRED system_reboot_detected >= 1
condition (no observed reboot -> advisory silent).
- One event per 2800 record was spam: records are now buffered and debounced
(10 s) into ONE aggregated event per burst (rebootUris[]/uriCount/
firstRebootUri/recordCount/unparsedCount). Watermark is persisted at FLUSH,
not claim - a pre-emit reboot kill loses nothing, the post-restart backfill
re-emits the burst at its historical time.
Verified 2800 description shape recorded in code, tests, and concept doc:
"The following URI has triggered a reboot: (./Device/...)."; extraction
covered by a real-text regression test. Lesson added to tasks/lessons.md
(observation-level events vs outcome-gated claims; substring tokens appear
in framework bookkeeping records).
Tests: Agent.V2.Core 1867/1867, Functions 3099/3099, MCP 400 passed,
solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).
- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
remediation = reassign profiles device -> user groups; +20 confidence on
coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
message texts are unverified until E2E VM validation (tolerant matching).
Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oliver Kieselbach