The OfficeInstallDetector fired a false office_install_failed on devices with
pre-installed OEM/consumer inbox Office (e.g. O365HomePremRetail +
OneNoteFreeRetail) running a background C2R CLIENTUPDATE/maintenance scenario:
the already-resident Office was mis-read as a failed Microsoft 365 Apps install,
with a benign small-integer scenario value ("2") treated as a failure code.
Field telemetry (202 office_install_started events) showed 200/202 real installs
start with streamingFinished=false (a live download) and only 2 already-resident
— so the clean discriminator is "already on disk", not the scenario name. A
"require INSTALL scenario to start" gate was rejected: 47% of real starts fire
via the early DO-CDN job before C2R writes its registry.
Changes:
- New office_preinstalled_detected (Info): emitted when the core Office binaries
are already on disk at the first signal AND the product set is consumer/OEM
inbox (OfficeProductClassifier). Anchored on the filesystem, NOT the flaky
StreamingFinished value (absent on some machines). Enterprise Office already on
disk keeps the started->completed path (CSP/Win32-wrapper case).
- Does NOT latch terminal: an enrollment often uninstalls the inbox Office and
installs a fresh one, so the detector stays armed (emit-once guard +
StatePreinstalled persisted for cross-reboot dedup, ResumePreinstalled set
before the process watcher's startup probe).
- Error attribution scoped to the INSTALL scenario only (+ errorSource); a
CLIENTUPDATE error never fails an install.
- Registry idle-start gated on InstallScenarioPresent (not any scenario).
- IsNonZeroNumericCode rejects implausibly small positive decimals (<1000).
- DO bytes are seeded only once a real lifecycle opens, so a preinstalled/
CLIENTUPDATE download cannot leak into a later install's doSummary.
- Web InstallProgress renders a sky "Pre-installed" info row; MCP + web event
catalogs updated.
V2.Core 1454/1454, web tsc clean, MCP 240 passed. Inert until redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
disk_free_gb was previously only visible inside the Debug performance_snapshot
payload (filtered out of timelines), so a disk-starved enrollment carried no
actionable signal. PerformanceCollector now emits a dedicated disk_space_low
Warning (ImmediateUpload) when the system drive crosses below 2 GB free.
State-change-only (no heartbeat): strict <2 GB fire, re-arm only above 3 GB
(1 GB hysteresis) so a single enrollment that expands then cleans its temp
cache can warn again without flapping. Decision lives in pure, testable
PerformanceCollector.EvaluateDiskLowTransition(double, ref bool).
- new Constants.EventTypes.DiskSpaceLow = "disk_space_low"
- 'disk_space_low' added to MCP resource-catalog health_events (drift sync)
- PerformanceCollectorDiskLowTests (7 cases); guard + drift tests green
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The office_install_completed doSummary paired aggregateFileSize next to
totalBytesDownloaded with no context, making the (expected) gap look like a
bug: the completed event fires at core-binary completion while background
streaming of the remaining declared content is still in flight, so the peak
DO sample shows downloaded < declared. aggregateFileSize is the sum of the
DO jobs' declared target sizes at the peak instant, not a stable payload total.
Mirror downloadPercent (from OfficeDoSample.DownloadPercent) and
streamingFinished into the summary so the gap is self-explanatory. XML-doc
now states why aggregate exceeds downloaded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RealmJoinWatcher was always-on, gated only by a compile-time const. Most
tenants do not deploy via RealmJoin, so the watcher and its 60-min completion
gate were pure noise. Add a per-tenant portal toggle (Agent Analyzers ->
RealmJoin Watcher), default off, delivered to the agent via remote config.
Mirrors the EnableIntegrityBypassAnalyzer flow end-to-end, with one divergence:
RealmJoin is a collector host (not an IAgentAnalyzer), so the agent-side gate
lives in DefaultComponentFactory.CreateCollectorHosts, not AgentAnalyzerManager.
When disabled the host is not created; the existing null-guard in the
DesktopArrival callback handles the absent HKU arming. The compile-time const
RealmJoinTrackingEnabled is kept as a build-time master kill-switch
(effective enable = remote flag AND const).
- Shared: TenantConfiguration.EnableRealmJoinWatcher (bool?, null=agent default
false) + AnalyzerConfiguration.EnableRealmJoinWatcher (bool=false)
- Backend: TableConfigRepository write/read; GetAgentConfigFunction maps
?? false into Analyzers; ConfigVersion 29 -> 30
- Web: types + TenantConfigContext (state/load/save/reset/deps/context value) +
SectionAgentAnalyzers + new "RealmJoin Watcher" subsection toggle
Inert until agent redeploy: old agents ignore the flag and stay always-on.
Verified: solution build 0 errors, tsc --noEmit clean, 76 backend config +
99 V2.Core tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
V2 never wired the Legacy EnrollmentTracker's CollectAtEnrollmentStart /
CollectAtEnd (plan 5.8 TODO) - the device-info sweep ran only at agent
start. For IME-deployed agents the per-reboot restart re-ran CollectAll
and picked late values up eventually, but image-deployed agents
(--await-enrollment resumes the moment the MDM certificate appears,
i.e. at the very beginning of provisioning) collect an almost-empty
surface, and a session without a reboot between agent start and
completion never refreshed at all: aad_join_status stuck on
"Not Joined", autopilot profile / FirstSync ESP config stale, BitLocker
end state (enabled via policy DURING enrollment) never captured.
DeviceInfoHost now subscribes to the central signal stream (same
pattern as ProvisioningPackageHost) and mirrors the V1 behavior:
- CollectAtEnrollmentStart once on the first DeviceSetup phase signal
(re-fetches AAD join, autopilot profile, ESP config, TPM)
- CollectAtEnd once on FinalizingSetup or desktop arrival - desktop
arrival also serves the no-ESP / WDP v2 case and runs the
enrollment-start refresh if DeviceSetup was never seen
- repeat posts of the EspConfigDetected decision signal are explicitly
safe (reducer set-once with late-fill semantics)
This was deferred partly because every re-collect used to mean 4-6
duplicate events; with the StartupEventGate emit-on-change dedup the
re-collections only emit values that actually changed.
V2.Core tests 1410/1410 green (2 new); V2 runtime builds clean.
Inert until redeploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One enrollment session spans several reboots; every reboot restarts the
agent process and re-ran the one-shot startup checks, re-emitting ~20
events that carried no new information (device-info sweep, geo/timezone/
NTP probes, startup analyzers). Multi-reboot sessions multiplied this
into significant duplicate-event volume; the timezone probe even re-ran
tzutil on every boot.
New StartupEventGate (startup-event-state.json in the state directory,
fail-soft, atomic write, thread-safe) with two policies:
- emit-on-change: payload-fingerprint dedup for the 13 DeviceInfoCollector
events (central EmitDeviceInfoEvent hook; enrollment_type_detected now
routed through it too), software_inventory_analysis startup baseline,
local_admin_analysis (startup trigger only) and
integrity_bypass_analysis. Identical repeats are suppressed; a real
change (aad_join_status flipping after the Hybrid-Join reboot,
BitLocker conversion progressing) always re-emits. The gate only ever
wraps the EVENT emission - collection, decision signals and derived
facts (HasAadJoinedUser, EspConfigDetected reducer signal, inventory
delta base) run unchanged.
- retry-until-success: device_location, timezone_auto_set and
ntp_time_check skip entirely (no network call / tzutil) once they
succeeded in a previous run; failures keep retrying on the next start.
NTP latches only at offset <= 60s (shared constant with the severity).
Deliberately ungated: power_state_check (battery drain across the
enrollment is real per-boot information), boot_time, wifi_signal_info,
agent_started, and all shutdown emissions (the local-admin and
software-inventory start-vs-end delta comparisons stay intact).
The network_interface_info fingerprint excludes linkSpeedMbps (the
negotiated WiFi rate varies per association and would defeat the dedup).
One shared gate instance per process, created in AgentRuntimeHost and
passed to DefaultComponentFactory, AgentAnalyzerManager and
StartupEnvironmentProbes.
V2.Core tests 1408/1408 green (25 new); V2 runtime builds clean.
No ConfigVersion bump, no new event types; inert until redeploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cold starts spent 35-55s (measured across 8 boots today) embedding the
static corpus — 119 knowledge docs + 144 event types — through MiniLM
on the 0.25 vCPU container before the server even started listening.
The corpus is baked into the image, so its embeddings now are too:
- precompute-embeddings.ts runs in the Docker build stage (also serving
as the model prefetch), writing search-index.json with per-section
sha256 corpus hashes.
- At boot, index.ts hydrates both vector indexes from that file in
milliseconds (135 ms measured locally). Model name + corpus hash are
validated first; any mismatch (rules changed without rebuild, model
bump, malformed file) logs loudly and falls back to computing at
boot — slow but never stale. SEARCH_BACKEND=fuse and the fuse boot
fallback are unaffected.
- The query embedder is no longer needed at boot; it warms in the
background after the indexes are ready. getEmbedder now memoizes the
promise instead of the resolved pipeline — a query racing the warmup
previously could have constructed the model twice (~2x model RAM on
a 0.5 Gi container).
- 14 new model-free unit tests: hash stability/sensitivity, validation
rejection paths, serialize/hydrate roundtrip.
Expected cold start drops from ~60-80s to infra-only (~15-25s).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OfficeInstallDetector state was in-memory only, so every agent
restart (an enrollment spans several reboots) re-armed the detector
from Idle and all three start triggers fired falsely on an
already-installed Office: Scenario\INSTALL persists post-install,
OfficeC2RClient.exe runs again for update checks, and Office updates
stream from the same CDN. Each restart then emitted a duplicate
started+completed pair (~0s duration) via the binary watcher's
initial scan; a mid-install reboot also reset the start time.
New OfficeInstallStatePersistence (office-install-state.json in the
state directory, fail-soft, atomic write) carries the lifecycle:
- persisted terminal (Completed/Failed) -> the factory does not arm
the detector at all on the next run (no WMI/RegNotify watchers)
- persisted Active -> the host resumes the open lifecycle without a
second started; a completion proven in this run reports the true
total duration plus the persisted DO peak as doSummary (also
catches completions missed across a mid-install reboot)
- AbandonSilently deliberately does NOT persist a terminal: it fires
on every dispose and would block that resume
- DO peak growth re-persists throttled (30s) while Active
V2.Core tests 1383/1383 green (14 new); V2 runtime builds clean.
No ConfigVersion bump (no new remote setting); inert until redeploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub forces all JS actions onto Node 24 starting 2026-06-16 and
removes Node 20 from runners on 2026-09-16. Bump every affected action
to the first major that officially declares node24, avoiding majors
with functional changes (checkout v6, upload-artifact v7):
actions/checkout v4 -> v5
azure/login v2 -> v3
actions/setup-dotnet v4 -> v5
actions/setup-node v4 -> v5
actions/upload-artifact v4 -> v6 (v5 still defaulted to node20)
softprops/action-gh-release v2 -> v3
stefanzweifel/git-auto-commit-action v5 -> v7 (v7 reverted the v6
input removals; only standard inputs used here anyway)
Azure/functions-action@v1 (floating tag, node24 since v1.5.4) and
Azure/static-web-apps-deploy@v1 (Docker action) need no change.
All target tags verified to exist via git ls-remote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The CI image build died on a single HuggingFace connect timeout:
transformers.js does not retry its model fetch and undici aborts after
10s, so one flaky connection failed the whole deploy. The prefetch step
now retries up to 5 times with 15s backoff, still fails the build
loudly when the model genuinely cannot be fetched or lands outside
HF_CACHE_DIR, and --build-arg PREBAKE_MODEL=0 skips pre-baking during
an extended HF outage (boot-time download / fuse fallback take over).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
H2: search_events caller keywords are now lowercased and MERGED with
auto-extraction via resolveQueryKeywords() — previously `??` replaced
extraction entirely and case-sensitive keywords like ["BitLocker"]
silently returned zero results.
H3: the ~23 MB MiniLM model is pre-baked at Docker build time (warmup
via the built provider keeps the model name single-sourced) into
HF_CACHE_DIR=/app/.hf-cache and copied into the final stage, so a
scale-to-zero cold start never depends on the HuggingFace CDN; boot
indexing falls back to the fuse backend with a loud log instead of
crash-looping on init failure.
M8: new SearchProvider.semanticCapable flag (vector=true, fuse=false);
selectEventTypeCandidates gates the semantic path on it so inverted
Fuse scores no longer flow through cosine-calibrated thresholds, and
the search_events response now reports the real searchBackend
(keyword-only vs hybrid) plus the provider name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gate-starvation dead-ends (caa6cf50, 1ec8f4c6) were ultimately caused
by a single required user-targeted app that never started installing.
Until now the session just hung anonymously; this makes the starving app
a named, actionable customer hint.
New ImeLogTracker.GetStarvedUserEspApps(): required (Install/Uninstall
intent) apps in the current AccountSetup phase with no download/install
activity ever observed, not terminal, not failed. Apps that are alive
(Downloading/Installing) stay excluded; error apps belong to the failure
path. Same conservative thread-safety contract as AreUserEspAppsSettled.
Two timer-free trigger points, one-shot per appId:
- Live: when the ESP page exits and the user-apps-settled probe reports
false, EspAndHelloTracker emits app_install_starved (Warning) per
starved app; dedupe via a lock-guarded HashSet exposed as
StarvedAppsReported.
- Terminal: EnrollmentTerminationHandler sweeps leftovers on every
outcome (before app_tracking_summary, skipped on WhiteGlove Part 1) and
consults the live path dedupe set via the read-model surface, so no
app is ever reported twice.
Wiring: ImeLogHost bridge -> DefaultComponentFactory probe ->
EspAndHelloTracker; CollectorSurfaces + IAppTrackingReadModel +
OrchestratorAppTrackingReadModel expose the starved list and the
already-reported set to the termination handler.
New event type synced to MCP resource catalog (app_events) and Web
gather-rules eventTypes.
V2.Core 1365/1365, MCP vitest 220, Web 458, full solution build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When the engine blocks or defers a completion attempt it now says what it
is still waiting on instead of going silent. New Info event
completion_waiting with data.missingPrerequisites built from stable
literals (account_setup_provisioning_complete, hello_resolution,
desktop_arrival, realmjoin_resolution), plus trigger, stage,
armedDeadlines and — on the guard-blocked esp_exiting site — the
AdvisoryCompletion window due-time.
Emission sites: CompleteThroughFinalizingOrDefer gate-closed defer,
HandleDesktopArrivedV1 desktop-first / blocked fast-path,
HandleEspExitingV1 post-AccountSetup guard-blocked exit,
HandleHelloResolvedV1 + HandleHelloSafetyDeadlineFired AwaitingDesktop
branches, HandleAccountSetupProvisioningCompleteV1 no-promote branch.
Pre-AccountSetup handoff exits stay silent (normal flow).
Dedupe via new additive-nullable state fact CompletionWaitingFingerprint
(comma-joined missing-set; ctor-end param, no schema bump — same pattern
as ImeUserSessionCompletedUtc in a77ad21e). A repeat with the same set
emits nothing; a set change emits one new event. SnapshotBuilder facts
16 -> 17; serializer roundtrip + legacy-snapshot compat covered.
New event type synced to MCP resource catalog (esp/enrollment group) and
Web gather-rules eventTypes.
DecisionCore 400/400, V2.Core 1347/1347, MCP vitest 220, Web 458, tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DecisionEngine is edge-triggered: facts get recorded, but nothing
guarantees another signal ever re-asks the completion question — a session
can be parked with no deadline armed. The three known dead-end variants
(8bc1180f, caa6cf50, 1ec8f4c6) are covered by dedicated arming sites; this
adds the tripwire for the unknown variant 4.
DecisionStepProcessor now checks the post-step state: non-terminal, past
AccountSetup entry, dead-end zone entered (ESP final exit at-or-after
AccountSetup entry OR advisory-defanged failure recorded) and NO
resolution-capable deadline armed (ClassifierTick does not count — it
re-arms itself and never resolves a session). One-shot Warning event via
InformationalEventPost (wired through the LazyIngressSinkRelay), no state
schema touch, no auto-action. Expected to never fire in production; every
occurrence is a bug report.
AwaitingDesktop without a deadline fires by design (visible parking); a
DesktopSafety deadline stays a possible later iteration.
New event type synced to the MCP resource catalog (health_events).
V2.Core 1347/1347, DecisionCore 387/387, MCP vitest 220 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session 1ec8f4c6 exposed the third completion-dead-end variant: Windows closed
the ESP page NORMALLY (Shell-Core 62407, errorCode=0) while the AccountSetup
Apps subcategory was still in_progress (a user-targeted app never started).
No EspTerminalFailure ever fired, so the AdvisoryCompletion arming site from
the 8bc1180f fix never ran - yet the dead-end is identical: the registry gate
can never open, the IME settle probe is starved, Hello is disabled, and the
session idles to the max-lifetime watchdog although the full completion
conjunction (real-user desktop + Hello disabled + genuine IME user-session
completion) was satisfied minutes in.
HandleEspExitingV1 now arms the shared AdvisoryCompletion resolution window
when a guard-blocked esp_exiting lands AFTER AccountSetup entry (fire-once;
pre-AccountSetup handoff exits stay arm-free; a later advisory deliberately
re-bases the window). The deadline handler accepts the new anchor (final exit
at-or-after AccountSetup entry) alongside the advisory anchor and fails the
no-evidence case with its own semantics: reason
esp_exit_without_completion_evidence and LastFailureTrigger=DeadlineFired -
NOT esp_terminal_failure, and the likely-stuck app promotion stays off,
because the ESP never reported a failure.
Updated the two tests that pinned "guard-blocked exit arms nothing" to the
new contract - notably the caa6cf50 gate-starvation shape now carries the
resolution window as a 30-min backstop underneath the synthesis fast path.
DecisionCore 387/387, V2.Core 1335/1335 green; solution builds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The V2 max-lifetime watchdog is deliberately "notbremse, not a session
verdict": the agent stops permanently and emits only
agent_shutting_down(reason=max_lifetime) - no enrollment_failed. Since the
backend flips session status exclusively on enrollment_complete /
enrollment_failed / esp_failure, such sessions stayed InProgress forever
(session 8bc1180f). V1 agents already emit enrollment_failed at max lifetime;
the gap is V2-only.
Both classification copies (legacy ingest + EventIngestProcessor, kept in
sync per the documented copy contract) now classify the event via a shared
IsMaxLifetimeAgentShutdown predicate and map it as the LOWEST-priority status
writer: any genuine terminal event in the same batch wins via the else-if
chain, already-terminal sessions are protected by the repository idempotency
guard, Pending (WhiteGlove) sessions are skipped, and Stalled sessions
finally resolve to Failed. failureReason carries the uptime;
failureSource=max_lifetime_watchdog.
This is the defense-in-depth half of the 8bc1180f fix - it works for every
field agent immediately on backend deploy, no agent redeploy needed. The
root-cause half (AdvisoryCompletion deadline in DecisionCore) resolves the
dead-end before max-lifetime is ever reached.
Backend tests 2284/2284 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session 8bc1180f: Windows dumped 689 EventID-100 "Autopilot policy not found"
records in one minute. Each was downgraded to Debug but still forwarded as an
individual modern_deployment_log event - 701 of 1116 session events (63%),
signal-ingress queue saturated (ingress_backpressure 256/256), and the burst
deferred the periodic-collector idle stop.
Two changes:
- ModernDeploymentTracker: harmless-downgraded EventIds (default 100/1005/1010,
Level 2/3 -> Debug) are now rollup-gated per EventId - first 3 occurrences
forwarded individually, afterwards only every 100th, carrying the cumulative
occurrenceCount (plus a rollup marker in Data). Counter-based, no timers;
warnings/errors and non-harmless IDs are untouched. The 689-burst now
produces 9 events instead of 689.
- SignalActivityClassifier: modern_deployment_log added to the non-activity
denylist - OS-eventlog forwarding is observability, not enrollment progress,
and a policy-cache re-read burst must not keep the periodic collectors
alive. modern_deployment_warning/error remain real activity.
V2.Core rollup + classifier tests 33/33 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An advisory-defanged EspTerminalFailure (ContinueAnyway + AccountSetup entered)
deliberately keeps the session non-terminal so the normal completion paths can
still win. Session 8bc1180f showed the dead-end variant: AccountSetup/Apps
failed (0x87d1041c), so AccountSetupProvisioningComplete can never arrive, the
IME settle probe was starved by a never-started user app, and Hello was
disabled - no completion path left. The session idled InProgress until the
max-lifetime watchdog, which by design issues no session verdict.
The advisory path now arms a 30-min AdvisoryCompletion deadline (AgentBoot-
floored, not re-armed by enrichment duplicates). On fire the engine resolves
the session either way:
- Completed (through Finalizing) when the real-user conjunction holds:
desktop_arrived (DAD-validated, defaultuser0/SYSTEM excluded at the
detector) AND Hello resolved-or-policy-disabled AND the new
ImeUserSessionCompletedUtc fact at-or-after AccountSetupEnteredUtc. The
IME timestamp gate kills defaultuser0 ghosts - OOBE/technician IME sessions
complete in the pre-AccountSetup frame and can never satisfy it.
- Failed otherwise, un-defanging the advisory: enrollment_failed with
reason=esp_terminal_failure and LastFailureTrigger=EspTerminalFailure so the
EnrollmentTerminationHandler likely-stuck app promotion keeps firing.
ImeUserSessionCompletedUtc is an additive nullable DecisionState fact
(set-once in HandleImeUserSessionCompletedV1, serialized snapshot-compatible,
surfaced in the signal census as ime_user_session_completed).
DecisionCore.Tests 381/381 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session 4d5a0b78: WMI Win32_Process.GetOwner failed on every single DAD poll
(wmiErrorCount == pollCount from the first poll on), so the explorer.exe
owner was never resolved, desktop_arrived never fired, and the Desktop half
of the completion AND-gate starved — the session sat in AwaitingDesktop
forever even though the gate-starvation synthesis (d3abbacd) had correctly
opened the AccountSetup gate.
Owner resolution is now WTSQuerySessionInformation(WTSUserName/WTSDomainName)
keyed by the process's session id as the primary path — a single kernel
round-trip with no WinMgmt dependency, and semantically the right question
anyway (explorer.exe runs as the session's logged-on user). The previous WMI
GetOwner path remains as fallback when WTS yields no user.
- New WtsNativeMethods interop (wtsapi32) with buffer-safe QuerySessionString
- DesktopArrivalDetector.ResolveOwner: WTS first, WMI fallback, injectable
test seams (SessionOwnerResolver/ProcessOwnerResolver)
- first_poll/no_candidate liveness payloads now carry wtsErrorCount alongside
wmiErrorCount so the broken path is visible per device
V2.Core 1327/1327 green; WTS path live-verified (AzureAD\user shape matches
the existing IsExcludedUser contract).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
realmjoin_package_started/completed events now render as first-class
install rows in the session-detail Install Progress, alongside Intune
(app_install_*) and Office (office_install_*) entries. The "RJ: " name
prefix keeps mixed Intune/RealmJoin installs distinguishable and doubles
as the map key, avoiding collisions with IME app names.
RealmJoin has no separate failed event — completed with success="false"
routes to the Failed branch, and lastExitCode maps onto the existing
exit-code badge (error-code catalog lookup included).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renders productVersion (+ releaseChannel when not "release") from the
existing realmjoin_detected event, right below the IME Agent Version
row. Pure client-side read of the event stream - no agent or backend
change needed, works retroactively for existing sessions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session caa6cf50: a policy-skipped user-ESP app leaves the ESP registry's
Apps subcategory permanently inProgress, so Windows never writes
AccountSetupCategory.Status.categorySucceeded before tearing down the ESP
page. Both registry-driven AccountSetupProvisioningComplete paths (normal +
all-subcategories fallback) starve, ShouldTransitionToAwaitingHello never
opens, and the session stalls after 60 min despite a successful enrollment
(esp_exiting + desktop_arrived + helloPolicy=false all recorded).
Fix, in three parts:
- ImeLogTracker.AreUserEspAppsSettled(): conservative probe — true only in
AccountSetup phase with >=1 required app, all required apps terminal
(Installed/Skipped/Postponed) and zero errors. Exposed via ImeLogHost.
- EspAndHelloTracker: on every Shell-Core exit forward, fire-once synthesis
of AccountSetupProvisioningComplete when the probe reports settled, plus
an esp_provisioning_status observability event
(fallbackReason=esp_exited_user_apps_settled_category_unresolved).
Wired through a new optional userEspAppsSettledProbe ctor param;
DefaultComponentFactory connects it to ImeLogHost via closure.
- HandleAccountSetupProvisioningCompleteV1: deferred-completion parity.
When the strong gate arrives last (Win11 ordering: explorer.exe runs
under the User-ESP page, so Desktop arrives before the final exit) and
Hello is resolved or policy-disabled, complete through Finalizing with
HelloOutcome="Skipped" instead of parking in AwaitingHello until
HelloSafety stamps a misleading "Timeout" 300 s later.
Sessions with failed or in-flight user apps keep the existing conservative
stall behaviour. DecisionCore 366/366, V2.Core 1321/1321 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RealmJoin.exe inverts the usual .NET SDK version-resource layout: the
FileVersion STRING carries the full SemVer including the channel tag
("4.21.6-canary+476277.d320cac0") while ProductVersion is the bare
"4.21.6". The previous ProductVersion-first read therefore never saw
the channel.
- RealmJoinInfo: read FileVersion first, parse SemVer into bare version
+ release channel (prerelease tag = channel; untagged = "release",
confirmed by the RJ developer; build metadata dropped)
- realmjoin_detected signal + timeline event carry releaseChannel
- RealmJoinFacts.ReleaseChannel set-once fact + audit-trail field
realmjoinReleaseChannel
Suites: DecisionCore 360, V2.Core 1307 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EnrollmentTerminationHandler's 25-parameter constructor (17 optional,
9 Func<> accessors, wired via ~80 lambda lines in AgentRuntimeHost)
becomes 11 parameters: four cohesive surfaces plus direct infra deps.
- TerminationSessionContext (concrete, pure data): configuration,
stateDirectory, agentStartTimeUtc, agentVersion, sessionPersistence,
dialogExePathOverride, lazy IsWhiteGlovePart2.
- IAppTrackingReadModel: currentState, packageStates, appTimings,
ignoredCount, promoteActiveInstallsToStuck, lastEspTerminalFailure.
Production: OrchestratorAppTrackingReadModel over the orchestrator's
CollectorSurfaces (builds on ARCH-F4).
- IDrainStatus: spool/ingress drain counts behind CanObserve* capability
flags (replaces the accessor-null wiring variants); production:
OrchestratorDrainStatus with the V1-parity 10s budget.
- IShutdownGate: signalShutdown, triggerReboot, writeCleanExitMarker,
tryClaimShutdownEvent; production: RuntimeShutdownGate in the host
project (owns the shutdown.exe logic formerly DefaultTriggerReboot).
Per-parameter rationale comments (c117946b, 080edee9, 7dd4e593, Codex
Finding 2, Option 1/2 hardening, plan §11) moved onto the interface
members. Test rig builds recording fakes of the same interfaces, so the
optional-default wiring variants production never ran are gone.
No behavior change. V2.Core 1297/1297, DecisionCore 359/359, full
solution build green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CreateCollectorHosts now returns an immutable CollectorSurfaces bundle
(hosts + typed live read-model views) and takes the telemetry spool as a
parameter. Deletes DefaultComponentFactory's mutable host fields, all
lazy read-model properties, and SetTelemetrySpool; deletes the
'is DefaultComponentFactory' downcast in EnrollmentOrchestrator (the
IComponentFactory seam is the contract again). The orchestrator stores
the surfaces and exposes them via CollectorSurfaces; AgentRuntimeHost
reads IME app tracking / EspAndHelloHost / AadJoinHost from there.
No behavior change. V2.Core tests 1297/1297 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
P2 (MON-C2): HelloTracker.ResetForEspResumption did not clear the new
_helloUnknownGraceUsed one-shot flag. After a grace-consumed run, a fresh
post-resume Hello-wait that hit an undetected policy would skip its grace
re-arm and resolve straight to not_configured. Reset the flag + add a
regression test (unknown timeout -> reset -> unknown timeout re-grants grace).
P3 (ARCH-F1): the completion-gate docs overpromised "add a gate to
s_completionGates and every site inherits it". Tightened to state precisely
that the collection governs only the completion-ATTEMPT sites routed through
CompleteThroughFinalizingOrDefer; the RealmJoin release path (pre-resolution
state) and the SelfDeploying deadline path (gate-specific deferral) do NOT
consult the collection and need explicit wiring for a second gate. Doc-only.
V2.Core.Tests 1297/1297; DecisionCore.Tests 359/359.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wave 4 / ARCH-F1. The cross-cutting "may the session complete now?" AND-gate
was hand-inlined at every terminal site across the Classic / Shared engine
partials — each with a near-identical RealmJoinGateOpen check plus its own
":RealmJoinGateClosed" defer block. That duplication was the origin of the
premature/duplicate enrollment_complete regressions (8b8d611d, 08c99638,
330f73f3) and meant a second completion gate (WDP v2 device-association
settle) would have to be woven into every site.
New DecisionEngine.Completion.cs introduces:
* CompletionGate (name + IsOpen predicate) + an append-only s_completionGates
collection (RealmJoin is the sole gate today) + FirstClosedCompletionGate.
* CompleteThroughFinalizingOrDefer — the single router every "both
prerequisites in -> finish via Finalizing" site now calls. All gates open
-> TransitionToFinalizing; a gate closed -> the canonical defer step with
the trigger tagged ":<GateName>Closed" (derived from the gate, not
hardcoded).
The four identical Classic/Shared sites (HelloResolved, DesktopArrived normal
+ Hello-disabled fast-path, Hello-safety timeout) now route through the helper.
The SelfDeploying deadline path keeps its gate-specific deferral (it owns the
SelfDeployingDeferredCompletion flag, released by the RealmJoin handlers); the
RealmJoin release path completes directly because the completion gates read the
pre-resolution state where RJ is still closed.
No behavior change — proven by the replay-deterministic scenario tests passing
bit-identically (same terminal stages, outcomes, hashes). Adding a gate is now
one entry in s_completionGates. New forward-proof test locks the gate-name ->
trigger-suffix derivation.
DecisionCore.Tests: 359/359 (was 358 + 1 new). Full solution builds; V2.Core
engine integration green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HelloTracker.OnHelloWaitTimeout branched on `if (_isHelloPolicyEnabled)`,
whose false case force-resolved HelloOutcome=not_configured and fired
HelloCompleted. But `_isHelloPolicyEnabled == false` also holds when the
policy has simply not been detected yet (_isPolicyConfigured == false) —
a slow MDM/CSP sync that lands the WHfB policy read after the 30s wait
would then prematurely resolve Hello on a device that is actually
configured.
Make the timeout a 3-way decision:
* policy known ENABLED -> extended wait (long completion timer)
* policy NOT YET DETECTED -> one bounded grace re-arm (~wait window)
so a slow policy sync can land; only
then fall through to not_configured
* policy known DISABLED (or still
unknown after the grace) -> not_configured completion (unchanged)
The bounded grace (a single re-arm of the existing one-shot wait timer)
is deliberately NOT the 300s completion timer: DetectHelloPolicy returns
(null,null) forever on a genuinely-unconfigured device, so an unconditional
extended wait would add a 5-minute completion delay to every such device.
The grace caps the extra wait and still closes the race.
Telemetry: hello_wait_timeout events now carry policyDetected + a distinct
action ("unknown_policy_grace") so the undetected path is observable.
V2.Core.Tests: 1296/1296 (3 new MON-C2 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wave 4 / ARCH-F7. HelloTrackerAdapter, ShellCoreTrackerAdapter and
ProvisioningStatusTrackerAdapter had zero production instantiations —
production runs solely through EspAndHelloTrackerAdapter (wired in
EspAndHelloHost), which already exposes every Trigger*FromTest seam the
sub-adapter tests exercised. Tests on the sub-adapters proved nothing
about production behavior.
- Delete the 3 sub-adapter sources + their test files.
- Port the meaningful coverage to EspAndHelloTrackerAdapterTests:
HelloOutcome verbatim-propagation theory + empty-failureType fallback.
- Rewrite ShellCoreToEngineEspExitingTests test 1 to assert the tracker's
EspExited contract directly (HandleBackfillRecord -> EspExited + source
timestamp); drop the redundant adapter-driven reducer test (coordinator
test 5 already covers the full production chain).
- Drop the duplicate ProvisioningStatusTrackerAdapter test in
EspFailureDetectedHResultTests (coordinator test above it is identical).
- Refresh stale ShellCoreTrackerAdapter references in ShellCoreTracker +
DefaultComponentFactory doc comments.
V2.Core.Tests: 1293/1293 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three sources flooded the App Insights warning/exception channels and worked
against traceability without any functional value:
1. Per-batch "still InProgress — may be stuck" warning removed from both ingest
sites (EventIngestProcessor + IngestEventsFunction.Processing). It was
observability-only (no status/StalledAt/OpsEvent) and fired on every ingest
of a >4h session (~440 warnings/10h for 7 sessions). Stuck handling is owned
authoritatively by MaintenanceService.MarkStalledSessionsAsTimedOutAsync
(Stalled@2h silence, Failed@SessionTimeoutHours + SessionTimeouts OpsEvent),
which acts strictly earlier than the 4h log — zero functional loss.
2. MergeSessionIndexAsync: specialize the catch on RequestFailedException 404
(benign race — index row not created yet / awaiting dual-write backfill) and
downgrade to LogDebug so the handled exception stops polluting the exceptions
table. Other failures still log as Warning.
3. StoreEventsBatchAsync: dedup each chunk by RowKey ({Timestamp:ms}_{Sequence})
before SubmitTransactionAsync. Colliding events collapse under UpsertReplace
anyway, but Azure rejects the whole batch with InvalidDuplicateRow, forcing a
per-event fallback and tracking a noisy TableTransactionFailedException.
Keeping last-occurrence (matches UpsertReplace last-wins) lets the batch
succeed cleanly.
Build: 0 warnings/errors. Tests: 123 (SessionIndex/StoreEvents/EventIngest/
Maintenance/Stalled) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MSAL runs with navigateToLoginRequestUrl=false and redirectUri=origin, so
after an interactive login the user lands on the bare root and AuthGate routes
them role-based to /dashboard — dropping the deep link they opened (e.g. a
/sessions/[id] link in a new tab, which has no shared sessionStorage token and
must re-auth).
Stash the intended in-app path before login() in ProtectedRoute and restore it
in AuthGate after auth settles. New lib/postLoginReturn.ts persists it in
sessionStorage (survives the same-tab redirect round-trip) and guards against
open redirects (same-origin "/..." paths only). Do NOT flip
navigateToLoginRequestUrl — it is intentional for the www->portal cross-origin
flow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploy-verified on f4d51ea4: the worker-side adaptive-sampling exclusion
(EnableAdaptiveSampling=false + manual Configure<TelemetryConfiguration>
UseAdaptiveSampling(excludedTypes:"Request;Event")) does NOT take effect in the
Functions isolated worker. Live data showed the enriched worker request copy
(Source=WorkerMiddleware) sampled to ~63% while the redundant host request item
(Source="") stayed fully billed — strictly worse (lossy enriched copy, host
duplicate unchanged). host.json "Host.Results": "Error" likewise did not drop
the host request item (emitted by the host process, not governed by that level).
Restores known-good: default worker sampling (enriched copy complete at normal
traffic, burst-capped) + host.json Host.Results back to Information. L1 storage
dependency filter and L2 Function-trace suppression are unaffected and still
verified live. L4 deferred pending a mechanism verified against the isolated-
worker pipeline. H2/H3/O1/O2 from f4d51ea4 verified healthy (0 exceptions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Telemetry cost (L4): worker-side adaptive sampling now excludes Request;Event
so the enriched RequestTelemetryMiddleware copy is never thinned; host.json
Host.Results=Error drops the duplicate unenriched host request item. One
complete request row per call instead of two (one unsampled+unenriched, one
sampled+enriched).
Ingest hot path (H2/H3): BlockedDeviceService caches negative answers (30s
revalidation, symmetric with positive entries) so healthy devices no longer
pay a BlockedDevices point-read per request; SessionDeletionGuard returns its
already-loaded row and IncrementSessionEventCountAsync returns the post-merge
snapshot, collapsing ~4 Sessions reads/batch to ~2.
Queue-worker correctness (O1/O2): new GetSessionEventsStrictAsync propagates
storage failures on the analyze + vulnerability-correlate paths so a transient
read fault retries/poisons instead of silently deleting the message after
"analyzing" an empty stream. Fail-soft variant kept for UI/metrics surfaces.
Review fixes: parse prefetched Status as string (the actual wire format, was
int -> optimization never fired); session-aware block over a cached negative
entry keeps its session scope instead of widening to whole-device.
2274/2274 backend tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Setting requestTelemetry.Context.Location.Ip never reached the requests
table: App Insights uses the IP only for geo-lookup then masks the stored
client_IP to 0.0.0.0 (verified — every request in the resource shows
0.0.0.0). On top of that, the only IP reaching the isolated worker is an
Azure-infra hop (West Europe / Amsterdam), not the device's real egress IP
(device geo = Falkenstein). A usable device IP would need a non-masked
custom property AND the correct forwarded header (Front-Door-topology
dependent) — not worth it: TenantId + CorrelationId already isolate a
device (they dissolved the IngestTelemetry "multiplication" illusion).
Remove the dead Context.Location.Ip block + now-unused Security using,
replace with a NOTE documenting why client_IP is intentionally absent so it
is not re-added. ClientIpExtractor.GetClientEgressIp stays (still used by
ReportDistressFunction's stored SourceIp, which is not App-Insights-masked).
Full backend suite 2249/2249 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Makes the requests table actually usable for diagnosing agent traffic and rate
limits, without adding any new telemetry items — only properties on the already-
emitted, sampling-excluded Request telemetry (host.json excludedTypes "Request;
Event"), consistent with StorageDependencyFilterProcessor.
RequestTelemetryMiddleware:
- client_IP: the isolated worker behind Front Door/App Service left it 0.0.0.0.
Set Context.Location.Ip from X-Azure-ClientIP (Front Door's true client egress
IP) else trusted rightmost X-Forwarded-For. Header values are joined via
StringValues.ToString() so the rightmost trusted hop wins (not a spoofable
left entry). Diagnostic only — the rate limiter independently uses the trusted
hop, so a spoofed value here cannot affect throttling.
- TenantId: agent ingest skips PolicyEnforcementMiddleware so reqCtx.TenantId was
empty. Fall back to X-Tenant-Id, but ONLY on an exact allowlist of device
endpoints that actually validate it (agent telemetry/ingest/error/distress,
bootstrap ingest/error) and only when it is a well-formed GUID — a prefix match
would let anonymous/management routes (bootstrap/validate, bootstrap/sessions,
agent/config, …) be tagged with a spoofed tenant.
RateLimitService: 429 warning is now structured ({RequestsInWindow}/{MaxRequests}
/{WindowSeconds}/{RateLimitKeyPrefix}) so it's queryable; privacy-safe (8-char
prefix only — the key may be a cert thumbprint, bootstrap token, or UPN).
ReportDistressFunction: store the real device egress IP in SourceIp (table entry
+ AgentDistressReport event) via new ClientIpExtractor.GetClientEgressIp; the
rate-limit key keeps the un-spoofable trusted hop.
Full backend suite 2249/2249 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent never sent X-Correlation-ID, so the backend's CorrelationIdMiddleware
minted a fresh id per inbound request — meaning a single logical upload that
fans out into multiple backend executions could not be tied back together (the
blocker that left the IngestTelemetry request-multiplication question
unanswerable). Wire up the correlation the infra was designed for:
- BackendTelemetryUploader: generate a unique X-Correlation-ID per upload (per
attempt, so a retry is a distinct traceable request) and log `corr=` on every
outcome line (flushing/OK/TRANSIENT/TIMEOUT/NETWORK/PERMANENT).
- BackendApiClient.AddSecurityHeaders: same per-request id for register/config/
diag/error, with a Debug line carrying corr + path.
Payoff: `requests | where name=='IngestTelemetry' | summarize count() by
tostring(customDimensions.CorrelationId)` now gives the per-request
multiplication factor deterministically (1 = clean, N = multiplied).
Test: updated the uploader logging assertion to the new `ingest OK (corr=` +
`items=N` format. Full V2.Core suite 1266/1266 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to 28885f7b. RequestStop() called UnregisterWait(), nulling
_registeredWait. When a Changed handler calls RequestStop() and then
queues Stop()/Dispose() to another thread while the callback is still
running — exactly the RealmJoinWatcher appearance-handler shape — the
queued Stop() then saw a null _registeredWait, skipped the blocking
Unregister(waitObject), and returned BEFORE the in-flight callback
finished. That defeated the Thread.Join() parity guarantee on the most
important RequestStop path.
RequestStop() now only sets _stopped and Set()s the signal, leaving the
live registration in place so Stop()/Dispose() can wait on it. The Set()
wakes any still-pending wait, which drains as a no-op (proceed is false
once _stopped is set); a callback already in flight is awaited by Stop().
No deadlock: Stop runs on a different thread and never holds _sync during
the wait.
Test: deterministic guard — park the callback inside the handler, call
RequestStop(), then assert Stop() blocks until the handler completes
(release fired 250ms later on another thread). Proven to FAIL when the
UnregisterWait() is reintroduced into RequestStop(). Full V2.Core suite
1266/1266 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oliver Kieselbach