The 2026-09-01 WG-seal soak on post-SignalIngress-fix builds (memory:
project_wg_seal_postfix_soak_residual) found sessions whose DecisionEngine
never sealed Part 1 while the agent kept running for hours - and telemetry
could not distinguish a frozen ingress worker from a deliberate Weak
sealing-classifier verdict, because the metrics snapshots carried no
pipeline counters at all.
Every agent_metrics_snapshot now carries the block. The fields are MONOTONIC
totals plus a monotonic queue peak - deliberately no momentary queue length:
a snapshot samples the integral, so any two snapshots prove whether the
worker ran in between (a frozen worker keeps signal_pending_count growing
while processed/applied stand still; a momentary gauge would alias between
ticks). Same field pattern the snapshot already uses for the spool
(spool_total_enqueued_count / spool_peak_pending_item_count).
- signal_last_ordinal / signal_processed_total (new ProcessedSignalCount) /
signal_pending_count / signal_reentrant_total: SignalIngress worker
progress; signal_queue_peak is a new lock-free Interlocked-max recorded in
Post (fast + slow path; worker-local follow-up items deliberately excluded)
- decision_stage + decision_last_applied_ordinal: from the immutable
DecisionState - answers the WG question directly ("minutes after the
tracker's whiteglove_complete and still not WhiteGloveSealed")
- wg_sealing_level / wg_sealing_score: only after the sealing classifier
produced at least one verdict (key ABSENCE, not Unknown/0 noise, for every
non-WG session) - separates "classifier said Weak" from "never ran"
Wiring is a read-only probe chain following the imeTrackerHealthProbe
pattern: EnrollmentOrchestrator passes () => _processor?.CurrentState into
IComponentFactory.CreateCollectorHosts (new optional decisionStateProbe;
test fakes updated), PeriodicCollectorLifecycleHost combines it with its
existing concrete-SignalIngress reference in BuildSignalPipelineProbe, and
AgentSelfMetricsCollector writes the fields fail-soft (a throwing probe
degrades to a snapshot without the block - pinned by test). The snapshot
event travels the collector/event path, which is independent of the signal
pipeline it measures - field-proven by the freeze cases, whose perf/metrics
events kept flowing while the engine stood still.
No backend/web/MCP change needed: no new event type, DataJson is free-form.
Tests: 5 new collector facts (full block, absence semantics x2, fail-soft,
no-probe compatibility) + 2 new SignalIngress facts (processed total stable
across drain; queue peak survives the queue draining - the property a
momentary gauge would not have). Agent suite 2812 green, solution builds
with 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 KiB
type, title, description, resource, tags, timestamp
| type | title | description | resource | tags | timestamp | ||||
|---|---|---|---|---|---|---|---|---|---|
| Reference | V2 Agent — Decision Engine (DecisionCore) | The pure-reducer state machine that decides enrollment completion/failure — signals, state, effects, the A/B/C completion arms, and its invariants. | /src/Shared/AutopilotMonitor.DecisionCore |
|
2026-08-30T00:00:00+02:00 |
V2 Agent — Decision Engine (DecisionCore)
DecisionCore (src/Shared/AutopilotMonitor.DecisionCore, netstandard2.0) is a pure
reducer: a deterministic state machine with no I/O, no clock reads, no threads. The
agent feeds it an ordered stream of signals; it returns a new immutable state plus a
list of effects the host must execute. Because it is pure, the same persisted signal
stream always reproduces the same journal and terminal outcome — that is what makes
crash recovery a replay instead of a guess (determinism guarantee "L.5",
Engine/IDecisionEngine.cs).
The reducer contract
DecisionStep step = engine.Reduce(DecisionState oldState, DecisionSignal signal)
// DecisionStep = { NewState, Transition, Effects[] }
Reducenever throws: a handler exception advances only bookkeeping and journals areducer_exceptiondead-end (fail-safe L.16).- Dispatch is a switch on
(signal.Kind, signal.KindSchemaVersion); unknown pairs become journaled dead-ends, never crashes. - Every step yields a
DecisionTransitionjournal record:Taken(state change vs dead-end + reason),StepIndex,FromStage/ToStage,Trigger,Guards,ReducerVersion.
State
DecisionState (schema v4) is an immutable value object; "change" = builder copy.
Key content:
- Identity + progress:
SessionId,TenantId,Stage,Outcome,StepIndex,LastAppliedSignalOrdinal,AgentBootUtc(replay-safety anchor). - Facts as
SignalFact<T>— value plusSourceSignalOrdinalprovenance, e.g.AccountSetupEnteredUtc,EspFinalExitUtc,DesktopArrivedUtc,HelloResolvedUtc,ImeUserSessionCompletedUtc,HelloPolicyEnabled,HelloOutcome. - Aggregates:
AppInstallFacts,ScenarioProfile+ScenarioObservations(enrollment-scenario classification, monotonic),ClassifierOutcomes,RealmJoinFacts. Deadlines— active timers the host has been asked to arm.
Stages: SessionStarted → EspDeviceSetup → EspAccountSetup → AwaitingHello → AwaitingDesktop → Finalizing → Completed (plus Failed, WhiteGloveCandidate/Sealed,
AwaitingDeviceOnlyEsp for self-deploying, and edge stages). Terminal =
Completed | Failed | WhiteGloveSealed. Finalizing is deliberately non-terminal
(grace window). Outcomes: EnrollmentComplete, EnrollmentFailed,
WhiteGlovePart1Sealed, Aborted, AdminPreempted.
Signals
DecisionSignal is the only input. Fields: Kind + KindSchemaVersion (dispatch key),
SessionSignalOrdinal (monotonic, assigned only by the ingress worker — never by
collectors), OccurredAtUtc, mandatory Evidence (classified Raw | Derived | Synthetic — a signal without evidence is a bug), Payload (string dict) and
TypedPayload (structured sidecar).
Signal kinds by family (Signals/DecisionSignalKind.cs):
- Raw observations:
EspPhaseChanged,EspExiting,EspResumed,EspTerminalFailure,DesktopArrived,HelloResolved,ImeUserSessionCompleted,DeviceSetupProvisioningComplete,AccountSetupProvisioningComplete,AppInstallCompleted/Failed,AadUserJoinedLate,SystemRebootObserved,EspConfigDetected,HelloPolicyDetected, WhiteGlove + RealmJoin kinds, … - Synthetic:
DeadlineFired(scheduler timer),ClassifierVerdictIssued,EffectInfrastructureFailure. - Lifecycle:
SessionStarted,SessionAborted,AdminPreemptionDetected. InformationalEvent— the single-rail pass-through: telemetry that must appear on the Events timeline but never influences decisions.
Ordering is the ingest ordinal, never wall-clock. Replayed IME/CMTrace lines carry
backdated source timestamps; all before/after comparisons in the reducer therefore
compare SourceSignalOrdinals (e.g. "final exit after AccountSetup entry").
Effects — the only side-channel
DecisionEffect kinds and how the host's EffectRunner treats failures:
| Kind | Purpose | Failure class |
|---|---|---|
ScheduleDeadline / CancelDeadline |
arm/cancel OS timers that come back as DeadlineFired |
critical — failure aborts the session (a lost timer means silent parking) |
EmitEventTimelineEntry |
timeline events incl. terminal enrollment_complete / enrollment_failed |
transient — retried 100/400/1600 ms |
PersistSnapshot |
hint to write the recovery snapshot | transient |
RunClassifier |
scenario classification; verdict returns as a ClassifierVerdictIssued signal |
optional — exception ⇒ Inconclusive, never aborts |
Terminal and state-changing timeline effects carry a structured audit trail in
TypedPayload (built by DecisionAuditTrailBuilder): decisionSource, trigger,
sessionStage, stepIndex, signalsSeen, signalEvidence, signalTimestamps,
scenario — so every completion/failure in the portal is explainable from its own
event payload.
Completion logic
The A/B/C arms
The arms are not three pipelines — they are three alternative sufficient conditions
inside one guard, ShouldTransitionToAwaitingHello
(Engine/DecisionEngine.Shared.cs), which gates promotion out of the ESP stages toward
Hello/desktop completion:
- Arm A — registry-strong AccountSetup success:
AccountSetupProvisioningSucceededUtcis set (the ESP registry reportedcategorySucceeded). Merely entering AccountSetup is insufficient — Shell-Core 62407 fires at every ESP page transition. - Arm B — SkipUser flow:
ScenarioObservations.SkipUserEsp == true— with no User-ESP page, the firstesp_exitingis the genuine final exit. - Arm C — 4-fact eager completion (all four mandatory):
- AccountSetup was entered, and
- a post-AccountSetup ESP final exit occurred (ordinal-ordered), and
- a genuine IME user-session completion (at-or-after the AccountSetup anchor —
guards against the
defaultuser0ghost), and - a DAD-validated real-user desktop arrival.
Arm C applies eagerly the same conjunction the 30-minute
AdvisoryCompletionbackstop would eventually trust.
Layering note: Arm C's app-install hardening lives in the adapter, not the
reducer. ImeLogTrackerAdapter defers posting ImeUserSessionCompleted while required
user-ESP Install-intent apps are still pending (IME logs "Completed user session" per
pass, not once per session — sessions 14690fc2/6cb01530). The reducer stays free of
app-pending guards; adapters sanitize evidence before it becomes a signal.
Zero user-targeted apps (sessions 81daa77f/75d6ae8e): IME never writes its
"Completed user session" line at all when the user has no assigned apps — its zero-app
branch returns first — so fact 3 was unreachable by construction and every such device
fell through to the backstop. IME-USER-SESSION-ZERO-APPS reads the line IME does
write there and posts the same signal, guarded to the AccountSetup phase. The event
carries an evidence field (zero_user_apps_enumerated vs user_session_completed)
naming which observation carried the completion. See
zero-user-apps-completion-gate.md.
Completion attempt sites and sequence
Completion can be attempted from whichever fact arrives last: HelloResolved,
DesktopArrived (including the Hello-disabled fast-path when HelloPolicyEnabled == false and an arm holds), EspExiting, ImeUserSessionCompleted,
AccountSetupProvisioningComplete, the Hello safety deadline, or the 30-minute
advisory backstop. All routes converge on CompleteThroughFinalizingOrDefer:
-
Completion gates (currently only the RealmJoin AND-gate): if RealmJoin was detected and has no terminal outcome yet, completion is deferred — the session stays in stage and emits
completion_waiting. Three terminals open the gate:Resolved(DeploymentPhase 110),FirstDeploymentIncomplete(phase left the first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted, typically by an interactive logon reclassifying the run as secondary-user; emits therealmjoin_first_deployment_incompleteWarning, session 224b2087), andTimeout(60-min deadline from detection). The abort rule reads the persistedRealmJoinFacts.LastDeploymentPhase(updated by the typedRealmJoinPhaseChangedsignal), so it survives agent restarts — theRealmJoinDetectedreplay path evaluates it too. A first observation of 210 does not release (RJ may have completed before agent boot and deploy afterwards).Activity-based timeout extension (report 55e6afd61c9d — a large RJ package catalog outlived the 60-min window mid-install): when the timeout deadline fires while the first deployment is demonstrably active —
LastDeploymentPhase100/101 and deployment activity (RealmJoinPhaseChangedor a package observation, tracked inRealmJoinFacts.LastActivityUtc; the phase captured at detection deliberately does not count) within the last 60 min — the handler re-arms the deadline tolastActivity + 60 mininstead of timing out, emitting an Inforealmjoin_timeout_extendedevent. Re-arms clamp to an absolute cap of detection + 4 h. Package activity is only observable at package completion (RJ writes the registry subkey when the install finishes), so a single install that is registry-silent for over 60 min still times out — the eventualrealmjoin_timeoutcarries areasonpayload (hard_timeout/inactivity/absolute_cap) telling the three shapes apart. A stale fire of an older deadline incarnation (armed due later than the fire time;OccurredAtUtc = DueAtUtcper scheduler contract) dead-ends asrealmjoin_timeout_stale_superseded_by_rearm. The idle case (detected, no deployment activity ever) times out at 60 min unchanged, and phase 200/210 activity never extends. Self-update observation (gktatooine session 946ccbd6): the version onRealmJoinDetectedis read once fromRealmJoin.exe, but the tray'sAppUpdateroften replaces the MSI-shipped binary on its first start — minutes after detection, before the deployment begins.RealmJoinWatchertherefore arms aRegNotifyChangeKeyValuewatcher onHKLM\...\Uninstall\RealmJoin(the only registry write of the updater; appearance-watch on theUninstallparent when the entry does not exist yet) once Detected fired, re-reads the binary version after a 2 s debounce (registryDisplayVersionas fallback while the file is locked) and posts the typedRealmJoinAutoUpdateDetectedsignal plus therealmjoin_autoupdate_detectedtimeline event when the version differs from the baseline. The reducer overridesRealmJoinFacts.ProductVersion/ReleaseChannel(WithVersionOverride— the one legitimate later write past the set-once helpers); it never touches the gate, the deadline, orLastActivityUtc. An unreadable baseline at detection is seeded silently by the first successful re-read. A closed gate also outranks the 30-minuteAdvisoryCompletionbackstop: that handler re-arms (trigger suffix:CompletionGateHolding) instead of failing while the RealmJoin gate is closed and its own deadline is still armed, because the gate carries a bounded resolution of its own. Before this, the shorter timer always won — session 75d6ae8e was failed 2 min after its 25th successful RealmJoin package. -
Finalizingstage + ~5 sFinalizingGracedeadline. -
Grace fires →
Completed,EnrollmentCompleteoutcome,enrollment_completetimeline event with full audit payload.
Self-deploying/kiosk is a separate rail: no Hello, no user desktop —
AwaitingDeviceOnlyEsp with a detection deadline as the sole terminal entry, protected
by stale-fire and race guards. A registry-confirmed self-deploying profile also waives
IME false-positive AccountSetup entries.
Invariants (violating any of these is a bug)
- Post-terminal dispatch guard: after
Completed/Failed, every signal is a bookkept dead-end — exceptInformationalEvent(single-rail pass-through must keep flowing).WhiteGloveSealedis unguarded on purpose (Part 2 resumes fresh). InformationalEventnever mutates decision state — exactly one pass-through timeline effect, payload copied 1:1.AadUserJoinedLateis observation-only — never completion evidence.- Hello policy is orthogonal to completion —
HelloPolicyEnabledtunes wait cadence, never gates the outcome. - Raw IME "Completed user session" is weak evidence — can run under
defaultuser0; only meaningful with a real-user desktop and the AccountSetup anchor (Arm C). - Deadline bases are floored at
AgentBootUtc— replayed backdated signals must not arm past-due timers at boot. - Reducer-side deadline cancels need an explicit
CancelDeadlineeffect — a builder-only removal leaves the OS timer live. - No silent parking: a non-terminal post-AccountSetup session must always hold a
resolution-capable deadline; a 10-min dwell tripwire emits
session_parked_without_deadline. - Every payload-derived string stored into state is bounded (
FactStringBounds, 256 chars): RealmJoin package id/version/scope (scope normalised tomachine/user), failed app ids,HelloOutcome,ImeMatchedPatternId,EspAdvisoryFailureCategory, RealmJoin product version/channel. The bound is applied before any dedupe comparison so a forged multi-hundred-KB key neither costs a full-length compare per stored row nor lands in snapshots/audit trails. It never truncates real telemetry — the sources are registry key names (Windows caps them at 255), GUIDs and enum-like strings. The same engine replays device-uploaded signals server-side (ReducerVerifier), which is why the bound lives in the reducer, not in the agent adapter alone.
Hosting & threading (inside the agent)
Wired by EnrollmentOrchestrator.Start (src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/):
SignalIngress— bounded queue (256, back-pressure) + one worker thread. Per signal, in order: assign ordinals →SignalLog.Append(durable, before reducing; append failure ⇒ reducer does not run, no ordinal gap) →engine.Reduce→DecisionStepProcessor.ApplyStep. APostissued by the worker thread itself (theEffectRunnerpostingClassifierVerdictIssuedfrom insideApplyStep) never enters the bounded queue: it goes to a worker-local follow-up queue that is drained, ahead of the channel, right after the current item — the worker is the queue's only consumer, so blocking on a full channel there would freeze the pipeline for the rest of the run.DecisionStepProcessor.ApplyStep(same thread, no locks): journal append (the only hard-throwing path; 3 consecutive failures escalate to quarantine) → effects run synchronously viaEffectRunner→ best-effort snapshot → advanceCurrentState→ one-shot terminal-stage hook (raises the agent'sTerminatedevent off-thread).- The
DeadlineScheduler'sFiredevent loops back in as syntheticDeadlineFiredsignals. - Pipeline health in
agent_metrics_snapshot(2026-09-01, motivated by the WG-seal soak in which a frozen engine and a deliberate Weak sealing verdict were telemetrically indistinguishable): every snapshot carries monotonic totals fromSignalIngress(signal_last_ordinal,signal_processed_total,signal_pending_count,signal_reentrant_total,signal_queue_peak— a monotonic peak, deliberately no momentary queue length, so snapshot sampling cannot alias it) plusdecision_stage,decision_last_applied_ordinaland — once the sealing classifier has produced a verdict —wg_sealing_level/wg_sealing_scorefrom the immutableDecisionState. Two snapshots therefore prove whether the worker ran in between: a frozen worker keepssignal_pending_countgrowing while the processed/applied counters stand still. The probe chain is read-only (EnrollmentOrchestrator→IComponentFactory. CreateCollectorHosts(decisionStateProbe)→PeriodicCollectorLifecycleHost→AgentSelfMetricsCollector); the snapshot event itself travels the collector/event path, which is independent of the signal pipeline it measures — field-proven by the freeze cases, whose perf/metrics events kept flowing while the engine stood still. - On restart,
RecoveryCoordinatorfolds the persisted signal log through a transientnew DecisionEngine()before the live pipeline starts — see logs & persistence.