Files
AutopilotMonitor/docs/agent/decision-engine.md
T
Oliver Kieselbach 52ae6541a7 feat(agent): signal-pipeline health block in agent_metrics_snapshot
The 2026-09-01 WG-seal soak on post-SignalIngress-fix builds (memory:
project_wg_seal_postfix_soak_residual) found sessions whose DecisionEngine
never sealed Part 1 while the agent kept running for hours - and telemetry
could not distinguish a frozen ingress worker from a deliberate Weak
sealing-classifier verdict, because the metrics snapshots carried no
pipeline counters at all.

Every agent_metrics_snapshot now carries the block. The fields are MONOTONIC
totals plus a monotonic queue peak - deliberately no momentary queue length:
a snapshot samples the integral, so any two snapshots prove whether the
worker ran in between (a frozen worker keeps signal_pending_count growing
while processed/applied stand still; a momentary gauge would alias between
ticks). Same field pattern the snapshot already uses for the spool
(spool_total_enqueued_count / spool_peak_pending_item_count).

- signal_last_ordinal / signal_processed_total (new ProcessedSignalCount) /
  signal_pending_count / signal_reentrant_total: SignalIngress worker
  progress; signal_queue_peak is a new lock-free Interlocked-max recorded in
  Post (fast + slow path; worker-local follow-up items deliberately excluded)
- decision_stage + decision_last_applied_ordinal: from the immutable
  DecisionState - answers the WG question directly ("minutes after the
  tracker's whiteglove_complete and still not WhiteGloveSealed")
- wg_sealing_level / wg_sealing_score: only after the sealing classifier
  produced at least one verdict (key ABSENCE, not Unknown/0 noise, for every
  non-WG session) - separates "classifier said Weak" from "never ran"

Wiring is a read-only probe chain following the imeTrackerHealthProbe
pattern: EnrollmentOrchestrator passes () => _processor?.CurrentState into
IComponentFactory.CreateCollectorHosts (new optional decisionStateProbe;
test fakes updated), PeriodicCollectorLifecycleHost combines it with its
existing concrete-SignalIngress reference in BuildSignalPipelineProbe, and
AgentSelfMetricsCollector writes the fields fail-soft (a throwing probe
degrades to a snapshot without the block - pinned by test). The snapshot
event travels the collector/event path, which is independent of the signal
pipeline it measures - field-proven by the freeze cases, whose perf/metrics
events kept flowing while the engine stood still.

No backend/web/MCP change needed: no new event type, DataJson is free-form.

Tests: 5 new collector facts (full block, absence semantics x2, fail-soft,
no-probe compatibility) + 2 new SignalIngress facts (processed total stable
across drain; queue peak survives the queue draining - the property a
momentary gauge would not have). Agent suite 2812 green, solution builds
with 0 warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-01 18:26:22 +02:00

17 KiB

type, title, description, resource, tags, timestamp
type title description resource tags timestamp
Reference V2 Agent — Decision Engine (DecisionCore) The pure-reducer state machine that decides enrollment completion/failure — signals, state, effects, the A/B/C completion arms, and its invariants. /src/Shared/AutopilotMonitor.DecisionCore
agent
decision-engine
reducer
completion
2026-08-30T00:00:00+02:00

V2 Agent — Decision Engine (DecisionCore)

DecisionCore (src/Shared/AutopilotMonitor.DecisionCore, netstandard2.0) is a pure reducer: a deterministic state machine with no I/O, no clock reads, no threads. The agent feeds it an ordered stream of signals; it returns a new immutable state plus a list of effects the host must execute. Because it is pure, the same persisted signal stream always reproduces the same journal and terminal outcome — that is what makes crash recovery a replay instead of a guess (determinism guarantee "L.5", Engine/IDecisionEngine.cs).

The reducer contract

DecisionStep step = engine.Reduce(DecisionState oldState, DecisionSignal signal)
// DecisionStep = { NewState, Transition, Effects[] }
  • Reduce never throws: a handler exception advances only bookkeeping and journals a reducer_exception dead-end (fail-safe L.16).
  • Dispatch is a switch on (signal.Kind, signal.KindSchemaVersion); unknown pairs become journaled dead-ends, never crashes.
  • Every step yields a DecisionTransition journal record: Taken (state change vs dead-end + reason), StepIndex, FromStage/ToStage, Trigger, Guards, ReducerVersion.

State

DecisionState (schema v4) is an immutable value object; "change" = builder copy. Key content:

  • Identity + progress: SessionId, TenantId, Stage, Outcome, StepIndex, LastAppliedSignalOrdinal, AgentBootUtc (replay-safety anchor).
  • Facts as SignalFact<T> — value plus SourceSignalOrdinal provenance, e.g. AccountSetupEnteredUtc, EspFinalExitUtc, DesktopArrivedUtc, HelloResolvedUtc, ImeUserSessionCompletedUtc, HelloPolicyEnabled, HelloOutcome.
  • Aggregates: AppInstallFacts, ScenarioProfile + ScenarioObservations (enrollment-scenario classification, monotonic), ClassifierOutcomes, RealmJoinFacts.
  • Deadlines — active timers the host has been asked to arm.

Stages: SessionStarted → EspDeviceSetup → EspAccountSetup → AwaitingHello → AwaitingDesktop → Finalizing → Completed (plus Failed, WhiteGloveCandidate/Sealed, AwaitingDeviceOnlyEsp for self-deploying, and edge stages). Terminal = Completed | Failed | WhiteGloveSealed. Finalizing is deliberately non-terminal (grace window). Outcomes: EnrollmentComplete, EnrollmentFailed, WhiteGlovePart1Sealed, Aborted, AdminPreempted.

Signals

DecisionSignal is the only input. Fields: Kind + KindSchemaVersion (dispatch key), SessionSignalOrdinal (monotonic, assigned only by the ingress worker — never by collectors), OccurredAtUtc, mandatory Evidence (classified Raw | Derived | Synthetic — a signal without evidence is a bug), Payload (string dict) and TypedPayload (structured sidecar).

Signal kinds by family (Signals/DecisionSignalKind.cs):

  • Raw observations: EspPhaseChanged, EspExiting, EspResumed, EspTerminalFailure, DesktopArrived, HelloResolved, ImeUserSessionCompleted, DeviceSetupProvisioningComplete, AccountSetupProvisioningComplete, AppInstallCompleted/Failed, AadUserJoinedLate, SystemRebootObserved, EspConfigDetected, HelloPolicyDetected, WhiteGlove + RealmJoin kinds, …
  • Synthetic: DeadlineFired (scheduler timer), ClassifierVerdictIssued, EffectInfrastructureFailure.
  • Lifecycle: SessionStarted, SessionAborted, AdminPreemptionDetected.
  • InformationalEvent — the single-rail pass-through: telemetry that must appear on the Events timeline but never influences decisions.

Ordering is the ingest ordinal, never wall-clock. Replayed IME/CMTrace lines carry backdated source timestamps; all before/after comparisons in the reducer therefore compare SourceSignalOrdinals (e.g. "final exit after AccountSetup entry").

Effects — the only side-channel

DecisionEffect kinds and how the host's EffectRunner treats failures:

Kind Purpose Failure class
ScheduleDeadline / CancelDeadline arm/cancel OS timers that come back as DeadlineFired critical — failure aborts the session (a lost timer means silent parking)
EmitEventTimelineEntry timeline events incl. terminal enrollment_complete / enrollment_failed transient — retried 100/400/1600 ms
PersistSnapshot hint to write the recovery snapshot transient
RunClassifier scenario classification; verdict returns as a ClassifierVerdictIssued signal optional — exception ⇒ Inconclusive, never aborts

Terminal and state-changing timeline effects carry a structured audit trail in TypedPayload (built by DecisionAuditTrailBuilder): decisionSource, trigger, sessionStage, stepIndex, signalsSeen, signalEvidence, signalTimestamps, scenario — so every completion/failure in the portal is explainable from its own event payload.

Completion logic

The A/B/C arms

The arms are not three pipelines — they are three alternative sufficient conditions inside one guard, ShouldTransitionToAwaitingHello (Engine/DecisionEngine.Shared.cs), which gates promotion out of the ESP stages toward Hello/desktop completion:

  • Arm A — registry-strong AccountSetup success: AccountSetupProvisioningSucceededUtc is set (the ESP registry reported categorySucceeded). Merely entering AccountSetup is insufficient — Shell-Core 62407 fires at every ESP page transition.
  • Arm B — SkipUser flow: ScenarioObservations.SkipUserEsp == true — with no User-ESP page, the first esp_exiting is the genuine final exit.
  • Arm C — 4-fact eager completion (all four mandatory):
    1. AccountSetup was entered, and
    2. a post-AccountSetup ESP final exit occurred (ordinal-ordered), and
    3. a genuine IME user-session completion (at-or-after the AccountSetup anchor — guards against the defaultuser0 ghost), and
    4. a DAD-validated real-user desktop arrival. Arm C applies eagerly the same conjunction the 30-minute AdvisoryCompletion backstop would eventually trust.

Layering note: Arm C's app-install hardening lives in the adapter, not the reducer. ImeLogTrackerAdapter defers posting ImeUserSessionCompleted while required user-ESP Install-intent apps are still pending (IME logs "Completed user session" per pass, not once per session — sessions 14690fc2/6cb01530). The reducer stays free of app-pending guards; adapters sanitize evidence before it becomes a signal.

Zero user-targeted apps (sessions 81daa77f/75d6ae8e): IME never writes its "Completed user session" line at all when the user has no assigned apps — its zero-app branch returns first — so fact 3 was unreachable by construction and every such device fell through to the backstop. IME-USER-SESSION-ZERO-APPS reads the line IME does write there and posts the same signal, guarded to the AccountSetup phase. The event carries an evidence field (zero_user_apps_enumerated vs user_session_completed) naming which observation carried the completion. See zero-user-apps-completion-gate.md.

Completion attempt sites and sequence

Completion can be attempted from whichever fact arrives last: HelloResolved, DesktopArrived (including the Hello-disabled fast-path when HelloPolicyEnabled == false and an arm holds), EspExiting, ImeUserSessionCompleted, AccountSetupProvisioningComplete, the Hello safety deadline, or the 30-minute advisory backstop. All routes converge on CompleteThroughFinalizingOrDefer:

  1. Completion gates (currently only the RealmJoin AND-gate): if RealmJoin was detected and has no terminal outcome yet, completion is deferred — the session stays in stage and emits completion_waiting. Three terminals open the gate: Resolved (DeploymentPhase 110), FirstDeploymentIncomplete (phase left the first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted, typically by an interactive logon reclassifying the run as secondary-user; emits the realmjoin_first_deployment_incomplete Warning, session 224b2087), and Timeout (60-min deadline from detection). The abort rule reads the persisted RealmJoinFacts.LastDeploymentPhase (updated by the typed RealmJoinPhaseChanged signal), so it survives agent restarts — the RealmJoinDetected replay path evaluates it too. A first observation of 210 does not release (RJ may have completed before agent boot and deploy afterwards).

    Activity-based timeout extension (report 55e6afd61c9d — a large RJ package catalog outlived the 60-min window mid-install): when the timeout deadline fires while the first deployment is demonstrably active — LastDeploymentPhase 100/101 and deployment activity (RealmJoinPhaseChanged or a package observation, tracked in RealmJoinFacts.LastActivityUtc; the phase captured at detection deliberately does not count) within the last 60 min — the handler re-arms the deadline to lastActivity + 60 min instead of timing out, emitting an Info realmjoin_timeout_extended event. Re-arms clamp to an absolute cap of detection + 4 h. Package activity is only observable at package completion (RJ writes the registry subkey when the install finishes), so a single install that is registry-silent for over 60 min still times out — the eventual realmjoin_timeout carries a reason payload (hard_timeout / inactivity / absolute_cap) telling the three shapes apart. A stale fire of an older deadline incarnation (armed due later than the fire time; OccurredAtUtc = DueAtUtc per scheduler contract) dead-ends as realmjoin_timeout_stale_superseded_by_rearm. The idle case (detected, no deployment activity ever) times out at 60 min unchanged, and phase 200/210 activity never extends. Self-update observation (gktatooine session 946ccbd6): the version on RealmJoinDetected is read once from RealmJoin.exe, but the tray's AppUpdater often replaces the MSI-shipped binary on its first start — minutes after detection, before the deployment begins. RealmJoinWatcher therefore arms a RegNotifyChangeKeyValue watcher on HKLM\...\Uninstall\RealmJoin (the only registry write of the updater; appearance-watch on the Uninstall parent when the entry does not exist yet) once Detected fired, re-reads the binary version after a 2 s debounce (registry DisplayVersion as fallback while the file is locked) and posts the typed RealmJoinAutoUpdateDetected signal plus the realmjoin_autoupdate_detected timeline event when the version differs from the baseline. The reducer overrides RealmJoinFacts.ProductVersion / ReleaseChannel (WithVersionOverride — the one legitimate later write past the set-once helpers); it never touches the gate, the deadline, or LastActivityUtc. An unreadable baseline at detection is seeded silently by the first successful re-read. A closed gate also outranks the 30-minute AdvisoryCompletion backstop: that handler re-arms (trigger suffix :CompletionGateHolding) instead of failing while the RealmJoin gate is closed and its own deadline is still armed, because the gate carries a bounded resolution of its own. Before this, the shorter timer always won — session 75d6ae8e was failed 2 min after its 25th successful RealmJoin package.

  2. Finalizing stage + ~5 s FinalizingGrace deadline.

  3. Grace fires → Completed, EnrollmentComplete outcome, enrollment_complete timeline event with full audit payload.

Self-deploying/kiosk is a separate rail: no Hello, no user desktop — AwaitingDeviceOnlyEsp with a detection deadline as the sole terminal entry, protected by stale-fire and race guards. A registry-confirmed self-deploying profile also waives IME false-positive AccountSetup entries.

Invariants (violating any of these is a bug)

  • Post-terminal dispatch guard: after Completed/Failed, every signal is a bookkept dead-end — except InformationalEvent (single-rail pass-through must keep flowing). WhiteGloveSealed is unguarded on purpose (Part 2 resumes fresh).
  • InformationalEvent never mutates decision state — exactly one pass-through timeline effect, payload copied 1:1.
  • AadUserJoinedLate is observation-only — never completion evidence.
  • Hello policy is orthogonal to completion — HelloPolicyEnabled tunes wait cadence, never gates the outcome.
  • Raw IME "Completed user session" is weak evidence — can run under defaultuser0; only meaningful with a real-user desktop and the AccountSetup anchor (Arm C).
  • Deadline bases are floored at AgentBootUtc — replayed backdated signals must not arm past-due timers at boot.
  • Reducer-side deadline cancels need an explicit CancelDeadline effect — a builder-only removal leaves the OS timer live.
  • No silent parking: a non-terminal post-AccountSetup session must always hold a resolution-capable deadline; a 10-min dwell tripwire emits session_parked_without_deadline.
  • Every payload-derived string stored into state is bounded (FactStringBounds, 256 chars): RealmJoin package id/version/scope (scope normalised to machine/user), failed app ids, HelloOutcome, ImeMatchedPatternId, EspAdvisoryFailureCategory, RealmJoin product version/channel. The bound is applied before any dedupe comparison so a forged multi-hundred-KB key neither costs a full-length compare per stored row nor lands in snapshots/audit trails. It never truncates real telemetry — the sources are registry key names (Windows caps them at 255), GUIDs and enum-like strings. The same engine replays device-uploaded signals server-side (ReducerVerifier), which is why the bound lives in the reducer, not in the agent adapter alone.

Hosting & threading (inside the agent)

Wired by EnrollmentOrchestrator.Start (src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/):

  • SignalIngress — bounded queue (256, back-pressure) + one worker thread. Per signal, in order: assign ordinals → SignalLog.Append (durable, before reducing; append failure ⇒ reducer does not run, no ordinal gap) → engine.Reduce → DecisionStepProcessor.ApplyStep. A Post issued by the worker thread itself (the EffectRunner posting ClassifierVerdictIssued from inside ApplyStep) never enters the bounded queue: it goes to a worker-local follow-up queue that is drained, ahead of the channel, right after the current item — the worker is the queue's only consumer, so blocking on a full channel there would freeze the pipeline for the rest of the run.
  • DecisionStepProcessor.ApplyStep (same thread, no locks): journal append (the only hard-throwing path; 3 consecutive failures escalate to quarantine) → effects run synchronously via EffectRunner → best-effort snapshot → advance CurrentState → one-shot terminal-stage hook (raises the agent's Terminated event off-thread).
  • The DeadlineScheduler's Fired event loops back in as synthetic DeadlineFired signals.
  • Pipeline health in agent_metrics_snapshot (2026-09-01, motivated by the WG-seal soak in which a frozen engine and a deliberate Weak sealing verdict were telemetrically indistinguishable): every snapshot carries monotonic totals from SignalIngress (signal_last_ordinal, signal_processed_total, signal_pending_count, signal_reentrant_total, signal_queue_peak — a monotonic peak, deliberately no momentary queue length, so snapshot sampling cannot alias it) plus decision_stage, decision_last_applied_ordinal and — once the sealing classifier has produced a verdict — wg_sealing_level/wg_sealing_score from the immutable DecisionState. Two snapshots therefore prove whether the worker ran in between: a frozen worker keeps signal_pending_count growing while the processed/applied counters stand still. The probe chain is read-only (EnrollmentOrchestrator → IComponentFactory. CreateCollectorHosts(decisionStateProbe) → PeriodicCollectorLifecycleHost → AgentSelfMetricsCollector); the snapshot event itself travels the collector/event path, which is independent of the signal pipeline it measures — field-proven by the freeze cases, whose perf/metrics events kept flowing while the engine stood still.
  • On restart, RecoveryCoordinator folds the persisted signal log through a transient new DecisionEngine() before the live pipeline starts — see logs & persistence.