mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 12:42:38 +00:00
52ae6541a7
The 2026-09-01 WG-seal soak on post-SignalIngress-fix builds (memory:
project_wg_seal_postfix_soak_residual) found sessions whose DecisionEngine
never sealed Part 1 while the agent kept running for hours - and telemetry
could not distinguish a frozen ingress worker from a deliberate Weak
sealing-classifier verdict, because the metrics snapshots carried no
pipeline counters at all.
Every agent_metrics_snapshot now carries the block. The fields are MONOTONIC
totals plus a monotonic queue peak - deliberately no momentary queue length:
a snapshot samples the integral, so any two snapshots prove whether the
worker ran in between (a frozen worker keeps signal_pending_count growing
while processed/applied stand still; a momentary gauge would alias between
ticks). Same field pattern the snapshot already uses for the spool
(spool_total_enqueued_count / spool_peak_pending_item_count).
- signal_last_ordinal / signal_processed_total (new ProcessedSignalCount) /
signal_pending_count / signal_reentrant_total: SignalIngress worker
progress; signal_queue_peak is a new lock-free Interlocked-max recorded in
Post (fast + slow path; worker-local follow-up items deliberately excluded)
- decision_stage + decision_last_applied_ordinal: from the immutable
DecisionState - answers the WG question directly ("minutes after the
tracker's whiteglove_complete and still not WhiteGloveSealed")
- wg_sealing_level / wg_sealing_score: only after the sealing classifier
produced at least one verdict (key ABSENCE, not Unknown/0 noise, for every
non-WG session) - separates "classifier said Weak" from "never ran"
Wiring is a read-only probe chain following the imeTrackerHealthProbe
pattern: EnrollmentOrchestrator passes () => _processor?.CurrentState into
IComponentFactory.CreateCollectorHosts (new optional decisionStateProbe;
test fakes updated), PeriodicCollectorLifecycleHost combines it with its
existing concrete-SignalIngress reference in BuildSignalPipelineProbe, and
AgentSelfMetricsCollector writes the fields fail-soft (a throwing probe
degrades to a snapshot without the block - pinned by test). The snapshot
event travels the collector/event path, which is independent of the signal
pipeline it measures - field-proven by the freeze cases, whose perf/metrics
events kept flowing while the engine stood still.
No backend/web/MCP change needed: no new event type, DataJson is free-form.
Tests: 5 new collector facts (full block, absence semantics x2, fail-soft,
no-probe compatibility) + 2 new SignalIngress facts (processed total stable
across drain; queue peak survives the queue draining - the property a
momentary gauge would not have). Agent suite 2812 green, solution builds
with 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
272 lines
17 KiB
Markdown
272 lines
17 KiB
Markdown
---
|
|
type: Reference
|
|
title: V2 Agent — Decision Engine (DecisionCore)
|
|
description: The pure-reducer state machine that decides enrollment completion/failure — signals, state, effects, the A/B/C completion arms, and its invariants.
|
|
resource: /src/Shared/AutopilotMonitor.DecisionCore
|
|
tags:
|
|
- agent
|
|
- decision-engine
|
|
- reducer
|
|
- completion
|
|
timestamp: 2026-08-30T00:00:00+02:00
|
|
---
|
|
|
|
# V2 Agent — Decision Engine (DecisionCore)
|
|
|
|
DecisionCore (`src/Shared/AutopilotMonitor.DecisionCore`, netstandard2.0) is a **pure
|
|
reducer**: a deterministic state machine with no I/O, no clock reads, no threads. The
|
|
agent feeds it an ordered stream of *signals*; it returns a new immutable state plus a
|
|
list of *effects* the host must execute. Because it is pure, the same persisted signal
|
|
stream always reproduces the same journal and terminal outcome — that is what makes
|
|
crash recovery a replay instead of a guess (determinism guarantee "L.5",
|
|
`Engine/IDecisionEngine.cs`).
|
|
|
|
# The reducer contract
|
|
|
|
```
|
|
DecisionStep step = engine.Reduce(DecisionState oldState, DecisionSignal signal)
|
|
// DecisionStep = { NewState, Transition, Effects[] }
|
|
```
|
|
|
|
- `Reduce` never throws: a handler exception advances only bookkeeping and journals a
|
|
`reducer_exception` dead-end (fail-safe L.16).
|
|
- Dispatch is a switch on `(signal.Kind, signal.KindSchemaVersion)`; unknown pairs
|
|
become journaled dead-ends, never crashes.
|
|
- Every step yields a `DecisionTransition` journal record: `Taken` (state change vs
|
|
dead-end + reason), `StepIndex`, `FromStage/ToStage`, `Trigger`, `Guards`,
|
|
`ReducerVersion`.
|
|
|
|
# State
|
|
|
|
`DecisionState` (schema **v4**) is an immutable value object; "change" = builder copy.
|
|
Key content:
|
|
|
|
- Identity + progress: `SessionId`, `TenantId`, `Stage`, `Outcome`, `StepIndex`,
|
|
`LastAppliedSignalOrdinal`, `AgentBootUtc` (replay-safety anchor).
|
|
- **Facts** as `SignalFact<T>` — value **plus** `SourceSignalOrdinal` provenance, e.g.
|
|
`AccountSetupEnteredUtc`, `EspFinalExitUtc`, `DesktopArrivedUtc`, `HelloResolvedUtc`,
|
|
`ImeUserSessionCompletedUtc`, `HelloPolicyEnabled`, `HelloOutcome`.
|
|
- Aggregates: `AppInstallFacts`, `ScenarioProfile` + `ScenarioObservations`
|
|
(enrollment-scenario classification, monotonic), `ClassifierOutcomes`, `RealmJoinFacts`.
|
|
- `Deadlines` — active timers the host has been asked to arm.
|
|
|
|
Stages: `SessionStarted → EspDeviceSetup → EspAccountSetup → AwaitingHello →
|
|
AwaitingDesktop → Finalizing → Completed` (plus `Failed`, `WhiteGloveCandidate/Sealed`,
|
|
`AwaitingDeviceOnlyEsp` for self-deploying, and edge stages). Terminal =
|
|
`Completed | Failed | WhiteGloveSealed`. `Finalizing` is deliberately non-terminal
|
|
(grace window). Outcomes: `EnrollmentComplete`, `EnrollmentFailed`,
|
|
`WhiteGlovePart1Sealed`, `Aborted`, `AdminPreempted`.
|
|
|
|
# Signals
|
|
|
|
`DecisionSignal` is the only input. Fields: `Kind` + `KindSchemaVersion` (dispatch key),
|
|
`SessionSignalOrdinal` (monotonic, assigned **only** by the ingress worker — never by
|
|
collectors), `OccurredAtUtc`, mandatory `Evidence` (classified `Raw | Derived |
|
|
Synthetic` — a signal without evidence is a bug), `Payload` (string dict) and
|
|
`TypedPayload` (structured sidecar).
|
|
|
|
Signal kinds by family (`Signals/DecisionSignalKind.cs`):
|
|
|
|
- **Raw observations**: `EspPhaseChanged`, `EspExiting`, `EspResumed`,
|
|
`EspTerminalFailure`, `DesktopArrived`, `HelloResolved`, `ImeUserSessionCompleted`,
|
|
`DeviceSetupProvisioningComplete`, `AccountSetupProvisioningComplete`,
|
|
`AppInstallCompleted/Failed`, `AadUserJoinedLate`, `SystemRebootObserved`,
|
|
`EspConfigDetected`, `HelloPolicyDetected`, WhiteGlove + RealmJoin kinds, …
|
|
- **Synthetic**: `DeadlineFired` (scheduler timer), `ClassifierVerdictIssued`,
|
|
`EffectInfrastructureFailure`.
|
|
- **Lifecycle**: `SessionStarted`, `SessionAborted`, `AdminPreemptionDetected`.
|
|
- **`InformationalEvent`** — the single-rail pass-through: telemetry that must appear
|
|
on the Events timeline but never influences decisions.
|
|
|
|
**Ordering is the ingest ordinal, never wall-clock.** Replayed IME/CMTrace lines carry
|
|
backdated source timestamps; all before/after comparisons in the reducer therefore
|
|
compare `SourceSignalOrdinal`s (e.g. "final exit after AccountSetup entry").
|
|
|
|
# Effects — the only side-channel
|
|
|
|
`DecisionEffect` kinds and how the host's `EffectRunner` treats failures:
|
|
|
|
| Kind | Purpose | Failure class |
|
|
| --- | --- | --- |
|
|
| `ScheduleDeadline` / `CancelDeadline` | arm/cancel OS timers that come back as `DeadlineFired` | **critical** — failure aborts the session (a lost timer means silent parking) |
|
|
| `EmitEventTimelineEntry` | timeline events incl. terminal `enrollment_complete` / `enrollment_failed` | transient — retried 100/400/1600 ms |
|
|
| `PersistSnapshot` | hint to write the recovery snapshot | transient |
|
|
| `RunClassifier` | scenario classification; verdict returns as a `ClassifierVerdictIssued` signal | optional — exception ⇒ Inconclusive, never aborts |
|
|
|
|
Terminal and state-changing timeline effects carry a structured **audit trail** in
|
|
`TypedPayload` (built by `DecisionAuditTrailBuilder`): `decisionSource`, `trigger`,
|
|
`sessionStage`, `stepIndex`, `signalsSeen`, `signalEvidence`, `signalTimestamps`,
|
|
`scenario` — so every completion/failure in the portal is explainable from its own
|
|
event payload.
|
|
|
|
# Completion logic
|
|
|
|
## The A/B/C arms
|
|
|
|
The arms are **not** three pipelines — they are three alternative sufficient conditions
|
|
inside one guard, `ShouldTransitionToAwaitingHello`
|
|
(`Engine/DecisionEngine.Shared.cs`), which gates promotion out of the ESP stages toward
|
|
Hello/desktop completion:
|
|
|
|
- **Arm A — registry-strong AccountSetup success**: `AccountSetupProvisioningSucceededUtc`
|
|
is set (the ESP registry reported `categorySucceeded`). Merely *entering* AccountSetup
|
|
is insufficient — Shell-Core 62407 fires at every ESP page transition.
|
|
- **Arm B — SkipUser flow**: `ScenarioObservations.SkipUserEsp == true` — with no
|
|
User-ESP page, the first `esp_exiting` *is* the genuine final exit.
|
|
- **Arm C — 4-fact eager completion** (all four mandatory):
|
|
1. AccountSetup was entered, and
|
|
2. a post-AccountSetup ESP final exit occurred (ordinal-ordered), and
|
|
3. a *genuine* IME user-session completion (at-or-after the AccountSetup anchor —
|
|
guards against the `defaultuser0` ghost), and
|
|
4. a DAD-validated real-user desktop arrival.
|
|
Arm C applies eagerly the same conjunction the 30-minute `AdvisoryCompletion`
|
|
backstop would eventually trust.
|
|
|
|
**Layering note:** Arm C's app-install hardening lives in the **adapter**, not the
|
|
reducer. `ImeLogTrackerAdapter` defers posting `ImeUserSessionCompleted` while required
|
|
user-ESP Install-intent apps are still pending (IME logs "Completed user session" per
|
|
*pass*, not once per session — sessions 14690fc2/6cb01530). The reducer stays free of
|
|
app-pending guards; adapters sanitize evidence before it becomes a signal.
|
|
|
|
**Zero user-targeted apps** (sessions 81daa77f/75d6ae8e): IME never writes its
|
|
"Completed user session" line at all when the user has no assigned apps — its zero-app
|
|
branch returns first — so fact 3 was unreachable by construction and every such device
|
|
fell through to the backstop. `IME-USER-SESSION-ZERO-APPS` reads the line IME *does*
|
|
write there and posts the same signal, guarded to the AccountSetup phase. The event
|
|
carries an `evidence` field (`zero_user_apps_enumerated` vs `user_session_completed`)
|
|
naming which observation carried the completion. See
|
|
[zero-user-apps-completion-gate.md](zero-user-apps-completion-gate.md).
|
|
|
|
## Completion attempt sites and sequence
|
|
|
|
Completion can be attempted from whichever fact arrives last: `HelloResolved`,
|
|
`DesktopArrived` (including the Hello-disabled fast-path when `HelloPolicyEnabled ==
|
|
false` and an arm holds), `EspExiting`, `ImeUserSessionCompleted`,
|
|
`AccountSetupProvisioningComplete`, the Hello safety deadline, or the 30-minute
|
|
advisory backstop. All routes converge on `CompleteThroughFinalizingOrDefer`:
|
|
|
|
1. **Completion gates** (currently only the RealmJoin AND-gate): if RealmJoin was
|
|
detected and has no terminal outcome yet, completion is *deferred* — the session
|
|
stays in stage and emits `completion_waiting`. Three terminals open the gate:
|
|
`Resolved` (DeploymentPhase 110), `FirstDeploymentIncomplete` (phase left the
|
|
first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted,
|
|
typically by an interactive logon reclassifying the run as secondary-user; emits
|
|
the `realmjoin_first_deployment_incomplete` Warning, session 224b2087), and
|
|
`Timeout` (60-min deadline from detection). The abort rule reads the *persisted*
|
|
`RealmJoinFacts.LastDeploymentPhase` (updated by the typed `RealmJoinPhaseChanged`
|
|
signal), so it survives agent restarts — the `RealmJoinDetected` replay path
|
|
evaluates it too. A first observation of 210 does **not** release (RJ may have
|
|
completed before agent boot and deploy afterwards).
|
|
|
|
**Activity-based timeout extension** (report 55e6afd61c9d — a large RJ package
|
|
catalog outlived the 60-min window mid-install): when the timeout deadline fires
|
|
while the first deployment is demonstrably active — `LastDeploymentPhase` 100/101
|
|
*and* deployment activity (`RealmJoinPhaseChanged` or a package observation, tracked
|
|
in `RealmJoinFacts.LastActivityUtc`; the phase captured at detection deliberately
|
|
does not count) within the last 60 min — the handler re-arms the deadline to
|
|
`lastActivity + 60 min` instead of timing out, emitting an Info
|
|
`realmjoin_timeout_extended` event. Re-arms clamp to an absolute cap of
|
|
**detection + 4 h**. Package activity is only observable at package *completion*
|
|
(RJ writes the registry subkey when the install finishes), so a single install
|
|
that is registry-silent for over 60 min still times out — the eventual
|
|
`realmjoin_timeout` carries a `reason` payload (`hard_timeout` / `inactivity` /
|
|
`absolute_cap`) telling the three shapes apart. A stale fire of an older deadline
|
|
incarnation (armed due later than the fire time; `OccurredAtUtc = DueAtUtc` per
|
|
scheduler contract) dead-ends as `realmjoin_timeout_stale_superseded_by_rearm`.
|
|
The idle case (detected, no deployment activity ever) times out at 60 min
|
|
unchanged, and phase 200/210 activity never extends.
|
|
**Self-update observation** (gktatooine session 946ccbd6): the version on
|
|
`RealmJoinDetected` is read once from `RealmJoin.exe`, but the tray's `AppUpdater`
|
|
often replaces the MSI-shipped binary on its first start — minutes after detection,
|
|
before the deployment begins. `RealmJoinWatcher` therefore arms a
|
|
`RegNotifyChangeKeyValue` watcher on `HKLM\...\Uninstall\RealmJoin` (the only
|
|
registry write of the updater; appearance-watch on the `Uninstall` parent when the
|
|
entry does not exist yet) once Detected fired, re-reads the binary version after a
|
|
2 s debounce (registry `DisplayVersion` as fallback while the file is locked) and
|
|
posts the typed `RealmJoinAutoUpdateDetected` signal plus the
|
|
`realmjoin_autoupdate_detected` timeline event when the version differs from the
|
|
baseline. The reducer overrides `RealmJoinFacts.ProductVersion` / `ReleaseChannel`
|
|
(`WithVersionOverride` — the one legitimate later write past the set-once helpers);
|
|
it never touches the gate, the deadline, or `LastActivityUtc`. An unreadable
|
|
baseline at detection is seeded silently by the first successful re-read.
|
|
A closed gate also outranks the 30-minute `AdvisoryCompletion` backstop: that
|
|
handler re-arms (trigger suffix `:CompletionGateHolding`) instead of failing while
|
|
the RealmJoin gate is closed *and* its own deadline is still armed, because the gate
|
|
carries a bounded resolution of its own. Before this, the shorter timer always won —
|
|
session 75d6ae8e was failed 2 min after its 25th successful RealmJoin package.
|
|
2. `Finalizing` stage + ~5 s `FinalizingGrace` deadline.
|
|
3. Grace fires → `Completed`, `EnrollmentComplete` outcome, `enrollment_complete`
|
|
timeline event with full audit payload.
|
|
|
|
**Self-deploying/kiosk** is a separate rail: no Hello, no user desktop —
|
|
`AwaitingDeviceOnlyEsp` with a detection deadline as the sole terminal entry, protected
|
|
by stale-fire and race guards. A registry-confirmed self-deploying profile also waives
|
|
IME false-positive AccountSetup entries.
|
|
|
|
# Invariants (violating any of these is a bug)
|
|
|
|
- **Post-terminal dispatch guard**: after `Completed`/`Failed`, every signal is a
|
|
bookkept dead-end — **except `InformationalEvent`** (single-rail pass-through must
|
|
keep flowing). `WhiteGloveSealed` is unguarded on purpose (Part 2 resumes fresh).
|
|
- **`InformationalEvent` never mutates decision state** — exactly one pass-through
|
|
timeline effect, payload copied 1:1.
|
|
- **`AadUserJoinedLate` is observation-only** — never completion evidence.
|
|
- **Hello policy is orthogonal to completion** — `HelloPolicyEnabled` tunes wait
|
|
cadence, never gates the outcome.
|
|
- **Raw IME "Completed user session" is weak evidence** — can run under `defaultuser0`;
|
|
only meaningful with a real-user desktop and the AccountSetup anchor (Arm C).
|
|
- **Deadline bases are floored at `AgentBootUtc`** — replayed backdated signals must
|
|
not arm past-due timers at boot.
|
|
- **Reducer-side deadline cancels need an explicit `CancelDeadline` effect** — a
|
|
builder-only removal leaves the OS timer live.
|
|
- **No silent parking**: a non-terminal post-AccountSetup session must always hold a
|
|
resolution-capable deadline; a 10-min dwell tripwire emits
|
|
`session_parked_without_deadline`.
|
|
- **Every payload-derived string stored into state is bounded** (`FactStringBounds`,
|
|
256 chars): RealmJoin package id/version/scope (scope normalised to `machine`/`user`),
|
|
failed app ids, `HelloOutcome`, `ImeMatchedPatternId`, `EspAdvisoryFailureCategory`,
|
|
RealmJoin product version/channel. The bound is applied *before* any dedupe comparison
|
|
so a forged multi-hundred-KB key neither costs a full-length compare per stored row nor
|
|
lands in snapshots/audit trails. It never truncates real telemetry — the sources are
|
|
registry key names (Windows caps them at 255), GUIDs and enum-like strings. The same
|
|
engine replays device-uploaded signals server-side (`ReducerVerifier`), which is why
|
|
the bound lives in the reducer, not in the agent adapter alone.
|
|
|
|
# Hosting & threading (inside the agent)
|
|
|
|
Wired by `EnrollmentOrchestrator.Start` (`src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/`):
|
|
|
|
- `SignalIngress` — bounded queue (256, back-pressure) + **one** worker thread. Per
|
|
signal, in order: assign ordinals → `SignalLog.Append` (durable, *before* reducing;
|
|
append failure ⇒ reducer does not run, no ordinal gap) → `engine.Reduce` →
|
|
`DecisionStepProcessor.ApplyStep`. A `Post` issued *by the worker thread itself*
|
|
(the `EffectRunner` posting `ClassifierVerdictIssued` from inside `ApplyStep`) never
|
|
enters the bounded queue: it goes to a worker-local follow-up queue that is drained,
|
|
ahead of the channel, right after the current item — the worker is the queue's only
|
|
consumer, so blocking on a full channel there would freeze the pipeline for the rest
|
|
of the run.
|
|
- `DecisionStepProcessor.ApplyStep` (same thread, no locks): journal append (the only
|
|
hard-throwing path; 3 consecutive failures escalate to quarantine) → effects run
|
|
synchronously via `EffectRunner` → best-effort snapshot → advance `CurrentState` →
|
|
one-shot terminal-stage hook (raises the agent's `Terminated` event off-thread).
|
|
- The `DeadlineScheduler`'s `Fired` event loops back in as synthetic `DeadlineFired`
|
|
signals.
|
|
- **Pipeline health in `agent_metrics_snapshot`** (2026-09-01, motivated by the WG-seal
|
|
soak in which a frozen engine and a deliberate Weak sealing verdict were telemetrically
|
|
indistinguishable): every snapshot carries monotonic totals from `SignalIngress`
|
|
(`signal_last_ordinal`, `signal_processed_total`, `signal_pending_count`,
|
|
`signal_reentrant_total`, `signal_queue_peak` — a monotonic peak, deliberately no
|
|
momentary queue length, so snapshot sampling cannot alias it) plus `decision_stage`,
|
|
`decision_last_applied_ordinal` and — once the sealing classifier has produced a
|
|
verdict — `wg_sealing_level`/`wg_sealing_score` from the immutable `DecisionState`.
|
|
Two snapshots therefore prove whether the worker ran in between: a frozen worker keeps
|
|
`signal_pending_count` growing while the processed/applied counters stand still. The
|
|
probe chain is read-only (`EnrollmentOrchestrator` → `IComponentFactory.
|
|
CreateCollectorHosts(decisionStateProbe)` → `PeriodicCollectorLifecycleHost` →
|
|
`AgentSelfMetricsCollector`); the snapshot event itself travels the collector/event
|
|
path, which is independent of the signal pipeline it measures — field-proven by the
|
|
freeze cases, whose perf/metrics events kept flowing while the engine stood still.
|
|
- On restart, `RecoveryCoordinator` folds the persisted signal log through a transient
|
|
`new DecisionEngine()` before the live pipeline starts — see
|
|
[logs & persistence](logs-and-persistence.md).
|