mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-25 13:10:13 +00:00
Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9e24b92fbf | |||
| 45d7c03ccf | |||
| 90012d9a66 | |||
| e8ef4f53ae | |||
| a0d10b8471 | |||
| ad3632e9fb | |||
| 05e1569503 | |||
| ba3581e253 |
@@ -0,0 +1,125 @@
|
||||
---
|
||||
type: Concept
|
||||
title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap
|
||||
description: Why a registered EnrollmentStatusTracking CSP policy provider that never sets TrackingPoliciesCreated parks the user ESP at "Apps (Identifying)" indefinitely, the pre-provisioning + ConfigMgr field case, and how the always-on esp_policy_provider_stalled tripwire (15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces it.
|
||||
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs
|
||||
tags:
|
||||
- agent
|
||||
- esp
|
||||
- enrollment-status-tracking
|
||||
- co-management
|
||||
- stall-detection
|
||||
timestamp: 2026-07-23T00:00:00+02:00
|
||||
---
|
||||
|
||||
# ESP Policy-Provider Stall Detection
|
||||
|
||||
## Schema
|
||||
|
||||
The Enrollment Status Page decides when its "Apps" category may progress from
|
||||
"Identifying" using the **EnrollmentStatusTracking CSP**
|
||||
(<https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>),
|
||||
mirrored in the registry under
|
||||
`HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\{Device|<UserSID>}\…`.
|
||||
Two contracts matter:
|
||||
|
||||
1. **`Setup/Apps/PolicyProviders/<Name>`** — per the CSP doc: *"Existence of this node
|
||||
indicates to the ESP that it shouldn't show the tracking status message until the
|
||||
TrackingPoliciesCreated node has been set to true."* This wait has **no timeout**. A
|
||||
registered provider that never sets `TrackingPoliciesCreated=1` parks the ESP at
|
||||
"Apps (Identifying)" indefinitely — observed for days in the field. Exists in device
|
||||
scope (`Device`) and user scope (`S-<SID>`, appears at sign-in).
|
||||
2. **`DevicePreparation/PolicyProviders/<Name>/InstallationState`** — must reach 2
|
||||
(NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. Device scope only. The ESP
|
||||
applies a default 15-minute timeout here, so these stalls self-surface as ESP errors —
|
||||
the detector still records them for the diagnostic trail.
|
||||
|
||||
Known provider names: **`Sidecar` is the Intune Management Extension** (the CSP doc names
|
||||
"Intune's agents, such as SideCar"); **`ConfigMgr`** is the Configuration Manager client,
|
||||
registered during Autopilot-into-co-management — the troubleshooting section of
|
||||
<https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment> queries
|
||||
exactly these registry paths. The same document states that ESP policy providers **are not
|
||||
aware of each other** and that **pre-provisioning is unsupported** with Autopilot into
|
||||
co-management.
|
||||
|
||||
## The field case
|
||||
|
||||
A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app.
|
||||
Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for
|
||||
hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a
|
||||
`ConfigMgr` key (no `Sidecar`)** and `TrackingPoliciesCreated` is never set. Manually
|
||||
renaming the key to `Sidecar` unblocks the ESP immediately (the IME then finds its
|
||||
provider node and reports tracking-policies-created). The mechanism is exactly the
|
||||
documented no-timeout wait; the unsupported pre-provisioning + co-management combination
|
||||
explains why only some devices are affected. Why the Sidecar registration is missing or
|
||||
displaced on those devices is not publicly documented — irrelevant for detection, since
|
||||
the signature is deterministic.
|
||||
|
||||
## Detection
|
||||
|
||||
`EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone
|
||||
of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes.
|
||||
`EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick
|
||||
from the always-on `EspPolicyProviderStallHost`:
|
||||
|
||||
* A provider **continuously incomplete for ≥ 15 minutes** (wall-clock) is stalled. The
|
||||
dwell clock starts at first incomplete observation and resets on completion or
|
||||
disappearance. Missing values parse as incomplete (fail-toward-observation).
|
||||
* Missing root key / empty provider list = normal early enrollment: clears all dwell
|
||||
state, never fires.
|
||||
* One `esp_policy_provider_stalled` **Warning** per newly stalled provider set
|
||||
(`ImmediateUpload`), one-shot per provider in-process, cross-restart deduped via a
|
||||
`StartupEventGate` fingerprint over the sorted currently-stalled key set — a restart
|
||||
into the same stall stays silent, a new provider joining the stall re-reports.
|
||||
* Payload: `providers[]` (full table: name, scope, kind, raw values, complete flag),
|
||||
`stalledProviders[]` with `stalledForMinutes`, `sidecarRegistered`, `dwellMinutes`.
|
||||
* **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt
|
||||
pass-through) — never touches decision state. No config toggle, no ConfigVersion bump
|
||||
(precedent: `disk_space_low`).
|
||||
|
||||
Deliberately **not** built on `StallProbeCollector` (its idle clock resets on any session
|
||||
activity — a provider stall coexists with a chatty session) and not on the periodic
|
||||
collectors (idle-stopped after 15 minutes — exactly the dormant window the dwell must
|
||||
observe).
|
||||
|
||||
## Guardrail companion
|
||||
|
||||
`rules/guardrails.json` allow-lists `SOFTWARE\Microsoft\Windows\Autopilot` (category
|
||||
"Autopilot / OOBE / Setup") so tenants can build their own gather rules on the subtree —
|
||||
e.g. a `not_exists` registry rule on the `Sidecar` provider key. The subtree carries
|
||||
deployment state only (provider registrations, tracking lists, app IDs, per-user SIDs);
|
||||
sensitivity is at or below existing entries. Enforcement is agent-side (embedded
|
||||
resource), so both the detector and the guardrail change ship with the agent release.
|
||||
|
||||
## Examples
|
||||
|
||||
Customer-case event (abbreviated):
|
||||
|
||||
```json
|
||||
{
|
||||
"eventType": "esp_policy_provider_stalled",
|
||||
"severity": "Warning",
|
||||
"data": {
|
||||
"dwellMinutes": 15,
|
||||
"sidecarRegistered": false,
|
||||
"providers": [
|
||||
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps",
|
||||
"trackingPoliciesCreated": null, "installationState": null, "complete": false }
|
||||
],
|
||||
"stalledProviders": [
|
||||
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps", "stalledForMinutes": 15.0 }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`sidecarRegistered=false` while `setupApps` providers exist is the co-management
|
||||
signature; `sidecarRegistered=true` with a stalled third-party provider generalizes the
|
||||
same contract violation.
|
||||
|
||||
# Citations
|
||||
|
||||
* EnrollmentStatusTracking CSP — <https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>
|
||||
* Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — <https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment>
|
||||
* Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts)
|
||||
* Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md)
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
type: Concept
|
||||
title: Secure Boot CA-2023 Detection — firmware evidence vs. Windows servicing status
|
||||
description: Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly (GetFirmwareEnvironmentVariableExW + SeSystemEnvironmentPrivilege) instead of trusting the SecureBoot\Servicing registry key, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant rule suppression across old and new agents.
|
||||
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs
|
||||
tags:
|
||||
- agent
|
||||
- secureboot
|
||||
- ca2023
|
||||
- analyze-rules
|
||||
- backward-compatibility
|
||||
timestamp: 2026-07-23T00:00:00+02:00
|
||||
---
|
||||
|
||||
# Secure Boot CA-2023 Detection
|
||||
|
||||
## Schema
|
||||
|
||||
Two independent evidence sources describe the Windows UEFI CA 2023 rollout state:
|
||||
|
||||
1. **Servicing registry** (`HKLM\SYSTEM\CurrentControlSet\Control\SecureBoot\Servicing\UEFICA2023Status`)
|
||||
— reflects whether the *Windows Update* rollout ran and self-verified. Lags reality:
|
||||
devices whose firmware received the certificate another way (OEM factory image, WinCS
|
||||
CLI, manual deployment) or where WU simply has not run yet show `unknown`/`notfound`.
|
||||
2. **Firmware itself** — the UEFI variables `db` (signature database, GUID
|
||||
`{d719b2cb-3d3a-4596-a3bc-dad00e67656f}`) and `KEK` (GUID
|
||||
`{8be4df61-93ca-11d2-aa0d-00e098032b8c}`). Authoritative. Certificate subject CNs are
|
||||
ASCII bytes inside the DER signature lists, so a byte-substring search for
|
||||
`Windows UEFI CA 2023` / `Microsoft Corporation KEK 2K CA 2023` decides presence —
|
||||
the same check `Get-SecureBootUEFI` enables from PowerShell.
|
||||
|
||||
`UefiSecureBootCertReader` (agent, `Monitoring/Interop`) reads both variables natively via
|
||||
`GetFirmwareEnvironmentVariableExW` — no PowerShell runspace during OOBE. The call needs
|
||||
`SeSystemEnvironmentPrivilege`: SYSTEM holds it default-disabled, so the reader enables it
|
||||
on the process token (`AdjustTokenPrivileges`) and restores the previous state afterwards.
|
||||
Fail-soft contract like `OobeStateReader`: never throws; failures map to status strings
|
||||
(`not_uefi` = legacy BIOS via `ERROR_INVALID_FUNCTION`, `variable_not_found`,
|
||||
`privilege_denied`, `error_<code>`).
|
||||
|
||||
Fields land on the existing `secureboot_status` event (once per agent start,
|
||||
restart-deduped by the StartupEventGate): `uefiFirmwareReadStatus`, per-cert booleans
|
||||
(`uefiDbHasWindowsUefiCa2023`, `uefiDbHasWindowsProductionPca2011`,
|
||||
`uefiKekHasMicrosoftKek2kCa2023`, …) — emitted only when the respective read succeeded —
|
||||
plus the marker below.
|
||||
|
||||
## The one-sided marker (absence-tolerant suppression)
|
||||
|
||||
Rule-engine constraints (pinned by `RuleEnginePreconditionTests`):
|
||||
|
||||
* **Conditions** never match on an absent field (null guard) — new fields must stay
|
||||
`required: false` or the rule dies for every old agent.
|
||||
* **Preconditions**: `not_exists` passes while the field is absent/empty and fails as soon
|
||||
as the field is present with *any* value; comparison operators fail closed on absence.
|
||||
|
||||
A two-sided boolean therefore cannot drive suppression: `not_exists` on
|
||||
`uefiDbHasWindowsUefiCa2023` would also suppress the rule when the field is present with
|
||||
`false`. Hence the **one-sided marker** `uefiCa2023FirmwareConfirmed = true`, emitted
|
||||
ONLY when the db read succeeded AND the 2023 certificate was found — never as `false`.
|
||||
ANALYZE-SEC-001 v3 adds a `not_exists` precondition on it:
|
||||
|
||||
| Device state | Marker | Rule |
|
||||
|---|---|---|
|
||||
| Old agent (no firmware fields) | absent | evaluates exactly like v2 |
|
||||
| New agent, cert missing or firmware unreadable | absent | evaluates; optional conditions `uefiDbHasWindowsUefiCa2023=false` (+20) / `uefiKekHasMicrosoftKek2kCa2023=false` (+10) raise confidence |
|
||||
| New agent, firmware confirms cert | present | silently suppressed (the customer-reported false positive) |
|
||||
|
||||
Because the marker exists nowhere until the new agent ships, the v3 rule is safe to seed
|
||||
**before** the agent rollout.
|
||||
|
||||
## Examples
|
||||
|
||||
Customer-equivalent verification from PowerShell (authoritative, matches the reader):
|
||||
|
||||
```powershell
|
||||
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023')
|
||||
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name KEK).Bytes).Contains('Microsoft Corporation KEK 2K CA 2023')
|
||||
```
|
||||
|
||||
## Citations
|
||||
|
||||
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs` — native reader
|
||||
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/DeviceInfo/DeviceInfoCollector.NetworkAndSecurity.cs` — `CollectSecureBootStatus` / `AppendFirmwareCertFields`
|
||||
* `rules/analyze/ANALYZE-SEC-001.json` — v3.0.0 rule definition
|
||||
* `src/Backend/AutopilotMonitor.Functions.Tests/RuleEnginePreconditionTests.cs` — pinned precondition semantics + v3 mirror tests
|
||||
* Microsoft: [Act now — Secure Boot certificates expire June 2026](https://techcommunity.microsoft.com/blog/windows-itpro-blog/act-now-secure-boot-certificates-expire-in-june-2026/4426856)
|
||||
@@ -21,6 +21,8 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
|
||||
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
|
||||
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
|
||||
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
|
||||
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait: a registered provider that never sets TrackingPoliciesCreated (co-management "ConfigMgr" without the IME's "Sidecar") parks the user ESP at "Apps (Identifying)" for days; the always-on esp_policy_provider_stalled tripwire and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
|
||||
|
||||
# Rules
|
||||
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
|
||||
## 2026-07-23
|
||||
|
||||
* **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key).
|
||||
|
||||
* **Creation**: Added `agent/secureboot-ca2023-detection.md` — ANALYZE-SEC-001 warned on devices whose firmware already carried the 2023 certificate, because the agent only read the `SecureBoot\Servicing\UEFICA2023Status` registry value, which mirrors the Windows Update rollout rather than the firmware (customer report). The agent now reads the UEFI `db`/`KEK` variables natively (`GetFirmwareEnvironmentVariableExW` with `SeSystemEnvironmentPrivilege` enabled on the process token — first privilege-adjustment code in the repo) and emits per-certificate booleans on `secureboot_status`. Suppression rides on a one-sided marker, `uefiCa2023FirmwareConfirmed`, emitted only on positive confirmation: the engine's `not_exists` precondition fails on any present value, so a two-sided boolean could not distinguish "confirmed" from "checked and missing", while the marker's absence keeps old agents and firmware-missing devices on the v2 path. Rule v3.0.0 is therefore safe to seed before the agent release.
|
||||
|
||||
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
|
||||
|
||||
* **Update**: `agent/decision-engine.md` — the RealmJoin completion gate has a third terminal, `FirstDeploymentIncomplete`. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. `realmjoin_phase_changed` is now a typed `RealmJoinPhaseChanged` signal, so `RealmJoinFacts.LastDeploymentPhase` is persisted across agent restarts (this also fixes every `realmjoin_timeout` event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the `realmjoin_first_deployment_incomplete` Warning, and the `RealmJoinDetected` replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.
|
||||
|
||||
@@ -2,10 +2,10 @@
|
||||
"$schema": "../schema/analyze-rule.schema.json",
|
||||
"ruleId": "ANALYZE-SEC-001",
|
||||
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
|
||||
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
|
||||
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
|
||||
"severity": "warning",
|
||||
"category": "security",
|
||||
"version": "2.0.0",
|
||||
"version": "3.0.0",
|
||||
"enabled": true,
|
||||
"baseConfidence": 70,
|
||||
"preconditions": [
|
||||
@@ -16,6 +16,14 @@
|
||||
"operator": "equals",
|
||||
"value": "false",
|
||||
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
|
||||
},
|
||||
{
|
||||
"source": "event_data",
|
||||
"eventType": "secureboot_status",
|
||||
"dataField": "uefiCa2023FirmwareConfirmed",
|
||||
"operator": "not_exists",
|
||||
"value": "",
|
||||
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
|
||||
}
|
||||
],
|
||||
"conditions": [
|
||||
@@ -54,13 +62,33 @@
|
||||
"operator": "exists",
|
||||
"value": "",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "db_ca2023_firmware_missing",
|
||||
"source": "event_data",
|
||||
"eventType": "secureboot_status",
|
||||
"dataField": "uefiDbHasWindowsUefiCa2023",
|
||||
"operator": "equals",
|
||||
"value": "false",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "kek_2023_missing",
|
||||
"source": "event_data",
|
||||
"eventType": "secureboot_status",
|
||||
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
|
||||
"operator": "equals",
|
||||
"value": "false",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [
|
||||
{ "signal": "ca2023_notfound", "condition": "exists", "weight": 20 },
|
||||
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 }
|
||||
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 },
|
||||
{ "signal": "db_ca2023_firmware_missing", "condition": "exists", "weight": 20 },
|
||||
{ "signal": "kek_2023_missing", "condition": "exists", "weight": 10 }
|
||||
],
|
||||
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
|
||||
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Deploy UEFI CA 2023 certificate now",
|
||||
@@ -69,7 +97,7 @@
|
||||
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
|
||||
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
|
||||
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
|
||||
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
|
||||
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
|
||||
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
|
||||
]
|
||||
}
|
||||
|
||||
+40
-4
@@ -2178,10 +2178,10 @@
|
||||
{
|
||||
"ruleId": "ANALYZE-SEC-001",
|
||||
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
|
||||
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
|
||||
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
|
||||
"severity": "warning",
|
||||
"category": "security",
|
||||
"version": "2.0.0",
|
||||
"version": "3.0.0",
|
||||
"enabled": true,
|
||||
"baseConfidence": 70,
|
||||
"preconditions": [
|
||||
@@ -2192,6 +2192,14 @@
|
||||
"operator": "equals",
|
||||
"value": "false",
|
||||
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
|
||||
},
|
||||
{
|
||||
"source": "event_data",
|
||||
"eventType": "secureboot_status",
|
||||
"dataField": "uefiCa2023FirmwareConfirmed",
|
||||
"operator": "not_exists",
|
||||
"value": "",
|
||||
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
|
||||
}
|
||||
],
|
||||
"conditions": [
|
||||
@@ -2230,6 +2238,24 @@
|
||||
"operator": "exists",
|
||||
"value": "",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "db_ca2023_firmware_missing",
|
||||
"source": "event_data",
|
||||
"eventType": "secureboot_status",
|
||||
"dataField": "uefiDbHasWindowsUefiCa2023",
|
||||
"operator": "equals",
|
||||
"value": "false",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "kek_2023_missing",
|
||||
"source": "event_data",
|
||||
"eventType": "secureboot_status",
|
||||
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
|
||||
"operator": "equals",
|
||||
"value": "false",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [
|
||||
@@ -2242,9 +2268,19 @@
|
||||
"signal": "ca2023_error",
|
||||
"condition": "exists",
|
||||
"weight": 10
|
||||
},
|
||||
{
|
||||
"signal": "db_ca2023_firmware_missing",
|
||||
"condition": "exists",
|
||||
"weight": 20
|
||||
},
|
||||
{
|
||||
"signal": "kek_2023_missing",
|
||||
"condition": "exists",
|
||||
"weight": 10
|
||||
}
|
||||
],
|
||||
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
|
||||
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Deploy UEFI CA 2023 certificate now",
|
||||
@@ -2253,7 +2289,7 @@
|
||||
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
|
||||
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
|
||||
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
|
||||
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
|
||||
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
|
||||
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -43,7 +43,8 @@
|
||||
"prefixes": [
|
||||
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
|
||||
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
|
||||
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon"
|
||||
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
|
||||
"SOFTWARE\\Microsoft\\Windows\\Autopilot"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -272,13 +272,16 @@ try {
|
||||
$downloadAttempt++
|
||||
try {
|
||||
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
|
||||
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
|
||||
Invoke-WebRequest `
|
||||
-Uri $AgentDownloadUrl `
|
||||
-OutFile $zipPath `
|
||||
-UseBasicParsing `
|
||||
-TimeoutSec 30 `
|
||||
-ErrorAction Stop
|
||||
Write-Log "Downloaded agent to $zipPath"
|
||||
$downloadStopwatch.Stop()
|
||||
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
|
||||
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
|
||||
break
|
||||
}
|
||||
catch {
|
||||
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
#nullable enable
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Analyzers
|
||||
{
|
||||
/// <summary>
|
||||
/// LocalAdminAnalyzer.MatchesAllowedEntry — the allowed-list matching contract.
|
||||
/// Entries without wildcard characters match exactly (case-insensitive); entries
|
||||
/// containing * or ? are glob patterns with the same semantics as the hardware
|
||||
/// whitelist. The same matcher is applied to local accounts and profile folders.
|
||||
/// </summary>
|
||||
public sealed class LocalAdminAnalyzerTests
|
||||
{
|
||||
// -------------------------------------------------------------- literal entries
|
||||
|
||||
[Theory]
|
||||
[InlineData("SupportAdmin", "SupportAdmin")]
|
||||
[InlineData("supportadmin", "SupportAdmin")]
|
||||
[InlineData("SUPPORTADMIN", "supportadmin")]
|
||||
public void Literal_entry_matches_exactly_case_insensitive(string name, string entry)
|
||||
{
|
||||
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("SupportAdmin2", "SupportAdmin")]
|
||||
[InlineData("SupportAdmin", "SupportAdmin2")]
|
||||
[InlineData("Support", "SupportAdmin")]
|
||||
public void Literal_entry_does_not_substring_match(string name, string entry)
|
||||
{
|
||||
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------- glob patterns
|
||||
|
||||
[Theory]
|
||||
[InlineData("adm-12345", "adm-*")]
|
||||
[InlineData("adm-", "adm-*")]
|
||||
[InlineData("ADM-9F3B2C", "adm-*")]
|
||||
[InlineData("lapsAdmin01", "lapsAdmin??")]
|
||||
[InlineData("svc-backup-01", "svc-*-01")]
|
||||
public void Wildcard_entry_matches_glob(string name, string entry)
|
||||
{
|
||||
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("administrator2", "adm-*")] // no literal "adm-" prefix
|
||||
[InlineData("xadm-123", "adm-*")] // pattern is anchored at start
|
||||
[InlineData("lapsAdmin1", "lapsAdmin??")] // ? is exactly one char
|
||||
[InlineData("adm.123", "adm-*")] // "-" is literal, not a regex class
|
||||
public void Wildcard_entry_does_not_overmatch(string name, string entry)
|
||||
{
|
||||
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Regex_metacharacters_in_entry_are_treated_literally()
|
||||
{
|
||||
// "adm.123" as an entry has no wildcard chars → exact match only,
|
||||
// the dot must not act as a regex any-char.
|
||||
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.123"));
|
||||
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.123"));
|
||||
|
||||
// Mixed: wildcard present, remaining metacharacters stay literal.
|
||||
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.*"));
|
||||
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.*"));
|
||||
}
|
||||
}
|
||||
}
|
||||
+139
@@ -405,5 +405,144 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
|
||||
var started = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorStarted);
|
||||
Assert.Equal(0, started.Data["noCandidateThresholdMinutes"]);
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// OOBE-state flip (oobe_state_completed) — observational one-shot on the poll
|
||||
// ============================================================================
|
||||
// The WinRT OutOfBoxExperienceState flip in_progress->completed is an owner-
|
||||
// independent desktop corroboration for DAD-starved sessions (4d5a0b78 class).
|
||||
// Tests drive polls synchronously via PollOnceForTest after Stop() so the real
|
||||
// timer cannot race, and pin the owner seams to an excluded user so a real
|
||||
// explorer.exe on the test machine can never set _desktopArrived (which would
|
||||
// short-circuit PollForDesktop before the flip check).
|
||||
|
||||
private static DesktopArrivalDetector BuildFlipDetector(TempDirectory tmp, TraceCapture trace, System.Func<string> oobeProvider)
|
||||
{
|
||||
var detector = new DesktopArrivalDetector(new AgentLogger(tmp.Path, AgentLogLevel.Info), noCandidateTimeoutMinutes: 0)
|
||||
{
|
||||
OnTraceEvent = trace.Sink,
|
||||
OobeStateProvider = oobeProvider,
|
||||
SessionOwnerResolver = _ => "SYSTEM",
|
||||
ProcessOwnerResolver = _ => "SYSTEM",
|
||||
};
|
||||
return detector;
|
||||
}
|
||||
|
||||
/// <summary>Returns the queued states in order, then repeats the last one forever.</summary>
|
||||
private static System.Func<string> StateQueue(params string[] states)
|
||||
{
|
||||
var queue = new Queue<string>(states);
|
||||
var last = states[states.Length - 1];
|
||||
return () => queue.Count > 0 ? queue.Dequeue() : last;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OobeFlip_emits_exactly_once_with_payload()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var trace = new TraceCapture();
|
||||
// Baseline sample happens synchronously inside Start()/ArmTimer.
|
||||
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "completed"));
|
||||
|
||||
detector.Start();
|
||||
detector.Stop();
|
||||
detector.PollOnceForTest(); // reads "completed" -> flip
|
||||
detector.PollOnceForTest(); // still "completed" -> no second emit
|
||||
|
||||
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
|
||||
var flip = trace.Events.Find(e => e.EventType == SharedEventTypes.OobeStateCompleted);
|
||||
Assert.Equal("in_progress", flip.Data["previousState"]);
|
||||
Assert.True(flip.Data.ContainsKey("pollCount"));
|
||||
Assert.True(flip.Data.ContainsKey("minutesSinceStart"));
|
||||
Assert.False((bool)flip.Data["desktopArrived"]);
|
||||
Assert.True(flip.Data.ContainsKey("wmiErrorCount"));
|
||||
Assert.True(flip.Data.ContainsKey("wtsErrorCount"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OobeFlip_does_not_emit_when_state_is_already_completed()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var trace = new TraceCapture();
|
||||
using var detector = BuildFlipDetector(tmp, trace, StateQueue("completed"));
|
||||
|
||||
detector.Start();
|
||||
detector.Stop();
|
||||
detector.PollOnceForTest();
|
||||
detector.PollOnceForTest();
|
||||
|
||||
// Productive machines read "completed" from the very first sample — the flip
|
||||
// event must never fire there (the agent_started field already records the state).
|
||||
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OobeFlip_does_not_emit_when_in_progress_was_never_seen()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var trace = new TraceCapture();
|
||||
using var detector = BuildFlipDetector(tmp, trace, StateQueue("unavailable", "completed"));
|
||||
|
||||
detector.Start();
|
||||
detector.Stop();
|
||||
detector.PollOnceForTest();
|
||||
|
||||
// Old builds without the WinRT contract report "unavailable"; a later
|
||||
// "completed" without a positively observed "in_progress" is not a flip.
|
||||
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OobeFlip_survives_transient_unavailable_between_polls()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var trace = new TraceCapture();
|
||||
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "unavailable", "completed"));
|
||||
|
||||
detector.Start();
|
||||
detector.Stop();
|
||||
detector.PollOnceForTest(); // transient reflection failure — must NOT clear the baseline
|
||||
detector.PollOnceForTest(); // "completed" -> flip still detected
|
||||
|
||||
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OobeFlip_provider_throwing_never_emits_and_never_pollutes_wmi_error_count()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var trace = new TraceCapture();
|
||||
using var detector = BuildFlipDetector(tmp, trace, () => throw new System.InvalidOperationException("boom"));
|
||||
|
||||
detector.Start(); // baseline sample throws -> swallowed
|
||||
detector.Stop();
|
||||
detector.PollOnceForTest();
|
||||
detector.PollOnceForTest();
|
||||
|
||||
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
|
||||
// The flip check has its own try/catch — a throw must not fall into
|
||||
// PollForDesktop's outer catch, which increments the WMI error counter the
|
||||
// session-4d5a0b78 liveness diagnostics rely on. first_poll carries that counter.
|
||||
var firstPoll = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorFirstPoll);
|
||||
Assert.Equal(0, firstPoll.Data["wmiErrorCount"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OobeFlip_one_shot_flag_survives_ResetForRealUserSwitch()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var trace = new TraceCapture();
|
||||
using var detector = BuildFlipDetector(tmp, trace,
|
||||
StateQueue("in_progress", "completed", "in_progress", "completed"));
|
||||
|
||||
detector.Start();
|
||||
detector.Stop();
|
||||
detector.PollOnceForTest(); // flip #1
|
||||
detector.ResetForRealUserSwitch(); // re-arms + re-samples baseline ("in_progress")
|
||||
detector.Stop();
|
||||
detector.PollOnceForTest(); // "completed" again — must NOT re-emit
|
||||
|
||||
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+11
@@ -290,9 +290,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
|
||||
[InlineData(@"SAM\SAM\Domains\Account\Users")]
|
||||
[InlineData(@"SECURITY\Policy\Secrets")]
|
||||
[InlineData(@"HKLM\SOFTWARE\Microsoft\EnrollmentsSomethingElse")] // prefix spoofing
|
||||
[InlineData(@"HKLM\SOFTWARE\Microsoft\Windows\AutopilotSomethingElse")] // sibling spoof of the Autopilot prefix
|
||||
public void RegistryCollector_OffAllowlistPath_IsBlockedBeforeTheHiveIsOpened(string target)
|
||||
=> AssertBlocked(new RegistryCollector().Execute(Rule("registry", target), Context()), target);
|
||||
|
||||
// Session 5d735290: the EnrollmentStatusTracking subtree (ESP policy providers / tracking
|
||||
// state) is allow-listed so tenants can build gather rules on the co-management stall
|
||||
// signature the esp_policy_provider_stalled detector reports (e.g. not_exists on the
|
||||
// Sidecar provider key).
|
||||
[Theory]
|
||||
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot")]
|
||||
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders\Sidecar")]
|
||||
public void RegistryGuard_AutopilotEnrollmentTrackingSubtree_IsAllowed(string subPath)
|
||||
=> Assert.True(GatherRuleGuards.IsRegistryPathAllowed(subPath));
|
||||
|
||||
[Theory]
|
||||
[InlineData("SELECT * FROM Win32_UserAccount")]
|
||||
[InlineData("SELECT * FROM Win32_ShadowCopy")]
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
#nullable enable
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
|
||||
{
|
||||
/// <summary>
|
||||
/// Smoke tests for the WinRT <c>SystemSetupInfo.OutOfBoxExperienceState</c> reader.
|
||||
/// The actual state on the test machine is unknown (CI runners are post-OOBE; Server
|
||||
/// SKUs may lack the UniversalApiContract entirely), so only the fail-soft contract
|
||||
/// is pinned: never throw, always return a member of the known value set.
|
||||
/// </summary>
|
||||
public sealed class OobeStateReaderTests
|
||||
{
|
||||
[Fact]
|
||||
public void Read_never_throws_and_returns_known_value()
|
||||
{
|
||||
var value = OobeStateReader.Read();
|
||||
|
||||
Assert.False(string.IsNullOrEmpty(value));
|
||||
Assert.True(
|
||||
value == "not_started"
|
||||
|| value == "in_progress"
|
||||
|| value == "completed"
|
||||
|| value == OobeStateReader.Unavailable
|
||||
|| value.StartsWith("unknown_"),
|
||||
$"unexpected reader value '{value}'");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Read_is_stable_across_repeated_calls()
|
||||
{
|
||||
// Exercises the cached-PropertyInfo path (second call skips resolution).
|
||||
var first = OobeStateReader.Read();
|
||||
var second = OobeStateReader.Read();
|
||||
|
||||
Assert.Equal(first, second);
|
||||
}
|
||||
}
|
||||
}
|
||||
+212
@@ -0,0 +1,212 @@
|
||||
using System;
|
||||
using System.Diagnostics;
|
||||
using System.IO;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Security;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
|
||||
{
|
||||
/// <summary>
|
||||
/// Executes the generated cleanup script against REAL scratch trees with a real
|
||||
/// powershell.exe — end-to-end replays of the 2026-07-22 incident mechanics that the
|
||||
/// string pins in <see cref="CleanupServiceScriptTests"/> cannot prove:
|
||||
/// <list type="bullet">
|
||||
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
|
||||
/// marker + state around the locks and orphaned the binaries),</item>
|
||||
/// <item>a straggler agent process (the incident's second instance) must be found by
|
||||
/// NAME, killed, and the cleanup must then complete,</item>
|
||||
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
|
||||
/// </list>
|
||||
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
|
||||
/// runs in seconds. Every test uses GUID-unique process/task/tree names so nothing real
|
||||
/// (a dev-box agent, a real Scheduled Task) can ever be touched.
|
||||
/// </summary>
|
||||
[Collection("SerialThreading")]
|
||||
public class CleanupServiceScriptIntegrationTests : IDisposable
|
||||
{
|
||||
private readonly string _root;
|
||||
private readonly string _basePath;
|
||||
private readonly string _taskName = $"AmTest-NoSuchTask-{Guid.NewGuid():N}";
|
||||
private readonly string _fakeProcessName = $"AmTestAgent{Guid.NewGuid().ToString("N").Substring(0, 8)}";
|
||||
|
||||
public CleanupServiceScriptIntegrationTests()
|
||||
{
|
||||
_root = Path.Combine(Path.GetTempPath(), $"am-cleanup-it-{Guid.NewGuid():N}");
|
||||
_basePath = Path.Combine(_root, "AutopilotMonitor");
|
||||
Directory.CreateDirectory(Path.Combine(_basePath, "Agent"));
|
||||
Directory.CreateDirectory(Path.Combine(_basePath, "State"));
|
||||
Directory.CreateDirectory(Path.Combine(_basePath, "Logs"));
|
||||
File.WriteAllText(Path.Combine(_basePath, "Agent", "fake.exe"), "x");
|
||||
File.WriteAllText(Path.Combine(_basePath, "State", "enrollment-complete.marker"), "x");
|
||||
File.WriteAllText(Path.Combine(_basePath, "Logs", "agent.log"), "x");
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
try { Directory.Delete(_root, recursive: true); } catch { }
|
||||
}
|
||||
|
||||
private static string LockPath => Path.Combine(Path.GetTempPath(), "AutopilotMonitor-Cleanup.lock");
|
||||
|
||||
private string BuildScript(bool keepLogs = false, int processWaitSeconds = 1)
|
||||
=> CleanupService.BuildCleanupScript(
|
||||
agentProcessName: _fakeProcessName,
|
||||
agentBasePath: _basePath,
|
||||
keepLogs: keepLogs,
|
||||
scheduledTaskName: _taskName,
|
||||
rebootOnComplete: false,
|
||||
initialDelaySeconds: 0,
|
||||
processWaitSeconds: processWaitSeconds,
|
||||
renameRetries: 2,
|
||||
renameRetryDelaySeconds: 1);
|
||||
|
||||
/// <summary>Runs the script synchronously in a real powershell.exe. Returns the script path.</summary>
|
||||
private string RunScript(string script)
|
||||
{
|
||||
var scriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
|
||||
File.WriteAllText(scriptPath, script);
|
||||
using (var proc = Process.Start(new ProcessStartInfo
|
||||
{
|
||||
FileName = SystemPaths.PowerShell,
|
||||
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{scriptPath}\"",
|
||||
UseShellExecute = false,
|
||||
CreateNoWindow = true,
|
||||
}))
|
||||
{
|
||||
Assert.True(proc.WaitForExit(90_000), "cleanup script did not finish within 90s");
|
||||
}
|
||||
return scriptPath;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Happy_path_deletes_tree_and_removes_lock_and_script()
|
||||
{
|
||||
var scriptPath = RunScript(BuildScript());
|
||||
|
||||
Assert.False(Directory.Exists(_basePath), "tree should be fully deleted");
|
||||
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
|
||||
Assert.False(File.Exists(scriptPath), "script should self-delete");
|
||||
Assert.False(File.Exists(LockPath), "cleanup lock should be released");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Locked_tree_aborts_completely_marker_and_files_survive()
|
||||
{
|
||||
// The incident regression: with a file locked inside the tree the OLD script
|
||||
// deleted everything else (incl. the enrollment-complete marker) and left the
|
||||
// binaries orphaned with the task already gone. The hardened script must treat
|
||||
// the failed probe-rename as "someone still holds this" and walk away entirely.
|
||||
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
|
||||
FileMode.Open, FileAccess.Read, FileShare.None))
|
||||
{
|
||||
RunScript(BuildScript());
|
||||
}
|
||||
|
||||
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")), "binaries must stay in place");
|
||||
Assert.True(File.Exists(Path.Combine(_basePath, "State", "enrollment-complete.marker")),
|
||||
"marker must survive so the next boot retries cleanup");
|
||||
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
|
||||
Assert.False(File.Exists(LockPath));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Straggler_process_holding_the_tree_is_killed_and_cleanup_completes()
|
||||
{
|
||||
// Full incident replay: a SECOND agent-named process (here: a renamed
|
||||
// powershell.exe, matching how only the process NAME identifies agents) holds a
|
||||
// file in the tree open. The old script waited on the launcher PID only and ran
|
||||
// its force-delete against the live locks. The hardened script must find the
|
||||
// straggler by name, kill it after the wait deadline, and then delete everything.
|
||||
var fakeExe = Path.Combine(_root, _fakeProcessName + ".exe");
|
||||
File.Copy(SystemPaths.PowerShell, fakeExe);
|
||||
|
||||
var lockTarget = Path.Combine(_basePath, "Agent", "fake.exe").Replace("'", "''");
|
||||
Process straggler = null;
|
||||
try
|
||||
{
|
||||
straggler = Process.Start(new ProcessStartInfo
|
||||
{
|
||||
FileName = fakeExe,
|
||||
Arguments = "-NoProfile -Command \"$f = [IO.File]::Open('" + lockTarget +
|
||||
"', 'Open', 'Read', 'None'); Start-Sleep -Seconds 120\"",
|
||||
UseShellExecute = false,
|
||||
CreateNoWindow = true,
|
||||
});
|
||||
|
||||
// Give the straggler time to acquire its file handle before cleanup starts.
|
||||
var acquireDeadline = DateTime.UtcNow.AddSeconds(20);
|
||||
var acquired = false;
|
||||
while (!acquired && DateTime.UtcNow < acquireDeadline)
|
||||
{
|
||||
try
|
||||
{
|
||||
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
|
||||
FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
|
||||
{
|
||||
}
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
acquired = true; // share violation ⇒ straggler holds the file
|
||||
}
|
||||
if (!acquired) System.Threading.Thread.Sleep(200);
|
||||
}
|
||||
Assert.True(acquired, "straggler never acquired its file lock");
|
||||
|
||||
RunScript(BuildScript(processWaitSeconds: 2));
|
||||
|
||||
Assert.True(straggler.HasExited, "straggler must be force-killed by the name-based backstop");
|
||||
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
|
||||
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
|
||||
}
|
||||
finally
|
||||
{
|
||||
try { if (straggler != null && !straggler.HasExited) straggler.Kill(); } catch { }
|
||||
try { straggler?.Dispose(); } catch { }
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Fresh_foreign_lock_backs_off_without_touching_the_tree()
|
||||
{
|
||||
File.WriteAllText(LockPath, "held by another cleanup");
|
||||
try
|
||||
{
|
||||
var scriptPath = RunScript(BuildScript());
|
||||
|
||||
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")),
|
||||
"loser script must not touch the tree — the lock holder owns the deletion");
|
||||
Assert.True(File.Exists(LockPath), "foreign lock must not be deleted by the loser");
|
||||
Assert.False(File.Exists(scriptPath), "loser still self-deletes its script");
|
||||
}
|
||||
finally
|
||||
{
|
||||
try { File.Delete(LockPath); } catch { }
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stale_foreign_lock_is_taken_over_and_cleanup_proceeds()
|
||||
{
|
||||
File.WriteAllText(LockPath, "crashed cleanup from a previous boot");
|
||||
File.SetLastWriteTime(LockPath, DateTime.Now.AddMinutes(-15));
|
||||
|
||||
RunScript(BuildScript());
|
||||
|
||||
Assert.False(Directory.Exists(_basePath), "stale lock must be taken over, cleanup must run");
|
||||
Assert.False(File.Exists(LockPath));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void KeepLogs_variant_deletes_everything_except_logs()
|
||||
{
|
||||
RunScript(BuildScript(keepLogs: true));
|
||||
|
||||
Assert.True(File.Exists(Path.Combine(_basePath, "Logs", "agent.log")), "Logs must survive");
|
||||
Assert.False(Directory.Exists(Path.Combine(_basePath, "Agent")));
|
||||
Assert.False(Directory.Exists(Path.Combine(_basePath, "State")));
|
||||
Assert.Empty(Directory.GetFileSystemEntries(_basePath, "*.del-*"));
|
||||
}
|
||||
}
|
||||
}
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
|
||||
{
|
||||
/// <summary>
|
||||
/// Pins the LIFE-F7 hardening invariants of the self-destruct cleanup script
|
||||
/// (incident 2026-07-22: a second agent instance held exe + loaded DLLs while the
|
||||
/// old script — waiting on the launcher PID only — force-deleted around the locks
|
||||
/// AFTER removing the Scheduled Task, orphaning the binaries with no retry path).
|
||||
/// These are string-level pins on <see cref="CleanupService.BuildCleanupScript"/>:
|
||||
/// weakening any of them silently re-introduces the orphaned-binaries failure mode.
|
||||
/// </summary>
|
||||
public class CleanupServiceScriptTests
|
||||
{
|
||||
private const string ProcessName = "AutopilotMonitor.Agent";
|
||||
private const string BasePath = @"C:\ProgramData\AutopilotMonitor";
|
||||
private const string TaskName = "AutopilotMonitorAgent";
|
||||
|
||||
private static string Script(bool keepLogs = false, bool reboot = false)
|
||||
=> CleanupService.BuildCleanupScript(ProcessName, BasePath, keepLogs, TaskName, reboot);
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)]
|
||||
[InlineData(true)]
|
||||
public void Waits_by_process_name_not_launcher_pid(bool keepLogs)
|
||||
{
|
||||
var script = Script(keepLogs);
|
||||
|
||||
// LIFE-F7 #2: name-based wait covers marker-retry / self-update / manual instances
|
||||
// and is immune to PID reuse. Wait-Process -Id saw exactly one PID and nothing else.
|
||||
Assert.Contains($"Get-Process -Name '{ProcessName}'", script);
|
||||
Assert.DoesNotContain("Wait-Process", script);
|
||||
|
||||
// Backstop: stragglers are force-killed — their images are deleted right after.
|
||||
Assert.Contains("Stop-Process -Force", script);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)]
|
||||
[InlineData(true)]
|
||||
public void Never_deletes_the_original_path(bool keepLogs)
|
||||
{
|
||||
var script = Script(keepLogs);
|
||||
|
||||
// LIFE-F7 #3: deletion happens exclusively on successfully renamed targets
|
||||
// ($entry.Renamed). A Remove-Item against the original tree is what deleted
|
||||
// marker + state around locked binaries in the incident.
|
||||
Assert.DoesNotContain($"Remove-Item -Path '{BasePath}'", script);
|
||||
Assert.Contains("Remove-Item -Path $entry.Renamed", script);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)]
|
||||
[InlineData(true)]
|
||||
public void Task_removal_is_gated_on_probe_and_deletion_on_task_gone(bool keepLogs)
|
||||
{
|
||||
var script = Script(keepLogs);
|
||||
|
||||
// Probe-rename first; only a successful probe reaches task removal, and only a
|
||||
// removed task reaches deletion. Both failure branches rename BACK so the next
|
||||
// boot retries with task + marker + files intact.
|
||||
Assert.Contains("if ($probeOk)", script);
|
||||
Assert.Contains("if ($taskGone)", script);
|
||||
Assert.Contains("Rename-Item -Path $entry.Renamed -NewName $entry.Original", script);
|
||||
|
||||
// LIFE-F6 task-removal loop + module-independent fallback stay intact.
|
||||
Assert.Contains("Unregister-ScheduledTask -TaskName $taskName", script);
|
||||
Assert.Contains("schtasks.exe /Delete /TN $taskName /F", script);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)]
|
||||
[InlineData(true)]
|
||||
public void Concurrent_cleanup_scripts_are_serialized_by_lock_file(bool keepLogs)
|
||||
{
|
||||
var script = Script(keepLogs);
|
||||
|
||||
// LIFE-F7 #1: CreateNew is the atomic single-flight gate; the loser exits without
|
||||
// touching the tree. Stale locks (>10 min) are taken over.
|
||||
Assert.Contains("AutopilotMonitor-Cleanup.lock", script);
|
||||
Assert.Contains("[System.IO.FileMode]::CreateNew", script);
|
||||
Assert.Contains("$lockAge.TotalMinutes -gt 10", script);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)]
|
||||
[InlineData(true)]
|
||||
public void Rename_targets_carry_a_guid_suffix(bool keepLogs)
|
||||
{
|
||||
// A fixed '.del' target collides with leftovers of earlier attempts and with a
|
||||
// concurrent script's rename — the GUID makes every attempt's target unique.
|
||||
Assert.Contains(".del-' + [Guid]::NewGuid().ToString('N')", Script(keepLogs));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void KeepLogs_variant_excludes_logs_directory()
|
||||
{
|
||||
var script = Script(keepLogs: true);
|
||||
Assert.Contains("-Exclude 'Logs'", script);
|
||||
Assert.DoesNotContain($"Rename-Item -Path '{BasePath}' -NewName", script);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Full_variant_renames_the_whole_tree()
|
||||
{
|
||||
var script = Script(keepLogs: false);
|
||||
Assert.Contains($"Rename-Item -Path '{BasePath}' -NewName $renamedPath", script);
|
||||
Assert.DoesNotContain("-Exclude 'Logs'", script);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)]
|
||||
[InlineData(true)]
|
||||
public void Reboot_flag_controls_shutdown_invocation(bool reboot)
|
||||
{
|
||||
var script = Script(keepLogs: false, reboot: reboot);
|
||||
if (reboot)
|
||||
Assert.Contains("shutdown.exe /r /t 10", script);
|
||||
else
|
||||
Assert.DoesNotContain("shutdown.exe", script);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)]
|
||||
[InlineData(true)]
|
||||
public void Script_and_lock_are_released_in_finally(bool keepLogs)
|
||||
{
|
||||
var script = Script(keepLogs);
|
||||
Assert.Contains("finally", script);
|
||||
Assert.Contains("$lockStream.Dispose()", script);
|
||||
Assert.Contains("Remove-Item -Path $scriptPath -Force", script);
|
||||
}
|
||||
}
|
||||
}
|
||||
+359
@@ -0,0 +1,359 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Persistence;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Session 5d735290 — pure detection logic of <see cref="EspPolicyProviderStallDetector"/>:
|
||||
/// the CSP completion rule and the dwell/one-shot transition. No registry, no timers.
|
||||
/// </summary>
|
||||
public sealed class EspPolicyProviderStallLogicTests
|
||||
{
|
||||
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc);
|
||||
private static readonly TimeSpan Dwell = TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes);
|
||||
|
||||
private static PolicyProviderState SetupApps(string name, int? tracking, string scope = "device") =>
|
||||
new PolicyProviderState(name, scope, EspPolicyProviderProbe.KindSetupApps, tracking, null);
|
||||
|
||||
private static PolicyProviderState DevicePrep(string name, int? installState) =>
|
||||
new PolicyProviderState(name, "device", EspPolicyProviderProbe.KindDevicePreparation, null, installState);
|
||||
|
||||
private static EspPolicyProviderSnapshot Snapshot(params PolicyProviderState[] providers) =>
|
||||
new EspPolicyProviderSnapshot(providers);
|
||||
|
||||
// ------------------------------------------------- completion rule (CSP contract)
|
||||
|
||||
[Theory]
|
||||
[InlineData(1, true)] // TrackingPoliciesCreated=true
|
||||
[InlineData(0, false)] // explicit false (the CSP default)
|
||||
[InlineData(null, false)] // value missing — the field case
|
||||
[InlineData(2, false)] // anything else is not "true"
|
||||
public void SetupApps_provider_complete_only_at_tracking_1(int? tracking, bool expected)
|
||||
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
|
||||
EspPolicyProviderProbe.KindSetupApps, tracking, null));
|
||||
|
||||
[Theory]
|
||||
[InlineData(2, true)] // NotRequired
|
||||
[InlineData(3, true)] // Completed
|
||||
[InlineData(1, false)] // NotInstalled
|
||||
[InlineData(4, false)] // Error
|
||||
[InlineData(null, false)]
|
||||
public void DevicePreparation_provider_complete_at_state_2_or_3(int? state, bool expected)
|
||||
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
|
||||
EspPolicyProviderProbe.KindDevicePreparation, null, state));
|
||||
|
||||
[Fact]
|
||||
public void TryReadInt_tolerates_dword_string_and_bool_forms()
|
||||
{
|
||||
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(1));
|
||||
Assert.Equal(3, EspPolicyProviderProbe.TryReadInt(3L));
|
||||
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(" 1 "));
|
||||
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(true));
|
||||
Assert.Equal(0, EspPolicyProviderProbe.TryReadInt(false));
|
||||
Assert.Null(EspPolicyProviderProbe.TryReadInt(null));
|
||||
Assert.Null(EspPolicyProviderProbe.TryReadInt("not-a-number"));
|
||||
Assert.Null(EspPolicyProviderProbe.TryReadInt(new byte[] { 1 }));
|
||||
}
|
||||
|
||||
// ------------------------------------------------- dwell transition
|
||||
|
||||
[Fact]
|
||||
public void Empty_or_missing_snapshot_clears_dwell_state_and_never_stalls()
|
||||
{
|
||||
var previous = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
|
||||
|
||||
var missing = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0.AddHours(2), Dwell);
|
||||
Assert.Empty(missing.UpdatedFirstSeenUtc);
|
||||
Assert.Empty(missing.NewlyStalled);
|
||||
|
||||
var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(), previous, new HashSet<string>(), T0.AddHours(2), Dwell);
|
||||
Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc);
|
||||
Assert.Empty(registeredButEmpty.NewlyStalled);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Incomplete_provider_accrues_dwell_but_does_not_stall_before_threshold()
|
||||
{
|
||||
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
|
||||
|
||||
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
|
||||
Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]);
|
||||
Assert.Empty(round1.NewlyStalled);
|
||||
|
||||
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(), T0.AddMinutes(14), Dwell);
|
||||
Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved
|
||||
Assert.Empty(round2.NewlyStalled);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Provider_stalls_at_dwell_threshold_with_correct_minutes()
|
||||
{
|
||||
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 0));
|
||||
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
|
||||
|
||||
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, seen, new HashSet<string>(), T0.AddMinutes(15), Dwell);
|
||||
|
||||
var stalled = Assert.Single(eval.NewlyStalled);
|
||||
Assert.Equal("ConfigMgr", stalled.Provider.Name);
|
||||
Assert.Equal(15.0, stalled.StalledForMinutes, precision: 3);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Completion_or_disappearance_resets_the_dwell_clock()
|
||||
{
|
||||
var key = "setupApps|device|ConfigMgr";
|
||||
var seen = new Dictionary<string, DateTime> { [key] = T0 };
|
||||
|
||||
// Completes mid-dwell → dropped from the map.
|
||||
var completed = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), T0.AddMinutes(10), Dwell);
|
||||
Assert.Empty(completed.UpdatedFirstSeenUtc);
|
||||
|
||||
// Re-appears incomplete much later → dwell restarts from the new now, no stall yet.
|
||||
var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc,
|
||||
new HashSet<string>(), T0.AddHours(3), Dwell);
|
||||
Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]);
|
||||
Assert.Empty(reappeared.NewlyStalled);
|
||||
|
||||
// Disappearance behaves the same as completion.
|
||||
var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary<string, DateTime> { [key] = T0 },
|
||||
new HashSet<string>(), T0.AddMinutes(20), Dwell);
|
||||
Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Already_fired_provider_is_never_reported_again()
|
||||
{
|
||||
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
|
||||
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
|
||||
var fired = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" };
|
||||
|
||||
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, seen, fired, T0.AddHours(5), Dwell);
|
||||
|
||||
Assert.Empty(eval.NewlyStalled);
|
||||
Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Mixed_kinds_and_scopes_are_tracked_under_distinct_keys()
|
||||
{
|
||||
var snapshot = Snapshot(
|
||||
SetupApps("ConfigMgr", tracking: null),
|
||||
SetupApps("ConfigMgr", tracking: null, scope: "user:S-1-5-21-1111"),
|
||||
DevicePrep("ConfigMgr", installState: 1));
|
||||
|
||||
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
|
||||
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), T0, Dwell);
|
||||
|
||||
Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count);
|
||||
Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
|
||||
Assert.Contains("setupApps|user:S-1-5-21-1111|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
|
||||
Assert.Contains("devicePreparation|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SidecarRegistered_matches_setupApps_sidecar_only()
|
||||
{
|
||||
Assert.True(Snapshot(SetupApps("Sidecar", tracking: 0)).SidecarRegistered);
|
||||
Assert.True(Snapshot(SetupApps("sidecar", tracking: 1, scope: "user:S-1-5-21-1")).SidecarRegistered);
|
||||
Assert.False(Snapshot(SetupApps("ConfigMgr", tracking: 1)).SidecarRegistered);
|
||||
Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind
|
||||
Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Emission surface of the detector via the real <see cref="InformationalEventPost"/> rail.
|
||||
/// Static probe override → serial collection (same contract as
|
||||
/// <c>EspConfigDetectedTrackingFieldsTests</c>).
|
||||
/// </summary>
|
||||
[Collection("SerialThreading")]
|
||||
public sealed class EspPolicyProviderStallDetectorEmissionTests : IDisposable
|
||||
{
|
||||
private static readonly DateTime At = new DateTime(2026, 7, 23, 12, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private readonly TempDirectory _tmp = new TempDirectory();
|
||||
|
||||
public void Dispose() => _tmp.Dispose();
|
||||
|
||||
private (EspPolicyProviderStallDetector sut, FakeSignalIngressSink sink, VirtualClock clock)
|
||||
BuildDetector(StartupEventGate? gate = null)
|
||||
{
|
||||
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
|
||||
var sink = new FakeSignalIngressSink();
|
||||
var clock = new VirtualClock(At);
|
||||
var post = new InformationalEventPost(sink, clock);
|
||||
var sut = new EspPolicyProviderStallDetector("S1", "T1", post, logger, clock, gate);
|
||||
return (sut, sink, clock);
|
||||
}
|
||||
|
||||
private static EspPolicyProviderSnapshot ConfigMgrOnly() => new EspPolicyProviderSnapshot(new[]
|
||||
{
|
||||
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
|
||||
trackingPoliciesCreated: null, installationState: null),
|
||||
});
|
||||
|
||||
private static EspPolicyProviderSnapshot SidecarCompleteConfigMgrIncomplete() => new EspPolicyProviderSnapshot(new[]
|
||||
{
|
||||
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
|
||||
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
|
||||
});
|
||||
|
||||
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> StalledEvents(FakeSignalIngressSink sink) =>
|
||||
sink.Posted.Where(p =>
|
||||
p.Payload != null
|
||||
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
|
||||
&& et == "esp_policy_provider_stalled").ToList();
|
||||
|
||||
[Fact]
|
||||
public void CustomerScenario_configmgr_without_sidecar_emits_one_warning_after_dwell()
|
||||
{
|
||||
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
|
||||
var (sut, sink, clock) = BuildDetector();
|
||||
|
||||
sut.Tick(); // first sighting — dwell starts
|
||||
Assert.Empty(StalledEvents(sink));
|
||||
|
||||
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
|
||||
sut.Tick(); // threshold crossed → emit
|
||||
|
||||
var evt = Assert.Single(StalledEvents(sink));
|
||||
Assert.Equal(DecisionSignalKind.InformationalEvent, evt.Kind);
|
||||
Assert.Equal("Warning", evt.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("true", evt.Payload[SignalPayloadKeys.ImmediateUpload]);
|
||||
Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]);
|
||||
|
||||
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
|
||||
Assert.Equal(false, data["sidecarRegistered"]);
|
||||
Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]);
|
||||
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
|
||||
var entry = Assert.Single(stalled);
|
||||
Assert.Equal("ConfigMgr", entry["name"]);
|
||||
Assert.Equal(EspPolicyProviderProbe.KindSetupApps, entry["kind"]);
|
||||
Assert.Equal(15.0, entry["stalledForMinutes"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag()
|
||||
{
|
||||
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => SidecarCompleteConfigMgrIncomplete());
|
||||
var (sut, sink, clock) = BuildDetector();
|
||||
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
|
||||
sut.Tick();
|
||||
|
||||
var evt = Assert.Single(StalledEvents(sink));
|
||||
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
|
||||
Assert.Equal(true, data["sidecarRegistered"]);
|
||||
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
|
||||
Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]);
|
||||
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
|
||||
Assert.Equal(2, providers.Count); // full table always attached
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Latch_prevents_second_emission_for_the_same_provider()
|
||||
{
|
||||
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
|
||||
var (sut, sink, clock) = BuildDetector();
|
||||
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromMinutes(30));
|
||||
sut.Tick();
|
||||
sut.Tick();
|
||||
|
||||
Assert.Single(StalledEvents(sink));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Startup_gate_suppresses_same_stall_across_restart_but_reports_new_provider_set()
|
||||
{
|
||||
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
|
||||
|
||||
// Run 1: fires and persists the fingerprint.
|
||||
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
|
||||
{
|
||||
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
|
||||
sut.Tick();
|
||||
Assert.Single(StalledEvents(sink));
|
||||
}
|
||||
|
||||
// Run 2 (simulated restart — fresh detector, gate reloaded from the same state dir):
|
||||
// identical stall re-accrues dwell but lands on the same fingerprint → suppressed.
|
||||
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
|
||||
{
|
||||
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
|
||||
sut.Tick();
|
||||
Assert.Empty(StalledEvents(sink));
|
||||
}
|
||||
|
||||
// Run 3: a SECOND provider joins the stall → fingerprint changes → reported.
|
||||
var widened = new EspPolicyProviderSnapshot(new[]
|
||||
{
|
||||
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, null, null),
|
||||
new PolicyProviderState("Contoso", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
|
||||
});
|
||||
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => widened))
|
||||
{
|
||||
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
|
||||
sut.Tick();
|
||||
Assert.Single(StalledEvents(sink));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Healthy_or_absent_tracking_never_emits()
|
||||
{
|
||||
var healthy = new EspPolicyProviderSnapshot(new[]
|
||||
{
|
||||
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
|
||||
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindDevicePreparation, null, 3),
|
||||
});
|
||||
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => healthy))
|
||||
{
|
||||
var (sut, sink, clock) = BuildDetector();
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromHours(4));
|
||||
sut.Tick();
|
||||
Assert.Empty(StalledEvents(sink));
|
||||
}
|
||||
|
||||
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => EspPolicyProviderSnapshot.Empty))
|
||||
{
|
||||
var (sut, sink, clock) = BuildDetector();
|
||||
sut.Tick();
|
||||
clock.Advance(TimeSpan.FromHours(4));
|
||||
sut.Tick();
|
||||
Assert.Empty(StalledEvents(sink));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Net.Http;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
|
||||
{
|
||||
/// <summary>
|
||||
/// Pins the anti-flood semantics of <see cref="EmergencyReporter.TrySendAsync"/> after the
|
||||
/// delivery-hardening change (2026-07-23): the once-per-session dedup and the per-session
|
||||
/// budget count DELIVERED reports. The previous mark-before-send permanently suppressed an
|
||||
/// error key after its first transport failure — fatal for single-shot callers like the 48h
|
||||
/// emergency break, whose process exits right after the attempt.
|
||||
/// </summary>
|
||||
public class EmergencyReporterTests
|
||||
{
|
||||
private sealed class FakeApiClient : BackendApiClient
|
||||
{
|
||||
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
|
||||
public readonly Queue<bool> Results = new Queue<bool>();
|
||||
|
||||
public FakeApiClient() : base(
|
||||
httpClient: new HttpClient(),
|
||||
baseUrl: "http://localhost",
|
||||
manufacturer: string.Empty,
|
||||
model: string.Empty,
|
||||
serialNumber: string.Empty,
|
||||
useBootstrapTokenAuth: false,
|
||||
bootstrapToken: null,
|
||||
agentVersion: "0.0.0",
|
||||
logger: null)
|
||||
{
|
||||
}
|
||||
|
||||
public int Calls => Timeouts.Count;
|
||||
|
||||
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
|
||||
{
|
||||
Timeouts.Add(timeout);
|
||||
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
|
||||
}
|
||||
}
|
||||
|
||||
private static EmergencyReporter Reporter(FakeApiClient client)
|
||||
=> new EmergencyReporter(client, "session", "tenant", "0.0.0", logger: null);
|
||||
|
||||
[Fact]
|
||||
public async Task Failed_send_rolls_back_reservation_so_a_later_call_retries()
|
||||
{
|
||||
var client = new FakeApiClient();
|
||||
client.Results.Enqueue(false); // first call fails
|
||||
client.Results.Enqueue(true); // second call succeeds
|
||||
var reporter = Reporter(client);
|
||||
|
||||
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
|
||||
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
|
||||
|
||||
// Old behaviour: second call suppressed as "already reported" ⇒ 1 call.
|
||||
Assert.Equal(2, client.Calls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Delivered_send_is_deduplicated_per_key()
|
||||
{
|
||||
var client = new FakeApiClient();
|
||||
client.Results.Enqueue(true);
|
||||
var reporter = Reporter(client);
|
||||
|
||||
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
|
||||
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
|
||||
|
||||
Assert.Equal(1, client.Calls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Retries_stay_inside_one_reservation_and_stop_on_success()
|
||||
{
|
||||
var client = new FakeApiClient();
|
||||
client.Results.Enqueue(false);
|
||||
client.Results.Enqueue(true); // second attempt delivers — third must not fire
|
||||
var reporter = Reporter(client);
|
||||
|
||||
await reporter.TrySendAsync(
|
||||
AgentErrorType.SessionAgeEmergencyBreak, "m",
|
||||
attempts: 3, perAttemptTimeout: TimeSpan.FromMilliseconds(50), retryDelay: TimeSpan.Zero);
|
||||
|
||||
Assert.Equal(2, client.Calls);
|
||||
|
||||
// Delivered ⇒ dedup holds for the session.
|
||||
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
|
||||
Assert.Equal(2, client.Calls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Per_attempt_timeout_is_forwarded_to_the_api_client()
|
||||
{
|
||||
var client = new FakeApiClient();
|
||||
client.Results.Enqueue(true);
|
||||
var reporter = Reporter(client);
|
||||
|
||||
var timeout = TimeSpan.FromSeconds(15);
|
||||
await reporter.TrySendAsync(
|
||||
AgentErrorType.SessionAgeEmergencyBreak, "m",
|
||||
attempts: 1, perAttemptTimeout: timeout);
|
||||
|
||||
Assert.Equal(timeout, Assert.Single(client.Timeouts));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task All_attempts_failing_rolls_back_after_exhausting_attempts()
|
||||
{
|
||||
var client = new FakeApiClient(); // queue empty ⇒ every attempt fails
|
||||
var reporter = Reporter(client);
|
||||
|
||||
await reporter.TrySendAsync(
|
||||
AgentErrorType.SessionAgeEmergencyBreak, "m",
|
||||
attempts: 3, retryDelay: TimeSpan.Zero);
|
||||
Assert.Equal(3, client.Calls);
|
||||
|
||||
// Rolled back ⇒ a later call may try again (budget + dedup + cooldown restored).
|
||||
client.Results.Enqueue(true);
|
||||
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
|
||||
Assert.Equal(4, client.Calls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Different_keys_do_not_suppress_each_other_after_rollback()
|
||||
{
|
||||
var client = new FakeApiClient();
|
||||
client.Results.Enqueue(false); // break report fails → rollback (incl. cooldown restore)
|
||||
client.Results.Enqueue(true); // unrelated report must still go out
|
||||
var reporter = Reporter(client);
|
||||
|
||||
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
|
||||
await reporter.TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
|
||||
|
||||
Assert.Equal(2, client.Calls);
|
||||
}
|
||||
}
|
||||
}
|
||||
+178
@@ -0,0 +1,178 @@
|
||||
#nullable enable
|
||||
using System.Collections.Generic;
|
||||
using System.Text;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
|
||||
{
|
||||
/// <summary>
|
||||
/// Pins the pure parts of the UEFI Secure Boot certificate probe: the ASCII substring
|
||||
/// search over raw signature-list bytes and the <c>secureboot_status</c> payload shaping
|
||||
/// (in particular the one-sided <c>uefiCa2023FirmwareConfirmed</c> marker whose ABSENCE
|
||||
/// is load-bearing for ANALYZE-SEC-001's not_exists precondition). The actual firmware
|
||||
/// read is only smoke-tested: test machines may be VMs, non-elevated, or legacy BIOS,
|
||||
/// so only the fail-soft contract is asserted.
|
||||
/// </summary>
|
||||
public sealed class UefiSecureBootCertReaderTests
|
||||
{
|
||||
// --- ContainsAscii ---
|
||||
|
||||
[Fact]
|
||||
public void ContainsAscii_finds_needle_embedded_in_binary_noise()
|
||||
{
|
||||
var bytes = Concat(new byte[] { 0x00, 0xA1, 0x30, 0x82 },
|
||||
Encoding.ASCII.GetBytes("Windows UEFI CA 2023"),
|
||||
new byte[] { 0x00, 0xFF });
|
||||
|
||||
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ContainsAscii_finds_needle_at_the_very_end()
|
||||
{
|
||||
var bytes = Concat(new byte[] { 0x01, 0x02 }, Encoding.ASCII.GetBytes("KEK 2K CA 2023"));
|
||||
|
||||
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "KEK 2K CA 2023"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ContainsAscii_rejects_partial_overlap_and_absent_needle()
|
||||
{
|
||||
var bytes = Encoding.ASCII.GetBytes("Windows UEFI CA 2011 something else");
|
||||
|
||||
Assert.False(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
|
||||
// Repeated prefix bytes must not confuse the scan.
|
||||
var tricky = Encoding.ASCII.GetBytes("ababac");
|
||||
Assert.True(UefiSecureBootCertReader.ContainsAscii(tricky, "abac"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ContainsAscii_handles_null_empty_and_oversized_needle()
|
||||
{
|
||||
Assert.False(UefiSecureBootCertReader.ContainsAscii(null, "x"));
|
||||
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[0], "x"));
|
||||
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, ""));
|
||||
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, "abc"));
|
||||
}
|
||||
|
||||
// --- Payload shaping ---
|
||||
|
||||
[Fact]
|
||||
public void Marker_emitted_only_when_db_ok_and_ca2023_present()
|
||||
{
|
||||
var data = new Dictionary<string, object>();
|
||||
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
|
||||
{
|
||||
DbStatus = UefiSecureBootCertReader.StatusOk,
|
||||
KekStatus = UefiSecureBootCertReader.StatusOk,
|
||||
DbHasWindowsUefiCa2023 = true,
|
||||
KekHasMicrosoftKek2kCa2023 = true
|
||||
});
|
||||
|
||||
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
|
||||
Assert.Equal("ok", data["uefiFirmwareReadStatus"]);
|
||||
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
|
||||
Assert.Equal(true, data["uefiKekHasMicrosoftKek2kCa2023"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Marker_absent_when_ca2023_missing_even_though_read_succeeded()
|
||||
{
|
||||
var data = new Dictionary<string, object>();
|
||||
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
|
||||
{
|
||||
DbStatus = UefiSecureBootCertReader.StatusOk,
|
||||
KekStatus = UefiSecureBootCertReader.StatusOk,
|
||||
DbHasWindowsUefiCa2023 = false,
|
||||
DbHasWindowsProductionPca2011 = true
|
||||
});
|
||||
|
||||
// The marker must be OMITTED, not false — not_exists fails on any present value.
|
||||
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
|
||||
Assert.Equal(false, data["uefiDbHasWindowsUefiCa2023"]);
|
||||
Assert.Equal(true, data["uefiDbHasWindowsProductionPca2011"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Failed_db_read_emits_status_but_no_cert_booleans_and_no_marker()
|
||||
{
|
||||
var data = new Dictionary<string, object>();
|
||||
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
|
||||
{
|
||||
DbStatus = UefiSecureBootCertReader.StatusPrivilegeDenied,
|
||||
KekStatus = UefiSecureBootCertReader.StatusPrivilegeDenied
|
||||
});
|
||||
|
||||
Assert.Equal(UefiSecureBootCertReader.StatusPrivilegeDenied, data["uefiFirmwareReadStatus"]);
|
||||
Assert.False(data.ContainsKey("uefiDbHasWindowsUefiCa2023"));
|
||||
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
|
||||
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Db_ok_but_kek_failed_keeps_db_booleans_and_reports_kek_status()
|
||||
{
|
||||
var data = new Dictionary<string, object>();
|
||||
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
|
||||
{
|
||||
DbStatus = UefiSecureBootCertReader.StatusOk,
|
||||
KekStatus = UefiSecureBootCertReader.StatusVariableNotFound,
|
||||
DbHasWindowsUefiCa2023 = true
|
||||
});
|
||||
|
||||
Assert.Equal(UefiSecureBootCertReader.StatusVariableNotFound, data["uefiFirmwareReadStatus"]);
|
||||
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
|
||||
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
|
||||
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Null_snapshot_leaves_payload_untouched()
|
||||
{
|
||||
var data = new Dictionary<string, object> { { "uefiSecureBootEnabled", true } };
|
||||
DeviceInfoCollector.AppendFirmwareCertFields(data, null);
|
||||
|
||||
Assert.Single(data);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DescribeFirmwareVerdict_covers_all_shapes()
|
||||
{
|
||||
Assert.Equal("unavailable", DeviceInfoCollector.DescribeFirmwareVerdict(null));
|
||||
Assert.Equal(UefiSecureBootCertReader.StatusNotUefi, DeviceInfoCollector.DescribeFirmwareVerdict(
|
||||
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusNotUefi }));
|
||||
Assert.Equal("CA2023 confirmed", DeviceInfoCollector.DescribeFirmwareVerdict(
|
||||
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = true }));
|
||||
Assert.Equal("CA2023 missing", DeviceInfoCollector.DescribeFirmwareVerdict(
|
||||
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = false }));
|
||||
}
|
||||
|
||||
// --- Reader smoke (real firmware; machine state unknown) ---
|
||||
|
||||
[Fact]
|
||||
public void Read_never_throws_and_reports_a_status_for_both_variables()
|
||||
{
|
||||
var snapshot = UefiSecureBootCertReader.Read();
|
||||
|
||||
Assert.NotNull(snapshot);
|
||||
Assert.False(string.IsNullOrEmpty(snapshot.DbStatus));
|
||||
Assert.False(string.IsNullOrEmpty(snapshot.KekStatus));
|
||||
}
|
||||
|
||||
private static byte[] Concat(params byte[][] parts)
|
||||
{
|
||||
var total = 0;
|
||||
foreach (var part in parts) total += part.Length;
|
||||
var result = new byte[total];
|
||||
var offset = 0;
|
||||
foreach (var part in parts)
|
||||
{
|
||||
System.Array.Copy(part, 0, result, offset, part.Length);
|
||||
offset += part.Length;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
}
|
||||
}
|
||||
+75
@@ -61,6 +61,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
internal Func<int, string> SessionOwnerResolver { get; set; }
|
||||
internal Func<int, string> ProcessOwnerResolver { get; set; }
|
||||
|
||||
// Test seam for the WinRT OOBE-state sample (null = OobeStateReader.Read).
|
||||
internal Func<string> OobeStateProvider { get; set; }
|
||||
|
||||
// OOBE-state flip tracking (observational only — see OobeStateReader contract).
|
||||
// _oobeCompletedEmitted is one-shot per agent lifetime and deliberately NOT reset
|
||||
// by ResetForRealUserSwitch: the OOBE flip is a global OS transition, not per-user.
|
||||
private string _lastOobeState;
|
||||
private bool _oobeCompletedEmitted;
|
||||
|
||||
/// <summary>
|
||||
/// System/service account names that should NOT be considered real users.
|
||||
/// </summary>
|
||||
@@ -161,6 +170,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
{
|
||||
_startedAtUtc = DateTime.UtcNow;
|
||||
|
||||
// Baseline OOBE-state sample so the first poll can already detect a flip
|
||||
// (self-contained try/catch inside, like the Emit* helpers).
|
||||
CheckOobeStateFlip();
|
||||
|
||||
_pollingTimer?.Dispose();
|
||||
_pollingTimer = new Timer(
|
||||
PollForDesktop,
|
||||
@@ -186,6 +199,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
|
||||
_pollCount++;
|
||||
|
||||
CheckOobeStateFlip();
|
||||
|
||||
try
|
||||
{
|
||||
var explorerProcesses = Process.GetProcessesByName("explorer");
|
||||
@@ -381,6 +396,66 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
catch (Exception ex) { _logger.Verbose($"DesktopArrivalDetector: detector-started emit failed: {ex.Message}"); }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Samples the WinRT OOBE state (<see cref="OobeStateReader"/>) and emits the one-shot
|
||||
/// <c>oobe_state_completed</c> trace event on the in_progress->completed flip.
|
||||
/// Observational only — an owner-independent desktop corroboration for sessions where
|
||||
/// WTS+WMI owner resolution starves and <c>desktop_arrived</c> never fires. Often absent
|
||||
/// in healthy sessions: the polling timer is disposed at desktop arrival, which is when
|
||||
/// the flip lands (empirical probe, session 9c404ae9).
|
||||
/// <para>
|
||||
/// Self-contained try/catch: a throw here must NOT reach <see cref="PollForDesktop"/>'s
|
||||
/// outer catch, which would inflate <see cref="_wmiErrorCountSinceStart"/> and skew the
|
||||
/// session-4d5a0b78 liveness diagnostics. "unavailable" readings never overwrite the last
|
||||
/// known state, so a transient reflection failure between two polls cannot mask the flip.
|
||||
/// One-shot flag survives <see cref="ResetForRealUserSwitch"/> — the flip is a global OS
|
||||
/// transition, not per-user.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void CheckOobeStateFlip()
|
||||
{
|
||||
try
|
||||
{
|
||||
var current = (OobeStateProvider ?? OobeStateReader.Read)();
|
||||
if (string.IsNullOrEmpty(current) || current == OobeStateReader.Unavailable)
|
||||
return;
|
||||
|
||||
if (!_oobeCompletedEmitted
|
||||
&& _lastOobeState == "in_progress"
|
||||
&& current == "completed")
|
||||
{
|
||||
_oobeCompletedEmitted = true;
|
||||
var minutesSinceStart = Math.Round((DateTime.UtcNow - _startedAtUtc).TotalMinutes, 1);
|
||||
_logger.Info("DesktopArrivalDetector: OOBE state flipped in_progress->completed (WinRT SystemSetupInfo)");
|
||||
OnTraceEvent?.Invoke(
|
||||
SharedEventTypes.OobeStateCompleted,
|
||||
$"OOBE state flipped in_progress->completed (WinRT SystemSetupInfo, poll {_pollCount}) — owner-independent desktop corroboration.",
|
||||
new Dictionary<string, object>
|
||||
{
|
||||
{ "previousState", "in_progress" },
|
||||
{ "pollCount", _pollCount },
|
||||
{ "minutesSinceStart", minutesSinceStart },
|
||||
{ "desktopArrived", _desktopArrived },
|
||||
{ "wmiErrorCount", _wmiErrorCountSinceStart },
|
||||
{ "wtsErrorCount", _wtsErrorCountSinceStart },
|
||||
});
|
||||
}
|
||||
|
||||
_lastOobeState = current;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Verbose($"DesktopArrivalDetector: OOBE state check failed: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Drives a single synchronous poll for unit tests — the production poll is private and
|
||||
/// timer-driven with a fixed 30 s interval (precedent: <see cref="ResolveOwner"/> is
|
||||
/// likewise internal for direct testing).
|
||||
/// </summary>
|
||||
internal void PollOnceForTest() => PollForDesktop(null);
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the owning user of an explorer.exe candidate. Session 4d5a0b78 fix
|
||||
/// (2026-06-11): the WMI <c>Win32_Process.GetOwner</c> call failed on every poll on
|
||||
|
||||
+301
@@ -0,0 +1,301 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using Microsoft.Win32;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Session 5d735290 (2026-07-23) — reads the ESP's registered policy providers from
|
||||
/// <c>HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking</c>, the registry
|
||||
/// mirror of the EnrollmentStatusTracking CSP
|
||||
/// (https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp).
|
||||
/// <para>
|
||||
/// CSP contract the ESP enforces (quotes from the doc above):
|
||||
/// <list type="bullet">
|
||||
/// <item><c>Setup/Apps/PolicyProviders/<Name></c> — "Existence of this node indicates
|
||||
/// to the ESP that it shouldn't show the tracking status message until the
|
||||
/// TrackingPoliciesCreated node has been set to true." This wait has NO timeout: a
|
||||
/// registered provider that never sets <c>TrackingPoliciesCreated=1</c> parks the ESP at
|
||||
/// "Apps (Identifying)" indefinitely (observed for days in the field).</item>
|
||||
/// <item><c>DevicePreparation/PolicyProviders/<Name>/InstallationState</c> — must reach
|
||||
/// 2 (NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. ESP applies a default
|
||||
/// 15-minute timeout here, so stalls in this stage are self-surfacing — still reported
|
||||
/// for the diagnostic trail.</item>
|
||||
/// </list>
|
||||
/// Registry layout: <c>Device</c> subkey for device scope, <c>S-<SID></c> subkeys for user
|
||||
/// scope (appear at sign-in), each containing <c>Setup\Apps\PolicyProviders\<Name></c>;
|
||||
/// <c>DevicePreparation\PolicyProviders</c> exists in device scope only.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Known provider names: <c>Sidecar</c> is the Intune Management Extension ("Intune's agents,
|
||||
/// such as SideCar" per the CSP doc); <c>ConfigMgr</c> is the Configuration Manager client
|
||||
/// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in
|
||||
/// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft
|
||||
/// warns that ESP policy providers are not aware of each other and that pre-provisioning is
|
||||
/// unsupported with co-management. Field case that motivated this probe: a pre-provisioned
|
||||
/// device with the ConfigMgr client installed as a blocking app ended up with ONLY
|
||||
/// <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
|
||||
/// <c>Sidecar</c>) and <c>TrackingPoliciesCreated</c> never set — the user ESP hung at
|
||||
/// "Apps (Identifying)" for hours to days; manually renaming the key to <c>Sidecar</c>
|
||||
/// unblocked it immediately. <see cref="EspPolicyProviderStallDetector"/> turns this contract
|
||||
/// into the <c>esp_policy_provider_stalled</c> tripwire.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Fail-soft like <see cref="EspTrackingInfoProbe"/>: any failure is Debug-logged and yields
|
||||
/// <see cref="EspPolicyProviderSnapshot.Empty"/>; a missing root key (early enrollment,
|
||||
/// non-Autopilot device) is not an error. Values are parsed defensively — a missing or
|
||||
/// unparseable value counts as incomplete (fail-toward-observation).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static class EspPolicyProviderProbe
|
||||
{
|
||||
internal const string TrackingRootKeyPath =
|
||||
@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking";
|
||||
|
||||
/// <summary>Provider kind: <c>Setup\Apps\PolicyProviders</c> (TrackingPoliciesCreated wait, no timeout).</summary>
|
||||
internal const string KindSetupApps = "setupApps";
|
||||
|
||||
/// <summary>Provider kind: <c>DevicePreparation\PolicyProviders</c> (InstallationState wait, 15-min OS timeout).</summary>
|
||||
internal const string KindDevicePreparation = "devicePreparation";
|
||||
|
||||
/// <summary>The IME's provider name — its absence while other providers are registered is the field-case signature.</summary>
|
||||
internal const string SidecarProviderName = "Sidecar";
|
||||
|
||||
internal const string DeviceScope = "device";
|
||||
|
||||
private const string TrackingPoliciesCreatedValueName = "TrackingPoliciesCreated";
|
||||
private const string InstallationStateValueName = "InstallationState";
|
||||
|
||||
/// <summary>
|
||||
/// Test seam — when non-null, <see cref="Read"/> delegates to this func instead of
|
||||
/// touching the live registry. Use <see cref="ScopedOverride"/> to guarantee cleanup.
|
||||
/// </summary>
|
||||
internal static Func<AgentLogger, EspPolicyProviderSnapshot> TestOverride;
|
||||
|
||||
/// <summary>
|
||||
/// Reads all registered ESP policy providers. <see cref="EspPolicyProviderSnapshot.HasData"/>
|
||||
/// is <c>true</c> when the EnrollmentStatusTracking root key existed and was readable
|
||||
/// (even with no providers registered); <see cref="EspPolicyProviderSnapshot.Empty"/> otherwise.
|
||||
/// </summary>
|
||||
/// <param name="logger">Optional logger — Debug-level trace only; no warn/error.</param>
|
||||
public static EspPolicyProviderSnapshot Read(AgentLogger logger = null)
|
||||
{
|
||||
var probe = TestOverride;
|
||||
if (probe != null) return probe(logger);
|
||||
|
||||
try
|
||||
{
|
||||
using (var rootKey = Registry.LocalMachine.OpenSubKey(TrackingRootKeyPath))
|
||||
{
|
||||
if (rootKey == null)
|
||||
{
|
||||
logger?.Debug("EspPolicyProviderProbe: EnrollmentStatusTracking registry key not found");
|
||||
return EspPolicyProviderSnapshot.Empty;
|
||||
}
|
||||
|
||||
var providers = new List<PolicyProviderState>();
|
||||
|
||||
foreach (var scopeName in rootKey.GetSubKeyNames())
|
||||
{
|
||||
var isDevice = string.Equals(scopeName, "Device", StringComparison.OrdinalIgnoreCase);
|
||||
// User scope repeats the Setup structure under the signed-in user's SID.
|
||||
var isUserSid = scopeName.StartsWith("S-", StringComparison.OrdinalIgnoreCase);
|
||||
if (!isDevice && !isUserSid) continue;
|
||||
|
||||
var scopeLabel = isDevice ? DeviceScope : "user:" + scopeName;
|
||||
using (var scopeKey = rootKey.OpenSubKey(scopeName))
|
||||
{
|
||||
if (scopeKey == null) continue;
|
||||
|
||||
CollectProviders(
|
||||
scopeKey, @"Setup\Apps\PolicyProviders", scopeLabel, KindSetupApps,
|
||||
TrackingPoliciesCreatedValueName, providers, logger);
|
||||
|
||||
if (isDevice)
|
||||
{
|
||||
CollectProviders(
|
||||
scopeKey, @"DevicePreparation\PolicyProviders", scopeLabel, KindDevicePreparation,
|
||||
InstallationStateValueName, providers, logger);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return new EspPolicyProviderSnapshot(providers);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger?.Debug($"EspPolicyProviderProbe: read threw: {ex.Message}");
|
||||
return EspPolicyProviderSnapshot.Empty;
|
||||
}
|
||||
}
|
||||
|
||||
private static void CollectProviders(
|
||||
RegistryKey scopeKey,
|
||||
string providersSubPath,
|
||||
string scopeLabel,
|
||||
string kind,
|
||||
string stateValueName,
|
||||
List<PolicyProviderState> providers,
|
||||
AgentLogger logger)
|
||||
{
|
||||
try
|
||||
{
|
||||
using (var providersKey = scopeKey.OpenSubKey(providersSubPath))
|
||||
{
|
||||
if (providersKey == null) return;
|
||||
|
||||
foreach (var providerName in providersKey.GetSubKeyNames())
|
||||
{
|
||||
using (var providerKey = providersKey.OpenSubKey(providerName))
|
||||
{
|
||||
if (providerKey == null) continue;
|
||||
var value = TryReadInt(providerKey.GetValue(stateValueName));
|
||||
providers.Add(new PolicyProviderState(
|
||||
name: providerName,
|
||||
scope: scopeLabel,
|
||||
kind: kind,
|
||||
trackingPoliciesCreated: kind == KindSetupApps ? value : null,
|
||||
installationState: kind == KindDevicePreparation ? value : null));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger?.Debug($"EspPolicyProviderProbe: '{providersSubPath}' ({scopeLabel}) read threw: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Defensive value parse: REG_DWORD expected per CSP (booleans stored as 0/1), but string
|
||||
/// forms are tolerated. Anything missing or unparseable is <c>null</c> — which the
|
||||
/// completion rule treats as incomplete.
|
||||
/// </summary>
|
||||
internal static int? TryReadInt(object rawValue)
|
||||
{
|
||||
switch (rawValue)
|
||||
{
|
||||
case null: return null;
|
||||
case int i: return i;
|
||||
case long l when l >= int.MinValue && l <= int.MaxValue: return (int)l;
|
||||
case string s when int.TryParse(s.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed):
|
||||
return parsed;
|
||||
case bool b: return b ? 1 : 0;
|
||||
default: return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure completion rule per the CSP contract: <c>setupApps</c> providers are complete when
|
||||
/// <c>TrackingPoliciesCreated == 1</c>; <c>devicePreparation</c> providers when
|
||||
/// <c>InstallationState</c> is 2 (NotRequired) or 3 (Completed).
|
||||
/// </summary>
|
||||
internal static bool IsProviderComplete(string kind, int? trackingPoliciesCreated, int? installationState)
|
||||
=> kind == KindSetupApps
|
||||
? trackingPoliciesCreated == 1
|
||||
: installationState == 2 || installationState == 3;
|
||||
|
||||
/// <summary>
|
||||
/// Disposable scope that sets <see cref="TestOverride"/> for the lifetime of the scope
|
||||
/// and restores the previous value on Dispose. Nestable. Internal test-only helper.
|
||||
/// </summary>
|
||||
internal sealed class ScopedOverride : IDisposable
|
||||
{
|
||||
private readonly Func<AgentLogger, EspPolicyProviderSnapshot> _previous;
|
||||
private int _disposed;
|
||||
|
||||
public ScopedOverride(Func<AgentLogger, EspPolicyProviderSnapshot> probe)
|
||||
{
|
||||
_previous = TestOverride;
|
||||
TestOverride = probe;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (System.Threading.Interlocked.Exchange(ref _disposed, 1) == 1) return;
|
||||
TestOverride = _previous;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One registered ESP policy provider with its completion-relevant state.
|
||||
/// <see cref="Key"/> is the stable identity used for dwell tracking and one-shot latching.
|
||||
/// </summary>
|
||||
internal readonly struct PolicyProviderState
|
||||
{
|
||||
public PolicyProviderState(string name, string scope, string kind, int? trackingPoliciesCreated, int? installationState)
|
||||
{
|
||||
Name = name ?? string.Empty;
|
||||
Scope = scope ?? string.Empty;
|
||||
Kind = kind ?? string.Empty;
|
||||
TrackingPoliciesCreated = trackingPoliciesCreated;
|
||||
InstallationState = installationState;
|
||||
IsComplete = EspPolicyProviderProbe.IsProviderComplete(kind, trackingPoliciesCreated, installationState);
|
||||
}
|
||||
|
||||
/// <summary>Provider key name, e.g. "Sidecar" or "ConfigMgr".</summary>
|
||||
public string Name { get; }
|
||||
|
||||
/// <summary><c>"device"</c> or <c>"user:S-1-5-21-…"</c>.</summary>
|
||||
public string Scope { get; }
|
||||
|
||||
/// <summary><see cref="EspPolicyProviderProbe.KindSetupApps"/> or <see cref="EspPolicyProviderProbe.KindDevicePreparation"/>.</summary>
|
||||
public string Kind { get; }
|
||||
|
||||
/// <summary>Raw <c>TrackingPoliciesCreated</c> value (setupApps only); <c>null</c> when missing/unparseable.</summary>
|
||||
public int? TrackingPoliciesCreated { get; }
|
||||
|
||||
/// <summary>Raw <c>InstallationState</c> value (devicePreparation only); <c>null</c> when missing/unparseable.</summary>
|
||||
public int? InstallationState { get; }
|
||||
|
||||
/// <summary>Completion per <see cref="EspPolicyProviderProbe.IsProviderComplete"/>.</summary>
|
||||
public bool IsComplete { get; }
|
||||
|
||||
/// <summary>Stable dwell/latch identity: <c>kind|scope|name</c>.</summary>
|
||||
public string Key => Kind + "|" + Scope + "|" + Name;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Snapshot of all registered ESP policy providers. <see cref="HasData"/> is <c>false</c> only
|
||||
/// when the EnrollmentStatusTracking root key was missing or unreadable — an empty provider
|
||||
/// list with <see cref="HasData"/> = <c>true</c> is valid data (nothing registered yet).
|
||||
/// </summary>
|
||||
internal readonly struct EspPolicyProviderSnapshot
|
||||
{
|
||||
public static readonly EspPolicyProviderSnapshot Empty = default;
|
||||
|
||||
public EspPolicyProviderSnapshot(IReadOnlyList<PolicyProviderState> providers)
|
||||
{
|
||||
HasData = true;
|
||||
Providers = providers ?? Array.Empty<PolicyProviderState>();
|
||||
}
|
||||
|
||||
/// <summary><c>true</c> when the EnrollmentStatusTracking root key existed and was readable.</summary>
|
||||
public bool HasData { get; }
|
||||
|
||||
/// <summary>All registered providers across scopes and kinds; empty when none registered.</summary>
|
||||
public IReadOnlyList<PolicyProviderState> Providers { get; }
|
||||
|
||||
/// <summary>
|
||||
/// <c>true</c> when a <c>Setup\Apps</c> provider named "Sidecar" (the IME) is registered in
|
||||
/// any scope — its absence while other providers are registered is the co-management
|
||||
/// field-case signature.
|
||||
/// </summary>
|
||||
public bool SidecarRegistered
|
||||
{
|
||||
get
|
||||
{
|
||||
var providers = Providers;
|
||||
if (providers == null) return false;
|
||||
return providers.Any(p =>
|
||||
p.Kind == EspPolicyProviderProbe.KindSetupApps &&
|
||||
string.Equals(p.Name, EspPolicyProviderProbe.SidecarProviderName, StringComparison.OrdinalIgnoreCase));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+226
@@ -0,0 +1,226 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Persistence;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Turns the ESP policy-provider contract read by <see cref="EspPolicyProviderProbe"/> into
|
||||
/// the one-shot <c>esp_policy_provider_stalled</c> Warning: a registered provider that stays
|
||||
/// continuously incomplete for <see cref="DwellMinutes"/> minutes is reported once, with the
|
||||
/// full provider table in the payload. See the probe's header for the CSP contract, the
|
||||
/// official sources, and the co-management field case (ConfigMgr registered without Sidecar
|
||||
/// → user ESP parked at "Apps (Identifying)" with no OS timeout) this tripwire exists for.
|
||||
/// <para>
|
||||
/// Dwell semantics: the dwell clock per provider starts when it is first observed incomplete
|
||||
/// and RESETS when the provider completes or disappears — only <i>continuously</i> incomplete
|
||||
/// providers fire. A missing root key / empty provider list is normal early enrollment and
|
||||
/// clears all dwell state (probe D4). Purely observational: emission goes through
|
||||
/// <see cref="InformationalEventPost"/> (InformationalEvent pass-through, dispatch-guard
|
||||
/// exempt), never into decision state.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Duplicate suppression: in-process one-shot per provider key, plus a
|
||||
/// <see cref="StartupEventGate"/> fingerprint over the sorted set of currently-stalled keys so
|
||||
/// an agent restart into the same stall does not re-report it — while a NEW provider joining
|
||||
/// the stall changes the fingerprint and is reported (the event carries the full list).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class EspPolicyProviderStallDetector
|
||||
{
|
||||
/// <summary>
|
||||
/// Wall-clock minutes a provider must stay continuously incomplete before it is reported.
|
||||
/// Mirrors the CSP's own DevicePreparation default timeout; the Setup/Apps wait this
|
||||
/// tripwire mainly targets has no OS timeout at all.
|
||||
/// </summary>
|
||||
internal const int DwellMinutes = 15;
|
||||
|
||||
internal const string SourceLabel = "EspPolicyProviderStallDetector";
|
||||
|
||||
private readonly string _sessionId;
|
||||
private readonly string _tenantId;
|
||||
private readonly InformationalEventPost _post;
|
||||
private readonly AgentLogger _logger;
|
||||
private readonly IClock _clock;
|
||||
private readonly StartupEventGate? _startupGate;
|
||||
private readonly TimeSpan _dwell;
|
||||
|
||||
private readonly Dictionary<string, DateTime> _firstSeenIncompleteUtc =
|
||||
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
|
||||
private readonly HashSet<string> _firedKeys =
|
||||
new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
public EspPolicyProviderStallDetector(
|
||||
string sessionId,
|
||||
string tenantId,
|
||||
InformationalEventPost post,
|
||||
AgentLogger logger,
|
||||
IClock clock,
|
||||
StartupEventGate? startupGate = null,
|
||||
TimeSpan? dwell = null)
|
||||
{
|
||||
_sessionId = sessionId;
|
||||
_tenantId = tenantId;
|
||||
_post = post ?? throw new ArgumentNullException(nameof(post));
|
||||
_logger = logger;
|
||||
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
|
||||
_startupGate = startupGate;
|
||||
_dwell = dwell ?? TimeSpan.FromMinutes(DwellMinutes);
|
||||
}
|
||||
|
||||
/// <summary>Host timer callback and test seam — one probe + evaluation round.</summary>
|
||||
internal void Tick()
|
||||
{
|
||||
var snapshot = EspPolicyProviderProbe.Read(_logger);
|
||||
var eval = EvaluateStallTransition(snapshot, _firstSeenIncompleteUtc, _firedKeys, _clock.UtcNow, _dwell);
|
||||
|
||||
_firstSeenIncompleteUtc.Clear();
|
||||
foreach (var kv in eval.UpdatedFirstSeenUtc)
|
||||
_firstSeenIncompleteUtc[kv.Key] = kv.Value;
|
||||
|
||||
if (eval.NewlyStalled.Count > 0)
|
||||
EmitStalled(snapshot, eval);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure evaluation of one probe round — the entire detection decision, testable without
|
||||
/// registry, timer, or ingress (pattern: <c>PerformanceCollector.EvaluateDiskLowTransition</c>).
|
||||
/// </summary>
|
||||
/// <param name="snapshot">Current provider snapshot.</param>
|
||||
/// <param name="firstSeenIncompleteUtc">Dwell state from the previous round (provider key → first-seen-incomplete).</param>
|
||||
/// <param name="alreadyFired">Provider keys already reported this run (never re-report).</param>
|
||||
/// <param name="nowUtc">Evaluation timestamp.</param>
|
||||
/// <param name="dwell">Continuous-incompleteness threshold.</param>
|
||||
internal static StallEvaluation EvaluateStallTransition(
|
||||
EspPolicyProviderSnapshot snapshot,
|
||||
IReadOnlyDictionary<string, DateTime> firstSeenIncompleteUtc,
|
||||
ISet<string> alreadyFired,
|
||||
DateTime nowUtc,
|
||||
TimeSpan dwell)
|
||||
{
|
||||
var result = new StallEvaluation();
|
||||
|
||||
// Missing root key or no registered providers = normal early enrollment: clear all
|
||||
// dwell state (a provider must be continuously OBSERVED incomplete to accrue dwell).
|
||||
if (!snapshot.HasData || snapshot.Providers.Count == 0)
|
||||
return result;
|
||||
|
||||
foreach (var provider in snapshot.Providers)
|
||||
{
|
||||
if (provider.IsComplete)
|
||||
continue; // completed → falls out of the dwell map (reset)
|
||||
|
||||
var firstSeen = firstSeenIncompleteUtc.TryGetValue(provider.Key, out var seen)
|
||||
? seen
|
||||
: nowUtc;
|
||||
result.UpdatedFirstSeenUtc[provider.Key] = firstSeen;
|
||||
|
||||
var stalledFor = nowUtc - firstSeen;
|
||||
if (stalledFor >= dwell && !alreadyFired.Contains(provider.Key))
|
||||
result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes));
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval)
|
||||
{
|
||||
foreach (var stalled in eval.NewlyStalled)
|
||||
_firedKeys.Add(stalled.Provider.Key);
|
||||
|
||||
// Cross-restart dedup: fingerprint = sorted set of all fired keys that are still
|
||||
// incomplete. A restart into the same stall re-accrues dwell and lands on the same
|
||||
// fingerprint → suppressed; a new provider joining the stall changes it → re-emitted.
|
||||
var currentlyStalledKeys = snapshot.Providers
|
||||
.Where(p => !p.IsComplete && _firedKeys.Contains(p.Key))
|
||||
.Select(p => p.Key)
|
||||
.OrderBy(k => k, StringComparer.OrdinalIgnoreCase)
|
||||
.ToList();
|
||||
var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys));
|
||||
if (_startupGate != null &&
|
||||
!_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint))
|
||||
{
|
||||
_logger?.Debug(
|
||||
"EspPolicyProviderStallDetector: stalled provider set already reported by a " +
|
||||
"previous agent run (fingerprint match) — suppressed.");
|
||||
return;
|
||||
}
|
||||
|
||||
var stalledNames = string.Join(", ",
|
||||
eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})"));
|
||||
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["dwellMinutes"] = DwellMinutes,
|
||||
["sidecarRegistered"] = snapshot.SidecarRegistered,
|
||||
["providers"] = snapshot.Providers.Select(p => new Dictionary<string, object?>(StringComparer.Ordinal)
|
||||
{
|
||||
["name"] = p.Name,
|
||||
["scope"] = p.Scope,
|
||||
["kind"] = p.Kind,
|
||||
["trackingPoliciesCreated"] = p.TrackingPoliciesCreated,
|
||||
["installationState"] = p.InstallationState,
|
||||
["complete"] = p.IsComplete,
|
||||
}).ToList(),
|
||||
["stalledProviders"] = eval.NewlyStalled.Select(s => new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["name"] = s.Provider.Name,
|
||||
["scope"] = s.Provider.Scope,
|
||||
["kind"] = s.Provider.Kind,
|
||||
["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1),
|
||||
}).ToList(),
|
||||
};
|
||||
|
||||
_post.Emit(new EnrollmentEvent
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
TenantId = _tenantId,
|
||||
Timestamp = _clock.UtcNow,
|
||||
EventType = Constants.EventTypes.EspPolicyProviderStalled,
|
||||
Severity = EventSeverity.Warning,
|
||||
Source = SourceLabel,
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
ImmediateUpload = true,
|
||||
Message = $"ESP policy provider(s) incomplete for >= {DwellMinutes} min: {stalledNames} " +
|
||||
$"(sidecarRegistered={snapshot.SidecarRegistered}) — the ESP waits on " +
|
||||
"Setup/Apps providers without any timeout",
|
||||
Data = data
|
||||
});
|
||||
_startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled);
|
||||
|
||||
_logger?.Warning(
|
||||
$"EspPolicyProviderStallDetector: {stalledNames} incomplete for >= {DwellMinutes} min " +
|
||||
$"(sidecarRegistered={snapshot.SidecarRegistered}).");
|
||||
}
|
||||
|
||||
/// <summary>Result of one <see cref="EvaluateStallTransition"/> round.</summary>
|
||||
internal sealed class StallEvaluation
|
||||
{
|
||||
/// <summary>Next round's dwell state — only currently-incomplete providers survive.</summary>
|
||||
public Dictionary<string, DateTime> UpdatedFirstSeenUtc { get; } =
|
||||
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported.</summary>
|
||||
public List<StalledProvider> NewlyStalled { get; } = new List<StalledProvider>();
|
||||
}
|
||||
|
||||
internal readonly struct StalledProvider
|
||||
{
|
||||
public StalledProvider(PolicyProviderState provider, double stalledForMinutes)
|
||||
{
|
||||
Provider = provider;
|
||||
StalledForMinutes = stalledForMinutes;
|
||||
}
|
||||
|
||||
public PolicyProviderState Provider { get; }
|
||||
public double StalledForMinutes { get; }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
using System;
|
||||
using System.Reflection;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
|
||||
{
|
||||
/// <summary>
|
||||
/// Reads WinRT <c>Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState</c>
|
||||
/// (NotStarted=0 / InProgress=1 / Completed=2) via the CLR's built-in WinRT projection —
|
||||
/// no NuGet package, no winmd reference; the type is resolved at runtime with
|
||||
/// <c>ContentType=WindowsRuntime</c>, the same mechanism the bootstrap script uses from
|
||||
/// PowerShell 5.1.
|
||||
/// <para>
|
||||
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
|
||||
/// Requires Windows 10 1809+ (build 17763, UniversalApiContract 7.0) and is callable
|
||||
/// from the SYSTEM service context (empirically validated, probe session 9c404ae9).
|
||||
/// On older builds or SKUs without the contract the type resolves to null and
|
||||
/// <see cref="Read"/> returns <c>"unavailable"</c>. <see cref="Read"/> never throws.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Empirical timing (Win11 24H2 probe): the state flips InProgress→Completed at desktop
|
||||
/// arrival — it is a corroboration signal, not an early one. WDP reseal and placeholder
|
||||
/// (fooUser) flip semantics are unverified; another reason this must stay observational.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static class OobeStateReader
|
||||
{
|
||||
public const string Unavailable = "unavailable";
|
||||
|
||||
// Resolved once per process. Benign race: concurrent first calls resolve the same
|
||||
// PropertyInfo and the assignment is idempotent — no lock needed.
|
||||
private static PropertyInfo _stateProperty;
|
||||
private static bool _initTried;
|
||||
|
||||
/// <summary>
|
||||
/// Returns <c>"not_started"</c> / <c>"in_progress"</c> / <c>"completed"</c>,
|
||||
/// <c>"unknown_<n>"</c> for unexpected enum values, or <c>"unavailable"</c>
|
||||
/// when the WinRT contract is absent or any part of the read fails. Never throws.
|
||||
/// </summary>
|
||||
public static string Read()
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!_initTried)
|
||||
{
|
||||
var type = Type.GetType(
|
||||
"Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType=WindowsRuntime",
|
||||
throwOnError: false);
|
||||
_stateProperty = type?.GetProperty(
|
||||
"OutOfBoxExperienceState",
|
||||
BindingFlags.Public | BindingFlags.Static);
|
||||
_initTried = true;
|
||||
}
|
||||
|
||||
if (_stateProperty == null)
|
||||
return Unavailable;
|
||||
|
||||
var value = Convert.ToInt32(_stateProperty.GetValue(null));
|
||||
switch (value)
|
||||
{
|
||||
case 0: return "not_started";
|
||||
case 1: return "in_progress";
|
||||
case 2: return "completed";
|
||||
default: return "unknown_" + value;
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
return Unavailable;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+316
@@ -0,0 +1,316 @@
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
|
||||
{
|
||||
/// <summary>
|
||||
/// Reads the UEFI Secure Boot signature databases <c>db</c> and <c>KEK</c> directly from
|
||||
/// firmware via <c>GetFirmwareEnvironmentVariableExW</c> and checks which Microsoft
|
||||
/// signing certificates they contain. This is the authoritative source for the 2023
|
||||
/// certificate rollout state — the <c>SecureBoot\Servicing</c> registry key only reflects
|
||||
/// whether the *Windows Update* rollout ran and verified, and lags behind when the
|
||||
/// certificate reached the firmware another way (OEM factory image, WinCS, manual).
|
||||
/// <para>
|
||||
/// The certificate subject CNs are stored as ASCII bytes inside the DER-encoded
|
||||
/// signature lists, so a plain byte-substring search is sufficient — the same check
|
||||
/// Microsoft documents via <c>Get-SecureBootUEFI</c> (which wraps this exact API).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
|
||||
/// <see cref="Read"/> never throws. Reading firmware variables requires
|
||||
/// <c>SeSystemEnvironmentPrivilege</c>, which SYSTEM holds but must be enabled on the
|
||||
/// process token first; the previous privilege state is restored after the read.
|
||||
/// On legacy BIOS the API fails with <c>ERROR_INVALID_FUNCTION</c> → status
|
||||
/// <c>not_uefi</c>; every failure maps to a status string instead of an exception.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static class UefiSecureBootCertReader
|
||||
{
|
||||
public const string StatusOk = "ok";
|
||||
public const string StatusNotUefi = "not_uefi";
|
||||
public const string StatusVariableNotFound = "variable_not_found";
|
||||
public const string StatusPrivilegeDenied = "privilege_denied";
|
||||
|
||||
// EFI_IMAGE_SECURITY_DATABASE_GUID — namespace of the "db" variable.
|
||||
private const string DbVariableGuid = "{d719b2cb-3d3a-4596-a3bc-dad00e67656f}";
|
||||
// EFI_GLOBAL_VARIABLE — namespace of the "KEK" variable.
|
||||
private const string KekVariableGuid = "{8be4df61-93ca-11d2-aa0d-00e098032b8c}";
|
||||
|
||||
// Subject CNs searched for in db. 2023 set per Microsoft's June-2026 expiry guidance;
|
||||
// 2011 set kept for context (which legacy CAs are still present).
|
||||
internal const string CertWindowsUefiCa2023 = "Windows UEFI CA 2023";
|
||||
internal const string CertMicrosoftUefiCa2023 = "Microsoft UEFI CA 2023";
|
||||
internal const string CertMicrosoftOptionRomCa2023 = "Microsoft Option ROM UEFI CA 2023";
|
||||
internal const string CertWindowsProductionPca2011 = "Microsoft Windows Production PCA 2011";
|
||||
internal const string CertMicrosoftUefiCa2011 = "Microsoft Corporation UEFI CA 2011";
|
||||
// Subject CNs searched for in KEK.
|
||||
internal const string CertMicrosoftKek2kCa2023 = "Microsoft Corporation KEK 2K CA 2023";
|
||||
internal const string CertMicrosoftKekCa2011 = "Microsoft Corporation KEK CA 2011";
|
||||
|
||||
private const int ErrorInvalidFunction = 1; // legacy BIOS / no UEFI
|
||||
private const int ErrorInsufficientBuffer = 122;
|
||||
private const int ErrorEnvvarNotFound = 203;
|
||||
private const int ErrorPrivilegeNotHeld = 1314;
|
||||
private const int ErrorNotAllAssigned = 1300;
|
||||
|
||||
private const int InitialBufferSize = 64 * 1024;
|
||||
private const int RetryBufferSize = 256 * 1024;
|
||||
|
||||
/// <summary>
|
||||
/// Reads db and KEK and evaluates the certificate presence flags. Never throws;
|
||||
/// per-variable failures surface as <see cref="UefiSecureBootCertSnapshot.DbStatus"/> /
|
||||
/// <see cref="UefiSecureBootCertSnapshot.KekStatus"/>. Certificate flags are only
|
||||
/// meaningful when the corresponding status is <see cref="StatusOk"/>.
|
||||
/// </summary>
|
||||
public static UefiSecureBootCertSnapshot Read()
|
||||
{
|
||||
var snapshot = new UefiSecureBootCertSnapshot();
|
||||
try
|
||||
{
|
||||
using (var privilege = SystemEnvironmentPrivilege.TryEnable())
|
||||
{
|
||||
if (!privilege.Enabled)
|
||||
{
|
||||
snapshot.DbStatus = StatusPrivilegeDenied;
|
||||
snapshot.KekStatus = StatusPrivilegeDenied;
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
byte[] db;
|
||||
snapshot.DbStatus = TryReadVariable("db", DbVariableGuid, out db);
|
||||
if (snapshot.DbStatus == StatusOk)
|
||||
{
|
||||
snapshot.DbHasWindowsUefiCa2023 = ContainsAscii(db, CertWindowsUefiCa2023);
|
||||
snapshot.DbHasMicrosoftUefiCa2023 = ContainsAscii(db, CertMicrosoftUefiCa2023);
|
||||
snapshot.DbHasMicrosoftOptionRomCa2023 = ContainsAscii(db, CertMicrosoftOptionRomCa2023);
|
||||
snapshot.DbHasWindowsProductionPca2011 = ContainsAscii(db, CertWindowsProductionPca2011);
|
||||
snapshot.DbHasMicrosoftUefiCa2011 = ContainsAscii(db, CertMicrosoftUefiCa2011);
|
||||
}
|
||||
|
||||
byte[] kek;
|
||||
snapshot.KekStatus = TryReadVariable("KEK", KekVariableGuid, out kek);
|
||||
if (snapshot.KekStatus == StatusOk)
|
||||
{
|
||||
snapshot.KekHasMicrosoftKek2kCa2023 = ContainsAscii(kek, CertMicrosoftKek2kCa2023);
|
||||
snapshot.KekHasMicrosoftKekCa2011 = ContainsAscii(kek, CertMicrosoftKekCa2011);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
var status = "error_" + ex.GetType().Name;
|
||||
if (snapshot.DbStatus == null) snapshot.DbStatus = status;
|
||||
if (snapshot.KekStatus == null) snapshot.KekStatus = status;
|
||||
}
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads one firmware variable into <paramref name="bytes"/> (trimmed to the actual
|
||||
/// length). Returns a status string; <paramref name="bytes"/> is null unless the
|
||||
/// status is <see cref="StatusOk"/>.
|
||||
/// </summary>
|
||||
private static string TryReadVariable(string name, string guid, out byte[] bytes)
|
||||
{
|
||||
bytes = null;
|
||||
var buffer = new byte[InitialBufferSize];
|
||||
var length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
|
||||
if (length == 0)
|
||||
{
|
||||
var error = Marshal.GetLastWin32Error();
|
||||
if (error == ErrorInsufficientBuffer)
|
||||
{
|
||||
buffer = new byte[RetryBufferSize];
|
||||
length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
|
||||
if (length == 0)
|
||||
error = Marshal.GetLastWin32Error();
|
||||
}
|
||||
|
||||
if (length == 0)
|
||||
{
|
||||
switch (error)
|
||||
{
|
||||
case ErrorInvalidFunction: return StatusNotUefi;
|
||||
case ErrorEnvvarNotFound: return StatusVariableNotFound;
|
||||
case ErrorPrivilegeNotHeld: return StatusPrivilegeDenied;
|
||||
default: return "error_" + error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bytes = new byte[length];
|
||||
Array.Copy(buffer, bytes, (int)length);
|
||||
return StatusOk;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Byte-level ASCII substring search (ordinal, case-sensitive). The signature lists
|
||||
/// are at most a few hundred KB and read once per agent start — a naive scan is fine.
|
||||
/// </summary>
|
||||
internal static bool ContainsAscii(byte[] haystack, string needle)
|
||||
{
|
||||
if (haystack == null || string.IsNullOrEmpty(needle) || needle.Length > haystack.Length)
|
||||
return false;
|
||||
|
||||
for (var i = 0; i <= haystack.Length - needle.Length; i++)
|
||||
{
|
||||
var j = 0;
|
||||
while (j < needle.Length && haystack[i + j] == (byte)needle[j])
|
||||
j++;
|
||||
if (j == needle.Length)
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Enables <c>SeSystemEnvironmentPrivilege</c> on the process token for the lifetime
|
||||
/// of the instance and restores the previous privilege state on dispose. SYSTEM holds
|
||||
/// the privilege (default-disabled); without elevation <see cref="Enabled"/> stays
|
||||
/// false and the caller reports <see cref="StatusPrivilegeDenied"/>.
|
||||
/// </summary>
|
||||
private sealed class SystemEnvironmentPrivilege : IDisposable
|
||||
{
|
||||
public bool Enabled { get; private set; }
|
||||
|
||||
private IntPtr _token = IntPtr.Zero;
|
||||
private NativeMethods.TOKEN_PRIVILEGES _previousState;
|
||||
private bool _restoreNeeded;
|
||||
|
||||
public static SystemEnvironmentPrivilege TryEnable()
|
||||
{
|
||||
var instance = new SystemEnvironmentPrivilege();
|
||||
try
|
||||
{
|
||||
if (!NativeMethods.OpenProcessToken(NativeMethods.GetCurrentProcess(),
|
||||
NativeMethods.TOKEN_ADJUST_PRIVILEGES | NativeMethods.TOKEN_QUERY, out instance._token))
|
||||
return instance;
|
||||
|
||||
NativeMethods.LUID luid;
|
||||
if (!NativeMethods.LookupPrivilegeValueW(null, "SeSystemEnvironmentPrivilege", out luid))
|
||||
return instance;
|
||||
|
||||
var newState = new NativeMethods.TOKEN_PRIVILEGES
|
||||
{
|
||||
PrivilegeCount = 1,
|
||||
Luid = luid,
|
||||
Attributes = NativeMethods.SE_PRIVILEGE_ENABLED
|
||||
};
|
||||
|
||||
uint returnLength;
|
||||
if (!NativeMethods.AdjustTokenPrivileges(instance._token, false, ref newState,
|
||||
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
|
||||
out instance._previousState, out returnLength))
|
||||
return instance;
|
||||
|
||||
// AdjustTokenPrivileges returns TRUE even when the privilege is not held —
|
||||
// the real verdict is ERROR_NOT_ALL_ASSIGNED in the last error.
|
||||
if (Marshal.GetLastWin32Error() == ErrorNotAllAssigned)
|
||||
return instance;
|
||||
|
||||
// Restore only when the privilege was not already enabled before.
|
||||
instance._restoreNeeded = instance._previousState.PrivilegeCount > 0
|
||||
&& (instance._previousState.Attributes & NativeMethods.SE_PRIVILEGE_ENABLED) == 0;
|
||||
instance.Enabled = true;
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Fail-soft: Enabled stays false.
|
||||
}
|
||||
return instance;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (_token != IntPtr.Zero)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (_restoreNeeded)
|
||||
{
|
||||
NativeMethods.TOKEN_PRIVILEGES ignored;
|
||||
uint returnLength;
|
||||
NativeMethods.AdjustTokenPrivileges(_token, false, ref _previousState,
|
||||
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
|
||||
out ignored, out returnLength);
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Best effort — the process token privilege state is not load-bearing
|
||||
// for anything else the agent does.
|
||||
}
|
||||
NativeMethods.CloseHandle(_token);
|
||||
_token = IntPtr.Zero;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static class NativeMethods
|
||||
{
|
||||
public const uint TOKEN_ADJUST_PRIVILEGES = 0x0020;
|
||||
public const uint TOKEN_QUERY = 0x0008;
|
||||
public const uint SE_PRIVILEGE_ENABLED = 0x0002;
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
public struct LUID
|
||||
{
|
||||
public uint LowPart;
|
||||
public int HighPart;
|
||||
}
|
||||
|
||||
// Single-privilege variant of the variable-length native struct — sufficient here
|
||||
// because exactly one privilege is adjusted per call.
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
public struct TOKEN_PRIVILEGES
|
||||
{
|
||||
public uint PrivilegeCount;
|
||||
public LUID Luid;
|
||||
public uint Attributes;
|
||||
}
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
public static extern uint GetFirmwareEnvironmentVariableExW(
|
||||
string lpName, string lpGuid, byte[] pBuffer, uint nSize, IntPtr pdwAttributes);
|
||||
|
||||
[DllImport("kernel32.dll")]
|
||||
public static extern IntPtr GetCurrentProcess();
|
||||
|
||||
[DllImport("advapi32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
public static extern bool OpenProcessToken(IntPtr processHandle, uint desiredAccess, out IntPtr tokenHandle);
|
||||
|
||||
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
public static extern bool LookupPrivilegeValueW(string lpSystemName, string lpName, out LUID lpLuid);
|
||||
|
||||
[DllImport("advapi32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
public static extern bool AdjustTokenPrivileges(IntPtr tokenHandle, bool disableAllPrivileges,
|
||||
ref TOKEN_PRIVILEGES newState, uint bufferLength, out TOKEN_PRIVILEGES previousState, out uint returnLength);
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
public static extern bool CloseHandle(IntPtr handle);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Result of one firmware read. Certificate flags are only meaningful when the
|
||||
/// corresponding status equals <see cref="UefiSecureBootCertReader.StatusOk"/>.
|
||||
/// </summary>
|
||||
internal sealed class UefiSecureBootCertSnapshot
|
||||
{
|
||||
public string DbStatus { get; set; }
|
||||
public string KekStatus { get; set; }
|
||||
|
||||
public bool DbHasWindowsUefiCa2023 { get; set; }
|
||||
public bool DbHasMicrosoftUefiCa2023 { get; set; }
|
||||
public bool DbHasMicrosoftOptionRomCa2023 { get; set; }
|
||||
public bool DbHasWindowsProductionPca2011 { get; set; }
|
||||
public bool DbHasMicrosoftUefiCa2011 { get; set; }
|
||||
|
||||
public bool KekHasMicrosoftKek2kCa2023 { get; set; }
|
||||
public bool KekHasMicrosoftKekCa2011 { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -31,84 +31,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
|
||||
{
|
||||
_logger.Info($"Executing FULL SELF-DESTRUCT (Scheduled Task + Files{(_configuration.RebootOnComplete ? " + Reboot" : "")})");
|
||||
|
||||
var currentProcessId = Process.GetCurrentProcess().Id;
|
||||
var currentProcess = Process.GetCurrentProcess();
|
||||
var agentBasePath = Path.GetFullPath(Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
"AutopilotMonitor"
|
||||
));
|
||||
|
||||
var logDir = Path.GetFullPath(Path.Combine(agentBasePath, "Logs"));
|
||||
var keepLogs = _configuration.KeepLogFile;
|
||||
|
||||
// Create a self-deleting PowerShell script
|
||||
var cleanupScript = $@"
|
||||
$scriptPath = $MyInvocation.MyCommand.Path
|
||||
|
||||
# Wait for agent process to exit
|
||||
Start-Sleep -Seconds 2
|
||||
Wait-Process -Id {currentProcessId} -ErrorAction SilentlyContinue -Timeout 30
|
||||
Start-Sleep -Seconds 1
|
||||
|
||||
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan task firing
|
||||
# against a deleted exe spams Task Scheduler errors every boot forever and, once the files (incl.
|
||||
# the enrollment-complete marker) are gone, has no cleanup-retry path. So confirm the task is gone
|
||||
# first; if it cannot be removed, SKIP the file deletion and leave everything intact for a clean
|
||||
# cleanup retry on the next boot (the marker survives, so bootstrap re-runs cleanup).
|
||||
$taskName = '{_configuration.ScheduledTaskName}'
|
||||
$taskGone = $false
|
||||
for ($i = 1; $i -le 5; $i++) {{
|
||||
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
$still = $null
|
||||
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
if (-not $still) {{ $taskGone = $true; break }}
|
||||
Start-Sleep -Seconds 1
|
||||
}}
|
||||
if (-not $taskGone) {{
|
||||
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
|
||||
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
|
||||
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
|
||||
}}
|
||||
|
||||
{(keepLogs ? $@"
|
||||
if ($taskGone) {{
|
||||
# Delete everything except Logs directory
|
||||
Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' | ForEach-Object {{
|
||||
$dest = $_.FullName + '.del'
|
||||
try {{ Rename-Item -Path $_.FullName -NewName $dest -Force -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
Remove-Item -Path $dest -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}}
|
||||
}}
|
||||
" : $@"
|
||||
if ($taskGone) {{
|
||||
# Rename folder then delete. Retry a few times to let the OS release handles.
|
||||
$renamedPath = '{agentBasePath}.del'
|
||||
$renamed = $false
|
||||
for ($i = 1; $i -le 10; $i++) {{
|
||||
try {{
|
||||
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
|
||||
$renamed = $true
|
||||
break
|
||||
}} catch {{
|
||||
Start-Sleep -Seconds 2
|
||||
}}
|
||||
}}
|
||||
if ($renamed) {{
|
||||
Remove-Item -Path $renamedPath -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}} else {{
|
||||
Remove-Item -Path '{agentBasePath}' -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}}
|
||||
}}
|
||||
")}
|
||||
{(_configuration.RebootOnComplete ? @"
|
||||
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
|
||||
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
|
||||
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
|
||||
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
|
||||
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
|
||||
" : "")}
|
||||
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
|
||||
";
|
||||
var cleanupScript = BuildCleanupScript(
|
||||
agentProcessName: currentProcess.ProcessName,
|
||||
agentBasePath: agentBasePath,
|
||||
keepLogs: _configuration.KeepLogFile,
|
||||
scheduledTaskName: _configuration.ScheduledTaskName,
|
||||
rebootOnComplete: _configuration.RebootOnComplete);
|
||||
|
||||
// Write cleanup script to temp location (outside of agent folder)
|
||||
var tempScriptPath = Path.Combine(Path.GetTempPath(), $"AutopilotMonitor-Cleanup-{Guid.NewGuid()}.ps1");
|
||||
@@ -143,5 +77,171 @@ Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the self-deleting cleanup PowerShell script. Extracted (internal static, pure)
|
||||
/// so tests can pin the hardened invariants without launching processes.
|
||||
/// <para>
|
||||
/// LIFE-F7 hardening (incident 2026-07-22 — exe + loaded DLLs orphaned forever):
|
||||
/// </para>
|
||||
/// <list type="number">
|
||||
/// <item><b>Single-flight lock</b> — the enrollment-complete marker is written BEFORE
|
||||
/// cleanup launches, so any agent start during an in-flight cleanup spawns a SECOND
|
||||
/// cleanup script. Two unsynchronized scripts interleave their rename/delete phases
|
||||
/// and leave partial trees behind. A CreateNew lock file in the SYSTEM temp dir
|
||||
/// serializes them; a stale lock (>10 min) is taken over.</item>
|
||||
/// <item><b>Wait by process NAME, not launcher PID</b> — marker-retry instances,
|
||||
/// self-update restarts and manual runs all load the same images but have PIDs the
|
||||
/// old <c>Wait-Process -Id</c> never saw (and an already-freed PID can be reused by
|
||||
/// an unrelated boot-time process). After the deadline any straggler is force-killed:
|
||||
/// its images are about to be deleted anyway.</item>
|
||||
/// <item><b>Probe-rename BEFORE task removal, never delete the original path</b> — the
|
||||
/// rename is the atomic lock probe. If it fails, something still holds the tree: abort
|
||||
/// with task + files + marker intact so the next boot retries (file-side symmetry to
|
||||
/// the LIFE-F6 task-side rule). If the task cannot be removed afterwards, rename BACK
|
||||
/// and abort for the same reason. Only a fully renamed tree with the task gone is
|
||||
/// deleted — the old <c>Remove-Item</c>-on-original fallback deleted marker + state
|
||||
/// around locked binaries AFTER the task was gone, converting a transient lock into a
|
||||
/// permanently orphaned exe.</item>
|
||||
/// </list>
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The timing parameters exist ONLY so integration tests can replay the incident
|
||||
/// scenarios (locked tree, straggler process) in seconds instead of minutes —
|
||||
/// production call sites always use the defaults.
|
||||
/// </remarks>
|
||||
internal static string BuildCleanupScript(
|
||||
string agentProcessName,
|
||||
string agentBasePath,
|
||||
bool keepLogs,
|
||||
string scheduledTaskName,
|
||||
bool rebootOnComplete,
|
||||
int initialDelaySeconds = 2,
|
||||
int processWaitSeconds = 60,
|
||||
int renameRetries = 10,
|
||||
int renameRetryDelaySeconds = 2)
|
||||
{
|
||||
return $@"
|
||||
$scriptPath = $MyInvocation.MyCommand.Path
|
||||
|
||||
# Single-flight lock (LIFE-F7 #1): serialize concurrent cleanup scripts. CreateNew is
|
||||
# atomic; the loser backs off and lets the winner own the deletion.
|
||||
$lockPath = Join-Path ([System.IO.Path]::GetTempPath()) 'AutopilotMonitor-Cleanup.lock'
|
||||
$lockStream = $null
|
||||
try {{
|
||||
if (Test-Path $lockPath) {{
|
||||
$lockAge = (Get-Date) - (Get-Item $lockPath -ErrorAction Stop).LastWriteTime
|
||||
if ($lockAge.TotalMinutes -gt 10) {{ Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue }}
|
||||
}}
|
||||
$lockStream = [System.IO.File]::Open($lockPath, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
|
||||
}} catch {{
|
||||
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
|
||||
exit 0
|
||||
}}
|
||||
|
||||
try {{
|
||||
# Wait for EVERY agent process to exit, by NAME (LIFE-F7 #2) — covers marker-retry
|
||||
# instances, self-update restarts and manual runs, and is immune to PID reuse.
|
||||
Start-Sleep -Seconds {initialDelaySeconds}
|
||||
$deadline = (Get-Date).AddSeconds({processWaitSeconds})
|
||||
while ((Get-Date) -lt $deadline) {{
|
||||
if (-not (Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue)) {{ break }}
|
||||
Start-Sleep -Seconds 2
|
||||
}}
|
||||
# Backstop: force-kill stragglers — their images are about to be deleted anyway.
|
||||
Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
|
||||
Start-Sleep -Seconds 1
|
||||
|
||||
{(keepLogs ? $@"
|
||||
# Probe-rename all children except Logs (LIFE-F7 #3). Any failure ⇒ rename back and
|
||||
# abort with the Scheduled Task intact so the next boot retries cleanly.
|
||||
$renamed = @()
|
||||
$probeOk = $true
|
||||
foreach ($item in @(Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' -ErrorAction SilentlyContinue)) {{
|
||||
$dest = $item.FullName + '.del-' + [Guid]::NewGuid().ToString('N')
|
||||
$itemOk = $false
|
||||
for ($i = 1; $i -le {renameRetries}; $i++) {{
|
||||
try {{
|
||||
Rename-Item -Path $item.FullName -NewName $dest -Force -ErrorAction Stop
|
||||
$itemOk = $true
|
||||
break
|
||||
}} catch {{
|
||||
Start-Sleep -Seconds {renameRetryDelaySeconds}
|
||||
}}
|
||||
}}
|
||||
if ($itemOk) {{ $renamed += @{{ Original = $item.FullName; Renamed = $dest }} }}
|
||||
else {{ $probeOk = $false; break }}
|
||||
}}
|
||||
" : $@"
|
||||
# Probe-rename the whole tree (LIFE-F7 #3). Failure ⇒ abort with task + files + marker
|
||||
# intact so the next boot retries cleanly. GUID suffix avoids collisions with leftovers
|
||||
# of earlier attempts.
|
||||
$renamedPath = '{agentBasePath}.del-' + [Guid]::NewGuid().ToString('N')
|
||||
$probeOk = $false
|
||||
for ($i = 1; $i -le {renameRetries}; $i++) {{
|
||||
try {{
|
||||
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
|
||||
$probeOk = $true
|
||||
break
|
||||
}} catch {{
|
||||
Start-Sleep -Seconds {renameRetryDelaySeconds}
|
||||
}}
|
||||
}}
|
||||
$renamed = @(@{{ Original = '{agentBasePath}'; Renamed = $renamedPath }})
|
||||
")}
|
||||
if ($probeOk) {{
|
||||
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan
|
||||
# task firing against a deleted exe spams Task Scheduler errors every boot forever
|
||||
# and, once the files (incl. the enrollment-complete marker) are gone, has no
|
||||
# cleanup-retry path.
|
||||
$taskName = '{scheduledTaskName}'
|
||||
$taskGone = $false
|
||||
for ($i = 1; $i -le 5; $i++) {{
|
||||
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
$still = $null
|
||||
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
if (-not $still) {{ $taskGone = $true; break }}
|
||||
Start-Sleep -Seconds 1
|
||||
}}
|
||||
if (-not $taskGone) {{
|
||||
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
|
||||
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
|
||||
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
|
||||
}}
|
||||
|
||||
if ($taskGone) {{
|
||||
# Task gone + tree renamed ⇒ safe to delete. Never touches the original path.
|
||||
foreach ($entry in $renamed) {{
|
||||
Remove-Item -Path $entry.Renamed -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}}
|
||||
}} else {{
|
||||
# Task survived ⇒ rename back and leave everything for the next-boot retry
|
||||
# (the marker survives, so bootstrap re-runs cleanup).
|
||||
foreach ($entry in $renamed) {{
|
||||
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
}}
|
||||
}}
|
||||
}} else {{
|
||||
# Probe failed ⇒ something still holds the tree. Rename back whatever succeeded and
|
||||
# abort WITHOUT deleting anything — task + marker stay, next boot retries.
|
||||
foreach ($entry in $renamed) {{
|
||||
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
|
||||
}}
|
||||
}}
|
||||
{(rebootOnComplete ? @"
|
||||
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
|
||||
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
|
||||
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
|
||||
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
|
||||
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
|
||||
" : "")}
|
||||
}} finally {{
|
||||
if ($lockStream) {{ $lockStream.Dispose() }}
|
||||
Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue
|
||||
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
|
||||
}}
|
||||
";
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
+43
-4
@@ -3,6 +3,7 @@ using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Management;
|
||||
using System.Text.RegularExpressions;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
@@ -28,6 +29,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
|
||||
///
|
||||
/// Emits a single "local_admin_analysis" event at startup and at shutdown,
|
||||
/// enabling delta detection between pre- and post-enrollment state.
|
||||
///
|
||||
/// Tenant-supplied allowed accounts may contain glob wildcards (e.g. "adm-*");
|
||||
/// see <see cref="MatchesAllowedEntry"/>.
|
||||
/// </summary>
|
||||
public class LocalAdminAnalyzer : IAgentAnalyzer
|
||||
{
|
||||
@@ -246,6 +250,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Allowed-list matching
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Matches an account or profile-folder name against one allowed-list entry.
|
||||
/// Entries containing <c>*</c> or <c>?</c> are glob patterns (same semantics as the
|
||||
/// hardware whitelist: <c>*</c> = any run, <c>?</c> = one char, case-insensitive);
|
||||
/// all other entries match exactly. No ambiguity with literal names — <c>*</c> and
|
||||
/// <c>?</c> are invalid characters in Windows account names.
|
||||
/// Internal for direct unit-testing via InternalsVisibleTo.
|
||||
/// </summary>
|
||||
internal static bool MatchesAllowedEntry(string name, string entry)
|
||||
{
|
||||
if (entry.IndexOf('*') < 0 && entry.IndexOf('?') < 0)
|
||||
return string.Equals(entry, name, StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
var regexPattern = "^" + Regex.Escape(entry)
|
||||
.Replace("\\*", ".*")
|
||||
.Replace("\\?", ".") + "$";
|
||||
|
||||
try
|
||||
{
|
||||
// Timeout guards against pathological patterns (ReDoS)
|
||||
return Regex.IsMatch(name, regexPattern, RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
|
||||
}
|
||||
catch (RegexMatchTimeoutException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool IsAllowed(string name, List<string> allowedAccounts)
|
||||
{
|
||||
return allowedAccounts.Any(a => MatchesAllowedEntry(name, a));
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Individual checks
|
||||
// -----------------------------------------------------------------------
|
||||
@@ -308,8 +349,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
|
||||
if (disabled)
|
||||
continue;
|
||||
|
||||
if (!allowedAccounts.Any(a =>
|
||||
string.Equals(a, name, StringComparison.OrdinalIgnoreCase)))
|
||||
if (!IsAllowed(name, allowedAccounts))
|
||||
{
|
||||
unexpected.Add(name);
|
||||
_logger.Debug($"{Name}: Unexpected local account: {name}");
|
||||
@@ -349,8 +389,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
|
||||
|
||||
allFound.Add(folderName);
|
||||
|
||||
if (!allowedAccounts.Any(a =>
|
||||
string.Equals(a, folderName, StringComparison.OrdinalIgnoreCase)))
|
||||
if (!IsAllowed(folderName, allowedAccounts))
|
||||
{
|
||||
unexpected.Add(folderName);
|
||||
_logger.Debug($"{Name}: Unexpected profile folder: {folderName}");
|
||||
|
||||
+55
-1
@@ -8,6 +8,7 @@ using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Security;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
@@ -493,7 +494,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
|
||||
}
|
||||
}
|
||||
|
||||
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus, $"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}", data);
|
||||
// Direct firmware evidence (UEFI db/KEK variables) — authoritative for the
|
||||
// CA-2023 rollout state. The Servicing key above only mirrors whether the
|
||||
// Windows Update rollout ran; it lags behind when the certificate reached
|
||||
// the firmware another way (OEM factory image, WinCS, manual deployment).
|
||||
var firmware = UefiSecureBootCertReader.Read();
|
||||
AppendFirmwareCertFields(data, firmware);
|
||||
|
||||
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus,
|
||||
$"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}, firmware: {DescribeFirmwareVerdict(firmware)}",
|
||||
data);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -501,6 +511,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps a <see cref="UefiSecureBootCertSnapshot"/> onto the <c>secureboot_status</c>
|
||||
/// payload. Certificate booleans are only emitted when the respective variable read
|
||||
/// succeeded — an absent boolean means "firmware state unknown", never "missing".
|
||||
/// </summary>
|
||||
internal static void AppendFirmwareCertFields(Dictionary<string, object> data, UefiSecureBootCertSnapshot firmware)
|
||||
{
|
||||
if (firmware == null) return;
|
||||
|
||||
data["uefiFirmwareReadStatus"] = firmware.DbStatus != UefiSecureBootCertReader.StatusOk
|
||||
? firmware.DbStatus
|
||||
: firmware.KekStatus;
|
||||
|
||||
if (firmware.DbStatus == UefiSecureBootCertReader.StatusOk)
|
||||
{
|
||||
data["uefiDbHasWindowsUefiCa2023"] = firmware.DbHasWindowsUefiCa2023;
|
||||
data["uefiDbHasMicrosoftUefiCa2023"] = firmware.DbHasMicrosoftUefiCa2023;
|
||||
data["uefiDbHasMicrosoftOptionRomCa2023"] = firmware.DbHasMicrosoftOptionRomCa2023;
|
||||
data["uefiDbHasWindowsProductionPca2011"] = firmware.DbHasWindowsProductionPca2011;
|
||||
data["uefiDbHasMicrosoftUefiCa2011"] = firmware.DbHasMicrosoftUefiCa2011;
|
||||
|
||||
// One-sided marker consumed by ANALYZE-SEC-001's not_exists precondition.
|
||||
// The engine's not_exists passes only while the field is absent/empty, so the
|
||||
// marker must be OMITTED (never emitted as false) unless the firmware db
|
||||
// positively confirms the 2023 certificate — old agents and unconfirmed
|
||||
// devices then keep evaluating the rule.
|
||||
if (firmware.DbHasWindowsUefiCa2023)
|
||||
data["uefiCa2023FirmwareConfirmed"] = true;
|
||||
}
|
||||
|
||||
if (firmware.KekStatus == UefiSecureBootCertReader.StatusOk)
|
||||
{
|
||||
data["uefiKekHasMicrosoftKek2kCa2023"] = firmware.KekHasMicrosoftKek2kCa2023;
|
||||
data["uefiKekHasMicrosoftKekCa2011"] = firmware.KekHasMicrosoftKekCa2011;
|
||||
}
|
||||
}
|
||||
|
||||
internal static string DescribeFirmwareVerdict(UefiSecureBootCertSnapshot firmware)
|
||||
{
|
||||
if (firmware == null) return "unavailable";
|
||||
if (firmware.DbStatus != UefiSecureBootCertReader.StatusOk) return firmware.DbStatus ?? "unavailable";
|
||||
return firmware.DbHasWindowsUefiCa2023 ? "CA2023 confirmed" : "CA2023 missing";
|
||||
}
|
||||
|
||||
private void CollectBitLockerStatus()
|
||||
{
|
||||
try
|
||||
|
||||
@@ -169,9 +169,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
|
||||
/// <summary>
|
||||
/// Sends a critical error report to the emergency channel endpoint.
|
||||
/// Fire-and-forget: swallows all exceptions so a failure here never cascades.
|
||||
/// Uses a 5-second per-request timeout so it cannot block the upload loop.
|
||||
/// Default 5-second per-request timeout so it cannot block the upload loop; the
|
||||
/// emergency-break path passes a longer <paramref name="timeout"/> because its first
|
||||
/// request of a freshly booted process pays cold TLS + DNS + a possible backend
|
||||
/// cold start (observed 503s take well over 5 s).
|
||||
/// Returns <c>true</c> when the HTTP round-trip completed (the endpoint always
|
||||
/// answers 200), <c>false</c> on timeout/transport failure — the caller-side
|
||||
/// anti-flood layer uses this to decide whether the report counts as delivered.
|
||||
/// Virtual for tests.
|
||||
/// </summary>
|
||||
public async Task ReportAgentErrorAsync(AgentErrorReport report)
|
||||
public virtual async Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -190,16 +197,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
|
||||
httpRequest.Headers.Add("X-Tenant-Id", report.TenantId);
|
||||
AddSecurityHeaders(httpRequest);
|
||||
|
||||
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
|
||||
using (var cts = new CancellationTokenSource(timeout ?? TimeSpan.FromSeconds(5)))
|
||||
using (var response = await _httpClient.SendAsync(httpRequest, cts.Token).ConfigureAwait(false))
|
||||
{
|
||||
// Response status is deliberately ignored — endpoint always returns 200.
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger?.Debug($"ReportAgentErrorAsync: emergency channel failed: {ex.Message}");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -57,19 +57,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
|
||||
/// (intended to be called as fire-and-forget via <c>_ = reporter.TrySendAsync(...)</c>).
|
||||
///
|
||||
/// All anti-flood checks are applied synchronously before the async HTTP call,
|
||||
/// so repeated calls are cheap when suppressed.
|
||||
/// so repeated calls are cheap when suppressed. The anti-flood reservation is
|
||||
/// taken up front (so concurrent same-key calls cannot double-send) but ROLLED
|
||||
/// BACK when every attempt fails: the budget and the once-per-session dedup are
|
||||
/// meant to count DELIVERED reports — the previous mark-before-send made the very
|
||||
/// first transport failure permanently suppress that error key for the session
|
||||
/// (found via the 48h emergency-break report, which is a single shot at boot).
|
||||
///
|
||||
/// <paramref name="attempts"/>/<paramref name="perAttemptTimeout"/>/<paramref name="retryDelay"/>
|
||||
/// exist for callers whose process is about to exit and cannot come back later
|
||||
/// (emergency break): retries stay INSIDE the one reservation, so they never
|
||||
/// multiply against the per-session budget.
|
||||
/// </summary>
|
||||
public virtual async Task TrySendAsync(
|
||||
AgentErrorType errorType,
|
||||
string message,
|
||||
int? httpStatusCode = null,
|
||||
long? sequenceNumber = null)
|
||||
long? sequenceNumber = null,
|
||||
int attempts = 1,
|
||||
TimeSpan? perAttemptTimeout = null,
|
||||
TimeSpan? retryDelay = null)
|
||||
{
|
||||
// Deduplicate by error type + status code: same failure category sent only once per session.
|
||||
// All anti-flood checks are protected by a lock to prevent race conditions when
|
||||
// TrySendAsync is called concurrently (fire-and-forget from multiple threads).
|
||||
var key = $"{errorType}:{httpStatusCode}";
|
||||
int currentReportCount;
|
||||
DateTime? previousLastReportTime;
|
||||
|
||||
lock (_antiFloodLock)
|
||||
{
|
||||
@@ -92,6 +106,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
|
||||
return;
|
||||
}
|
||||
|
||||
previousLastReportTime = _lastReportTime;
|
||||
_reportedKeys.Add(key);
|
||||
_reportCount++;
|
||||
_lastReportTime = DateTime.UtcNow;
|
||||
@@ -113,15 +128,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
|
||||
Timestamp = DateTime.UtcNow
|
||||
};
|
||||
|
||||
var delivered = false;
|
||||
var totalAttempts = attempts < 1 ? 1 : attempts;
|
||||
try
|
||||
{
|
||||
await _apiClient.ReportAgentErrorAsync(report);
|
||||
for (var attempt = 1; attempt <= totalAttempts && !delivered; attempt++)
|
||||
{
|
||||
if (attempt > 1 && retryDelay.HasValue)
|
||||
{
|
||||
await Task.Delay(retryDelay.Value).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
delivered = await _apiClient.ReportAgentErrorAsync(report, perAttemptTimeout).ConfigureAwait(false);
|
||||
if (!delivered && totalAttempts > 1)
|
||||
{
|
||||
_logger?.Debug($"[EmergencyChannel] Attempt {attempt}/{totalAttempts} failed: {key}");
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Swallow — emergency channel must never cascade failures into the caller
|
||||
_logger?.Debug($"[EmergencyChannel] Failed to send report: {ex.Message}");
|
||||
}
|
||||
|
||||
if (!delivered)
|
||||
{
|
||||
lock (_antiFloodLock)
|
||||
{
|
||||
_reportedKeys.Remove(key);
|
||||
_reportCount--;
|
||||
_lastReportTime = previousLastReportTime;
|
||||
}
|
||||
_logger?.Debug($"[EmergencyChannel] Report not delivered — reservation rolled back: {key}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,6 +36,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
/// <item><b>StallProbeHost</b> — <see cref="StallProbeCollector"/> +
|
||||
/// <see cref="StallProbeCollectorAdapter"/>. Owns its 60-s idle-check timer (the
|
||||
/// collector itself has no timer — it's a pure probe invoked from outside).</item>
|
||||
/// <item><b>EspPolicyProviderStallHost</b> — always-on 60-s wall-clock tick for the
|
||||
/// <c>esp_policy_provider_stalled</c> tripwire (registered EnrollmentStatusTracking CSP
|
||||
/// provider continuously incomplete ≥ 15 min — the no-timeout ESP wait).</item>
|
||||
/// </list>
|
||||
/// Optional peripheral hosts (driven by <see cref="CollectorConfiguration"/> toggles):
|
||||
/// <list type="bullet">
|
||||
@@ -303,6 +306,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
harmlessModernDeploymentEventIds: collectors.ModernDeploymentHarmlessEventIds));
|
||||
}
|
||||
|
||||
// ESP policy-provider stall tripwire — always-on kernel host (no config gate, like
|
||||
// disk_space_low): a registered EnrollmentStatusTracking CSP provider that never
|
||||
// completes (e.g. co-management "ConfigMgr" without "Sidecar" on a pre-provisioned
|
||||
// device) parks the user ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock
|
||||
// dwell, deliberately outside the StallProbe gate: the idle clock there resets on any
|
||||
// session activity, and the stall must accrue regardless of it.
|
||||
hosts.Add(new EspPolicyProviderStallHost(
|
||||
sessionId: sessionId,
|
||||
tenantId: tenantId,
|
||||
logger: logger,
|
||||
ingress: ingress,
|
||||
clock: clock,
|
||||
startupGate: _startupEventGate));
|
||||
|
||||
// Deterministic update corroboration — compares the persisted OS build (CurrentBuild.UBR)
|
||||
// across agent restarts and emits os_build_changed when it differs (session 7443317c:
|
||||
// build jump proved an OOBE quality update the WU channel never showed). Must be added
|
||||
|
||||
+94
@@ -0,0 +1,94 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Threading;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Persistence;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
{
|
||||
/// <summary>
|
||||
/// Drives the <see cref="EspPolicyProviderStallDetector"/> from a 60-s wall-clock tick.
|
||||
/// Deliberately NOT idle-coupled (a policy-provider stall coexists with a chatty session —
|
||||
/// <see cref="StallProbeHost"/>'s idle clock would keep resetting) and NOT idle-stopped
|
||||
/// (<see cref="PeriodicCollectorLifecycleHost"/> would sleep through exactly the dormant
|
||||
/// window the dwell must observe). Always-on kernel host, no config gate — precedent:
|
||||
/// the <c>disk_space_low</c> tripwire.
|
||||
/// <para>
|
||||
/// First tick fires after 60 s, not at Start: the EnrollmentStatusTracking key legitimately
|
||||
/// doesn't exist at agent start, and nothing can be 15 min stale at t=0.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class EspPolicyProviderStallHost : ICollectorHost
|
||||
{
|
||||
public string Name => "EspPolicyProviderStallDetector";
|
||||
|
||||
private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(60);
|
||||
|
||||
private readonly EspPolicyProviderStallDetector _detector;
|
||||
private readonly AgentLogger _logger;
|
||||
private Timer? _tickTimer;
|
||||
private int _disposed;
|
||||
|
||||
public EspPolicyProviderStallHost(
|
||||
string sessionId,
|
||||
string tenantId,
|
||||
AgentLogger logger,
|
||||
ISignalIngressSink ingress,
|
||||
IClock clock,
|
||||
StartupEventGate? startupGate)
|
||||
{
|
||||
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
|
||||
if (clock == null) throw new ArgumentNullException(nameof(clock));
|
||||
_logger = logger;
|
||||
var post = new InformationalEventPost(ingress, clock);
|
||||
_detector = new EspPolicyProviderStallDetector(
|
||||
sessionId: sessionId,
|
||||
tenantId: tenantId,
|
||||
post: post,
|
||||
logger: logger,
|
||||
clock: clock,
|
||||
startupGate: startupGate);
|
||||
}
|
||||
|
||||
public void Start()
|
||||
{
|
||||
_tickTimer = new Timer(
|
||||
_ => SafeTick(),
|
||||
state: null,
|
||||
dueTime: PollInterval,
|
||||
period: PollInterval);
|
||||
_logger.Info($"EspPolicyProviderStallHost: started (tick every {PollInterval.TotalSeconds}s).");
|
||||
}
|
||||
|
||||
public void Stop()
|
||||
{
|
||||
try
|
||||
{
|
||||
_tickTimer?.Dispose();
|
||||
_tickTimer = null;
|
||||
}
|
||||
catch (Exception ex) { _logger.Warning($"EspPolicyProviderStallHost: timer dispose failed: {ex.Message}"); }
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
|
||||
Stop();
|
||||
}
|
||||
|
||||
private void SafeTick()
|
||||
{
|
||||
try
|
||||
{
|
||||
_detector.Tick();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Error("EspPolicyProviderStallHost: tick failed.", ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Configuration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Runtime;
|
||||
@@ -93,6 +94,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
|
||||
// correction between agent start and termination no longer skews bootToStartSeconds.
|
||||
private readonly ulong _bootAnchorTickMs;
|
||||
private readonly DateTime _bootAnchorUtc;
|
||||
// WinRT OOBE state at construction (handler is built at agent start) — corroborates
|
||||
// the late-start heuristic: "completed" here means OOBE was already over on arrival.
|
||||
private readonly string _oobeStateAtAgentStart;
|
||||
private int _handled;
|
||||
|
||||
public EnrollmentTerminationHandler(
|
||||
@@ -129,6 +133,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
|
||||
|
||||
_bootAnchorTickMs = ObservationCoverage.CurrentTickMs();
|
||||
_bootAnchorUtc = DateTime.UtcNow;
|
||||
_oobeStateAtAgentStart = OobeStateReader.Read();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -658,6 +663,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
|
||||
{ "agentUptimeSeconds", Math.Round(uptimeSeconds, 1) },
|
||||
{ "deviceBootUtc", deviceBootUtc.ToString("o") },
|
||||
{ "agentStartUtc", _agentStartTimeUtc.ToString("o") },
|
||||
{ "oobeStateAtAgentStart", _oobeStateAtAgentStart },
|
||||
{ "outcome", args.Outcome.ToString() },
|
||||
{ "note", "The agent's bootstrap (an Intune platform script) ran only near the end of the enrollment, so the agent observed the already-decided end-state rather than the failure window. Treat the diagnosis as a post-mortem; check for a platform/remediation script that hung ahead of the bootstrap (see script_timeout_suspected)." },
|
||||
},
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using System;
|
||||
using System.Net.NetworkInformation;
|
||||
using System.Threading;
|
||||
using AutopilotMonitor.Agent.V2.Core.Configuration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
@@ -15,28 +17,78 @@ namespace AutopilotMonitor.Agent.V2.Runtime
|
||||
/// classifier terminalize the session instead of waiting out the grace. Fully swallowed — a send
|
||||
/// failure (no network, no cert) must never delay the cleanup/exit the break exists to perform.
|
||||
/// See tasks/enrollment-status-reclassification.md.
|
||||
/// <para>
|
||||
/// Delivery hardening (2026-07-23): this is the FIRST request of a freshly booted process and
|
||||
/// the process never comes back — a miss is gone forever. The old single shot with a 5 s cap
|
||||
/// died on boot-time realities (NIC still associating, cold TLS/DNS, backend cold starts
|
||||
/// observed as >5 s 503s). Now: wait briefly for the NIC, then up to
|
||||
/// <see cref="SendAttempts"/> attempts with a per-attempt timeout that survives a Functions
|
||||
/// cold start. The session is 48 h dead — spending up to ~75 s here is irrelevant, losing the
|
||||
/// only signal that the agent gave up is not.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static class EmergencyBreakReporter
|
||||
{
|
||||
private static readonly TimeSpan NetworkWaitMax = TimeSpan.FromSeconds(15);
|
||||
private const int SendAttempts = 3;
|
||||
private static readonly TimeSpan PerAttemptTimeout = TimeSpan.FromSeconds(15);
|
||||
private static readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(5);
|
||||
// Upper bound for the blocking wait: NIC wait + 3×15s attempts + 2×5s delays + slack.
|
||||
private static readonly TimeSpan OverallBudget = TimeSpan.FromSeconds(75);
|
||||
|
||||
public static void TrySend(AgentConfiguration agentConfig, string agentVersion, AgentLogger logger)
|
||||
{
|
||||
try
|
||||
{
|
||||
WaitForNetwork(logger);
|
||||
|
||||
var auth = BackendClientFactory.BuildAuthClients(agentConfig, agentVersion, logger);
|
||||
var message =
|
||||
$"Agent absolute session-age emergency break fired (cap {agentConfig.AbsoluteMaxSessionHours}h) — cleaning up and exiting.";
|
||||
|
||||
// The process is about to exit, so block briefly on the otherwise fire-and-forget send
|
||||
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP errors;
|
||||
// the auth bundle's HttpClients are intentionally left for process teardown.
|
||||
// The process is about to exit, so block on the otherwise fire-and-forget send
|
||||
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP
|
||||
// errors and keeps the retries inside one anti-flood reservation; the auth
|
||||
// bundle's HttpClients are intentionally left for process teardown.
|
||||
auth.EmergencyReporter
|
||||
.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, message)
|
||||
.Wait(TimeSpan.FromSeconds(5));
|
||||
.TrySendAsync(
|
||||
AgentErrorType.SessionAgeEmergencyBreak,
|
||||
message,
|
||||
attempts: SendAttempts,
|
||||
perAttemptTimeout: PerAttemptTimeout,
|
||||
retryDelay: RetryDelay)
|
||||
.Wait(OverallBudget);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger?.Debug($"Emergency-break report send failed (best-effort): {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Boot-time NIC grace: the break fires during bootstrap, often seconds after boot while
|
||||
/// Wi-Fi is still associating. Polls the cheap link-level signal — it cannot prove backend
|
||||
/// reachability (the send attempts handle that), it only avoids burning the first attempts
|
||||
/// into a known-dead link. Best-effort: probe errors abort the wait, never the send.
|
||||
/// </summary>
|
||||
private static void WaitForNetwork(AgentLogger logger)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (NetworkInterface.GetIsNetworkAvailable()) return;
|
||||
|
||||
logger?.Info($"Emergency-break report: no network link yet — waiting up to {NetworkWaitMax.TotalSeconds:F0}s.");
|
||||
var deadline = DateTime.UtcNow + NetworkWaitMax;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
Thread.Sleep(TimeSpan.FromSeconds(1));
|
||||
if (NetworkInterface.GetIsNetworkAvailable()) return;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger?.Debug($"Emergency-break report: network probe failed ({ex.Message}) — sending anyway.");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ using System;
|
||||
using System.Collections.Generic;
|
||||
using AutopilotMonitor.Agent.V2.Core.Configuration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Security;
|
||||
@@ -68,6 +69,10 @@ namespace AutopilotMonitor.Agent.V2.Runtime
|
||||
{ "configVersion", configVersion },
|
||||
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
|
||||
{ "remoteConfigOutcome", outcome.ToString() },
|
||||
// WinRT OOBE state sampled at this start. Together with previousExitType
|
||||
// this makes late starts provable: "completed" on a first_run means OOBE
|
||||
// was already over when the agent arrived (e.g. IME queue blocked).
|
||||
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
|
||||
};
|
||||
|
||||
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
|
||||
|
||||
@@ -1,7 +1,15 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Net.Http;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Functions.Ingest;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Functions.Services.Notifications;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
@@ -80,4 +88,116 @@ public class ReportAgentErrorFunctionTests
|
||||
var evt = ReportAgentErrorFunction.BuildAgentEmergencyBreakEvent(Report(Now, message: ""), TenantId, null!, Now);
|
||||
Assert.Contains("emergency break", evt.Message, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// MaterializeEmergencyBreakArtifactsAsync — the artifact set per break report
|
||||
// (2026-07-23 hardening): timeline event + cross-session EventType index row +
|
||||
// AgentEmergencyBreak ops event, once per session. The index upsert is load-bearing:
|
||||
// StoreEventsBatchAsync alone leaves the event invisible to every search-by-eventType
|
||||
// surface (found in the 2026-07-22 incident analysis).
|
||||
// =========================================================================
|
||||
|
||||
private sealed class Harness
|
||||
{
|
||||
public readonly Mock<ISessionRepository> SessionRepo = new();
|
||||
public readonly List<OpsEventEntry> OpsEvents = new();
|
||||
public readonly OpsEventService OpsService;
|
||||
|
||||
public Harness(List<EnrollmentEvent>? existingEvents = null)
|
||||
{
|
||||
SessionRepo.Setup(r => r.GetSessionEventsAsync(TenantId, SessionId, It.IsAny<int>()))
|
||||
.ReturnsAsync(existingEvents ?? new List<EnrollmentEvent>());
|
||||
SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
|
||||
.ReturnsAsync((List<EnrollmentEvent> e) => e);
|
||||
SessionRepo.Setup(r => r.UpsertEventTypeIndexBatchAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()))
|
||||
.Returns(Task.CompletedTask);
|
||||
|
||||
var opsRepo = new Mock<IOpsEventRepository>();
|
||||
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
|
||||
.Callback<OpsEventEntry>(e => { lock (OpsEvents) OpsEvents.Add(e); })
|
||||
.Returns(Task.CompletedTask);
|
||||
|
||||
var adminConfig = new Mock<AdminConfigurationService>(
|
||||
Mock.Of<IConfigRepository>(), NullLogger<AdminConfigurationService>.Instance,
|
||||
new MemoryCache(new MemoryCacheOptions()));
|
||||
var alertDispatch = new OpsAlertDispatchService(
|
||||
adminConfig.Object,
|
||||
new TelegramNotificationService(new HttpClient(), Mock.Of<IConfigRepository>(),
|
||||
NullLogger<TelegramNotificationService>.Instance),
|
||||
new WebhookNotificationService(new HttpClient(),
|
||||
NullLogger<WebhookNotificationService>.Instance),
|
||||
NullLogger<OpsAlertDispatchService>.Instance);
|
||||
OpsService = new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance, alertDispatch);
|
||||
}
|
||||
|
||||
public Task RunAsync(AgentErrorReport report)
|
||||
=> ReportAgentErrorFunction.MaterializeEmergencyBreakArtifactsAsync(
|
||||
report, TenantId, SessionRepo.Object, OpsService, NullLogger<ReportAgentErrorFunction>.Instance);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task First_break_report_materializes_event_index_row_and_ops_event()
|
||||
{
|
||||
var h = new Harness();
|
||||
|
||||
await h.RunAsync(Report(Now));
|
||||
|
||||
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
|
||||
It.Is<List<EnrollmentEvent>>(e => e.Count == 1
|
||||
&& e[0].EventType == AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak)), Times.Once);
|
||||
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
|
||||
TenantId, SessionId, It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Once);
|
||||
|
||||
var ops = Assert.Single(h.OpsEvents);
|
||||
Assert.Equal("AgentEmergencyBreak", ops.EventType);
|
||||
Assert.Equal(TenantId, ops.TenantId);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Already_materialized_session_produces_no_second_artifact_set()
|
||||
{
|
||||
var h = new Harness(new List<EnrollmentEvent>
|
||||
{
|
||||
new() { EventType = AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak, Sequence = 45 },
|
||||
});
|
||||
|
||||
await h.RunAsync(Report(Now));
|
||||
|
||||
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Never);
|
||||
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Never);
|
||||
Assert.Empty(h.OpsEvents);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Non_break_error_types_and_missing_session_id_are_ignored()
|
||||
{
|
||||
var h = new Harness();
|
||||
|
||||
var other = Report(Now);
|
||||
other.ErrorType = AgentErrorType.ConfigFetchFailed;
|
||||
await h.RunAsync(other);
|
||||
|
||||
var noSession = Report(Now);
|
||||
noSession.SessionId = null!;
|
||||
await h.RunAsync(noSession);
|
||||
|
||||
h.SessionRepo.Verify(r => r.GetSessionEventsAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()), Times.Never);
|
||||
Assert.Empty(h.OpsEvents);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Storage_failure_is_swallowed_and_never_throws()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
|
||||
.ThrowsAsync(new InvalidOperationException("storage down"));
|
||||
|
||||
// Must not throw — the always-200 emergency channel must never turn into a retry loop.
|
||||
await h.RunAsync(Report(Now));
|
||||
|
||||
Assert.Empty(h.OpsEvents);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -235,6 +235,80 @@ public class RuleEnginePreconditionTests
|
||||
Assert.Single(outcome.Results);
|
||||
}
|
||||
|
||||
// ===== ANALYZE-SEC-001 v3: firmware-confirmation marker (uefiCa2023FirmwareConfirmed) =====
|
||||
|
||||
/// <summary>
|
||||
/// v3 core fix: when the agent's direct UEFI db read confirms the 2023 certificate, the
|
||||
/// one-sided marker is present and the not_exists precondition suppresses the rule — even
|
||||
/// though the servicing status ("unknown") would still satisfy both required conditions.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Sec001V3_FirmwareConfirmed_MarkerSuppressesRule()
|
||||
{
|
||||
var rule = MakeSecureBootRuleV3();
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
HardwareSpecEvent(isVirtualMachine: "false"),
|
||||
SecureBootStatusEvent(uefiCA2023Status: "unknown", secureBootEnabled: "true",
|
||||
extraData: new Dictionary<string, object>
|
||||
{
|
||||
["uefiDbHasWindowsUefiCa2023"] = true,
|
||||
["uefiCa2023FirmwareConfirmed"] = true
|
||||
})
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
Assert.Single(outcome.EvaluatedRules);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Backward compatibility: an old-agent payload (no firmware fields at all) must behave
|
||||
/// exactly like v2 — marker absent → precondition passes → rule fires on the servicing
|
||||
/// status, at base confidence (no firmware factors).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Sec001V3_OldAgentPayload_FiresLikeV2()
|
||||
{
|
||||
var rule = MakeSecureBootRuleV3();
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
HardwareSpecEvent(isVirtualMachine: "false"),
|
||||
SecureBootStatusEvent(uefiCA2023Status: "unknown", secureBootEnabled: "true")
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
|
||||
Assert.Single(outcome.Results);
|
||||
Assert.Equal(70, outcome.Results[0].ConfidenceScore);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// New agent, firmware-verified missing: the marker is omitted (never emitted as false),
|
||||
/// so the rule fires and the firmware evidence raises confidence (+20 db, +10 KEK).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Sec001V3_FirmwareVerifiedMissing_RaisesConfidence()
|
||||
{
|
||||
var rule = MakeSecureBootRuleV3();
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
HardwareSpecEvent(isVirtualMachine: "false"),
|
||||
SecureBootStatusEvent(uefiCA2023Status: "unknown", secureBootEnabled: "true",
|
||||
extraData: new Dictionary<string, object>
|
||||
{
|
||||
["uefiDbHasWindowsUefiCa2023"] = false,
|
||||
["uefiKekHasMicrosoftKek2kCa2023"] = false
|
||||
})
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
|
||||
Assert.Single(outcome.Results);
|
||||
Assert.Equal(100, outcome.Results[0].ConfidenceScore);
|
||||
}
|
||||
|
||||
// ===== Helpers =====
|
||||
|
||||
private static AnalyzeRule MakeSecureBootRule(List<RulePrecondition>? preconditions)
|
||||
@@ -258,6 +332,29 @@ public class RuleEnginePreconditionTests
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Mirrors the shipped ANALYZE-SEC-001 v3.0.0 (rules/analyze/ANALYZE-SEC-001.json):
|
||||
/// VM-skip + firmware-confirmed-marker preconditions, the two required servicing
|
||||
/// conditions, and the optional firmware conditions with their confidence weights.
|
||||
/// Keep in sync when the rule JSON changes.
|
||||
/// </summary>
|
||||
private static AnalyzeRule MakeSecureBootRuleV3()
|
||||
{
|
||||
var rule = MakeSecureBootRule(preconditions: new List<RulePrecondition>
|
||||
{
|
||||
new() { Source = "event_data", EventType = "hardware_spec", DataField = "isVirtualMachine", Operator = "equals", Value = "false" },
|
||||
new() { Source = "event_data", EventType = "secureboot_status", DataField = "uefiCa2023FirmwareConfirmed", Operator = "not_exists", Value = "" }
|
||||
});
|
||||
rule.Conditions.Add(new RuleCondition { Signal = "db_ca2023_firmware_missing", Source = "event_data", EventType = "secureboot_status", DataField = "uefiDbHasWindowsUefiCa2023", Operator = "equals", Value = "false", Required = false });
|
||||
rule.Conditions.Add(new RuleCondition { Signal = "kek_2023_missing", Source = "event_data", EventType = "secureboot_status", DataField = "uefiKekHasMicrosoftKek2kCa2023", Operator = "equals", Value = "false", Required = false });
|
||||
rule.ConfidenceFactors = new List<ConfidenceFactor>
|
||||
{
|
||||
new() { Signal = "db_ca2023_firmware_missing", Condition = "exists", Weight = 20 },
|
||||
new() { Signal = "kek_2023_missing", Condition = "exists", Weight = 10 }
|
||||
};
|
||||
return rule;
|
||||
}
|
||||
|
||||
private static EnrollmentEvent HardwareSpecEvent(string isVirtualMachine) => new()
|
||||
{
|
||||
EventId = Guid.NewGuid().ToString(),
|
||||
@@ -280,20 +377,29 @@ public class RuleEnginePreconditionTests
|
||||
Data = new Dictionary<string, object> { ["edition"] = edition }
|
||||
};
|
||||
|
||||
private static EnrollmentEvent SecureBootStatusEvent(string uefiCA2023Status, string secureBootEnabled) => new()
|
||||
private static EnrollmentEvent SecureBootStatusEvent(string uefiCA2023Status, string secureBootEnabled,
|
||||
Dictionary<string, object>? extraData = null)
|
||||
{
|
||||
EventId = Guid.NewGuid().ToString(),
|
||||
TenantId = TenantId,
|
||||
SessionId = SessionId,
|
||||
EventType = "secureboot_status",
|
||||
Timestamp = DateTime.UtcNow,
|
||||
Sequence = 3,
|
||||
Data = new Dictionary<string, object>
|
||||
var data = new Dictionary<string, object>
|
||||
{
|
||||
["uefiCA2023Status"] = uefiCA2023Status,
|
||||
["uefiSecureBootEnabled"] = secureBootEnabled
|
||||
}
|
||||
};
|
||||
};
|
||||
if (extraData != null)
|
||||
foreach (var kvp in extraData)
|
||||
data[kvp.Key] = kvp.Value;
|
||||
|
||||
return new EnrollmentEvent
|
||||
{
|
||||
EventId = Guid.NewGuid().ToString(),
|
||||
TenantId = TenantId,
|
||||
SessionId = SessionId,
|
||||
EventType = "secureboot_status",
|
||||
Timestamp = DateTime.UtcNow,
|
||||
Sequence = 3,
|
||||
Data = data
|
||||
};
|
||||
}
|
||||
|
||||
private static async Task<AnalysisOutcome> RunAsync(AnalyzeRule rule, List<EnrollmentEvent> events)
|
||||
{
|
||||
|
||||
+65
-26
@@ -37,6 +37,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
private readonly TelemetryClient _telemetryClient;
|
||||
private readonly BootstrapSessionService _bootstrapSessionService;
|
||||
private readonly ISessionRepository _sessionRepo;
|
||||
private readonly OpsEventService _opsEventService;
|
||||
|
||||
public ReportAgentErrorFunction(
|
||||
ILogger<ReportAgentErrorFunction> logger,
|
||||
@@ -48,7 +49,8 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
DeviceAssociationValidator deviceAssociationValidator,
|
||||
TelemetryClient telemetryClient,
|
||||
BootstrapSessionService bootstrapSessionService,
|
||||
ISessionRepository sessionRepo)
|
||||
ISessionRepository sessionRepo,
|
||||
OpsEventService opsEventService)
|
||||
{
|
||||
_logger = logger;
|
||||
_configService = configService;
|
||||
@@ -60,6 +62,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
_telemetryClient = telemetryClient;
|
||||
_bootstrapSessionService = bootstrapSessionService;
|
||||
_sessionRepo = sessionRepo;
|
||||
_opsEventService = opsEventService;
|
||||
}
|
||||
|
||||
[Function("ReportAgentError")]
|
||||
@@ -169,35 +172,71 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
["AgentTimestamp"] = report.Timestamp.ToString("O"),
|
||||
});
|
||||
|
||||
// Materialize the agent's silent 48h emergency break as a timeline event so it shows in the
|
||||
// session and the timeout classifier can see it (tasks/enrollment-status-reclassification.md).
|
||||
// Best-effort — a failure here must never turn the always-200 emergency channel into a retry loop.
|
||||
if (report.ErrorType == AgentErrorType.SessionAgeEmergencyBreak
|
||||
&& !string.IsNullOrEmpty(report.SessionId))
|
||||
{
|
||||
try
|
||||
{
|
||||
var existing = await _sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
|
||||
// Idempotency: the agent's emergency channel can send up to a few reports per session;
|
||||
// only ever materialize one timeline event.
|
||||
var alreadyMaterialized = existing.Any(e =>
|
||||
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
|
||||
if (!alreadyMaterialized)
|
||||
{
|
||||
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
|
||||
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
|
||||
}
|
||||
}
|
||||
await MaterializeEmergencyBreakArtifactsAsync(report, tenantId, _sessionRepo, _opsEventService, _logger);
|
||||
|
||||
return req.CreateResponse(HttpStatusCode.OK);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Materializes the agent's silent 48h emergency break as (1) a timeline event so it shows
|
||||
/// in the session and the timeout classifier can see it, (2) the cross-session EventType
|
||||
/// index row, and (3) an <c>AgentEmergencyBreak</c> ops event for operator visibility
|
||||
/// (tasks/enrollment-status-reclassification.md). Static seam with explicit dependencies so
|
||||
/// tests can pin the artifact set without booting the Functions HTTP stack.
|
||||
/// Best-effort — a failure here must never turn the always-200 emergency channel into a
|
||||
/// retry loop.
|
||||
/// </summary>
|
||||
internal static async Task MaterializeEmergencyBreakArtifactsAsync(
|
||||
AgentErrorReport report,
|
||||
string tenantId,
|
||||
ISessionRepository sessionRepo,
|
||||
OpsEventService opsEventService,
|
||||
ILogger logger)
|
||||
{
|
||||
if (report.ErrorType != AgentErrorType.SessionAgeEmergencyBreak
|
||||
|| string.IsNullOrEmpty(report.SessionId))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var existing = await sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
|
||||
// Idempotency: the agent's emergency channel can send up to a few reports per session;
|
||||
// only ever materialize one timeline event (and one ops event).
|
||||
var alreadyMaterialized = existing.Any(e =>
|
||||
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
|
||||
if (alreadyMaterialized)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
|
||||
await sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
|
||||
|
||||
// StoreEventsBatchAsync writes the Events partition only — the cross-session
|
||||
// EventType index is normally written by EventIngestProcessor, which this
|
||||
// backend-materialized event never passes through. Without the upsert the event
|
||||
// exists on the session timeline but is invisible to every search-by-eventType
|
||||
// surface (portal cross-session search, MCP search_sessions_by_event /
|
||||
// query_raw_events) — found the hard way in the 2026-07-22 incident analysis.
|
||||
await sessionRepo.UpsertEventTypeIndexBatchAsync(
|
||||
tenantId, report.SessionId, new List<EnrollmentEvent> { evt });
|
||||
|
||||
// Operator visibility: an emergency break means an agent silently gave up at its
|
||||
// absolute age cap — exactly the "are we losing agents?" signal. Emitted only on
|
||||
// first materialization so repeat reports for the same session cannot flood the
|
||||
// ops feed. OpsEventService never throws.
|
||||
await opsEventService.RecordAgentEmergencyBreakAsync(
|
||||
tenantId, report.SessionId, report.AgentVersion, evt.Message);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogWarning(ex,
|
||||
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the backend-materialized <c>agent_emergency_break</c> timeline event from the agent's
|
||||
/// best-effort emergency report. Static + pure (analog to
|
||||
|
||||
@@ -439,6 +439,13 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var timeoutEvent = BuildSessionTimeoutEvent(session, timeoutHours, sessionEvents, now);
|
||||
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { timeoutEvent });
|
||||
|
||||
// Backend-materialized events bypass EventIngestProcessor and thus the
|
||||
// cross-session EventType index — without this upsert session_timeout is
|
||||
// invisible to every search-by-eventType surface (portal cross-session
|
||||
// search, MCP search_sessions_by_event / query_raw_events).
|
||||
await _sessionRepo.UpsertEventTypeIndexBatchAsync(
|
||||
session.TenantId, session.SessionId, new List<EnrollmentEvent> { timeoutEvent });
|
||||
|
||||
// StoreEventsBatchAsync only bumps the orphan side-index, not the
|
||||
// session's EventCount, and the terminal reconcile already ran inside
|
||||
// UpdateSessionStatusAsync *before* this synthetic event existed. Recount
|
||||
|
||||
@@ -493,6 +493,20 @@ namespace AutopilotMonitor.Functions.Services
|
||||
$"{sessionCount} session(s) timed out after {timeoutHours}h",
|
||||
tenantId, "System.Maintenance", new { sessionCount, timeoutHours });
|
||||
|
||||
/// <summary>
|
||||
/// An agent reported its absolute session-age emergency break (48h cap) over the
|
||||
/// emergency channel — it cleaned itself up and exited on a session that never reached
|
||||
/// a terminal state. This is the "are we silently losing agents?" signal; emitted by
|
||||
/// <see cref="Functions.Ingest.ReportAgentErrorFunction"/> once per session (guarded by
|
||||
/// the timeline-event idempotency check). Warning-tier so operators can wire a Telegram
|
||||
/// rule; if real-world volume turns out noisy, downgrade or remove — the timeline event
|
||||
/// on the session is the durable record.
|
||||
/// </summary>
|
||||
public Task RecordAgentEmergencyBreakAsync(string tenantId, string sessionId, string? agentVersion, string message)
|
||||
=> WriteAsync(OpsEventCategory.Agent, "AgentEmergencyBreak", OpsEventSeverity.Warning,
|
||||
$"Agent emergency break on session {sessionId} (agent {agentVersion ?? "?"}): {message}",
|
||||
tenantId, "System.EmergencyChannel", new { sessionId, agentVersion });
|
||||
|
||||
public Task RecordExcessiveSessionEventsAsync(string tenantId, string sessionId, int eventCount, int threshold)
|
||||
=> WriteAsync(OpsEventCategory.Agent, "ExcessiveSessionEvents", OpsEventSeverity.Warning,
|
||||
$"Session {sessionId} has {eventCount} events (threshold {threshold}) — likely agent loop bug",
|
||||
|
||||
@@ -34,6 +34,7 @@ export const EVENT_TYPES_CATALOG = {
|
||||
'desktop_detector_started',
|
||||
'desktop_detector_first_poll',
|
||||
'desktop_detector_no_candidate',
|
||||
'oobe_state_completed',
|
||||
],
|
||||
whiteglove_events: [
|
||||
'whiteglove_complete',
|
||||
@@ -166,6 +167,7 @@ export const EVENT_TYPES_CATALOG = {
|
||||
'telemetry_upload_blocked',
|
||||
'session_parked_without_deadline',
|
||||
'disk_space_low',
|
||||
'esp_policy_provider_stalled',
|
||||
'agent_late_start',
|
||||
],
|
||||
metrics_events: ['performance_snapshot', 'agent_metrics_snapshot', 'ingress_backpressure'],
|
||||
@@ -248,6 +250,12 @@ export const DEVICE_PROPERTIES_CATALOG = {
|
||||
},
|
||||
secureboot_status: {
|
||||
'secureboot_status.uefiSecureBootEnabled': { type: 'boolean', description: 'Secure Boot enabled' },
|
||||
'secureboot_status.uefiCA2023Status': { type: 'string', description: 'Windows servicing rollout status of the UEFI CA 2023 certificate (registry; "Updated" = rollout verified). Lags behind actual firmware state — prefer the firmware fields below' },
|
||||
'secureboot_status.uefiFirmwareReadStatus': { type: 'string', description: 'Result of the direct UEFI variable read: ok, not_uefi (legacy BIOS), variable_not_found, privilege_denied, or error_<code>' },
|
||||
'secureboot_status.uefiDbHasWindowsUefiCa2023': { type: 'boolean', description: 'Firmware db contains "Windows UEFI CA 2023" (authoritative 2023-cert deployment state; read directly from UEFI, only present when the read succeeded)' },
|
||||
'secureboot_status.uefiDbHasWindowsProductionPca2011': { type: 'boolean', description: 'Firmware db still contains the expired "Microsoft Windows Production PCA 2011"' },
|
||||
'secureboot_status.uefiKekHasMicrosoftKek2kCa2023': { type: 'boolean', description: 'Firmware KEK contains "Microsoft Corporation KEK 2K CA 2023" (required to keep receiving db/dbx updates after the 2011 KEK expiry)' },
|
||||
'secureboot_status.uefiCa2023FirmwareConfirmed': { type: 'boolean', description: 'One-sided marker: present (true) only when the firmware db positively confirms the 2023 certificate — suppresses ANALYZE-SEC-001 via not_exists precondition. Never emitted as false' },
|
||||
},
|
||||
bitlocker_status: {
|
||||
'bitlocker_status.systemDriveProtected': { type: 'boolean', description: 'System drive (C:) BitLocker protection' },
|
||||
|
||||
@@ -317,6 +317,12 @@ namespace AutopilotMonitor.Shared
|
||||
public const string DesktopDetectorStarted = "desktop_detector_started"; // 1x on Start() / ResetForRealUserSwitch()
|
||||
public const string DesktopDetectorFirstPoll = "desktop_detector_first_poll"; // 1x after first PollForDesktop() completes
|
||||
public const string DesktopDetectorNoCandidate = "desktop_detector_no_candidate"; // 1x after threshold polls without resolution (configurable)
|
||||
// 1x when WinRT SystemSetupInfo.OutOfBoxExperienceState flips InProgress->Completed,
|
||||
// sampled on the DAD 30s poll. Observational only (no decision-engine involvement) —
|
||||
// an owner-independent desktop corroboration for sessions where WTS+WMI owner
|
||||
// resolution starves and desktop_arrived never fires. Often absent in healthy
|
||||
// sessions: the DAD timer is disposed at desktop arrival, which is when the flip lands.
|
||||
public const string OobeStateCompleted = "oobe_state_completed";
|
||||
public const string CompletionCheck = "completion_check";
|
||||
// Liveness plan PR2 — emitted (state-change-only, deduped via the
|
||||
// CompletionWaitingFingerprint state fact) whenever the DecisionEngine blocks or
|
||||
@@ -362,6 +368,7 @@ namespace AutopilotMonitor.Shared
|
||||
public const string TelemetryUploadBlocked = "telemetry_upload_blocked"; // Warning (one-shot) when the drain is blocked by a retained unknown permanent 4xx (400 contract bug / 404 route mismatch / tenant-validation). Data is NOT discarded. LOCAL diagnostic (agent log + diag ZIP): this marker is queued behind the blocking batch and only reaches the backend after the block clears — no out-of-band bypass by design.
|
||||
public const string SessionParkedWithoutDeadline = "session_parked_without_deadline"; // One-shot Warning tripwire (liveness plan PR1): a non-terminal session entered the post-AccountSetup dead-end zone (ESP final exit or advisory failure) with NO resolution-capable deadline armed. After the 2026-06-12 dead-end fixes this must never fire in production — every occurrence is a bug report for an unknown parking variant.
|
||||
public const string DiskSpaceLow = "disk_space_low"; // One-shot Warning (state-change-only, 1 GB hysteresis — NOT periodic) when the system drive drops below 2 GB free. Review MON-B10: disk-full was previously only visible inside Debug performance_snapshot payloads (filtered out of timelines), so a disk-starved enrollment carried no actionable threshold signal.
|
||||
public const string EspPolicyProviderStalled = "esp_policy_provider_stalled"; // One-shot-per-provider Warning: a registered EnrollmentStatusTracking CSP policy provider stayed incomplete >= 15 min (Setup/Apps TrackingPoliciesCreated != 1, or DevicePreparation InstallationState not in {2,3}). The ESP waits on Setup/Apps providers WITHOUT any OS timeout — e.g. pre-provisioning + co-management leaves only "ConfigMgr" registered (no "Sidecar" = IME) and the user ESP sits at "Apps (Identifying)" for days. Observational only; payload carries the full provider list. See EspPolicyProviderProbe.
|
||||
public const string KeepAwakeEngaged = "keep_awake_engaged"; // Info (state-change, NOT periodic) emitted by the opt-in UserEspKeepAwake host when it holds the device awake (system + display) on entering the User-ESP/AccountSetup phase, so idle standby cannot stall app installs / account provisioning. Reboots are unaffected. Per-tenant toggle (AnalyzerConfiguration.KeepAwakeDuringUserEsp, default off).
|
||||
public const string KeepAwakeReleased = "keep_awake_released"; // Info (Warning when reason=safety_cap) emitted when the keep-awake hold is released — reason = account_setup_complete | host_stop | safety_cap. Brackets KeepAwakeEngaged; the OS also auto-clears the hold on process exit/reboot.
|
||||
|
||||
|
||||
@@ -95,7 +95,17 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
|
||||
"TenantTrialExpiring",
|
||||
"TenantTrialExpired",
|
||||
],
|
||||
Agent: ["BlobStorageMissing", "BlobStorageUnreachable", "NewImeVersionDetected", "ExcessiveSessionEvents"],
|
||||
Agent: [
|
||||
"BlobStorageMissing",
|
||||
"BlobStorageUnreachable",
|
||||
"NewImeVersionDetected",
|
||||
"ExcessiveSessionEvents",
|
||||
// 48h session-age emergency break reported over the agent's emergency channel — the
|
||||
// "are we silently losing agents?" signal. Emitted once per session by
|
||||
// ReportAgentErrorFunction (RecordAgentEmergencyBreakAsync). Dual-register per memory
|
||||
// feedback_ops_event_types_dual_register.
|
||||
"AgentEmergencyBreak",
|
||||
],
|
||||
};
|
||||
|
||||
const SEVERITIES = ["Info", "Warning", "Error", "Critical"];
|
||||
|
||||
@@ -32,6 +32,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
|
||||
description: "Pre-Provisioning Part 2 resumed after the user powered on the device." },
|
||||
{ value: "desktop_arrived", label: "desktop_arrived", category: "enrollment",
|
||||
description: "User desktop detected (explorer.exe under real user)." },
|
||||
{ value: "oobe_state_completed", label: "oobe_state_completed", category: "enrollment",
|
||||
description: "Windows OOBE state flipped InProgress→Completed (WinRT SystemSetupInfo, sampled on the desktop-arrival poll). One-shot and observational only; valuable as an owner-independent desktop corroboration in sessions where desktop_arrived never fires." },
|
||||
{ value: "completion_check", label: "completion_check", category: "enrollment",
|
||||
description: "Throttled state-machine snapshot during completion evaluation." },
|
||||
{ value: "completion_waiting", label: "completion_waiting", category: "enrollment",
|
||||
@@ -56,6 +58,8 @@ export const KNOWN_EVENT_TYPES: EventTypeEntry[] = [
|
||||
description: "Stall probe found an anomaly (ModernDeployment error, DeploymentErrorCode≠0, IME EnforcementState Failed)." },
|
||||
{ value: "stall_probe_check", label: "stall_probe_check", category: "stall",
|
||||
description: "Trace-level heartbeat from a stall probe — proves the logic ran (default: only the 15 min probe)." },
|
||||
{ value: "esp_policy_provider_stalled", label: "esp_policy_provider_stalled", category: "stall",
|
||||
description: "A registered ESP policy provider (EnrollmentStatusTracking CSP) stayed incomplete for 15+ minutes — e.g. co-management leaves only ConfigMgr registered without the Intune Sidecar provider, which parks the user ESP at 'Apps (Identifying)' with no OS timeout. One-shot Warning; payload carries the full provider table, per-provider state, and sidecarRegistered." },
|
||||
{ value: "agent_late_start", label: "agent_late_start", category: "stall",
|
||||
description: "Low observation coverage: the agent started long after device boot (≥10 min) AND lived only briefly (≤5 min) before a terminal outcome — it arrived after the enrollment had already decided, so its diagnosis is a post-mortem of the end-state, not a live observation. Carries bootToAgentStartSeconds, agentUptimeSeconds and outcome. Often paired with script_timeout_suspected (a script that hung ahead of the agent bootstrap)." },
|
||||
|
||||
|
||||
+19
-1
@@ -297,7 +297,25 @@ export default function DeviceDetailsCard({ events, latestAgentVersion }: { even
|
||||
{(bitLockerStatus || secureBootStatus || tpmStatus) && (
|
||||
<DetailSection title="Security">
|
||||
{secureBootStatus && (
|
||||
<DetailRow label="SecureBoot" value={secureBootStatus.uefiSecureBootEnabled ? "Enabled" : "Disabled"} />
|
||||
<>
|
||||
<DetailRow label="SecureBoot" value={secureBootStatus.uefiSecureBootEnabled ? "Enabled" : "Disabled"} />
|
||||
{/* Firmware verdict (direct UEFI db read, new agents) beats the servicing
|
||||
registry status, which only reflects the Windows Update rollout. */}
|
||||
{secureBootStatus.uefiDbHasWindowsUefiCa2023 !== undefined ? (
|
||||
<DetailRow
|
||||
label="UEFI CA 2023 (firmware)"
|
||||
value={secureBootStatus.uefiDbHasWindowsUefiCa2023 ? "Deployed" : "Missing"}
|
||||
/>
|
||||
) : secureBootStatus.uefiFirmwareReadStatus ? (
|
||||
<DetailRow
|
||||
label="UEFI CA 2023 (firmware)"
|
||||
value={`Unknown (${secureBootStatus.uefiFirmwareReadStatus})`}
|
||||
/>
|
||||
) : null}
|
||||
{secureBootStatus.uefiCA2023Status && (
|
||||
<DetailRow label="UEFI CA 2023 (servicing)" value={`${secureBootStatus.uefiCA2023Status}`} />
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
{tpmStatus && (
|
||||
<>
|
||||
|
||||
+3
-28
@@ -3,7 +3,7 @@
|
||||
import { useMemo } from "react";
|
||||
import { EnrollmentEvent, Session } from "@/types";
|
||||
import { V1_PHASE_NAMES, V2_PHASE_NAMES, V1_PHASE_ORDER, V2_PHASE_ORDER } from "../utils/phaseConstants";
|
||||
import { computeWhiteGloveSplitSequence, groupEventsByPhase } from "../utils/eventHelpers";
|
||||
import { computeWhiteGloveDurations, computeWhiteGloveSplitSequence, groupEventsByPhase } from "../utils/eventHelpers";
|
||||
|
||||
interface PhaseGrouping {
|
||||
eventsByPhase: Record<string, EnrollmentEvent[]>;
|
||||
@@ -176,33 +176,8 @@ export function useSessionDerivedData(
|
||||
if (!isWhiteGloveSession) {
|
||||
return { preProvDuration: null as string | null, userEnrollDuration: null as string | null, combinedDuration: null as string | null };
|
||||
}
|
||||
|
||||
const preProvEvts = whiteGloveSplitSequence < 0 ? events : events.filter(e => e.sequence <= whiteGloveSplitSequence);
|
||||
const userEnrollEvts = whiteGloveSplitSequence < 0 ? [] : events.filter(e => e.sequence > whiteGloveSplitSequence);
|
||||
|
||||
const calcMs = (evts: EnrollmentEvent[]): number => {
|
||||
if (evts.length === 0) return 0;
|
||||
const ts = evts.map(e => new Date(e.timestamp).getTime());
|
||||
return Math.max(...ts) - Math.min(...ts);
|
||||
};
|
||||
|
||||
const fmt = (ms: number): string | null => {
|
||||
const sec = Math.round(ms / 1000);
|
||||
if (sec < 1) return null;
|
||||
if (sec < 60) return `${sec}s`;
|
||||
if (sec < 3600) return `${Math.floor(sec / 60)}m ${sec % 60}s`;
|
||||
return `${Math.floor(sec / 3600)}h ${Math.floor((sec % 3600) / 60)}m`;
|
||||
};
|
||||
|
||||
const preProvMs = calcMs(preProvEvts);
|
||||
const userEnrollMs = calcMs(userEnrollEvts);
|
||||
|
||||
return {
|
||||
preProvDuration: fmt(preProvMs),
|
||||
userEnrollDuration: fmt(userEnrollMs),
|
||||
combinedDuration: fmt(preProvMs + userEnrollMs),
|
||||
};
|
||||
}, [events, isWhiteGloveSession, whiteGloveSplitSequence]);
|
||||
return computeWhiteGloveDurations(events, whiteGloveSplitSequence, session?.startedAt);
|
||||
}, [events, isWhiteGloveSession, whiteGloveSplitSequence, session?.startedAt]);
|
||||
|
||||
// Group events by phase — single timeline for normal sessions, two groups for WhiteGlove
|
||||
const { eventsByPhase, orderedPhases } = useMemo(() => {
|
||||
|
||||
@@ -66,6 +66,69 @@ export function computeWhiteGloveSplitSequence(events: EnrollmentEvent[]): numbe
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Pure helper — per-block durations for WhiteGlove sessions.
|
||||
// Duration 1 = pre-provisioning, Duration 2 = user enrollment, combined = D1 + D2
|
||||
// (the sealed pause between reseal and resume is excluded).
|
||||
//
|
||||
// Block spans are anchored on the SEQUENCE-canonical edge events (first/last event by
|
||||
// sequence), NOT on min/max over all timestamps in the block. Two real-world cases break
|
||||
// the min/max approach:
|
||||
// - Events re-emitted after the Part-2 resume can legitimately carry historical event
|
||||
// times (e.g. an app_install_started the ImeLogTracker re-parses from the IME log,
|
||||
// stamped with the Part-1 install start). min() would stretch the User Enrollment
|
||||
// block back across the sealed pause.
|
||||
// - A single clock-skewed source timestamp (e.g. an IME log line with a wrong timezone
|
||||
// bias that slips under the agent's 1h future-skew clamp) would stretch the block
|
||||
// the other way.
|
||||
// Sequence is the canonical event order, so the block edges by sequence are the robust
|
||||
// span anchors. Part 1 additionally starts at session.startedAt (registration) when
|
||||
// provided — the same anchor the backend uses for DurationSeconds.
|
||||
export interface WhiteGloveDurations {
|
||||
preProvDuration: string | null;
|
||||
userEnrollDuration: string | null;
|
||||
combinedDuration: string | null;
|
||||
}
|
||||
|
||||
export function computeWhiteGloveDurations(
|
||||
events: EnrollmentEvent[],
|
||||
splitSequence: number,
|
||||
startedAt?: string,
|
||||
): WhiteGloveDurations {
|
||||
const preProvEvts = splitSequence < 0 ? events : events.filter(e => e.sequence <= splitSequence);
|
||||
const userEnrollEvts = splitSequence < 0 ? [] : events.filter(e => e.sequence > splitSequence);
|
||||
|
||||
const spanMs = (evts: EnrollmentEvent[], startOverrideMs?: number): number => {
|
||||
if (evts.length === 0) return 0;
|
||||
let first = evts[0];
|
||||
let last = evts[0];
|
||||
for (const e of evts) {
|
||||
if (e.sequence < first.sequence) first = e;
|
||||
if (e.sequence > last.sequence) last = e;
|
||||
}
|
||||
const start = startOverrideMs ?? new Date(first.timestamp).getTime();
|
||||
const end = new Date(last.timestamp).getTime();
|
||||
return Math.max(0, end - start);
|
||||
};
|
||||
|
||||
const fmt = (ms: number): string | null => {
|
||||
const sec = Math.round(ms / 1000);
|
||||
if (sec < 1) return null;
|
||||
if (sec < 60) return `${sec}s`;
|
||||
if (sec < 3600) return `${Math.floor(sec / 60)}m ${sec % 60}s`;
|
||||
return `${Math.floor(sec / 3600)}h ${Math.floor((sec % 3600) / 60)}m`;
|
||||
};
|
||||
|
||||
const startedAtMs = startedAt ? new Date(startedAt).getTime() : NaN;
|
||||
const preProvMs = spanMs(preProvEvts, Number.isFinite(startedAtMs) ? startedAtMs : undefined);
|
||||
const userEnrollMs = spanMs(userEnrollEvts);
|
||||
|
||||
return {
|
||||
preProvDuration: fmt(preProvMs),
|
||||
userEnrollDuration: fmt(userEnrollMs),
|
||||
combinedDuration: fmt(preProvMs + userEnrollMs),
|
||||
};
|
||||
}
|
||||
|
||||
// Pure helper — groups a flat event list into phase buckets.
|
||||
// Extracted from the useMemo so it can be called multiple times for WhiteGlove split timelines.
|
||||
//
|
||||
|
||||
@@ -64,9 +64,13 @@ export default function AgentAnalyzersSection({
|
||||
trimmed !== "" &&
|
||||
(localAdminAllowedAccounts.some((a) => a.toLowerCase() === trimmed.toLowerCase()) ||
|
||||
BUILTIN_ACCOUNTS.some((a) => a.toLowerCase() === trimmed.toLowerCase()));
|
||||
// Wildcard patterns (* / ?) need at least 2 literal characters — a bare "*"
|
||||
// would silently disable the analyzer for the whole tenant.
|
||||
const isTooBroad =
|
||||
/[*?]/.test(trimmed) && trimmed.replace(/[*?]/g, "").length < 2;
|
||||
|
||||
const addAccount = () => {
|
||||
if (!trimmed || isDuplicate) return;
|
||||
if (!trimmed || isDuplicate || isTooBroad) return;
|
||||
setLocalAdminAllowedAccounts([...localAdminAllowedAccounts, trimmed]);
|
||||
setNewAllowedAccount("");
|
||||
};
|
||||
@@ -114,6 +118,10 @@ export default function AgentAnalyzersSection({
|
||||
<p className="font-medium text-gray-900 mb-1">Allowed Local Accounts</p>
|
||||
<p className="text-sm text-gray-500 mb-3">
|
||||
Accounts listed here are considered expected on enrolled devices and will not trigger alerts. These are merged with the built-in defaults on the agent.
|
||||
Wildcards are supported: <code className="px-1 bg-gray-100 rounded">*</code> matches any sequence of characters,{" "}
|
||||
<code className="px-1 bg-gray-100 rounded">?</code> exactly one — e.g.{" "}
|
||||
<code className="px-1 bg-gray-100 rounded">adm-*</code> allows generated accounts like <code className="px-1 bg-gray-100 rounded">adm-83921</code>.
|
||||
Matching is case-insensitive and applies to both account names and profile folders.
|
||||
</p>
|
||||
|
||||
{/* Built-in accounts (read-only) */}
|
||||
@@ -155,7 +163,7 @@ export default function AgentAnalyzersSection({
|
||||
<div className="flex gap-2 mt-2">
|
||||
<input
|
||||
type="text"
|
||||
placeholder="Account name (e.g. SupportAdmin)"
|
||||
placeholder="Account name or pattern (e.g. SupportAdmin, adm-*)"
|
||||
value={newAllowedAccount}
|
||||
onChange={(e) => setNewAllowedAccount(e.target.value)}
|
||||
onKeyDown={(e) => { if (e.key === "Enter") { e.preventDefault(); addAccount(); } }}
|
||||
@@ -163,7 +171,7 @@ export default function AgentAnalyzersSection({
|
||||
/>
|
||||
<button
|
||||
onClick={addAccount}
|
||||
disabled={!trimmed || isDuplicate}
|
||||
disabled={!trimmed || isDuplicate || isTooBroad}
|
||||
className="px-4 py-1.5 bg-rose-600 text-white rounded-lg text-sm font-medium hover:bg-rose-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors whitespace-nowrap"
|
||||
>
|
||||
Add
|
||||
@@ -172,6 +180,11 @@ export default function AgentAnalyzersSection({
|
||||
{isDuplicate && (
|
||||
<p className="text-xs text-red-500 mt-1">This account is already in the list.</p>
|
||||
)}
|
||||
{isTooBroad && (
|
||||
<p className="text-xs text-red-500 mt-1">
|
||||
Pattern is too broad — it needs at least 2 characters besides <code>*</code>/<code>?</code>, otherwise the analyzer would be effectively disabled.
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* ── Divider ─────────────────────────────────────── */}
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { computeWhiteGloveDurations } from "../../app/sessions/[sessionId]/utils/eventHelpers";
|
||||
import type { EnrollmentEvent } from "@/types";
|
||||
|
||||
// Fixture builder — sequence and timestamp are the only fields that matter here.
|
||||
function ev(seq: number, eventType: string, isoTs: string): EnrollmentEvent {
|
||||
return {
|
||||
eventId: `e${seq}`,
|
||||
sessionId: "s1",
|
||||
timestamp: isoTs,
|
||||
eventType,
|
||||
severity: "Info",
|
||||
source: "Agent",
|
||||
phase: 0,
|
||||
message: eventType,
|
||||
sequence: seq,
|
||||
};
|
||||
}
|
||||
|
||||
describe("computeWhiteGloveDurations", () => {
|
||||
it("computes both blocks from the sequence-canonical edge events", () => {
|
||||
const events = [
|
||||
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
|
||||
ev(2, "esp_phase_changed", "2026-07-23T08:35:00Z"),
|
||||
ev(3, "agent_shutting_down", "2026-07-23T09:28:00Z"), // Part 1 ends (reseal)
|
||||
ev(4, "agent_started", "2026-07-23T11:00:00Z"), // Part 2 boot after sealed pause
|
||||
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
|
||||
];
|
||||
const d = computeWhiteGloveDurations(events, 3);
|
||||
expect(d.preProvDuration).toBe("57m 0s");
|
||||
expect(d.userEnrollDuration).toBe("56m 0s");
|
||||
expect(d.combinedDuration).toBe("1h 53m"); // pause 09:28→11:00 excluded
|
||||
});
|
||||
|
||||
it("ignores a historical event time re-emitted into the user block (post-resume IME re-parse)", () => {
|
||||
// Real case (session fcb44595): after the Part-2 resume the ImeLogTracker re-emits an
|
||||
// app_install_started carrying the ORIGINAL Part-1 install time. min/max over timestamps
|
||||
// stretched the user block back across the sealed pause (56m became 3h18m).
|
||||
const events = [
|
||||
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
|
||||
ev(2, "agent_shutting_down", "2026-07-23T09:28:00Z"),
|
||||
ev(3, "agent_started", "2026-07-23T11:00:00Z"),
|
||||
ev(4, "app_install_started", "2026-07-23T08:37:57Z"), // historical event time, Part-2 sequence
|
||||
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
|
||||
];
|
||||
const d = computeWhiteGloveDurations(events, 2);
|
||||
expect(d.userEnrollDuration).toBe("56m 0s");
|
||||
expect(d.combinedDuration).toBe("1h 53m");
|
||||
});
|
||||
|
||||
it("ignores a clock-skewed outlier timestamp inside the pre-provisioning block", () => {
|
||||
// Real case (session fcb44595): a do_telemetry event carried a +1h source timestamp
|
||||
// (IME log line with a wrong timezone bias, just under the agent's future-skew clamp).
|
||||
// min/max stretched Part 1 from 57m to 1h44m.
|
||||
const events = [
|
||||
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
|
||||
ev(2, "do_telemetry", "2026-07-23T10:13:42Z"), // +1h skewed, mid-block by sequence
|
||||
ev(3, "agent_shutting_down", "2026-07-23T09:28:00Z"),
|
||||
ev(4, "agent_started", "2026-07-23T11:00:00Z"),
|
||||
ev(5, "enrollment_complete", "2026-07-23T11:56:00Z"),
|
||||
];
|
||||
const d = computeWhiteGloveDurations(events, 3);
|
||||
expect(d.preProvDuration).toBe("57m 0s");
|
||||
});
|
||||
|
||||
it("anchors Part 1 at session.startedAt when provided (backend DurationSeconds parity)", () => {
|
||||
const events = [
|
||||
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
|
||||
ev(2, "agent_shutting_down", "2026-07-23T09:28:00Z"),
|
||||
ev(3, "agent_started", "2026-07-23T11:00:00Z"),
|
||||
ev(4, "enrollment_complete", "2026-07-23T11:56:00Z"),
|
||||
];
|
||||
const d = computeWhiteGloveDurations(events, 2, "2026-07-23T08:29:41Z");
|
||||
expect(d.preProvDuration).toBe("58m 19s"); // 08:29:41 → 09:28:00
|
||||
});
|
||||
|
||||
it("treats everything as Part 1 while no split point exists yet", () => {
|
||||
const events = [
|
||||
ev(1, "agent_started", "2026-07-23T08:31:00Z"),
|
||||
ev(2, "esp_phase_changed", "2026-07-23T08:45:00Z"),
|
||||
];
|
||||
const d = computeWhiteGloveDurations(events, -1);
|
||||
expect(d.preProvDuration).toBe("14m 0s");
|
||||
expect(d.userEnrollDuration).toBeNull();
|
||||
expect(d.combinedDuration).toBe("14m 0s");
|
||||
});
|
||||
|
||||
it("returns nulls for an empty event list", () => {
|
||||
const d = computeWhiteGloveDurations([], -1);
|
||||
expect(d.preProvDuration).toBeNull();
|
||||
expect(d.userEnrollDuration).toBeNull();
|
||||
expect(d.combinedDuration).toBeNull();
|
||||
});
|
||||
|
||||
it("never returns a negative span when edge timestamps are disordered", () => {
|
||||
const events = [
|
||||
ev(1, "agent_started", "2026-07-23T09:00:00Z"),
|
||||
ev(2, "agent_shutting_down", "2026-07-23T08:59:00Z"), // straggler stamped before the boot
|
||||
];
|
||||
const d = computeWhiteGloveDurations(events, 2, "2026-07-23T09:30:00Z");
|
||||
expect(d.preProvDuration).toBeNull(); // clamped to 0 → formatted as null
|
||||
});
|
||||
});
|
||||
@@ -57,6 +57,7 @@ export const REGISTRY_PREFIX_CATEGORIES: readonly GuardrailCategory[] = [
|
||||
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
|
||||
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
|
||||
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
|
||||
"SOFTWARE\\Microsoft\\Windows\\Autopilot",
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -242,6 +243,7 @@ export const ALLOWED_REGISTRY_PREFIXES: readonly string[] = [
|
||||
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
|
||||
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
|
||||
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
|
||||
"SOFTWARE\\Microsoft\\Windows\\Autopilot",
|
||||
"SYSTEM\\CurrentControlSet\\Services\\TPM",
|
||||
"SOFTWARE\\Microsoft\\Tpm",
|
||||
"SYSTEM\\CurrentControlSet\\Control\\SecureBoot",
|
||||
|
||||
Reference in New Issue
Block a user